JP2019036239A - Information processing method, information processing program, information processing system, and information processing device - Google Patents

Abstract

To provide an information processing method capable of further improving user experience in a virtual space.SOLUTION: An information processing method is executed by a computer in an information processing system comprising a user terminal including a HMD 110 worn on a head of a user and a server connected to enable communication with the user terminal. The information processing method includes the steps of: (a) generating visual field image data indicating the visual field image displayed in the HMD 110 on the basis of the virtual space data for defining the virtual space and the movement of the HMD 110; (b) identifying the feeling of the user; and (c) updating an evaluation value of the content associated with the virtual space on the basis of the identified feeling of the user.SELECTED DRAWING: Figure 13

Description

  The present disclosure relates to an information processing method, an information processing program, an information processing system, and an information processing apparatus.

  It is known to share a virtual space via a network among a plurality of users wearing a head mounted device (hereinafter simply referred to as HMD). For example, Non-Patent Document 1 discloses a technique for controlling the operations of a plurality of avatars associated with a plurality of users in accordance with the operations of the plurality of users after sharing one VR space among the plurality of users. . In particular, Non-Patent Document 1 discloses a technology that promotes communication among a plurality of users in a virtual space by expressing the expression of each avatar richly.

“Facebook Social VR Demo-Oculus Connect 2016”, [online], October 6, 2016, Road to VR, [Search April 21, 2017], Internet <https://www.youtube.com/ watch? v = YuIgyKLPt3s>

  By the way, Non-Patent Document 1 discloses that the user's facial expression is reflected in the facial expression of the avatar associated with the user, thereby promoting communication between the users in the virtual space. In addition, the use of the identified user's emotion and communication behavior for the user experience in the virtual space after specifying the user's communication behavior is not studied. In particular, there is room for further improving the user experience in the virtual space by evaluating the contents of the virtual space based on the user's emotions and communication behavior.

  An object of this indication is to provide the information processing method which can further improve the user experience in virtual space. Furthermore, this indication aims at providing the information processing program, information processing system, and information processing apparatus which can further improve the user experience in virtual space.

An information processing method executed by a computer in an information processing system including a user terminal having a head-mounted device mounted on a user's head and a server connected to the user terminal so as to be communicable,
(A) generating visual field image data indicating a visual field image displayed on the head mounted device based on virtual space data defining a virtual space and movement of the head mounted device;
(B) identifying the user's emotion;
(C) updating the evaluation value of the content associated with the virtual space based on the identified user's emotion;
Including an information processing method.

  According to the present disclosure, the user experience in the virtual space can be further improved.

It is the schematic which shows the virtual space delivery system which concerns on embodiment (henceforth this embodiment only) of this invention. It is the schematic which shows a user terminal. It is a figure which shows the head of the user with which HMD was mounted | worn. It is a figure which shows the hardware constitutions of a control apparatus. It is a figure which shows an example of the specific structure of an external controller. It is a flowchart which shows the process which displays a visual field image on HMD. It is xyz space figure which shows an example of virtual space. The state (a) is a yx plan view of the virtual space shown in FIG. The state (b) is a zx plan view of the virtual space shown in FIG. It is a figure which shows an example of the visual field image displayed on HMD. It is a figure which shows the hardware constitutions of the server shown in FIG. The state (a) is a diagram showing a virtual space provided to the user A. The state (b) is a diagram showing a virtual space provided to the user B. It is a sequence figure for explaining an example of processing which synchronizes movement of each avatar between user terminals. It is a figure which shows an example of the virtual space provided to the user A for demonstrating the information processing method which concerns on 1st Embodiment. It is a sequence diagram for demonstrating an example of the information processing method which concerns on 1st Embodiment. It is a figure which shows an example of the content evaluation table in 1st Embodiment. It is a figure which shows a mode that a part of virtual space provided to the user A is updated when the number of the laughing users is more than a predetermined threshold value. It is a sequence diagram for demonstrating an example of the information processing method which concerns on 2nd Embodiment. It is a figure which shows an example of the content evaluation table in 2nd Embodiment. It is an example of the virtual space provided to the user D for demonstrating the information processing method which concerns on 3rd Embodiment. It is a sequence diagram for demonstrating the information processing method which concerns on 3rd Embodiment. It is a figure which shows an example of the visual field image displayed on HMD which the user D wears.

[Description of Embodiments Presented by the Present Disclosure]
An overview of an embodiment indicated by the present disclosure will be described.
(1) An information processing method executed by a computer in an information processing system including a user terminal having a head mounted device mounted on a user's head and a server connected to the user terminal so as to be communicable,
(A) generating visual field image data indicating a visual field image displayed on the head mounted device based on virtual space data defining a virtual space and movement of the head mounted device;
(B) identifying the user's emotion;
(C) updating the evaluation value of the content associated with the virtual space based on the identified user's emotion;
Including an information processing method.

  According to the above method, a real reaction to the user's content can be reflected in the content evaluation. Therefore, the user experience in the virtual space is improved.

(2) The step (b)
Identifying the user's emotion based on at least one of the facial expression of the user, voice output from the user, and movement of the user's hand,
The information processing method according to item (1).

  According to the above method, the user's emotion can be accurately specified by using at least one of the facial expression of the user, the voice output from the user, and the movement of the user's hand.

(3) The step (b)
Determining whether the user's emotion is a first emotion,
The step (c)
Updating the evaluation value of the content when the identified user's emotion is the first emotion,
The information processing method according to item (1) or (2).

  According to the above method, the user's first emotion can be reflected in the content evaluation.

(4) The virtual space is viewed by a plurality of users,
The evaluation value of the content corresponds to the number of users having the first emotion among the plurality of users.
The information processing method according to item (3).

  According to the above method, since the evaluation value of the content corresponds to the number of users having a first emotion (for example, laughing emotion), the content can be evaluated based on the number of users having the first emotion. it can.

  (5) The evaluation value of the content corresponds to an item (4) corresponding to the number of users having the first emotion and having a line of sight toward the first area of the virtual space among the plurality of users. The information processing method described in).

  According to the above method, content is evaluated based on the number of users who have a first emotion and have their eyes directed to the same region in the virtual space (for example, a region where an avatar performing) exists. Can do. In particular, since the number of users who look at the same area affects the content evaluation, it is possible to evaluate the content more accurately.

(6) The step (b)
Identifying a level of the user's first emotion,
The step (c)
Updating an evaluation value of the content based on the identified first emotion level;
The information processing method according to item (1) or (2).

  According to the above method, since the level of the first emotion of the user can be directly reflected in the content evaluation, the content can be evaluated in more detail.

(7) The virtual space is viewed by a plurality of users,
The information evaluation method according to item (6), wherein the evaluation value of the content corresponds to a cumulative value of the first emotion level of each user.

  According to the above method, the content can be evaluated based on the accumulated value of the first emotion level of each user, so that the content can be evaluated in more detail.

  (8) (d) The information processing according to any one of items (1) to (7), further including a step of updating at least a part of the virtual space data based on the evaluation value of the content. Method.

  According to the above method, for example, a scene in the virtual space is changed based on the content evaluation value, or a predetermined effect is executed in the virtual space by arranging a predetermined object in the virtual space. Thus, since at least a part of the virtual space data is updated based on the evaluation value of the content, it is possible to improve the entertainment property of the virtual space.

(9) The step (d)
The information processing method according to item (8), including a step of updating at least a part of the virtual space data when the evaluation value of the content reaches a first value.

  According to the above method, the user can recognize that the content evaluation value has reached the first value by looking at the updated virtual space. Furthermore, since at least a part of the virtual space data is updated when the evaluation value of the content is equal to or higher than the first value, the entertainment property of the virtual space can be improved.

  (10) (e) In response to a transmission request from the user terminal, the method further includes a step of transmitting data related to the content stored in the server to the user terminal. The information processing method as described in any one of them.

  According to the above method, when the user views the content distributed from the server, the content evaluation value is updated based on the emotion of the user viewing the content. Accordingly, since the content can be evaluated based on the user's emotion, the user's real feeling (reaction) can be reflected in the content evaluation, and it is possible to save time and effort for the user for the content evaluation. Become. In particular, it is not necessary for the user to evaluate the content by pressing an evaluation button (such as a “like” button) provided on the moving image distribution platform, so that it is possible to save the user operation for content evaluation.

(11) The step (e)
Transmitting data related to the content stored in the server and information about the evaluation value of the content to the user terminal,
(F) The information processing method according to item (10), further including a step of presenting information related to the evaluation value of the content to the user.

  According to the above method, the information related to the evaluation value of the content is presented to the user, so that the user can know the evaluation of the other user with respect to the content to be viewed.

(12) An information processing method executed by a computer in an information processing system including a user terminal having a head-mounted device mounted on a user's head and a server connected to the user terminal in a communicable manner,
(A) generating visual field image data indicating a visual field image displayed on the head mounted device based on virtual space data defining a virtual space and movement of the head mounted device;
(B) identifying the user's communication behavior;
(C) updating an evaluation value of content associated with the virtual space based on the identified communication behavior;
Including an information processing method.

  According to the above method, a real reaction to the user's content can be reflected in the content evaluation. Therefore, the user experience in the virtual space is improved.

  (13) An information processing program for causing a computer to execute the information processing method according to any one of items (1) to (12).

  According to the above, it is possible to provide an information processing program that can further improve the user experience in the virtual space.

(14) a user terminal having a head-mounted device mounted on the user's head;
A server communicably connected to the user terminal;
An information processing system configured to execute the information processing method according to any one of items (1) to (12).

  According to the above, it is possible to provide an information processing system that can further improve the user experience in the virtual space.

(15) a processor;
An information processing apparatus comprising a memory for storing computer-readable instructions,
When the computer-readable instruction is executed by the processor, the information processing apparatus executes the information processing method according to any one of items (1) to (12).

  According to the above, it is possible to provide an information processing apparatus that can further improve the user experience in the virtual space.

[Details of Embodiments Presented by the Present Disclosure]
Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In addition, about the element which has the same reference number as the element already demonstrated in description of this embodiment, the description is not repeated for convenience of explanation.

  First, an outline of the configuration of the virtual space distribution system 100 (an example of an information processing system) will be described with reference to FIG. FIG. 1 is a schematic diagram of a virtual space distribution system 100 (hereinafter simply referred to as distribution system 100). As shown in FIG. 1, the distribution system 100 includes a user terminal 1A operated by a user A wearing a head mounted device (HMD) 110, a user terminal 1B operated by a user B wearing the HMD 110, and a server 2 With. User terminals 1A and 1B are communicably connected to a server 2 via a communication network 3 such as the Internet. In the present embodiment, the virtual space includes a VR (Virtual Reality) space, an AR (Arranged Reality) space, and an MR (Mixed Reality) space. Further, hereinafter, for convenience of explanation, the user terminals 1A and 1B may be simply referred to as the user terminal 1 in some cases. Further, the users A and B may be simply referred to as a user U. In the present embodiment, the user terminals 1A and 1B are assumed to have the same configuration.

  Next, the configuration of the user terminal 1 will be described with reference to FIG. FIG. 2 is a schematic diagram showing the user terminal 1. As shown in FIG. 2, the user terminal 1 includes a head mounted device (HMD) 110, a headphone 116, a face camera 117, a microphone 118, a position sensor 130, and an external controller that are mounted on the user U's head. 320 and a control device 120.

  The HMD 110 includes a display unit 112, an HMD sensor 114, a gaze sensor 140, a face camera 113, and a mounting sensor 115. The display unit 112 includes a non-transmissive display device configured to completely cover the field of view (field of view) of the user U wearing the HMD 110. Thereby, the user U can immerse in the virtual space by viewing only the visual field image displayed on the display unit 112. The display unit 112 includes a left-eye display unit configured to provide an image to the user U's left eye and a right-eye display unit configured to provide an image to the user U's right eye. Also good. Further, the HMD 110 may include a transmissive display device. In this case, the transmissive display device may be temporarily configured as a non-transmissive display device by adjusting the transmittance.

  The HMD sensor 114 is mounted in the vicinity of the display unit 112 of the HMD 110. The HMD sensor 114 includes at least one of a geomagnetic sensor, an acceleration sensor, and a tilt sensor (such as an angular velocity sensor and a gyro sensor), and detects various movements (tilt and the like) of the HMD 110 mounted on the user U's head. be able to.

  The gaze sensor 140 has an eye tracking function that detects the line of sight of the user U. The gaze sensor 140 may include, for example, a right eye gaze sensor and a left eye gaze sensor. The right eye gaze sensor irradiates, for example, infrared light to the right eye of the user U, and detects reflected light reflected from the right eye (particularly the cornea and iris), thereby acquiring information related to the rotation angle of the right eye's eyeball. May be. On the other hand, the left eye gaze sensor irradiates the left eye of the user U with, for example, infrared light, and detects reflected light reflected from the left eye (particularly the cornea and iris), thereby providing information on the rotation angle of the left eye's eyeball. May be obtained.

  The face camera 113 is configured to acquire an image (particularly, a moving image) in which the eyes (left eye and right eye) and eyebrows (left eyebrow and right eyebrow) of the user U are displayed with the HMD 110 mounted on the user U. Has been. The face camera 113 is arranged at a predetermined position inside the HMD 110 so that the user's U eyes and eyebrows can be imaged. The frame rate of the moving image acquired by the face camera 113 may be larger than the frame rate of the visual field image (described later) displayed on the HMD 110.

  The wearing sensor 115 is configured to generate event information for indicating whether or not the user U is wearing the HMD 110. Specifically, when the user U removes the HMD 110 (when the state of the HMD 110 transitions from the wearing state to the non-wearing state), the wearing sensor 115 generates event information and then controls the event information. To device 120.

  Headphones 116 (audio output units) are attached to the left and right ears of the user U, respectively. The headphones 116 are configured to receive audio data (electrical signals) from the control device 120 and output audio based on the received audio data. The microphone 118 (voice input unit) is configured to collect voice uttered by the user U and generate voice data (electrical signal) based on the collected voice. Furthermore, the microphone 118 is configured to transmit audio data to the control device 120.

  The face camera 117 is configured to acquire an image (particularly, a moving image) in which the mouth of the user U and its periphery are displayed. The face camera 117 may be arranged at a position facing the user U's mouth so that the user U's mouth and its periphery can be imaged. The face camera 117 may be connected to the HMD 110. The frame rate of the moving image acquired by the face camera 117 may be larger than the frame rate of the visual field image displayed on the HMD 110.

  The position sensor 130 is configured by, for example, a position tracking camera, and is configured to detect the positions of the HMD 110 and the external controller 320. The position sensor 130 is communicably connected to the control device 120 by wireless or wired communication, and is configured to detect information on the position, inclination, or light emission intensity of a plurality of detection points (not shown) provided in the HMD 110. . Further, the position sensor 130 is configured to detect information on the position, inclination and / or light emission intensity of a plurality of detection points (not shown) provided in the external controller 320. The detection point is, for example, a light emitting unit that emits infrared light or visible light. The position sensor 130 may include an infrared sensor and a plurality of optical cameras.

  The external controller 320 detects the movement of a part of the body of the user U (a part other than the head, which is the user U's hand in the present embodiment), and thereby the avatar hand displayed in the virtual space. Used to control the operation. The external controller 320 is an external controller 320R for right hand operated by the right hand of the user U (hereinafter simply referred to as controller 320R) and an external controller 320L for left hand operated by the left hand of the user U (hereinafter simply referred to as controller 320L). And). The controller 320R is a device that indicates the position of the right hand of the user U and the movement of the finger of the right hand. Further, the right hand of the avatar that exists in the virtual space moves according to the movement of the controller 320R. The controller 320L is a device that indicates the position of the left hand of the user U and the movement of the finger of the left hand. Further, the left hand of the avatar that exists in the virtual space moves according to the movement of the controller 320L.

  The control device 120 is a computer configured to control the HMD 110. The control device 120 specifies the position information of the HMD 110 based on the information acquired from the position sensor 130, and based on the specified position information, the position of the virtual camera (avatar position) in the virtual space, and the reality. The position of the user U wearing the HMD 110 in the space can be accurately associated. Further, the control device 120 identifies the operation of the external controller 320 based on information acquired from the position sensor 130 and / or a sensor built in the external controller 320, and based on the identified operation of the external controller 320. Thus, the motion of the avatar's hand displayed in the virtual space can be accurately associated with the motion of the external controller 320 in the real space. In particular, the control device 120 identifies the operation of the controller 320L based on the information acquired from the position sensor 130 and / or the sensor built in the controller 320L, and determines the virtual operation based on the identified operation of the controller 320L. The motion of the left hand of the avatar displayed in the space and the motion of the controller 320L in the real space (the motion of the left hand of the user U) can be accurately associated. Similarly, the control device 120 specifies the operation of the controller 320R based on the information acquired from the position sensor and / or the sensor built in the controller 320R, and based on the operation of the specified controller 320R, the virtual space The motion of the right hand of the avatar displayed inside the motion of the controller 320R in the real space (the motion of the right hand of the user U) can be accurately associated.

  Further, the control unit 121 can associate the facial expression (state) of the user U with the facial expression (state) of the avatar of the user U based on the moving images acquired by the face cameras 113 and 117. Details of the process of associating the facial expression of the user U with the facial expression of the avatar will be described later.

  Further, the control device 120 specifies the gaze of the right eye and the gaze of the left eye of the user U based on the information transmitted from the gaze sensor 140 (the left eye gaze sensor and the right eye gaze sensor), and the right eye gaze. It is possible to specify the point of gaze that is the intersection of the left eye gaze. Furthermore, the control device 120 can specify the line of sight of both eyes of the user U (the line of sight of the user U) based on the specified gazing point. Here, the line of sight of the user U is the line of sight of both eyes of the user U, and coincides with the direction of a straight line passing through the middle point of the line connecting the right eye and the left eye of the user U and the gazing point.

  Next, with reference to FIG. 3, a method for acquiring information related to the position and inclination of the HMD 110 will be described. FIG. 3 is a diagram illustrating the head of the user U wearing the HMD 110. Information on the position and inclination of the HMD 110 that is linked to the movement of the head of the user U wearing the HMD 110 can be detected by the position sensor 130 and / or the HMD sensor 114 mounted on the HMD 110. As shown in FIG. 2, three-dimensional coordinates (uvw coordinates) are defined centering on the head of the user U wearing the HMD 110. The vertical direction in which the user U stands up is defined as the v-axis, the direction orthogonal to the v-axis and passing through the center of the HMD 110 is defined as the w-axis, and the direction orthogonal to the v-axis and the w-axis is defined as the u-axis. The position sensor 130 and / or the HMD sensor 114 is an angle around each uvw axis (that is, a yaw angle indicating rotation around the v axis, a pitch angle indicating rotation around the u axis, and a center around the w axis). The inclination determined by the roll angle indicating rotation) is detected. The control device 120 determines angle information for controlling the visual axis of the virtual camera based on the detected angle change around each uvw axis.

  Next, the hardware configuration of the control device 120 will be described with reference to FIG. FIG. 4 is a diagram illustrating a hardware configuration of the control device 120. As illustrated in FIG. 4, the control device 120 includes a control unit 121, a storage unit 123, an I / O (input / output) interface 124, a communication interface 125, and a bus 126. The control unit 121, the storage unit 123, the I / O interface 124, and the communication interface 125 are connected to each other via a bus 126 so as to communicate with each other.

  The control device 120 may be configured as a personal computer, a smartphone, a fablet, a tablet, or a wearable device separately from the HMD 110, or may be built in the HMD 110. In addition, some functions of the control device 120 may be mounted on the HMD 110, and the remaining functions of the control device 120 may be mounted on another device separate from the HMD 110.

  The control unit 121 includes a memory and a processor. The memory includes, for example, a ROM (Read Only Memory) in which various programs are stored, a RAM (Random Access Memory) having a plurality of work areas in which various programs executed by the processor are stored, and the like. The processor is, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), and / or a GPU (Graphics Processing Unit), and a program specified from various programs embedded in the ROM is expanded on the RAM. It is comprised so that various processes may be performed in cooperation with.

  In particular, the control unit 121 may control various operations of the control device 120 by a processor developing a control program on a RAM and executing the control program in cooperation with the RAM. The control unit 121 displays a field image on the display unit 112 of the HMD 110 based on the field image data. Thereby, the user U can be immersed in the virtual space.

  The storage unit (storage) 123 is a storage device such as an HDD (Hard Disk Drive), an SSD (Solid State Drive), or a USB flash memory, and is configured to store programs and various data. The storage unit 123 may store a control program for causing a computer to execute at least a part of the information processing method according to the present embodiment, or a control program for realizing sharing of a virtual space by a plurality of users. The storage unit 123 may store data related to the user U authentication program, various images, and objects (for example, an avatar). Furthermore, a database including tables for managing various data may be constructed in the storage unit 123.

  The I / O interface 124 is configured to communicably connect the HMD 110, the position sensor 130, the external controller 320, the headphones 116, the face camera 117, and the microphone 118 to the control device 120, respectively. For example, it is configured by a USB (Universal Serial Bus) terminal, a DVI (Digital Visual Interface) terminal, an HDMI (registered trademark) (High-Definition Multimedia Interface) terminal, and the like. The control device 120 may be wirelessly connected to each of the HMD 110, the position sensor 130, the external controller 320, the headphones 116, the face camera 117, and the microphone 118.

  The communication interface 125 is configured to connect the control device 120 to a communication network 3 such as a LAN (Local Area Network), a WAN (Wide Area Network), or the Internet. The communication interface 125 includes various wired connection terminals for communicating with an external device such as the server 2 via the communication network 3 and various processing circuits for wireless connection, and communicates via the communication network 3. It is configured to conform to the communication standard.

  Next, an example of a specific configuration of the external controller 320 will be described with reference to FIG. Since the controller 320R and the controller 320L have substantially the same configuration, only the specific configuration of the controller 320R will be described below with reference to FIG. In the following description, the controllers 320L and 320R may be simply referred to as the external controller 320 for convenience.

  As shown in FIG. 5, the controller 320R includes an operation button 302, a plurality of detection points 304, a sensor (not shown), and a transceiver (not shown). Only one of the detection point 304 and the sensor may be provided. The operation button 302 includes a plurality of button groups configured to accept an operation input from the user U. The operation button 302 includes a push button, a trigger button, and an analog stick. The push-type button is a button operated by an operation of pressing with a thumb. For example, two push buttons 302 a and 302 b are provided on the top surface 322. The trigger type button is a button operated by an operation of pulling a trigger with an index finger or a middle finger. For example, a trigger button 302 e is provided on the front surface portion of the grip 324, and a trigger button 302 f is provided on the side surface portion of the grip 324. The trigger type buttons 302e and 302f are operated by the index finger and the middle finger, respectively. The analog stick is a stick-type button that can be operated by being tilted 360 degrees from a predetermined neutral position in an arbitrary direction. For example, an analog stick 320i is provided on the top surface 322, and is operated using a thumb.

  The controller 320R includes a frame 326 that extends from both side surfaces of the grip 324 in a direction opposite to the top surface 322 to form a semicircular ring. A plurality of detection points 304 are embedded on the outer surface of the frame 326. The plurality of detection points 304 are, for example, a plurality of infrared LEDs arranged in a line along the circumferential direction of the frame 326. After the position sensor 130 detects information on the position, inclination, or light emission intensity of the plurality of detection points 304, the control device 120 detects the position or orientation (inclination / posture) of the controller 320R based on the information detected by the position sensor 130. Get information about (orientation).

  The sensor of the controller 320R may be, for example, a magnetic sensor, an angular velocity sensor, an acceleration sensor, or a combination thereof. When the user U moves the controller 320R, the sensor outputs a signal (for example, a signal indicating information related to magnetism, angular velocity, or acceleration) according to the orientation and position of the controller 320R. The control device 120 acquires information related to the position and orientation of the controller 320R based on the signal output from the sensor.

  The transceiver of the controller 320R is configured to transmit and receive data between the controller 320R and the control device 120. For example, the transceiver may transmit an operation signal corresponding to the operation input of the user U to the control device 120. The transceiver may receive an instruction signal for instructing the controller 320 </ b> R to emit light at the detection point 304 from the control device 120. Further, the transceiver may send a signal to the controller 120 indicating the value detected by the sensor.

  Next, processing for displaying a field-of-view image on the HMD 110 will be described with reference to FIGS. 6 to 9. FIG. 6 is a flowchart showing a process for displaying the visual field image on the HMD 110. FIG. 7 is an xyz space diagram showing an example of the virtual space 200. The state (a) in FIG. 8 is a yx plan view of the virtual space 200 shown in FIG. The state (b) of FIG. 8 is a zx plan view of the virtual space 200 shown in FIG. FIG. 9 is a diagram illustrating an example of the visual field image V displayed on the HMD 110.

  As shown in FIG. 6, in step S1, the control unit 121 (see FIG. 4) generates virtual space data indicating the virtual space 200 including the virtual camera 300 and various objects. As shown in FIG. 7, the virtual space 200 is defined as an omnidirectional sphere centered on the center position 210 (in FIG. 7, only the upper half celestial sphere is shown). In the virtual space 200, an xyz coordinate system with the center position 210 as the origin is set. The virtual camera 300 defines a visual axis L for specifying the visual field image V (see FIG. 9) displayed on the HMD 110. The uvw coordinate system that defines the visual field of the virtual camera 300 is determined so as to be linked to the uvw coordinate system that is defined around the head of the user U in the real space. Further, the control unit 121 may move the virtual camera 300 in the virtual space 200 in conjunction with the movement of the user U wearing the HMD 110 in the real space.

  Next, in step S2, the control unit 121 identifies the field of view CV (see FIG. 8) of the virtual camera 300. Specifically, the control unit 121 acquires information on the position and inclination of the HMD 110 based on data indicating the state of the HMD 110 transmitted from the position sensor 130 and / or the HMD sensor 114. Next, the control unit 121 identifies the position and orientation of the virtual camera 300 in the virtual space 200 based on information regarding the position and tilt of the HMD 110. Next, the control unit 121 determines the visual axis L of the virtual camera 300 from the position and orientation of the virtual camera 300 and specifies the visual field CV of the virtual camera 300 from the determined visual axis L. Here, the visual field CV of the virtual camera 300 corresponds to a part of the virtual space 200 visible to the user U wearing the HMD 110 (in other words, a part of the virtual space 200 displayed on the HMD 110). Equivalent to Further, the visual field CV includes a first region CVa set as an angular range of the polar angle α around the visual axis L in the xy plane shown in the state (a) of FIG. 8, and the state (b) of FIG. 8. The xz plane shown has a second region CVb set as an angle range of the azimuth angle β with the visual axis L as the center. The control unit 121 identifies the line of sight of the user U based on the data indicating the line of sight of the user U transmitted from the gaze sensor 140, and based on the information regarding the identified line of sight of the user U and the position and inclination of the HMD 110. The orientation of the virtual camera 300 (virtual camera field axis L) may be determined. As will be described later, the control unit 121 may determine the orientation of the face of the user U's avatar based on information on the position and inclination of the HMD 110.

  As described above, the control unit 121 can specify the visual field CV of the virtual camera 300 based on the data from the position sensor 130 and / or the HMD sensor 114. Here, when the user U wearing the HMD 110 moves, the control unit 121 updates the visual field CV of the virtual camera 300 based on the data indicating the movement of the HMD 110 transmitted from the position sensor 130 and / or the HMD sensor 114. be able to. That is, the control unit 121 can update the visual field CV according to the movement of the HMD 110. Similarly, when the line of sight of the user U changes, the control unit 121 may update the visual field CV of the virtual camera 300 based on the data indicating the line of sight of the user U transmitted from the gaze sensor 140. That is, the control unit 121 may change the visual field CV according to the change in the line of sight of the user U.

  Next, in step S <b> 3, the control unit 121 generates visual field image data indicating the visual field image V displayed on the display unit 112 of the HMD 110. Specifically, the control unit 121 generates visual field image data based on virtual space data defining the virtual space 200 and the visual field CV of the virtual camera 300.

  Next, in step S4, the control unit 121 displays the field image V on the display unit 112 of the HMD 110 based on the field image data (see FIG. 9). As described above, the visual field CV of the virtual camera 300 is changed according to the movement of the user U wearing the HMD 110, and the visual field image V displayed on the display unit 112 of the HMD 110 is changed. I can immerse myself in 200.

  Note that the virtual camera 300 may include a left-eye virtual camera and a right-eye virtual camera. In this case, the control unit 121 generates left-eye view image data indicating the left-eye view image based on the virtual space data and the view of the left-eye virtual camera. Further, the control unit 121 generates right-eye view image data indicating a right-eye view image based on the virtual space data and the view of the right-eye virtual camera. Thereafter, the control unit 121 displays the left-eye field image on the left-eye display unit based on the left-eye field image data, and also displays the right-eye field image on the right-eye display unit based on the right-eye field image data. Is displayed. In this way, the user U can visually recognize the visual field image three-dimensionally due to the parallax between the left-eye visual field image and the right-eye visual field image. As will be described later, the virtual camera may be arranged at the position of the avatar eye operated by the user. For example, the left-eye virtual camera may be placed in the left eye of the avatar, while the right-eye virtual camera may be placed in the right eye of the avatar.

  Further, the processes in steps S1 to S4 shown in FIG. 6 may be executed for each frame (a still image constituting a moving image). For example, when the frame rate of the moving image is 90 fps, the processes in steps S1 to S4 may be repeatedly executed at intervals of ΔT = 1/90 (seconds). As described above, since the processes in steps S1 to S4 are repeatedly performed at predetermined intervals, the visual field of the virtual camera 300 is updated according to the operation of the HMD 110, and the visual field image V displayed on the display unit 112 of the HMD 110. Is updated.

  Next, the hardware configuration of the server 2 shown in FIG. 1 will be described with reference to FIG. FIG. 10 is a diagram illustrating a hardware configuration of the server 2. As shown in FIG. 10, the server 2 includes a control unit 23, a storage unit 22, a communication interface 21, and a bus 24. The control unit 23, the storage unit 22, and the communication interface 21 are communicably connected to each other via a bus 24. The control unit 23 includes a memory and a processor. The memory is composed of, for example, a ROM and a RAM, and the processor is composed of, for example, a CPU, an MPU, and / or a GPU.

  The storage unit (storage) 22 is, for example, a large-capacity HDD. The storage unit 22 may store a control program for causing a computer to execute at least a part of the information processing method according to the present embodiment, or a control program for realizing sharing of a virtual space by a plurality of users. Moreover, the memory | storage part 22 may store the data regarding the user management information for managing each user, various images, and objects (for example, avatar etc.). The communication interface 21 is configured to connect the server 2 to the communication network 3.

  Next, an example of processing for synchronizing the movements of the avatars 4A and 4B between the user terminal 1A and the user terminal 1B will be described with reference to FIGS. The state (a) in FIG. 11 is a diagram showing the virtual space 200A provided to the user A. The state (b) in FIG. 11 is a diagram showing the virtual space 200B provided to the user B. FIG. 12 is a sequence diagram for explaining an example of processing for synchronizing the movements of the avatars 4A and 4B between the user terminal 1A and the user terminal 1B. In this description, as a precondition, as shown in FIG. 11, the avatar 4A associated with the user terminal 1A (user A) and the avatar 4B associated with the user terminal 1B (user B) share the same virtual space. Suppose you are. That is, it is assumed that the user A and the user B share one virtual space via the communication network 3.

  As shown in the state (a) of FIG. 11, the virtual space 200A of the user A includes an avatar 4A and an avatar 4B. The avatar 4A is operated by the user A and interlocked with the operation of the user A. The avatar 4A includes a left hand (a part of the virtual body of the avatar 4A) linked to an operation of the controller 320L of the user terminal 1A (an operation of the left hand of the user A) and an operation of the controller 320R of the user terminal 1A (the right hand of the user A A right hand that is linked to the action) and a face whose facial expression is linked to the facial expression of the user A. The face of the avatar 4A has a plurality of face parts (for example, eyes, eyebrows, mouths, etc.). The avatar 4B is operated by the user B and interlocked with the operation of the user B. The avatar 4B has a left hand linked to the operation of the controller 320L of the user terminal 1B showing the operation of the user B's left hand, a right hand linked to the operation of the controller 320R of the user terminal 1B showing the operation of the user B's right hand, It has a face that is linked to the facial expression of user B. The face of the avatar 4B has a plurality of face parts (for example, eyes, eyebrows, mouths, etc.).

  In the virtual space 200A provided to the user A, a case where the avatar 4A is not visualized is also assumed. In this case, the avatar 4A arranged in the virtual space 200A includes at least the virtual camera 300 that is linked to the movement of the HMD 110 of the user terminal 1A.

  Further, the positions of the avatars 4A and 4B may be specified according to the position of the HMD 110 of the user terminals 1A and 1B. Similarly, the face orientations of the avatars 4A and 4B may be specified according to the inclination of the HMD 110 of the user terminals 1A and 1B. Furthermore, the movement of the hands of the avatars 4A and 4B may be specified according to the movement of the external controller 320 of the user terminals 1A and 1B. In particular, the action of the left hand of the avatars 4A and 4B is specified according to the action of the controller 320L of the user terminals 1A and 1B, and the action of the right hand of the avatars 4A and 4B is determined according to the action of the controller 320R of the user terminals 1A and 1B. May be specified. The facial expressions of the avatars 4A and 4B may be specified according to the facial expressions (states) of the users A and B. In particular, the position of the black eye for the white eyes of the avatars 4A and 4B may be specified according to the position of the black eye for the white eyes of the users A and B.

  Moreover, the virtual camera 300 (FIG. 7) may be arrange | positioned at each eyes of avatar 4A, 4B. In particular, a left-eye virtual camera may be disposed on the left eyes of the avatars 4A and 4B, and a right-eye virtual camera may be disposed on the right eyes of the avatars 4A and 4B. In the following description, it is assumed that the virtual camera 300 is disposed at the eye positions of the avatars 4A and 4B. For this reason, the visual field CV of the avatar 4A matches the visual field CV (see FIG. 7) of the virtual camera 300 arranged on the avatar 4A. Similarly, it is assumed that the field of view CV of the avatar 4B matches the field of view CV (see FIG. 7) of the virtual camera 300 arranged on the avatar 4B.

  As shown in the state (b) of FIG. 11, the virtual space 200B of the user B includes an avatar 4A and an avatar 4B. The positions of the avatars 4A and 4B in the virtual space 200A may correspond to the positions of the avatars 4A and 4B in the virtual space 200B.

  In the virtual space 200B provided to the user B, a case where the avatar 4B is not visualized is also assumed. In this case, the avatar 4B arranged in the virtual space 200B includes at least the virtual camera 300 that is linked to the movement of the HMD 110 of the user terminal 1B.

  Next, referring to FIG. 12, in step S10, the control unit 121 of the user terminal 1A generates voice data of the user A. For example, when the user A inputs voice to the microphone 118 (voice input unit) of the user terminal 1A, the microphone 118 generates voice data indicating the input voice. Thereafter, the microphone 118 transmits the generated audio data to the control unit 121 via the I / O interface 124.

  Next, in step S11, the control unit 121 of the user terminal 1A generates control information of the avatar 4A, and then generates audio data indicating the control information of the generated avatar 4A and the voice of the user A (voice data of the user A) ) To the server 2. Thereafter, the control unit 23 of the server 2 receives the control information of the avatar 4A and the voice data of the user A from the user terminal 1A (step S12).

  Here, the control information of the avatar 4A is information necessary for controlling the operation of the avatar 4A. The control information of the avatar 4A includes information about the position of the avatar 4A (position information), information about the face orientation of the avatar 4A (face orientation information), and information about the state of the avatar 4A's hand (left hand and right hand) (hand information). ) And information (face information) related to the facial expression of the avatar 4A.

  The face information of the avatar 4A has information indicating the states of a plurality of face parts. The information indicating the state of the plurality of face parts includes information (eye information) indicating the eye state of the avatar 4A (the shape of the white eye and the position of the black eye with respect to the white eye), and information on the eyebrow of the avatar 4A (the position and shape of the eyebrows) Etc.) and information (mouth information) indicating the mouth state (mouth position and shape, etc.) of the avatar 4A.

  Specifically, the eye information of the avatar 4A includes information indicating the state of the left eye of the avatar 4A and information indicating the state of the right eye of the avatar 4A. The eyebrow information of the avatar 4A includes information indicating the state of the left eyebrow of the avatar 4A and information indicating the state of the right eyebrow of the avatar 4A.

  The control unit 121 of the user terminal 1A acquires an image showing the eyes (left eye and right eye) and eyebrow (left eyebrow and right eyebrow) of the user A from the face camera 113 mounted on the HMD 110, and then acquires the acquired image. Based on a predetermined image processing algorithm, the state of user A's eyes (left eye and right eye) and eyebrows (left eyebrows and right eyebrows) is specified. Next, based on the eye and eyebrow state of the specified user A, the control unit 121 includes information indicating the eye state of the avatar 4A (left eye and right eye) and the eyebrow of the avatar 4A (left eyebrow and right eyebrow). Information indicating the state is generated.

  Similarly, the control unit 121 of the user terminal 1A acquires an image indicating the mouth of the user A and the periphery thereof from the face camera 117, and then based on the acquired image and a predetermined image processing algorithm, the control unit 121 of the user A The state of is identified. Next, the control unit 121 generates information indicating the mouth state of the avatar 4A based on the identified mouth state of the user A.

  As described above, the control unit 121 of the user terminal 1A includes the eye information of the avatar 4A corresponding to the eyes of the user A, the eyebrow information of the avatar 4A corresponding to the user A's eyebrows, and the avatar 4A corresponding to the mouth of the user A. Mouth information can be generated.

  In addition, the control unit 121 uses a plurality of types of faces stored in the storage unit 123 or the memory based on an image captured by the face camera 113, an image captured by the face camera 117, and a predetermined image processing algorithm. The facial expression (for example, smile) of the user A may be specified from among the facial expressions (for example, smile, sad expression, no expression, angry expression, surprised expression, troubled expression, etc.). Thereafter, the control unit 121 can generate face information indicating a facial expression (for example, a smile) of the avatar 4A based on the identified facial expression of the user A. In this case, expression data including a plurality of types of facial expressions of the avatar 4A and facial information of the plurality of avatars 4A each associated with one of the plurality of types of facial expressions is stored in the storage unit 123. May be.

  For example, when the facial expression of the avatar 4A is specified as a smile, the control unit 121 acquires facial information indicating the smile of the avatar 4A based on the facial expression (smile) of the avatar 4A and the facial expression data. Here, the face information indicating the smile of the avatar 4A includes eye information, eyebrow information, and mouth information of the avatar 4A when the facial expression of the avatar 4A is smiling.

  Next, in step S <b> 13, the control unit 121 of the user terminal 1 </ b> B generates control information for the avatar 4 </ b> B and transmits the generated control information for the avatar 4 </ b> B to the server 2. Thereafter, the control unit 23 of the server 2 receives the control information of the avatar 4B from the user terminal 1B (step S14).

  Here, the control information of the avatar 4B is information necessary for controlling the operation of the avatar 4B. The control information of the avatar 4B includes information about the position of the avatar 4B (position information), information about the face direction of the avatar 4B (face orientation information), and information about the state of the avatar 4B's hands (left hand and right hand) (hand information). ) And information (face information) related to the facial expression of the avatar 4B.

  The face information of the avatar 4B has information indicating the state of a plurality of face parts. The information indicating the state of the plurality of face parts includes information (eye information) indicating the eye state of the avatar 4B (white eye shape and position of the black eye with respect to the white eye), and eyebrow information of the avatar 4B (eyebrows position and shape). Etc.) and information (mouth information) indicating the mouth state (mouth position and shape, etc.) of the avatar 4B. The method for acquiring the face information of the avatar 4B is the same as the method for acquiring the face information of the avatar 4A.

  Next, the server 2 transmits the control information of the avatar 4B to the user terminal 1A (step S15), and transmits the control information of the avatar 4A and the voice data of the user A to the user terminal 1B (step S19). Thereafter, in step S16, the control unit 121 of the user terminal 1A receives the control information of the avatar 4B and then updates the state of the avatars 4A and 4B based on the control information of the avatars 4A and 4B. Virtual space data indicating 200A (see state (a) in FIG. 11) is updated (step S17).

  Specifically, the control unit 121 of the user terminal 1A updates the positions of the avatars 4A and 4B based on the position information of the avatars 4A and 4B. The control unit 121 updates the face orientation of the avatars 4A and 4B based on the face orientation information of the avatars 4A and 4B. The control unit 121 updates the hands of the avatars 4A and 4B based on the hand information of the avatars 4A and 4B. The control unit 121 updates the facial expressions of the avatars 4A and 4B based on the face information of the avatars 4A and 4B. Thus, the virtual space data indicating the virtual space 200A including the updated avatars 4A and 4B is updated.

  Thereafter, the control unit 121 of the user terminal 1A specifies the visual field CV of the avatar 4A (virtual camera 300) according to the position and inclination of the HMD 110, and then updates the virtual space data and the visual field CV of the avatar 4A. Based on this, the visual field image displayed on the HMD 110 is updated (step S18).

  On the other hand, the control unit 121 of the user terminal 1B updates the state of the avatars 4A and 4B based on the control information of the avatars 4A and 4B after receiving the control information of the avatar 4A and the voice data of the user A in step S20. Above, the virtual space data which shows virtual space 200B (refer the state (b) of FIG. 11) is updated (step S21).

  Specifically, the control unit 121 of the user terminal 1B updates the positions of the avatars 4A and 4B based on the position information of the avatars 4A and 4B. The control unit 121 updates the face orientation of the avatars 4A and 4B based on the face orientation information of the avatars 4A and 4B. The control unit 121 updates the hands of the avatars 4A and 4B based on the hand information of the avatars 4A and 4B. The control unit 121 updates the facial expressions of the avatars 4A and 4B based on the face information of the avatars 4A and 4B. Thus, the virtual space data indicating the virtual space including the updated avatars 4A and 4B is updated.

  Thereafter, the control unit 121 of the user terminal 1B specifies the visual field CV of the avatar 4B (virtual camera 300) according to the position and inclination of the HMD 110, and then updates the virtual space data and the visual field CV of the avatar 4B. Based on this, the visual field image displayed on the HMD 110 is updated (step S22).

  Thereafter, the control unit 121 of the user terminal 1B applies the received voice data of the user A, information about the position of the avatar 4A included in the control information of the avatar 4A, information about the position of the avatar 4B, and a predetermined voice processing algorithm. Based on this, the voice data of user A is processed. Thereafter, the control unit 121 transmits the processed audio data to the headphones 116 (audio output unit), and then the headphones 116 outputs the user A's audio based on the processed audio data (step S23). Thus, voice conversation between users (between avatars) can be realized in the virtual space.

  In the present embodiment, after the user terminals 1A and 1B transmit the control information of the avatar 4A and the control information of the avatar 4B to the server 2, the server 2 transmits the control information of the avatar 4A to the user terminal 1B, while the avatar 4B. Is transmitted to the user terminal 1A. As described above, the movements of the avatars 4A and 4B can be synchronously controlled between the user terminal 1A and the user terminal 1B.

(First embodiment)
Next, the information processing method according to the first embodiment will be described with reference to FIGS. FIG. 13 is a diagram illustrating an example of the virtual space 200A provided to the user A for explaining the information processing method according to the first embodiment. FIG. 14 is a sequence diagram for explaining an example of the information processing method according to the first embodiment. FIG. 15 is a diagram illustrating an example of the content evaluation table 400. FIG. 16 is a diagram illustrating a state in which a part of the virtual space 200A provided to the user A is updated when the number N of laughing users is equal to or greater than a predetermined threshold Nth.

  As shown in FIG. 13, in the first embodiment, the virtual space 200A provided to the user A is operated by the avatar 4A operated by the user A (user terminal 1A) and the user B (user terminal 1B). The avatar 4B, the avatar 4C operated by the user C (user terminal 1C), and a large number of avatars operated by a large number of users (not shown) are included. Similar to the virtual space distribution system 100 illustrated in FIG. 1, the user terminals 1 </ b> A to 1 </ b> C are connected to the server 2 via the communication network 3 so as to be communicable. The user terminals 1A to 1C are assumed to have the same configuration. In the virtual space 200A, the user C performs a performance such as comedy or one performance in real time through the avatar 4C. On the other hand, users A and B are spectators who watch the performance of user C in real time through avatars 4A and 4B. In addition, many users (not shown) are spectators who watch the performance of the user C in real time through their avatars. The performance performed in real time by the user C is the content Y associated with the virtual space 200A. In the first embodiment, the evaluation value of the content Y corresponds to the number N of users (users having the first emotion) who laughed by viewing the performance of the user C.

  Next, a process of updating the number N of laughing users corresponding to the evaluation value of the content Y when the user A laughs will be described with reference to FIG. As shown in FIG. 14, in step S30, the control unit 121 of the user terminal 1A determines whether or not the emotion of the user A is a laughing emotion (first emotion) (that is, identifies the emotion of the user A). To do). Specifically, in step S30, the control unit 121 of the user terminal 1A causes the facial expression of the user A, the sound output from the user A, and the movement of the user A's hand (that is, the movement of the external controller 320). The emotion of the user A is specified based on at least one of the above.

  Specifically, the control unit 121 may determine whether the user A has laughed after specifying the facial expression of the user A based on the moving images acquired from the face cameras 113 and 117. In addition, the control unit 121 may determine whether or not the user A has laughed after analyzing the voice of the user A input to the microphone 118. Further, the control unit 121 analyzes the movement of the user A's hand after obtaining a signal indicating the movement of the user A's hand (external controller 320) from the position sensor 130 and / or a sensor built in the external controller 320. It may be determined whether or not the user A has laughed. For example, the control unit 121 may determine whether or not the user A has laughed when it is determined that the user A has struck the hand. The control unit 121 specifies the facial expression of the user A, the voice of the user A, and the movement of the hand of the user A, and then comprehensively determines the state of the user A so that the user A You may determine whether you laughed. As described above, it is possible to accurately specify whether or not the user A laughs by using at least one of the facial expression of the user A, the voice of the user A, and the movement of the hand of the user A. .

  Next, when it is determined that the user A has laughed (YES in step S30), the control unit 121 transmits information indicating that the user A has laughed to the server 2 (step S31). On the other hand, when it is determined that the user A is not laughing (NO in step S30), the control unit 121 executes the process of step S30 again. Thereafter, the control unit 23 (see FIG. 10) of the server 2 receives the information indicating that the user A laughed (step S32), and then the content evaluation table 400 (see FIG. 15) stored in the storage unit 22. ) Is updated (step S33).

  As shown in FIG. 15, the content evaluation table 400 includes information on the number N of laughing users for each content scene. In this embodiment, when the initial performance of the user C corresponds to the scene 1 of the content Y, the control unit 23 of the server 2 receives the information indicating that the user A has laughed, and receives the content evaluation table 400. The number N of laughing users is updated from 60 to 61. Although the control unit 23 counts the number of users for each content scene, the control unit 23 may count the number of users for each content.

  Next, the control unit 23 determines whether or not the number N of laughing users corresponding to the evaluation value of the content Y has reached a predetermined threshold Nth (step S34). When it is determined that the number N of laughing users has reached a predetermined threshold Nth (YES in step S34), the control unit 23 transmits a signal instructing updating of the virtual space data to each user terminal (step S35). . On the other hand, when the determination result of step S34 is NO, the process of step S35 is not executed. In step S35, even if a signal instructing to update the virtual space data is transmitted to the user terminal 1C and the user terminals of all users who are viewing the content Y in real time (for example, the user terminals 1A and 1B). Good.

  Thereafter, when the user terminal 1A receives a signal instructing to update the virtual space data from the server 2, the user terminal 1A updates the virtual space data indicating the virtual space 200A (step S36). For example, in the process of step S36, as shown in FIG. 16, effects such as the trough object 350 falling from the sky of the virtual space 200A may be performed. Alternatively, an object indicating that the number N of laughing users has reached a predetermined number Nth may appear on the virtual space 200A. Further, the background image displayed on the celestial sphere of the virtual space 200A may be updated. Furthermore, the scene of the virtual space 200A may be updated. Thus, the entertainment property of the virtual space can be improved by updating the virtual space 200A. Furthermore, the user A can recognize that the number N of laughing users has reached a predetermined number Nth by visually recognizing the change in the virtual space 200A.

  According to the present embodiment, when it is determined whether the user A has laughed and the user A laughs, the number N of laughing users corresponding to the evaluation value of the content Y associated with the virtual space 200A is determined. Updated. As described above, since the user's real reaction to the content Y can be reflected in the content evaluation, the user experience in the virtual space can be improved.

  In the first embodiment, the number N of laughing users is counted as the evaluation value of the content. However, even if the number N of users laughing with the line of sight toward the avatar 4C is counted as the evaluation value of content. Good. In this case, in step S <b> 30, the control unit 121 of the user terminal 1 </ b> A may determine whether or not the user A has laughed with the line of sight of the user A (avatar 4 </ b> A) facing the avatar 4 </ b> C. For example, after acquiring the line-of-sight information of the user A based on the information transmitted from the gaze sensor 140, the control unit 121 determines that the line of sight of the user A is the avatar 4C based on the line-of-sight information of the user A and the positional information of the avatar 4C. It may be determined whether it is suitable for. Thus, since the number of users who smiled with their eyes on the avatar 4C affects the content evaluation, the content can be evaluated more accurately.

  Further, the server 2 may transmit information on the number N of laughing users shown in the content evaluation table 400 to each user terminal at a predetermined cycle. In this case, the control unit 121 of each user terminal may display the information on the field-of-view image V displayed on the HMD 110 after receiving information about the number N of laughing users from the server 2.

(Second Embodiment)
Next, an information processing method according to the second embodiment will be described below mainly with reference to FIGS. 13, 17, and 18. FIG. 17 is a sequence diagram for explaining an example of the information processing method according to the second embodiment. FIG. 18 is a diagram illustrating an example of the content evaluation table 420 according to the second embodiment. In the first embodiment, the number N of laughing users is adopted as the content evaluation value. In the second embodiment, the cumulative value of each user's laughter level (first emotion level) is the content evaluation value. Adopted as a value.

  As illustrated in FIG. 17, in step S <b> 40, the control unit 121 of the user terminal 1 </ b> A specifies the level of emotion of the user A laughing. Specifically, the control unit 121 of the user terminal 1A is at least one of the facial expression of the user A, the sound output from the user A, and the movement of the user A's hand (movement of the external controller 320). Based on the above, the level of the feeling of laughter of the user A is specified. For example, the user's laughing emotion level may be classified into four levels (level 3: big laughter, level 2: medium laughter, level 1: small laughter, level 0: not laughing). Next, when the level of laughing emotion of user A is not level 0 (that is, user A is laughing) (YES in step S41), control unit 121 indicates the level of laughing emotion of user A. Information is transmitted to the server 2 (step S42). For example, when the user A is laughing big, the control unit 121 transmits information indicating the level 3 corresponding to the emotional level of the user A laughing to the server 2. On the other hand, when user A is not laughing (that is, the level of laughing emotion is level 0) (NO in step S41), the process of step S40 is executed again.

  Next, the control unit 23 of the server 2 receives the information indicating the level of the laughing feeling of the user A (step S43), and then uses the content evaluation table 420 (see FIG. 18) stored in the storage unit 22. Update (step S44). As shown in FIG. 18, the content evaluation table 420 includes information regarding a cumulative value P of the level of laughter of each user for each content scene (hereinafter simply referred to as “cumulative value P of laughter level”). For example, in this embodiment, it is assumed that the initial performance of the user C corresponds to the scene 1 of the content Y. Furthermore, when the level of the laughter feeling of the user A is level 1, +1 point is added to the accumulated value P of the laughter level. When the level of the laughing feeling of the user A is level 2, +2 points are added to the accumulated value P of the laughing level. When the level of emotion of laughter of user A is level 3, +3 points are added to the accumulated value of laughter level. In this case, when the control unit 23 of the server 2 receives information indicating the emotion level (level 3) of the laughter of the user A, the cumulative value of the laughter level shown in the content evaluation table 420 is increased from 97 points to 100 points. change. The control unit 23 counts the accumulated value P of the laughing level for each content scene, but may count the accumulated value P of the laughing level for each content.

  Next, the control unit 23 determines whether or not the accumulated value P of the laughter level corresponding to the evaluation value of the content Y has reached a predetermined threshold value Pth (step S45). When determining that the accumulated value P of the laughter level has reached the predetermined threshold value Pth (YES in step S45), the control unit 23 transmits a signal instructing to update the virtual space data to each user terminal (step S46). . On the other hand, when the determination result of step S45 is NO, the process of step S46 is not executed. Thereafter, when the user terminal 1A receives a signal instructing to update the virtual space data from the server 2, the user terminal 1A updates the virtual space data indicating the virtual space 200A (step S47).

  According to this embodiment, when the level of laughing emotion of user A is specified and the level of laughing emotion of user A is not level 0, the accumulated value of the laughing level corresponding to the evaluation value of content Y P is updated. As described above, since the user's real reaction to the content Y can be reflected in the content evaluation, the user experience in the virtual space can be improved. In particular, since the level of user laughter can be directly reflected in the evaluation value of the content, the content can be evaluated in more detail.

  In the second embodiment, the content evaluation table 420 is updated based on the level of laughing emotion of the user A without considering the user A's line of sight. However, in the process of step S40, the content of the user A's line of sight is updated. Direction may be taken into account. In this case, in step S40, the control unit 121 of the user terminal 1A determines whether or not the line of sight of the user A (avatar 4A) faces the avatar 4C and specifies the level of laughing emotion of the user A. Good. For example, if the control unit 121 determines that the line of sight of the user A is not facing the avatar 4C, the control unit 121 may not transmit information indicating the level of the laughing emotion of the user A to the server 2. That is, in this case, the content evaluation table 420 is not updated. Thus, since the number of users who smiled with their eyes on the avatar 4C affects the content evaluation, the content can be evaluated more accurately.

  Further, the server 2 may transmit information on the accumulated value of the laughter level shown in the content evaluation table 400 to each user terminal at a predetermined cycle. In this case, the control unit 121 of each user terminal may display the information on the field-of-view image V displayed on the HMD 110 after receiving information on the accumulated value of the laughter level from the server 2.

(Third embodiment)
Next, an information processing method according to the third embodiment will be described below with reference to FIGS. FIG. 19 is an example of a virtual space 200D provided to the user D for explaining the information processing method according to the third embodiment. FIG. 20 is a sequence diagram for explaining the information processing method according to the third embodiment. FIG. 21 is a diagram illustrating an example of the visual field image V displayed on the HMD 110 worn by the user D. In the first and second embodiments, when each user other than the user C views the performance (content Y) performed by the user C (avatar 4C) in the virtual space in real time, the performance (content Y) is I explained how to evaluate by feelings (especially feelings of laughter). In the third embodiment, a method for evaluating the performance (content Y) performed by the user C through the user's emotion (particularly, laughing emotion) when the user views the performance through the video platform will be described.

  As illustrated in FIG. 19, the virtual camera 300 is disposed at the center of the virtual space 200 </ b> D provided to the user D. The virtual camera 300 is linked to the movement of the user D's HMD 110. Similar to the distribution system 100 shown in FIG. 1, the user terminal 1 </ b> D is communicably connected to the server 2 via the communication network 3. The user terminal 1D is assumed to have the same configuration as the user terminals 1A and 1B described above. A 360-degree spatial image of content Y (video content) indicating the performance of user C described in the first embodiment is displayed on the celestial sphere Sd of the virtual space 200D. For example, 360 degree spatial image data indicating a 360 degree spatial image of the content Y (hereinafter simply referred to as 360 degree spatial image data) is arranged at a predetermined position (for example, a central position) in the virtual space 200A shown in FIG. It can be acquired through a 360-degree camera object for shooting. 360 degree | times spatial image data may be acquired by the control part 23 of the server 2, for example. The 360-degree spatial image data acquired by the control unit 23 of the server 2 may be stored in the storage unit 22 of the server 2 or may be stored in a storage unit of another server for the moving image platform. In the present embodiment, it is assumed that 360 degree spatial image data is stored in the storage unit 22 of the server 2.

  As shown in FIG. 20, in step S50, the control unit 121 of the user terminal 1D transmits to the server 2 a signal for requesting viewing of 360-degree spatial image data of the content Y (hereinafter referred to as a viewing request signal). To do. For example, the control unit 121 generates a viewing request signal through a predetermined input operation by the user D, and transmits the viewing request signal to the server 2 via the communication network 3. The viewing request signal may include information related to the IP address of the user terminal 1D and information related to the IP address of the server 2.

  Next, when the server 2 receives the viewing request signal from the user terminal 1D, the server 2 transmits the 360-degree spatial image data stored in the storage unit 22 and information on the number N of laughing users to the user terminal 1D. (Step S51). Here, the server 2 may acquire information on the number N of laughing users by referring to the content evaluation table 400 (see FIG. 15) stored in the storage unit 22. Here, the number N of laughing users is not the number of users laughing by viewing the content Y in real time, but the number of users laughing by viewing the content Y through the video platform.

  Next, the control unit 121 of the user terminal 1D receives 360-degree spatial image data and information regarding the number N of laughing users from the server 2 and then presents information regarding the number of laughing users to the user D ( Step S52). For example, as illustrated in FIG. 21, information regarding the number of laughing users may be displayed on the view image V displayed on the HMD 110 of the user terminal 1D. Moreover, the information regarding the number of users who laughed may be presented auditorily to the user D through the headphones 116 of the user terminal 1D. Thus, the information regarding the evaluation value of the content Y is presented to the user D, so that the user D can know the evaluation of other users with respect to the content Y before viewing.

  Next, in step S53, the control unit 121 displays a 360 degree spatial image corresponding to the 360 degree spatial image data received from the server 2 on the celestial sphere Sd of the virtual space 200D. In this way, the control unit 121 of the user terminal 1D presents the 360-degree spatial image of the content Y indicating the performance of the user C to the user D. In particular, the control unit 121 specifies the field of view of the virtual camera 300 based on the movement of the HMD 110, and then generates field image data based on the virtual space data indicating the virtual space 200 </ b> D and the field of view of the virtual camera 300. Thereafter, the control unit 121 displays a field image on the HMD 110 based on the field image data. As shown in FIG. 21, the content Y (the avatar 4C performing the performance) is displayed on the visual field image V displayed on the HMD 110 of the user terminal 1D. Note that the server 2 may transmit the 360-degree spatial image data to the user terminal 1D by a streaming method, or may transmit the 360-degree spatial image data to the user terminal 1D in a batch download format.

  Next, in step S54, the control unit 121 of the user terminal 1D determines whether or not the emotion of the user D is a laughing emotion (first emotion). Specifically, in step S54, the control unit 121 selects at least one of the facial expression of the user D, the voice output from the user D, and the movement of the user D's hand (that is, the movement of the external controller 320). Based on one, the emotion of user D is specified.

  When it is determined that the user D has laughed (YES in step S54), the control unit 121 transmits information indicating that the user D has laughed to the server 2 (step S55). On the other hand, when it is determined that the user D is not laughing (NO in step S54), the control unit 121 executes the process of step S54 again. Thereafter, the control unit 23 of the server 2 receives the information indicating that the user D laughed (step S56), and then updates the content evaluation table 400 (see FIG. 15) stored in the storage unit 22 (see FIG. 15). Step S57).

  Thus, according to the present embodiment, when the user D laughs when viewing the content Y distributed from the server 2, the number of laughing users corresponding to the evaluation value of the content Y is Updated. Therefore, since the content Y can be evaluated based on the emotion of the user who views the content Y, the user's realistic emotion (reaction) can be reflected in the evaluation of the content Y. Further, the user who views the content Y can omit the operation input for the content evaluation. In particular, it is not necessary for the user to evaluate the content by pressing an evaluation button (such as a “like” button) provided on the moving image distribution platform, so that it is possible to save the user operation for content evaluation.

  In the present embodiment, the number of users who laughed is adopted as the evaluation value of the content to be viewed. However, the cumulative value of the laughing level of each user may be adopted as the evaluation value of the content. Furthermore, in the present embodiment, the server 2 transmits the 360-degree spatial image data of the content Y to the user terminal 1D, but the virtual space data of the content Y may be transmitted to the user terminal 1D. Here, the virtual space data of the content Y is virtual space data indicating a virtual space in a period in which the performance of the user C (avatar 4C) is performed. The virtual space data may be acquired by the control unit 23 of the server 2. The virtual space data acquired by the control unit 23 of the server 2 may be stored in the storage unit 22 of the server 2 or may be stored in a storage unit of another server for the moving image platform.

  In the description of the first to third embodiments, the content evaluation table stored in the storage unit 22 of the server 2 is updated when the user's emotion is a laughing emotion. In other words, the feeling of laughter is adopted as the content evaluation value. On the other hand, the content evaluation table may be updated when the user's emotion is an emotion other than the laughing emotion (for example, sadness, anger, surprise, fear, etc.). That is, emotions other than the laughing emotion may be adopted as the content evaluation value. For example, in the case of content intended to scare viewers, the user's fear of fear may be adopted as the content evaluation value. In this way, user emotions other than laughing emotions can be adopted as content evaluation values.

  Furthermore, instead of the user's laughing feeling, user communication behavior may be adopted as the content evaluation value. For example, in step S30 illustrated in FIG. 14, the control unit 121 of the user terminal 1A specifies the communication behavior of the user A. Next, the control unit 121 indicates that the user A is evaluating the content Y when it is determined that the identified communication behavior of the user A is an action for evaluating the performance (content Y) of the user C. Information is transmitted to the server 2. For example, when the user A gives a comment such as “User C's performance is amazing!” Or “Please look at it for a moment!” To the user B, the control unit 121 of the user terminal 1A controls the user A's It is determined that the communication behavior of the user A is the behavior for evaluating the performance (content Y) of the user C by performing language analysis on the statement. Thereafter, the control unit 23 of the server 2 updates the content evaluation table stored in the storage unit 22 of the server 2 based on information indicating that the user A is evaluating the content Y. Here, the content evaluation table may include information on evaluation points for each content. When the evaluation point of the content Y shown in the content evaluation table is n, the control unit 23 updates the content evaluation point to n + 1 after receiving information indicating that the user A is evaluating the content Y. Also good.

  Further, in order to implement various processes executed by the control unit 121 of the user terminal 1 by software, a control program for causing the computer (processor) to execute various processes may be incorporated in the storage unit 123 or the memory in advance. Good. Alternatively, the control program can be a magnetic disk (HDD, floppy disk), optical disk (CD-ROM, DVD-ROM, Blu-ray disk, etc.), magneto-optical disk (MO, etc.), flash memory (SD card, USB memory, SSD). Etc.) may be stored in a computer-readable storage medium. In this case, when the storage medium is connected to the control device 120, the control program stored in the storage medium is incorporated into the storage unit 123. The control unit 121 executes various processes by loading the control program incorporated in the storage unit 123 onto the RAM and executing the loaded program.

  The control program may be downloaded from a computer on the communication network 3 via the communication interface 125. Similarly in this case, the downloaded control program is incorporated into the storage unit 123.

  As mentioned above, although embodiment of this indication was described, the technical scope of this invention should not be limitedly interpreted by description of this embodiment. This embodiment is an example, and it is understood by those skilled in the art that various modifications can be made within the scope of the invention described in the claims. The technical scope of the present invention should be determined based on the scope of the invention described in the claims and the equivalents thereof.

  In this embodiment, the virtual space is used to provide a virtual experience such as VR (Virtual Reality), AR (Arranged Reality), and MR (Mixed Reality) to the user. When the virtual space provides VR, background data stored in the memory is used as the background of the virtual space. When the virtual space provides AR or MR, the real space is used as the background. In this case, the HMD 110 includes a transmissive display device (optical see-through or video see-through display device), so that the real space can be used as a background. If virtual space is applied to MR, the object may be influenced by real space. As described above, when the virtual space includes at least a part of a virtual scene such as a background or a virtual object, the user can be provided with a virtual experience capable of interacting with the virtual scene.

1, 1A-1D: User terminal 2: Server 3: Communication networks 4A, 4B, 4C: Avatar 21: Communication interface 22: Storage unit 23: Control unit 24: Bus 100: Virtual space distribution system (distribution system)
110: Head mounted device (HMD)
112: Display unit 113: Face camera 114: Sensor 115: Wear sensor 116: Headphone 117: Face camera 118: Microphone 120: Control device 121: Control unit 123: Storage unit 124: I / O interface 125: Communication interface 126: Bus 130: Position sensor 140: Gaze sensor 200, 200A, 200B, 200C, 200D: Virtual space 210: Center position 300: Virtual camera 302: Operation buttons 302a, 302b: Push buttons 302e, 302f: Trigger buttons 304: Detection points 320: External controller 320i: Analog stick 320L: Left hand external controller (controller)
320R: External controller for right hand (controller)
322: Top surface 324: Grip 326: Frame 400, 420: Content evaluation table

Claims (15)

An information processing method executed by a computer in an information processing system including a user terminal having a head-mounted device mounted on a user's head and a server connected to the user terminal so as to be communicable,
(A) generating visual field image data indicating a visual field image displayed on the head mounted device based on virtual space data defining a virtual space and movement of the head mounted device;
(B) identifying the user's emotion;
(C) updating the evaluation value of the content associated with the virtual space based on the identified user's emotion;
Including an information processing method. The step (b)
Identifying the user's emotion based on at least one of the facial expression of the user, voice output from the user, and movement of the user's hand,
The information processing method according to claim 1. The step (b)
Determining whether the user's emotion is a first emotion,
The step (c)
Updating the evaluation value of the content when the identified user's emotion is the first emotion,
The information processing method according to claim 1 or 2. The virtual space is viewed by a plurality of users,
The evaluation value of the content corresponds to the number of users having the first emotion among the plurality of users.
The information processing method according to claim 3.   5. The evaluation value of the content according to claim 4, wherein the evaluation value of the content corresponds to the number of users having the first emotion and directing a line of sight to the first region of the virtual space among the plurality of users. Information processing method. The step (b)
Identifying a level of the user's first emotion,
The step (c)
Updating an evaluation value of the content based on the identified first emotion level;
The information processing method according to claim 1 or 2. The virtual space is viewed by a plurality of users,
The information processing method according to claim 6, wherein the evaluation value of the content corresponds to a cumulative value of the first emotion level of each user.   (D) The information processing method according to any one of claims 1 to 7, further comprising a step of updating at least a part of the virtual space data based on the evaluation value of the content. The step (d)
The information processing method according to claim 8, further comprising a step of updating at least a part of the virtual space data when the evaluation value of the content reaches a first value.   8. The method according to claim 1, further comprising: (e) transmitting data related to the content stored in the server to the user terminal in response to a transmission request from the user terminal. The information processing method described. The step (e)
Transmitting data related to the content stored in the server and information about the evaluation value of the content to the user terminal,
The information processing method according to claim 10, further comprising: (f) presenting information related to the evaluation value of the content to the user. An information processing method executed by a computer in an information processing system including a user terminal having a head-mounted device mounted on a user's head and a server connected to the user terminal so as to be communicable,
(A) generating visual field image data indicating a visual field image displayed on the head mounted device based on virtual space data defining a virtual space and movement of the head mounted device;
(B) identifying the user's communication behavior;
(C) updating an evaluation value of content associated with the virtual space based on the identified communication behavior;
Including an information processing method.   An information processing program for causing a computer to execute the information processing method according to any one of claims 1 to 12. A user terminal having a head-mounted device mounted on the user's head;
A server communicably connected to the user terminal;
An information processing system configured to execute the information processing method according to any one of claims 1 to 12. A processor;
An information processing apparatus comprising a memory for storing computer-readable instructions,
The information processing apparatus that executes the information processing method according to any one of claims 1 to 12, when the computer-readable instruction is executed by the processor.

Images