JP2019083029A - Information processing method, information processing program, information processing system, and information processing device - Google Patents

Abstract

To provide an information processing method, an information processing program, an information processing system, and an information processing device, capable of appropriately identifying an attention range of content.SOLUTION: An information processing method in a virtual space distribution system 100 comprising a head mount device (HMD 110) includes the steps of: generating virtual space data for defining a virtual space; generating visual field image data indicating a visual field image displayed in the HMD on the basis of the movement of the HMD and the virtual space data; identifying the emotions of users A and B wearing the HMD; and updating an evaluation value of a first region in the virtual space according to the direction of the HMD at the time at which a predetermined first condition is satisfied when the identified emotion satisfies the predetermined first condition.SELECTED DRAWING: Figure 1

Description

  The present disclosure relates to an information processing method, an information processing program, an information processing system, and an information processing apparatus.

  Patent Document 1 discloses that the attention range of video content is determined based on the overlapping of the view range of a plurality of users.

International publication 2016/009864 gazette

  In Patent Document 1, although the attention range of video content is determined based on the visual information of the user, since the action of the user focusing on the content is not reflected only in the visual information, specifying the attention range in the content There is room for improvement.

  An object of the present disclosure is to provide an information processing method, an information processing program, an information processing system, and an information processing apparatus capable of appropriately specifying the attention range of content.

According to one aspect the present disclosure shows, an information processing method in a system comprising a head mounted device is:
(A) generating virtual space data defining a virtual space;
(B) generating view image data indicating a view image to be displayed on the head mounted device based on the movement of the head mounted device and the virtual space data;
(C) identifying an emotion of a user wearing the head mount device;
(D) evaluation of the first region in the virtual space according to the orientation of the head mounted device at the time when the specified first condition is satisfied when the specified emotion satisfies the first predetermined condition Updating the values,
including.

  According to the present disclosure, it is possible to provide an information processing method, an information processing program, an information processing system, and an information processing apparatus capable of appropriately specifying the attention range of content.

FIG. 1 is a schematic view showing a virtual experience content distribution system according to an embodiment of the present invention (hereinafter simply referred to as the present embodiment). It is the schematic which shows the portable terminal which is a user terminal. It is a figure which shows the hardware constitutions of a portable terminal. It is the schematic which shows the structure of the user terminal containing a head mounted device (HMD). It is a figure showing a user's head equipped with HMD. It is a figure which shows the hardware constitutions of a control apparatus. It is a flowchart which shows the process which displays a visual field image on HMD. It is a xyz space figure which shows an example of virtual space. (A) is yx top view of virtual space shown in FIG. FIG. 8B is a zx plan view of the virtual space shown in FIG. It is a figure which shows an example of the visual field image displayed on HMD. It is a figure which shows the hardware constitutions of the server shown in FIG. It is a flowchart for demonstrating the operation | movement of the virtual experience content distribution system which concerns on this embodiment. It is a flowchart for demonstrating the operation | movement of the virtual experience content distribution system which concerns on this embodiment. It is a flowchart which shows each step in step 120 (step which updates the evaluation value data of each partial area | region) of FIG. It is a figure which shows the virtual space which comprises virtual experience content. It is a figure which shows another example of the virtual space which comprises virtual experience content. FIG. 6 is a schematic view showing an arrangement configuration of virtual cameras in a virtual space for generating a digest video.

[Description of Embodiments Presented by Present Disclosure]
An outline of an embodiment indicated by the present disclosure will be described.
(1) An information processing method in a system including a head mounted device according to an embodiment of the present disclosure,
(A) generating virtual space data defining a virtual space;
(B) generating view image data indicating a view image to be displayed on the head mounted device based on the movement of the head mounted device and the virtual space data;
(C) identifying an emotion of a user wearing the head mount device;
(D) evaluation of the first region in the virtual space according to the orientation of the head mounted device at the time when the specified first condition is satisfied when the specified emotion satisfies the first predetermined condition Updating the values,
including.

  According to this method, it is possible to appropriately identify the attention area (the attention area) of the content in the virtual space. That is, since the time when a specific emotion of the user is detected is specified as the excitement point of the content, and the evaluation point of the attention area is updated according to the direction (field of view) of the HMD at that point, the true excitement of the content Never miss a point.

  (2) (e) If the user's action satisfies the second condition after satisfying the first condition, the method may further include the step of further updating the evaluation value.

  By further taking into account the user's behavior after a specific emotion is detected, it is possible to enhance the eligibility of the characteristics of the attention range by adjusting the importance (the attention degree) of the attention region.

  (3) The second condition may include that a conversation has been made among a plurality of users in the virtual space.

  Communication behavior among multiple users can be used to adjust the importance of the attention area.

  (4) The conversation may include a linguistic expression pointing to the first region.

  (5) The conversation may include a tense expression pointing to the time.

  By using the communication behavior between users for the attention area, it is possible to easily adjust the importance of the attention area.

  (6) The second condition may include the user taking a picture of the first area in the virtual space.

  By using the photographing action for the attention area, the importance of the attention area can be easily adjusted.

  (7) The second condition may include an action of the user posting a photograph of the first area taken in the virtual space to a social networking site.

  Posting behavior to SNS can be used to adjust the importance of the area of interest.

(8) (f) designating the first region at the time as the region of interest when the evaluation value satisfies the third condition;
(G) generating an image relating to the region of interest over a first duration including the time of day;
May be further included.

  According to this method, it is possible to generate an advertisement video and the like using the attention area.

  (9) The video is generated based on the position of the region of interest in the virtual space, and a virtual camera disposed in the virtual space to define the view image based on the movement of the head mounted device. May be

  According to this method, advertisement video and the like can be easily and appropriately generated.

(10) An information processing method in a system including a head mount device according to an embodiment of the present disclosure,
(A ') generating virtual space data defining a virtual space;
(B ') generating, based on the movement of the head mounted device and the virtual space data, visual field image data indicating a visual field image displayed on the head mounted device;
(C ') identifying an action of a user wearing the head mount device;
(D ′) When the specified action satisfies a predetermined second condition, the first area in the virtual space according to the orientation of the head mount device at the time when the second condition is satisfied Updating the evaluation value;
including.

  According to this method, it is possible to appropriately identify the attention area of the content without missing the true excitement point of the content.

  (11) An information processing program according to an embodiment of the present disclosure is an information processing program for causing a computer to execute the information processing method according to any one of (1) to (10).

  According to the above, it is possible to provide an information processing program capable of appropriately specifying the attention range of the content.

  (12) An information processing system using a head mounted device according to an embodiment of the present disclosure is configured to execute the information processing method according to any one of (1) to (10). It is.

  According to the above, it is possible to provide an information processing system capable of appropriately specifying the attention range of content.

(13) An information processing apparatus according to an embodiment of the present disclosure,
A processor,
An information processing apparatus comprising: a memory for storing computer readable instructions;
The information processing apparatus is an information processing apparatus that executes the information processing method according to any one of (1) to (10) when the computer readable instruction is executed by the processor.

  According to the above, it is possible to provide an information processing apparatus capable of appropriately specifying the attention range of the content. It should be noted that the information processing apparatus is either a user terminal or a server.

[Details of Embodiment Presented by Present Disclosure]
Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. For the elements having the same reference numerals as the elements already described in the description of the present embodiment, the description thereof will not be repeated for convenience of the description.

  First, an outline of the configuration of the virtual space distribution system 100 (information processing system) will be described with reference to FIG. FIG. 1 is a schematic view of a virtual space distribution system 100 (hereinafter simply referred to as a distribution system 100). As shown in FIG. 1, the distribution system 100 includes a user terminal 1A (first user terminal) operated by a user A wearing a head mounted device (HMD) 110 and a user operated by a user B wearing an HMD 110 A terminal 1B and a server 2 for distributing virtual experience content are provided. The user terminals 1A and 1B are communicably connected to the server 2 via a communication network 3 such as the Internet. Hereinafter, for convenience of description, each of the user terminals 1A and 1B may be simply referred to as the user terminal 1. The users A and B may be simply referred to as the user U. In the present embodiment, the user terminals 1A and 1B are assumed to have the same configuration. The virtual experience content is video content or the like for providing the user of the user terminals 1A and 1B with a virtual experience, and is a virtual space including a plurality of objects or a virtual space including a 360 degree space image such as an omnidirectional image. May be provided as In a virtual space including a 360 degree space image, a 360 degree space image is displayed on the surface of the virtual space. The virtual space includes a VR (Virtual Reality) space, an AR (Argumented Reality) space, and an MR (Mixed Reality) space.

  Next, the configuration of the user terminal 1 will be described with reference to FIG. FIG. 2 is a schematic view showing the user terminal 1. As shown in FIG. 2, the user terminal 1 includes a head mounted device (HMD) 110 mounted on the head of the user U, a headphone 116, a face camera 117, a microphone 118, a position sensor 130, and an external controller 320 and a controller 120.

  The HMD 110 includes a display unit 112, an HMD sensor 114, a gaze sensor 140, and a face camera 113. The display unit 112 includes a non-transmissive display device configured to completely cover the field of view (field of view) of the user U wearing the HMD 110. Accordingly, the user U can immerse in the virtual space by viewing only the view image displayed on the display unit 112. The display unit 112 is configured of a display unit for the left eye configured to provide an image to the left eye of the user U, and a display unit for the right eye configured to provide an image to the right eye of the user U It is also good. The HMD 110 may also include a transmissive display device. In this case, the transmissive display device may be temporarily configured as a non-transmissive display device by adjusting the transmittance thereof.

  The HMD sensor 114 is mounted near the display unit 112 of the HMD 110. The HMD sensor 114 includes at least one of a geomagnetic sensor, an acceleration sensor, and an inclination sensor (such as an angular velocity sensor or a gyro sensor), and detects various movements (such as inclination) of the HMD 110 mounted on the head of the user U be able to.

  The gaze sensor 140 has an eye tracking function of detecting the line of sight of the user U. The gaze sensor 140 may include, for example, a right eye gaze sensor and a left eye gaze sensor. The right eye gaze sensor emits infrared light, for example, to the right eye of the user U, and detects reflected light reflected from the right eye (in particular, the cornea and the iris) to obtain information on the rotation angle of the right eye You may On the other hand, the gaze sensor for the left eye irradiates, for example, infrared light to the left eye of the user U, and detects the reflected light reflected from the left eye (in particular, the cornea and the iris). You may get The acquisition rate of detection data acquired by the gaze sensor 140 is larger than the transmission rate of avatar control information (described later) and the frame rate of a view image displayed on the HMD 110. For example, the acquisition rate of detection data is 200 Hz.

  The face camera 113 is configured to obtain an image (in particular, a moving image) in which the eyes of the user U (left and right eyes) and the eyebrows (left eyebrow and right eyebrow) are displayed while the HMD 110 is attached to the user U It is done. The face camera 113 is disposed at a predetermined position inside the HMD 110 so as to be able to image the eyes and eyebrows of the user U. The frame rate of the moving image acquired by the face camera 113 is larger than the transmission rate of the avatar control information and the frame rate of the view image displayed on the HMD 110. For example, the frame rate of the acquired moving image is 200 fps.

  The headphones 116 (audio output unit) are attached to the left ear and the right ear of the user U, respectively. The headphones 116 are configured to receive audio data (electrical signals) from the control device 120, and to output audio based on the received audio data. The microphone 118 (voice input unit) is configured to collect voice uttered by the user U and to generate voice data (electrical signal) based on the collected voice. Furthermore, the microphone 118 is configured to transmit audio data to the control device 120.

  The face camera 117 is configured to acquire an image (in particular, a moving image) in which the mouth of the user U and the periphery thereof are displayed. The face camera 117 may be disposed at a position facing the user U's mouth so that the user U's mouth and the periphery thereof can be imaged. Also, the face camera 117 may be coupled to the HMD 110. The frame rate of the moving image acquired by the face camera 117 may be larger than the frame rate of the view image displayed on the HMD 110.

  The position sensor 130 is configured by, for example, a position tracking camera, and is configured to detect the position of the HMD 110 and the external controller 320. The position sensor 130 is communicably connected to the control device 120 wirelessly or by wire, and is configured to detect information on the position, inclination or light emission intensity of a plurality of detection points (not shown) provided on the HMD 110. . Furthermore, the position sensor 130 is configured to detect information on the position, the inclination, and / or the light emission intensity of a plurality of detection points (not shown) provided in the external controller 320. The detection point is, for example, a light emitting unit that emits infrared light or visible light. The position sensor 130 may also include an infrared sensor and a plurality of optical cameras.

  The external controller 320 is used to control the motion of the hand of the avatar displayed in the virtual space by detecting the motion of the hand of the user U. The external controller 320 is an external controller 320R for the right operated by the user U's right hand (hereinafter simply referred to as the controller 320R) and an external controller 320L for the left hand operated by the left hand of the user U (hereinafter referred to simply as the controller 320L). And. The controller 320R is a device that indicates the position of the user U's right hand and the movement of the right hand's finger. In addition, the avatar's right hand existing in the virtual space moves in response to the movement of the controller 320R. The controller 320L is a device that indicates the position of the user U's left hand and the movement of fingers of the left hand. In addition, the left hand of the avatar present in the virtual space moves in response to the movement of the controller 320L.

  Control device 120 is a computer configured to control HMD 110. The control device 120 specifies the position information of the HMD 110 based on the information acquired from the position sensor 130, and the position of the virtual camera in the virtual space (the position of the avatar) and the reality based on the specified position information. The position of the user U wearing the HMD 110 in the space can be accurately associated. Furthermore, the control device 120 specifies the operation of the external controller 320 based on the information acquired from the position sensor 130 and / or the sensor built in the external controller 320, and based on the operation of the specified external controller 320. Thus, the motion of the avatar hand displayed in the virtual space can be accurately correlated with the motion of the external controller 320 in the real space. In particular, the control device 120 specifies the operation of the controller 320L based on the information acquired from the sensor incorporated in the position sensor 130 and / or the controller 320L, and based on the operation of the specified controller 320L, a virtual The action of the left hand of the avatar displayed in the space and the action of the controller 320L in the real space (the action of the left hand of the user U) can be accurately correlated. Similarly, the control device 120 specifies the operation of the controller 320R based on the information acquired from the position sensor and / or the sensor incorporated in the controller 320R, and based on the operation of the specified controller 320R, the virtual space The action of the avatar's right hand displayed in the inside and the action of the controller 320R in the real space (the action of the user U's right hand) can be accurately correlated.

  In addition, the control device 120 can associate the facial expression (state) of the user U with the facial expression (state) of the avatar of the user U based on the moving image acquired by the face cameras 113 and 117. The details of the process of correlating the facial expression of the user U with the facial expression of the avatar will be described later.

  Further, the control device 120 can identify the line of sight of the right eye and the line of sight of the left eye of the user U, respectively, and can specify a gaze point which is an intersection of the line of sight of the right eye and the line of sight of the left eye. Furthermore, the control device 120 can specify the line of sight of the user U (the line of sight of the user U) based on the specified fixation point. Here, the line of sight of the user U is the line of sight of the user U, and coincides with the direction of a straight line passing through the midpoint of the line connecting the right eye and the left eye of the user U and the gaze point. In addition, the control device 120 is based on detection data (eye tracking data) transmitted from the gaze sensor 140 (right eye gaze sensor and right eye gaze sensor), the center position of the right eye of the user U and the left eye of the user U And the center position of the black eye can be specified.

  Next, with reference to FIG. 3, a method of acquiring information on the position and tilt of the HMD 110 will be described below. FIG. 3 is a view showing the head of the user U wearing the HMD 110. As shown in FIG. Information on the position and tilt of the HMD 110 interlocked with the movement of the head of the user U wearing the HMD 110 can be detected by the position sensor 130 and / or the HMD sensor 114 mounted on the HMD 110. As shown in FIG. 2, three-dimensional coordinates (uvw coordinates) are defined around the head of the user U wearing the HMD 110. A vertical direction in which the user U stands is defined as a v-axis, a direction perpendicular to the v-axis and passing through the center of the HMD 110 as a w-axis, and a direction orthogonal to the v-axis and the w-axis as a u-axis. The position sensor 130 and / or the HMD sensor 114 are angled about each uvw axis (ie yaw angle indicating rotation about v axis, pitch angle indicating rotation about u axis, w axis) The tilt determined by the roll angle indicating rotation is detected. The control device 120 determines angle information for controlling the visual axis of the virtual camera based on the detected angle change around each uvw axis.

  Next, the hardware configuration of the control device 120 will be described with reference to FIG. FIG. 4 is a diagram showing a hardware configuration of the control device 120. As shown in FIG. As shown in FIG. 4, the control device 120 includes a control unit 121, a storage unit 123, an I / O (input / output) interface 124, a communication interface 125, and a bus 126. The control unit 121, the storage unit 123, the I / O interface 124, and the communication interface 125 are communicably connected to one another via a bus 126.

  The control device 120 may be configured as a personal computer, a smartphone, a fablet, a tablet or a wearable device separately from the HMD 110, or may be built in the HMD 110. Moreover, while the one part function of the control apparatus 120 is mounted in HMD110, the remaining function of the control apparatus 120 may be mounted in another apparatus separate from HMD110.

  The control unit 121 includes a memory and a processor. The memory is configured of, for example, a ROM (Read Only Memory) storing various programs and the like, and a RAM (Random Access Memory) having a plurality of work areas storing various programs and the like executed by the processor. The processor is, for example, a central processing unit (CPU), a micro processing unit (MPU) and / or a graphics processing unit (GPU), and develops a program specified from various programs incorporated in the ROM on the RAM. The system is configured to execute various processes in cooperation with the

  In particular, the control unit 121 may control various operations of the control device 120 by the processor developing the control program on the RAM and executing the control program in cooperation with the RAM. The control unit 121 displays the view image on the display unit 112 of the HMD 110 based on the view image data. Thereby, the user U can immerse in the virtual space.

  The storage unit (storage) 123 is, for example, a storage device such as a hard disk drive (HDD), a solid state drive (SSD), or a USB flash memory, and is configured to store programs and various data. The storage unit 123 may store a control program for causing a computer to execute at least a part of the information processing method according to the present embodiment, or a control program for realizing sharing of a virtual space by a plurality of users. In addition, the storage unit 123 may store an authentication program of the user U and data regarding various images and objects (for example, avatars and the like). Furthermore, in the storage unit 123, a database including a table for managing various data may be constructed.

  The I / O interface 124 is configured to communicably connect the HMD 110, the position sensor 130, the external controller 320, the headphone 116, the face camera 117, and the microphone 118 to the control device 120, respectively. For example, it is configured by a USB (Universal Serial Bus) terminal, a DVI (Digital Visual Interface) terminal, an HDMI (registered trademark) (High-Definition Multimedia Interface) terminal, and the like. The control device 120 may be wirelessly connected to each of the HMD 110, the position sensor 130, the external controller 320, the headphone 116, the face camera 117, and the microphone 118.

  The communication interface 125 is configured to connect the control device 120 to a communication network 3 such as a Local Area Network (LAN), a Wide Area Network (WAN), or the Internet. The communication interface 125 includes various wired connection terminals for communicating with an external device such as the server 2 via the communication network 3 and various processing circuits for wireless connection, and communicates via the communication network 3. Are configured to meet the communication standards of

  Next, an example of a specific configuration of the external controller 320 will be described with reference to FIG. The external controller 320 detects the movement of a part of the user U's body (a part other than the head, and in this embodiment, the user U's hand) to detect the movement of the hand object displayed in the virtual space. Used to control the The external controller 320 is an external controller 320R for the right operated by the user U's right hand (hereinafter simply referred to as the controller 320R) and an external controller 320L for the left hand operated by the left hand of the user U (hereinafter referred to simply as the controller 320L). And. The controller 320R is a device that indicates the position of the user U's right hand and the movement of the right hand's finger. Further, the right hand object 400R (see FIG. 16) existing in the virtual space moves in accordance with the movement of the controller 320R. The controller 320L is a device that indicates the position of the user U's left hand and the movement of fingers of the left hand. Also, in accordance with the movement of the controller 320L, the left hand object 400L (see FIG. 16) existing in the virtual space is moved. The controller 320R and the controller 320L have substantially the same configuration, so only the specific configuration of the controller 320R will be described below with reference to FIG. In the following description, the controllers 320L and 320R may be simply referred to as the controller 320 for convenience.

  As shown in FIG. 5, the controller 320R includes an operation button 302, a plurality of detection points 304, a sensor (not shown), and a transceiver (not shown). Only one of the detection point 304 and the sensor may be provided. The operation button 302 is configured of a plurality of button groups configured to receive an operation input from the user U. The operation button 302 includes a push button, a trigger button, and an analog stick. The push-type button is a button operated by an operation of pressing with the thumb. For example, on the top surface 322, two push-type buttons 302a and 302b are provided. The trigger button is a button operated by an operation such as pulling a trigger with the index finger or the middle finger. For example, the front portion of the grip 324 is provided with a trigger button 302e, and the side portion of the grip 324 is provided with a trigger button 302f. The trigger buttons 302 e and 302 f are assumed to be operated by the index finger and the middle finger, respectively. The analog stick is a stick-type button that can be operated by tilting it in any direction 360 degrees from a predetermined neutral position. For example, it is assumed that the analog stick 320i is provided on the top surface 322 and operated using the thumb.

  The controller 320R includes a frame 326 extending from both sides of the grip 324 in the direction opposite to the top surface 322 to form a semicircular ring. A plurality of sensing points 304 are embedded in the outer surface of the frame 326. The plurality of detection points 304 are, for example, a plurality of infrared LEDs aligned in a row along the circumferential direction of the frame 326. After the position sensor 130 detects information on the positions, inclinations, or light emission intensities of the plurality of detection points 304, the control device 120 detects the position or attitude of the controller 320R based on the information detected by the position sensor 130. Get information on

  The sensor of the controller 320R may be, for example, either a magnetic sensor, an angular velocity sensor, or an acceleration sensor, or a combination thereof. When the user U moves the controller 320R, the sensor outputs a signal (for example, a signal indicating information on magnetism, angular velocity, or acceleration) according to the direction or movement of the controller 320R. The control device 120 acquires information on the position and orientation of the controller 320R based on the signal output from the sensor.

  The transceivers of controller 320R are configured to transmit and receive data between controller 320R and controller 120. For example, the transceiver may transmit an operation signal corresponding to the operation input of the user U to the controller 120. The transceiver may also receive an instruction signal from the controller 120 instructing the controller 320R to emit light at the detection point 304. Additionally, the transceiver may transmit a signal to controller 120 indicating the value detected by the sensor.

  Next, processing for displaying a view image on the HMD 110 will be described with reference to FIGS. 6 to 9. FIG. 6 is a flowchart showing a process of displaying a view image on the HMD 110. FIG. 7 is an xyz space diagram showing an example of the virtual space 200. The state (a) of FIG. 8 is a yx plan view of the virtual space 200 shown in FIG. The state (b) of FIG. 8 is a zx plan view of the virtual space 200 shown in FIG. FIG. 9 is a view showing an example of the view image V displayed on the HMD 110. As shown in FIG.

  As shown in FIG. 6, in step S1, the control unit 121 (see FIG. 4) generates virtual space data indicating a virtual space 200 including the virtual camera 300 and various objects. As shown in FIG. 7, the virtual space 200 is defined as an omnidirectional sphere centered on the center position 210 (in FIG. 7, only the upper half celestial sphere is shown). In the virtual space 200, an xyz coordinate system having the center position 210 as an origin is set. The virtual camera 300 defines a viewing axis L for specifying a view image V (see FIG. 9) displayed on the HMD 110. The uvw coordinate system defining the field of view of the virtual camera 300 is determined to interlock with the uvw coordinate system defined around the head of the user U in the real space. The control unit 121 may move the virtual camera 300 in the virtual space 200 in conjunction with the movement of the user U wearing the HMD 110 in the real space.

  Next, in step S2, the control unit 121 specifies the field of view CV (see FIG. 8) of the virtual camera 300. Specifically, based on the data indicating the state of the HMD 110 transmitted from the position sensor 130 and / or the HMD sensor 114, the control unit 121 acquires information on the position and the inclination of the HMD 110. Next, the control unit 121 specifies the position and the orientation of the virtual camera 300 in the virtual space 200 based on the information on the position and the inclination of the HMD 110. Next, the control unit 121 determines the visual axis L of the virtual camera 300 from the position and the orientation of the virtual camera 300, and identifies the visual field CV of the virtual camera 300 from the determined visual axis L. The field of view CV of the virtual camera 300 corresponds to a partial area of the virtual space 200 visible to the user U wearing the HMD 110 (in other words, corresponds to a partial area of the virtual space 200 displayed on the HMD 110 ). The visual field CV is a first area CVa set as an angular range of the polar angle α centered on the visual axis L in the xy plane shown in the state (a) of FIG. 8 and xz shown in the state (b) in FIG. In the plane, it has a second region CVb set as an angular range of the azimuth angle β centered on the visual axis L. The control unit 121 identifies the line of sight of the user U based on the data indicating the line of sight of the user U transmitted from the gaze sensor 140, and based on the identified line of sight of the user U and information regarding the position and inclination of the HMD 110, The orientation of the camera 300 (the visual axis L of the virtual camera) may be determined. Further, as described later, the control unit 121 may determine the orientation of the face of the avatar of the user U based on the information on the position and the inclination of the HMD 110.

  Thus, the control unit 121 can specify the field of view CV of the virtual camera 300 based on the data from the position sensor 130 and / or the HMD sensor 114. When the user U wearing the HMD 110 moves, the control unit 121 can update the field of view CV of the virtual camera 300 based on the data indicating the movement of the HMD 110 transmitted from the position sensor 130 and / or the HMD sensor 114. . That is, the control unit 121 can update the visual field CV according to the movement of the HMD 110. Similarly, when the line of sight of the user U changes, the control unit 121 may update the field of view CV of the virtual camera 300 based on the data indicating the line of sight of the user U transmitted from the gaze sensor 140. That is, the control unit 121 may change the visual field CV in accordance with the change in the line of sight of the user U.

  Next, in step S3, the control unit 121 generates view image data indicating the view image V displayed on the display unit 112 of the HMD 110. Specifically, the control unit 121 generates view image data based on virtual space data defining the virtual space 200 and the view CV of the virtual camera 300.

  Next, in step S4, the control unit 121 displays the view image V on the display unit 112 of the HMD 110 based on the view image data (see FIG. 9). Thus, according to the movement of the user U wearing the HMD 110, the visual field CV of the virtual camera 300 changes, and the visual field image V displayed on the display unit 112 of the HMD 110 changes. Can be immersed in 200.

  The virtual camera 300 may include a left-eye virtual camera and a right-eye virtual camera. In this case, the control unit 121 generates left-eye view image data indicating a left-eye view image based on the virtual space data and the view of the left-eye virtual camera. Further, the control unit 121 generates right-eye view image data indicating a right-eye view image based on the virtual space data and the view of the right-eye virtual camera. Thereafter, the control unit 121 displays the left-eye view image on the left-eye display unit based on the left-eye view image data, and the right-eye view image on the right-eye display unit based on the right-eye view image data. Display In this manner, the user U can visually recognize the view image three-dimensionally by the parallax between the left-eye view image and the right-eye view image. The virtual camera may be placed at the eye position of the avatar operated by the user. For example, the left-eye virtual camera may be placed on the avatar's left eye, while the right-eye virtual camera may be placed on the avatar's right eye.

  The processes in steps S1 to S4 shown in FIG. 6 may be performed for each frame. For example, when the frame rate of a moving image is 90 fps, the processes of steps S1 to S4 may be repeatedly performed at an interval of ΔT = 1/90 (seconds). As described above, since the processes of steps S1 to S4 are repeatedly executed at predetermined intervals, the visual field V of the virtual camera 300 is updated according to the operation of the HMD 110, and the visual field image V displayed on the display unit 112 of the HMD 110. Is updated.

  Next, the hardware configuration of the server 2 shown in FIG. 1 will be described with reference to FIG. FIG. 10 is a diagram showing a hardware configuration of the server 2. As shown in FIG. 10, the server 2 includes a control unit 23, a storage unit 22, a communication interface 21, and a bus 24. The control unit 23, the storage unit 22, and the communication interface 21 are communicably connected to each other via the bus 24. The control unit 23 includes a memory and a processor, and the memory includes, for example, a ROM and a RAM, and the processor includes, for example, a CPU, an MPU, and / or a GPU.

  The storage unit (storage) 22 is, for example, a large capacity HDD or the like. The storage unit 22 may store a control program for causing a computer to execute at least a part of the information processing method according to the present embodiment, or a control program for realizing sharing of a virtual space by a plurality of users. In addition, the storage unit 22 may store user management information for managing each user, data regarding various images and objects (for example, hand objects and the like). The communication interface 21 is configured to connect the server 2 to the communication network 3.

  The server 2 can provide a moving image distribution service to the user terminal 1A, 1B or a terminal other than the user terminal 1A, 1B (for example, a personal computer or a tablet terminal) via an application executed on a web browser. . The storage unit 22 may also store an application for providing a moving image distribution service. The application stored in the storage unit 22 is distributed to the terminal device in response to the distribution request. The terminal device can also download the application from a server other than the server 2 (such as a server providing an application market) or the like.

  As described above, the server 2 can manage a website for providing a moving image delivery service, and can deliver a web page (HTML data) constituting the website in response to a request from a terminal device. The server 2 can transmit and receive various data (including data necessary for screen display) necessary for providing a moving image distribution service with a terminal device. In addition, the server 2 can store various data for each identification information (for example, user ID) for identifying each user, and can manage the provision status of the moving image distribution service for each user. Although detailed description is omitted, the server 2 can also have a function of performing user authentication processing, charging processing, and the like.

[Description on motion control between avatars]
Next, with reference to FIGS. 1, 11 and 12, an example of processing for synchronously controlling the movement of each avatar 4A, 4B between the user terminal 1A and the user terminal 1B will be described. The state (a) of FIG. 11 is a diagram showing a virtual space 200A provided to the user A. The state (b) of FIG. 11 is a diagram showing a virtual space 200B provided to the user B. FIG. 12 is a flowchart for explaining an example of processing for synchronizing the movements of the avatars 4A and 4B between the user terminal 1A and the user terminal 1B. In this description, as a precondition, as shown in FIG. 11, the avatar 4A associated with the user terminal 1A (user A) and the avatar 4B associated with the user terminal 1B (user B) share the same virtual space It shall be done. That is, it is assumed that the user A and the user B share one virtual space via the communication network 3.

  As shown in the state (a) of FIG. 11, the virtual space 200A of the user A includes an avatar 4A and an avatar 4B. The avatar 4A is operated by the user A and interlocked with the operation of the user A. Avatar 4A is interlocked with the left hand interlocked with the operation of controller 320L of user terminal 1A (the operation of the left hand of user A), the right hand interlocked with the operation of controller 320R of user terminal 1A, and the expression is interlocked with the facial expression of user A Have a face to The face of the avatar 4A has a plurality of face parts (eg, eyes, eyebrows, mouth, etc.). The avatar 4B is operated by the user B and interlocked with the operation of the user B. The avatar 4B has a left hand interlocked with the operation of the controller 320L of the user terminal 1B indicating the operation of the left hand of the user B, a right hand interlocked with the operation of the controller 320R of the user terminal 1B indicating the operation of the right hand of the user B, and an expression It has a face interlocked with the facial expression of the user B's face. The face of the avatar 4B has a plurality of face parts (for example, eyes, eyebrows, mouth, etc.).

  Further, the positions of the avatars 4A and 4B may be specified according to the positions of the HMDs 110 of the user terminals 1A and 1B. Similarly, the orientation of the face of the avatars 4A and 4B may be specified according to the inclination of the HMD 110 of the user terminals 1A and 1B. Furthermore, according to the operation of the external controller 320 of the user terminal 1A, 1B, the operation of the hand of the avatars 4A, 4B may be specified. Further, the facial expressions of the avatars 4A and 4B may be specified according to the facial expressions (states) of the users A and B. In particular, the positions of the black eyes with respect to the white eyes of the avatars 4A and 4B may be specified according to the positions of the black eyes with respect to the white eyes of the users A and B.

  As shown in the state (b) of FIG. 11, the virtual space 200B of the user B includes an avatar 4A and an avatar 4B. The positions of the avatars 4A and 4B in the virtual space 200A may correspond to the positions of the avatars 4A and 4B in the virtual space 200B.

  Next, referring to FIG. 12, in step S <b> 10, the control unit 121 of the user terminal 1 </ b> A generates voice data of the user A. For example, when the user A inputs a voice into the microphone 118 (voice input unit) of the user terminal 1A, the microphone 118 generates voice data indicating the input voice. Thereafter, the microphone 118 transmits the generated audio data to the control unit 121 via the I / O interface 124.

  Next, in step S11, the control unit 121 of the user terminal 1A generates control information of the avatar 4A, and then generates control data of the generated avatar 4A and voice data of the voice of the user A (voice data of the user A ) To the server 2. Thereafter, the control unit 23 of the server 2 receives the control information of the avatar 4A and the voice data of the user A from the user terminal 1A (step S12).

  Here, the control information of the avatar 4A is information necessary to control the operation of the avatar 4A. The control information of the avatar 4A includes information on the position of the avatar 4A (position information), information on the orientation of the face of the avatar 4A (face direction information), and information on the state of the hand (left hand and right hand) of the avatar 4A (hand information And information on the facial expression of the avatar 4A (face information) may be included.

  The face information of the avatar 4A has information indicating the state of a plurality of face parts. The information indicating the state of a plurality of face parts is information (eye information) indicating the eye state (the shape of the white eye and the position of the black eye relative to the white eye) of the avatar 4A, and the information (the position and shape of the eyebrow of the avatar 4A) Etc.) and information (mouth information) indicating the state of the mouth of the avatar 4A (position and shape of the mouth, etc.).

  Specifically, the eye information of the avatar 4A includes information indicating the state of the left eye of the avatar 4A and information indicating the state of the right eye of the avatar 4A. The eyelash information of the avatar 4A includes information indicating the left eyebrow state of the avatar 4A and information indicating the right eyebrow state of the avatar 4A.

  The control unit 121 of the user terminal 1A acquires an image showing the eyes of the user A (left eye and right eye) and eyebrows (left eyebrow and right eyebrow) from the face camera 113 mounted on the HMD 110, and then the acquired image And the state of the eye of the user A (left eye and right eye) and the eyebrow (left eyebrow and right eyebrow) based on a predetermined image processing algorithm. Next, the control unit 121 controls the information indicating the state of the eyes (left and right eyes) of the avatar 4A and the eyebrows (left and right eyelashes) of the avatar 4A based on the identified states of the user A's eyes and eyebrows. Each generates information indicating the status. Note that among the information indicating the state of the eye of the user A, the information indicating the position (center position) of the black eye of the user A may be acquired by the gaze sensor 140 instead of the face camera 113.

  Similarly, the control unit 121 of the user terminal 1A acquires an image showing the mouth of the user A and the periphery thereof from the face camera 117, and then the mouth of the user A based on the acquired image and a predetermined image processing algorithm. Identify the state of Next, the control unit 121 generates information indicating the state of the mouth of the avatar 4A based on the specified state of the user A's mouth.

  Thus, the control unit 121 of the user terminal 1A includes the eye information of the avatar 4A corresponding to the eyes of the user A, the eyebrow information of the avatar 4A corresponding to the eyelash of the user A, and the avatar 4A corresponding to the mouth of the user A. It is possible to generate mouth information of

  Further, the control unit 121 controls the image captured by the face camera 113, the image captured by the face camera 117, and a plurality of types of faces stored in the storage unit 123 or the memory based on a predetermined image processing algorithm. The facial expression (for example, smile) of the user A may be specified from among the facial expressions (for example, smile, sad expression, expressionlessness, angry expression, surprised expression, troubled expression, etc.). After that, the control unit 121 can generate face information indicating the facial expression (for example, smile) of the face of the avatar 4A based on the specified facial expression of the user A. In this case, the storage unit 123 stores an expression database including facial expressions of a plurality of types of faces of the avatar 4A and face information of a plurality of avatars 4A each associated with one of the facial expressions of the plurality of types. May be

  For example, when the control unit 121 specifies that the facial expression of the face of the avatar 4A is a smile, the control unit 121 acquires face information indicating a smile. Here, the face information indicating the smile of the avatar 4A includes eye information, eyebrow information and mouth information of the avatar 4A when the facial expression of the face of the avatar 4A is a smile.

  Next, in step S13, the control unit 121 of the user terminal 1B generates control information of the avatar 4B, and then transmits control information of the generated avatar 4B to the server 2. Then, control part 23 of server 2 receives control information on avatar 4B from user terminal 1B (Step S14).

  Here, the control information of the avatar 4B is information necessary to control the operation of the avatar 4B. The control information of the avatar 4B includes information on the position of the avatar 4B (position information), information on the orientation of the face of the avatar 4B (face direction information), and information on the state of the hand (left hand and right hand) of the avatar 4B (hand information And information on the facial expression of the avatar 4B (face information) may be included.

  The face information of the avatar 4B has information indicating the states of a plurality of face parts. Information indicating the state of a plurality of face parts is information (eye information) indicating the eye state (the shape of the white eye and the position of the black eye with respect to the white eye) of the avatar 4B, and the information (the position and shape of the eyebrow of the avatar 4B) Etc.) and information (mouth information) indicating the state of the mouth of the avatar 4B (position and shape of the mouth, etc.). The method of acquiring face information of the avatar 4B is the same as the method of acquiring face information of the avatar 4A.

  Next, the server 2 transmits control information of the avatar 4B to the user terminal 1A (step S15), and transmits control information of the avatar 4A and voice data of the user A to the user terminal 1B (step S19). Thereafter, the control unit 121 of the user terminal 1A updates the states of the avatars 4A and 4B based on the control information of the avatars 4A and 4B after receiving the control information of the avatar 4B in step S16. The virtual space data indicating the state (see the state (a) of FIG. 11) is updated (step S17).

  Specifically, the control unit 121 of the user terminal 1A updates the positions of the avatars 4A and 4B based on the position information of the avatars 4A and 4B. The control unit 121 updates the face orientation of the avatars 4A and 4B based on the face orientation information of the avatars 4A and 4B. The control unit 121 updates the hands of the avatars 4A and 4B based on the hand information of the avatars 4A and 4B. The control unit 121 updates the facial expression of the avatars 4A and 4B based on the facial information of the avatars 4A and 4B. Thus, virtual space data indicating the virtual space 200A including the updated avatars 4A and 4B is updated.

  Thereafter, the control unit 121 of the user terminal 1A specifies the visual field CV of the avatar 4A (virtual camera 300) according to the position and the inclination of the HMD 110, and then updates the virtual space data of the avatar 4A and the visual field CV of the avatar 4A. Based on the result, the view image displayed on the HMD 110 is updated (step S18).

  On the other hand, after receiving the control information of avatar 4A and the voice data of user A at step S20, control unit 121 of user terminal 1B updates the states of avatars 4A and 4B based on the control information of avatars 4A and 4B. Above, virtual space data indicating the virtual space 200B (see the state (b) in FIG. 11) is updated (step S21).

  Specifically, the control unit 121 of the user terminal 1B updates the positions of the avatars 4A and 4B based on the position information of the avatars 4A and 4B. The control unit 121 updates the face orientation of the avatars 4A and 4B based on the face orientation information of the avatars 4A and 4B. The control unit 121 updates the hands of the avatars 4A and 4B based on the hand information of the avatars 4A and 4B. The control unit 121 updates the facial expression of the avatars 4A and 4B based on the facial information of the avatars 4A and 4B. Thus, virtual space data indicating the virtual space including the updated avatars 4A and 4B is updated.

  Thereafter, the control unit 121 of the user terminal 1B specifies the visual field CV of the avatar 4B (virtual camera 300) according to the position and the inclination of the HMD 110, and then updates the virtual space data and the visual field CV of the avatar 4B. Based on the result, the view image displayed on the HMD 110 is updated (step S22).

  Next, the control unit 121 of the user terminal 1B receives the received voice data of the user A, information regarding the position of the avatar 4A included in the control information of the avatar 4A, information regarding the position of the avatar 4B, and a predetermined voice processing algorithm Process the voice data of the user A based on. Thereafter, the control unit 121 transmits the processed audio data to the headphone 116 (audio output unit), and the headphone 116 outputs the audio of the user A based on the processed audio data (step S23). Thus, voice chat (VR chat) between users (between avatars) can be realized on the virtual space.

  In the present embodiment, after the user terminals 1A and 1B transmit the control information of the avatar 4A and the control information of the avatar 4B to the server 2, respectively, the server 2 transmits the control information of the avatar 4A to the user terminal 1B. Control information is transmitted to the user terminal 1A. As described above, it is possible to synchronize the motions of the avatars 4A and 4B between the user terminal 1A and the user terminal 1B.

[Description of Information Processing Method According to First Embodiment]
Next, the information processing method according to the present embodiment will be described with reference to FIGS. 7 and 13 to 16. 13 and 14 are flowcharts for explaining the information processing method according to the present embodiment. FIG. 15 is a schematic view showing a method of updating the evaluation value of each partial area of the virtual space 200. As shown in FIG. In this embodiment, as shown in FIG. 15, the avatar 4A associated with the user terminal 1A (user A), the avatar 4B associated with the user terminal 1B (user B), and the user terminal 1C (user C) It is assumed that the associated avatar 4C shares the same virtual space 200. That is, it is assumed that the user A, the user B, and the user C share one virtual space 200 via the communication network 3. Although illustration is omitted, the virtual camera 300 corresponding to the HMD 110 of the user terminal 1A is arranged in association with the avatar 4A in the virtual space 200, and the virtual camera 300 corresponding to the HMD 110 of the user terminal 1B is a virtual space The virtual camera 300 corresponding to the HMD 110 of the user terminal 1C is arranged in association with the avatar 4B in 200, and is arranged in association with the avatar 4C in the virtual space 200. That is, the position and / or orientation of each avatar 4A, 4B, 4C is changed based on the movement of the HMD 110 of each user terminal 1A, 1B, 1C.

  In FIG. 7, for convenience of explanation, a plurality of parallel lines lo and meridians la are omitted on the front side (−z side) of the surface S of the virtual space 200, while the back side of the surface S of the virtual space 200 is omitted. On the (+ Z side), a plurality of parallel lines lo and meridians la and a plurality of partial areas R are illustrated. In FIG. 15, illustration of the parallel lines lo, the meridians la, and the plurality of partial areas R on the surface of the virtual space 200 is omitted.

  As shown in FIG. 13, in step S <b> 100, the control unit 23 (see FIG. 10) of the server 2 generates virtual space data indicating the virtual space 200. Specifically, control unit 23 reads virtual experience content (360-degree content) stored in storage unit 22 in accordance with input signals from user terminals 1A, 1B and 1C, and the virtual space or all-sky image Virtual space data including a 360 degree space image such as The 360 degree content may include a 360 degree moving image. The 360-degree content may include, for example, different contents in a plurality of regions of 360-degree spatial video. Thus, each user A, B, C sharing the virtual space 200 views various areas in the virtual space 200 and recommends other user's attention points to the avatars 4A, 4B, 4C, etc. You can enjoy communication in

  Next, in step S102, the control unit 23 generates a plurality of surfaces S of the virtual space 200 by using a plurality of longitude lines lo arranged at predetermined angular intervals and a plurality of meridians la arranged at predetermined angular intervals. Divide into partial regions R. In other words, the control unit 23 sets a plurality of partial regions R on the surface S by providing a grid composed of the latitude line lo and the longitude line la on the surface S of the hemispherical virtual space 200 (see FIG. 7) .

  The number of parallel lines lo is set in accordance with the angular interval of the adjacent parallel lines lo. For example, when the angular interval of adjacent latitudes lo is 15 degrees, (90 degrees / 15 degrees) -1 = 5, and the number of latitudes lo is five. Similarly, the number of meridians la is set according to the angular spacing of adjacent meridians la. For example, when the angle interval of adjacent meridians la is 15 degrees, 360 degrees / 15 degrees = 24, and the number of meridians la is 24. The control unit 121 can appropriately set the number of latitudes lo and longitudes la according to the type of virtual space data.

  The number of partial regions R is determined according to the number of meridians la and the number of latitudes lo. For example, when the number of meridians la is n and the number of latitudes lo is m, the total number N of partial regions R is N = n × (m + 1) (where n ≧ 2).

  Next, in step S104, the control unit 23 of the server 2 transmits virtual space data in a state in which a plurality of partial areas R are set on the surface S of the virtual space 200 to each of the user terminals 1A, 1B, and 1C. Next, in step S106, the control unit 121 (see FIG. 4) of each user terminal 1A, 1B, 1C displays the virtual space data distributed from the control unit 23 of the server 2 on the display unit 112.

  Next, in step S108, the control unit 121 of each user terminal 1A, 1B, 1C specifies the position and orientation of the HMD 110. Specifically, the control unit 121 may specify the position and orientation of the HMD 110 by receiving data indicating the state of the HMD 110 from the position sensor 130 and / or the HMD sensor 114. Alternatively, the control unit 121 may obtain information on the position and tilt of the HMD 110 based on data indicating the state of the HMD 110 transmitted from the position sensor 130 and / or the HMD sensor 114.

  Next, in step S110, the control unit 121 of each user terminal 1A, 1B, 1C determines the position of each avatar 4A, 4B, 4C (virtual camera 300) or the like in the virtual space 200 based on the position and orientation of the HMD 110. While updating the direction, the position of the intersection C where the line of sight direction (visual axis L) of each avatar 4A, 4B, 4C intersects with the surface S of the virtual space 200 is determined. For example, the intersection C is determined based on the radius r of the virtual space 200 and the latitude θ and longitude φ of the visual axis L shown in FIG. Here, since the radius r of the virtual space 200 is a known value, the control unit 121 determines the intersection C based on the latitude θ and the longitude φ of the visual axis L. The visual axis L is determined based on the position of each avatar 4A, 4B, 4C and the direction of the line of sight of each avatar 4A, 4B, 4C.

  In this step S110, the control unit 121 of each user terminal 1A, 1B, 1C specifies the line of sight direction of each avatar 4A, 4B, 4C based on the position and orientation of the HMD 110, but instead, each user The gaze direction of each avatar 4A, 4B, 4C may be specified based on the gazes of A, B, C. In this case, the control unit 121 may determine that the line of sight of each of the users A, B, and C has changed by receiving information from the gaze sensor 140. Alternatively, the control unit 121 acquires information related to the line of sight of each of the users A, B, and C based on the information transmitted from the gaze sensor 140, and each of the users A, B, and C based on the acquired information. It may be determined whether the line of sight of has changed.

  Next, in step S112, the control unit 121 of each of the user terminals 1A, 1B, and 1C specifies a partial area including the intersection C among the plurality of partial areas R dividing the surface S of the virtual space 200. . Here, since the partial region R is set by a plurality of latitude lines lo and a plurality of meridians la, each partial region R has an angular range of latitude θ and longitude φ. Further, as described above, the intersection C is determined based on the latitude θ and the longitude φ of the visual axis L. Thus, by determining the latitude θ and the longitude φ of the visual axis L after movement, the control unit 121 can determine in which partial region R the intersection C is included without performing complicated calculations. it can.

  In addition, since the sight line direction (visual axis L) of each avatar 4A, 4B, 4C may change from time to time, in step S112, visual field information (each avatar 4A showing the partial area R including the intersection C at each time , 4B, and 4C are specified in association with time information defining time. The time information may be defined corresponding to the actual time, or may be defined as an elapsed time from the playback start time of the 360-degree content transmitted to each user terminal 1A, 1B, 1C as virtual space data .

  For example, the control unit 121 of the user terminal 1A specifies the partial area R (partial area R1 in FIG. 15) including the intersection CA where the sight line direction LA of the avatar 4A and the surface S of the virtual space 200 intersect with time information Do. Similarly, each control unit 121 of the user terminal 1B associates the partial area R (partial area R1 in FIG. 15) including the intersection CB where the line-of-sight direction LB of the avatar 4B and the surface S of the virtual space 200 intersect with the time information To identify. Similarly, each control unit 121 of the user terminal 1C associates the partial area R (partial area R3 in FIG. 15) including the intersection CC at which the sight line direction LC of the avatar 4C and the surface S of the virtual space 200 intersect with the time information. To identify.

  Next, in step S114, the control unit 121 of each user terminal 1A, 1B, 1C mixes the face information of each avatar 4A, 4B, 4C at each time, and the voice information of each avatar 4A, 4B, 4C at each time. To get As described above, the control unit 121 specifies the facial expression of each avatar 4A, 4B, 4C based on the image captured by the face camera 113 or the face camera 117, and based on the specified expression, the face information is get. Further, the control unit 121 receives, from the microphone 118, voice data generated based on voices uttered from the respective users A, B, and C, and acquires voice information based on the received voice data.

  Next, in step S116, the control unit 121 of each of the user terminals 1A, 1B, and 1C transmits the acquired time information, view information, face information, and voice information to the server 2. Next, in step S118, the server 2 updates the visual field information data stored in the storage unit 22. An example of visual field information data is shown below.

(Table 1)

  Here, the visual field information data shown in Table 1 includes time information, visual field information of each avatar 4A, 4B, 4C at each time, emotion information, and an emotion evaluation value. The emotion information is specified based on the face information and voice information of each avatar 4A, 4B, 4C. Emotion information may be identified based on at least one of face information and voice information. The emotion evaluation value is determined based on the emotion information of each avatar 4A, 4B, 4C and the attribute of the 360-degree content. For example, if the 360-degree content delivered to the virtual space 200 is an entertainment program (a comedy program, a quiz program, etc.) or an entertainment movie, a positive emotion evaluation value is obtained if the emotion information is a smile or surprise face. If the emotion information is an expressionless face or an angry face, a negative emotion evaluation value may be attached. On the other hand, if the 360-degree content is a serious drama or social film, a positive emotion evaluation value may be attached if the emotion information is a sad expression or a surprise face. Also, the emotion evaluation value may not be obtained in the case of specific emotion information (for example, expressionless). In this example, when the emotion information is a smile, the emotion evaluation value is “+1”, when the emotion information is a surprise face, the emotion evaluation value is “+2”, and when the emotion information is expressionless The emotion evaluation value is set to "-1".

  As shown in FIG. 15, for example, avatar 4A looks at partial area R1 between times T1 and T5 (the intersection CA is included in partial area R1 at times T1 to T5), and between times T1 and T5. Suppose that the face information of avatar 4A is expressionless. Further, it is assumed that the user A utters a voice “Not right” at a predetermined time (for example, time T2) between time T1 to T5. In this case, the control unit 23 of the server 2 that has received the time information and the view information from the control unit 121 of the user terminal 1A specifies the view information of the avatar 4A at each time T1 to T5 as the partial area R1. The visual field information of the avatar 4A at times T1 to T5 of the visual field information data shown is updated. Furthermore, based on the face information of the avatar 4A at each of the times T1 to T5 received from the control unit 121 of the user terminal 1A and the voice information of the user A at the time T2, the control unit 23 Identify the information. At this time, the control unit 23 can linguistically analyze the received voice information and adjust emotion information based on the analysis result of the voice information. In the case of the present example, for example, it is determined that the voice information "not right" is neither positive evaluation nor negative evaluation. Therefore, based on the face information (an expressionless face) of avatar 4A between times T1 to T5 and the voice information that is zero evaluation, control unit 121 "without expression" for emotion information of avatar 4A at times T1 to T5. And (see Table 1). Then, the control unit 23 updates the emotion evaluation value of the avatar 4A based on the specified emotion information and the attribute of the 360-degree content distributed to the virtual space 200. For example, when the 360-degree content distributed to the virtual space 200 is an entertainment program, the control unit 23 evaluates the emotion of the avatar 4A at times T1 to T5 based on that the emotion information of the avatar 4A is expressionless. Update the value to "-1".

  As shown in FIG. 15, for example, avatar 4B gazes at partial region R1 at times T1 and T2, and then, as shown in FIG. 16, avatar 4C (partial region R2) at times T3 and T4. Then, it is assumed that the partial region R1 is gazed again at time T5. It is assumed that the face information of the avatar 4B is a smile through time T1 to T5. In addition, it is assumed that the user B utters a voice “I am fine” at a predetermined time (for example, time T2) between time T1 to T5. In this case, the control unit 23 of the server 2 that has received the time information and the view information from the control unit 121 of the user terminal 1B identifies the view information of the avatar 4B at each of the times T1, T2 and T5 as the partial area R1. The visual field information of the avatar 4B at T3 and T4 is specified as the partial region R2, and the visual field information of the avatar 4B at times T1 to T5 of the visual field information data shown in Table 1 is updated. Furthermore, based on the face information of the avatar 4B at each of the times T1 to T5 received from the control unit 121 of the user terminal 1B and the voice information of the user B at the time T2, the control unit 23 Identify the information. In the case of this example, for example, it is determined that the voice information "Ii." Therefore, based on the face information (smile) of avatar 4B between times T1 and T5 and voice information that is a positive evaluation, control unit 121 specifies the emotion information of avatar 4B at times T1 to T5 as "smile". Yes (see Table 1). Then, the control unit 23 updates the emotion evaluation value of the avatar 4B based on the specified emotion information and the attribute of the 360-degree content distributed to the virtual space 200. Since the 360-degree content distributed to the virtual space 200 is an entertainment program, the control unit 23 sets the emotion evaluation value of the avatar 4B at times T1 to T5 based on the fact that the emotion information of the avatar 4B is a smile. Update "+1".

  If the avatar 4B is doing a specific gesture while laughing (for example, a gesture that covers the mouth of the avatar with a hand object or a gesture that strikes a hand object), it is determined that the degree of laughing is high. You may further increase the emotion evaluation value. That is, not only the face information and voice information of the avatar but also the gesture by the hand object may be added to specify the emotion. Also, depending on the type of gesture, the degree of smile may be differentiated. For example, when the face information of the avatar is a smile, for example, the control unit 23 determines that the degree of laughter is mild (little laughter) if there is no specific gesture, and further adds an emotion evaluation value. I do not do it. When a gesture is made to cover the mouth of the avatar with a hand object, the control unit 23 determines that the degree of laughter is medium (middle laughter), and further adds an emotion evaluation value. When a gesture is made to hit the hand object, the control unit 23 determines that the degree of laughter is high (big laughter), and adds an emotion evaluation value more than in the case of middle laughter.

  As shown in FIG. 15, for example, it is assumed that the avatar 4C gazes at a partial region R3 different from the partial region R1 or the partial region R2 during time T1 to T5. It is assumed that the face information of the avatar 4C at times T1 to T5 is a surprise face. In addition, it is assumed that the user C utters a voice “oh!” At a predetermined time (for example, time T2) between time T1 to T5. In this case, the control unit 23 of the server 2 that has received the time information and the view information from the control unit 121 of the user terminal 1C specifies the view information of the avatar 4C at each of the times T1 to T5 as the partial area R3. The visual field information of the avatar 4C at times T1 to T5 of the visual field information data shown is updated. Furthermore, based on the face information of the avatar 4C at each of the times T1 to T5 received from the control unit 121 of the user terminal 1C and the voice information of the user C at the time T2, the control unit 23 Identify the information. In the case of this example, for example, it is determined that the voice information "o!" Is a positive evaluation. Therefore, the emotion information of avatar 4C at time T1 to T5 is specified as "surprised face" based on the face information (surprised face) of avatar 4C between time T1 to T5 and the voice information that is positive evaluation (table 1). Then, the control unit 23 updates the emotion evaluation value of the avatar 4C based on the specified emotion information and the attribute of the 360-degree content distributed to the virtual space 200. Since the 360-degree content distributed to virtual space 200 is an entertainment program, control unit 23 determines that avatars at times T1 to T5 are based on the fact that the emotion information of avatar 4C at times T1 to T5 is a surprise face. Update the 4C emotion evaluation value to "+2". In addition, when avatar 4C is performing a specific gesture (for example, a gesture which spreads a both-hands object) while being surprised, it may be judged that the degree of the surprise is strong, and may further add a positive emotion evaluation value .

  Next, in step S120, the server 2 updates the evaluation value data of each partial area. Table 2 shows an example of evaluation value data.

(Table 2)

  The evaluation value data is data indicating an evaluation value of each partial area (for example, partial areas R1 and R3) at each time (for example, time T1 to T5). The evaluation value of each partial area includes an emotion total evaluation value, a communication evaluation value, and an overall evaluation value obtained by adding the emotion total evaluation value and the communication evaluation value. The emotion total evaluation value is an evaluation value obtained by adding up the emotion evaluation values of the plurality of avatars 4A, 4B, 4C at each time (time T1 to T5). The communication evaluation value is, for example, an evaluation value associated with the communication action made between the avatars 4A, 4B, 4C in the virtual space 200. The communication action may include, for example, a conversation made between the avatars 4A, 4B, 4C, and an action (such as a gesture) of each avatar 4A, 4B, 4C. In this example, particularly, when a conversation or a specific gesture is made to urge the avatar to gaze at a specific area of the content 360 degrees to another avatar, the evaluation on the partial area corresponding to the specific area is performed. Add the value.

FIG. 14 is a flowchart showing each step in step S120 for updating the evaluation value data of each partial region.
As shown in FIG. 14, first, in step S120-1, the control unit 23 of the server 2 adds up the emotion evaluation values of the plurality of avatars 4A, 4B, 4C at each time (time T1 to T5), Update the emotion total evaluation value for each partial area. Subsequently, in step S120-2, the control unit 23 determines whether or not a communication action has been taken between the avatars 4A, 4B, and 4C. When it is determined that the communication action is performed between the avatars 4A, 4B, 4C (Yes in step S120-2), in step S120-3, the control unit 23 determines information related to the communication action (hereinafter referred to as action information). Is received from each user terminal 1A, 1B, 1C. Then, in step S120-4, the control unit 23 updates the communication evaluation value based on the action information.

The communication action between the avatars 4A, 4B and 4C will be described below with reference to FIG.
As shown in FIG. 16, for example, while pointing at partial region R1 with right-hand object 400R during a period from time T3 to T4, avatar 4B turns to avatar 4C and speaks "that part was good." I suppose. At this time, the control unit 121 of the user terminal 1B transmits, to the server 2, conversation information "That part is good" and a pointing gesture (gesture information) by the right hand object 400R, together with time information. The control unit 23 of the server 2 language-analyzes the conversation information received from the user terminal 1B, and based on the analysis result of the conversation information and the gesture information, determines whether or not the communication action is performed. Here, the control unit 23 extracts various expressions included in the conversation information of the avatar 4A, for example, a linguistic expression (instruction word) pointing to a specific region or a temporal expression, and analyzes the conversation information. For example, in the case of conversation information "that part was good.", The control unit 23 determines that the instruction word "that part" indicates the partial region R1 based on the gesture information of the avatar 4B. . Then, since the wording “It was good” includes a tense expression in the past form, the control unit 23 determines that the communication action associated with the partial region R1 is performed on the avatar 4C from the avatar 4C. Then, the control unit 23 sets the communication evaluation value of the partial region R1 to, for example, “+2 " In addition, when the conversation information includes a tense expression in the present form or the future form, it can not always be said that an instruction word (for example, “that part”) included in the conversation information indicates the partial region R1 Therefore, a positive communication evaluation value may not be given to the partial region R.

  Further, as shown in FIG. 16, it is assumed that the avatar 4C, which has been told from the avatar 4B that "the part was good", responds to the avatar 4B as "yes". In this case, the control unit 121 of the user terminal 1C transmits, to the server 2, the conversation information of "Yes" along with the view information of the avatar 4C. The control unit 23 of the server 2 language-analyzes the conversation information received from the user terminal 1C, and based on the analysis result and the visual field information, determines that the avatar 4C has uttered that it agrees to the utterance of the avatar 4B. Do. The utterance of the avatar 4C does not include an instruction word indicating a specific partial region, and since the tense is in the present form, it is determined that the utterance does not increase the communication evaluation value of the partial region. Therefore, the control unit 23 does not attach a communication evaluation value to the response "that's right." Of the avatar 4C. If the utterance of avatar 4C includes an instruction word indicating a specific partial area and the tense is in the past tense, it is an utterance that raises the communication evaluation value of the specific partial area. It can be evaluated.

  Next, in step S120-5, the control unit 23 of the server 2 adds up the emotion total evaluation value identified as described above and the communication evaluation value, and updates the comprehensive evaluation value. For example, as shown in Table 2, the comprehensive evaluation value at time T1 and T2 of partial region R1 is "0", and the comprehensive evaluation value at time T3 and T4 of partial region R1 is "+1", and the time of partial region R1 is The comprehensive evaluation value at T5 is "+3".

  Returning to FIG. 13, next, in step S122, the control unit 23 of the server 2 determines whether there is a partial region in which the comprehensive evaluation value is equal to or greater than a threshold (for example, “+3”). If it is determined that there is no partial area in which the comprehensive evaluation value is equal to or greater than the threshold (No in step S122), the processes after step S108 are repeated. On the other hand, when it is determined that there is a partial area in which the comprehensive evaluation value is equal to or more than the threshold (Yes in step S122), the control unit 23 determines the partial area in which the comprehensive evaluation value is equal to or more than the threshold in step S124. Designate as an attention area. For example, in the example shown in Table 2, since the comprehensive evaluation value of partial region R1 at time T5 is "+4", control unit 23 designates partial region R1 at time T5 as a region of interest.

  Next, in step S126, the control unit 23 of the server 2 displays an image (hereinafter referred to as an attention image) on the attention area in a predetermined time width (an example of a first time width) including the time when the comprehensive evaluation value becomes the threshold Generate). As the image of interest, for example, for the partial region R1, an image of 5 seconds before and after the time including the time T5 at which the comprehensive evaluation value is “+3” can be generated. As shown in FIG. 17, the video of interest is captured, for example, by imaging a region of interest R1 in the virtual space 200 with a virtual camera 300A for generating video of interest disposed at the center of the celestial sphere that makes up the virtual space 200. It may be generated. The avatars 4A, 4B, 4C and the virtual camera 300 associated with the avatars 4A, 4B, 4C are not necessarily located at the center of the celestial sphere that constitutes the virtual space 200, but at the center of the virtual space 200. By imaging the area of interest with the virtual camera 300 disposed and generating an image of interest, an appropriate image of interest can be generated from an easy-to-view angle. The video of interest may be generated as a three-dimensional video, or may be generated as a two-dimensional video. When a plurality of target videos are generated in step S126, the control unit 23 may generate a digest video content by connecting the plurality of target videos.

  Next, in step S128, the control unit 23 of the server 2 distributes the generated digest video content (or video of interest) to the moving picture distribution service on the web browser through the network 3. At this time, the control unit 23 collects moving image viewing information of many users, refers to the viewing history information of each user, and the history data regarding the attention area (for example, partial area R1) is an attribute of the digest video content of interest. A user terminal (user ID) similar to the above may be specified, and the digest video content may be distributed to the user terminal. For example, the control unit 23 estimates the user attribute of each user from the collected viewing history information, and provides the digest video content to the user whose user attribute is close to the attribute of the target digest video content. The digest video content is, for example, of social networking services (eg, LINE (registered trademark), Facebook (FACEBOOK (registered trademark)), Twitter (TWITTER (registered trademark))) registered by each user terminal. Delivered to the timeline. In addition, as described above, the control unit 23 of the server 2 picks up a user who may be interested in the digest video content of interest from a large number of users, and recommends the digest video content to the user. It can enhance the advertising effectiveness of 360-degree content. Also, when the user A, B, C who has finished viewing the content 360 degrees after mounting the HMD 110 removes the HMD 110 and then mounting the HMD 110 again, and wants to start viewing another 360 degree content, The control unit 23 of the server 2 sets digest videos of various 360-degree contents according to the interest of each of the users A, B, and C as recommendation videos based on the history information of the attention areas of the respective users A, B, and C. You may make it display on the display part 112 of HMD110 which each user A, B, C mounts | wears.

  As described above, according to the present embodiment, the control unit 23 of the server 2 specifies the emotions of the users A, B, and C wearing the HMD 110, and the specified emotions are predetermined conditions (first When an example of the condition is satisfied, the evaluation value of the area (an example of the first area) in the virtual space 200 according to the direction of the HMD 110 at the time when the condition is satisfied is updated. Thereby, the time when the specific emotion of the users A, B, C is detected is specified as the excitement time of the 360 degree content, and the direction (view) of the HMD 110 worn by each of the users A, B, C at that time is identified. The evaluation value of the attention area can be updated accordingly. Therefore, it is possible to appropriately specify a focused area (focused range) of the 360-degree content in the virtual space 200 without missing the true excitement point of the 360-degree content.

  Further, when the emotions of the users A, B, and C satisfy the first condition, the control unit 23 determines that the communication behavior of the users A, B, and C satisfies a predetermined condition (an example of the second condition). The evaluation value may be further updated. In addition to the emotions of the users A, B, and C, the accuracy of specifying the attention range can be increased by adjusting the importance (degree of attention) of the attention region based on the communication actions of the users A, B, and C. it can.

  The condition regarding the communication behavior of the users A, B, and C may include that the plurality of avatars 4A, 4B, and 4C have a conversation in the virtual space 200. In order to adjust the importance of the attention area, it is preferable to use a conversation made between the plurality of avatars 4A, 4B and 4C. In addition, users A, B, and C can also talk directly with each other without the avatars 4A, 4B, and 4C arranged in the virtual space 200. In this case, the communication behavior between the users A, B, and C is It may be used to update the evaluation value. Alternatively, the users A, B, and C can talk directly with one another by calling the real space from the virtual space 200, which is a VR space.

  If conversations made between avatars 4A, 4B, 4C or between users A, B, C include a linguistic expression pointing to a specific partial region, the evaluation points may be further updated. In addition, when a conversation made between avatars 4A, 4B, 4C or between users A, B, C includes a tense expression indicating time, the evaluation point may be further updated. As described above, by using various expressions in the conversation, it is possible to easily adjust the degree of importance of the region of interest.

  The communication actions of the users A, B and C used to update the evaluation value are not limited to the conversations made between the avatars 4A, 4B and 4C or between the users A, B and C. For example, when the users A, B, and C take a picture (moving image) of a specific partial area in the virtual space 200, the evaluation value of the photographed area may be updated. Furthermore, when the user A, B, C takes a picture of a specific area (for example, partial area R1) in the virtual space 200 and posts the picture to a social networking site (SNS), the area (partial area R1) The evaluation value of) may be updated. As described above, it is also possible to use photography in the virtual space 200 or posting to a SNS as communication action for adjusting the degree of importance of the attention area.

  In the above embodiment, the evaluation value of a specific partial area is updated based on the visual field information and face information of avatars 4A, 4B, 4C, and additionally, avatars 4A, 4B, 4C or user A, Although the evaluation value of each partial region R is further updated based on the communication behavior between B and C, the present invention is not limited to this example. For example, based on only communication behavior between avatars 4A, 4B, 4C or users A, B, C, without updating the evaluation value of each partial region R based on face information of avatars 4A, 4B, 4C. The evaluation value of each partial region R may be updated.

  Although the embodiments of the present disclosure have been described above, the technical scope of the present invention should not be construed as limited by the description of the present embodiments. It is understood by those skilled in the art that the present embodiment is an example, and that modifications of various embodiments are possible within the scope of the invention described in the claims. The technical scope of the present invention should be determined based on the scope of the invention described in the claims and the scope of equivalents thereof.

  A display control program for causing a computer (processor) to execute the display control method according to the present embodiment is incorporated in advance in the storage unit 123 or the ROM in order to realize various processes executed by the control unit 121 by software. It is also good. Alternatively, the display control program may be a magnetic disk (HDD, floppy disk), an optical disk (CD-ROM, DVD-ROM, Blu-ray disk etc.), a magneto-optical disk (MO etc.), a flash memory (SD card, USB memory, etc.) It may be stored in a computer readable storage medium such as an SSD). In this case, the storage medium is connected to the control device 120, whereby the program stored in the storage medium is incorporated into the storage unit 123. Then, the display control program incorporated in the storage unit 123 is loaded onto the RAM, and the processor executes the loaded program, whereby the control unit 121 executes the display control method according to the present embodiment.

  The display control program may be downloaded from a computer on the communication network 3 via the communication interface 125. Also in this case, the downloaded program is incorporated in the storage unit 123.

  Although the description of the present embodiment assumes that virtual space data indicating the virtual space 200 is updated on the user terminal 1 side, the virtual space data may be updated on the server 2 side. Furthermore, although it is premised that the view image data corresponding to the view image is updated on the user terminal 1 side, the view image data may be updated on the server 2 side. In this case, the user terminal 1 displays the view image on the HMD 110 based on the view image data transmitted from the server 2.

  Further, even if a control program for causing a computer (processor) to execute various processes is incorporated in advance in the storage unit 123 or the memory in order to realize various processes executed by the control unit 121 of the user terminal 1 by software. Good. Alternatively, the control program may be a magnetic disk (HDD, floppy disk), an optical disk (CD-ROM, DVD-ROM, Blu-ray (registered trademark) disk, etc.), a magneto-optical disk (MO, etc.), a flash memory (SD card, It may be stored in a computer readable storage medium such as a USB memory, an SSD, etc. In this case, by connecting the storage medium to the control device 120, the control program stored in the storage medium is incorporated in the storage unit 123. Then, the control program incorporated in the storage unit 123 is loaded onto the RAM, and the processor executes the loaded program, whereby the control unit 121 executes various processing.

  Also, the control program may be downloaded from a computer on the communication network 3 via the communication interface 125. Also in this case, the downloaded control program is incorporated in the storage unit 123.

  In this embodiment, the virtual space is used to provide the user with a virtual experience such as VR (Virtual Reality), AR (Argumented Reality) and MR (Mixed Reality). When the virtual space provides a VR, background data stored in memory is used as the background of the virtual space. When virtual space provides AR or MR, real space is used as the background. In this case, real space can be used as a background by providing the HMD 110 with a transmissive display (optical see-through or video see-through display). If a virtual space is applied to the MR, the object may be influenced by the real space. In this manner, the virtual space includes at least a virtual scene such as a background or a virtual object, thereby providing the user with a virtual experience capable of interacting with the virtual scene. Also, when virtual space is applied to AR or MR, the user's hand may be used instead of the hand object. In this case, the user's left hand is placed in the virtual space 200 instead of the left hand object 400L shown in FIG. 16, and the user's right hand is placed in the virtual space 200 instead of the right hand object 400R. Then, the server 2 can determine whether the communication action has been performed based on the gesture by the user's hand.

1 (1A to 1C): user terminal (an example of a terminal device)
2: Server 3: Communication network 4A to 4C: Avatar 110: Head mounted device (HMD)
112: display unit 114: sensor 116: headphone 118: microphone 120: control device 121: control unit 123: storage unit (storage)
124: I / O interface 125: communication interface 130: position sensor 140: gaze sensor 200, 200A: virtual space 300: virtual camera 320: external controller 400L: left hand object 400R: right hand object C: intersection R (R1 to R3): Partial area

Claims (13)

An information processing method in a system comprising a head mounted device, comprising:
(A) generating virtual space data defining a virtual space;
(B) generating view image data indicating a view image to be displayed on the head mounted device based on the movement of the head mounted device and the virtual space data;
(C) identifying an emotion of a user wearing the head mount device;
(D) evaluation of the first region in the virtual space according to the orientation of the head mounted device at the time when the specified first condition is satisfied when the specified emotion satisfies the first predetermined condition Updating the values,
Information processing methods, including:   (E) The information processing method according to claim 1, further comprising the step of further updating the evaluation value when the action of the user satisfies the second condition after satisfying the first condition.   The information processing method according to claim 2, wherein the second condition includes that a conversation is made between a plurality of users in the virtual space.   The information processing method according to claim 3, wherein the conversation includes a linguistic expression pointing to the first area.   The information processing method according to claim 3, wherein the conversation includes a tense expression pointing to the time.   The information processing method according to any one of claims 2 to 5, wherein the second condition includes the user taking a picture of the first area in the virtual space.   The information processing method according to claim 6, wherein the second condition includes an action of the user posting a photograph of the first area taken in the virtual space to a social networking site. (F) designating the first region at the time as a region of interest when the evaluation value satisfies a third condition;
(G) generating an image relating to the region of interest over a first duration including the time of day;
The information processing method according to any one of claims 1 to 7, further comprising   The video is generated based on a position of the attention area in the virtual space, and a virtual camera disposed in the virtual space to define the view image based on the movement of the head mounted device. An information processing method according to Item 8. An information processing method in a system comprising a head mounted device, comprising:
(A ') generating virtual space data defining a virtual space;
(B ') generating, based on the movement of the head mounted device and the virtual space data, visual field image data indicating a visual field image displayed on the head mounted device;
(C ') identifying an action of a user wearing the head mount device;
(D ′) When the specified action satisfies a predetermined second condition, the first area in the virtual space according to the orientation of the head mount device at the time when the second condition is satisfied Updating the evaluation value;
Information processing methods, including:   An information processing program for causing a computer to execute the information processing method according to any one of claims 1 to 10.   An information processing system using a head mounted device, wherein the information processing system is configured to execute the information processing method according to any one of claims 1 to 10. A processor,
An information processing apparatus comprising: a memory for storing computer readable instructions;
An information processing apparatus, wherein the information processing apparatus executes the information processing method according to any one of claims 1 to 10 when the computer readable instruction is executed by the processor.

Images