JP6999538B2 - Information processing methods, information processing programs, information processing systems, and information processing equipment - Google Patents

Description

The present disclosure relates to information processing methods, information processing programs, information processing systems, and information processing devices.

Patent Document 1 discloses that the attention range of video content is determined based on the overlap of the visual field ranges of a plurality of users.

International Publication No. 2016/099864

In Patent Document 1, the attention range of the video content is determined based on the visual information of the user, but since the behavior of the user paying attention to the content is not reflected only in the visual information, the attention range in the content is specified. Has room for improvement.

An object of the present disclosure is to provide an information processing method, an information processing program, an information processing system, and an information processing apparatus capable of appropriately specifying the range of attention of the content.

According to one aspect of the present disclosure, the information processing method in a system provided with a head mount device is:
(A) Steps to generate virtual space data that defines virtual space, and
(B) A step of generating field image data indicating a field image displayed on the head mount device based on the movement of the head mount device and the virtual space data.
(C) A step of identifying the emotion of the user wearing the head mount device, and
(D) When the specified emotion satisfies the predetermined first condition, the evaluation of the first region in the virtual space according to the orientation of the head mount device at the time when the first condition is satisfied. Steps to update the value and
including.

According to the present disclosure, it is possible to provide an information processing method, an information processing program, an information processing system, and an information processing apparatus capable of appropriately specifying an attention range of contents.

It is a schematic diagram which shows the virtual experience content distribution system which concerns on embodiment of this invention (hereinafter, simply referred to as this embodiment). It is a schematic diagram which shows the mobile terminal which is a user terminal. It is a figure which shows the hardware configuration of a mobile terminal. It is a schematic diagram which shows the structure of the user terminal including the head mount device (HMD). It is a figure which shows the head of the user who wears an HMD. It is a figure which shows the hardware configuration of a control device. It is a flowchart which shows the process of displaying a field of view image on an HMD. It is an xyz space diagram which shows an example of a virtual space. (A) is a yx plan view of the virtual space shown in FIG. (B) is a zx plan view of the virtual space shown in FIG. It is a figure which shows an example of the field of view image displayed on the HMD. It is a figure which shows the hardware configuration of the server shown in FIG. It is a flowchart for demonstrating the operation of the virtual experience content distribution system which concerns on this embodiment. It is a flowchart for demonstrating the operation of the virtual experience content distribution system which concerns on this embodiment. It is a flowchart which shows each step in the step 120 (the step which updates the evaluation value data of each partial area) of FIG. It is a figure which shows the virtual space which constitutes the virtual experience content. It is a figure which shows another example of the virtual space which constitutes the virtual experience content. It is a schematic diagram which shows the arrangement composition of the virtual camera in the virtual space for generating a digest video.

[Explanation of Embodiments Shown in the Present Disclosure]
An outline of the embodiments shown in the present disclosure will be described.
(1) The information processing method in the system provided with the head mount device according to the embodiment of the present disclosure is
(A) Steps to generate virtual space data that defines virtual space, and
(B) A step of generating field image data indicating a field image displayed on the head mount device based on the movement of the head mount device and the virtual space data.
(C) A step of identifying the emotion of the user wearing the head mount device, and
(D) When the specified emotion satisfies the predetermined first condition, the evaluation of the first region in the virtual space according to the orientation of the head mount device at the time when the first condition is satisfied. Steps to update the value and
including.

According to this method, the attention area (attention range) of the content in the virtual space can be appropriately specified. That is, the time when a specific emotion of the user is detected is specified as the time of excitement of the content, and the evaluation point of the attention area is updated according to the direction (field of view) of the HMD at that time, so that the true excitement of the content is achieved. Never miss a point.

(2) (e) If the behavior of the user satisfies the second condition after the first condition is satisfied, the step of further updating the evaluation value may be further included.

By further taking into account the user's behavior after a specific emotion is detected and adjusting the importance (attention) of the attention area, it is possible to enhance the eligibility of the characteristic of the attention range.

(3) The second condition may include a conversation between a plurality of users in the virtual space.

Communication behavior among multiple users can be used to adjust the importance of the area of interest.

(4) The conversation may include a linguistic expression pointing to the first area.

(5) The conversation may include a tense expression pointing to the time.

By using the communication behavior between users with respect to the area of interest, the importance of the area of interest can be easily adjusted.

(6) The second condition may include the user taking a picture of the first area in the virtual space.

By using the shooting behavior for the region of interest, the importance of the region of interest can be easily adjusted.

(7) The second condition may include an action in which the user posts a photograph taken of the first area in the virtual space to a social networking site.

Posting behavior to SNS can be used to adjust the importance of the area of interest.

(8) (f) When the evaluation value satisfies the third condition, the step of designating the first region at the time as the region of interest, and
(G) A step of generating an image relating to the region of interest over the first time width including the time.
May be further included.

According to this method, it is possible to generate an advertisement video or the like using the region of interest.

(9) The image is generated based on the position of the area of interest in the virtual space and a virtual camera arranged in the virtual space to define the field image based on the movement of the head mount device. You may.

According to this method, an advertisement video or the like can be easily and appropriately generated.

(10) The information processing method in the system provided with the head mount device according to the embodiment of the present disclosure is
(A') A step to generate virtual space data that defines a virtual space, and
(B') A step of generating field image data indicating a field image displayed on the head mount device based on the movement of the head mount device and the virtual space data.
(C') A step of identifying the behavior of the user wearing the head mount device, and
(D') When the specified behavior satisfies the predetermined second condition, the first region in the virtual space according to the orientation of the head mount device at the time when the second condition is satisfied. Steps to update the evaluation value and
including.

According to this method, the area of interest of the content can be appropriately specified without missing the true excitement point of the content.

(11) The information processing program according to the embodiment of the present disclosure is an information processing program for causing a computer to execute the information processing method according to any one of (1) to (10).

According to the above, it is possible to provide an information processing program capable of appropriately specifying the attention range of the content.

(12) The information processing system using the head mount device according to the embodiment of the present disclosure is an information processing system configured to execute the information processing method according to any one of (1) to (10). Is.

According to the above, it is possible to provide an information processing system capable of appropriately specifying the attention range of the content.

(13) The information processing apparatus according to the embodiment of the present disclosure is
With the processor
An information processing device equipped with a memory for storing computer-readable instructions.
When the computer-readable instruction is executed by the processor, the information processing apparatus is an information processing apparatus that executes the information processing method according to any one of (1) to (10).

According to the above, it is possible to provide an information processing device capable of appropriately specifying the attention range of the content. Note that the information processing device is either a user terminal or a server.

[Details of Embodiments Shown in the Present Disclosure]
Hereinafter, embodiments shown in the present disclosure will be described with reference to the drawings. For convenience of explanation, the description of the element having the same reference number as the element already described in the description of the present embodiment will not be repeated.

First, the outline of the configuration of the virtual space distribution system 100 (information processing system) will be described with reference to FIG. FIG. 1 is a schematic diagram of a virtual space distribution system 100 (hereinafter, simply referred to as a distribution system 100). As shown in FIG. 1, the distribution system 100 is a user terminal 1A (first user terminal) operated by a user A equipped with a head-mounted device (HMD) 110, and a user operated by a user B equipped with the HMD 110. It includes a terminal 1B and a server 2 that distributes virtual experience contents. The user terminals 1A and 1B are communicably connected to the server 2 via a communication network 3 such as the Internet. Hereinafter, for convenience of explanation, the user terminals 1A and 1B may be collectively referred to as the user terminal 1. Users A and B may be collectively referred to as user U. In this embodiment, it is assumed that the user terminals 1A and 1B have the same configuration. The virtual experience content is video content or the like for providing a virtual experience to the users of the user terminals 1A and 1B, and is a virtual space including a plurality of objects or a virtual space including a 360-degree spatial image such as a spherical image. May be provided as. In a virtual space including a 360-degree spatial image, a 360-degree spatial image is displayed on the surface of the virtual space. The virtual space includes a VR (Virtual Reality) space, an AR (Augmented Reality) space, and an MR (Mixed Reality) space.

Next, the configuration of the user terminal 1 will be described with reference to FIG. FIG. 2 is a schematic view showing a user terminal 1. As shown in FIG. 2, the user terminal 1 includes a head mount device (HMD) 110 mounted on the head of the user U, a headphone 116, a face camera 117, a microphone 118, a position sensor 130, and an external controller. It includes 320 and a control device 120.

The HMD 110 includes a display unit 112, an HMD sensor 114, a gaze sensor 140, and a face camera 113. The display unit 112 includes a non-transparent display device configured to completely cover the field of view (field of view) of the user U wearing the HMD 110. As a result, the user U can immerse himself in the virtual space by seeing only the visual field image displayed on the display unit 112. The display unit 112 includes a left-eye display unit configured to provide an image to the left eye of the user U, and a right-eye display unit configured to provide an image to the right eye of the user U. May be good. Further, the HMD 110 may be provided with a transmissive display device. In this case, the transmissive display device may be temporarily configured as a non-transparent display device by adjusting its transmittance.

The HMD sensor 114 is mounted in the vicinity of the display unit 112 of the HMD 110. The HMD sensor 114 includes at least one of a geomagnetic sensor, an acceleration sensor, and a tilt sensor (angular velocity sensor, gyro sensor, etc.), and detects various movements (tilts, etc.) of the HMD 110 mounted on the head of the user U. be able to.

The gaze sensor 140 has an eye tracking function for detecting the line of sight of the user U. The gaze sensor 140 may include, for example, a gaze sensor for the right eye and a gaze sensor for the left eye. The gaze sensor for the right eye acquires information on the angle of rotation of the eyeball of the right eye by irradiating the user U's right eye with infrared light, for example, and detecting the reflected light reflected from the right eye (particularly the cornea or iris). You may. On the other hand, the gaze sensor for the left eye irradiates the left eye of the user U with infrared light, for example, and detects the reflected light reflected from the left eye (particularly the cornea or the iris) to obtain information on the angle of rotation of the eyeball of the left eye. May be obtained. The acquisition rate of the detection data acquired by the gaze sensor 140 is larger than the transmission rate of the avatar control information (described later) and the frame rate of the visual field image displayed on the HMD 110. For example, the acquisition rate of the detected data is 200 Hz.

The face camera 113 is configured to acquire an image (particularly a moving image) in which the eyes (left eye and right eye) and eyebrows (left eyebrows and right eyebrows) of the user U are displayed while the HMD 110 is attached to the user U. Has been done. The face camera 113 is arranged at a predetermined position inside the HMD 110 so that the eyes and eyebrows of the user U can be imaged. The frame rate of the moving image acquired by the face camera 113 is larger than the transmission rate of the avatar control information and the frame rate of the field image displayed on the HMD 110. For example, the frame rate of the acquired moving image is 200 fps.

The headphone 116 (audio output unit) is attached to the left ear and the right ear of the user U, respectively. The headphone 116 is configured to receive voice data (electrical signal) from the control device 120 and output voice based on the received voice data. The microphone 118 (voice input unit) is configured to collect voice uttered by the user U and generate voice data (electrical signal) based on the collected voice. Further, the microphone 118 is configured to transmit voice data to the control device 120.

The face camera 117 is configured to acquire an image (particularly, a moving image) in which the mouth of the user U and its surroundings are displayed. The face camera 117 may be arranged at a position facing the mouth of the user U so that the mouth of the user U and its surroundings can be imaged. Further, the face camera 117 may be connected to the HMD 110. The frame rate of the moving image acquired by the face camera 117 may be larger than the frame rate of the field image displayed on the HMD 110.

The position sensor 130 is configured, for example, by a position tracking camera and is configured to detect the positions of the HMD 110 and the external controller 320. The position sensor 130 is wirelessly or wiredly connected to the control device 120 and is configured to detect information regarding the position, inclination, or emission intensity of a plurality of detection points (not shown) provided on the HMD 110. .. Further, the position sensor 130 is configured to detect information regarding the positions, tilts and / or emission intensities of a plurality of detection points (not shown) provided on the external controller 320. The detection point is, for example, a light emitting unit that emits infrared rays or visible light. Further, the position sensor 130 may include an infrared sensor and a plurality of optical cameras.

The external controller 320 is used to control the movement of the avatar's hand displayed in the virtual space by detecting the movement of the user U's hand. The external controller 320 is referred to as a right-handed external controller 320R operated by the right hand of the user U (hereinafter, simply referred to as a controller 320R) and a left-handed external controller 320L operated by the left hand of the user U (hereinafter, simply referred to as a controller 320L). ) And. The controller 320R is a device that indicates the position of the right hand of the user U and the movement of the fingers of the right hand. Further, the right hand of the avatar existing in the virtual space moves according to the movement of the controller 320R. The controller 320L is a device that indicates the position of the left hand of the user U and the movement of the fingers of the left hand. Further, the left hand of the avatar existing in the virtual space moves according to the movement of the controller 320L.

The control device 120 is a computer configured to control the HMD 110. The control device 120 identifies the position information of the HMD 110 based on the information acquired from the position sensor 130, and based on the specified position information, the position of the virtual camera (the position of the avatar) in the virtual space and the reality. The position of the user U wearing the HMD 110 in space can be accurately associated. Further, the control device 120 identifies the operation of the external controller 320 based on the information acquired from the sensor built in the position sensor 130 and / or the external controller 320, and is based on the operation of the specified external controller 320. Therefore, the movement of the avatar's hand displayed in the virtual space can be accurately associated with the movement of the external controller 320 in the real space. In particular, the control device 120 identifies the operation of the controller 320L based on the information acquired from the sensor built in the position sensor 130 and / or the controller 320L, and virtualizes based on the operation of the specified controller 320L. The movement of the left hand of the avatar displayed in the space and the movement of the controller 320L in the real space (the movement of the left hand of the user U) can be accurately associated with each other. Similarly, the control device 120 identifies the operation of the controller 320R based on the information acquired from the position sensor and / the sensor built in the controller 320R, and the virtual space is based on the operation of the specified controller 320R. The movement of the right hand of the avatar displayed inside and the movement of the controller 320R in the real space (the movement of the right hand of the user U) can be accurately associated with each other.

Further, the control device 120 can associate the facial expression (state) of the user U with the facial expression (state) of the avatar of the user U based on the moving images acquired by the face cameras 113 and 117. The details of the process of associating the facial expression of the user U with the facial expression of the avatar will be described later.

Further, the control device 120 can specify the line of sight of the right eye and the line of sight of the left eye of the user U, respectively, and can specify the gazing point which is the intersection of the line of sight of the right eye and the line of sight of the left eye. Further, the control device 120 can specify the line of sight of both eyes of the user U (the line of sight of the user U) based on the specified gaze point. Here, the line of sight of the user U is the line of sight of both eyes of the user U, and coincides with the direction of the straight line passing through the midpoint of the line segment connecting the right eye and the left eye of the user U and the gazing point. Further, the control device 120 sets the center position of the black eye of the user U's right eye and the left eye of the user U based on the detection data (eye tracking data) transmitted from the gaze sensor 140 (the gaze sensor for the left eye and the gaze sensor for the right eye). It is possible to specify the center position of each of the black eyes.

Next, with reference to FIG. 3, a method for acquiring information regarding the position and inclination of the HMD 110 will be described below. FIG. 3 is a diagram showing the head of the user U wearing the HMD 110. Information regarding the position and tilt of the HMD 110 linked to the movement of the head of the user U wearing the HMD 110 can be detected by the position sensor 130 and / or the HMD sensor 114 mounted on the HMD 110. As shown in FIG. 2, three-dimensional coordinates (uvw coordinates) are defined centering on the head of the user U wearing the HMD 110. The vertical direction in which the user U stands upright is defined as the v-axis, the direction orthogonal to the v-axis and passing through the center of the HMD 110 is defined as the w-axis, and the v-axis and the direction orthogonal to the w-axis are defined as the u-axis. The position sensor 130 and / or the HMD sensor 114 has an angle around each uvw axis (that is, a yaw angle indicating rotation about the v axis, a pitch angle indicating rotation about the u axis, and a w axis as the center. The tilt determined by the roll angle indicating rotation) is detected. The control device 120 determines the angle information for controlling the visual axis of the virtual camera based on the detected angle change around each uvw axis.

Next, the hardware configuration of the control device 120 will be described with reference to FIG. FIG. 4 is a diagram showing a hardware configuration of the control device 120. As shown in FIG. 4, the control device 120 includes a control unit 121, a storage unit 123, an I / O (input / output) interface 124, a communication interface 125, and a bus 126. The control unit 121, the storage unit 123, the I / O interface 124, and the communication interface 125 are communicably connected to each other via the bus 126.

The control device 120 may be configured as a personal computer, a smartphone, a phablet, a tablet, or a wearable device separately from the HMD 110, or may be built in the HMD 110. Further, a part of the functions of the control device 120 may be mounted on the HMD 110, and the remaining functions of the control device 120 may be mounted on another device separate from the HMD 110.

The control unit 121 includes a memory and a processor. The memory is composed of, for example, a ROM (Read Only Memory) in which various programs and the like are stored, a RAM (Random Access Memory) having a plurality of work areas in which various programs and the like executed by the processor are stored, and the like. The processor is, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit) and / or a GPU (Graphics Processing Unit), and expands a program specified from various programs embedded in the ROM onto the RAM and develops the RAM. It is configured to execute various processes in collaboration with.

In particular, the control unit 121 may control various operations of the control device 120 by the processor deploying the control program on the RAM and executing the control program in cooperation with the RAM. The control unit 121 displays the field of view image on the display unit 112 of the HMD 110 based on the field of view image data. As a result, the user U can immerse himself in the virtual space.

The storage unit (storage) 123 is, for example, a storage device such as an HDD (Hard Disk Drive), an SSD (Solid State Drive), or a USB flash memory, and is configured to store programs and various data. The storage unit 123 may store a control program for causing a computer to execute at least a part of the information processing method according to the present embodiment, and a control program for realizing sharing of a virtual space by a plurality of users. Further, the storage unit 123 may store data related to the user U authentication program and various images and objects (for example, an avatar). Further, a database including a table for managing various data may be constructed in the storage unit 123.

The I / O interface 124 is configured to connect the HMD 110, the position sensor 130, the external controller 320, the headphones 116, the face camera 117, and the microphone 118 to the control device 120 so as to be communicable. For example, it is composed of a USB (Universal Serial Bus) terminal, a DVI (Digital Visual Interface) terminal, an HDMI (registered trademark) (High-Definition Multimedia Interface) terminal, and the like. The control device 120 may be wirelessly connected to the HMD 110, the position sensor 130, the external controller 320, the headphones 116, the face camera 117, and the microphone 118, respectively.

The communication interface 125 is configured to connect the control device 120 to a communication network 3 such as a LAN (Local Area Network), a WAN (Wide Area Network), or the Internet. The communication interface 125 includes various wired connection terminals for communicating with an external device such as a server 2 via the communication network 3 and various processing circuits for wireless connection, and is for communicating via the communication network 3. It is configured to meet the communication standards of.

Next, an example of a specific configuration of the external controller 320 will be described with reference to FIG. The external controller 320 detects the movement of a part of the body of the user U (a part other than the head, and the hand of the user U in this embodiment), so that the movement of the hand object displayed in the virtual space is performed. Used to control. The external controller 320 is referred to as a right-handed external controller 320R operated by the right hand of the user U (hereinafter, simply referred to as a controller 320R) and a left-handed external controller 320L operated by the left hand of the user U (hereinafter, simply referred to as a controller 320L). ) And. The controller 320R is a device that indicates the position of the right hand of the user U and the movement of the fingers of the right hand. Further, the right-handed object 400R (see FIG. 16) existing in the virtual space moves according to the movement of the controller 320R. The controller 320L is a device that indicates the position of the left hand of the user U and the movement of the fingers of the left hand. Further, the left-handed object 400L (see FIG. 16) existing in the virtual space moves according to the movement of the controller 320L. Since the controller 320R and the controller 320L have substantially the same configuration, only the specific configuration of the controller 320R will be described below with reference to FIG. In the following description, for convenience, the controllers 320L and 320R may be simply collectively referred to as the controller 320.

As shown in FIG. 5, the controller 320R includes an operation button 302, a plurality of detection points 304, a sensor (not shown), and a transceiver (not shown). Only one of the detection point 304 and the sensor may be provided. The operation button 302 is composed of a plurality of button groups configured to receive an operation input from the user U. The operation button 302 includes a push button, a trigger button and an analog stick. The push-type button is a button operated by the action of pressing with the thumb. For example, two push-type buttons 302a and 302b are provided on the top surface 322. The trigger type button is a button operated by an action such as pulling a trigger with the index finger or the middle finger. For example, a trigger type button 302e is provided on the front surface portion of the grip 324, and a trigger type button 302f is provided on the side surface portion of the grip 324. It is assumed that the trigger type buttons 302e and 302f are operated by the index finger and the middle finger, respectively. The analog stick is a stick-type button that can be operated by tilting 360 degrees in any direction from a predetermined neutral position. For example, an analog stick 320i is provided on the top surface 322, and it is assumed that the analog stick 320i is operated by using the thumb.

The controller 320R includes a frame 326 extending from both sides of the grip 324 in a direction opposite to the top surface 322 to form a semicircular ring. A plurality of detection points 304 are embedded in the outer surface of the frame 326. The plurality of detection points 304 are, for example, a plurality of infrared LEDs arranged in a row along the circumferential direction of the frame 326. After the position sensor 130 detects information regarding the position, tilt or emission intensity of the plurality of detection points 304, the control device 120 determines the position and orientation (tilt / tilt) of the controller 320R based on the information detected by the position sensor 130. Get information about orientation).

The sensor of the controller 320R may be, for example, any one of a magnetic sensor, an angular velocity sensor, an acceleration sensor, or a combination thereof. When the user U moves the controller 320R, the sensor outputs a signal corresponding to the direction and movement of the controller 320R (for example, a signal indicating information regarding magnetism, angular velocity, or acceleration). The control device 120 acquires information regarding the position and orientation of the controller 320R based on the signal output from the sensor.

The transceiver of the controller 320R is configured to send and receive data between the controller 320R and the control device 120. For example, the transceiver may transmit an operation signal corresponding to the operation input of the user U to the control device 120. Further, the transceiver may receive an instruction signal instructing the controller 320R to emit light at the detection point 304 from the control device 120. Further, the transceiver may send a signal indicating the value detected by the sensor to the control device 120.

Next, a process for displaying the field image on the HMD 110 will be described with reference to FIGS. 6 to 9. FIG. 6 is a flowchart showing a process of displaying a field image on the HMD 110. FIG. 7 is an xyz space diagram showing an example of the virtual space 200. The state (a) of FIG. 8 is a yx plan view of the virtual space 200 shown in FIG. 7. The state (b) of FIG. 8 is a zx plan view of the virtual space 200 shown in FIG. 7. FIG. 9 is a diagram showing an example of the field image V displayed on the HMD 110.

As shown in FIG. 6, in step S1, the control unit 121 (see FIG. 4) generates virtual space data indicating the virtual space 200 including the virtual camera 300 and various objects. As shown in FIG. 7, the virtual space 200 is defined as a spherical surface centered on the central position 210 (in FIG. 7, only the upper half of the celestial sphere is shown). In the virtual space 200, an xyz coordinate system with the center position 210 as the origin is set. The virtual camera 300 defines a visual axis L for specifying the visual field image V (see FIG. 9) displayed on the HMD 110. The uvw coordinate system that defines the field of view of the virtual camera 300 is determined to be linked to the uvw coordinate system defined around the head of the user U in the real space. The control unit 121 may move the virtual camera 300 in the virtual space 200 in conjunction with the movement of the user U wearing the HMD 110 in the real space.

Next, in step S2, the control unit 121 specifies the field of view CV (see FIG. 8) of the virtual camera 300. Specifically, the control unit 121 acquires information on the position and inclination of the HMD 110 based on the data indicating the state of the HMD 110 transmitted from the position sensor 130 and / or the HMD sensor 114. Next, the control unit 121 specifies the position and orientation of the virtual camera 300 in the virtual space 200 based on the information regarding the position and inclination of the HMD 110. Next, the control unit 121 determines the visual axis L of the virtual camera 300 from the position and orientation of the virtual camera 300, and identifies the visual field CV of the virtual camera 300 from the determined visual axis L. The field of view CV of the virtual camera 300 corresponds to a part of the virtual space 200 visible to the user U wearing the HMD 110 (in other words, corresponds to a part of the virtual space 200 displayed on the HMD 110). ). The visual field CV is the first region CVa set as the angle range of the polar angle α about the visual axis L in the xy plane shown in the state (a) of FIG. 8, and the xz shown in the state (b) of FIG. In the plane, it has a second region CVb set as an angle range of the azimuth angle β about the visual axis L. The control unit 121 identifies the line of sight of the user U based on the data indicating the line of sight of the user U transmitted from the gaze sensor 140, and virtually based on the information on the specified line of sight of the user U and the position and inclination of the HMD 110. The orientation of the camera 300 (visual axis L of the virtual camera) may be determined. Further, as will be described later, the control unit 121 may determine the orientation of the face of the user U's avatar based on the information regarding the position and inclination of the HMD 110.

In this way, the control unit 121 can specify the field of view CV of the virtual camera 300 based on the data from the position sensor 130 and / or the HMD sensor 114. When the user U wearing the HMD 110 moves, the control unit 121 can update the field of view CV of the virtual camera 300 based on the data indicating the movement of the HMD 110 transmitted from the position sensor 130 and / or the HMD sensor 114. .. That is, the control unit 121 can update the visual field CV according to the movement of the HMD 110. Similarly, when the line of sight of the user U changes, the control unit 121 may update the field of view CV of the virtual camera 300 based on the data indicating the line of sight of the user U transmitted from the gaze sensor 140. That is, the control unit 121 may change the visual field CV according to the change in the line of sight of the user U.

Next, in step S3, the control unit 121 generates visual field image data indicating the visual field image V displayed on the display unit 112 of the HMD 110. Specifically, the control unit 121 generates visual field image data based on the virtual space data defining the virtual space 200 and the visual field CV of the virtual camera 300.

Next, in step S4, the control unit 121 displays the field of view image V on the display unit 112 of the HMD 110 based on the field of view image data (see FIG. 9). In this way, the field of view CV of the virtual camera 300 changes according to the movement of the user U wearing the HMD 110, and the field of view image V displayed on the display unit 112 of the HMD 110 changes, so that the user U is in the virtual space. You can immerse yourself in 200.

The virtual camera 300 may include a virtual camera for the left eye and a virtual camera for the right eye. In this case, the control unit 121 generates left-eye field-of-view image data showing the field-of-view image for the left eye based on the virtual space data and the field of view of the left-eye virtual camera. Further, the control unit 121 generates right-eye field image data showing a field image for the right eye based on the virtual space data and the field of view of the right-eye virtual camera. After that, the control unit 121 displays the visual field image for the left eye on the left eye display unit based on the left eye visual field image data, and displays the visual field image for the right eye on the right eye display unit based on the right eye visual field image data. Is displayed. In this way, the user U can visually recognize the visual field image three-dimensionally by the parallax between the visual field image for the left eye and the visual field image for the right eye. The virtual camera may be placed at the eye position of the avatar operated by the user. For example, the left-eye virtual camera may be placed in the left eye of the avatar, while the right-eye virtual camera may be placed in the right eye of the avatar.

The processes of steps S1 to S4 shown in FIG. 6 may be executed every frame. For example, when the frame rate of the moving image is 90 fps, the processes of steps S1 to S4 may be repeatedly executed at intervals of ΔT = 1/90 (seconds). In this way, since the processes of steps S1 to S4 are repeatedly executed at predetermined intervals, the field of view of the virtual camera 300 is updated according to the operation of the HMD 110, and the field of view image V displayed on the display unit 112 of the HMD 110 is updated. Is updated.

Next, the hardware configuration of the server 2 shown in FIG. 1 will be described with reference to FIG. FIG. 10 is a diagram showing a hardware configuration of the server 2. As shown in FIG. 10, the server 2 includes a control unit 23, a storage unit 22, a communication interface 21, and a bus 24. The control unit 23, the storage unit 22, and the communication interface 21 are connected to each other so as to be able to communicate with each other via the bus 24. The control unit 23 includes a memory and a processor, and the memory is composed of, for example, a ROM and a RAM, and the processor is composed of, for example, a CPU, an MPU, and / or a GPU.

The storage unit (storage) 22 is, for example, a large-capacity HDD or the like. The storage unit 22 may store a control program for causing a computer to execute at least a part of the information processing method according to the present embodiment, and a control program for realizing sharing of a virtual space by a plurality of users. Further, the storage unit 22 may store user management information for managing each user and data related to various images and objects (for example, a hand object). The communication interface 21 is configured to connect the server 2 to the communication network 3.

The server 2 may provide a video distribution service to a terminal other than the user terminals 1A and 1B or the user terminals 1A and 1B (for example, a personal computer or a tablet terminal) via an application executed on a web browser. .. An application for providing a moving image distribution service may also be stored in the storage unit 22. The application stored in the storage unit 22 is distributed to the terminal device in response to the distribution request. The terminal device can also download the application from a server other than the server 2 (such as a server that provides an application market).

In this way, the server 2 manages a website for providing a video distribution service, and can distribute a web page (HTML data) constituting the website in response to a request from a terminal device. The server 2 can send and receive various data (including data necessary for screen display) necessary for providing the video distribution service to and from the terminal device. Further, the server 2 can store various data for each identification information (for example, user ID) that identifies each user, and can manage the provision status of the video distribution service for each user. Although detailed description will be omitted, the server 2 may also have a function of performing user authentication processing, billing processing, and the like.

[Explanation of motion control between avatars]
Next, an example of a process for synchronously controlling the movements of the avatars 4A and 4B between the user terminal 1A and the user terminal 1B will be described with reference to FIGS. 1, 11 and 12. The state (a) of FIG. 11 is a diagram showing a virtual space 200A provided to the user A. The state (b) of FIG. 11 is a diagram showing a virtual space 200B provided to the user B. FIG. 12 is a flowchart for explaining an example of a process of synchronizing the movements of the avatars 4A and 4B between the user terminal 1A and the user terminal 1B. In this description, as a prerequisite, as shown in FIG. 11, the avatar 4A associated with the user terminal 1A (user A) and the avatar 4B associated with the user terminal 1B (user B) share the same virtual space. It is assumed that you are doing. That is, it is assumed that the user A and the user B share one virtual space via the communication network 3.

As shown in the state (a) of FIG. 11, the virtual space 200A of the user A includes the avatar 4A and the avatar 4B. The avatar 4A is operated by the user A and is linked to the operation of the user A. The avatar 4A has a left hand linked to the operation of the controller 320L of the user terminal 1A (the operation of the left hand of the user A), a right hand linked to the operation of the controller 320R of the user terminal 1A, and a facial expression linked to the facial expression of the user A. Have a face to do. The face of Avatar 4A has a plurality of face parts (eg, eyes, eyebrows, mouth, etc.). The avatar 4B is operated by the user B and is linked to the operation of the user B. The avatar 4B has a left hand linked to the operation of the controller 320L of the user terminal 1B indicating the operation of the left hand of the user B and a right hand linked to the operation of the controller 320R of the user terminal 1B indicating the operation of the right hand of the user B. It has a face linked to the facial expression of the user B. The face of Avatar 4B has a plurality of face parts (for example, eyes, eyebrows, mouth, etc.).

Further, the positions of the avatars 4A and 4B may be specified according to the positions of the HMD 110s of the user terminals 1A and 1B. Similarly, the orientation of the face of the avatars 4A and 4B may be specified according to the inclination of the HMD 110 of the user terminals 1A and 1B. Further, the hand movements of the avatars 4A and 4B may be specified according to the movements of the external controllers 320 of the user terminals 1A and 1B. Further, the facial expressions of the avatars 4A and 4B may be specified according to the facial expressions (states) of the users A and B. In particular, the positions of the black eyes with respect to the white eyes of the avatars 4A and 4B may be specified according to the positions of the black eyes with respect to the white eyes of the users A and B.

As shown in the state (b) of FIG. 11, the virtual space 200B of the user B includes the avatar 4A and the avatar 4B. The positions of the avatars 4A and 4B in the virtual space 200A may correspond to the positions of the avatars 4A and 4B in the virtual space 200B.

Next, referring to FIG. 12, in step S10, the control unit 121 of the user terminal 1A generates the voice data of the user A. For example, when the user A inputs voice to the microphone 118 (voice input unit) of the user terminal 1A, the microphone 118 generates voice data indicating the input voice. After that, the microphone 118 transmits the generated voice data to the control unit 121 via the I / O interface 124.

Next, in step S11, the control unit 121 of the user terminal 1A generates the control information of the avatar 4A, and then the voice data indicating the control information of the generated avatar 4A and the voice of the user A (voice data of the user A). ) Is sent to the server 2. After that, the control unit 23 of the server 2 receives the control information of the avatar 4A and the voice data of the user A from the user terminal 1A (step S12).

Here, the control information of the avatar 4A is information necessary for controlling the operation of the avatar 4A. The control information of the avatar 4A includes information on the position of the avatar 4A (position information), information on the orientation of the face of the avatar 4A (face orientation information), and information on the state of the hands (left hand and right hand) of the avatar 4A (hand information). ) And information (face information) regarding the facial expression of the avatar 4A may be included.

The face information of the avatar 4A has information indicating the state of a plurality of face parts. Information indicating the state of a plurality of face parts includes information indicating the state of the eyes of the avatar 4A (shape of the white eyes and the position of the black eyes with respect to the white eyes) (eye information) and information on the eyebrows of the avatar 4A (position and shape of the eyebrows). Etc.) and information (eyebrow information) indicating the state of the mouth of the avatar 4A (position and shape of the mouth, etc.).

Specifically, the eye information of the avatar 4A has information indicating the state of the left eye of the avatar 4A and information indicating the state of the right eye of the avatar 4A. The eyebrow information of the avatar 4A has information indicating the state of the left eyebrow of the avatar 4A and information indicating the state of the right eyebrow of the avatar 4A.

The control unit 121 of the user terminal 1A acquires an image showing the eyes (left eye and right eye) and eyebrows (left eyebrow and right eyebrow) of the user A from the face camera 113 mounted on the HMD 110, and then obtains the acquired image. And, based on a predetermined image processing algorithm, the states of the eyes (left eye and right eye) and eyebrows (left eyebrows and right eyebrows) of the user A are specified. Next, the control unit 121 uses information indicating the state of the eyes (left eye and right eye) of the avatar 4A and the eyebrows (left eyebrow and right eyebrow) of the avatar 4A based on the specified eye and eyebrow states of the user A. Generate information indicating the status respectively. Of the information indicating the state of the eyes of the user A, the information indicating the position (center position) of the black eyes of the user A may be acquired by the gaze sensor 140 instead of the face camera 113.

Similarly, the control unit 121 of the user terminal 1A acquires an image showing the mouth of the user A and its surroundings from the face camera 117, and then based on the acquired image and a predetermined image processing algorithm, the mouth of the user A. Identify the state of. Next, the control unit 121 generates information indicating the mouth state of the avatar 4A based on the mouth state of the specified user A.

As described above, the control unit 121 of the user terminal 1A has eye information of the avatar 4A corresponding to the eyes of the user A, eyebrow information of the avatar 4A corresponding to the eyebrows of the user A, and avatar 4A corresponding to the mouth of the user A. It becomes possible to generate the mouth information of.

Further, the control unit 121 has an image captured by the face camera 113, an image captured by the face camera 117, and a plurality of types of facial expressions stored in the storage unit 123 or the memory based on a predetermined image processing algorithm. The facial expression (for example, smile) of the user A may be specified from the facial expressions (for example, smile, sad expression, expressionless expression, angry expression, surprised expression, troubled expression, etc.). After that, the control unit 121 can generate facial information indicating the facial expression (for example, a smile) of the avatar 4A based on the specified facial expression of the user A. In this case, a facial expression database containing a plurality of types of facial expressions of the avatar 4A and facial information of the plurality of avatars 4A each associated with one of the plurality of types of facial expressions is stored in the storage unit 123. You may.

For example, when the control unit 121 identifies that the facial expression of the avatar 4A is a smile, the control unit 121 acquires face information indicating the smile. Here, the face information indicating the smile of the avatar 4A includes eye information, eyebrow information, and mouth information of the avatar 4A when the facial expression of the avatar 4A is a smile.

Next, in step S13, the control unit 121 of the user terminal 1B generates the control information of the avatar 4B, and then transmits the generated control information of the avatar 4B to the server 2. After that, the control unit 23 of the server 2 receives the control information of the avatar 4B from the user terminal 1B (step S14).

Here, the control information of the avatar 4B is information necessary for controlling the operation of the avatar 4B. The control information of the avatar 4B includes information on the position of the avatar 4B (position information), information on the orientation of the face of the avatar 4B (face orientation information), and information on the state of the hands (left hand and right hand) of the avatar 4B (hand information). ) And information (face information) regarding the facial expression of the avatar 4B may be included.

The face information of the avatar 4B has information indicating the state of a plurality of face parts. The information indicating the state of a plurality of face parts includes information indicating the state of the eyes of the avatar 4B (shape of the white eyes and the position of the black eyes with respect to the white eyes) (eye information) and information on the eyebrows of the avatar 4B (position and shape of the eyebrows). Etc.) and information (eyebrow information) indicating the state of the mouth of the avatar 4B (position and shape of the mouth, etc.). The method of acquiring the face information of the avatar 4B is the same as the method of acquiring the face information of the avatar 4A.

Next, the server 2 transmits the control information of the avatar 4B to the user terminal 1A (step S15), while transmitting the control information of the avatar 4A and the voice data of the user A to the user terminal 1B (step S19). After that, the control unit 121 of the user terminal 1A receives the control information of the avatar 4B in step S16, updates the states of the avatars 4A and 4B based on the control information of the avatars 4A and 4B, and then the virtual space 200A. The virtual space data indicating (see the state (a) in FIG. 11) is updated (step S17).

Specifically, the control unit 121 of the user terminal 1A updates the positions of the avatars 4A and 4B based on the position information of the avatars 4A and 4B. The control unit 121 updates the face orientation of the avatars 4A and 4B based on the face orientation information of the avatars 4A and 4B. The control unit 121 updates the hands of the avatars 4A and 4B based on the hand information of the avatars 4A and 4B. The control unit 121 updates the facial expressions of the avatars 4A and 4B based on the facial information of the avatars 4A and 4B. In this way, the virtual space data indicating the virtual space 200A including the updated avatars 4A and 4B is updated.

After that, the control unit 121 of the user terminal 1A identifies the field of view CV of the avatar 4A (virtual camera 300) according to the position and tilt of the HMD 110, and then sets the updated virtual space data and the field of view CV of the avatar 4A. Based on this, the visual field image displayed on the HMD 110 is updated (step S18).

On the other hand, the control unit 121 of the user terminal 1B updates the states of the avatars 4A and 4B based on the control information of the avatars 4A and 4B after receiving the control information of the avatar 4A and the voice data of the user A in step S20. Above, the virtual space data indicating the virtual space 200B (see the state (b) in FIG. 11) is updated (step S21).

Specifically, the control unit 121 of the user terminal 1B updates the positions of the avatars 4A and 4B based on the position information of the avatars 4A and 4B. The control unit 121 updates the face orientation of the avatars 4A and 4B based on the face orientation information of the avatars 4A and 4B. The control unit 121 updates the hands of the avatars 4A and 4B based on the hand information of the avatars 4A and 4B. The control unit 121 updates the facial expressions of the avatars 4A and 4B based on the facial information of the avatars 4A and 4B. In this way, the virtual space data indicating the virtual space including the updated avatars 4A and 4B is updated.

After that, the control unit 121 of the user terminal 1B identifies the field of view CV of the avatar 4B (virtual camera 300) according to the position and tilt of the HMD 110, and then sets the updated virtual space data and the field of view CV of the avatar 4B. Based on this, the visual field image displayed on the HMD 110 is updated (step S22).

Next, the control unit 121 of the user terminal 1B receives the voice data of the user A, the information regarding the position of the avatar 4A included in the control information of the avatar 4A, the information regarding the position of the avatar 4B, and a predetermined voice processing algorithm. The voice data of the user A is processed based on the above. After that, the control unit 121 transmits the processed voice data to the headphone 116 (voice output unit), and the headphone 116 outputs the voice of the user A based on the processed voice data (step S23). In this way, voice chat (VR chat) between users (between avatars) can be realized in the virtual space.

In the present embodiment, after the user terminals 1A and 1B transmit the control information of the avatar 4A and the control information of the avatar 4B to the server 2, the server 2 transmits the control information of the avatar 4A to the user terminal 1B, while the avatar 4B. Control information is transmitted to the user terminal 1A. In this way, it is possible to synchronize the movements of the avatars 4A and 4B between the user terminal 1A and the user terminal 1B.

[Explanation of information processing method according to the first embodiment]
Next, the information processing method according to the present embodiment will be described with reference to FIGS. 7, 13 to 16. 13 and 14 are flowcharts for explaining the information processing method according to the present embodiment. FIG. 15 is a schematic diagram showing a method of updating the evaluation value of each partial region of the virtual space 200. In this embodiment, as shown in FIG. 15, the avatar 4A associated with the user terminal 1A (user A), the avatar 4B associated with the user terminal 1B (user B), and the user terminal 1C (user C). It is assumed that the associated avatar 4C shares the same virtual space 200. That is, it is assumed that the user A, the user B, and the user C share one virtual space 200 via the communication network 3. Although not shown, the virtual camera 300 corresponding to the HMD 110 of the user terminal 1A is arranged in association with the avatar 4A in the virtual space 200, and the virtual camera 300 corresponding to the HMD 110 of the user terminal 1B is a virtual space. It is assumed that the virtual camera 300, which is arranged in association with the avatar 4B in the 200 and corresponds to the HMD 110 of the user terminal 1C, is arranged in association with the avatar 4C in the virtual space 200. That is, the position and / or orientation of each avatar 4A, 4B, 4C is changed based on the movement of the HMD 110 of each user terminal 1A, 1B, 1C.

In FIG. 7, for convenience of explanation, the parallel parallels lo and the meridian la are omitted on the front side (−z side) of the surface S of the virtual space 200, while the back side of the surface S of the virtual space 200. A plurality of parallels lo and meridian la, and a plurality of partial regions R are illustrated on the (+ Z side). In FIG. 15, the latitude line lo, the meridian line la, and the plurality of partial regions R on the surface of the virtual space 200 are not shown.

As shown in FIG. 13, in step S100, the control unit 23 (see FIG. 10) of the server 2 generates virtual space data indicating the virtual space 200. Specifically, the control unit 23 reads out the virtual experience content (360 degree content) stored in the storage unit 22 in response to the input signals from the user terminals 1A, 1B, and 1C, and reads the virtual space or the whole celestial sphere image. Generate virtual spatial data including 360 degree spatial images such as. The 360-degree content may include a 360-degree moving image. The 360-degree content may include, for example, different content in a plurality of regions of the 360-degree spatial image. As a result, each user A, B, C sharing the virtual space 200 can view various areas in the virtual space 200 and recommend attention points to other users, and the avatars 4A, 4B, 4C can be used with each other. You can enjoy communication at.

Next, in step S102, the control unit 23 uses a plurality of latitude lines lo arranged at predetermined angular intervals and a plurality of meridians la arranged at predetermined angular intervals to form a plurality of surfaces S of the virtual space 200. It is divided into partial areas R. In other words, the control unit 23 sets a plurality of partial regions R on the surface S by providing a grid composed of latitude lines lo and meridian la on the surface S of the hemispherical virtual space 200 (see FIG. 7). ..

The number of parallels lo is set according to the angle interval of adjacent parallels lo. For example, when the angle interval between the adjacent parallels lo is 15 degrees, (90 degrees ÷ 15 degrees) -1 = 5, and the number of parallels lo is 5. Similarly, the number of meridians la is set according to the angular spacing of adjacent meridians la. For example, when the angle interval between adjacent meridians la is 15 degrees, 360 degrees ÷ 15 degrees = 24, and the number of meridians la is 24. The control unit 121 can appropriately set the number of parallels lo and meridian la according to the type of virtual space data.

The number of partial regions R is determined according to the number of meridians la and the number of parallels lo. For example, when the number of meridians la is n and the number of parallels lo is m, the total number N of the partial regions R is N = n × (m + 1) (however, n ≧ 2).

Next, in step S104, the control unit 23 of the server 2 transmits virtual space data in a state where a plurality of partial regions R are set on the surface S of the virtual space 200 to the user terminals 1A, 1B, and 1C. Next, in step S106, the control unit 121 (see FIG. 4) of each user terminal 1A, 1B, 1C displays the virtual space data distributed from the control unit 23 of the server 2 on the display unit 112.

Next, in step S108, the control unit 121 of each user terminal 1A, 1B, 1C specifies the position and orientation of the HMD 110. Specifically, the control unit 121 may specify the position and orientation of the HMD 110 by receiving data indicating the state of the HMD 110 from the position sensor 130 and / or the HMD sensor 114. Alternatively, the control unit 121 may acquire information regarding the position and tilt of the HMD 110 based on the data indicating the state of the HMD 110 transmitted from the position sensor 130 and / or the HMD sensor 114.

Next, in step S110, the control unit 121 of each user terminal 1A, 1B, 1C determines the position of each avatar 4A, 4B, 4C (virtual camera 300) in the virtual space 200 based on the position and orientation of the HMD 110. The orientation is updated, and the position of the intersection C where the line-of-sight direction (view axis L) of each avatar 4A, 4B, 4C and the surface S of the virtual space 200 intersect is determined. For example, the intersection C is determined based on the radius r of the virtual space 200, the latitude θ of the visual axis L shown in FIG. 7, and the longitude φ. Here, since the radius r of the virtual space 200 is a known value, the control unit 121 determines the intersection C based on the latitude θ and the longitude φ of the visual axis L. The visual axis L is determined based on the position of each avatar 4A, 4B, 4C and the direction of the line of sight of each avatar 4A, 4B, 4C.

In this step S110, the control unit 121 of each user terminal 1A, 1B, 1C specifies the line-of-sight direction of each avatar 4A, 4B, 4C based on the position and orientation of the HMD 110, but instead, each user The line-of-sight direction of each avatar 4A, 4B, 4C may be specified based on the line-of-sight of A, B, and C. In this case, the control unit 121 may determine that the line of sight of each user A, B, or C has changed by receiving information from the gaze sensor 140. Alternatively, the control unit 121 acquires information on the line of sight of each user A, B, C based on the information transmitted from the gaze sensor 140, and based on the acquired information, each user A, B, C It may be determined whether or not the line of sight of the person has changed.

Next, in step S112, the control unit 121 of each user terminal 1A, 1B, 1C identifies a partial region including the intersection C among the plurality of partial regions R that divide the surface S of the virtual space 200. .. Here, since the partial region R is set by the plurality of parallels lo and the plurality of meridians la, each partial region R has an angular range of latitude θ and longitude φ. Further, as described above, the intersection C is determined based on the latitude θ and the longitude φ of the visual axis L. In this way, the control unit 121 can determine which partial region R the intersection C is included in by determining the latitude θ and the longitude φ of the visual axis L after movement without performing complicated calculations. can.

Since the line-of-sight direction (visual axis L) of each avatar 4A, 4B, 4C may change from moment to moment, in this step S112, the visual field information (each avatar 4A) indicating the partial region R including the intersection C at each time. , 4B, 4C information indicating a partial region pointed to by the line-of-sight direction) is specified in association with the time information that defines the time. The time information may be specified corresponding to the actual time, or may be specified as the elapsed time from the playback start time of the 360-degree content transmitted to each user terminal 1A, 1B, 1C as virtual space data. ..

For example, the control unit 121 of the user terminal 1A identifies a partial region R (partial region R1 in FIG. 15) including an intersection CA where the line-of-sight direction LA of the avatar 4A and the surface S of the virtual space 200 intersect with the time information. do. Similarly, each control unit 121 of the user terminal 1B associates the partial region R (partial region R1 in FIG. 15) including the intersection CB where the line-of-sight direction LB of the avatar 4B and the surface S of the virtual space 200 intersect with the time information. To identify. Similarly, each control unit 121 of the user terminal 1C associates the partial region R (partial region R3 in FIG. 15) including the intersection CC where the line-of-sight direction LC of the avatar 4C and the surface S of the virtual space 200 intersect with the time information. To identify.

Next, in step S114, the control unit 121 of each user terminal 1A, 1B, 1C receives the face information of each avatar 4A, 4B, 4C at each time, and the voice information of each avatar 4A, 4B, 4C at each time. To get. As described above, the control unit 121 identifies the facial expressions of the avatars 4A, 4B, and 4C based on the images captured by the face camera 113 and the face camera 117, and obtains facial information based on the specified facial expressions. get. Further, the control unit 121 receives the voice data generated based on the voices uttered by the users A, B, and C from the microphone 118, and acquires the voice information based on the received voice data.

Next, in step S116, the control unit 121 of each user terminal 1A, 1B, 1C transmits the acquired time information, the visual field information, the face information, and the voice information to the server 2. Next, in step S118, the server 2 updates the visual field information data stored in the storage unit 22. An example of the visual field information data is shown below.

(Table 1)

Here, the visual field information data shown in Table 1 includes time information, visual field information of each avatar 4A, 4B, 4C at each time, emotion information, and emotion evaluation value. The emotional information is specified based on the facial information and voice information of each avatar 4A, 4B, 4C. Emotional information may be specified based on at least one of facial information and voice information. The emotion evaluation value is determined based on the emotion information of each avatar 4A, 4B, 4C and the attribute of the 360-degree content. For example, when the 360-degree content distributed in the virtual space 200 is an entertainment program (laughter program, quiz program, etc.) or an entertainment movie, a positive emotion evaluation value is obtained when the emotion information is a smile or a surprised face. If the emotional information is expressionless or angry, a negative emotional evaluation value may be attached. On the other hand, when the 360-degree content is a serious drama or a social movie, a positive emotion evaluation value may be attached when the emotion information is a sad expression or a surprised face. In addition, it is not necessary to obtain an emotional evaluation value in the case of specific emotional information (for example, expressionlessness). In this example, when the emotional information is a smile, the emotional evaluation value is set to "+1", when the emotional information is a surprised face, the emotional evaluation value is set to "+2", and when the emotional information is expressionless. The emotional evaluation value is "-1".

As shown in FIG. 15, for example, the avatar 4A gazes at the partial region R1 between the times T1 and T5 (the intersection CA is included in the partial region R1 at the times T1 to T5), and between the times T1 and T5. It is assumed that the face information of the avatar 4A is expressionless. Further, it is assumed that the user A utters a voice saying "Not so" at a predetermined time between times T1 and T5 (for example, time T2). In this case, the control unit 23 of the server 2 that has received the time information and the field of view information from the control unit 121 of the user terminal 1A specifies the field of view information of the avatars 4A at each time T1 to T5 as the partial area R1 and shows in Table 1. The visual field information of the avatar 4A at the times T1 to T5 of the visual field information data shown is updated. Further, the control unit 23 has the emotions of the avatar 4A in Table 1 based on the face information of the avatar 4A at each time T1 to T5 received from the control unit 121 of the user terminal 1A and the voice information of the user A at the time T2. Identify the information. At this time, the control unit 23 can linguistically analyze the received voice information and adjust the emotion information based on the analysis result of the voice information. In the case of this example, for example, it is judged that the voice information "Not so" is neither a positive evaluation nor a negative evaluation. Therefore, based on the face information (expressionless) of the avatar 4A between the times T1 and T5 and the voice information which is a zero evaluation, the control unit 121 "expresses" the emotional information of the avatar 4A at the times T1 to T5. (See Table 1). Then, the control unit 23 updates the emotion evaluation value of the avatar 4A based on the specified emotion information and the attributes of the 360-degree content distributed in the virtual space 200. For example, when the 360-degree content distributed in the virtual space 200 is an entertainment program, the control unit 23 evaluates the emotions of the avatar 4A at times T1 to T5 based on the expressionless emotion information of the avatar 4A. Update the value to "-1".

As shown in FIG. 15, for example, the avatar 4B gazes at the partial region R1 at the times T1 and T2, and subsequently, as shown in FIG. 16, the avatar 4C (partial region R2) is watched at the times T3 and T4. Then, at time T5, the partial region R1 is gazed again. It is assumed that the face information of the avatar 4B is a smile through the times T1 to T5. Further, it is assumed that the user B utters a voice "like" at a predetermined time between times T1 and T5 (for example, time T2). In this case, the control unit 23 of the server 2 that has received the time information and the field of view information from the control unit 121 of the user terminal 1B specifies the field of view information of the avatar 4B at each time T1, T2, T5 as the partial area R1. The visual field information of the avatar 4B in T3 and T4 is specified as the partial region R2, and the visual field information of the avatar 4B at the times T1 to T5 of the visual field information data shown in Table 1 is updated. Further, the control unit 23 has the emotions of the avatar 4B in Table 1 based on the face information of the avatar 4B at each time T1 to T5 received from the control unit 121 of the user terminal 1B and the voice information of the user B at the time T2. Identify the information. In the case of this example, for example, the voice information "like" is judged to be a positive evaluation. Therefore, the control unit 121 identifies the emotional information of the avatar 4B at the times T1 to T5 as "smile" based on the face information (smile) of the avatar 4B between the times T1 and T5 and the voice information which is a positive evaluation. (See Table 1). Then, the control unit 23 updates the emotion evaluation value of the avatar 4B based on the specified emotion information and the attribute of the 360-degree content distributed in the virtual space 200. Since the 360-degree content distributed in the virtual space 200 is an entertainment program, the control unit 23 sets the emotion evaluation value of the avatar 4B at times T1 to T5 to "" based on the emotion information of the avatar 4B being a smile. +1 "is updated.

If the avatar 4B is laughing and making a specific gesture (for example, a gesture that covers the avatar's mouth with a hand object or a gesture that hits a hand object), it is judged that the degree of laughter is high. , The emotional evaluation value may be further increased. That is, not only the face information and voice information of the avatar but also the gesture by the hand object may be added to specify the emotion. In addition, the degree of smile may be different depending on the type of gesture. For example, when the face information of the avatar is a smile, the control unit 23 determines, for example, that the degree of laughter is mild (smiley) if there is no specific gesture, and further adds an emotion evaluation value. I don't do that. When the hand object makes a gesture that covers the mouth of the avatar, the control unit 23 determines that the degree of laughter is moderate (medium laughter), and further adds an emotion evaluation value. When a gesture such as hitting a hand object is performed, the control unit 23 determines that the degree of laughter is high (loud laughter), and adds an emotion evaluation value even more than in the case of middle laughter.

As shown in FIG. 15, for example, it is assumed that the avatar 4C is gazing at the partial region R3 different from the partial region R1 and the partial region R2 between the times T1 and T5. It is assumed that the face information of the avatar 4C at times T1 to T5 is a surprised face. Further, it is assumed that the user C utters the voice "Oh!" At a predetermined time between times T1 and T5 (for example, time T2). In this case, the control unit 23 of the server 2 that has received the time information and the field of view information from the control unit 121 of the user terminal 1C identifies the field of view information of the avatar 4C at each time T1 to T5 as the partial area R3, and is shown in Table 1. The visual field information of the avatar 4C at the times T1 to T5 of the visual field information data shown is updated. Further, the control unit 23 has the emotions of the avatar 4C in Table 1 based on the face information of the avatar 4C at each time T1 to T5 received from the control unit 121 of the user terminal 1C and the voice information of the user C at the time T2. Identify the information. In the case of this example, for example, the voice information "Oh!" Is judged to be a positive evaluation. Therefore, the emotional information of the avatar 4C at the times T1 to T5 is specified as the "surprise face" based on the face information (surprise face) of the avatar 4C between the times T1 and T5 and the voice information which is a positive evaluation (table). See 1). Then, the control unit 23 updates the emotion evaluation value of the avatar 4C based on the specified emotion information and the attribute of the 360-degree content distributed in the virtual space 200. Since the 360-degree content distributed in the virtual space 200 is an entertainment program, the control unit 23 determines the avatar at times T1 to T5 based on the fact that the emotional information of the avatar 4C at times T1 to T5 is a surprised face. The emotion evaluation value of 4C is updated to "+2". If the avatar 4C is surprised to make a specific gesture (for example, a gesture that spreads a two-handed object), it may be judged that the degree of surprise is strong and a positive emotion evaluation value may be further added. ..

Next, in step S120, the server 2 updates the evaluation value data of each partial region. Table 2 shows an example of evaluation value data.

(Table 2)

The evaluation value data is data showing the evaluation value of each partial region (for example, partial regions R1 and R3) at each time (for example, times T1 to T5). The evaluation value of each subregion includes a total emotion evaluation value, a communication evaluation value, and a comprehensive evaluation value obtained by adding the total emotion evaluation value and the communication evaluation value. The total emotion evaluation value is an evaluation value obtained by adding up the emotion evaluation values of a plurality of avatars 4A, 4B, and 4C at each time (time T1 to T5). The communication evaluation value is, for example, an evaluation value associated with the communication behavior performed between the avatars 4A, 4B, and 4C in the virtual space 200. The communication action may include, for example, a conversation between avatars 4A, 4B, and 4C, and actions (gestures, etc.) of each avatar 4A, 4B, 4C. In this example, in particular, when a conversation or a specific gesture that encourages another avatar to gaze at a specific area of 360-degree content is made, an evaluation of the partial area corresponding to the specific area is made. Add the values.

FIG. 14 is a flowchart showing each step in step S120 for updating the evaluation value data of each partial region.
As shown in FIG. 14, first, in step S120-1, the control unit 23 of the server 2 adds up the emotion evaluation values of the plurality of avatars 4A, 4B, and 4C at each time (time T1 to T5). Update the total emotion evaluation value for each sub-region. Subsequently, the control unit 23 determines in step S120-2 whether or not a communication action has been performed between the avatars 4A, 4B, and 4C. When it is determined that a communication action has been performed between the avatars 4A, 4B, and 4C (Yes in step S120-2), in step S120-3, the control unit 23 uses information related to the communication action (hereinafter referred to as action information). ) Is received from each user terminal 1A, 1B, 1C. Then, in step S120-4, the control unit 23 updates the communication evaluation value based on the behavior information.

The communication behavior between the avatars 4A, 4B, and 4C will be described below with reference to FIG.
As shown in FIG. 16, for example, the avatar 4B faces the avatar 4C while pointing at the partial area R1 by the right-hand object 400R between the times T3 and T4, and speaks "that part was good." Suppose. At this time, the control unit 121 of the user terminal 1B transmits the conversation information "that part was good" and the gesture (gesture information) of pointing by the right-hand object 400R to the server 2 together with the time information. The control unit 23 of the server 2 linguistically analyzes the conversation information received from the user terminal 1B, and determines whether or not the communication action has been performed based on the analysis result of the conversation information and the gesture information. Here, the control unit 23 extracts various expressions included in the conversation information of the avatar 4A, for example, a linguistic expression (demonstrative word) or a tense expression pointing to a specific area, and analyzes the conversation information. For example, in the case of conversation information "that part was good", the control unit 23 determines that the demonstrative word "that part" indicates the partial area R1 based on the gesture information of the avatar 4B. .. Then, since the wording "good" includes the past tense expression, the control unit 23 determines that the communication action associated with the partial region R1 has been performed from the avatar 4B to the avatar 4C. Then, the control unit 23 sets the communication evaluation value of the partial region R1 to, for example, "+2" with respect to the time when the communication action is performed from the avatar 4B to the avatar 4C (for example, the times T3 to T4 while the utterance is being made). ". When the conversation information includes the present tense and the future tense, the demonstrative (for example, "that part") included in the conversation information does not necessarily indicate the partial area R1. Therefore, it may be possible not to give a positive communication evaluation value to the partial area R.

Further, as shown in FIG. 16, it is assumed that Avatar 4C, who was told by Avatar 4B that "that part was good", replied "That's right" to Avatar 4B. In this case, the control unit 121 of the user terminal 1C transmits the conversation information "That's right" to the server 2 together with the visual field information of the avatar 4C. The control unit 23 of the server 2 linguistically analyzes the conversation information received from the user terminal 1C, and determines that the avatar 4C has made an utterance to the effect that the avatar 4C agrees with the utterance of the avatar 4B based on the analysis result and the visual field information. do. The utterance of this avatar 4C does not include a demonstrative word indicating a specific subregion, and since the tense is the present tense, it is judged that the utterance is not an utterance that raises the communication evaluation value of the subregion. Therefore, the control unit 23 does not give a communication evaluation value to the reply "That's right" of this avatar 4C. If the utterance of Avatar 4C contains a demonstrative word indicating a specific subregion and the tense is in the past tense, the utterance raises the communication evaluation value of the specific subregion. Can be evaluated as.

Next, in step S120-5, the control unit 23 of the server 2 adds up the total emotion evaluation value and the communication evaluation value specified as described above, and updates the comprehensive evaluation value. For example, as shown in Table 2, the comprehensive evaluation value at the time T1 and T2 of the partial region R1 is “0”, the comprehensive evaluation value at the time T3 and T4 of the partial region R1 is “+1”, and the time of the partial region R1. The overall evaluation value at T5 is "+3".

Returning to FIG. 13, next, in step S122, the control unit 23 of the server 2 determines whether or not there is a partial region whose comprehensive evaluation value is equal to or greater than the threshold value (for example, “+3”). If it is determined that there is no partial region whose comprehensive evaluation value is equal to or greater than the threshold value (No in step S122), the processes after step S108 are repeated. On the other hand, when it is determined that there is a partial region whose comprehensive evaluation value is equal to or higher than the threshold value (Yes in step S122), in step S124, the control unit 23 determines the partial region whose comprehensive evaluation value is equal to or higher than the threshold value. Specify as the area of interest. For example, in the example shown in Table 2, since the comprehensive evaluation value of the partial region R1 at the time T5 is “+4”, the control unit 23 designates the partial region R1 at the time T5 as the region of interest.

Next, in step S126, the control unit 23 of the server 2 has a video relating to the region of interest (hereinafter referred to as a video of interest) in a predetermined time width (example of the first time width) including the time when the comprehensive evaluation value becomes equal to or higher than the threshold value. To generate). As the video of interest, for example, for the partial region R1, it is possible to generate a video of 5 seconds before and after including the time T5 whose overall evaluation value is “+3”. As shown in FIG. 17, this attention image is obtained by capturing the attention area R1 in the virtual space 200 with, for example, a virtual camera 300A for generating the attention image arranged at the center of the celestial sphere constituting the virtual space 200. It may be generated. The avatars 4A, 4B, 4C and the virtual camera 300 associated with these avatars 4A, 4B, 4C are not necessarily located at the center of the celestial sphere constituting the virtual space 200, but at the center of the virtual space 200. By imaging the region of interest with the arranged virtual camera 300 and generating the image of interest, it is possible to generate an appropriate image of interest from an easy-to-see angle. The attention image may be generated as a three-dimensional image or may be generated as a two-dimensional image. When a plurality of attention videos are generated in step S126, the control unit 23 may generate digest video contents by connecting the plurality of attention videos.

Next, in step S128, the control unit 23 of the server 2 distributes the generated digest video content (or attention video) to the video distribution service on the web browser through the network 3. At this time, the control unit 23 collects the video viewing information of a large number of users, refers to the viewing history information of each user, and the history data regarding the region of interest (for example, the partial region R1) is the attribute of the target digest video content. A user terminal (user ID) similar to the above may be specified, and the digest video content may be distributed to the user terminal. For example, the control unit 23 estimates the user attribute of each user from the collected viewing history information, and provides the digest video content to the user whose user attribute is close to the attribute of the target digest video content. The digest video content is, for example, a social networking service registered by each user terminal (for example, LINE (registered trademark), Facebook (FACEBOOK (registered trademark)), Twitter (TWITTER (registered trademark))). Delivered to the timeline. Further, in this way, the control unit 23 of the server 2 picks up a user who will be interested in the target digest video content from a large number of users, and recommends the digest video content to the user. It is possible to enhance the advertising effect of 360-degree content. Further, when users A, B, and C who have completed viewing the 360-degree content by wearing the HMD 110 attach the HMD 110 again after removing the HMD 110 and try to start viewing another 360-degree content. The control unit 23 of the server 2 recommends digest images of various 360-degree contents according to the interests of each user A, B, C based on the history information of the attention area of each user A, B, C. It may be displayed on the display unit 112 of the HMD 110 worn by each user A, B, C.

As described above, according to the present embodiment, the control unit 23 of the server 2 identifies the emotions of the users A, B, and C wearing the HMD 110, and the identified emotions are predetermined conditions (first). When the condition (an example of the condition) is satisfied, the evaluation value of the region (an example of the first region) in the virtual space 200 according to the orientation of the HMD 110 at the time when the condition is satisfied is updated. As a result, the time when a specific emotion of the users A, B, and C is detected is specified as the time when the 360-degree content is excited, and the direction (field of view) of the HMD 110 worn by each user A, B, C at that time is set. The evaluation value of the region of interest can be updated accordingly. Therefore, the true excitement point of the 360-degree content is not overlooked, and the attention area (attention range) of the 360-degree content in the virtual space 200 can be appropriately specified.

Further, when the communication behavior of the users A, B, and C satisfies a predetermined condition (an example of the second condition) after the emotions of the users A, B, and C satisfy the first condition, the control unit 23 determines. The evaluation value may be further updated. By adjusting the importance (attention) of the attention area based on the communication behavior of users A, B, C in addition to the emotions of users A, B, C, it is possible to improve the accuracy of specifying the attention range. can.

The condition regarding the communication behavior of the users A, B, and C may include a conversation between a plurality of avatars 4A, 4B, and 4C in the virtual space 200. It is preferable to use conversations made between a plurality of avatars 4A, 4B, 4C in order to adjust the importance of the region of interest. It should be noted that the users A, B, and C can also have a direct conversation without going through the avatars 4A, 4B, and 4C arranged in the virtual space 200, and the communication behavior between the users A, B, and C in this case can be performed. It may be used for updating the evaluation value. Alternatively, users A, B, and C can have a direct conversation with each other by making a call from the virtual space 200, which is a VR space, to the real space.

If the conversation between the avatars 4A, 4B, 4C or between the users A, B, C contains a linguistic expression pointing to a specific subregion, the evaluation score may be further updated. Further, when the conversation between the avatars 4A, 4B, 4C or between the users A, B, and C includes a tense expression indicating the time, the evaluation score may be further updated. By using various expressions in the conversation in this way, the importance of the region of interest can be easily adjusted.

The communication behavior of the users A, B, and C used to update the evaluation value is not limited to the conversation between the avatars 4A, 4B, and 4C or between the users A, B, and C. For example, when users A, B, and C take a picture (moving image) of a specific partial area in the virtual space 200, the evaluation value of the pictured area may be updated. Further, when users A, B, and C take a picture of a specific area (for example, a partial area R1) in the virtual space 200 and post the picture to a social networking site (SNS), the area (partial area R1). ) May be updated. As described above, as a communication action for adjusting the importance of the area of interest, it is also possible to use a photography action in the virtual space 200 or a posting action to the SNS.

In the above embodiment, the evaluation value of the specific partial region is updated based on the visual field information and the face information of the avatars 4A, 4B, 4C, and in addition, between the avatars 4A, 4B, 4C, or the user A, The evaluation value of each subregion R is further updated based on the communication behavior between B and C, but the present invention is not limited to this example. For example, based on the communication behavior between avatars 4A, 4B, 4C, or between users A, B, C without updating the evaluation value of each partial region R based on the face information of avatars 4A, 4B, 4C. , The evaluation value of each partial region R may be updated.

Although the embodiments of the present disclosure have been described above, the technical scope of the present invention should not be construed as being limited by the description of the present embodiments. This embodiment is an example, and it is understood by those skilled in the art that various embodiments can be modified within the scope of the invention described in the claims. The technical scope of the present invention should be determined based on the scope of the invention described in the claims and the equivalent scope thereof.

In order to realize various processes executed by the control unit 121 by software, a display control program for causing a computer (processor) to execute the display control method according to the present embodiment is preliminarily incorporated in the storage unit 123 or the ROM. May be good. Alternatively, the display control program includes a magnetic disk (HDD, floppy disk), an optical disk (CD-ROM, DVD-ROM, Blu-ray disk, etc.), an optical magnetic disk (MO, etc.), a flash memory (SD card, USB memory, etc.). It may be stored in a computer-readable storage medium such as SSD). In this case, by connecting the storage medium to the control device 120, the program stored in the storage medium is incorporated into the storage unit 123. Then, the display control program incorporated in the storage unit 123 is loaded onto the RAM, and the processor executes the loaded program, so that the control unit 121 executes the display control method according to the present embodiment.

The display control program may be downloaded from a computer on the communication network 3 via the communication interface 125. In this case as well, the downloaded program is incorporated into the storage unit 123.

In the description of the present embodiment, it is assumed that the virtual space data indicating the virtual space 200 is updated on the user terminal 1 side, but the virtual space data may be updated on the server 2 side. Further, although it is premised that the visual field image data corresponding to the visual field image is updated on the user terminal 1, the visual field image data may be updated on the server 2 side. In this case, the user terminal 1 displays the field of view image on the HMD 110 based on the field of view image data transmitted from the server 2.

Further, in order to realize various processes executed by the control unit 121 of the user terminal 1 by software, even if a control program for causing a computer (processor) to execute various processes is preliminarily incorporated in the storage unit 123 or the memory. good. Alternatively, the control program includes a magnetic disk (HDD, floppy disk), an optical disk (CD-ROM, DVD-ROM, Blu-ray (registered trademark) disk, etc.), an optical magnetic disk (MO, etc.), a flash memory (SD card, SD card, etc.). It may be stored in a computer-readable storage medium such as a USB memory, SSD, etc.). In this case, by connecting the storage medium to the control device 120, the control program stored in the storage medium is incorporated into the storage unit 123. Then, the control program incorporated in the storage unit 123 is loaded onto the RAM, and the processor executes the loaded program, so that the control unit 121 executes various processes.

Further, the control program may be downloaded from the computer on the communication network 3 via the communication interface 125. In this case as well, the downloaded control program is incorporated into the storage unit 123.

In the present embodiment, the virtual space is used to provide a virtual experience such as VR (Virtual Reality), AR (Augmented Reality) and MR (Mixed Reality). When the virtual space provides VR, the background data stored in the memory is used as the background of the virtual space. If the virtual space provides AR or MR, the real space is used as the background. In this case, the HMD 110 is provided with a transmissive display device (optical see-through or video see-through type display device), so that the real space can be used as a background. If virtual space is applied to MR, the object may be influenced by real space. As described above, by including the virtual scene such as the background and the virtual object in at least a part of the virtual space, the user can be provided with a virtual experience capable of interacting with the virtual scene. Further, when the virtual space is applied to AR or MR, the user's hand may be used instead of the hand object. In this case, the user's left hand is placed in the virtual space 200 instead of the left-hand object 400L shown in FIG. 16, and the user's right hand is placed in the virtual space 200 instead of the right-hand object 400R. Then, the server 2 can determine whether or not the communication action has been performed based on the gesture by the user.

1 (1A-1C): User terminal (an example of a terminal device)
2: Server 3: Communication network 4A-4C: Avatar 110: Head-mounted device (HMD)
112: Display unit 114: Sensor 116: Headphones 118: Microphone 120: Control device 121: Control unit 123: Storage unit (storage)
124: I / O interface 125: Communication interface 130: Position sensor 140: Gaze sensor 200, 200A: Virtual space 300: Virtual camera 320: External controller 400L: Left-hand object 400R: Right-hand object C: Intersection R (R1 to R3): Subregion

Claims (13)

Information processing method in a system equipped with a head mount device.
Steps to generate virtual space data that defines virtual space,
A step of generating field image data indicating a field image displayed on the head mount device based on the movement of the head mount device and the virtual space data.
The step of identifying the behavior of the user wearing the head mount device, and
When the specified action satisfies a predetermined first condition, a step of updating the evaluation value of the first region in the virtual space according to the orientation of the head mount device, and a step of updating the evaluation value.
A step of generating an image relating to the first region based on the evaluation value, and
Including
At least the facial expression of the user is specified in the step of identifying the behavior of the user.
The step of updating the evaluation value is an information processing method including evaluation of information regarding the facial expression . The information processing method according to claim 1, further comprising a step of further updating the evaluation value when the action satisfies the second condition. The information processing method according to claim 2, wherein the second condition includes at least one of a conversation and a gesture being performed between a plurality of users in the virtual space. The information processing method according to claim 3, wherein the conversation includes a linguistic expression pointing to the first region. The information processing method according to claim 3 or 4, wherein the conversation includes a tense expression indicating a time when the first condition is satisfied. The information processing method according to any one of claims 2 to 5, wherein the second condition includes the user taking a picture of the first region in the virtual space. The information processing method according to claim 6, wherein the second condition includes an action of the user posting a photograph of the first area taken in the virtual space to a social networking site. When the evaluation value satisfies the third condition, the step of designating the first region as the region of interest at the time when the first condition is satisfied, and
A step of generating an image relating to the region of interest over the first time width including the time.
The information processing method according to any one of claims 1 to 7, further comprising. The image is generated based on the position of the region of interest in the virtual space and a virtual camera placed in the virtual space to define the field image based on the movement of the head mount device. Item 8. The information processing method according to Item 8. Information processing method in a system equipped with a head mount device.
Steps to generate virtual space data that defines virtual space,
A step of generating field image data indicating a field image displayed on the head mount device based on the movement of the head mount device and the virtual space data.
The step of identifying the emotion of the user wearing the head mount device, and
When the specified emotion satisfies a predetermined second condition, a step of updating the evaluation value of the first region in the virtual space according to the orientation of the head mount device, and
A step of generating an image relating to the first region based on the evaluation value, and
Information processing methods, including. An information processing program for causing a computer to execute the information processing method according to any one of claims 1 to 10. An information processing system comprising a head mount device and an information processing device communicably connected to the head mount device to execute the information processing method according to any one of claims 1 to 10. With the processor
An information processing device equipped with a memory for storing computer-readable instructions.
An information processing apparatus that, when the computer-readable instruction is executed by the processor, executes the information processing method according to any one of claims 1 to 10.

Images