JP6947661B2 - A program executed by a computer capable of communicating with the head mount device, an information processing device for executing the program, and a method executed by a computer capable of communicating with the head mount device. - Google Patents

Description

The disclosure relates to a technique for capturing the interest of a user using a headmount device, and more specifically to a technique for capturing the interest of a user based on the line of sight.

In recent years, technological developments for providing virtual reality using a head-mounted device (HMD) device have been actively carried out.

For example, Patent Document 1 discloses a technique for acquiring the line of sight of a user wearing an HMD. Further, Non-Patent Document 1 discloses a technique of aiming at an object by using the line of sight of a user in a shooting game in a virtual space.

U.S. Patent Application Publication No. 2016/0038069

"The experience changes so far with just one line of sight. VR headset with eye tracking system" FOVE "", [online], [Search on April 20, 2017], Internet <URL: http://www.gizmodo.jp /2016/09/tgs2016-vr-fove.html>

By the way, in recent years, various techniques for analyzing user behavior have been developed for marketing purposes. For example, there is a technique for indicating where many touches are made for each screen of an application that uses a touch panel. The technology for analyzing such user behavior is also required in the technical field of virtual space.

The present disclosure has been made to solve the above-mentioned problems, and an object in a certain aspect is to provide a technique for analyzing a user who uses an HMD.

According to certain embodiments, a program is provided that is executed by a computer capable of communicating with the head mount device. This program gives the computer a step of defining a virtual space, a step of acquiring the user's viewpoint position in the virtual space based on the output of the headmount device, and a step of receiving a signal representing the user's operation or operation. And, when the user's operation or action represented by the signal indicates the user's interest, the step of saving the viewpoint position in the memory is executed.

The above and other objectives, features, aspects and advantages of the disclosed technical features will become apparent from the following detailed description of the invention as understood in connection with the accompanying drawings.

It is a figure which shows the outline of the structure of the HMD system. It is a block diagram which shows an example of the hardware composition of the computer which follows a certain aspect. It is a figure which conceptually represents the uvw field-of-view coordinate system set in the HMD according to a certain embodiment. It is a figure that conceptually represents one aspect of expressing a virtual space according to a certain embodiment. It is a figure which showed the head of the user who wears an HMD according to a certain embodiment from the top. It is a figure which shows the YZ cross section which looked at the visual area in the virtual space from the X direction. It is a figure which shows the XZ cross section which looked at the visual area in a virtual space from the Y direction. It is a block diagram which shows the computer according to a certain embodiment as a module structure. FIG. 5 is a schematic diagram showing a situation in which each of a plurality of HMDs provides a virtual space to each of a plurality of users in a network. FIG. 9A is a diagram showing a user's field of view image in FIG. 9A. It is a figure explaining the process of detecting a mouth from a user's face image. It is a figure (the 1) for demonstrating the process which the motion detection module detects the shape of a mouth. It is a figure (the 2) for demonstrating the process which the motion detection module detects the shape of a mouth. It is a figure which shows an example of the structure of face tracking data. It is a figure explaining the hardware configuration and the module configuration of a server. It is a flowchart which shows the process which a server communicates with a computer and updates a user information. It is a figure which shows the field of view image which a user visually recognizes. It is a figure which shows the virtual space corresponding to the state of FIG. It is a figure which shows an example of the data structure of a viewpoint position information. It is a figure which shows the feature point of the face acquired when the user is expressionless. It is a figure which shows the feature point of the face acquired when the user is surprised. It is a flowchart which shows the process of storing a viewpoint position in storage according to a certain embodiment. It is a flowchart which shows the process of associating and saving a viewpoint position and an emotion type. It is a figure which shows the heat map based on the viewpoint position information. It is a flowchart which shows an example of a series of processing until the advertisement is delivered by specifying the target which the user looks at. It is a figure which shows an example of the data structure of a panoramic image DB. It is a figure which shows an example of the data structure of the 1st table. It is a figure which shows an example of the data structure of the 2nd table. It is a figure for demonstrating the process which recommends a panoramic image to a user. It is a figure for demonstrating the process when the viewpoint position is not saved in the viewpoint position information. It is a flowchart which shows the process of canceling the process of saving a viewpoint position in the viewpoint position information. It is a flowchart for demonstrating the process of step S2920.

Hereinafter, embodiments of this technical idea will be described in detail with reference to the drawings. In the following description, the same parts are designated by the same reference numerals. Their names and functions are the same. Therefore, the detailed description of them will not be repeated. In addition, each embodiment described below may be selectively combined as appropriate.

[HMD system configuration]
The configuration of the HMD (Head-Mounted Device) system 100 will be described with reference to FIG. FIG. 1 is a diagram showing an outline of the configuration of the HMD system 100. The HMD system 100 is provided as a home system or a business system.

The HMD system 100 includes an HMD (Head-Mounted Device) set 105A, 105B, 105C, 105D, a network 19, and a server 150. Each of the HMD sets 105A, 105B, 105C, and 105D is configured to be able to communicate with the server 150 via the network 19. Hereinafter, the HMD set 105A, 105B, 105C, and 105D are collectively referred to as the HMD set 105. The number of HMD sets 105 constituting the HMD system 100 is not limited to four, and may be three or less or five or more. The HMD set 105 includes an HMD 110, an HMD sensor 120, a controller 160, and a computer 200. The HMD 110 includes a monitor 112, a first camera 115, a second camera 117, a speaker 118, a microphone 119, and a gaze sensor 140. The controller 160 may include a motion sensor 130.

In some aspects, the computer 200 can connect to the Internet or other network 19 and communicate with the server 150 or other computer connected to the network 19 (eg, another computer in the HMD set 105). In another aspect, the HMD 110 may include a sensor 114 instead of the HMD sensor 120.

The HMD 110 may be worn on the head of user 190 and provide virtual space to user 190 during operation. More specifically, the HMD 110 displays an image for the right eye and an image for the left eye on the monitor 112, respectively. When each eye of the user 190 visually recognizes the respective image, the user 190 can recognize the image as a three-dimensional image based on the parallax of both eyes. The HMD 100 may include both a so-called head-mounted display including a monitor and a head-mounted device capable of mounting a smartphone or other terminal having a monitor.

The monitor 112 is realized as, for example, a non-transparent display device. In one aspect, the monitor 112 is located in the body of the HMD 110 so that it is located in front of both eyes of the user 190. Therefore, the user 190 can immerse himself in the virtual space when he / she visually recognizes the three-dimensional image displayed on the monitor 112. In certain embodiments, the virtual space includes, for example, a background, objects that can be manipulated by user 190, and images of menus that can be selected by user 190. In certain embodiments, the monitor 112 can be realized as a liquid crystal monitor or an organic EL (Electro Luminescence) monitor included in a so-called smartphone or other information display terminal.

In another aspect, the monitor 112 can be realized as a transmissive display device. In this case, the HMD 110 may be an open type such as a glasses type rather than a closed type that covers the eyes of the user 190 as shown in FIG. The transmissive monitor 112 may be temporarily configured as a non-transparent display device by adjusting its transmittance. Further, the monitor 112 may include a configuration in which a part of the image constituting the virtual space and the real space are displayed at the same time. For example, the monitor 112 may display an image of the real space taken by the camera mounted on the HMD 110, or may make the real space visible by setting a part of the transmittance to be high.

In some aspects, the monitor 112 may include a sub-monitor for displaying an image for the right eye and a sub-monitor for displaying an image for the left eye. In another aspect, the monitor 112 may be configured to display the image for the right eye and the image for the left eye as a unit. In this case, the monitor 112 includes a high speed shutter. The high-speed shutter operates so that the image for the right eye and the image for the left eye can be alternately displayed so that the image is recognized by only one of the eyes.

In one aspect, the HMD 110 includes a plurality of light sources (not shown). Each light source is realized by, for example, an LED (Light Emitting Diode) that emits infrared rays. The HMD sensor 120 has a position tracking function for detecting the movement of the HMD 110. More specifically, the HMD sensor 120 reads a plurality of infrared rays emitted by the HMD 110 and detects the position and inclination of the HMD 110 in the real space.

In another aspect, the HMD sensor 120 may be realized by a camera. In this case, the HMD sensor 120 can detect the position and tilt of the HMD 110 by executing the image analysis process using the image information of the HMD 110 output from the camera.

In another aspect, the HMD 110 may include a sensor 114 as a position detector in place of the HMD sensor 120 or in addition to the HMD sensor 120. The HMD 110 may use the sensor 114 to detect the position and tilt of the HMD 110 itself. For example, if the sensor 114 is an angular velocity sensor, a geomagnetic sensor, or an acceleration sensor, the HMD 110 may use any of these sensors instead of the HMD sensor 120 to detect its position and tilt. As an example, when the sensor 114 is an angular velocity sensor, the angular velocity sensor detects the angular velocity around the three axes of the HMD 110 in real space over time. The HMD 110 calculates the temporal change of the angle around the three axes of the HMD 110 based on each angular velocity, and further calculates the inclination of the HMD 110 based on the temporal change of the angle.

The first camera 115 captures the lower part of the face of the user 190. More specifically, the first camera 115 captures the nose, mouth, and the like of the user 190. The second camera 117 photographs the eyes, eyebrows, and the like of the user 190. The housing on the user 190 side of the HMD 110 is defined as the inside of the HMD 110, and the housing on the side opposite to the user 190 of the HMD 110 is defined as the outside of the HMD 110. In some aspects, the first camera 115 may be located outside the HMD 110 and the second camera 117 may be located inside the HMD 110. The images generated by the first camera 115 and the second camera 117 are input to the computer 200.

The speaker 118 converts the voice signal into voice and outputs it to the user 190. The microphone 119 converts the utterance of the user 190 into an audio signal (electric signal) and outputs it to the computer 200. In another aspect, the HMD 110 may include earphones instead of the speaker 118.

The gaze sensor 140 detects the direction (line of sight) in which the eyes of the right eye and the left eye of the user 190 are directed. The detection of the direction is realized by, for example, a known eye tracking function. The gaze sensor 140 is realized by a sensor having the eye tracking function. In certain aspects, the gaze sensor 140 preferably includes a sensor for the right eye and a sensor for the left eye. The gaze sensor 140 may be, for example, a sensor that irradiates the right eye and the left eye of the user 190 with infrared light and detects the angle of rotation of each eyeball by receiving the reflected light from the cornea and the iris with respect to the irradiation light. .. The gaze sensor 140 can detect the line of sight of the user 190 based on each of the detected rotation angles.

The server 150 may send the program to the computer 200. In another aspect, the server 150 may communicate with another computer 200 to provide virtual reality to the HMD used by other users. For example, in an amusement facility, when a plurality of users play a participatory game, each computer 200 communicates a signal based on the operation of each user with another computer 200, and the plurality of users are common in the same virtual space. Allows you to enjoy the game.

The controller 160 is connected to the computer 200 by wire or wirelessly. The controller 160 receives an instruction input from the user 190 to the computer 200. In one aspect, the controller 160 is configured to be grippable by the user 190. In another aspect, the controller 160 is configured to be wearable on a body or part of clothing of the user 190. In another aspect, the controller 160 may be configured to output at least one of vibration, sound, and light based on a signal transmitted from the computer 200. In another aspect, the controller 160 receives from the user 190 an operation for controlling the position and movement of an object arranged in the virtual space.

The motion sensor 130 is attached to the hand of the user 190 in a certain aspect to detect the movement of the hand of the user 190. For example, the motion sensor 130 detects the rotation speed, the number of rotations, and the like of the hand. The detected signal is sent to the computer 200. The motion sensor 130 is provided in, for example, a glove-shaped controller 160. In certain embodiments, for safety in real space, it is desirable that the controller 160 be attached to something that does not easily fly by being attached to the user 190's hand, such as a glove type. In another aspect, a sensor not attached to the user 190 may detect the movement of the user 190's hand. For example, the signal of the camera that captures the user 190 may be input to the computer 200 as a signal indicating the operation of the user 190. As an example, the motion sensor 130 and the computer 200 are wirelessly connected to each other. In the case of wireless communication, the communication mode is not particularly limited, and for example, Bluetooth (registered trademark) or other known communication method is used.

[Hardware configuration]
The computer 200 according to the present embodiment will be described with reference to FIG. FIG. 2 is a block diagram showing an example of the hardware configuration of the computer 200 according to a certain aspect. The computer 200 includes a processor 10, a memory 11, a storage 12, an input / output interface 13, and a communication interface 14 as main components. Each component is connected to the bus 15.

The processor 10 executes a series of instructions included in the program stored in the memory 11 or the storage 12 based on the signal given to the computer 200 or when a predetermined condition is satisfied. In a certain aspect, the processor 10 is realized as a CPU (Central Processing Unit), an MPU (Micro Processor Unit), an FPGA (Field-Programmable Gate Array) or other device.

The memory 11 temporarily stores programs and data. The program is loaded from storage 12, for example. The data includes data input to the computer 200 and data generated by the processor 10. In a certain aspect, the memory 11 is realized as a RAM (Random Access Memory) or other volatile memory.

The storage 12 permanently holds programs and data. The storage 12 is realized as, for example, a ROM (Read-Only Memory), a hard disk device, a flash memory, or other non-volatile storage device. The program stored in the storage 12 includes a program for providing a virtual space in the HMD system 100, a simulation program, a game program, a user authentication program, and a program for realizing communication with another computer 200. The data stored in the storage 12 includes data, objects, and the like for defining the virtual space.

In another aspect, the storage 12 may be realized as a detachable storage device such as a memory card. In yet another aspect, a configuration that uses programs and data stored in an external storage device may be used instead of the storage 12 built into the computer 200. According to such a configuration, for example, in a scene where a plurality of HMD systems 100 are used such as an amusement facility, it is possible to update programs and data at once.

In certain embodiments, the input / output interface 13 communicates signals with the HMD 110, the HMD sensor 120, and the motion sensor 130. In a certain aspect, the first camera 115, the second camera 117, the speaker 118, and the microphone 119 included in the HMD 110 may communicate with the computer 200 via the input / output interface 13 of the HMD 110. In a certain aspect, the input / output interface 13 is realized by using USB (Universal Serial Bus), DVI (Digital Visual Interface), HDMI (registered trademark) (High-Definition Multimedia Interface) and other terminals. The input / output interface 13 is not limited to the above.

In certain embodiments, the input / output interface 13 may further communicate with the controller 160. For example, the input / output interface 13 receives input of signals output from the controller 160 and the motion sensor 130. In another aspect, the input / output interface 13 sends an instruction output from the processor 10 to the controller 160. The command instructs the controller 160 to vibrate, output voice, emit light, and the like. Upon receiving the instruction, the controller 160 executes either vibration, audio output, or light emission in response to the instruction.

The communication interface 14 is connected to the network 19 and communicates with another computer (for example, the server 150) connected to the network 19. In a certain aspect, the communication interface 14 is realized as, for example, a LAN (Local Area Network) or other wired communication interface, or a WiFi (Wireless Fidelity), Bluetooth (registered trademark), NFC (Near Field Communication) or other wireless communication interface. Will be done. The communication interface 14 is not limited to the above.

In one aspect, the processor 10 accesses the storage 12, loads one or more programs stored in the storage 12 into the memory 11, and executes a series of instructions contained in the program. The one or more programs may include an operating system of a computer 200, an application program for providing a virtual space, game software that can be executed in the virtual space, and the like. The processor 10 sends a signal for providing a virtual space to the HMD 110 via the input / output interface 13. The HMD 110 displays an image on the monitor 112 based on the signal.

In the example shown in FIG. 2, the computer 200 is configured to be provided outside the HMD 110, but in another aspect, the computer 200 may be built in the HMD 110. As an example, a portable information communication terminal (for example, a smartphone) including a monitor 112 may function as a computer 200.

Further, the computer 200 may have a configuration commonly used for a plurality of HMD 110s. According to such a configuration, for example, the same virtual space can be provided to a plurality of users, so that each user can enjoy the same application as other users in the same virtual space.

In certain embodiments, the HMD system 100 has a preset global coordinate system. The global coordinate system has three reference directions (axes) that are parallel to the vertical direction in the real space, the horizontal direction orthogonal to the vertical direction, and the front-back direction orthogonal to both the vertical direction and the horizontal direction. In the present embodiment, the global coordinate system is one of the viewpoint coordinate systems. Therefore, the horizontal direction, the vertical direction (vertical direction), and the front-back direction in the global coordinate system are defined as the x-axis, the y-axis, and the z-axis, respectively. More specifically, in the global coordinate system, the x-axis is parallel to the horizontal direction of real space. The y-axis is parallel to the vertical direction in real space. The z-axis is parallel to the front-back direction of the real space.

In some aspects, the HMD sensor 120 includes an infrared sensor. When the infrared sensor detects infrared rays emitted from each light source of the HMD 110, the presence of the HMD 110 is detected. The HMD sensor 120 further detects the position and inclination (orientation) of the HMD 110 in the real space according to the movement of the user 190 wearing the HMD 110 based on the value of each point (each coordinate value in the global coordinate system). do. More specifically, the HMD sensor 120 can detect a temporal change in the position and inclination of the HMD 110 by using each value detected over time.

The global coordinate system is parallel to the coordinate system in real space. Therefore, each inclination of the HMD 110 detected by the HMD sensor 120 corresponds to each inclination of the HMD 110 around three axes in the global coordinate system. The HMD sensor 120 sets the uvw field of view coordinate system to the HMD 110 based on the tilt of the HMD 110 in the global coordinate system. The uvw field-of-view coordinate system set in the HMD 110 corresponds to the viewpoint coordinate system when the user 190 wearing the HMD 110 sees an object in the virtual space.

[Uvw field coordinate system]
The uvw field coordinate system will be described with reference to FIG. FIG. 3 is a diagram conceptually representing the uvw field coordinate system set in the HMD 110 according to an embodiment. The HMD sensor 120 detects the position and tilt of the HMD 110 in the global coordinate system when the HMD 110 is activated. The processor 10 sets the uvw field of view coordinate system to the HMD 110 based on the detected values.

As shown in FIG. 3, the HMD 110 sets a three-dimensional uvw visual field coordinate system centered (origin) on the head of the user 190 wearing the HMD 110. More specifically, the HMD 110 defines the global coordinate system in the horizontal, vertical, and front-back directions (x-axis, y-axis, z-axis) by the inclination of the HMD 110 around each axis in the global coordinate system. The three directions newly obtained by tilting each around the axis are set as the pitch axis (u axis), the yaw axis (v axis), and the roll axis (w axis) of the uvw field coordinate system in the HMD 110.

In a certain aspect, when the user 190 wearing the HMD 110 is upright and visually recognizing the front, the processor 10 sets the uvw field coordinate system parallel to the global coordinate system to the HMD 110. In this case, the horizontal direction (x-axis), vertical direction (y-axis), and front-back direction (z-axis) in the global coordinate system are the pitch axis (u-axis) and yaw-axis (v-axis) of the uvw field coordinate system in the HMD 110. , And the roll axis (w axis).

After the uvw field coordinate system is set to the HMD 110, the HMD sensor 120 can detect the tilt of the HMD 110 in the set uvw field coordinate system based on the movement of the HMD 110. In this case, the HMD sensor 120 detects the pitch angle (θu), yaw angle (θv), and roll angle (θw) of the HMD 110 in the uvw visual field coordinate system as the inclination of the HMD 110, respectively. The pitch angle (θu) represents the tilt angle of the HMD 110 around the pitch axis in the uvw visual field coordinate system. The yaw angle (θv) represents the tilt angle of the HMD 110 around the yaw axis in the uvw visual field coordinate system. The roll angle (θw) represents the tilt angle of the HMD 110 around the roll axis in the uvw field coordinate system.

The HMD sensor 120 sets the uvw field coordinate system in the HMD 110 after the HMD 110 has moved to the HMD 110 based on the detected inclination of the HMD 110. The relationship between the HMD 110 and the uvw field coordinate system of the HMD 110 is always constant regardless of the position and inclination of the HMD 110. When the position and inclination of the HMD 110 change, the position and inclination of the uvw visual field coordinate system of the HMD 110 in the global coordinate system change in conjunction with the change of the position and inclination.

In one aspect, the HMD sensor 120 is based on the infrared light intensity acquired based on the output from the infrared sensor and the relative positional relationship between the points (eg, the distance between the points). The position of the light in the real space may be specified as a relative position with respect to the HMD sensor 120. Further, the processor 10 may determine the origin of the uvw visual field coordinate system of the HMD 110 in the real space (global coordinate system) based on the specified relative position.

[Virtual space]
The virtual space will be further described with reference to FIG. FIG. 4 is a diagram conceptually representing one aspect of expressing the virtual space 2 according to a certain embodiment. The virtual space 2 has an all-sky spherical structure that covers the entire center 21 in the 360-degree direction. In FIG. 4, the celestial sphere in the upper half of the virtual space 2 is illustrated so as not to complicate the explanation. Each mesh is defined in the virtual space 2. The position of each mesh is predetermined as a coordinate value in the XYZ coordinate system defined in the virtual space 2. The computer 200 associates each partial image constituting the panoramic image 22 (still image, moving image, etc.) expandable in the virtual space 2 with each corresponding mesh in the virtual space 2.

In a certain aspect, the virtual space 2 defines an XYZ coordinate system with the center 21 as the origin. The XYZ coordinate system is, for example, parallel to the global coordinate system. Since the XYZ coordinate system is a kind of viewpoint coordinate system, the horizontal direction, the vertical direction (vertical direction), and the front-back direction in the XYZ coordinate system are defined as the X-axis, the Y-axis, and the Z-axis, respectively. Therefore, the X-axis (horizontal direction) of the XYZ coordinate system is parallel to the x-axis of the global coordinate system, and the Y-axis (vertical direction) of the XYZ coordinate system is parallel to the y-axis of the global coordinate system. The Z-axis (front-back direction) is parallel to the z-axis of the global coordinate system.

When the HMD 110 is activated, that is, in the initial state of the HMD 110, the virtual camera 1 is arranged at the center 21 of the virtual space 2. In a certain aspect, the processor 10 displays an image captured by the virtual camera 1 on the monitor 112 of the HMD 110. The virtual camera 1 moves in the virtual space 2 in the same manner in conjunction with the movement of the HMD 110 in the real space. As a result, changes in the position and inclination of the HMD 110 in the real space can be similarly reproduced in the virtual space 2.

As in the case of the HMD 110, the virtual camera 1 is defined with an uvw field-of-view coordinate system. The uvw field-of-view coordinate system of the virtual camera in the virtual space 2 is defined to be linked to the uvw field-of-view coordinate system of the HMD 110 in the real space (global coordinate system). Therefore, when the inclination of the HMD 110 changes, the inclination of the virtual camera 1 also changes accordingly. Further, the virtual camera 1 can also move in the virtual space 2 in conjunction with the movement of the user 190 wearing the HMD 110 in the real space.

The processor 10 of the computer 200 defines the viewing area 23 in the virtual space 2 based on the position and inclination (reference line of sight 5) of the virtual camera 1. The viewing area 23 corresponds to the area of the virtual space 2 that is visually recognized by the user 190 wearing the HMD 110. That is, the position of the virtual camera 1 can be said to be the viewpoint of the user 190 in the virtual space 2.

The line of sight of the user 190 detected by the gaze sensor 140 is a direction in the viewpoint coordinate system when the user 190 visually recognizes an object. The uvw field-of-view coordinate system of the HMD 110 is equal to the viewpoint coordinate system when the user 190 visually recognizes the monitor 112. Further, the uvw field-of-view coordinate system of the virtual camera 1 is linked to the uvw field-of-view coordinate system of the HMD 110. Therefore, the HMD system 100 according to a certain aspect can consider the line of sight of the user 190 detected by the gaze sensor 140 as the line of sight of the user 190 in the uvw field of view coordinate system of the virtual camera 1.

[User's line of sight]
The determination of the line of sight of the user will be described with reference to FIG. FIG. 5 is a top view of the head of the user 190 wearing the HMD 110 according to an embodiment.

In one aspect, the gaze sensor 140 detects each line of sight of the user 190's right and left eyes. In one aspect, when the user 190 is looking closer, the gaze sensor 140 detects the lines of sight R1 and L1. In another aspect, when the user 190 is looking far away, the gaze sensor 140 detects the lines of sight R2 and L2. In this case, the angle formed by the lines of sight R2 and L2 with respect to the roll axis w is smaller than the angle formed by the lines of sight R1 and L1 with respect to the roll axis w. The gaze sensor 140 transmits the detection result to the computer 200.

When the computer 200 receives the detection values of the lines of sight R1 and L1 from the gaze sensor 140 as the detection result of the line of sight, the computer 200 identifies the gaze point N1 which is the intersection of the lines of sight R1 and L1 based on the detected values. On the other hand, when the computer 200 receives the detected values of the lines of sight R2 and L2 from the gaze sensor 140, the computer 200 identifies the intersection of the lines of sight R2 and L2 as the gaze point. The computer 200 identifies the line of sight N0 of the user 190 based on the position of the identified gazing point N1. The computer 200 detects, for example, the extending direction of the straight line passing through the midpoint of the straight line connecting the right eye R and the left eye L of the user 190 and the gazing point N1 as the line of sight N0. The line of sight N0 is the direction in which the user 190 actually directs the line of sight with both eyes. Further, the line of sight N0 corresponds to the direction in which the user 190 actually directs the line of sight with respect to the viewing area 23.

In another aspect, the HMD system 100 may include a television broadcast receiving tuner. According to such a configuration, the HMD system 100 can display a television program in the virtual space 2.

In yet another aspect, the HMD system 100 may include a communication circuit for connecting to the Internet or a telephone function for connecting to a telephone line.

[Visibility area]
The viewing area 23 will be described with reference to FIGS. 6 and 7. FIG. 6 is a diagram showing a YZ cross section of the visual recognition area 23 viewed from the X direction in the virtual space 2. FIG. 7 is a diagram showing an XZ cross section of the visual recognition area 23 viewed from the Y direction in the virtual space 2.

As shown in FIG. 6, the visible region 23 in the YZ cross section includes the region 24. The region 24 is defined by the position of the virtual camera 1, the reference line of sight 5, and the YZ cross section of the virtual space 2. The processor 10 defines a range including the polar angle α centered on the reference line of sight 5 in the virtual space as a region 24.

As shown in FIG. 7, the visible region 23 in the XZ cross section includes the region 25. The area 25 is defined by the position of the virtual camera 1, the reference line of sight 5, and the XZ cross section of the virtual space 2. The processor 10 defines a range including the azimuth angle β centered on the reference line of sight 5 in the virtual space 2 as a region 25. The polar angles α and β are determined according to the position of the virtual camera 1 and the inclination (orientation) of the virtual camera 1.

In one aspect, the HMD system 100 provides the user 190 with a field of view in virtual space by displaying the field of view image 26 on the monitor 112 based on a signal from the computer 200. The visual field image 26 corresponds to a portion of the panoramic image 22 corresponding to the visual field area 23. When the user 190 moves the HMD 110 attached to the head, the virtual camera 1 also moves in conjunction with the movement. As a result, the position of the visual recognition area 23 in the virtual space 2 changes. As a result, the field-of-view image 26 displayed on the monitor 112 is updated to an image of the panoramic image 22 superimposed on the visual-view area 23 in the direction in which the user 190 faces in the virtual space 2. The user 190 can visually recognize a desired direction in the virtual space 2.

As described above, the inclination of the virtual camera 1 corresponds to the line of sight (reference line of sight 5) of the user 190 in the virtual space 2, and the position where the virtual camera 1 is arranged corresponds to the viewpoint of the user 190 in the virtual space 2. Therefore, by changing the position or tilt of the virtual camera 1, the image displayed on the monitor 112 is updated and the field of view of the user 190 is moved.

While wearing the HMD 110, the user 190 can visually recognize only the panoramic image 22 developed in the virtual space 2 without visually recognizing the real world. Therefore, the HMD system 100 can give the user 190 a high sense of immersion in the virtual space 2.

In a certain aspect, the processor 10 may move the virtual camera 1 in the virtual space 2 in conjunction with the movement of the user 190 wearing the HMD 110 in the real space. In this case, the processor 10 identifies an image area (visual area 23) projected on the monitor 112 of the HMD 110 based on the position and inclination of the virtual camera 1 in the virtual space 2.

According to certain embodiments, the virtual camera 1 may include two virtual cameras, i.e., a virtual camera for providing an image for the right eye and a virtual camera for providing an image for the left eye. Further, appropriate parallax is set in the two virtual cameras so that the user 190 can recognize the three-dimensional virtual space 2. In the present embodiment, the virtual camera 1 includes two virtual cameras, and the roll axis (w) generated by synthesizing the roll axes of the two virtual cameras is adapted to the roll axis (w) of the HMD 110. The technical idea of the present disclosure is illustrated as being configured as such.

[HMD control device]
The control device of the HMD 110 will be described with reference to FIG. In certain embodiments, the control device is implemented by a computer 200 having a well-known configuration. FIG. 8 is a block diagram showing a computer 200 according to an embodiment as a module configuration.

As shown in FIG. 8, the computer 200 includes a display control module 220, a virtual space control module 230, a memory module 240, and a communication control module 250. The display control module 220 includes a virtual camera control module 221, a view area determination module 222, a view image generation module 223, a tilt identification module 224, a facial organ detection module 225, and a motion detection module 226 as submodules. Includes viewpoint identification module 227. The virtual space control module 230 includes a virtual space definition module 231, a virtual object generation module 232, an operation object control module 233, and an avatar control module 234 as submodules.

In certain embodiments, the display control module 220 and the virtual space control module 230 are implemented by the processor 10. In another embodiment, a plurality of processors 10 may operate as the display control module 220 and the virtual space control module 230. The memory module 240 is realized by the memory 11 or the storage 12. The communication control module 250 is realized by the communication interface 14.

In one aspect, the display control module 220 controls the image display on the monitor 112 of the HMD 110.

The virtual camera control module 221 arranges the virtual camera 1 in the virtual space 2. Further, the virtual camera control module 221 controls the position of the virtual camera 1 in the virtual space 2 and the inclination (orientation) of the virtual camera 1. The field-of-view area determination module 222 defines the field-of-view area 23 according to the inclination of the head of the user 190 wearing the HMD 110 and the position of the virtual camera 1. The field-of-view image generation module 223 generates a field-of-view image 26 to be displayed on the monitor 112 based on the determined field-of-view area 23.

The tilt identification module 224 identifies the tilt of the HMD 110 based on the output of the HMD sensor 120. In another aspect, the tilt identification module 224 identifies the tilt of the HMD 110 based on the output of the sensor 114, which functions as a motion sensor. The facial organ detection module 225 detects organs (for example, mouth, eyes, eyebrows) constituting the face of the user 190 from the images of the face of the user 190 generated by the first camera 115 and the second camera 117. The motion detection module 226 detects the motion (shape) of each organ detected by the facial organ detection module 225. 10 to 12, the control contents of the facial organ detection module 225 and the motion detection module 226 will be described later.

The viewpoint identification module 227 detects the line of sight of the user 190 in the virtual space 2 based on the signal from the gaze sensor 140. Next, the viewpoint specifying module 227 detects the viewpoint position (coordinate value in the XYZ coordinate system) where the line of sight of the detected user 190 and the celestial sphere in the virtual space 2 intersect. More specifically, the viewpoint specifying module 227 detects the viewpoint position based on the line of sight of the user 190 defined by the uvw coordinate system and the position and inclination of the virtual camera 1. The viewpoint identification module 227 transmits the detected viewpoint position to the server 150. In another aspect, the viewpoint identification module 227 may be configured to transmit line-of-sight information representing the line of sight of the user 190 to the server 150. In such a case, the viewpoint position can be calculated based on the line-of-sight information received by the server 150.

The virtual space control module 230 controls the virtual space 2 provided to the user 190. The virtual space definition module 231 defines the virtual space 2 in the HMD system 100 by generating virtual space data representing the virtual space 2.

The virtual object generation module 232 generates an object to be arranged in the virtual space 2. Objects can include, for example, landscapes, animals, etc., including forests, mountains, etc., which are arranged as the story of the game progresses.

The operation object control module 233 arranges an operation object for receiving the operation of the user 190 in the virtual space 2 in the virtual space 2. By operating the operation object, the user 190 operates, for example, an object arranged in the virtual space 2. In some aspects, the operating object may include, for example, a hand object corresponding to the hand of user 190. In a certain aspect, the operation object control module 233 moves the hand object in the virtual space 2 so as to be linked to the movement of the user 190's hand in the real space based on the output of the motion sensor 130. In some aspects, the manipulation object may correspond to the hand portion of the avatar object described below.

The avatar control module 234 generates data for arranging the avatar object of the user 190 of the other computer 200 connected via the network in the virtual space 2. In a certain aspect, the avatar control module 234 generates data for arranging the avatar object of the user 190 in the virtual space 2. In one aspect, the avatar control module 234 creates an avatar object that mimics the user 190 based on an image that includes the user 190. In another aspect, the avatar control module 234 sets the avatar object in the virtual space 2 to be selected by the user 190 from among a plurality of types of avatar objects (for example, an object imitating an animal or a deformed human object). Generate data for placement.

The avatar control module 234 reflects the movement of the HMD 110 detected by the HMD sensor 120 on the avatar object. For example, the avatar control module 234 detects that the HMD 110 is tilted and generates data for tilting and arranging the avatar object. Further, in a certain aspect, the avatar control module 234 reflects the movement of the controller 160 on the avatar object. In this case, the controller 160 includes a motion sensor, an acceleration sensor, a plurality of light emitting elements (for example, infrared LEDs), and the like for detecting the movement of the controller 160. Further, the avatar control module 234 reflects the movement of the facial organ detected by the motion detection module 226 on the face of the avatar object arranged in the virtual space 2.

When each of the objects arranged in the virtual space 2 collides with another object, the virtual space control module 230 detects the collision. The virtual space control module 230 can detect, for example, the timing at which a certain object and another object touch each other, and when the detection is made, a predetermined process is performed. The virtual space control module 230 can detect the timing when the object and the object are separated from the touching state, and when the detection is made, the virtual space control module 230 performs a predetermined process. The virtual space control module 230 can detect that the object and the object are in contact with each other. Specifically, the operation object control module 233 detects that the operation object touches the other object when the operation object touches the other object, and performs a predetermined process. ..

The memory module 240 holds data used by the computer 200 to provide the virtual space 2 to the user 190. In a certain aspect, the memory module 240 holds the spatial information 241 and the object information 242, the user information 243, and the face template 244.

The spatial information 241 holds one or more templates defined to provide the virtual space 2.

The object information 242 includes a plurality of panoramic images 22 constituting the virtual space 2 and data for arranging the objects in the virtual space 2. The panoramic image 22 may include a still image and a moving image. Further, the panoramic image 22 may include an image in the unreal space and an image in the real space (for example, computer graphics).

The user information 243 holds a user ID that identifies the user 190. The user ID may be, for example, an IP (Internet Protocol) address or a MAC (Media Access Control) address set in the computer 200 used by the user. In other aspects, the user ID may be set by the user. The user information 243 includes a program for operating the computer 200 as a control device of the HMD system 100 and the like.

In the face template 244, the face organ detection module 225 holds a template stored in advance for detecting the face organ of the user 190. In certain embodiments, the face template 244 holds a mouth template 245, an eye template 246, and an eyebrow template 247. Each template can be an image corresponding to the organs that make up the face. For example, the mouth template 245 can be an image of the mouth. In addition, each template may include a plurality of images.

The data and programs stored in the memory module 240 are input by the user 190 of the HMD 110. Alternatively, the processor 10 downloads a program or data from a computer (for example, a server 150) operated by a business operator that provides the content, and stores the downloaded program or data in the memory module 240.

The communication control module 250 can communicate with the server 150 and other information communication devices via the network 19.

In certain aspects, the display control module 220 and the virtual space control module 230 can be implemented using, for example, Unity® provided by Unity Technologies. In another aspect, the display control module 220 and the virtual space control module 230 can also be realized as a combination of circuit elements that realize each process.

The processing in the computer 200 is realized by the hardware and the software executed by the processor 10. Such software may be pre-stored in a hard disk or other memory module 240. The software may also be stored on a CD-ROM or other computer-readable non-volatile data recording medium and distributed as a program product. Alternatively, the software may be provided as a downloadable program product by an information provider connected to the Internet or other networks. Such software is read from a data recording medium by an optical disk drive or other data reader, or downloaded from a server 150 or other computer via a communication control module 250, and then temporarily stored in a storage module. .. The software is read from the storage module by the processor 10 and stored in RAM in the form of an executable program. Processor 10 executes the program.

[Avatar Object]
An avatar object according to this embodiment will be described with reference to FIGS. 9A and 9B. Hereinafter, it is a figure explaining the avatar object of each user of the HMD set 105A, 105B. Hereinafter, the user of the HMD set 105A is referred to as a user 190A, the user of the HMD set 105B is referred to as a user 190B, the user of the HMD set 105C is referred to as a user 190C, and the user of the HMD set 105D is referred to as a user 190D. Further, A is attached to the reference code of each component related to the HMD set 105A, B is attached to the reference code of each component related to the HMD set 105B, and C is attached to the reference code of each component related to the HMD set 105C. A D is added to the reference code of each component with respect to the HMD set 105D. For example, the HMD 110A is included in the HMD set 105A.

FIG. 9A is a schematic diagram showing a situation in which each of the plurality of HMDs provides a virtual space to each of the plurality of users in the network. With reference to FIG. 9A, the computers 200A-200D provide the virtual spaces 2A-2D to the users 190A-190D via the HMD110A-110D, respectively. In the example shown in FIG. 9A, the virtual space 2A and the virtual space 2B are composed of the same data. In other words, the computer 200A and the computer 200B share the same virtual space. In the virtual space 2A and the virtual space 2B, the avatar object 900A of the user 190A and the avatar object 900B of the user 190B exist. The avatar object 900A in the virtual space 2A and the avatar object 900B in the virtual space 2B are each equipped with an HMD, but this is for the sake of easy explanation, and in reality, these objects have an HMD. Not installed.

In one aspect, the virtual camera control module 221A may place the virtual camera 1A, which captures the view image 26A of the user 190A, at the eye position of the avatar object 900A.

FIG. 9B is a diagram showing a field of view image 910 of the user 190A in FIG. 9A. The field of view image 910 is an image displayed on the monitor 112A of the HMD 110A. The field view image 910 is an image generated by the virtual camera 1A. Further, the avatar object 900B of the user 190B is displayed in the field of view image 910. Although not shown in particular, the avatar object 900A of the user 190A is also displayed in the field of view image of the user 190B.

In the state of FIG. 9B, the user 190A can communicate with the user 190B through a virtual space by dialogue. More specifically, the voice of the user 190A acquired by the microphone 119A is transmitted to the HMD 110B of the user 190B via the server 150, and is output from the speaker 118B provided in the HMD 110B. Further, the voice of the user 190B is transmitted to the HMD110A of the user 190A via the server 150, and is output from the speaker 118A provided in the HMD110A.

Further, the operation of the user 190B (the operation of the HMD 110B and the operation of the controller 160B) is reflected in the avatar object 900B arranged in the virtual space 2A by the avatar control module 234A. As a result, the user 190A can recognize the operation of the user 190B through the avatar object 900B.

[Face tracking]
Hereinafter, a specific example for detecting a user's facial expression (face movement) will be described with reference to FIGS. 10 to 12. 10 to 12 show, as an example, a specific example of detecting the movement of the mouth of the user 190. The detection method described with reference to FIGS. 10 to 12 is not limited to the movement of the mouth of the user 190, but is the detection of the movement of other organs (for example, eyes, eyebrows, nose, cheeks) constituting the face of the user 190. Can also be applied to.

FIG. 10 is a diagram illustrating control for detecting a mouth from a user's face image 1000. The face image 1000 generated by the first camera 115 includes the nose and mouth of the user 190.

The face organ detection module 225 identifies the mouth region 1010 from the face image 1000 by pattern matching using the mouth template 245 stored in the face template 244. In a certain aspect, the face organ detection module 225 sets a comparison area on a rectangle in the face image 1000, and while changing the size, position, and angle of the comparison area, the image of the comparison area and the mouth template 245 Calculate the similarity with the image. The facial organ detection module 225 may identify a comparison region for which a similarity greater than a predetermined threshold is calculated as the mouth region 1010.

The facial organ detection module 225 further determines that the comparison region is based on the relative relationship between the position of the comparison region where the calculated similarity is greater than the threshold value and the position of other facial organs (eg, eyes, nose). It is possible to judge whether or not it corresponds to the mouth area.

The motion detection module 226 detects a more detailed mouth shape from the mouth region 1010 detected by the facial organ detection module 225.

FIG. 11 is a diagram (No. 1) for explaining the process of detecting the shape of the mouth by the motion detection module 226. With reference to FIG. 11, the motion detection module 226 sets a contour detection line 1100 for detecting the shape of the mouth (the contour of the lips) included in the mouth region 1010. A plurality of contour detection lines 1100 are set at predetermined intervals in a direction orthogonal to the height direction of the face.

The motion detection module 226 can detect a change in the brightness value of the mouth region 1010 along each of the plurality of contour detection lines 1100, and can specify a position where the change in the brightness value is abrupt as a contour point. More specifically, the motion detection module 226 can specify as a contour point a pixel whose luminance difference (that is, change in luminance value) from adjacent pixels is equal to or greater than a predetermined threshold value. The brightness value of the pixel is obtained, for example, by integrating the RBG value of the pixel with a predetermined weighting.

The motion detection module 226 identifies two types of contour points from the image corresponding to the mouth region 1010. The motion detection module 226 identifies contour points 1110 corresponding to the outer contour of the mouth (lips) and contour points 1120 corresponding to the inner contour of the mouth (lips). In a certain aspect, when three or more contour points are detected on one contour detection line 1100, the motion detection module 226 can specify the contour points at both ends as the outer contour points 1110. In this case, the motion detection module 226 can specify contour points other than the outer contour points 1110 as inner contour points 1120. Further, when two or less contour points are detected on one contour detection line 1100, the motion detection module 226 can specify the detected contour points as the outer contour points 1110.

FIG. 12 is a diagram (No. 2) for explaining the process of detecting the shape of the mouth by the motion detection module 226. In FIG. 12, the outer contour points 1110 are shown as white circles, and the inner contour points 1120 are shown as hatched circles.

The motion detection module 226 identifies the mouth shape 1200 by complementing between the inner contour points 1120. In certain aspects, the motion detection module 226 may identify the mouth shape 1200 using a non-linear interpolation method such as spline interpolation. In another aspect, the motion detection module 226 may specify the mouth shape 1200 by complementing the outer contour points 1110. In yet another aspect, the motion detection module 226 excludes contour points that deviate significantly from the assumed mouth shape (a predetermined shape that can be formed by the upper and lower lips of a person), and the remaining contour points make the mouth. The shape 1200 may be specified. In this way, the motion detection module 226 can identify the motion (shape) of the user's mouth. The method for detecting the mouth shape 1200 is not limited to the above, and the motion detection module 226 may detect the mouth shape 1200 by another method. In addition, the motion detection module 226 can detect the motion of the user's eyes and eyebrows in the same manner. The motion detection module 226 may be configured to be able to detect the shape of an organ such as a cheek or a nose.

FIG. 13 shows an example of the structure of face tracking data. The motion detection module 226 generates face tracking data representing the facial expression of the user 190. The face tracking data represents the position coordinates of the feature points constituting the shape of each organ to be detected in the uvw visual field coordinate system. For example, the points m1, m2, ... Shown in FIG. 13 correspond to the outer contour points 1110 constituting the mouth shape 1200. In a certain aspect, the face tracking data is a coordinate value in the uvw field of view coordinate system with the position of the first camera 115 as a reference (origin). In another aspect, the face tracking data is a coordinate value in a coordinate system with a predetermined feature point as a reference (origin) for each organ. As an example, the points m1, m2, ... Are coordinate values in a coordinate system with the origin of any one of the outer contour points 1110 corresponding to the corner of the mouth.

The computer 200 transmits the generated face tracking data to the server 150. The server 150 transfers this data to another computer 200 that communicates with the computer 200. The other computer 200 reflects the received face tracking data in the avatar object corresponding to the user of the receiving computer 200.

In the example shown in FIG. 9B, the computer 200A receives face tracking data representing the facial expression of the user 190B from the computer 200B. The computer 200A reflects the received data on the avatar object 900B. As an example, among the vertices of the polygons constituting the avatar object 900B, the vertices corresponding to the face tracking data are set. The computer 200A moves the positions of the corresponding vertices based on the face tracking data. As a result, the facial expression of the user 190B is reflected in the avatar object 900B. As a result, the user 190A can recognize the facial expression of the user 190B via the avatar object 900B.

[Control structure of server 150]
FIG. 14 is a diagram illustrating a hardware configuration and a module configuration of the server 150. In certain embodiments, the server 150 comprises a communication interface 1410, a processor 1420, and storage 1440 as main components.

The communication interface 1410 functions as a communication module for wireless communication that performs modulation / demodulation processing for transmitting / receiving signals to / from an external communication device such as a computer 200. The communication interface 1410 is realized by a tuner, a high frequency circuit, or the like.

The processor 1420 controls the operation of the server 150. By executing various control programs stored in the storage 1440, the processor 1420 executes a transmission / reception unit 1421, a server processing unit 1422, a matching unit 1423, a viewpoint acquisition unit 1424, an emotion judgment unit 1425, a map generation unit 1426, and a cutting unit. It functions as 1427, the target identification unit 1428, and the filter unit 1429.

The transmission / reception unit 1421 transmits / receives various information to / from each computer 200. For example, the transmission / reception unit 1421 receives a request for arranging an object in the virtual space 2, a request for deleting the object from the virtual space 2, a request for moving the object, a user's voice, information for defining the virtual space 2, and the like. Send to computer 200.

The server processing unit 1422 updates the user information 1444 described later based on the information received from the computer 200.

The matching unit 1423 performs a series of processes for associating a plurality of users. For example, when a plurality of users perform an input operation for sharing the same virtual space 2, the matching unit 1423 performs a process of associating each user ID of the plurality of users belonging to the virtual space 2.

The viewpoint acquisition unit 1424 acquires the viewpoint position of the user 190 in the virtual space 2 (XYZ coordinate system) based on the line-of-sight information received from the computer 200. When the viewpoint specifying module 227 of the computer 200 is configured to specify the viewpoint position and transmit the specified information to the server 150, the processor 1420 does not have to function as the viewpoint acquisition unit 1424.

The emotion determination unit 1425 determines the emotion of the user 190 based on the face tracking data received from the computer 200. The map generation unit 1426 generates a map based on the viewpoint position of the user 190.

The cutout unit 1427 cuts out a peripheral image of the viewpoint position of the user 190 in the panoramic image 22 constituting the virtual space 2. The target identification unit 1428 identifies the content (the target to which the user gazes) included in the peripheral image cut out by the cutout unit 1427. The filter unit 1429 determines whether or not to save the viewpoint position of the user 190 in the storage 1440.

The storage 1440 includes virtual space designation information 1441, object designation information 1442, panoramic image DB1443, user information 1444, advertisement DB1445, first table TL1, second table TL2, reference data DB1446, and facial expression identification. It holds the device DB1447 and the object classifier DB1448.

The virtual space designation information 1441 is information used by the virtual space definition module 231 of the computer 200 to define the virtual space 2. For example, the virtual space designation information 1441 includes information that specifies the size or shape of the virtual space 2.

The object designation information 1442 specifies an object to be arranged (generated) in the virtual space 2 by the virtual object generation module 232 of the computer 200. The panorama image DB 1443 stores a plurality of panoramic images 22 to be distributed to the computer 200.

The user information 1444 includes a user ID received from each computer 200. That is, the user information 1444 includes information that identifies each of the plurality of users.

The user information 1444 further includes facial expression information 1451, position information 1452, tilt information 1453, and viewpoint position information 1454. The facial expression information 1451 is face tracking data for each user. As an example, the facial expression information 1451 is data in which the user ID and the face tracking data are associated with each other.

The position information 1452 is data in which the user ID and the user's viewpoint (position of the virtual camera 1) are associated with each other. The tilt information 1453 is data in which the user ID and the tilt of the virtual camera 1 (HMD110) are associated with each other. The viewpoint position information 1454 is data in which the user ID, the panoramic image ID, and the viewpoint position are associated with each other. Details of the viewpoint position information 1454 will be described later. The user information 1444 is updated at any time by the server processing unit 1422 based on the information input from each computer 200.

The advertisement DB 1445 holds a plurality of advertisements to be delivered to the computer 200. The first table TL1 holds each of the plurality of advertisements and the target specified by the target identification unit 1428 in association with each other. The second table TL2 holds the target specified by the target identification unit 1428 and the type of the panoramic image 22 in association with each other. Details of the first table TL1 and the second table TL2 will be described later.

The reference data DB 1446 holds the reference data used for comparison with the face tracking data in association with the user ID. The facial expression classifier DB1447 includes a facial expression classifier for each type of facial expression. As an example, the facial expression classifier DB1447 includes four types of facial expression classifiers 1461 to 1464. The facial expression classifier 1461 functions as a program for identifying a smiling facial expression. The facial expression classifier 1462 functions as a program for identifying an angry facial expression. The facial expression classifier 1463 functions as a program for identifying a surprised facial expression. The facial expression classifier 1464 functions as a program for identifying a sad facial expression. For example, the facial expression classifier 1461 learns the weighting coefficient using the face tracking data of a plurality of laughing persons as training data.

The object classifier DB1448 includes an object classifier for each type of object. In the example shown in FIG. 14, the object classifier DB1448 includes object classifiers 1471, 1472, 1473, .... For example, the object classifier 1471 functions as a program for identifying cats.

[User information update process]
FIG. 15 is a flowchart showing a process in which the server 150 communicates with the computers 200A and 200B to update the user information 1444. The process shown in FIG. 15 is realized by the processor 10 of the computer 200 executing the control program stored in the memory 11 or the storage 12, and the processor 1420 of the server 150 executing the control program stored in the storage 1440. obtain.

In step S1502, the processor 1420 of the server 150 defines the virtual space based on the information input from the computers 200A and 200B (for example, the information specifying one of the plurality of panoramic images 22). The processor 1420, as the transmission / reception unit 1421, transmits the virtual space designation information 1441 corresponding to the defined virtual space to the computers 200A and 200B. At this time, each computer 200 transmits the user ID to the server 150 together with the virtual space designation information 1441. Processor 1420 may further associate users 190A and 190B with each other as matching unit 1423, assuming that users 190A and 190B share the same virtual space.

In step S1504, the processor 10A of the computer 200A defines the virtual space 2A as the virtual space definition module 231A. Specifically, the processor 10A configures the virtual space 2A by using the panoramic image 22A based on the received virtual space designation information 1441. In step S1506, processor 10B of computer 200B defines virtual space 2B in the same way as processor 10A.

In step S1508, the processor 10A photographs the face of the user 190A by the first camera 115A and the second camera 117A. At this time, the processor 10A displays a message on the monitor 112A prompting the photographer to take a picture without expression. The processor 10A generates face tracking data based on the acquired image. The face tracking generated at this time functions as reference data. The processor 10A transmits the generated reference data to the server 150. In step S1510, the processor 10B also generates reference data and transmits it to the server 150. When the processors 10A and 10B transmit some data to the server 150, they also transmit the user ID.

In step S1512, the server 150 updates the reference data DB 1446 based on the reference data received from each computer 200.

In step S1514, the processor 10A arranges the user 190A's own avatar object 900A (denoted as “own avatar object” in FIG. 15) in the virtual space 2A as the avatar control module 234A. The processor 10A further places the virtual camera 1A at the position of the avatar object 900A (eg, the eye position). The processor 10A transmits the position information of the avatar object 900A (that is, the viewpoint information in the virtual space 2A of the user 190A) and the modeling data to the server 150. When the avatar object 900A is an avatar selected from a predetermined type, the processor 10A can transmit information specifying the type of the avatar to the server 150.

In step S1516, the processor 1420 updates the position information 1452 corresponding to the user 190A (user ID) based on the received position information of the avatar object 900A. The processor 1420 further transmits the information received from the computer 200A to the computer 200B communicating with the computer 200A.

In step S1518, the processor 10B arranges the avatar object 900A in the virtual space 2B as the avatar control module 234B based on the received information.

In steps S152 to S1524, avatar objects 900B (denoted as "other avatar objects" in FIG. 15) are generated in the virtual spaces 2A and 2B, and the position information 1452 corresponding to the user 190B is updated, as in steps S1514 to S1518. NS.

In step S1526, the processor 10A photographs the face of the user 190A by the first camera 115A and the second camera 117A to generate a face image including depth information. The processor 10A, as the face organ detection module 225A and the motion detection module 226A, generates face tracking data based on the face image and transmits it to the server 150.

In step S1528, the processor 10A identifies the viewpoint position of the user 190A in the virtual space 2A as the viewpoint specifying module 227A, and transmits the viewpoint position to the server 150.

In step S1530, the processor 10A updates the position and tilt of the virtual camera 1A based on the output of the HMD sensor 120 and / or the output of the controller 160. The processor 10A transmits the updated information indicating the position and tilt of the virtual camera 1A to the server 150.

In steps S1532 to S1536, the processor 10B transmits face tracking data, a viewpoint position in the virtual space 2B of the user 190B, and information indicating the position and inclination of the virtual camera 1B to the server 150, as in the processes of steps S1526 to S1530. do.

In step S1538, the processor 1420 updates the user information 1444 based on various information received from the computers 200A and 200B. In a certain aspect, the processor 1420 may store only the viewpoint positions that satisfy the predetermined conditions regarding the operation or operation of the user 190 in the viewpoint position information 1454 among the viewpoint positions received from each computer 200. Details of this process will be described later.

The processor 1420 further transmits the information received from the computer 200A to the computer 200B, and transmits the information received from the computer 200B to the computer 200A.

In step S1540, the processor 10A reflects the information received from the server 150 on the avatar object 900B arranged in the virtual space 2A. In step S1542, the processor 10A outputs the field of view image captured by the virtual camera 1A to the monitor 112A. As a result, the user 190A can visually recognize the avatar object 900B that reflects the movement and facial expression of the user 190B. After that, the processor 10A executes the process of step S1526 again.

In steps S1544 to S1546, the processor 10B executes the same processing as the processing of steps S1540 to S1542. After that, the processor 10B executes the process of step S1532 again.

In certain embodiments, the repetitive processes of steps S1526-S1546 may be performed at 1/60 second or 1/30 second intervals.

In another aspect, the repeatedly executed process may include a process of transmitting the voice of the user 190 to the other computer 200, and another process of promoting communication between users in the virtual space 2.

[Process to save the viewpoint position in memory]
The user 190 can evaluate the panoramic image 22 that constitutes the virtual space 2. If the user 190 evaluates the panoramic image 22 itself, the distributor of the panoramic image 22 may not be able to grasp what the user 190 is interested in. The reason is that since the panoramic image 22 is developed in all directions of 360 degrees, it is not possible to grasp what the user 190 saw and evaluated. Hereinafter, processes that can solve such problems will be described.

(Save process based on user operation)
First, with reference to FIGS. 16 and 17, a process of saving the viewpoint position of the user 190 in the memory will be described based on the user operation. FIG. 16 represents a field of view image 1600 viewed by the user 190A. FIG. 17 represents a virtual space 2A corresponding to the state of FIG.

In a certain aspect, a panoramic image 22 representing a cityscape in a real space is developed in the virtual space 2A. The visual field image 1600 is an image of a portion of the panoramic image 22 corresponding to the visual field area 23. The field of view image 1600 includes a cat 1610 that is part of the panoramic image 22. The view image 1600 further includes an avatar object 900B, a viewpoint object 1620, an operation object 1630, a UI object 1640, and an evaluation object 1650.

The viewpoint object 1620 represents the viewpoint position in the virtual space 2A of the user 190A. In one aspect, this object represents a viewpoint position in the panoramic image 22. In the example of FIG. 16, user 190A is gazing at cat 1610.

With reference to FIG. 17, the processor 10A identifies the line-of-sight 1710 of the user 190A as the viewpoint identification module 227A. Next, the viewpoint specifying module 227A identifies the coordinate value 1720 at which the line of sight 1710 and the celestial sphere in the virtual space 2A intersect. Processor 10A places the viewpoint object 1620 at the specified coordinate value 1720.

In one aspect, the processor 10A transmits the coordinate values 1720 in the XYZ coordinate system identified by the viewpoint identification module 227A to the server 150.

With reference to FIG. 16 again, the operation object 1630 is a hand object and moves in conjunction with the movement of the hand of the user 190A. More specifically, the processor 10A, as the operation object control module 233A, generates data for moving the operation object 1630 based on the output of the motion sensor 130A.

The UI object 1640 functions as a user interface for accepting the evaluation of the user 190A for the content included in the panoramic image 22. As an example, the UI object 1640 contains a positive wording (“Good!” In the example of FIG. 16). In a certain aspect, the processor 10A moves the UI object 1640 in conjunction with the virtual camera 1. As a result, the user 190A can always see the UI object 1640.

The user 190A operates the UI object 1640 when there is a favorite content in the field of view image 1600. As an example, the user 190A brings the operation object 1630 into contact with the UI object 1640 in a state where the line of sight 1710 is focused on the favorite content. The processor 10A associates the coordinate value 1720 at the timing when they are in contact with the information indicating that they are in contact (first operation information) and transmits the information to the server 150. The server 150 stores the viewpoint position associated with the first operation information in the viewpoint position information 1454 of the storage 1440.

In another aspect, the processor 10A sets the coordinate value 1720 at the timing when the predetermined button (UI) of the controller 160A for receiving the interest of the user 190 is pressed, and the information indicating that the button is pressed ( It may be transmitted to the server 150 together with the second operation information).

The first and second operation information is a signal representing the operation of the user 190A. Further, the operation of the user 190A represented by the first and second operation information indicates the interest of the user 190A. Hereinafter, the first operation information and the second operation information are also collectively referred to as "operation information".

FIG. 18 shows an example of the data structure of the viewpoint position information 1454. The viewpoint position information 1454 holds the user ID, the panoramic image ID, the viewpoint position, and the timing in association with each other. The panoramic image ID identifies each of the plurality of panoramic images 22. This timing represents the timing (timing at which the viewpoint position is acquired) in which the operation information during reproduction of the moving image is input when the panoramic image 22 is a moving image.

The viewpoint position information 1454 shown in FIG. 18 indicates that the user 190A is gazing at the viewpoint position (X1, Y1, Z1) at the time when 5 minutes and 3 seconds have passed since the panoramic image 22A started to be reproduced.

According to the above, the server 150 can store the user's viewpoint position associated with the operation information in the storage 1440. As a result, the distributor of the panoramic image 22 can grasp which user is interested in which content included in the panoramic image 22 by referring to the viewpoint position stored in the storage 1440.

With reference to FIG. 16 again, the evaluation object 1650 represents the viewpoint position at the timing when the user 190A or another user has input the operation information in the past. By visually recognizing the evaluation object 1650, the user 190A can grasp which content of the panoramic image 22 the other user is interested in.

The evaluation object 1650 may prevent the user 190A from visually recognizing the panoramic image 22. Therefore, in certain embodiments, the evaluation object 1650 is set to transparent (eg, 50% transmittance).

(Save process based on user behavior)
In the above example, the server 150 is configured to store the viewpoint position in the storage based on the operation of the user 190. In such a case, the distributor of the panoramic image may not be able to fully grasp the interest of the user 190. For example, when the panoramic image 22 is a moving image, the user 190 may not be able to input the operation information into the computer 200 at the timing when the content of interest is visually recognized. In addition, the user 190 may find it troublesome to input the operation information. Hereinafter, processes that can solve such problems will be described.

FIG. 19A shows facial feature points acquired when the user 190A is expressionless. FIG. 19B shows facial feature points acquired when user 190A is surprised. The feature points P shown in FIGS. 19A and 19B represent the facial feature points of the user 190A acquired by the motion detection module 226A.

As described in step S1508 of FIG. 15, the computer 200A generates face tracking data (reference data) of the expressionless user 190A. The feature point P shown in FIG. 19A corresponds to this reference data. On the other hand, the feature point P shown in FIG. 19B corresponds to the face tracking data generated in step S1526. In the example shown in FIG. 19B, since the user 190A is surprised, the feature points P of the eyes spread in the height direction of the face, and the feature points P of the eyebrows move upward. That is, the amount of change in the face tracking data with respect to the reference data represents the degree of interest of the user 190A in the content.

Therefore, when the fluctuation amount of the face tracking data with respect to the reference data exceeds a predetermined fluctuation amount, the processor 1420 of the server 150 stores the viewpoint position corresponding to the face tracking data in the viewpoint position information 1454. As an example, the viewpoint position corresponding to the face tracking data may be the viewpoint position input at the timing closest to the input timing of the face tracking data.

In a certain aspect, the processor 10A calculates the amount of fluctuation of the face tracking data with respect to the reference data for each feature point, and makes the above determination based on the sum total. In another aspect, the processor 10A calculates the amount of fluctuation only for predetermined feature points (for example, feature points corresponding to the corners of the mouth) having a large degree of change due to emotions, and makes the above determination based on the sum.

According to the above, the server 150 can increase the possibility that the user 190A can acquire the viewpoint position when he / she is interested in the content. Further, since the user 190 does not need to perform any operation, he / she can concentrate on viewing the panoramic image 22.

(Control structure)
FIG. 20 is a flowchart showing a process of storing the viewpoint position according to a certain embodiment in the storage 1440. The process shown in FIG. 20 is performed by processor 1420 of server 150.

In step S2010, the processor 1420 defines the virtual space 2 based on the virtual space designation information 1441. The processor 1420 further configures the virtual space 2 by using the panoramic image 22 designated by the computer 200 among the plurality of panoramic images 22 stored in the panoramic image DB 1443.

In step S2020, the processor 1420 receives input from the computer 200 of face tracking data, the position and tilt of the virtual camera 1, the viewpoint position, and signals representing the operation of the user 190. This face tracking data can be said to be a signal representing the operation of the user 190. The signal representing the operation includes, for example, the output of the controller 160. In other aspects, the signal representing the operation includes information indicating that the operation object has come into contact with another object.

In step S2030, the processor 1420 determines whether or not the viewpoint position and the operation information are associated with each other. When the processor 1420 determines that the operation information is associated with the viewpoint position (YES in step S2030), the processor 1420 saves the viewpoint position in the storage 1440 (step S2060). If not (NO in step S2030), processor 1420 proceeds to step S2040.

In step S2040, the processor 1420 calculates the amount of variation of the face tracking data with respect to the reference data. More specifically, the processor 1420 refers to the reference data DB 1446 to specify the reference data corresponding to the user ID of the source of the face tracking data. The processor 1420 compares the specified reference data with the received face tracking data to calculate the amount of fluctuation.

In step S2050, the processor 1420 determines whether or not the calculated fluctuation amount exceeds a predetermined value. When the processor 1420 determines that the calculated fluctuation amount exceeds a predetermined value (YES in step S2050), the processor 1420 saves the viewpoint position in the storage 1440 (step S2060). If not (NO in step S2050), processor 1420 re-executes the process of step S2020.

According to the above, the server 150 according to an embodiment can acquire the viewpoint position when the operation or operation of the user 190 indicates the interest of the user 190.

In the above example, the server 150 is configured to store the position information of the panoramic image 22 that the user 190 is interested in in the storage 1440, but in another aspect, the object that the user 190 is interested in. The information representing the above may be stored in the storage 1440. As an example, when the user 190 is looking at a predetermined object and the condition for storing the viewpoint position described above is satisfied, the server 150 provides information representing the object (for example, each object). ID) is stored in the storage 1440.

(Saving the viewpoint position by voice)
In certain embodiments, the server 150 receives input of a voice signal corresponding to the utterance of the user 190 from the computer 200 in step S1820. The server 150 may store the viewpoint position in the storage 1440 when the audio signal satisfies a predetermined condition. In one aspect, the server 150 stores the viewpoint position in storage 1440 when the input audio signal exceeds a predetermined level.

In another aspect, the server 150 estimates the emotion of the user 190 from the input audio signal and stores the viewpoint position in the storage 1440 based on the estimated emotion. For example, the server 150 extracts a character string from the audio signal and estimates the emotion from the character string. Such processing can be realized by, for example, the "emotion analysis API" provided by Metadata. In another aspect, the server 150 estimates emotions from the waveform of the audio signal. Such processing can be realized by, for example, the "ST Emotion SDK" provided by AGI.

By the above processing, the server 150 estimates the emotion type of the user 190 from among a plurality of emotion types (for example, "joy", "anger", "sorrow", and "comfort") based on the audio signal. The server 150 stores the viewpoint position in the storage 1440 when the estimated emotion type indicates the user's interest (for example, when the emotion type is "joy" or "comfort"). At this time, the server 150 may store the estimated emotion type in the storage 1440 in association with the viewpoint position.

(Guessing emotions)
In the example of FIG. 20, the processor 1420 is configured to store the viewpoint position in the storage 1440 when the amount of fluctuation of the face tracking data with respect to the reference data is large. In such a case, the distributor of the panoramic image 22 can grasp the content that the user 190 is interested in, but does not know what kind of feeling the user 190 has about the content. Therefore, the server 150 according to a certain embodiment guesses what kind of feeling the user 190 has about the content.

The processor 1420, as the emotion determination unit 1425, calculates the feature amount from the face tracking data in which the amount of fluctuation with respect to the reference data is equal to or greater than the threshold value. The emotion determination unit 1425 uses the facial expression classifiers 1461 to 1464 to specify the type of facial expression corresponding to the calculated feature amount.

As an example, the emotional judgment unit 1425 uses facial expression classifiers 1461 to 1464 that follow a plurality of SVMs (Support Vector Machines), and uses facial expression types (laughter, anger, surprise) from the features derived by the CNN (Convolutional Neural Network). , Sadness). The method for specifying the type of facial expression is not limited to this method, and known machine learning can be applied.

In another aspect, the emotional determination unit 1425 can identify the type of facial expression based on the arrangement pattern of the face tracking data. In still another aspect, the emotion determination unit 1425 can accept the input of the face image of the user 190 (the image taken by the first camera 115 and the second camera 117) and specify the type of facial expression based on the face image. ..

FIG. 21 is a flowchart showing a process of associating and saving the viewpoint position and the type of emotion. Of the processes shown in FIG. 21, the same processes as those described above are designated by the same reference numerals. Therefore, the description of the process will not be repeated.

In step S1910, the processor 1420, as the emotion determination unit 1425, identifies the type of facial expression based on the face tracking data in which the amount of fluctuation with respect to the reference data exceeds a predetermined value.

In step S1920, the processor 1420 associates the viewpoint position with the specified facial expression type and stores it in the storage 1440 (viewpoint position information 1454).

According to the above, the server 150 according to a certain embodiment can store the position information (viewpoint position) of the content that the user 190 is interested in and the emotion (facial expression) of the user 190 with respect to the content in the storage 1440. As a result, the distributor of the panoramic image 22 can obtain the user 190's evaluation of the panoramic image 22 in more detail.

(Visualization of viewpoint)
In the above example, the processor 1420 is configured to store the viewpoint position of interest to the user 190 in the storage 1440. In such a case, the distributor of the panoramic image 22 needs to check the correspondence between the viewpoint position (coordinate value) and the panoramic image 22. Therefore, the processor 1420 according to a certain embodiment creates a graph based on the panoramic image 22 and the viewpoint position information 1454 as the map generation unit 1426. This graph visualizes the viewpoint position (the position where the user 190 is interested) in the panoramic image 22.

FIG. 22 shows a heat map 2200 based on the viewpoint position information 1454. As an example, the processor 1420 generates a heat map 2200 by expressing a region on the panoramic image 22 where the viewpoint position is dense in red and a region where the viewpoint position is sparse in blue. In the example of FIG. 22, the region 2210 is a region where the viewpoint position is dense and is hatched in red.

By looking at the heat map 2200, the distributor of the panoramic image 22 can easily understand which content of the panoramic image 22 the user 190 is interested in.

In a certain aspect, when the processor 1420 stores the viewpoint position in the viewpoint position information 1454, the processor 1420 stores the viewpoint position in association with whether or not the user who transmits the viewpoint position communicates with another user. For example, in the example shown in FIG. 17, when the processor 1420 receives the coordinate value 1720 representing the viewpoint position from the computer 200A, the user 190A determines that the user 190A is communicating with the user 190B, and determines that fact and the viewpoint position. It is associated and saved in the viewpoint position information 1454.

A processor 1420 according to an embodiment may generate a heat map based on a viewpoint position when communicating with another user and a heat map based on a viewpoint position when not communicating with another user. In such a case, the distributor of the panoramic image 22 is interested in the content when the user is viewing the panoramic image 22 by one person and the content which is interested when the user is viewing the panoramic image 22 by a plurality of users. The difference with can be easily understood.

[Identify the content that the user is interested in]
In the above example, the processor 1420 is configured to store the viewpoint position in the storage 1440, and does not specify the content displayed at the viewpoint position. Therefore, the distributor of the panoramic image 22 must investigate the correspondence between the viewpoint position (coordinate value) and the panoramic image 22 in order to understand the content that the user is interested in. Therefore, the processor 1420 according to a certain embodiment identifies the content displayed at the viewpoint position.

With reference to FIG. 16, the viewpoint object 1620 of the user 190A is superimposed on the cat 1610. In one aspect, the computer 200A transmits the viewpoint position where the viewpoint object 1620 is placed to the server 150.

The processor 1420 of the server 150 cuts out the peripheral image 1660 around the viewpoint position received from the panoramic image 22 developed in the virtual space 2A as the cutout unit 1427. In a certain aspect, the cutting unit 1427 cuts out a predetermined rectangular area centered on the viewpoint position as a peripheral image 1660. In another aspect, the cutout unit 1427 cuts out the bounding box in which the content at the viewpoint position exists as the peripheral image 1660 by using a known object detection technique. For example, the cutout portion 1427 sets a minimum region (for example, 3 × 3 pxel) centered on the viewpoint position by using the Selective Search method, and the peripheral image 1660 (bounding) is based on the range occupied by the region similar to the region. Box) is cut out.

Next, the processor 1420 identifies the content included in the peripheral image 1660, that is, the target to which the user 190A looks at, as the target identification unit 1428. Similar to the emotion determination unit 1425, the object identification unit 1428 identifies the object (content) using the object classifiers 1471, 1472, 1473, and so on. Therefore, the specific process will not be described repeatedly. The processor 1420 can associate the line-of-sight position with the specified target and store it in the viewpoint position information 1454.

Next, the processor 1420 identifies an advertisement that the user 190A will be interested in from the advertisement DB 1445 based on the specified target (content that the user 190A is interested in), and delivers the specified advertisement to the computer 200A. ..

(Control structure)
FIG. 23 is a flowchart showing an example of a series of processes until the user 190 identifies the target to be gazed at and delivers the advertisement. The process shown in FIG. 23 is executed by the processor 1420 of the server 150.

In step S2310, the processor 1420, as the cutting unit 1427, cuts out the peripheral image from the panoramic image 22 based on the viewpoint position received from the computer 200.

In step S2320, the processor 1420 identifies the object classifier to be used as the target identification unit 1428 based on the first tag associated with the panoramic image 22. This process will be described in more detail with reference to FIG. 24.

FIG. 24 shows an example of the data structure of the panoramic image DB 1443. The panoramic image DB 1443 according to a certain embodiment holds the panoramic image 22, the first tag, and the second tag in association with each other.

The first tag specifies an object (content) included in the panoramic image 22. The second tag specifies the type of panoramic image 22. As an example, the first and second tags may be set by the distributor of the panoramic image 22. In another aspect, the first and second tags may be set by the viewer (user) of the panoramic image 22.

In the example shown in FIG. 24, the panoramic image PA1 is associated with "ship" and "bridge" as the first tag, and "travel" and "Mediterranean" as the second tag, respectively. The target identification unit 1428 identifies the object classifier corresponding to each of the “ship” and the “bridge” from the object classifier DB1448 based on the first tag associated with the panoramic image PA1.

With reference to FIG. 23 again, in step S2330, the processor 1420 identifies the target (content) included in the peripheral image by using the specified object classifier as the target identification unit 1428.

In step S2340, the processor 1420 associates the specified target with the viewpoint position and stores it in the storage 1440 (viewpoint position information 1454).

In step S2350, the processor 1420 refers to the first table TL1 and delivers the advertisement associated with the identified target to the source computer 200 (connected to the HMD 110) at the viewpoint position. This process will be described more specifically with reference to FIG.

FIG. 25 shows an example of the data structure of the first table TL1. The first table TL1 holds the advertisement and the target (content) in association with each other. In one aspect, the processor 1420 identifies that the user 190 is looking at a "ship." In such a case, the processor 1420 refers to the first table TL1 and delivers the advertisement AD1 associated with the “ship” to the computer 200 of the source of the viewpoint position. The computer 200 outputs the received advertisement AD1 to the HMD 110. As a result, the user 190 visually recognizes the advertisement AD1.

With reference to FIG. 23 again, in step S2360, the processor 1420 refers to the second table TL2 to identify the panoramic image 22 from which the user 190 will be interested. This process will be described more specifically with reference to FIG.

FIG. 26 shows an example of the data structure of the second table TL2. The second table TL2 includes an object (content) and a type of panoramic image 22. In one aspect, the processor 1420 identifies that the user 190 is looking at a "ship." In such a case, the processor 1420 refers to the second table TL2 and identifies that the type corresponding to the "ship" is "travel". Processor 1420 also refers to the panoramic image DB 1443 to identify the panoramic image 22 associated with "travel" as the second tag.

With reference to FIG. 23 again, in step S2070, the processor 1420 delivers information recommending the identified panoramic image 22 to the source computer 200 (connected to the HMD 110) at the viewpoint position. This information may include, for example, a partial image of the identified panoramic image 22, a panoramic image ID.

The computer 200 recommends the identified panoramic image 22 to the user 190 based on the above information received from the server 150.

FIG. 27 is a diagram for explaining a process of recommending the panoramic image 22 to the user 190. In one aspect, the monitor 112 of the HMD 110 displays a field of view image 2700 for the user 190 to select the panoramic image 22.

The field of view image 2700 includes a selection area 2710, a recommended area 2720, and a viewpoint object 2730. The selection area 2710 includes a part of each of the plurality of panoramic images 22 stored in the panoramic image DB 1443. The recommended area 2720 includes a part of the panoramic image 22 specified based on the above information received from the server 150. That is, the panoramic image 22 included in the recommended area 2720 is likely to be of interest to the user 190. Therefore, the user 190 can easily find the panoramic image 22 of interest from the recommended area 2720.

In one aspect, the user 190 manipulates the viewpoint object 2730 to select the panoramic image 22. As an example, the user 190 superimposes the viewpoint object 2730 on a part of the panoramic image 22 of interest for a predetermined time (for example, 3 seconds). The computer 200 transmits information (for example, a panoramic image ID) representing the panoramic image 22 selected by the user 190 to the server 150. The server 150 transmits the panoramic image 22 selected by the user 190 to the computer 200. The computer 200 constructs the virtual space 2 by using the received panoramic image 22. As a result, the user 190 can visually recognize the virtual space 2 composed of the designated panoramic image 22.

According to the above, the server 150 according to a certain embodiment can identify the target (content) that the user 190 is interested in. At that time, since the server 150 can limit the object classifier to be used based on the first tag associated with the panoramic image 22, the load required for the process of identifying the target can be significantly reduced. In addition, the distributor of the panoramic image 22 can limit the content to be analyzed in advance among the plurality of contents included in the panoramic image 22 by setting the first tag.

In addition, the server 150 can efficiently deliver the advertisement and the panoramic image 22 that the user 190 is likely to be interested in based on the specified target.

[Filtering process]
FIG. 28 is a diagram for explaining a process when the viewpoint position is not stored in the viewpoint position information 1454. With reference to FIG. 28, a panoramic image 22A is developed in the virtual space 2A. Further, in the virtual space 2A, an avatar object 900A corresponding to the user 190A and an avatar object 900B corresponding to the user 190B are arranged.

In the above example, the processor 1420 is configured to store the viewpoint position of the user 190A in the storage 1440 based on the operation (facial expression, voice) of the user 190A. However, the user 190A can communicate with the user 190B on the virtual space 2A. Therefore, the operation of the user 190A may not be caused by the panoramic image 22A, but may be caused by the communication with the user 190B. In such a case, the distributor of the panoramic image 22 cannot accurately grasp the interest of the user 190A in the panoramic image 22. Therefore, when the processor 1420 of the server 150 according to a certain embodiment determines that communication is being performed between the users as the filter unit 1429, the processor 1420 of the server 150 cancels the saving process of the viewpoint position based on the user's operation. Hereinafter, the processing of the filter unit 1429 will be described with reference to FIG. 28.

In a certain aspect, the filter unit 1429 determines that the users 190A and 190B are communicating when they face each other in the virtual space.

The processor 1420 receives line-of-sight information representing the line-of-sight 2810 of the user 190A from the computer 200A. The processor 1420 receives line-of-sight information representing the line-of-sight 2820 of the user 190B from the computer 200B. As an example, the filter unit 1429 determines that the users 190A and 190B face each other in the virtual space when the angle formed by the line of sight 2810 and the line of sight 2820 is approximately 180 degrees (for example, 170 to 190 degrees).

In this case, the processor 1420 does not store the viewpoint position in the storage 1440 even when the operation (facial expression, voice) of the user 190 satisfies the above-mentioned predetermined conditions.

In another aspect, the filter unit 1429 determines that these users are communicating when the distance between the viewpoint of the user 190A and the viewpoint of 190B in the virtual space is narrow.

The processor 1420 receives the position information of the virtual camera 1A (the viewpoint information in the virtual space 2A of the user 190A) from the computer 200A. The processor 1420 receives the position information of the virtual camera 1B from the computer 200B. The filter unit 1429 calculates the distance D between the viewpoint of the user 190A and the viewpoint of the user 190B based on the received position information. The filter unit 1429 determines that the distance between two people in the virtual space is narrow when the distance D is less than a predetermined distance.

In yet another aspect, the filter unit 1429 determines that the users 190A and 190B are communicating when they are speaking.

The processor 1420 receives the first audio signal corresponding to the utterance of the user 190A from the computer 200A. The processor 1420 receives a second audio signal corresponding to the utterance of the user 190B from the computer 200B. The filter unit 1429 determines that the users 190A and 190B are speaking when the first and second audio signals are at or above a predetermined level.

(Control structure)
FIG. 29 is a flowchart showing a process of canceling the process of saving the viewpoint position in the viewpoint position information 1454. Of the processes shown in FIG. 29, the same processes as those described above are designated by the same reference numerals. Therefore, the description of the process will not be repeated.

In step S2910, the processor 1420 receives inputs of face tracking data, the position and tilt of the virtual camera 1, the viewpoint position, the line-of-sight direction, and the voice signal from the computers 200A and 200B. The position of the virtual camera 1 represents the viewpoint of the user 190 in the virtual space 2. The line-of-sight direction is the direction of the line-of-sight in the virtual space 2 of the user 190 specified by the viewpoint identification module 227.

When the operation information is not associated with the viewpoint position (NO in step S2030), the processor 1420 advances the process to step S2920.

In step S2920, the processor 1420 determines whether or not to cancel the process of saving the viewpoint position in the storage 1440 as the filter unit 1429. When the processor 1420 cancels the process (YES in step S2920), the processor 1420 executes the process of step S2710 again. On the other hand, if the processor 1420 does not stop the process (NO in step S2920), the processor 1420 proceeds to the process in step S2040.

FIG. 30 is a flowchart for explaining the process of step S2920. In step S3010, the processor 1420 determines whether or not the line of sight of the user 190A and the line of sight of the user 190B face each other. When the processor 1420 determines that these lines of sight face each other (YES in step S3010), the processor 1420 executes the process of step S2710 again.

In step S3020, the processor 1420 determines whether or not the distance D between the viewpoint of the user 190A and the viewpoint of the user 190B is less than a predetermined distance. When the processor 1420 determines that the interval D is less than the predetermined interval (YES in step S3020), the processor 1420 executes the process of step S2710 again.

In step S3030, the processor 1420 determines whether or not the first audio signal corresponding to the user 190A and the second audio signal corresponding to the user 190B are at or above a predetermined level. If the processor 1420 determines that the first and second audio signals are above a predetermined level (YES in step S3030), the processor 1420 re-executes the process of step S2710.

If the processor 1420 determines that the conditions of steps S301 to S3030 are not satisfied, the processor 1420 proceeds to the process in step S2040.

According to the above, when it is determined that the users are communicating with each other, the server 150 can cancel the saving process of the viewpoint position based on the operation of the user 190. As a result, the server 150 can more accurately acquire the interest of the user 190 in the panoramic image 22.

In the above example, the processor 1420 is configured to stop the saving process of the viewpoint position when at least one of the conditions of steps S301 to S3030 is satisfied. In another aspect, the processor 1420 may be configured to stop the saving process of the viewpoint position when a plurality of conditions among the conditions of steps S310 to S3030 are satisfied.

[composition]
The technical features disclosed above can be summarized as follows.

(Structure 1) According to an embodiment, a program executed by a server 150 capable of communicating with the HMD 110 is provided. This program has a step (S2010) of defining the virtual space 2 in the computer, a step (S2020) of acquiring the viewpoint position of the user 190 of the HMD 110 in the virtual space 2 based on the output of the HMD 110, and an operation or operation of the user 190. A step (S2020) of receiving a signal representing the above, and a step (S2060) of saving the viewpoint position in the storage 1440 (viewpoint position information 1454) when the operation or operation of the user 190 represented by the signal indicates the interest of the user 190. Let it run.

In a certain aspect, the computer 200 outputs the viewpoint position in the virtual space 2 of the user 190 calculated from the output of the gaze sensor 140 provided in the HMD 110 to the server 150. As a result, the server 150 acquires the viewpoint position of the user 190. In another aspect, the computer 200 transmits to the server 150 the line-of-sight information representing the line-of-sight in the virtual space 2 of the user 190 calculated from the output of the gaze sensor 140 provided in the HMD 110. The computer 200 may acquire the viewpoint position in the virtual space 2 of the user 190 based on the received line-of-sight information.

(Structure 2) In (Structure 1), the step of defining the virtual space 2 includes a step of configuring the virtual space 2 using a panoramic moving image. The step of storing the viewpoint position in the storage 1440 includes associating the viewpoint position with the timing at which the viewpoint position is acquired during playback of the moving image and storing the viewpoint position in the storage 1440 (FIG. 18).

(Structure 3) In any of (Structure 1) and (Structure 2), the operation of the user 190 indicating the interest of the user 190 includes an operation on the user interface for accepting the interest of the user 190. This user interface can be, for example, a particular button provided on the controller 160. As another example, this user interface can be a UI object 1640 placed in virtual space 2.

(Structure 4) In any one of (Structure 1) to (Structure 3), the signal representing the operation of the user 190 includes face tracking data representing the facial expression of the user 190.

(Structure 5) The program according to (Structure 4) further includes a step (S1512) of accepting input of reference data used for comparison with face tracking data. The fact that the face tracking data shows the interest of the user 190 includes that the amount of fluctuation of the face tracking data with respect to the reference data exceeds a predetermined amount of fluctuation (S2050).

(Structure 6) In any of (Structure 4) and (Structure 5), the step of storing the viewpoint position in the storage 1440 is among a plurality of facial expression types when the face tracking data indicates the interest of the user 190. From (S2110), the type of facial expression corresponding to the face tracking data is specified, and the type of the specified facial expression and the viewpoint position are associated with each other and stored in the storage 1440 (S2120).

(Structure 7) In any one of (Structure 1) to (Structure 6), the signal representing the operation of the user 190 includes an audio signal corresponding to the utterance of the user 190.

(Structure 8) The program according to (Structure 7) further includes a step of estimating the emotion type of the user 190 corresponding to the voice signal from the plurality of emotion types. The step of storing the viewpoint position in the storage 1440 is to associate the estimated user 190 emotion type with the viewpoint position in the storage 1440 when the estimated user 190 emotion type indicates the interest of the user 190. Including saving.

(Structure 9) In any of (Structure 1) to (Structure 8), the step of saving the viewpoint position in the storage 1440 is such that when the signal indicates the interest of the user 190, the line of sight of the user 190 is noted in the virtual space 2. This includes identifying the target to be removed (S2330), associating the target with the viewpoint position, and storing the target in the storage 1440 (S2340).

(Structure 10) In (Structure 9), the object classifier DB1447 stored in the storage 1440 stores the object classifiers 1471, 1472, 1473, ... For each type of target. The step of defining the virtual space 2 includes a step of constructing the virtual space 2 using the panoramic image 22. To specify the object, cut out the peripheral image of the viewpoint position from the panoramic image (S2310), calculate the feature amount from the peripheral image, and object classifiers 1471, 1472, 1473 stored for each type of object. , ... Includes specifying the target corresponding to the feature amount calculated using, (S2320).

(Structure 11) In (Structure 10), the panoramic image 22 includes first tag information representing an object included in the panoramic image 22 (FIG. 24). To identify the target, among a plurality of object classifiers 1471, 1472, 1473, ... Stored in the storage device, the target object corresponding to the feature amount is used by using the target object classifier indicated by the first tag information. (S2330).

(Structure 12) A program according to any one of (Structure 9) to (Structure 11) further includes a step (S2350) of delivering an advertisement relating to the specified target to the HMD 110.

(Structure 13) In any of (Structure 10) to (Structure 12), the storage 1440 includes a panoramic image DB 1443 that stores a plurality of panoramic images 22. The program according to any one of (Structure 10) to (Structure 12) recommends the step (S2360) of identifying the panoramic image related to the identified target from the plurality of panoramic images 22 and the identified panoramic image. It further includes a step (S2370) of delivering the information to the HMD 110.

(Structure 14) In (Structure 13), the panoramic image 22 includes second tag information indicating the type of the panoramic image 22 (FIG. 24). The storage 1440 includes a panoramic image DB 1443 that holds a correspondence between the object and the type of panoramic image. The step of identifying the panoramic image 22 includes identifying the panoramic image 22 including the second tag information of the type corresponding to the identified object with reference to the panoramic image DB 1443.

(Structure 15) In any one of (Structure 1) to (Structure 14), the step of defining the virtual space 2 includes a step of configuring the virtual space 2 using the panoramic image 22. The program according to any one of (Structure 1) to (Structure 14) further includes a step of generating a graph from the panoramic image 22 and the viewpoint position stored in the storage 1440. The heat map 2200 is an example of this graph.

(Structure 16) A program according to any one of (Structure 1) to (Structure 15) is a first line-of-sight direction in the virtual space 2A of the user 190A and a third in the virtual space 2B of the user 190B using another HMD 110B different from the HMD 110A. 2 The step (S2910) for receiving the input in the line-of-sight direction is further provided. The step of storing the viewpoint position in the storage 1440 includes stopping the storage of the viewpoint position in the storage 1440 (S3010) when the first line-of-sight direction and the second line-of-sight direction face each other.

(Structure 17) A program according to any of (Structure 1) to (Structure 16) is a first viewpoint in the virtual space 2A of the user 190 and a second in the virtual space 2B of the user 190B using another HMD 110B different from the HMD 110A. A step (YES in S3010) for accepting the input of the viewpoint is further provided. The step of saving the viewpoint position in the storage 1440 is to stop saving the viewpoint position in the storage 1440 when the distance D between the first viewpoint and the second viewpoint in the virtual space 2A is less than a predetermined distance (S3020). Including YES).

(Structure 18) The program according to any one of (Structure 1) to (Structure 17) accepts the input of the first audio signal of the user 190 and the second audio signal of the user 190B using another HMD 110B different from the HMD 110A. (S2910) is further provided. The step of storing the viewpoint position in the storage 1440 includes stopping the storage of the viewpoint position in the storage 1440 (YES in S3030) when the first and second audio signals are above a predetermined level. ..

It should be considered that the embodiments disclosed this time are exemplary in all respects and not restrictive. The scope of the present invention is shown by the scope of claims rather than the above description, and it is intended to include all modifications within the meaning and scope equivalent to the scope of claims.

1 virtual camera, 2 virtual space, 10,1420 processor, 11 memory, 12,1440 storage, 22 panoramic image, 23 viewing area, 26,910,1600,2700 view image, 100 HMD system, 105 HMD set, 112 monitor, 114, 120 sensor, 115 1st camera, 117 2nd camera, 118 speaker, 119 microphone, 130 motion sensor, 140 gaze sensor, 150 server, 160 controller, 190 users, 200 computers, 220 display control module, 221 virtual camera control Module, 222 Visibility area determination module, 223 Visibility image generation module, 224 Tilt identification module, 225 Face organ detection module, 226 Motion detection module, 227 Perspective identification module, 230 Virtual space control module, 231 Virtual space definition module, 232 Virtual object Generation module, 233 Operation object control module, 234 Avatar control module, 240 memory module, 241 spatial information, 242 object information, 243, 1444 user information, 244 face template, 245 mouth template, 246 eye template, 247 eyebrow template, 250 communication Control module, 900A, 900B avatar object, 1000 face image, 1010 mouth area, 1100 contour detection line, 1110, 1120 contour point, 1200 mouth shape, 1421 transmitter / receiver, 1422 server processing unit, 1423 matching unit, 1424 viewpoint acquisition unit, 1425 Emotion judgment part, 1426 map generation part, 1427 cutout part, 1428 target identification part, 1429 filter part, 1441 virtual space designation information, 1442 object designation information, 1445 advertisement DB, 1446 reference data DB, 1451 facial expression information, 1452 position information , 1453 tilt information, 1454 viewpoint position information, 1471 object classifier, 1461 facial expression classifier, 1620, 2730 viewpoint object, 1630 operation object, 1640 object, 1650 evaluation object, 1660 peripheral image, 1710, 2810, 2820. Line of sight, 1720 coordinate values, 2200 heatmap, 2710 selection area, 2720 recommended area, P feature points, TL1 first table, TL2 second table.

Claims (5)

A program run by a computer capable of communicating with a headmount device
The program is on the computer
Steps to define virtual space and
A step of acquiring the user's viewpoint position of the head mount device in the virtual space based on the output of the head mount device, and
A step of receiving a signal representing the user's operation or operation, and
A step of distributing information corresponding to the viewpoint position when the operation or operation of the user represented by the signal satisfies a predetermined condition of the user.
A step of acquiring the position of another user different from the user and the virtual camera of the other user in the virtual space is provided.
When it is determined that the user is communicating with the other user, the distribution step includes a step of controlling not to distribute information corresponding to the viewpoint position .
When the distance between the position of the virtual camera in the virtual space of the user and the position of the virtual camera in the virtual space of the other user is equal to or less than the threshold value, it is determined that the communication is performed. The step of delivering the information is a step of controlling the information corresponding to the viewpoint position so as not to be delivered.
Program comprising. The program according to claim 1, wherein the information corresponding to the viewpoint position is an advertisement or a panoramic image based on the content corresponding to the viewpoint position. Further comprising a step of receiving an audio signal corresponding to a conversation between the user and another user.
When the voice level of the voice signal corresponding to the conversation between the user and the other user is equal to or higher than a predetermined level, it is determined that the communication is being performed, and the delivery step is the viewpoint. The program according to any one of claims 1 and 2, which controls not to distribute the information corresponding to the position. An information processing device including a memory for storing the program according to any one of claims 1 to 3 and a processor for executing the program. To the computer of the head mount device,
Steps to define virtual space and
A step of acquiring the user's viewpoint position of the head mount device in the virtual space based on the output of the head mount device, and
A step of receiving a signal representing the user's operation or operation, and a step of distributing information corresponding to the viewpoint position when the user's operation or operation represented by the signal satisfies the predetermined condition of the user.
A step of acquiring the positions of another user different from the user and the virtual camera of the other user in the virtual space.
The delivery step is a step of controlling the user not to deliver the information corresponding to the viewpoint position when it is determined that the user is communicating with another user .
When the distance between the position of the virtual camera in the virtual space of the user and the position of the virtual camera in the virtual space of the other user is equal to or less than the threshold value, it is determined that the communication is performed. The step of delivering the information is a step of controlling the information corresponding to the viewpoint position so as not to be delivered.
A program that executes.

Images