JP6911070B2 - Programs and methods that are executed on a computer to provide virtual space, and information processing devices that execute the programs. - Google Patents

Description

This disclosure relates to a technique for providing a virtual space, and more specifically to a technique for changing a panoramic image developed in the virtual space.

A technique for providing a virtual space using a head-mounted device (HMD) is known. For example, Japanese Patent Application Laid-Open No. 2016-140078 (Patent Document 1) discloses a technique for displaying a panoramic image on a head-mounted display.

Further, in recent years, various technologies have been proposed to enrich the user's experience in virtual space. For example, Non-Patent Document 1 discloses a technique in which avatars of a plurality of users are arranged in a virtual space and communication between users is achieved through these avatars.

Japanese Unexamined Patent Publication No. 2016-140078

"Facebook announces" Spaces ", an app that allows you to meet friends in the VR world", [online], [Search on August 25, 2017], Internet <URL: http://itpro.nikkeibp.co.jp/atcl / idg / 14/481709/042000319/>

In one aspect, the user wants to change the panoramic image developed in the virtual space. Although Non-Patent Document 1 discloses a technique for presenting a panoramic image of a switching candidate to a user, the panoramic image of the switching candidate may not include the panoramic image desired by the user. .. Therefore, there is a need for a technique for presenting a candidate for a panoramic image desired by the user.

The present disclosure has been made to solve the above-mentioned problems, and the purpose in a certain aspect is to present the user with a panoramic image desired by the user in order to change the panoramic image developed in the virtual space. To provide the technology to do.

According to certain embodiments, a headmount device provides a program that runs on a computer to provide virtual space. This program has a step of expanding one panoramic image contained in a plurality of predetermined panoramic images into a virtual space on a computer, and another panorama of a panoramic image expanded in the virtual space from a user of a head mount device. A step of accepting a trigger input for changing to an image, a step of accepting a search key input from a user after accepting a trigger input, and one or more panoramic images searched by a search key. A step of presenting information for identifying a panoramic image in a virtual space, a step of accepting an input from a user to select one panoramic image from one or more panoramic images, and a virtual step of virtualizing the selected one panoramic image. Perform steps to expand into space.

A program according to an embodiment can present a panoramic image desired by the user to the user in changing the panoramic image developed in the virtual space.

The above and other objectives, features, aspects and advantages of the disclosed technical features will become apparent from the following detailed description of the invention, which is understood in connection with the accompanying drawings.

It is a figure which shows the outline of the structure of the HMD system according to a certain embodiment. It is a block diagram which shows an example of the hardware composition of the computer which follows one aspect. It is a figure which conceptually represents the uvw field-of-view coordinate system set in the HMD according to a certain embodiment. It is a figure that conceptually represents one aspect of expressing a virtual space according to a certain embodiment. It is a figure which showed the head of the user who wears an HMD according to a certain embodiment from the top. It is a figure which shows the YZ cross section which looked at the visual field area from the X direction in the virtual space. It is a figure which shows the XZ cross section which looked at the field of view area from the Y direction in a virtual space. It is a figure which shows the schematic structure of the controller according to a certain embodiment. It is a block diagram which shows the computer according to a certain embodiment as a module structure. FIG. 5 is a sequence chart representing a portion of the processing performed in an HMD system according to an embodiment. It is a figure (the 1) explaining the process for changing the panoramic image developed in a virtual space. It is a figure (the 2) explaining the process for changing a panoramic image. It is a figure (the 3) explaining the process for changing a panoramic image. It is a figure for demonstrating the structure of the HMD system according to a certain embodiment. It is a figure explaining the control which detects the mouth from the face image of a user. It is a figure (the 1) explaining the process which the face tracking module detects the shape of a mouth. It is a figure (the 2) for demonstrating the process which the face tracking module detects the shape of a mouth. An example of the structure of face tracking data is shown. It is a block diagram which shows the hardware configuration and the functional configuration of a server. An example of the data structure of the panoramic image DB is shown. An example of the data structure of the learning DB is shown. It is a field-of-view image showing how a user is drawing a line diagram in a virtual space. It is a field-of-view image showing how a user selects a panoramic image. It is a flowchart which shows the process for changing a panoramic image based on a diagram. It is a field-of-view image showing a state of accepting input of a three-dimensional diagram. It is a figure explaining the process of making a three-dimensional diagram into two dimensions. It is a flowchart which shows the process for changing a panoramic image based on face tracking data. It is a visual field image showing a state in which a user inputs a search key by utterance. It is a flowchart which shows the process for switching a panoramic image based on the character string extracted from an audio signal. It is a figure explaining the network when a plurality of users share a virtual space. Represents a visual field image visually recognized by the user.

Hereinafter, embodiments of this technical idea will be described in detail with reference to the drawings. In the following description, the same parts are designated by the same reference numerals. Their names and functions are the same. Therefore, the detailed description of them will not be repeated. In addition, each embodiment and each modification described below may be selectively combined as appropriate.

[HMD system configuration]
The configuration of the HMD system 100 will be described with reference to FIG. FIG. 1 is a diagram showing an outline of the configuration of the HMD system 100 according to an embodiment. In one aspect, HM
The D system 100 is provided as a home system or a business system.

The HMD system 100 includes an HMD 110, an HMD sensor 120, and a controller 16.
0 and a computer 200 are provided. The HMD 110 includes a monitor 112 and a gaze sensor 1.
40, a speaker 115, and a microphone 119 are included. The controller 160 may include a motion sensor 130.

In one aspect, the computer 200 is an internet or other network 19
It is possible to connect to the server 150 and other computers connected to the network 19. In another aspect, the HMD 110 may include a sensor 114 instead of the HMD sensor 120.

The HMD 110 is attached to the head of the user 190, and the virtual space 2 is used by the user 190 during operation.
Can be provided to. More specifically, the HMD 110 displays an image for the right eye and an image for the left eye on the monitor 112, respectively. When each eye of the user 190 visually recognizes the respective image, the user 190 can recognize the image as a three-dimensional image based on the parallax of both eyes.

The monitor 112 is realized as, for example, a non-transparent display device. In one aspect, the monitor 112 is located in the body of the HMD 110 so that it is located in front of both eyes of the user 190. Therefore, the user 190 can immerse himself / herself in the virtual space 2 when he / she visually recognizes the three-dimensional image displayed on the monitor 112. In certain embodiments, the virtual space 2 includes, for example, a background, an object that the user 190 can manipulate, and an image of a menu that the user 190 can select. If a plurality of computers 200 pass signals based on the actions of each user so that a plurality of users can experience a virtual experience in one virtual space 2, an avatar object corresponding to each user is presented in the virtual space 2. Will be done.

The object is a virtual object existing in the virtual space 2. In one aspect, the object is an avatar object corresponding to the user, virtual accessories and virtual clothing worn by the avatar object, a virtual panel that mimics a panel showing information about the user, a virtual letter that mimics a letter, and a post. Includes imitated virtual posts. Further, the avatar object is a character symbolizing the user 190 in the virtual space 2, and includes, for example, a human type, an animal type, a robot type, and the like. Objects come in many shapes.
The user 190 selects a favorite object from the predetermined objects in the virtual space 2
It may be presented to the virtual space 2 or the object created by oneself may be presented to the virtual space 2.

In certain embodiments, the monitor 112 can be realized as a liquid crystal monitor or an organic EL (Electro Luminescence) monitor included in a so-called smartphone or other information display terminal.

In some aspects, the monitor 112 may include a subscreen for displaying an image for the right eye and a subscreen for displaying an image for the left eye. In another aspect, the monitor 112 may be configured to display the image for the right eye and the image for the left eye as a unit.
In this case, the monitor 112 includes a high speed shutter. The high-speed shutter operates so that the image for the right eye and the image for the left eye can be alternately displayed so that the image is recognized by only one of the eyes.

The gaze sensor 140 detects the direction (line-of-sight direction) in which the eyes of the right eye and the left eye of the user 190 are directed. The detection of the direction is realized by, for example, a known eye tracking function. The gaze sensor 140 is realized by a sensor having the eye tracking function. In certain aspects, the gaze sensor 140 preferably includes a sensor for the right eye and a sensor for the left eye. The gaze sensor 140 may be, for example, a sensor that irradiates the right eye and the left eye of the user 190 with infrared light and detects the angle of rotation of each eyeball by receiving the reflected light from the cornea and the iris with respect to the irradiation light. .. The gaze sensor 140 can detect the line-of-sight direction of the user 190 based on each detected rotation angle.

The speaker 115 outputs the voice (utterance) corresponding to the voice data received from the computer 200 to the outside. The microphone 119 outputs voice data corresponding to the utterance of the user 190 to the computer 200. The user 190 can use the microphone 119 to speak to another user, while the speaker 115 can hear the voice (utterance) of the other user.

More specifically, when the user 190 speaks into the microphone 119, the user 19
The voice data corresponding to the utterance of 0 is input to the computer 200. Computer 200
Outputs the voice data to the server 150 via the network 19. Server 15
0 outputs the voice data received from the computer 200 to another computer 200 via the network 19. The other computer 200 outputs the voice data received from the server 150 to the speaker 115 of the HMD 110 worn by another user. As a result, other users can hear the voice of the user 190 through the speaker 115 of the HMD 110. Similarly, utterances from other users are output from the speaker 115 of the HMD 110 worn by the user 190.

The computer 200 monitors an image that moves another avatar object corresponding to the other user according to the voice data received from the computer 200 of the other user.
Display on 12. For example, in a certain situation, the computer 200 displays an image that moves the mouth of another avatar object on the monitor 112, as if it were a virtual space 2.
The virtual space 2 is expressed as if the avatar objects are talking to each other. By transmitting and receiving voice data between the plurality of computers 200 in this way, a conversation (chat) between a plurality of users is realized in one virtual space 2.

The HMD sensor 120 includes a plurality of light sources (not shown). Each light source is realized by, for example, an LED (Light Emitting Diode) that emits infrared rays. The HMD sensor 120 has a position tracking function for detecting the movement of the HMD 110. The HMD sensor 120 uses this function to detect the position and tilt of the HMD 110 in the real space.

In another aspect, the HMD sensor 120 may be realized by a camera. In this case, the HMD sensor 120 can detect the position and tilt of the HMD 110 by executing the image analysis process using the image information of the HMD 110 output from the camera.

In another aspect, the HMD 110 may include the sensor 114 as the position detector instead of the HMD sensor 120. The HMD 110 uses the sensor 114 and the HMD 11
The position and tilt of 0 itself can be detected. For example, when the sensor 114 is an angular velocity sensor, a geomagnetic sensor, an acceleration sensor, a gyro sensor, or the like, the HMD 110 is H.
Instead of the MD sensor 120, any of these sensors can be used to detect its position and tilt. As an example, when the sensor 114 is an angular velocity sensor, the angular velocity sensor detects the angular velocity around the three axes of the HMD 110 in real space over time. HMD11
0 calculates the temporal change of the angle around the three axes of the HMD 110 based on each angular velocity, and further calculates the inclination of the HMD 110 based on the temporal change of the angle.

Further, the HMD 110 may be provided with a transmissive display device. In this case, the transmissive display device may be temporarily configured as a non-transparent display device by adjusting its transmittance. Further, the field-of-view image may include a configuration for presenting the real space as a part of the image constituting the virtual space 2. For example, an image taken by a camera mounted on the HMD 110 may be superimposed on a part of the field of view image and displayed, or by setting a high transmittance of a part of the transmissive display device, the field of view image can be displayed. The real space may be visible from a part.

The server 150 may send the program to the computer 200. In other aspects,
The server 150 may communicate with another computer 200 to provide virtual reality to the HMD 110 used by other users. For example, in an amusement facility, when a plurality of users play a participatory game, each computer 200 communicates a signal based on the operation of each user with another computer 200, and the plurality of users are common in the same virtual space 2. Allows you to enjoy the game. Further, as described above, the plurality of computers 200 transmit and receive signals based on the actions of each user, so that a plurality of users can enjoy the conversation in one virtual space 2.

The controller 160 receives an instruction input from the user 190 to the computer 200. In one aspect, the controller 160 is configured to be grippable by the user 190. In another aspect, the controller 160 is configured to be wearable on a body or part of clothing of the user 190. In another aspect, the controller 160 may be configured to output at least one of vibration, sound, and light based on a signal sent from the computer 200. In another aspect, the controller 160 accepts operations given by the user 190 to control the position and movement of objects placed in the space that provides virtual reality.

The motion sensor 130 is attached to the hand of the user 190 in a certain aspect.
Detects the movement of the user 190's hand. For example, the motion sensor 130 detects the rotation speed, the number of rotations, and the like of the hand. Data representing the detection result of the hand movement of the user 190 obtained by the motion sensor 130 (hereinafter, also referred to as detection data) is sent to the computer 200. The motion sensor 130 is provided in, for example, a glove-shaped controller 160. In one embodiment, for safety in real space, controller 160
Is preferably attached to something that does not easily fly by being attached to the user 190's hand, such as a glove type. In another aspect, a sensor not attached to the user 190 may detect the movement of the user 190's hand. For example, the signal of the camera that captures the user 190 may be input to the computer 200 as a signal representing the operation of the user 190.
The motion sensor 130 and the computer 200 are connected to each other by wire or wirelessly. In the case of wireless, the communication form is not particularly limited, and for example, Bluetooth (
Registered trademark) Other known communication methods are used.

In another aspect, the HMD system 100 may include a television broadcast receiving tuner. According to such a configuration, the HMD system 100 can display a television program in the virtual space 2.

In still other aspects, the HMD system 100 may include a communication circuit for connecting to the Internet or a telephone function for connecting to a telephone line.

[Computer hardware configuration]
The computer 200 according to the present embodiment will be described with reference to FIG. Figure 2 shows
It is a block diagram which shows an example of the hardware composition of the computer 200 which follows one aspect. The computer 200 includes a processor 10, a memory 11, a storage 12, an input / output interface 13, and a communication interface 14 as main components. Each component is connected to the bus 15.

The processor 10 executes a series of instructions included in the program stored in the memory 11 or the storage 12 based on the signal given to the computer 200 or when a predetermined condition is satisfied. In a certain situation, the processor 10
Is a CPU (Central Processing Unit), MPU (Micro Processor Unit), FP
It is realized as a GA (Field-Programmable Gate Array) and other devices.

The memory 11 temporarily stores programs and data. The program is loaded from storage 12, for example. The data includes data input to the computer 200 and data generated by the processor 10. In a certain situation, the memory 11
Is realized as RAM (Random Access Memory) or other volatile memory.

The storage 12 permanently holds programs and data. Storage 12
For example, it is realized as a ROM (Read-Only Memory), a hard disk device, a flash memory, or other non-volatile storage device. The program stored in the storage 12 is H.
The MD system 100 includes a program for providing the virtual space 2, a simulation program, a game program, a user authentication program, and a program for realizing communication with another computer 200. The data stored in the storage 12 includes data and objects for defining the virtual space 2.

In another aspect, the storage 12 may be realized as a detachable storage device such as a memory card. In still other aspects, configurations may be used that use programs and data stored in an external storage device instead of the storage 12 built into the computer 200. According to such a configuration, for example, in a scene where a plurality of HMD systems 100 are used such as an amusement facility, it is possible to update programs and data at once.

In certain embodiments, the input / output interface 13 communicates a signal with the HMD 110, the HMD sensor 120, or the motion sensor 130. In a certain aspect, the input / output interface 13 is a USB (Universal Serial Bus) interface, D.
VI (Digital Visual Interface), HDMI (Registered Trademark) (High-Definition Multi)
media Interface) Realized using other terminals. Input / output interface 1
3 is not limited to the above.

In certain embodiments, the input / output interface 13 further comprises a controller 16.
Can communicate with 0. For example, the input / output interface 13 receives an input of a signal output from the motion sensor 130. In another aspect, the input / output interface 13 is
The instruction output from the processor 10 is sent to the controller 160. The command is vibration,
Instruct the controller 160 to output audio, emit light, and the like. Upon receiving the instruction, the controller 160 executes either vibration, audio output, or light emission in response to the instruction.

The communication interface 14 is connected to the network 19 and is connected to another computer (for example, a server 150, another user's computer 2) connected to the network 19.
Communicate with (00, etc.). In one aspect, the communication interface 14 may be, for example, L.
AN (Local Area Network) or other wired communication interface, or WiFi
(Wireless Fidelity), Bluetooth®, NFC (Near Field Com)
munication) Realized as another wireless communication interface. The communication interface 14 is not limited to the above.

In one aspect, the processor 10 accesses the storage 12 and the storage 12
One or more programs stored in the memory 11 are loaded into the memory 11 and a series of instructions included in the program are executed. The one or more programs may include an operating system for the computer 200, an application program for providing the virtual space 2, game software that can be executed in the virtual space 2 using the controller 160, and the like. The processor 10 sends a signal for providing the virtual space 2 to the HMD 110 via the input / output interface 13. The HMD 110 displays an image on the monitor 112 based on the signal.

In the example shown in FIG. 2, the computer 200 is configured to be provided outside the HMD 110, but in other aspects, the computer 200 may be built in the HMD 110. As an example, a portable information communication terminal (for example, a smartphone) including a monitor 112 may function as a computer 200.

Further, the computer 200 may have a configuration commonly used for a plurality of HMD 110s. According to such a configuration, for example, the same virtual space 2 can be provided to a plurality of users, so that each user can enjoy the same application as other users in the same virtual space 2.

In one embodiment, the HMD system 100 has a preset global coordinate system. The global coordinate system is parallel to the vertical direction in the real space, the horizontal direction orthogonal to the vertical direction, and the front-back direction orthogonal to both the vertical direction and the horizontal direction, respectively.
It has one reference direction (axis). In the present embodiment, the global coordinate system is one of the viewpoint coordinate systems. Therefore, the horizontal direction, the vertical direction (vertical direction), and the front-back direction in the global coordinate system are defined as the x-axis, the y-axis, and the z-axis, respectively. More specifically, in the global coordinate system, the x-axis is parallel to the horizontal direction of real space. The y-axis is parallel to the vertical direction in real space. The z-axis is parallel to the front-back direction of the real space.

In some aspects, the HMD sensor 120 includes an infrared sensor. Infrared sensor is H
When the infrared rays emitted from each light source of the MD110 are detected, the presence of the HMD110 is detected. The HMD sensor 120 further detects the position and inclination of the HMD 110 in the real space according to the movement of the user 190 wearing the HMD 110 based on the value of each point (each coordinate value in the global coordinate system). More specifically, the HMD sensor 120
Each value detected over time can be used to detect temporal changes in the position and tilt of the HMD 110.

The global coordinate system is parallel to the coordinate system in real space. Therefore, the HMD sensor 120
Each slope of the HMD 110 detected by the above corresponds to each slope of the HMD 110 around three axes in the global coordinate system. The HMD sensor 120 is an HMD in the global coordinate system.
Based on the tilt of 110, the uvw field coordinate system is set to HMD110. The uvw field-of-view coordinate system set in the HMD 110 corresponds to the viewpoint coordinate system when the user 190 wearing the HMD 110 sees an object in the virtual space 2.

[Uvw field coordinate system]
The uvw field coordinate system will be described with reference to FIG. FIG. 3 is a diagram conceptually representing the uvw field coordinate system set in the HMD 110 according to an embodiment. HMD sensor 12
0 detects the position and tilt of the HMD 110 in the global coordinate system when the HMD 110 is activated. Processor 10 HMDs the uvw field coordinate system based on the detected values.
Set to 110.

As shown in FIG. 3, the HMD 110 sets a three-dimensional uvw visual field coordinate system centered (origin) on the head of the user 190 wearing the HMD 110. More specifically, HMD1
Reference numeral 10 denotes tilting the horizontal direction, the vertical direction, and the front-back direction (x-axis, y-axis, z-axis) that define the global coordinate system around each axis by the inclination around each axis of the HMD 110 in the global coordinate system. Three new directions obtained by uvw in HMD110
It is set as the pitch direction (u-axis), yaw direction (v-axis), and roll direction (w-axis) of the visual field coordinate system.

In a certain aspect, when the user 190 wearing the HMD 110 is upright and visually recognizing the front, the processor 10 HMD the uvw field coordinate system parallel to the global coordinate system.
Set to 110. In this case, the horizontal direction (x-axis) and the vertical direction (x-axis) in the global coordinate system (
The y-axis) and the anteroposterior direction (z-axis) coincide with the pitch direction (u-axis), yaw direction (v-axis), and roll direction (w-axis) of the uvw visual field coordinate system in the HMD 110.

After the uvw field coordinate system is set to HMD110, the HMD sensor 120 will be HMD11.
Based on the movement of 0, the inclination (change amount of inclination) of the HMD 110 in the set uvw visual field coordinate system can be detected. In this case, the HMD sensor 120 sets the inclination of the HMD 110 as
The pitch angle (θu), yaw angle (θv), and roll angle (θw) of the HMD 110 in the uvw field coordinate system are detected, respectively. The pitch angle (θu) represents the tilt angle of the HMD 110 around the pitch direction in the uvw visual field coordinate system. The yaw angle (θv) represents the tilt angle of the HMD 110 around the yaw direction in the uvw visual field coordinate system. The roll angle (θw) represents the tilt angle of the HMD 110 around the roll direction in the uvw visual field coordinate system.

The HMD sensor 120 determines the HMD 110 based on the detected tilt angle of the HMD 110.
The uvw field-of-view coordinate system in the HMD 110 after the movement is set to the HMD 110. HM
The relationship between D110 and the uvw field coordinate system of HMD110 is always constant regardless of the position and inclination of HMD110. When the position and inclination of the HMD 110 change, the position and inclination of the uvw visual field coordinate system of the HMD 110 in the global coordinate system change in conjunction with the change of the position and inclination.

In one aspect, the HMD sensor 120 is based on the infrared light intensity obtained based on the output from the infrared sensor and the relative positional relationship between the points (eg, the distance between the points). The position of the above in the real space may be specified as a relative position with respect to the HMD sensor 120. Further, the processor 10 may determine the origin of the uvw visual field coordinate system of the HMD 110 in the real space (global coordinate system) based on the specified relative position.

[Virtual space]
The virtual space 2 will be further described with reference to FIG. FIG. 4 is a diagram conceptually representing one aspect of expressing the virtual space 2 according to a certain embodiment. The virtual space 2 is 360 in the center 21.
It has an all-sky spherical structure that covers the entire direction. In FIG. 4, the celestial sphere in the upper half of the virtual space 2 is illustrated so as not to complicate the explanation. Each mesh is defined in the virtual space 2. The position of each mesh is predetermined as a coordinate value in the XYZ coordinate system defined in the virtual space 2. The computer 200 associates each partial image constituting the contents (still image, moving image, etc.) expandable in the virtual space 2 with each corresponding mesh in the virtual space 2, and the virtual space image visible to the user 190. The virtual space 2 in which 22 is deployed is provided to the user 190.

In a certain aspect, the virtual space 2 defines an XYZ coordinate system with the center 21 as the origin. The XYZ coordinate system is, for example, parallel to the global coordinate system. Since the XYZ coordinate system is a kind of viewpoint coordinate system, the horizontal direction, the vertical direction (vertical direction), and the front-back direction in the XYZ coordinate system are defined as the X-axis, the Y-axis, and the Z-axis, respectively. Therefore, X in the XYZ coordinate system
The axis (horizontal direction) is parallel to the x-axis of the global coordinate system, the Y-axis (vertical direction) of the XYZ coordinate system is parallel to the y-axis of the global coordinate system, and the Z-axis (front-back direction) of the XYZ coordinate system is. It is parallel to the z-axis of the global coordinate system.

When the HMD110 is started, that is, in the initial state of the HMD110, the virtual camera 1 is
It is arranged at the center 21 of the virtual space 2. The virtual camera 1 moves in the virtual space 2 in the same manner in conjunction with the movement of the HMD 110 in the real space. As a result, HMD1 in the real space
The changes in the position and orientation of 10 are similarly reproduced in the virtual space 2.

As in the case of the HMD 110, the virtual camera 1 is defined with an uvw field-of-view coordinate system. The uvw field-of-view coordinate system of the virtual camera in the virtual space 2 is defined to be linked to the uvw field-of-view coordinate system of the HMD 110 in the real space (global coordinate system). Therefore, H
When the tilt of the MD 110 changes, the tilt of the virtual camera 1 also changes accordingly. also,
The virtual camera 1 can also move in the virtual space 2 in conjunction with the movement of the user 190 wearing the HMD 110 in the real space.

Since the orientation of the virtual camera 1 is determined according to the position and tilt of the virtual camera 1, the user 1
When the 90 visually recognizes the virtual space image 22, the reference line of sight (reference line of sight 5) is the virtual camera 1.
It depends on the direction of. The processor 10 of the computer 200 defines the field of view 23 in the virtual space 2 based on the reference line of sight 5. The field of view 23 is H in the virtual space 2.
It corresponds to the field of view of the user 190 wearing the MD110.

The line-of-sight direction of the user 190 detected by the gaze sensor 140 is a direction in the viewpoint coordinate system when the user 190 visually recognizes an object. The uvw field-of-view coordinate system of the HMD 110 is equal to the viewpoint coordinate system when the user 190 visually recognizes the monitor 112. Also, u of the virtual camera 1
The vw field-of-view coordinate system is linked to the uvw field-of-view coordinate system of the HMD 110. Therefore, the HMD system 100 according to a certain aspect is the user 190 detected by the gaze sensor 140.
The line-of-sight direction can be regarded as the line-of-sight direction of the user 190 in the uvw field-of-view coordinate system of the virtual camera 1.

[User's line of sight]
The determination of the line-of-sight direction of the user 190 will be described with reference to FIG. FIG. 5 is a top view of the head of the user 190 wearing the HMD 110 according to an embodiment.

In one aspect, the gaze sensor 140 detects each line of sight of the user 190's right and left eyes. In some situations, if the user 190 is looking closer, the gaze sensor 140 will
The lines of sight R1 and L1 are detected. In another aspect, when the user 190 is looking far away, the gaze sensor 140 detects the lines of sight R2 and L2. In this case, the angle formed by the lines of sight R2 and L2 with respect to the roll direction w is smaller than the angle formed by the lines of sight R1 and L1 with respect to the roll direction w. The gaze sensor 140 transmits the detection result to the computer 200.

When the computer 200 receives the detection values of the lines of sight R1 and L1 from the gaze sensor 140 as the detection result of the line of sight, the computer 200 identifies the gaze point N1 which is the intersection of the lines of sight R1 and L1 based on the detected values. On the other hand, when the computer 200 receives the detected values of the lines of sight R2 and L2 from the gaze sensor 140, the computer 200 identifies the intersection of the lines of sight R2 and L2 as the gaze point. The computer 200 specifies the line-of-sight direction N0 of the user 190 based on the position of the specified gazing point N1. For example, the computer 200 detects the midpoint of the straight line connecting the right eye R and the left eye L of the user 190 and the extending direction of the straight line passing through the gazing point N1 as the line-of-sight direction N0. The line-of-sight direction N0 is the direction in which the user 190 actually directs the line of sight with both eyes. Further, the line-of-sight direction N0 corresponds to the direction in which the user 190 actually directs the line of sight with respect to the field of view area 23.

[Visibility area]
The field of view 23 will be described with reference to FIGS. 6 and 7. FIG. 6 is a diagram showing a YZ cross section of the view area 23 viewed from the X direction in the virtual space 2. FIG. 7 is a diagram showing an XZ cross section of the view area 23 viewed from the Y direction in the virtual space 2.

As shown in FIG. 6, the field of view region 23 in the YZ cross section includes the region 24. Area 24
Is defined by the reference line of sight 5 of the virtual camera 1 and the YZ cross section of the virtual space 2. The processor 10 defines a range including the polar angle α centered on the reference line of sight 5 in the virtual space 2 as a region 24.

As shown in FIG. 7, the field of view region 23 in the XZ cross section includes the region 25. Area 25
Is defined by the reference line of sight 5 and the XZ cross section of the virtual space 2. The processor 10 defines a range including the azimuth angle β centered on the reference line of sight 5 in the virtual space 2 as a region 25.

In one aspect, the HMD system 100 provides the user 190 with the virtual space 2 by displaying a field of view image on the monitor 112 based on a signal from the computer 200. The field-of-view image corresponds to a portion of the virtual space image 22 that is superimposed on the field-of-view area 23.
When the user 190 moves the HMD 110 attached to the head, the virtual camera 1 also moves in conjunction with the movement. As a result, the position of the field of view 23 in the virtual space 2 changes. This will
The visual field image displayed on the monitor 112 is updated to an image superimposed on the visual field region 23 in the direction in which the user 190 faces in the virtual space 2 among the virtual space images 22. User 190
The desired direction in the virtual space 2 can be visually recognized.

While wearing the HMD 110, the user 190 can visually recognize only the virtual space image 22 developed in the virtual space 2 without visually recognizing the real world. Therefore, the HMD system 10
0 can give the user 190 a high sense of immersion in the virtual space 2.

In a certain aspect, the processor 10 may move the virtual camera 1 in the virtual space 2 in conjunction with the movement of the user 190 wearing the HMD 110 in the real space. in this case,
The processor 10 HM is based on the position and orientation of the virtual camera 1 in the virtual space 2.
The image area projected on the monitor 112 of the D110 (that is, the view area 23 in the virtual space 2) is specified.

According to certain embodiments, the virtual camera 1 preferably includes two virtual cameras, a virtual camera for providing an image for the right eye and a virtual camera for providing an image for the left eye. Further, it is preferable that an appropriate parallax is set in the two virtual cameras so that the user 190 can recognize the three-dimensional virtual space 2. In the present embodiment, the virtual camera 1 includes two virtual cameras, and the roll direction (w) generated by combining the roll directions of the two virtual cameras is adapted to the roll direction (w) of the HMD 110. The technical idea according to the present disclosure will be illustrated as being configured as such.

[controller]
An example of the controller 160 will be described with reference to FIG. FIG. 8 is a diagram showing a schematic configuration of a controller 160 according to an embodiment.

As shown in the segment (A) of FIG. 8, in some aspects, the controller 160 may include a right controller 800 and a left controller (not shown). Right controller 800
Is operated with the right hand of user 190. The left controller is operated by the left hand of the user 190. In one aspect, the right controller 800 and the left controller are symmetrically configured as separate devices. Therefore, the user 190 can freely move the right hand holding the right controller 800 and the left hand holding the left controller respectively. In another aspect, the controller 160 may be an integrated controller that accepts operations of both hands. Hereinafter, the right controller 800 will be described.

The right controller 800 includes a grip 30, a frame 31, and a top surface 32. The grip 30 is configured to be gripped by the right hand of the user 190. For example, the grip 30 may be held by the palm of the user 190's right hand and three fingers (middle finger, ring finger, little finger).

The grip 30 includes buttons 33 and 34 and a motion sensor 130. Button 33
Is arranged on the side surface of the grip 30 and accepts an operation by the middle finger of the right hand. The button 34 is arranged on the front surface of the grip 30 and accepts an operation by the index finger of the right hand. In one aspect, the buttons 33, 34 are configured as trigger-type buttons. The motion sensor 130 is built in the housing of the grip 30. If the operation of the user 190 can be detected from around the user 190 by a camera or other device, the grip 30 may be used.
The motion sensor 130 may not be provided.

The frame 31 includes a plurality of infrared LEDs 35 arranged along its circumferential direction. The infrared LED 35 emits infrared rays as the program progresses while the program using the controller 160 is being executed. The infrared rays emitted from the infrared LED 35 can be used to detect each position and orientation (tilt, orientation) of the right controller 800 and the left controller. In the example shown in FIG. 8, infrared LEDs 35 arranged in two rows are shown, but the number of arrays is not limited to that shown in FIG. An array of one column or three or more columns may be used.

The top surface 32 includes buttons 36 and 37 and an analog stick 38. Button 36,
37 is configured as a push-type button. Buttons 36 and 37 accept operations by the thumb of the user 190's right hand. The analog stick 38 accepts an operation 360 degrees in any direction from the initial position (neutral position) in a certain aspect. The operation is
For example, it includes an operation for moving an object arranged in the virtual space 2.

In one aspect, the right controller 800 and the left controller are infrared LEDs 35.
Includes batteries for driving other components. Batteries include, but are not limited to, rechargeable, button type, dry cell type and the like. In another aspect, the right controller 800 and the left controller may be connected to, for example, the USB interface of the computer 200. In this case, the right controller 800 and the left controller do not require batteries.

FIG. 8B shows an example of a hand object 810 arranged in the virtual space 2 corresponding to the right hand of the user 190 holding the right controller 800. For example, user 19
The yaw, roll, and pitch directions are defined for the hand object 810 corresponding to the right hand of 0. For example, when the input operation is performed on the button 34 of the right controller 800, the index finger of the hand object 810 is grasped, and when the input operation is not performed on the button 34, the segmentation diagram ( As shown in B), the index finger of the hand object 810 can be extended. For example, in the hand object 810, when the thumb and the index finger are extended, the direction in which the thumb is extended is the yaw direction, and the direction in which the index finger is extended is the roll direction, the yaw direction axis, and the direction perpendicular to the plane defined by the roll direction axis. The direction is defined in the hand object 810 as the pitch direction.

[HMD control device]
The control device of the HMD 110 will be described with reference to FIG. In certain embodiments, the control device is implemented by a computer 200 having a well-known configuration. FIG. 9 is a block diagram showing a computer 200 according to an embodiment as a module configuration.

As shown in FIG. 9, the computer 200 includes a display control module 220, a virtual space control module 230, a voice control module 225, and a memory module 240.
It includes a communication control module 250.

The display control module 220 is a virtual camera control module 22 as a sub module.
1, the field of view area determination module 222, the field of view image generation module 223, the reference line of sight identification module 224, the line of sight detection module 226, and the hand tracking module 2.
27 and a face tracking module 228.

The virtual space control module 230 is a sub-module, and the virtual space definition module 2
31, virtual object generation module 232, hand object control module 23
3 and is included.

In one embodiment, the display control module 220, the virtual space control module 230
, And the voice control module 225 is realized by the processor 10. In other embodiments, the plurality of processors 10 may operate as the display control module 220, the virtual space control module 230, and the voice control module 225. The memory module 240 is realized by the memory 11 or the storage 12. Communication control module 2
50 is realized by the communication interface 14.

In one aspect, the display control module 220 controls the image display on the monitor 112 of the HMD 110. The virtual camera control module 221 has a virtual camera 1 in the virtual space 2.
Is arranged to control the behavior, orientation, etc. of the virtual camera 1. The visual field determination module 222 defines the visual field region 23 according to the orientation of the head of the user 190 wearing the HMD 110.
The visual field image generation module 223 generates visual field image data (also referred to as visual field image data) displayed on the monitor 112 based on the determined visual field region 23. Further, the field of view image generation module 223 generates the field of view image data based on the data received from the virtual space control module 230. The visual field image data generated by the visual field image generation module 223 is output to the HMD 110 by the communication control module 250. The reference line-of-sight identification module 224 detects the reference line-of-sight (inclination of the HMD 110) based on the signal from the HMD sensor 120 or the sensor 114. The line-of-sight detection module 226 is a gaze sensor 1
The line of sight of the user 190 is identified based on the signal from 40. The hand tracking module 227 is the movement of the controller 160 worn by the user 190, that is, the user 19
Detects 0 hand movements. More specifically, the HMD sensor 120 is a controller 160.
The infrared rays emitted by the infrared rays LED 35 provided in the computer 200 are detected and output to the computer 200. The hand tracking module 227 is a controller 160 (each of the right controller 800 and the left controller) based on the detection result input from the HMD sensor 120.
Detect the position of. The face tracking module 228 detects the facial expression of the user 190. Details of the face tracking module 228 will be described later.

The virtual space control module 230 controls the virtual space 2 provided to the user 190.
The virtual space definition module 231 defines the virtual space 2 in the HMD system 100 by generating virtual space data representing the virtual space 2.

The virtual object generation module 232 generates data of an object arranged in the virtual space 2. Objects can include, for example, other avatar objects, virtual panels, virtual letters, and virtual posts. The data generated by the virtual object generation module 232 is output to the field of view image generation module 223.

The hand object control module 233 arranges the hand object in the virtual space 2. The hand object corresponds, for example, to the right or left hand of the user 190 holding the controller 160. In a certain aspect, the hand object control module 233 generates data for arranging the hand object corresponding to the right hand or the left hand in the virtual space 2. Further, the hand object control module 233 generates data for moving the hand object in response to the operation of the controller 160 by the user 190. The data generated by the hand object control module 233 is output to the field of view image generation module 223.

In another aspect, if the movement of a part of the body of the user 190 (eg, the movement of the left hand, right hand, left foot, right foot, head, etc.) is associated with the controller 160, the virtual space control module 230 will be the user 190. Generate data for arranging a partial object corresponding to a part of the body in the virtual space 2. When the user 190 operates the controller 160 using a part of the body, the virtual space control module 230 generates data for moving the partial object. These data are output to the field of view image generation module 223.

When the voice control module 225 detects an utterance using the microphone 119 of the user 190 from the HMD 110, the voice control module 225 identifies the computer 200 to which the voice data to be transmitted corresponding to the utterance is transmitted. The voice data is the computer 2 specified by the voice control module 225.
It is sent to 00. When the voice control module 225 receives voice data from another user's computer 200 via the network 19, the voice control module 225 outputs the voice (utterance) corresponding to the voice data from the speaker 115.

The memory module 240 holds data used by the computer 200 to provide the virtual space 2 to the user 190. In one aspect, the memory module 240
Is spatial information 241 and object information 242, user information 243, and face information 244.
And holds.

Spatial information 241 holds one or more templates defined to provide virtual space 2.

The object information 242 holds the content to be reproduced in the virtual space 2 and the information for arranging the object used in the content. The content is
For example, it may include a game, content representing a landscape similar to the real world, and the like. Moreover,
The object information 242 includes data for arranging a hand object corresponding to the hand of the user 190 who operates the controller 160 in the virtual space 2, data for arranging the avatar object of each user in the virtual space 2, and a virtual panel. It includes data for arranging other objects such as the above in the virtual space 2.

The user information 243 holds a program for operating the computer 200 as a control device of the HMD system 100, an application program for using each content held in the object information 242, and the like. The data and programs stored in the memory module 240 are input by the user 190 of the HMD 110. Alternatively, the processor 10 is a computer operated by a business operator that provides the content (
For example, a program or data is downloaded from the server 150), and the downloaded program or data is stored in the memory module 240. Face information 24
4 includes a template required to detect the facial organs of user 190. In certain embodiments, the face information 244 includes a mouth template 245, an eye template 246, and an eyebrow template 247. Each template can be an image corresponding to the organs that make up the face. For example, the mouth template 245 can be an image of the mouth. In addition, each template may include a plurality of images.

The communication control module 250 can communicate with the server 150 and other information communication devices via the network 19.

In certain aspects, the display control module 220 and the virtual space control module 230 can be implemented using, for example, Unity® provided by Unity Technologies. In another aspect, the display control module 220 and the virtual space control module 230 can also be realized as a combination of circuit elements that realize each process.

The processing in the computer 200 is realized by the hardware and the software executed by the processor 10. Such software may be pre-stored in a hard disk or other memory module 240. Also, the software
It may be stored on a CD-ROM or other computer-readable non-volatile data recording medium and distributed as a program product. Alternatively, the software may be provided as a downloadable program product by an information provider connected to the Internet or other networks. Such software is read from a data recording medium by an optical disk drive or other data reader, or downloaded from a server 150 or other computer via a communication control module 250, and then temporarily stored in a storage module. .. The software is processor 1
Read from the storage module by 0 and stored in RAM in the form of an executable program. Processor 10 executes the program.

The hardware that constitutes the computer 200 is general. Therefore, it can be said that the most essential part according to the present embodiment is the program stored in the computer 200. Since the operation of the hardware of the computer 200 is well known, the detailed description will not be repeated.

The data recording medium is not limited to CD-ROM, FD (Flexible Disk), and hard disk, but also magnetic tape, cassette tape, and optical disk (MO (Magnetic Optical).
Disc) / MD (Mini Disc) / DVD (Digital Versatile Disc)), IC (Integr
ated Circuit) Card (including memory card), optical card, mask ROM, EPROM
(Electronically Programmable Read-Only Memory), EEPROM (Electronicall)
y Erasable Programmable Read-Only Memory), a non-volatile data recording medium such as a semiconductor memory such as a flash ROM that stably carries a program may be used.

The program referred to here may include not only a program that can be directly executed by the processor 10, but also a program in the form of a source program, a compressed program, an encrypted program, and the like.

[Control structure of HMD system]
The control structure of the HMD system 100 will be described with reference to FIG. FIG. 10 is a sequence chart showing a part of the processing performed in the HMD system 100 according to an embodiment.

As shown in FIG. 10, in step S1010, the processor 10 of the computer 200 specifies the virtual space image data as the virtual space definition module 231 and defines the virtual space 2.

In step S1020, the processor 10 initializes the virtual camera 1. for example,
The processor 10 arranges the virtual camera 1 at a predetermined center point in the virtual space 2 in the work area of the memory, and directs the line of sight of the virtual camera 1 in the direction in which the user 190 is facing.

In step S1030, the processor 10 serves as the field of view image generation module 223.
Generates field of view image data for displaying the initial field of view image. The generated visual field image data is output to the HMD 110 by the communication control module 250.

In step S1032, the monitor 112 of the HMD 110 displays the field of view image based on the field of view image data received from the computer 200. The user 190 wearing the HMD 110 can recognize the virtual space 2 when he / she visually recognizes the field of view image.

In step S1034, the HMD sensor 120 detects the position and tilt of the HMD 110 based on the plurality of infrared rays emitted from the HMD 110. The detection result is output to the computer 200 as motion detection data.

In step S1040, the processor 10 identifies the viewing direction of the user 190 wearing the HMD 110 based on the position and inclination included in the motion detection data of the HMD 110.

In step S1050, the processor 10 executes the application program.
An object is presented in the virtual space 2 based on the instruction included in the application program. The object presented at this time includes other avatar objects.

In step S1060, the controller 160 detects the operation of the user 190 based on the signal output from the motion sensor 130, and outputs the detection data representing the detected operation to the computer 200. In another aspect, the operation of the controller 160 by the user 190 may be detected based on an image from a camera arranged around the user 190.

In step S1065, the processor 10 detects the operation of the controller 160 by the user 190 based on the detection data acquired from the controller 160.

In step S1070, the processor 10 generates visual field image data for presenting the hand object in the virtual space 2.

In step S1080, processor 10 is controller 160 by user 190.
Generates field of view image data based on the operation of. The generated visual field image data is output to the HMD 110 by the communication control module 250.

In step S1092, the HMD 110 updates the visual field image based on the received visual field image data, and displays the updated visual field image on the monitor 112.

[Technical Thought]
Next, the technical idea according to the present disclosure will be described with reference to FIGS. 11 to 13. FIG. 11 is a diagram (No. 1) for explaining a process for changing the virtual space image 22 (hereinafter, also referred to as “panorama image 22”) developed in the virtual space 2.

With reference to FIG. 11, a panoramic image 22 including a flower 1110 is developed in the virtual space 2. Further, the virtual space 2 includes the speaker object 1120. In a certain aspect, the user 190 wants to change the panoramic image 22 developed in the virtual space 2. In such a case, the user 190 triggers the computer 200 to change the panoramic image 22.
Is input to (step S1). As an example, the trigger involves bringing the speaker object 1120 into contact with the hand object 810.

After that, the user 190 inputs the search key to the computer 200 (step S2).
.. In the example shown in FIG. 11, the user 190 speaks "animal". Computer 2
00 extracts the character string "animal" from the audio signal input from the microphone 119. The computer 200 transmits the search key 1130 representing the extracted character string to the server 150 (step S3). The server 150 has a storage 1930 that holds a panoramic image DB (database) 1934. The panoramic image DB 1934 holds a plurality of panoramic images (including moving images) in association with information for identifying the panoramic image (hereinafter, also referred to as “panoramic image ID”).

When the search key 1130 is input from the computer 200, the server 150 searches for one or more panoramic images from the plurality of panoramic images stored in the panoramic image DB 1934 based on the search key 1130 (step S4). ). The server 150 generates thumbnails 1140 of one or more searched panoramic images. The thumbnail 1140 functions as information for identifying the panoramic image. The server 150 generated the thumbnail 1
The 140 and the panoramic image ID 1150 are transmitted to the computer 200 (step S5).
..

FIG. 12 is a diagram (No. 2) for explaining the process for changing the panoramic image 22. The field of view image 1200 shown in FIG. 12 includes thumbnails 121-10260. These thumbnails correspond to one or more received thumbnails 1140. Moreover, these thumbnails may be either a still image or a moving image.

The user 190 selects one thumbnail (panoramic image) of the thumbnails 121 to 1260 with the hand object 810 (step S6). For example, the user 190 can bring the thumbnail 1210 containing the dog into contact with the hand object 810.
Select thumbnail 1210.

FIG. 13 is a diagram (No. 3) for explaining the process for changing the panoramic image 22. The computer 200 has a panoramic image I corresponding to a thumbnail selected from the user 190.
D1310 is transmitted to the server 150 (step S7). The server 150 transmits the panoramic image 1320 corresponding to the received panoramic image ID to the computer 200. The computer 200 expands the received panoramic image 1320 in the virtual space 2 (step S9).
). As a result, the user 190 can use the panoramic image 1 including the dog 1330 in the virtual space 2.
320 can be visually recognized.

According to the above, the computer 200 can present to the user 190 one or more panoramic images related to the search key input to the user 190, in other words, the panoramic image (identifying information) desired by the user 190. Hereinafter, more specific processing for changing the panoramic image will be described.

[Configuration of HMD system 1400]
FIG. 14 is a diagram for explaining the configuration of the HMD system 1400 according to an embodiment. The HMD system 1400 differs from the HMD system 100 described with reference to FIG. 1 in that it has an HMD 1410 instead of the HMD 110.

The HMD 1410 is H in that it has a first camera 116 and a second camera 117.
It is different from MD110. The first camera 116 captures the lower part of the user 190's face. More specifically, the first camera 116 captures the nose, cheeks, mouth, etc. of the user 190. The second camera 117 photographs the eyes, eyebrows, and the like of the user 190. The housing on the user 190 side of the HMD 1410 is defined as the inside of the HMD 1410, and the housing on the side opposite to the user 190 of the HMD 1410 is defined as the outside of the HMD 1410. In one aspect, the first camera 116 is the HMD1
Located outside the 410, the second camera 117 may be located inside the HMD 1410.
The images generated by the first camera 116 and the second camera 117 are input to the computer 200.

The face tracking module 228 detects organs (for example, mouth, eyes, eyebrows) constituting the face of the user 190 from the images of the face of the user 190 generated by the first camera 116 and the second camera 117 of the user 190. The face tracking module 228 intermittently detects the position of the feature point for each detected organ. In other words, the face tracking module 228 detects the facial expression of the user 190. Hereinafter, more specific processing of the face tracking module 228 will be described.

[Face tracking]
Hereinafter, a specific example for detecting a user's facial expression (face movement) will be described with reference to FIGS. 15 to 18. 15 to 18 show, as an example, a specific example of detecting the movement of the mouth of the user 190. The detection method described with reference to FIGS. 15 to 18 is described by the user 19.
Not limited to 0 mouth movements, other organs that make up the user 190's face (eg, eyes, eyebrows, nose, etc.)
It can also be applied to the detection of cheek) movement.

FIG. 15 is a diagram illustrating control for detecting a mouth from a user's face image 1500.
The face image 1500 generated by the first camera 116 includes the nose and mouth of the user 190.

The face tracking module 228 identifies the mouth region 1510 from the face image 1500 by pattern matching using the mouth template 245 stored in the face information 244. In a certain aspect, the face tracking module 228 sets a comparison area on a rectangle in the face image 1500, and changes the size, position, and angle of the comparison area to obtain an image of the comparison area and an image of the mouth template 245. Calculate the similarity with. The face tracking module 228 may identify a comparison region for which a similarity greater than a predetermined threshold is calculated as the mouth region 1510.

The face tracking module 228 is further based on the relative relationship between the position of the comparison region where the calculated similarity is greater than the threshold and the position of other facial organs (eg, eyes, nose).
It can be determined whether or not the comparison region corresponds to the mouth region.

The face tracking module 228 detects a more detailed mouth shape based on the identified mouth region 1510.

FIG. 16 is a diagram (No. 1) illustrating a process by which the face tracking module 228 detects the shape of the mouth. With reference to FIG. 16, the face tracking module 228
A plurality of contour detection lines 16 for detecting the shape of the mouth (contour of the lips) included in the mouth region 1510.
Set 00. Each contour detection line 1600 is set at a predetermined interval in a direction orthogonal to the height direction of the face.

The face tracking module 228 can detect a change in the brightness value of the mouth region 1510 along each of the plurality of contour detection lines 1600, and can specify a position where the change in the brightness value is abrupt as a contour point. More specifically, the face tracking module 228 can specify as a contour point a pixel whose luminance difference (that is, change in luminance value) from adjacent pixels is equal to or greater than a predetermined threshold value. The brightness value of the pixel is obtained, for example, by integrating the RBG value of the pixel with a predetermined weighting.

The face tracking module 228 identifies two types of contour points from the image corresponding to the mouth region 1510. The face tracking module 228 identifies contour points 1610 corresponding to the outer contour of the mouth (lips) and contour points 1620 corresponding to the inner contour of the mouth (lips). In a certain aspect, when three or more contour points are detected on one contour detection line 1600, the face tracking module 228 sets the contour points at both ends to the outer contour points 16.
Can be specified as 10. In this case, the face tracking module 228 can identify contour points other than the outer contour points 1610 as the inner contour points 1620. Further, when two or less contour points are detected on one contour detection line 1600, the face tracking module 228 can specify the detected contour points as the outer contour points 1610.

FIG. 17 is a diagram (No. 2) for explaining a process in which the face tracking module 228 detects the shape of the mouth. In FIG. 17, the outer contour points 1610 are shown as white circles, and the inner contour points 1620 are shown as hatched circles.

The face tracking module 228 identifies the mouth shape 1700 by complementing between the inner contour points 1620. In this case, the contour point 1620 can be said to be a feature point of the mouth.
In certain aspects, the face tracking module 228 may identify the mouth shape 1700 using a non-linear complementation method such as spline complementation. In another aspect, the face tracking module 228 may specify the mouth shape 1700 by complementing the outer contour points 1610. In yet another aspect, the face tracking module 228 has an assumed mouth shape (a predetermined shape that can be formed by a person's upper and lower lips).
Therefore, the contour points that deviate significantly may be excluded, and the mouth shape 1700 may be specified by the remaining contour points. In this way, the face tracking module 228 can identify the movement (shape) of the user's mouth. The method for detecting the mouth shape 1700 is not limited to the above, and the face tracking module 228 may detect the mouth shape 1700 by another method. In addition, the face tracking module 228 may detect the movements of the user's eyes and eyebrows in the same manner. The face tracking module 228 may be configured to be able to detect the shape of organs such as cheeks and nose.

FIG. 18 shows an example of the structure of face tracking data. The face tracking data represents the position coordinates of a plurality of feature points constituting the shape of each organ in the uvw visual field coordinate system. For example, the points m1, m2, ... Shown in FIG. 18 correspond to the inner contour points 1620 constituting the mouth shape 1700. In one aspect, face tracking data
It is a coordinate value in the uvw field of view coordinate system with the position of the first camera 116 or the second camera 117 as a reference (origin). In another aspect, the face tracking data is a coordinate value in a coordinate system with a predetermined feature point as a reference (origin) for each organ. As an example, the points m1, m2, ... Are coordinate values in the coordinate system with the origin of any one of the inner contour points 1620 corresponding to the corner of the mouth.

[Configuration of server 150]
FIG. 19 is a block diagram showing a hardware configuration and a functional configuration of the server 150. In certain embodiments, the server 150 comprises a communication interface 1910, a processor 1920, and a storage 1930 as main hardware.

The communication interface 1910 functions as a communication module for wireless communication that performs modulation / demodulation processing for transmitting / receiving signals to / from an external communication device such as a computer 200. The communication interface 1910 is realized by a tuner, a high frequency circuit, or the like.

The processor 1920 controls the operation of the server 150. The processor 1920 executes various control programs stored in the storage 1930, so that the transmission / reception unit 19
It functions as 21, a server processing unit 1922, a search engine 1923, an object identification unit 1924, an emotion judgment unit 1925, and a learning unit 1926.

The transmission / reception unit 1921 transmits / receives various information to / from each computer 200. For example, the transmission / reception unit 1921 transmits a request for arranging an object in the virtual space 2, a request for deleting the object from the virtual space 2, a request for moving the object, a user's voice, and the like to each computer 200.

The server processing unit 1922 updates the user DB 1933 and the learning DB 1935, which will be described later, based on the information received from the computer 200. Search engine 1923
Based on the search key received from the computer 200, one or more panoramic images are searched from among the plurality of panoramic images included in the panoramic image DB 1934.

The object identification unit 1924 uses the learning model 1936 described later to be used in the computer 200.
Identify the candidate objects corresponding to the diagram data input from. The emotion determination unit 1925 determines the emotion candidates of the user 190 corresponding to the face tracking data received from the computer 200 by using the learning model 1937 described later. The learning unit 1926 updates the learning models 1936 and 1937 described later.

The storage 1930 has virtual space designation information 1931 and object designation information 1932.
, The user DB 1933, the panoramic image DB 1934, the learning DB 1935, and the learning models 1936 and 1937.

The virtual space designation information 1931 is information used by the virtual space definition module 231 of the computer 200 to define the virtual space 2. For example, virtual space designation information 193
1 contains information that specifies the size or shape of the virtual space 2. Object specification information 19
32 specifies an object to be arranged (generated) in the virtual space 2 by the virtual object generation module 232 of the computer 200.

The user DB 1933 includes information for identifying each user 190 of the plurality of computers 200 connected to the network 19 (hereinafter, also referred to as "user ID"), and user attribute information. Attribute information includes, for example, age, place (address, etc.), gender, hobbies, and the like. The panorama image DB 1934 holds the panorama image ID and the panorama image (including the moving image) in association with each other.

The learning DB 1935 holds the search key and the panoramic image selected by the user 190 among the one or more panoramic images searched by the search key in association with each other. In a certain aspect, the learning DB 1935 may further associate and hold a user ID with this information. The data structures of the panoramic image DB 1934 and the learning DB 1935 will be described later.

The learning model 1936 holds a program for identifying an object corresponding to the diagram data input from the computer 200. The learning model 1937 holds a program for determining the emotion of the user 190 corresponding to the face tracking data input from the computer 200. As an example, learning models 1936, 1937 are neural networks that include an input layer, multiple intermediate layers, and an output layer. The learning unit 1926 updates the weighting coefficient between each node of the learning models 1936 and 1937 based on the learning data held in the learning DB 1935. As an example, the learning unit 1926 updates the weighting coefficient according to the backpropagation method.

(Panorama image DB1934)
FIG. 20 shows an example of the data structure of the panoramic image DB 1934. Panorama image DB1
The 934 holds the panoramic image ID, the image data, the title, and the tag in association with each other.

The image data may include moving image data. The image data is, for example, the computer 20.
It is posted (registered) to the server 150 by the user of 0 or the administrator of the server 150. The title is set by the poster of the image data. The tag is set by the poster of the image data or the viewer of the panoramic image. The tag specifies the genre of the panoramic image, the object included in the panoramic image, and the like. Tags can also include emotion tags. The emotion tag represents an emotion presumed to be obtained by the user 190 viewing the panoramic image.
As an example, emotion tags such as "fun", "relaxation", and "satisfaction" are set in the panoramic image of the comedy moving image.

(DB1935 for learning)
FIG. 21 shows an example of the data structure of the learning DB 1935. The learning DB 1935 includes tables 2110, 2120, and 2130. Table 2110 displays a search key as line diagram data, an object represented by the line diagram, and a panoramic image ID of a panoramic image selected by the user 190 among one or more panoramic images searched by the search key. Keep related to each other. Table 2120 shows a search key as face tracking data, emotions represented by the face tracking data, and a panoramic image ID of a panoramic image selected by the user 190 among one or more panoramic images searched by the search key. Are associated with each other and held. Table 2130 shows a search key as a character string extracted from an audio signal, and a user 19 of one or more panoramic images searched by the search key.
The panoramic image ID of the panoramic image selected as 0 is associated with each other and held.

Hereinafter, as the search key, line diagram data representing a line diagram drawn by the user 190 in the virtual space 2, face tracking data representing the facial expression of the user 190, and a character string extracted from an audio signal based on the utterance of the user 190 are used. An embodiment will be described.

[Use diagram data as search key]
FIG. 22 is a field of view image 2 showing a user 190 drawing a diagram in the virtual space 2.
It is 200. The field of view image 2200 includes a desk object 2210. Desk object 2
An input object 2220 for the computer 200 to receive the input of the diagram from the user 190 is arranged on the 210. The shape of the input object 2220 is a substantially rectangular parallelepiped with a flat surface.

In one aspect, the pen object 2230 is placed on top of the desk object 2210. Further, a message object 2240 for prompting input of a diagram to the input object 2220 is arranged in the vicinity of the input object 2220.

In the example shown in FIG. 22, the hand object 810 and the pen object 2230
Is associated with. For example, the processor 10 of the computer 200 detects that the hand object 810 and the pen object 2230 are in contact with each other and associates these objects with each other.

User 190 runs controller 160 with these objects associated with it. The processor 10 is a hand tracking module 227, and the controller 1
The movement of 60 is reflected in the hand object 810. As a result, the pen object 22
30 moves in conjunction with the hand object 810.

The user 190 inputs a diagram to the input object 2220 using the pen object 2230. More specifically, the processor 10 inputs a locus object 2250 representing a locus of contact between the pen object 2230 and the input object 2220 as the input object 2.
Place on 220. As a result, the user 190 can recognize the line diagram (corresponding to the locus object 2250) drawn by the user 190. In certain embodiments, the processor 10 uses the pen object 22 only while a predetermined button on the controller 160 is pressed.
It may be configured to track the trajectory of contact between 30 and the input object 2220.

The computer 200 transmits the data of the locus object 2250 (the diagram data representing the diagram drawn by the user 190) to the server 150 as a search key.

The processor 1920 of the server 150 serves as an object identification unit 1924 and is a computer 20.
The candidate of the object represented by the diagram data input from 0 is specified. More specifically, the object identification unit 1924 inputs the diagram data to each input layer of the learning model 1936, and identifies the candidate object represented by the diagram data based on the values output from each output layer. .. As an example, the object identification unit 1
924 specifies an object corresponding to an output layer that outputs a value exceeding a predetermined threshold value as a candidate for an object represented by the diagram data. The object identification unit 1924 outputs information (for example, a character string) representing a candidate of the specified object to the search engine 1923.

The search engine 1923 searches for one or more panoramic images from a plurality of panoramic images stored in the panoramic image DB 1934 based on the information input from the object identification unit 1924. For example, the search engine 1923 accepts input of character strings of "cat" and "dog" from the object identification unit 1924. The search engine 1923 uses the entered character string to perform the search engine 1923.
A panoramic image containing the character string "cat" or "dog" in the title or tag is searched from the panoramic image DB 1934. The processor 1920 generates thumbnails from the image data of the searched panoramic image. The processor 1920 has generated thumbnails and
The panoramic image ID corresponding to the thumbnail is transmitted to the computer 200.

FIG. 23 is a field of view image 2300 showing how the user 190 selects a panoramic image. The field of view image 2300 is a screen 231 representing a search key entered by the user 190.
0 and thumbnails 2320 to 2350 received from the server 150 are included.

The user 190 selects one thumbnail (panoramic image) of the thumbnails 2320 to 2350 with the hand object 810. The computer 200 transmits the panoramic image ID corresponding to the selected thumbnail to the server 150. The server 150 transmits the image data corresponding to the received panoramic image ID to the computer 200. Computer 2
00 expands the received image data into the virtual space 2. As a result, the user 190 can view the desired panoramic image. When the panoramic image is a moving image, the computer 200 may download the panoramic image in advance and then develop it in the virtual space 2, or the panoramic image may be streamed and reproduced.

The processor 1920 of the server 150 identifies the object represented by the search key based on the tag associated with the panoramic image ID received from the computer 200. The processor 1920 stores the search key (line diagram data), the object represented by the search key, and the panoramic image ID in the table 2110 of the search DB 1935. In a certain aspect, the processor 1920, as the learning unit 1926, executes learning (supervised learning) of the learning model 1936 based on the diagram data and the object represented by the diagram data.

According to the above, the server 150 is the user 190 based on the diagram drawn by the user 190.
Can increase the possibility that the user 190 can be provided with the desired panoramic image candidate. Further, as shown in the field of view image 2300 of FIG. 23, the user 190 uses the search key (search key (1) entered by himself / herself.
Since the panoramic image (thumbnail) searched by the search key is presented together, the panoramic video corresponding to the search key (character string instead of the line diagram) that you have in mind is accurately selected. Can be done. As a result, the server 150 can acquire high-quality learning data.
Learning efficiency can be improved.

In one aspect, the field of view image 2300 includes a question object 2360 for asking the user 190 what the object corresponding to the search key is. For example, in the question object 2360, whether or not the object corresponding to the search key is the object corresponding to the output layer that outputs the largest value (that is, the object having the highest probability of being the object intended by the user). Inquire. The question object 2360 includes the answer objects 2370 and 2380 for accepting the input of the answer to the inquiry. The user 190 selects one of the answer objects 2370 and 2380 with the hand object 810. The computer 200 transmits the response result of the user 190 to the inquiry to the server 150. The processor 1920 of the server 150 may learn the learning model 1936 based on the received answer result and the diagram data.

(Control structure for panoramic image change processing based on diagram)
FIG. 24 is a flowchart showing a process for changing a panoramic image based on a diagram. The processing shown in FIG. 24 is the processor 10 and the server 15 of the computer 200.
It is realized by the processor 1920 of 0.

In step S2405, the processor 10 is used as the virtual space definition module 231.
A panoramic image is developed in the virtual space 2. In step S2410, the processor 10 arranges various objects such as an input object, a pen object, a hand object functioning as an operation object, and a virtual camera in the virtual space 2.

In step S2415, the processor 10 determines whether or not the operation object and the pen object have come into contact with each other. When the processor 10 determines that these objects have touched (YES in step S2415), the processor 10 associates these objects. Further, the processor 10 accepts the contact of these objects as a trigger for changing the panoramic image. Otherwise (NO in step S2415), processor 10 waits until it accepts the trigger.

In step S2420, the processor 10 accepts a diagram of the input object (for example, the locus of contact points between the pen object and the input object) as a search key.
In step S2425, the processor 10 determines whether or not the association between the operation object and the pen object has been released. For example, user 190 is the controller 16
By pressing a predetermined button provided at 0, the above association is released.
When the processor 10 determines that the association between the operation object and the pen object has been released (YES in step S2425), the processor 10 transmits the diagram data input in step S2420 to the server 150 (step S2430). If not (step S24)
No at 25), the processor 10 re-executes the process of step S2420.

In step S2435, the processor 1920 receives the diagram data (search key) from the computer 200. In step S2440, the processor 1920 identifies an object (eg, a character string representing the object) based on the diagram data. In step S2445, the processor 1920 uses the panoramic image DB 1934 based on the character string representing the identified object.
Search for one or more panoramic images from multiple panoramic images stored in. Step S
At 2450, the processor 1920 generates thumbnails for each of the searched one or more panoramic images and transmits the thumbnails and the panoramic image ID to the computer 200.

In step S2455, the processor 10 receives the thumbnail and the panoramic image ID from the server 150. In step S2460, the processor 10 presents in the virtual space 2 a thumbnail that functions as information for identifying the searched panoramic image. In step S2465, the processor 10 accepts the selection of one panoramic image (thumbnail) from one or more panoramic images (thumbnails) from the user 190. Step S24
At 70, the processor 10 transmits the panoramic image ID corresponding to the selected thumbnail to the server 150.

In step S2475, the processor 1920 assigns the panoramic image ID to the computer 2.
Receive from 00. In step S2480, the processor 1920 has a panoramic image DB.
With reference to 1934, the image data corresponding to the received panoramic image ID is input to the computer 2
Send to 00.

In step S2485, the processor 10 is used as the virtual space definition module 231.
The received image data is expanded into the celestial sphere that constitutes the virtual space 2. As a result, user 190
Can visually recognize the panoramic image desired by itself.

In step S2490, the processor 1920 learns (updates) the learning model 1936 as the learning unit 1926 based on the diagram data input as the search key and the object represented by the diagram data.

(Search based on a three-dimensional diagram)
In the above example, the computer 200 is configured to accept input of a planar diagram drawn by the user with respect to a flat input object. In another aspect, the computer 200 may be configured to accept a three-dimensional diagram drawn by the user as a search key.

FIG. 25 is a field of view image 2500 showing a state of accepting input of a three-dimensional diagram. The field of view image 2500 includes an input object 2510. In the example shown in FIG. 25, the input object 2510 is a cube, and each side constituting the cube is indicated by a broken line.

User 190 is the hand object 81 with which the pen object 2230 is associated.
By manipulating 0, a three-dimensional diagram can be drawn in the area surrounded by the input object 2510. In the example shown in FIG. 25, an airplane 2520 is depicted.

The computer 200 makes the airplane 2520 (three-dimensional diagram) two-dimensional. More specifically, the airplane 2520 is transferred to each plane of the cube constituting the input object 2510. It is assumed that the input object 2510 is arranged parallel to each axis set in the virtual space 2. In such a case, the computer 200 is an airplane 2520, as shown in FIG.
Is transferred to the XY plane, the YZ plane, and the ZX plane, respectively. FIG. 26 (A) is a diagram in which the airplane 2520 is transferred to the XY plane. FIG. 26B is a diagram in which the airplane 2520 is transferred to the YZ plane. FIG. 26C is a diagram of the airplane 2520 transferred to the ZX plane.

The computer 200 transmits the obtained two-dimensional diagram data to the server 150 as a search key. Since the subsequent processing is as described above, the description of the processing will not be repeated.

According to the above, the user 190 can input the three-dimensional diagram drawn on the virtual space 2 into the computer 200 and the server 150 as a search key. In one aspect, user 190
May communicate with other users via the virtual space 2. In such a case, the user 190
Whether or not the server 150 can correctly recognize the object intended by the user 190 based on the input diagram can be enjoyed together with other users as if it were a game.

[Use face tracking data as a search key]
Next, an embodiment in the case where the face tracking data, that is, the facial expression of the user 190 is used as the search key will be described.

FIG. 27 is a flowchart showing a process for changing a panoramic image based on face tracking data. Of the processes shown in FIG. 27, the same processes as those in FIG. 24 are designated by the same reference numerals. Therefore, the description of the process will not be repeated.

In step S2710, the processor 10 of the computer 200 determines whether or not the trigger for changing the panoramic image developed in the virtual space 2 has been accepted. For example, the processor 10 receives the trigger in response to the pressing of a predetermined button provided on the controller 160.

In step S2720, the processor 10 acquires face tracking data. At this time, the processor 10 may inquire of the user 190 about the current emotion (mood) or the emotion desired to be obtained by viewing the panoramic image. User 190 makes a facial expression in response to the inquiry. The first camera 116 and the second camera 117 generate an image of the face of the user 190 at that time. Processor 10 is the face tracking module 228
To generate face tracking data based on the input face image. In step S2730, the processor 10 transmits the generated face tracking data to the server 150 as a search key.

In step S2740, the processor 1920 of the server 150 is the computer 200.
Accepts face tracking data input from. In step S2750, the processor 1920 identifies the emotion candidate of the user 190 represented by the input face tracking data as the emotion determination unit 1925. More specifically, the emotion determination unit 1925 inputs face tracking data to each input layer of the learning model 1937, and identifies emotion candidates represented by the face tracking data based on the values output from each output layer. .. As an example, the emotion determination unit 1925 specifies an emotion corresponding to an output layer that outputs a value exceeding a predetermined threshold value as an emotion candidate represented by the face tracking data. Step S2
At 760, the processor 1920 searches for one or more panoramic images including the emotion tag of the specified emotion from the plurality of panoramic images stored in the panoramic image DB 1934.

In step S2790, the processor 1920 learns (updates) the learning model 1937 as the learning unit 1926 based on the face tracking data input as the search key and the emotion represented by the face tracking data.

According to the above, the user 190 simply makes a facial expression (for example, laughs) and feels like himself (for example, laughing).
You can receive candidates for panoramic images that match your emotions. Further, the server 150 can increase the possibility of providing the user 190 with a candidate for a panoramic image desired by the user 190 based on the facial expression (representing face tracking data) of the user 190.

In the above example, the server 150 is configured to estimate the emotion of the user 190 based on the face tracking data, but in other aspects, the emotion of the user 190 is estimated based on other data. It may be configured in. For example, server 150
An audio signal based on the utterance of the user 190 may be received from the computer 200, and the emotion of the user 190 may be estimated based on the audio signal. For example, the server 150 extracts a character string from the audio signal and estimates the emotion from the character string. Such processing is, for example,
It can be realized by the "emotion analysis API" provided by Metadata. In another aspect, the server 150 estimates emotions from the waveform of the audio signal. Such processing is, for example, AG
It can be realized by "ST Emotion SDK" provided by Company I. Note that these processes may be configured to be executed by the computer 200 instead of the server 150. In such a case, the computer 200 transmits information for identifying the estimated emotion to the server 150.

[Use the character string extracted from the audio signal as the search key]
Next, an embodiment in the case where a character string extracted from the voice signal corresponding to the utterance of the user 190 is used as the search key will be described.

FIG. 28 is a field of view image 2800 showing a user 190 inputting a search key by utterance.
Is. The field of view image 2800 includes a desk object 2210. Desk object 2210
A speaker object 2810 is arranged above the speaker object 2810. Further, the pointer object 2820 is arranged so as to be superimposed on the speaker object 2810.

The pointer object 2820 represents the viewpoint of the user 190 in the virtual space 2. As the line-of-sight detection module 226, the processor 10 detects the line-of-sight of the user 190 in the real space based on the output of the gaze sensor 140. The line-of-sight detection module 226 converts the line of sight of the user 190 in the real space into the line of sight in the virtual space 2 based on the position and inclination (reference line of sight) of the virtual camera 1. The virtual object generation module 232 is a pointer object 2820 at a position where the line of sight in the virtual space 2 collides with the object.
To place.

In one aspect, the user 190 selects the speaker object 2810 to trigger the computer 20 to change the panoramic image developed in the virtual space 2.
Enter 0. As an example, the user 190 selects the speaker object 2810 by bringing the hand object 810 and the speaker object 2810 into contact with each other.
As another example, the user 190 selects the speaker object 2810 by gazing at the speaker object 2810 for a predetermined time, that is, by superimposing the pointer object 2820 on the speaker object 2810 for a predetermined time.

The processor 10 of the computer 200 notifies the user 190 that the trigger has been input. For example, the processor 10 outputs a voice or a message prompting the user 190 to speak, such as "Speak what you want to search." According to this, user 19
0 speaks the content to be searched. The processor 10 receives an audio signal from the microphone 119 and extracts a character string from the audio signal. As an example, the processor 10 extracts a character string by collating the waveform data divided by a predetermined time unit (for example, 10 msec unit) from the beginning of the audio signal with an acoustic model (not shown) stored in the storage 12. do. The acoustic model represents the features of each phoneme such as vowels and consonants. As an example, the processor 10 matches an audio signal with an acoustic model based on a hidden Markov model. The processor 10 transmits the extracted character string to the server 150.

The processor 1920 of the server 150 searches for a panoramic image including a character string (or a word included in the character string) input in the title or tag from a plurality of panoramic images stored in the panoramic image DB 1934. Subsequent processing is the same as the above processing and is not repeated.

According to the above, the user 190 can input the search key into the computer 200 by simply speaking. Further, in the real world, voice communication between a device having a speaker function (for example, Amazon Echo (registered trademark)) and a user has become common. Therefore, the user 190 can easily imagine that the speaker object having the shape of the speaker has the voice input and output functions. In another aspect, a microphone object having a microphone shape may be arranged instead of the speaker object.

(Control structure)
FIG. 29 is a flowchart showing a process for switching a panoramic image based on a character string extracted from an audio signal. Of the processes shown in FIG. 29, the same processes as those in FIG. 24 are designated by the same reference numerals. Therefore, the description of the process will not be repeated.

In step S2910, the processor 10 of the computer 200 determines whether or not the trigger for changing the panoramic image developed in the virtual space 2 has been accepted. For example, when the speaker object is selected by the user 190, the processor 10 accepts the trigger and executes the process of step S2920. Otherwise (NO in step S2910), processor 10 waits until it accepts the trigger.

In step S2920, the processor 10 receives the input of the audio signal corresponding to the utterance of the user 190 from the microphone 119. In step S2930, the processor 10 extracts a character string from the audio signal and transmits the extracted character string to the server 150 as a search key. In another aspect, the processor 10 may be configured to transmit an audio signal to the server 150, and the server 150 may be configured to extract a character string from the audio signal.

In step S2940, the processor 1920 of the server 150 receives the character string as the search key from the computer 200.

According to the above, the user 190 can receive a candidate for a panoramic image desired by the user 190 simply by speaking.

(Search panoramic images in collaboration with other users)
As described above, the user 190 can communicate with other users on the virtual space 2 via the network 19. At this time, the user 190 and other users are viewing the same panoramic image. In one aspect, the user 190 may want to change the panoramic image in consultation with another user. The reason is that the user 190 arbitrarily changes the panoramic image 22 to prevent other users from being surprised. Therefore, the process of changing the panoramic image based on the input of a plurality of users will be described below.

FIG. 30 is a diagram illustrating a network 19 when a plurality of users share a virtual space. In the example shown in FIG. 30, computers 200A, 2 on network 19
00B is connected. The computer 200A is included in the HMD system 1400A and
The computer 200B is included in the HMD system 1400B. HMD system 1400
The configurations of A and the HMD system 1400B are the same as those of the above-mentioned HMD system 1400. Hereinafter, the user of the HMD system 1400A is referred to as the user 190A, and the HMD system 1
Let the user of 400B be the user 190B. Further, A is added to the reference code of each component related to the HMD system 1400A, and B is added to the reference code of each component related to the HMD system 1400B.

In the example shown in FIG. 30, the virtual space 2A provided by the computer 200A.
And the virtual space 2B provided by the computer 200B is composed of the same data. In other words, the computer 200A and the computer 200B share the same virtual space. In the virtual space 2A and the virtual space 2B, there are an avatar object 3000A corresponding to the user 190A and an avatar object 3000B corresponding to the user 190B. The avatar object 30 in the virtual space 2A
The avatar object 3000B in 00A and the virtual space 2B are each equipped with an HMD, but this is for the sake of clarity, and in reality, these objects are not equipped with an HMD.

The computer 200A is a virtual camera 1 at the eye position of the avatar object 3000A.
Place A. As a result, the user 190A shares the field of view of the avatar object 3000A in the virtual space 2A. Computer 200B is an avatar object 30
The virtual camera 1B is placed at the eye position of 00B. As a result, the user 190B shares the field of view of the avatar object 3000B in the virtual space 2B. In virtual space 2A and virtual space 2B, avatar object 3000A and avatar object 300
They face each other with 0B. Therefore, the user 190A visually recognizes the avatar object 3000B corresponding to the user 190B, and the user 190B visually recognizes the avatar object 3000A corresponding to the user 190A. In this state, the user 190A and the user 190B communicate with each other.

For example, when the computer 200B detects that the controller 160B has moved, the computer 200 transmits data representing the movement of the controller 160B via the server 150.
Send to A. The computer 200A uses the received data as the avatar object 300.
Reflect in the hand of 0B. As a result, the user 190A can use the avatar object 3000B.
The movement of the user 190B can be recognized via.

FIG. 31 represents a field of view image 3100 visually recognized by the user 190A. The field of view image 3100 is
It is displayed on the monitor 112A connected to the computer 200A. Field of view image 3100
Includes a desk object 2210, a speaker object 2810 placed on the desk object 2210, an avatar object 3000B, and a thumbnail group 3110.

In one aspect, the user 190B selects the speaker object 2810 and then "
Enter the search key into the computer 200B by saying "animal". Server 150
Thumbnail group 3110 corresponding to a plurality of panoramic images searched based on the search key input from the computer 200B is provided to the computer 200B that receives the search key and the computer 200A that shares the virtual space with the computer 200B. Send.

The computer 200A presents the received thumbnail group 3110 in the virtual space 2A.
As a result, the user 190A visually recognizes the thumbnail group 3110 representing the panoramic image searched by the search key input by the user 190B to the computer 200B.

The user 190A can input a search key for further limiting the plurality of panoramic images represented by the thumbnail group 3110 into the computer 200A. For example, user 190A
Superimposes the pointer object 3120 representing its own gazing point on the speaker object 2810 for a predetermined time, and selects the speaker object 2810. After that, the user 190A inputs the search key into the computer 200A by speaking "cat".

When the server 150 receives the input of another search key from the computer 200A, the server 150 searches for one or more panoramic images related to the cat from among the plurality of panoramic images related to the animal. The server 150 generates thumbnails of each of the searched panoramic images and sends them to the computers 200A and 200B.

When the computer 200A receives one or more new thumbnails from the server 150,
Instead of the currently presented thumbnail group 3110, one or more newly received thumbnails are displayed. The computer 200B also operates in the same manner as the computer 200A.

Either user 190A or 190B selects one thumbnail from one or more thumbnails presented in the virtual space. As a result, panoramic images corresponding to the selected thumbnails are developed in the virtual spaces 2A and 2B.

According to the above, the users 190A and 190B sharing the virtual space can determine the panoramic image to be changed while communicating.

[Constitution]
The technical features disclosed above can be summarized as follows.

(Structure 1) The HMD 110 provides a program executed by the computer 200 to provide the virtual space 2. This program expands one panoramic image included in a plurality of panoramic images into the virtual space 2 on the computer 200 (step S2).
405) and the step (step S) of receiving the input of the trigger for changing the panoramic image developed in the virtual space 2 to another panoramic image from the user 190 of the HMD 110.
2415), the step of accepting the input of the search key from the user 190 after accepting the input of the trigger (step S2420), and specifying one or more panoramic images searched by the search key from the plurality of panoramic images. A step of presenting information for the purpose to the virtual space 2 (step S2460), a step of accepting an input from the user 190 to select one panoramic image from one or more panoramic images (step S2465), and one selected. The step (step S2485) of developing the panoramic image of the above in the virtual space 2 is executed.

In the above example, the process of searching the panoramic image by the server 150 has been described, but in other embodiments, the computer 200 may perform the search. For example, the computer 200 may search for one or more panoramic images based on the search key from the plurality of panoramic images stored in the storage 12.

In the above example, the computer 200 has described the process of presenting the thumbnail generated from the panoramic image to the virtual space 2 as the information for identifying the panoramic image, but as another example, it is associated with the panoramic image. The title and tag may be presented in the virtual space 2.

According to the above, the computer 200 can present to the user 190 one or more panoramic images related to the search key input to the user 190, in other words, the panoramic image (identifying information) desired by the user 190.

(Structure 2) In a certain embodiment, the above program transmits the input search key and the panoramic image ID of one selected panoramic image to the computer 200 to the server 150 capable of communicating with the computer 200. (Step S2430, Step S
2470) and is executed.

(Structure 3) The above search key includes a diagram drawn by the user 190 in the virtual space 2.
According to the above, even if the user 190 does not know or forgets the name of the object of interest, for example, by drawing a diagram of the object in the virtual space, the computer 2
You can enter a search key for the object of interest in 00.

(Structure 4) In one embodiment, the program is on the computer 200 and the user 19
Further execution of the step of detecting the movement of the hand of 0. The step of accepting the input of the search key from the user 190 includes accepting a diagram based on the movement of the hand of the user 190 as the search key.

In one aspect, the computer 200 moves the controller 160 to the user 190.
Detected as the movement of the hand. In another aspect, the computer 200 may photograph the user 190's hand with an infrared camera (not shown) and detect the movement of the user 190's hand based on the imaging result. Such a technique is, for example, L provided by Leap Motion.
It can be realized by eep Motion®. According to the above, user 190
Can easily draw a diagram on the virtual space 2.

(Structure 5) The step of accepting the input of the search key includes accepting a diagram drawn by the user 190 as a search key for the input object arranged in the virtual space 2.

(Structure 6) The step of accepting the input of the above search key is performed by the user 1 in a predetermined space in the virtual space 2 (for example, a space surrounded by the input object 2510).
It includes accepting an input of a three-dimensional object (for example, an airplane 2520) formed by 90, and accepting a diagram obtained by making the three-dimensional object two-dimensional as a search key.

According to the above, the user 190 can input the three-dimensional diagram as a search key into the computer 200. Further, since the server 150 or the computer 200 can perform a search based on a three-dimensional diagram having a larger amount of information than a planar diagram, it is possible to present the panoramic image desired by the user 190 to the user 190. Can enhance sex.

(Structure 7) A program according to an embodiment causes the computer 200 to further perform a step (step S2720) of detecting face tracking data representing the facial expression of the user 190. The search key contains face tracking data for user 190.

In a certain aspect, it is conceivable that the computer 200 arranges a virtual keyboard in the virtual space 2 and accepts the input of the search key from the user 190 based on the virtual keyboard. However, since the virtual keyboard has no tactile sensation, the usability is different from that of the hardware keyboard in the real space. Therefore, the user 190 may not be able to successfully enter the search key on the virtual keyboard. On the other hand, the user 190 can input the search key to the computer 200 simply by making a facial expression.

(Structure 8) The step of accepting the input of the search key includes accepting the input of the voice of the user 190 and accepting the character string extracted from the voice of the user 190 as the search key.

According to the above, the user 190 can input the search key to the computer 200 by simply speaking.

(Structure 9) A program according to an embodiment causes the computer 200 to further perform a step of arranging a speaker object or a microphone object in the virtual space 2.
The step of accepting the input of the trigger includes accepting a predetermined action of the user 190 with respect to the speaker object or the microphone object as a trigger.

In a certain aspect, when the computer 200 comes into contact with a hand object linked to the movement of the user 190's hand and a speaker object or a microphone object, the computer 200 makes contact with the hand object.
Accepts the input of the above trigger. In another aspect, the computer 200 is the user 19
When it is detected that 0 has stared at the speaker object or the microphone object for a predetermined time, the input of the trigger is accepted.

(Structure 10) The step of presenting one or more panoramic images to the virtual space 2 includes presenting a search key or information based on the search key and one or more panoramic images to the virtual space 2.

According to the above, since the user 190 is presented with the search key entered by himself / herself and the panoramic image (thumbnail) searched by the search key, he / she intended from among one or more panoramic images. You can select the panoramic video accurately. As a result, the server 150 or the computer 200 can acquire high-quality learning data and can improve the learning efficiency.

The information based on the search key indicates whether or not the content most likely intended by the user 190 among the plurality of contents (objects, emotions, etc.) searched based on the search key corresponds to the search key. It can be information to contact. By obtaining the response of the user 190 to the inquiry, the server 150 or the computer 200 can acquire high-quality learning data and can improve the learning efficiency.

(Structure 11) In a certain embodiment, the program includes a step of arranging an avatar object 3000B corresponding to the user 190B of the computer 200B in the virtual space 2 on the computer 200A, and a user 190 of the computer 200B from a plurality of panoramic images. The step of presenting one or more panoramic images searched by the search key input to the computer 200B to the virtual space 2 is further executed.

According to the above, the user 190A of the computer 200A can confirm the search result for the search key input by the user 190B who shares the virtual space. As a result, the computer 200A uses the user 19 when the panoramic image is changed unintentionally by the user 190A.
0A can suppress surprises.

(Structure 12) The step of accepting the input of the above search key is the computer 200B.
Includes accepting the input of a search key for one or more panoramic images searched by the search key input by the user 190B of.

According to the above, the users 190A and 190B sharing the virtual space can change the panoramic image while forming a consensus with each other in the process of changing the panoramic image.

(Structure 13) According to a certain embodiment, a computer 200 including a storage 12 storing the program described in any of the above and a processor 10 for executing the program is provided.

(Structure 14) According to certain embodiments, the HMD 110 provides a method performed on the computer 200 to provide the virtual space 2. In this method, one panoramic image included in a plurality of panoramic images is developed in the virtual space 2 (step S2405).
And the step of receiving the input of the trigger for changing the panoramic image developed in the virtual space 2 to another panoramic image from the user 190 of the HMD 110 (step S2415).
), The step of accepting the input of the search key from the user 190 after accepting the input of the trigger (step S2420), and to identify one or more panoramic images searched by the search key from the plurality of panoramic images. (Step S2460), and a step (step S2465) of accepting an input for selecting one panoramic image from one or more panoramic images from the user 190, and one of the selected panoramic images. It includes a step (step S2485) of developing a panoramic image in the virtual space 2.

(Structure 15) In one embodiment, a program executed by a server 150 capable of communicating with a computer 200 that provides the virtual space 2 to the HMD 110 is provided. This program is stored in the panoramic image DB 1934 of the storage 1930 of the server 150 and the step (step S2435) of receiving the input of the search key for changing the panoramic image developed on the monitor 112 of the HMD 110 from the computer 200 in the server 150. The computer 200 provides a step of searching for one or more panoramic images based on a search key (step S2445) and information (for example, thumbnails) for identifying the searched one or more panoramic images from a plurality of panoramic images. (Step S24)
50) and the panoramic image I of one panoramic image selected from one or more panoramic images.
The step of accepting the input of D from the computer 200 (step S2475) and the step of learning for the search based on the identification information of the panoramic image and the search key (step S2490) are executed.

According to the above, the server 150 follows the search key and the user 190 of the computer 200.
Can provide the user 190 with the desired panoramic image candidates. Further, the server 150 learns based on the search key and the information (for example, tag, title) associated with the panoramic image selected by the user 190, so that the user 190 receives the input search key. Can increase the possibility that the desired panoramic image can be provided to the user 190.

(Structure 16) The above search step is to detect a character string from a search key and one or more panoramas in which the detected character string is associated with a plurality of panoramic images stored in the panorama image DB 1934. Includes searching for images.

(Structure 17) The search key includes face tracking data of user 190 of computer 200. The search step described above includes inferring the emotion of the user 190 from the face tracking data of the user 190, and searching for one or more panoramic images based on the inferred emotion.

(Configuration 18) According to an embodiment, a server 150 including a storage 1930 storing the program according to any one of (configuration 15) to (configuration 17) and a processor 1920 for executing the program is provided. NS.

(Structure 19) According to an embodiment, there is provided a method executed by a server 150 capable of communicating with a computer 200 that provides the HMD 110 with a virtual space 2. This method includes a step (step S2435) of receiving input of a search key for changing a panoramic image developed on the monitor 112 of the HMD 110 from the computer 200, and a server 15
A step of searching for one or more panoramic images based on a search key from a plurality of panoramic images stored in the panoramic image DB 1934 of the storage 1930 of 0 (step S24).
45) and information (eg, thumbnails) for identifying one or more searched panoramic images.
To the computer 200 (step S2450), and input the panoramic image ID of one panoramic image selected from one or more panoramic images to the computer 20.
It includes a step of accepting from 0 (step S2475) and a step of learning for searching based on the identification information of the panoramic image and the search key (step S2490).

It should be considered that the embodiments disclosed this time are exemplary in all respects and not restrictive. The scope of the present invention is shown by the scope of claims rather than the above description, and it is intended to include all modifications within the meaning and scope equivalent to the scope of claims.

1 virtual camera, 2 virtual space, 5 reference line of sight, 10,1920 processor, 11
Memory, 12, 1930 storage, 19 networks, 22 panoramic images, 10
0,1400 HMD system, 112 monitors, 114 sensors, 115 speakers,
116 1st camera, 117 2nd camera, 119 microphone, 120 HMD sensor, 1
30 motion sensor, 140 gaze sensor, 150 server, 160 controller, 190 users, 200 computers, 220 display control module, 221 virtual camera control module, 222 view area determination module, 223 view image generation module, 224 reference line-of-sight identification module, 225 Voice control module, 226 line-of-sight detection module, 227 hand tracking module, 228 face tracking module, 230 virtual space control module, 231 virtual space definition module, 23
2 Virtual object generation module, 233 Hand object control module, 240
Memory module, 241 spatial information, 242 object information, 243 user information, 244 face information, 810 hand objects 1120, 2810 speaker objects, 1130 search keys, 1140, 1210, 1260, 2320, 2350 thumbnails, 1200, 2200, 2300, 2500 , 2800, 3100 view image,
1923 search engine, 1924 object identification part, 1925 emotion judgment part, 1926 learning part, 1934 panoramic image DB, 1935 learning DB, 1936, 1937 learning model, 2220, 2510 input object, 2230 pen object, 224
0 Message object, 2250 Trajectory object, 2310 screen, 2
520 airplane, 2820, 3120 pointer object, 3000A, 3000B
Avatar object.

Claims (13)

A program that is executed on a computer to provide a virtual space shared by a plurality of users.
A step of developing one panoramic image included in a plurality of predetermined panoramic images in the virtual space, and
A step of arranging a first avatar object corresponding to the first user and a second avatar object corresponding to the second user in the virtual space, and
A step of accepting an input of a trigger for changing a panoramic image developed in the virtual space to another panoramic image from the first user, and
After accepting the input of the trigger, the step of accepting the input of the search key from the first user, and
A step of presenting a display object including information for identifying one or more panoramic images searched by the search key from the plurality of panoramic images in the virtual space shared by the plurality of users. ,
The movement of the first avatar object moved by the first user and the movement of the second user with respect to the information for identifying the searched one or more panoramic images included in the display object. A step of accepting an input for selecting one panoramic image from the one or more panoramic images in response to the movement of the second avatar object.
A program that executes a step of developing the selected panoramic image in the virtual space. The program causes the computer to execute a step of transmitting the input search key and the identification information of the selected panoramic image to an information processing terminal capable of communicating with the computer. The program described in. The program according to claim 1 or 2, wherein the search key includes a diagram drawn by the user in the virtual space. The program causes the computer to further perform a step of detecting the movement of the user's hand.
The program according to claim 3, wherein the step of accepting the input of the search key from the user includes accepting a diagram based on the movement of the user's hand as the search key. The step according to claim 3 or 4, wherein the step of accepting the input of the search key includes accepting a diagram drawn by the user for a predetermined object arranged in the virtual space as the search key. program. Input accepting step of the search key is
Accepting the input of the three-dimensional object formed by the user in the predetermined space in the virtual space,
The program according to claim 3 or 4, wherein a diagram obtained by converting the three-dimensional object into two dimensions is accepted as the search key. The program causes the computer to further perform a step of detecting data representing the user's facial expression.
The program according to claim 1 or 2, wherein the search key includes data representing the facial expression of the user. The step of accepting the input of the search key is
Accepting the user's voice input
The program according to claim 1 or 2, comprising accepting a character string extracted from the user's voice as the search key. The program causes the computer to further perform a step of placing a speaker object or a microphone object in the virtual space.
The program according to claim 8, wherein the step of accepting the input of the trigger includes accepting a predetermined action of the user with respect to the speaker object or the microphone object as the trigger. The step of presenting the one or more panoramic images to the virtual space includes presenting the search key or information based on the search key and the one or more panoramic images to the virtual space. The program according to any one of 9. The program according to claim 10 , wherein the step of accepting the input of the search key includes accepting the input of the search key for one or more panoramic images searched by the search key input by the user of the other computer. A memory in which the program according to any one of claims 1 to 11 is stored, and
An information processing device including a processor for executing the program. A method performed on a computer to provide a virtual space shared by multiple users.
A step of developing one panoramic image included in a plurality of predetermined panoramic images in the virtual space, and
A step of arranging a first avatar object corresponding to the first user and a second avatar object corresponding to the second user in the virtual space, and
A step of accepting an input of a trigger for changing a panoramic image developed in the virtual space to another panoramic image from the first user, and
After accepting the input of the trigger, the step of accepting the input of the search key from the first user, and
A step of presenting a display object including information for identifying one or more panoramic images searched by the search key from the plurality of panoramic images in the virtual space shared by the plurality of users. ,
The movement of the first avatar object moved by the first user and the movement of the second user with respect to the information for identifying the searched one or more panoramic images included in the display object. A step of accepting an input for selecting one panoramic image from the one or more panoramic images in response to the movement of the second avatar object.
A step of expanding the selected panoramic image into the virtual space,
A method comprising the steps of transmitting the input search key and the identification information of the selected panoramic image to an information processing terminal capable of communicating with the computer.

Images