JP6461850B2 - Simulation system and program - Google Patents

Description

  The present invention relates to a simulation system and a program.

  Conventionally, using a plurality of surround speakers such as 5.1ch to create a realistic sound field that simulates the sound field of the real world, you can enjoy games, movies, etc. Simulation systems are known. As a conventional technique of such a simulation system, for example, there is a technique disclosed in Patent Document 1.

JP-A-8-243256

  However, in a conventional simulation system that realizes surround sound using a plurality of speakers, it is assumed that the user is positioned at a predetermined reference position in a predetermined reference direction with respect to the arrangement positions of the plurality of speakers. The sound field was formed. Therefore, when the user moves in real space and the user's position and direction change, there is a problem that an appropriate sound field such as surround cannot be maintained.

  According to some aspects of the present invention, it is possible to provide a simulation system, a program, and the like that can form an appropriate sound field even when the position or direction of a user changes.

  One aspect of the present invention is an input processing unit that acquires information on a position and a direction of a user who moves in a real space in which first to Mth speakers (M is an integer of 3 or more) are arranged; When the position and direction of the virtual user corresponding to the user change as the user's position and direction change, the virtual user follows the virtual user while maintaining a predetermined relative positional relationship. The first to Nth virtual speakers are set by adjusting the volume balance of the first to Mth speakers for the first to Nth positions (N is an integer of 2 or more), and the virtual The present invention relates to a simulation system including a sound processing unit that performs a sound field forming process using a virtual sound source in space using the first to Nth virtual speakers. The present invention also relates to a program that causes a computer to function as each of the above-described units, or a computer-readable information storage medium that stores the program.

  According to one aspect of the present invention, information on the position and direction of a user who moves in a real space in which the first to Mth speakers are arranged is acquired. With respect to the first to Nth positions that follow the virtual user while maintaining a predetermined relative positional relationship even when the position and direction of the virtual user change with changes in the position and direction of the user. First to Nth virtual speakers are set. These first to Nth virtual speakers are set by adjusting the volume balance of the first to Mth speakers arranged in the real space. And the formation process of the sound field by the virtual sound source in virtual space is performed using the set 1st-Nth virtual speaker. As described above, according to one aspect of the present invention, even when the position and direction of a virtual user change in accordance with the change in the position and direction of the user, the first to Nth virtual speakers are predetermined with respect to the virtual user. Following relative position is maintained. The sound field of the virtual sound source in the virtual space is formed using the first to Nth virtual speakers that follow the virtual user while maintaining a predetermined relative positional relationship. Therefore, it is possible to provide a simulation system or the like that can form an appropriate sound field even when the position or direction of the user changes.

  In the aspect of the invention, the sound processing unit may include information on a position of a first speaker group of the first to Mth speakers and a first position of the first to Nth positions. The first virtual speaker among the first to Nth virtual speakers is set by adjusting the volume balance of the first speaker group based on the information of the first to Mth speakers. By adjusting the volume balance of the second speaker group based on the position information of the second speaker group of the speakers and the information of the second position of the first to Nth positions. A second virtual speaker among the first to Nth virtual speakers may be set.

  In this way, each virtual speaker can be set by adjusting the volume balance of the speaker group based on the position information of the speaker group corresponding to each virtual speaker and the position information of each virtual speaker. become.

  In the aspect of the invention, the sound processing unit causes the first speaker group including at least three speakers to output the sound of the first channel, and includes the second speaker including at least three speakers. The sound of the second channel may be output to the speaker group.

  Thus, if each virtual speaker is set using a speaker group including at least three speakers, the virtual speaker can be set to an arbitrary position.

  In the aspect of the invention, the sound processing unit may adjust the volume balance of the first speaker group so that the first virtual speaker is set at a position corresponding to the left ear of the virtual user. The volume balance of the second speaker group may be adjusted so that the second virtual speaker is set at a position corresponding to the right ear of the virtual user.

  In this way, it is possible to achieve an acoustic effect as if wearing virtual headphones, for example.

  In the aspect of the invention, the sound processing unit may cause the first and second speaker groups to output a sound from a binaural sound source.

  In this way, by making the first and second speaker groups output the sound from the binaural sound source, it is possible to realize more realistic and realistic stereoscopic sound.

  In the aspect of the invention, the sound processing unit may include information on a position of a first speaker group of the first to Mth speakers and a first position of the first to Nth positions. The first virtual speaker among the first to Nth virtual speakers is set by adjusting the volume balance of the first speaker group based on the information of the first to Mth speakers. By adjusting the volume balance of the second speaker group based on the position information of the second speaker group of the speakers and the information of the second position of the first to Nth positions. , Setting a second virtual speaker among the first to Nth virtual speakers, and information on a position of a third speaker group among the first to Mth speakers and the first to Nth virtual speakers. Based on the information of the third position among the positions, the volume balance of the third speaker group By performing an integer, it may be set a third virtual speaker of the virtual speaker of the first to N.

  In this way, the first, second, and third virtual speakers can be set by adjusting the volume balance of the first, second, and third speaker groups of the first to Mth speakers. Become. Even if the position and direction of the virtual user change as the user position and direction change, these first, second, and third virtual speakers follow the virtual user while maintaining a predetermined relative positional relationship. As a result, a more appropriate surround effect or the like can be realized.

  In the aspect of the invention, the sound processing unit may be configured based on the first to Nth position information of the first to Nth virtual speakers and the position information of the virtual sound source. The sound field forming process by the virtual sound source may be performed by adjusting the volume balance of the Nth virtual speaker.

  In this way, the sound field by the virtual sound source in the virtual space is appropriately formed by adjusting the volume balance of the first to Mth virtual speakers that follow the virtual user while maintaining a predetermined relative positional relationship. It becomes possible to do.

  In one aspect of the present invention, a movable range may be set for the user in the real space, and the first to Mth speakers may be arranged at positions corresponding to the movable range.

  In this way, even when the user moves within the movable range set in the real space and the position or direction changes, an appropriate sound field that does not depend on the change in the position or direction can be formed. Is possible.

  Moreover, in one aspect of the present invention, a display processing unit that performs a display process of an image seen from the viewpoint of the virtual user corresponding to the user may be included.

  In this way, when the user's position and direction change, the virtual user's position and direction change, so that the image that can be seen from the virtual user's viewpoint also changes. Will be displayed. Even when the image changes in this way, a sound field of an appropriate virtual sound source can be formed using a virtual speaker, so that the virtual reality of the user can be improved.

  In the aspect of the invention, the display processing unit may perform an image display process from a first-person viewpoint where the virtual user is not displayed.

  In this way, the user can obtain a virtual reality as if he / she became a virtual user. And since the sound field by a virtual sound source can be appropriately formed using a virtual speaker, it can prevent that the said virtual reality is impaired.

  In the aspect of the invention, the display processing unit may perform processing for displaying an image on the head-mounted display device worn by the user.

  In this way, even when the user's field of view is covered by the head-mounted display device, the sound field by the virtual sound source can be appropriately formed using the virtual speaker, so that the virtual reality of the user can be improved. It can be improved.

  In one aspect of the present invention, the input processing unit is based on information from a sensor unit provided in the head-mounted display device when the user moves while wearing the head-mounted display device. Then, information on the position and direction of the user may be acquired.

  If it does in this way, based on the information from the sensor part provided in the head-mounted display device, the information on the user's position and direction is acquired, and the change of the user's position and direction causes the virtual user's position to change. The position and direction can be changed.

  In the aspect of the invention, the display processing unit may perform an image display process for reproducing a sound generation state by the virtual sound source in the virtual space.

  In this way, it is possible to appropriately form the sound field of the virtual sound source while reproducing the sound generation state of the virtual sound source in the virtual space with the image displayed to the user, Reality can be further improved.

  In one aspect of the present invention, the sound processing unit detects an approach of the user to an i-th speaker (1 ≦ i <M) among the first to M-th speakers in the real space. In this case, a process of adjusting at least one of the volume of the sound output from the i-th speaker, the high frequency component, and the reverberation may be performed.

  In this way, when the user approaches a real-world speaker, the presence of the speaker is recognized by the user, and the virtual reality of the user can be prevented from being impaired.

  In the aspect of the invention, the sound processing unit may invalidate the settings of the first to Nth virtual speakers when it is determined that a predetermined situation has occurred.

  If the virtual speaker setting is invalidated in this way, it becomes possible for the user to grasp the position of the speaker in the real space, and it becomes possible to deal with the occurrence of a predetermined situation such as the occurrence of an emergency.

The block diagram which shows the structural example of the simulation system of this embodiment. FIG. 2A and FIG. 2B are examples of the HMD used in this embodiment. 3A and 3B are other examples of the HMD used in the present embodiment. Explanatory drawing of the private room which is a play area. Explanatory drawing of the private room which is a play area. The example of the game image produced | generated by this embodiment. The example of the game image produced | generated by this embodiment. Explanatory drawing about the movement of the user (virtual user) on a stage. FIG. 9A and FIG. 9B are explanatory diagrams about the problem of sound field formation when the position of the user changes in real space. Explanatory drawing about the problem of sound field formation when a user's direction changes in real space. FIG. 11A and FIG. 11B are explanatory diagrams of a stereo virtual speaker setting method. Explanatory drawing about the method of setting the virtual speaker of FIG. 11 (A) and FIG. 11 (B) by the volume balance adjustment of a speaker group. FIG. 13A and FIG. 13B are explanatory diagrams of a method for setting a virtual speaker at a position corresponding to the ear of the virtual user. Explanatory drawing about the method of setting the virtual speaker of FIG. 13 (A) and FIG. 13 (B) by the volume balance adjustment of a speaker group. FIG. 15A and FIG. 15B are explanatory diagrams of a technique using a binaural sound source. FIG. 16A and FIG. 16B are explanatory diagrams of a technique for causing a virtual user to follow a surround virtual speaker. Explanatory drawing about the virtual sound source in virtual space. Explanatory drawing of the method of forming the sound field by a virtual sound source by volume balance adjustment of a virtual speaker. Explanatory drawing about the speaker arrange | positioned corresponding to the user's movable range with which HMD was mounted | worn, and the movable range. Explanatory drawing of the specific example of a virtual sound source. FIG. 21A and FIG. 21B are diagrams for explaining an example of a sound generation state of a virtual sound source. 22A and 22B are diagrams for explaining another example of the sound generation state of the virtual sound source. Explanatory drawing of the process performed when a user approaches a speaker. The flowchart which shows the detailed process example of this embodiment.

  Hereinafter, this embodiment will be described. In addition, this embodiment demonstrated below does not unduly limit the content of this invention described in the claim. In addition, all the configurations described in the present embodiment are not necessarily essential configuration requirements of the present invention.

1. Configuration FIG. 1 shows a configuration example of a simulation system (game system, video display system, simulation apparatus) of the present embodiment. Note that the simulation system of the present embodiment is not limited to the configuration shown in FIG. 1, and various modifications such as omitting some of the components (each unit) or adding other components are possible.

  The input device 160 is a device for the user to input various input information. The input device 160 can include a sound input device 161 and a vibration device 164. The input device 160 may have a game controller function for the user to input game operation information. The game controller is realized by, for example, an operation button, a direction instruction key, a joystick or a lever. In this case, the game controller and the sound input device 161 may be realized by an integral casing or may be realized by a separate casing.

  The sound input device 161 is a device for the user to input sound information. The sound input device 161 can input user voices such as a user's singing voice, calling voice, and shouting voice. The sound input device 161 can be realized by, for example, a microphone 162 described with reference to FIG. Note that the shape of the sound input device 161 is not limited to the microphone 162 having the shape shown in FIG. 2A, and various types of devices such as a headset type microphone having a headband and a small microphone can be used. . The sound input device 161 may be a sound input device (pickup microphone or the like) in a musical instrument or a device imitating a musical instrument. Examples of musical instruments include stringed instruments (guitar), percussion instruments (drums, drums), and keyboard instruments (piano, keyboard).

  The vibration device 164 (vibration generating unit) is a device that generates vibration for warning or the like, and is realized by, for example, a vibration motor (vibrator). For example, the vibration motor generates vibration by rotating an eccentric weight. Specifically, eccentric weights are attached to both ends of the drive shaft so that the motor itself swings. The vibration device 164 is not limited to a vibration motor, and may be realized by, for example, a piezoelectric element.

  The storage unit 170 stores various types of information. The storage unit 170 functions as a work area such as the processing unit 100 or the communication unit 196. The game program and game data necessary for executing the game program are held in the storage unit 170. The function of the storage unit 170 can be realized by a semiconductor memory (DRAM, VRAM), HDD (hard disk drive), SDD, optical disk device, or the like. The storage unit 170 includes a spatial information storage unit 172, a music information storage unit 174, a sound data storage unit 175, a parameter storage unit 176, and a drawing buffer 178.

  The information storage medium 180 (a computer-readable medium) stores programs, data, and the like, and its function can be realized by an optical disk (DVD, BD, CD), HDD, semiconductor memory (ROM), or the like. . The processing unit 100 performs various processes of the present embodiment based on a program (data) stored in the information storage medium 180. That is, in the information storage medium 180, a program for causing a computer (an apparatus including an input device, a processing unit, a storage unit, and an output unit) to function as each unit of the present embodiment (a program for causing the computer to execute processing of each unit). Is memorized.

  The head-mounted display device 200 (HMD) is a device that is mounted on the user's head and displays an image in front of the user's eyes. The HMD 200 is preferably a non-transmissive type, but may be a transmissive type. The HMD 200 may be a so-called glasses-type HMD.

  The HMD 200 includes a sensor unit 210, a display unit 220, and a processing unit 240. A modification in which a light emitting element is provided in the HMD 200 is also possible. The sensor unit 210 is for realizing tracking processing such as head tracking, for example. For example, the position and direction of the HMD 200 are specified by tracking processing using the sensor unit 210. By specifying the position and direction of the HMD 200, the position and direction of the user can be specified. The position and direction of the virtual user (virtual player) in the virtual space corresponding to the user (player) are specified by the position and direction of the user. The user's position and direction are, for example, the user's viewpoint position and line-of-sight direction. The position and direction of the virtual user are, for example, the viewpoint position and line-of-sight direction of the virtual user.

  Various methods can be adopted as the tracking method. In the first tracking method, which is an example of the tracking method, a plurality of light receiving elements (photodiodes and the like) are provided as the sensor unit 210, as will be described in detail with reference to FIGS. 2A and 2B described later. Then, by receiving light (laser or the like) from a light emitting element (LED or the like) provided outside by the plurality of light receiving elements, the position of the HMD 200 (user's head) in the real world three-dimensional space, In the second tracking method for specifying the direction, a plurality of light emitting elements (LEDs) are provided in the HMD 200, as will be described in detail with reference to FIGS. 3A and 3B described later. And the position and direction of HMD200 are pinpointed by imaging the light from these light emitting elements with the imaging part provided outside. In the third tracking method, a motion sensor is provided as the sensor unit 210, and the position and direction of the HMD 200 are specified using this motion sensor. The motion sensor can be realized by, for example, an acceleration sensor or a gyro sensor. For example, by using a 6-axis motion sensor using a 3-axis acceleration sensor and a 3-axis gyro sensor, the position and direction of the HMD 200 in a three-dimensional space in the real world can be specified. Note that the position and direction of the HMD 200 may be specified by a combination of the first tracking method and the second tracking method, or a combination of the first tracking method and the third tracking method.

  The display unit 220 of the HMD 200 can be realized by, for example, a liquid crystal display (LCD) or an organic EL display. For example, the HMD 200 is provided with a first display arranged in front of the user's left eye and a second display arranged in front of the right eye as the display unit 220. For example, stereoscopic display is possible. ing. When performing stereoscopic display, for example, a left-eye image and a right-eye image with different parallaxes are generated, the left-eye image is displayed on the first display, and the right-eye image is displayed on the second display.

  The processing unit 240 of the HMD 200 performs various processes necessary for the HMD 200. For example, the processing unit 240 performs control processing of the sensor unit 210, display control processing of the display unit 220, and the like. Further, the processing unit 240 may perform a three-dimensional sound (stereoscopic sound) process to realize reproduction of a three-dimensional sound direction, distance, and spread.

  The sound output unit 192 outputs the sound generated by the present embodiment, and can be realized by, for example, a speaker or headphones.

  The I / F (interface) unit 194 performs interface processing with the portable information storage medium 195, and the function can be realized by an ASIC for I / F processing or the like. The portable information storage medium 195 is for a user to save various types of information, and is a storage device that retains storage of such information even when power is not supplied. The portable information storage medium 195 can be realized by an IC card (memory card), a USB memory, a magnetic card, or the like.

  The communication unit 196 communicates with the outside (another apparatus) via a wired or wireless network, and functions thereof are hardware such as a communication ASIC or communication processor, or communication firmware. Can be realized.

  Note that a program (data) for causing a computer to function as each unit of the present embodiment is obtained from an information storage medium of a server (host device) via an information storage medium 180 (or storage unit 170, auxiliary storage) via a network and communication unit 196. May be distributed to the device 194). Use of an information storage medium by such a server (host device) can also be included in the scope of the present invention.

  The processing unit 100 (processor) performs game processing, game results calculation processing based on input information from the input device 160, tracking information (HMD position, direction, or viewpoint position, line-of-sight direction) in the HMD 200, a program, and the like. Display processing or sound processing.

  Each processing (each function) of this embodiment performed by each unit of the processing unit 100 can be realized by a processor (a processor including hardware). For example, each process of the present embodiment can be realized by a processor that operates based on information such as a program and a memory that stores information such as a program. In the processor, for example, the function of each unit may be realized by individual hardware, or the function of each unit may be realized by integrated hardware. The processor may be, for example, a CPU (Central Processing Unit). However, the processor is not limited to the CPU, and various processors such as a GPU (Graphics Processing Unit) or a DSP (Digital Processing Unit) can be used. The processor may be an ASIC hardware circuit. The memory (storage unit 170) may be a semiconductor memory such as SRAM or DRAM, or may be a register. Alternatively, it may be a magnetic storage device such as a hard disk device (HDD) or an optical storage device such as an optical disk device. For example, the memory stores instructions that can be read by a computer, and the processing (function) of each unit of the processing unit 100 is realized by executing the instructions by the processor. The instruction here may be an instruction set constituting a program, or an instruction for instructing an operation to the hardware circuit of the processor.

  The processing unit 100 includes an input processing unit 102, an arithmetic processing unit 110, and an output processing unit 140. The arithmetic processing unit 110 includes a game processing unit 111, a game result calculation unit 118, a display processing unit 120, and a sound processing unit 130. As described above, each process of the present embodiment executed by these units can be realized by a processor (or a processor and a memory). Various modifications such as omitting some of these components (each unit) or adding other components are possible.

  The input processing unit 102 (input processing program module) performs, as input processing, processing for receiving input information and tracking information, processing for reading information from the storage unit 170, and processing for receiving information via the communication unit 196. . For example, the input processing unit 102 obtains input information input by the user using the input device 160 and tracking information (such as user position, direction, or line-of-sight information) detected by the sensor unit 210 of the HMD 200, A process of reading information specified by the read command from the storage unit 170 and a process of receiving information from an external device (such as a server) via a network are performed as input processes. Here, the reception process includes a process of instructing the communication unit 196 to receive information, a process of acquiring information received by the communication unit 196, and writing the information in the storage unit 170, and the like.

  The arithmetic processing unit 110 performs various arithmetic processes. For example, arithmetic processing such as game processing, game results calculation processing, display processing, or sound processing is performed.

  The game processing unit 111 (game processing program module) performs various game processes for the user to play the game. The game processing unit 111 includes a game progress processing unit 112, an evaluation processing unit 113, a character processing unit 114, a parameter processing unit 115, an object space setting unit 116, and a virtual camera control unit 117.

  The game progress processing unit 112 performs a process of starting a game when the game start condition is satisfied, a process of progressing the game, or a process of ending the game when the game end condition is satisfied. The evaluation processing unit 113 performs an evaluation process for the user's game play. For example, a user's performance in a music game and an evaluation process for game operation are performed. Music information used in the music game is stored in the music information storage unit 174.

  The character processing unit 114 performs various processes related to the character. For example, a process of moving a character in an object space (virtual space, game space) or a process of moving a character is performed. For example, the process of moving the character can be realized by motion processing (motion reproduction or the like) using motion data. The parameter processing unit 115 performs calculation processing of various parameters (game parameters) used in the game. For example, a process of increasing or decreasing the parameter value is performed. The parameter information is stored in the parameter storage unit 176.

  The object space setting unit 116 performs processing for setting an object space (a virtual space in a broad sense) in which a plurality of objects are arranged. For example, various objects (polygons, free-form surfaces, subdivision surfaces, etc.) representing display objects such as characters (people, animals, robots, etc.), maps (terrain), buildings, auditoriums, courses (roads), trees, walls, water surfaces, etc. The object is arranged and set in the object space. In other words, the position and rotation angle of the object in the world coordinate system (synonymous with direction and direction) are determined, and the rotation angle (rotation angle around the X, Y, and Z axes) is determined at that position (X, Y, Z). Arrange objects. Specifically, the spatial information storage unit 172 of the storage unit 170 stores information such as the positions and rotation angles (directions) of a plurality of objects (part objects) in the object space as spatial information. The object space setting unit 116 performs a process of updating the space information for each frame, for example.

  The virtual camera control unit 117 performs control processing of a virtual camera (viewpoint, reference virtual camera) for generating an image that can be seen from a given (arbitrary) viewpoint in the object space. Specifically, processing for controlling the position (X, Y, Z) or rotation angle (rotation angle about the X, Y, Z axis) of the virtual camera (processing for controlling the viewpoint position, the line-of-sight direction or the angle of view) I do. This virtual camera corresponds to the viewpoint of the user. In the case of stereoscopic display, a first viewpoint for the left eye (first virtual camera for the left eye) and a second viewpoint for the right eye (second virtual camera for the right eye) are set.

  The game score calculation unit 118 performs processing for calculating the user's game score. For example, calculation processing of game results such as scores and points obtained by the user's game play is performed.

  The display processing unit 120 (display processing program module) performs display processing of a game image. For example, a drawing process is performed based on the results of various processes (game process, simulation process) performed by the processing unit 100, thereby generating an image and displaying it on the display unit 220 of the HMD 200. Specifically, geometric processing such as coordinate transformation (world coordinate transformation, camera coordinate transformation), clipping processing, perspective transformation, or light source processing is performed. Based on the processing result, drawing data (the position of the vertex of the primitive surface) Coordinates, texture coordinates, color data, normal vector, α value, etc.) are created. Based on the drawing data (primitive surface data), the object (one or a plurality of primitive surfaces) after perspective transformation (after geometry processing) is converted into image information in units of pixels such as a drawing buffer 178 (frame buffer, work buffer, etc.). Draw in a buffer that can be stored. As a result, an image that can be seen from the virtual camera (given viewpoint, left eye and right eye first and second viewpoints) in the object space is generated. Note that the drawing processing performed by the display processing unit 120 can be realized by vertex shader processing, pixel shader processing, or the like.

  The sound processing unit 130 (sound processing program module) performs sound processing based on the results of various processes performed by the processing unit 100. Specifically, game sounds such as music (music, BGM), sound effects, or sounds are generated, and the game sounds are output to the sound output unit 192. Sound data output (reproduced) during the game is stored in the sound data storage unit 175. Note that part of the sound processing of the sound processing unit 130 (for example, three-dimensional sound processing) may be realized by the processing unit 240 of the HMD 200.

  The output processing unit 140 performs various types of information output processing. For example, the output processing unit 140 performs processing for writing information in the storage unit 170 and processing for transmitting information via the communication unit 196 as output processing. For example, the output processing unit 140 performs a process of writing information specified by a write command in the storage unit 170 or a process of transmitting information to an external apparatus (server or the like) via a network. The transmission process is a process of instructing the communication unit 196 to transmit information, or instructing the communication unit 196 to transmit information.

  For example, in the present embodiment, the game processing unit 111 performs processing of a game played by the user in a virtual space (game space) where a plurality of objects are arranged. For example, a plurality of objects such as characters are arranged in a virtual space that is an object space, and the game processing unit 111 performs various game processes (game progress process, character process) for realizing a game in the virtual space. , An object space setting process, or a virtual camera control process). Then, the display processing unit 120 displays the game image seen from a given viewpoint (first and second viewpoints for left eye and right eye) in the virtual space on the display unit 220 (first and second displays) of the HMD 200. Process to display. That is, in the object space that is a virtual space, a process of displaying a game image that can be seen from the viewpoint (virtual camera) of the virtual user (user) is performed.

  For example, when a user in the real world wearing the HMD 200 moves in the play area of the private room shown in FIGS. 4 and 5 described later, the movement direction changes, the head is shaken, and the squat is crouched. When it changes, the position and direction of the virtual user in the virtual space corresponding to the user also change. Since the position and direction of the user in the real world can be specified by the tracking process of the HMD 200, the position and direction of the virtual user in the virtual space can also be specified. The position and direction of the user and the virtual user are, for example, the viewpoint position and the line-of-sight direction of the user and the virtual user. When the virtual user is not displayed as a character, the display image of the HMD 200 is a first-person viewpoint image, and when the virtual user is displayed as a character, the display image is a third-person viewpoint image.

  The play area (action area) is an area that is set as a range in which the user can move, for example, and is an area that is defined in advance as an area (field, space) in which the user performs an action such as game play. . This play area (movable range in a broad sense) is an area including a range in which, for example, user position information can be tracked. The play area may be, for example, an area surrounded by walls, but may be an open space area.

  In this embodiment, the input processing unit 102 acquires information on the position and direction of the user who moves in the real space. For example, information on the position (viewpoint position) and direction (line-of-sight direction, movement direction) of the user specified by the tracking process of the HMD 200 is acquired. First to Mth speakers (M is an integer of 3 or more) speakers (real speakers) are arranged in the real space (the range in which the user can move). For example, three or more speakers are arranged so as to surround the user. These speakers may be speakers installed on a wall or ceiling of a private room that is a play area, or may be speakers that are supported by a stand in an open area play area.

  The sound processing unit 130 performs a virtual speaker setting process, and performs a sound field forming process in a virtual space using the set virtual speaker. The virtual speaker setting process is performed by the virtual speaker setting unit 132, and the sound field forming process is performed by the sound field forming process unit 134. Specifically, the sound processing unit 130 (virtual speaker setting unit) is a virtual corresponding to the user in accordance with a change in the position and direction (viewpoint position, line-of-sight direction) of the user in the real space (play area, action area). Even when the position and direction (viewpoint position, line-of-sight direction) of the user in the virtual space (object space) change, the first to Nth (N Is a process of setting the first to Nth virtual speakers for a position of 2 or more. For example, the first to Nth virtual speakers are set by adjusting the volume balance of the first to Mth speakers arranged in the real space. The sound processing unit 130 (sound field forming processing unit) performs sound field forming processing using a virtual sound source in the virtual space using the first to Nth virtual speakers. For example, by adjusting the volume balance of the set first to Nth virtual speakers and outputting virtual sounds from these first to Nth virtual speakers, it is set at an arbitrary position in the virtual space The sound field is formed by the virtual sound source. For example, a sound field forming process is performed such that the sound from the virtual sound source is localized at the position of the virtual sound source.

  The virtual speaker is not a speaker that actually exists, but a virtual speaker that is set in a virtual space by computer calculation processing. The virtual speaker is realized by adjusting the volume balance of a plurality of speakers in the real space (real world). In the present embodiment, the relative positional relationship between the virtual speaker and the virtual user is determined in advance, for example. This relative positional relationship is a relative relationship between at least one of the position and direction (line-of-sight position and line-of-sight direction) of the virtual user and the position of the virtual speaker. For example, in the case of stereo, two virtual speakers are set on the line connecting the left and right ears of the virtual user. In the case of surround, each virtual speaker of a predetermined number of virtual speakers is arranged at a plurality of predetermined positions around the virtual user. For example, virtual speakers are set in a virtual user's left diagonal forward direction, right diagonal forward direction, left diagonal rear direction, right diagonal rear direction, and the like. Alternatively, in addition to these, a virtual speaker is set in the front direction or directly behind the virtual user. Since the virtual speakers are realized by computer arithmetic processing, the set number is an arbitrary number of 2 or more, and the set position is also arbitrary.

  The virtual sound source is a virtual sound source realized by computer arithmetic processing as a sound generation source in a virtual space, for example. The sound of the virtual sound source is, for example, a sound generated from an object appearing in the virtual space. For example, the sound of the virtual sound source is voice (chance, shout, singing voice) or performance sound (performance sound of an instrument such as a guitar, drum, or piano) appearing in a virtual space (game space, object space). The sound of the virtual sound source may be an operation sound that can be heard from a moving object such as a car, an airplane, or a robot appearing in the virtual space, or may be a sound effect such as an explosive sound generated by an explosive in the virtual space. Good. In this embodiment, these virtual sound sources are simulated by computer arithmetic processing. And the sound field by a virtual sound source is formed of a virtual speaker. For example, by adjusting the volume balance of a plurality of virtual speakers, a sound field by a virtual sound source is formed. Information on the position and direction of the virtual sound source is stored in the spatial information storage unit 172.

  In addition, the sound processing unit 130 is based on the information on the position of the first speaker group among the first to Mth speakers and the information on the first position among the first to Nth positions. By adjusting the volume balance of one speaker group, the first virtual speaker among the first to Nth virtual speakers is set (configured). The volume of the second speaker group based on the position information of the second speaker group of the first to Mth speakers and the information of the second position of the first to Nth positions. By performing the balance adjustment, the second virtual speaker among the first to Nth virtual speakers is set (configured).

  Here, each of the first speaker group and the second speaker group includes a plurality of speakers among the first to Mth speakers. There may be overlapping speakers between the plurality of speakers constituting the first speaker group and the plurality of speakers constituting the second speaker group.

  The first virtual speaker is realized by adjusting the volume balance of the first speaker group. For example, if the volume of the m-th speaker in the first speaker group is increased, the first virtual speaker approaches the position corresponding to the m-th speaker, and if the volume of the n-th speaker is increased, the first virtual speaker is increased. The 1 virtual speaker approaches a position corresponding to the nth speaker. Then, how to adjust the volume balance of each speaker of the first speaker group is determined by information on the position of each speaker in the first speaker group and information on the first position where the first virtual speaker is set. Based on the above.

  Similarly, the second virtual speaker is realized by adjusting the volume balance of the second speaker group. Then, how to adjust the volume balance of each speaker of the second speaker group depends on the information on the position of each speaker in the second speaker group and the information on the second position where the second virtual speaker is set. Based on the above.

  The sound processing unit 130 causes the first speaker group including at least three speakers to output the sound of the first channel, and causes the second speaker group including at least three speakers to output the second channel. Output sound. The sound of the first channel is, for example, the sound of the left channel, and the sound of the second channel is, for example, the sound of the right channel. In this way, the sound of the first channel is virtually output from the first virtual speaker set by the volume balance adjustment of the first speaker group, and is set by the volume balance adjustment of the second speaker group. The second channel sound is virtually output from the second virtual speaker. Then, by using the first and second speaker groups each including at least three speakers, the first and second virtual speakers can be set at arbitrary positions. For example, when the first channel is the left channel and the second channel is the right channel, at an arbitrary position on the line connecting the position corresponding to the left ear of the virtual user and the position corresponding to the right ear, It becomes possible to set the first and second virtual speakers. The first and second channels may be arbitrary channels in surround, for example.

  For example, the sound processing unit 130 adjusts the volume balance of the first speaker group so that the first virtual speaker is set at a position corresponding to the left ear of the virtual user, and a position corresponding to the right ear of the virtual user. The volume balance of the second speaker group may be adjusted so that the second virtual speaker is set at the same time. In this way, it is possible to make the user listen to the sound as if wearing headphones.

  Further, the sound processing unit 130 may cause the first and second speaker groups to output a sound from a binaural sound source. In this way, the sound that reaches the eardrum of the ear is virtually output from the first and second virtual speakers on the left and right, so that you can feel as if you were there. A certain sound can be heard. The sound produced by the binaural sound source may be a sound virtually simulated by the computer arithmetic processing of the sound of the binaural sound source, or may be a sound including a sound obtained by binaural recording. For example, in addition to the sound waves that directly reach the left and right ears from the virtual sound source, a sound simulation process is performed in which sound waves that are complicatedly diffracted and reflected by each part of the earlobe and body are superimposed and output from the virtual speaker.

  In addition, the sound processing unit 130 adjusts the volume balance of the first speaker group based on the position information of the first speaker group and the information of the first position as described above, so that the first virtual group is adjusted. While setting a speaker and adjusting the volume balance of the second speaker group based on the information on the position of the second speaker group and the information on the second position, the second virtual speaker is set, Volume balance adjustment of the third speaker group based on the position information of the third speaker group of the first to Mth speakers and the information of the third position of the first to Nth positions. By performing the above, a third virtual speaker among the first to Nth virtual speakers may be set. In this way, at least three virtual speakers such as the first, second, and third virtual speakers can be set in the virtual space, and a surround effect can be realized.

  Note that the number of virtual speakers set may be four or more. For example, when five virtual speakers are set, the first and second virtual speakers are set to the left front of the virtual user and the right front of the virtual user, and the third virtual speaker is set to the front of the virtual user. What is necessary is just to set the 4th, 5th virtual speaker behind the virtual user diagonally to the left and diagonally behind the right. In this case, even when the position and direction of the user change in the real space and the position and change of the virtual user change in the virtual space, the relative positional relationship of the first to fifth virtual speakers with respect to the virtual user is maintained. Will remain. Therefore, it is possible to realize a surround speaker (virtual surround speaker) that follows changes in the position and direction of the user (virtual user).

  The sound processing unit 130 adjusts the volume balance of the first to Nth virtual speakers based on the information on the first to Nth positions of the first to Nth virtual speakers and the information of the position of the virtual sound source. By doing so, the sound field is formed by the virtual sound source.

  For example, the volume balance adjustment of the first virtual speaker group is performed based on the position information of the first virtual speaker group among the first to Nth virtual speakers and the position information of the first virtual sound source. Thus, the sound field forming process is performed by the first virtual sound source. Further, the volume balance of the second virtual speaker group is adjusted based on the position information of the second virtual speaker group among the first to Nth virtual speakers and the position information of the second virtual sound source. Thus, the sound field forming process is performed by the second virtual sound source. Each of the first and second virtual speaker groups includes at least three virtual speakers. By adjusting the volume balance of these three virtual speakers based on the relative positional relationship between the positions of these three virtual speakers and the virtual sound source, sound is generated as if from the position of the virtual sound source. Create a sound field that sounds like it is.

  In the present embodiment, a movable range is set for the user in the real space, and the first to Mth speakers are arranged at positions corresponding to the movable range. For example, the first to Mth speakers are arranged around the movable range, or a part thereof is arranged within the movable range. The user's movable range is, for example, a user's play area, and the play area may be an area surrounded by a wall or an open space area.

  The display processing unit 120 performs display processing of an image that can be seen from the viewpoint of the virtual user corresponding to the user. For example, in an object space that is a virtual space, display processing of an image seen from the viewpoint is performed. This image is an image that can be seen in the viewing direction at the viewpoint of the virtual user. This image is preferably a so-called stereoscopic image.

  In addition, the display processing unit 120 performs an image display process from a first-person viewpoint where a virtual user is not displayed. That is, the character corresponding to the virtual user does not appear in the virtual space, and an image in the virtual user's viewpoint position and line-of-sight direction is displayed.

  The display processing unit 120 performs processing for displaying an image on the HMD 200 (head-mounted display device) worn by the user. When the HMD 200 is thus mounted and the field of view is covered, the user cannot grasp the positions of the plurality of speakers arranged around the user. Also in this case, in the present embodiment, a plurality of virtual speakers realized by volume balance adjustment of the plurality of speakers is set with a predetermined relative positional relationship with respect to the virtual user corresponding to the user. become. That is, the relative positional relationship is maintained even when the position or direction of the user changes and the position or direction of the virtual user changes. Therefore, it is possible to form an appropriate (accurate) sound field even when the position or direction of the user wearing the HMD 200 changes.

  Further, when the user moves while wearing the HMD 200, the input processing unit 102 acquires information on the position and direction of the user based on information from the sensor unit 210 provided in the HMD 200. For example, information on the position and direction of the user is acquired by a tracking method as shown in FIGS. Then, based on the acquired position and direction information of the user, the position and direction of the virtual user corresponding to the user are specified, a plurality of virtual speakers following the virtual user are set, and the sound field by the virtual sound source is set. Form.

  In addition, the display processing unit 130 performs an image display process for reproducing the sound generation state by the virtual sound source in the virtual space. For example, when a spectator appearing in the virtual space is a virtual sound source, a reproduction image of a situation in which sound is generated when the spectator shouts, shouts, or claps is displayed. When a player such as a guitarist appearing in the virtual space is a virtual sound source, a reproduction image of a situation in which the player performs while moving, for example, and a performance sound (guitar sound, etc.) is generated is displayed. When the explosive is a virtual sound source, an image that reproduces the situation where the explosive is dropped and an explosion sound is generated by the explosion is displayed. By doing so, the sound field of the virtual sound source using the virtual speaker and the image of the sound generation status by the virtual sound source are linked, and the virtual reality of the user can be further improved.

  The sound processing unit 130 is output from the i-th speaker when the user's approach to the i-th speaker (1 ≦ i <M) among the first to M-th speakers in the real space is detected. The process of adjusting at least one of the volume of the sound, the high frequency component, and the reverberation is performed. For example, the presence of the i-th speaker is achieved by lowering the volume of the sound of the i-th speaker, reducing the high frequency component of the sound of the i-th speaker, or increasing the reverberation (echo) of the sound of the i-th speaker. Process to hide. In this way, when the user approaches the i-th speaker, it is possible to prevent the user from recognizing the presence of the i-th speaker. This makes it difficult for the user to feel that the virtual speaker is formed by these real space speakers.

  Further, when the sound processing unit 130 determines that a predetermined situation has occurred, the sound processing unit 130 invalidates the settings of the first to Nth virtual speakers. For example, when it is determined that a predetermined situation such as the occurrence of an emergency has occurred, the settings of the first to Nth virtual speakers are invalidated and the virtual sound output by the virtual speakers is invalidated. The process of invalidating the settings of the first to Nth virtual speakers can be realized by preventing the first to Mth speakers in the real space from performing volume balance adjustment for setting the virtual speakers. For example, when each virtual speaker of the first to Nth virtual speakers is set by volume balance adjustment of each speaker group corresponding to each virtual speaker, the volume balance adjustment of each speaker group is not performed. Thus, the setting of each virtual speaker can be invalidated.

2. Next, the method of this embodiment will be described in detail. In the following description, a case where the game to which the method of the present embodiment is applied is a music game (live stage game, karaoke game, etc.) for performing a song will be mainly described as an example. However, the game to which the method of the present embodiment is applied is not limited to this, for example, music that plays an instrument such as a stringed instrument (guitar, etc.), a percussion instrument (drum, drum, etc.), or a keyboard instrument (keyboard, piano). It may be a game (game that competes for rhythm and skill). In addition, the method of the present embodiment includes a communication game (human relationship simulation game) with a heterosexual character, a conversation game (forensic battle game, cross game), a battle game, an RPG game, a robot game, and a card game. It can also be applied to various games such as sports games or action games, and playback of video content and music content.

2.1 HMD and Play Area FIG. 2A shows an example of the HMD 200 used in the simulation system of this embodiment. As shown in FIG. 2A, the HMD 200 is provided with a plurality of light receiving elements 201, 202, and 203 (photodiodes). The light receiving elements 201 and 202 are provided on the front side of the HMD 200, and the light receiving element 203 is provided on the right side of the HMD 200. In addition, a light receiving element (not shown) is also provided on the left side, upper surface, and the like of the HMD.

  The user PL has the input devices 160-1 and 160-2 with the left hand and the right hand. Similar to the HMD 200, the input devices 160-1 and 160-2 are provided with a plurality of light receiving elements (not shown). The input device 160-1 is provided with a microphone 162 (a sound input device in a broad sense), and the user PL sings with the mouth toward the microphone 162 in a song playing game. The input devices 160-1 and 160-2 also function as game controllers, and are provided with operation buttons, direction instruction keys, and the like (not shown). Note that the number of input devices 160 held by the user may be one.

  Further, the HMD 200 is provided with a headband 260 and the like so that the user PL can stably wear the HMD 200 on the head with a better wearing feeling. Then, the user PL operates the input devices 160-1 and 160-2 functioning as a game controller, performs a head movement operation and a head swing operation, and inputs operation information to enjoy game play. The whirling motion and the swing motion can be detected by the sensor unit 210 of the HMD 200 or the like.

  As shown in FIG. 2B, base stations 280 and 284 are installed in the play area of the user PL. The base station 280 is provided with light emitting elements 281 and 282, and the base station 284 is provided with light emitting elements 285 and 286. The light emitting elements 281, 282, 285, and 286 are realized by LEDs that emit laser (infrared laser or the like), for example. The base stations 280 and 284 use these light emitting elements 281, 282, 285, and 286 to emit, for example, a laser beam radially. The light receiving elements 201 to 203 and the like provided in the HMD 200 in FIG. 2A receive the lasers from the base stations 280 and 284, whereby the tracking of the HMD 200 is realized, and the position of the head of the user PL and the direction in which the user PL faces. (The position and direction of the user in a broad sense) can be detected. The light receiving elements (not shown) provided in the input devices 160-1 and 160-2 receive lasers from the base stations 280 and 284, thereby realizing tracking of the input devices 160-1 and 160-2. The positions and directions of the input devices 160-1 and 160-2 can be detected. Thereby, for example, an image of a microphone corresponding to the input device 160-1 can be displayed on the game image.

  FIG. 3A shows another example of the HMD 200. In FIG. 3A, a plurality of light emitting elements 231 to 236 are provided for the HMD 200. These light emitting elements 231 to 236 are realized by, for example, LEDs. The light emitting elements 231 to 234 are provided on the front side of the HMD 200, and the light emitting element 235 and the light emitting element 236 (not shown) are provided on the back side. These light emitting elements 231 to 236 emit (emit) light in a visible light band, for example. Specifically, the light emitting elements 231 to 236 emit light of different colors. 3B is installed on the front side of the user PL, and the light of the light emitting elements 231 to 236 is imaged by the imaging unit 150. That is, the spot light of these light emitting elements 231 to 236 is reflected in the captured image of the imaging unit 150. And the tracking of the user's PL head (HMD) is implement | achieved by performing the image process of this captured image. That is, the three-dimensional position of the head of the user PL and the facing direction (user position and direction) are detected.

  For example, as illustrated in FIG. 3B, the imaging unit 150 includes first and second cameras 151 and 152, and the first and second cameras 151 and 152 of the first and second cameras 151 and 152 are provided. By using the captured image, it is possible to detect the position and the like of the head of the user PL in the depth direction. Further, based on the motion detection information of the motion sensor provided in the HMD 200, the rotation angle (line of sight) of the head of the user PL can also be detected. Therefore, by using such an HMD 200, when the user PL is directed in any direction of all 360 degrees around the user, an image (a user's virtual space) is displayed in the corresponding virtual space (virtual three-dimensional space). Image viewed from a virtual camera corresponding to the viewpoint) can be displayed on the display unit 220 of the HMD 200. Note that instead of visible light, infrared LEDs may be used as the light emitting elements 231 to 236. Further, the position or movement of the user's head may be detected by another method such as using a depth camera.

  Note that the tracking processing method for detecting the user's viewpoint position and line-of-sight direction (user's position and direction) is not limited to the method described with reference to FIGS. For example, the tracking process may be realized by a single unit of the HMD 200 using a motion sensor or the like provided in the HMD 200. That is, tracking processing is realized without providing external devices such as the base stations 280 and 284 in FIG. 2B and the imaging unit 150 in FIG. Or you may detect viewpoint information, such as a user's viewpoint position and a gaze direction, by various viewpoint tracking methods, such as well-known eye tracking, face tracking, or head tracking.

  FIG. 4 and FIG. 5 show examples of play areas where the game of this embodiment is realized. This play area is realized by a box-shaped soundproof private room. As shown in FIGS. 4 and 5, the box private room has walls 301, 302, 303, 304, a ceiling 305, and a door 306. Sound insulating materials 311, 312, 313, 314, and 315 that also function as cushioning materials are provided inside the walls 301, 302, 303, and 304 and the ceiling 305. The base stations 280 and 284 and the lighting fixtures 290 and 292 are installed on the ceiling 305.

  Front speakers 330 and 331 and a center speaker 332 are installed on the front side of the user PL, and rear speakers 333 and 334 and a woofer 335 are installed on the rear side. Surround sound is realized by these speakers. And the winding device 50 is accommodated in the accommodation box in which the woofer 335 is accommodated. The winding device 50 includes a rotating reel 62, and the cable 20 is wound around the rotating reel 62 through a cable passage port 52 provided in a storage box (shelf).

  The user PL opens the door 306 in FIG. 5 and enters the private room to play a game. This space in the private room becomes the play area (play space, real space) of the user PL. And as shown in FIG. 5, the area AR assumed as a movement range (movable range) of the user PL is set in the play area of the private room of the box. Within this area AR, it is possible to track the position and direction (viewpoint position, line-of-sight direction) of the user PL using the stations 280, 284 and the like. On the other hand, reliable tracking cannot be realized at a position beyond the boundary BD of the area AR. Further, when the user PL exceeds the boundary BD of the area AR, there is a possibility that the user PL will hit the walls 301, 302, 303, 304, which is not desirable in terms of safety. The area AR can be set, for example, by the initialization setting of the game device.

  As shown in FIGS. 4 and 5, the user PL is wearing the waist belt 30. A housing portion 32 is attached to the waist belt 30, and a relay point RP of the cable 20 is provided in the housing portion 32. The cable 20 is wound up by the winding device 50 from the HMD 200 via the relay point RP. A point between the cable portion 21 and the cable portion 22 becomes a relay point RP. The cable 20 is provided with a stopper 26 for loosening the cable 20 when the user PL is standing at the reference position.

2.2 Outline of Game Next, an outline of the game realized by the method of the present embodiment will be described. The game realized by the present embodiment is a music game in which a vocal performance is achieved by becoming a vocal of a band in a sense of reality like a real live stage. The user can obtain a strong feeling of cheering from his fans all over the body while feeling an unprecedented sense of singing in front of a large audience. With HMD and high-power surround speakers, you can feel as if you are performing on a live stage and singing with your fans.

  The audience around the stage responds to the user's vocal performance and stage action and interactively returns flashy cheering and various actions. On the live stage as if standing on the spot by HMD, in front of the crowd in front of the crowd that fills the venue, including the fans in the front row who can see the expression, Meet the audience's expectations by singing and performing live.

  Users can make reservations and play settings for entry times in the reception space provided in the common space, and experience live appearance in a secure private room with cushioning material (soundproofing material) as shown in Figs. Enjoy.

  The user selects the concert appearance mode in the play setting before the stage appearance. After that, the song to be sung is selected and the appearance stage is selected. Then, the devices such as the HMD 200 and the input devices 160-1 and 160-2 described in FIG. 2A and FIG. The store operator explains the precautions and assists the user in mounting and adjusting the device. The operator calibrates (initializes) the private room space, which is the play area, in advance.

  The input device 160-1 in FIG. 2A is a microphone and game controller. In the VR (virtual reality) space, the arms and hands of the user (virtual user) are not drawn, but the position of the input device 160-1 etc. held by the user is sensed, and a microphone image is drawn at the same position. The microphone image moves according to the user's movement.

  The user performs vocal key adjustment in the standby room of the VR space. The standby room is a waiting space below the stage. The space where the user stands in the VR space is a large lift, and rises on the stage during the performance.

  As the lift rises and the stage approaches, the cheers and shouts of the hall that were heard from a distance gradually increase, increasing the force and changing vividly. When a user appears on the stage, a spotlight of backlighting is applied from the front toward the user, and a loud cheer occurs that has reached its peak.

  In the live performance, users enjoy singing on stage as much as they want. 6 and 7 are examples of game images (images in the VR space) displayed on the user's HMD 200 on the stage. As shown in FIGS. 6 and 7, a full audience is projected in front of the user. FIG. 6 is a game image when the user is facing the front, and FIG. 7 is a game image when the user is facing the right.

  As shown in FIGS. 6 and 7, in the game device of this embodiment using the HMD 200, the world of the VR space, which is a virtual space, extends across the entire direction of the user. For example, when the user wearing the HMD 200 faces forward, the game image of FIG. 6 is displayed on the HMD 200, and when the user wears the right direction, the game image of FIG. 7 is displayed. If you look back, you can see an image of a performance band. Therefore, in a huge concert hall where a large number of spectators cheer, it is possible to give the user a virtual reality feeling as if they are performing vocals, and the degree of immersion in the game can be greatly improved.

  In the present embodiment, the movement or cheering of the audience changes in accordance with the inflection of the song. In addition, the movement and cheer of the audience change according to the user's action. For example, in FIG. 6, spectators AD1 to AD7 near the stage where the user stands are represented by polygon model objects composed of a large number of polygons, for example. The same applies to the audiences AD8 to AD11 in FIG. On the other hand, the audience at a position far from the stage is represented by an image drawn on a billboard polygon facing the user's line of sight.

  The movement of the polygon model audience (AD1 to AD11) is expressed by, for example, motion reproduction using motion data. These spectators perform basic actions (basic motion) according to the rhythm of the song. The audience reacts interactively according to the voice and movement of the user. When the user performs actions such as “singing in a specific direction” or “waving hands”, the tension of the audience in that direction increases, and the basic action becomes one stage, flashy, or Suddenly exciting action that has nothing to do with rhythm.

  FIG. 8 is a diagram illustrating a state where the stage SG where the user PL (virtual user) stands in the VR space is seen from above. The stage SG is divided by the boundary BDS, and spectators acting as shown in FIGS. 6 and 7 are arranged in the audience seats SE1, SE2, and SE3 around the stage SG. The boundary BDS of the stage SG corresponds to, for example, the boundary BD of the area AR of the real world play area in FIG. As an example, the boundary BDS of the stage SG in the VR space is set at a position corresponding to, for example, a predetermined distance from the boundary BD of the real world area AR in FIG.

  In this embodiment, the user can perform a performance for exciting the audience. A spectator who performs some appealing action toward the user becomes the target of the performance. For example, in FIG. 6, the audience AD 4 raises his right hand to perform an appealing action to the user, and this audience AD 4 is the target. In FIG. 7, the audience AD10 who is performing an appealing action is the target.

  When the user performs an action on these appealing spectators, the value of the enthusiastic parameter (enthusiasm gauge) of these spectators increases. When the enthusiasm parameter reaches the maximum value, these spectators will perform an enthusiasm action that expresses delight.

  The first action that the user performs on the appealing audience is an action of singing with a line of sight toward the target audience. This first action is performed in the singing part among the singing part and the interlude part of the music.

  The second action performed by the user for the appealing audience is an action for making a call to the audience. The audience responds to the user's call with cheering, cheering, or applause. The user performs an action of raising an arm in accordance with cheering, cheering, and applause from the audience.

  Audiences that are targets of these first and second actions occur, for example, randomly in the concert hall. The number of targets in the performance of one song by the user is determined to be a predetermined number. For all target audiences of the predetermined number, when the enthusiasm parameter reaches the maximum value and the excitement is successful, it is all cleared. For example, if the number of target audiences is 10, and all of these 10 audiences have reached the maximum value, the target is completely cleared. In this case, the type of the last stage effect changes according to the number of target clears. For example, when all 10 targets are cleared, the most flashy effect occurs in the last stage. For example, when eight targets are cleared, a more flashy effect occurs than when five targets are cleared.

  In this embodiment, the user's singing ability is evaluated. That is, the pitch and rhythm of the user's song is detected and evaluated. Specifically, it is evaluated whether or not the utterance can be adapted to the rhythm. For example, when the user can speak in accordance with the designated timing of playing the bass or drum, points are added. In addition, the skill of long tone and the accuracy of rest are evaluated. Also evaluate the accuracy of the pitch. In other words, the pitch of the user's song is determined, and the pitch of the song sung with the correct pitch is displayed graphically. Note that if the user's volume level is zero for a certain period of time, most of the audience will shift to a standby motion, during which time the user will not be able to earn points.

  When the user's performance is completed and the user's evaluation result through the live stage satisfies a certain standard, the audience asks for an encore. By responding to the encore, the user can play one more song. Then, at the end of the live stage, the lights become darker in the cheering that fades out, and it ends with a blackout. After that, a message such as “Thank you very much, please remove the HMD” and a guide voice are played. Then, the user removes the equipment and leaves the private room, and the game play ends.

2.3 Virtual Speaker In FIG. 9A, the user PL is movable in a play area that is a movable range, and there are five speakers FL, FR, FC, around the play area (movable range). RL and RR are arranged. FL and FR are front speakers, FC is a center speaker, and RL and RR are rear speakers. By arranging such speakers FL, FR, FC, RL, and RR, so-called surround sound can be realized, and realistic sound in which a sense of depth and a sense of breadth are expressed can be provided to the user PL.

  However, the surround by these speakers FL, FR, FC, RL, and RR is such that the user PL is located at the center (reference position) of the play area and faces the front direction (reference direction) as shown in FIG. The sound field of the virtual space is formed so as to be optimal in the case of Therefore, the user PL moves as shown in FIG. 9B, and the position PPL of the user PL changes to a place away from the center of the play area, or the direction DPL toward the user PL as shown in FIG. If it changes, the sound field will be incorrect (inaccurate). Here, the position PPL of the user PL is, for example, the viewpoint position of the user, and the direction DPL of the user PL is, for example, the line-of-sight direction of the user PL. Note that it may be assumed that the moving direction of the user PL matches the line of sight of the user.

  For example, the user PL feels that the position of the virtual sound source (for example, a band player) in the virtual space immediately behind in FIG. 9 (A) exists at an oblique right position in FIG. 9 (B). . In addition, the user PL feels that the position of the virtual sound source (for example, the spectator) that is directly in front in FIG. 9A exists diagonally to the left in FIG. For this reason, there exists a possibility that problems, such as the virtual reality of user PL being impaired, or losing a sense of direction, may arise. In particular, when the user PL is wearing the HMD 200, the field of view is covered by the HMD 200, and the real world situation cannot be seen, so there is a high possibility of losing a sense of direction or the like.

  Therefore, in the present embodiment, a technique is adopted that enables an accurate sound field environment (for example, a surround environment) to be maintained even if the position PPL and the direction DPL of the user PL change. Specifically, a plurality of virtual speakers are set in the virtual space by adjusting the volume balance of the plurality of speakers in the real space, and a virtual sound source in the virtual space is expressed using the plurality of virtual speakers.

  First, a stereo virtual speaker setting method will be described with reference to FIGS. 11A and 11B. For example, in FIGS. 11A and 11B, the left-channel virtual speaker VS1 is set (configured) by adjusting the volume balance of the speakers SP1 and SP2. In addition, the right channel virtual speaker VS2 is set (configured) by adjusting the volume balance of the speakers SP3 and SP4. And the sound of the left channel in stereo reproduction | regeneration is output from virtual speaker VS1. Further, the right channel sound in stereo reproduction is output from the virtual speaker VS2. As a result, a stereo effect in the correct orientation can be given to the user.

  Here, the speakers SP1, SP2, SP3, and SP4 in FIGS. 11A and 11B are speakers arranged in the real space, and the movable range of the user PL as shown in FIG. 9A. Corresponds to the speakers FL, FR, FC, RL, RR installed corresponding to. P1, P2, P3, and P4 are the positions of the speakers SP1, SP2, SP3, and SP4. PS1 and PS2 are the positions (first and second positions) of the virtual speakers VS1 and VS2 (first and second virtual speakers).

  Further, the virtual user PV (virtual player) in FIGS. 11A and 11B corresponds to the user PL (player) in FIG. 9A, and the positions PPL and DPL of the user PL in the real space. Changes, the position PP and the direction DP of the virtual user PV in the virtual space also change accordingly. The position PP is, for example, the viewpoint position of the virtual user PV, and the direction DP is, for example, the visual line direction of the virtual user PV.

  For example, when the user PL is wearing the HMD 200, the position and direction (viewpoint position, line-of-sight direction) of the user PL are specified by detecting the position and direction of the HMD 200, and thereby the position and direction of the virtual user PV. (Viewpoint position, line-of-sight direction) is identified. Therefore, when the position PPL and the direction DPL of the user PL change as shown in FIGS. 9A to 10, the position PP and the direction DP of the virtual user PV in the virtual space can be changed accordingly. Even in a system in which the user PL does not wear the HMD 200, the position PPL and the direction DPL of the user PL can be specified by using various sensors, and the position PP and the direction DP of the virtual user PV can be changed accordingly. Is possible. For example, by using a sensor such as an image sensor (RGB sensor) or a depth sensor, the skeleton information and position information of the user PL may be acquired to specify the position PPL and the direction DPL of the user PL.

  In the present embodiment, even when the position PP and the direction DP of the virtual user PV change in accordance with the change in the position PPL and the direction DPL of the user PL, the virtual speakers VS1 and VS2 have a predetermined relative relation to the virtual user PV. Follow the target position. Specifically, in FIGS. 11A and 11B, virtual speakers are positioned at positions PS1 and PS2 (first and second positions) that follow the virtual user PV while maintaining a relative positional relationship. VS1 and VS2 (first and second virtual speakers) are set. For example, the relative positional relationship that the positions PS1 and PS2 of the virtual speakers VS1 and VS2 are set on the line connecting the position of the left ear EL and the position of the right ear ER of the virtual user PV is maintained. For example, in FIG. 11A and FIG. 11B, the position PP and the direction DP of the virtual user PV are different, but the relative positional relationship of the virtual speakers VS1 and VS2 with respect to the virtual user PV is maintained. That is, the virtual speakers VS1 and VS2 follow the virtual user PV so that the positions PS1 and PS2 are set on the line connecting the left ear EL and the right ear ER of the virtual user PV.

  Then, a sound field is formed by a virtual sound source (for example, a spectator or a player described later) in the virtual space using these virtual speakers VS1 and VS2. For example, volume balance adjustment of the virtual speakers VS1 and VS2 is performed so that the sound from the virtual sound source is localized at the position of the virtual sound source. In this way, even when the position PPL and the direction DPL of the user PL change and the position PP and the direction DP of the virtual user PV change, an appropriate sound field that does not depend on these changes can be formed. Become.

  11A and 11B, by adjusting the volume balance of the speakers SP1 and SP2, the virtual speaker VS1 is set at the position PS1, and the volume balance of the speakers SP3 and SP4 is adjusted. A virtual speaker VS2 is set at the position PS2.

  FIG. 12 is an explanatory diagram of a method for setting the virtual speakers VS1 and VS2 of FIGS. 11A and 11B by adjusting the volume balance of the speakers SP1 to SP4.

  In FIG. 12, information on the positions P1 and P2 of the speakers SP1 and SP2, which are the first speaker group among the speakers SP1 to SP4 (first to Mth speakers in a broad sense), and the position PS1 (first position). ), The virtual speaker VS1 (first virtual speaker) is set by adjusting the volume balance of the speakers SP1 and SP2. Further, based on the information on the positions P3 and P4 of the speakers SP3 and SP4, which are the second speaker group of the speakers SP1 to SP4, and the information on the position PS2 (second position), the volume balance of the speakers SP3 and SP4. By performing the adjustment, the virtual speaker VS2 (second virtual speaker) is set.

  For example, as described in FIGS. 11A and 11B, the positions PS1 and PS2 are set to positions that maintain a relative positional relationship with respect to the position PP and the direction DP of the virtual user PV. As shown in FIG. 12, the position is set on a line LLR connecting the left ear EL and the right ear ER. Specifically, the position PS1 is the position of the intersection of the line LLR and the line L12 connecting the positions P1 and P2. The position PS2 is the position of the intersection of the line LLR and the line L34 connecting the positions P3 and P4.

  Then, the volume balance of the speakers SP1 and SP2 (first speaker group) at the positions P1 and P2 is adjusted so that the virtual speaker VS1 is set at the position PS1. For example, the distance between the positions P1 and PS1 is D1, and the distance between the positions P2 and PS1 is D2. In this case, as D1 becomes longer and D2 becomes shorter, the volume of the speaker SP1 is reduced and the volume of the speaker SP2 is increased. On the other hand, as D1 becomes shorter and D2 becomes longer, the volume of speaker SP1 is increased and the volume of speaker SP2 is decreased.

  Further, the volume balance of the speakers SP1 and SP2 (second speaker group) at the positions P3 and P4 is adjusted so that the virtual speaker VS2 is set at the position PS2. For example, the distance between the positions P3 and PS2 is D3, and the distance between the positions P4 and PS2 is D4. In this case, the volume of the speaker SP3 is decreased and the volume of the speaker SP4 is increased as D3 becomes longer and D4 becomes shorter. On the other hand, as D3 becomes shorter and D4 becomes longer, the volume of speaker SP3 is increased and the volume of speaker SP4 is decreased. By doing so, the virtual speakers VS1 and VS2 can be localized at the positions PS1 and PS2.

2.4 Setting of Virtual Speaker by Three or More Speakers In the method of FIGS. 11A to 12, only two speakers are used as a speaker group for setting each virtual speaker. Cannot set a virtual speaker. For this reason, there is a problem that it is difficult to realize a strictly correct stereo effect. For example, in FIG. 11A, when the position PP of the virtual user PV moves to the left side, the distance between the virtual speaker VS1 and the left ear EL becomes short, and the distance between the virtual speaker VS2 and the right ear ER becomes long. Further, when the position PP of the virtual user PV moves to the right side, the distance between the virtual speaker VS1 and the left ear EL increases, and the distance between the virtual speaker VS2 and the right ear ER decreases. Therefore, it is difficult to realize a correct stereo effect.

  In this regard, in the present embodiment, as described with reference to FIGS. 13A to 14, three or more speakers are used as a speaker group for setting each virtual speaker, whereby the virtual speaker at an arbitrary position is used. It is possible to set.

  For example, in FIGS. 13A and 13B, the virtual speaker VS1 is set by the first speaker group including at least three speakers SP1, SP2, and SP3. The virtual speaker VS2 is set by a second speaker group including at least three speakers SP2, SP3, SP4.

  In this way, in FIGS. 13A and 13B, the virtual speaker VS1 is set at a position corresponding to the left ear EL of the virtual user PV, and the virtual speaker VS2 is set at a position corresponding to the right ear ER. Can be set. Here, the position corresponding to the left ear EL is, for example, the position of the left speaker of the virtual headphone worn by the virtual user PV, and the position corresponding to the right ear ER is, for example, the position of the right speaker of the virtual headphone. It is. By doing so, an effect close to the state in which headphones are worn can be created, and there is an advantage that binaural sound sources can be reproduced, for example. Further, since the positions PS1 and PS2 of the virtual speakers VS1 and VS2 can be set freely, there is an advantage that a correct stereo effect can be realized without depending on the position of the user PL (virtual user PV).

  FIG. 14 is an explanatory diagram of a method for setting the virtual speakers VS1 and VS2 of FIGS. 13A and 13B by adjusting the volume balance of the speakers SP1 to SP4.

  In FIG. 14, the information of the positions P1, P2, and P3 of the speakers SP1, SP2, and SP3, which are the first speaker group among the speakers SP1 to SP4 (first to Mth speakers), and the information of the position PS1. Based on this, the virtual speaker VS1 is set by adjusting the volume balance of the speakers SP1, SP2, and SP3. Further, based on the information on the positions P2, P3, and P4 of the speakers SP2, SP3, and SP4 which are the second speaker group among the speakers SP1 to SP4, the volume balance of the speakers SP2, SP3, and SP4 is based on the information on the position PS2. By performing the adjustment, the virtual speaker VS2 is set. That is, each virtual speaker is set by adjusting the volume balance of a speaker group including at least three speakers.

  For example, as shown in FIGS. 13A and 13B, the positions PS1 and PS2 are set to positions that maintain a relative positional relationship with respect to the position PP and the direction DP of the virtual user PV. Specifically, as shown in FIG. 14, PS1 and PS2 are set at the same distance DE from the position PP at a position on the line LLR connecting the left ear EL and the right ear ER. For example, the line LLR is orthogonal to the direction DP of the virtual user PV.

  In this embodiment, the volume balance of the speakers SP1, SP2, and SP3 (first speaker group) at the positions P1, P2, and P3 is adjusted so that the virtual speaker VS1 is set at the position PS1.

  For example, in FIG. 14, a line connecting the positions P1 and P2 of the speakers SP1 and SP2 is L12, and an intersection with a perpendicular line from the position PS1 of the virtual speaker VS1 to the line L12 is PS1Y. The distance between the position P1 and the intersection PS1Y is Y1, and the distance between the position P2 and the intersection PS1Y is Y2. Further, a line connecting the positions P2 and P3 of the speakers SP2 and SP3 is L23, and an intersection with a perpendicular line from the position PS1 of the virtual speaker VS1 to the line L23 is PS1X. The distance between the position P2 and the intersection PS1X is X2, and the distance between the position P3 and the intersection PS1X is X3.

  In this case, the volume balance of the speakers SP1 and SP2 is adjusted based on the distances Y1 and Y2, and the volume balance of the speakers SP2 and SP3 is adjusted based on the distances X2 and X3. For example, as Y1 becomes longer and Y2 becomes shorter, the volume of the speaker SP1 is reduced and the volume of the speaker SP2 is increased. On the other hand, as Y1 becomes shorter and Y2 becomes longer, the volume of speaker SP1 is increased and the volume of speaker SP2 is decreased. Further, as X2 becomes longer and X3 becomes shorter, the volume of the speaker SP2 is reduced and the volume of the speaker SP3 is increased. On the other hand, as X2 becomes shorter and X3 becomes longer, the volume of speaker SP2 is increased and the volume of speaker SP3 is decreased.

  In the present embodiment, the volume balance of the speakers SP2, SP3, SP4 (second speaker group) at the positions P3, P3, P4 is adjusted so that the virtual speaker VS2 is set at the position PS2.

  For example, in FIG. 14, a line connecting the positions P3 and P4 of the speakers SP3 and SP4 is L34, and an intersection with a perpendicular line from the position PS2 of the virtual speaker VS2 to the line L34 is PS2Y. The distance between the position P3 and the intersection PS2Y is Y3, and the distance between the position P4 and the intersection PS2Y is Y4. Further, a line connecting the positions P4 and P1 of the speakers SP4 and SP1 is L41, and an intersection with a perpendicular line from the position PS2 of the virtual speaker VS2 to the line L41 is PS2X. The distance between the position P4 and the intersection PS2X is X4, and the distance between the position P1 and the intersection PS2X is X1.

  In this case, the volume balance of the speakers SP3 and SP4 is adjusted based on the distances Y3 and Y4, and the volume balance of the speakers SP4 and SP1 is adjusted based on the distances X4 and X1. For example, as Y3 becomes longer and Y4 becomes shorter, the volume of the speaker SP3 is decreased and the volume of the speaker SP4 is increased. On the other hand, as Y3 becomes shorter and Y4 becomes longer, the volume of speaker SP3 is increased and the volume of speaker SP4 is decreased. Further, as X4 becomes longer and X1 becomes shorter, the volume of speaker SP4 is decreased and the volume of speaker SP1 is increased. On the other hand, as X4 becomes shorter and X1 becomes longer, the volume of speaker SP4 is increased and the volume of speaker SP1 is decreased. As described above, the virtual speakers VS1 and VS2 can be localized at the positions PS1 and PS2.

  FIGS. 15A and 15B are explanatory diagrams of binaural recording. As shown in FIG. 15A, for example, a dummy head DMH that reproduces the acoustic effect of a human head is prepared, and a plurality of sound sources are arranged around the dummy head DMH. Microphones MCL and MCR are installed at the positions of the left and right ears of the dummy head DMH. And the sound from these several sound sources is recorded using microphone MCL and MCR. The sound recorded in this way is output from the left speaker HDL and the right speaker HDR of the headphone HDP as shown in FIG. 15B, so that a sound with a sense of presence can be reproduced as if it were present. . Sound corresponding to such binaural recording is simulated by computer calculation processing. For example, a model (model object) corresponding to the dummy head DMH is prepared. Then, in addition to the sound waves that reach the left and right ears of the model from the virtual sound source in the virtual space, a simulation process of the sound that superimposes the sound waves that are diffracted and reflected by each part of the model's earlobe and body is performed to obtain a binaural sound source Generate sound. Then, the left channel sound from the binaural sound source is output from the virtual speaker VS1 in FIG. 13A, and the right channel sound from the binaural sound source is output from the virtual speaker VS2. Specifically, after performing the volume balance adjustment as described above, the left channel sound by the binaural sound source is output from the first speaker group corresponding to the virtual speaker VS1, and the second speaker corresponding to the virtual speaker VS2 is output. The right channel sound from the binaural sound source is output from the speaker group. By doing so, it is possible to generate a realistic and realistic sound field that is very close to the real world.

2.5 Surround FIGS. 16A and 16B are explanatory diagrams of a technique for realizing surround sound that increases the number of virtual speakers and follows the movement of the virtual user PV (user).

  In FIG. 16A and FIG. 16B, for example, four virtual speakers VS1, VS2, VS3, and VS4 are set to surround the virtual user PV. As shown in FIGS. 16A and 16B, even if the position PP and the direction DP of the virtual user PV change, the position that follows the virtual user PV while maintaining a predetermined relative positional relationship. Virtual speakers VS1, VS2, VS3, and VS4 (first to Nth virtual speakers) are set for PS1, PS2, PS3, and PS4 (first to Nth positions). For example, the virtual speaker VS1 is set by adjusting the volume balance of the first speaker group (SP4, SP1, SP2), and the virtual speaker VS2 is set by adjusting the volume balance of the second speaker group (SP1, SP2, SP3). The The virtual speaker VS3 is set by adjusting the volume balance of the third speaker group (SP2, SP3, SP4), and the virtual speaker VS4 is set by adjusting the volume balance of the fourth speaker group (SP3, SP4, SP1). .

  In addition, each speaker group which sets each virtual speaker can be changed based on position PP etc. of virtual user PV. In FIG. 16A and FIG. 16B, four virtual speakers are set in order to realize surround sound, but this embodiment is not limited to this. In order to realize surround sound, at least three or more virtual speakers may be set, and the number of virtual speakers is arbitrary. For example, a total of three virtual speakers, such as two front virtual speakers and one center virtual speaker, may be realized, or two rear speakers may be provided for these three virtual speakers. These virtual speakers may be added to realize a total of five virtual speakers. Since it is clear from FIG. 14 that a virtual speaker can be set at an arbitrary position for the virtual user PV, a detailed description of the setting of the virtual speakers VS1 to VS4 in FIGS. 16A and 16B is provided. Is omitted.

  16A and 16B, for example, the position PPL and the direction DPL of the user PL change as shown in FIGS. 9A to 10, and the position of the virtual user PV is changed accordingly. Even if the PP and the direction DP change, the surround virtual speakers VS1 to VS4 are set so as to follow the change. That is, the surround virtual speakers VS1 to VS4 follow the virtual user PV while maintaining a predetermined relative positional relationship even when the position PP and the direction DP of the virtual user PV change. For example, in FIG. 16A and FIG. 16B, the virtual speakers VS1, VS2, VS3, and VS4 are respectively left diagonally forward, diagonally left backward, diagonally right backward, and diagonally forward right for the virtual user PV. Fixed in position. Therefore, it is possible to realize surround sound with an appropriate sound field always in the correct direction.

2.6 Sound Field Formation by Virtual Sound Source In this embodiment, the sound field is formed by the virtual sound source by adjusting the volume balance of the virtual speaker based on the position information of the virtual speaker and the position information of the virtual sound source. Processing is in progress. FIGS. 17 and 18 are diagrams for explaining a method of forming a sound field of a virtual sound source using such a virtual speaker.

  In FIG. 17, virtual sound sources SR1 and SR2 are set at positions PR1 and PR2 in the virtual space. These virtual sound sources SR1 and R2 are sources of sound generated by, for example, a spectator, a player, etc. in a virtual space as will be described later. In this embodiment, the sound field by these virtual sound sources SR1 and SR2 is formed using the virtual speakers VS1 to VS4. As shown in FIGS. 16A and 16B, these virtual speakers VS1 to VS4 have a predetermined relative relation to the virtual user PV even when the position PP and the direction DP of the virtual user PV change. Follow the target position.

  For example, in the present embodiment, sound field formation by the virtual sound source SR1 is realized by adjusting the volume balance of the virtual speakers VS1, VS2, and VS3.

  For example, in FIG. 18, a line connecting the positions PS1 and PS2 of the virtual speakers VS1 and VS2 is LS12, and an intersection with a perpendicular line from the position PR1 of the virtual sound source SR1 to the line LS12 is PR1Y. The distance between the position PS1 and the intersection PR1Y is Y1R, and the distance between the position PS2 and the intersection PR1Y is Y2R. Also, a line connecting the positions PS2 and PS3 of the virtual speakers VS2 and VS3 is LS23, and an intersection with a perpendicular line from the position PR1 of the virtual sound source SR1 to the line LS23 is PR1X. The distance between the position PS2 and the intersection PR1X is X2R, and the distance between the position PS3 and the intersection PR1X is X3R.

  In this case, in the present embodiment, the volume balance of the virtual speakers VS1 and VS2 is adjusted based on the distances Y1R and Y2R, and the volume balance of the virtual speakers VS2 and VS3 is adjusted based on the distances X2R and X3R. SR1 is localized at position PR1. For example, as Y1R becomes longer and Y2R becomes shorter, the volume of the virtual speaker VS1 is decreased and the volume of the virtual speaker VS2 is increased. On the other hand, as Y1R becomes shorter and Y2R becomes longer, the volume of the virtual speaker VS1 is increased and the volume of the virtual speaker VS2 is decreased. As X2R becomes longer and X3R becomes shorter, the volume of the virtual speaker VS2 is reduced and the volume of the virtual speaker VS3 is increased. On the other hand, as X2R becomes shorter and X3R becomes longer, the volume of the virtual speaker VS2 is increased and the volume of the virtual speaker VS3 is decreased. In this way, a sound field in which the virtual sound source SR1 is localized at the position PR1 can be formed.

  The sound field formation by the virtual sound source SR2 can be realized by adjusting the volume balance of the virtual speakers VS3, VS4, and VS1, for example. In this case, the sound field formation for locating the virtual sound source SR2 to the position PR2 can be realized by the same method as in the case of the virtual sound source SR1, and thus detailed description thereof is omitted. Each virtual speaker group for setting each virtual sound source can be changed based on the position of each virtual sound source.

  As described above, in the present embodiment, information on the position PPL and the direction DPL of the user PL moving in the real space where the speakers SP1 to SP4 (first to Mth speakers) are arranged is acquired. Then, the position PP and the direction DP in the virtual space of the virtual user PV corresponding to the user PL are changed along with the change in the position PPL and the direction DPL of the user PL in the real space as shown in FIGS. And In this case, for example, as described in FIG. 16A, FIG. 16B, FIG. 13A to FIG. 14, etc., a position that follows the virtual user PV while maintaining a predetermined relative positional relationship. Virtual speakers VS1 to VS4 (first to Nth virtual speakers) are set for PS1 to PS4 (first to Nth positions). These virtual speakers VS1 to VS4 are set by adjusting the volume balance of the speakers SP1 to SP4. Then, sound field formation processing by the virtual sound sources SR1 and SR2 in the virtual space shown in FIG. 17 is performed using the virtual speakers VS1 to VS4. Specifically, as described in FIG. 18, the volume balance of the virtual speakers VS1 to VS4 is based on the information on the positions PS1 to PS4 of the virtual speakers VS1 to VS4 and the information on the positions PR1 and PR2 of the virtual sound sources SR1 and SR2. By performing the adjustment, a sound field forming process is performed by the virtual sound sources SR1 and SR2.

  As described above, in this embodiment, the virtual speakers are not simply localized in the virtual space, but the virtual speakers (VS1 to VS1) are positioned at positions (PS1 to PS4) that follow the virtual user while maintaining a predetermined relative positional relationship. VS4) is set. That is, even if the position and direction of the user change and the position and direction of the virtual user change, the virtual speaker is caused to follow so that the relative positional relationship is maintained. And the sound field of the virtual sound source which appears in virtual space is formed using a plurality of virtual speakers which follow a virtual user in this way. In this way, for example, the sound field of a virtual sound source (audience, performer, explosion, etc.) appearing in various forms at various positions in a game or the like can be made independent of changes in the position and direction of the user. It becomes possible to form accurately. That is, even when the position or direction of the user changes, it is possible to form an appropriate (accurate) sound field for the virtual sound source, and an unprecedented simulation system can be realized.

  In this embodiment, as described with reference to FIGS. 13A to 14, the first speaker group including at least three speakers (SP1, SP2, SP3) is connected to the left channel (first in a broad sense). (Channel) sound is output, and the right channel (second channel in a broad sense) sound is output to the second speaker group including at least three speakers (SP2, SP3, SP4). Then, the volume balance of the first speaker group is adjusted so that the virtual speaker VS1 is set at the position of the left ear EL of the virtual user PV. Further, the volume balance of the second speaker group is adjusted so that the virtual speaker VS2 is set at the position of the right ear ER of the virtual user PV. By doing so, it is possible to realize a stereo effect in which the left channel sound is output from the left speaker of the virtual headphone and the right channel sound is output from the right speaker of the virtual headphone. Then, by making the first and second speaker groups output sound from a binaural sound source, it is possible to realize more realistic and realistic three-dimensional sound.

  In the present embodiment, as shown in FIG. 19, a movable range ARM is set for the user PL in the real space. The user PL can watch and enjoy the sounds and images generated by the simulation system while freely moving in the movable range ARM. And speaker SP1-SP4 (1st-Mth speaker) for implement | achieving a virtual speaker is arrange | positioned in the position corresponding to this movable range ARM. For example, in FIG. 19, speakers SP1 to SP4 are arranged around the movable range ARM. That is, speakers SP1 to SP4 are arranged so as to surround the user PL. A part of these speakers can be arranged in the movable range ARM.

  In the present embodiment, even when the user PL freely moves in the movable range ARM set in this way and changes its position and direction, it does not depend on the change of the position and direction. (Accurate) sound field can be formed. For example, the sound field of various virtual sound sources generated in a virtual space in which a virtual user corresponding to the user PL moves can be accurately and accurately reproduced, and an unprecedented simulation system can be realized.

  Further, in the present embodiment, an image display process that is visible from the viewpoint of the virtual user corresponding to the user is performed. Specifically, as shown in FIGS. 6 and 7, an image display process is performed from a first-person viewpoint in which the virtual user is not displayed. By displaying such an image of the first person viewpoint, the user plays a game or the like while feeling the virtual reality as if entering the virtual space. For example, in the present embodiment, when the user in the real world moves and changes its position and direction, the position and direction of the virtual user in the virtual space changes. It feels like In this case, in this embodiment, the virtual speaker follows a change in the position and direction of the virtual user while maintaining a predetermined relative positional relationship, and the sound field of the virtual sound source in the virtual space is detected by the virtual speaker. A forming process is performed. Therefore, the user can immerse in the VR world with an accurately formed sound field without feeling the presence of the speakers SP1 to SP4 in the real space as shown in FIG.

  Further, in the present embodiment, as shown in FIG. 19, a process for displaying an image on the HMD worn by the user PL is performed. In the simulation system in which the user PL wears the HMD that covers the field of vision in this way, the presence of the real-world speakers SP1 to SP4 disappears, and the method of the present embodiment that can form an accurate sound field by the virtual speaker is particularly validate. When the user PL moves while wearing the HMD as described above, information on the position and direction of the user can be acquired based on information from a sensor unit provided in the HMD. Then, based on the acquired position and direction of the user, the position and direction of the virtual user are specified, a virtual speaker that follows the virtual user is set, and a realistic sound field can be formed.

  In the present embodiment, an image display process for reproducing the sound generation state by the virtual sound source in the virtual space is performed.

  For example, in FIG. 20, the virtual user PV (user) is calling on the audience ADA, ADB, and ADC in the virtual space. For example, the line of sight VLP is called toward the audience ADA, ADB, and ADC. This is realized, for example, when the user directs the line of sight VLP toward the audience ADA, ADB, and ADC and inputs the calling voice to the microphone 162 in FIG. When the call voice is input in this way, the audience ADA, ADB, ADC perform cheers or applause in response to the input. That is, a response voice or a response sound is generated. In this case, the audience ADA, ADB, and ADC correspond to virtual sound sources in the virtual space, and response sounds and response sounds such as cheers and applause are sounds generated from the virtual sound sources. As described above, in this embodiment, an image that reproduces a situation in which sound is generated from the virtual sound source that is the audience is displayed.

  For example, in FIG. 21 (A), the virtual user PV (user) makes a call with the line of sight VLP toward the front audience ADA to ADC, and the front audience ADA to ADC responds to this. And doing applause. On the other hand, in FIG. 21B, the virtual user PV is calling with the line of sight VLP toward the audience ADD to DF on the right side, and the audience ADD to DF on the right side is cheering or applauding in response to this. It is carried out. As described above, in the present embodiment, sound is generated from the virtual sound source at the location corresponding to the direction of the line of sight VLP of the virtual user PV, and the sound generation state by the virtual sound source is changed in real time.

  In FIG. 22A, the virtual user PV stands on the stage SG and performs a song. The spectators ADA to ADC as described in FIG. 21A are arranged in the spectator seat SE1, and the spectators ADD to ADF as described in FIG. 21B are arranged in the spectator seat SE2. A band performance stage SGB is set behind the virtual user PV, and a guitarist GT and a drummer DR perform a guitar performance and a drum performance. In this case, in addition to the audiences ADA to ADC and ADD to ADF, the guitarist GT and the drummer DR also become virtual sound sources in the virtual space, and in this embodiment, an image that reproduces the sound generation state by these virtual sound sources. Is generated. Specifically, for example, as shown in FIGS. 22A and 22B, when the guitarist GT moves on the performance stage SGB, an image showing the movement is displayed. As the guitarist GT moves, the corresponding virtual sound source for playing the guitar also moves.

  In this case, in this embodiment, when the virtual user PV is facing the front audience seat SE1 as shown in FIG. 22A, an accurate sound field where the guitar performance can be heard from the left rear direction. Can be formed. Further, as shown in FIG. 22B, when the virtual user PV faces the audience seat SE2 on the right side and the guitarist GT moves, an accurate sound field is formed so that the guitar performance can be heard from the right side. can do. Therefore, it is possible to construct a VR world with higher virtual reality and improve the user's immersiveness into the VR world.

  As shown in FIG. 23, when the user PL is wearing the HMD, the HMD covers the field of view, and thus a situation occurs in which the user PL approaches the speaker SP1. When such a situation occurs, the presence of the speaker SP1 is known to the user PL by hearing a loud sound from the speaker SP1. That is, in this embodiment, setting a virtual speaker has an effect of eliminating the presence of the real-world speakers SP1 to SP4, but the user PL approaches the specific speaker SP1 as shown in FIG. If it does, the said effect will reduce.

  Therefore, in the present embodiment, for example, when the approach of the user PL to the speaker SP1 (i-th speaker among the first to Mth speakers) is detected in this way, the volume of the sound output from the speaker SP1. Then, processing for adjusting at least one of the high-frequency component and reverberation is performed. For example, by reducing the volume of the sound of the speaker SP, reducing the high frequency component of the sound (low-pass filter processing), or increasing the reverberation (echo component) of the sound, the presence of the speaker SP1 is increased. Perform the process of elimination. By doing so, even when the user PL approaches the speaker SP1 (or SP2, SP3, SP4), the presence can be eliminated as much as possible. Therefore, it is possible to prevent the occurrence of a situation in which the virtual reality is impaired by the user PL perceiving the presence of the speaker SP1. Note that the determination process of the user's approach to the speaker may be performed in multiple stages, for example, 40 cm approach or 60 cm approach.

  In this embodiment, when it is determined that a predetermined situation has occurred, for example, the settings of the virtual speakers VS1 to VS4 (first to Nth virtual speakers) in FIGS. 16A and 16B are invalidated. You may make it do. For example, when it is determined that a predetermined situation such as the occurrence of an emergency has occurred, the settings of the virtual speakers VS1 to VS4 are invalidated so that virtual sounds are not output from the virtual speakers. Taking FIG. 16A and FIG. 16B as an example, the setting of the virtual speaker VS1 is invalidated by not adjusting the volume balance of the first speaker group (SP4, SP1, SP2). it can. Further, the setting of the virtual speaker VS2 can be invalidated by not performing the volume balance adjustment of the second speaker group (SP1, SP2, SP3). Similarly, the settings of the virtual speakers VS3 and VS4 are disabled by not performing the volume balance adjustment by the third speaker group (SP2, SP3, SP4) and the fourth speaker group (SP3, SP4, SP1). Can be.

  In the present embodiment, as described above, by setting a virtual speaker, it is possible to eliminate the presence of the speaker in the real space. When the virtual speaker is set, it becomes difficult for the user to recognize the position of the speaker in the real world.

  However, in some cases, it may be better not to perform such virtual speaker settings. Therefore, in this embodiment, for example, when a predetermined situation such as an emergency situation occurs, the setting of the virtual speaker is invalidated. By doing so, for example, when an emergency situation occurs, the setting of the virtual speaker is invalidated, and it is possible to provide guidance during emergency evacuation using, for example, a speaker near the door. That is, the setting of the virtual speaker is invalid, and the sound of the speaker near the door can be heard from the door. Therefore, even when the user is wearing the HMD, the user can move to the door and evacuate using the emergency evacuation guidance voice from the speaker as a clue. Note that the predetermined situation in which the virtual speaker is disabled is not limited to such an emergency situation. For example, even when a situation in which it is better to disable the virtual speaker occurs in the course of the game, processing for invalidating the setting of the virtual speaker may be performed.

3. Detailed Processing Next, a detailed processing example of the present embodiment will be described with reference to the flowchart of FIG.

  First, the position and direction of the user are acquired (step S1). For example, the position and direction of the user are acquired by detecting the position and direction of the HMD worn by the user. Next, speaker volume balance adjustment for setting each virtual speaker corresponding to the position and direction of the user is executed (step S2). For example, the volume balance adjustment as described in FIGS. 13A, 13B, 14, 16A, 16B, and the like is performed. And the formation process of the sound field of the virtual sound source in virtual space is performed using the set virtual speaker (step S3). For example, the sound field forming process described with reference to FIGS. 17 and 18 is performed to simulate the generation of sound by the virtual sound source described with reference to FIGS.

  Next, it is determined whether or not the user has approached the speaker (step S4). When it is determined that the speaker is approaching (when it is determined that the speaker is closer than the predetermined distance), the volume, high-frequency component, or reverberation adjustment processing of the speaker is executed (step S5). That is, as described with reference to FIG. 23, an adjustment process for eliminating the presence of the speaker is executed.

  Although the present embodiment has been described in detail as described above, it will be easily understood by those skilled in the art that many modifications can be made without departing from the novel matters and effects of the present invention. Accordingly, all such modifications are intended to be included in the scope of the present invention. For example, in the specification or drawings, terms (microphone, play area, left) described at least once together with different terms (sound input device, movable range, first channel, second channel, etc.) having a broader meaning or the same meaning Channel, right channel, etc.) may be replaced by their different terms anywhere in the specification or drawings. The virtual speaker setting method, virtual sound source sound field forming method, volume balance adjustment method, image display method, simulation system configuration, and the like are not limited to those described in this embodiment, and are equivalent to these. Techniques, processing, and configurations are also included in the scope of the present invention. The present invention can be applied to various games. Further, the present invention can be applied to various simulation systems such as a business game device, a home game device, or a large attraction system in which a large number of users participate.

PL user, PV virtual user, PPL, PP position, DPL, DP direction,
SP1 to SP4 speakers (first to Mth speakers), P1 to P4 positions,
VS1 to VS4 virtual speakers (first to Nth virtual speakers),
PS1 to PS4 position (1st to Nth position), RP relay point, AR area, BD boundary,
100 processing units, 102 input processing units, 110 arithmetic processing units, 111 game processing units,
112 game progress processing unit, 113 evaluation processing unit, 114 character processing unit,
115 parameter processing unit, 116 object space setting unit,
117 virtual camera control unit, 118 game result calculation unit, 120 display processing unit,
130 sound processing unit, 132 virtual speaker setting unit, 134 sound field formation processing unit,
140 output processing unit, 150 imaging unit, 151, 152 camera,
160 (160-1, 160-2) input device, 161 sound input device,
162 microphone, 170 storage unit, 172 spatial information storage unit, 174 music information storage unit,
175 sound data storage unit, 176 parameter storage unit, 178 drawing buffer,
180 Information storage medium, 192 sound output unit, 194 I / F unit,
195 portable information storage medium, 196 communication unit,
200 HMD (head-mounted display device), 201-203 light receiving element, 210 sensor unit,
220 display unit, 231 to 236 light emitting element, 240 processing unit, 260 headband,
280, 284 stations, 281, 282, 285, 286 light emitting elements,
290, 292 lighting fixtures, 301-304 walls, 305 ceiling, 306 doors,
311 to 315 Soundproof material, 330, 331 Front speaker, 332 Rear speaker,
333, 334, rear speaker, 335 woofer,

Claims (17)

An input processing unit for acquiring information on the position and direction of the user moving in the real space in which the first to Mth speakers (M is an integer of 3 or more) are arranged;
A predetermined relative positional relationship with respect to the virtual user even when the position and direction of the virtual user corresponding to the user change in accordance with the change of the position and direction of the user in the real space. The first to Nth virtual speakers are set by adjusting the volume balance of the first to Mth speakers with respect to the first to Nth (N is an integer of 2 or more) positions that follow the A sound processing unit that performs a sound field formation process using a virtual sound source in the virtual space using the first to Nth virtual speakers;
Only including,
The sound processing unit
A simulation system characterized by invalidating the settings of the first to Nth virtual speakers when it is determined that a predetermined situation has occurred . An input processing unit for acquiring information on the position and direction of the user moving in the real space in which the first to Mth speakers (M is an integer of 3 or more) are arranged;
A predetermined relative positional relationship with respect to the virtual user even when the position and direction of the virtual user corresponding to the user change in accordance with the change of the position and direction of the user in the real space. The first to Nth virtual speakers are set by adjusting the volume balance of the first to Mth speakers with respect to the first to Nth (N is an integer of 2 or more) positions that follow the A sound processing unit that performs a sound field formation process using a virtual sound source in the virtual space using the first to Nth virtual speakers;
Only including,
The sound processing unit
By adjusting the volume balance of the first to Nth virtual speakers based on the information of the first to Nth positions of the first to Nth virtual speakers and the information of the position of the virtual sound source. A simulation system characterized in that the sound field is formed by the virtual sound source . An input processing unit for acquiring information on the position and direction of the user moving in the real space in which the first to Mth speakers (M is an integer of 3 or more) are arranged;
A predetermined relative positional relationship with respect to the virtual user even when the position and direction of the virtual user corresponding to the user change in accordance with the change of the position and direction of the user in the real space. The first to Nth virtual speakers are set by adjusting the volume balance of the first to Mth speakers with respect to the first to Nth (N is an integer of 2 or more) positions that follow the A sound processing unit that performs a sound field formation process using a virtual sound source in the virtual space using the first to Nth virtual speakers;
Only including,
The sound processing unit
Based on the information on the position of the first speaker group among the first to Mth speakers and the information on the first position of the first virtual speaker among the first to Nth positions, By adjusting the volume balance of the first speaker group, the first virtual speaker among the first to Nth virtual speakers is set,
Based on the information on the position of the second speaker group among the first to Mth speakers and the information on the second position of the second virtual speaker among the first to Nth positions, The simulation system , wherein the second virtual speaker among the first to Nth virtual speakers is set by adjusting a volume balance of the second speaker group . In claim 3 ,
The sound processing unit
Causing the first speaker group including at least three speakers to output the sound of the first channel, and causing the second speaker group including at least three speakers to output the sound of the second channel. A simulation system characterized by In claim 4 ,
The sound processing unit
The volume balance of the first speaker group is adjusted so that the first virtual speaker is set at a position corresponding to the left ear of the virtual user, and the first speaker is set at a position corresponding to the right ear of the virtual user. A simulation system, wherein volume balance adjustment of the second speaker group is performed so that two virtual speakers are set. In claim 4 or 5 ,
The sound processing unit
A simulation system characterized by causing the first and second speaker groups to output a sound from a binaural sound source. An input processing unit for acquiring information on the position and direction of the user moving in the real space in which the first to Mth speakers (M is an integer of 3 or more) are arranged;
A predetermined relative positional relationship with respect to the virtual user even when the position and direction of the virtual user corresponding to the user change in accordance with the change of the position and direction of the user in the real space. The first to Nth virtual speakers are set by adjusting the volume balance of the first to Mth speakers with respect to the first to Nth (N is an integer of 2 or more) positions that follow the A sound processing unit that performs a sound field formation process using a virtual sound source in the virtual space using the first to Nth virtual speakers;
Only including,
The sound processing unit
Based on the information on the position of the first speaker group among the first to Mth speakers and the information on the first position of the first virtual speaker among the first to Nth positions, By adjusting the volume balance of the first speaker group, the first virtual speaker among the first to Nth virtual speakers is set,
Based on the information on the position of the second speaker group among the first to Mth speakers and the information on the second position of the second virtual speaker among the first to Nth positions, By adjusting the volume balance of the second speaker group, the second virtual speaker among the first to Nth virtual speakers is set,
Based on the information on the position of the third speaker group among the first to Mth speakers and the information on the third position of the third virtual speaker among the first to Nth positions, The simulation system , wherein the third virtual speaker among the first to Nth virtual speakers is set by adjusting a volume balance of the third speaker group . In any of claims 3 to 7 ,
The sound processing unit
By adjusting the volume balance of the first to Nth virtual speakers based on the information of the first to Nth positions of the first to Nth virtual speakers and the information of the position of the virtual sound source. A simulation system characterized in that the sound field is formed by the virtual sound source. In any one of Claims 1 thru | or 8 .
A simulation system, wherein a movable range is set for the user in the real space, and the first to Mth speakers are arranged at positions corresponding to the movable range. In any one of Claims 1 thru | or 9 ,
The simulation system characterized by including the display process part which performs the display process of the image seen from the viewpoint of the said virtual user corresponding to the said user. In claim 10 ,
The display processing unit
A simulation system for performing processing for displaying an image on a head-mounted display device worn by the user. In claim 10 or 11 ,
The display processing unit
The simulation system characterized by performing the display process of the image which reproduces the sound generation condition by the said virtual sound source in the said virtual space. In any one of Claims 1 to 12 ,
The sound processing unit
The sound output from the i-th speaker is detected when the user's approach to the i-th speaker (1 ≦ i <M) of the first to M-th speakers in the real space is detected. A simulation system characterized by performing a process of adjusting at least one of volume, high frequency component, and reverberation. An input processing unit for acquiring information on the position and direction of the user moving in the real space in which the first to Mth speakers (M is an integer of 3 or more) are arranged;
A predetermined relative positional relationship with respect to the virtual user even when the position and direction of the virtual user corresponding to the user change in accordance with the change of the position and direction of the user in the real space. The first to Nth virtual speakers are set by adjusting the volume balance of the first to Mth speakers with respect to the first to Nth (N is an integer of 2 or more) positions that follow the As a sound processing unit that performs sound field formation processing using a virtual sound source in the virtual space using the first to Nth virtual speakers,
Make the computer work ,
The sound processing unit
A program for invalidating the settings of the first to Nth virtual speakers when it is determined that a predetermined situation has occurred . An input processing unit for acquiring information on the position and direction of the user moving in the real space in which the first to Mth speakers (M is an integer of 3 or more) are arranged;
A predetermined relative positional relationship with respect to the virtual user even when the position and direction of the virtual user corresponding to the user change in accordance with the change of the position and direction of the user in the real space. The first to Nth virtual speakers are set by adjusting the volume balance of the first to Mth speakers with respect to the first to Nth (N is an integer of 2 or more) positions that follow the As a sound processing unit that performs sound field formation processing using a virtual sound source in the virtual space using the first to Nth virtual speakers,
Make the computer work ,
The sound processing unit
By adjusting the volume balance of the first to Nth virtual speakers based on the information of the first to Nth positions of the first to Nth virtual speakers and the information of the position of the virtual sound source. A program for performing the process of forming the sound field by the virtual sound source . An input processing unit for acquiring information on the position and direction of the user moving in the real space in which the first to Mth speakers (M is an integer of 3 or more) are arranged;
A predetermined relative positional relationship with respect to the virtual user even when the position and direction of the virtual user corresponding to the user change in accordance with the change of the position and direction of the user in the real space. The first to Nth virtual speakers are set by adjusting the volume balance of the first to Mth speakers with respect to the first to Nth (N is an integer of 2 or more) positions that follow the As a sound processing unit that performs sound field formation processing using a virtual sound source in the virtual space using the first to Nth virtual speakers,
Make the computer work ,
The sound processing unit
Based on the information on the position of the first speaker group among the first to Mth speakers and the information on the first position of the first virtual speaker among the first to Nth positions, By adjusting the volume balance of the first speaker group, the first virtual speaker among the first to Nth virtual speakers is set,
Based on the information on the position of the second speaker group among the first to Mth speakers and the information on the second position of the second virtual speaker among the first to Nth positions, A program characterized in that the second virtual speaker among the first to Nth virtual speakers is set by adjusting the volume balance of the second speaker group . An input processing unit for acquiring information on the position and direction of the user moving in the real space in which the first to Mth speakers (M is an integer of 3 or more) are arranged;
A predetermined relative positional relationship with respect to the virtual user even when the position and direction of the virtual user corresponding to the user change in accordance with the change of the position and direction of the user in the real space. The first to Nth virtual speakers are set by adjusting the volume balance of the first to Mth speakers with respect to the first to Nth (N is an integer of 2 or more) positions that follow the As a sound processing unit that performs sound field formation processing using a virtual sound source in the virtual space using the first to Nth virtual speakers,
Make the computer work ,
The sound processing unit
Based on the information on the position of the first speaker group among the first to Mth speakers and the information on the first position of the first virtual speaker among the first to Nth positions, By adjusting the volume balance of the first speaker group, the first virtual speaker among the first to Nth virtual speakers is set,
Based on the information on the position of the second speaker group among the first to Mth speakers and the information on the second position of the second virtual speaker among the first to Nth positions, By adjusting the volume balance of the second speaker group, the second virtual speaker among the first to Nth virtual speakers is set,
Based on the information on the position of the third speaker group among the first to Mth speakers and the information on the third position of the third virtual speaker among the first to Nth positions, A program that sets the third virtual speaker among the first to Nth virtual speakers by adjusting a volume balance of the third speaker group .

Images