JP6515057B2 - Simulation system, simulation apparatus and program - Google Patents

Description

The present invention relates to a simulation system , a simulation apparatus, a program and the like.

  BACKGROUND ART Conventionally, a game system is known in which a spectator is made to appear in a soccer game or a baseball game, and the spectator cheers a team. As a prior art of such a game system, there exists a technique disclosed by patent document 1, for example. There is also known a game system in which a voice input unit is provided, and by recognizing a voice inputted by the voice input unit, an operation of a character is instructed to advance a game. As a prior art of such a game system, there exists a technique disclosed by patent document 2, for example.

JP, 2002-42164, A Japanese Patent Application Publication No. 2001-224851

  However, in the prior art of Patent Document 1, only the image of the spectator is displayed on the spectator seat in the game image, and a system in which the spectator responds to the user voice input by the user has not been proposed. Further, in the prior art of Patent Document 2, only the motion of the character can be instructed based on the user's voice input by the user, and the character does not respond to the user's voice with a response voice, a response tone, etc. .

  According to some aspects of the present invention, it is possible to provide a simulation system, a program and the like capable of outputting a response voice or response sound appropriate for the input of user voice.

  According to one aspect of the present invention, an input processing unit for acquiring information of user voice input by a user using a sound input device, a measurement processing unit for performing measurement processing of the user voice, and input of the user voice A sound processing unit that performs an output process of a response voice or a response sound different from the user voice, the measurement processing unit performs a measurement process of the volume and the length of the user voice, and the sound processing unit The present invention relates to a simulation system that performs output processing of the response sound or the response sound based on a result of measurement processing of volume and length of the user voice. The present invention also relates to a program that causes a computer to function as the above-described sections or a computer-readable information storage medium storing the program.

  According to one aspect of the present invention, when the user inputs a user voice using the sound input device, a response voice or response sound different from the user voice is output in response to the input of the user voice. That is, in response to the input of the user voice, the response voice or the response tone is output. In this case, in one aspect of the present invention, measurement processing of the volume and length of the input user voice is performed, and a response voice or response sound based on the result of the measurement processing is output. In this way, it is possible to reflect the result of the measurement processing of the volume and the length of the user voice on the response voice or the response voice outputted corresponding to the input of the user voice. Therefore, it is possible to provide a simulation system or the like capable of outputting an appropriate response voice or response sound to the input of the user voice.

  Further, in one aspect of the present invention, the sound processing unit may perform processing to make the response sound or the response sound different according to the volume and the length of the user sound.

  In this way, the response voice or the response tone becomes different depending on the volume and the length of the user voice, and it becomes possible to output a more appropriate response voice or response voice corresponding to the user voice.

  Further, in one aspect of the present invention, the measurement processing unit is configured such that the volume of the user's voice exceeds the i-th volume level (1 ≦ i ≦ N) of the first to Nth volume levels. And measuring the length up to the timing at which the sound volume level falls below the i-th sound level, and the sound processing unit performs processing to make the response sound or the response sound different according to the measured length. Good.

  In this way, the volume and the length of the user's voice can be measured by a process with a small load, so that the processing load can be reduced.

  Further, in one aspect of the present invention, the sound processing unit is configured to set a case where the volume of the user voice exceeds the i-th volume level, and the j-th (1 ≦ 1) of the first to N-th volume levels. The processing for differentiating the response sound or the response sound may be performed when the volume level of i <j ≦ N is exceeded.

  In this way, it is possible to make the output response voice or response sound different when the volume of the user voice exceeds the i-th volume level and the j-th volume level. It becomes possible to output an appropriate response voice or response tone by low-load processing.

  In one aspect of the present invention, the sound processing unit may perform processing for outputting the response sound or the response sound as sound or sound of a target of the user in a game.

  In this way, when the user inputs a user voice using the sound input device, a response voice or response sound to the input of the user voice can be output as the voice or sound of the user's target in the game.

  In one aspect of the present invention, the target may be a character of a spectator appearing in a game.

  In this way, the cheers and the like of the spectators appearing in the game can be output as a response sound or a response sound.

  Further, in one aspect of the present invention, the sound processing unit may perform the response voice or the response voice according to at least one of a type of the target, a positional relationship between the target and the user, and a gaze direction of the user with respect to the target. A process of changing the response sound may be performed.

  In this way, the response voice or the response sound changes according to the type of the target, the positional relationship with the target, and the direction of the user's line of sight with respect to the target, and a more appropriate response to user voice input. It becomes possible to output voice or response sound.

  In one aspect of the present invention, the measurement processing unit performs analysis processing of the feature amount of the user voice, and the sound processing unit performs the response according to a result of analysis processing of the feature amount of the user voice. A process of changing voice or the response sound may be performed.

  In this way, it is possible to output a response voice or response sound reflecting the feature of the user voice in response to the input of the user voice.

  Further, in one aspect of the present invention, the sound processing unit is a position and a direction of the sound input device held by the user, a position and a direction of the hand, and a position of a head mounted display worn by the user. Processing to change the response voice or the response sound according to at least one of the direction, the posture of the user, and the line of sight of the user, or the process of outputting the response voice or the response sound, or the response voice or A process of receiving the input of the user voice for outputting the response sound may be performed.

  In this way, the response voice or the response sound is selected according to the position and orientation of the sound input device, the position and orientation of the hand, the position and orientation of the head-mounted display device, the posture of the user, or the line of sight of the user. It is possible to make various changes, control the output timing of the response sound or the response sound, etc., and determine whether or not to receive the input of the user sound.

  In one aspect of the present invention, the sound processing unit may perform the process of changing the response sound or the response sound based on past play history information of the user.

  In this way, it is possible to output a response sound or response sound reflecting the player's past play history.

  In one aspect of the present invention, the sound processing unit further includes a sound data storage unit for storing sound data, and the sound processing unit selects sound data to be used from a plurality of sound data stored in the sound data storage unit. Alternatively, output processing of the response voice or the response sound may be performed by performing processing of combining a plurality of sound data.

  In this way, it is possible to output an appropriate response voice or response sound by selecting sound data from a plurality of sound data and combining sound data.

  In one aspect of the present invention, the input processing unit receives an input of the response voice or the user voice for causing the response sound to be output in a period other than an evaluation period in which an evaluation process on the user voice is performed. It is also good.

  In this way, it is possible to prevent the response voice or the input of the user voice for outputting the response voice from adversely affecting the evaluation process for the user voice.

  Further, according to another aspect of the present invention, there is provided an input processing unit for acquiring information of a user voice input by a user using a sound input device, and a response voice or response different from the user voice with respect to the user voice input. A sound processing unit that performs sound output processing, and the input processing unit is configured to output the response voice or the response sound in a period other than an evaluation period in which evaluation processing on the user voice is performed. It relates to a simulation system that accepts voice input. The present invention also relates to a program that causes a computer to function as the above-described sections or a computer-readable information storage medium storing the program.

  According to another aspect of the present invention, when the user inputs a user voice using the sound input device, a response voice or response sound different from the user voice is output in response to the input of the user voice. In this case, in one aspect of the present invention, the input of the user voice that causes such a response voice or the response sound to be output is accepted in a period other than the evaluation period in which the evaluation process on the user voice is performed. Therefore, it is possible to prevent a situation in which the input of the user voice adversely affects the evaluation process on the user voice.

FIG. 1 is a block diagram showing a configuration example of a simulation system of the present embodiment. FIGS. 2A and 2B show examples of the HMD used in the present embodiment. FIGS. 3A and 3B show other examples of the HMD used in the present embodiment. Explanatory drawing of the private room which is a play area. Explanatory drawing of the private room which is a play area. The example of the game image produced | generated by this embodiment. The example of the game image produced | generated by this embodiment. Explanatory drawing about the movement of the user (virtual user) in a stage. Explanatory drawing of the method made to respond with response voice and response sound with respect to the input of a user voice. Explanatory drawing of the method of making response sound and response sound different according to the volume and length of user's voice. FIGS. 11A and 11B are specific examples of processing for making response voices and response sounds different according to the volume and length of user voices. Explanatory drawing about an output timing of response sound and response sound. 13 (A) and 13 (B) are explanatory views of a method for changing response voice and response sound according to the type of audience as a target. FIGS. 14 (A) and 14 (B) are explanatory views of a method for changing the response voice and the response sound according to the positional relationship between the audience and the user. Explanatory drawing of the method of changing response sound and response sound according to the user's gaze direction with respect to an audience. 16 (A) and 16 (B) are explanatory views of a method of changing response voice and response sound according to the result of analysis processing of the feature amount of the user voice. Explanatory drawing of the method of changing a response audio | voice and response sound according to the position of microphone (hand), direction, position of HMD, direction, attitude | position of a user, and a user's gaze. 18 (A) and 18 (B) are explanatory views of a method of changing the response sound and the response sound according to the play history of the user. 19 (A) and 19 (B) are explanatory views of response sound and response sound output processing by sound data selection processing and sound data combination processing. 20 (A) and 20 (B) are explanatory diagrams about evaluation processing of a user's song. Explanatory drawing of the method of receiving the user's voice input for C & R in periods other than the evaluation period of a song. 3 is a flowchart showing an example of processing of the present embodiment.

  Hereinafter, the present embodiment will be described. Note that the embodiments described below do not unduly limit the contents of the present invention described in the claims. Further, not all of the configurations described in the present embodiment are necessarily essential configuration requirements of the present invention.

1. Configuration FIG. 1 shows a configuration example of a simulation system (game system, video display system, simulation apparatus) of the present embodiment. Note that the simulation system of the present embodiment is not limited to the configuration of FIG. 1, and various modifications may be made such as omitting some of the components (each part) or adding other components.

  The input device 160 is a device for the user to input various input information. The input device 160 can include a sound input device 161 and a vibrating device 164. Further, the input device 160 may have a function of a game controller for the user to input game operation information. The game controller is realized by, for example, an operation button, a direction indication key, a joystick or a lever. In this case, the game controller and the sound input device 161 may be realized by an integral housing or may be realized by a separate housing.

  The sound input device 161 is a device for the user to input sound information. The sound input device 161 can, for example, input user voices such as the user's singing voice, calling voice and squeal. The sound input device 161 can be realized by, for example, the microphone 162 described with reference to FIG. The shape of the sound input device 161 is not limited to the microphone 162 having a shape as shown in FIG. 2A, and various types of microphones such as a headset microphone having a headband and a small microphone can be used. . The sound input device 161 may be a sound input device (pickup microphone or the like) in a musical instrument or a device imitating a musical instrument. Musical instruments include string instruments (guitars), percussion instruments (drums, drums), and keyboard instruments (pianos, keyboards).

  The vibrating device 164 (vibration generating unit) is a device that generates vibration for warning or the like, and is realized by, for example, a vibrating motor (vibrator) or the like. The vibration motor generates vibration, for example, by rotating an eccentric weight. Specifically, eccentric weights are attached to both ends of the drive shaft so that the motor itself swings. The vibrating device 164 is not limited to the vibrating motor, and may be realized by, for example, a piezo element or the like.

  The storage unit 170 stores various types of information. The storage unit 170 functions as a work area of the processing unit 100, the communication unit 196, and the like. The game program and game data necessary for executing the game program are held in the storage unit 170. The function of the storage unit 170 can be realized by a semiconductor memory (DRAM, VRAM), an HDD (hard disk drive), an SDD, an optical disk device, or the like. The storage unit 170 includes a space information storage unit 172, a music information storage unit 174, a sound data storage unit 175, a parameter storage unit 176, and a drawing buffer 178.

  An information storage medium 180 (a computer readable medium) stores programs, data, etc., and its function can be realized by an optical disc (DVD, BD, CD), HDD, semiconductor memory (ROM), etc. . The processing unit 100 performs various processes of the present embodiment based on a program (data) stored in the information storage medium 180. That is, a program for causing a computer (an apparatus including an input device, a processing unit, a storage unit, and an output unit) to function as each unit of the present embodiment in the information storage medium 180 (a program for causing the computer to execute processing of each unit) Is stored.

  The head mounted display 200 (HMD) is a device that is mounted on the head of the user and displays an image in front of the user's eyes. The HMD 200 is preferably non-transmissive, but may be transmissive. The HMD 200 may be a so-called glasses-type HMD.

  The HMD 200 includes a sensor unit 210, a display unit 220, and a processing unit 240. In addition, the modification implementation which provides a light emitting element in HMD200 is also possible. The sensor unit 210 is for implementing tracking processing such as head tracking, for example. For example, the position and the direction of the HMD 200 are specified by tracking processing using the sensor unit 210. By specifying the position and direction of the HMD 200, it is possible to specify the position and direction of the user. The position and direction of the virtual user (virtual player) in the virtual space corresponding to the user (player) are specified by the position and direction of the user. The position and direction of the user are, for example, the viewpoint position and the gaze direction of the user. The position and the direction of the virtual user are, for example, the viewpoint position and the sight line direction of the virtual user.

  Various methods can be employed as the tracking method. In the first tracking method, which is an example of the tracking method, as will be described in detail with reference to FIGS. 2A and 2B described later, a plurality of light receiving elements (photodiodes or the like) are provided as the sensor unit 210. The position of the HMD 200 (user's head) in the three-dimensional space of the real world by receiving light (laser etc.) from a light emitting element (LED etc.) provided outside by these plural light receiving elements, In the second tracking method for specifying the direction, a plurality of light emitting elements (LEDs) are provided in the HMD 200 as described in detail in FIG. 3A and FIG. 3B described later. And the position and direction of HMD200 are pinpointed by imaging the light from these several light emitting elements with the imaging part provided outside. In the third tracking method, a motion sensor is provided as the sensor unit 210, and the position and direction of the HMD 200 are specified using this motion sensor. The motion sensor can be realized by, for example, an acceleration sensor or a gyro sensor. For example, by using a six-axis motion sensor using a three-axis acceleration sensor and a three-axis gyro sensor, it is possible to specify the position and direction of the HMD 200 in the three-dimensional space of the real world. The position and the direction of the HMD 200 may be specified by a combination of the first tracking method and the second tracking method, or a combination of the first tracking method and the third tracking method.

  The display unit 220 of the HMD 200 can be realized by, for example, a liquid crystal display (LCD) or an organic EL display. For example, the HMD 200 is provided with a first display disposed in front of the user's left eye and a second display disposed in front of the right eye as the display unit 220. For example, stereoscopic display is enabled. ing. When stereoscopic display is performed, for example, a left-eye image and a right-eye image having different parallaxes may be generated, the left-eye image may be displayed on the first display, and the right-eye image may be displayed on the second display.

  The processing unit 240 of the HMD 200 performs various processes necessary for the HMD 200. For example, the processing unit 240 performs control processing of the sensor unit 210, display control processing of the display unit 220, and the like. In addition, the processing unit 240 may perform three-dimensional sound (three-dimensional sound) processing to realize three-dimensional reproduction of the direction, distance, and the spread of sound.

  The sound output unit 192 outputs the sound generated according to the present embodiment, and can be realized by, for example, a speaker or headphones.

  An I / F (interface) unit 194 performs interface processing with the portable information storage medium 195, and the function thereof can be realized by an ASIC or the like for I / F processing. The portable information storage medium 195 is a storage device for storing various types of information by the user, and is a storage device that retains storage of the information even when the power is not supplied. The portable information storage medium 195 can be realized by an IC card (memory card), a USB memory, a magnetic card, or the like.

  The communication unit 196 communicates with the outside (another device) via a wired or wireless network, and the function thereof is hardware such as a communication ASIC or a communication processor, or communication firmware. Can be realized by

  The program (data) for causing the computer to function as each part of the present embodiment is from the information storage medium possessed by the server (host device) via the network and communication unit 196 via the information storage medium 180 (or storage unit 170, auxiliary storage) It may be distributed to the device 194). The use of an information storage medium by such a server (host device) can also be included within the scope of the present invention.

  The processing unit 100 (processor) performs game processing and game score calculation processing based on input information from the input device 160, tracking information in the HMD 200 (position, direction, viewpoint position, gaze direction of the HMD), programs, and the like. , Display processing or sound processing.

  Each process (each function) of this embodiment which each part of processing part 100 performs is realizable by a processor (processor containing hardware). For example, each process of the present embodiment can be realized by a processor that operates based on information such as a program and a memory that stores information such as a program. In the processor, for example, the function of each unit may be realized by separate hardware, or the function of each unit may be realized by integral hardware. The processor may be, for example, a CPU (Central Processing Unit). However, the processor is not limited to the CPU, and various processors such as a graphics processing unit (GPU) or a digital processing unit (DSP) can be used. The processor may also be a hardware circuit with an ASIC. The memory (storage unit 170) may be a semiconductor memory such as SRAM or DRAM, or may be a register. Alternatively, it may be a magnetic storage device such as a hard disk drive (HDD) or an optical storage device such as an optical disk drive. For example, the memory stores an instruction readable by a computer, and the processing (function) of each unit of the processing unit 100 is realized by the instruction being executed by the processor. The instructions here may be an instruction set that configures a program, or may be instructions that instruct an operation to a hardware circuit of a processor.

  The processing unit 100 includes an input processing unit 102, an arithmetic processing unit 110, and an output processing unit 140. The arithmetic processing unit 110 includes a game processing unit 111, a game score calculation unit 118, a measurement processing unit 119, a display processing unit 120, and a sound processing unit 130. As described above, each process of the present embodiment executed by these units can be realized by the processor (or processor and memory). Note that various modifications may be made such as omitting some of these components (each part) or adding other components.

  The input processing unit 102 (program module for input processing) performs processing for receiving input information and tracking information, processing for reading information from the storage unit 170, and processing for receiving information via the communication unit 196 as input processing. . For example, the input processing unit 102 acquires input information input by the user using the input device 160 or tracking information (such as information on the position, direction, or line of sight of the user) detected by the sensor unit 210 of the HMD 200 or the like. A process of reading out information designated by the read command from the storage unit 170 and a process of receiving information from an external apparatus (server or the like) via a network are performed as input processes. Here, the reception process is a process of instructing the communication unit 196 to receive information, or a process of acquiring the information received by the communication unit 196 and writing the information in the storage unit 170.

  The arithmetic processing unit 110 performs various arithmetic processing. For example, calculation processing such as game processing, game score calculation processing, display processing, or sound processing is performed.

  The game processing unit 111 (program module for game processing) performs various game processing for the user to play a game. The game processing unit 111 includes a game progress processing unit 112, an evaluation processing unit 113, a character processing unit 114, a parameter processing unit 115, an object space setting unit 116, and a virtual camera control unit 117.

  The game progress processing unit 112 performs a process of starting a game when the game start condition is satisfied, a process of advancing the game, or a process of ending the game when the game end condition is satisfied. The evaluation processing unit 113 performs an evaluation process of the user's game play. For example, evaluation processing of the user's performance in the music game and the game operation is performed. The music information used for the music game is stored in the music information storage unit 174.

  The character processing unit 114 performs various processes related to the character. For example, processing of moving a character in an object space (virtual space, game space) or processing of moving a character is performed. For example, processing for moving a character can be realized by motion processing (motion reproduction and the like) using motion data. The parameter processing unit 115 performs arithmetic processing of various parameters (game parameters) used for the game. For example, processing is performed to increase or decrease the value of the parameter. The parameter information is stored in the parameter storage unit 176.

  The object space setting unit 116 performs setting processing of an object space (virtual space in a broad sense) in which a plurality of objects are arranged. For example, various objects (polygons, free-form surfaces, subdivision surfaces, etc.) representing displayed objects such as characters (people, animals, robots etc.), maps (terrain), buildings, audience seats, courses (roads), trees, walls, water surfaces etc. Processing of arranging and setting objects in the object space). That is, the position and rotation angle (equivalent to direction and direction) of the object in the world coordinate system are determined, and the rotation angle (rotation angle around X, Y, Z axis) is determined at that position (X, Y, Z) Place an object. Specifically, the space information storage unit 172 of the storage unit 170 stores, as space information, information such as positions of plural objects (part objects) in the object space and rotation angles (directions). The object space setting unit 116 performs, for example, a process of updating the space information for each frame.

  The virtual camera control unit 117 performs control processing of a virtual camera (viewpoint, reference virtual camera) for generating an image viewed from a given (arbitrary) viewpoint in the object space. Specifically, processing for controlling the position (X, Y, Z) or rotation angle (rotation angle around the X, Y, Z axes) of the virtual camera (processing for controlling the viewpoint position, sight direction or angle of view) I do. This virtual camera corresponds to the viewpoint of the user. In the case of stereoscopic display, a first viewpoint for the left eye (first virtual camera for the left eye) and a second viewpoint for the right eye (the second virtual camera for the right eye) are set.

  The game score calculation unit 118 performs processing to calculate the game score of the user. For example, calculation processing of game scores such as points, points, etc. obtained by the user's game play is performed.

  The measurement processing unit 119 performs various measurement processing on sound. For example, measurement processing of the user voice input by the sound input device 161 is performed.

  The display processing unit 120 (program module for display processing) performs display processing of a game image. For example, drawing processing is performed based on the results of various processing (game processing, simulation processing) performed by the processing unit 100, and thereby an image is generated and displayed on the display unit 220 of the HMD 200. Specifically, geometry processing such as coordinate conversion (world coordinate conversion, camera coordinate conversion), clipping processing, perspective conversion, or light source processing is performed, and based on the processing result, drawing data (the position of the vertex of the primitive surface) Coordinates, texture coordinates, color data, normal vectors or α values, etc. are created. Then, based on the drawing data (primitive plane data), the object (one or a plurality of primitive planes) after perspective transformation (after geometry processing) is converted into image information in pixel units such as the drawing buffer 178 (frame buffer, work buffer, etc.) Draw in a buffer that can be stored. As a result, an image viewed from a virtual camera (given viewpoints; first and second viewpoints for the left and right eyes) in the object space is generated. The drawing processing performed by the display processing unit 120 can be realized by vertex shader processing, pixel shader processing, or the like.

  The sound processing unit 130 (program module for sound processing) performs sound processing based on the results of various processing performed by the processing unit 100. Specifically, game sound such as music (music, BGM), sound effect, or voice is generated, and the game sound is output to the sound output unit 192. Sound data to be output (reproduced) during the game is stored in the sound data storage unit 175. A part of the sound processing of the sound processing unit 130 (for example, three-dimensional sound processing) may be realized by the processing unit 240 of the HMD 200.

  The output processing unit 140 performs output processing of various types of information. For example, the output processing unit 140 performs a process of writing information in the storage unit 170 and a process of transmitting information via the communication unit 196 as an output process. For example, the output processing unit 140 performs a process of writing information designated by the write command in the storage unit 170, and a process of transmitting information to an external device (such as a server) via the network. The transmission process is a process of instructing the communication unit 196 to transmit information or instructing the communication unit 196 to transmit information.

  For example, in the present embodiment, the game processing unit 111 performs processing of a game played by the user in a virtual space (game space) in which a plurality of objects are arranged. For example, in a virtual space which is an object space, a plurality of objects such as characters are arranged, and the game processing unit 111 performs various game processing (game progress processing, character processing) for realizing the game in this virtual space. , Object space setting processing, or virtual camera control processing). Then, the display processing unit 120 causes the game image viewed from a given viewpoint (first and second viewpoints for left and right eyes) in the virtual space to be displayed on the display unit 220 (first and second displays) of the HMD 200. Perform processing to display. That is, in the object space which is a virtual space, a process of displaying a game image viewed from the viewpoint (virtual camera) of the virtual user (user) is performed.

  For example, a user in the real world wearing the HMD 200 moves, changes the moving direction, shakes the head or squats in the play area of the individual room in FIGS. 4 and 5 described later, and the position and direction are When it changes, the position and the direction of the virtual user in the virtual space corresponding to the user also change. Since the position and direction of the user in the real world can be specified by the tracking process of the HMD 200, the position and direction of the virtual user in the virtual space can also be specified. The position and the direction of the user and the virtual user are, for example, the viewpoint position and the gaze direction of the user and the virtual user. When the virtual user is not displayed as a character, the display image of the HMD 200 is an image of a first person viewpoint, and when the virtual user is displayed as a character, the display image is an image of a third person viewpoint.

  The play area (action area) is, for example, an area set as a range in which the user can move, and is an area defined in advance as an area (field, space) in which the user performs an action such as game play, for example. . The play area (a movable range in a broad sense) is an area including a range in which tracking of position information of the user, for example, is possible. The play area may be, for example, an area surrounded by a wall, but may be an open space area.

  Then, in the present embodiment, the input processing unit 102 acquires information of the user voice input by the user using the sound input device 161. As an example, A / D conversion is performed on an analog voice signal input by the sound input device 161, and acquired as digital user voice information.

  The measurement processing unit 119 performs measurement processing of the acquired user voice. For example, various measurement processes are performed on the waveform of the user voice. Further, the sound processing unit 130 performs an output process of a response sound or a response sound different from the user's voice in response to the input of the user's voice. For example, processing for selecting sound data from a plurality of sound data (sound data files) stored in the sound data storage unit 175, processing for combining sound data, etc. It is generated (composed) and output by a sound output unit 192 (speaker or the like).

  And in this embodiment, the measurement process of the volume and length of a user's voice is performed. For example, the volume of the user voice is measured by measuring whether or not the signal waveform of the user voice exceeds a predetermined volume level (sound pressure). Further, by measuring the length at which the signal waveform of the user voice exceeds a predetermined volume level, the length of the user voice (sound measure) is measured. Then, the sound processing unit 130 performs an output process of the response voice or the response sound based on the result of the measurement process of the volume and the length of the user voice. The response sound processing unit 132 performs output processing or change processing of the response sound or the response sound.

  For example, when the result of the measurement process of the volume and the length of the user voice is the first measurement result, the sound processing unit 130 may respond to the input of the user voice in response to the first response voice or the first response voice. A response sound is output to the sound output unit 192. On the other hand, when the result of the measurement process of the volume and the length of the user voice is the second measurement result, the second response voice or the second response sound is output corresponding to the input of the user voice. It is output to the part 192. Similarly, when the result of the measurement process of the volume and length of the user voice is the measurement result of the Kth (K is an integer of 2 or more), the Kth response voice or the corresponding response to the input of the user voice The Kth response sound is output to the sound output unit 192.

  For example, the sound processing unit 130 performs processing to make the response sound or the response sound different according to the volume and the length of the user sound. That is, the response voice or the response tone to be output in response to the input of the user voice is made to differ according to the volume and the length of the user voice. For example, the volume of the user voice is LV (m), and the length of the user voice is TL (n) (m and n are integers of 2 or more). In this case, for example, a response voice or a response tone as represented by RS (m, n) = RS (LV (m), TL (n)) is output. For example, when the volume and length are LV (1) and TL (1), RS (1, 1) is output as a response voice or response sound, and the volume and length are LV (2) and TL (1). In the case of), RS (2, 1) is output as a response voice or response tone. And RS (1, 1) and RS (2, 1) are different response voices or response tones. When the volume and length are LV (1) and TL (2), RS (1, 2) is output as a response voice or response sound, and the volume and length are LV (2) and TL (2). In the case of), RS (2, 2) is output as a response voice or response tone. And RS (1, 2) and RS (2, 2) are different response voices or response tones. Further, RS (1, 1), RS (2, 1), RS (1, 2), and RS (2, 2) are also mutually different response voices or response tones.

  The response sound or response sound being different means that, for example, the type of response sound or response sound (sound data, waveform, attribute, etc.), volume (intensity), height (pitch), timbre, etc. are different.

  More specifically, the measurement processing unit 119 sets the i-th volume from the timing when the volume of the user's voice exceeds the i-th volume level (1 ≦ i ≦ N) of the first to N-th volume levels. Measure the length to the timing below the level. For example, in the signal waveform of the user voice, the volume rises, and the timing exceeding the i-th volume level is taken as TM1. Thereafter, the volume of the signal waveform of the user's voice decreases, and the timing below the ith volume level is set to TM2. In this case, TM2-TM1 is measured as user voice length T = TM2-TM1. Then, the sound processing unit 130 performs processing to make the response sound or the response sound different according to the measured length T = TM2-TM1. That is, the response voice or the response tone is made different between when the measured length is T = T1 and T = T2 (T1 and T2 are different lengths).

  Further, the sound processing unit 130 sets the jth (1 ≦ i <j ≦ N) volume level of the first to Nth volume levels when the volume of the user voice exceeds the i-th volume level. In the case of exceeding, the processing of making the response voice or the response tone different is performed. For example, it is assumed that the first response sound or the first response sound is output when the volume of the user voice exceeds the i-th volume. In this case, when the volume of the user voice is further increased and exceeds the j-th volume level, the second response voice or the second response voice different from the first response voice or the first response voice is selected. Output. In this case, both the first response sound or the first response sound and the second response sound or the second response sound may be output, or only the second response sound or the second response sound may be output. You may output it. Also, when the response voice or response voice is cheering, scream or applause, the second response voice or second response voice is more excited and frenzy than the first response voice or the first response voice. Cheers, screams or applauses that have increased in frequency.

  Further, the sound processing unit 130 performs a process of outputting the response sound or the response sound as a sound or a sound of the target of the user in the game. Here, the target is a target of the user's game play, and is, for example, a character appearing in the game. In this case, the response sound is a sound uttered by the character, and the response sound is an operation sound or the like generated by the motion of the character. The target may be an object other than a character appearing in the game (an object which does not move).

  For example, the target is a character of a spectator appearing in the game. For example, it is a character of a spectator in a music game, a performance game or a sports game. In this case, the response voice is a voice such as cheers, screams and calls, and the response tone is a clap, a whistling, a stepping tone and the like. Alternatively, the target character may be an opponent character in a match game, a cooperative character of the user, or the like.

  Further, the sound processing unit 130 performs processing of changing the response voice or the response sound according to at least one of the type of target, the positional relationship between the target and the user, and the direction of the user's gaze with respect to the target. For example, when a target response voice or response tone is output in response to user voice input, the response voice or response tone is differentiated according to the type of the target. The type of target is, for example, a target (character) according to gender (male, female), age group (child, adult, etc.), personality (enthusiastic type, coolness type, etc.) or attribute (fire, soil, water, etc.) Difference in the classification of The positional relationship between the target and the user represents, for example, the relationship between at least one of the position and orientation of the target and at least one of the position and orientation of the user. The gaze direction of the user with respect to the target is, for example, what direction the gaze direction of the user is with respect to the position of the target.

  The target may represent a place in a virtual space (object space) (for example, a seat in a spectator seat). In this case, attribute data (control data) is stored in association with each place (target) in the virtual space, and it is determined that the user's gaze direction is directed to the place (target). The process of changing the response voice or the response sound is performed using the attribute data (data representing the attribute of the place) associated with the place. This attribute data can be stored, for example, in the space information storage unit 172.

  Further, the process of changing the response voice or the response sound is, for example, a process of processing the response voice or the response sound by applying an effect, a process of synthesizing a different voice or sound for the response voice or the response voice, or a response voice or It is a process of changing the combination, type, and number of sound data for generating a response sound.

  The measurement processing unit 119 also performs analysis processing of the feature amount of the user voice. Then, the sound processing unit 130 performs processing of changing the response sound or the response sound according to the result of the analysis processing of the feature amount of the user sound. The feature amount of the user voice is, for example, a feature amount related to the size, the height, the timbre, or the type of the sound of the user voice. The analysis processing of the feature amount of the user voice can be realized by, for example, frequency analysis processing. For example, frequency analysis is performed on user speech data for each predetermined interval (several tens of ms) to obtain an acoustic spectrum that is frequency component information of the speech, and analysis processing of the feature amount of the user speech is performed using this acoustic spectrum as a clue. . Then, the response voice or the response sound is changed according to the result of the analysis processing. For example, when the gender or age group of the user is determined from the result of analysis processing of the feature amount of the user voice, the response voice or the response sound is changed according to the determined sex or age group. For example, when the user is a male, a male response voice or response sound is output, and when the user is a female, a female response voice or response sound is output. When the user is an adult, a response voice or response sound for adults is output, and when the user is a child, a response voice or response sound for children is output.

  Further, the sound processing unit 130 is at least one of the position and direction (or the position and direction of the hand) of the sound input device 161 held by the user, the position and direction of the HMD 200 worn by the user, the posture of the user, and the line of sight of the user. Depending on one, processing to change response voice or response sound, or processing to output response voice or response sound, or processing to receive input of user voice for outputting response voice or response sound is performed. For example, the sound processing unit 130 changes or responds to the response voice or the response tone depending on what position the sound input device 161 (or a hand holding the same) is at or in which direction it is facing. It performs output control (control of output timing, etc.) of voice or response sound, and determines whether or not to receive an input of user voice. Also, depending on the position of the HMD 200 or in which direction it is facing, the response voice or the response sound is changed, the output control of the response voice or the response sound is performed, or the user voice is input. Determine whether to accept. Alternatively, whether the state of the user's posture or line of sight is detected and the response voice or the response sound is changed according to the detection result, the output control of the response voice or the response sound is performed, or the input of the user voice is accepted To judge.

  Further, the sound processing unit 130 performs processing of changing the response sound or the response sound based on the past play history information of the user. For example, information of the past play history of the user is stored in the storage unit 170. Then, when the user inputs a user voice through the sound input device 161, the information of the play history is read from the storage unit 170, and the response voice or the response sound is changed according to the information of the play history. For example, the response voice or response sound to be output is made different depending on whether the user playing the game frequently inputs the user voice and the user rarely playing the game inputs the user voice. .

  The simulation system of the present embodiment also includes a sound data storage unit 175 that stores sound data as shown in FIG. The sound data storage unit 175 stores a plurality of sound data (sound data files) for outputting a response sound or a response sound. In this case, the sound processing unit 130 performs a process of selecting sound data to be used from a plurality of sound data stored in the sound data storage unit 175 or a process of combining a plurality of sound data, Perform response sound output processing. For example, the process of changing the response voice or the response sound (process of changing) is realized by changing the sound data to be selected from the plurality of sound data, or by changing the sound data combination.

  Further, the input processing unit 102 receives an input of a user voice for outputting a response voice or a response sound in a period other than the evaluation period in which the evaluation process on the user voice is performed. The evaluation process for the user voice is an evaluation process for singing power when the user voice is a singing voice. And the evaluation period in which the evaluation process with respect to singing ability is performed is a period during singing, for example, and periods other than an evaluation period are a period of a prelude, a period of an interlude, etc., for example. It is not desirable for a response voice or response sound to be output in response to the user's singing while singing. Therefore, the response sound or the response sound is not output during the singing period, and the response sound or the response sound is output during the prelude period or the interlude period other than the singing period.

2. Method of this Embodiment Next, the method of this embodiment will be specifically described. In the following, the case where the game to which the method of the present embodiment is applied is a music game (a game on a live stage, a karaoke game, etc.) for playing a song will be mainly described as an example. However, the game to which the method of this embodiment is applied is not limited to this, and music playing an instrument such as a stringed instrument (guitar etc.), percussion instrument (drum, drum etc.) or keyboard instrument (keyboard, piano) etc. It may be a game (a game in which the player competes in rhythm and performance). In addition, the method according to the present embodiment includes a communication game with a heterosexual character (human relationship simulation game), a conversational game (talk battle game, interplay game), a battle game, an RPG game, a robot game, a card game, etc. The present invention is also applicable to various games such as sports games and action games, and reproduction of video content and music content.

2.1 HMD, Play Area FIG. 2A shows an example of the HMD 200 used in the simulation system of this embodiment. As shown in FIG. 2A, the HMD 200 is provided with a plurality of light receiving elements 201, 202, and 203 (photodiodes). The light receiving elements 201 and 202 are provided on the front side of the HMD 200, and the light receiving element 203 is provided on the right side of the HMD 200. A light receiving element (not shown) is also provided on the left side surface, the upper surface, etc. of the HMD.

  The user PL also has input devices 160-1 and 160-2 with the left hand and the right hand. Similar to the HMD 200, the input devices 160-1 and 160-2 are provided with a plurality of light receiving elements (not shown). Further, the input device 160-1 is provided with a microphone 162 (in a broad sense, a sound input device), and the user PL sings the mouth to the microphone 162 in a song playing game. The input devices 160-1 and 160-2 also function as game controllers, and are provided with operation buttons, direction instruction keys, and the like (not shown). The number of input devices 160 held by the user may be one.

  Further, the headband 260 and the like are provided in the HMD 200, and the user PL can wear the HMD 200 on the head stably with a better feeling of wearing. Then, the user PL operates the input devices 160-1 and 160-2 functioning as a game controller, and performs operation of swinging or swinging the head to input operation information and enjoy game play. The twisting operation and the swinging operation can be detected by the sensor unit 210 or the like of the HMD 200.

  As shown in FIG. 2 (B), base stations 280 and 284 are installed in the play area of the user PL. The base station 280 is provided with light emitting elements 281, 282, and the base station 284 is provided with light emitting elements 285, 286. The light emitting elements 281, 282, 285, 286 are realized by, for example, LEDs emitting laser (infrared laser etc.). The base station 280, 284 radiates, for example, a laser radially using the light emitting elements 281, 282, 285, 286. When the light receiving elements 201 to 203 and the like provided in the HMD 200 of FIG. 2A receive the lasers from the base stations 280 and 284, tracking of the HMD 200 is realized, and the head position and direction of the user PL It becomes possible to detect (the position and direction of the user in a broad sense). Also, when light receiving elements (not shown) provided in the input devices 160-1 and 160-2 receive the lasers from the base stations 280 and 284, tracking of the input devices 160-1 and 160-2 is realized, Positions and directions of the input devices 160-1 and 160-2 can be detected. Thus, for example, an image of a microphone corresponding to the input device 160-1 can be displayed on a game image.

  Another example of the HMD 200 is shown in FIG. In FIG. 3A, a plurality of light emitting elements 231 to 236 are provided for the HMD 200. These light emitting elements 231 to 236 are realized by, for example, LEDs. The light emitting elements 231 to 234 are provided on the front side of the HMD 200, and the light emitting element 235 and the light emitting element 236 (not shown) are provided on the back side. The light emitting elements 231 to 236 emit (emit) light in a visible light band, for example. Specifically, the light emitting elements 231 to 236 emit light of different colors. Then, the imaging unit 150 illustrated in FIG. 3B is installed on the front side of the user PL, and the imaging unit 150 images the light of the light emitting elements 231 to 236. That is, spot lights of these light emitting elements 231 to 236 appear on the captured image of the imaging unit 150. Then, by performing image processing of this captured image, tracking of the head (HMD) of the user PL is realized. That is, the three-dimensional position and the direction (the position and direction of the user) of the head of the user PL are detected.

  For example, as shown in FIG. 3B, the imaging unit 150 is provided with first and second cameras 151 and 152. The first and second cameras 151 and 152 of these first and second cameras 151 and 152 are provided. By using the captured image, the position or the like of the user PL in the depth direction of the head can be detected. Further, based on motion detection information of a motion sensor provided in the HMD 200, it is also possible to detect the rotation angle (gaze line) of the head of the user PL. Therefore, by using such an HMD 200, when the user PL faces in any direction of all 360 degrees of the surrounding, an image (user's image) in the corresponding virtual space (virtual three-dimensional space) It becomes possible to display on the display unit 220 of the HMD 200 an image viewed from the virtual camera corresponding to the viewpoint. In addition, you may use LED of not infrared rays but visible light as the light emitting elements 231-236. Further, the position, the movement, and the like of the user's head may be detected by another method such as using a depth camera.

  Note that the method of tracking processing for detecting the viewpoint position of the user and the gaze direction (the position and direction of the user) is not limited to the method described in FIGS. 2 (A) to 3 (B). For example, a tracking process may be realized with the HMD 200 alone using a motion sensor or the like provided in the HMD 200. That is, tracking processing is realized without providing an external device such as the base stations 280 and 284 in FIG. 2B and the imaging unit 150 in FIG. 3B. Alternatively, viewpoint information and the like such as the viewpoint position of the user and the gaze direction may be detected by various viewpoint tracking methods such as known eye tracking, face tracking, or head tracking.

  4 and 5 show an example of a play area in which the game of the present embodiment is realized. This play area is realized by a box-like soundproof private room. As shown in FIGS. 4 and 5, the box compartment has walls 301, 302, 303, 304, a ceiling 305, and a door 306. Soundproofing materials 311, 312, 313, 314, and 315, which also function as cushioning materials, are provided inside the walls 301, 302, 303, and 304, and the ceiling 305. In the ceiling 305, the above-mentioned base stations 280 and 284 and lighting fixtures 290 and 292 are installed.

  Front speakers 330 and 331 and a center speaker 332 are installed on the front side of the user PL, and rear speakers 333 and 334 and a woofer 335 are installed on the rear side. Surround sound is realized by these speakers. The winding device 50 is accommodated in the accommodation box in which the woofer 335 is accommodated. The winding device 50 has a rotating reel 62, and the cable 20 is taken up by the rotating reel 62 through a cable passage port 52 provided in a storage box (shelf).

  The user PL opens the door 306 of FIG. 5, enters the compartment, and plays the game. The space of this individual room becomes the play area (play space, real space) of the user PL. Then, as shown in FIG. 5, an area AR assumed as a movement range (movable range) of the user PL is set in the play area of the boxed room. In this area AR, tracking of the position and direction (viewpoint position and gaze direction) of the user PL using the stations 280 and 284 is possible. On the other hand, in a position beyond the boundary BD of the area AR, reliable tracking can not be realized. In addition, if the user PL crosses the boundary BD of the area AR, there is a possibility that the wall 301, 302, 303, 304 may collide, which is not desirable in terms of safety. The setting of the area AR can be set, for example, by the initialization setting of the game apparatus.

  Then, as shown in FIG. 4 and FIG. 5, the user PL wears the waist belt 30. A storage portion 32 is attached to the waist belt 30, and a relay point RP of the cable 20 is provided in the storage portion 32. The cable 20 is wound by the winding device 50 from the HMD 200 via the relay point RP. The point between the cable portion 21 and the cable portion 22 is the relay point RP. The cable 20 is provided with a stopper 26 for loosening the cable 20 when the user PL is standing at the reference position.

2.2 Outline of Game Next, an outline of a game realized by the method of the present embodiment will be described. The game realized by the present embodiment is a music game in which a vocal performance is performed while becoming a vocal of a band in the presence like a real live stage. The user can obtain an intense feeling of exhilarating sensation from his fans while feeling the sensation of exhilaration that he sings in front of a large audience. The HMD and high-output surround speakers give you a sense of presence, as if you were singing on a real live stage and being surrounded by your fans.

  The audience around the stage interactively returns loud cheers and various actions in response to the user's vocal performance and stage actions. In front of a crowd of spectators who fill the hall, including the front row fans who can see facial expressions on a live stage with a sense of realism, such as standing on the spot by HMD, with their band members playing their own Meet the audience's expectations by singing and performing live performances.

  Users make room reservation and play settings in the reception space provided in the common space, and as shown in Fig. 4 and Fig. 5, they can experience live appearances in a safe private room with a cushioning material (soundproofing material) affixed. Enjoy.

  The user selects the concert appearance mode in the play setting before stage appearance. Then, select the song to sing, and select the appearance stage. Then, the devices such as the HMD 200 and the input devices 160-1 and 160-2 described with reference to FIGS. 2A and 2B and the like, and the waist belt 30 are attached. The operator of the store explains the notes and the like, and assists the installation and adjustment of the user's device and the like. The operator performs the calibration (initialization) of the room space as the play area in advance.

  The input device 160-1 in FIG. 2A is a microphone & game controller. In the VR (Virtual Reality) space, although the user's (virtual user) 's arms and hands are not drawn, the position of the input device 160-1 etc. held by the user is sensed and a microphone image is drawn at the same position. The microphone image moves according to the movement of the user.

  The user performs vocal key adjustment in the standby room of the VR space. The standby room is a standby space below the stage. The space where the user stands in the VR space is a large lift, and during production, it rises on the stage.

  As the lift goes up and the stage approaches, the cheers and screeches of the hall that were heard from a distance gradually increase, becoming more powerful and vibrantly changing. When the user appears on the stage, the backlight is backlit from the front toward the user, and a loud cheer that reaches the climax occurs when the user appears.

  In live performances, the user enjoys the most enjoyable on stage performance. 6 and 7 show examples of game images (images in the VR space) displayed on the HMD 200 of the user on the stage. As shown in FIGS. 6 and 7, a full audience is shown in front of the user. FIG. 6 is a game image when the user turns to the front, and FIG. 7 is a game image when the user turns to the right.

  As shown in FIGS. 6 and 7, in the game device of the present embodiment using the HMD 200, the world of the VR space, which is a virtual space, spreads in the direction of the entire periphery of the user. For example, if the user wearing the HMD 200 faces the front, the game image of FIG. 6 is displayed on the HMD 200, and if the user faces the right, the game image of FIG. 7 is displayed. If you look backwards, an image such as a performance band will be displayed. Accordingly, in a large concert hall where a large number of spectators cheer up, it is possible to give the user a virtual reality as if vocal performance is being performed, and the degree of immersion in the game can be remarkably improved.

  Further, in the present embodiment, the movement (action) and cheers of the audience change in accordance with the tonation of the song. Also, the movements and cheers of the audience change so as to respond to the user's action. For example, in FIG. 6, the spectators AD1 to AD7 near the stage on which the user stands are represented by objects of a polygon model configured of, for example, a large number of polygons. The same applies to the audiences AD8 to AD11 in FIG. On the other hand, a spectator at a position far from the stage is represented by an image drawn on a billboard polygon facing the user's gaze.

  The movement of the spectators (AD1 to AD11) of the polygon model is represented by motion reproduction using motion data, for example. These spectators perform basic actions (basic motions) according to the rhythm of the song. The audience reacts interactively according to the voice and movement of the user. For example, when the user performs an action such as "sing to a specific direction" or "swing", the audience's tension in that direction is increased, and the basic action becomes one step, becomes brighter, or It will perform a sudden upsurge action that has nothing to do with the rhythm.

  FIG. 8 is a view showing the stage SG where the user PL (virtual user) stands in the VR space as viewed from above. The stage SG is divided by the boundary BDS, and in the audience seats SE1, SE2, SE3 around the stage SG, spectators who perform actions as shown in FIGS. 6 and 7 are arranged. The boundary BDS of the stage SG corresponds to, for example, the boundary BD of the area AR of the real-world play area shown in FIG. As an example, the boundary BDS of the stage SG in the VR space is set at a position corresponding to, for example, a predetermined distance inward from the boundary BD of the area AR of the real world in FIG.

  In the present embodiment, the user can perform a performance to boost the audience. A spectator who makes some appealing action towards the user is the target of the performance. For example, in FIG. 6, the spectator AD 4 raises the right hand to perform an appealing action to the user, and the spectator AD 4 becomes a target. Further, in FIG. 7, a spectator AD 10 performing an action of appeal is a target.

  As the user performs an action on these appealing spectators, the value of the enthusiasticness parameter (the enthusiasticity gauge) of these spectators increases. And when the enthusiasm parameter reaches its maximum value, these spectators will perform enthusiasm actions that express joy.

  The first action performed by the user on the appealing audience is an action of singing and aiming at the target audience. The first action is performed in the singing part of the singing part and the interlude part of the music.

  The second action performed by the user on the appealing audience is an action for vocalizing the audience or the like. The audience responds to the user's vocalization by cheering, cheering, applause, etc. The user performs an action such as raising an arm in accordance with cheers, screams and applause from the audience.

  Audiences targeted by these first and second actions occur, for example, randomly in the concert hall. The number of targets in one song played by the user is fixed to a predetermined number. If the enthusiasm parameter reaches its maximum value for all the predetermined number of target audiences to be targeted and if the enthusiasm is successful, then all will clear. For example, in the case where the number of target audiences is 10, when the enthusiasm parameter of all the 10 audiences reaches the maximum value, the target becomes completely clear. In this case, the type of the last stage effect changes in accordance with the number of cleared targets. For example, if all 10 targets are cleared, the most spectacular effect occurs on the last stage. Further, for example, when eight targets are cleared, a more flashy effect occurs than when five targets are cleared.

  In the present embodiment, the user's singing ability is evaluated. That is, the pitch and rhythm of the user's song are detected and evaluated. Specifically, it is evaluated whether a voice matched to the rhythm has been achieved. For example, points can be added when the user can utter at the designated timing of the bass or drum performance. In addition, the accuracy of the long tone, the accuracy of the rest, etc. are also evaluated. Also assess the accuracy of the pitch. That is, it determines the pitch of the user's song, and graphically displays the pitches sang at the correct pitch. It should be noted that when the user's volume continues to be zero for a certain period of time, most of the spectators shift to the standby motion, during which the user can not obtain point addition.

  When the user's performance is finished and the evaluation result of the user through the live stage satisfies a certain standard, the spectator comes for encore. Responding to the encore allows the user to play one more song additionally. And at the end of the live stage, in the faded out cheers, the lights get darker, black out and finished. After that, a message such as "Thank you for your kindness, please remove the HMD" and a guide sound are heard. Then, the user removes the equipment and leaves the room, and the game play ends.

2.3 Response to User Voice Input In this embodiment, a method of outputting a response voice and a response tone to respond to the user's voice input is adopted. For example, when the user makes a call to the audience (in a broad sense, the target in FIG. 6 or FIG. 7) by inputting a voice to the microphone 162, cheers from the audience, screams, applause, footsteps or whistling etc. Response is returned. That is, response voices such as cheers, screams, applauses, footsteps or whistles, and response sounds are output from the speakers 330 to 334 in FIGS. 4 and 5. In the following, the case where the sound input device 161 is a microphone 162 shaped like a hand microphone as shown in FIG. 2A will be described as an example.

  For example, in FIG. 9, the user PL turns the line of sight VLP toward the audience ADA, ADB, and ADC, and makes a call "Yay." That is, the user voice "Yay" is input to the microphone 162. When such user voice input is performed, after a given time has elapsed, as shown in FIG. 9, the spectators ADA, ADB, ADC respond with cheers, applause, whistling, and the like. By adopting such a call & response (C & R) method, the user can feel the audience in the virtual space as if it were a real human audience, and the virtual reality of the user is greatly enhanced. It can improve.

  In this case, if the audience's response to the user's call (call) is a one-pattern non-variate response, the user is as if it were calling to the robot. It has a sense and virtual reality is lost.

  On the other hand, if it is attempted to generate a response voice or a response sound of a spectator by processing of a complex algorithm by performing voice recognition processing of user voice, etc., the processing load becomes excessive, and other processing performed by the simulation system ( It adversely affects the image generation processing etc.).

  So, in this embodiment, the measurement process of the volume and length of the user's voice inputted by microphone 162 (sound input device in a broad sense) is performed. And the method of outputting the response sound and the response sound based on the result of the measurement process of the volume and length of the user voice is adopted.

  In the example of FIG. 9, when the user PL inputs the voice of the call ("Yay") using the microphone 162, the volume and length of the voice of the call are measured. Then, depending on the volume of the voice of the call, the response voices such as cheers and applauses of the audience ADA to ADC and the response sounds are made different. Also, depending on the length of the voice of the call, the cheering of the audience ADA to ADC, the response sound such as applause, and the response sound are made different. For example, the response sound, the type of response sound, the volume (intensity), the pitch or tone of the sound, etc. are made different. For example, the sound data selected from among the plurality of sound data in the sound data storage unit 175 of FIG. 1 and the combination of the sound data are made different according to the volume and the length of the voice of the call. For example, different sound data may be selected or sound data of different combinations may be used according to the volume of the calling voice. Also, different sound data may be selected or sound data of different combinations may be used according to the length of the calling voice.

  FIG. 10 is a view for explaining a method of making the response sound and the response sound different according to the volume and the length of the user sound. The response sounds and response sounds of RS11 to RS35 in FIG. 10 are realized by sound data selected from a plurality of sound data in the sound data storage unit 175 or a combination of sound data.

  For example, when the volume of the user voice input to the microphone 162 is LV1 (when LV1 is exceeded or 0 to LV1), RS11, RS12, RS13 according to the length (scale) of the user voice. The response sound of either RS14 or RS15 or the response sound (sound data or a combination of sound data) is selected and output. For example, when the length of the user voice is T1 (when T1 is exceeded or 0 to T1), the response voice and response tone of RS11 are output. When the length of the user voice is T2 (when T2 is exceeded or T1 to T2), the response voice and response tone of RS12 are output. Similarly, when the length of the user voice is T3, T4, or T5, the response voices and response tones of RS13, RS14, and RS15 are output, respectively.

  When the volume of the user's voice input to the microphone 162 is LV2 (when LV2 is exceeded or LV1 to LV2), RS21, RS22, RS23, RS24, and so on according to the length of the user's voice. Any response voice and response tone of RS 25 is selected and output. For example, when the lengths of the user voices are T1, T2, T3, T4, and T5, response voices and response sounds of RS21, RS22, RS23, RS24, and RS25 are output, respectively.

  When the volume of the user's voice input to the microphone 162 is LV3 (when LV3 is exceeded or LV2 to LV3), RS31, RS32, RS33, RS34, and so on according to the length of the user's voice. Any response voice and response tone of RS35 is selected and output. For example, when the lengths of the user voices are T1, T2, T3, T4, and T5, response voices and response sounds of RS31, RS32, RS33, RS34, and RS35 are output, respectively.

  If such a method is adopted, for example, if the volume and length of the voice of the call inputted by the user PL differ in FIG. 9, for example, the cheers, screams, applauses, and whistles of the audience ADA to ADC differ accordingly. It becomes a sound.

  For example, when the volume of the voice of the call is small, the cheers and squeals become low voices, and the volume of the cheers and squeaks become smaller. Also, the ratio of clapping and whistling is reduced. On the other hand, when the volume of the calling voice is large, the cheering voice becomes a high voice (sound containing high frequency components) or the volume of the cheering voice or the cheering voice becomes large. In addition, the ratio of clapping and whistling also increases.

  In addition, when the length of the voice of the call is short, the length of the cheers and the hail also becomes short. In addition, clapping and whistling do not mix, and in the second half, only a little clapping and whistling sounds. On the other hand, when the length of the voice of the call is long, the length of the cheers and the hail also becomes long. You may also want to raise the pitch of the cheers and urges, or raise the volume. In addition, when the length of the voice of the call is long, many applauses and whistles will be mixed, and in addition to this, footsteps will be mixed and an encore voice will be mixed.

  As described above, in the present embodiment, the response sound and the response sound have different sounds according to the volume and the length of the user voice. Therefore, it is possible to give the user a feeling as if a real human spectator is cheering or applauding, and the virtual reality of the user can be improved. That is, since the response from the spectators' cheers and applause is not one pattern, but varied and full of change, it does not give the user a sense of being opposed to the robot, and the user's degree of immersion in virtual reality Can be increased.

  FIGS. 11A and 11B are diagrams showing a specific example of the process of making the response sound and the response sound different according to the volume and length of the user voice.

  In FIG. 11A, A1 is a waveform of user voice. Then, in FIG. 11A, from the timing when the volume of the user's voice exceeds the volume level LV1 (the i'th volume level among the first to Nth volume levels), to the timing when the volume falls below the volume level LV1. Measure the length TL1. Then, the response sound and the response sound are made different according to the measured length TL1. For example, the response sound and the response sound are made different depending on whether the length TL1 is long or short.

  Further, the length TL2 from the timing when the volume of the user's voice exceeds the volume level LV2 (the jth volume level of the first to Nth volume levels) to the timing when the volume falls below the volume level LV2 is measured. Then, the response sound and the response sound are made different according to the measured length TL2. For example, the response sound and the response sound are made different between when the length TL2 is long and when the length TL2 is short. In this case, the response sound and the response sound are differentiated depending on whether the volume of the user voice exceeds the volume level LV1 (i-th volume level) and the volume level LV2 (j-th volume level). .

  Also, the length TL3 from the timing when the volume of the user voice exceeds the volume level LV3 to the timing when the volume falls below the volume level LV3 is measured. Then, the response sound and the response sound are made different according to the measured length TL3. For example, the response sound and the response sound are made different between when the length TL3 is long and when the length TL3 is short. In this case, the response sound and the response sound are made different depending on whether the volume of the user's voice exceeds the volume levels LV1 and LV2 or LV3.

  In FIG. 11B, the signal waveform of the user voice is different from that of FIG. For example, the sound lengths TL1, TL2, and TL3 at the volume levels LV1, LV2, and LV3 are different between FIG. 11 (A) and FIG. 11 (B). For example, in FIG. 11 (B), the length TL3 at the volume level LV3 is considerably shorter than that of FIG. 11 (A). Therefore, the response sound and the response sound to be output also have different sounds. As shown in FIG. 11 (B), when a back rising type user voice is input, the time for the volume to reach the volume level LV3 is delayed, so that the corresponding response voice and response voice are also delayed.

  For example, in FIGS. 11 (A) and 11 (B), when the volume of the user's voice is the volume level LV1, the cheering and screeching become low, and the volume of the cheering and screeching becomes relatively small. When the length TL1 is increased, for example, applause etc. will be mixed in the second half.

  Further, when the volume of the user's voice is the volume level LV2, cheers and cheers become relatively high sounds (sounds in which high frequency components are mixed) and the volume becomes larger as compared with the case of the volume level LV1. Then, when the length TL2 becomes longer, for example, in the second half, in addition to the applause, the whistling becomes mixed.

  In the case where the volume of the user's voice is the volume level LV3, the cheers and cheers become a higher sound (a sound in which high frequency components are further mixed) and the volume is even larger than in the case of the volume levels LV1 and LV2. Become. When the length TL3 is increased, for example, in the second half, in addition to clapping and whistling, footsteps and encore voices become mixed.

  For example, when the volume of the user's voice is large, the user who is the vocal is calling out with a loud voice to excite the audience, and the cheers of the audience also become a mixed sound of high frequency components so as to respond to the enthusiasm. The volume also increases. Also, if the length of the voice of the user voice is long, it is considered that the user who is vocal is trying to transmit some kind of message, and the audience applauds, whistling or stepping in response to it. . In this way, voice communication and bargaining are created between the user and the audience, and it is possible to give the user a virtual reality as if they were playing against the real human audience. Become.

  When the sound volume level LV2 is exceeded, both the response sound for the sound volume level LV1 and the response sound for the LV2 may be output. Similarly, when the volume level LV3 is exceeded, all of the response sounds for the volume levels LV1, LV2, LV3 and the response sound may be output.

  FIG. 12 is a diagram for explaining the output timing of the response sound and the response sound. For example, in FIG. 12, the volume of the user's voice exceeds the volume level LV1. In this case, after a given time TD1, a response sound according to the volume level LV1 and the length TL1, RS = RS (LV1, TL1) which is a response sound, is output. The time TD1 is, for example, about one second.

  That is, when the spectator immediately returns a response voice such as cheers or a response sound in response to a call from the user, it sounds as if the robot is cheering or the like. Therefore, as shown in FIG. 12, the audience responds after the lapse of time TD1, which is the delay time, from the user's call. In this way, by giving a response to the response, it is possible to give the user a feeling as if a real human audience is responding, and virtual reality can be improved.

  It should be noted that various modifications can be made as processing for differentiating the response sound and the response sound according to the volume and length of the user voice.

  For example, when the volume of the user's voice is the volume level LV1 in FIG. 11A, the file of sound data corresponding to the volume level LV1 is selected. Then, the reproduction time of this sound data may be changed according to the length TL1 of FIG. For example, the reproduction sound based on the sound data is faded out at a timing when a time corresponding to the length TL1 has elapsed. For example, when the length TL1 is short, the timing at which the reproduced sound of the sound data is faded out is advanced, and when the length TL1 is long, the timing at which the faded out is delayed. Similarly, even when the volume of the user's voice is at the volume levels LV2 and LV3, the file of sound data corresponding to the volume levels LV2 and LV3 is selected, and the files of the sound data are selected according to the lengths TL2 and TL3. Change playback time. That is, it fades out at a length corresponding to the lengths TL2 and TL3.

  Alternatively, based on the length of the user voice (TL1, TL2, TL3), use the microphone 162 to infer the content of the user's talk and replace the file of the response voice and the sound data of the response tone accordingly. You may do so. For example, a plurality of (for example, five) sound data files corresponding to the length of the user voice are prepared, and the sound data files to be used are replaced according to the length of the user voice.

  For example, if the length of the user's voice is very short (for example, about 0.1 second), it is presumed that the user is making a meaningful call (a hail) and the audience response is also a corresponding response. . For example, the audience only responds with short cheers. On the other hand, when the length of the user's voice is long (for example, 1 to 2 seconds or more), it is inferred that the user speaks a meaningful word to the audience. Then, laughter is mixed with the audience response, and a voice that agrees with the user's speech is mixed. For example, if the user is speaking in a quiet tone for a long time (when the volume of the user's voice is small and long), it is inferred that, for example, "the next album comes out" Make the audience respond with a happy feeling. By doing this, it is possible to respond to a more human-like audience.

  Further, in the present embodiment, the response voice to the input of the user voice and the response tone are output as the target voice and sound of the user in the game. Specifically, it is output as the voice and sound of the spectator character appearing in the game. In this case, in the present embodiment, the response voice and the response sound are changed according to the type of audience (target), the positional relationship between the audience and the user, or the direction of the user's line of sight with respect to the audience.

  For example, in FIG. 13A, the user PL is calling for a male audience ADX. In this case, the response sound of the audience ADX, the response sound becomes the response sound for the male audience, the response sound. For example, the kind of cheering that men normally use (for example, "O") comes back in male voice. On the other hand, in FIG. 13 (B), the user PL has made a call to the female audience ADY. In this case, the response voice of the audience ADY and the response tone become the response voice and response tone for the female audience. For example, the kind of cheering that women normally use (for example, "Kah") comes back in the female voice.

  As described above, in the present embodiment, the response sound and the response sound change in accordance with the type of audience (target). For example, depending on whether the audience is male or female, the response sound and the response sound change. In this case, various types of audiences (targets) can be assumed. For example, the response sound and the response sound may be changed according to whether the audience is an adult or a child.

  Further, in FIG. 14A, the distance between the user PL (virtual user) and the audiences ADA to ADC is long. In this case, the volume of the response sound of the cheers of the audience ADA to ADC, the applause, etc., and the response sound is reduced. On the other hand, in FIG. 14B, the distance between the user PL and the audience ADA to ADC is short. In this case, the loudness of response sounds such as cheers and applauses of the audience ADA to ADC and the response sound is increased.

  As described above, in the present embodiment, the response sound and the response sound are changed according to the positional relationship between the audience (target) and the user. That is, since the positional relationship between the audience ADA to ADC and the user PL in FIG. 14A and the positional relationship between the audience ADA to ADC and the user PL in FIG. 14B are different, the volume of the response sound and the response sound is different. It is changing. In this case, the response sound, the type of the sound of the response sound, the pitch or timbre of the sound may be changed according to the positional relationship.

  Further, in FIG. 15, the line of sight VLP of the user PL is directed to the audience ADA to ADC on the front side, and is not directed to the audience ADD to ADF on the right. In such a state, when the user PL inputs the voice of the call, the volume of the response sound and the response sound of the audience ADA to ADC to which the user PL points the sight line VLP becomes large. On the other hand, the volume of the response sound and the response sound of the spectators ADD to ADF in which the user PL is not pointing the line of sight VLP is reduced.

  As described above, in the present embodiment, the response sound and the response sound are changed according to the direction of the user's line of sight with respect to the audience (target).

  In FIG. 15, the response sound and the volume of the response sound are changed according to the direction of the user's line of sight with respect to the audience, but the response sound, the type of the sound of the response sound, and the pitch or timbre of the sound are changed. May be

  The target may also represent a place in a virtual space (object space). For example, the target may be a spot such as each seat of a spectator seat in a music game or a sports game. In this case, attribute data (control data) is stored in association with, for example, a place such as a seat. The first attribute data is for enthusiastic fan seats (for example, the front row seats), and the second attribute data is for the seats of young people such as teenagers (for example, the second floor seats), The third attribute data is associated with the seat for parties concerned that can hardly expect any reaction. Then, when it is determined that the user's gaze direction is directed to these seats, the response voice and the response sound are changed and output based on the attribute data associated with the seat. For example, when the gaze direction of the user is directed toward the enthusiastic fan's seat (first attribute data), response voices such as enthusiastic cheers and applause, and response sounds are output. When the line of sight is directed to the seat (second attribute data) of the customer who has a good loin, a response voice such as a good cheer or applause and a response sound are output.

  Further, in FIG. 16A, analysis processing of the feature amount of the user's voice input to the microphone 162 is performed. That is, the feature of the user voice is detected. In the present embodiment, the response sound and the response sound are changed according to the result of the analysis process of the feature amount of the user sound. That is, the response sound and the response sound are changed according to the feature (sex, age group, etc.) of the user sound. The feature amount analysis process is a process of extracting and analyzing feature amounts related to the size, height, timbre, or type of the sound of the user's voice. This is realized, for example, by obtaining an acoustic spectrum or the like by frequency analysis processing.

  Then, for example, it is assumed that the gender of the user is determined to be male by the feature amount analysis of FIG. 16 (A). In this case, as shown in FIG. 16 (B), the spectators' cheers are a mixture of male and female voices such that the ratio of males' cheers is RA and the ratio of females' cheers is RC. Response voice, output as response tone Since the gender of the user is male, for example, the ratio RA of male cheers is lowered and the ratio RC of female cheers is increased.

  Further, it is assumed that the gender of the user is determined to be female by the feature amount analysis of FIG. In this case, as shown in FIG. 16 (B), the spectators' cheers are a mixture of male and female voices such that the ratio of males' cheers is RB and the ratio of females' cheers is RD. Response voice, output as response tone Since the gender of the user is female, for example, the ratio RB of male cheers is increased, and the ratio RD of female cheers is decreased.

  For example, the age group of the user may be determined by the feature amount analysis of FIG. Then, when the age group of the user is a young group, the cheers for the young group are output as the response sound and the response sound of the audience. On the other hand, when the user's age group is the elderly, the cheers for the elderly are output as the audience's response sound and response sound.

  In this way, analyzing the characteristics of the user's voice and outputting the response voice and the response sound according to it analyzes the characteristics of the user's gender, age group, etc. in the cheers of the audience, etc. The virtual reality of can be further improved. Note that the feature of the user voice may be extracted, and the processing corresponding to the feature may be applied to the response voice and the response tone. For example, only the sequence of vowels in the user's voice is extracted, and response voices such as cheers and response sounds are processed so as to be a sequence of vowels similar to the sequence of vowels. That is, processing such as a so-called vocoder is performed. In this way, it is possible to more directly reflect the characteristics of the user's voice on cheers and the like.

  Further, in the present embodiment, the response is made according to the position and direction (or the position and direction of the hand) of the microphone 162 held by the user, the position and direction of the HMD 200 worn by the user, the posture of the user, or the line of sight of the user. It is possible to perform processing to change voice and response sound, or to perform processing to output response voice and response sound, or to receive input of user voice for outputting response voice and response sound. Good.

  For example, in the game according to the present embodiment, as described with reference to FIGS. 6 and 7, the user PL performs various actions on the audience, and the result of the action is evaluated. For example, in FIG. 17, the user PL performs an action of directing the line of sight VLP toward the audience AD and directing the microphone 162 toward the audience AD. Then, the microphone 162 is urged to make cheers and screams. When the user PL performs such an action, the enthusiasticness parameter (excitation degree parameter) of the audience AD rises, and the performance of the user PL is highly evaluated.

  For example, in FIG. 17, the position and direction (at least one of position and direction) of the microphone 162 and the HMD 200 can be detected by the tracking process or the like of the microphone 162 and the HMD 200 described in FIG. For example, the position and direction of the microphone 162 can be detected by detecting the position and direction of the input device 160-1 in FIG. 2A. The position and the direction of the user PL holding the microphone 162 may be detected to detect the position and the direction of the microphone 162. Further, by detecting the position and direction of the HMD 200, it is possible to detect the direction of the line of sight VLP in FIG. 17 and the gaze target of the line of sight VLP of the user.

  The posture of the user PL can be detected based on the first position of the HMD 200 of FIG. 2A, the second position of the input device 160-1, and the third position of the input device 160-2. That is, based on these first, second and third positions, it is possible to detect the general posture state of the user. A motion sensor that detects the posture of the user may be provided, and the posture of the user may be detected based on the motion sensor. For example, the posture of the user may be detected by estimating the shape of the skeleton of the user based on a color image and depth information acquired by an imaging sensor (a color image sensor, a depth sensor or the like) which is a motion sensor.

  Then, in FIG. 17, depending on whether or not the user PL is gazing toward the spectator AD with respect to the eye gaze VLP (the line of sight of the HMD 200), the response sound such as cheers and applause of the spectator AD is changed. Further, depending on whether or not the user PL directs the direction (directed direction) of the microphone 162 to the spectator AD, the response sound such as cheers and applause of the spectator AD and the response sound are changed. Alternatively, depending on whether or not the posture for performance such as the crouching posture of the user PL shown in FIG. 17 is taken, the response sound of the cheers of the audience AD, the applause, etc., and the response sound are changed.

  For example, if the user PL inputs a voice such as a call to the microphone 162, then the gaze VLP is directed to the audience AD, the microphone 162 is directed to the audience AD, or a posture for performance is taken. , A response voice such as cheers of a spectator AD, a screeching, a stepping sound or a clap, a response sound is changed into a response sound representing a more enthusiastically excited state, a response sound. In this way, the response sound and the response sound of the audience can be interactively changed according to the direction of the line of sight VLP of the user PL, the direction in which the microphone 162 is directed, the posture, etc. You will be able to improve your sense of reality.

  Further, for example, although FIG. 12 illustrates control of outputting a response sound and a response sound after a given time TD1 has passed since the volume of the user voice exceeds the volume level LV1, this embodiment is not limited to this. For example, response control of the response sound or the response sound (output timing control or the like) may be performed based on the position or direction of the microphone 162 or the HMD 200, the posture of the user, or the line of sight of the user. For example, after the user PL inputs a voice such as a call to the microphone 162 in FIG. 17, the microphone 162 is directed toward the audience AD, the direction of the HMD 200 (user's gaze) towards the audience AD, or the user When a predetermined posture is taken, a response voice such as cheers and applause by the audience AD or a response sound may be output.

  Alternatively, it may be determined based on the position and direction of the microphone 162 and the HMD 200, the posture of the user, the line of sight of the user, or the like whether or not to receive the input of the user's voice.

For example, in FIG. 2A, the position of the HMD 200 can be considered to correspond to the position of the head (mouth) of the user PL. Therefore, when it is determined that the distance between the microphone 162 and the HMD 200 (head and mouth) is short, or it is determined that the direction (directing direction) of the microphone 162 is directed to the HMD 200 (head and mouth). It can be determined that the user PL is trying to input a voice such as a call to the microphone 162. Therefore, in this case, the input of the user's voice is accepted. That is, the microphone input (the input channel of the microphone) is turned on. On the other hand, when the distance between the microphone 162 and the HMD 200 is long, or when the direction (the pointing direction) of the microphone 162 is not directed to the HMD 200, it is determined that the user PL has no intention to input a voice such as a call. it can. Therefore, in this case, the input of the user voice is not accepted, and the microphone input is turned off. As described above, by turning off the microphone input, the direction of the microphone 162 (orientation direction) is directed to the speaker in the real world, and it is possible to prevent howling or the like from occurring. The response sound and the response sound may be changed based on the information of the user's past play history. For example, FIG. 18A shows an example of the information of the play history. In this play history, the date and time when the user played the game of FIGS. 6 and 7 are recorded. It is desirable that the information of the play history is stored in the storage unit 170 of FIG. 1 and then transmitted to the server system via the network.

  Then, when the user carries, for example, an IC card (portable information storage medium 195 in FIG. 1) in which user information and the like are stored and performs touch operation on a card reader of the simulation system, information on the user's play history is sent from the server system It is transferred and stored in the storage unit 170 of FIG. Then, the information of the play history is read from the storage unit 170, and the response sound and the response sound are changed based on the read information of the play history.

  For example, if it is judged based on the information of the play history that the user has not played the game in the simulation system for a long time, the response voice file as shown in FIG. 18B is selected. Then, the processing of the response sound and the response sound is performed so that the voices such as "After a long time" and "I waited" are mixed in the cheers of the audience. In this way, the response sound of the spectator and the response sound change in various ways according to the user's past play history, and it is possible to output the response sound and the response sound that make the human body feel more.

  FIG. 19A is an explanatory diagram of the sound data storage unit 175. As shown in FIG. As shown in FIG. 19A, the sound data storage unit 175 stores a plurality of sound data SD1, SD2, SD3, SD4. In the present embodiment, processing for selecting sound data to be used from a plurality of sound data (sound data files) stored in the sound data storage unit 175 realizes output processing of a response sound or a response sound. doing. For example, sound data corresponding to the measurement result is selected from the plurality of sound data based on the measurement result of the user voice. For example, when the volume and length of the user voice are measured, sound data corresponding to the user voice is selected, reproduction processing based on the selected sound data is performed, and a response voice and a response sound are output. Alternatively, depending on the type of target such as a spectator, the positional relationship between the target and the user, the gaze direction of the user with respect to the target, the analysis result of the feature of the user voice, the position and direction of the microphone 162, the posture of the user, or the gaze of the user Accordingly, sound data to be selected may be made different.

  Further, in the present embodiment, as shown in FIG. 19B, a response voice and a response sound are output by performing a process of combining a plurality of sound data read from the sound data storage unit 175. For example, mixing processing for combining a plurality of sound data is performed, and a response sound and a response sound are output. In this case, the combination of sound data is changed based on the measurement result of the user's voice, and output as the response voice and the response sound. For example, when the volume and length of the user voice are measured, the combination processing (mixing processing) of sound data according to the measurement is performed, and the response voice and the response sound are output. Alternatively, depending on the type of target such as a spectator, the positional relationship between the target and the user, the gaze direction of the user with respect to the target, the analysis result of the feature of the user voice, the position and direction of the microphone 162, the posture of the user, or the gaze of the user In response, the combination of sound data may be changed and output as a response sound or response sound.

2.4 Reception Timing of User Voice When outputting a response voice and a response sound to a user voice input by a user, it is also necessary to consider the reception timing of the user voice.

  For example, in the present embodiment, evaluation processing is performed on the user's voice input by the microphone 162. Specifically, when the user sings a song using the microphone 162, evaluation processing (score processing) of the song is performed.

  For example, as shown in FIG. 20 (A), a pitch display of lyrics or a song is shown to the user. This pitch display is for the user to visually grasp the pitch which becomes the reference (model) of the evaluation processing. In addition, the user can grasp the rhythm which is the standard (model) of the evaluation process by this pitch display. Note that only the lyrics may be displayed without performing such pitch display.

  Then, when the user sings a song, comparison processing with reference data of pitches and rhythms as shown in FIG. 20 (B) is performed. That is, processing of extracting the data of the user's song's song and rhythm from the information of the singing voice input by using the microphone 162 is performed. Then, evaluation processing of the user's song (vocal performance) is performed by performing comparison processing of the extracted user's pitch and rhythm data with the pitch and rhythm reference data. In this way, it is possible to properly evaluate the user's singing ability. In addition to the reference data of the pitch and the rhythm, reference data of volume may be prepared, and the user's vocal volume control may be evaluated as well.

  In this embodiment, during a period in which evaluation processing is performed on user voice such as the user's singing voice, response voice, user voice input for outputting the response voice, that is, user voice for call & response Do not accept input. Specifically, as shown in FIG. 21, while the user is singing and the user's song is being evaluated, the user voice input for C & R (for call & response) is not accepted. Then, in a period other than such a song evaluation period, the input of the user voice for C & R is accepted. Specifically, as shown in FIG. 21, the input of the user's voice for C & R is accepted in the prelude period and the interlude period. Then, in response to such user voice input, the response voice and the response sound are output by the method described in FIGS. 9 to 19B.

  For example, in the evaluation period of a song in which the user is singing, the user will input into the microphone 162 the words constituting the lyrics of the song. Then, in response to the speech of this word, response voices such as cheers and the like as shown in FIG. 9 and the like are output, which may disturb the evaluation process of the song or cause the user to feel unnatural. There's a problem.

  In this respect, in FIG. 21, in the evaluation period of the song in which the user is singing, the input of the user voice for C & R is not accepted, and the corresponding response voice and response sound are not output. Then, the input of the user voice for C & R is accepted only in a period other than such an evaluation period, and the response voice and the response sound are output. Therefore, as described above, it is possible to prevent the occurrence of the problem that the response voice or the response tone interferes with the evaluation process of the song, or the user feels unnatural.

3. Detailed Processing Next, a detailed processing example of the present embodiment will be described using the flowchart of FIG.

  First, it is determined whether it is a reception period of user voice input for C & R (for call & response) (step S1). Then, as described in FIG. 21, when it is not the reception period (prelude period, interlude period, etc.), the user voice input for C & R is not received. On the other hand, in the case of the reception period, the user voice input is received, and measurement processing of the volume and length of the user voice is performed (step S2). And based on the result of a measurement process, the process which selects the sound data of response voice or a response sound is performed (step S3). That is, as described with reference to FIGS. 10 to 12, processing for varying the response sound or the response sound is performed according to the volume and the length of the user sound.

  Next, based on target information (positional relationship etc.) of audience etc., feature amount (sex, age group etc.) of user voice, position of microphone 162 or HMD 200, posture of user, user's gaze, user's play history etc. Then, change processing (processing processing) of the response voice or the response sound is executed (step S4). That is, the change processing of the response sound or the response sound as described in FIGS. 13A to 17 is executed. Then, the response voice or response tone after the change processing (after the processing) is output (step S5).

  It should be understood by those skilled in the art that although the present embodiment has been described in detail as described above, many modifications can be made without departing substantially from the novel matters and effects of the present invention. Accordingly, all such modifications are intended to be included within the scope of the present invention. For example, the terms (microphone, audience, play area, etc.) described together with different terms (sound input device, target, movable range, etc.) more broadly or synonymously at least once in the specification or figures are the specification or drawings In any part of, the different terms can be substituted. Also, in this embodiment, user voice acquisition processing, user voice measurement processing, response voice, response sound output processing, response voice, response sound change processing, user voice input acceptance processing, simulation system configuration, etc. The present invention is not limited to the above-described ones, and equivalent methods, processes and configurations are also included in the scope of the present invention. The present invention is also applicable to various games. Further, the present invention can be applied to various simulation systems such as a arcade game machine, a home game machine, or a large attraction system in which a large number of users participate.

100 processing unit, 102 input processing unit, 110 arithmetic processing unit, 111 game processing unit,
112 game progress processing unit, 113 evaluation processing unit, 114 character processing unit,
115 parameter processing unit 116 object space setting unit
117 virtual camera control unit, 118 game result calculation unit, 119 measurement processing unit,
120 display processing unit, 130 sound processing unit, 132 response sound processing unit,
140 output processing unit, 150 imaging unit, 151, 152 cameras,
160 (160-1, 160-2) input device, 161 sound input device,
162 microphone, 170 storage unit, 172 space information storage unit, 174 music information storage unit,
175 sound data storage unit, 176 parameter storage unit, 178 drawing buffer,
180 information storage medium, 192 sound output unit, 194 I / F unit,
195 portable information storage medium, 196 communication unit,
200 HMD (head mounted display), 201 to 203 light receiving elements, 210 sensor unit,
220 display unit, 231 to 236 light emitting elements, 240 processing unit, 260 head band,
280, 284 stations, 281, 282, 285, 286 light emitting elements,
290, 292 lighting fixtures, 301-304 walls, 305 ceilings, 306 doors,
311 to 315 soundproofing materials, 330, 331 front speakers, 332 rear speakers,
333, 334, rear speakers, 335 woofer,

Claims (15)

An input processing unit that acquires information on user voice input by a user using a sound input device, and information on a viewpoint position and a gaze direction of a virtual user in a virtual space corresponding to the user;
A measurement processing unit that performs measurement processing of the user voice;
A sound processing unit that performs an output process of a response voice or a response sound different from the user voice in response to the input of the user voice;
A virtual space setting unit that performs setting processing of the virtual space;
A display processing unit that generates an image viewed from the viewpoint of the virtual user in the virtual space and displays the image on a display unit of a head mounted display worn by the user;
Including
The measurement processing unit
Perform measurement processing of the volume and length of the user voice;
The sound processing unit
The response voice or the response tone based on the result of the measurement process of the volume and length of the user voice is output as the voice or sound of the target of the user in the virtual space,
The sound processing unit
Type of the target the virtual user directs the gaze direction in the virtual space in response to the at least one virtual user's gaze direction relative to the target in 及 beauty the virtual space, the response voice or the response sound A simulation system characterized by performing processing to change. In claim 1,
The sound processing unit
The distance between the sound input device and the head-mounted display device, or whether the sound input device faces the head-mounted display device, whether the user's voice input is accepted or not The simulation system characterized by judging according to. In claim 1 or 2,
The sound processing unit
A simulation system characterized by performing processing of changing the response voice or the response sound according to whether or not the virtual user gazes at a gaze direction toward the target in the virtual space. In any one of claims 1 to 3,
The sound processing unit
Processing for detecting the posture of the user based on the position of the head-mounted display device and the position of the voice input device, and changing the response voice or the response sound according to the detected posture of the user A simulation system characterized by performing. In any one of claims 1 to 4,
The sound processing unit
A simulation system characterized in that the response voice or the response tone is differentiated according to the volume and length of the user voice. In any one of claims 1 to 5,
The measurement processing unit
The length from the timing at which the volume of the user voice exceeds the i-th volume level (1 ≦ i ≦ N) of the first to N-th volume levels to the timing at which the volume falls below the i-th volume level Measure the
The sound processing unit
A simulation system characterized by performing processing to make the response voice or the response sound different according to the measured length. In claim 6,
The sound processing unit
When the volume of the user voice exceeds the i-th volume level and when the j-th (1 ≦ i <j ≦ N) volume level of the first to N-th volume levels is exceeded Processing the response voice or the response sound differently. In any one of claims 1 to 7,
The simulation system characterized in that the target is a character of a spectator appearing in a game. In any one of claims 1 to 8,
The measurement processing unit
The analysis process of the feature amount of the user voice is performed, and the sound processing unit is configured to perform the process of changing the response voice or the response sound according to the result of the analysis process of the feature amount of the user voice. Simulation system. In any one of claims 1 to 9,
The sound processing unit
The position and the direction of the sound input device held by the user, the position and the direction of the hand, the position and the direction of the head mounted display worn by the user, the posture of the user, and the line of sight of the user According to at least one, processing to change the response sound or the response sound, or processing to output the response sound or the response sound, or input of the user voice for outputting the response sound or the response sound A simulation system characterized by performing processing to receive In any one of claims 1 to 10,
The sound processing unit
A simulation system characterized by performing processing to change said response sound or said response sound based on past play history information of said user. In any one of claims 1 to 11,
A sound data storage unit for storing sound data;
The sound processing unit
The response sound or the response sound is output by performing processing of selecting sound data to be used from a plurality of sound data stored in the sound data storage unit or processing of combining a plurality of sound data. Simulation system characterized by In any one of claims 1 to 12,
The input processing unit
In an evaluation period in which evaluation processing is performed on the user voice, the response voice or the input of the user voice for outputting the response sound is not received, and the response voice or the response sound is received in a period other than the evaluation period. Receiving the input of the user's voice for outputting The simulation system according to any one of claims 1 to 13.
The sound input device;
The head mounted display;
Simulation apparatus characterized by including. An input processing unit that acquires information on user voice input by a user using a sound input device, and information on a viewpoint position and a gaze direction of a virtual user in a virtual space corresponding to the user;
A measurement processing unit that performs measurement processing of the user voice;
A sound processing unit that performs an output process of a response voice or a response sound different from the user voice in response to the input of the user voice;
A virtual space setting unit that performs setting processing of the virtual space;
A display processing unit that generates an image viewed from the viewpoint of the virtual user in the virtual space and displays the image on a display unit of a head mounted display worn by the user.
Make the computer work
The measurement processing unit
Perform measurement processing of the volume and length of the user voice;
The sound processing unit
The response voice or the response tone based on the result of the measurement process of the volume and length of the user voice is output as the voice or sound of the target of the user in the virtual space,
The sound processing unit
Type of the target the virtual user directs the gaze direction in the virtual space in response to the at least one virtual user's gaze direction relative to the target in 及 beauty the virtual space, the response voice or the response sound A program characterized by performing a process of changing.

Images