JP2025045040A - Program and sound control device - Google Patents

Abstract

To improve user experience.SOLUTION: A sound control device includes: a virtual-space control section that controls a virtual space such that a virtual portion moves in the virtual space in response to a movement of a predetermined portion of a user; an image reproducing section that allows a display to reproduce an image in the virtual space; a sound generating section that, if a condition for reproducing effect sound responsive to a predetermined movement of the virtual portion is established, generates effect sound in response to a moving speed of the predetermined portion; and a sound reproducing section that allows a speaker to reproduce the effect sound generated by the sound generating section.SELECTED DRAWING: Figure 5

Description

This disclosure relates to a program and an audio control device.

Some game programs play sound effects in a virtual space. In the example of Patent Document 1, gunshot sounds are played in a shooting game.

JP 2014-094160 A

When various services (e.g. games) are provided within a virtual space, it is desirable from the perspective of user experience to play audio appropriate to the situation.

The purpose of this disclosure is to improve the user experience in services provided in virtual space.

A first aspect is a program for causing a computer system having one computer or a plurality of computers working together to function as a virtual space control unit, an image playback unit, an audio generation unit, and an audio playback unit,
The computer system is used with a display, a speaker, and a sensor;
The sensor outputs a signal indicative of a movement of a predetermined part of the user's body;
the virtual space control unit controls the virtual space such that a virtual part moves within the virtual space in response to the movement of the predetermined part indicated by the signal;
The image playback unit plays an image of the virtual space on the display,
the sound generation unit generates a sound effect according to a moving speed of the virtual body part when a condition for playing the sound effect according to a predetermined movement of the virtual body part is satisfied;
The sound reproducing unit is a program that reproduces the sound effects generated by the sound generating unit through the speaker.

In a first aspect,
The computer system may be used in conjunction with a controller that is worn or held by the user and operated,
The sensor may be included in the controller;
The virtual space control unit may control the virtual space based on a signal output from the sensor and an operation input to the controller.

In a first aspect,
The predetermined part may be a hand,
The virtual part may be a virtual hand.

In a first aspect,
A virtual object that can be held by the virtual hand may be placed in the virtual space,
The virtual space control unit may control the virtual space such that, when a gripping condition is satisfied, the gripping operation is performed in which the virtual hand grips the virtual object in the virtual space.
When the gripping condition is satisfied, the sound generation unit may generate a gripping sound, which is a sound effect corresponding to the gripping action, depending on the movement speed of the specified part during a gripping start period starting from the time when the gripping condition is satisfied.

In a first aspect,
The grasping condition may be a condition that an operation for performing the grasping action is input to the controller with the virtual hand entering a predetermined range based on a position of the virtual object in the virtual space.

In a first aspect,
The virtual object may be portable in a state where it is held by the virtual hand.
The virtual space control unit may control the virtual space such that, when a carrying condition is satisfied, which is a condition for performing a carrying action in which the virtual hand carries the virtual object in a state of holding the virtual object, after the grasping condition is satisfied in the virtual space, the carrying action is performed,
When the carrying condition is met after the holding condition is met, the sound generation unit may generate a carrying sound, which is a sound effect corresponding to the carrying action, depending on the movement speed of the specified part during a carrying start period from the time when the holding condition is met to the time when the carrying condition is met.

In a first aspect,
The carrying condition may be a condition that, while the virtual hand is continuing to grip the virtual object in the virtual space, a movement distance of the specified part after the gripping condition is satisfied reaches a specified distance.

The second aspect is
A storage unit that stores the program of the first aspect;
and a control unit that executes the program.

This disclosure can improve the user experience.

1 is a block diagram illustrating a configuration of a game device according to an embodiment. FIG. 1 is a diagram illustrating an example of the appearance of a head mounted display. 1 is a schematic diagram illustrating a player wearing a head mounted display and a controller. FIG. FIG. 11 is a schematic diagram for explaining a gripping operation. 11 is a flowchart illustrating a flow of sound processing in response to a gripping motion. FIG. 11 is a schematic diagram for explaining a carrying operation. 11 is a flowchart illustrating a flow of sound processing in response to a carrying operation.

The following describes the embodiment in detail with reference to the drawings. Note that the same or corresponding parts in the drawings are given the same reference numerals and their description will not be repeated. In addition, the following describes an example in which the program is implemented as a game program and the sound control device is implemented as a game device. The game device is an example of a computer system having one computer or multiple computers working together.

(Game Description)
In the following description, a game refers to a game (video game) that is played in a virtual space. The virtual space is a three-dimensional game space, and is displayed as an image on a display. Objects are placed in the virtual space. Examples of objects include a player character operated by a player (a user who plays the game) and a non-player character operated by a computer.

The objects also include sound source objects that act as virtual sound sources in the virtual space. Examples of sound source objects include objects that operate in the virtual space and objects that only output sound in the virtual space. Player characters and non-player characters can be sound source objects. In the games described below, sounds are played. Sounds played in the game include sounds output from sound source objects, background music (BGM), sound effects, and the like.

In the game described below, the player character is a virtual player (virtual human) placed in a virtual space, and moves in the virtual space in response to the operation and movement of the player in the real world. Specifically, a virtual part moves in the virtual space in response to the movement of a specific part of the player's body. The virtual part is a virtual part (a specific part of the player character's body) that corresponds to a specific part of the player. For example, the player character's virtual hand (virtual hand) moves in the virtual space in response to the movement of the player's hand in the real world.

In the following description, virtual objects that can be grasped by a virtual hand (the hand of the player character) are placed in the virtual space. The virtual objects can be carried while being grasped by the virtual hand. An example of a virtual object is a weapon equipped to the player character. Examples of weapons include firearms such as handguns and knives. The player character performs various actions such as a grasping action and a carrying action. A grasping action is an action in which the virtual hand grasps a virtual object in the virtual space. A carrying action is an action in which the virtual hand grasps and carries a virtual object.

For ease of explanation, the following will use an example of a game as a "combat action game in which a player character attacks and defeats enemy characters." There are no limitations to the type of game. Additionally, an example of a virtual space will be a "virtual reality space in which virtual reality (VR) can be experienced." Additionally, an example will be given of a game in which an image of the virtual space is displayed on a display from a first-person perspective and the game progresses.

In the following, an example will be given in which the predetermined part of the user's body is the "user's right hand", the virtual part corresponding to the predetermined part is the "virtual right hand", and the virtual object that can be held by the virtual hand is a "handgun". The player character can equip the handgun in his virtual right hand and attack enemy characters with the handgun.

(Gaming device)
1 illustrates a configuration of a game device 10 according to an embodiment. The game device 10 executes a game in response to operations by a user. In this example, the game device 10 is used together with a head mounted display 20, a speaker 30, and a controller 40, and is connected to these devices by wire or wirelessly. The game device 10 is an example of an acoustic control device.

The game device 10 can be, for example, a personal computer, PlayStation (registered trademark), XBox (registered trademark), PlayStation Vita (registered trademark), Nintendo Switch (registered trademark), or other commercially available device.

The game progresses on the game device 10 based on the installed game program and game data. Game devices 10 can communicate data with each other using a communication network (not shown) or a short-range wireless communication device (not shown).

[Head-mounted display]
The head mounted display 20 includes a display 21 and a motion sensor 22 .

The display 21 reproduces (displays) images. For example, the display 21 is a liquid crystal display.

The motion sensor 22 outputs a motion signal that indicates the movement of the player's head. For example, the motion sensor 22 includes an acceleration sensor that detects acceleration, a gyro sensor that detects angular velocity (changes in rotation and orientation), and the like. The motion signal output from the motion sensor 22 is transmitted to the game device 10.

As shown in FIG. 2, the head mounted display 20 has a main body 25 and a headband 26. As shown in FIG. 3, the head mounted display 20 is worn on the head of the player 50 with the main body 25 covering both eyes of the player 50. In addition, a display 21 is fixed to the inside of the main body 25 (the side facing the eyes of the player 50). The display surface of the display 21 faces both eyes of the player 50 when the head mounted display 20 is worn on the head of the player 50.

The head mounted display 20 attached to the head of the player 50 moves in conjunction with the movement of the head of the player 50. Therefore, based on the movement signal output from the movement sensor 22 included in the head mounted display 20, it is possible to derive the "position and orientation of the head of the player 50" and the "position and orientation of the display 21 included in the head mounted display 20."

In addition, vector Vd in the head mounted display 20 is a reference line of the image of the virtual space represented from the first person perspective. Vector Vd extends in the normal direction of the display surface of display 21. Vector Vd can be derived from the position and orientation of display 21 included in the head mounted display 20.

[Speaker]
The speakers 30 reproduce sound (game sounds in this example). In this example, a plurality of speakers 30 are provided so that three-dimensional sound can be reproduced.

〔controller〕
The controller 40 is operated while being held or worn by the player 50. In this example, the controller 40 is operated while being held in the hand of the player 50. As shown in FIG 1, the controller 40 includes an operator 41 and a motion sensor 42.

The operator 41 is composed of various buttons and the like. The controller 50 outputs an operation signal to the game device 10 according to an operation input to the operator 41 (e.g., pressing a button). With this configuration, the player 50 operates the operator 41 of the controller 40 to input an operation signal (a signal indicating a command, data, etc.) to the game device 10.

The motion sensor 42 outputs a motion signal that indicates the movement of the player's hand. For example, the motion sensor 22 includes an acceleration sensor that detects acceleration, a gyro sensor that detects angular velocity (changes in rotation and orientation), and the like. The motion signal output from the motion sensor 42 is transmitted to the game device 10.

The controller 40 held (or worn) in the hand of the player 50 moves in conjunction with the movement of the player's 50 hand. Therefore, the position and orientation of the player's 50 hand can be derived based on the movement signal output from the movement sensor 42 included in the controller 40. The movement sensor 42 is an example of a sensor that outputs a signal indicating the movement of a specific part of the user's body.

As shown in FIG. 3, in this example, a first controller 40a and a second controller 40b are provided as the controller 40. The configuration of each of the first controller 40a and the second controller 40b is the same as that of the controller 40.

The first controller 40a is held and operated by the right hand 55 of the player 50, and moves in conjunction with the movement of the right hand 55 of the player 50. The second controller 40b is held and operated by the left hand 56 of the player 50, and moves in conjunction with the movement of the left hand 56 of the player 50. Therefore, the position and orientation of the right hand 55 of the player 50 can be derived based on the movement signal output from the movement sensor 42 included in the first controller 40a. Also, the position and orientation of the left hand 56 of the player 50 can be derived based on the movement signal output from the movement sensor 42 included in the second controller 40b.

[Hardware configuration of the game device]
1, the game device 10 includes a network interface 11, a graphics processing unit 12, an audio processing unit 13, an operation processing unit 14, a storage unit 15, and a control unit 16. The network interface 11, the graphics processing unit 12, the audio processing unit 13, the operation processing unit 14, and the storage unit 15 are electrically connected to the control unit 16 via a bus 17.

The network interface 11 is communicatively connected to a communication network (not shown) to transmit and receive various data between, for example, other game devices 10 and an external server device (not shown).

The graphics processing unit 12 is connected to the display 21 of the head-mounted display 20 by wire or wirelessly. The graphics processing unit 12 renders game images including various objects in a virtual space in a moving image format according to game image information (image data) output from the control unit 16. The game images (images of the virtual space) rendered in a moving image format are displayed on the display 21 as a game screen.

The audio processing unit 13 is connected to the speaker 30 via a wired or wireless connection. The audio processing unit 13 plays digital game sounds according to instructions from the control unit 16. Specifically, the audio processing unit 13 converts the audio data output by the control unit 16 into an analog signal and outputs it to the speaker 30.

In this example, the audio processing unit 13 is configured to be capable of multi-channel audio output in order to reproduce three-dimensional sound. Multiple speakers 30 are connected to the game device 10. The multiple speakers 30 are positioned to the left and right, front and rear, above, etc. of the listener (player) in order to reproduce three-dimensional sound.

The operation processing unit 14 is connected to each of the head mounted display 20 and the controller 40 by wire or wirelessly. The operation processing unit 14 transmits and receives signals between each of the head mounted display 20 and the controller 40. In this example, the operation processing unit 14 transmits to the control unit 16 the operation signal output from the motion sensor 22 of the head mounted display 20, the operation signal transmitted from the controller 40 (a signal corresponding to an operation input to the operator 41), and the operation signal output from the motion sensor 42 of the controller 40.

The storage unit 15 stores various types of information and data. The storage unit 15 is composed of a HDD, SSD, RAM, ROM, etc. For example, the storage unit 15 stores game data, various programs including game programs, etc. Examples of game data include game media, user account information, etc.

The storage unit 15 also stores various types of audio data indicating various types of sounds that have been prepared in advance. The various types of audio data include audio data indicating sounds output from sound source objects, audio data indicating music (BGM) played in the virtual space, audio data indicating sound effects, and the like.

The control unit 16 controls the operation of the game device 10. The control unit 16 transmits and receives various information and data, and processes various information and data. The control unit 16 has a CPU (microcomputer) and a semiconductor memory. The semiconductor memory stores programs and data for operating the CPU.

[Functional configuration of the control section]
The control unit 16 of the game device 10 has a game progression unit 100, an image playback unit 101, a sound generation unit 102, and a sound playback unit 103. Specifically, the control unit 16 functions as the game progression unit 100, the image playback unit 101, the sound generation unit 102, and the sound playback unit 103 by executing a game program.

<Game Progression Section (Virtual Space Control Section)>
The game progression unit 100 progresses a game that takes place in a virtual space. The game progression unit 100 is an example of a virtual space control unit that controls the virtual space.

In this example, the game progression unit 100 progresses the game that takes place in a virtual space by controlling (generating or updating) image data showing the virtual space based on the movement (movement of the head of the player 50) indicated in the movement signal output from the movement sensor 22, the movement (movement of the hand of the player 50) indicated in the movement signal output from the movement sensor 42, and the operation (operation input to the operator 41 of the controller 40) indicated in the operation signal output from the controller 40 so that these movements and operations are reflected in the virtual space.

Specifically, the game progression unit 100 controls the movement of the player character in the virtual space in accordance with the above movements and operations. The game progression unit 100 also controls the movement of other objects (such as non-player characters) in the virtual space in accordance with the progress of the game. For example, AI (artificial intelligence) is used by the game progression unit 100 to control the movement of other objects.

Also, as shown in Figures 4 and 6, the game progression unit 100 controls the virtual space so that a virtual part 61 (a specified part of the player character 60) moves within the virtual space in response to the "movement of a specified part 51 of the body of the player 50" indicated in the movement signal output from the movement sensor 42.

In this example, the game progression unit 100 controls the virtual space so that a virtual right hand 65 (the right hand of the player character 60) moves within the virtual space in response to the "movement of the right hand 55 of the player 50" indicated in the movement signal output from the movement sensor 42 included in the first controller 40a. The game progression unit 100 also controls the virtual space so that a virtual left hand 66 (the left hand of the player character 60) moves within the virtual space in response to the "movement of the left hand 56 of the player 50" indicated in the movement signal output from the movement sensor 42 included in the second controller 40b.

<Image playback section>
The image playback unit 101 plays an image of the virtual space controlled by the game progression unit 100 on the display 21. In this example, the image playback unit 101 supplies image data (image data showing an image of the virtual space) controlled by the game progression unit 100 to the graphics processing unit 12. As a result, the image of the virtual space shown in the image data is played (displayed) on the display 21 of the head mounted display 20.

<Speech Generation Unit>
The sound generation unit 102 generates sounds (game sounds) in the virtual space controlled by the game progression unit 100. In this example, the sound generation unit 102 generates sounds according to the situation in the virtual space (such as the movement of objects). Specifically, the sound generation unit 102 selects sound data to be reproduced from the sound data (various types of sound data prepared in advance) stored in the storage unit 15 according to the situation in the virtual space (the progress of the game in this example), and generates sound data indicating a sound according to the situation in the virtual space based on the selected sound data (or new sound data obtained by synthesizing the selected multiple sound data).

In this example, the sound generation unit 102 also performs sound image localization processing. Sound image localization processing is processing that localizes the position of a sound source as perceived by a human (player) to a predetermined virtual position in the virtual space (e.g., the position of a sound source object).

In the sound image localization process, the sound generation unit 102 places a virtual microphone (not shown) in a virtual space and generates sound data including sound obtained by the virtual microphone. The virtual microphone is a virtual listener (sound collection point) and is the reference point for sound image localization in the sound image localization process. For example, the virtual microphone is placed near the player character and moves in accordance with the movement of the player character.

Specifically, in the sound image localization process, the sound generation unit 102 generates sound data according to the sound collection conditions of the virtual microphone (e.g., the position and direction of the virtual microphone relative to the sound source object in the virtual space, the volume and direction of the sound output from the sound source object, etc.) so that the sound played (output) from the speaker 30 includes sound obtained by the virtual microphone.

By processing the audio data generated by the sound image localization process by the audio processing unit 13, it becomes possible to output the sound output from the virtual sound source (sound source object) in the virtual space from the speaker 30 with an acoustic representation as if the sound was collected by a virtual microphone.

Furthermore, when the conditions for playing sound effects corresponding to a predetermined movement of a virtual part (a predetermined part of the player character) in the virtual space are met, the sound generation unit 102 generates sound effects corresponding to the moving speed of a predetermined part of the user's body in the real world. Specifically, when the above conditions are met, the sound generation unit 102 generates sound effects corresponding to the moving speed of the predetermined part within a predetermined period (a predetermined period starting or ending at the time when the above conditions are met) immediately before or after the above conditions are met. The generation of sound effects (sound effects corresponding to a predetermined movement of a virtual part) by the sound generation unit 102 will be described in detail later.

<Audio playback section>
The sound reproducing unit 103 reproduces the sound generated by the sound generating unit 102 (sound including sound effects corresponding to the predetermined movement of the virtual body part) on the speaker 30. In this example, the sound reproducing unit 103 supplies the sound data generated by the sound generating unit 102 to the audio processing unit 13. As a result, the sound of the virtual space indicated by the sound data (sound including sound effects corresponding to the predetermined movement of the virtual body part) is reproduced (output) from the speaker 30.

[Processing related to gripping operation]
Next, a process related to a gripping motion will be described with reference to Fig. 4. The gripping motion is a motion in which a virtual right hand 65 (virtual hand) grips a handgun 71 (virtual object 70) in a virtual space. The gripping motion is an example of a predetermined movement of the virtual part 61.

When a gripping condition, which is a condition for performing a gripping action, is met, the game progression unit 100 controls the virtual space so that the gripping action is performed. Specifically, when the gripping condition is met, the game progression unit 100 causes the virtual right hand 65 of the player character 60 to grip a handgun 71 in the virtual space.

When the gripping condition is met, the sound generation unit 102 generates a gripping sound, which is a sound effect corresponding to the gripping action, according to the movement speed of the right hand 55 (predetermined part 51) of the player 50 during the gripping start period starting from the time when the gripping condition is met. The sound processing corresponding to the gripping action will be explained in detail later.

The gripping condition is a condition in which an operation to perform a gripping motion is input to the operator 41 of the controller 40 with the virtual right hand 65 (virtual hand) entering a predetermined range R1 based on the position Q1 of the handgun 71 (virtual object 70) in the virtual space. The gripping condition is an example of a condition for playing a sound effect corresponding to a predetermined movement of the virtual part 61.

[Gripping sound]
In this example, the gripping sound is the sound (e.g., a "grab" sound) of the imaginary right hand 65 gripping the handgun 71 that is removably housed in a holster attached to the player character 60. The gripping sound is a sound effect that imitates the sound made when gripping an object (a handgun in this example) with a hand.

In the real world, the sound made when grasping an object with one's hand changes depending on the speed at which the hand moves toward the object to grasp it. For example, the sound made when grasping an object with one's hand tends to become louder and more intense the faster the hand moves toward the object. Also, the faster the hand moves toward the object, the faster the hand moves with momentum (excessive momentum) immediately after grasping the object.

The relationship between the above "sound produced when grasping an object with the hand" and the "movement speed of the hand moving toward the object in order to grasp the object (movement speed in a specified period of time)" can be derived through experiments and simulations. Similarly, the relationship between the above "sound produced when grasping an object with the hand" and the "movement speed of the hand moving with momentum immediately after grasping an object with the hand (movement speed in a specified period of time)" can be derived through experiments and simulations.

[Sound processing according to gripping motion]
Next, with reference to Fig. 5, an acoustic process (a process for generating sound effects) corresponding to a gripping operation will be described. In this example, a gripping sound (a plurality of gripping sounds) corresponding to a gripping operation of the virtual object 70 is prepared for each type of virtual object 70 and for each range of the moving speed of a predetermined part 51 in a gripping start period described later. Sound data (a plurality of sound data) indicating these gripping sounds is stored in the storage unit 15. In this example, the range of the moving speed of the virtual part 61 is three ranges, namely, a low speed range (a range of moving speed higher than zero), a medium speed range (a range of moving speed higher than the low speed range), and a high speed range (a range of moving speed higher than the medium speed range).

<Step S10>
The sound generation unit 102 determines whether or not the virtual part 61 is within a predetermined range R1 based on the position Q1 of the virtual object 70. If the virtual part 61 is within the predetermined range R1, the process of step S11 is performed.

As shown in FIG. 4, in this example, the predetermined range R1 is a spherical range centered on the position of the handgun 71, which is position Q1 of the virtual object 70, and is a range within which the virtual right hand 65, which is the virtual part 61, is deemed to be in a position where it can grip the handgun 71. The sound generation unit 102 acquires the position (e.g., coordinates) of the virtual right hand 65 in the virtual space, and when the position of the virtual right hand 65 is included within the predetermined range R1, it determines that the virtual right hand 65 has entered the predetermined range R1.

<Step S11>
The sound generation unit 102 determines whether or not an operation for gripping the virtual object 70 has been input to the controller 40. If an operation for gripping the virtual object 70 has been input to the controller 40, the process of step S12 is performed, and if not, the process of step S10 is performed.

In this example, the operation for gripping the handgun 71, which is the virtual object 70, is the operation of pressing a specific button included in the operator 41 of the controller 40. The sound generation unit 102 determines whether or not an operation for gripping the handgun 71 (the operation of pressing a specific button) has been input to the operator 41 of the controller 40, based on an operation signal (a signal corresponding to an operation input to the operator 41) output from the controller 40.

<Step S12>
When an operation to grasp the virtual object 70 is input to the controller 40 while the virtual part 61 is within the predetermined range R1, the grasping condition is met. Note that the determination of whether the grasping condition is met (the processing of steps S10 and S11) may be performed by the game progression unit 100. When the grasping condition is met, the sound generation unit 102 identifies the type of virtual object 70 that is the target of the grasping operation. In this example, the sound generation unit 102 identifies that the type of virtual object 70 that is the target of the grasping operation is a "handgun 71."

<Step S13>
Next, the sound generation unit 102 derives the "movement speed of a predetermined part 51 of the body of the player 50 in the real world" during the gripping start period, which begins at the time when the gripping condition is established.

In this example, the sound generation unit 102 derives the "movement speed of the player's 50's right hand 55 in the real world" during the grip start period based on the movement signal output from the movement sensor 42 of the first controller 40a. For example, the sound generation unit 102 derives the movement speed of the right hand 55 during the grip start period by integrating the "acceleration indicated in the movement signal output from the movement sensor 42" during the grip start period.

<Step S14>
Next, the voice generation unit 102 determines whether the moving speed of the predetermined part 51 derived in step S13 (in this example, the moving speed of the right hand 55) falls within a predetermined "low speed range,""medium speed range," or "high speed range." If the moving speed of the predetermined part 51 falls within the low speed range, processing of step S15 is performed. If the moving speed of the predetermined part 51 falls within the medium speed range, processing of step S16 is performed. If the moving speed of the predetermined part 51 falls within the high speed range, processing of step S17 is performed.

<Step S15>
When the moving speed of the predetermined part 51 (in this example, the moving speed of the right hand 55) is included in the low speed range, the voice generating unit 102 generates a gripping sound corresponding to the low speed. The voice reproducing unit 103 reproduces the "gripping sound corresponding to the low speed" generated by the voice generating unit 102 on the speaker 30.

In this example, the sound generation unit 102 selects a plurality of pieces of sound data corresponding to the type of virtual object 70 identified in step S12 (handgun 71 in this example) from among the plurality of pieces of sound data (sound data indicating gripping sounds) stored in the storage unit 15. Next, the sound generation unit 102 selects sound data indicating a "grip sound corresponding to low speed" corresponding to the low speed range from among the plurality of pieces of sound data (sound data prepared for each range of moving speeds of the virtual part 61 in the grip start period). Then, the sound reproduction unit 103 supplies the sound data indicating the "grip sound corresponding to low speed" selected by the sound generation unit 102 to the audio processing unit 13, thereby causing the speaker 30 to reproduce the grip sound corresponding to low speed.

<Step S16>
When the moving speed of the predetermined part 51 (in this example, the moving speed of the right hand 55) is within the medium speed range, the voice generating unit 102 generates a gripping sound corresponding to the medium speed. The voice reproducing unit 103 reproduces the "gripping sound corresponding to the medium speed" generated by the voice generating unit 102 on the speaker 30. For example, the gripping sound corresponding to the medium speed is more intense and louder than the gripping sound corresponding to the low speed.

The sound generating unit 102 selects, from among the multiple pieces of sound data (sound data indicating gripping sounds) stored in the storage unit 15, sound data indicating a "gripping sound corresponding to medium speed" that corresponds to the type of virtual object 70 identified in step S12 (handgun 71 in this example) and corresponds to the medium speed range. The selection of sound data indicating a "gripping sound corresponding to medium speed" by the sound generating unit 102 is similar to the selection of sound data indicating a "gripping sound corresponding to low speed" by the sound generating unit 102. The sound reproducing unit 103 supplies the sound data indicating the "gripping sound corresponding to medium speed" selected by the sound generating unit 102 to the audio processing unit 13, thereby reproducing the gripping sound corresponding to medium speed on the speaker 30.

<Step S17>
When the moving speed of the predetermined part 51 (in this example, the moving speed of the right hand 55) is included in the high speed range, the voice generating unit 102 generates a gripping sound corresponding to the high speed. The voice reproducing unit 103 reproduces the "gripping sound corresponding to the high speed" generated by the voice generating unit 102 on the speaker 30. For example, the gripping sound corresponding to the high speed is more intense and louder than the gripping sound corresponding to the medium speed.

The sound generating unit 102 selects, from among the multiple pieces of sound data (sound data indicating gripping sounds) stored in the memory unit 15, sound data indicating a "grip sound corresponding to high speed" that corresponds to the type of virtual object 70 identified in step S12 (handgun 71 in this example) and corresponds to the high speed range. The selection of sound data indicating a "grip sound corresponding to high speed" by the sound generating unit 102 is similar to the selection of sound data indicating a "grip sound corresponding to low speed" by the sound generating unit 102. The sound reproducing unit 103 supplies the sound data indicating the "grip sound corresponding to high speed" selected by the sound generating unit 102 to the audio processing unit 13, thereby reproducing the grip sound corresponding to high speed on the speaker 30.

[Processing related to carrying operation]
Next, the carrying motion will be described with reference to Fig. 6. The carrying motion is a motion in which the virtual right hand 65 (virtual hand) holds and carries a handgun 71 (virtual object 70). The carrying motion is an example of a predetermined movement of the virtual part 61.

After the gripping condition is met in the virtual space, the game progression unit 100 controls the virtual space so that the carrying action is performed when the carrying condition, which is a condition for performing the carrying action, is met. Specifically, when the carrying condition is met, the game progression unit 100 controls the movement of the virtual right hand 65 in conjunction with the movement of the right hand 55 of the player 50 so that the virtual right hand 65 grips and carries the handgun 71.

When the carrying condition is satisfied after the holding condition is satisfied, the sound generating unit 102 generates a carrying sound, which is a sound effect corresponding to the carrying action, according to the movement speed of the right hand 55 (predetermined part 51) during the carrying start period from the time when the holding condition is satisfied to the time when the carrying condition is satisfied. The sound processing corresponding to the carrying action will be described in detail later.

The carrying condition is a condition in which the virtual right hand 65 (virtual hand) continues to grip a handgun 71 (virtual object 70) in the virtual space, and the movement distance of the right hand 55 (predetermined part 51) after the gripping condition is met reaches a predetermined distance. The carrying condition is an example of a condition for playing a sound effect corresponding to a predetermined movement of the virtual part 61.

[Carrying noise]
In this example, the carrying sound is the sound (e.g., a "whoosh" sound) of handgun 71 housed in a holster attached to player character 60 being removed from the holster while rubbing against the inner surface of the holster. The carrying sound is a sound effect that imitates the sound made when carrying (moving) an object (a handgun in this example) while rubbing it with the hand.

In the real world, the sound made when carrying an object while rubbing it in the hand changes depending on the moving speed of the hand that is holding and carrying the object. For example, the sound made when carrying an object while rubbing it in the hand tends to become louder and more intense the faster the hand that is holding and carrying the object moves. The relationship between the above "sound made when carrying an object while rubbing it in the hand" and "moving speed of the hand that is holding and carrying the object (moving speed in a specified period of time)" can be derived through experiments and simulations.

[Audio processing in response to carrying movements]
Next, an acoustic process (a process for generating sound effects) corresponding to a carrying action will be described with reference to Fig. 7. In this example, carrying sounds (a plurality of carrying sounds) corresponding to the carrying action of the virtual object 70 are prepared for each type of virtual object 70 and for each range of the moving speed of a predetermined part 51 during a carrying start period described later. Audio data (a plurality of audio data) indicating these carrying sounds is stored in the storage unit 15.

<Step S20>
The voice generating unit 102 determines whether or not the gripping condition is satisfied. If the gripping condition is satisfied, the process of step S21 is performed.

<Step S21>
The sound generator 102 starts measuring the moving distance of the virtual part 61 from the gripping position (the position when the gripping motion is started) of the virtual object 70. Next, the process of step S22 is performed.

In this example, the voice generation unit 102 acquires the position (e.g., coordinates) of the virtual right hand 65, which is the virtual part 61, when the gripping action of the virtual right hand 65 to grip the handgun 71, which is the virtual object 70, is started, and sets the position of the virtual right hand 65 as the "measurement start position of the movement distance". The voice generation unit 102 then tracks the position (e.g., coordinates) of the virtual right hand 65 in the virtual space, and derives the distance of the trajectory of the movement of the virtual right hand 65 as the "movement distance of the virtual right hand 65 from the grip position of the handgun 71". Alternatively, the voice generation unit 102 may derive the straight-line distance between the current position of the virtual right hand 65 in the virtual space and the measurement start position as the "movement distance of the virtual right hand 65 from the grip position of the handgun 71".

<Step S22>
The sound generation unit 102 determines whether or not an operation for releasing the grip of the virtual object 70 has been input to the controller 40. If an operation for releasing the grip of the virtual object 70 has not been input to the controller 40, the process of step S23 is performed. On the other hand, if an operation for releasing the grip of the virtual object 70 has been input to the controller 40, the process ends.

In this example, the operation for releasing the grip of the handgun 71, which is the virtual object 70, is an operation of releasing the press of a specific button (a button pressed to grip the handgun 71) included in the operator 41 of the controller 40. The sound generation unit 102 determines whether or not the operation for gripping the handgun 71 (the operation of pressing the specific button) has been released, based on an operation signal (a signal corresponding to an operation input to the operator 41) output from the controller 40. If the operation for gripping the handgun 71 is continued without being released, the process of step S23 is performed. On the other hand, if the operation for gripping the handgun 71 is released, the process ends.

<Step S23>
Next, the sound generation unit 102 determines whether or not the movement distance of the virtual part 61 (the virtual right hand 65 in this example) from the grip position of the virtual object 70 (the handgun 71 in this example) has reached a predetermined distance. If the movement distance of the virtual part 61 from the grip position of the virtual object 70 has reached the predetermined distance, the process of step S24 is performed, and if not, the process of step S22 is performed.

<Step S24>
When the movement distance of the virtual part 61 from the gripping position of the virtual object 70 reaches a predetermined distance without an operation for releasing the grip of the virtual object 70 being input to the controller 40, the carrying condition is met. Note that the determination of whether or not the carrying condition is met (the processing of steps S21 to S23) may be performed by the game progression unit 100. When the gripping condition is met, the sound generation unit 102 identifies the type of virtual object 70 that is the target of the carrying action. In this example, the sound generation unit 102 identifies the type of virtual object 70 that is the target of the carrying action as "handgun 71."

<Step S25>
Next, the sound generation unit 102 derives the "movement speed of a predetermined part 51 of the body of the player 50 in the real world" during the carrying start period from the point when the gripping condition is met to the point when the carrying condition is met.

In this example, the sound generation unit 102 derives the "movement speed of the player's 50's right hand 55 in the real world" during the start of carrying period based on the movement signal output from the movement sensor 42 of the first controller 40a. For example, the sound generation unit 102 derives the movement speed of the right hand 55 during the start of carrying period by integrating the "acceleration indicated in the movement signal output from the movement sensor 42" during the start of carrying period.

<Step S26>
Next, the voice generation unit 102 determines whether the moving speed of the predetermined part 51 derived in step S25 (in this example, the moving speed of the right hand 55) falls within a predetermined "low speed range,""medium speed range," or "high speed range." If the moving speed of the predetermined part 51 falls within the low speed range, processing of step S27 is performed. If the moving speed of the predetermined part 51 falls within the medium speed range, processing of step S28 is performed. If the moving speed of the predetermined part 51 falls within the high speed range, processing of step S29 is performed.

<Step S27>
When the moving speed of the predetermined part 51 (in this example, the moving speed of the right hand 55) is included in the low speed range, the sound generating unit 102 generates a carrying sound corresponding to the low speed. The sound reproducing unit 103 causes the speaker 30 to reproduce the "carrying sound corresponding to the low speed" generated by the sound generating unit 102.

In this example, the sound generation unit 102 selects a plurality of pieces of sound data corresponding to the type of virtual object 70 identified in step S24 (handgun 71 in this example) from among the plurality of pieces of sound data (sound data indicating the carrying sound) stored in the storage unit 15. Next, the sound generation unit 102 selects sound data indicating a "carrying sound corresponding to low speed" corresponding to the low speed range from among the plurality of pieces of sound data (sound data prepared for each range of the moving speed of the virtual part 61 during the carrying start period). Then, the sound reproduction unit 103 supplies the sound data indicating the "carrying sound corresponding to low speed" selected by the sound generation unit 102 to the audio processing unit 13, thereby causing the speaker 30 to reproduce the carrying sound corresponding to low speed.

<Step S28>
When the moving speed of the predetermined part 51 (in this example, the moving speed of the right hand 55) is within the medium speed range, the voice generating unit 102 generates a carrying sound corresponding to the medium speed. The voice reproducing unit 103 causes the speaker 30 to reproduce the "carrying sound corresponding to the medium speed" generated by the voice generating unit 102. For example, the carrying sound corresponding to the medium speed is more intense and louder than the carrying sound corresponding to the low speed.

The sound generation unit 102 selects, from among the multiple pieces of sound data (sound data indicating carrying sounds) stored in the storage unit 15, sound data indicating a "carrying sound corresponding to medium speed" that corresponds to the type of virtual object 70 identified in step S24 (handgun 71 in this example) and corresponds to the medium speed range. The selection of sound data indicating a "carrying sound corresponding to medium speed" by the sound generation unit 102 is similar to the selection of sound data indicating a "carrying sound corresponding to low speed" by the sound generation unit 102. The sound reproduction unit 103 supplies the sound data indicating the "carrying sound corresponding to medium speed" selected by the sound generation unit 102 to the audio processing unit 13, thereby causing the speaker 30 to reproduce the carrying sound corresponding to medium speed.

<Step S29>
When the moving speed of the predetermined part 51 (in this example, the moving speed of the right hand 55) is included in the high speed range, the voice generating unit 102 generates a carrying sound corresponding to the high speed. The voice reproducing unit 103 causes the speaker 30 to reproduce the "carrying sound corresponding to the high speed" generated by the voice generating unit 102. For example, the carrying sound corresponding to the high speed is more intense and louder than the carrying sound corresponding to the medium speed.

The sound generation unit 102 selects, from among the multiple pieces of sound data (sound data indicating carrying sounds) stored in the storage unit 15, sound data indicating a "carrying sound corresponding to high speed" that corresponds to the type of virtual object 70 identified in step S24 (handgun 71 in this example) and corresponds to the high speed range. The selection of sound data indicating a "carrying sound corresponding to high speed" by the sound generation unit 102 is similar to the selection of sound data indicating a "carrying sound corresponding to low speed" by the sound generation unit 102. The sound reproduction unit 103 supplies the sound data indicating the "carrying sound corresponding to high speed" selected by the sound generation unit 102 to the audio processing unit 13, thereby causing the speaker 30 to reproduce a gripping sound corresponding to high speed.

[Overview of the embodiment]
In summary, the program (game program) of the embodiment is a program that causes a computer system (game device 10) having one computer or multiple computers working together to function as a virtual space control unit (game progression unit 100), an image reproduction unit 101, a sound generation unit 102, and a sound reproduction unit 103. The computer system is used together with a display 21, a speaker 30, and a sensor (motion sensor 42). The sensor outputs a signal indicating the movement of a predetermined part 51 in the body of a user (player 50). The virtual space control unit controls the virtual space so that a virtual part 61 moves in the virtual space according to the movement of the predetermined part 51 indicated by the signal output from the sensor. The image reproduction unit 101 reproduces an image of the virtual space on the display 21. The sound generation unit 102 generates a sound effect according to the moving speed of the predetermined part 51 when a condition for reproducing a sound effect according to the predetermined movement of the virtual part 61 is established. The sound reproduction unit 103 reproduces the sound effect generated by the sound generation unit 102 on the speaker 30.

The acoustic control device (game device 10) of the embodiment includes a storage unit 15 that stores the above program, and a control unit 16 that executes the above program.

[Effects of the embodiment]
In the game device 10 of the embodiment, when a condition for playing a sound effect corresponding to a predetermined movement of a virtual part in the virtual space (a virtual part that moves in conjunction with the movement of a predetermined part of the player's body) is met, the above sound effect is generated according to the moving speed of the predetermined part of the player's body in the real world. This can improve the user experience. For example, the reality of the sound effect can be improved, thereby improving interest.

In addition, in the game device 10 of the embodiment, a gripping sound corresponding to a gripping action in which a virtual hand grips a virtual object in a virtual space, and a carrying sound corresponding to a carrying action in which the virtual hand carries the virtual object while gripping it, are generated separately. This can further improve the user experience. For example, the realism of sound effects corresponding to a series of actions, such as a gripping action followed by a carrying action, can be improved, thereby improving interest.

Other Embodiments
In the above description, the following configuration or setting may be adopted.

The predetermined part of the player's (user's) body is not limited to the hands. For example, the predetermined part of the player's body may be the player's torso, the player's legs, or some other part.

The sensor that outputs a signal indicating the movement of a specific part of the player's (user's) body is not limited to the motion sensor 42 of the controller 40. For example, such a sensor may be a combination of a camera (image sensor) that captures an image of a specific part of the player's body, and a recognition processing unit that recognizes the movement of the specific part based on the "image of the specific part" obtained by the camera.

The virtual object that can be held by the virtual hand is not limited to a firearm such as a handgun. For example, such a virtual object may be another weapon such as a knife, may be armor worn by the player character, or may be another object (virtual object) that is not a weapon or armor.

The voice data does not necessarily need to be prepared in advance. For example, the voice generation unit 102 may synthesize or generate voice data each time based on some data (e.g., voice data stored in the storage unit 15) or information.

The speaker 30 may be a stationary speaker or a wearable speaker such as an earphone or a headphone.

The player character may or may not be displayed on the display. In other words, the player character may or may not be included in the image of the virtual space that is displayed (played) on the display. For example, when an image of the virtual space is displayed on the display from a first-person perspective, only the virtual hand (virtual body part), which is part of the player character, may be displayed, or the virtual hand may not be displayed and only a handgun (virtual object) held in the virtual hand may be displayed.

The image of the virtual space may be from a first-person perspective or a third-person perspective.

In the above description, the game program may be implemented as a game program for a so-called online game. In this case, the processing performed in the control unit 16 of the game device 10 may be performed in a control unit of a server device (not shown), or may be shared between the game device 10 and the control unit of the server device. The game device 10 may be a mobile information terminal such as a smartphone or tablet.

In the above explanation, a virtual space in which a game progresses (game space) has been given as an example of a virtual space, but this is not limiting. For example, the virtual space may be a virtual space in which avatars communicate with each other (metaverse space). In other words, the services provided in the virtual space are not limited to games. For example, the services provided in the virtual space may be the provision of a place for commercial transactions, the provision of a place for communication, etc.

Even when these other embodiments are adopted, the effects of the present invention are achieved. In addition, this embodiment can be combined with other embodiments, and other embodiments can be combined with each other as appropriate. The above embodiments are essentially preferred examples, and are not intended to limit the scope of the present invention, its applications, or its uses.

10 Game device (sound control device)
11 Network interface 12 Graphics processing unit 13 Audio processing unit 14 Operation processing unit 20 Head mounted display 21 Display 22 Motion sensor 25 Main body unit 26 Headband 30 Speaker 40 Controller 40a First controller 40b Second controller 41 Operator 42 Motion sensor (sensor)
50 Player (user)
51 designated part 55 right hand (hand)
56 Left hand (hand)
60 Player character (virtual user)
61 Virtual part 65 Virtual right hand (virtual hand)
66 Virtual left hand (virtual hand)
70 Virtual object 71 Handgun 100 Game progression unit (virtual space control unit)
101 Image playback unit 102 Audio generation unit 103 Audio playback unit

Claims (8)

A program for causing a computer system having one computer or a plurality of computers working together to function as a virtual space control unit, an image playback unit, an audio generation unit, and an audio playback unit,
The computer system is used with a display, a speaker, and a sensor;
The sensor outputs a signal indicative of a movement of a predetermined part of the user's body;
the virtual space control unit controls the virtual space such that a virtual part moves within the virtual space in response to the movement of the predetermined part indicated by the signal;
The image playback unit plays an image of the virtual space on the display,
the sound generation unit generates a sound effect according to a moving speed of the virtual body part when a condition for playing the sound effect according to a predetermined movement of the virtual body part is satisfied;
The sound playback unit is a program that causes the speaker to play back the sound effects generated by the sound generation unit. In the program of claim 1,
the computer system is used together with a controller that is worn or held by the user and operated;
The sensor is included in the controller;
The virtual space control unit is a program that controls the virtual space based on the signal output from the sensor and the operation input to the controller. In the program of claim 2,
the predetermined part is a hand,
The program, wherein the virtual part is a virtual hand. In the program of claim 3,
a virtual object that can be held by the virtual hand is placed in the virtual space;
the virtual space control unit controls the virtual space such that, when a gripping condition is satisfied, the gripping operation is performed such that the virtual hand grips the virtual object in the virtual space;
The sound generation unit is a program that, when the gripping condition is satisfied, generates a gripping sound, which is a sound effect corresponding to the gripping action, depending on the movement speed of the specified part during a gripping start period starting from the time when the gripping condition is satisfied. In the program of claim 4,
The program, wherein the grasping condition is a condition that an operation to perform the grasping action is input to the controller with the virtual hand entering a predetermined range based on the position of the virtual object in the virtual space. In the program of claim 4,
the virtual object is portable in a state of being held by the virtual hand;
the virtual space control unit, when a carrying condition is satisfied that is a condition for performing a carrying action in which the virtual hand carries the virtual object in a state of holding the virtual object after the grasping condition is satisfied in the virtual space, controls the virtual space so that the carrying action is performed;
The sound generation unit is a program that, when the carrying condition is met after the holding condition is met, generates a carrying sound, which is a sound effect corresponding to the carrying action, depending on the movement speed of the specified part during the carrying start period from the time when the holding condition is met to the time when the carrying condition is met. In the program of claim 6,
The program, wherein the carrying condition is a condition that, while the virtual hand is continuing to grasp the virtual object in the virtual space, a movement distance of the specified part after the grasping condition is satisfied reaches a specified distance. A storage unit that stores the program according to any one of claims 1 to 7;
and a control unit for executing the program.