JP2025091813A - Program, sound control device, and system - Google Patents

Abstract

[Problem] To improve user experience. [Solution] A detection unit detects the orientation of a user P1 based on a signal output from a sensor. A display processing unit changes the orientation of a virtual camera 61 arranged in a virtual space so as to follow the orientation of the user P1 detected by the detection unit, and causes an image of the virtual space obtained by the virtual camera 61 to be displayed on a display. An audio processing unit causes a plurality of speakers 30 to output sounds in the virtual space so that the orientation of a sound field formed by sounds in the virtual space output from the plurality of speakers 30 does not change even if the orientation of the virtual camera 61 changes. [Selected Figure] Figure 8

Description

This disclosure relates to a program, an audio control device, and a system.

Some game programs play sounds within a virtual space. For example, in Patent Document 1, sounds such as gunshots are played in a shooting game.

JP 2014-094160 A

It is conceivable that a system equipped with a display (e.g., a head-mounted display) that follows the user's orientation and multiple speakers arranged around the user could be used to provide the user with images and sounds from a virtual space in which various services (e.g., games) are offered. From the perspective of user experience, in such a system, it is desirable that sounds from the virtual space corresponding to images in the virtual space displayed on the display are reproduced from multiple speakers in an appropriate position.

The purpose of this disclosure is to improve the user experience in services provided in virtual space.

A first aspect is a program for causing a computer system having one or more computers working together to function as a detection unit, a display processing unit, and an audio processing unit, the program comprising:
The computer system is used with a display that tracks the orientation of a user, a plurality of speakers arranged around the user, and a sensor that outputs a signal indicative of the user's movements, including changes in the user's orientation;
The detection unit detects a direction of the user based on a signal output from the sensor,
the display processing unit changes an orientation of a virtual camera disposed in a virtual space so as to follow the orientation of the user detected by the detection unit, and displays an image of the virtual space obtained by the virtual camera on the display;
The acoustic processing unit is a program that causes the multiple speakers to output audio within the virtual space so that the direction of the sound field formed by the audio within the virtual space output from the multiple speakers does not change even when the direction of the virtual camera changes.

In a first aspect,
the acoustic processing unit may move a virtual microphone disposed in the virtual space so as to follow movement of the virtual camera, and output sound in the virtual space obtained by the virtual microphone to the plurality of speakers,
The orientation of the virtual microphone may be a reference direction for the orientation of the virtual microphone in the virtual space, and may be maintained in a virtual reference direction that does not change in response to changes in the orientation of the user detected by the detection unit.

In a first aspect,
The computer system may be used in conjunction with a user-operated controller;
When a direction change operation for changing a direction of the virtual camera from a front direction corresponding to the direction of the user detected by the detection unit is input to the controller, the display processing unit may change the direction of the virtual camera from the front direction in response to the direction change operation,
When a direction of the virtual camera changes from the front direction in response to the direction change operation, the sound processing unit may change a direction of the virtual microphone to follow the direction of the virtual camera.

In a first aspect,
When a reset operation for resetting an orientation of the virtual camera to the front direction is input to the controller, the display processing unit may reset the orientation of the virtual camera to the front direction in response to the reset operation,
When the orientation of the virtual camera is reset to the front direction in response to the reset operation, the sound processing unit may reset the orientation of the virtual microphone to the virtual reference direction.

A second aspect includes a storage unit that stores the program of the first aspect;
and a control unit for executing the program.

The third aspect is
A display that follows the user's orientation;
A plurality of speakers arranged around the user;
a sensor that outputs a signal indicative of the user's movement, including a change in the user's orientation;
A control unit.
The control unit is
A detection process for detecting a direction of the user based on a signal output from the sensor;
a display process of changing an orientation of a virtual camera arranged in a virtual space so as to follow the orientation of the user detected by the detection process, and displaying an image of the virtual space obtained by the virtual camera on the display;
The system performs acoustic processing to output sound within the virtual space to the multiple speakers so that the direction of the sound field formed by the sound within the virtual space output from the multiple speakers does not change even when the direction of the virtual camera changes.

This disclosure can improve the user experience.

FIG. 1 is a block diagram illustrating a configuration of a game system according to an embodiment. FIG. 1 is a diagram illustrating an example of the appearance of a head mounted display. 1 is a schematic diagram illustrating a player wearing a head mounted display and a controller. FIG. FIG. 1 is a schematic diagram illustrating the directions of multiple speakers and the orientation of a player. 1 is a schematic diagram illustrating directions in a virtual space, the orientation of a player character, the orientation of a virtual camera, and the orientation of a virtual microphone. 11A to 11C are schematic diagrams illustrating examples of the orientation of a virtual camera, a virtual microphone, and a sound direction of a speaker when a player faces forward in an embodiment. 13 is a schematic diagram illustrating the direction of the virtual camera, the direction of the virtual microphone, and the direction of the sound from the speaker when the player faces to the right in the comparative example. FIG. 11 is a schematic diagram illustrating an example of the direction of a virtual camera, a virtual microphone, and a sound direction of a speaker when a player faces right in an embodiment. FIG. 11A to 11C are schematic diagrams illustrating examples of the orientation of the virtual camera, the orientation of the virtual microphone, the orientation of the sound of the speaker, and an image displayed on the display after a direction change operation is input.

The following describes the embodiment in detail with reference to the drawings. Note that the same or corresponding parts in the drawings are given the same reference numerals and their description will not be repeated. In addition, the following describes an example in which the program is implemented as a game program, the sound control device is implemented as a game device, and the system is implemented as a game system. The game device is an example of a computer system having one computer or multiple computers working together.

(Game Description)
In the following description, a game refers to a game (video game) that is played in a virtual space. The virtual space is a three-dimensional game space, and is displayed as an image on a display. Objects are placed in the virtual space. Examples of objects include a player character operated by a player (a user who plays the game) and a non-player character operated by a computer. A player is an example of a user.

The objects also include sound source objects that act as virtual sound sources in the virtual space. Examples of sound source objects include objects that operate in the virtual space and objects that only output sound in the virtual space. Player characters and non-player characters can be sound source objects. In the games described below, sounds are played. Sounds played in the game include sounds output from sound source objects, background music (BGM), sound effects, and the like.

In the game described below, the player character is a virtual player (virtual human) placed in a virtual space, and moves in the virtual space in response to the operation and movement of the player in the real world. Specifically, a virtual part moves in the virtual space in response to the movement of a specific part of the player's body. The virtual part is a virtual part (a specific part of the player character's body) that corresponds to a specific part of the player. For example, the virtual head (virtual head) of the player character moves in the virtual space in response to the movement of the player's head in the real world.

For ease of explanation, the following will use an example of a game as a "combat action game in which a player character attacks and defeats enemy characters." There are no limitations to the type of game. Additionally, an example of a virtual space will be a "virtual reality space in which virtual reality (VR) can be experienced." Additionally, an example will be given of a case in which an image of the virtual space is displayed on a display from a first-person perspective as the game progresses.

(Game System)
1 illustrates an example of the configuration of a game system 1 according to an embodiment. The game system 1 includes a game device 10, a head-mounted display 20, a plurality of speakers 30, and a controller 40.

[Game device]
The game device 10 executes a game in response to operations by a user. In this example, the game device 10 is used together with a head mounted display 20, a plurality of speakers 30, and a controller 40, and is connected to these devices by wire or wirelessly. The game device 10 is an example of an acoustic control device.

The game device 10 can be, for example, a personal computer, PlayStation (registered trademark), XBox (registered trademark), PlayStation Vita (registered trademark), Nintendo Switch (registered trademark), or other commercially available device.

The game progresses on the game device 10 based on the installed game program and game data. Game devices 10 can communicate data with each other using a communication network (not shown) or a short-range wireless communication device (not shown).

[Head-mounted display]
The head mounted display 20 includes a display 21 and a motion sensor 22 .

The display 21 plays (displays) images. For example, the display 21 is a liquid crystal display. The display 21 is an example of a display that follows the player's orientation.

The motion sensor 22 outputs a motion signal that indicates the movement of the player's head. For example, the motion sensor 22 includes an acceleration sensor that detects acceleration, a gyro sensor that detects angular velocity (changes in rotation and orientation), and the like. The motion signal output from the motion sensor 22 is transmitted to the game device 10.

As shown in FIG. 2, the head mounted display 20 has a main body 25 and a headband 26. As shown in FIG. 3, the head mounted display 20 is worn on the head of the player P1 with the main body 25 covering both eyes of the player P1. In addition, a display 21 is fixed to the inside of the main body 25 (the side facing the eyes of the player P1). The display surface of the display 21 faces the eyes of the player P1 when the head mounted display 20 is worn on the head of the player P1.

The head mounted display 20 attached to the head of the player P1 moves in conjunction with the movement of the player P1's head. Therefore, the "position and orientation of the head of the player P1" and the "position and orientation of the display 21 included in the head mounted display 20" can be derived based on the movement signal output from the movement sensor 22 included in the head mounted display 20. The movement sensor 22 is an example of a sensor that outputs a signal indicating the movement of the player P1, including changes in the orientation of the player P1.

In addition, vector Vd in the head mounted display 20 is a reference line of the image of the virtual space represented from the first person perspective. Vector Vd extends in the normal direction of the display surface of display 21. Vector Vd can be derived from the position and orientation of display 21 included in the head mounted display 20.

[Speaker]
The multiple speakers 30 are arranged around the player and play back sounds (game sounds in this example). In this example, the multiple speakers 30 are provided so that sounds can be played back three-dimensionally.

As shown in FIG. 4, in this example, the multiple speakers 30 are composed of a front center speaker 50, a front right speaker 51, a front left speaker 52, a surround right speaker 53, a surround left speaker 54, a surround backright speaker 55, and a surround back left speaker 56.

The front center speaker 50 is positioned in front of reference point Q. The front right speaker 51 is positioned to the right of reference point Q. The front left speaker 52 is positioned to the left of reference point Q. The surround right speaker 53 is positioned to the right of reference point Q. The surround left speaker 54 is positioned to the left of reference point Q. The surround backright speaker 55 is positioned to the right of reference point Q. The surround back left speaker 56 is positioned to the left of reference point Q.

The reference point Q is a reference point (position) for determining the directions (front, back, left, right) of the multiple speakers 30. For example, the reference point Q is the initial position (reference position) of the player. The directions of the multiple speakers 30 will be explained in detail later.

〔controller〕
The controller 40 is operated while being held or worn by a player. In this example, the controller 40 is operated while being held in the player's hand. As shown in FIG 1, the controller 40 includes an operator 41 and a motion sensor 42.

The operators 41 are composed of various buttons and the like. The controller 40 outputs an operation signal to the game device 10 according to an operation input to the operators 41 (e.g., pressing a button). With this configuration, the player inputs an operation signal (a signal indicating a command, data, etc.) to the game device 10 by operating the operators 41 of the controller 40.

The motion sensor 42 outputs a motion signal that indicates the movement of the player's hand. For example, the motion sensor 22 includes an acceleration sensor that detects acceleration, a gyro sensor that detects angular velocity (changes in rotation and orientation), and the like. The motion signal output from the motion sensor 42 is transmitted to the game device 10.

The controller 40 held (or worn) by the player's hand moves in conjunction with the movement of the player's hand. Therefore, the position and orientation of the player's hand can be derived based on the movement signal output from the movement sensor 42 included in the controller 40.

As shown in FIG. 3, in this example, a first controller 40a and a second controller 40b are provided as the controller 40. The configuration of each of the first controller 40a and the second controller 40b is the same as that of the controller 40.

The first controller 40a is held and operated in the right hand of the player P1, and moves in conjunction with the movement of the player P1's right hand. The second controller 40b is held and operated in the left hand of the player P1, and moves in conjunction with the movement of the player P1's left hand. Therefore, the position and orientation of the player P1's right hand can be derived based on the movement signal output from the movement sensor 42 included in the first controller 40a. Also, the position and orientation of the player P1's left hand can be derived based on the movement signal output from the movement sensor 42 included in the second controller 40b.

[Hardware configuration of the game device]
1, the game device 10 includes a network interface 11, a graphics processing unit 12, an audio processing unit 13, an operation processing unit 14, a storage unit 15, and a control unit 16. The network interface 11, the graphics processing unit 12, the audio processing unit 13, the operation processing unit 14, and the storage unit 15 are electrically connected to the control unit 16 via a bus 17.

The network interface 11 is communicatively connected to a communication network (not shown) to transmit and receive various data, for example, between other game devices 10 and external server devices (not shown).

The graphics processing unit 12 is connected to the display 21 of the head-mounted display 20 by wire or wirelessly. The graphics processing unit 12 renders game images including various objects in a virtual space in a moving image format according to game image information (image data) output from the control unit 16. The game images (images of the virtual space) rendered in a moving image format are displayed on the display 21 as a game screen.

The audio processing unit 13 is connected to multiple speakers 30 via a wired or wireless connection. The audio processing unit 13 plays digital game sounds according to instructions from the control unit 16. Specifically, the audio processing unit 13 converts the audio data output by the control unit 16 into an analog signal and outputs it to the multiple speakers 30. In this example, the audio processing unit 13 is configured to be able to output multi-channel audio in order to play back three-dimensional sound.

The operation processing unit 14 is connected to each of the head mounted display 20 and the controller 40 by wire or wirelessly. The operation processing unit 14 transmits and receives signals between each of the head mounted display 20 and the controller 40. In this example, the operation processing unit 14 transmits to the control unit 16 the operation signal output from the motion sensor 22 of the head mounted display 20, the operation signal transmitted from the controller 40 (a signal corresponding to an operation input to the operator 41), and the operation signal output from the motion sensor 42 of the controller 40.

The storage unit 15 stores various types of information and data. The storage unit 15 is composed of a HDD, SSD, RAM, ROM, etc. For example, the storage unit 15 stores game data, various programs including game programs, etc. Examples of game data include game media, user account information, etc.

The storage unit 15 also stores various types of audio data indicating various types of sounds that have been prepared in advance. The various types of audio data include audio data indicating sounds output from sound source objects, audio data indicating music (BGM) played in the virtual space, audio data indicating sound effects, and the like.

The control unit 16 controls the operation of the game device 10. The control unit 16 transmits and receives various information and data, and processes various information and data. The control unit 16 has a CPU (microcomputer) and a semiconductor memory. The semiconductor memory stores programs and data for operating the CPU.

[Functional configuration of the control unit]
The control unit 16 of the game device 10 has a game progression unit 100, a detection unit 101, a display processing unit 102, and a sound processing unit 103. Specifically, the control unit 16 functions as the game progression unit 100, the detection unit 101, the display processing unit 102, and the sound processing unit 103 by executing a game program.

In other words, the control unit 16 performs game progression processing (virtual space control processing), detection processing, display processing, and sound processing. The detection unit 101 performs detection processing. The game progression processing is processing by the game progression unit 100. The detection processing is processing by the detection unit 101. The display processing is processing by the display processing unit 102. The sound processing is processing by the sound processing unit 103.

<Game Progression Section (Virtual Space Control Section)>
The game progression unit 100 progresses a game that takes place in a virtual space. The game progression unit 100 is an example of a virtual space control unit that controls the virtual space. The game progression processing is an example of virtual space control processing (processing by the virtual space control unit).

In this example, the game progression unit 100 progresses (controls) the game taking place in the virtual space by controlling (generating or updating) the three-dimensional data representing the virtual space based on the movements (movement of the player's head) indicated in the movement signals output from the movement sensor 22, the movements (movement of the player's hands) indicated in the movement signals output from the movement sensor 42, and the operations (operations input to the operators 41 of the controller 40) indicated in the operation signals output from the controller 40 so that these movements and operations are reflected in the virtual space.

The three-dimensional data representing the virtual space includes well-known data for constructing the virtual space, such as data on a coordinate system (e.g., a world coordinate system) that serves as the reference for directions within the virtual space, data on the coordinate systems (e.g., a local coordinate system) of various objects placed within the virtual space, and data on the shapes of various objects. By controlling the three-dimensional data representing the virtual space, it is possible to control the movement of objects within the virtual space.

For example, the game progression unit 100 controls the movement of the player character in the virtual space in accordance with the above movements and operations. The game progression unit 100 also controls the movement of other objects (such as non-player characters) in the virtual space in accordance with the progress of the game. For example, AI (artificial intelligence) is used to control the movement of other objects by the game progression unit 100.

Specifically, the game progression unit 100 controls the virtual space so that a virtual part (a specific part of the player character) moves within the virtual space in response to the "movement of a specific part of the player's body (head, right hand, left hand)" indicated in the movement signal output from the movement sensor (movement sensor 22 or movement sensor 42). For example, the game progression unit 100 controls the virtual space so that a virtual head (the head of the player character) moves within the virtual space in response to the "movement of the player's head" indicated in the movement signal output from the movement sensor 22. By moving the virtual head, the orientation of the player character can be changed.

<Detection section>
The detection unit 101 detects the orientation of the player based on the signal output from the motion sensor 22. The orientation of the player is the direction in which the front (front face) of the player faces. In this example, the orientation of the player is a vector Vd extending in the normal direction of the display surface of the display 21. For example, the detection unit 101 derives the position and orientation of the display 21 included in the head mounted display 20 based on the motion signal (acceleration and angular velocity) output from the motion sensor 22 included in the head mounted display 20, and derives the vector Vd based on the position and orientation of the display 21.

Display Processing Section
The display processing unit 102 generates an image (a game image in this example) of the virtual space controlled by the game progression unit 100. Specifically, the display processing unit 102 generates image data showing an image of the virtual space corresponding to the vector Vd (vector Vd indicating the orientation of the player) detected by the detection unit 101, based on the three-dimensional data (three-dimensional data indicating the virtual space) controlled by the game progression unit 100 and the vector Vd (vector Vd indicating the orientation of the player). The process of generating an image corresponding to the vector Vd from the three-dimensional data is a well-known process (such as coordinate transformation or projection transformation).

Then, the display processing unit 102 reproduces (displays) the image generated as described above on the display 21. In this example, the display processing unit 102 supplies the image data generated as described above to the graphics processing unit 12. As a result, the image of the virtual space shown in the image data is reproduced (displayed) on the display 21 of the head-mounted display 20.

As shown in FIG. 5, a virtual camera 61 is placed in the virtual space. The virtual camera 61 is placed near the player character C1. The virtual camera 61 is a virtual viewer (viewpoint), and is, for example, a reference point for reproducing the viewpoint (first-person viewpoint) of the player character. The display processing unit 102 changes the orientation of the virtual camera 61 so as to follow the orientation of the player detected by the detection unit 101. The display processing unit 102 also moves the virtual camera 61 so as to follow the movement of the player character C1. Then, the display processing unit 102 displays an image of the virtual space obtained by the virtual camera 61 on the display 21.

The process of deriving the orientation of the virtual camera 61 from the orientation of the player (e.g., vector Vd) is a well-known process (such as coordinate conversion). The movement of the player character C1 is performed in accordance with the movement or operation of the player (operation input to the controller 40).

Specifically, the display processing unit 102 generates image data showing the image according to the shooting conditions of the virtual camera 61 (e.g., the position and direction of the virtual camera 61 relative to the object in the virtual space, the orientation of the virtual camera 61, the orientation of the object, etc.) so that the image displayed on the display 21 is an image including an image obtained by the virtual camera 61. The image (image data) obtained by the virtual camera 61 is an image (image data) generated according to the shooting conditions of the virtual camera 61.

<Audio Processing Section>
The sound processing unit 103 generates sounds (game sounds in this example) in the virtual space controlled by the game progression unit 100. The sounds in the virtual space are sounds corresponding to the situation in the virtual space (such as the movement of objects). Specifically, the sound processing unit 103 selects sound data to be played from the sound data (various types of sound data prepared in advance) stored in the storage unit 15 according to the situation in the virtual space (the progress of the game in this example), and generates sound data indicating sound corresponding to the situation in the virtual space based on the selected sound data (or new sound data obtained by synthesizing the selected multiple sound data).

Then, the acoustic processing unit 103 reproduces (outputs) the audio generated as described above from the multiple speakers 30. In this example, the acoustic processing unit 103 supplies the audio data generated as described above to the audio processing unit 13. As a result, the audio of the virtual space represented in the audio data is reproduced (output) from the multiple speakers 30.

Sound field maintenance processing
The sound processing unit 103 performs a sound field maintenance process. In the sound field maintenance process, the sound processing unit 103 causes the multiple speakers 30 to output sounds in the virtual space so that the direction of the sound field formed by the sounds in the virtual space output from the multiple speakers 30 does not change even if the direction of the virtual camera 61 changes. The sound field maintenance process is included in the sound processing.

Sound image localization processing
In this example, the sound processing unit 103 performs sound image localization processing. Sound image localization processing is processing for localizing the position of a sound source as perceived by a human (player) at a predetermined virtual position (e.g., the position of a sound source object) in a virtual space. Sound image localization processing is included in sound processing.

As shown in FIG. 5, a virtual microphone 62 is placed in the virtual space. The virtual microphone 62 is placed near the player character C1 together with the virtual camera 61. The virtual microphone 62 is a virtual listener (sound collection point) and is a reference point for sound image localization in the sound image localization process. The sound processing unit 103 moves the virtual microphone 62 so as to follow the movements of the player character C1 and the virtual camera 61. The sound processing unit 103 then reproduces (outputs) the sound obtained by the virtual microphone 62 to the multiple speakers 30.

Specifically, in the sound image localization process, the acoustic processing unit 103 generates audio data indicating the above-mentioned sound according to the sound collection conditions of the virtual microphone 62 (e.g., the position and direction of the virtual microphone 62 relative to the sound source object in the virtual space, the volume and direction of the sound output from the sound source object, the orientation of the virtual microphone 62, the orientation of the sound source object, etc.) so that the sound played (output) from the multiple speakers 30 is sound including the sound obtained by the virtual microphone 62. The sound (audio data) obtained by the virtual microphone 62 is sound (audio data) generated according to the sound collection conditions of the virtual microphone 62. The sound output from the sound source object is sound associated with the sound source object (sound prepared in advance).

By processing the audio data generated by the sound image localization process by the audio processing unit 13, it becomes possible to output the sound output from a virtual sound source (sound source object) in the virtual space from multiple speakers 30 with an acoustic representation as if the sound was collected by a virtual microphone 62.

In this example, the acoustic processing unit 103 performs a sound field maintenance process in the sound image localization process. The sound field maintenance process is a process for maintaining (fixing) the orientation of the virtual microphone 62 in the sound image localization process in a virtual reference direction. Specifically, the acoustic processing unit 103 moves the virtual microphone 62 arranged in the virtual space so as to follow the movement of the virtual camera 61, and outputs the sound in the virtual space obtained by the virtual microphone 62 to the multiple speakers 30. The orientation of the virtual microphone 62 is maintained (fixed) in a predetermined virtual reference direction. In other words, the acoustic processing unit 103 moves the virtual microphone 62 so as to follow the movement of the virtual camera 61 while maintaining the orientation of the virtual microphone 62 in the virtual reference direction.

The virtual reference direction is a reference direction for the orientation of the virtual microphone 62 in the virtual space, and is a direction that does not change according to changes in the orientation of the player detected by the detection unit 101. The virtual reference direction will be explained in detail later.

[Various directions]
4 , the directions (front, back, left, right) of the multiple speakers 30 are determined based on a reference point Q surrounded by the multiple speakers 30. For example, the "front" of the multiple speakers 30 corresponds to the front of the reference point Q, and the speaker 30 located "forward" among the multiple speakers 30 is a "front center speaker 50" disposed in front of the reference point Q.

The orientation of player P1 is the direction in which player P1 faces forward. In the example of FIG. 4, player P1 faces the "forward" direction of the multiple speakers 30.

As shown in FIG. 5, directions in the virtual space (front, back, left, right, up, down) are predetermined. For example, "front, back, left, right" in the virtual space correspond to "north, south, west, east" which are predetermined in the virtual space, and "up and down" in the virtual space correspond to "heaven and earth" which are predetermined in the virtual space. Directions in the virtual space (front, back, left, right, up, down) are expressed in a three-dimensional coordinate system (for example, the world coordinate system).

The orientation of the virtual camera 61 is the direction in which the front of the virtual camera 61 faces. In this example, the orientation of the virtual camera 61 is the orientation corresponding to the player's orientation (specifically, vector Vd) derived based on the signal output from the motion sensor 42 of the head-mounted display 20.

The orientation of the virtual microphone 62 is the direction in which the front of the virtual microphone 62 faces. The directions (front, back, left, right) of the virtual microphone 62 correspond to the directions (front, back, left, right) of the multiple speakers 30. For example, sound output from the "front" of the virtual microphone 62 toward the virtual microphone 62 (sound in the virtual space) is output from the front center speaker 50 (speaker 30 located "in front"), which corresponds to the "front" of the multiple speakers 30.

In the embodiment, the orientation of the virtual microphone 62 is maintained (fixed) in the virtual reference direction. The virtual reference direction (the direction that serves as the reference for the orientation of the virtual microphone 62 in the virtual space) is set to the orientation of the virtual microphone 62 when the relationship between the direction in the virtual microphone 62 and the direction in the virtual space is a predetermined relationship. Specifically, the virtual reference direction is set to the orientation of the virtual microphone 62 (the direction in which the front of the virtual microphone 62 faces) when the direction in the virtual microphone 62 (front, back, left, right) matches the direction in the virtual space (front, back, left, right).

In the example of FIG. 5, the virtual reference direction is the direction facing "forward (north)" in the virtual space. When the front of the virtual microphone 62 faces "forward (north)" in the virtual space, the directions in the virtual microphone 62 (front/back, left/right) and the directions in the virtual space (front/back, left/right) match.

[Details of processing by the control unit]
Next, the process performed by the control unit 16 will be described in detail with reference to Fig. 6. In the following, a case where the player P1 faces "forward" of the speaker 30 will be described as an example.

When the player P1 faces "forward" on the multiple speakers 30, the player character C1 faces "forward (north)" in the virtual space. Similarly, the virtual camera 61 faces "forward (north)" in the virtual space. As a result, an image including an enemy character C2 located "forward (north)" of the player character C1 in the virtual space is displayed on the display 21 of the head-mounted display 20 worn by the player P1. The enemy character C2 is an example of a sound source object.

The virtual microphone 62 is also maintained facing "forward (north)" in the virtual space. As a result, the character voice A2 output from the enemy character C2, which is located "forward (north)" of the player character C1 in the virtual space, is output from the front center speaker 50, which is positioned "forward" of the player P1.

By the above processing, an image in the virtual space corresponding to the orientation of player P1 is displayed on the display 21 of the head-mounted display 20. Furthermore, the direction of the sound heading toward the virtual microphone 62 in the virtual space (the orientation of the sound field in the virtual space) coincides with the direction of the sound heading toward player P1 from the multiple speakers 30 (the orientation of the sound field in the real world). In this way, the sound in the virtual space corresponding to the image in the virtual space displayed on the display 21 is output from the multiple speakers 30 with appropriate positioning.

[Comparison between the embodiment and the comparative example]
Next, the embodiment and the comparative example will be compared. In the following, for convenience of explanation, the reference numerals of the components in the comparative example are used in the explanation of the components in the comparative example. The comparative example in the following explanation differs from the embodiment in that the orientation of the virtual microphone 62 is changed so as to follow the orientation of the virtual camera 61. Other processes in the comparative example are similar to those in the embodiment. Note that in conventional game systems (e.g., VR systems), it is common to change the orientation of the virtual microphone 62 so as to follow the orientation of the virtual camera 61.

As shown in FIG. 7, in the comparative example, when the player P1 faces "to the right" of the multiple speakers 30, the player character C1 faces "to the right (east)" in the virtual space. Similarly, the virtual camera 61 faces "to the right (east)" in the virtual space. As a result, an image of the player character C1 "to the right (east)" in the virtual space is displayed on the display 21 of the head-mounted display 20 worn by the player P1. The enemy character C2 is located "to the left (north)" of the player character C1.

The virtual microphone 62, like the virtual camera 61, faces "to the right (east)" in the virtual space. Therefore, the character voice A2 output from the enemy character C2 located "to the left (north)" of the player character C1 in the virtual space is output from the surround left speaker 54 located "behind" the player P1.

As such, in the comparative example, the direction of the sound in the virtual space toward the virtual microphone 62 (the direction of the sound field in the virtual space) does not match the direction of the sound from the multiple speakers 30 toward the player P1 (the direction of the sound field in the real world). Therefore, the sound in the virtual space corresponding to the image in the virtual space displayed on the display 21 cannot be output from the multiple speakers 30 in an appropriate position. In the example of Figure 7, character sound A2 should originally be output toward player P1 from the "left" of player P1, but is instead output toward player P1 from "behind" player P1.

On the other hand, as shown in FIG. 8, in this embodiment, the virtual microphone 62 is maintained facing "forward (north)" in the virtual space. Therefore, the character voice A2 output from the enemy character C2 located "to the left (north)" of the player character C1 in the virtual space is output from the front center speaker 50 located "to the left" of the player P1.

In this way, by maintaining the orientation of the virtual microphone 62 in the virtual reference direction ("forward (north)" in the example of Figure 8), it is possible to maintain a state in which the direction of sound heading toward the virtual microphone 62 in the virtual space (the direction of the sound field in the virtual space) matches the direction of sound heading toward player P1 from the multiple speakers 30 (the direction of the sound field in the real world). In this way, sound in the virtual space corresponding to the image in the virtual space displayed on the display 21 is output from the multiple speakers 30 with appropriate positioning.

[Overview of the embodiment]
To sum up, the program (game program) of the embodiment is a program that causes a computer system having one or a plurality of computers working together to function as a detection unit 101, a display processing unit 102, and an audio processing unit 103. The computer system is used together with a display 21 that follows the orientation of a user (player), a plurality of speakers 30 arranged around the user, and a sensor (motion sensor 22) that outputs a signal indicating the user's movement including a change in the user's orientation. The detection unit 101 detects the orientation of the user based on the signal output from the sensor. The display processing unit 102 changes the orientation of a virtual camera 61 arranged in a virtual space so as to follow the orientation of the user detected by the detection unit 101, and causes the display 21 to display an image of the virtual space obtained by the virtual camera 61. The audio processing unit 103 causes the plurality of speakers 30 to output sounds in the virtual space so that the orientation of the sound field formed by the sounds in the virtual space output from the plurality of speakers 30 does not change even if the orientation of the virtual camera 61 changes.

The acoustic control device (game device 10) of the embodiment includes a storage unit 15 that stores the above program, and a control unit 16 that executes the above program.

The system (game system 1) of the embodiment includes a display 21 that tracks the orientation of a user (player), multiple speakers 30 arranged around the user, a sensor (motion sensor 22) that outputs a signal indicating the user's movement including changes in the user's orientation, and a control unit 16. The control unit 16 performs detection processing, display processing, and acoustic processing. In the detection processing, the control unit (detection unit 101) detects the orientation of the user based on the signal output from the sensor. In the display processing, the control unit (display processing unit 102) changes the orientation of a virtual camera 61 arranged in the virtual space so as to track the orientation of the user detected by the detection processing, and causes the display 21 to display an image of the virtual space obtained by the virtual camera 61. In the acoustic processing, the control unit (acoustic processing unit 103) causes the multiple speakers 30 to output sounds in the virtual space so that the orientation of the sound field formed by the sounds in the virtual space output from the multiple speakers 30 does not change even if the orientation of the virtual camera 61 changes.

[Effects of the embodiment]
As described above, in the embodiment, the sound in the virtual space is output to the multiple speakers 30 so that the direction of the sound field formed by the sound in the virtual space output from the multiple speakers 30 (hereinafter referred to as the "direction of the sound field in the real world") does not change even if the orientation of the virtual camera 61 changes.

The above control can prevent the orientation of the sound field in the real world from changing due to a change in the orientation of player P1, so that the "orientation of the sound field in the real world" does not deviate from the "orientation of the sound field in the virtual space (the orientation of the sound field formed by the sound output from the sound source object in the virtual space)". This allows the sound in the virtual space corresponding to the image in the virtual space displayed on the display 21 to be output from the multiple speakers 30 with appropriate positioning, improving the user experience. For example, the realism of how the sound is heard in the virtual space can be improved, thereby improving interest.

(Modification of the embodiment)
The game system 1 according to the modification of the embodiment differs from the game system 1 according to the embodiment in the processing by the display processing unit 102 and the sound processing unit 103. The other configurations and processing of the game system 1 according to the modification of the embodiment are similar to the configurations and processing of the game system 1 according to the embodiment.

In a modified embodiment, a direction change operation and a reset operation are input to the controller 40. The direction change operation is an operation for changing the orientation of the virtual camera 61 from the forward direction corresponding to the orientation of the player detected by the detection unit 101 to another direction. In the direction change operation, a direction other than the forward direction (for example, a direction that provides a third-person viewpoint) is specified. The reset operation is an operation for resetting the orientation of the virtual camera 61 to the forward direction.

The forward direction is the direction that coincides with the orientation of the player character C1 corresponding to the orientation of the player detected by the detection unit 101. For example, when the player P1 faces "forward" in the multiple speakers 30, the player character C1 faces "forward (north)" in the virtual space, and the forward direction is "forward (north)" in the virtual space.

When a direction change operation is input to the controller 40, the display processing unit 102 changes the orientation of the virtual camera 61 from the front direction in accordance with the direction change operation. When the orientation of the virtual camera 61 changes from the front direction in accordance with the direction change operation, the sound processing unit 103 changes the orientation of the virtual microphone 62 so as to follow the orientation of the virtual camera 61.

When a reset operation is input to the controller 40, the display processing unit 102 resets the orientation of the virtual camera 61 to the front direction in response to the reset operation. When the orientation of the virtual camera 61 is reset to the front direction in response to the reset operation, the sound processing unit 103 resets the orientation of the virtual microphone 62 to a virtual reference direction.

In this example, the display processing unit 102 selectively performs a first display process and a second display process. The first display process is a process of changing the orientation of the virtual camera 61 so as to follow the orientation of the player detected by the detection unit 101, and displaying an image of the virtual space obtained by the virtual camera 61 on the display 21. The second display process is a process of setting the orientation of the virtual camera 61 to a direction different from the forward direction (a direction specified by a direction change operation), and displaying an image of the virtual space obtained by the virtual camera 61 on the display 21.

When a direction change operation is input to the controller 40, the display processing unit 102 ends the first display process and starts the second display process. Also, when a reset operation is input to the controller 40, the display processing unit 102 ends the second display process and starts the first display process.

In this example, the acoustic processing unit 103 selectively performs a first sound image localization process and a second sound image localization process. The first sound image localization process is a process in which the virtual microphone 62 is moved to follow the movement of the virtual camera 61 while maintaining the orientation of the virtual microphone 62 in a virtual reference direction, and sound in the virtual space obtained by the virtual microphone 62 is output to the multiple speakers 30. The second sound image localization process is a process in which the orientation of the virtual microphone 62 is changed to follow the orientation of the virtual camera 61, and the virtual microphone 62 is moved to follow the movement of the virtual camera 61, and sound in the virtual space obtained by the virtual microphone 62 is output to the multiple speakers 30.

When the orientation of the virtual camera 61 changes from the front in response to a direction change operation, the sound processing unit 103 ends the first sound image localization process and starts the second sound image localization process. Also, when the orientation of the virtual camera 61 is reset to the front in response to a reset operation, the sound processing unit 103 ends the second sound image localization process and starts the first sound image localization process.

[Details of processing by the control unit]
Next, the process by the control unit 16 in the modified embodiment will be described in detail with reference to Fig. 9. In the following, a case where the player P1 faces "forward" of the speaker 30 will be described as an example. Note that in this example, when the orientation of the virtual camera 61 changes from the front direction to another direction in response to a direction change operation, the image displayed on the display 21 changes from a "first-person perspective image" to a "third-person perspective image."

When player P1 faces "forward" on the multiple speakers 30, if a direction change operation is input to the controller 40 to change the orientation of the virtual camera 61 from "forward (north)" (front direction) in the virtual space to "left (west)" in the virtual space, the virtual camera 61 faces "left (west)" in the virtual space. As a result, an image of the player character C1 as seen from the "right (east)" in the virtual space is displayed on the display 21 of the head-mounted display 20 worn by player P1.

The left-right direction in the image displayed on the display 21 corresponds to the front-back direction (north-south direction) in the virtual space. In the image displayed on the display 21, the enemy character C2 is located to the "right" of the player character C1. In the example of FIG. 9, the image displayed on the display 21 is a third-person perspective image including the player character C1 and the enemy character C2. The display processing unit 102 changes the orientation of the virtual camera 61 and moves the virtual camera 61 so that a third-person perspective image is obtained by the virtual camera 61.

The orientation of the virtual microphone 62 also changes from "forward (north)" (front direction) in the virtual space to "leftward (westward)" in the virtual space so as to follow the orientation of the virtual camera 61. As a result, the character voice A2 output from the enemy character C2 located to the right of the player character C1 in the image displayed on the display 21 is output from the surround right speaker 53 located to the "right" of the player P1.

By the above processing, an image (third-person perspective image) of the virtual space obtained by the virtual camera 61 facing in a direction different from the forward direction corresponding to the orientation of the player P1 is displayed on the display 21 of the head-mounted display 20. Furthermore, the direction of the sound directed toward the player character C1 in the image displayed on the display 21 (the direction of the sound field in the virtual space) matches the direction of the sound directed toward the player P1 from the multiple speakers 30 (the direction of the sound field in the real world). In this way, the sound in the virtual space corresponding to the image in the virtual space displayed on the display 21 is output from the multiple speakers 30 with appropriate positioning.

[Effects of the Modification of the Embodiment]
In the modified example of the embodiment, the same effects as those of the embodiment can be obtained.

In addition, in a modified embodiment, even when a direction change operation is input to the controller 40 to change the orientation of the virtual camera 61 from the forward direction corresponding to the orientation of the player detected by the detection unit 101 to another direction, sounds in the virtual space corresponding to the images in the virtual space displayed on the display 21 can be output from the multiple speakers 30 in an appropriate position, improving the user experience.

Other Embodiments
In the above description, the following configuration or setting may be adopted.

The audio data does not necessarily need to be prepared in advance. For example, the acoustic processing unit 103 may synthesize or generate audio data each time based on some data (e.g., audio data stored in the memory unit 15) or information.

The player character may or may not be displayed on the display. In other words, the player character may or may not be included in the image of the virtual space that is displayed (played) on the display. For example, when an image of the virtual space is displayed on the display from a first-person perspective, only the virtual hand (virtual body part), which is part of the player character, may be displayed, or the virtual hand may not be displayed and only a handgun (virtual object) held in the virtual hand may be displayed.

The image of the virtual space may be an image from a first-person perspective, or an image from a third-person perspective (for example, an image of the player character viewed from behind the player character).

Multiple speakers 30 may be positioned to the left and right, in front and behind, above, etc., of the listener (player) to enable three-dimensional sound reproduction.

The sensor that outputs a signal indicating the player's movements, including changes in the player's orientation, is not limited to the motion sensor 22 included in the head-mounted display 20. For example, such a sensor may be a combination of a camera (image sensor) that captures an image of a specific part of the player's body (e.g., the head), and a recognition processing unit that recognizes the player's movements, including changes in the player's orientation, based on the "image of the specific part" obtained by the camera.

The game program may be implemented as a game program for a so-called online game. In this case, the processing performed in the control unit 16 of the game device 10 may be performed in a control unit of a server device (not shown), or may be shared between the game device 10 and the control unit of the server device. The game device 10 may be a mobile information terminal such as a smartphone or tablet.

In the above explanation, a virtual space in which a game progresses (game space) has been given as an example of a virtual space, but this is not limiting. For example, the virtual space may be a virtual space in which avatars communicate with each other (metaverse space). In other words, the services provided in the virtual space are not limited to games. For example, the services provided in the virtual space may be the provision of a place for commercial transactions, the provision of a place for communication, etc.

Even when these other embodiments are adopted, the effects of the present invention are achieved. In addition, this embodiment can be combined with other embodiments, and other embodiments can be combined with each other as appropriate. The above embodiments are essentially preferred examples, and are not intended to limit the scope of the present invention, its applications, or its uses.

1. Game System (System)
10 Game device (sound control device)
11 Network interface 12 Graphics processing unit 13 Audio processing unit 14 Operation processing unit 20 Head mounted display 21 Display 22 Motion sensor 25 Main body unit 26 Headband 30 Speaker 40 Controller 40a First controller 40b Second controller 41 Operator 42 Motion sensor (sensor)
50 Front center speaker 51 Front right speaker 52 Front left speaker 53 Surround right speaker 54 Surround left speaker 55 Surround back right speaker 56 Surround back left speaker 61 Virtual camera 62 Virtual microphone 100 Game progression unit (virtual space control unit)
101 Detection unit 102 Display processing unit 103 Sound processing unit P1 Player (user)
C1 Player character (user character)
C2 Enemy character (sound source object)
A2 Character voice (voice in virtual space)

Claims (6)

A program for causing a computer system having one or more computers working together to function as a detection unit, a display processing unit, and an audio processing unit,
The computer system is used with a display that tracks the orientation of a user, a plurality of speakers arranged around the user, and a sensor that outputs a signal indicative of the user's movements, including changes in the user's orientation;
The detection unit detects a direction of the user based on a signal output from the sensor,
the display processing unit changes an orientation of a virtual camera disposed in a virtual space so as to follow the orientation of the user detected by the detection unit, and displays an image of the virtual space obtained by the virtual camera on the display;
The acoustic processing unit is a program that causes the multiple speakers to output audio within the virtual space so that the direction of the sound field formed by the audio within the virtual space output from the multiple speakers does not change even if the direction of the virtual camera changes. In the program of claim 1,
the acoustic processing unit moves a virtual microphone disposed in the virtual space so as to follow the movement of the virtual camera, and outputs sound in the virtual space obtained by the virtual microphone to the plurality of speakers;
A program in which the orientation of the virtual microphone is a reference direction for the orientation of the virtual microphone in the virtual space, and is maintained in a virtual reference direction that does not change in response to changes in the orientation of the user detected by the detection unit. In the program of claim 2,
the computer system is used in conjunction with a controller operated by the user;
when a direction change operation for changing a direction of the virtual camera from a front direction corresponding to the direction of the user detected by the detection unit is input to the controller, the display processing unit changes the direction of the virtual camera from the front direction in response to the direction change operation;
The sound processing unit is a program that changes the orientation of the virtual microphone to follow the orientation of the virtual camera when the orientation of the virtual camera changes from the front direction in response to the direction change operation. In the program of claim 3,
when a reset operation for resetting the orientation of the virtual camera to the front direction is input to the controller, the display processing unit resets the orientation of the virtual camera to the front direction in response to the reset operation;
The sound processing unit is a program that resets the orientation of the virtual microphone to the virtual reference direction when the orientation of the virtual camera is reset to the front direction in response to the reset operation. A storage unit that stores the program according to any one of claims 1 to 4;
and a control unit for executing the program. A display that follows the user's orientation;
A plurality of speakers arranged around the user;
a sensor that outputs a signal indicative of the user's movement, including a change in the user's orientation;
A control unit.
The control unit is
A detection process for detecting a direction of the user based on a signal output from the sensor;
a display process of changing an orientation of a virtual camera arranged in a virtual space so as to follow the orientation of the user detected by the detection process, and displaying an image of the virtual space obtained by the virtual camera on the display;
and acoustic processing that causes the multiple speakers to output audio within the virtual space so that the direction of the sound field formed by the audio within the virtual space output from the multiple speakers does not change even when the direction of the virtual camera changes.