JP2018026701A - Sound recording device, image/sound processing program, and game device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To reproduce a sound in a VR space in which a HMD is used so as to provide a high reality.SOLUTION: In a game in which a user plays with a HMD 300, a camera control part 52 controls a virtual camera in accordance with a motion of a head of the user detected by the HMD 300, and previously prepares sound data in each direction of a virtual space, and a sound processing part 54 outputs sound data in a direction corresponding to an orientation of the virtual camera. Thus, since the sound corresponded to the virtual space where the user is viewing is outputted, the sound can be reproduced with higher reality. In a case where the sound processing part 54 produces a sound effect, the sound effect is outputted by setting a plurality of virtual microphones in accordance with each direction, determining each virtual microphone capable of collecting the sound effect in accordance with the direction of the virtual camera under the condition that the sound effect is assumed to be produced in the direction of the virtual microphone capable of collecting the sound effect.SELECTED DRAWING: Figure 6

Description

  The present invention relates to a technique for reproducing sound in a virtual reality space.

  In recent years, virtual reality (VR) using a head mounted display (HMD) is becoming widespread. In the VR game corresponding to the virtual reality using the HMD, the user can feel as if it actually exists in the virtual three-dimensional space.

Japanese Patent No. 5565258 Japanese Patent No. 5869177

  In the VR space, the user can look around the virtual three-dimensional space by moving his head.

  In a game that is not a conventional VR game, the game is played with the eyes on the monitor, so there is no need to reproduce the sound in consideration of the direction of the eyes. However, when sound is output without considering the direction that the user is viewing in the VR game, there is a problem that the same sound can be heard in any direction. In the VR game, since the game screen is displayed at 360 degrees, it is necessary to reproduce the sound in consideration of the direction of the line of sight.

  The present invention has been made in view of the above, and an object of the present invention is to reproduce sound so that a sense of reality can be further obtained in a VR space using an HMD.

  A recording apparatus according to a first aspect of the present invention includes a plurality of microphones arranged at equal intervals on a concentric circle, and a sound signal collected in each of the plurality of microphones in a direction corresponding to each of the plurality of microphones. And a recording means for recording as follows.

  A video / audio processing program according to a second aspect of the present invention is a video / audio processing program for operating a computer as an apparatus for viewing a video by a user wearing a head-mounted display, wherein the computer is connected to the user from the head-mounted display. Input means for inputting head movement information and detecting the direction in which the user is facing, video processing means for displaying an image corresponding to the direction in which the user is facing on the head-mounted display, facing the user It is characterized by functioning as sound processing means for outputting sound corresponding to the direction.

  A game apparatus according to a third aspect of the present invention includes a program storage unit that stores the video / audio processing program and a computer that executes the video / audio processing program stored in the program storage unit.

  According to the present invention, the direction in which the user is facing is detected based on the information from the head mounted display, the video according to the direction in which the user is facing is displayed, and the sound according to the direction in which the user is facing. By outputting the sound, it is possible to reproduce the sound so as to give a more realistic feeling in the VR space using the HMD.

It is a functional block diagram which shows the structure of the recording system which concerns on this embodiment. It is a figure which shows the example of the recorded audio | voice signal. It is the schematic which shows the structure of the game system which concerns on this embodiment. It is a block diagram which shows the hardware constitutions of a game machine. It is a block diagram which shows the hardware constitutions of a head mounted display. It is a functional block diagram which shows the functional structure of the game device of this embodiment. It is a figure explaining the sound data read when the direction of a virtual camera is a specific direction. It is a figure explaining the sound data read when the direction of a virtual camera is an area | region boundary. It is a figure explaining adjusting the sound volume output according to the direction of a virtual camera. It is a figure explaining the example which outputs sound data by stereo. It is a figure explaining the direction which arranges a virtual microphone in virtual space and can hear a sound effect. It is a figure explaining calculation of the volume of the sound effect sounded within the virtual space. It is a figure which shows a mode that the virtual microphone was arrange | positioned to the soccer field in virtual space. It is a figure which shows the example which displays a part of soccer field. It is a figure which shows the example which displays another part of a soccer field. It is a whole lineblock diagram containing the virtual reality system concerning this embodiment. It is a figure which shows an example of a microphone array. It is a functional block diagram which shows the structure of the audiovisual processing apparatus which concerns on this embodiment. It is a flowchart which shows the flow of a process of the audiovisual processing apparatus which concerns on this embodiment.

<First embodiment>
[Recording system]
FIG. 1 is a functional block diagram showing the configuration of the recording system according to the present embodiment.

  The recording system shown in FIG. 1 includes a microphone array 10 and a recording device 20.

  The microphone array 10 includes six microphones C, L, R, Ls, Rs, and B arranged at equal intervals on a concentric circle. The angle formed by the line connecting the center of the concentric circle and each of the microphones C, L, R, Ls, Rs, B is 60 degrees. The microphones C, L, R, Ls, Rs, and B may be unidirectional microphones. In FIG. 1, the number of microphones is six, but this is not a limitation. Since this recording system only needs to be able to process audio signals for six channels, a 5.1ch recording system can be used.

  When a sound to be recorded around the microphone array 10 is played or output, the recording device 20 records the sound collected by each microphone C, L, R, Ls, Rs, B.

  The recording device 20 includes an amplification unit 21, an analog / digital (A / D) conversion unit 22, a recording unit 23, and a storage unit 24. Audio signals from the microphones C, L, R, Ls, Rs, and B are amplified by the amplifying unit 21 and converted into digital data by the A / D converting unit 22. The recording unit 23 stores each digital data in the storage unit 24.

  FIG. 2 shows an example of audio signals collected by the microphones C, L, R, Ls, Rs, and B recorded by the recording device 20. The sounds collected by the microphones C, L, R, Ls, Rs, and B are individually converted into digital data and stored in the storage unit 24. In the present embodiment, sound data for six channels is stored in the storage unit 24.

  As described above, according to the present embodiment, the microphones C, L, R, Ls, Rs, B are arranged on the concentric circles at equal intervals, and the microphones C, L, R, Ls, Rs, B are arranged. By recording for each voice collected at, voice in each direction around the microphone array 10 can be recorded by direction.

[Game system]
Next, an example of a VR game played by a user wearing an HMD will be described as an apparatus for reproducing the sound recorded by the recording system.

  One example of a game that a user plays with wearing an HMD is a first-person view game. In a first-person viewpoint game, a virtual three-dimensional space is displayed from the viewpoint of the player character operated by the user. When the user moves his / her head, the HMD detects the movement of the user's head. The game program receives information on the movement of the user's head from the HMD, controls a virtual camera that captures a virtual three-dimensional space in accordance with the movement of the user's head, and displays an image corresponding to the orientation of the user's head. . In the present embodiment, the game program determines the direction in the virtual three-dimensional space that the user is viewing and outputs sound data in that direction.

  FIG. 3 is a schematic diagram showing the configuration of the game system according to the present embodiment.

  The game system shown in FIG. 3 includes a game machine 100, a controller 200, and an HMD 300. The game machine 100 is a computer including an arithmetic processing device and a storage device that can execute a game program. The game machine 100 may be a home game machine, a personal computer, a mobile terminal such as a smartphone, or an arcade game machine. The controller 200 transmits the operation input by the user to the game machine 100. The controller 200 is connected to the game machine 100 by wire or wireless. The HMD 300 is mounted on the user's head, detects the rotation angle and movement of the user's head, and transmits the detected result to the game machine 100. The HMD 300 displays an image received from the game machine 100 and outputs a sound received from the game machine 100.

  FIG. 4 is a block diagram illustrating a hardware configuration of the game machine 100. The game machine 100 includes a central processing unit (CPU) 101, a read only memory (ROM) 102, a random access memory (RAM) 103, a graphics processing unit (GPU) 104, a sound processing unit (SPU) 105, an interface 106, and DRIVE 107 is provided. The CPU 101 executes a game program. The ROM 102 stores a system program. The RAM 103 stores a game program and various data. The GPU 104 generates a virtual space image. The SPU 105 processes audio. The interface 106 connects the controller 200 and the HMD 300, and inputs operation information and HMD information. The interface 106 outputs images and sounds processed by the GPU 104 and the SPU 105. The DRIVE 107 reads the game program from the recording medium on which the game program is recorded and stores it in the RAM 103. The recording medium may be an optical disk such as a CD, DVD, Blu-Ray (registered trademark) Disk, a magnetic disk, or a semiconductor memory. The game machine 100 may have a communication function and acquire a game program via a network.

  FIG. 5 is a block diagram showing a hardware configuration of the HMD 300. As shown in FIG. The HMD 300 includes display units 301A and 301B, a sound output unit 302, a gyro sensor 303, an acceleration sensor 304, and a control unit 305. The display units 301A and 301B display a right-eye image and a left-eye image, respectively. When the right-eye image and the left-eye image have parallax, the image viewed by the user is a three-dimensional stereoscopic image. The sound output unit 302 outputs the sound received from the game machine 100. The sound output unit 302 may be a headphone included in the HMD 300 or an audio output terminal for connecting a headphone or the like. The gyro sensor 303 detects the rotation angle of the user's head. With the head of the user wearing the HMD 300 as the origin, the roll axis is in the front direction of the user, the pitch axis is in the left direction of the user, and the yaw axis is in the upward direction of the user. The gyro sensor 303 detects the rotation angle (roll angle, pitch angle, yaw angle) of the user's head for each axis. The acceleration sensor 304 detects the movement of the user's head. The control unit 305 receives images from the game machine 100 and displays them on the display units 301 </ b> A and 301 </ b> B, receives sound from the game machine 100, and outputs the sound to the sound output unit 302. In addition, the control unit 305 transmits data detected by the gyro sensor 303 and the acceleration sensor 304 to the game machine 100.

  Note that the HMD 300 may not be provided with the sound output unit 302, and headphones or the like may be connected to the game machine 100.

[Game device]
FIG. 6 is a functional block diagram showing a functional configuration of the game apparatus 50 according to the present embodiment. The game device 50 shown in FIG. 6 includes a character control unit 51, a camera control unit 52, a rendering unit 53, a sound processing unit 54, and a storage unit 55. When the game machine 100 executes the game program according to the present embodiment, the game machine 100 operates as the game apparatus 50 including the functional unit illustrated in FIG. Although the game machine 100 also exhibits functions that are not shown, only functional units related to the present embodiment are illustrated here.

  The character control unit 51 controls the position and orientation of the player character in accordance with a user operation.

  The camera control unit 52 sets the orientation of the virtual camera based on the orientation of the player character and the orientation of the user's head included in the HMD information received from the HMD 300. For example, when the player character faces the north direction in the virtual three-dimensional space and the user's head turns to the right, the camera control unit 52 turns the virtual camera from the north to the east.

  The rendering unit 53 generates a two-dimensional image obtained by photographing a virtual three-dimensional space with a virtual camera. When the rendering unit 53 generates a right-eye image and a left-eye image having parallax and the HMD 300 displays each image on each of the display units 301A and 301B, the user views a three-dimensional stereoscopic image on the HMD 300. Can do.

  The sound processing unit 54 reads out sound data in a direction corresponding to the direction of the virtual camera from the storage unit 55 and outputs the sound data. Further, the sound processing unit 54 determines the direction in which the sound effect generated in the virtual three-dimensional space can be heard, and determines the sound volume for outputting the sound effect based on the direction of the virtual camera and the direction in which the sound effect can be heard.

  The accumulation unit 55 accumulates sound data and sound effect data corresponding to each direction in the virtual three-dimensional space. In the present embodiment, the storage unit 55 includes a center (C) channel, a left (L) channel, a right (R) channel, and a left corresponding to the directions of the microphones C, L, R, Ls, Rs, and B in FIG. Sound data of the surround (Ls) channel, the light surround (Rs) channel, and the back (B) channel is stored.

[Sound processing]
Next, a process in which the sound processing unit outputs sound data in a direction corresponding to the direction of the virtual camera will be described.

  The sound processing unit 54 divides the entire circumference around the yaw axis (upward direction of the player character) of the virtual three-dimensional space into six regions, and associates the C channel with the north region of the virtual three-dimensional space, Around, the R channel, Rs channel, B channel, Ls channel, and L channel are associated with each region. The sound processing unit 54 outputs sound data in an area corresponding to the direction of the virtual camera.

  For example, as shown in FIG. 7A, when the virtual camera is facing north, the sound processing unit 54 reads out and outputs sound data of the C channel.

  The sound processing unit 54 may synthesize and output sound data of a plurality of channels according to the orientation of the virtual camera. For example, when the virtual camera is facing the boundary of the area, the sound data of the channels in both areas sandwiching the boundary is output in a size of 50%. In the example of FIG. 7B, the virtual camera faces the boundary between the C channel region and the R channel region. In this case, the sound processing unit 54 outputs the C channel sound data and the R channel sound data at a size of 50%. In the example of FIG. 7C, since the virtual camera faces the direction closer to the Ls channel between the direction of the B channel and the direction of the Ls channel, the sound processing unit 54 converts the sound data of the Ls channel to 75%, B channel sound data is output at 25% size. More specifically, the orientation of the virtual camera between the B channel and the Ls channel is assumed to be 0 when the virtual camera is facing the B channel and 100 when the virtual camera is facing the Ls channel. Is represented by a numerical value from 0 to 100. The sound processing unit 54 determines the volume of each channel based on a numerical value indicating the direction of the virtual camera. In the example of FIG. 7C, the numerical value indicating the direction of the virtual camera is 75. Therefore, the sound processing unit 54 outputs the sound data of the Ls channel in a size of 75% and the sound data of the B channel in a size of 25%.

  Next, processing for reproducing in stereo will be described.

  The sound processing unit 54 may determine sound data to be output from each of the stereo left channel and the right channel on the basis of the orientation of the virtual camera.

  In the example of FIG. 8, the sound processing unit 54 determines sound data to be output from the stereo left channel based on the direction rotated 30 degrees counterclockwise with reference to the orientation of the virtual camera, and 30 degrees clockwise. The sound data to be output from the right channel of stereo is determined based on the rotated direction. Since the standard of the stereo left channel indicates the boundary between the C channel region and the L channel region, the C channel and L channel sound data are output from the stereo left channel at a size of 50%. Since the stereo right channel reference indicates the boundary between the C channel region and the R channel, the stereo channel right channel outputs the sound data of the C channel and the R channel at 50% each.

  Note that the angles and ratios shown above are examples, and other values may be used.

[Sound effect processing]
Next, processing in which the sound processing unit outputs sound effects will be described.

  So far, the process of outputting sound data corresponding to each direction in the virtual three-dimensional space according to the orientation of the virtual camera has been described. If the position and direction in which sound is produced is known in advance, sound data in that direction can be prepared in advance. It is unclear which direction sound data should be prepared for the sound of an object whose position in the virtual three-dimensional space is not determined in advance.

  Therefore, in the present embodiment, the sound processing unit 54 sets a plurality of virtual microphones corresponding to each direction in the virtual three-dimensional space, and when an object in the virtual three-dimensional space sounds, A virtual microphone that can collect sound and the volume of sound are determined. Then, the sound processing unit 54 outputs sound according to the direction of the virtual camera, assuming that sound is produced in the direction of the virtual microphone that has collected sound. The distance between the virtual microphone and the virtual camera is 50 cm, for example, but can be arbitrarily set according to the size of the virtual three-dimensional space.

  First, the process for determining the direction in which the sound effect can be heard will be described.

  FIG. 9 is a diagram for explaining the direction in which a sound effect can be heard. In the example of FIG. 9, it is assumed that a sound effect sounds at the point P. A circle 60 in the figure indicates a range where a sound effect produced at the point P can be heard. The sound processing unit 54 sets six virtual microphones C, L, R, Ls, Rs, and B in the virtual three-dimensional space. In the example of FIG. 9, sound effects are collected by the two virtual microphones R and Rs. Therefore, the sound processing unit 54 treats the sound effect at the point P as a sound effect sounded in the directions of the R channel and the Rs channel. The magnitude of the sound effect is attenuated according to the distance from the point P to the virtual microphones R and Rs. For example, it is assumed that the sound effect collected by the virtual microphone R is attenuated by 80% and can be heard at a volume of 20%.

  Next, processing for outputting sound effects collected by the virtual microphone will be described.

  The sound effect collected by the virtual microphone determines whether or not the sound is output and the volume according to the direction of the virtual camera.

  FIG. 10 is a diagram for explaining the calculation of the sound effect volume. In the example of FIG. 10, the virtual camera faces the boundary between the C channel region and the R channel region. In this case, the sound processing unit 54 outputs the C channel and the R channel at a volume of 50%. Therefore, the sound processing unit 54 outputs the sound effect collected by the virtual microphone R at a volume of 50%. If the sound volume of the sound effect at point P is Vp and the sound effect at point P is heard at a volume of 20% in the virtual microphone R, the sound processing unit 54 sets the sound effect at point P to 50% × 20% × Vp. Output at volume.

<Second embodiment>
A second embodiment will be described. The second embodiment is an example in which a user is watching a soccer game from a spectator seat.

  In the soccer game, the game program displays a part of the soccer field according to the movement of the head of the user wearing the HMD 300 without displaying the entire soccer field. For example, the position of the virtual camera is set to the audience seat on the extension of the center line of the soccer field, and the display position is changed by controlling the direction of the virtual camera in accordance with the movement of the user's head. When the user faces the left direction, the vicinity of the goal in the left direction is displayed, and when the user faces the front, the vicinity of the center line is displayed.

  Also in the soccer game, the direction of the virtual camera, that is, the channel to be output is determined according to the video to be displayed.

  FIG. 11 is a diagram illustrating a state in which a virtual microphone is arranged on a soccer field in a game. In the example of the figure, virtual microphones are arranged around the soccer field, and the virtual microphones are divided into nine groups L1 to L3, C1 to C3, and R1 to R3. Sound generated in the soccer field is collected by a virtual microphone. For example, the sound effect at the point P is heard up to the range indicated by the circle 60 and collected by the virtual microphones of the groups L2, L3, and C1. The sound effect at the point P attenuates according to the distance from the point P to the virtual microphone.

  Whether or not the sound collected by the virtual microphone is output and the volume is determined for each of the groups L1 to L3, C1 to C3, and R1 to R3 according to the direction of the virtual camera.

  For example, the direction of the virtual camera is represented by a numerical value from 0 to 100. The numerical value when the virtual camera is facing the leftmost is 0, the numerical value when the virtual camera is facing the center is 50, and the numerical value is 100 when the virtual camera is facing the rightmost. The volume of each of the groups L1 to L3, C1 to C3, and R1 to R3 is determined based on a numerical value representing the direction of the virtual camera.

  The group L1 including the microphone arranged at the left end of the soccer field has a maximum volume when the numerical value is 0, and the volume decreases as the numerical value approaches 100. The groups L2 and L3 have the maximum volume when the numerical values are 10 and 20, respectively, and the volume decreases as the numerical values approach 0 or 100. For the other groups C1 to C3 and R1 to R3, the relationship between the numerical value representing the direction of the virtual camera and the volume is determined.

  When the user is facing left and the virtual camera is facing left, the vicinity of the goal in the left direction is displayed as shown in FIG. 12A. At this time, the numerical value indicating the orientation of the virtual camera is zero. The sound processing unit 54 obtains the volume of each of the groups L1 to L3, C1 to C3, and R1 to R3 when the numerical value is zero. If the obtained volumes are 100%, 90%, and 80% for the groups L1 to L3, respectively, and the other groups C1 to C3, R1 to R3 are 0%, the sound processing unit 54 The sounds collected by the virtual microphones of the groups L1, L2, and L3 are output at a volume of 100%, 90%, and 80%. As another example, as shown in FIG. 12B, when the center of the soccer field is displayed, the sound processing unit 54 collects 90% of the sounds collected by the virtual microphones of the groups C1, C2, and C3. Output at 100,90% volume. When the audience cheers, the sound may be played back at a high volume regardless of the orientation of the virtual camera.

  As described above, according to the present embodiment, in a game where the user wears the HMD 300 and plays, the camera control unit 52 controls the virtual camera according to the movement of the user's head detected by the HMD 300, and the virtual Sound data in each direction in the space is prepared, and the sound processing unit 54 outputs sound data in a direction corresponding to the direction of the virtual camera, so that a sound corresponding to the video viewed by the user is output. Therefore, it is possible to reproduce the sound so that it feels more realistic.

  According to the present embodiment, when the sound processing unit 54 plays a sound effect, it sets a plurality of virtual microphones corresponding to each direction, determines a virtual microphone capable of collecting the sound effects, and collects the sound effects. Even if a sound effect sounds in the direction of the virtual microphone that can make sound, by outputting the sound effect according to the direction of the virtual camera, a more realistic sound can be reproduced. The first embodiment and the second embodiment may be combined. For example, sounds generated on a soccer field are output according to the second embodiment, and sounds generated at a spectator seat are output according to the first embodiment.

<Third embodiment>
A third embodiment will be described. The third embodiment is an example of a virtual reality system in which a user wears an HMD and views a video shot by a camera.

  This virtual reality system displays not only a virtual three-dimensional space created by a computer but also video captured by a camera on the HMD and outputs audio collected in the shooting environment in accordance with the video displayed on the HMD.

  FIG. 13 is an overall configuration diagram including a virtual reality system according to the present embodiment.

  The virtual reality system according to the present embodiment includes a video / audio processing device 70 and an HMD 300. The video / audio processing device 70 cuts out a part of the panoramic video shot by the camera 30 and displays it on the HMD 300 according to the movement of the head of the user wearing the HMD 300, and a microphone according to the video displayed on the HMD 300. The sound collected by the array 10 is output.

  The microphone array 10 is composed of a plurality of microphones arranged in the shooting environment, and collects and transmits the sound of the shooting environment. FIG. 14 is a diagram illustrating an example of the microphone array 10. The microphone array 10 shown in the figure is composed of a plurality of microphones arranged around the soccer field. The plurality of microphones are divided into nine groups L1 to L3, C1 to C3, and R1 to R3. The microphone array 10 transmits the audio signal input from the microphones of each group to the video / audio processing device 70 via the network for each group.

  The camera 30 is arranged in a spectator seat on the extension of the center line of the soccer field, captures the entire soccer field, and transmits the captured video to the video / audio processing device 70 via the network. As the camera 30, a camera capable of panoramic shooting of the entire soccer field may be used, or each area of the soccer field may be shot with a plurality of cameras, and the shot videos may be combined to form a video of the entire soccer field.

  The video / audio processing device 70 receives a video of the entire soccer field from the camera 30, and cuts out a part of the received video and displays it on the HMD 300 according to the orientation of the head of the user wearing the HMD 300. Also, the audio / video processing device 70 receives audio signals of each group from the microphone array 10, selects audio signals according to the direction of the head of the user wearing the HMD 300, mixes the selected audio signals, and outputs them. .

  FIG. 15 is a functional block diagram showing the configuration of the video / audio processing apparatus 70 according to the present embodiment. The video / audio processing apparatus 70 shown in the figure includes a mixer 71, an audio processing unit 72, a video input unit 73, a video processing unit 74, and a control unit 75. The video / audio processing device 70 may be configured by a computer including an arithmetic processing device, a storage device, and the like, and the processing of each unit may be executed by a program. This program is stored in a storage device included in the video / audio processing apparatus 70, and can be recorded on a recording medium such as a magnetic disk, an optical disk, or a semiconductor memory, or provided through a network.

  The mixer 71 inputs audio signals in units of groups from the microphone array 10 and mixes the audio signals based on the audio selection information. The audio selection information is information received from the audio processing unit 72 and includes information for specifying a group. For example, when the audio selection information indicates the group L1, the mixer 71 turns on the input of the audio signal of the group L1 and turns off the input of the audio signal of the other group. The audio selection information may include a plurality of groups, or may include a proportion of the magnitude of the audio signal input for the group to be turned on. For example, when the audio selection information includes information indicating that the group L1 is 100% and the group L2 is 90%, the mixer 71 inputs the audio signal of the group L1 with a magnitude of 100% and the audio signal of the group L2 is 90%. The audio signals of the groups L1 and L2 are mixed.

  The audio processing unit 72 generates audio selection information for selecting an audio signal to be mixed based on the direction the user is facing received from the control unit 75 and transmits the audio selection information to the mixer 71. Similar to the process of determining the volume of each group in the above-described soccer game, the sound processing unit 72 determines the size of the sound signal input for each group and generates sound selection information. For example, when the user is facing the left direction, the sound processing unit 72 generates sound selection information for selecting the groups L1 to L3 on the left side of the microphone array 10. Also, the audio processing unit 72 receives the audio signal mixed by the mixer 71 and transmits it to the HMD 300.

  The video input unit 73 decodes the panoramic video received from the camera 30.

  The video processing unit 74 cuts out a video to be displayed from the decoded panoramic video based on the direction of the user that is received from the control unit 75 and transmits the video to the HMD 300. For example, when the user is facing left, the video processing unit 74 cuts out the video from the left side of the panoramic video and transmits it to the HMD 300.

  The control unit 75 determines the direction of the user's head based on the HMD information received from the HMD 300, and transmits the direction in which the user is facing to the audio processing unit 72 and the video processing unit 74.

  Next, the operation of the video / audio processing apparatus 70 according to the present embodiment will be described.

  FIG. 16 is a flowchart showing a process flow of the video / audio processing apparatus 70 according to the present embodiment.

The control unit 75 inputs HMD information including the orientation of the user's head from the HMD 300, and detects the direction in which the user is facing (step S11).
The video processing unit 74 receives the direction in which the user is facing from the control unit 75, and determines an area to be cut out from the panoramic video received by the video input unit 73 based on the direction in which the user is facing (step S12). For example, if the user is facing left, the left area of the entire soccer field video is cut out and output to the HMD 300. When the user is facing the front, the central area of the video of the entire soccer field is cut out and output to the HMD 300.

  The sound processing unit 72 receives the direction in which the user is facing from the control unit 75, and determines the sound signal to be mixed by the mixer 71 based on the direction in which the user is facing (step S13). For example, when the user is facing the left direction, the audio signal to be mixed is determined so that the audio on the left side of the soccer field is output. When the user is facing the front, the audio signal to be mixed is determined so that the audio at the center of the video of the entire soccer field is output.

  When the audio processing unit 72 determines the audio signal to be mixed based on the direction in which the user is facing, the audio suitable for the video displayed on the HMD 300 can be output.

  Note that 360-degree images may be taken by a 360-degree omnidirectional camera and collected by a microphone array in which microphones are arranged at equal intervals on concentric circles, like the microphone array of the recording system of FIG. Also in this case, the video / audio processing device 70 specifies the direction in which the user is facing based on the information from the HMD 300, displays the video in the direction in which the user is facing on the HMD 300, and also displays the direction in which the user is facing. Outputs the audio signal input by the microphone.

  As described above, according to the present embodiment, in a virtual reality system in which a user wears a head-mounted display and views an image captured by a camera, the control unit 75 is based on information from the HMD 300. The video processing unit 74 detects the direction in which the user is facing, displays the video in the direction in which the user is facing, and the audio processing unit 72 receives a plurality of audio signals received from the microphone array 10 based on the direction in which the user is facing. By selecting an audio signal to be mixed from among them, it is possible to output audio that matches the video displayed on the HMD 300.

DESCRIPTION OF SYMBOLS 10 ... Microphone array 20 ... Recording apparatus 21 ... Amplification part 22 ... Conversion part 23 ... Recording part 24 ... Accumulation part 30 ... Camera 50 ... Game device 51 ... Character control part 52 ... Camera control part 53 ... Rendering part 54 ... Sound processing part 55 ... Accumulation unit 70 ... Video / audio processing device 71 ... Mixer 72 ... Audio processing unit 73 ... Video input unit 74 ... Video processing unit 75 ... Control unit 100 ... Game machine 101 ... CPU
102 ... ROM
103 ... RAM
104 ... GPU
105 ... SPU
106 ... interface 107 ... DRIVE
200 ... Controller 300 ... HMD
301A, 301B ... Display unit 302 ... Sound output unit 303 ... Gyro sensor 304 ... Acceleration sensor 305 ... Control unit

Claims (8)

A plurality of microphones arranged at equal intervals on a concentric circle;
Recording means for recording audio signals collected by each of the plurality of microphones as audio signals in directions corresponding to the plurality of microphones,
A recording apparatus comprising: A video / audio processing program for operating a computer as a device for a user to wear a head-mounted display and view a video,
Input means for inputting movement information of the user's head from the head-mounted display and detecting the direction in which the user is facing;
Video processing means for displaying a video corresponding to the direction the user is facing on the head-mounted display;
Audio processing means for outputting audio in accordance with the direction the user is facing;
A video / audio processing program characterized by functioning as   If the direction in which the user is facing is between a plurality of directions corresponding to sound data, the plurality of sound processing means may correspond to the direction in which the user is facing between the plurality of directions. The video / audio processing program according to claim 2, wherein sound data corresponding to the direction is synthesized. Further causing the computer to function as storage means for storing sound data corresponding to a plurality of directions,
The video / audio processing program according to claim 2 or 3, wherein the audio processing means reads out and outputs the sound data corresponding to the direction in which the user is facing from the storage means.   5. The video / audio processing program according to claim 4, wherein the storage unit holds sound data collected by each of the plurality of microphones corresponding to the plurality of directions. The video processing means displays a video taken of a virtual space,
The sound processing means sets a virtual microphone corresponding to a plurality of directions when sounding a sound effect in the virtual space, determines the virtual microphone collecting the sound effect, and determines the direction of the sound effect The video / audio processing program according to claim 2, wherein a sound volume for outputting the sound effect is determined based on a direction in which the user is facing and a direction of the sound effect. Allowing the computer to further function as audio input means for receiving audio signals in a plurality of directions from each of a plurality of microphones;
The voice processing means selects and outputs the voice signal to be output from the voice signals in the plurality of directions received by the voice input means based on the direction in which the user is facing. The video / audio processing program according to claim 2 or 3. A program storage unit storing the video / audio processing program according to claim 2;
A computer for executing the video / audio processing program stored in the program storage unit;
A game device comprising:

Images