JP2024043429A - Realistic sound field reproduction device and realistic sound field reproduction method - Google Patents

Abstract

[Problem] To provide a realistic sound field reproducing device and method for reproducing with high sensitivity in a sound field reproducing space a sense of realism including sound images from various sound sources in a sound field recording space. [Solution] In a realistic sound field reproducing system 100, a satellite venue includes a satellite venue side communication unit that acquires performer sound signals recorded by multiple headset microphones worn by people, zone sound signals recorded by multiple zone microphones, and position information of the people from a live venue, a sound image localization zone determination unit that determines a main peripheral microphone among the multiple zone microphones based on the position information, and a sound field reproducing unit (audience side realistic parameter calculation unit 16, mixing/level balance adjustment unit, sound field reproducing processing unit) that executes sound field reproducing processing to reproduce the sound field of the live venue using multiple satellite speakers based on speech signals from the headset microphone, a first peripheral sound signal from the main peripheral microphone, and a second peripheral sound signal from a zone microphone other than the main peripheral microphone. [Selected Figure] Figure 1

Description

The present disclosure relates to a realistic sound field reproducing device and a realistic sound field reproducing method.

Recently, scene-based 3D sound reproduction technology has been attracting attention for its ability to reproduce (play) sound fields in real time. Scene-based 3D sound reproduction technology is a method of reproducing (playing) a 3D sound field in real time using speakers arranged to surround the viewing environment (space) by performing signal processing on multi-channel signals recorded (sound pickup) using an Ambisonics microphone, which has multiple directional microphone elements arranged on a solid sphere or a hollow sphere, and using this to reproduce (play) a 3D sound field in real time as if the listener were present at the location where the Ambisonics microphone was installed.

As a prior art related to sound field reproduction, for example, Patent Document 1 is known. Patent Document 1 discloses an audio recording device that receives a sound signal from a wireless microphone attached to a subject and generates a multichannel audio signal based on each audio signal collected by a plurality of microphones. There is. This audio recording device allocates a sound signal picked up by a wireless microphone to one or more arbitrary channels of a multi-channel audio signal, synthesizes them at an arbitrary synthesis ratio, and records them on a recording medium together with a captured image signal.

Japanese Patent Application Publication No. 2006-314078

Here, it is assumed that the above-mentioned scene-based stereophonic reproduction technology is used to reproduce a sound field from various sound sources (e.g., performers such as actors on stage, sound effects) recorded at a live venue such as a large concert hall in one or more satellite venues different from the live venue. Patent Document 1 does not disclose in detail the relationship between the atmosphere of the sound field picked up by a microphone and the atmosphere of the sound field picked up by a wireless microphone, and it is considered difficult to apply the technology of Patent Document 1 to realize the above-mentioned assumption. In addition, in order to reproduce (reproduce) the sense of realism of the live venue with high sensitivity in the satellite venue, there is a possibility that the sound (in other words, the sound image) when the performer on the stage of the live venue speaks alone may not be sufficient in terms of sound field reproduction for listeners in the satellite venue. However, Patent Document 1 does not present a solution for reproducing a sense of realism with high sensitivity using the above-mentioned various sound sources.

The present disclosure has been devised in consideration of the above-mentioned conventional situation, and aims to provide a realistic sound field reproduction device and a realistic sound field reproduction method that reproduces with high sensitivity in a sound field reproduction space a sense of realism including sound images from various sound sources in the sound field recording space.

The present disclosure provides a realistic sound field reproduction device comprising: an acquisition unit that acquires at least a speech sound signal recorded by a person microphone worn by at least one person capable of moving in an activity area in a sound field recording space, ambient sound signals recorded by multiple peripheral microphones arranged around the activity area, and position information of the person; a determination unit that determines a main peripheral microphone, which is one of the multiple peripheral microphones and has a recording area where the person is located in the activity area, based on the person's position information; and a sound field reproduction unit that executes a sound field reproduction process to reproduce a sound field in the sound field recording space using multiple speakers arranged in a sound field reproduction space different from the sound field recording space, based on at least the speech sound signal from the person microphone, a first ambient sound signal from the main peripheral microphone, and a second ambient sound signal from a peripheral microphone other than the main peripheral microphone.

The present disclosure also provides a method for reproducing a realistic sound field using a realistic sound field reproducing device, in which utterances are recorded by a person microphone worn by at least one person who can move in an activity area within a sound field recording space. a step of acquiring at least a signal, ambient sound signals recorded by a plurality of peripheral microphones arranged around the activity area, and position information of the person; determining a main surrounding microphone that is one of the surrounding microphones and whose recording area is a location where the person is located in the activity area; a plurality of speakers arranged in a sound field reproduction space different from the sound field recording space based on a first peripheral sound signal from the main peripheral microphone and a second peripheral sound signal from the peripheral microphone other than the main peripheral microphone. A method for reproducing a realistic sound field is provided, comprising: executing a sound field reproducing process for reproducing the sound field in the sound field recording space using the sound field recording space.

Note that these comprehensive or specific aspects may be realized by a system, an apparatus, a method, an integrated circuit, a computer program, or a recording medium. It may be realized by any combination of the following.

According to the present disclosure, a sense of presence including sound images from various sound sources in the sound field recording space can be reproduced with high sensitivity in the sound field reproduction space.

A block diagram showing an example of a system configuration of a realistic sound field reproduction system according to Embodiment 1. A diagram showing an example of the arrangement of stage end zone microphones and ceiling-mounted zone microphones when looking at the stage of a live venue from the ceiling. Diagram showing examples of recording ranges for stage edge zone microphones and ceiling-mounted zone microphones Diagram showing an example of directivity of a ceiling-mounted zone microphone A diagram showing an example of the arrangement of stage edge zone microphones and ceiling-suspended zone microphones when looking from the left (right) stage edge to the right (left) stage edge of a live venue stage. Transition diagram showing an example of regular updating of the matching table A diagram schematically showing an example of the operation overview of selecting a zone sound signal according to the position information of the performer. Diagram showing an example of satellite speaker placement in a satellite venue A sequence diagram chronologically showing an example of the operation procedure of the realistic sound field reproduction system according to Embodiment 1. A schematic diagram showing the concept of scene-based 3D sound reproduction using Ambisonics microphones, from sound field capture to sound field reproduction. FIG. 1 shows an example of a basis of Ambisonics components based on a spherical harmonic expansion for order n and degree m. A schematic diagram showing an example of the operation of a sound field realism reproduction system. FIG. 13 is a block diagram showing an example of a system configuration of a sound field realism reproduction system according to a second embodiment. FIG. 14 is a diagram showing an example of an outline of operations from sound field realism pickup to sound field realism reproduction in the sound field realism reproduction system of FIG. 13; Flowchart chronologically showing an example of an operation procedure for reproducing a sense of presence in a sound field by the sense of presence in a sound field according to the second embodiment FIG. 13 is a block diagram showing an example of a system configuration of a sound field realism reproduction system according to a modification of the second embodiment. Figure 16 is a diagram illustrating an example of an outline of the operation from sound field realistic sound collection to sound field realistic reproduction in the sound field realistic feeling reproduction system. Flowchart chronologically showing an example of an operation procedure for reproducing a sense of presence in a sound field by a sense of presence in a sound field according to a modification of the second embodiment. Block diagram showing a system configuration example of a sound field realistic sensation reproduction system according to Embodiment 3 Figure 19 is a diagram illustrating an example of an outline of the operation from sound field realistic sound collection to sound field realistic reproduction in the sound field realistic feeling reproduction system. 11 is a flowchart showing an example of an operation procedure for reproducing a sense of realism in a sound field by the sound field realism reproduction device according to the third embodiment. Block diagram showing a system configuration example of a sound field realistic sensation reproduction system according to a modification of Embodiment 3 Figure 22 is a diagram showing an example of an outline of the operation from sound field realistic sound collection to sound field realistic reproduction in the sound field realistic feeling reproduction system. Flowchart chronologically showing an example of an operation procedure for reproducing a sense of presence in a sound field by a sense of presence in a sound field according to a modification of the third embodiment.

DESCRIPTION OF EMBODIMENTS Hereinafter, embodiments specifically disclosing a realistic sound field reproducing device and a realistic sound field reproducing method according to the present disclosure will be described in detail with reference to the drawings as appropriate. However, more detailed explanation than necessary may be omitted. For example, detailed explanations of well-known matters and redundant explanations of substantially the same configurations may be omitted. This is to avoid unnecessary redundancy in the following description and to facilitate understanding by those skilled in the art. The accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter of the claims.

(Embodiment 1)
In the first embodiment, a performer such as an actor (an example of a person) speaks while a play is being performed on a stage set up in a sound field recording space (for example, a live venue such as a wide concert hall). We recorded sound source signals (sound signals) from various sound sources such as speech sounds when performers performed, sounds near the performers' feet, applause, cheers, and murmurs from the audience seats away from the stage, and created a sound field based on the sound source signals. An example will be described in which the sense of presence is reproduced in a sound field reproduction space (for example, a satellite venue such as a movie theater) that is different from the sound field recording space.

First, with reference to FIG. 1, an example of the system configuration of the realistic sound field reproduction system according to the first embodiment will be described. FIG. 1 is a block diagram showing an example of a system configuration of a realistic sound field reproduction system 1000 according to the first embodiment.

The realistic sound field reproduction system 1000 includes various types of equipment placed on the live venue LV1 side and various types of equipment placed on the satellite venue STL1 side. The live venue side communication unit 6, which is one of the devices on the live venue LV1 side, and the satellite venue side communication unit 11, which is one of the devices on the satellite venue STL1 side, are connected to each other to enable data communication via the network NW1. has been done. Network NW1 may be a wired network or a wireless network. The wired network may be, for example, at least one of a wired LAN (Local Area Network), a wired WAN (Wide Area Network), and a power line communication (PLC), and may be any other network configuration that allows wired communication. On the other hand, the wireless network includes at least one of wireless LAN such as Wi-Fi (registered trademark), wireless WAN, short-range wireless communication such as Bluetooth (registered trademark), and mobile mobile communication network such as 4G or 5G. , or other network configurations that allow wireless communication.

Various devices placed on the live venue LV1 side include, for example, headset microphones HM1, ..., HM4, zone microphones ZM1, ..., ZM7, ambisonics microphone AMB1, mixer 1, performer tracking system 2, and the like. It includes a face database 3, an encoder 4, a matching table database 5, and a live venue side communication section 6. In addition, in order to simplify the explanation, FIG. 1 shows an example of four headset microphones and seven zone microphones, but the number of headset microphones is not limited to four, and furthermore, the number of zone microphones is seven. Not limited.

Each of the headset microphones HM1 to HM4 is an example of a person microphone, and is a wireless microphone worn by at least one person (for example, a performer such as an actor) who can move around the stage STG1 (see FIG. 2) of the live venue LV1. It is possible to perform wireless communication of various control signals or data signals with the mixer 1. Each of the microphone elements of the headset microphones HM1 to HM4 is configured using, for example, an omnidirectional microphone or a directional microphone. Each of the headset microphones HM1 to HM4 collects (records) the speech voice of the wearer using its microphone element. Note that each of the headset microphones HM1 to HM4 may have a speaker, and may output the sound signal input to the speaker. Each of the headset microphones HM1 to HM4 is worn so as to face the mouth of the wearer. For example, headset microphone HM1 is worn by performer A, headset microphone HM2 is worn by performer B, headset microphone HM3 is worn by performer C, and headset microphone HM4 is worn by performer D. The speech signals collected (recorded) by each of the headset microphones HM1 to HM4 are identified by the identification information of the corresponding headset microphone (for example, the sound source ID, which is the identification number of the performer who wears it (see below)). It is also wirelessly transmitted to the mixer 1 as a performer sound signal.

Each of the zone microphones ZM1 to ZM7 is an example of a peripheral microphone, and is placed around the stage STG1 (see FIG. 2) of the live venue LV1, and is used for wired communication or communication of various control signals or data signals with the mixer 1. It is possible to perform wireless communication. Each of the microphone elements of the zone microphones ZM1 to ZM7 is configured using, for example, an omnidirectional microphone or a directional microphone. Each of the zone microphones ZM1 to ZM7 uses its microphone element to collect (record) sounds occurring near the feet or above the head of the person wearing the headset microphone. The sound signals picked up (recorded) by each of the zone microphones ZM1 to ZM7 are wirelessly transmitted to the mixer 1 as a zone sound signal. Details of arrangement examples of the zone microphones ZM1 to ZM7 will be described later with reference to FIGS. 2 to 5.

Ambisonics microphone AMB1 is an example of a sound recording device, and is placed in a peripheral area (for example, near the space between seats on the audience side) away from stage STG1 (see Figure 2) of live venue LV1, and between mixer 1 and It is possible to perform wired or wireless communication of various control signals or data signals. The ambisonics microphone AMB1 includes four microphone elements Mc1, Mc2, Mc3, and Mc4 (see Figure 10), and the microphone element Mc1 records sound from the front upper left direction of the casing of the ambisonics microphone AMB1 (see Figure 10). , microphone element Mc2 records the sound in the front lower right direction of the housing (see Figure 10), microphone element Mc3 records the sound in the rear lower left direction of the housing (see Figure 10), and microphone element Mc4 records the sound in the rear lower left direction of the housing (see Figure 10). Record the sound from the rear and upper right direction of the same housing (see Figure 10). For example, the ambisonics microphone AMB1 is used in an area different from the stage STG1 (see Figure 2) in the live venue LV1 (specifically, recording sounds such as applause, cheers, and murmurs from the audience side (in other words, a sense of presence on the audience side). ) is recorded. Note that the ambisonics microphone AMB1 may include more microphone elements having unidirectionality than the four microphone elements Mc1, Mc2, Mc3, and Mc4 arranged in the hollow, or may include microphone elements arranged on a hard sphere. A microphone element having omnidirectionality may be included. By using an ambisonics microphone equipped with a large number of microphone elements, it becomes possible to synthesize ambisonics signals of second order or higher order in the audience seat side realism parameter calculation unit 16, which will be described later. The audio signal from the audience seat side that is picked up (recorded) by the ambisonics microphone AMB1 is wirelessly transmitted to the mixer 1 as a realistic acoustic signal from the audience seat side.

The mixer 1 is configured using a processor such as a CPU (Central Processing Unit) included in the computer device P1. Mixer 1 receives performer sound signals with identification information (see above) from each of the headset microphones HM1 to HM4, zone sound signals from each of zone microphones ZM1 to ZM7, and audience sound signals from ambisonics microphone AMB1. The side realistic sensor sound signal is inputted, mixed as a live venue sound signal, and outputted to the encoder 4. Note that the mixer 1 mixes the zone sound signals from each of the zone microphones ZM1 to ZM7 and the ambience sound signal from the audience seat side from the ambisonics microphone AMB1 and outputs the mixture to the encoder 4, and also outputs the mixture to the encoder 4. The performer sound signals from each of the performers may be individually output to the encoder 4. Further, the mixer 1, the encoder 4, and the live venue side communication section 6 may be configured by one or more computer devices P1 (for example, a personal computer or a server computer). For example, the computer device P1 may be called a sound field recording device.

The performer tracking system 2 can perform wired or wireless communication of data signals with each of the face database 3, encoder 4, and matching table database 5. The performer tracking system 2 tracks performers who are moving or standing still on the stage STG1 (see FIG. 2) of the live venue LV1 while a play is being performed, by referring to the face database 3. After recognizing the performer, identification information of the performer is constantly issued or reconfirmed, and furthermore, the position information of the performer is acquired. As a result, the performer tracking system 2 arranges records having performer identification information (sometimes referred to as "sound source ID") and performer position information for each performer while the upper limit is set for a play, etc. Matching table data is generated and stored in the matching table database 5. The performer tracking system 2 includes an AI camera equipped with, for example, AI (Artificial Intelligence), and combines an image of the performer captured by the AI camera with a face photo of the performer registered in advance in the face database 3. The performer's identification information and location information are obtained by comparing the information. Note that the performer tracking system 2 does not need to be limited to acquiring the performer's identification information and location information using the AI camera and the face database 3, and may obtain the performer's identification information and location information using other equipment (for example, a sensor individually worn by the performer). ) may be performed based on the information detected. The performer tracking system 2 sends to the encoder 4 identification information and position information for each performer, and a data signal of a captured image of the performer.

The face database 3 is configured using, for example, a flash memory or a hard disk drive, and stores face photo data of each performer performing a play or the like on the stage STG1 (see FIG. 2) of the live venue LV1 as the performer's identification information. Registered in association. The face database 3 is connected so as to be accessible by the performer tracking system 2, and is referred to by the performer tracking system 2 when recognizing the performer and acquiring location information. In FIG. 1, the database is shown as "DB" for convenience.

The encoder 4 is configured using a processor such as a CPU included in the computer device P1. The encoder 4 inputs and acquires the data signal of the live venue sound signal from the mixer 1 and the identification information of each recognized performer, position information, and captured video data from the performer tracking system 2. The encoder 4 receives data from the performer tracking system 2 (for example, performer identification information) having performer identification information that matches the performer identification information (for example, sound source ID) attached to the performer sound signal data of the live venue sound signal. information and location information) may be obtained by referring to the matching table database 5. The encoder 4 associates (for example, IP (Internet Protocol) packetization) the performer sound signal data of the performers having common identification information with the identification information, position information, and captured video data, so that the data is transmitted via the network NW1. Generates IP packets for transmission and reception. Furthermore, the encoder 4 uses the zone sound signals from each of the zone microphones ZM1 to ZM7 and the audience-side realistic sound signal from the ambisonics microphone AMB1 to generate IP packets for transmission and reception via the network NW1.

The matching table database 5 is configured using, for example, a flash memory or a hard disk drive, and stores data of matching tables MaTL1, MaTL2, . . . (see FIG. 6) generated or updated by the performer tracking system 2. Details of the data in the matching tables MaTL1, MaTL2, . . . will be described later with reference to FIG.

The live venue communication unit 6 is configured using a communication circuit capable of realizing wired or wireless communication, for example, and performs wired or wireless communication of various control signals or data signals with the satellite venue communication unit 11 via the network NW1. The live venue communication unit 6 transmits various IP packets generated by the encoder 4 to the satellite venue communication unit 11 via the network NW1.

The various devices arranged on the satellite venue STL1 side include, for example, a satellite venue side communication unit 11, a decoder 12, a memory unit 13, a video output unit 14, a sound image localization zone determination unit 15, an audience side realism parameter calculation unit 16, a mixing/level balance adjustment unit 17, a sound field reproduction processing unit 18, and satellite speakers SPk1, ..., SPkp. p indicates the number of satellite speakers arranged and is a constant integer equal to or greater than 2.

The satellite venue side communication unit 11 is configured using a communication circuit that can realize communication using wired or wireless communication, for example, and exchanges various control signals or data signals with the live venue side communication unit 6 via the network NW1. Performs wired or wireless communication. The satellite venue side communication unit 11 receives various IP packets sent from the live venue side communication unit 6 via the network NW1 and outputs them to the decoder 12. Also, the satellite venue side communication section 11, the decoder 12, the storage section 13, the video output section 14, the sound image localization zone determination section 15, the audience seat side realistic sensation parameter calculation section 16, the mixing/level balance adjustment section 17, and the sound field reproduction processing section. 18 may be configured by one or more computer devices P1 (for example, a personal computer or a server computer). For example, the computer device P2 may be called a sound field reproduction device.

The decoder 12 is configured using a processor such as a CPU provided in the computer device P2. The decoder 12 decodes the IP packets from the satellite venue communication unit 11 to separate the performer sound signals, identification information, position information, and captured video data for each performer picked up (recorded) by each of the headset microphones HM1 to HM4, the zone sound signals from each of the zone microphones ZM1 to ZM7, and the audience-side realistic sound signal from the Ambisonics microphone AMB1. The decoder 12 sends the captured video data for each performer to the video output unit 14. The decoder 12 outputs the performer sound signals, identification information, position information, and captured video data for each performer picked up (recorded) by each of the headset microphones HM1 to HM4, and the zone sound signals from each of the zone microphones ZM1 to ZM7 to the sound image localization zone determination unit 15. The decoder 12 outputs the audience-side realism sound signal from the Ambisonics microphone AMB1 to the audience-side realism parameter calculation unit 16.

The storage unit 13 is configured using, for example, a flash memory or a hard disk drive, and stores sound image localization parameters calculated in advance for each of the satellite speakers SPk1 to SPkp or various data acquired by the decoder 12. A method for calculating the sound image localization parameter for each satellite speaker is disclosed in, for example, Japanese Patent Application Laid-Open No. 09-149500, so a description thereof will be omitted here. That is, the sound image localization parameters are, for each satellite speaker, the delay of another satellite speaker (volume auxiliary speaker) for supplementing the volume from that satellite speaker (localization speaker), and the attenuation level of that volume auxiliary speaker. This sound image localization parameter is used when the sound field reproduction processing section 18 executes sound image localization processing. In addition, the various data acquired by the decoder 12 include, for example, performer sound signals, identification information, position information, and captured video data for each performer collected (recorded) by each of the headset microphones HM1 to HM4. , zone sound signals from each of the zone microphones ZM1 to ZM7, and an audience-side realistic sound signal from the ambisonics microphone AMB1. Furthermore, the storage unit 13 stores position information (for example, coordinate information in three-dimensional space) within the satellite venue STL1 of each of the satellite speakers SPk1 to SPkp.

The video output unit 14 is configured using, for example, a projector, and projects and displays the captured video data for each performer from the decoder 12 on a screen SCR1 (see FIG. 8) arranged in the satellite venue STL1. Thereby, a plurality of audience members seated in the satellite venue STL1 can view the captured video imaged at the live venue LV1 so as to focus on each performer via the screen SCR1.

The sound image localization zone determination unit 15 is configured using a processor such as a CPU included in the computer device P2. The sound image localization zone determining unit 15 is an example of a determining unit, and based on the data from the decoder 12 (that is, the identification information and position information data for each performer), the sound image localization zone determination unit 15 determines the location of the stage STG1 (see FIG. 2) of the live venue LV1. A sound image localization zone is determined for reproducing (reproducing) a sound field (atmosphere) formed by the voice of the performer's utterances and the sound of footsteps caused by stopping or moving in the satellite venue STL1. The sound image localization zone referred to here is one of the sound collection zones ZN1 to ZN7 shown in FIG. 3. Details of this example of determining the sound image localization zone will be described later with reference to FIG.

The audience seat side realism parameter calculation unit 16 is configured using a processor such as a CPU included in the computer device P2. The audience seat-side presence sensation parameter calculation unit 16 is an example of a sound field reproduction unit, and is based on the data from the decoder 12 (that is, the audience-side presence sensation signal from the ambisonics microphone AMB1) and the satellite data stored in the storage unit 13. Based on the position information of each of the speakers SPk1 to SPkp, parameters for stereoscopic reproduction are calculated for three-dimensionally reproducing (reproducing) the realistic sound field on the audience seat side in the live venue LV1 at the satellite venue STL1. The audience seat side realism parameter calculation unit 16 uses this stereoscopic reproduction parameter to generate a speaker drive signal for each of the satellite speakers SPk1 to SPkp, and outputs the generated speaker drive signal to the mixing/level balance adjustment unit 17. Details of an example of calculating parameters for stereoscopic reproduction and an example of generating a speaker drive signal will be described in detail with reference to Embodiment 2.

The mixing/level balance adjustment unit 17 is configured using a processor such as a CPU provided in the computer device P2. The mixing/level balance adjustment unit 17 is an example of a sound field reproduction unit, and adjusts the balance of each signal level between the performer sound signals picked up (recorded) by each of the headset microphones HM1 to HM4 for each performer, the zone sound signals from each of the zone microphones ZM1 to ZM7, and the speaker drive signals for each satellite speaker. The mixing/level balance adjustment unit 17 mixes (i.e., mixes) the signals whose signal level balance has been adjusted. A detailed example of adjusting the balance of each signal level will be described later with reference to FIG. 7.

The sound field reproduction processing unit 18 is configured using a processor such as a CPU provided in the computer device P2. The sound field reproduction processing unit 18 is an example of a sound field reproduction unit, and executes a sound field reproduction process to reproduce (reproduce) the sound field (atmosphere) formed by the voice of the performer in the live venue LV1 and the foot sounds due to standing still or moving, etc., and the sound field (atmosphere) formed by the presence of the audience in the live venue LV1, in the satellite venue STL1, using the sound image localization parameters for each satellite speaker read from the storage unit 13 and the signal output from the mixing/level balance adjustment unit 17. Specifically, the sound field reproduction processing unit 18 executes a process of localizing the sound image of the performer's speech and the foot sounds via each satellite speaker, using the signal output from the mixing/level balance adjustment unit 17, in order to reproduce the sound field (atmosphere) formed by the voice of the performer in the live venue LV1 and the foot sounds due to standing still or moving, etc., in the satellite venue STL1. Furthermore, in order to reproduce the sound field (atmosphere) formed by the sense of realism of the audience seats in the live venue LV1 in the satellite venue STL1, the sound field reproduction processing unit 18 outputs the signals output from the mixing/level balance adjustment unit 17 (particularly the speaker drive signals for each satellite speaker) via the corresponding satellite speakers.

Each of the satellite speakers SPk1, . The number of satellite speakers installed can be changed depending on the sound field you want to reproduce (reproduce), and when you do not want to reproduce (reproduce) a specific direction, or when using general methods such as a transaural system or VBAP (Vector Based Amplitude Panning) method. By combining virtual sound image generation methods known in the art, a sound field may be reproduced using fewer than p satellite speakers. Conversely, the sound field may be reproduced using more than p speakers. Further, the installation position of the satellite speakers is not limited as long as they are installed so as to surround the reference position of the satellite venue STL1 (for example, the central position in the satellite venue STL1 where the listener PS1 is seated).

Next, details of the zone microphone shown in FIG. 1 will be described with reference to FIGS. 2 to 5. FIG. 2 is a diagram showing an example of the arrangement of zone microphones in the direction of the stage of a live venue viewed from the ceiling. FIG. 3 is a diagram showing an example of a recording range of a zone microphone. FIG. 4 is a diagram showing an example of directivity of a zone microphone. FIG. 5 is a diagram illustrating an example of the arrangement of zone microphones in a direction when looking from the left (right) stage edge to the right (left) stage edge of the stage at a live venue. In FIGS. 2 to 5, the Z axis is perpendicular to both the X and Y axes and indicates a height direction parallel to the direction of gravity, and the X axis is perpendicular to both the Z and Y axes and indicates the height direction of the stage STG1. The Y axis is perpendicular to both the Z axis and the Y axis and represents the depth direction of the stage STG1. In addition, seats for the audience of the live venue LV1 are provided in the positive direction of the Y-axis, and a backyard of the stage STG1 is provided in the negative direction of the Y-axis.

As shown in FIG. 2, four zone microphones ZM1, ZM2, ZM3, and ZM4 are arranged at the edge of the stage STG1 of the live venue LV1. On the other hand, three zone microphones ZM5, ZM6, and ZM7 are suspended from the upper part (for example, the ceiling) of the stage STG1.

As shown in FIG. 3, each of the zone microphones ZM1, ZM2, ZM3, and ZM4 is located within, for example, approximately elliptical (including circular) sound collection zones ZN1, ZN2, ZN3, and ZN4 (recording area or sound image localization zone). Collect (record) the sounds (including voice) generated by The length in the longitudinal direction of the sound collection zones ZN1 to ZN4 is 4 [m]. Therefore, the footstep sound when a performer, such as an actor, stops and takes steps or dances on one of the zone microphones ZM1 to ZM4 is mainly collected by the zone microphone corresponding to the sound collection zone that includes the position of the performer. Sound (recorded). However, it is not excluded that the footstep sound is collected (recorded) by a zone microphone corresponding to a sound collection zone adjacent to the sound collection zone including the performer's position.

The length of the stage STG1 in the depth direction is 12 [m], and each of the zone microphones ZM1, ZM2, ZM3, and ZM4 is placed at the end of the audience seat side of the stage STG1, and each of the zone microphones ZM5, ZM6, and ZM7 is placed at the end of the stage STG1 on the audience seat side. is suspended from the ceiling at a position approximately 6 [m] away from the audience seat side end of stage STG1 and at a height of 4 [m]. These dimensions are merely examples and are not limiting.

As shown in FIG. 3, each of the zone microphones ZM5, ZM6, and ZM7 collects (records) sounds (including audio) occurring within, for example, approximately circular sound collection zones ZN5, ZN6, and ZN7 (recording areas). do. The length of the sound collection zones ZN5 to ZN7 in the diametrical direction is 4 [m]. Therefore, when a performer, such as an actor, stops directly under or near one of the zone microphones ZM5 to ZM7 and makes a movement that produces sound near his or her head, the sound will be heard in the sound collection zone that includes the performer's overhead position. Sound is mainly collected (recorded) by the corresponding zone microphone. However, this does not preclude that the sound may be collected (recorded) by a zone microphone corresponding to a sound collection zone adjacent to the sound collection zone including the position above the performer's head.

As shown in FIG. 4, each of the zone microphones ZM5 to ZM7 has a sound collection zone up to 2.5 [m] ahead within a beam range (directivity) of 80 degrees when viewed from the center of the housing of the zone microphone. It is possible to collect (record) sounds generated in ZN5 to ZN7 (that is, the approximately circular area with a diameter of 4 [m] shown in FIG. 3). It is assumed that a portion above the head and near the mouth of the performer ACT1, such as an actor who can move on the stage STG1, is located 2.5 [m] ahead of the housing of each of the zone microphones ZM5 to ZM7.

Therefore, as shown in FIG. 5, when a performer ACT1 wearing a headset microphone HM1 utters lines or the like on the stage STG1, not only the performer's sound signal picked up by the headset microphone HM1 but also the A zone sound signal picked up by at least one zone microphone having the position of the performer ACT1 as a sound pickup zone is recorded and input to the mixer 1. Although FIG. 5 shows an example in which the height of the performer ACT1 is 1.5 [m] for explanation, the height does not have to be limited to 1.5 [m].

Next, with reference to FIG. 6, the matching table generated or updated by the performer tracking system 2 will be described. FIG. 6 is a transition diagram showing an example of regular updating of the matching table.

The matching tables MaTL1, MaTL2, ... show that each performer (for example, performer A, performer B, performer C, performer D, ...) performing a play on the stage STG1 (see FIG. 2) is tracked by the performer tracking system 2. This is data that defines identification information (sound source ID) and position information (for example, two-dimensional coordinates of (X, Y)) for each performer when the performer is recognized. Note that the performer's position information is shown as two-dimensional coordinates with a specific position on the stage STG1 (see FIG. 2) as the origin. This matching table is updated periodically, for example, at 1 second intervals. As shown in FIG. 6, matching table MaTL1 is generated at time t=t1, and matching table MaTL2 is generated one second later at t=t2. Note that the update interval does not need to be limited to 1 second.

For example, in the matching table MaTL1 of FIG. 6, at least performer A, performer B, and performer C are recognized by the performer tracking system 2 at time t=t1. In other words, performer A's sound source ID (1) and position information (4, 1), performer B's sound source ID (2) and position information (-3, 2), and performer C's sound source ID (3) and position information. (1, 0) are at least associated in the matching table MaTL1.

Meanwhile, in matching table MaTL2, at time t=t2, performer A and performer B are recognized by performer tracking system 2 in the same way as at time t=t1, but performer C is not recognized and performer D is newly recognized. In other words, performer A's sound source ID (1) and position information (1, 5), performer B's sound source ID (2) and position information (-3, 0), and performer D's sound source ID (4) and position information (-2, 3) are at least associated in matching table MaTL2. For example, it is suggested that performer C has disappeared from stage STG1 (see Figure 2) at time t=t2, and performer D has newly appeared in stage STG1 (see Figure 2) at time t=t2.

In this way, even if the performers move on the stage STG 1 (see FIG. 2) during a play, etc., the performer tracking system 2 recognizes and tracks each performer's identification information and location information. The data of the matching table indicating the location information of the performers is updated as needed.

Next, with reference to FIG. 7, selection of a zone sound signal according to position information of a performer will be explained. FIG. 7 is a diagram schematically showing an example of an outline of the operation of selecting a zone sound signal according to position information of a performer. To simplify the explanation of FIG. 7, it is assumed that a total of four zone microphones are arranged at the live venue LV1, and a total of four satellite speakers are arranged at the satellite venue STL1. Note that p, which indicates the number of satellite speakers arranged, does not have to be limited to 12.

It is assumed that the performer tracking system 2 acquires the position information of the performer A at a "certain time" and that the position indicated by the position information is within the sound collection zone ZN2 of the zone microphone ZM2. In the description of FIG. 7, this "certain time" may be referred to as a "specific time." In this case, the sound image localization zone determination unit 15 places the main peripheral microphone that mainly picks up (records) sounds near the head of the performer A or near the feet of the performer A into a zone based on the position information of the performer A at the "specific time." I decided on Mike ZM2. In addition, as the audio signal of performer A, not only the performer sound signal collected (recorded) by the headset microphone of performer A and the zone sound signal of zone microphone ZM2, but also the sound signal collected (recorded) by other zone microphones ZM1, ZM3, and ZM4 are included. The recorded zone sound signal is also input to the mixing/level balance adjustment section 17.

Further, it is assumed that the performer tracking system 2 acquires position information of the performer B at the same "specific time" (see above), and that the position indicated by the position information is within the sound collection zone ZN3 of the zone microphone ZM3. In this case, the sound image localization zone determining unit 15 places the main peripheral microphone that mainly picks up (records) sounds near the head or feet of the performer B into a zone based on the position information of the performer B at the "specific time." I decided on Mike ZM3. Furthermore, as the audio signal of performer B, not only the performer sound signal collected (recorded) by the headset microphone of performer B and the zone sound signal of zone microphone ZM3, but also the sound signal collected (recorded) by other zone microphones ZM1, ZM2, and ZM4 are included. The recorded zone sound signal is also input to the mixing/level balance adjustment section 17.

The mixing/level balance adjustment unit 17 adjusts the levels of the performer sound signals of performers A and B and the zone sound signals of zone microphones ZM2 and ZM3 to be relatively higher than the other performer sound signals and zone sound signals, and then mixes each signal, in order to reproduce the atmosphere of the sound field for performers A and B (for example, sounds generated by speech, or sounds generated overhead or near the feet due to the movements of performers A and B) in the satellite venue STL1. This mixing process is performed for each of the satellite speakers SPk1 to SPk4.

7, the mixing/level balance adjustment unit 17 mixes the performer sound signal of performer A, the performer sound signal of performer B, the zone sound signal of zone microphone ZM1, the zone sound signal of zone microphone ZM2, the zone sound signal of zone microphone ZM3, and the zone sound signal of zone microphone ZM4 for the satellite speaker SPk1. The mixing/level balance adjustment unit 17 also mixes the performer sound signal of performer A, the performer sound signal of performer B, the zone sound signal of zone microphone ZM1, the zone sound signal of zone microphone ZM2, the zone sound signal of zone microphone ZM3, and the zone sound signal of zone microphone ZM4 for the satellite speaker SPk2. The mixing/level balance adjustment unit 17 also mixes the performer sound signal of performer A, the performer sound signal of performer B, the zone sound signal of zone microphone ZM1, the zone sound signal of zone microphone ZM2, the zone sound signal of zone microphone ZM3, and the zone sound signal of zone microphone ZM4 for the satellite speaker SPk3. The mixing/level balance adjustment unit 17 also mixes the performer sound signal of performer A, the performer sound signal of performer B, the zone sound signal of zone microphone ZM1, the zone sound signal of zone microphone ZM2, the zone sound signal of zone microphone ZM3, and the zone sound signal of zone microphone ZM4 for the satellite speaker SPk4. The signals mixed for each satellite speaker are input to the sound field reproduction processing unit 18.

Using the input signals for each satellite speaker, the sound field reproduction processing unit 18 uses the input signals for each satellite speaker to adjust the sound field to the sound collection zone ZN2 of the live venue LV1 so that the performer A is located in the sound collection zone ZN2 of the live venue LV1 at the satellite venue STL1. The sound image localization parameters corresponding to the corresponding satellite speakers are acquired from the storage unit 13 and sound image localization (reproduction) processing is performed. In addition, the sound field reproduction processing unit 18 uses the input signals for each satellite speaker to adjust the sound collection zone of the satellite venue STL1 so that the performer B is located in the sound collection zone ZN3 of the live venue LV1. The sound image localization parameters corresponding to the satellite speakers corresponding to ZN3 are acquired from the storage unit 13, and sound image localization (reproduction) processing is performed.

Next, referring to FIG. 8, the arrangement of satellite speakers in a satellite venue will be described. FIG. 8 is a diagram showing an example of the arrangement of satellite speakers in a satellite venue. To make the explanation of FIG. 8 easier to understand, it is assumed that a total of 12 satellite speakers are arranged in satellite venue STL1. Note that p, which indicates the number of satellite speakers arranged, does not need to be limited to 12.

As shown in Figure 8, satellite venue STL1 has many seats in the center where multiple spectators can sit, and a screen SCR1 is provided directly in front of each seat. The video output unit 14 is, for example, a projector, and projects and outputs data signals of the captured video of the performers sent from the live venue communication unit 6 of the live venue LV1 onto the screen SCR1. This allows visitors to the satellite venue STL1 to watch the captured video projected onto the screen SCR1 and experience a performance such as a play being shown on the stage STG1 (see Figure 2) of the live venue LV1.

For example, four satellite speakers SPk1, SPk2, SPk3, and SPk4 are arranged at a certain distance from each other on the rear side of the screen SCR1. Since each of the satellite speakers SPk1 to SPk4 is arranged on the rear side of the screen SCR1, it is preferable that the sound field reproduction processing unit 18 uses them mainly for sound image localization of the sound field based on the speech of the performers on the stage STG1 (see FIG. 2) of the live venue LV1 and the sounds generated near the head or feet. In addition, the sound field reproduction processing unit 18 can also localize the sound field (sound image SIm1) of the performers in the live venue LV1 at the midpoint between the satellite speakers SPk1 and SPk2 by using the sound image localization parameters for the satellite speakers SPk1 and SPk2 from the memory unit 13. Similarly, the sound field reproduction processing unit 18 can localize the sound field (sound image SIm2) of the performer in the live venue LV1 at the midpoint between the satellite speakers SPk2 and SPk3 by using the sound image localization parameters for the satellite speakers SPk2 and SPk3 from the storage unit 13. Similarly, the sound field reproduction processing unit 18 can localize the sound field (sound image SIm3) of the performer in the live venue LV1 at the midpoint between the satellite speakers SPk3 and SPk4 by using the sound image localization parameters for the satellite speakers SPk3 and SPk4 from the storage unit 13.

Here, adjacent satellite speakers SPk1 and SPk2 are arranged so that the angle between the vector to satellite speaker SPk1 and the vector to satellite speaker SPk2 seen from listener PS1 seated in the audience seat is within 20 degrees. It is preferable that This allows the sound field reproduction processing section 18 to perform sound image localization processing with high accuracy. Note that the same holds true not only for the angle formed between the satellite speakers SPk1, SPk2 and the listener PS1, but also for adjacent satellite speakers arranged on the back side of the screen SCR1. That is, the same applies to the angle between the satellite speakers SPk2, SPk3 and the listener PS1, and the angle between the satellite speakers SPk3, SPk4 and the listener PS1.

Further, satellite speakers SPk5, SPk6, SPk7, SPk8, SPk9, SPk10, SPk11, and SPk12 are arranged at a certain distance apart so as to cover the sides of the wide audience seats in the satellite venue STL1. Since each of the satellite speakers SPk5 to SPk12 is arranged so as to cover the side surface of the audience seats in the satellite venue STL1, the sound field reproduction processing unit 18 mainly generates applause, cheers, and noise from the audience seats in the live venue LV1. It is preferable to use it for outputting a sense of presence on the audience seat side. For details on outputting the sense of presence on the audience seat side using each of these satellite speakers SPk5 to SPk12 (in other words, reproducing the speaker drive signal for each of the satellite speakers SPk5 to SPk12), see Embodiment 2. This will be explained in detail.

Next, with reference to FIG. 9, the operating procedure of the realistic sound field reproduction system 1000 according to the first embodiment will be described. FIG. 9 is a sequence diagram chronologically showing an example of the operation procedure of the realistic sound field reproduction system 1000 according to the first embodiment. A series of processes from step St1 to step St7 is executed within the live venue LV1, and this series of processes is repeated as a unit of processing while a play or the like is being performed. In order to simplify the explanation of FIG. 9, an example will be explained in which there is a performer A (see FIG. 6) such as one actor on the stage STG1 (see FIG. 2), but the number of performers There can be 2 or more people.

In Figure 9, the performer tracking system 2 recognizes performer A on stage STG1 (see Figure 2) of the live venue LV1, and sends the identification information (sound source ID), position information, and captured video data signal of performer A as the recognition result to the encoder 4 via the mixer 1 (step St1). In addition, by recognizing performer A, the performer tracking system 2 generates or updates data in a matching table (see Figure 6) containing performer A's identification information (sound source ID) and position information, and stores the data in the matching table database 5 (step St2).

The headset microphone HM1 worn by performer A picks up (records) the speech of performer A on the stage STG1 (see FIG. 2), and sends the performer sound signal of performer A, which is recorded, to the encoder 4 via the mixer 1 (step St3). Each of the zone microphones ZM1 to ZM7 arranged around the stage STG1 (see FIG. 2) picks up (records) the sound near the head or feet of performer A, and sends the zone sound signal of performer A, which is recorded, to the encoder 4 via the mixer 1 (step St4). The Ambisonics microphone AMB1 arranged in a peripheral area (e.g., near the seats on the audience side) away from the stage STG1 (see FIG. 2) picks up (records) an audience side realistic sound signal, which is a recording sound of the realistic sensation (specifically, applause, cheers, commotion, etc. on the audience side) in the audience side area away from the stage STG1 (see FIG. 2), and sends it to the encoder 4 via the mixer 1 (step St5).

The encoder 4 receives data from the performer tracking system 2 (for example, the performer's identification information and location information) is acquired by referring to the data sent in step St1 or the matching table database 5 (step St6). The encoder 4 associates data of the performer sound signal of performer A having common identification information with the data of the identification information, position information, and captured video (see step St1 or step ST6) (for example, by converting it into an IP packet). An IP packet for transmission and reception via the network NW1 is generated (step St6). Furthermore, the encoder 4 uses the zone sound signals from each of the zone microphones ZM1 to ZM7 and the audience-side realistic sound signal from the ambisonics microphone AMB1 to generate IP packets for transmission and reception via the network NW1 ( Step St6). The encoder 4 sends each IP packet generated in step St6 to the satellite venue side communication unit 11 via the live venue side communication unit 6 and the network NW1 (step St7).

The decoder 12 acquires each IP packet sent in step St7 via the satellite venue side communication unit 11, performs decoding processing, and extracts various data signals (step St8).

The sound image localization zone determining unit 15 distinguishes between the voice uttered by the performer A on the stage STG1 (see FIG. 2) of the live venue LV1 and A sound image localization zone for reproducing (reproducing) a sound field (atmosphere) formed by footstep sounds caused by standing still or moving, etc., within the satellite venue STL1 is determined (Step St9). The sound image localization zone referred to here is one of the sound collection zones ZN1 to ZN7 shown in FIG. 3. The audience seat side presence sensation parameter calculation unit 16 calculates the data from the decoder 12 (that is, the audience seat side presence sensation signal from the ambisonics microphone AMB1) and the position information of each of the satellite speakers SPk1 to SPkp stored in the storage unit 13. Based on this, parameters for stereoscopic reproduction are calculated for three-dimensionally reproducing (reproducing) the realistic sound field on the audience seat side in the live venue LV1 at the satellite venue STL1 (step St10). The audience seat-side presence feeling parameter calculation unit 16 uses this stereoscopic reproduction parameter to generate a speaker drive signal (an example of stereoscopic reproduction sound) for each of the satellite speakers SPk1 to SPkp, and sends the signal to the mixing/level balance adjustment unit 17. Output (step St11).

The sound image localization zone determination unit 15 reads and acquires the sound image localization parameters for each of the satellite speakers SPk1 to SPkp from the memory unit 13 (step St12). The sound image localization zone determination unit 15 sends the information on the sound image localization zone determined in step St9 and the information on the sound image localization parameters for each of the satellite speakers SPk1 to SPkp acquired in step St12 to the mixing/level balance adjustment unit 17 (step St13).

The mixing/level balance adjustment unit 17 uses the information sent in step St13 to adjust the balance of each signal level between the performer sound signals picked up (recorded) by each of the headset microphones HM1-HM4 for each performer, the zone sound signals from each of the zone microphones ZM1-ZM7, and the speaker drive signals for each satellite speaker (step St14). The mixing/level balance adjustment unit 17 mixes (i.e., mixes) each signal whose signal level balance has been adjusted (step St14), and sends the adjusted and mixed signal to the sound field reproduction processing unit 18 (step St15).

The sound field reproduction processing unit 18 uses the sound image localization parameters for each satellite speaker and the signal output from the mixing/level balance adjustment unit 17 to distinguish between the voices of the performers in the live venue LV1 and the footsteps of the performers who are standing still or moving. Execute sound field reproduction processing to reproduce (reproduce) the sound field (atmosphere) formed by the sound and the sound field (atmosphere) formed by the sense of presence on the audience side in the live venue LV1 at the satellite venue STL1. (Step St16). Specifically, the sound field reproduction processing unit 18 reproduces, in the satellite venue STL1, a sound field (atmosphere) formed by voices uttered by the performers in the live venue LV1 and footstep sounds caused by stopping or moving. Then, using the signal output from the mixing/level balance adjustment section 17, a process is performed to localize the sound image of the performer's speech and footstep sounds via each satellite speaker (Step St16). Furthermore, the sound field reproduction processing section 18 uses the output from the mixing/level balance adjustment section 17 in order to reproduce the sound field (atmosphere) formed by the sense of presence on the audience side in the live venue LV1 in the satellite venue STL1. A signal (particularly a speaker drive signal for each satellite speaker) is outputted via the corresponding satellite speaker (step St16).

As described above, in the realistic sound field reproducing system 1000 according to the first embodiment, the realistic sound field reproducing device is configured to support at least one person who can move in the activity area (stage STG1) in the sound field recording space (live venue LV1). Speech audio signals (performer sound signals) recorded by a person microphone (headset microphone HM1, etc.) worn by a person (performer ACT1) (performer ACT1), and multiple peripheral microphones (zone microphones ZM1 to ZM7) placed around the activity area. an acquisition unit (satellite venue side communication unit 11) that acquires at least an ambient sound signal recorded by the person and location information of the person; A determining unit (sound image localization zone determining unit 15) that determines the main peripheral microphone (for example, zone microphone ZM2) whose recording area is the location where the person is located in the activity area, at least the speech signal from the person microphone, and the main peripheral microphone. a first peripheral sound signal (for example, a zone sound signal of zone microphone ZM2), and a second peripheral sound signal by peripheral microphones other than the main peripheral microphone (for example, zone sound signals of zone microphones ZM1, ZM3 to ZM7). Based on this, to reproduce the sound field in the sound field recording space using multiple speakers (satellite speakers SPk1 to SPkp) placed in a sound field reproduction space (satellite venue STL1) different from the sound field recording space. The sound field reproduction section (audience seat side realism parameter calculation section 16, mixing/level balance adjustment section 17, sound field reproduction processing section 18) that executes the sound field reproduction processing of. As a result, the immersive sound field playback device reproduces the sense of presence, including the sound images from various sound sources (for example, at least one performer, etc.) in the sound field recording space (live venue LV1), using sounds different from those in the sound field recording space. Highly sensitive reproduction can be performed within the venue reproduction space (satellite venue STL1).

Further, the sound field reproduction unit (sound field reproduction processing unit 18) converts the first ambient sound signal (zone sound signal of zone microphone ZM2) into a second ambient sound signal (for example, zone sound signal) in the sound field reproduction space (satellite venue STL1). Sound field reproduction processing is performed so as to reproduce the sound field in the sound field recording space (live venue LV1) with more emphasis than the zone sound signals of the microphones ZM1 and ZM3 to ZM7. As a result, the realistic sound field reproducing device relatively emphasizes the speech sounds of the performers who are on the activity area (stage STG1) and are speaking their lines, etc., and the sounds near the head or feet of the satellite venue STL1. Even within a theater, the sound field can be reproduced so that the voice uttered by the performer and the sounds near the head or feet of the performer stand out, and the reproducibility of the sound field reproduction can be improved.

In addition, the acquisition unit (satellite venue side communication unit 11) acquires a sound recording device (ambisonics Further, the surrounding area sound signal (audience seat side realistic sound signal) recorded by the microphone AMB1) is acquired. The sound field reproduction unit (sound field reproduction processing unit 18) executes signal processing to reproduce a sound field based on the peripheral area sound signal in an area in the sound field reproduction space corresponding to the peripheral area in the sound field recording space. . As a result, the immersive sound field reproducing device sub-distributes sounds such as applause, cheers, and murmurs from the audience seats in the live venue LV1 that have been collected (recorded) by the sound recording device (audience side ambiance) as a secondary satellite. The sound field can be reproduced (reproduced) in the area corresponding to the audience seats in the venue LV1.

The sound field reproduction unit (mixing/level balance adjustment unit 17) also outputs a speech audio signal (performer sound signal of the performer), a first peripheral sound signal (zone sound signal of zone microphone ZM2), and a second peripheral sound signal (e.g. Mixing with the zone sound signals of the zone microphones ZM1 and ZM3 to ZM7 and balance adjustment of the signal level is performed, and output via each of a plurality of speakers (satellite speakers SPk1 to SPkp). As a result, the realistic sound field reproducing device can reproduce not only the speech sounds such as the performer's lines that are occurring on the activity area (stage STG1) in the live venue LV1, but also the sounds near the performer's head or feet. The atmosphere of the sound field created by the performers can be reproduced (reproduced) with high precision even within the satellite venue STL1.

The sound field reproduction unit (mixing/level balance adjustment unit 17) also outputs a speech audio signal (performer sound signal of the performer), a first peripheral sound signal (zone sound signal of zone microphone ZM2), and a second peripheral sound signal (e.g. It mixes the zone sound signals of the zone microphones ZM1, ZM3 to ZM7) and the surrounding area sound signals (audience seat-side immersive sound signals) and adjusts the signal level balance, Output via each. As a result, the immersive sound field reproducing device can reproduce not only speech sounds such as the performer's lines that are occurring on the activity area (stage STG1) in the live venue LV1, sounds near the performer's head or feet, but also the live performance. The sound atmosphere of the auditorium side in the venue LV1 can be reproduced (reproduced) with high precision even in the satellite venue STL1.

Further, the acquisition unit (satellite venue side communication unit 11) repeatedly acquires position information of a person, which can be updated periodically. The determining unit (sound image localization zone determining unit 15) determines the main peripheral microphone every time the position information of the person is updated. As a result, the realistic sound field reproducing device can acquire zone sound signals from the main peripheral microphones that match the position information of the person (performer) every time the person (performer) moves on the stage STG1 in the live venue LV1. Even if the position information of the performer changes, the sound field can be adaptively reproduced within the satellite venue STL1.

Furthermore, there are a plurality of people (performers such as actors), and the determination unit (sound image localization zone determination unit 15) determines the main peripheral microphone for each person based on the position information of the person. Thereby, the realistic sound field reproducing device can reproduce the sound image localization for each performer with high precision even when there are multiple performers on the stage STG1 in the live venue LV1.

In addition, the sound field reproduction unit (for example, the sound field reproduction processing unit 18) generates a speech sound signal (a performer's sound signal of a performer), a first ambient sound signal (a zone sound signal of zone microphone ZM2), and a second ambient sound signal (for example, Using the respective zone sound signals of the zone microphones ZM1 and ZM3 to ZM7 as reference signals, erasing processing for erasing the reference signal components included in the surrounding area sound signal (audience side realistic sound signal) is executed as signal processing. As a result, the realistic sound field reproduction device can effectively eliminate components of the reference signal that are thought to have leaked into the surrounding area sound signal, and can generate a high-quality surrounding area sound signal. .

Further, the sound field reproduction unit (for example, the sound field reproduction processing unit 18) encodes the signal after the erasure processing, and based on the signal after the encoding processing, performs a coding process on each of the plurality of speakers (satellite speakers SPk1 to SPkp). , generates a speaker drive signal for reproducing the sense of presence of the sound field in the sound field recording space in the sound field reproduction space. As a result, the realistic sound field reproducing device can reproduce the sense of presence on the audience seat side in the live venue LV1 by taking into account the position information of each of the multiple satellite speakers SPk1 to SPkp arranged in the space of the satellite venue STL1. A speaker drive signal can be generated.

Further, the sound field reproduction unit (for example, the sound field reproduction processing unit 18) outputs a speaker drive signal for each generated speaker (satellite speakers SPk1 to SPkp) from the corresponding speaker (satellite speaker). For example, the sound field reproduction processing unit 18 outputs a speaker drive signal generated for the satellite speaker SPk1 from the satellite speaker SPk1. Speaker drive signals generated for other satellite speakers are similarly output from the corresponding satellite speakers. As a result, the realistic sound field reproducing device reproduces the sense of presence on the audience seat side in the live venue LV1 by outputting speaker drive signals for each of the plurality of satellite speakers SPk1 to SPkp arranged in the space of the satellite venue STL1. It can be reproduced with high precision within the satellite venue STL1.

(Circumstances leading to Embodiment 2 and beyond)
Recently, scene-based stereophonic sound reproduction technology has been attracting attention in order to reproduce (reproduce) a sound field in real time. Scene-based 3D sound reproduction technology is a multi-channel signal collected using an ambisonics microphone in which multiple omnidirectional microphone elements are placed on a rigid sphere or multiple directional microphones are placed on a hollow sphere. By applying signal processing, speakers placed around the listening environment (space) can be used to create a three-dimensional sound field as if the listener were present at the location where the ambisonics microphone is installed. This is a method of reproducing (playing) in real time.

As a prior art related to sound field reproduction, for example, Reference Patent Document 1 is known. Reference Patent Document 1 discloses an audio recording device that receives a sound signal from a wireless microphone attached to a subject and generates a multichannel audio signal based on each audio signal collected by a plurality of microphones. ing. This audio recording device allocates a sound signal picked up by a wireless microphone to one or more arbitrary channels of a multi-channel audio signal, synthesizes them at an arbitrary synthesis ratio, and records them on a recording medium together with a captured image signal.

(Reference Patent Document 1) Japanese Patent Application Publication No. 2006-314078

Here, using the scene-based stereophonic sound reproduction technology described above, for example, an ambisonics microphone (see above) is placed on the audience side of a live performance venue such as a wide concert hall, and the ambisonics microphone (see above) is placed on the audience side of a live performance venue such as a wide concert hall. It captures the sense of presence such as applause, roars, murmurs, cheers, etc. from the audience seats during a performance (hereinafter sometimes referred to as "the sense of presence from the audience seats"), and makes the sense of presence from the audience seats different from that of a live venue. It is assumed that it will be reproduced at the above satellite venues. Reference Patent Document 1 does not disclose in detail the relationship between the atmosphere of the sound field picked up by the microphone and the atmosphere of the sound field picked up by the wireless microphone. It is considered difficult to apply this technology. In addition, even if the ambisonics microphone is placed on the audience side, it not only provides a sense of presence on the audience side, but also provides utterances such as the speech of actors on stage, sound effects, BGM (Background Music), original sound sources, etc. There is a high possibility that the performance sound will pick up the sound signal that propagated through the space inside the live venue. In this case, other sound components other than the sense of presence on the audience side in the live venue are mixed in, making it difficult to reproduce the sound field of the sense of presence on the audience side with high precision for the listeners in the satellite venue. Conceivable. Patent Document 1 does not present a solution for reproducing with high sensitivity in a satellite venue the sound field based on the sense of presence on the audience seat side, where the sound inside the live venue is collected.

Therefore, in Embodiment 2 and later described below, a sound field presence reproduction device that reproduces with high precision the atmosphere of presence on the audience seat side in a sound collection space in which sound is collected using an ambisonics microphone is provided in at least one satellite venue. An example of a method for reproducing a sense of presence in a sound field will be explained. Note that "realistic sound field reproduction" in Embodiment 1 and "sound field realistic reproduction" in each of the following embodiments are synonymous (in other words, the terms may be replaced with each other). I'll keep it.

In the following embodiments, a scene-based stereophonic sound reproduction technology using an ambisonics microphone as a sound collection device for collecting sound source signals such as sounds, music, and human voices in a sound collection space (for example, a live venue) will be exemplified. and explain. In this scene-based stereophonic sound reproduction technology, a point sound source that can be expressed as a signal picked up by multiple microphone elements constituting an ambisonics microphone (acoustic signal) or a monaural signal is converted into an intermediate representation ITMR1 ( (see FIG. 1) or by expressing (encoding) it as a B format signal, the sound field arriving from all directions is handled uniformly in the ambisonics signal domain (see below). Further, by decoding this intermediate representation, a speaker drive signal is generated, thereby realizing a desired sound field reproduction within a reproduction space (for example, a satellite venue).

Hereinafter, a "sound field" is defined as a space (including a location) in which sound spreads. Sound propagating within the sound field includes sounds from one or more sound sources propagating within the target space. Here, the sound sources include not only the sound sources of various performances (e.g., band performances, musical plays) being performed on the main stage of the sound collection space such as the live venue LV1, but also the sound sources from the main stage in the live venue LV1. It also includes sounds that give a sense of presence, such as cheers, murmurs, roars, and applause that occur in the audience seats that are far away.

(Embodiment 2)
First, the concept of scene-based stereophonic sound reproduction technology will be explained with reference to FIG. FIG. 10 is a diagram schematically showing the concept from sound field collection to sound field reproduction in the scene-based stereophonic sound reproduction technology using the ambisonics microphone AMB1. The ambisonics microphone AMB1 is arranged at a predetermined position on the audience seat side in the sound collection space (for example, the live venue LV1). In the live venue LV1, sound signals propagating within the space are collected by the ambisonics microphone AMB1. For example, if a band is performing by a plurality of people on the main stage of the live venue LV1, sound signals from various sound sources such as vocals (singing voices), bass, guitar, drums, etc. are collected. Furthermore, if a musical play is being performed, audio signals uttered by one or more performers (sound sources) such as actors are collected. On the other hand, sound signals that give a sense of presence on the audience side, such as cheers, murmurs, roars, and applause, generated on the audience side are also collected by the ambisonics microphone AMB1.

An ambisonics microphone AMB1 as an example of a sound collection device includes four microphone elements Mc1, Mc2, Mc3, and Mc4. Each of the microphone elements Mc1 to Mc4 is hollowly arranged so as to face four vertices from the center of the cube CB1 in FIG. 10 when the direction Dr1 is the front direction, and has unidirectivity in each vertex direction. ing. The microphone element Mc1 faces the front left up (FLU) of the ambisonics microphone AMB1, and mainly picks up sound in the front left up (FLU) direction. The microphone element Mc2 faces toward the front right down (FRD) of the ambisonics microphone AMB1, and mainly collects sounds in the front right down (FRD) direction. The microphone element Mc3 faces toward the back left down (BLD) of the ambisonics microphone AMB1, and mainly picks up sound in the back left direction. The microphone element Mc4 faces toward the back right up (BRU) of the ambisonics microphone AMB1, and mainly picks up sound from the rear upper right direction.

The collected signals of sounds from these four directions (i.e., FLU, FRD, BLD, BRU) are called A-format signals. The A-format signals cannot be used as is and are converted into B-format signals as intermediate representations ITMR1 having directional characteristics (directivity). The B-format signals have, for example, a B-format signal W for sounds from all directions (omnidirectional), a B-format signal X for sounds in the front-rear direction, a B-format signal Y for sounds in the left-right direction, and a B-format signal Z for sounds in the up-down direction. The A-format signals are converted into B-format signals using the following conversion formula:

W = FLU + FRD + BLD + BRU
X = FLU + FRD - BLD - BRU
Y = FLU - FRD + BLD - BRU
Z = FLU - FRD - BLD + BRU

By combining the B format signals W, X, Y, and Z, sound signals in all directions, front and back, left and right, and up and down are obtained. By changing the signal levels of the B-format signals W, X, Y, and Z and synthesizing them, it is possible to generate a sound signal having arbitrary directional characteristics in all directions: front, rear, left, right, and top and bottom. can. For example, as shown in FIG. 10, a total of eight satellite speakers SPk1, SPk2, SPk3, SPk4, SPk5, SPk6, SPk7, and SPk8 are installed at each vertex in a reproduction space modeled as a cube (for example, satellite venue STL1). Consider a three-dimensional coordinate system in which the sound collection space (for example, the live venue LV1) is arranged (that is, the front-rear, left-right, and up-down directions are parallel or in the same direction). Note that although the number of satellite speakers is exemplified as eight here to make the explanation easier to understand, it goes without saying that the number is not limited to eight.

The position of each of the satellite speakers SPk1 to SPk8 can be specified by a predetermined distance and angle (azimuth angle θ _i and elevation angle φ _i ) from a reference position (e.g., central position LSP1) of the reproduction space (e.g., satellite venue STL1). In Fig. 10, i is a variable indicating the satellite speaker located in the reproduction space (e.g., satellite venue STL1), and takes any integer from 1 to 8 in the example of Fig. 10.

It is assumed that a listener (listener) who is a user exists at the center position LSP1 of the reproduction space (for example, satellite venue STL1) and is facing the front direction (Front). Under such circumstances, the data and reproduction space of the B format signals W, The sound field in the sound collection space (for example, live venue LV1) is freely reproduced in the reproduction space (for example, satellite venue STL1) based on the respective directions of satellite speakers SPk1 to SPk8 in (for example, satellite venue STL1). be able to. In other words, when there is a listener who is a user in the reproduction space (for example, satellite venue STL1), the front direction of the listener is taken as the reference direction, and sound in any three-dimensional direction from that reference direction is reproduced and output. becomes possible.

Next, with reference to FIG. 11, the basis of the ambisonics component based on the spherical harmonic expansion for the order n and the power m will be described. FIG. 11 is a diagram showing an example of the basis of an ambisonics component based on spherical harmonic expansion with respect to order n and power m.

The horizontal axis (m) in FIG. 11 indicates degree, and the vertical axis (n) in FIG. 11 indicates order. The frequency m takes a value from -n to +n. The spherical harmonics up to order n=N include a total of (N+1) ^two bases. For example, when n=N=0, one basis (that is, an omnidirectional B format signal W) is obtained. For example, when n = N = 1, the omnidirectional B format signal W corresponding to the four bases (that is, (n, m) = (0, 0), (n, m) = (1, B-format signal X in the front-rear direction corresponding to -1), B-format signal Z in the vertical direction corresponding to (n, m) = (1, 0), corresponding to (n, m) = (1, 1) A horizontal B format signal Y) is obtained. Note that the same applies to n=N=2 and thereafter, so the explanation will be omitted.

It is known that spherical harmonics have a property of increasing spatial periodicity as n and m increase. Therefore, it is possible to express B format signals with different directional patterns (directional characteristics) depending on the combination of n and m. If the dimensions for order n and degree m are defined as K=n(n+1)+m based on Ambisonics Channel Numbering (ACN), the spherical harmonic function can be expressed in vector form as shown in equation (1). be. In formula (1), the superscript T indicates transposition.

Next, with reference to FIG. 12, an example of an outline of the operation of the sound field realistic sensation reproduction system will be described. FIG. 12 is a diagram schematically showing an example of an outline of the operation of the sound field realistic sensation reproduction system. In FIG. 12, the sound collection space in which the ambisonics microphone AMB1 is arranged will be explained by exemplifying a live venue LV1 where a band performance is performed using various sound sources such as vocals, bass, guitar, drums, etc. However, as mentioned above, in the live venue LV1, which is a sound collection space, it is not limited to band performances, but also musical performances performed by one or more actors, concerts with multiple musical instruments, or orchestra performances. The same applies below.

As shown in FIG. 12, a main stage STG1 is provided in the live venue LV1, and a band is performing on this main stage STG1. In a band performance, for example, an audio signal SS2 such as a singing voice from a vocalist (an example of a sound source), a bass sound signal SS1 from a bass (an example of a sound source), and a sound signal SS3 from a guitar (an example of a sound source) are widely used at the live venue. It propagates within the space within LV1 and reaches the audience seats. These signals may be directly transmitted through the space from each sound source position and reach the audience seats, or they may be reproduced through loudspeaker equipment such as speakers installed in the live venue LV1 and reach the audience seats. . The ambisonics microphone AMB1 is placed at a predetermined position on the audience side of the live venue LV1 (for example, the center position of the audience seats) for the purpose of mainly collecting sound that gives a sense of presence on the audience side. For this reason, the ambisonics microphone AMB1 mainly picks up sounds that give a sense of presence on the audience side, such as cheers, roars, murmurs, and applause from the audience side during the band performance.

However, as described above, the bass sound signal SS1, vocal sound signal SS2, and guitar sound signal SS3 during the band performance propagate within the space of the live venue LV1. Therefore, the components DS1, DS2, and DS3 of the diffused sounds (including reverberant sounds; the same applies hereinafter) of the sound signal SS1, the audio signal SS2, and the sound signal SS3 are picked up as sound signals by the ambisonics microphone AMB1. Therefore, the ambisonics microphone AMB1 picks up diffuse sound components DS1, DS2, and DS3 that are not originally wanted to be picked up. It was difficult to accurately reproduce the realistic sound from the audience seats.

Therefore, in the following embodiments, an example of a sound field immersion reproduction system that reproduces with high precision the atmosphere of presence on the audience seat side in a sound collection space in which sound is collected using an ambisonics microphone is provided in at least one satellite venue. Explain.

Next, with reference to FIGS. 13 and 14, the system configuration and operational outline of the sound field realistic sensation reproduction system 100 according to the second embodiment will be described. FIG. 13 is a block diagram showing an example of a system configuration of a sound field realistic sensation reproduction system 100 according to the second embodiment. FIG. 14 is a diagram illustrating an example of an outline of the operation from sound field presence sound collection to sound field presence reproduction in the sound field presence reproduction system 100 of FIG. 13.

The sound field presence reproduction system 100 includes a sound field presence sound pickup device 10 and a sound field presence reproduction device 20. The sound field presence sound pickup device 10 and the sound field presence reproduction device 20 are connected to each other via a network NW1 so as to be capable of data communication. Network NW1 may be a wired network or a wireless network. The wired network may be, for example, at least one of a wired LAN (Local Area Network), a wired WAN (Wide Area Network), and a power line communication (PLC), and may be any other network configuration that allows wired communication. On the other hand, the wireless network includes at least one of wireless LAN such as Wi-Fi (registered trademark), wireless WAN, short-range wireless communication such as Bluetooth (registered trademark), and mobile mobile communication network such as 4G or 5G. , or other network configurations that allow wireless communication.

The sound field realistic sound pickup device 10 is arranged in a sound pickup space (for example, a live venue LV1), and includes an ambisonics microphone AMB1, an A/D converter 7, and individual sound pickup microphones M1, . . . , Mn. Here, n indicates the number of individual sound sources (for example, independent sound sources such as vocals, bass, guitar, etc. in the case of a band performance) in the live venue LV1, and is specifically an integer of 2 or more. Note that the sound field presence sound pickup device 10 only needs to have at least the ambisonics microphone AMB1, and the A/D converter 7 may be provided in the sound field presence reproduction device 20.

Ambisonics microphone AMB1 includes four microphone elements Mc1, Mc2, Mc3, and Mc4. Microphone element Mc1 collects sound in the front upper left direction (see FIG. 10), and microphone element Mc2 collects sound in the front lower right direction (see FIG. 10). ), the microphone element Mc3 collects the sound in the rear lower left direction (see FIG. 10), and the rear upper right direction (see FIG. 10). Note that the ambisonics microphone AMB1 may include more microphone elements having unidirectionality than the four microphone elements Mc1, Mc2, Mc3, and Mc4 arranged in the hollow, or may include microphone elements arranged on a hard sphere. A microphone element having omnidirectionality may be included. By using an ambisonics microphone equipped with a large number of microphone elements, it becomes possible to synthesize ambisonics signals of second order or higher order in the encoding unit 22 of the sound field reality reproduction device 20. Signals (collected sound signals) collected by each microphone element constituting the ambisonics microphone AMB1 are input to the A/D converter 7.

At least the A/D conversion unit 7 is configured by a semiconductor chip or dedicated hardware that implements at least one of the following electronic devices: a CPU (Central Processing Unit), a DSP (Digital Signal Processor), a GPU (Graphical Processing Unit), an FPGA (Field Programmable Gate Array), etc.

The A/D converter 7 converts the analog sound pickup signal from each microphone element constituting the ambisonics microphone AMB1 into a digital sound pickup signal. This converted sound pickup signal is transmitted to the sound field presence reproduction device 20 via a communication interface (not shown) included in the sound field presence sound pickup device 10 and the network NW1.

The individual sound pickup microphone M1 is generated from a unique sound source (for example, vocals of a band performance or performers of a musical theater performance) during an event such as a band performance or a musical theater performance on the main stage STG1 (see FIG. 12) of the live venue LV1. Individual sound source sound (first sound source) is collected. The individual sound pickup microphone M1 may be, for example, a headset microphone worn by a vocalist in a band performance or a performer in a musical play. The individual sound source sound signal collected by the individual sound collection microphone M1 is transmitted to the sound field presence reproduction device 20 via a communication interface (not shown) included in the sound field presence sound pickup device 10 and the network NW1. .

Similarly, the individual sound pickup microphone Mn is configured to listen to a unique sound source (for example, a guitar in a band performance, or a guitar in a musical play) during an event such as a band performance or musical play on the main stage STG1 (see FIG. 12) of the live venue LV1. An individual sound source sound (nth sound source) generated from sound effects or BGM (Background Music) is collected. The individual sound collection microphone Mn may be, for example, a headset microphone worn by a guitar player playing in a band, or a microphone capable of collecting sound effects or BGM during a musical play. The individual sound source sound signals collected by the individual sound collection microphones Mn are transmitted to the sound field presence reproduction device 20 via a communication interface (not shown) included in the sound field presence sound pickup device 10 and the network NW1. .

The sound field realism reproduction device 20 is placed in a reproduction space (for example, satellite venue STL1) and includes echo cancellation units 21, ..., 2n, an encoding unit 22, a microphone element direction designation unit 23, a speaker direction designation unit 24, a decoding unit 25, a sound field reproduction unit 26, and satellite speakers SPk1, ..., SPkp. Here, p indicates the number of satellite speakers placed in the satellite venue STL1, and is specifically an integer of 2 or more. Also, n indicating the number of echo cancellation units 21 to 2n is the same as n indicating the number of individual sound pickup microphones M1 to Mn. In other words, the sound field realism reproduction device 20 is provided with the same number of echo cancellation units as the number of types of sound sources picked up by the individual sound pickup microphones. Note that the configurations of the echo cancellation units 21 to 2n, the encoding unit 22, the microphone element direction designation unit 23, the speaker direction designation unit 24, the decoding unit 25, and the sound field reproduction unit 26 in FIG. 13 correspond to the configurations of the audience seat side presence parameter calculation unit 16 and the sound field reproduction processing unit 18 in embodiment 1 (see FIG. 1).

The echo canceling unit 21 inputs the sound pickup signal for each microphone element of the ambisonics microphone AMB1 sent from the sound field immersive sound pickup device 10 (A/D converter 7 side), and further performs sound field ambience sensing. The individual sound source sound signal of the first sound source (see above) sent from the sound device 10 (individual sound collection microphone M1 side) is input as the first reference signal M1S. The echo cancellation unit 21 cancels the component of the first reference signal M1S (that is, the individual sound source sound signal collected by the individual sound collection microphone M1) included in the sound signal collected by each microphone element of the ambisonics microphone AMB1. Executes erasure processing (for example, echo cancellation processing). The echo canceling unit 21 outputs the signal after the cancellation process (first collected sound signal) to the encoding unit 22.

Similarly, the echo canceling unit 2n inputs the sound pickup signal for each microphone element of the ambisonics microphone AMB1 sent from the sound field presence sound pickup device 10 (A/D conversion unit 7 side), and The individual sound source sound signal of the nth sound source (see above) sent from the realistic sound pickup device 10 (individual sound collection microphone Mn side) is input as the nth reference signal MnS. The echo canceling unit 2n cancels the component of the n-th reference signal MnS (that is, the individual sound source sound signal collected by the individual sound collection microphone Mn) included in the sound signal collected by each microphone element of the ambisonics microphone AMB1. Executes erasure processing (for example, echo cancellation processing). The echo canceling unit 2n outputs the signal after the cancellation process (the nth collected sound signal) to the encoding unit 22.

Here, each of the echo canceling units 21 to 2n may be configured as an echo canceller using an adaptive filter that operates in the time domain, for example. This echo canceller can be configured as a Single Channel EchoCanceller, for example. Therefore, as shown in FIG. 14, each of the echo canceling units 21 to 2n can be configured using the same number of Single Channel Echo Cancellers as the number of microphone elements (for example, 4) of the ambisonics microphone AMB1. This Single Channel EchoCanceller may have the configuration disclosed in the referenced non-patent document, for example. By using this configuration, the ambisonics microphone AMB1 It becomes possible to highly accurately eliminate (suppress) the component of the reference signal (that is, the individual sound source sound signal collected by the individual sound collection microphone) included in the sound pickup signal for each microphone element. Further, the echo canceling units 21 to 2n may be implemented as echo canceling processing using an adaptive filter on the frequency domain or subband domain after forwardly transforming the time domain signal using DFT (Discrete Fourier Transform) or the like. The post-cancelled signal may be inversely transformed into the time domain by IFFT (Inverse Fast Fourier Transform) or the like, and then the subsequent processing may be performed.

<Reference non-patent literature> Chapter 5 Acoustic echo canceller, "Configuration example of adaptive filter" (see Figures 5 and 2), p4/(17), Institute of Electronics, Information and Communication Engineers, 2012, [September 2, 2020] Search], Internet <URL: https://www. ieice-hbkb. org/files/02/02gun_06hen_05. pdf>

Each of the echo cancellation units 21 to 2n is provided for the purpose of erasing (suppressing) the individual sound source sound propagated in the sound collection space (for example, live venue LV1). For this reason, each of the echo cancellation units 21 to 2n may be provided on the sound collection space (for example, live venue LV1) side together with the encoding unit 22, or on the reproduction space (for example, satellite venue STL) side together with the encoding unit 22. In this case, when provided on the sound collection space (for example, live venue LV1) side, only the component of the first-order Ambisonics signal (i.e., the realism of the audience seats) output from the encoding unit 22 is sent to the sound field realism reproduction device 20. On the other hand, when provided on the reproduction space (for example, satellite space STL1) side, the component of the first-order Ambisonics signal (i.e., the realism of the audience seats) output from the encoding unit 22 and the individual sound source sound signal are sent to the sound field realism reproduction device 20. Also, only the echo cancellation units 21 to 2n may be provided in the sound collection space (for example, the live venue LV1), and the encoding unit 22 may be provided in the reproduction space (for example, the satellite venue STL). In this case, only the output signal component of the echo cancellation unit 2n is sent to the sound field realism reproduction device 20.

Furthermore, on the side of the reproduction space (for example, satellite venue STL1), the sound field presence reproduction device 20 receives individual sound source sound signals picked up by each of the individual sound pickup microphones M1 to Mn of the sound field presence sound pickup device 10. For the purpose of reproducing a sense of presence in a sound field, the signal may be output from each of the satellite speakers SPk1 to SPkp, or from another satellite speaker (not shown) provided for an individual sound source sound signal.

Here, details of the encoding process by the encoding unit 22 will be explained.

In general, the sound pressure p observed (collected) at a position of radius r for any angle (θ, φ) on a spherical surface is expressed as a wave number k as a solution to an internal problem in the spherical harmonic domain of the wave equation. On the other hand, it is known that Equation (2) can be expanded as Equation (4) using the spherical harmonic function as a basis. In equation (4), A ^m _n is an expansion coefficient and R _n (kr) is a radial function term. Furthermore, the infinite sum with respect to order n is approximated by truncating at a finite order N, and the accuracy of sound field reproduction changes according to this truncated order N. Hereinafter, the truncation order will be expressed as N.

In equation (6), i is an imaginary unit, j _n (kr) is an n-th spherical Bessel function, and j ^′ _n (kr) is its derivative. In the present disclosure, the expansion coefficient vector γ ^m _n for this plane wave is handled as a B format signal (intermediate representation) that is the output of the encoding process by the encoding unit 22. Hereinafter, this expansion coefficient vector may be referred to as an ambisonics domain signal on an ambisonics domain different from the time domain or simply an ambisonics signal.

More specifically, in the encoding process by the encoding unit 22, the collected sound signal is a time domain signal after the cancellation process of the components of the reference signals output from each of the echo canceling units 21 to 2n. is converted into an ambisonics signal (for example, a first-order ambisonics signal), and this ambisonics signal (for example, a first-order ambisonics signal) is decoded by the decoding section 25 and converted into a speaker drive signal.

The sound field reproduction unit 26 converts the digital speaker drive signal for each satellite speaker output from the decoding unit 25 into an analog speaker drive signal, amplifies the signal, and outputs (reproduces) the signal from the corresponding satellite speaker. .

The satellite speakers SPk1, . Regenerate (recreate) the place. Note that the number of speakers installed may be changed depending on the sound field to be reproduced, and may be used in cases where reproduction is not performed in a specific direction, or in cases where generally known virtual sound image generation methods such as the transaural system and VBAP (Vector Based Amplitude Panning) method are used. By combining these, the sound field may be reproduced using fewer than p satellite speakers (for example, eight in the example of FIG. 10). Conversely, sound field reproduction may be performed using more than p satellite speakers (for example, 8 in the example of FIG. 10). Further, the speaker installation position may be other than each vertex of the reproduction space (for example, the satellite venue STL1) as long as it is installed so as to surround the reference position (for example, the center position LSP1) of the satellite venue STL1. The sound field reproduction unit 26 may output the signal to a binaural reproduction device such as headphones or earphones worn by the listener (user) instead of the satellite speaker. Furthermore, when the sound field reproduction unit 28 supplies a signal to a reproduction device for both ears of the listener (user) (for example, the above-mentioned headphones or earphones), the sound field reproduction unit 28 corresponds to an azimuth angle of +-90° by decoding processing described later. Alternatively, a virtual sound image may be generated in multiple directions surrounding the head, and the user may perceive a 3D sound image such as HRTF (Head Related Transfer Function) corresponding to the multiple angles. The reproduction signal may be generated by multiplying the virtual sound image in the corresponding direction by the transfer characteristic for the reproduction in the frequency domain or by convolving it in the time domain. As a result, the sound field is not reproduced only from each of the satellite speakers SPk1 to SPkp placed in the satellite venue STL1, but the playback device ( For example, it is also possible to reproduce the sound field on the headphones and earphones mentioned above.

Here, we will explain the details of the processing performed by the decoding unit 25.

Next, with reference to FIG. 15, a description will be given of an operation procedure for reproducing a sound field sense of presence by the sound field sense of presence reproduction apparatus 20. FIG. 15 is a flowchart chronologically showing an example of an operation procedure for reproducing a sound field sense of presence by the sound field sense of presence reproduction apparatus 20 according to the second embodiment.

15, the Ambisonics microphone AMB1 of the sound field realism pickup device 10 picks up sounds (e.g., sounds that give a sense of realism to the audience) occurring around a specific position on the audience side located in the pickup space (e.g., live venue LV1) (step St21). The pickup signals for each microphone element of the Ambisonics microphone AMB1 picked up in step St21 are transmitted to the sound field realism reproduction device 20B. However, as described above, the sounds picked up in step St21 include not only sounds that give a sense of realism to the audience side, but also sounds from one or more sound sources due to performances or plays on the main stage STG1 (see FIG. 12) of the pickup space (e.g., live venue LV1). In addition, individual sound source sounds (in other words, the first reference signal M1S to the nth reference signal MnS) picked up by each of the individual pickup microphones M1 to Mn from one or more sound sources such as a performance or a play on the main stage STG1 (see FIG. 12) of the sound collection space (e.g., the live performance venue LV1) are also transmitted to the sound field realism reproduction device 20 (step St21).

In each of the echo canceling units 21 to 2n, the sound field reality reproduction device 20 uses the picked-up signal of each microphone element of the ambisonics microphone AMB1 as a main signal, and refers to the picked-up signal of each of the individual sound collecting microphones M1 to Mn as a main signal. The echo canceling process (see above) made into a signal is repeatedly executed for each reference signal on the time axis (step St22). More specifically, the echo canceling unit 21 of the sound field presence reproduction device 20 receives various signals sent in step St21 (specifically, the sound pickup signal for each microphone element of the ambisonics microphone AMB1, the corresponding The first reference signal M1S) is input, and the first reference signal M1S (that is, the individual sound source sound signal collected by the individual sound collection microphone M1) included in the sound collection signal for each microphone element of the ambisonics microphone AMB1 is Erasing processing (for example, echo canceling processing) for erasing the components is executed on the time axis (Step St22). Similarly, the echo canceling unit 2n of the sound field presence reproduction device 20 receives various signals sent in step St21 (specifically, the sound pickup signal for each microphone element of the ambisonics microphone AMB1, the corresponding n-th input the reference signal MnS) and erase the component of the n-th reference signal MnS (that is, the individual sound source sound signal picked up by the individual sound pickup microphone Mn) included in the sound pickup signal for each microphone element of the ambisonics microphone AMB1. Erasing processing (for example, echo canceling processing) for this purpose is executed on the time axis (Step St22).

As described above, the sound field presence reproduction device 20 according to the second embodiment is configured to provide a sound pickup signal (diffusion an acquisition unit (echo cancellation units 21 to 2n) that acquires sound components DS1 to DS3) and sound source signals (sound signal SS1, audio signal SS2, and sound signal SS3) of one or more sound sources in the sound collection space; An erasing section (echo canceling section 21 to 2n) that uses the sound source signal as a reference signal and performs an erasing process on the time axis to erase the reference signal component included in the collected sound signal, and encodes the signal after the erasing process. Based on the encoding unit 22 to process and the signal after encoding processing, the recording is performed for each of the plurality of speakers (satellite speakers SPk1 to SPkp) arranged in a reproduction space (satellite venue STL1) different from the sound collection space. A generation unit (decoding unit 25) that generates a speaker drive signal for reproducing the sense of presence of a sound field in a reproduction space, and a sound output unit that outputs a speaker drive signal for each speaker from each of a plurality of speakers. A field reproduction section 26 is provided. Thereby, the sound field presence reproduction device 20 according to the second embodiment reproduces the atmosphere of presence on the audience seat side in the sound collection space (live venue LV1) mainly collected by the ambisonics microphone AMB1. By erasing on the time axis the components of one or more individual sound sources (reference signals) in the live venue LV1 that are picked up by the sound, it is possible to reproduce the sound with high precision in at least one satellite venue.

The sound field reality reproduction device 20 further includes a microphone element direction designation unit 23 that designates direction information of a plurality of microphone elements Mc1 to Mc4 included in the sound collection device (ambisonics microphone AMB1). The encoding unit 22 executes encoding processing using the direction information of each of the plurality of microphone elements Mc1 to Mc4 and the signal after the erasure processing. Thereby, the sound field presence reproduction device 20 generates an ambisonics signal having a plurality of directional resolutions (see the B format signal in FIG. 10) by taking into account the direction information of each of the microphone elements Mc1 to Mc4 included in the ambisonics microphone AMB1. can be generated.

Furthermore, the sound field presence reproduction device 20 further includes a speaker direction designation unit 24 that designates direction information of a plurality of speakers (satellite speakers SPk1 to SPkp) in the reproduction space (satellite venue STL1). The generation unit (decoding unit 25) generates a speaker drive signal for each of the plurality of speakers in the ambisonics region using the direction information of each of the plurality of speakers and the encoded signal. As a result, the sound field presence reproduction device 20 can move from the reference position (for example, refer to the center position LSP1 corresponding to the listener's position) of each of the plurality of satellite speakers SPk1 to SPkp arranged in the space of the satellite venue STL1. It is possible to generate a speaker drive signal that can reproduce the sense of presence on the audience seat side in the live venue LV1 by taking into account the direction information.

The elimination unit (echo cancellation unit 21-2n) of the sound field realistic reproduction device 20 is composed of a number of single echo cancellers (see FIG. 14) determined based on the number of microphone elements Mc1-Mc4 of the sound collection device (Ambisonics microphone AMB1) and the number of sound sources. Each single echo canceller inputs a sound source signal of the corresponding sound source (for example, a sound signal or a voice signal from an individual sound collection microphone) and performs an elimination process (echo cancellation process) on the time axis. As a result, particularly when there is no correlation between the reference signals (in other words, when it is less than a predetermined threshold value that can be considered to be substantially free of crosstalk components), it is possible to highly accurately eliminate (suppress) the components of the reference signal (i.e., the individual sound source sound signal collected by the individual sound collection microphone Mn) contained in the collected signal for each microphone element of the Ambisonics microphone AMB1. Substantially no crosstalk components are present when, for example, the voice of a vocalist singing on the main stage STG1 (see FIG. 12) is not picked up by other individual microphones, or if it is picked up, the sound pressure level is below the predetermined threshold value mentioned above.

(Modification of Embodiment 2)
In the second embodiment, an example has been described in which each of the echo canceling units 21 to 2n is configured as a Single Echo Canceller in the sound field realistic feeling reproduction device. In a modification of the second embodiment, an example will be described in which a sound field realistic sensation reproducing apparatus is configured as a multi-channel echo canceller that handles a plurality of audio channels instead of the echo canceling sections 21 to 2n. In addition, in the modification of the second embodiment, configurations and contents that overlap with those of the second embodiment will be given corresponding common reference numerals to simplify or omit the explanation, and different contents will be explained.

First, with reference to FIGS. 16 and 17, the system configuration and operational outline of a sound field realistic sensation reproduction system 100A according to a modification of the second embodiment will be described. FIG. 16 is a block diagram showing a system configuration example of a sound field realistic sensation reproduction system 100A according to a modification of the second embodiment. FIG. 17 is a diagram illustrating an example of an outline of operations from sound field presence sound collection to sound field presence reproduction in the sound field presence reproduction system 100A of FIG. 16.

The sound field presence reproduction system 100A includes a sound field presence sound pickup device 10 and a sound field presence reproduction device 20A. The sound field presence sound pickup device 10 and the sound field presence reproduction device 20A are connected to each other via a network NW1 so as to be capable of data communication.

The sound field presence reproduction device 20A is arranged in a reproduction space (for example, the satellite venue STL1), and includes a multichannel echo canceling section 21A, an encoding section 22, a microphone element direction specifying section 23, a speaker direction specifying section 24, It includes a decoding section 25, a sound field reproduction section 26, and satellite speakers SPk1, . . . , SPkp. That is, in the modified example of the second embodiment, instead of the echo canceling units 21 to 2n of the second embodiment, a multi-channel echo canceller inputs the sound source signal of the sound source picked up by each of the n individual sound collection microphones. The section 21A is provided in the sound field realistic sensation reproduction device 20A. Note that the configurations of the multichannel echo canceling section 21A, encoding section 22, microphone element direction specifying section 23, speaker direction specifying section 24, decoding section 25, and sound field reproduction section 26 in FIG. 16 are the same as in the first embodiment. This corresponds to the configuration of the audience seat side realism parameter calculation unit 16 and sound field reproduction processing unit 18 (see FIG. 1).

The multi-channel echo canceling unit 21A inputs the sound pickup signal for each microphone element of the ambisonics microphone AMB1 sent from the sound field realistic sound pickup device 10 (A/D conversion unit 7 side), and further inputs the sound pickup signal for each microphone element of the ambisonic microphone AMB1. The respective individual sound source sound signals of the first sound source (see above) to the nth sound source (see above) sent from the sound pickup device 10 (individual sound collection microphone M1 side) are converted into first reference signals M1S to Nth sound source (see above). n reference signal MnS. The multi-channel echo cancellation unit 21A is configured to convert the first reference signal M1S (that is, the individual sound source sound signal collected by the individual sound collection microphone M1) to the nth reference signal MnS, which are included in the sound signal collected by the ambisonics microphone AMB1. (that is, the individual sound source sound signal picked up by the individual sound collection microphone Mn), an erasure process (for example, multi-channel echo cancellation process) is executed in the time domain. The multi-channel echo canceling unit 21A outputs the signals after the cancellation process (first to nth collected sound signals) to the encoding unit 22.

Here, the multi-channel echo canceller 21A may be configured as an echo canceller using an adaptive filter that operates in the time domain, for example. This echo canceller can be configured as a multi-channel echo canceller, for example. Note that the configuration of this multi-channel echo canceller may refer to the stereo echo canceller shown in the reference non-patent document, and this stereo echo canceller is an example in which two reference signals are input to the multi-channel echo canceller. Therefore, as shown in FIG. 17, the multi-channel echo canceller 21A can be configured using a multi-channel echo canceller or a stereo echo canceller in the same number as the number of microphone elements (for example, 4) of the Ambisonics microphone AMB1. This Multi Channel Echo Canceller or stereo echo canceller may be, for example, a configuration disclosed in a reference non-patent document or a configuration obtained by referring to that configuration. By using this configuration, even if there is a correlation between the reference signals (in other words, when the correlation is equal to or exceeds a predetermined threshold value to the extent that it can be considered that no crosstalk components are included), it is possible to highly accurately erase (suppress) the components of the reference signal (i.e., the individual sound source signal picked up by the individual sound pickup microphone Mn) contained in the sound pickup signal of the Ambisonics microphone AMB1. In addition, the multi-channel echo cancellation unit 21A may be realized as an echo cancellation process using an adaptive filter in the frequency domain or subband domain after forward transforming the time domain signal by DFT or the like, or the canceled signal may be inversely transformed to the time domain by IFFT or the like before performing subsequent processing.

<Reference non-patent literature> Chapter 5 Acoustic echo canceller, "Configuration example of stereo echo canceller" (see Figures 5 and 8), Institute of Electronics, Information and Communication Engineers, 2012, [Retrieved September 2, 2020], Internet < URL: https://www. ieice-hbkb. org/files/02/02gun_06hen_05. pdf>

Next, with reference to FIG. 18, an explanation will be given of an operation procedure for reproducing a sound field sense of presence by the sound field sense of presence reproduction apparatus 20A. FIG. 18 is a flowchart chronologically showing an example of an operation procedure for reproducing a sound field sense of presence by the sound field sense of presence reproduction apparatus according to a modification of the second embodiment. In the explanation of FIG. 18, the same step numbers are given to the contents that overlap with the explanation of FIG. 15 to simplify or omit the explanation, and the different contents will be explained.

In FIG. 18, the sound field presence reproduction device 20A uses the picked-up signal of the ambisonics microphone AMB1 as the main signal and the picked-up signals of the individual sound collecting microphones M1 to Mn as the reference signal in the multi-channel echo canceling unit 21A. The multi-channel echo cancellation processing (see above) is executed in the time domain (Step St22A). More specifically, the multi-channel echo canceling unit 21A of the sound field presence reproduction device 20A receives the various signals sent in step St21 (specifically, the sound pickup signal for each microphone element of the ambisonics microphone AMB1, The first reference signal M1S to the n-th reference signal MnS) are input, and the first reference signal M1S (that is, the individual sound collection microphone M1 is Erasing processing (for example, multi-channel echo cancellation processing) for erasing the component of the individual sound source sound signal) to the n-th reference signal MnS (that is, the individual sound source sound signal picked up by the individual sound collection microphone Mn) is executed on the time axis (step St22A). Since the processing after step St22A is the same as that in FIG. 15, the explanation will be omitted.

As described above, in the sound field presence reproduction system 100A according to the modified example of the second embodiment, the canceling section (multichannel echo canceling section 21A) of the sound field presence reproduction device 20A is configured to use the sound pickup device (ambisonics microphone AMB1). It is constituted by a number of multi-channel echo cancellers determined based on the number of microphone elements Mc1 to Mc4 included in the device (see FIG. 17). Each multi-channel echo canceller receives a sound source signal (sound source sound signal) corresponding to each of a plurality of sound sources and executes cancellation processing (multi-cancellation echo cancellation processing). As a result, even if there is a correlation between the reference signals (in other words, if the correlation is greater than a predetermined threshold to the extent that it can be considered that crosstalk components are not included), the collected sound signal for each microphone element of the ambisonics microphone AMB1 It becomes possible to eliminate (eliminate) with high precision the component of the reference signal (that is, the individual sound source sound signal picked up by the individual sound pickup microphone) included in the reference signal.

(Embodiment 3)
In the second embodiment, an example is described in which, in a sound field realistic sensation reproduction device, before performing an encoding process for generating a first-order Ambisonics signal, echo cancellation processing is performed in the time domain using the pickup signal for each microphone element of the Ambisonics microphone AMB1 and a sound source sound signal (reference signal) for each sound source in the sound collection space (for example, live venue LV1). In the third embodiment, an example is described in which, in a sound field realistic sensation reproduction device, echo cancellation processing is performed in the Ambisonics domain using a first-order Ambisonics signal and a direction-specified sound source sound signal for each sound source in the sound collection space (for example, live venue LV1). Note that in the third embodiment, the configuration and contents that overlap with those in the second embodiment are given corresponding common reference symbols, and the description is simplified or omitted, and the different contents are described.

First, with reference to FIGS. 19 and 20, the system configuration and operational outline of the sound field realistic sensation reproduction system 100B according to the third embodiment will be described. FIG. 19 is a block diagram showing an example of a system configuration of a sound field realistic sensation reproduction system 100B according to the third embodiment. FIG. 20 is a diagram illustrating an example of an outline of the operation from sound field presence sound collection to sound field presence reproduction in the sound field presence reproduction system 100B of FIG. 19.

The sound field realism reproduction system 100B includes a sound field realism pickup device 10B and a sound field realism reproduction device 20B. The sound field realism pickup device 10B and the sound field realism reproduction device 20B are connected to each other via a network NW1 so that data communication is possible.

The sound field realism pickup device 10B is placed in a sound pickup space (e.g., a live venue LV1) and includes an Ambisonics microphone AMB1, an A/D conversion unit 7, an encoding unit 8, a microphone element direction designation unit 9, and individual pickup microphones M1, ..., Mn. Note that the sound field realism pickup device 10 only needs to have at least the Ambisonics microphone AMB1, and the A/D conversion unit 7, the encoding unit 8, and the microphone element direction designation unit 9 may be provided in the sound field realism reproduction device 20B.

The sound field presence reproduction device 20B is arranged in a reproduction space (for example, the satellite venue STL1), and includes echo canceling sections 21B, ..., 2nB, encoding sections 31, ..., 3n, and sound source position specifying sections 41, ..., 4n. , a speaker direction designation section 24, a decoding section 25B, a sound field reproduction section 26, and satellite speakers SPk1, . . . , SPkp. Further, n indicates the number of echo canceling sections 21 to 2n, n indicates the number of encoding sections 31 to 3n, n indicates the number of sound source position specifying sections 41 to 4n, and n indicates the number of individual sound collecting microphones M1 to Mn. This is the same as n indicating the number of objects. That is, the same number of echo canceling units, encoding units, and sound source position specifying units as the number of types of sound sources picked up by the individual sound pickup microphones are provided in the sound field presence reproduction device 20B. Note that the configurations of the echo canceling units 21B to 2nB, encoding units 31 to 3n, sound source position specifying units 41 to 4n, speaker direction specifying unit 24, decoding unit 25B, and sound field reproduction unit 26 in FIG. This corresponds to the configuration of the audience seat side realism parameter calculation section 16 and the sound field reproduction processing section 18 according to the first embodiment (see FIG. 1).

Here, each of the echo cancellation units 21B to 2nB may be configured as an echo canceller using an adaptive filter that operates in the Ambisonics domain, for example. This echo canceller can be configured as a Single Channel Echo Canceller, for example. Therefore, as shown in FIG. 20, each of the echo cancellation units 21B to 2nB can be configured using a Single Channel Echo Canceller with the same number of microphone elements as the number of microphone elements of the Ambisonics microphone AMB1 (for example, 4). This Single Channel Echo Canceller may have a configuration disclosed in, for example, a reference non-patent document. By using this configuration, even if there is no correlation between the reference signals (in other words, if it is below a certain threshold at which it can be assumed that there is substantially no crosstalk component), it is possible to highly accurately erase (suppress) the components of the reference signal (i.e., the individual sound source signal picked up by the individual sound pickup microphone) contained in the signal components based on the pickup signal for each microphone element of the Ambisonics microphone AMB1 (for example, a signal having resolution in each of the W, X, Y, and Z directions shown in FIG. 10).

<Reference non-patent literature> Chapter 5 Acoustic echo canceller, "Configuration example of adaptive filter" (see Figures 5 and 2), p4/(17), Institute of Electronics, Information and Communication Engineers, 2012, [September 2, 2020] Search], Internet <URL: https://www. ieice-hbkb. org/files/02/02gun_06hen_05. pdf>

Next, with reference to FIG. 21, an explanation will be given of an operation procedure for reproducing a sound field sense of presence by the sound field sense of presence reproduction apparatus 20B. FIG. 21 is a flowchart chronologically showing an example of an operation procedure for reproducing a sound field sense of presence by the sound field sense of presence reproduction apparatus 20B according to the third embodiment. In the explanation of FIG. 21, the same step numbers are assigned to the contents that overlap with the explanation of FIG. 15 to simplify or omit the explanation, and the different contents will be explained.

As described above, in the sound field presence reproduction system 100B according to the third embodiment, the sound field presence reproduction device 20B uses the sound collection device (ambisonics microphone AMB1) arranged in the sound collection space (live venue LV1). An acquisition section (echo canceling section 21B to 2nB) that acquires at least a collected sound signal (diffuse sound components DS1 to DS3), and a sound source signal of one or more sound sources in the sound collection space (sound signal SS1, audio Encoding units 31 to 3n encode the signal SS2, sound signal SS3), use the encoded sound source signal as a reference signal, and perform an erasing process to erase the reference signal component included in the collected sound signal. Based on the cancellation unit (echo cancellation unit 21B to 2nB) and the signal after cancellation processing, each of the plurality of speakers (satellite speakers SPk1 to SPkp) arranged in a reproduction space (satellite venue STL1) different from the sound collection space A generation unit (decoding unit 25B) that generates a speaker drive signal for reproducing the sound field presence in the sound collection space in the reproduction space, and a speaker drive signal for each speaker from each of the plurality of speakers. A sound field reproduction section 26 for outputting the sound field is provided. As a result, the sound field presence reproduction device 20B according to the third embodiment reproduces the atmosphere of presence on the audience seat side in the sound collection space (live venue LV1) mainly collected by the ambisonics microphone AMB1. By erasing the components of one or more individual sound sources (reference signals) in the live venue LV1, which are picked up by the LV1, in the ambisonics domain rather than in the time domain, the components of at least one individual sound source (reference signal) in the live venue LV1 are eliminated with high directional resolution. can be reproduced with high precision.

In addition, the sound signals acquired by the acquisition unit (echo cancellation units 21B to 2nB) are encoded using the direction information of each of the plurality of microphone elements Mc1 to Mc4 included in the sound collection device (Ambisonics microphone AMB1). This is the signal that was sent. Thereby, the sound field presence reproduction device 20B can obtain a first-order ambisonics signal having a high directional resolution as a signal to be input as a target of cancellation processing in the cancellation section (each of the echo cancellation sections 21B to 2nB).

Furthermore, the sound field presence reproduction device 20B further includes a speaker direction designation unit 24 that designates direction information of a plurality of speakers (satellite speakers SPk1 to SPkp) in the reproduction space (satellite venue STL1). The generation unit (decoding unit 25B) generates a speaker drive signal for each of the plurality of speakers using the direction information of each of the plurality of speakers and the signal after the erasure processing. As a result, the sound field presence reproduction device 20B performs decoding processing using the signal after the erasure processing in the ambisonics region, so that the reference position of each of the plurality of satellite speakers SPk1 to SPkp (for example, It is possible to reproduce the sense of presence on the audience seat side in the live venue LV1 by taking into account the direction information from the center position LSP1 (see center position LSP1 corresponding to the position of LV1), and to generate a speaker drive signal with high precision with high directional resolution.

In addition, the sound field presence reproduction device 20B further includes sound source position designating units 41 to 4n that designate position information of one or more sound sources in the sound collection space (live venue LV1). Each of the encoding units 31 to 3n executes encoding processing using the sound source signal and position information of the corresponding sound source. Thereby, the sound field presence reproduction device 20B takes into account the direction in which the individual sound sources in the live venue LV1 exist and generates a highly accurate reference signal that is necessary for the cancellation process of the cancellation section (echo cancellation section 21B to 2nB). Can be generated.

Further, the canceling section (echo canceling section 21B to 2nB) is determined based on the number of microphone elements Mc1 to Mc4 (for example, 4) and the number of sound sources (for example, n pieces) included in the sound collection device (Ambisonics microphone AMB1). (for example, 4n (=4×n)) single echo cancellers. The single echo canceller inputs a signal after the sound source signal of the corresponding sound source has been encoded and performs cancellation processing. As a result, the sound field presence reproduction device 20B is able to control the sound field, especially when there is no correlation between the reference signals (in other words, when the crosstalk component is less than a predetermined threshold that can be considered to be substantially free of crosstalk components). , a reference signal (that is, an individual sound collection microphone) included in a signal component (for example, a signal having resolution in each of the W, X, Y, and Z directions shown in FIG. 10) based on a sound pickup signal for each microphone element of the ambisonics microphone AMB1. It becomes possible to highly accurately eliminate (suppress) the component of the first-order ambisonics signal based on the individual sound source sound signal collected by Mn and the direction of the individual sound source.

(Modification of the third embodiment)
In the third embodiment, an example is described in which each of the echo cancellation units 21B to 2nB is configured as a single echo canceller in the sound field realistic reproduction device. In a modified version of the third embodiment, an example is described in which the echo cancellation units 21B to 2nB are configured as a multi-channel echo canceller that handles multiple audio channels in the Ambisonics domain instead of the time domain in the sound field realistic reproduction device. Note that in the modified version of the third embodiment, the configurations and contents that overlap with those of the first and second embodiments are given the corresponding common reference numerals, and the description is simplified or omitted, and only the different contents are described.

First, with reference to FIGS. 22 and 23, the system configuration and operational outline of a sound field realistic sensation reproduction system 100C according to a modification of the third embodiment will be described. FIG. 22 is a block diagram showing a system configuration example of a sound field realistic sensation reproduction system 100C according to a modification of the third embodiment. FIG. 23 is a diagram illustrating an example of an outline of the operation from sound field presence sound pickup to sound field presence reproduction in the sound field presence reproduction system 100C of FIG. 22.

The sound field presence reproduction system 100C includes a sound field presence sound pickup device 10B (see FIG. 19) and a sound field presence reproduction device 20C. The sound field presence sound pickup device 10B and the sound field presence reproduction device 20C are connected to each other via a network NW1 so as to be capable of data communication.

Here, the multichannel echo canceling unit 21C may be configured as an echo canceller using a plurality of adaptive filters that operate in the ambisonics region, for example. This echo canceller can be configured as a Multi Channel EchoCanceller, for example. Note that for the configuration of this Multi Channel EchoCanceller, the stereo echo canceler shown in the reference non-patent document may be referred to. An example. Therefore, as shown in FIG. 23, the multi-channel echo canceling section 21C can be configured using the same number of Multi Channel Echo Cancellers or stereo echo cancellers as the number of microphone elements (for example, 4) of the ambisonics microphone AMB1. This Multi Channel Echo Canceller or stereo echo canceller may be, for example, a configuration disclosed in a referenced non-patent document or a configuration obtained by referring to the configuration. By using this configuration, even if there is a correlation between the reference signals (in other words, if the correlation is greater than or equal to a predetermined threshold that allows it to be considered that crosstalk components are not included), the sound picked up by the ambisonics microphone AMB1 can be The component of the reference signal (that is, the individual sound source sound signal picked up by the individual sound collection microphone Mn) included in the signal component based on It becomes possible to erase (suppress) with high precision.

<Reference non-patent literature> Chapter 5 Acoustic echo canceller, "Configuration example of stereo echo canceller" (see Figures 5 and 8), Institute of Electronics, Information and Communication Engineers, 2012, [Retrieved September 2, 2020], Internet < URL: https://www. ieice-hbkb. org/files/02/02gun_06hen_05. pdf>

Next, with reference to FIG. 24, a description will be given of an operation procedure for reproducing a sound field sense of presence by the sound field sense of presence reproduction apparatus 20C. FIG. 24 is a flowchart chronologically showing an example of an operation procedure for reproducing a sound field sense of presence by the sound field sense of presence reproduction apparatus 20C according to a modification of the third embodiment. In the description of FIG. 24, the same step numbers will be given to the same steps as those in the description of FIG. 15, FIG. 18, or FIG. 21, and the description will be simplified or omitted, and the different contents will be explained.

As described above, in the sound field reality reproduction device 20C according to the modified example of the third embodiment, the canceling section (multichannel echo canceling section 21C) is configured to remove the microphone elements Mc1 to Mc4 included in the sound collection device (ambisonics microphone AMB1). It is constituted by the number of multichannel echo cancellers determined based on the number of echo cancellers. The multi-channel echo canceller inputs a signal obtained by encoding the sound source signal corresponding to each of a plurality of sound sources and executes cancellation processing (multi-channel echo cancellation processing). As a result, the sound field presence reproduction device 20C can use the ambisonics microphone even if there is a correlation between the reference signals (in other words, if the correlation is greater than or equal to a predetermined threshold at which it can be considered that crosstalk components are not included). A reference signal (that is, a reference signal included in a signal component based on a sound pickup signal for each microphone element of AMB1 (for example, a signal having resolution in each of the W, X, Y, and Z directions shown in FIG. 10) This makes it possible to eliminate (suppress) components of the individual sound source sound signal with high precision in the ambisonics region.

Although the embodiments have been described above with reference to the attached drawings, the present disclosure is not limited to such examples. It is clear that a person skilled in the art can conceive of various modifications, corrections, substitutions, additions, deletions, and equivalents within the scope of the claims, and it is understood that these also fall within the technical scope of the present disclosure. Furthermore, the components in the above-described embodiments may be combined in any manner as long as it does not deviate from the spirit of the invention.

In the above-described first embodiment, the realistic sound field reproducing device reproduces sound in real time using the performer's sound signal, the zone sound signal, and the audience seat side realistic sound signal collected (recorded) in the live venue LV1. An example has been described in which sound image localization and realistic reproduction are performed within the satellite venue STL1. However, the realistic sound field reproducing device stores in advance the performer's sound signal, the zone sound signal, and the audience seat side realistic sound signal recorded in advance at the live venue LV1 in the storage unit 13 in the satellite venue STL1, and real-time Instead, the atmosphere may be reproduced (sound image localization, realistic reproduction) using the sound field of the live venue LV1 at a later date than the recording date.

In the system configuration example shown in FIG. 1, at least one of the memory unit 13, sound image localization zone determination unit 15, audience seating side realism parameter calculation unit 16, and mixing/level balance adjustment unit 17 may be provided on the live venue LV side. In other words, the layout of the system configuration of the realism sound field reproduction system 1000 shown in FIG. 1 may be appropriately determined taking into consideration the performance of the computer device P1 provided on the live venue LV1 side and the performance of the computer device P2 provided on the satellite venue STL1 side.

INDUSTRIAL APPLICABILITY The present disclosure is useful as a realistic sound field reproducing device and a realistic sound field reproducing method that highly sensitively reproduces a sense of presence including sound images from various sound sources in a sound field recording space in a sound field reproduction space.

1 Mixer 2 Performer tracking system 3 Face database 4 Encoder 5 Matching table database 6 Live venue side communication section 7 A/D conversion section 8, 22, 31, 3n Encoding section 9, 23 Microphone element direction specifying section 10, 10B Sound field Presence sound pickup device 11 Satellite venue side communication section 12 Decoder 13 Storage section 14 Video output section 15 Sound image localization zone determination section 16 Audience seat side presence feeling parameter calculation section 17 Mixing/level balance adjustment section 18 Sound field reproduction processing section 20, 20A , 20B, 20C Sound field presence reproduction device 21, 2n, 21B, 2nB Echo cancellation unit 21A, 21C Multi-channel echo cancellation unit 24 Speaker direction designation unit 25, 25B, 25C Decoding unit 26 Sound field reproduction unit 41, 4n Sound source Position specifying section 100, 100A, 100B, 100C Sound field realistic reproduction system 1000 Realistic sound field reproduction system AMB1 Ambisonics microphone HM1, HM4 Headset microphone M1, Mn Individual sound collection microphone SPk1, SPk2, SPk3, SPk4, SPk5, SPk6, SPk7, SPk8, SPk9, SPk10, SPk11, SPk12, SPkp Satellite speaker ZM1, ZM2, ZM3, ZM4, ZM5, ZM6, ZM7 Zone microphone

Claims (11)

A speech audio signal recorded by a person microphone worn by at least one person who can move in an activity area in a sound field recording space, and an ambient sound signal recorded by a plurality of peripheral microphones arranged around the activity area. , and an acquisition unit that acquires at least the location information of the person;
a determining unit that determines a main peripheral microphone, which is one of the plurality of peripheral microphones and whose recording area is a location where the person is located within the activity area, based on position information of the person;
Based on at least the uttered audio signal from the person microphone, a first surrounding sound signal from the main surrounding microphone, and a second surrounding sound signal from the surrounding microphone other than the main surrounding microphone, the sound field recording space is a sound field reproduction unit that executes a sound field reproduction process for reproducing the sound field in the sound field recording space using a plurality of speakers arranged in a sound field reproduction space different from the sound field reproduction space;
Realistic sound field reproduction device. The sound field reproduction unit reproduces the sound field so as to reproduce the sound field in the sound field recording space by emphasizing the first peripheral sound signal more than the second peripheral sound signal in the sound field reproduction space. execute the process,
The realistic sound field reproducing device according to claim 1. The acquisition unit further acquires a peripheral area sound signal recorded by a sound recording device located in a peripheral area different from the activity area in the sound field recording space,
The sound field reproduction unit executes signal processing for reproducing a sound field based on the peripheral area sound signal in an area in the sound field reproduction space corresponding to the peripheral area in the sound field recording space.
The realistic sound field reproducing device according to claim 1. The sound field reproduction unit mixes the uttered audio signal, the first ambient sound signal, and the second ambient sound signal, and performs signal level balance adjustment, and outputs the mixture through each of the plurality of speakers.
The realistic sound field reproducing device according to claim 1. The sound field reproduction unit mixes the uttered audio signal, the first ambient sound signal, the second ambient sound signal, and the ambient area sound signal and adjusts the balance of signal levels, and controls each of the plurality of speakers. output via,
The realistic sound field reproducing device according to claim 3. The acquisition unit repeatedly acquires position information of the person that can be updated periodically,
The determining unit determines the main peripheral microphone each time the position information of the person is updated.
The realistic sound field reproducing device according to claim 1. the person is plural;
The determination unit determines the main peripheral microphone for each person based on position information of the person.
The realistic sound field reproducing device according to claim 1. A speech audio signal recorded by a person microphone worn by at least one person who can move in an activity area in a sound field recording space, and an ambient sound signal recorded by a plurality of peripheral microphones arranged around the activity area. , and an acquisition unit that acquires at least the location information of the person;
a determining unit that determines a main peripheral microphone, which is one of the plurality of peripheral microphones and whose recording area is a location where the person is located within the activity area, based on position information of the person;
Based on at least the uttered audio signal from the person microphone, a first surrounding sound signal from the main surrounding microphone, and a second surrounding sound signal from the surrounding microphone other than the main surrounding microphone, the sound field recording space is a sound field reproduction unit that executes a sound field reproduction process for reproducing the sound field in the sound field recording space using a plurality of speakers arranged in a sound field reproduction space different from the sound field reproduction space,
The acquisition unit further acquires a peripheral area sound signal recorded by a sound recording device located in a peripheral area different from the activity area in the sound field recording space,
The sound field reproduction section is
performing signal processing for reproducing a sound field based on the surrounding area sound signal in an area in the sound field reproduction space that corresponds to the surrounding area in the sound field recording space;
using the speech audio signal, the first ambient sound signal, and the second ambient sound signal as reference signals, and performing an erasure process as the signal processing for erasing a component of the reference signal included in the ambient area sound signal;
Realistic sound field reproduction device. The sound field reproduction section encodes the signal after the erasure processing, and reproduces the sense of presence of the sound field in the sound field recording space for each of the plurality of speakers based on the signal after the encoding processing. Generating a speaker drive signal for reproduction in a sound field reproduction space,
The realistic sound field reproducing device according to claim 8. The sound field reproduction unit outputs the generated speaker drive signal for each of the speakers from the corresponding speaker.
The realistic sound field reproducing device according to claim 9. A method for reproducing a realistic sound field using a realistic sound field reproducing device, the method comprising:
A speech audio signal recorded by a person microphone worn by at least one person who can move in an activity area in a sound field recording space, and an ambient sound signal recorded by a plurality of peripheral microphones arranged around the activity area. , and obtaining at least the location information of the person;
determining a main peripheral microphone, which is one of the plurality of peripheral microphones and whose recording area is a location where the person is located within the activity area, based on the position information of the person;
Based on at least the uttered audio signal from the person microphone, a first surrounding sound signal from the main surrounding microphone, and a second surrounding sound signal from the surrounding microphone other than the main surrounding microphone, the sound field recording space is executing a sound field reproduction process for reproducing the sound field in the sound field recording space using a plurality of speakers arranged in a sound field reproduction space different from the sound field reproduction space,
A realistic sound field reproduction method.

Images