JP2018074252A - Acoustic system and control method of same, signal generating device, computer program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technique enabling efficient processing in a configuration in which sound is acquired from a plurality of areas into which a space is divided to generate a reproduction signal.SOLUTION: The acoustic system includes: a microphone array for collecting a sound; separation means for separating the sound collected by the microphone array into sounds in a plurality of separation areas obtained by dividing a certain space; control means for controlling division into a plurality of separation areas of the certain space; and generating means for generating a reproduction signal based on the separated sounds.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an acoustic system, a control method thereof, a signal generation device, and a computer program.

  A technique is known in which a space is divided into a plurality of areas and audio for each area is acquired (Patent Document 1).

JP 2014-72708 A

  However, if the voice of the area divided into a plurality of areas is processed in real time and broadcasted, the processing and transmission may not be in time, data may be lost, and the voice may be interrupted.

  The present invention has been made in view of the above problems, and an object of the present invention is to provide a technique that enables efficient processing in a configuration in which audio is obtained from a plurality of areas divided into spaces and a reproduction signal is generated. And

In order to achieve the above object, an acoustic system according to the present invention comprises the following arrangement. That is,
A microphone array that collects audio,
Separation means for separating the sound collected by the microphone array into sound in a plurality of separation areas into which a certain space is divided;
Control means for controlling division of the constant space into the plurality of separation areas;
Generating means for generating a reproduction signal based on the separated sound.

  According to the present invention, it is possible to improve processing efficiency in a configuration in which sound is obtained from a plurality of areas divided into spaces and a reproduction signal is generated.

The block diagram which shows the structure of an audio | voice signal processing apparatus. Explanatory drawing of isolation | separation area control. Explanatory drawing showing the time change of isolation | separation area control. The block diagram which shows the hardware constitutions of an audio | voice signal processing apparatus. The flowchart which shows an audio | voice signal process. 6A and 6B illustrate a display device for separation area control. The figure explaining an acoustic system. The block diagram which shows the detail of a structure of an acoustic system. Explanatory drawing of isolation | separation area control. The flowchart which shows an audio | voice signal process.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. The following embodiments do not limit the present invention, and all the combinations of features described in the present embodiment are not necessarily essential to the solution means of the present invention. In addition, about the same structure, the same code | symbol is attached | subjected and demonstrated.

<Embodiment 1>
In the first embodiment (Embodiment 1) of the present invention, a configuration for reducing the number of separation areas to be used when the sound source separation processing is not in time for real-time reproduction will be described.

(Audio signal processor)
FIG. 1 is a block diagram showing the configuration of the audio signal processing apparatus 100. The audio signal processing apparatus 100 collects audio from a predetermined spatial area by a microphone array, separates the collected audio into a plurality of audio signals based on a plurality of separation areas, performs audio processing, performs mixing, and reproduces signals. Is a device that generates The audio signal processing device 100 includes a microphone array 111, a sound source separation unit 112, a separation area control unit 113, an audio signal processing unit 114, a storage unit 115, a real-time reproduction signal generation unit 116, and a replay reproduction signal generation unit 117. .

  The microphone array 111 is composed of a plurality of microphones. The microphone array 111 collects sound of the space in charge with a microphone. Since each microphone constituting the microphone array 111 collects sound, the sound collected by the microphone array 111 is a multi-channel signal composed of a plurality of sounds collected by each microphone as a whole. The microphone array 111 picks up the sound of the space with a microphone, performs A / D conversion (analog / digital conversion) on the picked up signal, and outputs the signal to the sound source separation unit 112.

  The sound source separation unit 112, the separation area control unit 113, the audio signal processing unit 114, the real-time reproduction signal generation unit 116, and the replay reproduction signal generation unit 117 are, for example, arithmetic processing devices such as a CPU (Central Processing Unit), a DSP, and an MPU. Consists of. DSP is an abbreviation for Digital Signal Processor, and MPU is an abbreviation for Micro-processing unit.

  The sound source separation unit 112 divides a signal input from the microphone array 111 into each of N (N> 1) areas (hereinafter, “separation areas”) when the space in which the microphone array 111 is in charge of sound collection is divided. Performs sound source separation processing to separate the sound in the separation area. As described above, the signal input from the microphone array 111 is a multi-channel signal composed of a plurality of sounds collected by each microphone. Therefore, based on the positional relationship between the microphones constituting the microphone array 111 and the separation area to be collected, the sound signals collected by the microphones are phase-controlled and weighted and added, so that the sound of an arbitrary separation area can be added. Can be reproduced. In this embodiment, an example in which the arrangement of the separation areas is predetermined will be described. The sound source separation unit 112 performs sound source separation processing so as to divide the space into N (N> 1) areas using the signal input from the microphone array 111. The separation process is performed every processing frame, that is, every predetermined time interval. For example, the beam forming process is performed every predetermined time, and the sound for each area is acquired. The separated and acquired sound is output to the sound signal processing unit 114 and the storage unit 115.

  The separation area control unit 113 controls the division into a plurality of separation areas in a certain space where the microphone array collects sound according to the processing load for performing sound source separation, reproduction signal generation, and the like. Specifically, the arrangement and the number of separation areas are controlled. For example, if the processing load of the processing device is large and the sound source separation processing for all areas is not in time for real-time playback, the separation area control unit 113 combines the sound source separation areas performed by the sound source separation unit 112 to determine the number of areas. Reduce. For example, in a state where the processing is sufficiently in time, for example, as shown in FIG. 2A, the sound collection space A1 is finely divided into 8 × 8 = 64 separation areas A2. When the processing is not in time, for example, it is determined whether or not the sound of the area is equal to or higher than a predetermined level in the processing of the previous frame, and areas less than the predetermined level are shown in FIG. To reduce the number of areas. While the voice above the predetermined level has a high probability of being a significant voice, the voice below the predetermined level has a high probability of being a non-significant voice such as noise. For this reason, a fine separation area is preferentially assigned to areas where the audio is above a predetermined level, so that significant audio can be faithfully reproduced, and in areas below the predetermined level, the separation area is integrated to speed up processing. Can be

  An example of the change in the separation size of the area is shown in FIG. FIG. 3D shows a state in which area control based on the processing load is being performed (area control ON) or not (area control OFF). fp to fp + 7 represent frame numbers. FIG. 3C shows a state where the sound level separated for each area is equal to or higher than a predetermined level (with sound) or lower than a predetermined level (no sound). Here, sound is present in frames fp + 1 and fp + 3. FIG. 3B shows the division size of the most finely divided area. This division size represents the area of the minimum area when the area of the sound collection space A1 is 1. For example, in the frame fp, since the space is equally divided into 64 areas, the minimum area size is 1/64. FIG. 3A shows a state in which each frame is separated into a plurality of areas.

  The time from fp + 1 to fp + 6 is the time when the processing load is large and the number of areas needs to be reduced. In the frame fp, the sound level did not exceed a predetermined value in any area (no sound in FIG. 3C). Therefore, in the frame fp + 1, the area size is a large area in which one side is 1/2 of the sound collection space and the sound collection space is divided into four (1/4 in FIG. 3B).

  In the frame fp + 1, there was an area where the sound level exceeded a predetermined value (with sound in FIG. 3C). For this reason, in the frame fp + 2, the area A3 where the voice is present is again divided into areas each having a side that is 1/8 smaller than the sound collection space A1 (1/64 in FIG. 3B).

  Subsequently, in the frame fp + 2, the sound level did not exceed the predetermined value in any area (no sound in FIG. 3C). Therefore, in the frame fp + 3, some areas are combined and one side is divided into an area having an intermediate size of 1/4 of the sound collection space (1/16 in FIG. 3B).

  In the frame fp + 3, there was an area where the sound level exceeded a predetermined value (with sound in FIG. 3C). For this reason, in the frame fp + 4, the area A3 where the sound is present is again divided into areas whose one side is 1/8 as small as the sound collection space (1/64 in FIG. 3B).

  In frames fp + 4 and fp + 5, the sound level did not exceed a predetermined value in any area (no sound in FIG. 3C). Therefore, the areas are combined, and in the frame fp + 6, one side is a half of the sound collection space and the sound collection space is divided into four.

  In this way, the separation area control unit 113 increases or decreases the number of separation areas according to the presence or absence of voice detection. Here, the example in which the separation area control unit 113 combines the sound source separation areas to reduce the number of areas has been described. Of course, the sound source separation unit 112 may have a filter for beam forming that separates into a plurality of area sizes, and the separation area control unit 113 may control the filter to be used.

  Further, the separation area control unit 113 manages the area information combined with the frame for the areas combined by the separation area control as a separation area control list. For example, when four areas are combined in the frame fq, the frame fq and the four areas are managed as a list. Here, the areas are previously identified with an ID or the like. The separation area control unit 113 instructs the sound source separation unit 112 to perform sound source separation of each area for the areas combined with the frames recorded in the separation area control list in response to a reduction in processing load. put out. When sound source separation is performed, the frame and area are deleted from the list.

  The audio signal processing unit 114 performs audio signal processing for each frame and area. The processing performed by the audio signal processing unit 114 is, for example, delay correction processing for correcting the influence of the distance between the area and the sound collection device, gain correction processing, echo removal, and the like.

  The storage unit 115 is a storage device such as an HDD (Hard Disk Drive), an SSD (Solid State Drive), or a memory. The storage unit 115 records the signals of all the audio channels of the frames subjected to the separation area control in the sound source separation unit 112 and the signals subjected to the audio signal processing in the audio signal processing unit 114 together with time information.

  The real-time playback signal generation unit 116 generates and outputs a signal for real-time playback by mixing the sound for each area obtained from the sound source separation unit 112 within a predetermined time from the collected sound. For example, a virtual listening point in a space that changes according to time, the direction of the virtual listener (hereinafter simply referred to as the listening point and the listener's direction), and reproduction environment information are acquired from the outside, and the sound source Mixing. Here, the playback environment is an environment related to the configuration of the playback device, such as whether the playback device that plays back the signal generated by the real-time playback signal generation unit 116 is a speaker (stereo, surround, other multi-channel) or headphones. . That is, in sound source mixing, a process of synthesizing and converting the audio signal of each divided area according to the environment such as the number of channels of the playback device is performed.

  When replay playback is requested, the replay playback signal generation unit 117 acquires data at the corresponding time from the storage unit 115, performs the same processing as the real time playback signal generation unit 116, and outputs the data.

  FIG. 4 is a block diagram illustrating a hardware configuration example of the audio signal processing apparatus 100. The audio signal processing apparatus 100 is realized by, for example, a personal computer (PC), an embedded system, a tablet terminal, a smartphone, or the like.

  In FIG. 4, a CPU 990 is a central processing unit, and controls the operation of the entire audio signal processing apparatus 100 in cooperation with other components based on a computer program. The ROM 991 is a read-only memory and stores basic programs, data used for basic processing, and the like. A RAM 992 is a writable memory and functions as a work area for the CPU 990.

  The external storage drive 993 can access a recording medium, and can load a computer program and data stored in a medium (recording medium) 994 such as a USB memory into this system. The storage 995 is a device that functions as a large-capacity memory such as an SSD (solid state drive). The storage 995 stores various computer programs and data.

  The operation unit 996 is a device that receives an instruction and a command input from a user, and corresponds to a keyboard, a pointing device, a touch panel, and the like. The display 997 is a display device that displays a command input from the operation unit 996, a response output of the audio signal processing device 100 with respect to the command, and the like. An interface (I / F) 998 is a device that relays data exchange with an external device. The microphone array 111 is connected to the audio signal processing apparatus 100 via the interface 998. The system bus 999 is a data bus that manages the flow of data in the audio signal processing apparatus 100.

  Each functional element in FIG. 1 is realized by the CPU 990 controlling the entire apparatus based on a computer program. In addition, it can also be comprised as an alternative of a hardware apparatus by the software which implement | achieves a function equivalent to the above each apparatus.

(Processing procedure)
Next, the procedure of processing executed by the audio signal processing apparatus 100 will be described with reference to FIG. FIG. 5A to FIG. 5C are flowcharts showing a procedure of processing executed by the audio signal processing apparatus 100 of the present embodiment.

  FIG. 5A is a flow from collecting sound to generating a real-time reproduction signal. First, sound in the space is collected in the microphone array 111 (S111). The collected audio signal of each channel is output to the sound source separation unit 112.

  Subsequently, the separation area control unit 113 determines whether the sound source separation is in time for real-time reproduction from the viewpoint of processing load (S112). As described with reference to FIG. 3, this process is performed based on the presence or absence of a predetermined level of sound.

  When it is determined that it is not in time for real-time reproduction (NO in S112), the separation area control unit 113 controls the number of areas so that the sound source separation area is reduced (S113). Specifically, for example, the number of separation areas is reduced by integrating separation areas with low importance such as areas in which sound of a certain level or higher is not detected. Then, information on which area is to be separated is output to the sound source separation unit 112. Further, the separation area control unit 113 creates a separation area control list.

  Subsequently, the audio signal of the frame subjected to the separation area control is recorded in the storage unit 115 (S114).

  When it is determined that it is in time for real-time reproduction, or after recording in S114, the sound source separation unit 112 performs sound source separation (S115). That is, the sound in each separation area is synthesized based on the multi-channel signal collected in S111. As described above, the sound of the separation area is added by performing phase control and weighting on the sound signal collected by each microphone based on the relationship between the microphones constituting the microphone array 111 and the position of the separation area. Can be reproduced. The separated audio signal for each area is output to the audio signal processing unit 114.

  Subsequently, the audio signal processing unit 114 processes the audio signal for each separation area (S116). As described above, the processing by the audio signal processing unit 114 is, for example, delay correction processing for correcting the influence of the distance between the separation area and the sound collection device, gain correction processing, noise processing by echo removal, and the like. The processed audio signal is output to the real-time reproduction signal generation unit 116 and the storage unit 115.

  Subsequently, the real-time playback signal generator 116 mixes the audio for real-time playback (S117). In mixing, signals are synthesized and converted so that playback can be performed in accordance with the specifications of the playback device (for example, the number of channels). Audio mixed for real-time playback is output as an external playback device or broadcast signal.

  Subsequently, the voice of each area is recorded in the storage unit 115 (S118). The audio signal for replay reproduction is created using the audio for each area of the storage unit 115.

  Next, referring to FIG. 5B, a description will be given of processing in the case where processing is not in time for real-time playback in S112 of FIG. 5A (NO in S112).

  The separation area control unit 113 reads data from the storage unit 115 based on the separation area control list when the load on the processing apparatus is lower than a predetermined value (S121).

  Subsequently, the sound source separation process is performed again on the area before combining the areas where the sound source separation is performed by combining the areas described in the separation area control list (S122). The processed audio signal is output to the audio signal processing unit 114. The corresponding frame and area are deleted from the separation area control list when the processing is completed. Since S123 is similar to S116, detailed description thereof is omitted.

  Subsequently, the storage unit 115 overwrites and records the audio signal of the input area on the previous data (S124).

  Next, a processing flow when replay is requested will be described with reference to FIG. When a replay is requested, the replay playback signal generation unit 117 reads out an audio signal for each area corresponding to the replay time from the storage unit 115 (S131).

  Subsequently, the replay playback signal generator 117 mixes the replay playback audio (S132). Audio mixed for replay playback is output as an external playback device or broadcast signal.

  As described above, the separation area is controlled according to the processing load. That is, in a certain space, control is performed so that a region where the processing load of at least one of sound source separation and reproduction signal generation is larger is divided into finer separation areas. For this reason, the degree of separation of the area where the sound volume level is lower than the predetermined value is lowered, but the area where the sound volume level is higher than the predetermined value is in time for generating the real-time reproduction signal with high resolution. Furthermore, when the processing load is light, the separation of the areas controlled by the separation area is performed, so that data with sufficient resolution can be obtained during replay.

  In the present embodiment, the microphone array 111 has been described as an example of a microphone, but may be a set with a structure such as a reflector. Further, the microphone used in the microphone array 111 may be non-directional, may be a directional microphone, or a mixture thereof.

  In the present embodiment, the sound source separation unit 112 has described an example in which sound collection for each area is performed using beamforming, but other sound source separation may be used. For example, the power spectral density (PSD) for each area may be estimated, and separation by a Wiener filter may be performed based on the estimated PSD.

  In the present embodiment, the example in which the separation area control unit 113 controls the separation area based on whether or not the audio level of the area is equal to or higher than a predetermined value has been described, but other determination criteria may be used. For example, even when the same voice is used, it is possible to provide a configuration for detecting a feature amount of a sound instead of a level and determine the presence or absence of the feature amount. Specifically, when the voice that shows a predetermined feature is detected, such as when screams, gunshots, ball sounds, car sounds, etc. are detected by voice feature analysis, the separation area is reduced. Thus, detailed audio may be reproduced. Further, for example, a space including all areas may be photographed, and the separation area may be controlled from the photographed moving image. For example, a specific subject such as a person, an animal, or a marker may be detected from the moving image, and control may be performed so that the size of the separation area around the subject becomes smaller.

  In live broadcasts such as television broadcasting, a system is generally known that broadcasts with a certain delay of several seconds to several minutes from actual shooting in order to adjust the time or cope with an unexpected situation. When such a system is used, the separation area control unit 113 may control the separation order according to events included in video and audio for a delay time. For example, when there is a delay of 2 minutes in a live sports broadcast, a sound source separation may be performed by setting a separation area from a game development of 2 minutes. For example, when a goal is decided in a game such as soccer, the movement of the player or the ball who decided the goal from the video for 2 minutes is detected, and the separation area around the trajectory may be set to be fine. . On the other hand, it is preferable that the separation area is set to be rough for areas where players and balls do not enter.

  In the present embodiment, the separation area control unit 113 reduces the number of areas as much as possible, but may calculate the number of areas according to the processing load to reduce the minimum number of areas.

  In this embodiment, the separation area control unit 113 controls the separation area using the audio level of the previous frame. However, the separation area may be controlled using information of the processing frame. That is, if the sound level of the separated area is greater than or equal to a predetermined value, the separation area control unit 113 instructs the sound source separation unit 112 to perform sound source separation in an area obtained by further dividing the area. The separation area control unit 113 and the sound source separation unit 112 repeat this process until the area is reduced to a predetermined size. In this way, the separation area control can be prevented from being delayed by one frame. However, this method increases the amount of processing as the number of sound sources increases, so it should be used in a scene where the number of sound sources is known to be small in advance, or the number of repetitions should be limited within the allowable processing load range. .

  In the present embodiment, the audio signal processing unit 114 performs the delay correction process, the gain correction process, and the echo removal, but other processes may also be performed. For example, noise removal processing for each area may be performed.

  In the present embodiment, the example in which the replay reproduction signal generation unit 117 and the real time reproduction signal generation unit 116 perform the same processing has been described. However, the replay playback signal generation unit 117 and the real time playback signal generation unit 116 may perform different mixing. For example, in the real-time playback signal generation unit 116, a voice with a rough separation area may be input, so that, for example, an area having a large area size is reduced during mixing depending on whether or not the processing has been performed. May be.

  Although not shown in the present embodiment, as shown in FIG. 6, display control for displaying the status of area control on the display device may be performed. For example, a time bar 501 and a time cursor 502, an area division display 503, an area division ratio display 504, and the like are displayed on the display screen. Here, the time bar 501 is a bar representing the recording time up to the present, and the position of the time cursor 502 represents the time on the display screen. The area division display 503 shows the area division state at the time indicated by the time cursor 502. The image indicating the division state may be displayed by being superimposed on an actual space image, a CG reproducing the actual space, or the like. The area division ratio display 504 displays a ratio for each area division size. Alternatively, a screen as shown in FIG. 3 may be displayed. By displaying in this way, it is possible to intuitively understand the state of area division. The display device may further include an input device such as a touch panel. For example, the user may select an area with a large area size by touch or the like, and may be configured to preferentially perform the process of finely dividing the area.

<Embodiment 2>
The second embodiment (Embodiment 2) of the present invention relates to an acoustic system in which a plurality of users each set a listening point and a sound corresponding to the listening point is played back by a playback device.

(Acoustic system)
FIG. 7 is a block diagram showing the configuration of the acoustic system 20. The acoustic system 20 includes a sound collection unit 21, a reproduction signal generation unit 22, and a plurality of reproduction units 23. The sound collection unit 21, the reproduction signal generation unit 22, and the plurality of reproduction units 23 perform data transmission / reception via a wired or wireless transmission path. A transmission path between the sound collection unit 21, the reproduction signal generation unit 22, and the reproduction unit 23 is realized by a dedicated communication path such as a LAN, but may also pass through a public communication network such as the Internet.

  8A is a block diagram showing the configuration of the sound collection unit 21, FIG. 8B is a block diagram showing the configuration of the reproduction signal generation unit 22, and FIG. 8B is a block diagram showing the configuration of the reproduction unit 23. is there. The sound collection unit 21 in FIG. 8A includes a microphone array 111 and a sound collection signal transmission unit 211. Since the microphone array 111 is the same as that of the first embodiment, detailed description thereof is omitted. The collected sound signal transmission unit 211 transmits the microphone signal input from the microphone array 111.

  8B includes a sound source separation unit 112, a separation area control unit 113, an audio signal processing unit 114, a storage unit 115, a collected sound signal reception unit 221, a listening point reception unit 222, and a reproduction signal. A generation unit 223 and a reproduction signal transmission unit 224 are provided. Since the sound source separation unit 112, the audio signal processing unit 114, and the storage unit 115 are substantially the same as those in the first embodiment, detailed description thereof is omitted.

  The separation area control unit 113 controls an area in which the sound source separation unit 112 performs sound source separation based on a plurality of listening points input from a listening point reception unit 222 described later. Here, the listening point is information including the position and orientation of the virtual listener in the space set by the user, and the time. For example, the separation area control unit 113 monitors the processing load of the reproduction signal generation unit 22 and controls the area so as to reduce the number of separation areas based on the distribution of listening points when the load increases. For example, it is assumed that the positions of listeners set by users who are listening in real time are distributed as shown in FIG. In that case, as shown in FIG. 9B, the area is controlled so that the periphery of the area where more listening points are set is finely divided, and the area with few listening points is roughly divided.

  Also, when the user has specified listening points at a past time, that is, when replay is requested, it is determined whether sound source separation processing is necessary based on the situation of the separation area at the time and the specified viewpoint If necessary, sound source separation is performed according to the processing load. For example, if area control is not performed at the specified time, or if area control is performed but the sound source is separated in a sufficiently fine area around the listening point specified this time, it is necessary to perform separation again Absent. On the other hand, when the area control is performed at the designated time and the area around the listening point designated this time is rough, the separation area control unit 113 performs sound source separation so as to make the area division around the listening point fine. A control signal is output to the unit 112.

  The collected sound signal receiving unit 221 receives the collected sound signal from the sound collecting unit 21. The listening point receiving unit 222 receives a listening point from each of the plurality of reproducing units 23. The received listening points are output to the separation area control unit 113 and the reproduction signal generation unit 223. The reproduction signal generation unit 223 has a function of combining the real-time reproduction signal generation unit 116 and the replay reproduction signal generation unit 117 of the first embodiment. A reproduction signal is generated according to the position and orientation of the listener input from the listening point receiver 222 and the time. If the input time is real time, it is the same as the real time reproduction signal generation unit 116, and if the input time is past, it is the same as the replay reproduction signal generation unit 117. The audio signal generated for each listening point is output to the reproduction signal transmission unit 224. The reproduction signal transmission unit 224 outputs the received audio signal for each listening point to each reproduction unit 23.

  The playback unit 23 in FIG. 8C includes a listening point input unit 231, a listening point transmission unit 232, a playback signal reception unit 233, and a speaker 234. The listening point input unit 231 is an input device that can set the time and the position of the virtual listener in the space where the user is collecting sound and the orientation of the listener. The listening point input unit 231 is realized by a keyboard, a pointing device, a touch panel, or the like. The set listening point is output to the listening point transmission unit 232.

  The listening point transmitting unit 232 outputs the listening point set by the user to the listening point receiving unit 222. The reproduction signal receiving unit 233 receives an audio signal corresponding to the listening point set by the listening point input unit 231 and outputs the audio signal to the speaker 234. The speaker 234 performs D / A conversion on the input audio signal and emits sound from the speaker.

(Processing procedure)
Subsequently, a procedure of processing executed by the acoustic system 20 will be described with reference to 94. FIG. 10A to FIG. 10C are flowcharts showing a procedure of processing executed by the acoustic system 20 of the present embodiment.

  As shown in FIG. 10A, first, sound in the space is collected in the microphone array 111 (S201). The collected sound is output to the collected sound signal transmission unit 211. Subsequently, the collected sound signal is transmitted from the collected sound signal transmitting unit 211 of the sound collecting unit 21 and received by the collected sound signal receiving unit 221 of the reproduction signal generating unit 22 (S202). The received sound collection signal is output to the sound source separation unit 112. Subsequently, listening points are input in the listening point input units 231 of the plurality of reproducing units 23 (S203). The input listening point is output to the listening point transmission unit 232.

  Subsequently, the listening point is transmitted from the listening point transmitting unit 232 and received by the listening point receiving unit 222 of the reproduction signal generating unit 22 (S204). The plurality of received listening points are output to the separation area control unit 113 and the reproduction signal generation unit 223.

  Subsequently, the separation area control unit 113 determines whether or not the processing is in time for real-time reproduction (S205). If it is determined that the real-time reproduction is in time (YES in S205), the process proceeds to S208. If it is determined that the real-time reproduction is not in time (NO in S205), the process proceeds to S206.

  In S206, the separation area control unit 113 controls the separation area. That is, in S206, a control for combining a plurality of areas and reducing the number of areas is output to the sound source separation unit 112. Further, a separation area control list is generated and control information of the separation area is managed. Subsequently, when the area is controlled in the sound source separation unit 112, the sound collection signal of the frame is output to the storage unit 115, and the sound collection signal input in the storage unit 115 is recorded (S207). Then, the process proceeds to S208.

  In S208, the sound source separation unit 112 performs sound source separation for each area. The separated audio signal for each area is output to the audio signal processing unit 114.

  Subsequently, the audio signal processing unit 114 performs audio signal processing (S209). The processed audio signal is output to the storage unit 115.

  Subsequently, an audio signal for each area processed in the storage unit 115 is recorded (S210). Subsequently, the reproduction signal generation unit 223 acquires sound for each area according to the times of the plurality of listening points input from the storage unit 115 from the listening point reception unit 222, and the reproduction sound is mixed for each listening point. This is performed (S211). The plurality of mixed reproduction signals are output to the reproduction signal transmission unit 224.

  Subsequently, a plurality of reproduction signals generated for each listening point are transmitted from the reproduction signal transmitting unit 224, and a reproduction signal corresponding to the input listening point is received by the reproduction signal receiving unit 233 of each reproducing unit 23 ( S212). Finally, the reproduction signal received by the reproduction signal receiving unit 233 is reproduced from the speaker (S213).

  Next, a process when the number of areas is reduced when it is determined in S205 of FIG. 10A that the process is not in time will be described with reference to FIG.

  When the processing load falls below a predetermined value, the separation area control unit 113 refers to the separation area control list and determines the time (frame) and area for separation (S221). Information on the area to be separated and the time is output to the sound source separation unit 112.

  Subsequently, the sound source separation unit 112 reads a sound collection signal based on the time information input from the storage unit 115 (S222). Since S223 to S225 are the same as S208 to S210, detailed description thereof is omitted.

  As described above, the number of areas is reduced by combining the separation areas based on the processing load and the distribution of the plurality of listening points. Therefore, it is possible to faithfully reproduce important audio signals, and to realize real-time processing by improving processing efficiency. Further, at the time of replay, a reproduction signal can be generated using the separated sound even in an area where transmission is not in time in real time.

  In the present embodiment, all the playback units 23 have the same configuration for simplicity, but the configurations may be different. Although not described in the present embodiment, it may be used in combination with a free viewpoint video generation system that generates a free viewpoint video. For example, a space substantially the same as a space where sound is collected by a plurality of imaging devices is imaged from all directions, and a free viewpoint video is generated from the captured images. In this case, the listening point may be calculated from the viewpoint, or a free viewpoint video may be generated in conjunction with the listening point.

  In the present embodiment, the reproduction signal generation unit 223 is configured in the reproduction signal generation unit 22, but may be configured in the reproduction unit 23. In the present embodiment, the separation area control unit 113 determines the separation area using only the positions of a plurality of listeners. However, as shown in FIG. The existing area may be divided finely and the rear part may be divided roughly.

  When area control is performed in the present embodiment, listening positions that can be input by the listening point input unit 231 may be limited. In the present embodiment, the playback unit 23 is treated uniformly, but may have a different weight for each listening point in order to control the separation area. Similarly to the first embodiment, a display device that displays the status of area control and an input device that instructs separation area control may be provided.

  In each embodiment of the present invention, even in real-time playback where the time until playback is limited, by controlling the number of areas for sound source separation, the entire space is collected and the resolution of important areas is maintained. Can be played.

<Other embodiments>
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

100: Audio signal processing device, 111: Microphone array, 112: Sound source separation unit, 113: Separation area control unit, 114: Audio signal processing unit, 115: Storage unit, 116: Signal generation unit for real-time reproduction, 117: Replay reproduction Signal generator

Claims (17)

A microphone array that collects audio,
Separation means for separating the sound collected by the microphone array into sound in a plurality of separation areas into which a certain space is divided;
Control means for controlling division of the constant space into the plurality of separation areas;
An acoustic system comprising: a generation unit that generates a reproduction signal based on the separated sound.   The acoustic system according to claim 1, wherein the control unit controls the arrangement and the number of the plurality of separation areas.   The acoustic system according to claim 2, wherein the control unit controls the arrangement and the number of the plurality of separation areas based on a processing load of at least one of the separation unit and the generation unit.   The control unit according to claim 1, wherein the control unit performs control so as to divide a region in which the processing load of at least one of the separation unit and the generation unit is larger in the fixed space into finer separation areas. 3. The acoustic system according to 3.   The acoustic system according to claim 2, wherein the control unit controls the arrangement and the number of the plurality of separation areas based on sound detected in the certain space.   The acoustic system according to claim 5, wherein the control unit performs control so as to divide a region in which a higher sound level is detected in the certain space into finer separation areas.   6. The control unit according to claim 5, wherein the control unit performs control so as to divide an area in which sound having a predetermined characteristic is detected in the certain space into smaller separation areas than other areas. The described acoustic system. A photographing means for photographing the fixed space and generating an image;
The acoustic system according to claim 2, wherein the control unit controls the arrangement and the number of the plurality of separation areas based on an image generated by the photographing unit.   9. The acoustic system according to claim 8, wherein the control unit performs control so as to divide an area in which more specific subjects are reflected in the image into finer separation areas.   3. The sound according to claim 2, wherein when the position of a listening point for listening to a sound is designated, the control unit performs control so as to divide a region closer to the listening point into finer separation areas. system.   11. The control unit according to claim 10, wherein when the listening direction at the listening point is designated, the control unit performs control so that a region existing in the listening direction is divided into finer separation areas. Acoustic system.   The acoustic system according to claim 10, wherein the generation unit generates a signal of a sound to be heard at the listening point as the reproduction signal.   The acoustic system according to any one of claims 1 to 12, further comprising display control means for causing a display means to display an image showing an arrangement of the plurality of separation areas divided by the control means.   The acoustic system according to claim 2, wherein the control unit controls the arrangement and the number of the plurality of separation areas according to an instruction from a user. A method for controlling an acoustic system including a microphone array for collecting sound,
A separation step of separating the sound collected by the microphone array into sound in a plurality of separation areas into which a certain space is divided;
Generating a reproduction signal based on the separated audio, and
A control method for an acoustic system, comprising: a control step of controlling division of the certain space into the plurality of separation areas before the separation step. A signal generation device that generates a reproduction signal based on sound collected by a microphone array that collects sound,
Separation means for separating the sound collected by the microphone array into sound in a plurality of separation areas into which a certain space is divided;
Control means for controlling division of the constant space into the plurality of separation areas;
A signal generation device comprising: a generation unit configured to generate a reproduction signal based on the separated sound.   The computer program for functioning a computer as each means with which the signal generation apparatus of Claim 16 is provided.

Images