JP6818445B2 - Sound data processing device and sound data processing method - Google Patents

Description

The present invention relates to a sound data processing apparatus and a sound data processing method.

Conventionally, there is known a technique of dividing a space into a plurality of areas and acquiring (sound collecting) sound for each area (Patent Document 1). In recent years, a sound collection technique with higher directivity has been proposed, and it has become possible to clearly record and emphasize distant sound (Patent Document 2). When sound is collected in the target area using a microphone that has sharper directivity and can record even long-distance voice as in Patent Document 2, it is possible to divide a fairly wide space into small areas and collect sound. Become.

Japanese Unexamined Patent Publication No. 2014-722708 Japanese Unexamined Patent Publication No. 2012-165189

However, if a wide space is divided into small areas and sound is collected for each area, the number of sound collecting areas increases. Since audio signal processing such as echo removal and noise removal is performed for each sound collection area, as the number of sound collection areas increases, the amount of audio signal processing also increases. If the number of sound collection areas is large and an attempt is made to reproduce the collected sound in real time, the sound signal processing may not be in time and the sound data may be lost (the sound may be interrupted).
The present invention has been devised to solve the above-mentioned problems of the prior art, and is an audio signal processing device that does not lose a predetermined (desired) audio even in real-time reproduction in which the time until the end of audio signal processing is limited. The purpose is to provide.

According to one aspect of the present invention, an acquisition means for acquiring sound data generated by collecting sounds in a sound collection target space, and at least one priority of a plurality of areas in the sound collection target space. A sound having a determination means for determining the above , and an output means for outputting processed data obtained by processing the sound data acquired by the acquisition means and corresponding to each of the plurality of areas. In the data processing device, the output means is sound data acquired by the acquisition means when the processing time related to the processing for obtaining the processed data corresponding to each of the plurality of areas is longer than a predetermined threshold value. 1 which is the first processed data obtained by performing the processing of the processing time equal to or less than the predetermined threshold value and is selected based on the priority determined by the determination means among the plurality of areas. after outputting the first processed data corresponding to the above area among the second processed a data before Symbol plurality of areas obtained by processing the acquired sound data by the acquisition unit the one or more of the area to output the second processed data corresponding to another one or more areas comprising different areas, the sound data processing device is provided.

According to the present invention, since the audio signals are processed or transmitted in the order set based on the predetermined conditions, the predetermined (desired) audio is not lost even in the real-time reproduction.

The block diagram which shows the functional structure of the audio signal processing apparatus of Embodiment 1. FIG. The flowchart which shows the audio signal processing of Embodiment 1. The block diagram which shows the hardware configuration of the audio signal processing apparatus of Embodiment 1. FIG. The block diagram which shows the structure of the audio signal processing system of Embodiment 2. The flowchart which shows the audio signal processing of Embodiment 2. The block diagram which shows the structure of the photographing processing system of Embodiment 3. The figure explaining the separation state of voice in Embodiment 3. The flowchart which shows the audio signal processing of Embodiment 3.

Hereinafter, embodiments of the audio signal processing device of the present invention will be described with reference to the accompanying drawings. The following embodiments do not limit the present invention. Moreover, not all combinations of features described in the present embodiment are essential for the means of solving the present invention. The configuration of the embodiment may be appropriately modified or changed depending on the specifications of the apparatus to which the present invention is applied and various conditions (use conditions, use environment, etc.). In the following description, the same configuration will be described with the same reference numerals. In the present specification, the voice is not limited to the voice uttered by a person, and means not only the voice of a person or an animal but also general sounds such as musical sounds and environmental sounds.

Embodiment 1
The audio signal processing device of the first embodiment will be described as being a part of an acoustic system. The sound system is a sound system used for a relay system that performs real-time broadcasting (live broadcasting) and replay broadcasting from a viewpoint (camera orientation) different from that of real-time broadcasting, for example, in sports broadcasting.
(Configuration of audio signal processing device)
FIG. 1 is a functional block diagram showing the configuration of the audio signal processing device 10. The audio signal processing device 10 includes a microphone array 11, a sound source separation unit 12, a processing order control unit 13, an audio signal processing unit 14, a storage unit 15, a real-time reproduction signal generation unit 16, and a replay reproduction signal generation unit 17. Each component of the audio signal processing device 10 is interconnected by a bus 18. The audio signal processing device 10 is connected to the playback device 19 by wire or wirelessly.

The microphone array 11 is composed of a plurality of microphones. The microphone array 11 is connected to the sound source separation unit 12. The microphone array 11 collects the sound in the space with a microphone, A / D-converts the sound signal of the collected sound, and then outputs the sound to the sound source separation unit 12. The microphone array 11 is a sound collecting unit.
The sound source separation unit 12 performs a sound source separation process so as to divide the space into N (N> 1) areas using the audio signal input from the microphone array 11. The sound source separation process is performed for each processing frame, that is, for each predetermined time (time interval, time unit). The sound source separation unit 12 performs beamforming processing, for example, at predetermined time intervals, and acquires an audio signal for each area (a). The acquired voice signal is output to the processing order control unit 13 together with the time (t) information.

The processing order control unit 13 receives the audio signal S (t, a) for each time (t) and area (a) from the sound source separation unit 12. The processing order control unit 13 outputs a signal for controlling the order (order) of audio signal processing for each time and area in the audio signal processing unit 14 based on the time and area information acquired from the audio signal S (t, a). Output. For example, the processing order control unit 13 compares adjacent areas with respect to the voice of the acquired time, and determines whether or not the voice signal of a certain area is masked by the sound of the adjacent area. The processing order control unit 13 sets the audio signal processing order to be delayed for the area that is masked as a result of the determination. In the present embodiment, the masked area is considered to have low importance (priority). That is, the processing order control unit 13 determines the priority for the plurality of areas, accelerates (prioritizes) the audio signal processing order of the high priority area, and delays the audio signal processing of the low priority area. Make settings. The processing order control unit 13 is a means for setting the processing order. The processing order is set from the one with the highest importance (priority) to the one with the lowest priority. The processing order control unit 13 sets the processing order of the audio signals for each area based on a predetermined condition of whether or not masking is performed.

The audio signal processing unit 14 is composed of a processing device such as a microprocessor or the like. The audio signal processing unit 14 receives an output signal (a signal that controls and sets the order of audio signal processing) of the processing order control unit 13. Further, the audio signal processing unit 14 sequentially monitors the processing amount (load state) of the audio signal processing device 10. The audio signal processing unit 14 processes audio signals for each time and area according to a control signal (setting signal) in the order of audio signal processing. The processing performed by the audio signal processing unit 14 includes, for example, a delay correction processing, a gain correction processing, an echo removal processing, and the like for correcting the influence of the distance from the sound collecting unit (microphone array 11) to the area. In the present embodiment, it is assumed that the order of the plurality of processes (delay correction process, gain correction process, echo removal process, etc.) performed by the audio signal processing unit 14 is fixed.

(When the processing amount of the processing device is smaller than the specified amount)
When the processing amount (load) of the processing device is smaller than the predetermined amount, the processing of the audio signal in the entire area of a certain processing frame can be completed within the predetermined time. In this case, the audio signal processing unit 14 outputs the audio signal that has been processed for all areas to the real-time reproduction signal generation unit 16 and the storage unit 15. If the processing of the audio signal in all areas of a certain processing frame can be completed within a predetermined time in the audio signal processing unit 14, the real-time reproduction signal generation unit 16 outputs the real-time reproduction signal in which all the processing is performed in all areas. Can be generated.

(When the processing amount of the processing device is more than the specified amount)
When the processing amount of the processing device is a predetermined amount or more, if the audio signals of the entire area for a certain time (a certain processing frame) are processed, the real-time reproduction signal cannot be generated. When the processing amount of the processing device is equal to or more than a predetermined amount, the audio signal processing unit 14 performs the following signal processing.
The audio signal processing unit 14 manages the remaining time until real-time reproduction, the time required to complete each process, and the like. Since the time related to each process changes according to the load of the processing device, the time until the end of the process changes according to the load of the processing device. When the time until the end of processing becomes longer than the remaining time until real-time reproduction, the audio signal processing unit 14 stores the audio signal in the area where some processing is not performed by the signal generation unit 16 for real-time reproduction and the storage unit. Output to 15. The case where the time until the end of processing is longer than the remaining time until real-time playback is the case where all the processing is not in time for real-time playback. In this case, according to the processing order set by the processing order control unit 13, all the processing is performed in descending order of importance (priority) within the range in time for real-time reproduction (for the area determined to be in time for real-time reproduction). .. Then, the processed audio signal is output to the real-time reproduction signal generation unit 16 and the storage unit 15. On the other hand, the area determined not to be in time for real-time reproduction is output to the real-time reproduction signal generation unit 16 and the storage unit 15 without performing some processing. Not performing some processing means, for example, that three processing should have been performed, but one processing is not performed. The three processes are prioritized, and if the processes are not in time, the processes are not performed from the subsequent processes. Information on the processing that has not been performed is recorded in the processing waiting list. For example, the processing waiting list describes the area where processing was not performed, the time, and the content of processing that was not performed.

Since the audio signal processing unit 14 determines whether or not the processing of all areas is in time for real-time reproduction, it is determined whether or not the processing of the audio signals of all areas can be executed within a predetermined time. It can be said that. Further, when the processing of the audio signals of all the areas cannot be executed within the predetermined time, the information of the area where the processing could not be executed is stored in the storage unit 15.
When the processing load is sufficient, the audio signal processing unit 14 acquires the audio signal of the area that has not been partially processed from the storage unit 15 based on the above-mentioned processing waiting list, performs the remaining processing, and stores the storage unit. Output to 15. Those that have finished processing are deleted from the waiting list.
The storage unit 15 is a storage device such as an HDD, an SSD, or a memory, and stores the audio signal output from the audio signal processing unit 14 in association with time and area.

The real-time reproduction signal generation unit 16 generates a real-time reproduction signal by mixing the sound for each area obtained from the audio signal processing unit 14, and outputs the signal to the reproduction device 19. For example, the real-time playback signal generation unit 16 acquires a virtual listening point (listener's position) in the space that changes with time from the outside, a virtual listener's orientation, and information on the playback environment. Mix the sound source. The reproduction environment means, for example, a type (speaker, headphones, etc.) of a reproduction device 19 that reproduces a signal generated by a real-time reproduction signal generation unit 16.
When the replay playback signal generation unit 17 specifies a time and the replay playback is requested, the replay playback signal generation unit 17 acquires audio data at the corresponding time from the storage unit 15 and performs the same processing as the real-time playback signal generation unit 16. Output. That is, the replay reproduction signal generation unit 17 generates a replay reproduction signal by mixing the audio signals for each area obtained from the storage unit 15, and outputs the audio signal to the reproduction device 19.

(Flow of audio signal processing)
2 (A) to 2 (C) are flowcharts showing the processes executed by the audio signal processing device 10 of the present embodiment.
FIG. 2A shows a flow from sound collection to generation of a real-time reproduction signal. As shown in FIG. 2A, first, the microphone array 11 picks up the sound in the space (S1). The audio signal of the audio of each channel collected is output from the microphone array 11 to the sound source separation unit 12.
Next, the sound source separation unit 12 performs sound source separation for acquiring a sound source for each area in which the space is divided based on a preset boundary (information for dividing the space) (S2). The audio signal for each separated area is output from the sound source separation unit 12 to the processing order control unit 13.

Next, the processing order control unit 13 performs the separation sound analysis (S3) and the processing order setting (S4). More specifically, in the separated sound analysis (S3) by the processing sequence control unit 13, the voice signal of the target area is compared with the voice signal of the surrounding area using the input voice signal for each area, and the target area is compared. Analyzes and determines whether or not the voice of is masked by the voice of the surrounding area. Based on the result of this analysis, in the processing order setting (S4) by the processing order control unit 13, the processing order of the audio signal for each area is set. The audio signal for each area for which the processing order is set is output from the processing order control unit 13 to the audio signal processing unit 14. The processing order setting may be referred to as processing order control.

Subsequently, the audio signal processing unit 14 determines whether or not the processing of the audio signal for each input area is in time for the real-time output (reproduction) (S5). More specifically, the audio signal processing unit 14 sequentially monitors the load state of the audio signal processing device 10 and determines whether or not the real-time output is in time according to the load state (S5). That is, in S5, the audio signal processing unit 14 determines whether the processing amount of signal processing is a predetermined amount or more. When the load of the audio signal processing device 10 is light and the processing can be completed in time (S5: Yes), the audio signal processing unit 14 performs processing on the audio signals corresponding to all areas. Since processing is performed on the voice signals corresponding to all areas in S6, S6 in FIG. 2A is described as "all processing". After that, the audio signal processing unit 14 outputs the processed audio signal to the real-time reproduction signal generation unit 16 and the storage unit 15. The case where the load of the audio signal processing device 10 is light is the case where the load is lower than a predetermined amount. When the process is in time (for real-time playback), the process is completed within a predetermined time.

When the determination in S5 is No, the audio signal processing unit 14 does not process the audio signals in some areas where the processing order is slow based on the order set by the processing order control unit 13, and outputs the audio signals in real time. It is output to the reproduction signal generation unit 16 and the storage unit 15 (S9). At the same time, the information of the area where the processing was not performed is recorded in the processing waiting list. In FIG. 2A, when the determination in S5 is No, the sound in a part of the area is not processed, so S9 is described as “processing except for a part”. In S9, processing is performed in the order of earliest processing order within the range in time for real-time reproduction, so that all processing is executed for the audio signal in the area where the processing order is earlier.

After S6 or S9, the real-time reproduction signal generation unit 16 mixes the audio for real-time reproduction (S7). The audio (signal) mixed for real-time reproduction is output to an external reproduction device 19 as a real-time reproduction signal. The real-time reproduction signal may be output (transmitted) from the audio signal processing device 10 to another device as a broadcasting signal without being output to the reproduction device 19.
After that, the storage unit 15 records the input audio signal in each area (S8).
The audio signal for replay reproduction is created by using the audio signal for each area of the storage unit 15.

Next, using FIG. 2B, processing when the load of the audio signal processing device 10 becomes small after S9 in FIG. 2A will be described. More specifically, processing when the load of the audio signal processing device 10 is reduced after the audio signal in the area where some processing is not performed is recorded in the storage unit 15 because the processing was not in time for real-time reproduction. Will be explained. As an example in which some processing is not performed, a case where delay correction processing, gain correction processing, and echo cancellation processing should have been performed, but echo removal processing is not performed will be described.

When the load of the audio signal processing device 10 becomes smaller than a predetermined amount, the audio signal processing unit 14 reads out the audio signal in the unprocessed area from the storage unit 15 based on the processing waiting list (S11). Since this step is signal processing in a part of the area, it is described as "partially read" in FIG. 2 (B).
Subsequently, the audio signal processing unit 14 performs the processing recorded in the processing waiting list (echo removal processing that was not performed in the audio signal processing unit 14) (S12). In the present embodiment, only the echo removal processing is performed on the audio signal in the area where the delay correction processing and the gain correction processing are performed but the echo removal processing is not performed. The audio signal subjected to the echo removal processing is output to the storage unit 15.

The storage unit 15 overwrites the audio signal in the input area with the previous data (audio signal) and records it (S13).
FIG. 2C shows a processing flow when replay playback is requested. When the replay playback is requested, the replay playback signal generation unit 17 reads out the audio signal for each area corresponding to the replay time from the storage unit 15 (S21).
Subsequently, the replay playback signal generation unit 17 mixes the audio signal for replay playback (S22). The audio signal mixed for replay reproduction is output to the reproduction device 19 or output to another device as a broadcast signal.

(Effect of Embodiment 1)
As described above, the audio signal processing device 10 of the present embodiment controls (sets) the signal processing order for each area based on the audio signal for each area, so that the area has a high importance (priority). On the other hand, audio processing can be reliably performed and real-time broadcasting can be performed. More specifically, when the processing amount of signal processing in the audio signal processing unit 14 is a predetermined amount or more, there is a possibility that the audio may be lost during real-time reproduction in the conventional method. On the other hand, in the present embodiment, when the processing amount of the signal processing by the audio signal processing unit 14 is a predetermined amount or more, all the signal processing by the audio signal processing unit 14 is performed in the order of the earliest processing order within the range in time for real-time reproduction. Then, the audio signal is supplied to the audio reproduction unit. Therefore, all audio processing is performed for areas of high importance so that they can be played back in real time.
Further, in the present embodiment, all the audio processing is performed later even for the area where all the audio processing is not in time during the real-time reproduction (the area with low importance), so that the audio processing is completely performed during the replay reproduction. Playback sounds can be generated using audio signals.

(Modification example)
Although the microphone array 11 is made of a microphone in the present embodiment, it may be made of a combination of the microphone and a structure such as a reflector.
The microphone used in the microphone array 11 may be an omnidirectional microphone, a directional microphone, or a combination thereof. That is, the microphone array 11 may include at least one directional microphone.
In the present embodiment, the sound source separation unit 12 collects sound for each area by using beamforming, but other sound source separation methods may be used. For example, the power spectral density (PSD) for each area may be estimated, and separation by a Wiener filter may be performed based on the estimated PSD. PSD is an abbreviation for Power Spectral Density.

In the present embodiment, the processing order control unit 13 sets the processing order according to the masking when the target area and the adjacent area are compared with each other, but other information / conditions may be used for setting the processing order. For example, the area where the event occurs may be set to be processed quickly. More specifically, in sports, the area near the goal may be set to be processed quickly, and in the city, the area where many people and cars pass, such as near an intersection, may be set to be processed quickly. May be good. Alternatively, the processing order may be set based on the characteristics of the voice signal acquired by collecting the sound from the area. For example, the audio signal processing device 10 includes means for determining whether or not the sound collection signal (audio signal) in the area includes an audio signal representing a human voice, and the processing order is determined according to the presence or absence of the human voice. May be made faster (priority). Alternatively, the levels of the sound pick-up signals (audio signals) in the area may be compared with each other, and the processing order of the audio signals in the high-level area may be accelerated based on the comparison result.

Alternatively, the processing order may be controlled (set) according to the virtual listening point (position of the virtual listener) and the orientation of the virtual listener set in the signal generation for real-time reproduction. For example, the order of processing may be earlier for the sound pick-up signal in the area closer to the virtual listening point. When comparing the effects of masking, instead of comparing with adjacent areas, set a predetermined angle range from the listening point and the orientation of the listener, and compare the effects of masking on the area within that range. You may. At that time, it is advisable to compare masking after performing delay correction and volume correction according to the distance from the virtual listening point. In this case, the audio signal processing device 10 includes a distance acquisition unit that acquires the distance from the microphone array 11 to each area.

Further, in the present embodiment, the processing order control unit 13 sets the same processing order for a plurality of processes (delay correction, gain correction, echo removal processing, etc.) (the processing order is fixed), but the sound collection status. Different processing orders may be set according to (long-distance sound collection, short-distance sound collection, etc.). Generally, it is considered that the farther (longer) the distance from the microphone array to the sound collecting area is, the stronger the influence of echo is. Therefore, when the distance from the microphone array to the sound collection area is long (long-distance sound collection), the order of the echo removal processing is set to be earlier than the delay correction processing and the gain correction processing. However, when the distance from the microphone array to the sound collection area is short (short-distance sound collection), the processing order of the echo removal processing may be delayed. In that case, the delay correction processing and the gain correction processing for the area are set so that the processing order is earlier than the echo removal processing. Further, the processing order may be determined in a complex manner according to a plurality of conditions as described above.
In the present embodiment, the audio signal processing unit 14 performs delay correction processing, gain correction processing, and echo cancellation, but other processing may also be performed. For example, the audio signal processing unit 14 may perform noise removal processing for each area.
Further, in the above-described embodiment, when the processing amount of the audio signal processing unit 14 is equal to or greater than a predetermined amount, it has been described that some processing is not performed in the area where the processing order is slow, but all the processing is not performed. It may be.

In the present embodiment, the replay reproduction signal generation unit 17 and the real-time reproduction signal generation unit 16 perform the same processing (mixing). However, the replay reproduction signal generation unit 17 and the real-time reproduction signal generation unit 16 may be mixed differently. For example, since there is a possibility that audio that has not been processed by the audio signal processing unit 14 may be input to the real-time reproduction signal generation unit 16, the real-time reproduction signal generation unit 16 determines whether or not the processing has been performed. The level (quality) of mixing may be changed accordingly. For example, when unprocessed audio is input, the signal generation unit 16 for real-time reproduction may lower the mixing level.
In the configuration of FIG. 1, the microphone array 11 is included in the audio signal processing device 10, but the microphone array 11 may not be included in the audio signal processing device 10. In that case, the sound source separation unit 12 receives an audio signal from the microphone array 11 which is an external device.

(Hardware configuration)
At least a part of the functional blocks shown in FIG. 1 may be realized by hardware. When it is realized by hardware, for example, by using a predetermined compiler, a dedicated circuit may be automatically generated on the FPGA from the program for realizing each step. FPGA is an abbreviation for Field Programmable Gate Array. Further, a Gate Array circuit may be formed in the same manner as the FPGA and realized as hardware. Further, it may be realized by an ASIC (Application Specific Integrated Circuit).

FIG. 3 is a diagram showing an example of the hardware configuration of the audio signal processing device 10. In this example, the audio signal processing device 10 includes a CPU 10a, a RAM 10b, a ROM 10c, an external memory 10d, a communication interface (I / F) 10e, a display unit 10f, an input unit 10g, and a microphone array 11. The CPU 10a can realize the functions of each part of the audio signal processing device 10 of the first embodiment described above by executing a program stored in the RAM 10b, the ROM 10c, the external memory 10d, or the like. The audio signal processing device 10 can realize each process shown in FIG. 2 by reading and executing a program required by the CPU 10a. The communication I / F10e is an interface for communicating with an external device or device. The audio signal processing device 10 can communicate with the playback device 19 of FIG. 1 via the communication I / F 10e. The display unit 10f can be configured by various displays. The display unit 10f can display, for example, the separation status of the sound source separation process. The input unit 10g can be composed of a keyboard, a pointing device (mouse or the like), a touch panel, and various switches. The user can request the audio signal processing device 10 to perform replay reproduction using the input unit 10g. The audio signal processing device 10 may further include an imaging unit that captures an image of a subject existing in the area.

Embodiment 2
In the first embodiment, the configuration in which the microphone array 11, the audio signal processing unit 14, the real-time reproduction signal generation unit 16, and the replay reproduction signal generation unit 17 are included in one device 10 has been described. It is not limited to the configuration. For example, the microphone array and the audio signal processing unit may be included in one device, and the reproduction signal generation unit may be implemented in a configuration (system) separated from the device. Further, the microphone array may adopt a directional microphone array. The voice signal of the voice picked up by the directional microphone array may be immediately processed by the voice signal processing unit. An audio signal processing system having such a configuration will be described as the second embodiment. The same reference numerals are given to the same components as in the first embodiment.

(Configuration of audio signal processing system 20)
FIG. 4 is a block diagram showing the configuration of the audio signal processing system 20 of the second embodiment. The audio signal processing system 20 includes a sound collection processing unit 20a and a reproduction signal generation unit 20b. The sound collection processing unit 20a and the reproduction signal generation unit 20b are interconnected by wire or wirelessly, and data is transmitted to and received from each other by a wired communication means or a wireless communication means.
The sound collection processing unit 20a includes a directional microphone array 21, an audio signal processing unit 22, a transmission order (transmission order) control unit 23, a transmission unit 24, and a first storage unit 25. The first storage unit 25 holds an untransmitted list. Each component of the sound collecting processing unit 20a is interconnected by a bus 18a.

The directional microphone array 21 is composed of, for example, a plurality of shotgun microphones, and is arranged so as to pick up the sound of each area in which the space is divided into N (N> 1) areas. The sound collected for each area is A / D converted and then output to the voice signal processing unit 22 for each predetermined time frame size.
The audio signal processing unit 22 performs delay correction processing, gain correction processing, noise removal processing, echo removal processing, and the like, as in the audio signal processing unit 14 of the first embodiment. The audio signal for each processed area is output from the audio signal processing unit 22 to the transmission sequence control unit 23.

Similar to the processing sequence control unit 13 of the first embodiment, the transmission sequence control unit 23 has the influence of masking of the area voice, the level of the area sound (large and small), the feature amount of the area sound, and the steady state of the area sound. Control (set) the transmission order based on gender. The transmission sequence control unit 23 determines, for example, the non-stationarity of the audio signal S (t, a) in the time t and the area a. When the non-stationary sound is generated within a predetermined time interval, the transmission sequence control unit 23 determines that there is a high possibility that an important event has occurred in the area, and the frequency of the non-stationary sound is in descending order. Set the transmission order to perform transmission. In other words, the area where the frequency of non-stationary sounds is high is considered to be the area with high importance (priority). The transmission order control unit 23 of the present embodiment determines the priority for a plurality of areas based on a predetermined condition of the frequency of non-stationary sounds, and accelerates (prioritizes) the transmission order of the high priority area. ), Set to slow down the transmission order in low priority areas.

Further, the transmission order control unit 23 determines the voice signal S (order of voice signal transmission) based on the signal transmission amount between the sound collection processing unit 20a and the reproduction signal generation unit 20b and the set transmission order (order of voice signal transmission). Whether or not t and a) are in time for real-time playback (broadcasting) is determined for each area. This determination is performed, for example, by whether or not the amount of signal transmission between the sound collection processing unit 20a and the reproduction signal generation unit 20b is equal to or greater than a predetermined amount. The audio signal in the area determined to be in time for real-time reproduction is output from the transmission sequence control unit 23 to the transmission unit 24. On the other hand, the audio signal in the area determined to be in time (the audio signal in the area where the signal transmission amount is a predetermined amount or more and the transmission order is slow) is output from the transmission order control unit 23 to the first storage unit 25. Will be done. The voice data that has not been transmitted is managed (stored) in the untransmitted list in the first storage unit 25. That is, when the transmission amount of the audio signal from the transmission unit 24 is equal to or more than a predetermined amount, the transmission unit 24 does not transmit in the area where the transmission order is slow.

When the signal transmission amount between the sound collection processing unit 20a and the reproduction signal generation unit 20b becomes smaller than a predetermined amount, the transmission sequence control unit 23 reads out the audio signal in the untransmitted area stored in the first storage unit 25. Output to the transmitter 24.
The transmission unit 24 sequentially transmits the audio signals for each area input from the transmission order control unit 23 to the reproduction signal generation unit 20b. The reproduction signal generation unit 20b can be referred to as an audio reproduction unit.
The reproduction signal generation unit 20b includes a reception unit 26, a real-time reproduction signal generation unit 27, a second storage unit 28, and a replay reproduction signal generation unit 29. Each component of the reproduction signal generation unit 20b is interconnected by a bus 18b.
The receiving unit 26 receives the audio signal output from the transmitting unit 24, and outputs the audio signal to the real-time reproduction signal generation unit 27 and the second storage unit 28.
The real-time reproduction signal generation unit 27, the second storage unit 28, and the replay reproduction signal generation unit 29 are substantially the same as the real-time reproduction signal generation unit 16, the storage unit 15, and the replay reproduction signal generation unit 17 of the first embodiment. Since it has a function, the description thereof will be omitted.

(Processing flow of audio signal processing system 20)
Next, the processing flow of the audio signal processing system 20 will be described with reference to FIGS. 5 (A) to 5 (C).
As shown in FIG. 5A, first, the directional microphone array 21 picks up the sound in the space (S31). The voice signal of the voice of each area collected is output from the directional microphone array 21 to the voice signal processing unit 22.
The audio signal processing unit 22 performs processing such as delay correction on the input audio signal (S32). The processed audio signal is output from the audio signal processing unit 22 to the transmission sequence control unit 23.

The transmission sequence control unit 23 detects (detects) the non-stationarity of the voice for each area using the input processed voice signal (S33). That is, the transmission sequence control unit 23 performs non-steady sound detection for detecting whether or not the voice signal for each area includes a voice signal representing a non-stationary sound.
Further, the transmission order control unit 23 sets the transmission order according to the result of the non-stationary sound detection in S33 (S34). For example, the transmission order control unit 23 sets the transmission order of the audio signals for each area so that the transmission order is faster (priority) for the area where the non-stationary sound is detected.
Further, the transmission order control unit 23 determines (S35) whether the transmission of the processed audio signal is in time for real-time reproduction. More specifically, with respect to the data (voice signal) in which the order is set for each area, the transmission order control unit 23 determines the voice signal according to the communication load state between the transmission unit 24 and the reception unit 26. Estimate the time until the transmission is completed, and determine whether or not it is in time for real-time playback. When it is determined based on the determination result that the real-time reproduction is in time, the audio data (audio signal) in the area is output from the transmission order control unit 23 to the transmission unit 24. Then, the transmission unit 24 transmits the voice data of the area to the reception unit 26 (S36).

The receiving unit 26 receives the audio signal of the area transmitted from the transmitting unit 24 (S37). The audio signal in the received area is output from the receiving unit 26 to the real-time reproduction signal generation unit 27 and the second storage unit 28.
Since S38 and S39 are the same as S17 and S18 (FIG. 2) of the first embodiment, the description thereof will be omitted.
On the other hand, when it is determined in S35 that the transmission of the processed audio signal (audio signal transmission) is not in time for real-time reproduction, the audio data (audio signal) in the area is transmitted from the transmission order control unit 23 to the first storage unit 25. Is output to and recorded in the first storage unit 25 (S40). The voice data is recorded in the first storage unit 25 so that the processing frame number (or time), the area of the sound, and the like can be known. That is, in the first storage unit 25, the voice data is recorded (stored) in association with predetermined information.

Next, in step S35 of FIG. 5 (A), when the processed voice transmission is not in time for real-time reproduction, the processing when the voice signal in the area is recorded in the first storage unit 25 is shown in FIG. 5 (B). It will be explained based on.
When the transmission load between the transmission unit 24 and the reception unit 26 becomes smaller than a predetermined amount, the transmission sequence control unit 23 reads out the audio signal in the untransmitted area from the first storage unit 25 (S41). The reading is performed from the one with the smallest frame number of the audio signal. That is, the reading is performed from the audio signal of the older time. The read audio signal is output from the transmission sequence control unit 23 to the transmission unit 24.
Since S42 is the same as S36, the description thereof will be omitted.
After S42, the receiving unit 26 receives the voice data (S43). The received voice data is output from the receiving unit 26 to the second storage unit 28.
Since S44 is the same as S39, the description thereof will be omitted.
The processing flow when the replay reproduction is requested is shown in FIG. 5 (C). Since S51 and S52 of this processing flow are the same as S21 and S22 (FIG. 2C) of the first embodiment, description thereof will be omitted.

(Effect of Embodiment 2)
As described above, in the second embodiment, the area of higher importance is obtained by controlling and setting the transmission order of the audio signals based on the non-stationarity of the audio signals (frequency of non-stationary sounds) for each area. On the other hand, it is possible to broadcast in real time with an audio signal that has been reliably processed by audio. More specifically, when the amount of signal transmitted from the transmitting unit 24 to the receiving unit 26 is a predetermined amount or more, there is a possibility that audio may be lost during real-time reproduction in the conventional method. On the other hand, in the present embodiment, the audio signal is transmitted from the transmission unit 24 according to the set transmission order within the range in time for real-time reproduction. Then, the transmission unit 24 does not transmit in the area where the transmission order is late. Therefore, in the area of high importance (the area where the transmission order is early), the voice signal is surely received by the receiving unit 26 and reaches the real-time reproduction signal generation unit 27. As described above, in the signal processing system 20 of the second embodiment, the highly important audio signal is transmitted to the audio reproduction unit in time for real-time reproduction.
Further, in the present embodiment, since the audio signal is later transmitted to the receiving unit 26 even in the area where the transmission (transmission) is not in time for the real-time reproduction (the area of low importance), all the areas during the replay reproduction are performed. Playback sounds can be generated using the audio signal sounds of the area.

(Modification example)
In the second embodiment, the directional microphone array 21 is used for collecting the sound for each area, but the sound source may be separated after the sound is collected by using the microphone array as in the first embodiment. Further, the omnidirectional microphone may be arranged at the substantially center of each of the set areas to acquire the sound.
In the configuration of FIG. 4, the directional microphone array 21 is included in the sound collecting processing unit 20a, but the directional microphone array 21 may not be included in the sound collecting processing unit 20a. In that case, the audio signal processing unit 22 receives the audio signal from the directional microphone array 21 which is an external device.

Embodiment 3
Although the audio signal processing device that processes only audio signals has been described in the first embodiment, the present invention can also be implemented in a system that processes audio signals and video signals. A system capable of audio signal processing and video signal processing is referred to as a shooting processing system and will be described below. The shooting processing system of the present embodiment can also be referred to as an audio / video output system. Further, the same reference numerals are given to the same components as in the first embodiment.
FIG. 6 is a block diagram showing the configuration of the photographing processing system 30. The photographing processing system 30 includes an audio sound picking control device 30a and a video photographing control device 30b. The audio sound collection control device 30a and the video capture control device 30b are interconnected by a bus 18.
The audio sound collection control device 30a includes a microphone array 31, a separation order control unit 32, a sound source separation unit 33, a storage unit 34, a real-time reproduction signal generation unit 35, a replay reproduction signal generation unit 36, and a separation status display unit 37. .. Each component of the voice pick-up control device 30a is interconnected by a bus 18.
Since the microphone array 31 is the same as the microphone array 11 of the first embodiment, the description of the microphone array 31 will be omitted.

The separation order control unit 32 divides the image captured by the image pickup unit 41 described later into a plurality of areas (for example, areas A, B, C, ...), And within the divided areas, for example, a person, a car, a musical instrument, etc. Determine if there is a subject that can be a sound source. Based on the determination result, the separation order control unit 32 sets the processing order of the sound source separation processing. More specifically, when there is a subject that can be a sound source in the area, the separation order control unit 32 is set to speed up the processing order (order) of the sound source separation processing in the area. When there are a plurality of subjects that can be sound sources in the area (for example, when there are a plurality of people in the area), the separation order control unit 32 is set to speed up the processing order of the sound source separation processing according to the number of subjects. To do. For example, when there is one subject in the area A and five subjects in the area B, the processing order is set so that the sound source separation process in the area B is faster than the sound source separation process in the area A. In other words, an area with a large number of subjects is considered to be an area with high importance (priority) because there are many objects (persons / objects) that are candidates for sound sources. The separation order control unit 32 of the present embodiment determines the priority for a plurality of areas based on a predetermined condition, accelerates the separation order of the high priority area, and delays the separation order of the low priority area. Make the settings.

Further, when the replay playback is set in the replay setting unit 44 described later, the separation order control unit 32 sets the start time and end time of the replay playback (broadcast), the remaining time until the replay broadcast, and the viewpoint in the replay (of the camera). Orientation) and the like are received from the replay setting unit 44. Then, based on the received information, the separation order control unit 32 resets the processing order of the sound source separation processing. More specifically, the separation order control unit 32 refers to the unseparated list described later and extracts the unseparated area from the start of the replay to the end of the replay. Then, the processing order of the sound source separation processing is reset so that the sound source separation processing order of the extracted area becomes faster (priority). If the sound source separation process for real-time playback is performed even when the replay playback is set, the replay playback process is executed so as not to interfere with the real-time playback process. Therefore, the ratio of the replay playback process and the real-time playback process when the replay playback is set may be set in advance. For example, when replay playback is set, real-time playback processing is executed for 80% of the processing priority in all areas, and the remaining time is allocated to replay playback processing (replay playback processing is for the remaining 20%). It may be executed against).

The separation order control unit 32 outputs the processing order information of the set sound source separation process to the sound source separation unit 33 and the separation status display unit 37.
The sound source separation unit 33 performs sound source separation processing according to the processing order input from the separation order control unit 32. Since the content of the sound source separation process is the same as that of the first embodiment, the description thereof will be omitted. The sound source separation unit 33 of the present embodiment further monitors the processing amount (load state) of the sound source separation processing, performs the sound source separation processing of the area in the designated order, and performs the sound source separation processing of the entire area. Determines if is in time for real-time playback. When it is determined that the real-time reproduction is not in time, the sound source separation unit 33 does not perform the sound source separation processing of some audio signals, and outputs information about the audio signals as an unseparated list to the separation order control unit 32. More specifically, the sound source separation processing is performed in the order of earliest processing order (in descending order of importance) within the range in time for real-time playback, and the sound source separation processing is not performed in the area where the processing order is late. Information such as the area and time of the audio signal that has not been separated from the sound source is recorded in the unseparated list (the time information may be information corresponding to the time such as the processing frame number). The sound source separation processed audio signal is output from the sound source separation unit 33 to the real-time reproduction signal generation unit 35 and the storage unit 34.

Regarding the voice of the unseparated list, when the load of the sound source separation processing becomes lower than the predetermined value, the voice signal specified by the information recorded in the unseparated list is transmitted from the storage unit 34 to the sound source separation unit 33. Read out. The sound source separation unit 33 separates and processes the audio signal, and outputs (records) it to the storage unit 34 again. The area and time information of the audio signal for which the sound source separation process has been executed is deleted from the unseparated list.
The storage unit 34 is connected to the microphone array 31, and the sounds of all the channels collected by the microphone array 31 are recorded in the storage unit 34. The storage unit 34 is also connected to the sound source separation unit 33, and the audio signal separated by the sound source separation unit 33 is recorded in the storage unit 34 in association with the area and time.

The real-time reproduction signal generation unit 35 generates an audio signal according to the viewpoint (camera orientation) switching from the real-time viewpoint switching unit 43 described later, and outputs the audio signal to the reproduction device 19a. The playback device 19a of the present embodiment can reproduce audio and video.
The replay reproduction signal generation unit 36 acquires a separated audio signal for each area of the corresponding time from the storage unit 34 according to the viewpoint from the replay setting unit 44 described later and the start time and end time of the replay. Then, the replay reproduction signal generation unit 36 outputs the separated audio signal as a replay reproduction audio signal to the reproduction device 19a.

The separation status display unit 37 is a display device, which is connected to the separation order control unit 32, and displays the audio separation status of each time area from the unseparated list. FIG. 7A shows an example of the display screen 60 of the separation status display unit 37. The display screen 60 includes a time bar 61, a time cursor 62, a separation end ratio display 63, an overall area display 64, an unseparated area 65, and a separated area 66.
The time bar 61 is a bar showing the recording time up to the present, and the position of the time cursor 62 represents the time on the display screen. The separation end ratio display 63 displays what percentage of the total area the separation process has been completed. In the example of FIG. 7A, there are 40 areas, and since the separation processing of 20 areas has been completed, the separation end ratio display 63 is 50%. The whole area display 64 shows the whole target area.
The small square in the whole area display 64 represents the area division, the area where the separation process has not been completed is displayed as a white area (unseparated area) 65, and the area where the separation process has been completed is a black area (separated area). ) 66 is displayed. The unseparated area 65 and the separated area 66 are displayed so that the difference can be seen.

The video capture control device 30b includes an imaging unit 41, an image storage unit 42, a real-time viewpoint switching unit 43, and a replay setting unit 44. Each component of the video capture control device 30b is interconnected by a bus 18.
The imaging unit 41 has a plurality of cameras (not shown). Each camera can shoot (capture) a moving image. The image pickup unit 41 takes an image of the space including the entire area where the sound is collected by the microphone array 31. The captured image (image) is output from the imaging unit 41 to the image storage unit 42.
The image storage unit 42 is a storage device, and records (stores) images taken by all the cameras.

The real-time viewpoint switching unit 43 is, for example, a switcher for switching a plurality of cameras of the imaging unit 41, and outputs the image of the switched (selected) camera to the playback device 19a as a real-time image output. Further, the real-time viewpoint switching unit 43 outputs the viewpoint information of the switched camera to the real-time reproduction signal generation unit 35.
The replay setting unit 44 sets the replay start time, the replay end time, the replay broadcast time, the replay video viewpoint, and the like. The replay setting unit 44 reads the video from the image storage unit 42 based on the set information and outputs the replay video output to the playback device 19a. Further, the replay setting unit 44 outputs the set information to the separation order control unit 32 and the replay reproduction signal generation unit 36.

(Processing flow of shooting processing system 30)
8 (A) and 8 (B) are flowcharts illustrating the operation and processing of the photographing processing system 30 of the present embodiment. In the following description, the real-time viewpoint switching unit 43 switches the camera of the imaging unit 41 at a predetermined timing (S73) (meaning that there are a plurality of cameras and a camera other than the camera currently in use is selected). And.
A case of performing real-time reproduction will be described with reference to FIG. 8 (A).
First, sound is picked up by the microphone array 31 (S61), and image pickup is performed by the imaging unit 41 (S71). The voice data (voice signal) acquired via the microphone array 31 is output to the storage unit 34 and the sound source separation unit 33. The image data (image signal) acquired via the image pickup unit 41 is output to the image storage unit 42, the real-time viewpoint switching unit 43, and the separation order control unit 32.

Next, the voice data input to the storage unit 34 is recorded in the storage unit 34 (S62). Further, the image data input to the image storage unit 42 is recorded in the image storage unit 42 (S72).
Subsequently, the separation order control unit 32 detects a subject that can be a sound source for each area from the input image data (S63). The separation order control unit 32 detects the subject by, for example, a face recognition technique or an object recognition technique.
Further, the separation order control unit 32 sets the order of sound source separation based on the number of subjects that can be sound sources for each area detected in S63 (S64). That is, in the present embodiment, the order of sound source separation is set based on the sound source position information. The set sound source separation order is output from the separation order control unit 32 to the sound source separation unit 33.

Next, the sound source separation unit 33 determines whether or not the sound source separation processing in all areas is in time for real-time reproduction according to the processing load status of the sound source separation (S65). This determination is made for each area. That is, within the range in time for real-time playback, it is determined (determined) to which area the sound source is separated in the order of earliest separation order.
The area determined in S65 that the sound source separation process is in time for real-time reproduction is subjected to sound source separation processing using the audio signals of a plurality of channels input from the microphone array 31 (the sound source separation unit 33 separates the audio signals in the area. And extract) (S66). The audio signal in the separated area is output from the sound source separation unit 33 to the storage unit 34 and the real-time reproduction signal generation unit 35.
The storage unit 34 records the voice signal of the separated voice of the area generated in S66 (S67).

Here, it is assumed that the viewpoint is switched by the real-time viewpoint switching unit 43 (S73). By this viewpoint switching, another camera is selected. The switched viewpoint information is output from the real-time viewpoint switching unit 43 to the real-time reproduction signal generation unit 35.
The real-time reproduction signal generation unit 35 generates an audio signal for real-time reproduction based on the separated sound (audio signal) for each area input from the sound source separation unit 33 and the viewpoint information input from the real-time viewpoint switching unit 43. Generate (S68). The generation of the audio signal for real-time reproduction in S68 is described as real-time rendering in FIG.
Finally, the audio signal for real-time reproduction generated in S68 and the image of the camera selected by the viewpoint switching in S73 are output to the reproduction device 19a (S69, S74).
For the area determined in S65 that the sound source separation process is not in time for real-time playback, the unseparated list is generated by recording (adding) the unseparated time information and the area information in the unseparated list. (S70).

Subsequently, the flow when the replay is set in the replay setting unit 44 will be described with reference to FIG. 8 (B).
First, the replay setting unit 44 sets the replay (S81). The set replay start time, end time, viewpoint information, etc. are output from the replay setting unit 44 to the separation order control unit 32 and the replay reproduction signal generation unit 36.
Subsequently, the separation order control unit 32 confirms the unseparated list (S82). The separation order control unit 32 extracts information on the unseparated area included in the replay time from the unseparated list. Then, the separation order control unit 32 resets the separation order based on the information of the extracted unseparated area (S83). The reset separation order is output from the separation order control unit 32 to the sound source separation unit 33. After that, sound source separation (S84) and recording (S85) are executed. Since S84 is the same process as S66 in FIG. 8 (A) and S85 is the same process as S67 in FIG. 8 (A), the details will be omitted.

Subsequently, the replay reproduction signal generation unit 36 reads out the audio signal of the corresponding time and area from the storage unit 34 based on the input from the replay setting unit 44 (S86).
On the other hand, the replay setting unit 44 reads out the video (image data) corresponding to the replay time and the viewpoint (selected camera) from the image storage unit 42 (S87).
Subsequently, the replay reproduction signal generation unit 36 generates a replay reproduction signal (S88). The generation of the audio signal for replay reproduction in S88 is described as replay rendering in FIG.
Finally, the audio signal generated in S88 is output from the replay reproduction signal generation unit 36 to the reproduction device 19a as the replay audio output (S89). Further, the video data read out in S87 in synchronization with this audio signal is output from the replay setting unit 44 to the playback device 19a as a replay video output (S90).

(Effect of Embodiment 3)
According to the present embodiment, by setting the order of sound source separation according to the sound source position information, even if the sound source separation of all areas is not in time for real-time playback due to the processing load, the sound of the area with the sound source (candidate) The signal is preferentially separated into sound sources and an audio signal can be acquired. Areas with sound source candidates are areas of high importance. Therefore, according to the present embodiment, real-time audio reproduction can be performed without lacking audio signals in areas of high importance.

(Modification example)
In the present embodiment, the separation order control unit 32 controls the separation order based on the image captured by the imaging unit 41, but the separation order control unit 32 may control and set the separation order based on other information. Good. For example, the photographing processing system 30 may further include a distance acquisition unit that acquires a distance from the microphone array 31 to the area. Then, the separation order control unit 32 may control the separation order based on the distance acquired by the distance acquisition unit. The time required for sound propagation differs between the area far from the microphone array 31 and the area near it. That is, the voice in a distant area takes a long time to propagate, so that the delay amount becomes large. Therefore, it is necessary to hurry the separation process for the voice in the street area. Therefore, the separation order control unit 32 may be set to advance the processing timing as the distance from the microphone array 31 increases.

In live broadcasting such as TV broadcasting, a system (technique) that broadcasts with a certain time delay (for example, a delay of several seconds to several minutes) from the actual shooting in order to adjust the time and respond to unexpected situations. It has been known. When such a system is used, the separation order control unit 32 may control the separation order according to the events included in the video for the delay time captured by the imaging unit 41. For example, in the case of broadcasting a live broadcast of a sports game with a delay of 2 minutes, even if the attention area is set from the 2-minute game development and the separation order of each area is controlled according to the attention area. Good.

Further, in order to ensure the continuity of audio, the separation order control unit 32 acquires the average volume of a plurality of processing frames for each area from the sound source separation result of the sound source separation unit 33, and separates the sound sources based on the average volume. You may control the order. For example, the degree to which a sound source exists in the area (sound source abundance) is estimated from the average volume of each area of a predetermined number of frames in the past (most recent), and the processing order of the area having a high sound source abundance is accelerated. May be set to. Further, since the sound source may move, the processing order may be set to be earlier in the vicinity of the area where the sound source is highly present.
In the present embodiment, the storage unit 34 records the audio signals of the audio of all the channels collected by the microphone array 31, but the recording amount of the audio signal may increase depending on the shooting time and the number of microphones. is there. In order to prevent the recording amount from becoming too large, the storage unit 34 may delete the audio signal corresponding to the time when the separation processing of all areas is performed.

In the present embodiment, the display screen 60 (FIG. 7A) of the separation status display unit 37 displays in black and white whether or not the separation process is completed, but it is set by the separation order control unit 32. The separation order may be displayed. For example, as shown in FIG. 7B, on the display screen 60, numbers indicating the separation order (numbers 1 to 13 in the figure) may be displayed in the area display. Alternatively, instead of the numbers 1 to 13 in FIG. 7 (B) (or in addition to the numbers 1 to 13), a display using colors (including gradation) according to the separation order may be used.

Further, the separation status display unit 37 superimposes the viewpoint at the time of real-time reproduction, the viewpoint at the time of replay reproduction, the position of the subject, and the like on the display screen 60 of FIG. 7A or FIG. 7B. May be good. Further, the separation status display unit 37 may superimpose the entire area display 64 of FIG. 7A or FIG. 7B on an image of the actual space.
The display by the separation status display unit 37 is not limited to the display as shown in FIGS. 7 (A) and 7 (B), and may be any display that shows the processing status (separation status). For example, as shown in FIG. 7C, the separation status may be shown in a graph with the separation end percentage (separation end ratio display) as the vertical axis and the time as the horizontal axis. FIG. 7C shows the separation end percentage in a time series as a line graph, and shows the separation status in a simple format.

In the present embodiment, the separation status display unit 37 has only a display function, but may also have a user interface function. For example, the display screen 60 may form a touch panel, and the touch panel may function as a user interface for the separation order control unit 32. With such a configuration, for example, the user selects at least one unseparated area 65 on the display screen 60 and selects a time interval using the time cursor 62 to select a designated area and a designated time. The order of separation of sections can be shortened.
The configuration shown in FIG. 6 does not include an audio signal processing unit (similar to the audio signal processing unit 14 in FIG. 1) that performs delay correction, gain correction, echo cancellation, etc., but if necessary. For example, an audio signal processing unit may be provided between the sound source separation unit 33 and the storage unit 34.
In the configuration of FIG. 6, the microphone array 31 is included in the voice pick-up control device 30a, but the microphone array 31 may not be included in the voice pick-up control device 30a. In that case, the sound source separation unit 33 receives an audio signal from the microphone array 31 which is an external device.

(Other embodiments)
The present invention supplies a program (computer program) that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device supply the program (computer program). It can also be realized by the process of reading and executing the program. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

10 ... Audio signal processing device, 11 ... Microphone array, 12 ... Sound source separation unit, 13 ... Processing order control unit, 14 ... Audio signal processing unit, 15 ... Storage unit, 16 ... Real-time playback signal generation unit

Claims (19)

Acquisition means for acquiring sound data generated by collecting sound in the sound collection target space, and
A determination means for determining at least one priority of a plurality of areas in the sound collection target space, and
It has an output means for outputting processed data obtained by processing the sound data acquired by the acquisition means and corresponding to each of the plurality of areas.
When the processing time related to the processing for obtaining the processed data corresponding to each of the plurality of areas is longer than the predetermined threshold value, the output means has the predetermined threshold value with respect to the sound data acquired by the acquisition means. wherein corresponding to one or more areas that are selected based on the priority determined by said determining means of the first processed data is a by the plurality of areas obtained by performing the processing of the following processing time after outputting the first processed data, and the one or more areas of the second processed a data before Symbol plurality of areas obtained by processing the sound data acquired by the acquiring means the sound data processing device and outputting a pre-Symbol second processed data that corresponds to the addition of one or more areas comprising different areas. The claim for obtaining the first processed data and the process for obtaining the second processed data include at least one of a delay correction process, a gain correction process, and an echo cancellation process. Item 2. The sound data processing device according to item 1 . The process for obtaining the first processed data and the process for obtaining the second processed data include a sound source separation process for extracting a sound in a specific area from the sound data. Item 2. The sound data processing apparatus according to item 1 . Further having a second acquisition means for acquiring the captured image of the sound collection target space,
The determination means according to any one of claims 1 to 3 , wherein the determination means determines the priority based on the position of a predetermined object in the captured image acquired by the second acquisition means. The sound data processing device described. The fourth aspect of the present invention is characterized in that the priority is determined so that a higher priority is set in the area including the position of the predetermined object as compared with the area not including the position of the predetermined object. The sound data processing device described. Further having a third acquisition means for acquiring viewpoint information corresponding to an image reproduced together with a sound based on the sound data acquired by the acquisition means.
The sound data processing apparatus according to any one of claims 1 to 3 , wherein the determination means determines the priority based on the viewpoint information acquired by the third acquisition means. The priority is determined so that a higher priority is set in the area including the position of the viewpoint corresponding to the viewpoint information as compared with the area not including the position of the viewpoint corresponding to the viewpoint information. The sound data processing apparatus according to claim 6 . The priority is set higher in the area including the position of the intersection of the line of sight and the object corresponding to the viewpoint information than in the area not including the position of the intersection of the line of sight and the object corresponding to the viewpoint information. The sound data processing apparatus according to claim 6 , wherein the sound data processing apparatus is determined to be used. Further having a fourth acquisition means for acquiring information on a listening point corresponding to a reproduced sound based on the sound data acquired by the acquisition means.
The sound data processing according to any one of claims 1 to 3 , wherein the determination means determines the priority based on the information regarding the listening point acquired by the fourth acquisition means. apparatus. The ninth aspect of the present invention is characterized in that the priority is determined so that a higher priority is set in the area including the listening point position than in the area not including the listening point position. Sound data processing device. Further, it has a determination means for determining the occurrence position of a predetermined event in the sound collection target space based on the sound data acquired by the acquisition means.
The sound data processing apparatus according to any one of claims 1 to 3 , wherein the determination means determines the priority based on the generation position determined by the determination means. The claim is characterized in that the priority is determined so that a higher priority is set in the area including the occurrence position of the predetermined event as compared with the area not including the occurrence position of the predetermined event. Item 11. The sound data processing apparatus according to item 11 . A setting means for setting the processing order of the plurality of areas based on the priority determined by the determination means, and
An execution means for executing processing for obtaining processed data corresponding to each of the plurality of areas according to the processing order set by the setting means, and an execution means.
Have more
Before Kide force means according to any one of claims 1 to 12, characterized in that outputs the first processed data obtained by processing included in by Ru processing execution by the execution unit Sound data processing device. The claim means that the determination means determines the priority based on the position of a microphone that collects sound in the sound collection target space in order to generate sound data acquired by the acquisition means. The sound data processing apparatus according to any one of 1 to 13 . The output by the pre Kide force means to the first processed data, data of the sound corresponding to said one or more areas that are selected based on the determined priority by the determination means is included, the The sound data processing apparatus according to any one of claims 1 to 14 , wherein sound data corresponding to an area different from one or more areas is not included. The determination means according to any one of claims 1 to 15 , wherein the determination means determines the priority based on at least one of the volume corresponding to each area and the influence of masking corresponding to each area. The sound data processing device described. The acquisition process to acquire the sound data generated by collecting the sound in the sound collection target space,
A determination step of determining at least one priority of a plurality of areas in the sound collection target space, and
It has a processed data obtained by processing the sound data acquired in the acquisition step and an output step of outputting the processed data corresponding to each of the plurality of areas.
In the output step, when the processing time related to the processing for obtaining the processed data corresponding to each of the plurality of areas is longer than the predetermined threshold value, the predetermined sound data acquired by the acquisition step is described. It is the first processed data obtained by performing the processing of the processing time equal to or less than the threshold value, and corresponds to one or more areas selected based on the priority determined in the determination step among the plurality of areas. The second processed data obtained by processing the sound data acquired in the acquisition step after the first processed data is output, which is one or more of the plurality of areas. audio data processing method characterized in that said second processed data corresponds to another one or more areas comprising different areas are outputted. It said second processed data, the sound data according to claim 17, characterized in that it is obtained by the sound data obtained by the obtaining step performs a process of processing time than the predetermined threshold Processing method. Program for causing to function as each means of the sound data processing apparatus according to computer what Re one of claims 1-16.

Images