JP6821390B2 - Sound processing equipment, sound processing methods and programs - Google Patents

Description

The present invention relates to an acoustic processing apparatus, an acoustic processing method and a program.

A technique for dividing a space into a plurality of areas and acquiring sound for each area is known (Patent Document 1).

Japanese Unexamined Patent Publication No. 2014-722708

However, when the audio of an area divided into a plurality of areas is processed in real time and an attempt is made to broadcast the audio, there is a possibility that the processing or transmission cannot be completed in time, data is lost, and the audio is interrupted.

The present invention has been made in view of the above problems, and an object of the present invention is to provide a technique capable of improving processing efficiency in a configuration in which audio is acquired from a plurality of areas in which a space is divided to generate a reproduction signal. And.

In order to achieve the above object, the sound processing apparatus according to the present invention has the following configurations. That is,
An acquisition means for acquiring a sound collection signal based on sound collection by a plurality of microphones that collect sound in the sound collection area, and
A determination means for determining the position and size of a plurality of partial areas in the sound collection area based on the position of the object detected in the sound collection area.
An extraction means for extracting a plurality of area acoustic signals corresponding to the plurality of partial areas determined by the determination means from the sound collection signal acquired by the acquisition means, and an extraction means.
It has a generation means for generating a reproduction acoustic signal according to the position and orientation of a designated virtual listening point by acoustic processing using two or more area acoustic signals extracted by the extraction means.
The determining means makes the size of the partial area including the position of the object detected in the sound collecting area smaller than the size of the partial area including the position of the object detected in the sound collecting area. , The feature is to determine the size of the plurality of partial areas .

According to the present invention, it is possible to improve the efficiency of processing in a configuration in which audio is acquired from a plurality of areas in which a space is divided to generate a reproduction signal.

The block diagram which shows the structure of the audio signal processing apparatus. Explanatory drawing of separation area control. Explanatory drawing which shows the time change of the separation area control. The block diagram which shows the hardware composition of the audio signal processing apparatus. A flowchart showing audio signal processing. The figure explaining the display device of the separation area control. The figure explaining the sound system. A block diagram showing details of the configuration of an audio system. Explanatory drawing of separation area control. A flowchart showing audio signal processing.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. It should be noted that the following embodiments do not limit the present invention, and not all combinations of features described in the present embodiment are essential for the means for solving the present invention. The same configuration will be described with the same reference numerals.

<Embodiment 1>
In the first embodiment (Embodiment 1) of the present invention, a configuration for reducing the number of separation areas to be used when the sound source separation process is not in time for real-time reproduction will be described.

(Audio signal processor)
FIG. 1 is a block diagram showing a configuration of an audio signal processing device 100. The voice signal processing device 100 collects voice from a predetermined space area by a microphone array, separates the collected voice into a plurality of voice signals based on a plurality of separated areas, performs voice processing, and mixes them to reproduce a signal for reproduction. Is a device that produces. The audio signal processing device 100 includes a microphone array 111, a sound source separation unit 112, a separation area control unit 113, an audio signal processing unit 114, a storage unit 115, a real-time reproduction signal generation unit 116, and a replay reproduction signal generation unit 117. ..

The microphone array 111 is composed of a plurality of microphones. The microphone array 111 collects the sound of the space in charge with a microphone. Since each microphone constituting the microphone array 111 collects sound, the sound collected by the microphone array 111 is a multi-channel signal composed of a plurality of sounds collected by each microphone as a whole. The microphone array 111 collects the sound in the space with a microphone, performs A / D conversion (analog / digital conversion) on the collected signal, and then outputs the sound to the sound source separation unit 112.

The sound source separation unit 112, separation area control unit 113, audio signal processing unit 114, real-time playback signal generation unit 116, and replay playback signal generation unit 117 are arithmetic processing units such as a CPU (central processing unit), DSP, and MPU. Consists of. DSP is an abbreviation for Digital Signal Processor, and MPU is an abbreviation for Micro-processing unit.

When the space in which the microphone array 111 is in charge of collecting sound is divided into N (N> 1) areas (hereinafter, “separation area”), the sound source separation unit 112 receives signals input from the microphone array 111. Performs sound source separation processing to separate into audio in the separation area. As described above, the signal input from the microphone array 111 is a multi-channel signal composed of a plurality of voices collected by each microphone. Therefore, based on the positional relationship between each microphone constituting the microphone array 111 and the separation area to be collected, the sound signal collected by the microphone is phase-controlled and weighted and added to obtain the sound in an arbitrary separation area. It can be reproduced. In this embodiment, an example in which the arrangement of the separation area is predetermined will be described. The sound source separation unit 112 performs sound source separation processing so as to divide the space into N (N> 1) areas using the signal input from the microphone array 111. The separation process is performed for each processing frame, that is, at predetermined time intervals. For example, beamforming processing is performed at predetermined time intervals, and sound for each area is acquired. The separately acquired voice is output to the voice signal processing unit 114 and the storage unit 115.

The separation area control unit 113 controls the division of a certain space in which the microphone array collects sound into a plurality of separation areas according to the processing load for separating the sound source and generating the reproduction signal. Specifically, the arrangement and number of a plurality of separation areas are controlled. For example, if the processing load of the processing device is large and the processing is not in time for real-time playback when the sound source separation processing of all areas is performed, the separation area control unit 113 combines the sound source separation areas performed by the sound source separation unit 112 to obtain the number of areas. To reduce. For example, when the processing is sufficiently in time, the sound collecting space A1 is finely divided into 8 × 8 = 64 separation areas A2 as shown in FIG. 2A, for example. When the processing is not in time, for example, it is determined whether or not the sound in the area is above the predetermined level in the processing of the previous frame, and the area below the predetermined level is the area as shown in FIG. 2 (B). To reduce the number of areas. Voices above a predetermined level are more likely to be significant voices, while voices below a predetermined level are more likely to be insignificant voices such as noise. Therefore, by preferentially assigning a fine separation area to the area where the voice is above the predetermined level, the significant voice is faithfully reproduced, and in the area below the predetermined level, the separation area is integrated to speed up the processing. Can be transformed into.

An example of the area separation size change is shown in FIG. FIG. 3D shows a state in which area control is performed based on the processing load (area control ON) or not (area control OFF). fp to fp + 7 represent frame numbers. FIG. 3C shows a state in which the sound level separated for each area is equal to or higher than a predetermined level (with sound) or lower than a predetermined level (without sound). Here, the frames fp + 1 and fp + 3 are in a state of having sound. FIG. 3B shows the division size of the most finely divided area. This division size represents the area of the minimum area when the area of the sound collecting space A1 is 1. For example, in the frame fp, the space is equally divided into 64 areas, so the minimum area size is 1/64. FIG. 3A shows how each frame is separated into a plurality of areas.

The time from fp + 1 to fp + 6 is the time when the processing load is large and the number of areas needs to be reduced. In the frame fp, the audio level did not exceed the predetermined value in any area (no sound in FIG. 3C). Therefore, in the frame fp + 1, the area size is a large area in which one side is 1/2 of the sound collecting space and the sound collecting space is divided into four (1/4 in FIG. 3B).

In frame fp + 1, there was an area where the sound level exceeded a predetermined value (with sound in FIG. 3C). Therefore, in the frame fp + 2, one side of the area A3 where the sound was present is again divided into a small area of 1/8 of the sound collecting space A1 (1/64 in FIG. 3B).

Subsequently, in the frame fp + 2, the audio level did not exceed the predetermined value in any area (no sound in FIG. 3C). Therefore, in the frame fp + 3, some areas are combined and one side is divided into an area having an intermediate size of 1/4 of the sound collecting space (1/16 in FIG. 3B).

In frame fp + 3, there was an area where the sound level exceeded a predetermined value (with sound in FIG. 3C). Therefore, in the frame fp + 4, the area A3 where the sound was present is once again divided into a small area with one side of 1/8 of the sound collecting space (1/64 in FIG. 3B).

In frames fp + 4 and fp + 5, the audio level did not exceed the predetermined value in any area (no sound in FIG. 3C). Therefore, the areas are combined, and in the frame fp + 6, one side is 1/2 of the sound collecting space, and the sound collecting space is divided into four large areas.

In this way, the separation area control unit 113 increases or decreases the number of separation areas depending on the presence or absence of voice detection. Here, the separation area control unit 113 has described an example in which the sound source separation areas are combined to reduce the number of areas. However, in reality, the sound source separation unit 112 may have a filter for beamforming that separates into a plurality of area sizes, and the separation area control unit 113 may control the filter to be used.

Further, the separation area control unit 113 manages the area information combined with the frame for the area combined by the separation area control as a separation area control list. For example, when four areas are combined in the frame fq, the frame fq and the four areas are managed as a list. Here, the areas are given an ID or the like in advance so that they can be distinguished. The separation area control unit 113 instructs the sound source separation unit 112 to separate the sound sources of each area for the area combined with the frame recorded in the separation area control list according to the reduction in the processing load. put out. When sound source separation is performed, the frame and area are deleted from the list.

The audio signal processing unit 114 processes the audio signal for each frame and area. The processing performed by the audio signal processing unit 114 is, for example, delay correction processing for correcting the influence of the distance between the area and the sound collecting device, gain correction processing, echo cancellation, and the like.

The storage unit 115 is a storage device such as an HDD (hard disk drive), an SSD (solid state drive), or a memory. The storage unit 115 records the signals of all the audio channels of the frame whose separation area is controlled by the sound source separation unit 112 and the signals processed by the audio signal processing unit 114 together with the time information.

The real-time reproduction signal generation unit 116 generates and outputs a real-time reproduction signal by mixing the sound for each area obtained from the sound source separation unit 112 within a predetermined time from the sound collection. For example, a sound source is obtained by acquiring information on a virtual listening point and a virtual listener's orientation (hereinafter, simply referred to as a listening point and a listener's orientation) in a space that changes with time from the outside, and information on a playback environment. Mixing. Here, the playback environment is an environment related to the configuration of the playback device, such as whether the playback device that reproduces the signal generated by the real-time playback signal generation unit 116 is a speaker (stereo, surround, or other multi-channel) or headphones. .. That is, in the mixing of sound sources, the audio signals of each divided area are combined and converted according to the environment such as the number of channels of the playback device.

When the replay reproduction is requested, the replay reproduction signal generation unit 117 acquires the data at the corresponding time from the storage unit 115, performs the same processing as the real-time reproduction signal generation unit 116, and outputs the data.

FIG. 4 is a block diagram showing a hardware configuration example of the audio signal processing device 100. The audio signal processing device 100 is realized by, for example, a personal computer (PC), an embedded system, a tablet terminal, a smartphone, or the like.

In FIG. 4, the CPU 990 is a central processing unit, which cooperates with other components based on a computer program to control the operation of the entire audio signal processing device 100. The ROM 991 is a read-only memory that stores a basic program, data used for basic processing, and the like. The RAM 992 is a writable memory and functions as a work area or the like of the CPU 990.

The external storage drive 993 realizes access to the recording medium, and can load computer programs and data stored in the medium (recording medium) 994 such as a USB memory into the system. The storage 995 is a device that functions as a large-capacity memory such as an SSD (solid state drive). Various computer programs and data are stored in the storage 995.

The operation unit 996 is a device that receives instructions and command inputs from the user, and corresponds to a keyboard, a pointing device, a touch panel, and the like. The display 997 is a display device that displays a command input from the operation unit 996 and a response output of the audio signal processing device 100 to the command. The interface (I / F) 998 is a device that relays the exchange of data with an external device. Further, the microphone array 111 is connected to the audio signal processing device 100 via the interface 998. The system bus 999 is a data bus that controls the flow of data in the audio signal processing device 100.

Each functional element of FIG. 1 is realized by the CPU 990 controlling the entire device based on a computer program. It should be noted that the software that realizes the same functions as each of the above devices can be configured as a substitute for the hardware device.

(Processing procedure)
Subsequently, the procedure of the processing executed by the audio signal processing device 100 will be described with reference to FIG. 5 (A) to 5 (C) are flowcharts showing a procedure of processing executed by the audio signal processing device 100 of the present embodiment.

FIG. 5A is a flow from sound collection to generation of a real-time reproduction signal. First, the microphone array 111 collects sounds in space (S111). The picked-up audio signals of each channel are output to the sound source separation unit 112.

Subsequently, the separation area control unit 113 determines whether or not the sound source separation is in time for real-time reproduction from the viewpoint of the processing load (S112). As described with reference to FIG. 3, this process is performed based on the presence or absence of a predetermined level of voice and the like.

When it is determined that the real-time reproduction is not in time (NO in S112), the separation area control unit 113 controls the number of areas so that the sound source separation area is reduced (S113). Specifically, for example, the number of separated areas is reduced by integrating less important separated areas such as areas where voice above a certain level is not detected. Then, the information on what kind of area is to be separated is output to the sound source separation unit 112. Further, the separation area control unit 113 creates a separation area control list.

Subsequently, the storage unit 115 records the audio signal of the frame whose separation area is controlled (S114).

When it is determined that the sound source is in time for real-time reproduction, or after recording in S114, the sound source separation unit 112 performs sound source separation (S115). That is, the voice in each separation area is synthesized based on the multi-channel signal collected in S111. As described above, the sound in the separation area is added by phase-controlling and weighting the sound signals collected by each microphone based on the relationship between the microphones constituting the microphone array 111 and the position of the separation area. It can be reproduced. The audio signal for each separated area is output to the audio signal processing unit 114.

Subsequently, the audio signal processing unit 114 processes the audio signal for each separation area (S116). As described above, the processing by the audio signal processing unit 114 includes, for example, delay correction processing for correcting the influence of the distance between the separation area and the sound collecting device, gain correction processing, noise processing by echo cancellation, and the like. The processed audio signal is output to the real-time reproduction signal generation unit 116 and the storage unit 115.

Subsequently, the real-time reproduction signal generation unit 116 mixes the real-time reproduction audio (S117). In mixing, signals are synthesized and converted so that they can be played back according to the specifications of the playback device (for example, the number of channels). The audio mixed for real-time playback is output as an external playback device or a broadcast signal.

Subsequently, the storage unit 115 records the voice of each area (S118). The audio signal for replay reproduction is created using the audio for each area of the storage unit 115.

Next, the process when the process is not in time for the real-time reproduction in S112 of FIG. 5 (A) (NO in S112) will be described with reference to FIG. 5 (B).

When the load of the processing device is lower than the predetermined value, the separation area control unit 113 reads data from the storage unit 115 based on the separation area control list (S121).

Subsequently, the sound source separation processing is performed again for the area before the combination for the area where the sound source separation is performed by combining the areas described in the separation area control list (S122). The processed audio signal is output to the audio signal processing unit 114. The corresponding frames and areas are deleted from the isolated area control list when processing is complete. Since S123 is the same as S116, detailed description thereof will be omitted.

Subsequently, the storage unit 115 overwrites the input area audio signal with the previous data and records it (S124).

Next, a processing flow when a replay is requested will be described with reference to FIG. 5 (C). When the replay is requested, the replay playback signal generation unit 117 reads out the audio signal for each area corresponding to the replay time from the storage unit 115 (S131).

Subsequently, the replay playback signal generation unit 117 mixes the replay playback audio (S132). The audio mixed for replay playback is output as an external playback device or a broadcast signal.

As described above, the separation area is controlled according to the processing load. That is, in a certain space, a region in which at least one of processing of sound source separation and reproduction signal generation has a larger processing load is controlled to be divided into finer separation areas. Therefore, the degree of separation of the area where the volume level is lower than the predetermined value is lowered, but the area where the volume level is higher than the predetermined value is in time for the real-time reproduction signal generation with high resolution. Furthermore, when the processing load is light, the separated area is separated so that data with sufficient resolution can be obtained during replay.

Although the example in which the microphone array 111 is composed of a microphone has been described in the present embodiment, it may be a set with a structure such as a reflector. Further, the microphone used in the microphone array 111 may be omnidirectional, may be a directional microphone, or may be a mixture thereof.

In the present embodiment, the sound source separation unit 112 has described an example in which sound is picked up for each area by using beamforming, but other sound source separation may be used. For example, the power spectral density (PSD) for each area may be estimated, and separation by a Wiener filter may be performed based on the estimated PSD.

In the present embodiment, the separation area control unit 113 has described an example in which the separation area is controlled depending on whether or not the sound level of the area is equal to or higher than a predetermined value, but other determination criteria may be provided. For example, even when the same voice is used, the presence or absence of the feature amount may be determined by providing a configuration for detecting the feature amount of the sound instead of the level. Specifically, when voices showing predetermined characteristics are detected, such as when screams, gunshots, ball sounds, car sounds, etc. are included in the voice feature analysis, the separation area is reduced. Then, the detailed sound may be reproduced. Further, for example, a space including all areas may be photographed, and the separation area may be controlled from the photographed moving image. For example, a specific subject such as a person, an animal, or a marker may be detected from the moving image, and the size of the separation area around the subject may be controlled to be smaller.

Further, in live broadcasting such as television broadcasting, a system is generally known that broadcasts with a certain delay of about several seconds to several minutes from the actual shooting in order to adjust the time and respond to an unexpected situation. When such a system is used, the separation area control unit 113 may control the separation order according to the events included in the video and audio for the delay time. For example, if there is a delay of 2 minutes in live sports broadcasting, a separation area may be set from the 2-minute game development to separate the sound sources. For example, when a goal is scored in a game such as soccer, the movement of the player or ball who scored the goal may be detected from a two-minute video, and the separation area around the trajectory may be set to be finer. .. On the other hand, for areas where players and balls do not enter, it is advisable to set the separation area to be rough.

Further, in the present embodiment, the separation area control unit 113 reduces the number of areas as much as possible, but the number of areas may be calculated according to the processing load to reduce the minimum number of necessary areas.

Further, in the present embodiment, the separation area control unit 113 controls the separation area by using the voice level of the previous frame, but the separation area may be controlled by using the information of the processing frame. That is, if the sound level of the separated area is equal to or higher than a predetermined value, the separation area control unit 113 instructs the sound source separation unit 112 to perform sound source separation in the area further divided into the areas. The separation area control unit 113 and the sound source separation unit 112 repeat this process until the area becomes smaller to a predetermined size. In this way, it is possible to prevent the separation area control from being delayed by one frame. However, since this method increases the amount of processing as the number of sound sources increases, it is better to use it in situations where it is known in advance that the number of sound sources is small, or to limit the number of repetitions within the allowable range of the processing load. ..

In the present embodiment, the audio signal processing unit 114 is supposed to perform delay correction processing, gain correction processing, and echo cancellation, but other processing may also be performed. For example, noise removal processing for each area may be performed.

In the present embodiment, an example in which the replay reproduction signal generation unit 117 and the real-time reproduction signal generation unit 116 perform the same processing has been described. However, the replay reproduction signal generation unit 117 and the real-time reproduction signal generation unit 116 may be mixed differently. For example, in the real-time playback signal generation unit 116, audio with a coarse separation area may be input, so for example, an area with a large area size may have a lower mixing level depending on whether or not processing has been performed. You may.

Further, although not shown in the present embodiment, display control for displaying the status of area control on the display device may be performed as shown in FIG. For example, a time bar 501, a time cursor 502, an area division display 503, an area division ratio display 504, and the like are displayed on the display screen. Here, the time bar 501 is a bar representing the recording time up to the present, and the position of the time cursor 502 represents the time on the display screen. The area division display 503 shows the area division state at the time pointed to by the time cursor 502. The image showing the divided state may be superimposed on an image of the actual space, a CG that reproduces the actual space, or the like. The area division ratio display 504 displays the ratio for each size of the area division. Alternatively, a screen as shown in FIG. 3 may be displayed. By displaying in this way, it is possible to intuitively understand the state of area division. Further, this display device may further include an input device such as a touch panel. For example, the user may be able to select an area having a large area size by touch or the like, and set it so that the process of finely dividing the area is preferentially performed.

<Embodiment 2>
A second embodiment (Embodiment 2) of the present invention relates to an acoustic system in which a plurality of users set listening points and reproduce sound according to the listening points with a reproduction device.

(Acoustic system)
FIG. 7 is a block diagram showing the configuration of the acoustic system 20. The sound system 20 includes a sound collecting unit 21, a reproduction signal generation unit 22, and a plurality of reproduction units 23. The sound collecting unit 21, the reproduction signal generation unit 22, and the plurality of reproduction units 23 transmit and receive data to and from each other through a wired or wireless transmission path. The transmission path between the sound collecting unit 21, the reproduction signal generation unit 22, and the reproduction unit 23 is realized by a dedicated communication path such as a LAN, but may be via a public communication network such as the Internet.

8 (A) is a block diagram showing the configuration of the sound collecting unit 21, FIG. 8 (B) is a block diagram showing the configuration of the reproduced signal generation unit 22, and FIG. 8 (B) is a block diagram showing the configuration of the reproduced unit 23. is there. The sound collecting unit 21 of FIG. 8A includes a microphone array 111 and a sound collecting signal transmitting unit 211. Since the microphone array 111 is the same as that of the first embodiment, detailed description thereof will be omitted. The sound pick-up signal transmission unit 211 transmits the microphone signal input from the microphone array 111.

The reproduction signal generation unit 22 of FIG. 8B is a sound source separation unit 112, a separation area control unit 113, an audio signal processing unit 114, a storage unit 115, a sound collection signal reception unit 221 and a listening point reception unit 222, and a reproduction signal. It includes a generation unit 223 and a reproduction signal transmission unit 224. Since the sound source separation unit 112, the audio signal processing unit 114, and the storage unit 115 are substantially the same as those in the first embodiment, detailed description thereof will be omitted.

The separation area control unit 113 controls an area for sound source separation of the sound source separation unit 112 based on a plurality of listening points input from the listening point receiving unit 222, which will be described later. Here, the listening point is information including the position and orientation of a virtual listener in the space set by the user, and the time. For example, the separation area control unit 113 monitors the processing load of the reproduction signal generation unit 22, and controls the area so as to reduce the number of separation areas based on the distribution of listening points when the load becomes large. For example, it is assumed that the positions of listeners set by the user listening in real time are distributed as shown in FIG. 9A. In that case, as shown in FIG. 9B, the area around the area where more listening points are set is finely divided, and the area is controlled so as to roughly divide the area where there are few listening points.

In addition, when the user has specified a listening point at a past time, that is, when replay is requested, it is determined whether or not sound source separation processing is necessary based on the status of the separation area at that time and the specified viewpoint. If necessary, the sound source is separated according to the processing load. For example, if the area is not controlled at the specified time, or if the area is controlled but the sound source is separated in a sufficiently fine area around the listening point specified this time, it is necessary to perform the separation again. Absent. On the other hand, when the area is controlled at the specified time and the area around the listening point specified this time is roughly divided, the separation area control unit 113 separates the sound sources so as to finely divide the area around the listening point. A control signal is output to unit 112.

The sound collecting signal receiving unit 221 receives the sound collecting signal from the sound collecting unit 21. The listening point receiving unit 222 receives listening points from each of the plurality of reproducing units 23. The received listening point is output to the separation area control unit 113 and the reproduction signal generation unit 223. The reproduction signal generation unit 223 has a function of combining the real-time reproduction signal generation unit 116 and the replay reproduction signal generation unit 117 of the first embodiment. A playback signal is generated according to the position and orientation of the listener input from the listening point receiving unit 222 and the time. If the input time is real-time, it is the same as the real-time reproduction signal generation unit 116, and if the time is in the past, it is the same as the replay reproduction signal generation unit 117. The audio signal generated for each listening point is output to the reproduction signal transmission unit 224. The reproduction signal transmission unit 224 outputs the received audio signal for each listening point to each reproduction unit 23.

The reproduction unit 23 of FIG. 8C includes a listening point input unit 231, a listening point transmitting unit 232, a reproduction signal receiving unit 233, and a speaker 234. The listening point input unit 231 is an input device capable of setting the position of a virtual listener and the orientation of the listener in the space where the user is collecting the time and sound. The listening point input unit 231 is realized by a keyboard, a pointing device, a touch panel, or the like. The set listening point is output to the listening point transmission unit 232.

The listening point transmitting unit 232 outputs the listening point set by the user to the listening point receiving unit 222. The reproduction signal receiving unit 233 receives the audio signal corresponding to the listening point set by the listening point input unit 231 and outputs the audio signal to the speaker 234. The speaker 234 D / A-converts the input audio signal and emits the sound from the speaker.

(Processing procedure)
Subsequently, the procedure of the process executed by the acoustic system 20 will be described with reference to 94. 10A to 10C are flowcharts showing a procedure of processing executed by the sound system 20 of the present embodiment.

As shown in FIG. 10A, first, sound is picked up in space in the microphone array 111 (S201). The picked-up voice is output to the pick-up signal transmission unit 211. Subsequently, the sound collection signal is transmitted from the sound collection signal transmission unit 211 of the sound collection unit 21, and is received by the sound collection signal reception unit 221 of the reproduction signal generation unit 22 (S202). The received sound pick-up signal is output to the sound source separation unit 112. Subsequently, listening points are input to the listening point input units 231 of the plurality of playback units 23 (S203). The input listening point is output to the listening point transmission unit 232.

Subsequently, the listening point is transmitted from the listening point transmitting unit 232 and received by the listening point receiving unit 222 of the reproduction signal generation unit 22 (S204). The plurality of received listening points are output to the separation area control unit 113 and the reproduction signal generation unit 223.

Subsequently, the separation area control unit 113 determines whether or not the processing is in time for real-time reproduction (S205). If it is determined that the real-time reproduction is in time (YES in S205), the process proceeds to S208, and if it is determined that the real-time reproduction is not in time (NO in S205), the process proceeds to S206.

In S206, the separation area control unit 113 controls the separation area. That is, in S206, a plurality of areas are combined and control for reducing the number of areas is output to the sound source separation unit 112. Furthermore, a separation area control list is generated, and control information of the separation area is managed. Subsequently, when the area is controlled by the sound source separation unit 112, the sound collection signal of the frame is output to the storage unit 115, and the sound collection signal input by the storage unit 115 is recorded (S207). Then, the process proceeds to S208.

In S208, the sound source separation unit 112 separates the sound sources for each area. The audio signal for each separated area is output to the audio signal processing unit 114.

Subsequently, the audio signal processing unit 114 processes the audio signal (S209). The processed audio signal is output to the storage unit 115.

Subsequently, the voice signal for each area processed by the storage unit 115 is recorded (S210). Subsequently, the reproduction signal generation unit 223 acquires audio for each area according to the times of a plurality of listening points input from the listening point receiving unit 222 from the storage unit 115, and mixes the audio for reproduction for each listening point. It is done (S211). The plurality of mixed reproduction signals are output to the reproduction signal transmission unit 224.

Subsequently, a plurality of reproduction signals generated for each listening point are transmitted from the reproduction signal transmitting unit 224, and the reproduction signal corresponding to the input listening point is received by the reproduction signal receiving unit 233 of each reproduction unit 23 ( S212). Finally, the reproduced signal received by the reproduced signal receiving unit 233 is reproduced from the speaker (S213).

Next, when it is determined in S205 of FIG. 10 (A) that the process is not in time using FIG. 10 (B), the process when the number of areas is reduced will be described.

When the processing load falls below a predetermined value, the separation area control unit 113 refers to the separation area control list and determines the time (frame) and area for separation (S221). Information on the area to be separated and the time is output to the sound source separation unit 112.

Subsequently, the sound source separation unit 112 reads out the sound collection signal based on the time information input from the storage unit 115 (S222). Since S223 to S225 are the same as S208 to S210, detailed description thereof will be omitted.

As described above, the number of areas is reduced by combining the separated areas based on the processing load and the distribution of a plurality of listening points. Therefore, an important audio signal can be faithfully reproduced, and processing can be streamlined to realize real-time processing. Furthermore, during replay, it is possible to generate a playback signal using the separated sound even for areas where transmission was not in time in real time.

In the present embodiment, the reproduction unit 23 has the same configuration for the sake of simplicity, but the configuration may be different. Although not described in this embodiment, it may be used in combination with a free viewpoint image generation system that generates a free viewpoint image. For example, a space substantially the same as the space in which sound is picked up by a plurality of imaging devices is imaged from all directions, and a free viewpoint image is generated from the captured images. In that case, the listening point may be calculated from the viewpoint, or the free viewpoint video may be generated in conjunction with the listening point.

In the present embodiment, the reproduction signal generation unit 223 is configured in the reproduction signal generation unit 22, but may be configured in the reproduction unit 23. In the present embodiment, the separation area control unit 113 determines the separation area using only the positions of a plurality of listeners, but as shown in FIG. 9C, the separation area is in front of the listening direction according to the direction of the listeners. The existing area may be finely divided and the rear part may be roughly divided.

When area control is performed in the present embodiment, the listening position that can be input by the listening point input unit 231 may be limited. In the present embodiment, the reproduction unit 23 is treated uniformly, but may have different weights for each listening point in order to control the separation area. Further, as in the first embodiment, a display device for displaying the status of area control and an input device for instructing separation area control may be provided.

In each embodiment of the present invention, the entire space is picked up by controlling the number of sound source separation areas even in real-time reproduction in which the time until reproduction is limited, and the resolution of important areas is maintained. Can be played.

<Other Embodiments>
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

100: Audio signal processing device, 111: Microphone array, 112: Sound source separation unit, 113: Separation area control unit, 114: Audio signal processing unit, 115: Storage unit, 116: Real-time playback signal generation unit, 117: Replay playback Signal generator

Claims (15)

An acquisition means for acquiring a sound collection signal based on sound collection by a plurality of microphones that collect sound in the sound collection area, and
A determination means for determining the position and size of a plurality of partial areas in the sound collection area based on the position of the object detected in the sound collection area.
An extraction means for extracting a plurality of area acoustic signals corresponding to the plurality of partial areas determined by the determination means from the sound collection signal acquired by the acquisition means, and an extraction means.
It has a generation means for generating a reproduction acoustic signal according to the position and orientation of a designated virtual listening point by acoustic processing using two or more area acoustic signals extracted by the extraction means.
The determining means makes the size of the partial area including the position of the object detected in the sound collecting area smaller than the size of the partial area including the position of the object detected in the sound collecting area. , A sound processing apparatus, characterized in that the size of the plurality of partial areas is determined . An acquisition means for acquiring a sound collection signal based on sound collection by a plurality of microphones that collect sound in the sound collection area, and
A determination means for determining the position and size of a plurality of partial areas in the sound collection area based on the position of the object detected in the sound collection area.
An extraction means for extracting a plurality of area acoustic signals corresponding to the plurality of partial areas determined by the determination means from the sound collection signal acquired by the acquisition means, and an extraction means.
It has a generation means for generating a reproduction acoustic signal according to the position and orientation of a designated virtual listening point by acoustic processing using two or more area acoustic signals extracted by the extraction means.
The determining means is a portion in which the size of the partial area including the first number of objects detected in the sound collecting area includes a second number of objects detected in the sound collecting area, which is smaller than the first number. An audio processing device characterized in that the size of the plurality of partial areas is determined so as to be smaller than the size of the area. An acquisition means for acquiring a sound collection signal based on sound collection by a plurality of microphones that collect sound in the sound collection area, and
A determination means for determining the position and size of a plurality of partial areas in the sound collection area based on the position of the object detected in the sound collection area.
An extraction means for extracting a plurality of area acoustic signals corresponding to the plurality of partial areas determined by the determination means from the sound collection signal acquired by the acquisition means, and an extraction means.
It has a generation means for generating a reproduction acoustic signal according to the position and orientation of a designated virtual listening point by acoustic processing using two or more area acoustic signals extracted by the extraction means.
The generation means is an acoustic processing apparatus characterized in that the reproduction acoustic signal is generated by synthesizing the two or more area acoustic signals based on the position and orientation of the virtual listening point. An acquisition means for acquiring a sound collection signal based on sound collection by a plurality of microphones that collect sound in the sound collection area, and
A determination means for determining the position and size of a plurality of partial areas in the sound collection area based on the position of the object detected in the sound collection area.
An extraction means for extracting a plurality of area acoustic signals corresponding to the plurality of partial areas determined by the determination means from the sound collection signal acquired by the acquisition means, and an extraction means.
A generation means for generating a reproduction acoustic signal according to the position and orientation of a designated virtual listening point by acoustic processing using two or more area acoustic signals extracted by the extraction means.
An audio processing apparatus comprising: a display control means for displaying an image showing an arrangement of the plurality of partial areas determined by the determination means on a display unit. The sound processing apparatus according to any one of claims 1 to 4, wherein the determination means determines the number of the plurality of partial areas included in the sound collecting region based on the position of the object. .. Claims 1 to 5 are characterized in that the determination means determines the number of the plurality of partial areas included in the sound collecting region based on the processing load of at least one of the extraction means and the generation means. The sound processing apparatus according to any one of the above. The acoustic processing according to any one of claims 1 to 6 , further comprising a detection means for detecting the position of an object in the sound collection region based on the sound collection signal acquired by the acquisition means. apparatus. The sound according to any one of claims 1 to 7 , further comprising a detecting means for detecting the position of an object in the sound collecting area based on an image obtained by photographing the sound collecting area. Processing equipment. An acquisition means for acquiring a sound collection signal based on sound collection by a plurality of microphones that collect sound in the sound collection area, and
A determining means for determining the position and size of a plurality of partial areas within the sound collecting region based on at least one of the positions and orientations of the designated virtual listening points.
An extraction means for extracting a plurality of area acoustic signals corresponding to the plurality of partial areas determined by the determination means from the sound collection signal acquired by the acquisition means, and an extraction means.
Acoustics characterized by having a generation means for generating a reproduction acoustic signal according to the position and orientation of the virtual listening point by acoustic processing using two or more area acoustic signals extracted by the extraction means. Processing equipment. The determination means determines the size of the plurality of partial areas so that the size of the partial area including the position of the virtual listening point is smaller than the size of the partial area not including the position of the virtual listening point. 9. The sound processing apparatus according to claim 9 . The determining means of the plurality of partial areas so that the size of the partial area located in the direction in which the virtual listening point is directed is smaller than the size of the partial area not located in the direction in which the virtual listening point is directed. The sound processing apparatus according to claim 9 , wherein the size is determined. The acquisition process of acquiring the sound collection signal based on the sound collection by multiple microphones that collect the sound in the sound collection area, and
A determination step of determining the position and size of a plurality of partial areas in the sound collection area based on the position of the object detected in the sound collection area.
An extraction step of extracting a plurality of area acoustic signals corresponding to each of the plurality of partial areas determined in the determination step from the sound collection signal acquired in the acquisition step, and an extraction step.
It has a generation step of generating a reproduction acoustic signal according to the position and orientation of a designated virtual listening point by acoustic processing using two or more area acoustic signals extracted in the extraction step.
In the determination step, the size of the partial area including the position of the object detected in the sound collecting area is smaller than the size of the partial area including the position of the object detected in the sound collecting area. , A sound processing method comprising determining the size of the plurality of partial areas . The acquisition process of acquiring the sound collection signal based on the sound collection by multiple microphones that collect the sound in the sound collection area, and
A determination step of determining the position and size of a plurality of partial areas in the sound collecting region based on at least one of the positions and orientations of the designated virtual listening points.
An extraction step of extracting a plurality of area acoustic signals corresponding to each of the plurality of partial areas determined in the determination step from the sound collection signal acquired in the acquisition step, and an extraction step.
Acoustics characterized by having a generation step of generating a reproduction acoustic signal according to the position and orientation of the virtual listening point by acoustic processing using two or more area acoustic signals extracted in the extraction step. Processing method. In the determination step, the size of the plurality of partial areas is determined so that the size of the partial area including the position of the virtual listening point is smaller than the size of the partial area not including the position of the virtual listening point. 13. The sound processing method according to claim 13 . A computer program for causing a computer to function as each means included in the sound processing apparatus according to any one of claims 1 to 11 .

Images