JP5143656B2 - Sound collection system and sound display method - Google Patents

Description

  The present invention relates to a sound collection system, and more particularly to a sound collection system capable of displaying the volume of collected sound.

  In a telephone conference system and a video conference system, voice collected from a voice input unit at a certain site is transmitted to another site after echo canceller or noise removal processing is executed. At this time, the transmitted own voice is not necessarily transmitted at a volume that is high enough to be heard by the far end (speech receiving side) partner. Also, the voice that you do not want to hear is not always transmitted at a volume that is low enough not to be heard by the far-end party.

  If the user can know how loud his / her voice is transmitted to the far end, the user can talk according to the volume of his / her voice while checking the volume of his / her voice.

However, it is not easy for the user to know how loud his / her voice is transmitted to the far end. For example, a method of presenting the volume of the received signal to the user is conceivable (see, for example, Patent Document 1).
JP 2006-67240 A M. Togami, T. Sumiyoshi, and A. Amano, "Stepwise phase difference restoration method for sound source localization using multiple microphone pairs", ICASSP2007, vol.1, pp.117-120, 2007. T.Takatani, T.Nishikawa, H.Saruwatari, and K.Shikano, "Blind separation of binaural sound mixture using SIMO-model-based independent component analysis", ICASSP2004, vol.4, pp.113-116, 2004.

  However, the volume that can be confirmed by the method described in Patent Document 1 is not the volume of only one's own voice, but the voice of a person who spoke at the same time, the sound output from another speaker, and the environmental noise It is only the volume of the signal that picks up the sound with the sound of multiple sound sources superimposed.

  An object of the present invention is to provide a sound collection system capable of confirming, for each sound source, how much the voice collected from the voice input unit is suppressed.

  A typical example of the present invention is as follows. That is, a sound collection system including a microphone array composed of two or more microphones and a processing unit that converts a signal output from the microphone array, wherein the processing unit is output from the microphone array. A sound source separation unit that separates a signal for each direction in which a sound source exists, a noise removal processing unit that removes noise from the signal output from the microphone array, a signal output from the sound source separation unit, and the noise removal processing And a direction-specific residual signal calculation unit that calculates a sound volume for each direction of the residual signal based on the residual signal output from the unit, and the sound collection system further includes a calculation result by the direction-specific residual signal calculation unit And a suppression amount display unit for displaying the volume of the residual signal for each direction.

  According to the embodiment of the present invention, it is possible to confirm for each sound source how much the collected voice is suppressed.

[First Embodiment]
Hereinafter, a video conference system using the present invention will be described as an example. In a video conference system using an IP network line, two or more bases connected by a network communicate with each other using a telephone conference facility composed of a microphone array and a speaker, and between speakers existing at each base. Realize conversation. Hereinafter, a video conference system centering on an arbitrary base will be described. The base is referred to as the near end, and the base other than the near end connected to the near end is referred to as the far end.

  FIG. 1 is a diagram showing a hardware configuration of the video conference system according to the first embodiment of the present invention.

  The video conference system includes a microphone array 101 including two or more microphone elements, an A / D / D / A converter 102, a central processing unit 103, a volatile memory 104, a storage medium 105, a suppression amount display unit 106, and a noise removal operation. Input unit 107, speaker 108, camera 109, image display device 110, hub 111, audio cable 112, digital cable 113, digital cable 114, digital cable 115, audio cable 116, digital cable 117, monitor cable 118, and LAN cable 119 Consists of

  The A / D-D / A converter 102 converts an analog signal of sound pressure output from the microphone array 101 into digital data. The central processing unit 103 manages the output of the A / D-D / A conversion device 102. The storage medium 105 stores information such as a program and physical coordinates of each microphone element of the microphone array 101, and is connected to the central processing unit 103.

  Multi-channel sound pressure data collected by each microphone element of the microphone array 101 is output to the A / D-D / A converter 102 via the audio cable 112. The multi-channel sound pressure data is converted into multi-channel digital sound pressure data by an A / D-D / A converter 102. The above-described conversion is executed in synchronization with the conversion timing between the sound pressure signals output from the microphone elements.

  The converted multi-channel digital sound pressure data is output to the central processing unit 103 via the digital cable 113. The central processing unit 103 performs acoustic signal processing on the input multi-channel digital sound pressure data. The signal subjected to the acoustic signal processing is transmitted to the network via the LAN cable 119 and the hub 111.

  Digital sound pressure data received from the far end via the network is output to the central processing unit 103 via the hub 111 and the LAN cable 119, and the central processing unit 103 executes acoustic signal processing. The digital sound pressure data subjected to the sound processing is output to the A / D-D / A converter 102 via the digital cable 113. The output digital sound pressure data is converted into analog sound pressure data by the A / D-D / A converter 102, and the analog sound pressure data converted via the audio cable 116 is output from the speaker 108.

  The noise removal operation input unit 107 is an input unit in which the user sets suppression direction data indicating whether or not to suppress the voice coming from each direction included in the collected multi-channel sound pressure data. The noise removal operation input unit 107 is an apparatus in which, for example, a plurality of buttons are installed so as to go around the side surface of the cylindrical housing. By operating the button, it is possible to set whether or not to suppress the voice coming from the direction in which the button is arranged. For example, if the voice coming from a certain direction is suppressed, the LED of the button in that direction is turned on. If the voice coming from a certain direction is not suppressed, the LED of the button in that direction is turned off. Can be shown to the user. The set suppression direction data is transmitted to the central processing unit 103 via the digital cable 115.

  The multichannel digital sound pressure data X collected by the microphone array 101 and output to the central processing unit 103 includes the sound output from the speaker 108 as an acoustic echo.

  The central processing unit 103 updates the multi-channel digital filter for removing the acoustic echo at each time based on the multi-channel digital sound pressure data X and the digital sound pressure data output from the hub 111, and is updated. The digital filter is stored in the volatile memory 104, and the acoustic echo is removed using the digital filter updated in each time zone. Further, the central processing unit 103 refers to the suppression direction data output from the noise removal operation input unit 107 and the physical coordinates of each microphone element of the microphone array 101 stored in the storage medium 105 to remove the acoustic echo. The noise removal process is executed on the multi-channel sound pressure data Y after the above.

  Further, the central processing unit 103 uses the multi-channel digital sound pressure data X to calculate the sound volume P_X in each direction of arrival included in the multi-channel digital sound pressure data X. Furthermore, the central processing unit 103 uses the multi-channel digital sound pressure data X and the digital sound pressure data Y on which the noise removal processing has been performed to convert the digital sound pressure data Y on which the noise removal processing has been performed into The volume P_Y of each included arrival direction is calculated. The calculated volume P_X and the calculated volume P_Y are output from the central processing unit 103 to the suppression amount display unit 106 via the digital cable 14.

  The suppression amount display unit 106 displays the calculated volume P_X and the calculated volume P_Y.

  An image signal captured by the camera 109 is output to the central processing unit 103 via the digital cable 117. The central processing unit 103 performs image signal processing on the input image signal. The image signal that has been subjected to the image signal processing is transmitted to the network via the LAN cable 119 and the hub 111.

  The image signal transmitted from the far end is output to the central processing unit via the hub 111 and the LAN cable 119. The central processing unit 103 performs image signal processing on the input image signal, and outputs the image signal on which image signal processing has been performed to the image display device 110 via the monitor cable 118. Display an image on the screen.

  As the digital cable 113, the digital cable 114, the digital cable 115, and the digital cable 117, a USB cable or the like is used.

  The suppression amount display unit 106 can indicate to the user the suppression amount of speech coming from each direction. The suppression amount display unit 106, for example, combines two columns of a sequence SEQ_X in which a plurality of green LEDs are arranged vertically and a sequence SEQ_Y in which a plurality of red LEDs are arranged vertically into one column SEQ_COMB. It is an apparatus arrange | positioned so that the side surface of a cylindrical housing | casing may go around. When the direction θ in which SEQ_COMB is arranged is included in the range of Θ = [θ_1, θ_2], SEQ_X_Θ is the level of the volume of the sound coming from the range of Θ included in the input multi-channel digital sound pressure data X Display using a meter. When the direction θ in which SEQ_COMB is arranged is included in the range of Θ = [θ_1, θ_2], SEQ_Y_Θ is the level of the sound volume coming from the range of Θ included in the digital sound pressure data Y from which noise is removed. Display using a meter.

  By displaying the volume for each range of the direction in which the voice comes, the user can check the suppression amount of his / her voice.

  In the present embodiment, it is desirable that the housings of the microphone array 101 and the suppression amount display unit 106 are physically fixed to each other and the relative positional relationship is fixed. Accordingly, when the microphone array 101 is moved, the display unit that displays the suppression amount is also moved together. Therefore, the user only has to consider the position of the microphone array 101 as a reference, and the direction in which the suppression is suppressed can be easily understood.

  Further, since it is not necessary to newly install a sensor, the configuration of the apparatus can be simplified. That is, if the relative positional relationship between the microphone array 101 and the suppression amount display unit 106 changes with time, the position where the suppression amount is displayed must be changed according to the relative positional relationship. For that purpose, it is necessary to obtain a relative positional relationship with various position sensors, such as acquiring a marker position with a magnetic sensor, an ultrasonic sensor, or a camera. However, if a sensor is introduced, the configuration of the apparatus becomes complicated. By fixing the relative positional relationship between the microphone array 101 and the suppression amount display unit 106, a sensor is unnecessary.

  In addition, it is desirable that the housings of the suppression amount display unit 106 and the noise removal operation input unit 107 are physically fixed to each other and the relative positional relationship is fixed. Thus, as described above, the configuration of the apparatus can be simplified by not using the sensor for estimating the relative positional relationship.

  Furthermore, it is desirable that the direction in which the LED row set SEQ_COMB of the suppression amount display unit 106 is arranged coincides with the direction in which the buttons of the noise removal operation input unit 107 are arranged. Accordingly, when the user specifies the direction in which the voice is desired to be suppressed, the shorter the distance between the position of the display unit that displays the suppression amount and the position of the button, the easier the user can operate.

  FIG. 2 is a diagram showing a usage example of the video conference system according to the first embodiment of the present invention.

  A user U1 and a user U2 exist at the base A, and a call is made with a user at the base B. At this time, the user U1 who wants to talk only at the site A has the shortest distance from himself among the buttons installed in the noise removal operation input unit 107 so that his / her voice cannot be heard by the user at the site B. Operate a button. In other words, the button corresponding to the position where the user exists is pressed. Then, the central processing unit 103 calculates a directional filter having a directivity pattern that often has a volume coming from the direction of the user U1. The central processing unit 103 applies the calculated filter to the signal after the echo canceller process, and transmits the sound in which the sound coming from the direction of the user U1 is suppressed to the base B.

  At the site B, the received signal is output from the speaker 208 via the central processing unit 203.

  The suppression amount display unit 106 at the site A displays the volume of voice coming from the direction in which the user U1 exists included in the input multi-channel digital sound pressure data X and the user U1 included in the digital sound pressure data after noise removal. Is displayed in SEQ_COMB arranged in the suppression amount display unit 106 corresponding to the direction in which the user U1 exists.

  The user U1 can talk without being heard by the user at the site B while checking whether or not the volume of the voice coming from the direction in which the user U1 exists is sufficiently suppressed by looking at the displayed suppression amount. Further, it is possible to have a conversation only at the site A while confirming that the conversation between the user U2 at the site A and the user at the site B is not disturbed. Further, when the conversation voice of the site A is not sufficiently suppressed, the user U1 and the user U2 of the site A operate the buttons of the noise removal operation input unit 107 so as to specify more directions, or more It is possible to talk only with the scripture A without listening to the conversation of the user at the base B by talking with a low voice.

  FIG. 3 shows the speech of each user, the input operation to the noise removal operation input unit 107, the display of the suppression amount display unit 106, and the transmission to the far end in the video conference system which is an embodiment of the sound collection system of the present invention. It is a figure which shows the example of the time chart of the volume relationship of the sound pressure data performed. In FIG. 3, the horizontal axis represents time.

  The user U1 and the user U2 speak in the time zone t1. At this time, the voices of the users U1 and U2 are transmitted to the far end without being suppressed. Even in the suppression amount display, it can be seen that the volume difference between the volume of the collected voice and the volume of the transmitted voice is small, and is hardly suppressed.

  At time t2, the user U1 operates the button B1 installed at the shortest position from the user U1 of the noise removal operation input unit 107. After this operation, the voice in the direction in which the user U1 exists is suppressed.

  User U1 and user U2 speak in time zone t3. At this time, the voice of the user U2 is transmitted without being suppressed as in the time zone t1. On the other hand, since the voice in the direction in which the user U1 exists is in a suppressed state, the voice of the user U1 is suppressed in the residual signal. Even in the suppression amount display, since the difference between the volume of the voice collected from the direction where the user U1 exists and the volume of the voice transmitted to the far end is large, it can be seen that the voice of the user U1 is sufficiently suppressed. The user U1 can perform a conversation only at the near end with peace of mind.

  At time t4, the user U1 operates the button B1 again. After this operation, the sound in the direction in which the user U1 exists returns from the suppressed state to the normal unsuppressed state.

  Since the voice in the direction of the user U1 is not suppressed in the time zone t5, the voice of the user U1 is transmitted to the far end without being suppressed as in the time zone t1. Even in the suppression amount display, it can be seen that the volume difference between the volume of the collected voice and the volume of the voice transmitted to the far end is very small, and is hardly suppressed.

  FIG. 13 is a flowchart showing a series of processes of the video conference system according to the first embodiment of this invention.

  After the video conference system is activated, first, the central processing unit 103 at its own base (near end) performs acoustic echo canceller adaptation processing (S1301). The acoustic echo canceller adaptive process outputs a white signal or a full-band signal whose frequency changes in the time direction from the speaker, and initializes the filter of the acoustic echo canceller. Thereafter, the central processing unit 103 determines whether or not a connection is requested from another base (far end) (S1302).

  When it is determined that a connection is requested from another base (far end), the central processing unit 103 connects to another base (far end) (S1304). When it is determined that a connection is not requested from another base (far end), the central processing unit 103 determines whether or not a connection is requested from the local base (near end) to another base (far end). (S1303).

  When it is determined that a connection is requested from its own base (near end) to another base (far end), the central processing unit 103 connects to another base (far end) (S1304). When it is determined that a connection is not requested from its own base (near end) to another base (far end), the central processing unit 103 returns to S1302.

  In S1304, after being connected to another site (far end), the central processing unit 103 reproduces the far end sound from the speaker (S1305), acoustic echo canceller (S1306), noise removal processing (S1307), and collection. The processing is executed in the order of sound source separation (S1308), calculation of sound volume by direction with respect to residual signal (S1309), presentation of suppression amount (S1310), and voice transmission to another base (far end) (S1311). . After the processing described above is executed, the central processing unit 103 determines whether or not the connection with the other base (far end) is broken (S1312).

  When it is determined that the connection with the other base (far end) is broken, the central processing unit 103 executes processing for disconnecting the connection with the other base (far end) (S1314), and a series of processing Exit. When it is determined that the connection with the other base (far end) is not broken, the central processing unit 103 determines whether or not the local base (near end) has requested disconnection from the other base (far end). (S1313).

  When it is determined that a disconnection request has been made from its own base (near end) to another base (far end), the central processing unit 103 executes processing for disconnecting from the other base (far end) (S1314). ), A series of processing ends. If it is determined that the disconnection is not requested from the own base (near end) to another base (far end), the process returns to S1315, and the same processing is performed thereafter.

  FIG. 4 is a block diagram showing the configuration of the video conference system according to the first embodiment of the present invention.

  Multi-channel analog sound pressure data input to each microphone element of the microphone array 101 is converted into multi-channel digital sound pressure data x_i (t) corresponding to each microphone element by a multi-channel A / D converter 401. Here, i is an index indicating the number of the microphone element. If the total number of microphone elements is M, i takes any value from 0 to M-1. T is a discrete time for each sampling period. The converted multi-channel digital sound pressure data x_i (t) is output to the multi-channel frame processing unit 402.

  The sound receiving unit 404 receives the digital sound pressure data ref (t) transmitted from the far end. The received digital sound pressure data ref (t) is digital sound pressure data using the TCP / IP protocol or the RTP protocol.

  In the case of a multi-site video conference system with a server at the center, the server receives audio signals from the multi-sites, mixes the received audio signals, and transmits the mixed audio signals to the respective bases. The voice receiving unit 404 receives the mixed voice signal transmitted from the server. In this case, the audio reception unit 404 transmits the mixed audio as it is to the D / A conversion unit 405 and the multi-channel frame processing unit 402 as digital sound pressure data ref (t).

  In the case of a multi-site video conference system in which communication is performed using multicast or the like without using a server in the center, the audio signal of each base is transmitted to each base, and the audio receiver 404 receives the audio signal of each base. Directly from each location. In this case, the voice receiving unit 404 mixes the voices of the respective bases, and then outputs the mixed voice to the D / A conversion unit 405 and the multi-channel frame processing unit 402 as digital sound pressure data ref (t). . The digital sound pressure data ref (t) output to the multi-channel frame processing unit 402 is used as a reference signal in the multi-channel frame processing unit 402, as will be described later.

  The D / A conversion unit 405 converts the input digital sound pressure data ref (t) into analog sound pressure data. The converted analog sound pressure data is output from the speaker 108 by the sound reproduction unit 406.

  The multi-channel frame processing unit 402 converts the input multi-channel digital sound pressure data x_i (t) into a multi-channel time domain frame signal Xf_i (t, τ) corresponding to a range from t = τS to t = τS + F−1 and a time. The signal is converted into a region reference signal Reff (t, τ).

  Note that f is an index representing a frequency band obtained by dividing the frequency at equal intervals. When the frequency is divided into N, f takes any value from 0 to N-1. Hereinafter, it is described as a frequency bin f. t represents time. τ is called a frame index, and τ is incremented by 1 after the processing from the multi-channel frame processing unit 402 to the voice transmission unit 413 is completed. S is called a frame shift, and means the number of samples shifted for each frame. F is called the frame size and means the number of samples processed at one time for each frame.

  The converted multi-channel time-domain frame signal Xf_i (t, τ) and the time-domain reference signal Reff (t, τ) are output to the multi-channel short-time frequency analysis unit 403. The multi-channel short-time frequency analysis unit 403 adds a DC component cut, a Hamming window, a Hanning window, and a time-domain reference signal Reff (t, τ) to the input multi-channel time domain frame signal Xf_i (t, τ). Perform window processing such as Blackman windows. Thereafter, the multi-channel short-time frequency analysis unit 403 further performs short-time Fourier transform to convert the multi-channel frequency domain frame signal Xf_i (f, τ) and the frequency domain reference signal Reff (f, τ). Here, the number of frequency bins f (hereinafter referred to as the number of frequency bins) is N.

  FIG. 14 is an explanatory diagram showing a data structure of the multi-channel frequency domain frame signal Xf_i (f, τ) in the arbitrary frame τ according to the first embodiment of this invention.

  A corresponding multi-channel frequency domain frame signal Xf_i (f, τ) is stored in each divided by the number of microphone elements and the number of frequency bins. The multi-channel frequency domain frame signal Xf_i (f, τ) takes a complex value.

  The multi-channel frequency domain frame signal Xf_i (f, τ) and the frequency domain reference signal Reff (f, τ) of each microphone element are output to the multi-channel acoustic echo canceller 407. Further, the multi-channel frequency domain frame signal Xf_i (f, τ) is also output to the sound source separation unit 408.

  The multi-channel acoustic echo canceller 407 receives an acoustic echo signal of a signal input from the speaker 108 from the multi-channel frequency domain frame signal Xf_i (f, τ) of each microphone element input from the multi-channel short-time frequency analyzer 403. Remove ingredients. The acoustic echo signal component is calculated based on the frequency domain reference signal Reff (f, τ) input from the multi-channel short-time frequency analysis unit 403. As the acoustic echo removal processing, for example, processing in which a transfer function of acoustic echo is sequentially adapted using a general algorithm such as an NLMS algorithm can be considered. Note that the difference in processing of the acoustic echo canceller is not an essential difference of the present invention.

  The multi-channel frequency domain frame signal after the acoustic echo component is removed by the multi-channel acoustic echo canceller 407 is defined as Ef_i (f, τ). The multi-channel frequency domain frame signal Ef_i (f, τ) calculated by the multi-channel acoustic echo canceller unit 407 is output to the noise removal processing unit 409.

  FIG. 15 is an explanatory diagram illustrating a data structure of the multi-channel frequency domain frame signal Ef_i (f, τ) in the arbitrary frame τ according to the first embodiment of this invention.

  A corresponding multi-channel frequency domain frame signal Ef_i (f, τ) is stored in each divided by the number of microphone elements and the number of frequency bins. The multi-channel frequency domain frame signal Ef_i (f, τ) takes a complex value.

  The noise removal operation input unit 107 outputs a signal indicating whether or not to perform noise removal for each of J mutually exclusive direction ranges Θ_j = [θ_j1, θ_j2]. However, j is an index indicating a direction. When all directions are divided into J, j takes any value from 0 to J-1.

  Specifically, the noise removal processing unit 409 includes a button B_j corresponding to each direction range Θ_j, and each time the button B_j is pressed,

IsReduced_j (τ) having a binary value as shown in FIG. However, IsReduced_j (τ) is a Boolean value that is true (value is 1) when the button B_j is pressed in an arbitrary frame.

  When IsReduced_j (τ) to be output is 0, a signal indicating that noise removal is not performed is output to the noise removal processing unit 409. When the output IsReduced_j (τ) is not 0, a signal indicating that noise removal is performed is output to the noise removal processing unit 409.

  FIG. 16 is an explanatory diagram illustrating a data structure of IsReduced_j (τ) in an arbitrary frame τ according to the first embodiment of this invention.

  Corresponding values are stored in an area obtained by dividing J in all directions. As shown in FIG. 16, it is a one-dimensional array.

  Note that the method of designating the direction of noise removal is not limited to the method of manual designation by the user, and may be a method of providing a preset value in the noise removal processing unit 409 in advance. In this case, it is not necessary to provide the noise removal operation input unit 107.

  The noise removal processing unit 409 is based on the multi-channel frequency domain frame signal Ef_i (f, τ) input from the multi-channel acoustic echo canceller unit 407 and IsReduced_j (τ) input from the noise removal operation input unit 107. The noise in the designated direction is removed from the multi-channel frequency domain frame signal Ef_i (f, τ). Specific processing will be described below.

  FIG. 5 is a block diagram illustrating a configuration example of the noise removal processing unit 409 using the minimum dispersion beamformer.

  The noise removal processing unit 409 includes a target sound / noise separation unit 501, a target sound steering vector update unit 502, a noise covariance matrix update unit 503, a filter update unit 504, and a filter multiplication unit 505.

  First, the properties of the input multi-channel frequency domain frame signal Ef_i (f, τ) will be described.

  FIG. 6 is a diagram schematically showing a signal of one channel of the input multi-channel frequency domain frame signal Ef_i (f, τ).

  As shown in FIG. 6, it is known that the collected speech is discrete for each frequency component. This property is called “sparseness”. Therefore, each frequency component can be assumed to be a component of only one voice. The present embodiment uses this assumption to separate the target sound and noise.

  First, the target sound / noise separation unit 501 suppresses the voice arriving from an arbitrary direction included in the data regarding the arrangement of the microphone elements from the microphone arrangement 410 and the multichannel sound pressure data collected from the noise removal operation input unit 107. Θ is calculated using the suppression direction data indicating whether or not and the multi-channel frequency domain frame signal Ef_i (f, τ) input from the multi-channel acoustic echo canceller 407. Note that θ is an amount representing the direction of voice arrival.

  As a calculation method of θ, for example, when the number of microphone elements of the microphone array 101 is two,

Is calculated using

  Here, ρ (f, τ) is a phase difference between the input signals of the two microphone elements at the frame τ and the frequency index f. As a calculation method of the phase difference ρ (f, τ), for example, as shown in FIG. 14, the multi-channel frequency domain frame signal Xf_1 (f, τ) of the microphone element 1 at an arbitrary frequency, that is, in an arbitrary row. And the multi-channel frequency domain frame signal Xf_1 (f, τ) of the microphone element i are calculated, and the phase difference is calculated from the multiplier.

  Freq (f) is the frequency of the frequency bin f,

Is calculated using Here, Fs is the sampling rate of the multi-channel A / D converter 401. d is the physical distance between the two microphone elements. c is the speed of sound. Strictly speaking, the speed of sound changes depending on the temperature and the density of the medium, but it may be fixed to one value such as 340 m / s.

  In the noise removal processing, the time-frequency is fixed based on the above-mentioned “sparseness” assumption, and the same processing is performed for each fixed time-frequency. Hereinafter, the fixed time-frequency suffix (f, τ) is omitted.

  When the number of microphone elements in the microphone array 101 is three or more, θ can be calculated by the SPIRE algorithm (see Non-Patent Document 1). In the SPIRE algorithm, the time-frequency is fixed based on the above-described assumption of “sparseness”, and the same processing is performed for each fixed time-frequency.

  FIG. 7 is a flowchart showing a θ calculation method (SPIRE algorithm) when the number of microphone elements of the microphone array 101 is three or more.

  First, the target sound / noise separation unit 501 reads data relating to the arrangement of the microphone elements (S701). Note that the storage medium 105 holds data regarding the microphone arrangement.

  Next, the target sound / noise separation unit 501 selects a combination of microphone elements to form a microphone pair having two microphone elements as one set (S702). At this time, it is desirable that the interval between the two selected microphone elements is selected so as to be different for each microphone pair.

  Next, the target sound / noise separation unit 501 rearranges the selected microphone pairs in descending order of the arrangement interval of the microphone elements, and stores them in the microphone pair queue (S703). Here, k is an index for specifying one microphone pair, k = 1 is a microphone pair with the shortest microphone element arrangement interval, and k = K is a microphone pair with the longest microphone element arrangement interval.

  The target sound / noise separation unit 501 determines whether or not the number of elements in the microphone pair queue is 0 (S704). That is, it is determined whether there is a microphone pair. When it is determined that the number of elements in the microphone pair queue is not 0, the target sound / noise separation unit 501 reads one microphone pair with the shortest microphone element arrangement interval from the microphone pair queue, and waits for the read microphone pair to wait for the microphone pair. Processing to remove from the matrix is performed (S705).

  The target sound / noise separation unit 501 calculates a phase difference for the read microphone pair. Specifically, the target sound / noise separation unit 501 firstly

An integer n_k that satisfies the above is calculated. Since the range enclosed by the inequality corresponds to 2π, there is always a solution.

  Next, the target sound / noise separation unit 501 uses the calculated integer n_k.

And the phase difference is calculated. The initial value in the case of k = 1 is

Defined by

  After S706, the process returns to S704 again, and the same processing is executed for all microphone pairs.

  If it is determined in S704 that the number of elements in the microphone pair queue is 0, the target sound / noise separation unit 501 uses the calculated phase difference.

And θ (f, τ), which is the voice arrival direction, is calculated. Here, d _k is the arrangement interval of the microphone elements of the kth microphone pair.

  The estimation accuracy of the calculation of the arrival direction of sound increases as the arrangement interval of the microphone elements of the microphone pair increases, but the arrangement interval of the microphone elements of the microphone pair having a half wavelength or more of the multi-channel frequency domain frame signal Ef_i (f, τ). Is longer, one direction cannot be specified from the phase difference of the arrangement intervals of the microphone elements of the microphone pair, and two or more directions having the same phase difference exist (spatial aliasing).

The above calculation method of the arrival direction is based on the arrangement interval of the microphone elements of the short microphone pair in the previous loop among the two or more directions originally obtained by [Formula 2] with respect to the arrangement interval of the microphone elements of the long microphone pair. The procedure is equivalent to selecting the direction of arrival θ (f, τ) of the voice obtained uniquely. Therefore, the direction of arrival of sound can be calculated with high accuracy even when spatial aliasing occurs.

The target sound / noise separation unit 501 converts the multi-channel frequency domain frame signal Ef_i (f, τ) into the target sound signal E _subject _i based on the calculated voice arrival direction θ (f, τ) for each time-frequency. (F, τ) and noise signal E _{noise —} i (f, τ) are separated.

  Specifically, in each frequency bin f, separation is performed as follows with respect to j in the direction range Θ_j in which the direction range Θ_j includes the voice arrival direction θ (f, τ).

FIG. 17 is a diagram illustrating a data structure of the target sound signal E _{subject —} i (f, τ) in an arbitrary frame τ according to the first embodiment of this invention. FIG. 18 is a diagram illustrating a data structure of the noise signal E _{noise —} i (f, τ) in an arbitrary frame τ according to the first embodiment of this invention.

The target sound signal E _{subject —} i (f, τ) is output from the target sound / noise separation unit 501 to the target sound steering vector update unit 502. The noise signal E _{noise —} i (f, τ) is output from the target sound / noise separation unit 501 to the noise covariance matrix update unit 503.

  The target sound steering vector update unit 502

, A target sound steering vector a _subject (f, τ) = [a — 0 (f, τ),..., A_M−1 (f, τ)] ^T is updated. However, for the sake of stability, it may be updated only when the absolute value of the target sound signal E _{subject —} i (f, τ) is sufficiently large. The updated target sound steering vector a _subject (f, τ) is output to the filter update unit 504.

  The noise covariance matrix update unit 503

Based on the above, the noise covariance matrix R _n (f, τ) is updated. However, the noise signal E _{noise —} i (f, τ) = [E _{noise —} 0 (f, τ),..., E _noise — M−1 (f, τ)] ^T, and γ _n is 0 or more and less than 1 Constant parameter. For the sake of stability, the noise signal E _{noise —} i (f, τ) may be updated only when the absolute value is sufficiently large. The updated noise covariance matrix R _n (f, τ) is output to the filter update unit 504.

The filter update unit 504 receives the target sound steering vector a _subject (f, τ) and the noise covariance matrix noise covariance matrix R _n (f, τ),

Based on the above, the filter w (f, τ) is calculated. However, γ _w is an appropriate constant parameter of 0 or more and less than 1.

  The filter multiplier 505 receives the filter w (f, τ) and the multi-channel frequency domain frame signal Ef_i (f, τ).

And the frequency domain frame signal y (f, τ) from which the sound coming from the designated direction is removed is calculated.

  The frequency domain frame signal y (f, τ) calculated by the above-described procedure is output to the time signal generation unit 411 and the direction-specific residual volume calculation unit 415. The time signal generation unit 411 performs inverse FFT on the input frequency domain frame signal y (f, τ) to convert it into a time domain frame signal y (t, τ). Further, the time signal generation unit 411 superimposes and adds the time domain frame signal y (t, τ) for each frame period, and multiplies the inverse of the window function to convert it into the time domain signal y (t). Then, the time signal generation unit 411 outputs the converted time domain signal y (t) to the voice transmission unit 413.

  When transmitting through the server, the voice transmission unit 413 transmits the time domain signal y (t) generated for each site to the server. When not passing through the server, the time domain signal y (t) is transmitted to each base using the TCP / IP or RTP protocol.

  The sound source separation unit 408 separates the multi-channel frequency frame region signal Xf_i (f, τ) into direction-specific frequency region frame signals Xf_j (f, τ), which are components in the respective directions. The separated sound volume calculation unit 415 outputs the separated direction-specific frequency domain frame signal Xf_j (f, τ). Hereinafter, processing in the sound source separation unit 408 will be described.

  FIG. 8 is a flowchart illustrating processing of the sound source separation unit 408 according to the first embodiment of this invention.

  First, the sound source separation unit 408 calculates the voice arrival direction θ (f, τ) from the input multi-channel frequency frame region signal Xf_i (f, τ) (S801). Note that the calculation method of the sound arrival direction θ (f, τ) is calculated using the SPIRE algorithm described above.

Next, the sound source separation unit 408 calculates the absolute value A _X (f, τ) of the amplitude for each frequency bin f.

(S802).

  When the calculated θ (f, τ) is included in the range of the direction range Θj,

Then, the direction-specific frequency domain frame signal Xf_j (f, τ) is calculated (S803).

  FIG. 19 is a diagram illustrating a data structure of the frequency domain frame signal Xf_j (f, τ) for each arbitrary frame τ direction according to the first embodiment of this invention.

  As shown in FIG. 19, the direction-specific frequency domain frame signal Xf_j (f, τ) calculated by the above processing stores data for each corresponding frequency and corresponding direction range.

  The volume calculation unit 414 calculates the volume P_j (τ) for each direction of the collected sound signal of the input frequency domain frame signal Xf_j (f, τ) for each direction.

Calculate based on The calculated volume P_j (τ) for each direction in the collected sound signal is output to the suppression amount display unit 106.

  The direction-specific residual volume calculation unit 415 includes the frequency domain frame signal y (f, τ) input from the noise removal processing unit 409 and the direction-specific frequency domain frame signal Xf_j (f, τ) input from the sound source separation unit 408. ) From which the sound coming from within the direction range Θj is suppressed, that is, the volume Q_j (τ) for each remaining signal in the direction. Hereinafter, the residual sound volume calculation unit 415 for each direction. Processing will be described.

  FIG. 9 is a flowchart illustrating a process of the direction-specific residual volume calculation unit 415 according to the first embodiment of this invention.

  First, the direction-specific residual sound volume calculation unit 415 performs initialization setting (S901). Specifically, the frequency bin f is set to 0, and the residual signal in-direction volume Q_j (τ) is set to 0 for all range directions Θj.

  Next, the direction-specific residual sound volume calculation unit 415 determines whether or not f = N−1 (S902). When it is determined that f = N−1 is not satisfied, the direction-specific residual volume calculation unit 415 sets j to 0 (S908).

  The direction-specific residual volume calculation unit 415 determines whether j = J (S903). If it is determined that j = J, the direction-specific residual volume calculation unit 415 proceeds to S907. When it is determined that j = J is not satisfied, the direction-specific residual volume calculation unit 415 determines whether the direction-specific frequency domain frame signal Xf_j (f, τ) = 0 (S904).

When it is determined that the direction-specific frequency domain frame signal Xf_j (f, τ) = 0, the direction-specific residual volume calculation unit 415 defines j + 1 as a new j (S905), and returns to S903. Directionally frequency domain frame signal Xf_j (f, τ) = 0 not equal when it is determined, by the residual volume calculation unit 415 direction, the residual signal in the direction-specific volume Q_j (τ) | y (f , τ) | 2 And the added value is defined as a new residual signal in-direction volume Q_j (τ) (S906).

  Next, the direction-specific residual volume calculation unit 415 defines f + 1 as a new frequency bin f (S907), returns to S902, and performs the same processing.

  When it is determined in S902 that f = N−1, the direction-specific residual sound volume calculation unit 415 outputs the residual signal in-direction sound volume Q_j (τ) to the suppression amount display unit 106.

  Here, the direction-specific frequency domain frame signal Xf_j (f, τ) is obtained by separating the components of each frequency f into a single range direction Θj in the sound source separation unit 408 based on the assumption of “sparseness”. Therefore, the direction-specific frequency domain frame signal Xf_j (f, τ) ≠ 0 only at most one f. Therefore, when one frequency bin f satisfying the direction-specific frequency domain frame signal Xf_j (f, τ) ≠ 0 is found in the loop of S903 to S905, the process can move to the next loop of S902 to S907. Thereby, high-speed processing can be performed.

  FIG. 10 is a diagram illustrating an example of display on the suppression amount display unit 106 according to the first embodiment of this invention.

  In the suppression amount display unit 106, a sound collection volume meter SEQ_X_Θj and a residual signal volume meter SEQ_Y_Θj are arranged in parallel to form a set of SEQ_COMB_Θj, and SEQ_COMB_Θj is arranged on the side surface of the cylindrical housing. Each SEQ_COMB_Θj corresponds to each direction range Θj. SEQ_COMB_Θj is preferably arranged in the direction of θjm = (θ_j1 + θ_j2) / 2 with respect to the direction range Θ_j = [θ_j1, θ_j2]. This is to make it easy to understand the correspondence between the direction in which each SEQ_COMB_Θj is arranged and the voice arrival direction.

  The sound collection volume meter SEQ_X_Θj displays the volume P_j (τ) for each direction in the sound collection signal. The residual signal volume meter SEQ_Y_Θj displays the volume Q_ (τ) for each residual signal inward direction.

  FIG. 11 is a diagram illustrating a correspondence between the number of LEDs that are turned on in the suppression amount display unit 106 according to the first embodiment of this invention and the value of the sound volume P_j (τ) for each direction in the collected sound signal. FIG. 12 is a diagram illustrating a correspondence between the number of LEDs that are turned on in the suppression amount display unit 106 according to the first embodiment of this invention and the value of the volume Q_j (τ) for each remaining signal in-direction.

  When the number of LEDs constituting the sound collection volume meter SEQ_X_Θj is 8, the number of LEDs to be lit is 0 to 8 with respect to the ratio of the volume P_j (τ) and Pmax according to the direction of the sound collection signal. What can be considered. However, Pmax in the table is the maximum value of the volume P_j (τ) for each direction in the collected sound signal, and the numbers of the LEDs are L1, L2,. The same applies to the volume Q_j (τ) for each remaining signal in-direction, as shown in FIG. However, Qmax is the maximum value of the volume Q_j (τ) for each remaining signal in the direction.

  The suppression amount display unit 106 is not limited to display using LEDs as in the present embodiment. For example, it may be another device such as an organic EL display or a liquid crystal display, or may be another display method having a function as a level meter.

  The present embodiment is not limited to a video conference system, and can be applied to, for example, a mobile phone videophone or a car navigation handsfree call device.

  The embodiment of the present invention is not limited to the shapes of the microphone array 101, the suppression amount display unit 106, and the noise removal operation input unit 107. For example, microphone elements may be arranged in the microphone array 101 in a hemispherical shape, and the suppression amount display unit 106 and the noise removal operation input unit 107 may be arranged so as to correspond to each microphone element. In this case, the sound can be separated and displayed not in the two-dimensional direction but in the three-dimensional direction including the height.

[Second Embodiment]
FIG. 20 is a block diagram showing the configuration of the video conference system according to the second embodiment of the present invention.

  In the second embodiment, the multi-channel frequency domain frame signal Ef_i (f, τ) calculated by the multi-channel acoustic echo canceller unit 407 is output to the sound source separation unit 408 and the noise removal processing unit 409.

  With the above-described configuration, the first embodiment displays the voice with the echo removed and the noise removed as the suppressed voice, whereas the second embodiment has the voice with the noise removed. Is displayed as suppressed speech.

[Third Embodiment]
The suppression amount display unit 106 is not limited to the form of displaying the sound volume P_j (τ) for each direction in the collected sound signal and the sound volume Q_j (τ) for each remaining signal direction.

  For example, the suppression amount display unit 106

As shown in FIG. 5, the difference between the volume P_j (τ) for each direction in the collected sound signal and the volume Q_j (τ) for each remaining signal direction is defined as a suppression amount R_j (τ), and the suppression amount R_j (τ) is displayed. Form may be sufficient.

  The suppression amount display unit 106

As shown in FIG. 3, the ratio of the volume P_j (τ) for each direction in the collected sound signal and the volume Q_j (τ) for each remaining signal direction is defined as the suppression amount R_j (τ), and the suppression amount R_j (τ) is displayed. It may be.

  It is desirable that the suppression amount display unit 106 displays a scale that shows a relative difference between the volume P_j (τ) for each direction in the collected sound signal and the volume Q_j (τ) for each remaining signal direction.

  FIG. 21 is a diagram illustrating an example of display on the suppression amount display unit 106 according to the third embodiment of the present invention.

  As shown in FIG. 21, the suppression amount display unit 106 may be configured to display the suppression amount R_j (τ) and the residual signal in-direction volume Q_j (τ) using display icons.

  FIG. 22 is a diagram illustrating associations between display icons and suppression amounts R_j (τ) according to the third embodiment of the present invention. FIG. 23 is a diagram illustrating a correspondence between display icons and residual signal in-direction volume Q_j (τ) according to the third embodiment of the present invention.

  As shown in FIG. 22, a method of associating the ratio between the suppression amount R_j (τ) and Rmax with the display icon is conceivable. However, Rmax is the maximum value of the suppression amount R_j (τ). In addition, as shown in FIG. 23, a method of associating the ratio between the volume Q_j (τ) for each remaining signal in-direction and Qmax with a display icon is conceivable. However, Qmax is the maximum value of the volume Q_j (τ) for each remaining signal in the direction.

  In the third embodiment described above, the user can grasp the volume more intuitively as compared with the first embodiment.

[Fourth Embodiment]
The processing method of the sound source separation unit 408 that separates sound for each sound source is not limited to the first embodiment, and can be separated by other processing.

  FIG. 24 is a block diagram showing a configuration of a video conference system according to the fourth embodiment of the present invention.

  As shown in FIG. 24, the multi-channel time domain frame signal Xf_i (t, τ) output from the multi-channel frame processing unit 402 is input to the sound source separation unit 2608. Hereinafter, processing of the sound source separation unit 2608 will be described.

  FIG. 25 is a flowchart illustrating processing of the sound source separation unit 2608 according to the fourth embodiment of this invention.

  The sound source separation unit 2608 receives the multi-channel time domain frame signal Xf_i (t, τ) as input, calculates a SIMO-ICA (see Non-Patent Document 2) filter, and updates the SIMO-ICA filter (S2701). In addition, the method described in the nonpatent literature 2 can be used for the calculation method and update method of a filter.

  Next, the sound source separation unit 2608 multiplies the updated SIMO-ICA filter by the multi-channel time domain frame signal Xf_i (t, τ), and separates each sound source (S2702). By the separation process described above, the signal is separated into S signals Xf_s_i (t, τ). Here, s is an integer from 0 to S-1, and is an index representing each sound source. Hereinafter, it is described as a sound source s. S is the maximum number of sound sources, and is a number equal to or less than the number of microphone elements (M). That is, the separated signal Xf_s_i (t, τ) indicates a signal in which the sound of the sound source s is input to the microphone element i.

  The sound source separation unit 2608 converts the separated signal Xf_s_i (t, τ) into a frequency domain frame signal Xf_s_i (f, τ) (S2703).

  The sound source separation unit 2608 sets the sound source s to 0 (S2704), and then determines whether or not s = S-1 (S2705).

  If it is determined that s = S−1 is not satisfied, the sound source separation unit 2608 sets the frequency bin f to 0 and further initializes the direction histogram h_s (θ) (S2706). Specifically, the direction histogram h_s (θ) = 0 is set for the arrival directions θ of all the sounds of the sound sources s. The direction histogram h_s (θ) is a histogram showing the angle distribution in an arbitrary sound source. Next, the sound source separation unit 2608 determines whether or not f = N−1 (S2707).

  If it is determined that the frequency bin f = N−1 is not satisfied, the sound source separation unit 2608 calculates the voice arrival direction θ (f, τ) (S2708). Note that the voice arrival direction θ (f, τ) is calculated using the SPIRE algorithm, as in the first embodiment.

Next, the sound source separation unit 2608 sets the calculated voice arrival direction θ (f, τ) as θ _h ,

Accordingly, the voting is performed for the direction histogram h_s (θ _h ) (S2709). Then, f + 1 is defined as a new frequency bin f (S2710), and the process returns to S2707. Thereafter, the processing from S2708 to S2710 is similarly performed for all frequencies (until f = N−1).

If it is determined in step S2707 that f = N−1, the sound source separation unit 2608 searches for a direction peak from the direction histogram h_s (θ _h ) created by the series of loop processing in steps S2707 to S2710 (S2711). . Due to the separation using the SIMO-ICA filter, the frequency domain frame signal Xf_s_i (f, τ) is theoretically a component of a single sound source. Therefore, the search for the direction peak described above may be performed by obtaining θ _h that takes the maximum value from the distribution of the direction histogram h_s (θ _h ). Further, the obtained θ _h is set as the sound arrival direction θ_s of the sound source s.

  After searching for the direction peak, the sound source separation unit 2608 defines s + 1 as a new sound source s (S2712), returns to S2705, and thereafter performs the same processing.

  When it is determined in S2705 that s = S-1, the sound source separation unit 2608 displays the arrival direction θ_s of the sound of each sound source s calculated by the loop processing of S2705 to S2712 and the suppression amount display. The range direction Θ_j of the unit 106 is associated (S2713).

  Through the above processing, the sound source separation unit 2608 calculates the direction-specific frequency domain frame signal Xf_j (f, τ) separated in each direction based on the calculated voice arrival direction θ_s of each sound source s. The direction-specific frequency domain frame signal Xf_j (f, τ) is output to the calculation unit 414 and the direction-specific residual volume calculation unit 2615. Hereinafter, the correlation between the calculated voice arrival direction θ_s of each sound source s and the display range Θ_j and the calculation method of the direction-specific frequency domain frame signal Xf_j (f, τ) will be described.

  As one method, for example, for j where Θ_j includes θ_s,

Based on the above, a frequency domain frame signal Xf_j (f, τ) is calculated for each direction.

  As another method, for example,

Consider the cost function C (δ) shown in FIG. Here, for δ (s, j), only one j is 1 for an arbitrary sound source s, 0 for the other j, and only one s for any j. The function is 1 and the other s are 0. δ (s, j) represents a one-to-one correspondence between the sound sources s and j. Also, dist (θ_s, Θ_j) is

A distance function as shown in FIG.

  As a method of association, for example, a method of associating the sound source s with the direction range j by obtaining δ that maximizes the cost function C (δ) is conceivable. Then, s separated signals Xf_s_i (t, τ) satisfying δ (s, j) = 1 are obtained.

And the frequency domain frame signal Xf_j (f, τ) is calculated for each direction. By using the method described above, even if the direction of the sound source s is close and the directions that can be displayed by the suppression amount display unit 106 exist discretely, the sound source s can be separated.

  Next, the direction-specific residual volume calculation unit 2615 will be described.

  FIG. 26 is a flowchart illustrating a process of the direction-specific residual sound volume calculation unit 2615 according to the fourth embodiment of this invention.

  The direction-specific residual volume calculation unit 2615 performs an initialization process (S2801). Specifically, the frequency bin f is set to 0, and the residual signal inner direction volume Q_j (τ) is set to 0. Next, the direction-specific residual volume calculation unit 2615 determines whether or not f = N−1 (S2802).

  When it is determined that f = N−1 is not satisfied, the direction-specific residual volume calculation unit 2615

Based on, to calculate the P _sum (S2803), it sets the direction range j to 0 (S2804). Next, the direction-specific residual sound volume calculation unit 2615 determines whether j = J−1 or not (S2805).

When it is determined that j = J−1 is not satisfied, the direction-specific residual volume calculation unit 2615 calculates the P _sum calculated in S2803.

And Q_j is calculated (S2806).

  Next, the direction-specific residual volume calculation unit 2615 defines j + 1 as a new direction range j (S2807), returns to S2805, and performs the same processing.

  When it is determined in S2805 that j = J, the direction-specific residual volume calculation unit 2615 defines f + 1 as a new frequency bin f (S2808), returns to S2802, and performs the same processing.

  When it is determined in S2802 that f = N−1, the direction-specific residual volume calculation unit 2615 suppresses the residual signal-specific direction volume Q_j (τ) calculated from the series of loop processing of S2802 to S2808. The data is output to the display unit 106.

  As a display method of the suppression amount display unit 106, the same method as that in the first embodiment or the third embodiment is used.

  The present embodiment is not limited to a video conference system, and can be applied to, for example, a mobile phone videophone or a car navigation handsfree call device. In addition, since the sound source separation unit 2608 uses a sound source separation method that does not require the assumption of speech sparsity, the sound source separation unit 2608 is not limited to conversational speech, and may be used for other types of sounds such as environmental sounds or musical sounds. Applicable.

[Fifth Embodiment]
The present invention is also applicable to a voice recording device such as an IC recorder.

  FIG. 27 is a diagram illustrating a hardware configuration example of a voice recording device according to the fifth embodiment of the present invention.

  The voice recording apparatus 2000 includes a microphone array 101 including one or more microphone elements, an A / D conversion apparatus 2002 that converts an analog sound pressure value input from the microphone array 101 into digital data, and an output from the A / D conversion apparatus 2002. A central processing unit 103 for processing the digital data to be processed, a volatile memory 104 connected to the central processing unit 103, a program and a physical arrangement of each microphone element of the microphone array 101 connected to the central processing unit 103, etc. A storage medium 105 that stores information, a suppression amount display unit 106, a noise removal operation input unit 107, an audio cable 112, a digital cable 113, a digital cable 114, and a digital cable 115 are included.

  Since the fifth embodiment does not require data exchange with the far end, images such as a camera and a monitor need not be handled. In addition, the A / D conversion device 2002 does not reproduce audio and therefore does not require D / A conversion. Therefore, the A / D converter 2002 only performs processing for converting the input multi-channel sound pressure data into multi-channel digital sound pressure data.

  The arrangement method of the microphone array 101, the suppression amount display unit 106, and the noise removal operation input unit 107 is the same as that in the first embodiment. The processing in the central processing unit 103, the suppression amount display device 2006, and the noise removal operation input unit 107 is the same as that in the first embodiment.

  FIG. 28 is a block diagram showing a configuration of a voice recording apparatus according to the fifth embodiment of the present invention.

  As shown in FIG. 28, the audio recording apparatus according to the present embodiment may not include the audio reception unit 404, the audio reproduction unit 406, and the multi-channel acoustic echo canceller unit 407. In addition, although the user may manually determine the direction of noise removal using the noise removal operation input unit 2007, the direction of noise removal may be determined by a value preset in the voice recording device. In that case, the noise removal operation input unit 2007 may not be included in the configuration of the voice recording device.

When the voice recording apparatus in the present embodiment is operated so as to remove noise or voice that is not desired to be recorded from a certain direction and removes the incoming sound from the direction from which the voice arrives as noise, the above-described noise or recording is performed. The user can make a conversation while confirming that the sound that he / she does not want to remove is sufficiently removed as noise and the sound he / she wants to record is recorded. The present invention can be applied to a mechanism as it is. Further, as in the fourth embodiment, the sound source separation unit 2608 and the direction-specific residual volume calculation unit 2615 using SIMO-ICA can be configured as a voice recording device. In this case, as described above in the fourth embodiment, the sound source separation unit 2608 uses a sound source separation method that does not require the assumption of speech sparsity. It can also be applied to other types of sounds.

  The embodiment of the present invention is not limited to the shape of the voice recording device. For example, a hemispherical shape may be used. In this case, the sound can be separated and displayed not in the two-dimensional direction but in the three-dimensional direction including the height.

It is the figure which showed the hardware constitutions of the video conference system in the 1st Embodiment of this invention. It is the figure which showed the usage example of the video conference system in the 1st Embodiment of this invention. In the video conference system which is an embodiment of the sound collection system of the present invention, each user's utterance, input operation to the noise removal operation input unit, display of the suppression amount display device, and sound pressure data transmitted to the far end It is a figure which shows the example of the time chart of a volume relationship. It is the block diagram which showed the structure of the video conference system in the 1st Embodiment of this invention. It is a block diagram which shows the structural example of the noise removal process part by a minimum dispersion beamformer. FIG. 6 is a diagram schematically showing a signal of one channel among input multi-channel frequency domain frame signals Ef_i (f, τ). It is a flowchart which shows the calculation method (SPIRE algorithm) of (theta) in case the number of microphone elements of a microphone array is three or more. It is a flowchart which shows the process of the sound source separation part of the 1st Embodiment of this invention. It is a flowchart which shows the process of the residual sound volume calculation part according to direction of the 1st Embodiment of this invention. It is a figure which shows an example of the display of the suppression amount display part of the 1st Embodiment of this invention. It is a figure which shows matching with the value of the number of LED lighted by the suppression amount display part of the 1st Embodiment of this invention, and the value of the sound volume P_j (τ) according to the direction of the collected sound signal. It is a figure which shows matching with the number of LED lighted by the suppression amount display part of the 1st Embodiment of this invention, and the value of the volume Q_j (τ) according to the direction in residual signal. It is the flowchart which showed a series of processes of the video conference system of the 1st Embodiment of this invention. It is explanatory drawing which showed the data structure of the multi-channel frequency domain frame signal Xf_i (f, (tau)) in arbitrary frames (tau) of the 1st Embodiment of this invention. It is explanatory drawing which showed the data structure of the multichannel frequency domain frame signal Ef_i (f, (tau)) in arbitrary frames (tau) of the 1st Embodiment of this invention. It is explanatory drawing which showed the data structure of IsReduced_j (τ) in the arbitrary frame τ according to the first embodiment of this invention. It illustrates a data structure of the target sound signal E _subject _i in any frame tau in the first embodiment of the present invention (f, τ). It illustrates a data structure of the noise signal E _noise _i in any frame tau in the first embodiment of the present invention (f, τ). It is a figure which shows the data structure of the frequency domain frame signal Xf_j (f, (tau)) according to the arbitrary frame (tau) direction of the 1st Embodiment of this invention. It is the block diagram which showed the structure of the video conference system in the 2nd Embodiment of this invention. It is a figure which shows an example of the display of the suppression amount display part in the 3rd Embodiment of this invention. It is a figure which shows matching with the display icon and suppression amount R_j ((tau)) in the 3rd Embodiment of this invention. It is a figure which shows matching with the display icon in the 3rd Embodiment of this invention, and volume Q_j ((tau)) according to residual signal inner direction. It is the block diagram which showed the structure of the video conference system in the 4th Embodiment of this invention. It is the flowchart which showed the process of the sound source separation part of the 4th Embodiment of this invention. It is a flowchart which shows the process of the residual sound volume calculation part according to direction of the 4th Embodiment of this invention. It is a figure which shows the hardware structural example of the audio | voice recording apparatus of the 5th Embodiment of this invention. It is a block diagram which shows the structure of the audio recording apparatus of the 5th Embodiment of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 Video conference system 101 Microphone array 102 A / D-D / A converter 103 Central processing unit 104 Volatile memory 105 Storage medium 106 Suppression amount display part 107 Noise removal operation input part 108 Speaker 109 Camera 110 Image display apparatus 111 Hub 112 Audio cables 113 to 115 Digital cable 116 Audio cable 117 Digital cable 118 Monitor cable 119 LAN cable U1, U2 User 203 Central processing unit 208 Speaker t1, t3, t5 Time zone t2, t4 Time 401 Multi-channel A / D converter 402 Multi Channel frame processing unit 403 Multi-channel short-time frequency analysis unit 404 Audio reception unit 405 D / A conversion unit 406 Audio reproduction unit 407 Multi-channel acoustic echo canceller unit 408 Sound source component Separation unit 409 Noise removal processing unit 410 Microphone arrangement 411 Time signal generation unit 413 Audio transmission unit 414 Volume calculation unit 415 Directional residual volume calculation unit 501 Target sound / noise separation unit 502 Target sound steering vector update unit 503 Noise covariance matrix update Unit 504 filter update unit 505 filter update unit S701 to S707 step S801 to S803 step S901 to S908 step S1301 to S1314 step L1 to L8 LED number 2000 voice recording device 2002 A / D converter 2608 sound source separation unit 2615 residual volume according to direction Calculation unit S2701 to S2713 Step S2801 to S2808 Step

Claims (10)

A sound collection system comprising: a microphone array composed of two or more microphones; and a processing unit that converts a signal output from the microphone array,
The processor is
A sound source separation unit that separates signals output from the microphone array for each direction in which a sound source exists;
A noise removal processing unit for removing noise from the signal output from the microphone array;
A residual signal calculation unit for each direction that calculates a volume for each direction of the residual signal based on the signal output from the sound source separation unit and the residual signal output from the noise removal processing unit;
The sound collection system further includes a suppression amount display unit that displays a volume of the residual signal for each direction based on a calculation result by the residual signal calculation unit for each direction. The sound source separation unit is
For each frequency in the time / frequency domain divided by the time component and frequency component, calculate the direction in which the sound source exists,
Based on the calculated direction, the signal output from the microphone array is separated for each direction in which a sound source exists,
The direction-specific residual signal calculator is
Determine whether the sound source exists in the time / frequency domain divided by the time component and the frequency component,
The sound collection system according to claim 1, wherein a sound volume for each direction of the residual signal is calculated based on the determination result. The sound source separation unit is
Calculate the direction that the sound source exists for each frequency of the sound collected by each microphone,
Based on the direction in which the sound source for each calculated frequency exists, the signal output from the microphone array is separated for each direction in which the sound source exists,
The direction-specific residual signal calculator is
Separated for each direction in which the sound source is present and separated from each direction in which the sound source is present and output from the microphone array with respect to the sum of the magnitudes of the signals output from the microphone array. The sound collection system according to claim 1, wherein a relative value of the magnitude of the signal in the direction is calculated as a volume for each direction of the residual signal. The processing unit further includes a volume calculation unit that calculates the volume of the signal output from the sound source separation unit,
The sound collection system according to claim 1, wherein the suppression amount display unit further displays a volume of the signal output from the sound source separation unit based on a calculation result by the volume calculation unit.   The sound collection system according to claim 1, wherein the suppression amount display unit displays a difference between a volume of the residual signal and a volume of a signal output from the sound source separation unit.   The sound collection system according to claim 1, wherein the suppression amount display unit includes a display that displays the volume of the residual signal for each direction in which a sound source exists.   The sound collection system according to claim 1, wherein the suppression amount display unit includes a display that displays a volume of a signal output from the sound source separation unit for each direction in which the sound source exists. Sound in a sound collection device comprising: a microphone array including two or more microphones; a processing unit that converts a signal output from the microphone array; and a suppression amount display unit that displays a volume of the converted signal. Display method,
The processor is
The signal output from the microphone array is separated for each direction in which a sound source exists,
Remove noise from the signal output from the microphone array,
Based on the residual signal from which the noise has been removed, calculate a volume for each direction of the residual signal,
The suppression amount display unit
A sound display method comprising displaying the calculated volume of each residual signal in each direction. The processor is
When calculating the volume for each direction of the residual signal,
For each frequency in the time / frequency domain divided by the time component and frequency component, calculate the direction in which the sound source exists,
Based on the calculated direction, the signal output from the microphone array is separated for each direction in which a sound source exists,
Determine whether the sound source exists in the time / frequency domain divided by the time component and the frequency component,
The sound display method according to claim 8, wherein the sound volume for each direction of the residual signal is calculated based on the determination result. The processor is
When calculating the volume for each direction of the residual signal,
Calculate the direction that the sound source exists for each frequency of the sound collected by each microphone,
Calculating the direction of the statistics sound source is present for each of the calculated frequency,
Based on the calculated statistics, the signal output from the microphone array is separated for each direction in which a sound source exists,
The signal output from the microphone array is separated for each direction in which the sound source is present in a predetermined direction with respect to the sum of the magnitudes of the signals output from the microphone array and separated in the direction in which the sound source is present. The sound display method according to claim 8 , wherein a relative value of the magnitude is calculated as a volume for each direction of the residual signal.

Images