RU2759715C2 - Sound recording using formation of directional diagram - Google Patents

Abstract

FIELD: acoustics.SUBSTANCE: tool for forming a directional diagram contains a microphone array that is connected to the first generator of a directional diagram and the second generator of a directional diagram. Generators of a directional diagram are filtering and combining generators of a directional diagram containing a set of filters of a shape of a directional diagram, each of which has an adaptive impulse response. A mismatch processor determines a mismatch indicator between directional diagrams of the first generator of a directional diagram and the second generator of a directional diagram in response to a comparison of adaptive impulse responses of two generators of a directional diagram. The mismatch indicator, for example, can be used to combine output signals of generators of a directional diagram. The mismatch indicator is chosen to be less sensitive to scattered noise.EFFECT: increase in the efficiency of noise suppression.14 cl, 8 dwg

Description

The technical field to which the invention relates

The invention relates to sound recording using beamforming, and in particular, but not exclusively, to voice recording using beamforming.

BACKGROUND OF THE INVENTION

Sound recording, and in particular speech recording, has become more and more important in recent decades. Indeed, sound recording has become increasingly important in a variety of applications, including telecommuting, teleconferencing, games, speech user interfaces, etc. However, a problem in many scenarios and applications is that the desired speech source is typically is not the only sound source in the environment. Rather, in typical acoustic environments, there are many other sound / noise sources that are picked up by the microphone. One of the critical issues facing many recording applications is how to best isolate speech from noisy environments. In order to respond to this problem, a number of different noise suppression approaches have been proposed.

Indeed, research into hands-free telephone systems is a topic of discussion that has attracted a lot of interest for decades. The first commercial systems available focused on professional (video) conferencing systems in environments with low background noise and low reverberation times. It has been found that a particularly useful approach for identifying and highlighting desired sound sources, such as a desired presenter, for example, is to use beamforming based on signals from a microphone array. Initially, a microphone array with a focused fixed pattern was often used, but later the use of adaptive patterns became more popular.

In the second half of the 1990s, hands-free systems for mobile phones began to be produced. They were meant to be used in many different environments, including reverberant rooms and (higher) background noise levels. Such acoustic environments provide for significantly more difficult tasks and, in particular, can complicate or worsen the adaptation of the formed radiation pattern.

Recording research for such environments initially focused on echo attenuation and later on noise cancellation. An example of a beamforming sound recording system is illustrated in FIG. 1. In an example, a directional array of a plurality of microphones 101 is coupled to a beamformer 103 that produces a sound source signal z (n) and one or more noise reference signals x (n).

Microphone array 101 may contain only two microphones in some embodiments, but will typically contain more.

The beamformer 103, more specifically, may be an adaptive beamformer, in which a single beamform can be directed to a speech source using a suitable adaptation algorithm.

For example, US 7,146,012 and US 7,602,926 disclose examples of adaptive beamforming devices that focus on speech, but also provide a reference signal that (almost) does not contain speech.

The beamformer creates an improved output, z (n), by coherently adding the desired portion of the microphone signals by filtering the received signals in direct match filters and adding the filtered outputs. In addition, the output signal is filtered in inverse adaptive filters having filter characteristics coupled to the forward filters (in the frequency domain corresponding to the time-reversed impulse responses in the time domain). The error signals are generated as the difference between the input signals and the outputs of the inverse adaptive filters, and the filter coefficients are adapted to minimize the error signals, thereby resulting in an audio beam steered in the direction of the dominant signal. The generated error signals x (n) can be considered as noise reference signals, which are particularly suitable for performing additional noise reduction on the improved output signal z (n).

The primary signal z (n) and the reference signal x (n) are typically both contaminated with noise. In the event that the noise in the two signals is coherent (eg, when there is an interfering point noise source), an adaptive filter 105 can be used to attenuate the coherent noise.

To this end, the noise reference signal x (n) is coupled to the input of the adaptive filter 105 with an output subtracted from the sound source signal z (n) to generate a compensated signal r (n). The adaptive filter 105 is adapted to minimize the power of the compensated signal r (n), typically when the desired sound source is inactive (eg, when there is no speech), and this results in suppression of coherent noise.

The compensated signal is fed to a post-processor 107 which performs noise reduction on the compensated signal r (n) based on the noise reference x (n). More specifically, the post processor 107 converts the compensated signal r (n) and the noise reference signal x (n) to the frequency domain using a windowed Fourier transform. Then, for each frequency bin, it modifies the amplitude R (ω) by subtracting the scaled version of the amplitude spectrum X (ω). The resulting complex spectrum is converted back to the time domain to give an output q (n) in which noise has been suppressed. This spectral subtraction technology was first described by S.F. Boll, “Suppression of Acoustic Noise in Speech using Spectral Subtraction,” IEEE Trans. Acoustics, Speech and Signal Processing, vol. 27, pp. 113-120, Apr. 1979.

In many recording systems, a variety of beamforming devices can be used and are capable of independently adapting to the sound source. For example, in order to track two different presenters in an acoustical environment, a sound recorder may include two independently adaptive beam drivers.

In systems using a plurality of independently adaptable beamformers, it can often be useful to determine how close the beamforms of the different beamformers are to each other. For example, when using two beamformators to track two separate presenters, it can be important to ensure that both of them do not both adapt to track the same presenter. This, for example, can be achieved by defining a misalignment factor, which is indicative of the difference between the radiation patterns. If the mismatch metric indicates that the difference is below the threshold, this can reset one of the beam drivers towards the other sound source.

In other systems, an audio recorder may use interoperable beamforming devices to provide improved sound recording, and in such systems it can be useful to determine how close different radiation patterns are to each other.

For example, although the system of FIG. 1 provides very efficient performance and useful action in many scenarios, it is not optimal in all scenarios. Indeed, while many conventional systems, including the example of FIG. 1 provide very good performance when the desired sound source / speaker is within the reverberation radius of the microphone array, that is, for applications where the forward energy of the desired sound source is (preferably significantly) more powerful than the reflection energy of the desired sound source, they tend to produce less optimal results when they are not. In typical environments, it has been found that the presenter should typically be within 1-1.5 meters of the microphone array.

However, there is a strong desire for recording-based solutions, applications and hands-free systems where the user can be located at great distances from the microphone array. This is, for example, desirable for many communication systems and applications as well as many voice control systems and applications. Systems that provide speech enhancement, including reverberation cancellation and noise cancellation for such situations, are in the art referred to as supersonic communication systems.

In more detail, when it comes to additional stray noise and a desirable speaker outside the reverberation radius, the following problems can arise:

• A beamformer can often have problems distinguishing between the echo of the desired speech and scattered background noise, resulting in speech distortion.

• The adaptive beamformer may converge more slowly towards the desired presenter. During the time that the adaptive radiation pattern has not converged yet, there will be speech loss in the reference signal, resulting in speech distortion if this reference signal is used for non-stationary noise cancellation and cancellation. The problem increases when there are more desirable sources that speak one after the other.

The solution to dealing with the more slowly converging adaptive filters (due to background noise) is to supplement them with a number of fixed radiation patterns aimed in different directions, as illustrated in FIG. 2. However, this approach is especially developed for scenarios in which the desired sound source is present within the reverberation radius. It can be less effective for sound sources outside the reverberation radius and can often lead to unreliable solutions in such cases, especially if there is also acoustic diffuse background noise.

In particular, in order to control and operate such a system, it is typically important to be able to measure how close different beamforms / beamformers are to each other. For example, it can be important to compare focused and unfocused beamformers with each other to select which radiation pattern to use to generate the audio output.

However, generating reliable misalignment readings can be very difficult in many scenarios, especially when the desired sound source is outside the reverberation radius. Typical mismatch metrics tend to be based on a comparison of the outputs produced by the beamformers, such as by comparing signal levels or by correlating outputs, for example. Another approach is to determine the direction of arrival (DoA) of the signal and compare them with each other.

However, while such mismatch rates may provide acceptable performance in many embodiments, they tend to be suboptimal in many practical scenarios. In particular, they tend not to be optimal in scenarios with high levels of noise and reflections, and especially in reverberant environments where the desired sound source is outside the reverberation radius.

This can be understood as follows: if the desired sound source is outside the reverberation radius, the energy of the direct sound field is small compared to the energy of the scattered sound field generated from the reflections. The ratio of the direct sound field to the scattered sound field will be further degraded if there is also scattered background noise. The energies of different radiation patterns will be approximately the same and, accordingly, this does not provide a useful indication of the similarity of radiation patterns. For the same reason, a system based on DoA measurements will not be robust: due to the low energy of the forward field, the cross-correlation of signals will not produce a sharp single peak and will result in large errors. For the same reason, direct signal correlations are unlikely to provide a clear indication. Becoming more robust detectors will often result in unsuccessful detection of the desired sound source, resulting in unfocused radiation patterns. The typical result is speech scatter in a noisy reference signal, and severe distortion will occur if an attempt is made to attenuate the noise in the primary signal based on the noise reference signal.

Hence, an improved recording approach would be useful and, in particular, an approach that provides an improved misalignment rate between different radiation patterns would be useful. More specifically, an approach would be useful to provide the possibility of reduced complexity, increased flexibility, easier implementation, reduced cost, improved recording, improved usability for recording outside the reverberation radius, reduced sensitivity to noise, improved speech recording, improved accuracy of the misalignment metric, improved control and / or improved performance.

SUMMARY OF THE INVENTION

Accordingly, the invention seeks to preferably mitigate, alleviate or eliminate one or more of the above disadvantages, alone or in any combination.

According to an aspect of the invention, there is provided a beamforming sound recorder comprising: a microphone array; a first beamformer coupled to the microphone array and configured to generate a first beamforming audio output signal, the first beamformer is a filtering and combining beamformer comprising a first plurality of beamforming filters, each of which has a first adaptive pulse characteristic; a second beamformer coupled to the microphone array and configured to generate a second beamforming audio output, the second beamformer is a filtering and combining beamformator comprising a second plurality of beamforming filters, each of which has a second adaptive pulse characteristic; and a mismatch processor for determining a mismatch metric between the beams of the first beamformer and the second beamformer in response to comparing the first adaptive impulse responses with the second adaptive impulse responses.

The invention, in many scenarios and applications, can provide an improved indication of difference / similarity between radiation patterns generated by two beam drivers. In particular, an improved misalignment rate can often be provided in scenarios where the direct path from the sound sources to which the beam drivers are adapted is not predominant. Improved performance can often be achieved for scenarios containing high levels of stray noise, reflections and / or late reflections.

An audio recorder, in many embodiments, may comprise an output unit for generating an audio output in response to a first beamformed audio output, a second beamformed audio output, and a misalignment metric. For example, the output unit may comprise a combinator for combining the first and second beamforming audio outputs in response to the mismatch metric. However, it will be appreciated that the misalignment metric can be used for many other purposes in other applications, such as, for example, to choose between different radiation patterns, to control the adaptation of the beamformer, etc.

The approach can reduce the sensitivity of the properties of audio signals (whether beamforming audio outputs or microphone signals) and accordingly can be less sensitive to noise, for example. In many scenarios, the mismatch indicator can form faster and, for example, in some scenarios, instantly. In particular, the mismatch metric can be generated based on the current filter parameters without any averaging.

The filtering and combining beamforming devices may comprise a beamforming filter for each microphone and a combiner for combining outputs from the beamforming filters to produce a beamforming audio output signal. The combiner, more specifically, may be a combiner, and the filtering and combining beamformers may be filtering and combining beamformers.

The beam shapers are adaptive beam shapers and may contain adaptation functionality to adapt the adaptive impulse response (thereby adapting the effective directivity of the microphone array).

The mismatch score is equivalent to the similarity score.

The filtering and combining beamformers may more specifically comprise finite impulse response (FIR) filters having a plurality of coefficients.

In accordance with an optional feature of the invention, the mismatch processor is configured, for each microphone in the microphone array, to determine a correlation between the first and second adaptive impulse responses for the microphone and to determine the mismatch metric in response to a combination of correlations for each microphone from the microphone array.

This can provide a particularly useful mismatch metric without requiring excessive complexity.

In accordance with an optional feature of the invention, the mismatch processor is configured to determine frequency domain representations of the first adaptive impulse responses and the second adaptive impulse responses; and determining a mismatch metric in response to the frequency domain representations of the first adaptive impulse responses and the second adaptive impulse responses.

This can further improve performance and / or ease handling. This, in many embodiments, can facilitate the determination of the mismatch metric. In some embodiments, adaptive impulse responses may be provided in the frequency domain, and frequency domain representations may be available without delay. However, in most embodiments, the adaptive impulse responses may be output in the time domain, for example, by means of an FIR filter coefficients, and the mismatch processor, for example, may be configured to apply a discrete Fourier transform (DFT) to the time domain impulse responses. to generate frequency representations.

In accordance with an optional feature of the invention, the mismatch processor is configured to determine mismatch frequency metrics for frequency domain representation frequencies; and determine the mismatch metric in response to the mismatch frequency metrics as applied to the frequency domain representation frequencies; the mismatch processor is configured to determine the mismatch frequency metric for the first frequency and the first microphone from the microphone array in response to the first frequency domain coefficient and the second frequency domain coefficient, the first frequency domain coefficient is the frequency domain coefficient for the first frequency in relation to the first adaptive impulse response for the first a microphone, and the second frequency domain factor is a frequency domain factor for a first frequency with respect to a second adaptive impulse response for a first microphone; and the mismatch processor is further configured to determine a mismatch frequency metric for the first frequency in response to a combination of mismatch frequency metrics for the plurality of microphones of the microphone array.

This can provide a particularly useful misalignment indicator, which, in particular, can provide an accurate indication of the difference between radiation patterns.

и

, соответственно, частотный показатель несовпадения для частоты ω и микрофона m может быть определен как:When designating the first and second frequency components for the frequency ω and microphone m as

and

, accordingly, the frequency mismatch index for the frequency ω and the microphone m can be defined as:

The (combined) frequency misalignment metric for the frequency ω for a plurality of microphones of the microphone array may be determined by combining the values for the pickup microphones. For example, for simple summing on M microphones:

The overall mismatch score can then be determined by combining the individual mismatch frequency scores. For example, a frequency dependent combination can be applied:

- пригодная частотная весовая функция.where

is a suitable frequency weighting function.

In accordance with an optional feature of the invention, the mismatch processor is configured to determine a mismatch frequency metric for the first frequency and the first microphone in response to multiplying the first frequency domain coefficient and the conjugate value of the second frequency domain coefficient.

This can provide a particularly useful misalignment indicator, which, in particular, can provide an accurate indication of the difference between radiation patterns. In some embodiments, the frequency mismatch metric for frequency ω and microphone m may be defined as:

In accordance with an optional feature of the invention, the mismatch processor is configured to determine a mismatch frequency metric for a first frequency in response to an actual portion of a first frequency mismatch frequency metric combination for a plurality of microphones in a microphone array.

This can provide a particularly useful misalignment indicator, which, in particular, can provide an accurate indication of the difference between radiation patterns.

In accordance with an optional feature of the invention, a mismatch processor is configured to determine a mismatch frequency metric for a first frequency in response to a first frequency mismatch frequency metric combination rate for a plurality of microphones in a microphone array.

This can provide a particularly useful misalignment indicator, which, in particular, can provide an accurate indication of the difference between radiation patterns. The rate, in particular, may be the L1 rate.

In accordance with an optional feature of the invention, the mismatch processor is configured to determine a mismatch frequency metric for the first frequency in response to at least one of the real part and combination rate of the mismatch frequency metrics for the first frequency for a plurality of microphone array microphones relative to the sum of the L2 rate function for the sum the first frequency domain coefficients and the L2 norm function for the sum of the second frequency domain coefficients for the plurality of microphone array microphones.

This can provide a particularly useful misalignment indicator, which, in particular, can provide an accurate indication of the difference between radiation patterns. Monotone functions, more precisely, can be quadratic functions.

In accordance with an optional feature of the invention, the mismatch processor is configured to determine a mismatch frequency metric for a first frequency in response to a first frequency mismatch frequency metric combination rate for a plurality of microphone array microphones with respect to the product of the L2 rate function for the sum of the first frequency domain coefficients and the L2 rate function for the sum of the second frequency domain coefficients for the plurality of microphone array microphones.

This can provide a particularly useful misalignment indicator, which, in particular, can provide an accurate indication of the difference between radiation patterns. Monotone functions, more precisely, can be functions of an absolute value.

In accordance with an optional feature of the invention, the mismatch processor is configured to determine the mismatch metric as a frequency selective weighted sum of the mismatch frequency metrics.

This can provide a particularly useful misalignment indicator, which, in particular, can provide an accurate indication of the difference between radiation patterns. In particular, it can provide expressiveness of particularly perceptually significant frequencies, such as the expressiveness of speech frequencies.

In accordance with an optional feature of the invention, the first plurality of beamform filters and the second plurality of beamform filters are finite impulse response filters having a plurality of coefficients.

This can provide efficient operation and implementation in many embodiments.

In accordance with an optional feature of the invention, the beamforming sound recorder further comprises: a plurality of restricted beamforming devices coupled to the microphone array and each configured to produce a restricted beamforming audio output, each restricted beamforming device being restricted to generate a beamforming area in a different area from other localized beam drivers, the second beamformer is a localized beamformer from a plurality of localized beam drivers; a first adapter for adapting the beamform parameters of the first beamformer; a second adapter for adapting the parameters of the limited beamforms for the plurality of limited beam drivers; in this case, the second adapter is configured to adapt the parameters of the limited beamforms only for limited beam drivers from among the plurality of limited beam drivers, for which a mismatch metric has been determined that satisfies the similarity criterion.

The invention can provide improved sound recording in many embodiments. In particular, improved performance can often be achieved in reverberant environments and / or for long distance sound sources. The approach, in particular, can provide improved speech recording in many more complex acoustic environments. In many embodiments, the approach can provide reliable and accurate beamforming while simultaneously providing rapid adaptation to new desired sound sources. An approach could provide for a recording device that has reduced sensitivity to, for example, noise, reverberation, and reflections. In particular, improved recording of sound sources outside the reverberation radius can often be achieved.

In some embodiments, the audio output from the audio recorder may be generated in response to the first beamforming audio output and / or limited beamforming audio output. In some embodiments, the audio output may be a combination of constrained audio outputs, and more specifically, selective combining may be used, such as selecting a single constrained audio output.

The misalignment metric may reflect the difference between the generated beamforms of the first beamformer and the constrained beamformer for which the misalignment metric is generated, such as measured as the difference between the beam directions. In some embodiments, the misalignment metric may be indicative of a difference between the beamforming filters of the first beamformer and the constrained beamformer. The mismatch metric may be a distance metric, such as, for example, a metric defined as the distance between the vectors of the beamform filters of the first beamformer and the constrained beamformer.

It will be appreciated that a similarity score may be equivalent to a mismatch score for the reason that a similarity score providing information related to the similarity between two features, in fact, also provides information related to the difference between them, and vice versa.

The similarity criterion, for example, may require that the mismatch score be indicative of a difference below a predetermined score, eg, it may require the mismatch score having increasing values for increasing difference to be below a threshold value.

The regions can be multi-path beamforming dependent and are typically not limited by the angular direction of the arrival regions. For example, areas may differ based on the distance to the microphone array. Constraining the constrained beam drivers to beam in different regions can be accomplished by constraining the filter parameters of the constrained beamformer's beamform filters such that the constrained range of filter parameters (e.g., ranges for filter coefficients) is different for different shaper drivers. directional patterns.

The adaptation of the beamformers can occur by adapting the filter parameters of the beamforming filters of the beamformer, for example, more precisely, by adapting the filter coefficients. The adaptation may seek to optimize (maximize or minimize) a given adaptation parameter, such as, for example, maximizing the output level when a sound source is detected, or minimizing it when only noise is detected. The adaptation may seek to modify the beamform filters to optimize the measured parameter.

The second adapter may be configured to adapt the bounded beamform parameters of the second beamformers only if the misalignment metric meets the similarity criterion.

In accordance with an optional feature of the invention, the beamforming audio recorder further comprises a sound source detector for detecting point sound sources in the second beamformed audio outputs; and wherein the second adapter is configured to adapt the bounded beamforming parameters only for the bounded beam drivers for which the presence of a point sound source is detected in the bounded beamforming audio output signal.

This can further improve performance and, for example, can provide more reliable performance resulting in improved sound recording. Different criteria can be used to identify a point sound source in different embodiments. A point sound source, more precisely, can be a correlated sound source for microphones in a microphone array. A point sound source, for example, can be considered detected if the correlation between the signals from the microphones from the microphone array (for example, after being filtered by the constrained beamformer beamform filters) exceeds a predetermined threshold value.

According to an aspect of the invention, there is provided a method of operation for a beamforming sound recording apparatus comprising:

microphone array;

a first beamformer coupled to the microphone array, the first beamformer being a filtering and combining beamforming device comprising a first plurality of beamforming filters each having a first adaptive impulse response; a second beamformer coupled to the microphone array, the second beamformer being a filtering and combining beamforming device comprising a second plurality of beamforming filters each having an adaptive impulse response; the method is that: the first driver generates a first beamforming audio output signal; the second driver generates a second beamforming audio output; and determining a mismatch ratio between the radiation patterns of the first beamformer and the second beamformer in response to comparing the first adaptive impulse responses with the second adaptive impulse responses.

These and other aspects, features and advantages of the invention will be apparent from and explained with reference to the embodiment (s) described below.

BRIEF DESCRIPTION OF DRAWINGS

Embodiments of the invention will be described, by way of example only, with reference to the drawings, of which

fig. 1 illustrates an example of elements of a beamforming sound recording system;

fig. 2 illustrates an example of a plurality of radiation patterns generated by a sound recording system;

fig. 3 illustrates an example of elements of a sound recorder in accordance with some embodiments of the invention;

fig. 4 illustrates an example of filtering and summing beamformer elements;

fig. 5 illustrates an example of elements of a sound recorder in accordance with some embodiments of the invention;

fig. 6 illustrates an example of elements of a sound recorder in accordance with some embodiments of the invention;

fig. 7 illustrates an example of elements of a sound recorder in accordance with some embodiments of the invention;

fig. 8 illustrates an example of a flowchart for an adaptation approach of limited beamforming audio recorders in accordance with some embodiments of the invention.

DETAILED DESCRIPTIONS OF CERTAIN EMBODIMENTS

The following description will focus on embodiments of the invention applicable to a beamforming voice recording system, but it will be appreciated that the approach is applicable to many other scripted audio recording systems.

FIG. 3 illustrates an example of some elements of a sound recorder in accordance with some embodiments of the invention.

The sound recorder includes a microphone array 301 that contains a plurality of microphones configured to capture sound in the environment.

A microphone array 301 is coupled to a first beamformer 303 (typically directly or via echo cancellers, amplifiers, D / A converters, etc., as will be well known to one of ordinary skill in the art).

The first beamformer 303 is configured to combine signals from the microphone array 301 so that an effective directional sonic response of the microphone array 301 is generated. The first beamformer 303 thus generates an output, referred to as the first beamforming audio output signal , which corresponds to the selective recording of sound in the environment. The first beamformer 303 is an adaptive beamformer, and the directivity can be controlled by setting parameters, referred to as first beamform parameters, the effect of the beamform of the first beamformer 303, and more specifically by setting parameters (typically coefficients) of filters beamforms.

The microphone array 301 is further coupled to a second beamformer 305 (typically directly or via echo cancellers, amplifiers, D / A converters, etc., as will be well known to one of ordinary skill in the art).

The second beamformer 305 is similarly configured to combine signals from the microphone array 301 so that an effective directional sonic response of the microphone array 301 is generated. The second beamformer 305 thus provides an output, referred to as a second beamformed audio output. a signal that corresponds to the selective recording of sound in the environment. The second beamformer 305 is also an adaptive beamformer, and the directivity may be controlled by setting parameters, referred to as second beamform parameters, an act of beamforming the second beamformer 305, and more specifically by setting parameters (typically, coefficients ) beamform filters.

The first and second beamformer 303, 305, respectively, are adaptive beamformers, where the directivity can be controlled by adapting the parameters of the beamform action.

More specifically, the beam drivers 303, 305 are filter and combiner (or more precisely, filter and sum in most embodiments) beam drivers. A waveform filter can be applied to each of the microphone signals, and the filtered outputs can be combined, typically by simply adding together.

In most embodiments, each of the waveform filters has a time domain impulse response that is not a simple Dirac pulse (corresponding to a simple delay, and thus gain and phase shift in the frequency domain), but rather has an impulse response that is typically continues for a time interval of no less than 2, 5, 10, or even 30 milliseconds.

Impulse responses can often be implemented with beamform filters, which are FIR (finite impulse response) filters with multiple coefficients. Formers 303, 305 beamforming in such an embodiment can adapt beamforming by adapting the filter coefficients. In many embodiments, FIR filters may have coefficients corresponding to constant time offsets (typically offsets in sample sampling times), where adaptation is achieved by adapting the coefficient values. In other embodiments, beamform filters typically can have significantly fewer coefficients (eg, only two or three), but with such timing (also) being adaptable.

A particular advantage of beamforming filters having extended impulse responses instead of a simple variable delay (or simple gain / phase adjustment in the frequency domain) is that they allow the beam shapers 303, 305 to adapt to more than just the strongest, typically directional component of the signal. Rather, they allow the beam shapers 303, 305 to adapt to accommodate additional signal paths typically associated with reflections. Accordingly, the approach allows for improved performance in most real world environments, and more specifically, allows for improved performance in reflective and / or reverberant environments, and / or for sound sources farther from the microphone array 301.

It will be appreciated that different adaptation algorithms may be used in different embodiments, and that different optimization parameters will be known to those skilled in the art. For example, beam shapers 303, 305 can adapt the beamform parameters to maximize the output value of the beam shapers 303, 305. As a specific example, consider a beamformer where the received microphone signals are filtered with direct matching filters, and where the filtered outputs are summed. The output signal is filtered by inverse adaptive filters having filter characteristics coupled to the forward filters (in the frequency domain corresponding to the time-reversed impulse responses in the time domain). The error signals are generated as the difference between the input signals and the outputs of the inverse adaptive filters, and the filter coefficients are adapted to minimize the error signals, thereby resulting in maximum output power. Further details of this approach can be found in US 7,146,012 and US 7,602,926.

It is noted that approaches such as in US 7,146,012 and US 7,602,926 are based on an adaptation based both on the sound source signal z (n) and on the noise reference signal (s) x (n) from the beam shaper and it will be appreciated that the same approach can be used for the system of FIG. 3.

The beam drivers 303, 305, more specifically, may in fact be the beam drivers as illustrated in FIG. 1 and disclosed in US 7,146,012 and US 7,602,926.

Beam drivers 303, 305 in the example are coupled to an (optional) output processor 307 that receives beamforming audio outputs from beam drivers 303, 305. The exact output signal generated from the audio recorder will depend on the particular preferences and requirements of the particular embodiment. Indeed, in some embodiments, the output from the audio recorder may simply consist of audio outputs from the beam drivers 303, 305.

In many embodiments, the output from the output processor 307 is generated as a combination of audio outputs from the beam drivers 303, 305. Indeed, in some embodiments, simple selective combining may be performed, such as selecting the audio outputs for which the signal-to-noise ratio is the highest, or simply the signal strength.

Thus, the selection and subsequent processing of the output of the output processor 307 may be specialized and / or different in different implementations / embodiments. For example, all possible focused beamforms outputs can be provided, selection can be made based on user-defined criteria (e.g., the most powerful presenter is selected), etc.

With regard to the application of speech control, for example, all output signals can be forwarded to a speech trigger recognizer that is configured to identify a specific word or phrase to initialize the speech control. In such an example, the audio output signal in which the trigger word or phrase is detected may accompany the trigger phrase to be used by the speech recognizer to elicit special commands.

For communication applications, for example, it may be useful to select the audio output that has the highest power, for example, for which a particular point sound source has been detected.

In some embodiments, post-processing, such as the noise suppression of FIG. 1 may be applied to the output of an audio recorder (eg, output processor 307). This can improve performance, for example, for voice communications. Non-linear operations may be included in such post-processing, although, for example, with some speech recognizers, it may be useful to limit the processing to include only linear processing.

In many systems using a plurality of beamforming devices, it can be useful to be able to determine if the beamforming devices have formed beamforms that are close to each other. In the systems of FIG. 3, the audio recorder comprises a mismatch processor 309 that is configured to determine a mismatch metric that is indicative of a difference between the radiation patterns generated by the first beamformer 303 and the second beamformer 305.

It will be appreciated that the use of such a mismatch metric may be different for different applications and implementations, and that the principles are not limited to a particular application. In the specific example of FIG. 3, a mismatch processor 309 is coupled to an output processor 307 and is used in generating an audio output from an output processor 307. For example, if the mismatch metric indicates that the two radiation patterns are very close to each other, the audio output can be generated by summing or averaging the outputs (eg, in the frequency domain). If the mismatch metric is indicative of a large difference (and thus indicating that the two radiation patterns are adapted for different sound sources), the output processor 307 may generate an audio output by selecting the beamformed audio output that has the highest energy level.

In conventional approaches for comparing beamformers and beamforms, the similarity between beamforms is assessed by comparing the generated audio outputs. For example, cross-correlation between audio outputs can be generated, with the similarity being indicated by the magnitude of the correlation. In some systems, the DoA can be determined by cross-correlating the audio signals for a pair of microphones and determining the DoA in response to the timing of the peak.

In the system of FIG. 3, the mismatch metric is not determined based solely on the property or comparison of audio signals, whether beamformed audio outputs from the beam drivers or microphone inputs, but rather the mismatch processor 309 of the audio recorder of FIG. 3 is configured to determine the misalignment metric in response to comparing the impulse responses of the beamforming filters of the first and second beamforming shapers 303, 305.

FIG. 4 illustrates a simplified example of a filtering and summing beamformer based on a microphone array containing only two microphones 401. In the example, each microphone 401 is coupled to a waveform filter 403, 405 whose outputs are summed in an adder 407 to form a shaped directional patterns of the audio output signal. The beamforming filters 403, 405 have impulse responses f1 and f2 that are adapted to beamform in a given direction. It will be appreciated that, typically, a microphone array will contain more than two microphones, and that the principle of FIG. 4 easily extends to more microphones by additionally including a beamform for each microphone in the filter.

The first and second beam drivers 303, 305 may include such a filtering and combining architecture for beamforming (eg, as for beam drivers of US 7,146,012 and US 7,602,926). It will be appreciated that, in many embodiments, the microphone array 301, however, may contain more than two microphones. In addition, it will be appreciated that beamformers 303, 305 include functionality for adapting beamform filters as previously described. Also, in a specific example, the beam drivers 303, 305 generate not only the beamforming audio output, but also a noise reference signal.

In the system of FIG. 3, the parameters of the shape filters for the first beamformer 303 are compared with the parameters of the shape filters of the second beamformer 305. The mismatch score can then be determined to reflect how close these parameters are to each other. More specifically, for each microphone, the respective beamform filters of the first beamformer 303 and the second beamformer 305 are compared with each other to generate an intermediate mismatch metric. The intermediate mismatch metrics are then combined into a single mismatch metric output from the mismatch processor 309.

The comparison parameters of the radiation pattern are typically filter coefficients. More specifically, the beamform filters can be FIR filters having a time-domain impulse response defined by a set of FIR filter coefficients. The mismatch processor 309 may be configured to compare the respective filters of the first beamformer 303 and the second beamformer 305, determining a correlation between the filters. The correlation value can be determined as the maximum correlation (ie, the correlation value for the time offset maximizing the correlation).

The mismatch processor 309 can then combine all of these individual correlation values into a single mismatch metric by simply summing them together, for example. In other embodiments, a weighted combination may be performed, for example, by determining the weights of the larger coefficients higher than the lower coefficients.

It will be appreciated that such a misalignment rate will have an increasing importance for increasing filter correlation, and thus that a higher value will indicate increased pattern similarity rather than increased difference. However, in embodiments in which it is desired that the mismatch rate increases for increasing difference, the monotonically decreasing function can simply be applied to the combined correlation.

Determining the misalignment metric based on a comparison of the impulse responses of the waveform filters instead of based on audio signals (beamformed audio signals or microphone signals) offers significant advantages in many systems and applications. In particular, the approach typically provides significantly improved performance and is indeed suitable for applications in reverberant sound environments and for long range sound sources, including specific sound sources outside the reverberation radius. Indeed, it provides significantly improved performance in scenarios where the direct path from the sound source is not dominant, but is preferred where the direct path and possibly early reflections, for example, are dominated by a diffuse sound field. Particularly in such scenarios, the misalignment estimate based on the audio signal will be strongly influenced by the spatial and temporal characteristics of the sound field, while the filter-based approach allows for a more direct estimation of radiation patterns based on filter parameters that not only reflect the direct sound field. field / path, but adapt to reflect direct sound field / path and early reflections (due to impulse responses having an extended duration to take these reflections into account).

Indeed, while traditional DoA metrics and audio correlations for assessing the similarity of two beam drivers are based on anechoic environments and thus perform well in environments where the desired users are close to the microphones (within the reverberation radius) so that the diffuse energy is dominant. sound field, the approach of FIG. 3 is not based on such assumptions and gives an excellent estimate even in the presence of many reflections and / or substantially diffuse acoustic noise.

Other advantages include that the misalignment metric can be determined instantly based on the current beamform parameters, and more precisely, based on the current filter coefficients. In most embodiments, there is no need for any parameter averaging, rather, the adaptation rate of the adaptive beamformers determines the tracking behavior.

A particularly useful aspect is that the comparison and mismatch metric can be based on impulse responses that have an extended duration. This makes it possible that the misalignment metric not only reflects the forward path delay or the angular direction of the radiation pattern, but rather allows the impulse response of a significant part, or even more so, of the entire acoustic room to be taken into account. Thus, the misalignment metric is not based solely on the subspace excited by the signals from the microphones, as in traditional approaches.

In some embodiments, the misalignment metric may more accurately be configured to compare impulse responses in the frequency domain rather than in the time domain. More specifically, the mismatch processor 309 may be configured to convert the adaptive impulse responses of the filters of the first beamformer 303 to the frequency domain. Likewise, the mismatch processor 309 may be configured to convert the adaptive impulse responses of the filters of the second beamformer 305 to the frequency domain. The transformation may more accurately be performed by applying, for example, Fast Fourier Transform (FFT) to the impulse responses of the beamforming filters of both the first beamformer 303 and the second beamformer 305.

The mismatch processor 309 may respectively generate a set of frequency domain coefficients for each filter of the first beamformer 303 and the second beamformer 305. It can then proceed to determine the mismatch metric based on the frequency representation. For example, with respect to each microphone from the microphone array 301, the mismatch processor 309 may compare the frequency domain coefficients of the two beamform filters. As a simple example, it can simply determine the magnitude of the mismatch vector calculated as the difference between the vectors of the frequency domain coefficients for the two filters. The mismatch score can then be determined by combining intermediate mismatch scores generated for the individual frequencies.

In the following, specific and highly useful approaches for determining the mismatch score will be described. The approaches are based on a comparison of adaptive impulse responses in the frequency domain. In an approach, mismatch processor 309 is configured to determine mismatch frequency metrics for frequency domain representation frequencies. More specifically, a frequency misalignment metric can be determined for each frequency in a frequency representation. The output mismatch metric is then generated from these individual mismatch frequency metrics.

A mismatch frequency metric, more specifically, may be generated for each frequency filter coefficient of each filter pair of beamform filters, where the filter pair are filters, respectively, of the first beamformer 303 and the second beamformer 305 for the same microphone. The frequency mismatch indicator for this pair of frequency coefficients is formed as a function of the two coefficients. Indeed, in some embodiments, the frequency mismatch metric for a pair of coefficients may be determined as the absolute difference between the coefficients.

However, for real-valued time-domain coefficients (i.e. real-valued impulse response), the frequency coefficients will generally be complex values and, in many applications, a particularly useful frequency mismatch metric for a pair of coefficients is determined in response to the multiplication of the first frequency-domain coefficient and the conjugate quantity a second frequency domain coefficient (that is, in response to multiplying the complex coefficient of one filter and the conjugate value of the complex coefficient of the other filter in the pair).

Thus, with respect to each frequency domain bin of the impulse responses of the beamform filters, a mismatch frequency metric may be generated for each microphone / filter pair. The combined mismatch frequency metric for frequency can then be generated by combining these microphone specific mismatch metrics for all microphones, for example, simply by adding them together.

In more detail, beam drivers 303, 305 may contain frequency domain filter coefficients for each microphone and for each frequency domain representation.

, а для второго формирователя 305 диаграммы направленности они могут быть обозначены

, где M - количество микрофонов.With regard to the first beamformer 303, these coefficients can be denoted

and for the second beamformer 305 they may be denoted

, where M is the number of microphones.

и

.The complete set of frequency domain filter coefficients of the beamform for a specific frequency and for all microphones, with respect to the first beamformer 303 and the second beamformer 305, respectively, may be denoted as

and

...

In this case, the frequency mismatch metric for a given frequency can be defined as:

By multiplying the complex-valued filter coefficients that belong to the same microphones, we obtain, for each frequency, the first form of the distance exponent, thus

представляет собой комплексно сопряженную величину. Это может использоваться в качестве показателя несовпадения для частоты ω применительно к микрофону m. Комбинированный частотный показатель несовпадения для всех микрофонов может формироваться в виде суммы таковых, то есть,where

is a complex conjugate value. This can be used as an indication of the misalignment for the frequency ω applied to the microphone m. The combined frequency mismatch index for all microphones can be formed as a sum of those, that is,

If the two filters are unconnected, that is, the adapted state of the filters and thus the generated radiation patterns are highly misaligned, this sum is expected to be close to zero and thus the mismatch frequency metric is close to zero. However, if the filter coefficients are the same, a large positive value is obtained. If the filter coefficients are of the opposite sign, a large negative value is obtained. Thus, the generated mismatch frequency metric is indicative of the similarity of the beamform filters with respect to that frequency.

The multiplication of two complex coefficients (including the conjugate value) results in a complex value and, in many embodiments, it may be desirable to convert it to a scalar value.

In particular, in many embodiments, the mismatch frequency metric for a given frequency is determined in response to the real part of the combination of mismatch frequency metrics for different microphones with respect to that frequency.

More precisely, the combined mismatch frequency metric can be defined as:

In this measure, the similarity score based on Re (S) results in the maximum value achieved when the filter coefficients are identical, while the minimum value is reached when the filter coefficients are the same but have opposite signs.

Another approach is to determine the combined mismatch metric for a given frequency in response to the mismatch combination rate for the microphones. The rate typically advantageously may be the L1 or L2 rate.

For example:

In some embodiments, the combined mismatch frequency metric for all microphones from the microphone array 301 is thus defined as the amplitude or absolute value of the sum of the complex mismatch frequency metrics for the individual microphones.

In many embodiments, it may be useful to normalize the mismatch score. For example, it may be useful to normalize the mismatch metric so that it falls within the interval [0; 1].

In some embodiments, the misalignment metrics described above may be normalized by being determined in response to the sum of a monotonic rate function of the sum of frequency domain coefficients applied to the first beamformer 303 and a monotonic rate function for the sum of frequency domain coefficients applied to the second beamformer 305. where the sums are performed over the microphones. The norm may predominantly be the L2 norm, and the monotonic function may predominantly be a quadratic function.

Thus, the mismatch indicators can be normalized relative to the following value:

Combined with the first approach described above, this results in combined frequency mismatch metrics given as:

, частотный показатель несовпадения имел значение единицы, а применительно к

, частотный показатель несовпадения имел значение нуля. Таким образом, формируется показатель несовпадения между 0 и 1, где возрастающее значение является указывающим уменьшающееся различие. Будет принято во внимание, что, если возрастающее значение желательно для возрастающего различия, это может просто достигаться посредством определения:where the shift ½ is introduced so that, as applied to

, the frequency mismatch indicator had the value of one, and in relation to

, the mismatch frequency exponent was zero. Thus, a mismatch score between 0 and 1 is generated, where an increasing value is indicative of a decreasing difference. It will be appreciated that if an increasing value is desired for an increasing difference, this can simply be achieved by specifying:

Similarly, for the second approach, the following mismatch frequency metric can be determined:

вновь давая в результате частотный показатель несовпадения, попадающий в интервал [0;1].

again resulting in a mismatch frequency metric falling within the interval [0; 1].

As another example, the rationing in some embodiments may be based on the multiplication of the norms, and more precisely the L2 norms, of the individual sums of the frequency mismatch indicators:

This, in particular in many applications, can give very useful performance characteristics for the last example of a misalignment metric (ie, based on the L1 rate for the coefficients). In particular, the following mismatch frequency metric can be used.

Specific frequency indicators of mismatch, respectively, can be defined as:

- скалярное произведение, а

- норма

.where

is the dot product, and

- norm

...

The mismatch processor 309 can then generate a mismatch metric from the mismatch frequency metrics, combining them into a single mismatch metric indicating how similar the radiation patterns of the first beamformer 303 and the second beamformer 305 are.

More specifically, the mismatch metric can be defined as a frequency selective weighted sum of the mismatch frequency metrics. A frequency selective approach can be particularly useful for applying a suitable frequency window, for example, by allowing the pre-emphasis to be superimposed on specific frequency bands, such as the audio band or fundamental speech frequency bands, for example. For example, (weighted) averaging can be used to generate a robust broadband mismatch metric.

More precisely, the mismatch score can be defined as:

- подходящая весовая функция.where

is a suitable weighting function.

может быть сконструирована, чтобы учитывать, что речь главным образом активна в определенных полосах частот, и/или что микрофонная решетка имеет тенденцию иметь низкую направленность для относительно низких частот.As an example, the weighting function

may be designed to take into account that speech is primarily active in certain frequency bands, and / or that the microphone array tends to have low directivity for relatively low frequencies.

It will be appreciated that, while the above equations are represented in the continuous frequency domain, they can easily be translated into the discrete frequency domain.

, можем рассчитать:For example, discrete time-domain filters can first be converted to discrete frequency-domain filters by applying a discrete Fourier transform, that is, in relation to

, we can calculate:

представляет собой характеристику фильтра в дискретном времени

'ого формирователя диаграммы направленности для

'ого микрофона,

- длина фильтров во временной области,

представляет собой дискретный фильтр в частотной области

'ого формирователя диаграммы направленности для

'ого микрофона, а

- длина фильтров формы диаграммы направленности в частотной области, типично выбираемая в качестве

(зачастую то же самого числа, что и коэффициентов во временной области, хотя это не обязательно имеет место. Например, что касается количества коэффициентов во временной области, иного чем 2N, вставка нулей может использоваться для облегчения преобразования в частотную область (например, с использованием БПФ)).where

is the discrete-time response of the filter

'th beamformer for

'wow microphone,

- the length of the filters in the time domain,

is a discrete filter in the frequency domain

'th beamformer for

'oh mic, ah

is the length of the frequency domain shape filters, typically selected as

(often the same number as the coefficients in the time domain, although this is not necessarily the case. FFT)).

и

являются векторы

и

, которые получаются посредством собирания коэффициентов фильтра частотной области по индексу

частоты для всех микрофонов в вектор.Equivalents in the discrete frequency domain of the vectors

and

are vectors

and

, which are obtained by collecting the frequency domain filter coefficients by the index

frequencies for all microphones into vector.

подобия затем может выполняться следующим образом:Essentially, for example, calculating an indicator

the similarity can then be done as follows:

moreover,

представляет собой комплексное сопряжение.where

represents complex conjugation.

подобия может, на основании весовой функции

рассчитываться, как изложено ниже:In conclusion, broadband metric

similarity can, based on the weight function

be calculated as follows:

ведет к широкополосному показателю подобия, который ограничен между нулем и единицей, и который ровно взвешивает все частоты.Selecting the weight function as

leads to a broadband similarity measure that is bounded between zero and one, and which evenly weighs all frequencies.

Alternative weighting functions may focus on a specific frequency range (eg, due to the fact that it is likely to contain speech). In such a case, a weighting function that results in a similarity measure bounded between zero and one, for example, in such a case, can be chosen as:

и

- индексы частоты, соответствующие границам требуемого диапазона частот.where

and

- frequency indices corresponding to the boundaries of the required frequency range.

The derived mismatch metric provides particularly effective performance with different characteristics that may be desirable in different embodiments. In particular, certain values may be sensitive to different pattern mismatch properties and, depending on the particular embodiment, different metrics may be preferred.

несовпадения/подобия может рассматриваться для измерения несовпадений фазы, затухания и направления между формирователями диаграммы направленности, тем временем,

вновь учитывает только несовпадения коэффициента передачи и направления. В заключение показатель

несовпадения учитывает только несовпадения направления и игнорирует несовпадения фазы и затухания.Indeed, the indicator

mismatch / similarity can be considered to measure phase, attenuation and direction mismatches between beamformers, meanwhile,

again takes into account only the discrepancy between the transmission ratio and the direction. In conclusion, the indicator

mismatch only considers directional mismatches and ignores phase mismatches and attenuation.

. В этом случае, коэффициенты фильтра формирователя диаграммы направленности могут быть разложены, как изложено ниже:These mismatches are related to the beamformer structure. More specifically, suppose that the beamformer filter coefficients share a common (frequency independent) factor across all microphones, which we refer to as

... In this case, the coefficients of the beamformer filter can be decomposed as follows:

. Затем, рассмотрим два варианта общего фактора

.In abbreviated notation, we have

... Then, consider two options for the common factor

...

, также известного как фазовый фильтр. Во втором случае, допустим, что общий фактор имеет произвольный коэффициент передачи и сдвиг фазы для каждой частоты. Три представленных показателя подобия обращаются с этими общими факторами по-разному.In the first case, let's say that the total factor consists only of a (independent of frequency) phase shift, that is

also known as a phase filter. In the second case, let's say that the common factor has an arbitrary gain and phase shift for each frequency. The three similarity metrics presented treat these common factors differently.

чувствительно к общим расхождениям амплитуды и фазы между формирователями диаграммы направленности.

sensitive to general amplitude and phase discrepancies between beamformers.

чувствительно к общим расхождениям амплитуды между формирователями диаграммы направленности.

sensitive to general amplitude discrepancies between the beamformers.

нечувствительно к общему фактору.

insensitive to the general factor.

This can be seen from the following examples:

Example 1:

, причем,

является произвольной фазой для каждой частоты, то есть, фазовым фильтром.For this example, consider a scenario with

, moreover,

is an arbitrary phase for each frequency, that is, a phase filter.

This results in the following results for the similarity scores:

Example 2:

, причем

- произвольные коэффициент передачи и фаза для каждой частоты. Это дает в результате следующие результаты для показателей подобия:For this example, consider a scenario with

, and

- arbitrary gain and phase for each frequency. This results in the following results for the similarity scores:

несовпадения в многих вариантах осуществления может давать особенно привлекательный показатель.In many practical embodiments, there can be general gain and phase differences between the beamformers and, accordingly, the metric

mismatches in many embodiments can provide a particularly attractive metric.

In the following, a sound recording apparatus will be described in which the generated mismatch metric interacts with the other described elements to provide a particularly useful sound recording system. In particular, the approach is extremely suitable for recording sound sources in noisy and reverberant environments. It provides particularly useful performance for applications where the desired sound source may be outside the reverberation radius and the sound captured by the microphones may be dominated by stray noise and late reflections or reverberations.

FIG. 5 illustrates an example of elements of such a sound recording apparatus in accordance with some embodiments of the invention. The elements and approach of the system of FIG. 3 may correspond to the system of FIG. 5 as follows.

The sound recorder comprises a microphone array 501 which may correspond directly to that of FIG. 3. In the example, the microphone array 501 is connected to an optional echo canceller 503 that can cancel out echo that comes from acoustic sources (for which a reference is available) that are linearly coupled to the echo in the microphone signal (s). This source can be, for example, a loudspeaker. An adaptive filter can be applied with a reference signal as the input signal, and with the output signal subtracted from the microphone signal to create an echo canceled signal. This can be repeated for each individual microphone.

It will be appreciated that echo canceler 503 is optional and may simply not be included in many embodiments.

A microphone array 501 is coupled to a first beamformer 505, typically directly or via an echo canceller 503 (and possibly also via amplifiers, digital-to-analog converters, etc., as will be well known to one of ordinary skill in the art). The first beamformer 505 may correspond directly to the first beamformer 303 of FIG. 3.

The first beamformer 505 is configured to combine signals from the microphone array 501 so that an effective directional sonic response of the microphone array 501 is generated. The first beamformer 505 thus generates an output, referred to as the first beamforming audio output signal , which corresponds to the selective recording of sound in the environment. The first beamformer 505 is an adaptive beamformer, and the directivity can be controlled by setting parameters, referred to as first beamform parameters, to the effect of the beamform of the first beamformer 505.

The first beamformer 505 is coupled to the first adapter 507, which is adapted to adapt the parameters of the first beamform. Thus, the first adapter 507 is configured to adapt the parameters of the first beamformer 505 so that the radiation pattern can be directionally controlled.

In addition, the sound recorder comprises a plurality of limited beam drivers 509, 511, each of which is configured to combine signals from the microphone array 501 so that an effective directional sonic response of the microphone array 501 is formed. Each of the limited beam drivers 509, 511 such is thus configured to generate an audio output, referred to as a limited beamforming audio output that corresponds to selectively recording audio in an environment. Similar to the first beamformer 505, the constrained beam drivers 509, 511 are adaptive beamformers, where the directivity of each constrained beamformer 509, 511 may be controlled by settings referred to as the constrained beamform parameters of the constrained beamformers 509, 511 ...

The audio recorder suitably comprises a second adapter 513 that is configured to adapt the bounded beamform parameters of the plurality of bounded beamforming devices, thereby adapting the radiation patterns generated therefrom.

The second beamformer 305 of FIG. 3 may correspond directly to the first constrained beamformer 509 of FIG. 5. It will also be appreciated that the rest of the constrained beamformer 511 may correspond to the first beamformer 303 and could be considered instances of it.

Both the first beamformer 505 and the constrained beamformer 509, 511, respectively, are adaptive beamformers for which the actual generated beamformator can dynamically adapt. More specifically, the beam drivers 505, 509, 511 are filter and combiner (or, more specifically, filter and combiner in most embodiments). A waveform filter can be applied to each of the microphone signals, and the filtered outputs can be combined, typically by simply adding together.

It will be appreciated that the comments made with respect to first beamformer 303 and second beamformer 305 (eg, with respect to waveform filters) equally apply to beamformators 505, 509, 511 of FIG. 5.

In many embodiments, the design and implementation of the first beamformer 505 and the constrained beamformer 509, 511 may be the same, for example, the beamform filters may have identical FIR filter designs with the same number of coefficients, etc.

However, the operation and parameters of the first beamformer 505 and the limited beamformer 509, 511 will be different and, in particular, the limited beamformer 509, 511 is somewhat limited, and the first beamformer 505 is not. More specifically, the adaptation of the limited beamformer 509, 511 will be different than the adaptation of the first beamformer 505, and more specifically will be subject to some limitations.

More specifically, the constrained beam drivers 509, 511 are subject to the constraint that the adaptation (updating the waveform filter parameters) is limited to situations where a criterion is met, while the first beamformer 505 will be allowed to adapt even when such a criterion is not met. Indeed, in many embodiments, the first adapter 507 may be allowed to always adapt the beamform filter without being limited by any properties of the sound captured by the first beamformer 505 (or any of the limited beamformators 509, 511).

The criterion for adapting the limited beam shapers 509, 511 will be described in more detail later.

In many embodiments, the adaptation rate for the first beamformer 505 is higher than for the constrained beamformer 509, 511. Thus, in many embodiments, the first adapter 507 can be configured to adapt to changes faster than the second adapter 513, and thus the first beamformer 505 can update faster than the constrained beam drivers 509, 511. This, for example, can be achieved by lowpass filtering a maximized or minimized value (e.g., output signal level or error signal magnitude) having a higher cutoff frequency for the first beamformer 505 than for the constrained beamformer 509, 511 directionality. As yet another example, the maximum change to update the beamform parameters (more specifically, the beamform filter coefficients) may be greater for the first beamformer 505 than for the constrained beamformers 509, 511.

Accordingly, in the system, a plurality of focused (adaptation constrained) beamformers that adapt slowly, and only when a certain criterion is met, are supplemented with a standalone faster adapting beamformer that does not suffer from this constraint. Slower and more focused beamformators will typically give slower, but more accurate and reliable adaptation to a specific sound environment than a stand-alone beamformer, which, however, will typically be able to quickly adapt over a wider range of parameters.

In the system of FIG. 5, these beamformers are used synergistically together to provide improved performance, as will be described in more detail later.

The first beamformer 505 and the limited beam drivers 509, 511 are coupled to an output processor 515 that receives beamforming audio outputs from the beamformers 505, 509, 511. The exact output signal generated from the audio recorder will depend on the particular preferences and requirements of the particular embodiment. Indeed, in some embodiments, the output from the audio recorder may simply consist of audio outputs from beamforming drivers 505, 509, 511.

In many embodiments, the output from the output processor 515 is generated as a combination of audio outputs from the beam drivers 505, 509, 511. Indeed, in some embodiments, simple selective combining may be performed, such as selecting the audio outputs for which the signal-to-noise ratio is the highest, or simply the signal strength.

Thus, the selection and subsequent processing of the output of the output processor 515 may be specialized and / or different in different implementations / embodiments. For example, all possible focused beamforms outputs can be provided, selection can be made based on user-defined criteria (e.g., the most powerful presenter is selected), etc.

With regard to the application of speech control, for example, all output signals can be forwarded to a speech trigger recognizer that is configured to identify a specific word or phrase to initialize the speech control. In such an example, the audio output signal in which the trigger word or phrase is detected may accompany the trigger phrase to be used by the speech recognizer to elicit special commands.

For communication applications, for example, it may be useful to select the audio output that has the highest power, for example, for which a particular point sound source has been detected.

In some embodiments, post-processing, such as the noise suppression of FIG. 1 may be applied to the output of an audio recorder (eg, output processor 515). This can improve performance, for example, for voice communications. Non-linear operations may be included in such post-processing, although, for example, with some speech recognizers, it may be useful to limit the processing to include only linear processing.

In the system of FIG. 5, a particularly useful approach is adopted for capturing an audio signal based on the synergy and relationship between the first beamformer 505 and the constrained beamformer 509, 511.

To this end, the audio recorder comprises a mismatch processor 517 that is configured to determine a mismatch metric between one or more of the constrained beam shapers 509, 511 and the first beam shaper 505. The misalignment metric is indicative of the difference between the radiation patterns generated respectively by the first beamformer 505 and the constrained beamformer 509, 511. Thus, the mismatch metric for the first constrained beamformer 509 may indicate a difference between the radiation patterns that are generated by the first beamformer 505 and the first constrained beamformer 509. Thus, the misalignment metric may be indicative of how closely the two beamforming devices 505, 509 are adapted to the same sound source.

The mismatch processor 517 directly corresponds to the mismatch processor 309 of FIG. 3, and the approach described in this regard is directly applicable to the mismatch processor 517 of FIG. 5. Thus, the system of FIG. 5 uses the described approach to determine the misalignment metric between the first beamformer 505 and one of the constrained beamformer 509, 511 in response to comparing the adaptive impulse responses of the waveform filters of the first beamformer 505 with the adaptive impulse responses of the waveform filters limited beam former 509, 511. It will be appreciated that, in many embodiments, the misalignment metric may be determined for such a constrained beamformer 509, 511.

Thus, in the system of FIG. 5, the misalignment metric is generated to reflect the difference between the beamforming parameters of the first beamformer 505 and the first restricted beamformer 509, and / or the difference between beamforming audio outputs thereof.

It will be appreciated that generating, defining and / or using a mismatch metric is directly equivalent to generating, defining and / or using a similarity metric. Indeed, one can typically be considered a monotonically decreasing function of the other, and thus the mismatch indicator is also an indicator of similarity (and vice versa), where one simply indicates increasing differences in increasing values, and the other does so in decreasing values.

A mismatch processor 517 is coupled to the second adapter 513 and provides a mismatch indicator to it. The second adapter 513 is configured to adapt the constrained beam drivers 509, 511 in response to the misalignment metric. More specifically, the second adapter 513 is configured to adapt the bounded beamform parameters only for the bounded beam drivers for which a misalignment metric has been determined that satisfies the similarity criterion. Thus, if the misalignment metric has not been determined for the data of the constrained beam drivers 509, 511, or if the determined misalignment metric for the given beamformer 509, 511 indicates that the beamforms of the first beamformer 505 and this constrained beamformer 509, 511 the directions are not sufficiently similar, then the adaptation is not performed.

Thus, in the audio recording apparatus of FIG. 5, constrained beamformers 509, 511 are constrained by beam adaptation. More specifically, they are constrained to adapt only if the current beam formed by the confined beamformer 509, 511 is close to the radiation pattern generated by the autonomous first beamformer 505, i.e., the individual constrained beamformer 509, 511 adapts only if the first beamformer 505 is currently adapted to be close enough to the individual beamformer 509, 511.

The result of this is that the adaptation of the constrained beam shapers 509, 511 is controlled by the operation of the first beamformer 505 so that, in fact, the radiation pattern generated by the first beamformer 505 controls which of the constrained beam shapers 509, 511 optimized / adapted. This approach, more specifically, may result in constrained beam drivers 509, 511 having a tendency to adapt only when the desired sound source is close to the current adaptation of constrained beamformer 509, 511.

An approach of requiring similarity between radiation patterns in order to provide adaptations found in practice result in significantly improved performance when the desired sound source, the desired speaker in this case, is outside the reverberation radius. Indeed, it has been found that this will give highly desirable performance, in particular for weak sound sources in reverberant environments with a predominantly direct path sound component.

In many embodiments, the adaptation constraint may be subject to additional requirements.

For example, in many embodiments, adaptation may be a requirement that the signal-to-noise ratio of the beamformed audio output exceeds a threshold. Thus, the adaptation for an individual limited beamformer 509, 511 can be limited to scenarios in which it is sufficiently adapted and the signal on which the adaptation is based reflects the desired audio signal.

It will be appreciated that different approaches for determining the signal-to-noise ratio may be used in different embodiments. For example, the noise floor of the microphone signals can be determined by tracking the minimum of the smoothed power estimate and, for each frame or time slot, the instantaneous power is compared to that minimum. As another example, the noise floor of the beamformer output can be determined and compared to the instantaneous output power of the beamformed output.

In some embodiments, the adaptation of the constrained beamformer 509, 511 is limited to when a speech component has been detected in the output of the constrained beamformer 509, 511. This will provide improved performance for speech recording applications. It will be appreciated that any suitable algorithm or approach for detecting speech in an audio signal can be used.

It will be appreciated that the system of FIG. 3-7 typically operates using block or block processing. Thus, consecutive time slots or frames are defined, and the described processing can be performed within each time slot. For example, the microphone signals can be divided into processing time slots and, for each processing time slot, the beam shapers 505, 509, 511 can generate beamforming audio output for the time domain, determine the misalignment metric, select limited shapers 509. , 511 beamforms and update / adapt this limited beamformer 509, 511, etc. The processing time slots in many embodiments may be between 5ms and 50ms in duration.

It will be appreciated that, in some embodiments, different processing times may be used for different aspects and functions of the audio recorder. For example, the mismatch metric and selection of the constrained beamformer 509, 511 for adaptation may be performed at a lower rate than, for example, the beamforming processing time slot.

In many embodiments, the adaptation may depend on the detection of point sound sources in the beamformed audio outputs. Accordingly, in many embodiments, the audio recorder may further comprise a sound source detector 601, as illustrated in FIG. 6.

Sound source detector 601, more specifically, in many embodiments may be configured to detect point sound sources in second beamforming audio signals, and accordingly, sound source detector 601 is connected to limited beam drivers 509, 511 and receives beamformed audio outputs therefrom.

A point source of sound acoustically is a sound that emanates from a point in space. It will be appreciated that the sound source detector 601 may use different algorithms or criteria to evaluate (detect) whether a point sound source is present in the beamformed audio output from a given restricted beamformer 509, 511, and that the skilled person will give to be aware of the various such approaches.

The approach, more specifically, may be based on identifying characteristics of a single or dominant point source captured by the microphones of the microphone array 501. A single or dominant point source, for example, may be identified by looking at the correlation between signals on the microphones. If there is a high correlation, then the dominant point source is considered to be present. If the correlation is low, then it is considered that there is no dominant point source, but that the recorded signals come from many uncorrelated sources. Thus, in many embodiments, a point sound source can be considered a spatially correlated sound source, where the spatial correlation is reflected by the correlation of the signals from the microphone.

In this case, the correlation is determined after being filtered by the beamform filters. More specifically, the correlation of the output signal of the beamforming filters of the limited beam drivers 509, 511 can be determined, and if it exceeds this threshold, the point sound source can be considered detected.

In other embodiments, the point source may be detected by evaluating the content of the beamforming audio signals. For example, the sound source detector 601 may analyze the beamformed output signals, and if sufficient speech power is detected in the beamformed audio output signal, this may be considered appropriate for a point sound source and thus detecting a strong speech component may be considered as detecting. point source of sound.

The detection result is forwarded from the sound source detector 601 to the second adapter 513, which is configured to apply an adaptation in response thereto. More specifically, the second adapter 513 may be configured to adapt only limited beam drivers 509, 511 for which the sound source detector 601 indicates that a point sound source has been detected.

Thus, the sound recorder is configured to limit the adaptation of the limited beam shapers 509, 511, so that only beam shapers 509, 511 adapt, in which a point sound source is present in the generated radiation pattern, and the generated radiation pattern is close to that formed first beam former 505. Thus, adaptation is typically limited to limited beam drivers 509, 511 that are already close to the (desired) point sound source. The approach allows for very stable and accurate beamforming that works exceptionally well in environments where the desired sound source may be outside the reverberation radius. In addition, by actuating and selectively updating a plurality of constrained beam drivers 509, 511, this robustness and accuracy can be complemented by relatively fast response times, allowing the overall system to adapt quickly to fast moving or emerging sound sources.

In many embodiments, the audio recorder may be configured to adapt only one limited beamformer 509, 511 at a time. Thus, the second adapter 513 in each adaptation slot can select one and the limited beamformer 509, 511 and adapt only it, updating the parameters of the beamform.

The selection of single limited beam shaper 509, 511 will typically occur automatically upon selection of the limited beam shaper 509, 511 for adaptation, only if the current generated radiation pattern is close to that generated by the first beam former 505, and if a point sound source is detected in the radiation pattern. ...

However, in some embodiments, it may be possible for a plurality of constrained beam drivers 509, 511 to satisfy the criteria simultaneously. For example, if a point sound source is located adjacent to areas covered by two different bounded beam drivers 509, 511 (or, for example, it is in an overlapping area), the point sound source can be detected in both radiation patterns, and both could be adapted. , so that it is close to each other, the one and the other being adjusted to the direction of the point sound source.

Thus, in such embodiments, the second adapter 513 may select one of the constrained beamformers 509, 511 satisfying two criteria and only adapt this one. This will reduce the risk that the two radiation patterns will adapt towards the same point sound source, and thus reduce the risk of operations with those interfering with each other.

In fact, adapting the constrained beam shapers 509, 511 under the constraint that the corresponding misalignment rate should be low enough, and selecting only one of the constrained beam shapers 509, 511 for adaptation (e.g., at each time slot / processing frame) will result in to adaptations differing between the different limited beamformers 509, 511. This will tend to result in the constrained beamformer 509, 511 adaptable to cover different areas of the closest constrained beamformer 509, 511, automatically being selected to adapt / follow the sound source detected by the first beamformer 505. However, for example, in contrast to the approach of FIG. 2, the regions are not stationary or predetermined, but rather are generated dynamically and automatically.

It should be noted that the regions may be multi-path beamforming dependent and are typically not limited by the angular direction of the arrival regions. For example, areas may differ based on the distance to the microphone array. Thus, the term region can be viewed as indicating a reference to the positions in space at which the sound source will result in an adaptation that satisfies the similarity requirement in terms of the mismatch metric. Thus, this includes considering not only the direct path, but, for example, also reflections, if they are considered in the parameters of the radiation pattern, and in particular are determined based on both spatial and temporal aspects (and more precisely, depend from the total impulse responses of the radiation pattern filters).

The selection of a single limited beamformer 509, 511, more specifically, may occur in response to a fixed sound level. For example, sound source detector 601 may determine the sound level of each of the beamformed audio outputs from the constrained beam drivers 509, 511 that meet the criteria, and it may select the constrained beamformer 509, 511 resulting in the highest level. In some embodiments, the sound source detector 601 may select the constrained beamformer 509, 511 for which the point sound source detected in the beamformed audio output has the highest value. For example, the sound source detector 601 may detect the speech component in the beamformed audio outputs from the two limited beam shapers 509, 511 and proceed to select the one having the highest level of the speech component.

In an approach, highly selective adaptation of the constrained shapers 509, 511 is thus performed, resulting in one adapting only in certain circumstances. This allows for very reliable beamforming by limited beamforming devices 509, 511, resulting in improved capture of the desired sound source. However, in many scenarios, beamforming constraints can also result in slower adaptability and, in fact, in many situations, can result in new sound sources (eg presenters) that are undetectable or only very slow to adapt.

FIG. 7 illustrates the sound recorder of FIG. 6, but with the addition of a beamformer controller 701, which is connected to the second adapter 513 and the sound source detector 601. The beamformer controller 701 is configured to initialize the limited beamformer 509, 511 in certain situations. More specifically, the controller 701 of the beamformer may initialize the constrained beamformer 509, 511 in response to the first beamformer 505, and more specifically, may initialize the constrained beamformer 509, 511 to form a beamforming that corresponds to that of the first beamformer 505 directional patterns.

The beamformer controller 701 more specifically sets the beamform parameters of one of the restricted beamformers 509, 511 in response to the beamform parameters of the first beamformer 505, hereinafter referred to as first beamform parameters. In some embodiments, the filters of the constrained beamformer 509, 511 and the first beamformer 505 may be identical, for example, they may have the same architecture. As a specific example, both filters of the constrained beamformer 509, 511 and the first beamformer 505 may be FIR filters of the same length (i.e., a predetermined number of coefficients), and the current adapted coefficient values from the filters of the first beamformer 505 may be simply copied to the constrained beamformer 509, 511, that is, the coefficients of the constrained beamformer 509, 511 can be set to values from the first beamformer 505. Thus, the constrained beamformer 509, 511 will be initialized with the same beamforming properties that are currently being adapted by the first beamformer 505.

In some embodiments, the filter setting of the limited beamformer 509, 511 may be determined from the filter parameters of the first beamformer 505, but instead of being used immediately, they may be adapted prior to use. For example, in some embodiments, the FIR filter coefficients may be modified to initialize the beamform of the constrained beamformer 509, 511 to be wider than the beamform of the first beamformer 505 (eg, generated in the same direction).

The beamformer controller 701, in many embodiments, in some circumstances, accordingly may initialize one of the constrained beamformers 509, 511 with an initial beamforming pattern corresponding to that of the first beamformer 505. The system can then proceed to process the constrained beamformer 509, 511 as previously described, and more specifically, may proceed to adapt the constrained beamformer 509, 511 when it meets the previously described criteria.

The criteria for initializing the constrained beamformer 509, 511 may be different in different embodiments.

In many embodiments, the beamforming controller 701 may be configured to initialize the constrained beamformer 509, 511 if the presence of a point sound source is detected in the first beamforming audio output, but not in any of the constrained beamforming directivity of audio output signals.

Thus, the sound source detector 601 can determine if a point sound source is present in any of the beamformed audio outputs from the constrained beam drivers 509, 511 or the first beamformer 505. The detection / evaluation results for each beamforming audio output can be sent to a beamformer controller 701, which can evaluate them. If a point sound source is detected only for the first beamformer 505, but for none of the limited beam drivers 509, 511, this may reflect a situation in which a point sound source, such as a speaker, is present and detected by the first beamformer 505 , but none of the limited beam shapers 509, 511 were identified and adapted for a point sound source. In this case, the constrained beam drivers 509, 511 may never (or only very slowly) adapt to a point sound source. Therefore, one of the constrained beamformers 509, 511 is initialized to beamform corresponding to a point sound source. As such, this radiation pattern should probably be close to a point sound source, and it will (typically slowly but surely) adapt to this new point sound source.

Thus, the approach can combine and provide useful results from both fast first beamformer 505 and robust limited beamformer 509, 511.

In some embodiments, the controller 701 of the beamformer may be configured to initialize the constrained beamformer 509, 511 only if the misalignment metric for the constrained beamformer 509, 511 exceeds a threshold value. More specifically, if the smallest determined misalignment rate for the constrained beam drivers 509, 511 is below a threshold value, no initialization is performed. In such a situation, it may be possible that the adaptation of the constrained beamformer 509, 511 is closer to the desired situation, while the less reliable adaptation of the first beamformer 505 is less accurate and may be adapted to be closer to the first beamformer 505. Thus, in scenarios where the mismatch rate is low enough, it may be useful to allow the system to try to adapt automatically.

In some embodiments, the beamformer controller 701, more specifically, may be configured to initialize the constrained beamformer 509, 511 when a point sound source is detected for both the first beamformer 505 and one of the constrained beam drivers 509. 511 radiation patterns, but the mismatch indicator for them fails to satisfy the similarity criterion. More specifically, the beamformer controller 701 may be configured to set the beamform parameters for the first constrained beamformer 509, 511 in response to the beamform parameters of the first beamformer 505 if a point sound source is detected as in a faulty beamforming beamforming audio output from the first beamformer 505 and beamforming audio output from the constrained beamformer 509, 511, and the misalignment ratio exceeds a threshold value.

Such a scenario may reflect a situation in which the confined beamformer 509, 511 could possibly adapt to and capture a point sound source that is, however, different from the point sound source captured by the first beamformer 505. Thus, more accurately, it can reflect that the constrained driver 509, 511 may have captured the "erroneous" point sound source. Accordingly, the constrained beamformer 509, 511 can be reinitialized to beamform in the direction of the desired point sound source.

In some embodiments, the number of constrained beam drivers 509, 511 that are operable may vary. For example, an audio recorder may include functionality for generating a potentially relatively high number of constrained beam shapers 509, 511. For example, it can implement, say, up to eight simultaneous limited beam shapers 509, 511. However, for example, in order to reduce power consumption and computational load, not all of them can be active at the same time.

Thus, in some embodiments, the set of operable limited beam drivers 509, 511 is selected from a larger set of beam drivers. This can more accurately be done when the beamformer 509, 511 is initialized. Thus, in the examples above, initialization of the constrained beamformer 509, 511 (e.g., if a point signal source is not detected in any of the active constrained beam drivers 509, 511) can be achieved by initializing an inactive constrained beamformer 509, 511 from the constellation, thereby increasing the number of operable limited beam shapers 509, 511.

If all of the constrained beamformer 509, 511 are collectively active, initialization of the constrained beamformer 509, 511 may be performed by initializing the currently active beamformer 509, 511. The constrained beamformer 509, 511 to be initialized may be selected according to some suitable criterion. For example, the constrained beam drivers 509, 511 having the highest misalignment rate or the lowest signal strength may be selected.

In some embodiments, the constrained beamformer 509, 511 may be disabled in response to a suitable criterion being met. For example, constrained beam drivers 509, 511 may be disabled if the misalignment metric increases above a predetermined threshold.

A specific approach for controlling adaptation and tuning of constrained beam drivers 509, 511 in accordance with many of the examples described above is illustrated by the flowchart of FIG. eight.

The method begins at block 801 by initializing the next processing timeslot (eg, waiting for the start of the next processing timeslot, collecting a set of parameters for the processing timeslot, etc.).

Step 801 is followed by step 803, where it is determined if there is a point sound source detected in any of the radiation patterns of the restricted pattern drivers 509, 511.

If so, the method continues to block 805, where it is determined whether the mismatch metric meets the similarity criterion, and more specifically, whether the mismatch metric is below a threshold.

If so, the method continues to block 807, where the constrained beamformer 509, 511 in which the point sound source was detected (or which has the highest signal strength in the event that the point sound source was detected in more than one shaper 509, 511 radiation pattern), adapts, that is, the parameters of the radiation pattern (filter) are updated.

If not, the method continues to block 809 where the constrained beamformer 509, 511 is initialized, the beamform parameters of the constrained beamformer 509, 511 are set regardless of the beamform parameters of the first beamformer 505. The constrained beamformer 509, 511 being initializable may be a new constrained beamformer 509, 511 (i.e., a beamformer from a plurality of dormant beamformers), or it may be an already operational constrained beamformer 509, 511 for which new parameters of the directional pattern are provided.

Following blocks 807 and 809, the method returns to block 801 and waits for the next processing time slot.

If it is determined in step 803 that a point sound source is not detected in the beamformed audio output of any of the constrained beam drivers 509, 511, the method proceeds to step 811, where it is determined whether a point sound source is detected in the first driver. 505 beamforming, that is, whether the current scenario corresponds to the capture of a point sound source by the first beamformer 505, but none of the limited beamformers 509, 511.

If not, the point sound source has not been detected at all and the method returns to step 801 to await the next processing time slot.

Otherwise, the method proceeds to block 813, where it is determined whether the mismatch metric meets the similarity criterion, and more specifically, whether the mismatch metric is below a threshold (which may be the same or a different threshold / criterion than that used in block 805 ).

If so, the method proceeds to 815 where the constrained beamformer 509, 511 adapts for which the misalignment metric is below the threshold (or if more than one constrained beamformer 509, 511 meets the criterion, one may be selected, for example, with the lowest mismatch indicator).

Otherwise, the method proceeds to 817 where the shaper 509, 511 is initialized, the beamform parameters of the constrained beamformer 509, 511 are set regardless of the shape parameters of the first beamformer 505. The constrained beamformer 509, 511 being initializable may be a new constrained beamformer 509, 511 (i.e., a beamformer from a plurality of dormant beamformers), or it may be an already operational constrained beamformer 509, 511 for which new parameters of the directional pattern are provided.

Following blocks 815 and 817, the method returns to block 801 and waits for the next processing time slot.

The described approach of the audio recorder of FIG. 5-7 can provide useful performance in many scenarios and, in particular, can tend to provide an audio recorder with the ability to dynamically generate focused, stable and accurate radiation patterns to capture sound sources. Beam patterns will tend to adapt to cover different areas, and the approach, for example, can automatically select and adapt the nearest constrained beamformer 509, 511.

Thus, in contrast to the approach, for example in FIG. 2, there is no need to directly impose any special constraints on the direction of the radiation pattern or on the filter coefficients. Rather, the individual regions can be automatically generated / shaped by allowing the constrained beamformer 509, 511 to (conventionally) adapt only when there is a single dominant sound source and when it is close enough to the beamforming beamformator 509, 511. This is more precisely determined by considering filter coefficients that take into account both the direct field and the (first) reflections.

It should be noted that the use of extended impulse response filters (as opposed to using simple delay filters, that is, single coefficient filters) also allows for reflections to arrive some (specific) time after the direct field. Accordingly, the radiation pattern is determined not only by the spatial characteristics (from which directions the direct wave and reflection field comes), but also determined by the temporal characteristics (at what moments in time after the direct wave field the reflections come). Thus, the reference to radiation patterns is not only limited by spatial considerations, but also reflects the temporal content of the pattern filters. Likewise, region references include both purely spatial and temporal effects of beamform filters.

The approach can thus be considered shaping regions that are defined by the misalignment metric difference between the autonomous beamforming of the first beamformer 505 and the beamforming of the constrained beamformer 509, 511. For example, assume that the constrained beamformer 509, 511 has a source-focused beamform (both spatial and temporal). Suppose the source is silent and the new source becomes active, with the first beamformer 505 adapting to focus on it. In such a case, each source with space-time characteristics such that the distance between the beam pattern of the first beamformer 505 and the beamforming beamformer 509, 511 does not exceed a threshold value, can be considered to be in the region of the restricted beamformer 509, 511 ... Thus, the constraint on the first constrained beamformer 509 can be considered to translate into constraint in space.

Distance criterion for adapting the constrained beamformer together with a beamforming initialization approach (eg, copying the waveform filter coefficients) typically provides that the constrained beamformer 509, 511 beamforms in different regions.

The approach typically results in the automatic generation of regions reflecting the presence of sound sources in the environment instead of a predefined stationary system like the system of FIG. 2. This flexible approach allows the system to rely on space-time characteristics, such as those caused by reflections, which would be very difficult and difficult to account for in a predefined and stationary system (since these characteristics depend on many parameters, such as characteristics of size, shape room reverb, etc.).

It will be appreciated that the above description, for clarity, has described embodiments of the invention with reference to various functional diagrams, blocks, and processors. However, it will be apparent that any suitable distribution of functionality between different functional circuits, blocks or processors can be used without detracting from the invention. For example, functionality illustrated by being executed by separate processors or controllers may be performed by the same processor or controllers. Hence, references to specific functional blocks or diagrams should be considered only as references to a suitable means for providing the described functionality, rather than indicating a strict logical or physical structure or organization.

The invention may be practiced in any suitable form, including hardware, software, firmware, or a combination thereof. The invention may optionally be implemented, at least in part, as computer software running on one or more data processors and / or digital signal processors. Elements and components of an embodiment of the invention may be physically, functionally and logically implemented in any suitable manner. Indeed, the functionality can be implemented in a single block, multiple blocks, or as part of other functional blocks. As such, the invention can be implemented in a single block, or it can be physically or functionally distributed among different circuits, blocks and processors.

Although the present invention has been described in connection with some of the options for implementation, it is not intended to be limited to certain forms set forth in the materials of this application. Rather, the scope of the present invention is limited only by the attached claims. Additionally, although a feature may appear described in connection with specific embodiments, one skilled in the art will appreciate that various features of the described embodiments may be combined in accordance with the invention. In the claims, the term does not exclude the presence of other elements or steps.

Moreover, although listed separately, a plurality of means, elements, circuits, or method steps may be implemented, for example, in a single circuit, block, or processor. Additionally, although individual features may be included in different claims, these may be combined in an advantageous manner as possible, and inclusion in different claims does not imply that a combination of features is not feasible and / or useful. In addition, the inclusion of a feature in one category of claims is not intended to be limited to that category, but rather indicates that the feature is equally applicable to the categories of another claim as appropriate. Moreover, the order of the features in the claims does not imply any particular order in which the features are to be processed, and in particular the order of the individual steps in the method claim does not imply that the steps are to be performed in that order. Rather, the steps can be performed in any suitable order. In addition, singular mentions do not exclude plurality. Expressions in the singular, "first", "second", etc., do not eliminate plurality. The reference characters in the claims are provided only as clarifying examples, which are not to be construed as limiting the scope of the claims in any way.

Claims (28)

1. A beamforming sound recording device comprising: microphone array (301); a first beamformer (303) coupled to a microphone array (301) and configured to generate a first beamforming audio output, the first beamforming device being a filtering and combining beamforming device comprising a first plurality of beamforming filters , each of which has a first adaptive impulse response; a second beamformer (305) coupled to a microphone array (301) and configured to generate a second beamforming audio output, the second beamforming device being a filtering and combining beamforming device comprising a second plurality of beamforming filters , each of which has a second adaptive impulse response; and a mismatch processor (309) for determining a mismatch metric between the first beam former (303) and the second beamformer (305) in response to comparing the first adaptive impulse responses with the second adaptive impulse responses. 2. The beamforming sound recorder of claim 1, wherein the mismatch processor (309) is configured for each microphone of the microphone array (301) to determine the correlation between the first and second adaptive impulse responses for the microphone and to determine the mismatch metric in response to the combination of correlations for each microphone from the microphone array (301). 3. The beamforming sound recorder of claim 1, wherein the mismatch processor (309) is configured to determine frequency domain representations of the first adaptive impulse responses and second adaptive impulse responses; and determining a mismatch metric in response to the frequency domain representations of the first adaptive impulse responses and the second adaptive impulse responses. 4. The beamforming sound recorder according to claim 3, wherein the mismatch processor (309) is configured to determine mismatch frequency metrics for frequency domain representation frequencies; and determine the mismatch metric in response to the mismatch frequency metrics for the frequency domain representation frequencies; wherein the mismatch processor (309) is configured to determine the mismatch frequency metric for the first frequency and the first microphone of the microphone array (301) in response to the first frequency domain coefficient and the second frequency domain coefficient, wherein the first frequency domain coefficient is the frequency domain coefficient for the first frequencies for the first adaptive impulse response for the first microphone, and the second frequency domain coefficient is a frequency domain coefficient for the first frequency for the second adaptive impulse response for the first microphone; and the mismatch processor (309) is further configured to determine a mismatch frequency metric for the first frequency in response to a combination of mismatch frequency metrics for a plurality of microphones of the microphone array (301). 5. The beamforming sound recorder of claim 4, wherein the mismatch processor (309) is configured to determine a mismatch frequency metric for the first frequency and the first microphone in response to multiplying the first frequency domain coefficient and a conjugate value of the second frequency domain coefficient. 6. The beamforming sound recorder of claim 5, wherein the mismatch processor (309) is configured to determine a mismatch frequency metric for the first frequency in response to a real part of the mismatch frequency metric combination for the first frequency for a plurality of microphones of the microphone array (301). 7. The beamforming sound recorder of claim 5, wherein the mismatch processor (309) is configured to determine a mismatch frequency metric for the first frequency in response to a first frequency mismatch frequency metric combination rate for a plurality of microphones of the microphone array (301). 8. A beamforming sound recorder according to claim 6 or 7, wherein the mismatch processor (309) is configured to determine the mismatch frequency metric for the first frequency in response to at least one of the real part and norm of the mismatch frequency metric combination for the first frequency for the plurality of microphones of the microphone array (301) with respect to the sum of the L2 norm function for the sum of the first frequency domain coefficients and the L2 norm function for the sum of the second frequency domain coefficients for the plurality of microphone array microphones (301). 9. The beamforming sound recorder according to claim 6 or 7, wherein the mismatch processor (309) is configured to determine the mismatch frequency metric for the first frequency in response to the rate of mismatch frequency metric combination for the first frequency for a plurality of microphones of the microphone array (301) with respect to the product of the L2 norm function for the sum of the first frequency domain coefficients and the L2 norm function for the sum of the second frequency domain coefficients for a plurality of microphones of the microphone array (301). 10. A beamforming sound recording device according to any one of paragraphs. 4-9, in which the mismatch processor (309) is configured to determine the mismatch metric as a frequency selective weighted sum of the mismatch frequency metrics. 11. The beamforming audio recorder as claimed in any one of the preceding claims, wherein the first plurality of beamform filters and the second plurality of beamforming filters are finite impulse response filters having a plurality of coefficients. 12. A beamforming sound recording device according to any of the preceding paragraphs, further comprising: a plurality of limited beam drivers (509, 511) connected to the microphone array (301) and each of which is configured to generate a limited beamforming audio output signal, with each limited beamformer of the plurality of limited beam drivers (509, 511 ) beamforming is limited to beamforming in an area different from other limited beamforming areas of the plurality of constrained beam drivers (509, 511), wherein the second beamformer is a constrained beamformer of the plurality of constrained beam drivers (509, 511) directional patterns; the first adapter (507) for adapting the parameters of the beamform of the first beam former (505); a second adapter (513) for adapting the bounded beamform parameters for the plurality of bounded beamformers (509, 511); while the second adapter (513) is configured to adapt the parameters of the limited beamforms only for limited beamformators from a plurality of limited beamformators (509, 511), for which a misalignment factor has been determined that satisfies the similarity criterion. 13. The beamforming sound recorder according to claim 12, further comprising a sound source detector for detecting point sound sources in the second beamforming audio output signals; and wherein the second adapter (513) is configured to adapt the bounded beamforming parameters only for the bounded beam drivers for which the presence of a point sound source is detected in the bounded beamforming audio output signal. 14. A method of operation for a beamforming sound recording device comprising: microphone array (301); a first beamformer (303) coupled to a microphone array (301), the first beamformer (303) being a filtering and combining beamforming device comprising a first plurality of beamforming filters, each of which has a first adaptive impulse response; a second beamformer (305) connected to the microphone array (301), the second beamformer (305) being a filtering and combining beamforming device comprising a second plurality of beamforming filters, each of which has an adaptive impulse response; in this case, the mentioned method consists in the fact that: the first beamformer (303) generates a first beamforming audio output signal; the second beamformer (305) generates a second beamforming audio output; and determining the mismatch index between the radiation patterns of the first beam former (303) and the second beamformer (305) in response to comparing the first adaptive impulse responses with the second adaptive impulse responses.

Images