EP4747965A1

EP4747965A1 - Content-aware real-time level management of audio content

Info

Publication number: EP4747965A1
Application number: EP24825259.5A
Authority: EP
Inventors: Chunghsin YEH; Shaofan YANG; Kai Li; Ripinder SINGH
Original assignee: Dolby International AB; Dolby Laboratories Licensing Corp
Current assignee: Dolby International AB; Dolby Laboratories Licensing Corp
Priority date: 2023-06-23
Filing date: 2024-06-19
Publication date: 2026-05-27
Also published as: JP2026521033A; CN121399853A; WO2024260365A1

Abstract

Systems and methods for content-aware real-time level management of audio content. One example provides a method for real-time level management of audio content, the method comprising receiving an input audio signal; generating a short-time signal based on the input audio signal, identifying, with a sound source classifier, one or more sources of interest associated with the short-time signal, estimating, with a long-term energy estimator, a long-term Root Mean Squared energy (long-term RMS) for respective sources of interest of the one or more sources of interest, estimating, with an automatic gain control, a set of short-time gains based at least in part on the long-term RMS of at least one of the respective sources of interest, and applying the short-time gain to the short-time signal.

Description

CONTENT-AWARE REAL-TIME LEVEL MANAGEMENT OF AUDIO CONTENT

TECHNICAL FIELD

This application relates generally to audio signal processing and, more specifically, content-aware level management by estimating and applying a time-varying gain value in order to boost or attenuate respective sound sources.

BACKGROUND

Unless otherwise indicated herein, the materials described in this section are not prior art to the claims in this application and are not admitted as prior art by inclusion in this section.
Existing enhancement algorithms for noisy speech perform poorly when applied to content mixed with speech, music, noise, and the like. For example, known levelling approaches based on analyzing the distribution of short-time energy are less effective when the signal-to-noise ratio (SNR) is low. Additionally, known levelling approaches cannot distinguish between different audio sources based solely on the energy distribution.
Known levelling approaches which rely on a Voice Activity Detector (VAD) treat non-speech sources, such as music sources and noise sources, as a single source which may result in suboptimal audio outputs (for example, boosting noise) . Additionally, known levelling approaches which rely on source separation followed by a single-source levelling algorithm are computationally complex and may not be suitable for a variety of implementations.
It is with respect to these and other considerations that the disclosure made herein is presented.
BRIEF SUMMARY OF THE DISCLOSURE
With the increasing diversity of real-time communication, real-time streaming, user-generated content creation, and other similar applications, content-aware processing provides for enhancing audio sources’a ssociated with the interest of a user. For example, a real-time communication for a music lesson would require respective enhancement of speech and music audio content. Playing a musical instrument is typically louder than speech. Accordingly, content-aware real-time processing algorithms described herein may decrease the music volume and increase the speech volume. As a majority of live content creators do not have a dedicated sound engineer nor the ability to balance audio levels manually in real-time, embodiments described herein provide automatic processing tools to assist with real-time content such as cloud-based video conferencing with a laptop device, live broadcasting using a smart phone device, and the like.
Techniques are described for processing audio signals. Examples found herein provide for systems, devices, and methods to process audio content and, more specifically, for estimating and applying a time-varying gain so that respective audio sources are boosted or attenuated to achieve a constant energy level.
Some examples provide a framework including a sound source classification module, a confidence activation tracker module, a long-term energy estimation module, and a gain control module. The sound source classification module is configured to receive an input audio signal and identify one or more sources of interest associated with the input audio signal. The confidence activation tracker module is configured to monitor the output of the sound source classification module of a plurality of frames and validate the output of the sound source classification module. The long-term energy estimation module is configured to estimate a long-term RMS for respective sources of interest of the one or more sources of interest. The automatic gain control module is configured to estimate a set of short-time gains based at least in part on the long-term RMS of at least one of the respective sources of interest.
In some aspects, the sound source classification module identifies and classifies audio sources within the input audio signal, such as speech sources, music sources, and noise sources. During short-time frames, the sound source classification module outputs source activation probabilities for each identified audio source. The sound source classification module may be, for example, a machine learning model or some other audio analysis algorithm.
In some aspects, the confidence activation tracker module receives source activation probabilities for each audio source from the sound source classification module. The confidence activation tracker module may validate whether audio sources are activated based on the respective source activation probabilities exceeding a threshold for a predetermined number of frames. The predetermined number of frames may be unique for each audio source. In some aspects, when a source activation probability drops below the threshold, the confidence activation tracker module may determine that the respective audio source is inactive and resets tracking of the respective audio source.
In some aspects, the long-term energy estimation module operates only on audio sources that are indicated as active and valid by the confidence activation tracker module. The long-term energy estimator may estimate both instantaneous energy and accumulated energy for the active audio sources. The instantaneous energy may be stored in a energy history buffer associated with the respective audio source, while the accumulated energy is a sum of the instantaneous energies.
In some aspects, the long-term energy estimation module calculates the long-term RMS of the audio source by dividing the accumulated energy by a number of frames. The long-term energy estimation module may track the audio source having the greatest long-term RMS.
In some aspects, the automatic gain control module updates the short-time gains based on the long-term RMS and a target RMS. For example, initial short-time gains may be set to 1. The short-time gains may updated by adding the short-time gains with a gain increment. In some instances, the target RMS is based on the long-term RMS of the audio source with the greatest source activation probability.
In some aspects, the speed of the gain increment is controlled by an attack time parameter and a release time parameter. The attack time parameter may be implemented to calculate the gain increment when the short-time gain is smaller than the target gain. The release time parameter may be implemented to calculate the gain increment when the short-time gain is larger than the target gain.
In some aspects, a maximally-allowed gain is implemented for clipping prevention. For example, a maximally-allowed gain may be retained in a local history buffer from which the minimum is used to limit the current short-time gain. The maximally-allowed gain may be calculated by dividing a maximally-allowed amplitude by the maximal amplitude of the short-time signal.
In some aspects, the short-time gain is a frequency-dependent gain that is based on an available estimation of noise spectrum. For example, the frequency-dependent gain may be derived for noise bins (bins having smaller or equal amplitude compared to the noise spectrum) and non-noise bins (bins having greater amplitude compared to the noise spectrum) . The short-time gain may be applied to non-noise bins and/or noise bins.
Another example provides a method for real-time level management of audio content, the method comprising receiving an input audio signal; generating a short-time signal based on the input audio signal, identifying, with a sound source classifier, one or more sources of interest associated with the short-time signal, estimating, with a long-term energy estimator, a long-term Root Mean Squared energy (long-term RMS) for respective sources of interest of the one or more sources of interest, estimating, with an automatic gain control, a set of short-time gains based at least in part on the long-term RMS of at least one of the respective sources of interest, and applying the short-time gain to the short-time signal.
Various aspects of the present disclosure provide for processing of audio signals, and effect improvements in at least the technical fields of audio processing, audio encoding, audio decoding, virtual reality, object-based audio, and the like.
The embodiments described herein may be generally described as techniques, where the term “technique” may refer to system (s) , device (s) , method (s) , computer-readable instruction (s) , module (s) , component (s) , hardware logic, and/or operation (s) as suggested by the context as applied herein.
Features and technical benefits other than those explicitly described above will be apparent from a reading of the following Detailed Description and a review of the associate drawings. This Summary is provided to introduce a selection of techniques in a simplified form, and not intended to identify key or essential features of the claimed subject matter, which are defined by the appended claims.
DESCRIPTION OF THE DRAWINGS
These and other more detailed and specific features of various embodiments are more fully disclosed in the following description, reference being had to the accompanying drawings, in which:
FIG. 1 illustrates a block diagram of an example audio coding system in which various aspects of the present disclosure can be practiced.
FIG. 2 illustrates a block diagram of a framework for audio level management according to some aspects of the present disclosure.
FIG. 3 illustrates a block diagram of another framework for audio level management according to some aspects of the present disclosure.
FIG. 4 illustrates a block diagram of various example methods for audio level management, which may be performed by the audio coding system of FIG. 1, in accordance with various aspects of the present disclosure.
FIG. 5A illustrates a schematic block diagram of an example device architecture that may be used to implement various aspects of the present disclosure.
FIG. 5B illustrates a schematic block diagram of an example CPU implemented in the device architecture of FIG. 5A that may be used to implement various aspects of the present disclosure.

DETAILED DESCRIPTION

In the following description, numerous details are set forth, such as audio device configurations, timings, operations, and the like, in order to provide an understanding of one or more aspects of the present disclosure. It will be readily apparent to one skilled in the art that these specific details are merely examples and not intended to limit the scope of this application.
As used herein, the term “includes” and its variants are to be read as open-ended terms that mean “includes, but is not limited to. ” The term “or” is to be read as “and/or” unless the context clearly indicates otherwise. Such terms are to be read as having an inclusive meaning. For example, “A and B” may mean at least the following: “both A and B” , “at least both A and B” . As another example, “A or B” may mean at least the following: “at least A” , “at least B” , “both A and B” , “at least both A and B” . As another example, “A and/or B” may mean at least the following: “A and B” , “A or B” . When an exclusive-or is intended, such will be specifically noted (e.g., “either A or B” , “at most one of A and B” ) . The term “based on” is to be read as “based at least in part on. ” The term “one example implementation” and “an example implementation” are to be read as “at least one example implementation. ” The term “another implementation” is to be read as “at least one other implementation. ” The terms “determined, ” “determines, ” or “determining” are to be read as obtaining, receiving, computing, calculating, estimating, predicting, or deriving. In addition, in the following description and claims, unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skills in the art to which this disclosure belongs.
Various Acronyms that may appear throughout this disclosure and in the associated claims and/or drawings are listed below. Other commonly used acronyms and terms of art may be excluded from this list in the interest of brevity. Thus, a short list of acronyms is provided below as an easy reference for the reader.
IVAS –Immersive Voice and Audio Services
RMS –Root Mean Squared
STG –Short-Time Levelling Gain
AT –Attack Time
RT –Release Time
FIG. 1 illustrates a block diagram of an audio coding system in which various aspects of the present invention may be incorporated. The example audio coding system 100 includes an encoder 110 and a decoder 120. The input of the encoder 110 corresponds to a first signal path 105, while an output of the encoder 110 corresponds to a second signal path 115. The input of the decoder 120 corresponds to the second signal path 115, while an output of the decoder corresponds to a third signal path 125.
The encoder 110 is configured to receive, from the first signal path 105, one or more streams of audio information representing one or more channels of audio signals. The encoder 110 is further configured to process the received streams of audio information and generate an encoded signal, which may be output to the second signal path 115. At the second signal path 115, the encoded signal may be stored (e.g., captured, buffered and/or recorded) , or transmitted (e.g., via a wired or wireless communication medium) . The decoder 120 is configured to receive the encoded signal from the second signal path 115. The decoder 120 is further configured to process the received encoded signal and generate a decoded signal, which may be output to the third signal path 125. The decoded signal that is generated by the decoder 120 corresponds to a replica of the audio information previously received by the encoder 110 from the first signal path 105. At the third signal path, the decoded signal may be stored (e.g., captured and/or recorded) , transmitted (e.g., via a wireless or wired electronic communication medium) , or output to a listening device (e.g., an audio processing device such as a receiver, speaker, soundbar, gaming system, portable multifunction device (e.g., mobile phone or tablet) , smart glasses, XR/VR/AR/MR headset, etc. ) . The audio coding system 100 may be an audio system capable of implementing an audio codec standard, such as the Immersive Voice and Audio Services (IVAS) standard. In such an instance, the encoded signal at the second signal path 115 may correspond to an IVAS bitstream.
In various examples described herein, the terms “replica” and “replica signal” are not intended to mean that the streams of audio information are “identical” . Instead, the term “replica” may indicate that the streams of audio information are approximately the same as the original audio information (e.g., reconstructions representing the original audio information) . For example, when the encoder 110 uses a lossless encoding technique to generate the encoded signal, the decoder 120 can in principle recover a lossless version that is approximately the same as the original audio information from the streams. However, in examples where the encoder 110 uses a lossy encoding technique, the content of the recovered replica signal is generally not identical to the content of the original stream but it may be perceptually indistinguishable from the original content. Thus, the terms “replica” and “replica signal” are intended to cover both lossless and lossy encoding techniques as used herein.
Examples described herein provide for real-time content-aware level management (e.g., loudness levels) of audio content. In some instances, a time-varying gain is estimated and applied to audio sources within the audio content (for example audio objects) . The time-varying gain may boost or attenuate the audio sources to, in some instances, achieve a constant loudness level or energy level of the audio content. Examples described herein primarily refer to three types of audio included in audio content: music, speech, and noise. However, example systems, methods, and devices described herein may also be utilized in other scenarios that include different audio types, fewer audio types, or greater audio types. Types of audio may be referred to herein as audio sources or sound sources. Noise sources may be undesired noise (for example, audio that does not include speech or music) , such as background noise, ambient sounds, and other disruptive noises. In some instances, quiet or overtly-loud music or speech may be considered noise. Noise as referred to herein generally includes sounds that may not be processed further by the framework 200 (see FIG. 2) or the framework 300 (see FIG. 3) .
FIG. 2 illustrates a block diagram of a framework 200 for audio level management according to some aspects of the present disclosure. The framework 200 may be implemented, for example, by the encoder 110, the decoder 120, or a combination thereof. The framework 200 includes a sound source classification module 202, a confidence activation tracker module 204, a validation module 206, a long-term energy estimation module 208, and a gain control module 212. The long-term energy estimation module 208 may include an energy history buffer 210. The gain control module 212 may include a max-gain history buffer 214.
The input of the sound source classification module 202 corresponds to a first path 201. The output of the sound source classification module 202 corresponds to a second path 203. The sound source classification module 202 is coupled to the confidence activation tracker module 204 via the second path 203. The sound source classification module 202 is configured to receive input audio content via the first path 201 (for example, the audio information received via the first signal path 105 in FIG. 1) .
The sound source classification module 202 processes the input audio content to identify and classify audio sources within the input audio content. For example, the sound source classification module 202 include an audio analysis algorithm, such as a trained machine-learning model, that identifies audio sources or sounds of interest within the input audio content. Audio sources may include, for example, speech, music, noise, and the like. In some instances, the audio sources may be more specific, such as a type of music, particular instruments, a particular speaker, or the like.
The sound source classification module 202 generates source activation probabilities, which are values indicating a likelihood (e.g., a probability or confidence value) that the received input audio content includes an associated audio source. For example, for one frame of input audio content, the sound source classification module 202 may determine (e.g., by calculation, estimation, or using the audio analysis algorithm) that there is a 90%likelihood the input audio content includes speech, 32%likelihood the input audio content includes music, and 22%likelihood the input audio content is noise. In some instances, the sound source classification module 202 operates in real-time (e.g., on a per-frame basis) . In other instances, the sound source classification module 202 operates using historical data (e.g., using previous frames) . The sound source classification module 202 is configured to provide the source activation probabilities to the confidence activation tracker module 204 via the second path 203.
The input of the confidence activation tracker module 204 corresponds to the second path 203. The confidence activation tracker module 204 is configured to receive the source activation probabilities from the sound source classification module 202 via the second path 203. The output of the confidence activation tracker module 204 corresponds to a third path 205. The confidence activation tracker module 204 is coupled to the validation module 206 via the third path 205.
The confidence activation tracker module 204 processes the source activation probabilities from the sound source classification module 202 to determine an activation status of each sound source identified by the sound source classification module 202. For example, the confidence activation tracker module 204 performs the operations of a buffer and stores an indication as to whether each sound source is active during each frame. The confidence activation tracker module 204 may include, for example, a sliding window buffer. A separate buffer may be provided for the source activation probabilities associated with each sound source identified by the sound source classification module 202 such that each sound source is separately monitored.
To determine that a sound source is active, the confidence activation tracker module 204 may compare the source activation probabilities to a confidence threshold at each frame. For example, consider a speech audio source. When the source activation probability of the speech audio source is above a confidence threshold (for example, 90%confidence) for a predetermined number of frames (for example, 10 frames) , the confidence activation tracker module 204 determines that the speech audio source is active for that frame. When the source activation probability of the speech audio source is below the confidence threshold for a single frame, the confidence activation tracker module 204 determines the speech audio source is not active (e.g., is inactive) for that frame.
Additionally, the confidence activation tracker module 204 may determine whether a sound source being active is valid over a plurality of frames. For example, when the speech audio source is active for a predetermined number of frames (for example, 10 frames) , the confidence activation tracker module 204 determines that the activation of the speech audio source is valid. When the speech audio source is not active for a frame, the confidence activation tracker module 204 determines that the activation of the speech audio source is invalid and may restart monitoring of the speech audio source.
Confidence thresholds and the predetermined number of frames implemented by the confidence activation tracker module 204 may be set separately for each different audio source. In this manner, the confidence threshold and predetermined number of frames for a music audio source may be different than the confidence threshold and predetermined number of frames for a speech audio source, and each audio source may be handled differently (e.g., with content-specific thresholds for activation tracking by the confidence activation tracker module 204) .
Such operations are performed for each audio source (e.g., the music audio source) . In some implementations, the confidence activation tracker module 204 ignores source activation probabilities associated with noise audio sources. The confidence activation tracker module 204 provides source activation validity (e.g., whether each sound source is valid) to the validation module 206 via the third path 205.
In some instances, the input audio content received by the confidence activation tracker module 204 is a short-time signal including short-time frames (e.g., subframes of audio content) . Short-time frames may be, for example, a 20ms waveform represented by coding coefficients. The short-time frames may also be waveforms of other lengths, such as a 10ms waveform or a 25ms waveform. In other instances, the confidence activation tracker module 204 receives the input audio signal and generates a short-time signal based on the input audio signal.
The inputs of the validation module 206 correspond to the first path 201 and the third path 205. The validation module 206 is coupled to the confidence activation tracker module 204 via the third path 205. The validation module 206 receives input audio content via the first path 201. The validation module 206 receives the source activation status from the confidence activation tracker module 204 via the third path 205. The output of the validation module 206 corresponds to a fourth path 207. The validation module 206 is coupled to the long-term energy estimation module 208 via the fourth path 207.
The validation module 206 is configured to provide the input audio content to the long-term energy estimation module 208 via the fourth path 207 (as transferred input audio content) when the source activation status from the confidence activation tracker module 204 indicates that one of the audio sources is valid. When one of the audio sources is invalid, the validation module 206 does not provide the input audio content to the long-term energy estimation module 208. That is, as illustrated in FIG. 2, the Transferred Input Audio Content provided by validation module 206 to the long-term energy estimation module 208 via the fourth path 207 may be a subset of the Input Audio Content received by Sound Source Classification Module 202 via first path 201 (e.g., the validation module 206 passes only validated portions of Input Audio Content to the long-term energy estimation module 208) , or may include all of the initially-received Input Audio Content. In some instances, the operations of the validation module 206 may be performed by the confidence activation tracker module 204 or the long-term energy estimation module 208 rather than being performed by a separate module.
As the sound source classification module 202 and the confidence activation tracker module 204 may operate in real-time, the input audio content may be noisy. However, the operations of the sound source classification module 202 and the confidence activation tracker module 204 provide for the use of noisy audio while rejecting incorrect detections of an activated sound source, thereby providing for reduced complexity in audio analysis.
The input of the long-term energy estimation module 208 corresponds to the fourth path 207. The long-term energy estimation module 208 is coupled to the validation module 206 via the fourth path 207. The long-term energy estimation module 208 is configured to receive the input audio content from the validation module 206 via the fourth path 207. The output of the long-term energy estimation module 208 corresponds to a fifth path 209. The long-term energy estimation module 208 is coupled to the gain control module 212 via the fifth path 209.
The long-term energy estimation module 208 tracks the instantaneous energy and the accumulated energy of the audio sources within the input audio content. The energy history buffer 210 may store the accumulated energy which is updated by adding the instantaneous energy at each frame to the energy history buffer 210. The long-term energy estimation module 208 may track the instantaneous energy and the accumulated energy of the audio sources as real-time root-mean-square (RMS) values. For example, when the confidence activation tracker module 204 indicates that an audio source is valid, the long-term energy estimation module 208 first adds the instantaneous energy of the audio source to the energy history buffer 210. The long-term RMS value for the audio source is calculated by the long-term energy estimation module 208 by dividing the accumulated energy in the energy history buffer 210 by the number of “active” frames (e.g., an active frame count) .
In some instances, the long-term energy estimation module 208 also tracks which audio source has a highest long-term RMS value. For example, the long-term energy estimation module 208 stores the highest long-term RMS value and an indicator of the associated audio source. After calculating the long-term RMS value for active audio sources at each frame, the long-term energy estimation module 208 determines whether the long-term RMS value exceeds the highest long-term RMS value currently stored. If the long-term RMS value for an active audio source in the current frame exceeds the stored highest long-term RMS value, the long-term energy estimation module 208 overwrites the highest long-term RMS value.
The long-term energy estimation module 208 only refers to audio content with an “active” audio source. Should contiguous activation of an audio source not be retained (e.g., the confidence activation tracker module 204 indicates that activation of an audio source is “invalid” ) , the operations of the long-term energy estimation module 208 are not performed for the associated frames. In some examples, the energy history buffer 210 is a rolling window buffer such that an oldest value of the accumulated energy is replaced with a newest value of the accumulated energy. The long-term energy estimation module 208 is configured to provide the long-term RMS values to the gain control module 212 via the fifth path 209. The long-term energy estimation module 208 may also provide the stored highest long-term RMS value and the indicator of the associated audio source to the gain control module 212 via the fifth path 209.
The input of the gain control module 212 corresponds to the fifth path 209. The gain control module 212 is coupled to the long-term energy estimation module 208 via the fifth path 209. The gain control module 212 is configured to receive the long-term RMS values from the long-term energy estimation module 208 via the fifth path 209. The output of the gain control module 212 corresponds to a sixth path 211.
The gain control module 212 is configured to determine (e.g., estimate or calculate) a short-time levelling gain to apply to the audio content based on the long-term RMS values provided by the long-term energy estimation module 208 and based on a pre-determined target RMS. The gain control module 212 may determine the short-time levelling gain once a predetermined number of frames are processed by the long-term energy estimation module 208 and the associated audio source is “ready” for a gain update.
When updating the short-time levelling gain, the gain control module 212 accounts for all audio sources that are “ready” (e.g., are active for the predetermined number of frames) . The target gain (in dB) may be calculated by the gain control module 212 according to Equation 1:
targetGain_dB=targetRMS_dB-LTRMS_dB Equation (1)
where targetRMS_dB is the pre-determined target RMS and LTRMS is the long-term RMS value provided by the long-term energy estimation module 208.
In some instances, when multiple audio sources are ready for a gain update, the gain control module 212 calculates a single value of long-term RMS from the multiple audio sources. For example, the gain control module 212 may select the audio source having the greatest source activation probability for the given frame (or greatest average source activation probability over multiple frames) . In another example, the gain control module 212 selects the audio source associated with the highest long-term RMS value as provided by the long-term energy estimation module 208. In yet another example, the gain control module 212 calculates a weighted sum of the long-term RMS values for the multiple audio sources using the source activation probability values associated with the audio sources.
In some implementations, to prevent abrupt gain changes, additional parameters are implemented to control the short-time levelling gain (STG) calculation. For example, an attack time (AT) parameter may indicate the number of frames required to increase from the STG to the target gain. A release time (RT) parameter may indicate the number of frames required to decrease from the STG to the target gain. The AT and RT parameters may be unique for each audio source. Using these parameters, with the initial gain set to a value of 1, the STG may be updated according to Equations 2-4:
STG_dB=STG_dB+ΔG_dB Equation (2)
where ΔG is the change in the STG, and where:
In some implementations, AT and RT are time-varying parameters. For example, the AT and RT parameters may be adapted by the gain control module 212 such that ΔG remains similar or remains a fixed amount during gain boosting or attenuation. Additionally, the gain control module 212 may include a max-gain history buffer 214 that stores a maximally-allowed STG. In some instances, the maximally-allowed STG may be 1 divided by the maximum amplitude of the short-time waveform of the audio content. To prevent clipping, the maximally-allowed STG may be tracked by the gain control module 212 and implemented according to Equation 5:
STG_dB=min (STG_dB, MSTG_dB) Equation (5)
where MSTG is the maximally-allowed STG.
In some instances, when the maximally-allowed STG is updated at each short-time frame, the resulting STG may fluctuate. However, if the maximally-allowed STG is updated as a global minimum, the resulting STG may be ineffective in gain levelling. Accordingly, the gain control module 212 may track the maximally-allowed STG in the max-gain history buffer 214 and update the maximally-allowed STG locally. The duration at which the max-gain history buffer 214 is updated may be selected for each audio source to achieve effective levelling without clipping.
In some instances, only boosting gain is implemented (and not attenuation) . In such instances, weaker audio sources are boosted to a minimal target level, resulting in sufficient loudness. In such cases, the target gain may be constrained to the running maximum of the target gain.
The gain control module 212 applies the resulting STG to the input short-time audio signal to generate a levelled output signal. The gain control module 212 provides the levelled output signal via the sixth path 211. With respect to FIG. 1, the levelled output signal may be encoded by the encoder 110 and provided within the encoded signal via the second signal path 115.
In some instances, noise estimation and reduction may be performed to achieve a frequency-dependent STG. FIG. 3 illustrates a block diagram of a framework 300 for audio level management according to some aspects of the present disclosure. The framework 300 may be implemented, for example, by the encoder 110, the decoder 120, or a combination thereof. The framework 300 includes the sound source classification module 202, the confidence activation tracker module 204, the validation module 206, the long-term energy estimation module 208, a gain control module 312, and a noise estimation reduction module 302. The sound source classification module 202, the confidence activation tracker module 204, the validation module 206, and the long-term energy estimation module 208 may operate substantially similar as described with respect to the framework 200. The gain control module 312 may include a frequency-dependent gain buffer 304.
The input of the noise estimation reduction module 302 corresponds to the first path 201.
The noise estimation reduction module 302 receives the input audio content via the first path 201. The output of the noise estimation reduction module 302 corresponds to a seventh path 303. The noise estimation reduction module 302 is coupled to the gain control module 312 via the seventh path 303.
The noise estimation reduction module 302 is configured to process the received input audio content and classify the observed spectral bins of the input audio content into noise bins (having a spectral amplitude lower than an estimated noise spectrum) and non-noise bins (having a spectral amplitude greater than an estimated noise spectrum) . The estimated noise spectrum may be a predetermined (for example, pre-loaded) spectrum stored by the noise estimation reduction module 302. The noise estimation reduction module 302 is configured to provide the noise bins and the non-noise bins to the gain control module 312.
The inputs of the gain control module 312 correspond to the seventh path 303 and the fifth path 209. The gain control module 312 is coupled to the long-term energy estimation module 208 via the fifth path 209. The gain control module 312 receives the long-term RMS values from the long-term energy estimation module 208 via the fifth path 209. The gain control module 312 is coupled to the noise estimation reduction module 302 via the seventh path 303. The gain control module 312 receives the noise bins and the non-noise bins from the noise estimation reduction module 302 via the seventh path 303.
The gain control module 312 may operate in a similar manner as the gain control module 212 as described with respect to FIG. 2. The gain control module 312 may also calculate the STG further based on the received noise bins and non-noise bins from the noise estimation reduction module 302. For example, for noise bins, using a noise parameter α, the gain control module 312 may apply (1-∝) *STG_dB when STG_dB≥0 for gain boosting, and may apply (1+∝) *STG_dB when STG_dB<0 for gain attenuation, where 0 ≤ α ≤ 1. In this manner, the gain control module 312 prevents re-boosting of noise that may have been suppressed by the noise estimation reduction module 302. Accordingly, the SNR may be further improved by implementing the noise estimation reduction module 302. In some examples, the STG is smoothed at each frequency by a one-pole low-pass filter.
As the estimated noise spectrum may fluctuate between frames, the gain control module 312 may apply a simple smoothing to calculate a frequency-dependent STG (FD-STG) , as provided by Equation 6:
FD-STG [i, b] =β*FD-STD [i-1, b] + (1-β) *FD-STG [i, b] Equation (6)
where β is a smoothing parameter, i is a time index, and b is a bin index. The FD-STG may be stored in the frequency-dependent gain buffer 304.
In some implementations, the gain control module 312 also performs the operations of the noise estimation reduction module 302. In these implementations, the noise reduction and FD-STG may be combined such that the gain processing is performed by the gain control module 312 in one instance, as provided by Equation 7:
FD-STG_dB=FD-NRG_dB+FD-STG_dB Equation (7)
where FD-NRG_dB is the frequency-dependent noise reduction.
The illustrated blocks of modules in the framework 200 of FIG. 2 and the framework 300 of FIG. 3 are merely examples for real-time level management of audio content. In other examples, the framework 200, 300 may include additional blocks, may omit block, may combine the functions of blocks, or may divide portions of the blocks into additional blocks.
FIG. 4 illustrates a block diagram of various example methods 400 for real-time level management of audio content, which may be performed by the framework 200 of FIG. 2. While primarily described with respect to the framework 200, the methods 400 may also be performed by the framework 300 of FIG. 3. The methods 400 may be performed by a processor, which may be configured to perform methods 400 via machine-executable instructions. The methods 400 may be broken into various blocks or partitions, such as blocks 405, 410, 415, 420, 425, and 430. The various process blocks illustrated in FIG. 4 provide examples of various methods disclosed herein, and it is understood that some blocks may be removed, added, combined, or modified without departing from the spirit of the present disclosure. For some examples, processing of the various blocks, which may be described as processes, methods, steps, blocks, operations, or functions, may commence at block 405.
At block 405, “Receiving An Input Audio Signal” , an example method 400 may include receiving an input audio block included in a frame. For example and with reference to FIG. 2, the sound source classification module 202 receives the input audio signal via the first path 201 in a frame, as previously described. The input audio signal includes an input audio block. Processing may proceed from block 405 to block 410.
At block 410, “Generating A Short-Time Signal Based On The Input Audio Signal” , an example method 400 may include generating a short-time signal based on the input audio signal. For example and with reference to FIG. 2, the sound source classification module 202 generates a short-time signal based on the input audio signal, as previously described. In other examples, the input audio signal may be a short-time signal. Processing may proceed from block 410 to block 415.
At block 415, “Identifying One Or More Sources Of Interest” , the example method 400 may include identifying one or more sources of interest associated with the short-time signal. For example, with reference to FIG. 2, the sound source classification module 202 identifies and classifies audio sources within the input audio signal, as previously described. Processing may proceed from block 415 to block 420.
At block 420, “Estimating A Long-Term RMS For Respective Sources of Interest, ” the example method 400 may include estimating a long-term RMS for respective sources of interest of the one or more sources of interest, as previously described with respect to the confidence activation tracker module 204, the validation module 206, and the long-term energy estimation module 208. Processing may proceed from block 420 to block 425.
At block 425, “Estimating A Set Of Short-Time Gains, ” the example method 400 may include estimating a set of short-time gains (STGs) based at least in part on the long-term RMS of at least one of the respective sources of interest, as previously described with respect to the long-term energy estimation module 208 and the gain control module 212. Processing may proceed from block 425 to block 430.
At block 430, “Applying The Short-Time Gain To The Short-Time Signal, ” the example method 400 may include applying the short-time gain to the short-time signal, as previously described with respect to the gain control module 212.
FIG. 5A illustrates a schematic block diagram of an example device architecture 500 (e.g., an apparatus 500) that may be used to implement various aspects of the present disclosure. Architecture 500 includes but is not limited to servers and client devices, systems, and methods as described in reference to FIGS. 1-4. As shown, the architecture 500 includes central processing unit (CPU) 501 which is capable of performing various processes in accordance with a program stored in, for example, read only memory (ROM) 502 or a program loaded from, for example, storage unit 508 to random access memory (RAM) 503. The CPU 501 may be, for example, an electronic processor 501. In RAM 503, the data required when CPU 501 performs the various processes is also stored, as required. CPU 501, ROM 502, and RAM 503 are connected to one another via bus 504. Input/output interface 505 is also connected to bus 504.
The following components are connected to I/O interface 505: input unit 506, that may include a keyboard, a mouse, or the like; output unit 507 that may include a display such as a liquid crystal display (LCD) and one or more speakers; storage unit 508 including a hard disk, or another suitable storage device; and communication unit 509 including a network interface card such as a network card (e.g., wired or wireless) .
In some implementations, input unit 506 includes one or more microphones in different positions (depending on the host device) enabling capture of audio signals in various formats (e.g., mono, stereo, spatial, immersive, and other suitable formats) .
In some implementations, output unit 507 include systems with various number of speakers. Output unit 507 (depending on the capabilities of the hose device) can render audio signals in various formats (e.g., mono, stereo, immersive, binaural, and other suitable formats) .
In some embodiments, communication unit 509 is configured to communicate with other devices (e.g., via a network) . Drive 510 is also connected to I/O interface 505, as required. Removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a flash drive or another suitable removable medium is mounted on drive 510, so that a computer program read therefrom is installed into storage unit 508, as required. A person skilled in the art would understand that although apparatus 500 is described as including the above-described components, in real applications, it is possible to add, remove, and/or replace some of these components and all these modifications or alteration all fall within the scope of the present disclosure.
In accordance with example embodiments of the present disclosure, the processes described above may be implemented as computer software programs or on a computer-readable storage medium. For example, embodiments of the present disclosure include a computer program product including a computer program tangibly embodied on a machine readable medium, the computer program including program code for performing methods. In such embodiments, the computer program may be downloaded and mounted from the network via the communication unit 509, and/or installed from the removable medium 511, as shown in FIG. 5A.
FIG. 5B illustrates a schematic block diagram of an example CPU 501 implemented in the device architecture 500 of FIG. 5A that may be used to implement various aspects of the present disclosure. The CPU 501 includes an electronic processor 520 and a memory 521. The electronic processor 520 is electrically and/or communicatively connected to the memory 521 for bidirectional communication. The memory 521 stores levelling software 522. In some examples, memory 521 may be located internal to the electronic processor 520, such as for an internal cache memory or some other internally located ROM, RAM, or flash memory. In other examples, memory 521 may be located external to the electronic processor 520, such as in a ROM 502, a RAM 503, flash memory or a removable medium 511, or another non-transitory computer readable medium that is contemplated for device architecture 500. In some instances, the electronic processor 520 may implement the levelling software 522 stored in the memory 521 to perform, among other things, any of the methods 400 of FIG. 4.
Generally, various example embodiments of the present disclosure may be implemented in hardware or special purpose circuits (e.g., control circuitry) , software, logic or any combination thereof. For example, the units and modules discussed above can be executed by control circuitry (e.g., CPU 501 in combination with other components of FIG. 5A) , thus, the control circuitry may be performing the actions described in this disclosure. Some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software which may be executed by a controller, microprocessor or other computing device (e.g., control circuitry) . While various aspects of the example embodiments of the present disclosure are illustrated and described as block diagrams, flowcharts, or using some other pictorial representation, it will be appreciated that the blocks, apparatus, systems, techniques or methods described herein may be implemented in, as non-limiting examples, hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controller or other computing devices, or some combination thereof.
Additionally, various blocks shown in the flowcharts may be viewed as method steps, and/or as operations that result from operation of computer program code, and/or as a plurality of coupled logic circuit elements constructed to carry out the associated function (s) . For example, embodiments of the present disclosure include a computer program product including a computer program tangibly embodied on a machine readable medium, the computer program containing program codes configured to carry out the methods as described above.
In the context of the disclosure, a machine-readable medium may be any tangible medium that may contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may be non-transitory and may include but not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random-access memory (RAM) , a read-only memory (ROM) , an erasable programmable read-only memory (EPROM or Flash memory) , an optical fiber, a portable compact disc read-only memory (CD-ROM) , an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
Computer program code for carrying out methods of the present disclosure may be written in any combination of one or more programming languages. These computer program codes may be provided to a processor of a general-purpose computer, special purpose computer, or other programmable data processing apparatus that has control circuitry, such that the program codes, when executed by the processor of the computer or other programmable data processing apparatus, cause the functions/operations specified in the flowcharts and/or block diagrams to be implemented. The program code may execute entirely on a computer, partly on the computer, as a stand-alone software package, partly on the computer and partly on a remote computer or entirely on the remote computer or server or distributed over one or more remote computers and/or servers.
A person skilled in the art realizes that the present invention by no means is limited to the embodiments described above. On the contrary, many modifications and variations are possible and considered within the scope of the appended claims. Various aspects and implementations of the present disclosure may also be appreciated from the following enumerated example embodiments (EEEs) , which are not claims, and which may represent systems, methods, and devices, all arranged in accordance with aspects of the present disclosure.
EEE1. A method for real-time level management of audio content, the method comprising: receiving an input audio signal; generating a short-time signal based on the input audio signal; identifying, with a sound source classifier, one or more sources of interest associated with the short-time signal; estimating, with a long-term energy estimator, a long-term Root Mean Squared energy (long-term RMS) for respective sources of interest of the one or more sources of interest; estimating, with an automatic gain control, a set of short-time gains based at least in part on the long-term RMS of at least one of the respective sources of interest; and applying the short-time gain to the short-time signal.
EEE2. The method according to EEE1, wherein while identifying, with the sound source classifier, one or more sources of interest associated with the short-time signal, outputs of the sound source classifier are processed using a confidence activation tracker.
EEE3. The method according to any one of EEE1 to EEE2, wherein the sound source classifier is trained to classify sources including at least one of: speech sources, music sources, and noise sources.
EEE4. The method according to any one of EEE1 to EEE3, wherein the sound source classifier outputs, for each short-time frame of the short-time signal, instantaneous source activation probabilities for the respective sources of interest.
EEE5. The method according to any one of EEE1 to EEE4, further comprising: determining a source activation status based in part on probability thresholds pre-selected for each respective source of interest.
EEE6. The method according to any one of EEE2 to EEE5, wherein the confidence activation tracker counts contiguous active frames of a respective source.
EEE7. The method according to any one of EEE2 to EEE6, wherein the confidence activation tracker indicates a source of interest as valid when a contiguous active frame count for the respective source of interest is larger than a pre-defined quantity of frames.
EEE8. The method according to EEE7, wherein the pre-defined quantity of frames is defined per respective source of interest.
EEE9. The method according to any one of EEE6 to EEE8, wherein the contiguous active frame count resets in accordance with a determination that a respective source of interest is inactive.
EEE10. The method according to any one of EEE1 to EEE9, wherein the long-term energy estimator estimates instantaneous energy and accumulated energy for respective sources of interest of the one or more sources of interest.
EEE11. The method according to any one of EEE9 to EEE10, wherein the instantaneous energy is stored in an energy history buffer in accordance with a determination that the respective source is active.
EEE12. The method according to EEE11, wherein the accumulated energy is a sum of the instantaneous energies.
EEE13. The method according to any one of EEE2 to EEE12, wherein the long-term energy estimator updates the long-term RMS estimate in accordance with a determination by the confident activation tracker of a valid activation.
EEE14. The method according to any one of EEE10 to EEE13, wherein the long-term RMS is calculated by dividing the accumulated energy by a number of active frames.
EEE15. The method according to any one of EEE11 to EEE14, wherein the accumulated energy is updated by adding the energy values stored in the energy history buffer.
EEE16. The method according to any one of EEE1 to EEE15, wherein the long-term energy estimator tracks the maximum of the long-term RMS.
EEE17. The method according to any one of EEE1 to EEE16, wherein automatic gain control updates the short-time gain based on the long-term RMS and a target RMS.
EEE18. The method according to any one of EEE2 to EEE17, wherein the automatic gain control is considered “ready” to update when the confident activation tracker reports “valid” for the first time.
EEE19. The method according to any one of EEE1 to EEE18, wherein initial short-time gains are set to 1.
EEE20. The method according to any one of EEE1 to EEE19, wherein a target gain is calculated using a difference between a target RMS and a long-term RMS.
EEE21. The method according to EEE20, wherein the long-term RMS is based on the long-term RMS of the source with maximal source activation probability.
EEE22. The method according to EEE20, wherein the long-term RMS is based on the source with maximal long-term RMS.
EEE23. The method according to EEE20, wherein the long-term RMS is based on mean of long-term RMS among all the “active” sources.
EEE24. The method according to any one of EEE1 to EEE23, wherein the short-time gain is updated by adding the short time gain with a gain increment.
EEE25. The method according to any one of EEE20 to EEE24, wherein the target gain is constrained as the running maximum to allow only gain boosting.
EEE26. The method according to any one of EEE24 to EEE25, wherein the gain increment is calculated as the average increment per frame to change from the short-time gain to the target gain.
EEE27. The method according to any one of EEE24 to EEE26, wherein the speed of gain increment is controlled by attack time and release time.
EEE28. The method according to any one of EEE24 to EEE27, wherein attack time is used to calculate the gain increment when the short-time gain is smaller than the target gain.
EEE29. The method according to any one of EEE27 to EEE28, wherein release time is used to calculate the gain increment when the short-time gain is larger than the target gain.
EEE30. The method according to any one of EEE1 to EEE29, wherein the short-time gain is adapted further with respective to a maximally allowed gain for clipping prevention.
EEE31. The method according to EEE30, wherein the maximally allowed gain is a maximally allowed amplitude divided by the maximal amplitude of the short-time signal.
EEE32. The method according to EEE30 to EEE31, wherein the maximally allowed gain is retained in a local history buffer from which the minimum is used to limit the current short-time gain.
EEE33. The method according to EEE32, wherein the duration of the local history buffer is defined respectively for each source of interest.
EEE34. The method according to any one of EEE1 to EEE33, wherein the short-time levelling gain is converted into frequency-dependent gain based on an available estimation of noise spectrum.
EEE35. The method according to EEE34, wherein the frequency-dependent gain is derived for noise bins and non-noise bins.
EEE36. The method according to EEE35, wherein the noise bins are those having smaller or equal amplitude compared to the noise spectrum.
EEE37. The method according to EEE36, wherein the non-noise bins are those having larger amplitude compared to the noise spectrum.
EEE38. The method according to any one of EEE34 to EEE37, wherein the short-time levelling gain is applied to non-noise bins.
EEE39. The method according to any one of EEE34 to EEE38, wherein the short-time levelling gain applied to noise bins is multiplied by a factor of (1-α) while gain boosting and a factor of (1+α) while gain attenuating, where α is between 0 and 1.
EEE40. The method according to any one of EEE34 to EEE39, wherein the frequency-dependent gain at each frequency is smoothed by a one-pole low-pass filter.
EEE41. The method according to any one of EEE34 to EEE40, wherein the resulting frequency-dependent gain is combined with noise reduction gain and is applied within a single process.
EEE42. An electronic device, comprising: one or more processors; and a memory including one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the methods of any of EEE1 to EEE41.
EEE43. A non-transitory computer-readable storage medium storing one or more programs configured to be executed one or more processors of an electronic device, the one or more programs including instructions for performing the methods of any of EEE1 to EEE41.
With regard to the processes, systems, methods, heuristics, etc. described herein, it should be understood that, although the steps of such processes, etc. have been described as occurring according to a certain ordered sequence, such processes could be practiced with the described steps performed in an order other than the order described herein. It further should be understood that certain steps could be performed simultaneously, that other steps could be added, or that certain steps described herein could be replaced, amended, or omitted. In other words, the descriptions of processes herein are provided for the purpose of illustrating certain embodiments and should in no way be construed so as to limit the claims.
Accordingly, it is to be understood that the above description is intended to be illustrative and not restrictive. Many embodiments and applications other than the examples provided would be apparent upon reading the above description. The scope should be determined, not with reference to the above description, but should instead be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled. It is anticipated and intended that future developments will occur in the technologies discussed herein, and that the disclosed systems and methods will be incorporated into such future embodiments. In sum, it should be understood that the application is capable of modification and variation.
All terms used in the claims are intended to be given their broadest reasonable constructions and their ordinary meanings as understood by those knowledgeable in the technologies described herein unless an explicit indication to the contrary in made herein. In particular, use of the singular articles such as “a, ” “the, ” “said, ” etc. should be read to recite one or more of the indicated elements unless a claim recites an explicit limitation to the contrary.
The Abstract of the Disclosure is provided to allow the reader to quickly ascertain the nature of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the foregoing Detailed Description, it can be seen that various features are grouped together in various embodiments for the purpose of streamlining the disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the claimed embodiments incorporate more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter lies in less than all features of a single disclosed embodiment. Thus, the following claims are hereby incorporated into the Detailed Description, with each claim standing on its own as a separately claimed subject matter.

Claims

A method for real-time level management of audio content, the method comprising:

receiving an input audio signal;

generating a short-time signal based on the input audio signal;

identifying, with a sound source classifier (202) , one or more sources of interest associated with the short-time signal;

estimating, with a long-term energy estimator (208) , a long-term RMS (Root Mean Squared energy) for respective sources of interest of the one or more sources of interest;

estimating, with an automatic gain control (212) , a set of short-time gains based at least in part on the long-term RMS of at least one of the respective sources of interest; and

applying the short-time gain to the short-time signal.
The method of claim 1, wherein while identifying, with the sound source classifier (202) , one or more sources of interest associated with the short-time signal, outputs of the sound source classifier (202) are processed using a confidence activation tracker (204) .
The method of any of claims 1-2, wherein the sound source classifier (202) is trained to classify sources including at least one of: speech sources, music sources, and noise sources.
The method of any of claims 1-3, wherein the sound source classifier (202) outputs, for each short-time frame of the short-time signal, instantaneous source activation probabilities for the respective sources of interest.
The method of any of claims 1-4, further comprising: determining a source activation status based in part on probability thresholds pre-selected for each respective source of interest.
The method of any of claims 2-5, wherein the confidence activation tracker (204) counts contiguous active frames of a respective source.
The method of any of claims 2-6, wherein the confidence activation tracker (204) indicates a source of interest as valid when a contiguous active frame count for the respective source of interest is larger than a pre-defined quantity of frames.
The method of claim 7, wherein the pre-defined quantity of frames is defined per respective source of interest.
The method of any of claims 6-8, wherein the contiguous active frame count resets in accordance with a determination that a respective source of interest is inactive.
The method of any of claims 1-9, wherein the long-term energy estimator (208) estimates instantaneous energy and accumulated energy for respective sources of interest of the one or more sources of interest.
The method of any of claims 9-10, wherein the instantaneous energy is stored in an energy history buffer (210) in accordance with a determination that the respective source is active.
The method of claim 11, wherein the accumulated energy is a sum of the instantaneous energies.
The method of any of claims 2-12, wherein the long-term energy estimator (208) updates the long-term RMS estimate in accordance with a determination by the confident activation tracker (204) of a valid activation.
The method of any of claims 10-13, wherein the long-term RMS is calculated by dividing the accumulated energy by a number of active frames.
The method of any of claims 11-14, wherein the accumulated energy is updated by adding the energy values stored in the energy history buffer (210) .
The method of any of claims 1-15, wherein the long-term energy estimator (208) tracks the maximum of the long-term RMS.
The method of any of claims 1-16, wherein automatic gain control (212) updates the short-time gain based on the long-term RMS and a target RMS.
The method of any of claims 2-17, wherein the automatic gain control (212) is considered “ready” to update when the confident activation tracker (204) reports “valid” for the first time.
The method of any of claims 1-18, wherein initial short-time gains are set to 1.
The method of any of claims 1-19, wherein a target gain is calculated using a difference between a target RMS and a long-term RMS.
The method of claim 20, wherein the long-term RMS is based on the long-term RMS of the source with maximal source activation probability.
The method of claim 20, wherein the long-term RMS is based on the source with maximal long-term RMS.
The method of claim 20, wherein the long-term RMS is based on mean of long-term RMS among all the “active” sources.
The method of any of claims 1-23, wherein the short-time gain is updated by adding the short time gain with a gain increment.
The method of any of claims 20-24, wherein the target gain is constrained as the running maximum to allow only gain boosting.
The method of any of claims 24-25, wherein the gain increment is calculated as the average increment per frame to change from the short-time gain to the target gain.
The method of claim any of claims 24-26, wherein the speed of gain increment is controlled by attack time and release time.
The method of any of claims 24-27, wherein attack time is used to calculate the gain increment when the short-time gain is smaller than the target gain.
The method of any of claims 27-28, wherein release time is used to calculate the gain increment when the short-time gain is larger than the target gain.
The method of any of claims 1-29, wherein the short-time gain is adapted further with respective to a maximally allowed gain for clipping prevention.
The method of claim 30, wherein the maximally allowed gain is a maximally allowed amplitude divided by the maximal amplitude of the short-time signal.
The method of any of claims 30-31, wherein the maximally allowed gain is retained in a local history buffer (214) from which the minimum is used to limit the current short-time gain.
The method of claim 32, wherein the duration of the local history buffer (214) is defined respectively for each source of interest.
The method of any of claims 1-33, wherein the short-time levelling gain is converted into frequency-dependent gain based on an available estimation of noise spectrum.
The method of claim 34, wherein the frequency-dependent gain is derived for noise bins and non-noise bins.
The method of claim 35, wherein the noise bins are those having smaller or equal amplitude compared to the noise spectrum.
The method of claim 36, wherein the non-noise bins are those having larger amplitude compared to the noise spectrum.
The method of any of claims 34-37, wherein the short-time levelling gain is applied to non-noise bins.
The method of any of claims 34-38, wherein the short-time levelling gain applied to noise bins is multiplied by a factor of (1-α) while gain boosting and a factor of (1+α) while gain attenuating, where α is between 0 and 1.
The method of any of claims 34-39, wherein the frequency-dependent gain at each frequency is smoothed by a one-pole low-pass filter.
The method of any of claims 34-40, wherein the resulting frequency-dependent gain is combined with noise reduction gain and is applied within a single process.
An electronic device, comprising:

one or more processors; and

a memory including one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the methods of any of claims 1-41.
A non-transitory computer-readable storage medium storing one or more programs configured to be executed one or more processors of an electronic device, the one or more programs including instructions for performing the methods of any of claims 1-41.