RU2807170C2 - Dialog detector - Google Patents

Abstract

FIELD: computer technology.

SUBSTANCE: invention relates to the extracting audio features in a dialogue detector in response to an input audio signal. The technical result consists in increasing the performance of extracting sound features when using several context windows, each of which contains a different number of frames to represent the frame in different contexts. The technical result is achieved by dividing the input audio signal into many frames; extracting frame audio features from each frame I; defining a set of context windows, where each context window contains a number of frames surrounding the current frame; deriving, for each context window, a corresponding contextual audio feature for the current frame based on the frame audio features of the frames in each corresponding context; performing concatenation on each contextual audio feature to form a combined feature vector to represent the current frame; and obtaining a speech confidence score representing the probability of dialogue occurring in the current frame using the combined feature vector, wherein the number of frames in one or more context windows is determined adaptively based on the extracted frame audio features.

EFFECT: increasing the performance of extracting sound features when using several context windows.

12 cl, 12 dwg

Description

Cross reference to related applications

This application claims PCT priority to Patent Application No. PCT/CN2019/083173 filed April 18, 2019, US Provisional Patent Application No. 62/840,839 filed April 30, 2019, and European Patent Application No. 19192553.6 filed August 20, 2019 years, each of which is incorporated by reference herein in its entirety.

Field of technology to which the invention relates

The present invention relates generally to audio signal processing, and in particular to a dialogue detector.

BACKGROUND OF THE INVENTION

The dialogue detector is a key component in many audio signal processing algorithms, such as dialogue enhancement, noise reduction, and loudness metering. Typically, in an existing dialogue detector, the input audio signal is first converted into a uniform format in a pre-processing component through sample rate conversion or down-mixing, etc. For example, as pre-processing, the input audio signal may be down-mixed into a mono audio signal . Then the processed audio signal is divided into short-term frames and audio features are extracted from a context window, which includes a fixed number of frames, to describe the characteristics of each frame. A classifier built using machine learning techniques is then used to automatically assign the audio features a confidence score representing the likelihood of dialogue occurring. Finally, post-processing, such as a median or mean filter, can be used to remove or smooth out unwanted fluctuations in the resulting confidence scores. If the confidence score is high, the signal will be classified as a dialogue. The dialogue signal may then be sent to an audio enhancement device such as a dialogue enhancer.

Brief description of the invention

A first aspect of the present invention relates to a method for extracting audio features in a dialogue detector in response to an input audio signal, the method comprising dividing the input audio signal into a plurality of frames, extracting frame audio features from each frame, defining a set of context windows, where each context window contains a number of frames surrounding the current frame, deriving for each context window a corresponding contextual audio feature for the current frame based on the frame audio features of the frames in each corresponding context, and performing concatenation on each contextual audio feature to form a combined feature vector to represent the current frame.

Thus, the present invention proposes the use of multiple context windows, each containing a different number of frames, to represent a frame in different contexts, where context windows of different lengths will play different roles in representing the audio property of the target frame. Context windows of varying lengths can improve responsiveness and improve reliability. To this end, the present invention introduces a new process, combined temporal context determination, for determining multiple, eg three, context windows of different lengths or ranges, eg short-term context, medium-temporal context and long-term context; then the audio features in the contexts are extracted in the audio feature extraction component.

In some embodiments, the frame feature extractor component extracts frame audio features (i.e., frame audio features) from each of a plurality of frames extracted from the input audio signal, and the combined temporal context determines the length or range of each context window. Then, the corresponding contextual audio feature is derived based on the frame audio features in each determined context. Each contextual audio feature is then concatenated to form a combined feature vector to represent the current frame.

In some embodiments, context windows include short-term context, medium-term context, and long-term context. Short-term context represents local information around the current frame. The mid-temporal context further contains many flashback frames. The long-term context further contains many long-term history frames.

In some embodiments, the length or range of one or more contexts (i.e., the number of frames in the respective context windows) may be predetermined. For example, if a lookahead buffer is available, the short-lived context may contain the current frame and the lookahead frames. The mid-temporal context may contain the current frame, look-ahead frames, and look-ahead frames. A long-term context can contain the current frame, lookahead frames, lookahead frames, and long-term history frames. In one embodiment, the length or range of lookahead frames may be predefined by up to 23 frames, and the length or range of lookahead frames may be predefined by up to 24 frames, and the length or range of long-term history frames may be predefined by 48 to 96 frames. . In another example, if the lookahead buffer is not available, the short-term context may contain the current frame and the first part of the lookahead frames. The medium-temporal context may comprise a current frame, a first portion of flashback frames, and a second portion of flashback frames. The long-term context may contain the current frame, a first portion of flashback frames, a second portion of flashback frames, and long-term history frames. Therefore, the length or range of the first portion of the flashback frames may be predetermined by a size of up to 23 frames, and the length or range of the second portion of the flashback frames may be predetermined by a size of up to 24 frames, and the length or range of long-term history frames may be predetermined by a size of 48 to 96 frames.

In some embodiments, the length or range of one or more contexts may be determined adaptively by analyzing the stationarity of a frame-level feature. For example, adaptive detection is based on information related to the amplitude of the input audio signal. In particular, one method for adaptively determining the length or range of a short-term context is based on detecting a strong onset or transition. In another example, the adaptive determination is based on information related to the spectrum of the input audio signal. In particular, one method for adaptively determining the length or range of short-term context is based on identifying the largest spectral mismatch using Bayesian information criteria. Additionally, in adaptive detection implementations, short-term context may propagate in both the lookahead and lookahead directions, or propagate in only one direction. In some embodiments, the length or range of contexts may be predetermined in combination with adaptive determination.

In addition, the present invention provides a pre-cleaning method for eliminating uncorrelated signal noise to improve detection accuracy in a low signal-to-noise ratio conversation. To this end, the present invention uses downmixing with time- and frequency-dependent gains, with greater emphasis on the correlated signal.

In some embodiments, the input audio signal is first divided into a plurality of frames, and then the frames in the left channel and the right channel are converted into a spectral representation of the frames. Uncorrelated signals in the left channel and right channel are removed by applying frequency-dependent gains to the spectrum in the left channel and right channel, respectively, to obtain the signal after downmixing. In some embodiments, frequency-dependent gains may be calculated from the covariance matrix.

In addition, the present invention provides a music content detector such that both the music fidelity score and the speech fidelity score can be jointly considered to correct the original dialogue fidelity score and obtain the final adjusted dialogue fidelity score to significantly reduce false positives in music.

In some embodiments, the speech content detector receives features extracted using context windows, and then the speech content detector determines a speech confidence score. Then the music content detector takes the features extracted using context windows, and then the music content detector determines the music confidence score. The speech fidelity score and the music fidelity score are combined to produce a final dialogue fidelity score. In some embodiments, the final dialogue fidelity score may be refined through a context-sensitive parameter, which may be calculated based on the proportion of frames identified as speech or music in the historical context. In some embodiments, the historical context may be ten seconds or more in duration.

Brief description of graphic materials

The accompanying figures are for illustrative purposes and serve only to present examples of possible operations for the disclosed innovative methods, system, and computer-readable media. These figures are in no way intended to limit any changes in form and detail that may be made by one skilled in the art within the spirit and scope of the disclosed embodiments.

Fig. 1 is a block diagram of a conversation detector 100 including a combined temporal context determination component 104 in accordance with some embodiments.

Fig. 2 is a block diagram of a component 102 for extracting audio features in a dialogue detector 100 in accordance with some embodiments.

Fig. 3 is a flow diagram of a method 200 in accordance with some embodiments for pre-determining the length or range of a combined temporal context using the combined temporal context determination component 104.

Fig. 4a is an example of a predetermined length or range of a combined temporal context if a lookahead buffer is available, in accordance with some embodiments.

Fig. 4b is an example of a predetermined length or range of a combined temporal context if the lookahead buffer is not available, in accordance with some embodiments.

Fig. 5a is a flow diagram of an example method 300, in accordance with some embodiments, for adaptively determining the length or range of a combined temporal context using the combined temporal context determining component 104.

Fig. 5b is a schematic view of a method 300 illustrated in a context boundary search range.

Fig. 6a is a flow diagram of another example of a method 400, in accordance with some embodiments, for adaptively determining the length or range of a combined temporal context using the combined temporal context determining component 104.

Fig. 6b is a schematic view of the method 400 illustrated in the Bayesian Information Criteria window.

In fig. 7 illustrates an example of an adaptively determined length or range of a combined temporal context if a lookahead buffer is available, in accordance with some embodiments.

Fig. 8 is a flow diagram of an example method 500 of downmixing a dialogue detector input audio signal that is performed in accordance with some embodiments.

Fig. 9 is a block diagram of a dialogue detector 600, further including a music content detector 606, in accordance with some embodiments.

Detailed description

As noted, in a conventional conventional conversation detector, each frame is represented by a context, namely a window containing a number of frames (eg, 32 or 48 frames), and is classified by audio features extracted from the frames in that context window. However, a problem with this conventional dialogue detector is that it can sometimes introduce a large delay in detection since the detector can only determine whether dialogue is present after identifying several frames of dialogue, which can be detrimental to real-time applications. In addition, it cannot extract more reliable rhythmic features that can contribute to speech recognition from singing voices or recitatives, and consequently, this may have a negative impact on the reliability of dialogue detection.

To solve these problems, the present invention proposes methods that use a set of context windows of different lengths to represent a frame at multiple scales, where context windows of different lengths will play different roles in representing the audio property of the target frame. The following is a description of some examples of methods, systems, and computer-readable media that implement these methods for extracting an audio feature of a dialogue detector given an input audio signal.

Fig. 1 is a block diagram of a dialog detector 100 including a combined time context window determination component 104 in accordance with some embodiments. In fig. 1, the pre-processing component 102 receives an input audio signal. At pre-processing component 102, the input audio signal may be downmixed into a monaural audio signal. It is then broken down into frames. Then, the combined temporal context determination component 104 and the audio feature extraction component 106 receive frames from the preprocessing component 102, respectively. Frame audio features are then extracted from each frame in the audio feature extraction component 106. In addition, in the combined time context definition component 104, the length or range of each context window is determined. Then, the audio feature extraction component 106 receives the determination result from the combined temporal determination component 104. Then, in the audio feature extraction component 106, the frame audio features in each context window are used to derive each context feature depending on the specific context window. Each set of contextual features is then concatenated or combined to form a joint feature vector. The classifier 108 then receives the vector of extracted features from the audio feature extraction component 106 . At classifier 108, a confidence score representing the probability of having a conversation is obtained. Finally, in post-processing component 110, the resulting confidence scores may be smoothed, for example, by a median or mean filter, to remove unwanted fluctuations therein.

Fig. 2 is a block diagram of an audio feature extraction component 106 in a dialogue detector 100 in accordance with some embodiments. In particular, this figure describes the extraction and combination of features of a combined temporal context. In fig. 2, in the audio feature extractor component 106, frame audio features are extracted from each received frame by the frame feature extractor component 1060. The combined temporal context definition component 104 then determines the length or range of each context window, in this case the short-term context window, the medium-temporal context window, and the long-term context window. Then, based on the frame audio features in the short-term context window, the medium-time context window, and the long-term context window, the short-term context audio feature, the mid-temporal context audio feature, and the long-term context audio feature are output respectively. Finally, these three sets of contextual features are concatenated to form a high-dimensional feature vector. For example, if, suppose, each contextual feature is 100-dimensional, then the resulting feature by concatenation will be 300-dimensional.

Therefore, instead of representing the current frame with a single context window, the present invention proposes the use of multiple context windows. In one embodiment, three context windows are used to represent the current frame, namely a short-term context window, a medium-term context window, and a long-term context window of varying lengths or ranges. In particular, short-term context represents local information around the target frame so that when dialogue appears, the detector can respond faster. Medium-temporal context is an element used in the existing detector because it can provide an appropriate time interval for analyzing audio content. The long-term context window represents more global information in which only rhythmic features are extracted, since the short-term context or medium-term context window is typically not long enough to allow reliable rhythmic features to be extracted. That is, the present invention proposes adding a short-term context window to improve performance and a long-term context to improve reliability. Therefore, the length of these three context windows should be determined when extracting features. To this end, the present invention provides a combined timing component for determining the length of a short-term context window, a medium-term context window, and a long-term context window.

In one example, the frame audio feature may comprise at least one of sub-band features or full-band features. Examples of subband features include: subband spectral energy distribution, subband spectral contrast, subband partial overshoot, pure tone Fourier cosine transform coefficients (MFCC), MFCC flux, and low pass energy. Examples of full range features include: spectral flux, spectral residual, and short-term energy.

In one example, a contextual audio feature may be derived from one or more frame audio features. For example, the contextual audio feature may include frame audio feature statistics such as mean, mode, median, variance, or standard deviation.

Additionally or alternatively, the contextual audio feature may include a rhythm-related feature, such as a 2D modulation feature, rhythm power, rhythm purity, rhythm regularity, average tempo, and/or window-level correlation (i.e., context-level correlation).

The above examples of frame audio features and contextual audio features are not exhaustive, and various other frame audio features and contextual audio features may be used instead of or in addition to the above features.

Fig. 3 is a flow diagram of a method 200, in accordance with some embodiments, for pre-determining the length or range of a combined time context using the combined time context determination component 104. In this exemplary embodiment, the length or range of the combined time contexts may be predetermined. In one example, if a lookahead buffer is available, at step 202 it may be determined that the short-term context contains only the current frame and a few lookahead frames, and the length or range of the lookahead frames may be predetermined to be 23 frames, thus the total length or range short-term context is 24 frames, so that the most recent incoming content can be analyzed. At step 204, the medium-temporal context may be determined to include a current frame, several lookahead frames, and several look-ahead frames, and the length or range of the flashback frames may be predetermined to be 24 frames, such that the total length or range of the medium-temporal context is 48 frames. The long-term context may then be determined at 206 to include a current frame, several lookahead frames, some look-ahead frames, and more history frames, wherein the length or range of the long-term history frames may be predetermined to be 48-96 frames; thus, the total length or range of long-term context is from 96 to 144 frames, so as to have an analysis of stable rhythmic features. In fig. 4a shows this example of a predetermined length or range of a combined temporal context. In another example, if there is no lookahead buffer, the short-term context may be determined at step 208 to contain only the current frame and a portion of the flashback frames, and the length or range of the portion of the flashback frames may be predetermined to be 23 frames, thus the total length or The short-term context range is 24 frames. At step 210, the medium-temporal context may be determined to include the current frame, a portion of the flashback frames, and additional flashback frames, wherein the length or range of the additional flashback frames may be predetermined to be 24 frames, such that the total length or range of the medium-temporal context is 48 frames. . The long-term context may then be determined at 212 to include the current frame, a portion of the flashback frames, additional flashback frames, and more history frames, wherein the length or range of the long-term history frames may be predetermined to be 48-96 frames; thus, the total length or range of the long-term context is from 96 to 144 frames. In fig. 4b shows this example of a predetermined length or range of a combined temporal context. In method 200, the length or range of the lookahead buffer, lookahead buffer, and long-term history can all be predetermined. Alternatively, a different number of frames may be used in addition to the above number of frames, as long as it ensures that the short-term context contains only frames with a property similar to the current frame, and the long-term context contains enough history frames to extract reliable rhythmic features.

Alternatively, the length or range of one or more context windows may be adaptively determined in the combined temporal context determination component 104 by analyzing the stationarity of frame-level features and grouping audio frames accordingly. Fig. 5a is a flow diagram of an example method 300, in accordance with some embodiments, for adaptively determining the length or range of a combined time context window using the combined time context determination component 104. In particular, short-term context is taken as an example to describe the method 300. Method 300 is based on power transient detection. First, at step 302, the short-term energy is calculated frame k according to the following formula (1):

Here [x _k,0 , …, x _{k, N-1} ] represent PCM samples of frame k. Before energy is calculated, samples can also be windowed/weighted and energy can be derived from either the full range or sub-range signal.

Then at step 304 the energy The frame is asymmetrically smoothed with a fast advance rate as energy increases and a slow decay rate as energy decreases, as presented in formula (2):

Where – smoothed short-term energy in kth sound frame. The parameter α is the smoothing coefficient.

A difference filter is then applied to the smoothed energy envelope at step 306, and values greater than a predetermined threshold δ may be considered to be the beginning of , as presented in formula (3):

Then at step 308 can further be normalized by the average value of short-term energy in the search range. Then, at any one of steps 310, 312, or 314, a boundary for the length or range of the short-term context may be determined. At step 310, the position with the largest taken as the boundary of the context. At step 312 the peak above a certain threshold value, for example 0.3 (it can be adjusted between 0 and 1), can be taken as the context boundary. Instead of a threshold value, step 314 may consider the distance between and a previously identified strong peak. In other words, only if it has a certain duration, for example one second, from the previous strong transient, will it be defined as a strong transient and selected as a context boundary. In addition, at step 314, if a strong transient is not detected in the search range, then full flashback frames and/or lookahead frames are used. Fig. 5b is a schematic view of a method 300 illustrated in a context boundary search range. In particular, it illustrates the original waveform of audio signals, short-term energy, differential short-term energy after normalization, and the range of a specific short-term context.

Instead of using amplitude information to determine the context range, adaptive context range determination may also be based on spectral information. For example, to determine the context range, Bayesian Information Criteria (BIC) can be used to find the largest spectral inconsistency. Fig. 6a represents a method 400 for determining boundaries using a BIC. A short-term context is also taken as an example to describe the method 400. First, at step 402, it is assumed that timestamp t in the BIC window is the true boundary and that the window is better represented by two separate normal distribution models separated at time t. It is then assumed at step 404 that timestamp t in the BIC window is not a true boundary and that the window would be better represented by just one normal distribution model. Delta BIC is then calculated at step 406 using the following formula (4):

Here, H ₀ is the hypothesis at step 402 and H ₁ is the hypothesis at step 404. In FIG. Figure 6b shows an example of a ΔBIC(t) curve in the BIC window, which represents the log-likelihood difference between two hypotheses. The delta BIC may then be normalized at step 408. Then, at step 410, if the peak ΔBIC(t) is greater than a threshold value (which can be adjusted between 0 and 1), this peak can be selected as the best possible context boundary position.

In fig. 7 illustrates an example of adaptively determining the length or range of a combined temporal context window if a lookahead buffer is available, in accordance with some embodiments. Specifically, the length or range of the short-term context window is adaptively determined by the method 300 or the method 400, and the length or range of the medium-term context and long-term context is predetermined by the method 200. As shown in FIG. 7, if a lookahead buffer is available, the short-term context can propagate in both the lookahead and lookahead directions. Alternatively, the short-lived context may only propagate in one direction, for example if there is no lookahead buffer (not shown). Method 300 or method 400 of the present invention takes short-term context as an example to describe adaptive determination, but the length or range of medium-term context can also be adaptively determined by methods similar to methods 300 or 400 described above.

As noted above, a modern dialogue detector is used to downmix to mono the L/R channels for a stereo signal or the L/R/C channels for a 5.1 signal to reduce computational complexity. However, mixing all channels together may reduce the dialogue signal-to-noise ratio and reduce dialogue detection accuracy. For example, dialogue with a lot of noise (such as in sports games) or dialogue in intense action scenes may be missed when detected. To solve this problem, a predominant center channel downmix is applied, as represented by formula (5), to reduce dialogue smearing, since most of the dialogue is in the C channel in a 5.1 signal.

Here C, L, R mean the complex spectrum for each time-spectral “tile” (that is, for each frame and each interval/band) in the central, left and right channels respectively, and g is a parameter between 0 and 1 to reduce the “contribution » from L and R channels. However, the above method works on a 5.1 signal, but is not applicable to a stereo signal, since dialogue is usually treated as a panned signal, and therefore correlated in L and R, in a stereo signal.

To solve this problem, the present invention proposes a new downmixing method for eliminating uncorrelated noise in a signal to make dialogue clearer after downmixing. Fig. 8 represents an example of a method 500 for downmixing a dialog detector audio input signal that is performed in accordance with some embodiments. First, at step 502, the input audio signal is divided into a plurality of frames. Next, at step 504, the frames in the left channel and the right channel are converted to a spectral representation of the frames. Then, at step 506, the uncorrelated signals are eliminated using formula (6) as follows:

Here L is the spectral representation of frames in the left channel, R is the spectral representation of frames in the right channel, and g ₁ and g ₂ represent two frequency-dependent gains rather than the wideband gains applied to L and R, respectively. For simplicity, the frequency band characteristic is omitted from the formula. In one embodiment, g ₁ and g ₂ can be calculated from the covariance matrix, which is calculated for each band in a certain period of time (and only the real part is taken into account, and the frequency response is also omitted), as presented in formula (7):

Then, after analyzing the eigenvectors and the idea of environment extraction in NGCS, g ₁ and g ₂ can be represented as follows.

Here a, c and d are alternative representations of the covariance coefficients respectively, to simplify the presentation of formulas (8) and (9). After step 506, the downmixed signal M is obtained at step 508.

Although the above method 500 is described and developed based on a stereo signal, it could be applied to a 5.1 signal. In one embodiment _, the 5.1 signal may first be converted to a stereo signal ( Lc and Rc ) with a dominant center channel downmix, as _represented in formulas (10) and (11):

L _c and R _c will then follow the method 500 to remove uncorrelated noise.

In addition to or instead of the uncorrelated signal elimination method 500, certain other methods may also be used. In some embodiments, a technique similar to echo cancellation may be used to reduce noise in the center channel C using (L+R)/2 as the reference noise signal. Alternatively, the NMR spectral basis can be constructed for either dialogue or both dialogue and noise, and can be used to extract the pure dialogue component.

In addition, in a modern detector, a musical signal, especially a singing voice in a cappella (without significant instrumental accompaniment) or recitative, which has many similar properties to dialogue, can be incorrectly classified as dialogue, so false positives can increase significantly. The applicant found that the music confidence score was also high for the same misclassified frames. Based on this, in parallel with the dialogue detector, the applicant introduces a music classifier so that the music credibility score can be used as a reference to refine or adjust the original dialogue credibility score to significantly reduce false positives in music.

Fig. 9 is a block diagram of a dialogue detector 600, further including a music content detector 606, in accordance with some embodiments. First, the input audio signal is divided into a plurality of frames and converted to a spectral representation using discrete Fourier transform (DFT) 602. Feature extraction component 604 then extracts features to represent each frame according to the procedure illustrated in FIG. 2. Then, the music content detector 606 takes the extracted features to obtain a music credibility score C _m (t); meanwhile, the speech content detector 608 also receives the extracted features to obtain a speech confidence score C _s (t). In addition, the music fidelity score C _m (t) and the speech fidelity score C _s (t) may be further smoothed by a median filter or an average filter. Additionally, in post-processing component 610, the music fidelity score C _m (t) and the speech fidelity score C _s (t) are combined to produce a final score reliability of the dialogue. In particular, the initial indicator of dialogue reliability is refined to obtain an updated final indicator dialogue authenticity in post-processing component 610. Typically, the original dialogue fidelity score can be reduced to some extent if the music fidelity score C _m (t) for the same frame is also high. However, this may result in an excessive reduction in the speech fidelity score C _s (t) since real dialogue content can also generate both a high dialogue fidelity score and a high music fidelity score if the dialogue is present with background music. To address this issue, historical context can be used to determine whether the music fidelity score C _s (t) can be confidently used to refine the dialogue fidelity score. If the historical context is predominant in the dialogue, it would be more conservative to specify the dialogue fidelity score, that is, with the intention of ignoring the music fidelity score. Therefore, in some embodiments, the final indicator the reliability of the dialogue is clarified using the following formula (12):

Here represents the refined dialogue confidence score at frame t, C _s (t) is the speech confidence score, C _m (t) is the music confidence score, and β is a context-sensitive parameter that determines how much the music confidence score influences the final dialogue confidence score. In one embodiment, β is calculated from the proportion of frames identified in the historical context as speech or music. For example, β may be defined as the ratio of frames identified in historical context as music by a simple binary method. In particular, β may be set equal to one if the context is dominated by music, and β may be set equal to zero if the context is dominated by dialogue, as represented in formula (13):

Here N _m represents the number of musical frames, N is the total number of frames in the historical context; r _th is a threshold value typically set to 0.5, although the threshold value can also be adjusted between 0 and 1 depending on how aggressive the music frames are. Alternatively, β can be represented as a continuous function, such as a linear function, as illustrated in formula (14), or as a sigmoid function, as illustrated in formula (15):

Here, a represents a scaling factor governing the shape of the sigmoidal function, and in the present invention may be set to 5. Moreover, the historical context used in determining the context-sensitive parameter β may be much longer than the history frames used to extract long-term features , for example, the length or range of the historical context may be set to 10 seconds or even longer.

The conversation detector techniques described herein could be implemented by one or more computing devices. For example, a special purpose computing device controller may be hardware implemented to perform or enable the described operations and may include digital electronic circuits, such as one or more application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs), permanently programmed to performing operations or ensuring the execution of operations. In some embodiments, custom hardware-based logic, ASICs, and/or FPGA devices with custom programming are combined to perform these methods.

In some other embodiments, a general purpose computing device could comprise a controller comprising a central processing unit (CPU) programmed to cause one or more of the described operations to be performed in accordance with software instructions in firmware, memory, other storage device, or a combination thereof. .

The term “computer readable medium” as used herein refers to any medium storing instructions and/or data causing a computer or other type of machine to operate in a particular manner. Any of the models, detector and operations described herein may be implemented or implemented by software code executed by the controller processor using an appropriate computer language. The software code may be stored as a sequence of instructions on a computer-readable storage medium. Examples of suitable computer-readable storage media include random access memory (RAM), read only memory (ROM), magnetic media, optical media, solid state drive, flash memory, and any other memory chip or cartridge. The computer readable storage medium may be any combination of these storage devices. Any such computer-readable storage media may be located on or in a single computing device or an entire computer system, and may be located among other computer-readable storage media on a system or network.

While the subject matter of the present invention is specifically shown and described with reference to embodiments thereof, those skilled in the art will appreciate that changes in form and detail may be made to the disclosed embodiments without departing from the spirit or scope of the present invention. Examples of some of these embodiments are illustrated in the accompanying graphics, and specific details are set forth to provide a thorough understanding thereof. It should be noted that embodiments may be practiced without some or all of these specific details. In addition, for the sake of clarity of presentation, well-known features may not have been described in detail. Finally, although advantages have been discussed herein with reference to certain embodiments, it will be understood that the scope of the present invention should not be limited by reference to these advantages. Rather, the scope should be determined by reference to the appended claims.

Various aspects of the present invention can be understood from the following numbered exemplary embodiments (NEX):

1. A method for extracting an audio feature in a dialogue detector in response to an input audio signal, comprising:

extracting, by the frame feature extraction component, frame features from each frame of the plurality of frames extracted from the input audio signal;

determining, by the combined temporal context determination component, the length or range of each context;

deriving the corresponding contextual feature based on the personnel features in each specific context; And

performing concatenation on each contextual feature and generating a combined feature vector to represent the current frame.

2. Method according to PPVO 1, characterized in that the combined time context contains:

short-term context, representing local information around the current frame;

a mid-temporal context further comprising a plurality of flashback frames; And

long-term context, additionally containing many long-term history frames.

3. A method according to PPVO 1 or 2, in which the length or range of one or more contexts can be predetermined.

4. The method according to OPVO 2 or 3, in which the short-term context contains the current frame and lookahead frames if a lookahead buffer is available; or the short-term context contains the current frame and the first part of the flashback frames if the lookahead buffer is not available.

5. The method according to VPVO 2 or 3, in which the medium-temporal context contains the current frame, lookahead frames, and lookahead frames if a lookahead buffer is available; or the mid-temporal context contains the current frame, the first part of the flashback frames, and the second part of the flashback frames if no lookahead buffer is available.

6. The method according to OPVO 2 or 3, in which the long-term context contains the current frame, lookahead frames, lookahead frames, and long-term history frames if a lookahead buffer is available; or the long-term context contains the current frame, the first part of the flashback frames, the second part of the flashback frames, and long-term history frames if the lookahead buffer is not available.

7. A method according to DPVO 1 or 2, in which the length or range of one or more contexts can be determined adaptively.

8. The method according to PPVO 7, in which the length or range of one or more contexts can be determined adaptively by analyzing the stationarity of frame-level features.

9. The method according to PPVO 8, wherein adaptively determining the length or range of one or more contexts is based on information related to the amplitude of the input audio signal.

10. A method according to PPVO 2 or 9, in which the adaptive determination of the length or range of the short-term context includes:

calculating the short-term energy of one frame from a plurality of frames of the input audio signal;

smoothing the calculated short-term energy;

determining the difference between the smoothed short-term energy of the frame and the smoothed short-term energy of the previous frame;

determining the start based on whether the difference meets a threshold;

normalization of the beginning;

defining a boundary for the length or range of a short-term context by one of the following steps:

taking the position having the greatest origin as the boundary; or

taking the peak onset as a boundary based on whether the peak satisfies a threshold; or

by taking the peak onset as a boundary based on both the threshold value and the distance between the onset and the strong transition immediately before the onset.

11. The method according to AFPR 8, wherein adaptively determining the length or range of one or more contexts is based on information related to the spectrum of the input audio signal.

12. The method according to PPVO 2 or 11, in which the adaptive determination of the length or range of the short-term context includes:

the assumption that the timestamp in the Bayesian Information Criteria window is a true boundary for the length or range of the short-term context;

the assumption that this timestamp in the Bayesian Information Criteria window is not a true boundary for the length or range of the short-term context;

determining the difference in Bayesian information criteria between two hypotheses;

normalization of differences in Bayesian information criteria; And

considering the Bayesian Information Criteria difference peak as the position of the true cutoff for the length or range of short-term context based on whether the peak satisfies a threshold.

13. The method as in any one of the previous OPVOs 2, 7-12, wherein the short-term context may propagate in both the look-ahead and look-ahead directions, or propagate in only one direction.

14. The method as in any one of the preceding DPWs, wherein the length or range of contexts may be predetermined in combination with adaptive determination.

15. A method for downmixing an input audio signal of a dialogue detector, comprising:

dividing the input audio signal into multiple frames;

converting frames in the left channel and right channel into a spectral representation of the frames;

eliminating uncorrelated signals in the left channel and right channel by applying frequency-dependent gains to the spectrum in the left channel and right channel, respectively;

receiving the downmix signal and

supplying the downmix signal to the dialogue detector in the manner of any of the previous DPVOs 1–14.

16. The method according to PPVO 15, in which frequency-dependent gains can be calculated from the covariance matrix.

17. A method for classifying the input audio signal of a dialogue detector, including:

receiving by the speech content detector the features extracted by the method according to any of the previous PPVOs 1–14;

determination by the speech content detector of the speech reliability indicator; And

receiving by the music content detector the features extracted by the method according to any of the previous PPVOs 1–14;

determination by the music content detector of the music reliability indicator; And

combining the speech fidelity score and the music fidelity score to produce a final dialogue fidelity score.

18. Method according to PPVO 17, in which the final indicator of the reliability of the dialogue can be clarified by means of a context-sensitive parameter.

19. The method of PPVO 18, wherein the context-sensitive parameter may be calculated based on the proportion of frames identified as speech or music in the historical context.

20. Method according to AAPR 19, in which the historical context may have a duration of ten seconds or more.

Claims (29)

1. A method for obtaining a confidence indicator representing the probability of the presence of dialogue in the current frame of an input audio signal, the method includes: dividing the input audio signal into multiple frames; extracting frame audio features from each frame I; defining a set of context windows, where each context window contains a number of frames surrounding the current frame; deriving, for each context window, a corresponding contextual audio feature for the current frame based on the frame audio features of the frames in each corresponding context; performing concatenation on each contextual audio feature to form a combined feature vector to represent the current frame; And obtaining a speech reliability indicator representing the probability of the presence of dialogue in the current frame using a combined feature vector, wherein the number of frames in one or more context windows is determined adaptively based on the extracted frame audio features. 2. The method according to claim 1, characterized in that the set of context windows contains: a short-term context window containing the current frame and a number of frames preceding and/or following the current frame; a medium-term context window containing short-term context window frames and a plurality of lookback frames; And a long-term context window containing frames of the medium-term context window and many long-term history frames. 3. The method according to claim 1 or claim 2, characterized in that the number of frames in one or more context windows is determined adaptively by analyzing the stationarity of frame-level audio features. 4. The method according to claim 3, characterized in that the adaptive determination of the number of frames in one or more context windows is based on information related to the amplitude of the input audio signal. 5. The method according to claim 4, characterized in that the adaptive determination of the number of frames in one or more context windows is based on information related to the spectrum of the input audio signal. 6. The method according to any of the previous paragraphs, characterized in that it additionally includes: converting frames in the left channel and right channel into a spectral representation of the frames; eliminating uncorrelated signals in the left channel and right channel by applying frequency-dependent gains to the spectral representation in the left channel and right channel, respectively; receiving a downmix signal from the left and right channels; And using the specified signal after downmixing as the specified input audio signal. 7. The method according to claim 6, characterized in that the frequency-dependent gains are calculated from the covariance matrix. 8. A method for classifying the input audio signal of a dialogue detector, including: obtaining an indicator of speech reliability using the method according to any of the previous paragraphs 1-7; determining the music reliability indicator based on the combined feature vector; And combining the speech fidelity score and the music fidelity score to produce a final dialogue fidelity score. 9. The method according to claim 8, characterized in that the final indicator of the reliability of the dialogue is clarified by means of a context-dependent parameter. 10. The method of claim 9, wherein the context-sensitive parameter is calculated based on the proportion of frames identified as speech or music in the contextual history window, for example, lasting at least ten seconds. 11. A dialogue detector containing means for performing the method according to any of paragraphs 1-7. 12. A computer-readable storage medium on which instructions are stored that, when executed, cause a computer to execute the method of any of paragraphs 1-7.

Images