RU2782364C1 - Apparatus and method for isolating sources using sound quality assessment and control - Google Patents

Abstract

FIELD: computing technology.

SUBSTANCE: invention relates to the field of computing technology for processing audio data. The technical result is achieved by determining the estimated target signal depending on the input audio signal; determining the resulting values, depending on the estimated sound quality of the estimated target signal, in order to obtain one or multiple parameter values; and forming an isolated audio signal, depending on the one or multiple parameter values and depending on one of the estimated target signal, the input audio signal, and the estimated difference signal; wherein the estimated difference signal is an estimate of a signal containing only a section of the difference audio signal, wherein the isolated audio signal is formed depending on the parameter values and depending on the linear combination of the estimated target signal and the input audio signal; or wherein the isolated audio signal is formed depending on the parameter values and depending on the linear combination of the estimated target signal and the estimated difference signal.

EFFECT: maximum reduction of noise level on the condition of absence of artifacts.

16 cl, 6 dwg

Description

The present invention relates to separation (separation) of audio sources, in particular, to adaptive signal quality control of the sound quality of separated output signals, and, in particular, to a device and method for separating sources using sound quality estimation and control.

As the sources are separated, the quality of the output signals deteriorates, and this degradation increases monotonically with the attenuation of the interfering signals.

The separation of audio sources has been done in the past.

при заданном совокупном сигнале

,The separation (selection) of audio signal sources is aimed at obtaining the target signal

for a given cumulative signal

,

(1),

(one),

содержит все сигналы помех и в дальнейшем называется "сигналом помех". Результатом отделения

является оценка целевого сигнала

,where

contains all interference signals and is referred to as "interference signal" in the following. The result of separation

is the estimate of the target signal

,

(2)

(2)

,and, possibly, an additional evaluation of the interference signal

,

(3)

(3)

Such processing usually introduces artifacts into the output signal that degrade the sound quality. This deterioration in sound quality increases monotonically with the amount of separation, attenuation of interference signals. In many applications, full separation is not required, but partial amplification, interference sounds are attenuated but still present in the output signal.

This has the added benefit that the audio quality is better than with completely separated signals because less artifact is introduced and interference signal leakage partially masks perceived artifacts.

Partial masking of an audio signal means that its loudness (eg perceived intensity) is partially reduced. In addition, it may be desirable and necessary that, instead of achieving a large attenuation, the output audio quality does not fall below a predetermined audio quality level.

An example of such an application is the enhancement of dialogue. Audio signals in television and radio broadcasts and sound in films are often a mixture of speech signals and background signals such as environmental sounds and music. When these signals are mixed in such a way that the speech level is too low compared to the background level, the listener may have difficulty understanding what has been said, or comprehension is very difficult to listen to, resulting in listener fatigue. In such scenarios, ways to automatically reduce the background level can be applied, but the result should be of high sound quality.

и

. Модель смешивания описывает характеристики того, как входные сигналы объединяются для получения смешенного сигнала

, здесь посредством сложения.In the prior art, there are various ways to separate sources. The separation of the target signal from the signal mix has been discussed in the prior art. These methods can be categorized into two approaches. The first category of methods is based on formulated assumptions about the signal model and/or mixing model. The signal model describes the characteristics of the input signals, here

and

. The mixing model describes the characteristics of how input signals are combined to produce a mixed signal.

, here by addition.

Based on these assumptions, the method is developed analytically or heuristically. For example, an Independent Component Analysis method can be obtained by assuming that the mix contains two original signals that are statistically independent, the mix was captured by two microphones, and the mix was obtained by adding both signals (producing instant mix). The inverse mixing process is then mathematically derived as the inverse of the mixing matrix, and the elements of this mixing separation matrix are calculated in accordance with the specified method. Most of the analytical methods are obtained by formulating the separation problem as a numerical optimization of a criterion, for example, the root mean square error between the true target and the estimated target.

The second category is data driven. In this case, the representation of the target signals is evaluated, or a set of parameters is evaluated to extract the target signals from the input mix. The score is based on a model that has been trained on the training dataset, hence the name "data driven". The score is obtained by optimizing the criterion, for example, by minimizing the mean square error between the true target and the estimated target given the training data. An example for this category are Artificial Neural Networks (ANNs) that have been trained to produce an estimate of a speech signal in the presence of a mixture of speech and noise signals. During training, the adjustable parameters of the artificial neural network are determined in such a way that the performance criterion calculated for the training dataset is optimized - on average over the entire dataset.

As far as source separation is concerned, the solution that is optimal in terms of rms error, or optimal in terms of any other numerical criterion, is not necessarily the highest audio quality solution that human listeners prefer.

The second problem is that the separation of sources always leads to two effects: firstly, to the desired attenuation of interference sounds and, secondly, to an undesirable deterioration in sound quality. Both effects are correlated, for example, an increase in the desired effect leads to an increase in the undesirable effect. The ultimate goal is to manage the compromise between them.

Sound quality can be quantified, for example, with a listening test or with computational sound quality models. Sound quality has many aspects, hereinafter referred to as Sound Quality Components (SQC).

For example, audio quality is determined by the perceived intensity of artifacts (these are signal components that have been introduced by signal processing, such as source separation, that degrade audio quality).

Or, for example, the sound quality is determined by the perceived intensity of the interfering signals, or, for example, speech intelligibility (when the target signal is speech), or, for example, the overall sound quality.

,

, где

обозначает количество компонентов качества звука.There are various sound quality computational models that calculate (estimate) sound quality components,

,

, where

indicates the number of sound quality components.

Such methods typically estimate the audio quality component given a target signal and an estimate of the target signal,

(4)

(four)

or considering also the interference signal,

(5).

(5).

(и сигналы помех

) не доступны, иначе не требовалось бы отделение. Когда доступны только входной сигнал

и оценки целевого сигнала

, компоненты качества звука не могут быть вычислены с помощью этих способов.In practical application, target signals

(and interference signals

) are not available, otherwise separation would not be required. When only the input signal is available

and target signal evaluation

, the sound quality components cannot be calculated using these methods.

Various computational models have been described in the prior art for evaluating aspects of sound quality, including intelligibility.

Blind Source Separation Evaluation (BSSEval) (see [1]) is a set of tools for multi-criteria performance evaluation. The estimated signal is decomposed by means of an orthogonal projection onto the target signal component, interference from other sources, and artifacts. The metrics are calculated as the energy ratios of these components and are expressed in dB. These metrics are Source to Distortion Ratio (SDR), Source to Interference Ratio (SIR), and Source to Artifact Ratio (SAR).

Perceptual Evaluation methods for Audio Source Separation (PEASS) (see [2]) were developed as a perceptually motivated successor to the BSSEval method. The signal is projected onto time segments using a gammaton filter bank.

PEMO-Q (see [3]) is used to provide multiple features. Four perception scores are obtained from these features using a neural network trained with subjective scores. The Perceptual Scores are: Overall Perceptual Score (OPS), Interference-related Perceptual Score (IPS), Artifact-related Perceptual Score (APS) and Perceptual Score, associated with the target signal (Target-related Perceptual Score, TPS).

Perceptual Evaluation of Audio Quality (PEAQ) (see [4]) is a metric developed for audio coding. It uses a peripheral ear model to calculate the basilar membrane representations of the reference and test signals. Aspects of difference between these representations are quantified by several output variables. Through a neural network trained with subjective data, these variables are combined to get the main result, for example, the overall difference score (Overall Difference Grade, ODG).

Perceptual Evaluation of Speech Quality (PESQ) (see [5]) is a metric developed for speech transmitted over telecommunication networks. Therefore, the method includes pre-processing that simulates a handset. The metrics for audio interference are calculated from a given signal loudness and combined in the PESQ scores. Based on them, the MOS estimate is predicted using a polynomial mapping function (see [6]).

ViSQOLAudio (see [7]) is a metric designed for music encoded at low bit rates, developed on top of the Virtual Speech Quality Objective Listener (ViSQOL). Both metrics are based on a model of the peripheral auditory system to create internal representations of signals called neurograms. They are compared through an adaptation of the Structural Similarity Index originally developed to evaluate the quality of compressed images.

The Hearing-Aid Audio Quality Index (HAAQI) (see [8]) is an index designed to predict the quality of music for hearing aid wearers. The index is based on an auditory periphery model extended to account for the effects of hearing loss. This corresponds to a database of quality ratings made by hearing-impaired or normal-hearing listeners. The simulation of hearing loss can be bypassed and the index becomes valid also for people with normal hearing. Based on the same auditory model, the HAAQI authors also proposed a speech quality index - the Hearing-Aid Speech Quality Index (HASQI) (see [9]), and a speech intelligibility index - the hearing aid speech perception index (Hearing -Aid Speech Perception Index, HASPI) (see [10]).

Short-Time Objective Intelligibility (STOI) (see [11]) is a measure that is expected to have a monotonous relationship with average speech intelligibility. It applies especially to speech processed with some time-frequency weighting.

In [12], an artificial neural network is trained to estimate the source-to-distortion ratio given only the input signal and the estimated target output signal, where the calculation of the source-to-distortion ratio would normally take as input also the true target and the interference signal. Many separation algorithms are executed in parallel on the same input signal. The source-to-distortion ratio estimates are used to select, for each time interval, the algorithm output with the best source-to-distortion ratio. Therefore, no control over the trade-off between sound quality and separation is formulated, and no control over the parameters of the separation algorithm is proposed. In addition, a source-to-distortion ratio is used, which is not motivated by perception and has been shown to be poorly correlated with perceived quality, for example in [13].

In addition, recently there have been works on speech enhancement using supervised learning, in which estimates of the sound quality components are integrated into cost functions, while traditionally speech enhancement models are optimized based on the root mean square error (MSE) between the estimated and clean speech. For example, [14], [15], [16] use cost functions based on STOI rather than MSE. In [17], reinforcement learning based on PESQ or PEASS is used. However, there is no control over the trade-off between sound quality and separation.

[18] proposes an audio processing device in which the audibility score is used in conjunction with the artifact identification score to control the time-frequency gain applied by the processing. This is done, for example, in order to provide the maximum reduction in noise level, provided there are no artifacts, the trade-off between sound quality and separation is fixed. In addition, the system does not involve supervised learning. Artifact detection uses the kurtosis factor, a metric that directly compares output and input signals (perhaps in non-speech segments), without the need to determine the true target and interference signal. This simple indicator is supplemented by an indicator of audibility.

The object of the present invention is to provide improved concepts for source separation. The task of the present invention is solved by means of the device according to claim 1, the method according to claim 16 and the computer program according to claim 17 of the claims.

An apparatus is provided for generating a separated audio signal from an input audio signal. The input audio signal contains a portion of the target audio signal and a portion of the difference audio signal. The difference audio section indicates the difference between the input audio signal and the target audio section. The device contains a source separator, a definition module, and a signal processor. The source splitter is configured to define an estimated target signal that depends on the input audio signal, the estimated target signal is an estimated signal that contains only a section of the target audio signal. The determining module is configured to determine one or more result values depending on the estimated audio quality of the estimated target signal to obtain one or more parameter values, where one or more parameter values are one or more result values or depend on one or more result values. The signal processor is configured to generate a separated audio signal depending on one or more parameter values and depending on at least one of the estimated target signal and the input audio signal and the estimated difference signal, wherein the estimated difference signal is a signal estimate that contains only a section of the difference audio signal.

In addition, a method for generating a separated audio signal from an input audio signal is provided. The input audio signal contains a portion of the target audio signal and a portion of the difference audio signal. The difference audio section indicates the difference between the input audio signal and the target audio section. The method contains:

- determining the estimated target signal, which depends on the input audio signal, the estimated target signal is an estimate of the signal, which contains only a portion of the target audio signal;

- determining one or more result values depending on the estimated audio quality of the estimated target signal to obtain one or more parameter values, where one or more parameter values represent one or more result values or depend on one or more result values; and

- generating a separated audio signal depending on one or more parameter values and depending on at least one of the estimated target signal and the input audio signal and the estimated difference signal, the estimated difference signal is an estimate of a signal that contains only a portion of the difference audio signal.

In addition, a computer program is provided for implementing the method described above when it is executed on a computer processor or a signal processor.

Hereinafter, embodiments of the present invention are described in more detail with reference to the following figures.

Fig. 1a illustrates an apparatus for generating a separated audio signal from an input audio signal according to an embodiment,

Fig. 1b illustrates an apparatus for generating a separated audio signal according to another embodiment, further comprising an artificial neural network,

Fig. 2 illustrates an apparatus according to an embodiment which is configured to use sound quality estimation and which is configured to perform post-processing,

Fig. 3 illustrates an apparatus according to another embodiment in which post-processing parameters are directly evaluated,

Fig. 4 illustrates an apparatus according to a further embodiment in which sound quality evaluation and secondary separation are performed, and

Fig. 5 illustrates an apparatus according to another embodiment in which separation parameters are directly estimated.

Fig. 1a illustrates an apparatus for generating a separated audio signal from an input audio signal according to an embodiment. The input audio signal contains a portion of the target audio signal and a portion of the difference audio signal. The difference audio section indicates the difference between the input audio signal and the target audio section.

The apparatus includes a source splitter 110, a determiner 120, and a signal processor 130.

The source splitter 110 is configured to determine an estimated target signal that depends on the input audio signal, the estimated target signal is a signal estimate that contains only a section of the target audio signal.

Determination module 120 is configured to determine one or more result values depending on the estimated audio quality of the estimated target signal to obtain one or more parameter values, wherein the one or more parameter values are one or more result values or depend on one or more result values. .

The signal processor 130 is configured to generate a separated audio signal depending on one or more parameter values and depending on at least one of the estimated target signal and the input audio signal and the estimated difference signal. The estimated difference signal is an estimate of a signal that contains only a portion of the difference audio signal.

Optionally, in an embodiment, determiner 120 may, for example, be configured to determine one or more result values depending on the estimated target signal and depending on at least one of the input audio signal and the estimated difference signal.

Embodiments provide perceptually motivated and signal-adaptive control of the trade-off between sound quality and separation using supervised learning. This can be achieved in two ways. The first method estimates the audio quality of the output signal and uses this estimate to adapt the separation parameters or post-processing of the separated signals. In the second embodiment, the regression method directly outputs the control parameters, resulting in the sound quality of the output signal meeting the predetermined requirements.

качества звука и определения параметров обработки на основе

, чтобы качество звука на выходе (при использовании определенных параметров обработки) было не нижнее заданного значения качества.In accordance with embodiments, analysis of the input signal and the output signal of the department is carried out to obtain an estimate

sound quality and determine processing parameters based on

so that the output audio quality (when using certain processing options) is not below the specified quality value.

в (9). Из показателя качества вычисляется управляющий параметр

в формуле (13) ниже (например, масштабный коэффициент), и окончательные выходные данные получаются посредством микширования начальных выходных данных и входных данных, как в формуле (13) ниже. Вычисление

может выполняться итерационно или посредством регрессии, причем параметры регрессии получаются в результате обучения из набора обучающих сигналов, см. фиг. 2. В вариантах осуществления управляющий параметр может представлять собой не масштабный коэффициент, а, например, параметр сглаживания и т.п.In some embodiments, the analysis produces a quality score

at 9). The control parameter is calculated from the quality index

in formula (13) below (eg, scaling factor), and the final output is obtained by mixing the initial output and input data as in formula (13) below. calculation

can be performed iteratively or by regression, with the regression parameters obtained as a result of training from a set of training signals, see FIG. 2. In embodiments, the control parameter may not be a scaling factor but, for example, a smoothing parameter or the like.

в (13) непосредственно, см. фиг. 3.In some embodiments, the analysis results in a control parameter

in (13) directly, see fig. 3.

Fig. 4 and FIG. 5 define additional embodiments.

Some embodiments achieve audio quality control in the post-processing step, as described below.

A subset of the embodiments described herein may be applied regardless of the separation method. Some of the embodiments described herein control the separation process.

преобразуется посредством оконного преобразования Фурье (STFT) или обрабатывается с помощью набора фильтров, что дает в результате комплекснозначные коэффициенты преобразования STFT или сигналы

частотных подполос, где

обозначает индекс временного кадра,

обозначает индекс частотного интервала или индекс частотной подполосы. Комплекснозначные коэффициенты преобразования STFT или сигналы частотных подполос требуемого сигнала представляют собой

, и сигнал помех представляет собой

.Source separation using spectral weighting processes signals in the time-frequency domain or short-term spectral domain. Input signal

transformed by a windowed Fourier transform (STFT) or processed by a filter bank, resulting in complex-valued STFT transform coefficients or signals

frequency subbands, where

denotes the time frame index,

denotes a frequency slot index or a frequency subband index. The complex valued STFT transform coefficients or frequency subband signals of the desired signal are

, and the interference signal is

.

The separated (dedicated) output signals are computed by spectral weighting as

(6),

(6)

поэлементно умножаются на входной сигнал. Цель состоит в том, чтобы ослабить элементы в

, где источник помех

является большим. С этой целью спектральные весовые коэффициенты могут быть вычислены на основе оценки цели

, оценки источника помех

или оценки отношения сигнала к источнику помех, например,where the spectral weights

element-wise multiplied by the input signal. The goal is to weaken the elements in

, where the interference source

is big. To this end, the spectral weights can be computed based on the target score

, interference source estimates

or an estimate of the signal-to-interference ratio, for example,

(7)

(7)

or

(8),

(eight),

и

- параметры, управляющие отделением. Например, увеличение

может привести к большему ослаблению источника помех, но также и к более сильному ухудшению качества звука. Спектральные весовые коэффициенты могут быть дополнительно модифицированы, например, посредством задания порога, чтобы

было больше порога. Модифицированные коэффициенты усиления

вычисляются какwhere

and

- parameters that control the department. For example, an increase

may lead to more attenuation of the interferer, but also to a greater deterioration in sound quality. The spectral weights can be further modified, for example by setting a threshold, so that

was over the threshold. Modified gains

calculated as

.

.

Increasing the threshold v reduces the attenuation of the interfering source and reduces the potential degradation in audio quality.

, источника помех

или отношения сигнала к источнику помех) является основой этих способов, и в прошлом были разработаны различные способы оценки. Они следуют одному из двух описанных выше подходов.Estimation of the required values (objectives

, interference source

or signal-to-interferer ratio) is the basis of these methods, and various evaluation methods have been developed in the past. They follow one of the two approaches described above.

вычисляется с использованием обратной обработки преобразования STFT или набора фильтров.Then the output signal

is computed using the inverse processing of the STFT transform or filter bank.

Next, source separation using target signal estimation according to the embodiments will be described.

The representation of the target signal can also be estimated directly from the input signal, for example, using an artificial neural network. Recently, various methods have been proposed in which an artificial neural network was trained to estimate a target timing signal, or its STFT coefficients, or STFT coefficient values.

для оценки результатов этой вычислительной модели,Regarding audio quality, the audio quality component (SQC) is obtained by applying a supervised learning model

to evaluate the results of this computational model,

(9).

(9).

реализован следующим образом.Method of supervised learning

implemented as follows.

с помощью обучаемых параметров,

входных переменных и

выходных переменных.1. Supervised learning model configuration

using learnable parameters,

input variables and

output variables.

и смешения

.2. Formation of a data set using sample signals for the target

and mixing

.

.3. Computing an estimate for target signals by separating sources,

.

из полученных сигналов посредством вычислительных моделей качества звука в соответствии с (9) или (10).4. Calculation of sound quality components

from the received signals by means of computational sound quality models in accordance with (9) or (10).

таким образом, чтобы она выдавала оценки

с учетом соответствующих сигналов-примеров для предполагаемой цели

(результата отделения источников) и смешения

. В качестве альтернативы, обучение модели контролируемого обучения

таким образом, чтобы она выдавала оценки

с учетом

и

(если)

.5. Supervised learning model training

in such a way that it gives ratings

with appropriate example signals for the intended purpose

(the result of separation of sources) and mixing

. Alternatively, supervised learning model training

in such a way that it gives ratings

taking into account

and

(if)

.

(результат отделения источников), полученную из смешения

с использованием способа отделения источников вместе со смешением

.6. In application, the trained model gets the estimated target

(the result of separation of sources), obtained from mixing

using the source separation method along with mixing

.

The use of supervised learning methods to control the quality of the separated output signal is provided.

The following describes sound quality evaluation using supervised learning in accordance with the embodiments.

Fig. 1b illustrates an embodiment in which determination module 120 comprises an artificial neural network 125. Artificial neural network 125, for example, may be configured to determine one or more result values depending on the estimated target signal. The artificial neural network 125, for example, may be configured to receive a plurality of input values, each of the plurality of input values depending on at least one of the estimated target signal and the estimated difference signal, and on the input audio signal. Artificial neural network 125, for example, may be configured to determine one or more result values as one or more output values of artificial neural network 125.

Optionally, in an embodiment, the artificial neural network 125, for example, may be configured to determine one or more result values depending on the estimated target signal and at least one signal from the input audio signal and the estimated difference signal.

In an embodiment, each of the plurality of input values, for example, may depend on at least one of the estimated target signal and the estimated difference signal, and on the input audio signal. One or more result values, for example, may indicate the estimated audio quality of the estimated target signal.

According to an embodiment, each of the plurality of input values may, for example, depend on at least one of the estimated target signal and the estimated difference signal, and on the input audio signal. One or more result values, for example, may be one or more parameter values.

In an embodiment, artificial neural network 125, for example, may be configured to train by receiving a plurality of training datasets, each of the plurality of training datasets comprising a plurality of artificial neural network 125 input training values and one or more artificial neural network 125 output training values, each of the plurality of output training values, for example, may depend on at least one of the training target signal and the training difference signal and on the training input signal, wherein each of the one or more output training values, for example, may depend on the sound quality score of the training target signal.

In embodiments, an estimate for the audio quality component is obtained by supervised learning using a supervised learning model (SLM), such as an Artificial Neural Network (ANN) 125. The artificial neural network 125, for example, may be a fully connected artificial neural network. 125 which contains an input layer with A blocks, at least one hidden layer with input levels each with at least two blocks, and an output layer with one or more blocks.

блоками, где

равно количеству этапов квантования.The supervised learning model can be implemented as a regression model or a classification model. The regression model evaluates one target value at the output of one block in the output layer. Alternatively, the regression problem can be formulated as a classification problem by quantizing the output value into at least 3 steps using an output layer with

blocks, where

is equal to the number of quantization steps.

One output block is used for each quantization step.

, оцененной цели

и компонента качества звука

, где компонент качества звука был вычислен из оцененной цели

и истинной цели

, например. Один элемент набора данных обозначен как

. Выходной результат модели контролируемого обучения здесь обозначен как

.The supervised learning model is first trained on a dataset that contains multiple mixed signal examples

, estimated goal

and sound quality component

, where the audio quality component was computed from the estimated target

and true purpose

, for example. One element of the dataset is denoted as

. The output of the supervised learning model is denoted here as

.

соответствует количеству входных значений. Вводы в модели вычисляются из входных сигналов. Каждый сигнал может быть факультативно обработан посредством набора фильтров частотно-временного преобразования, например, краткосрочного преобразования Фурье (STFT). Например, ввод может быть построен посредством конкатенации коэффициентов STFT, вычисленных из

смежных кадров из

и

, где

или

. Если

- общее количество спектральных коэффициентов на кадр, то общее количество входных коэффициентов равно

.Number of blocks in the input layer

corresponds to the number of input values. The inputs in the model are computed from the input signals. Each signal can optionally be processed through a bank of time-frequency transform filters, such as the Short Term Fourier Transform (STFT). For example, the input can be constructed by concatenating the STFT coefficients computed from

adjacent frames from

and

, where

or

. If a

is the total number of spectral coefficients per frame, then the total number of input coefficients is

.

Each block of artificial neural network 125 calculates its output value as a linear combination of input values, which are then optionally processed using a non-linear compression function,

(10),

(ten),

обозначает выход одного нейрона,

обозначают

входных значений,

обозначают

весовых коэффициентов для линейной комбинации, и

обозначают

дополнительных составляющих смещения. Для блоков в первом скрытом слое количество входных значений

равно количеству входных коэффициентов D. Все

и

являются параметрами искусственной нейронной сети 125, которые определяются в способе обучения.where

denotes the output of one neuron,

designate

input values,

designate

weighting factors for the linear combination, and

designate

additional bias components. For blocks in the first hidden layer, the number of input values

equals the number of input coefficients D. All

and

are the parameters of the artificial neural network 125, which are determined in the training method.

The blocks of one layer are connected to the blocks of the next layer, the outputs of the blocks of the previous layer are the inputs to the blocks of the next layer.

. Ошибка предсказания для всего набора данных или подмножества набора данных, используемого в качестве критерия оптимизации, является, например, среднеквадратичной ошибкой MSE или средней абсолютной ошибкой MAE, где

обозначает количество элементов в наборе данных.Training is performed by minimizing the prediction error using a numerical optimization method such as gradient descent. The prediction error for one element is a function of the difference

. The prediction error for the entire data set or a subset of the data set used as an optimization criterion is, for example, the MSE root mean square error or the MAE mean absolute error, where

denotes the number of elements in the dataset.

(11)

(eleven)

(12)

(12)

и дифференцируемыми. Кроме того, существуют другие структуры и элементы для построения искусственных нейронных сетей, например, слои сверточной нейронной сети или слои рекуррентной нейронной сети.Other error rates are possible for learning purposes if they are monotone functions

and differentiable. In addition, there are other structures and elements for building artificial neural networks, such as convolutional neural network layers or recurrent neural network layers.

и

), которые определяются в процедуре обучения посредством оптимизации скалярного критерия.They all have in common that they implement a mapping from a multidimensional input to a one or multidimensional output, with the mapping function controlled by a set of parameters (for example,

and

), which are determined in the learning procedure by optimizing the scalar criterion.

с учетом смешения без необходимости в истинной цели

.After training, a supervised learning model can be used to evaluate the sound quality of an unknown evaluated target.

subject to confusion without the need for a true target

.

With regard to sound quality computational models, experiments in accordance with embodiments have successfully used various computational models to evaluate aspects of sound quality (including intelligibility), such as the computational models described in [1]-[11], in particular the evaluation of blind source separation. (BSSEval) (see [1]), Perceptual Assessment Methods for Audio Source Separation (PEASS) (see [2]), PEMO-Q (see [3]), Perceptual Audio Quality Score (PEAQ) (see [4]), Perceptual Speech Quality Score (PESQ) (see [5] and [6]), ViSQOLAudio (see [7], Hearing Aid Audio Quality Index (HAAQI) (see [8]), Quality Index Hearing Aid Speech Perception (HASQI) (see [9], Hearing Aid Speech Perception Index (HASPI) (see [10]), and Short Term Objective Intelligibility (STOI) (see [11]).

Thus, according to an embodiment, the estimation of the sound quality of the training target signal, for example, may depend on one or more computational sound quality models.

For example, in an embodiment, the audio quality estimate of the training target signal may depend on one or more of the following audio quality computational models:

Evaluation of the blind separation of sources,

Perceptual evaluation methods for separating audio sources,

Audio quality perception evaluation,

Speech quality perception assessment,

Audio with virtual speech quality objective listener,

Hearing aid audio quality index,

Hearing aid speech quality index,

Hearing aid speech perception index, and

Short-term objective intelligibility.

Other sound quality computational models, for example, may also be used in other embodiments.

The following describes sound quality control.

(или не опускался ниже этого целевого значения).Audio quality control can be realized by estimating the audio quality component and calculating processing parameters based on the audio quality component estimate, or by directly estimating the optimal processing parameters so that the audio quality component matches the target value.

(or did not fall below this target value).

The evaluation of the audio quality component has been described above. Similarly, the optimal processing parameters can be estimated by training the regression method with the desired values of the optimal processing parameters. Optimal processing parameters are calculated as described below. This processing is hereinafter referred to as a Parameter Estimation Module (PEM).

будет определять компромисс между отделением и качеством звука. Этот параметр может управляться пользователем или указываться в зависимости от сценария воспроизведения звука. Воспроизведение звука в домашних условиях в спокойной обстановке на высококачественном оборудовании может извлечь преимущество из более высокого качества звука и меньшего отделения. Воспроизведение звука в транспортных средствах в шумной среде через динамики, встроенные в смартфон, может извлечь преимущество из более низкого качества звука, но более высокого отделения и разборчивости речи.Target value for sound quality

will determine the trade-off between separation and sound quality. This setting can be controlled by the user or specified depending on the audio playback scenario. Playing audio at home in a quiet environment on high quality equipment can benefit from higher sound quality and smaller separation. Audio playback in vehicles in noisy environments through speakers built into a smartphone can benefit from lower sound quality but higher separation and speech intelligibility.

In addition, the estimated values (either the audio quality component or the processing parameters) can be further applied either to control the post-processing or to control the secondary compartment.

Thus, four different concepts can be used to implement the proposed method. These concepts are illustrated in Fig. 2, fig. 3, fig. 4 and FIG. 5 and are described below.

Fig. 2 illustrates an apparatus according to an embodiment that is configured to use audio quality estimation and that is configured to perform post-processing.

According to such an embodiment, determining module 120 may, for example, be configured to evaluate, depending on at least one of the estimated target signal and the input audio signal and on the estimated difference signal, the sound quality value as one or more result values, wherein the value audio quality indicates the estimated audio quality of the estimated target signal. The determiner 120, for example, may be configured to determine one or more parameter values depending on the sound quality value.

Thus, according to an embodiment, determining module 120, for example, may be configured to determine, depending on the estimated audio quality of the estimated target signal, the control parameter as one or more parameter values. The signal processor 130, for example, may be configured to determine the separated audio signal depending on the control parameter and depending on at least one of the estimated target signal and the input audio signal and the estimated difference signal.

The following describes specific embodiments.

.At the first stage, separation is applied. The separated signal and the raw signal are input to a Quality Estimation Module (QEM). QEM calculates a score for audio quality components,

.

используются для вычисления набора параметров

для управления последующей обработкой.Sound Quality Evaluation Components

are used to calculate a set of parameters

to control post-processing.

,

,

и

могут изменяться во времени, но зависимость от времени в дальнейшем опущена для ясности обозначения.Variables

,

,

and

may change over time, but the dependence on time is omitted from here on for clarity of notation.

Such post-processing, for example, adds a scaled or filtered copy of the input signal to a scaled or filtered copy of the output signal and thereby reduces the attenuation of interference signals (for example, the effect of separation), for example,

(13),

(13),

управляет величиной отделения.where parameter

controls the size of the branch.

In other embodiments, for example, the formula may be used:

,

,

- оцененный разностный сигнал.where

is the estimated difference signal.

The reduction in separation leads to

1) reducing the number of artifacts and

2) increased leakage of interference sounds, which masks separation artifacts.

- отделенный аудиосигнал,

- оцененный целевой сигнал,

- входной аудиосигнал,

- управляющий параметр, и

- индекс.Thus, in an embodiment, the signal processor 120, for example, may be configured to determine the separated audio signal depending on formula (13), where

- separated audio signal,

- estimated target signal,

- input audio signal,

is the control parameter, and

- index.

и целевого показателя качества

,The parameter is calculated taking into account the sound quality rating

and quality target

,

(14).

(fourteen).

, например, может представлять собой итерационный экстенсивный поиск, как проиллюстрировано с помощью следующего псевдокода.This function

, for example, can be an iterative extensive search, as illustrated by the following pseudocode.

может быть вычислено следующим образом.Alternatively, the ratio

can be calculated as follows.

для набора значений

,

.1. Calculation

for a set of values

,

.

посредством интерполяции и экстраполяции.2. Calculation of the remaining values

through interpolation and extrapolation.

управляет последующей обработкой, как в уравнении (13),

вычисляется для фиксированного количества значений

, например, соответствующих 18, 12 и 6 дБ относительного усиления

.For example, when the processing parameter

controls post-processing as in equation (13),

calculated for a fixed number of values

, for example, corresponding to 18, 12 and 6 dB of relative gain

.

аппроксимируется, и

может быть выбрано.So the display

is approximated, and

can be chosen.

To summarize, in an embodiment, the signal processor 130, for example, may be configured to generate a separated audio signal by determining a first version of the separated audio signal and by changing the separated audio signal one or more times to obtain one or more intermediate versions of the separated audio signal. The determiner 120, for example, may be configured to change the sound quality value depending on one of one or more intermediate values of the separated audio signal. The signal processor 130, for example, may be configured to stop changing the separated audio signal if the audio quality value is greater than or equal to a predetermined quality value.

Fig. 3 illustrates an apparatus according to another embodiment in which post-processing parameters are directly evaluated.

и входного сигнала

. Это означает, что операция в уравнении 14 перемещена в фазу обучения, и регрессионный метод обучается оценивать

вместо

. Следовательно, производится обучение следующей функции.Separation is applied first. The separated signals are input to a Parameter Estimation Module (PEM). Estimated parameters are used to control post-processing. The PEM was trained to directly estimate p(n) based on the separated signal

and input signal

. This means that the operation in Equation 14 is moved to the learning phase and the regression method is trained to evaluate

instead of

. Therefore, the next function is trained.

(15).

(fifteen).

. Однако несколько моделей могут быть обучены на разных значениях

. Таким образом, окончательная гибкость в выборе

может быть сохранена.Obviously, this procedure has the advantage of requiring less computation than the procedure described above. This comes at the cost of less flexibility as the model is trained for a fixed setting.

. However, multiple models can be trained on different values

. So the ultimate flexibility in choosing

can be saved.

In an embodiment, signal processor 130, for example, may be configured to generate a separated audio signal depending on one or more parameter values and depending on subsequent processing of the estimated target signal.

Fig. 4 illustrates an apparatus according to a further embodiment in which sound quality evaluation and secondary separation are performed.

вводятся либо входной сигнал

, либо результат первого отделения

, линейная комбинация обоих

, где

и

являются весовыми коэффициентами, или промежуточный результат из первого отделения.Separation is applied first. The separated signals are entered into QEM. The estimated audio quality components are used to calculate a set of parameters for controlling the secondary compartment. To the secondary department

input or input signal

, or the result of the first division

, a linear combination of both

, where

and

are the weights, or the intermediate result from the first division.

Thus, in such an embodiment, the signal processor 130, for example, may be configured to generate a separated audio signal depending on one or more parameter values and depending on the linear combination of the estimated target signal and the input audio signal, or the signal processor 130, for example, may be configured to generate a separated audio signal depending on one or more parameter values and depending on a linear combination of the estimated target signal and the estimated difference signal.

Suitable parameters for controlling the secondary compartment are, for example, parameters that modify the spectral weights.

In FIG. 5 shows an apparatus according to another embodiment in which separation parameters are directly evaluated.

Separation is applied first. The separated signals are entered into the PEM. Estimated parameters control the secondary branch.

, линейная комбинация обоих

, где

и

являются весовыми коэффициентами, или промежуточный результат из первого отделения.Either the input signal x(n) or the result of the first branch are entered into the secondary branch z(n)

, a linear combination of both

, where

and

are the weights, or the intermediate result from the first division.

, и

из уравнений (5), (6) и

, как описано выше.For example, the following parameters are controlled:

, and

from equations (5), (6) and

as described above.

With respect to iterative processing according to the embodiments, FIG. 4 and 5 show iterative processing with one iteration. In the general case, it can be repeated several times and implemented in a loop.

Iterative processing (without intermediate quality evaluation) is very similar to other previous methods that perform the concatenation of several branches.

This approach, for example, may be suitable for combining several different ways (which is better than repeating one way).

Although some aspects have been described in the context of a device, it is clear that these aspects are also a description of the corresponding method, in which the module or device corresponds to a method step or a feature of a method step. Likewise, aspects described in the context of a method step are also descriptions of the respective module, element, or feature of the respective device. Some or all of the steps of the method may be executed by (or using) a hardware device such as, for example, a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, one or more of the most important steps of the method may be performed by such a device.

Depending on some implementation requirements, embodiments of the invention may be implemented in hardware or software, at least partially in hardware, or at least partially in software. Implementation can be done using digital storage media such as floppy disk, digital versatile disk (DVD), Blu-ray disc, compact disc (CD), read only memory (ROM; ROM), programmable read only memory (PROM; PROM). ), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), and flash memory having electronically readable signals stored thereon that interact (or are capable of interacting) with a programmable computer system. , resulting in the corresponding method being executed. Thus, the digital storage medium can be computer readable.

Some embodiments in accordance with the invention include a storage medium having electronically readable control signals that are capable of interacting with a programmable computer system, resulting in one of the methods described herein.

Typically, embodiments of the present invention may be implemented as a computer program product with program code, the program code performing one of the methods when the computer program product is executed on the computer. The program code may, for example, be stored on a computer-readable medium.

Other embodiments include a computer program for performing one of the methods described herein, stored on a computer-readable medium.

In other words, an embodiment of the method of the invention is thus a computer program having program code for performing one of the methods described herein when the computer program is executed on a computer.

An additional embodiment of the methods of the invention, therefore, is a storage medium (or digital storage medium, or computer-readable medium) containing a computer program recorded thereon for performing one of the methods described herein. The storage medium, digital storage medium, or recorded data medium is typically a tangible and/or non-transitory medium.

An additional embodiment of the method of the present invention is thus a data stream or signal sequence representing a computer program for performing one of the methods described herein. The data stream or signal sequence, for example, may be configured to be carried over a data connection, such as over the Internet.

An additional embodiment comprises a processing means, such as a computer or programmable logic device, configured or adapted to perform one of the methods described herein.

An additional embodiment comprises a computer having a computer program installed thereon for performing one of the methods described herein.

An additional embodiment in accordance with the invention comprises an apparatus or system configured to transfer to a receiver (eg, electronically or optically) a computer program to perform one of the methods described herein. The receiver, for example, may be a computer, a mobile device, a storage device, or the like. The device or system, for example, may include a file server for transferring a computer program to the receiver.

In some embodiments, a programmable logic device (eg, a field programmable gate array) may be used to perform some or all of the functionality of the methods described herein. In some embodiments, a user-programmable gate array may interface with a microprocessor to perform one of the methods described herein. In general, the methods are preferably performed by any hardware device.

The apparatus described herein may be implemented using a hardware device, using a computer, or using a combination of a hardware device and a computer.

The methods described herein may be performed using a hardware device, using a computer, or using a combination of a hardware device and a computer.

The embodiments described above are merely illustrative of the principles of the present invention. It is intended that modifications and variations of the arrangements and details described herein will be apparent to others skilled in the art. Thus, the invention is intended to be limited only by the scope of the following patent claims and not by the specific details provided by way of the description and explanation of the embodiments set forth herein.

Literature

[1] E. Vincent, R. Gribonval, and C. Févotte, “Performance measurement in blind audio source separation,” IEEE Transactions on Audio, Speech and Language Processing, vol. 14, no. 4, pp. 1462-1469, 2006.

[2] V. Emiya, E. Vincent, N. Harlander, and V. Hohmann, “Subjective and objective quality assessment of audio source separation,” IEEE Trans. Audio, Speech and Language Process., vol. 19, no. 7, 2011.

[3] R. Huber and B. Kollmeier, “PEMO-Q - a new method for objective audio quality assessment using a model of audatory perception,” IEEE Trans. Audio, Speech and Language Process., vol. 14, 2006.

[4] ITU-R Rec. BS.1387-1, “Method for objective measurements of perceived audio quality,” 2001.

[5] ITU-T Rec. P.862, “Perceptual evaluation of speech quality (PESQ): An objective method for end-to-end speech quality assessment of narrow-band telephone networks and speech codecs,” 2001.

[6] ITU-T Rec. P.862.1, “Mapping function for transforming P.862 raw results scores to MOS-LQO,” 2003.

[7] A. Hines, E. Gillen et al., “ViSQOLAudio: An Objective Audio Quality Metric for Low Bitrate Codecs,” J. Acoust. soc. Am., vol. 137, no. 6, 2015.

[8] J. M. Kates and K. H. Arehart, “The Hearing-Aid Audio Quality Index (HAAQI),” IEEE Trans. Audio, Speech and Language Process., vol. 24, no. 2, 2016, evaluation code kindly provided by Prof. J.M. Kates.

[9] J. M. Kates and K. H. Arehart, “The Hearing-Aid Speech Quality Index (HASQI) version 2,” Journal of the Audio Engineering Society, vol. 62, no. 3, pp. 99-117, 2014.

[10] J. M. Kates and K. H. Arehart, “The Hearing-Aid Speech Perception Index (HASPI),” Speech Communication, vol. 65, pp. 75-93, 2014.

[11] C. Taal, R. Hendriks, R. Heusdens, and J. Jensen, “An algorithm for intelligibility prediction of time-frequency weighted noisy speech,” IEEE Trans. Audio, Speech and Language Process., vol. 19, no. 7, 2011.

[12] E. Manilow, P. Seetharaman, F. Pishdadian, and B. Pardo, “Predicting algorithm efficacy for adaptive multi-cue source separation,” in Applications of Signal Processing to Audio and Acoustics (WASPAA), 2017 IEEE Workshop on , 2017, pp. 274-278.

[13] M. Cartwright, B. Pardo, G. J. Mysore, and M. Hoffman, “Fast and easy crowdsourced perceptual audio evaluation,” in Acoustics, Speech and Signal Processing (ICASSP), 2016 IEEE International Conference on, 2016.

[14] S.-W. Fu, T.-W. Wang, Y. Tsao, X. Lu, and H. Kawai, “End-to-end waveform utterance enhancement for direct evaluation metrics optimization by fully convolutional neural networks,” IEEE/ACM Transactions on Audio, Speech and Language Processing (TASLP) , vol. 26, no. 9, 2018.

[15] Y. Koizumi, K. Niwa, Y. Hioka, K. Koabayashi, and Y. Haneda, “Dnn-based source enhancement to increase objective sound quality assessment score,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2018.

[16] Y. Zhao, B. Xu, R. Giri, and T. Zhang, “Perceptually guided speech enhancement using deep neural networks,” in Acoustics, Speech and Signal Processing (ICASSP), 2018 IEEE International Conference on, 2018.

[17] Y. Koizumi, K. Niwa, Y. Hioka, K. Kobayashi, and Y. Haneda, “Dnn-based source enhancement self-optimized by reinforcement learning using sound quality measurements,” in Acoustics, Speech and Signal Processing ( ICASSP), 2017 IEEE International Conference on, 2017.

[18] J. Jensen and M. S. Pedersen, “Audio processing device comprising artifact reduction,” US Patent US 9,432,766 B2, Aug. 30, 2016.

Claims (63)

1. A device for generating a separated audio signal from an input audio signal, the input audio signal comprising a target audio signal section and a difference audio signal section, the difference audio signal section indicating a difference between the input audio signal and the target audio signal section, the device comprising: a source separator (110) for determining an estimated target signal that depends on the input audio signal, wherein the estimated target signal is an estimate of a signal that contains only a portion of the target audio signal, determination module (120), wherein the determination module (120) is configured to determine one or more resulting values depending on the estimated sound quality of the estimated target signal to obtain one or more parameter values, wherein one or more parameter values are one or more result values or depend on one or more result values, and a signal processor (130) for generating a separated audio signal depending on one or more parameter values and depending on at least one of the estimated target signal, and the input audio signal, and the estimated difference signal, where the estimated difference signal is an estimate of a signal that contains only section of the difference audio signal, wherein the signal processor (130) is configured to generate a separated audio signal depending on one or more parameter values and depending on a linear combination of the estimated target signal and the input audio signal, or wherein the signal processor (130) is configured to generate a separated audio signal depending on one or more parameter values and depending on a linear combination of the estimated target signal and the estimated difference signal. 2. The device according to claim 1, in which the determining module (120) is configured to determine, depending on the estimated sound quality of the estimated target signal, the control parameter as one or more parameter values, and wherein the signal processor is configured to determine the separated audio signal depending on the control parameter and depending on at least one of the estimated target signal and the input audio signal and the estimated difference signal. 3. Device according to claim 2, wherein the signal processor (130) is configured to determine the separated audio signal depending on:

, or depending on:

, where

- separated audio signal, where

- estimated target signal, where

- input audio signal, where

is the estimated difference signal, where

is the control parameter, and where

- index. 4. Device according to claim 2, in which the determining module (120) is configured to evaluate, depending on at least one of the estimated target signal, and the input audio signal, and the estimated difference signal, the sound quality value as one or more result values, and the sound quality value indicates the estimated quality the sound of the estimated target signal, and wherein the determination module (120) is configured to determine one or more parameter values depending on the sound quality value. 5. The device according to claim 4, wherein the signal processor (130) is configured to generate a separated audio signal by determining a first version of the separated audio signal and by changing the separated audio signal one or more times to obtain one or more intermediate versions of the separated audio signal, in which the determination module (120) is configured to change the sound quality value depending on one of one or more intermediate values of the separated audio signal, and wherein the signal processor (130) is configured to stop changing the separated audio signal if the sound quality value is greater than or equal to the predetermined quality value. 6. Device according to claim 1, wherein the determining module (120) is configured to determine one or more result values depending on the estimated target signal and depending on at least one of the input audio signal and the estimated difference signal. 7. The device according to claim 1, in which the determination module (120) contains an artificial neural network (125) to determine one or more resulting values depending on the estimated target signal, and the artificial neural network (125) is configured to receive a plurality of input values, each of the plurality of input values depends on from at least one of the estimated target signal, and the estimated difference signal, and the input audio signal, and wherein the artificial neural network (125) is configured to determine one or more resulting values as one or more output values of the artificial neural network (125). 8. Device according to claim 7, wherein each of the plurality of input values depends on at least one of the estimated target signal and the estimated difference signal and the input audio signal, and in which one or more result values indicate the estimated audio quality of the estimated target signal. 9. The device according to claim 7, wherein each set of input values depends on at least one of the estimated target signal and the estimated difference signal and the input audio signal, and in which one or more result values are one or more parameter values. 10. Device according to claim 7, wherein the artificial neural network (125) is configured to learn by receiving a plurality of training data sets, each of the plurality of training data sets comprising a plurality of artificial neural network (125) input training values and one or more artificial neural network (125) output training values , wherein each of the plurality of training output values depends on at least one of the training target signal, and the training difference signal, and the training input signal, with each of the one or more training output values dependent on the audio quality estimate of the training target signal. 11. The device according to claim 10, wherein the estimate of the sound quality of the training target signal depends on one or more computational sound quality models. 12. Device according to claim 11, wherein one or more computational sound quality models are at least one of the following models: Blind Source Separation Evaluation, methods of perceptual evaluation for separating audio sources, audio quality perception evaluation, assessment of perception of speech quality, audio with speech quality virtual objective listener, hearing aid audio quality index, hearing aid speech quality index, hearing aid speech perception index, and short-term objective intelligibility. 13. The device according to claim 7, wherein the artificial neural network (125) is configured to determine one or more result values depending on the estimated target signal and depending on at least one of the input audio signal and the estimated difference signal. 14. Device according to claim 1, wherein the signal processor (130) is configured to generate a separated audio signal depending on one or more parameter values and depending on subsequent processing of the estimated target signal. 15. A method for generating a separated audio signal from an input audio signal, the input audio signal comprising a target audio signal portion and a difference audio signal portion, the difference audio signal portion indicating a difference between the input audio signal and the target audio signal portion, the method comprising: determining an estimated target signal that depends on the input audio signal, wherein the estimated target signal is an estimate of a signal that contains only a portion of the target audio signal, determining one or more result values depending on the estimated audio quality of the estimated target signal to obtain one or more parameter values, where one or more parameter values are one or more result values or depend on one or more result values, and generating a separated audio signal depending on one or more parameter values and depending on at least one of the estimated target signal, and the input audio signal, and the estimated difference signal, wherein the estimated difference signal is an estimate of a signal that contains only a portion of the difference audio signal, moreover, the formation of the separated audio signal is carried out depending on one or more parameter values and depending on the linear combination of the estimated target signal and the input audio signal; or wherein the formation of the separated audio signal is carried out depending on one or more parameter values and depending on the linear combination of the estimated target signal and the estimated difference signal. 16. A computer-readable medium containing program code for performing the method of claim 15 when executed on a computer processor or signal processor.

Images