EP4600952A1

EP4600952A1 - Decoder

Info

Publication number: EP4600952A1
Application number: EP24156127.3A
Authority: EP
Inventors: Andreas BRENDEL; Kishan GUPTA; Nicola PIA; Guillaume Fuchs; Suraj Pandey; Markus Multrus
Original assignee: Fraunhofer Gesellschaft zur Foerderung der Angewandten Forschung eV
Current assignee: Fraunhofer Gesellschaft zur Foerderung der Angewandten Forschung eV
Priority date: 2024-02-06
Filing date: 2024-02-06
Publication date: 2025-08-13
Also published as: WO2025168598A1

Abstract

Decoder (20') for decoding an encoded input audio signal, the decoder comprising: a conditioning entity (20c), receiving an encoded style information derived from the encoded input audio signal and performing at least a learnable affine transform of an encoded content information derived from the encoded input audio signal to obtain a transformed encoded content information; and a decoder (20d) configured to decode the transformed encoded content information further.

Description

Embodiments of the present invention refer to a decoder for decoding the encoded input audio signal. The coding of the encoder and the decoder enable that the input data is compressed. Further embodiments refer to the corresponding methods and to a computer program. Preferred embodiments are in a field of neural audio codec using speaker embeddings.
State-of-the-art (SOTA) neural speech coding achieves good quality of the reconstructed speech at bitrates as low as 3.2kbps. SOTA neural speech coders [1,2,3] are typically trained in an end-to-end fashion, where an input audio signal is encoded by an encoding neural network, the output of the encoder (the so-called latent) is quantized and the resulting quantized latent is decoded by a decoding neural network (see Fig. 1). During application, the encoder runs on transmitter side providing quantized latents (or rather identifying indices of the used latents) which are transmitted to the receiver side where the decoder reproduces the input speech signal.
Schematic principle is illustrated by Fig. 1.
Fig. 1 shows exemplarily a codec having an encoder 10 receiving an input audio signal IS so as to encode the input audio signal to obtain the encoded input audio signal Q. This encoded audio signal Q can be decoded by use of the decoder 20. The decoder 20 is configured to provide based on the signal Q a reconstructed input audio signal IS'.
In this way, the submitted data stream Q must carry a long term information, like speaker identity, speaker style (called style in the following), etc., as well as short term information, like phonetic content (called content in the following). This way of coding has drawbacks with regard to the coding efficiency. Therefore, there is a need for an improved approach.
An objective of the present invention is to provide a concept for coding speech having high coding efficiency.
This objective is solved by the subject-matter of the independent claims.

Aspect Decoder

An embodiment provides a decoder for decoding an encoded input audio signal. The decoder comprises a conditioning entity and a content encoder. The conditioning entity is configured to receive an encoded style information derived from the encoded input audio signal and to perform at least a learned affine transform of at least one encoded content information derived from the encoded input audio signal to obtain a transformed encoded content information. The decoder is configured to further decode the transformed encoded content information.
Embodiments of the present invention are based on the finding that it is advantageous to disentangle content and style information. Background thereof is that the content information which may comprise a phonetic content mainly comprises short term information, while the style information which may comprise an information on the speaker identity or speaking style mainly carries a long term information. Thus, the decoder comprises the two entities conditioning entity and the decoder so as to enable the decoding of the two different information (content), namely the short-term information and the long-term information. The long-term information, also referred to as style information, is obtained using the conditioning entity while the short-term information, i.e., the content information, is decoded by the decoder taking into account the previously processed long-term information/style information.
According to embodiments, the decoder comprises at least one learnable layer. According to embodiments, the decoder may comprise at least one temporal adaptive DE-normalization (TADE) residual block. Note, decoders based on TADE layers are one possibility for designing the neural network. The decoder could be in principle any neural network decoder.
According to embodiments, the decoder is configured to use a default control signal or initial control signal before the conditioning entity provides the control signal based on the first data stream. For example, the default control signal or initial control signal represents an average style and/or an average style for a male speaker and/or an average style for a female speaker; alternatively, the default control signal or initial control signal represents an average style selected out of a plurality of average styles, wherein the selection is taken based on an external information.
According to embodiments, the decoder may comprise a conditioning entity and a decoder. The conditioning entity may according to embodiments output a control signal being a default control signal which comprises a style information. The decoder is configured to decode the data stream (comparable to the second data stream) of the encoded input audio signal comprising content information, wherein the decoder is controlled and/or adapted by the control signal. By providing the default style, the reconstructed speech may anonymized, i.e., the reconstructed speech signal does not allow for identifying the speaker as the style was changed (speaker privacy can be achieved according to this aspect).
According to embodiments, the default control signal representing a default style information may be selected out of a plurality of default styles. This may be done by use of a control signal.
According to embodiments, the first data stream, is transmitted less frequently or with longer transmission intervals than the second data stream.
According to embodiments, the conditioning entity and/or the decoder use neural codecs like NESC or other network codecs for encoding.
According to embodiments, the conditioning entity and/or the decoder are trained or trainable using machine learning techniques; additionally or alternatively a layer of the conditioning entity and/or decoder comprise recurrent neural network block and/or Temporal Adaptive DE-normalization layer and/or Pseudo Quadrature Mirror Filterbank.

Additional Decoder Aspect

Another embodiment provides a decoder for decoding an encoded input audio signal, like the input audio signal as encoded using the encoder as defined above. The decoder may comprise a conditioning entity and a decoder. The conditioning entity receives a first (latent) data stream of the encoded input audio signal (cf. above discussion) and is configured to output a control signal based on the first (latent) data stream. The first (latent) data stream comprises an encoder style information. The decoder is configured to decode the second (latent) data stream of the encoded input audio signal comprising content information (cf. above discussion). The decoder is controlled and/or adapted by the control signal of the conditioning entity.
According to embodiments, the decoder is configured to obtain a reconstructed signal of the encoded input audio signal based on the first and second (latent) data stream. According to embodiments, the reconstructed signal is output by the decoder taking into account the control signal of the conditioning entity. This control information is based on the first (latent) data stream comprising the encoded style information.
In case the style encoder could not immediately provide the control signal, e.g. at the beginning of the decoding, the decoder may be configured to use a default control signal or initial control signal before the conditioning entity provides the control signal based on the first data stream. For example, the default control signal may be stored in the decoder. The default control signal and/or the initial control signal may represent an average style. According to embodiments, different default control signals or different initial control signals may be used. For example, an average style information for a male speaker and/or an average style information for a female speaker may be provided so as to start the decoding based on the default control signal/initial control signal. According to embodiments, the default control signal or initial control signal represents an average style selected out of the plurality of average styles, wherein the selection is taken based on an external information.
Another embodiment provides a decoder for decoding an encoded input audio signal comprising a conditioning entity receiving the encoded input audio signal and configured to output a control signal based on the encoded input audio signal. The encoded input audio signal comprises an encoded style information. The decoder further comprises a decoder configured to decode the encoded input audio signal further comprising content information, wherein the decoder is controlled and/or adapted by the control signal. The decoder may be configured to use a default control signal or an initial control signal before the conditioning entity provides the control signal based on external information or the first (latent) data stream.
According to embodiments, the default control signal or initial control signal represents a given style selected out of a plurality of available styles, wherein the selection is taken based on an external information or control. In this case, a deliberated anonymization or desired voice conversion could be achieved.
According to further embodiments, the decoder is configured to decode an encoded input audio signal comprising a conditioning entity and a decoder. The conditioning entity is configured to output a control signal representing a style information. According to embodiments, the default control signal or initial control signal is used as control signal and thus, represents a given style selected out of one or more available styles. The decoder is configured to decode the encoded input audio signal comprising content information, wherein the decoder is controlled and/or adapted by the control signal. Since a default control signal or initial control signal is used, an anonymization of the content is achieved.
According to embodiments, the default control signal or initial control signal represents an average style and/or average style for a male speaker and/or average style for a female speaker. The default control signal or initial control signal may according to further embodiments represent an (average) style selected out of a plurality of (average) styles, wherein the selection is, for example, taken based on an (external) information. Note that it could also be a set of other candidate styles not necessarily an average of other styles i.e., any representation that produces a desired styling of the output.
According to embodiments, the conditioning may be performed by a TADE layer (one of the developed network building blocks based on StyleMelGAN and NESC as described below) or a conditioning network comprising one or several TADE layers.

Aspect Encoder

An embodiment provides an encoder for encoding an input audio signal like a speech signal. The encoder comprises a style encoder and a content encoder. The style encoder is configured to encode style information of the input audio signal to obtain a first (latent) data stream. The content encoder is configured to encode content information to obtain a second (latent) data stream.
Embodiments of the present invention are based on the finding that it is advantageous to disentangle content and style information. Background thereof is that the content information which may comprise a phonetic content mainly comprises short term information, while the style information which may comprise an information on the speaker identity or speaking style mainly carries a long term information. Starting from this finding, the encoder is subdivided into two parallel acting encoders, namely as style encoder and a content encoder. In this way, it is advantageously possible to increase the coding efficiency.
According to an embodiment, the first and the second (latent) data stream are output and/or transmitted separately. For example, the first and second (latent) data stream may be quantized separately before transmitting.
As already mentioned above, the first (latent) data stream which is output by the style encoder comprises (mainly) long term information, especially speaker identity information or speaking style information. Furthermore, the second (latent) data stream (output by the content encoder) may comprise short term information, especially phonetic content. Due to this, it is advantageously possible not only to separate the encoding, but also the transmission, so that the two data streams can be encoded with separate entities. Due to the long term and short term character of the first and second (latent) data stream, the first (latent) data stream may comprise an information which is sent at much lower frequencies relative to the second (latent) data stream. For example, the first (latent) data stream may comprise data blocks that are sent at low frequency, e.g., below 10 times per second as the coded style information is slowly time varying. The second (latent) data stream may be composed of data blocks that have to be sent more frequently, e.g., 10-100 times per second, to account for the short-term nature of the coded signal content. This means that the content information is sent more frequently than the style information.
According to embodiments, the above-defined encoder is especially advantageous for speech coding. Therefore the input audio signal may be a speech signal or may comprise a speech recording. For example, an input audio signal may be a speech signal of a person, the person being characterized by the first (latent) data stream with regard to a characteristic speech style, e.g. male speech style or female speech style or another speech style. Such speech style information is highly characteristic for a speech style and describes the speech behavior of a person whose speech should be coded very good, but general.
Below, possible implementation details for the encoders are given. According to embodiments, the style encoder and the content encoder use neural codecs like NESC or other neural network codecs for encoding. Such neural codecs are advantageously trained or trainable using machine learning techniques. Thus, according to embodiments, the style encoder and/or the content encoder are trained or trainable using machine learning techniques (artificial intelligence algorithms or neural network approaches). According to embodiments, the style encoder is configured to employ layers of a recurrent neural network or layers representing long-term information, like for example a dilated convolutional neural network or a transformer.
According to embodiments, the content encoder is configured to use normalization layers enforcing temporal whiteness or input perturbations or vocal tract length perturbations on the input audio signal waveform in order to emphasize short-term characteristics.
The disentanglement of content and style (in principle) also allows for separate manipulation of both representations and corresponding decoding, e.g., for speaker anonymization. Thus, according to embodiments, the encoder can just use the content encoder configured to encode content information to obtain a second data stream. In this way, the content can be transmitted without transmitting the identity sensitive style information.
On the decoder side, a default style information is used so that the content can be reproduced in an anonymized manner.
Expressed in other words: By manipulating the style information in an intended, user-defined way and transmitting the content information as it is, the same spoken content can be transmitted but the speaker identity is hidden, thereby preserving privacy of the conversation partner.

Methods

An embodiment provides a method for decoding an encoded input audio signal. The method comprising:

receiving an encoded style information derived from the encoded input audio signal;
performing at least a learnable affine transform of an encoded content information derived from the encoded input audio signal to obtain a transformed encoded content information; and
further decoding the transformed encoded content information using a decoder.

receiving a first (latent) data stream of the encoded input audio signal and outputting a control signal based on the first (latent) data stream, the first (latent) data stream comprising an encoded style information;
decoding a second (latent) data stream of the encoded input audio signal comprising content information, wherein the decoder is controlled and/or adapted by the control signal.

Another embodiment provides a method for encoding an input audio signal:

encoding style information of the input audio signal to obtain a first (latent) data stream;
encoding content information to obtain a second (latent) data stream.

Another embodiment provides a method for decoding an encoded input audio signal, and the method comprising:

receiving the encoded input audio signal and outputting a control signal based on the encoded input audio signal, the first (latent) data stream comprising an encoded style information;
using a default control signal or initial control signal before the conditioning entity provides the control signal based on the first (latent) data stream;
decoding the encoded input audio signal further comprising content information, wherein the decoder is controlled and/or adapted by the control signal.

Regarding the default control signal or initial control signal it should be noted that providing a manipulated or a default control signal may be used not only to cover the time span before the first encoded style information is received but also for anonymizing the speaker (here, the style information may not be transmitted, thereby keeping the identifying information of the speaker included in the audio signal private).
An embodiment provides a method for decoding an encoded audio signal comprising a conditioning entity configured to output a control signal being a default control signal and representing a style information; a decoder configured to decode a data stream of the encoded input audio signal comprising content information, wherein the decoder is controlled and/or adapted by the control signal.
According to an embodiment, the method may further comprise setting a default/manipulated style on the decoder side and not transmitting the style information for not disclosing identifying information about the speaker in the transmitted signals (encoder side).
According to further embodiments, the above-discussed methods may be computer implemented. Therefore, embodiments refer to a computer program for performing, when running on a computer, one of the methods as defined above.

Below, embodiments of the present invention will subsequently be discussed referring to the enclosed figures, wherein:

Fig. 1	shows a schematic block diagram illustrating a codec according to a conventional approach;
Fig. 2	shows the schematic block diagram illustrating a codec according to embodiment;
Fig.3a	shows a schematic block diagram illustrating an encoder according to an embodiment; and
Fig. 3b	shows a schematic block diagram illustrating a decoder according to embodiments.

Below, embodiments of the present invention will subsequently be discussed referring to the herewith enclosed figures, wherein identical reference numeral are provided to objects having identical or similar function, so that description thereof is interchangeable and mutually applicable.
Starting from the above discussed conventional codec having the encoder 10 and the decoder 20 an enhanced codec having the encoder 10' and the decoder 20' will be discussed taking reference to Fig. 2.
The encoder 10' receives the input audio signal IS, e.g., an audio signal comprising speech. The encoder 10' performs encoding so as to obtain two output data streams Q1 and Q2. These two audio data streams Q1 and Q2 are transmitted, e.g., separately to the decoder 20'. The decoder 20' performs a reconstruction so as to obtain the reconstructed signal IS'.
The encoder 10' comprising two encoding entities, namely a style encoder 10s and a content encoder 10c. Each of the two encoders 10s and 10c receives the input audio signal IS and performs an encoding of a portion of the input audio signal. The encoder 10s' outputs the coded audio signal Q1, while the encoder 10c outputs the coded audio signal Q2.
On the decoder side, the signal Q1 and Q2 are processed by a conditioning/PreNet 20c, and afterwards by the decoder 20d. decConsequently, decoder 20' uses the two signals Q1 and Q2 so as to output the reconstructed signal IS'.
Since now the structure of the codec according to an enhanced approach has been discussed, the functionality of the encoder 10' and the decoder 20' will be discussed separately taking reference to the entities 10s and 10c and 20c and 20d, respectively.
For the below discussion, references taken to Fig. 3a showing the encoder 10' comprising the style encoder 10s and the content encoder 10c.
The style codec 10s is configured to code style information of the input audio signal to obtain a first (latent) data stream Q1. The style information may comprise long term information, like a style information. For example, the style information may be dependent on the gender (male/female speakers) or dependent on the language. The style information may have an influence on the general frequency dependent distribution of power of the voice and/or to the general sound. Due to the encoding, the signal Q1 comprises the specific coding style information.
The content encoder 10c is configured to encode content information, i.e., short term information, like phonetic information.
Due to the proposed concept of separately encoding content and style information performed by the separate encoders 10s and 10c, embodiments of present invention are more efficient than the SOTA approaches to neural speech coding [1, 2, 3].
To this end, we use (at least) two individual encoding networks, e.g., one for encoding style information and one for encoding content information. Both encoders obtain as input the speech signal that is to be transmitted. The outputs of these encoders, i.e., latents, are quantized separately and transmitted to receiver side. Here, the style latent is combined with the content latent for decoding with the joint decoding network (cf. Fig. 2).
According to embodiments, the encoder 10 (as well as the decoder 20) may be realized by neural codec NESC [1]. Below, possible implementations for a neural codec will be discussed: generally speaking, any neural network that transforms a potentially preprocessed audio signal into a compact representation (typically reducing the dimensionality of the signal representation) can serve as an encoder for a neural codec. This means that a neural codec may be characterized by a preprocessing of the audio signal and a transformation of the preprocessed audio signal so as to obtain a compact representation of same. Here, typically convolutional neural network blocks are used that downsample the input audio signal successively and recurrent neural network blocks or transformers that model additional temporal context. Part of this encoder would be also a subsequently applied quantizer transforming that continuous output of the neural network to a discreet representation that can be potentially entropy-coded and transmitted. Formally, also a formatter for preparing the data stream, etc., may be part of it.
NESC is an example for a neural codec. NESC may, for example, consist of the following blocks:
At the encoder side, the input waveform is transformed into a two-dimensional representation by a rolling window frontend (The frontend is the first component of the encoder (receiving the input audio signal)). The result is processed by a convolutional layer within each frame, recurrent layers along different frames and again a convolutional layer operating within each frame. Together, this is termed dual path convolutional recurrent neural network (DPCRNN). After the DPCRNN, the signal is further processed by several convolutional neural network blocks. The output of these blocks is quantized, e.g., by residual vector quantization (NESC) or by scalar quantization.
The decoder 20' as already shown by Fig. 2 will be discussed taking reference to Fig. 3b showing the decoder 20d and the conditioning entity 20c as central components. According to an embodiment, the conditioning entity 20c may be configured to receive the encoded style information Q1 derived from the encoded input audio signal, e.g., via the data stream Q1, and performs at least a learnable affine transform of the at least one coded content information (signal) Q2 derived from the encoded input audio signal. The result is an obtained transformed encoded content information (signal) TC. This transformed content information signal TC is forwarded as kind of preprocessed content signal (transformed encoded content information) to the decoder 20d which performs a further decoding. The decoder 20d receives the contend of Q2 via 20c as transformed content TC and outputs IS'.
According to embodiments the preprocessing in terms of affine transform (e.g. performed by 20c) is controlled using an internal control signal C or an adapted control signal, e.g. derived from the encoded style information. Note that the encoded content information is transformed once with said control signal. Alternatively a stream of encoded content information(s), i.e. a plurality of encoded content information signals are transformed with the one control signal. This means in general that one or multiple intermediate representations of the encoded content information can be transformed with said control signal. As discussed above, the control signal may be a control signal without information on the identity of origin of the received audio signal, i.e. a default control signal having privacy or anonymous character.
Furthermore, the decoding performed by 20d has the purpose to obtain IS'. Note that according to embodiments, the data stream Q2 may suffice to be the input audio signal for 20c and 20d if just the content should be transmitted, e.g. in an anonymous manner.
At the decoder side, the quantized representation may according to further embodiments be preprocessed by recurrent neural network blocks (so as to implement an example neural codec based on NESC). The result is decoded by Streamwise-StyleMelGAN (SSMGAN). SSMGAN consists of a sequence of convolutional layers that subsequently upsample the input audio signal, where the input audio signal is used to condition each upsampling stage with a temporal adaptive DE-normalization (TADE) layer. The final signal is synthesized with a Pseudo-Quadrature Mirror Filter bank (PQMF).
According to embodiments, the neural codec used at the decoder side may comprise Streamwise-StyleMeIGAN, e.g., consisting of a sequence of convolutional layers that subsequently upsample the input audio signal. According to embodiments, the decoder may comprise an SSMGAN decoder including the TADE layers and the dequantization (mapping indices describing the discrete representation (or a stream of indices) to a latent waveform (representation) serving as a continuous input to the remaining part of the decoder. According to further embodiments, different network architectures may be used.
According to embodiments, the encoder comprises a style encoder and/or a content encoder which comprises the following elements:
a format definer configured to define a first multi-dimensional audio signal representation of the input audio signal (1), the first multi-dimensional audio signal representation of the input audio signal including at least:

a first dimension [e.g. inter frame dimension], so that a plurality of mutually subsequent frames is ordered according to the first dimension; and
a second dimension [e.g. intra frame dimension], so that a plurality of samples of at least one frame are ordered according to the second dimension,
and at least one learnable layer configured to process the first multidimensional audio signal representation of the input audio signal (1), or processed version of the first multi-dimensional audio signal representation

According to a comparison example, the decoder comprises a styling element of the content information from the first data stream or a processed version of it, the styling element being controlled by the control signal (C). Here, C is derived based on Q1. In doing this, a NESC decoder (SMGAN) involving the styling element can be used. According to embodiments, the Q1 and Q2 are processed by the conditioning prenet providing TC, so the decoder receives a preprocessed signal including the styling element and content element; the decoder further process TC to obtain a reconstructed signal (IS') of the encoded input audio signal based on the first (Q1) and second data stream (Q2). Here, styling elements/TADEs are used elsewhere in the decoder, e.g. using SSMGAN as decoder. For example, at least one of the styling elements comprises an adaptive element-wise modulation achieved by at least an element-wise multiplication or addition, or wherein at least one of the styling elements comprises a learnable affine transform, or wherein at least one of the styling elements is part of a weight layer in a residual block of a neural network (= rough/high level definitions of TADE and TADEResBlock).
As discussed above, the content encoder 10c captures short time relations within the input audio signal (e.g., phonetic information), while the style encoder captures more global information (e.g., the identity of the speaker or the speaking style). Hence, the content encoder produces latents at a much higher frequency (e.g., 10-100Hz) relative to the style encoder (less than 10Hz). Expressed in other words this means, that the frame rate used by the content encoder 10c is much higher when compared to the frame rate of the style encoder 10s. To be computationally efficient, both encoders may share some of their layers.
According to embodiments, layers capable of representing long-term information such as RNNs may be employed to enforce that the style encoder 10s learns long-term information to avoid leaking of style information to the content encoder, techniques such as normalization layers and forcing temporal whiteness or input perturbations like vocal tract length perturbations may be used.
Consequently, the two latents Q1 and Q2 comprise different information. It should be noted that the latent Q1 (output of the style encoder) is transmitted less often as the latent Q2 (cf. output of the content encoder 10c). Background is that style information is a long-term and/or global feature, while content is referring to the short time information.
Below, the decoder 20' performing the decoding based on the two signals Q1 and Q2 will be discussed. The decoder 20 comprises the two entities 20c and 20d. 20c receives the latents Q1 and Q2. Q1 comprises the style information. By use of Q1, the signal Q2 is transformed by the entity 20c which is also referred to as conditioning PreNet. For example learning affine transforms of intermediate representations can be used to obtain the transformed content signal. Said transformed content signal is than provided to the decoder 20d which performs a further decoding.
The decoder 20d receives the signal TC and performs a decoding so as to obtain the reconstructed signal IS'.
As the decoder 20c or 20d makes use of a long-term style information, which may not be available yet in an initial phase of decoding, initialization techniques may be used according to embodiments. For example, decoding with an average style or with an average style for male/female speakers may be used during the initiation phase. This means in other words, that the decoder 20c or 20d uses an initial control signal representing the average style or an average style of, e.g., male or female speakers . To select the (average) style out of a plurality of (average) styles an information may be used, e.g. extracted or included by Q1.
When selecting an average style or just a predefined style out of the plurality of styles, a control signal can be provided imprinting an anonymous identity to the decoded signal by use of the decoder 20d. In other words, this means that in case Q1 is not available, e.g., not transmitted, it is possible to decode the content in an anonymous manner, namely when selecting a style so as to obtain the control signal c for the decoder 20d. In other words, this means that it is, according to another aspect, beneficial not to transmit Q1 but just Q2, so that the content can be reproduced, while the identity information is not forwarded by the encoder 10'. Consequently, the decoder 20' which just uses a random style, enables to reproduce the content anonymously.
dec
Regarding the decoder 20d, it should be noted that same may comprise at least one learnable layer. For example, a convolutional layer or temporal adaptive DE normalization layer (TADE) may be used. This means that according to further embodiments, the decoder may comprise at least one temporal adaptive DE normalization residual block. According to embodiments, the decoder 20d comprises at least a styling element including a learnable element-wise modulation or a learnable affine transform (one of the key features of TADE and TADEresblock). For example, at least one of the styling elements is part of a weighting layer in a residual block of a neural network.
The proposed method achieves better quality of the reconstructed speech at very low bitrates and/or low complexity than comparable neural speech coders that do not leverage separate encoders for content and style. Especially the generalization ability of the trained codec may be significantly increased due to the proposed extra encoder. Following, e.g., the design of the neural speech codec NESC [1], both encoders can be realized in a computationally very efficient way such that an extra encoder does not pose significant extra computational burden.
Similar disentanglement ideas have been proposed for speech synthesis and low-rate speech coding [4], but latent fusion was achieved by simple concatenation rather than by the usage of a conditioning PreNet and an explicit style input to the decoder. Furthermore, [4] only relies on pretrained models, while the proposed method allows for end-to-end training, e.g., of the content encoder with the decoder and conditioning network or all network blocks together, which promises superior reconstruction quality.
As discussed above, embodiments of the present invention may be implemented as an apparatus, i.e., as decoder and encoder, or may be implemented as the corresponding methods for encoding and decoding.
Another embodiment refers to the training of the encoder and/or decoder or of the entities of the encoder and/or decoder. An embodiment provides a concept for training with at least two encoders, wherein each encoder is specialized to encode certain types of information, e.g., speaking style and content. This means that the training of the encoder 10s may be performed separately and/or specialized to the certain encoder type by the encoder 10s. Vice versa, the training of the encoder 10c may also be performed separately and/or specialized separately. For the training of the encoder 10s, speaking style information may be used, e.g., for different speaking styles, like a female or male speaking style, etc. For the training of the encoder 10c, content information may be used.
Another embodiment provides a method for decoding the data stream comprising Q1 and Q2. The decoding is based on at least two quantized latent data streams. Effective combination of these data streams by a styling and/or modulation, e.g., with the described PreNet.
Another embodiment provides a technique for avoiding style leakage into the content encoder. This may, for example, be achieved by differentiating the input audio signal, e.g., the speech or audio input audio signal, with respect to long-term and/or short-term information.
Another embodiment provides an implementation of a system, initialization techniques for being able to reproduce realistic speech with some default style information. This means that during an initial phase, the decoder 20d performs the decoding using initial/default style information as long as the decoder 20c does not provide the control signal C. As already discussed above, by selecting/using a default style information or random style information not being obtained based on Q1, the privacy of the speaker belonging to the input audio signal can be maintained, while the content is transmitted.
Although some aspects have been described in the context of an apparatus, it is clear that these aspects also represent a description of the corresponding method, where a block or device corresponds to a method step or a feature of a method step. Analogously, aspects described in the context of a method step also represent a description of a corresponding block or item or feature of a corresponding apparatus. Some or all of the method steps may be executed by (or using) a hardware apparatus, like for example, a microprocessor, a programmable computer or an electronic circuit. In some embodiments, some one or more of the most important method steps may be executed by such an apparatus.
The inventive encoded audio signal can be stored on a digital storage medium or can be transmitted on a transmission medium such as a wireless transmission medium or a wired transmission medium such as the Internet.
Depending on certain implementation requirements, embodiments of the invention can be implemented in hardware or in software. The implementation can be performed using a digital storage medium, for example a floppy disk, a DVD, a Blu-Ray, a CD, a ROM, a PROM, an EPROM, an EEPROM or a FLASH memory, having electronically readable control signals stored thereon, which cooperate (or are capable of cooperating) with a programmable computer system such that the respective method is performed. Therefore, the digital storage medium may be computer readable.
Some embodiments according to the invention comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.
Generally, embodiments of the present invention can be implemented as a computer program product with a program code, the program code being operative for performing one of the methods when the computer program product runs on a computer. The program code may for example be stored on a machine readable carrier.
Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.
In other words, an embodiment of the inventive method is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.
A further embodiment of the inventive methods is, therefore, a data carrier (or a digital storage medium, or a computer-readable medium) comprising, recorded thereon, the computer program for performing one of the methods described herein. The data carrier, the digital storage medium or the recorded medium are typically tangible and/or non-transitionary.
A further embodiment of the inventive method is, therefore, a data stream or a sequence of signals representing the computer program for performing one of the methods described herein. The data stream or the sequence of signals may for example be configured to be transferred via a data communication connection, for example via the Internet.
A further embodiment comprises a processing means, for example a computer, or a programmable logic device, configured to or adapted to perform one of the methods described herein.
A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.
A further embodiment according to the invention comprises an apparatus or a system configured to transfer (for example, electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may, for example, be a computer, a mobile device, a memory device or the like. The apparatus or system may, for example, comprise a file server for transferring the computer program to the receiver.
In some embodiments, a programmable logic device (for example a field programmable gate array) may be used to perform some or all of the functionalities of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor in order to perform one of the methods described herein. Generally, the methods are preferably performed by any hardware apparatus.
The above described embodiments are merely illustrative for the principles of the present invention. It is understood that modifications and variations of the arrangements and the details described herein will be apparent to others skilled in the art. It is the intent, therefore, to be limited only by the scope of the impending patent claims and not by the specific details presented by way of description and explanation of the embodiments herein.

References

[1] Pia, Nicola et al., "NESC: Robust Neural End-2-End Speech Coding with GANs", https://arxiv.org/abs/2207.03282
[2] Zeghidour, Neil et al. "SoundStream: An End-to-End Neural Audio Codec", https://arxiv.org/abs/2107.03312
[3] D6fossez Alexandre et al., "High Fidelity Neural Audio Compression", https://arxiv.org/abs/2210.13438
[4] Polyak et al., "Speech Resynthesis from Discrete Disentangled Self-Supervised Representations", 2021 https://arxiv.org/pdf/2104.00355.pdf

Claims

Decoder (20') for decoding an encoded input audio signal, the decoder comprising:
a conditioning entity (20c) receiving an encoded style information derived from the encoded input audio signal and configured to perform at least a learnable affine transform of at least one encoded content information derived from the encoded input audio signal to obtain a transformed encoded content information; and

a decoder (20d) configured to decode at least one transformed encoded content information further.
Decoder (20') according to claim 1, wherein the conditioning entity (20c) is configured to control the learnable affine transform of the encoded content information with a control signal or adapted control signal extracted from encoded style information.
Decoder (20') according to claim 2, wherein the encoded content information is transformed once with said control signal or a stream of encoded content information is transformed with the control signal or wherein one or multiple intermediate representations of the encoded content information are transformed with said control signal.
Decoder (20') according to one of the previous claims, wherein the decoder (20d) comprises at least one learnable layer.
Decoder (20') according to claim 4, wherein the decoder (20d) comprises at least one temporal adaptive DE-normalization residual block.
Decoder (20') according to claim 4 or 5, wherein the decoder (20d) comprises at least a styling element including a learnable element-wise modulation or a learnable affine transform.
Decoder (20') according to claim 4, 5, or 6, wherein at least one of the styling elements is part of a weighting layer in a residual block of a neural network.
Decoder (20') according to one of the previous claims, wherein the conditioning entity (20c) is configured to use a default style information or initial style information.
Decoder (20') according to claim 7 or 8, wherein the default control signal or initial control signal or the default style information or initial style information represents an anonymous style and/or average style and/or an average style for a male speaker and/or an average style for a female speaker; or
wherein the default control signal or initial control signal or the default style information or initial style information represents an style selected out of a plurality of styles, wherein the selection is taken based on an external information.
Decoder (20') according to one of the previous claims, wherein the first data stream (Q1), is transmitted less frequently or with longer transmission intervals than the second data stream.
Decoder (20') according to one of the previous claims, wherein the conditioning entity (20c) and/or the decoder (20d) use neural codecs like NESC or other network codecs for encoding.
Decoder (20') according to one of the previous claims, wherein the conditioning entity (20c) and/or the decoder (20d) are trained or trainable using machine learning techniques; and/or
wherein a layer of the conditioning entity (20c) and/or decoder (20d) comprises recurrent neural network blocks and/or Temporal Adaptive DE-normalization layers and/or Pseudo Quadrature Mirror Filterbanks.
Decoder (20') according to one of the previous claims, wherein a first data stream (Q1) of the encoded input audio signal comprises long term information, especially speaker identity information or speaking style information.
Decoder (20') according to one of the previous claims, wherein the encoded input audio signal (IS) is the speech signal; and/or
a second data stream (Q2) of the encoded input audio signal comprises short term information, especially phonetic content.
Method for decoding an encoded input audio signal, the method comprising the following steps:
receiving an encoded style information derived from the encoded input audio signal;

performing at least a learnable affine transform of at least one encoded content information derived from the encoded input audio signal to obtain a transformed encoded content information; and

further decoding the at least one transformed encoded content information using a decoder.
Computer program for performing, when running a computer one of the methods according to claim 15.