EP4600950A1

EP4600950A1 - Encoder and decoder

Info

Publication number: EP4600950A1
Application number: EP24156124.0A
Authority: EP
Inventors: Andreas BRENDEL; Kishan GUPTA; Nicola PIA; Guillaume Fuchs; Suraj Pandey; Markus Multrus
Original assignee: Fraunhofer Gesellschaft zur Foerderung der Angewandten Forschung eV
Current assignee: Fraunhofer Gesellschaft zur Foerderung der Angewandten Forschung eV
Priority date: 2024-02-06
Filing date: 2024-02-06
Publication date: 2025-08-13
Also published as: WO2025168600A1

Abstract

Encoder (10') for encoding an input audio signal (IS), comprising: a style encoder (10s) configured to encode style information of the input audio signal (IS) to obtain a first data stream (Q1); a content encoder (10c) configured to encode content information to obtain a second data stream (Q2).

Description

Embodiments of the present invention refer to an encoder for encoding the input audio signal and a decoder for decoding the encoded input audio signal. The coding of the encoder and the decoder enable that the input data is compressed. Further embodiments refer to the corresponding methods and to a computer program. Preferred embodiments are in a field of neural audio codec using speaker embeddings.
State-of-the-art (SOTA) neural speech coding achieves good quality of the reconstructed speech at bitrates as low as 3.2kbps. SOTA neural speech coders [1,2,3] are typically trained in an end-to-end fashion, where an input audio signal is encoded by an encoding neural network, the output of the encoder (the so-called latent) is quantized and the resulting quantized latent is decoded by a decoding neural network (see Fig. 1). During application, the encoder runs on transmitter side providing quantized latents (or rather identifying indices of the used quantized latents) which are transmitted to the receiver side where the decoder reproduces the input speech signal.
Schematic principle is illustrated by Fig.1.
Fig. 1 shows exemplarily a codec having an encoder 10 receiving an input audio signal IS so as to encode the input audio signal to obtain the encoded input audio signal Q. This encoded input audio signal Q can be decoded by use of the decoder 20. The decoder 20 is configured to provide based on the signal Q a reconstructed input audio signal IS'.
In this way, the submitted data stream Q must carry a long term information, like speaker identity, speaker style (called style in the following), etc., as well as short term information, like phonetic content (called content in the following). This way of coding has drawbacks with regard to the coding efficiency. Therefore, there is a need for an improved approach.
An objective of the present invention is to provide a concept for coding speech having high coding efficiency.
This objective is solved by the subject-matter of the independent claims.

Aspect Encoder

An embodiment provides an encoder for encoding an input audio signal like a speech signal. The encoder comprises a style encoder and a content encoder. The style encoder is configured to encode style information of the input audio signal to obtain a first (latent) data stream. The content encoder is configured to encode content information to obtain a second (latent) data stream.
Embodiments of the present invention are based on the finding that it is advantageous to disentangle content and style information. Background thereof is that the content information which may comprise a phonetic content mainly comprises short term information, while the style information which may comprise an information on the speaker identity or speaking style mainly carries a long term information. Starting from this finding, the encoder is subdivided into two parallel acting encoders, namely as style encoder and a content encoder. In this way, it is advantageously possible to increase the coding efficiency. Another advantage could lie in the transmission of the two data streams. Since information is disentangled and splits into two independent streams, they can be transmitted over different transmission channels (with different data rate, transmission intervals, and power consumption) and/or with unequal error protection (UEP).
According to an embodiment, the first and the second (latent) data stream are output and/or transmitted separately. For example, the first and second (latent) data stream may be quantized separately before transmitting. According to embodiments, the first data stream is quantized and/or transmitted and/or stored disjointly from the second data stream. Consequently, the two streams may be transmitted separately on different transmission channels.
As already mentioned above, the first (latent) data stream which is output by the style encoder comprises (mainly) long term information, especially speaker identity information or speaking style information. Furthermore, the second (latent) data stream (output by the content encoder) may comprise short term information, especially phonetic content. Due to this, it is advantageously possible not only to separate the encoding, but also the transmission, so that the two data streams can be encoded with separate entities. Due to the long term and short term character of the first and second (latent) data stream, the first (latent) data stream may comprise an information which is sent at much lower frequencies relative to the second (latent) data stream. For example, the first (latent) data stream may comprise data blocks that are sent at low frequency, e.g., below 10 times per second as the coded style information is slowly time varying. The second (latent) data stream may be composed of data blocks that have to be sent more frequently, e.g., 10-100 times per second, to account for the short-term nature of the coded signal content. This means that the content information is sent more frequently than the style information.
According to embodiments, the above-defined encoder is especially advantageous for speech coding. Therefore the input audio signal may be a speech signal or may comprise a speech record. For example, an input audio signal may be a speech signal of a person, the person being characterized by the first (latent) data stream with regard to a characteristic speech style, e.g. male speech style or female speech style or another speech style. Such speech style information is highly characteristic for a speech style and describes the speech behavior of a person whose speech should be coded very good, but general.
Below, possible implementations details for the encoders are given. According to embodiments, the style encoder and the content encoder use neural codecs like NESC or other neural network codecs for encoding. Such neural codecs are advantageously trained or trainable using machine learning techniques. Thus, according to embodiments, the style encoder and/or the content encoder are trained or trainable using machine learning techniques (artificial intelligence algorithms or neural network approaches). According to embodiments, the style encoder is configured to employ layers of a recurrent neural network or layers representing long-term information, like for example a dilated convolutional neural network or a transformer.
According to embodiments, the content encoder is configured to use normalization layers enforcing temporal whiteness or input perturbations or vocal tract length perturbations on the input audio signal waveform to emphasize short-term characteristics.
The disentanglement of content and style (in principle) also allows for separate manipulation of both representations and corresponding decoding, e.g., for speaker anonymization. Thus, according to embodiments, the encoder can just use the content encoder configured to encode content information to obtain a second data stream. In this way, the content can be transmitted without transmitting the identity sensitive style information.
On the decoder side, a default style information is used so that the content can be reproduced in an anonymized manner.
Expressed in other words: By manipulating the style information in an intended, user-defined way and transmitting the content information as it is, the same spoken content can be transmitted but the speaker identity is hidden, thereby preserving privacy of the conversation partner.

Aspect Decoder

Another embodiment provides a decoder for decoding an encoded input audio signal, like the input audio signal as encoded using the encoder as defined above. The decoder may comprise a conditioning entity and a content decoder. The conditioning entity receives a first (latent) data stream of the encoded input audio signal (cf. above discussion) and is configured to output a control signal based on the first (latent) data stream. The first (latent) data stream comprises an encoded style information. The content decoder is configured to decode the second (latent) data stream of the encoded input audio signal comprising content information (cf. above discussion). The content decoder is controlled and/or adapted by the control signal of the conditioning entity.
According to embodiments, the content decoder is configured to obtain a reconstructed signal of the encoded input audio signal based on the first and second (latent) data stream. According to embodiments, the reconstructed signal is output by the content decoder taking into account the control signal of the conditioning entity. This control information is based on the first (latent) data stream comprising the encoded style information.
In case the style encoder could not immediately provide the control signal, e.g. at the beginning of the decoding, the content decoder may be configured to use a default control signal or initial control signal before the conditioning entity provides the control signal based on the first data stream. For example, the default control signal may be stored in the decoder. The default control signal and/or the initial control signal represents an average style. According to embodiments, different default control signals or different initial control signals may be used. For example, an average style information for a male speaker and/or an average style information for a female speaker may be provided so as to start the decoding based on the default control signal/initial control signal. According to embodiments, the default control signal or initial control signal represents an average style selected out of the plurality of average styles, wherein the selection is taken based on an external information.
According to embodiments, the decoder may comprise a conditioning entity and a content decoder. The conditioning entity outputs a control signal being a default control signal which comprises a style information. The content decoder is configured to decode the data stream (comparable to the second data stream) of the encoded input audio signal comprising content information, wherein the content decoder is controlled and/or adapted by the control signal. By providing the default style, the reconstructed speech may be anonymized, i.e., the reconstructed speech signal does not allow for identifying the speaker as the style was changed (speaker privacy can be achieved according to this aspect).
According to embodiments, the default control signal representing a default style information may be selected out of a plurality of default styles. This may be done by use of a control signal.
Another embodiment provides a decoder for decoding an encoded input audio signal comprising a conditioning entity receiving the encoded input audio signal and configured to output a control signal based on the encoded input audio signal. The encoded input audio signal comprises an encoded style information. The decoder further comprises a content decoder configured to decode the encoded input audio signal further comprising content information, wherein the content decoder is controlled and/or adapted by the control signal. The content decoder may be configured to use a default control signal or an initial control signal before the conditioning entity provides the control signal based on external information or the first (latent) data stream.
According to embodiments, the default control signal or initial control signal represents a given style selected out of a plurality of available styles, wherein the selection is taken based on an external information or control. In this case, a deliberated anonymization or desired voice conversion could be achieved.
According to further embodiments, the decoder is configured to decode an encoded input audio signal comprising a conditioning entity and a content decoder. The conditioning entity is configured to output a control signal representing a style information. According to embodiments, the default control signal or initial control signal is used as control signal and thus, represents a given style selected out of one or more available styles. The content decoder is configured to decode the encoded input audio signal comprising content information, wherein the content decoder is controlled and/or adapted by the control signal.
Since a default control signal or initial control signal is used, an anonymization of the content is achieved.
According to embodiments, the default control signal or initial control signal represents, e.g., an average style and/or average style for a male speaker and/or average style for a female speaker. The default control signal or initial control signal may according to further embodiments represent an average style selected out of a plurality of (average) styles, wherein the selection is taken based on an external information. Note it could also be a set of other candidate styles not necessarily an average of other styles.

Additional Decoder Aspect

An embodiment provides a decoder for decoding an encoded input audio signal. The decoder comprises a conditioning entity and a content encoder. The conditioning entity is configured to receive an encoded style information derived from the encoded input audio signal and to perform at least a learned affine transform of an encoded content information derived from the encoded input audio signal to obtain a transformed encoded content information. The decoder is configured to further decode the transformed encoded content information.
Embodiments of the present invention are based on the finding that that it is advantageous to disentangle content and style information. Background thereof is that the content information which may comprise a phonetic content mainly comprises short term information, while the style information which may comprise an information on the speaker identity or speaking style mainly carries a long term information. Thus, the decoder comprises the two entities conditioning entity and the content decoder so as to enable the decoding of the two different information (content), namely the short-term information and the long-term information. The long-term information, also referred to as style information, is obtained using the conditioning entity while the short-term information, i.e., the content information, is decoded by the content decoder taking into account the previously processed long-term information/style information.
According to embodiments, the conditioning may be performed by a TADE layer (one of developed network building blocks based on StyleMelGAN and NESC as described below) or a conditioning network comprising one or several TADE layers.

Methods

Another embodiment provides a method for encoding an input audio signal:

encoding style information of the input audio signal to obtain a first (latent) data stream;
encoding content information to obtain a second (latent) data stream.

An embodiment provides a method for decoding an encoded input audio signal. The method comprising:

receiving a first (latent) data stream of the encoded input audio signal and outputting a control signal based on the first (latent) data stream, the first (latent) data stream comprising an encoded style information;
decoding a second (latent) data stream of the encoded input audio signal comprising content information, wherein the content decoder is controlled and/or adapted by the control signal.

Another embodiment provides a method for decoding an encoded input audio signal, and the method comprising:

receiving the encoded input audio signal and outputting a control signal based on the encoded input audio signal, the first (latent) data stream comprising an encoded style information;
using a default control signal or initial control signal before the conditioning entity provides the control signal based on the first (latent) data stream;
decoding the encoded input audio signal further comprising content information, wherein the content decoder is controlled and/or adapted by the control signal.

Regarding the default control signal or initial control signal it should be noted that providing a manipulated or a default control signal may be used not only to cover the time span before the first encoded style information is received but also for anonymizing the speaker (here, the style information may not be transmitted, thereby keeping the identifying information of the speaker included in the audio signal private).
An embodiment provides a method for decoding an encoded audio signal comprising a conditioning entity configured to output a control signal being a default control signal and representing a style information; a content decoder configured to decode a data stream of the encoded input audio signal comprising content information, wherein the content decoder is controlled and/or adapted by the control signal.
According to an embodiment, the method may further comprise setting a default/manipulated style on the decoder side and not transmitting the style information for not disclosing identifying information about the speaker in the transmitted signals (encoder side).
According to further embodiments, the above-discussed methods may be computer implemented. Therefore, embodiments refer to a computer program for performing, when running on a computer, one of the methods as defined above.
Below, embodiments of the present invention will subsequently be discussed referring to the enclosed figures, wherein:

Fig. 1: shows a schematic block diagram illustrating a codec according to a conventional approach;
Fig. 2: shows the schematic block diagram illustrating a codec according to embodiment;
Fig.3a: shows a schematic block diagram illustrating an encoder according to an embodiment; and
Fig. 3b: shows a schematic block diagram illustrating a decoder according to embodiments.

Below, embodiments of the present invention will subsequently be discussed referring to the herewith enclosed figures, wherein identical reference numeral are provided to objects having identical or similar function, so that description thereof is interchangeable and mutually applicable.
Starting from the above discussed conventional codec having the encoder 10 and the decoder 20 an enhanced codec having the encoder 10' and the decoder 20' will be discussed taking reference to Fig. 2.
The encoder 10' receives the input audio signal IS, e.g., an audio signal comprising speech. The encoder 10' performs encoding so as to obtain two output data streams Q1 and Q2. These two audio data streams Q1 and Q2 are transmitted, e.g., separately to the decoder 20'. The decoder 20' performs a reconstruction so as to obtain the reconstructed signal IS'. The encoder 10' comprising two encoding entities, namely a style encoder 10s and a content encoder 10c. Each of the two encoders 10s and 10c receives the input audio signal IS and performs an encoding of a portion of the input audio signal. The encoder 10s outputs the coded audio signal Q1, while the encoder 10c outputs the coded audio signal Q2.
On the decoder side, the signal Q1 is processed by a conditioning/PreNet 20c, while the encoded audio stream Q2 is decoded by the content decoder 20d. The content decoder 20d has an input for Q2 and a control signal generated by 20c. The decoder uses the two signals so as to output the reconstructed signal IS'.
Since now the structure of the codec according to an enhanced approach has been discussed, the functionality of the encoder 10' and the decoder 20' will be discussed separately taking reference to the entities 10s and 10c and 20c and 20d, respectively.
For the below discussion, references taken to Fig. 3a showing the encoder 10' comprising the style encoder 10s and the content encoder 10c.
The style codec 10s is configured to code style information of the input audio signal to obtain a first (latent) data stream Q1. The style information may comprise long term information, like a style information. For example, the style information may be dependent on the gender (male/female speakers) or dependent on the language. The style information may have an influence on the general frequency dependent power distribution of the voice and/or to the general sound. Due to the encoding, the signal Q2 comprises the specific coding style information.
The content encoder 10c is configured to encode content information, i.e., short term information, like phonetic information.
Due to the proposed concept of separately encoding content and style information performed by the separate encoders 10s and 10c, embodiments of present invention are more efficient than the SOTA approaches to neural speech coding [1, 2, 3].
To this end, we use (at least) two individual encoding networks, e.g., one for encoding style information and one for encoding content information. Both encoders obtain as input the speech signal that is to be transmitted. The outputs of these encoders, i.e., latents, are quantized separately and transmitted to receiver side. Here, the style latent is combined with the content latent for decoding with the joint decoding network (cf. Fig. 2).
According to embodiments, the encoder 10 (as well as the decoder 20) may be realized by neural codec NESC [1]. Below, possible implementations for a neural codec will be discussed: generally speaking, any neural network that transforms a potentially preprocessed audio signal into a compact representation (typically reducing the dimensionality of the signal representation) can serve as an encoder for a neural codec. This means that a neural codec may be characterized by a preprocessing of the audio signal and a transformation of the preprocessed audio signal so as to obtain a compact representation of same. Here, typically convolutional neural network blocks are used that downsample the input audio signal successively and recurrent neural network blocks or transformers that model additional temporal context. Part of this encoder would be also a subsequently applied quantizer transforming that continuous output of the neural network to a discrete representation that can be potentially entropy-coded and transmitted. Formally, also a formatter for preparing the data stream, etc., may be part of it.
NESC is an example for a neural codec. NESC may, for example, consist of the following blocks:
At the encoder side, the input waveform is transformed into a two-dimensional representation by a rolling window frontend (The frontend is the first part (receiving the input audio signal) of the encoder). The result is processed by a convolutional layer within each frame, recurrent layers along different frames and again a convolutional layer operating within each frame. Together, this is termed dual path convolutional recurrent neural network (DPCRNN). After the DPCRNN, the signal is further processed by several convolutional neural network blocks. The output of these blocks is quantized, e.g., by residual vector quantization (NESC) or by scalar quantization.
The decoder 20' as already shown by Fig. 2 will be discussed taking reference to Fig. 3b showing the decoder 20d as central component. The decoder 20d receives Q2 and outputs IS'. Furthermore, the decoder 20d has another input for the control signal c obtained using the condition PreNet 20c, also referred to as condition entity.
At the decoder side, the quantized representation may according to further embodiments be preprocessed by recurrent neural network blocks (so as to implement example neural codec based on NESC). The result is decoded by Streamwise-StyleMelGAN (SSMGAN). SSMGAN consists of a sequence of convolutional layers that subsequently upsample the input audio signal, where the input audio signal is used to condition each upsampling stage with a temporal adaptive DE-normalization (TADE) layer. The final signal is synthesized with a Pseudo-Quadrature Mirror Filter bank (PQMF).
According to embodiments, the neural codec used at the decoder side may comprise Streamwise-StyleMeIGAN, e.g., consisting of a sequence of convolutional layers that subsequently upsample the input audio signal. According to embodiments, the decoder may comprise a SSMGAN decoder including the TADE layers and the dequantization (mapping indices describing the discrete representation (or a stream of indices) to a latent waveform (representation) that serve as the input of the remaining part of the decoder. According to further embodiments, different network architectures may be used.
According to embodiments, the encoder comprises a style encoder and/or a content encoder which comprises the following elements:

a format definer configured to define a first multi-dimensional audio signal representation of the input audio signal (1), the first multi-dimensional audio signal representation of the input audio signal including at least:
- a first dimension [e.g. inter frame dimension], so that a plurality of mutually subsequent frames is ordered according to the first dimension; and
- a second dimension [e.g. intra frame dimension], so that a plurality of samples of at least one frame are ordered according to the second dimension,

According to embodiments, the content decoder comprises a styling element of the content information from the first data stream or a processed version of it, the styling element being controlled by the control signal (C). In doing this, a NESC decoder (SMGAN) involving the styling element can be used. According to further embodiments, the content decoder comprises at least a styling element to obtain a reconstructed signal (IS') of the encoded input audio signal based on the first (Q1) and second data stream (Q2). Here, styling elements/TADEs are used elsewhere in the decoder, e.g. using SSMGAN as content decoder. For example, at least one of the styling elements comprises an adaptive element-wise modulation achieved by at least an element-wise multiplication or addition, or wherein at least one of the styling elements comprises a learnable affine transform, or wherein at least one of the styling elements is part of a weight layer in a residual block of a neural network (= rough/high level definitions of TADE and TADEResBlock).
As discussed above, the content encoder 10c captures short time relations within the input audio signal (e.g., phonetic information), while the style encoder captures more global information (e.g., the identity of the speaker or the speaking style). Hence, the content encoder produces latents at a much higher frequency (e.g., 10-100Hz) relative to the style encoder (less than 10Hz). Expressed in other words this means that the frame rate used by the content encoder 10c is much higher when compared to the frame rate of the style encoder 10s. To be computationally efficient, both encoders may share some of their layers. According to embodiments, layers capable of representing long-term information such as RNNs may be employed to enforce that the style encoder 10s learns long-term information to avoid leaking of style information to the content encoder, techniques such as normalization layers and forcing temporal whiteness or input perturbations like vocal tract length perturbations may be used.
Consequently, the two latents Q1 and Q2 comprise different information. It should be noted that the latent Q1 (output of the style encoder) is transmitted less often as the latent Q2 (cf. output of the content encoder 10c). Background is that style information is a long-term and/or global feature, while content is referring to the short time information.
Below, the decoder 20' performing the decoding based on the two signals Q1 and Q2 will be discussed. The decoder 20 comprises the two entities 20c and 20d. 20c receives the latent Q1 comprising the style information and performs a processing so as to obtain a control signal. This entity 20c is also referred to as conditioning PreNet and is configured to prepare a control signal to specialize the decoder 20d towards a specific style. Consequently, the control signal output by the entity 20c serves as additional input for the decoder 20d. This can be achieved by learning affine transforms of intermediate representations in the decoder 20d. The decoder 20d receives the latent Q2 and performs a decoding so as to obtain the reconstructed signal IS'. The signal is obtained based on Q2 taking into account the control signal provided by 20c. Through the control signal the specific style information is received, so that the decoder 20d can perform the decoding not only based on the information included by Q2, but also in the information included by Q1.
As the decoder 20d makes use of a long-term style information, which may not be available yet in an initial phase of decoding, initialization techniques may be used according to embodiments. For example, decoding with an average style or with an average style for male/female speakers may be used during the initiation phase. This means in other words, that the decoder 20d uses an initial control signal representing the average style or an average style of a speaker female. To select the (average) style out of a plurality of (average) styles an information may be used, e.g. extracted or included by Q1.
When selecting an average style or just a predefined style out of the plurality of styles, a control signal can be provided imprinting an anonymous identity to the content decoded by use of the decoder 20d. In other words, this means that in case Q1 is not available, e.g., not transmitted, it is possible to decode the content in an anonymous manner, namely when selecting a style so as to obtain the control signal c for the content decoder 20d. In other words, this means that it is, according to another aspect, beneficial not to transmit Q1 but just Q2, so that the content can be reproduced, while the identity informations are not forwarded by the encoder 10'. Consequently, the decoder 20' which just uses a random style, enables to reproduce the content anonymously.
According to an embodiment, the conditioning entity 20c may be configured to receive the encoded style information derived from the encoded input audio signal, e.g., via the data stream Q1, and performs at least a learnable affine transform of the coded content information derived from the encoded input audio signal. The result is an obtained transformed encoded content information. This content information is forwarded as a kind of control signal C to the content decoder 20d which performs a further decoding. The further decoding may take into account the data stream Q2. Furthermore, the decoding performed by 20d has the purpose to obtain IS'. Note that according to embodiments, the data stream Q2 may suffice to be the input audio signal for 20c and 20d. Background thereof is that a long-term information can be derived from Q2, as well.
Regarding the content decoder 20d, it should be noted that same may comprise at least one learnable layer. For example, a convolutional layer or temporal adaptive DE-normalization layer (TADE) may be used. This means that according to further embodiments, the content decoder may comprise at least one temporal adaptive DE normalization residual block.
The proposed method achieves better quality of the reconstructed speech at very low bitrates and/or low complexity than comparable neural speech coders that do not leverage separate encoders for content and style. Especially the generalization ability of the trained codec may be significantly increased due to the proposed extra encoder. Following, e.g., the design of the neural speech codec NESC [1], both encoders can be realized in a computationally very efficient way such that an extra encoder does not pose significant extra computational burden.
Similar disentanglement ideas have been proposed for speech synthesis and low-rate speech coding [4], but latent fusion was achieved by simple concatenation rather than by the usage of a conditioning PreNet and an explicit style input to the decoder. Furthermore, [4] only relies on pretrained models, while the proposed method allows for end-to-end training, e.g., of the content encoder with the decoder and conditioning network or all network blocks together, which promises superior reconstruction quality.
As discussed above, embodiments of the present invention may be implemented as an apparatus, i.e., as decoder and encoder, or may be implemented as the corresponding methods for encoding and decoding.
Another embodiment refers to the training of the encoder and/or decoder or of the entities of the encoder and/or decoder. An embodiment provides a concept for training with at least two encoders, wherein each encoder is specialized to encode certain types of information, e.g., speaking style and content. This means that the training of the encoder 10s may be performed separately and/or specialized to the certain encoder type by the encoder 10s. Vice versa, the training of the encoder 10c may also be performed separately and/or specialized separately. For the training of the encoder 10s, speaking style information may be used, e.g., for different speaking styles, like a female or male speaking style, etc. For the training of the encoder 10c, content information may be used.
Another embodiment provides a method for decoding the data stream comprising Q1 and Q2. The decoding is based on at least two quantized latent data streams. Effective combination of these data streams by a styling and/or modulation, e.g., with the described PreNet.
Another embodiment provides a technique for avoiding style leakage into the content encoder. This may, for example, be achieved by differentiating the input audio signal, e.g., the speech or audio input audio signal, with respect to long-term and/or short-term information.
Another embodiment provides an implementation of a system, initialization techniques for being able to reproduce realistic speech with some default style information. This means that during an initial phase, the decoder 20d performs the decoding using initial/default style information as long as the decoder 20c does not provide the control signal C. As already discussed above, by selecting/using a default style information or random style information not being obtained based on Q1, the privacy of the speaker belonging to the input audio signal can be maintained, while the content is transmitted.
Although some aspects have been described in the context of an apparatus, it is clear that these aspects also represent a description of the corresponding method, where a block or device corresponds to a method step or a feature of a method step. Analogously, aspects described in the context of a method step also represent a description of a corresponding block or item or feature of a corresponding apparatus. Some or all of the method steps may be executed by (or using) a hardware apparatus, like for example, a microprocessor, a programmable computer or an electronic circuit. In some embodiments, some, one or more of the most important method steps may be executed by such an apparatus.
The inventive encoded audio signal can be stored on a digital storage medium or can be transmitted on a transmission medium such as a wireless transmission medium or a wired transmission medium such as the Internet.
Depending on certain implementation requirements, embodiments of the invention can be implemented in hardware or in software. The implementation can be performed using a digital storage medium, for example a floppy disk, a DVD, a Blu-Ray, a CD, a ROM, a PROM, an EPROM, an EEPROM or a FLASH memory, having electronically readable control signals stored thereon, which cooperate (or are capable of cooperating) with a programmable computer system such that the respective method is performed. Therefore, the digital storage medium may be computer readable.
Some embodiments according to the invention comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.
Generally, embodiments of the present invention can be implemented as a computer program product with a program code, the program code being operative for performing one of the methods when the computer program product runs on a computer. The program code may for example be stored on a machine readable carrier.
Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.
In other words, an embodiment of the inventive method is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.
A further embodiment of the inventive methods is, therefore, a data carrier (or a digital storage medium, or a computer-readable medium) comprising, recorded thereon, the computer program for performing one of the methods described herein. The data carrier, the digital storage medium or the recorded medium are typically tangible and/or non-transitionary.
A further embodiment of the inventive method is, therefore, a data stream or a sequence of signals representing the computer program for performing one of the methods described herein. The data stream or the sequence of signals may for example be configured to be transferred via a data communication connection, for example via the Internet.
A further embodiment comprises a processing means, for example a computer, or a programmable logic device, configured to or adapted to perform one of the methods described herein.
A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.
A further embodiment according to the invention comprises an apparatus or a system configured to transfer (for example, electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may, for example, be a computer, a mobile device, a memory device or the like. The apparatus or system may, for example, comprise a file server for transferring the computer program to the receiver.
In some embodiments, a programmable logic device (for example a field programmable gate array) may be used to perform some or all of the functionalities of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor in order to perform one of the methods described herein. Generally, the methods are preferably performed by any hardware apparatus.
The above described embodiments are merely illustrative for the principles of the present invention. It is understood that modifications and variations of the arrangements and the details described herein will be apparent to others skilled in the art. It is the intent, therefore, to be limited only by the scope of the impending patent claims and not by the specific details presented by way of description and explanation of the embodiments herein.

References

[1] Pia, Nicola et al., "NESC: Robust Neural End-2-End Speech Coding with GANs", https://arxiv.org/abs/2207.03282
[2] Zeghidour, Neil et al. "SoundStream: An End-to-End Neural Audio Codec", https://arxiv.org/abs/2107.03312
[3] D6fossez Alexandre et al., "High Fidelity Neural Audio Compression", https://arxiv.org/abs/2210.13438
[4] Polyak et al., "Speech Resynthesis from Discrete Disentangled Self-Supervised Representations", 2021 https://arxiv.org/pdf/2104.00355.pdf

Claims

Encoder (10') for encoding the input audio signal (IS), comprising:
a style encoder (10s) configured to encode style information of the input audio signal (IS) to obtain a first data stream (Q1);

a content encoder (10c) configured to encode content information to obtain a second data stream (Q2).
Encoder (10') according to claim 1, wherein the first and the second data stream (Q1; Q2) are output and/or transmitted separately: and/or
wherein the first and second data stream (Q1; Q2) are quantized separately.
Encoder (10') according to one of the previous claims, wherein the first data stream (Q1) comprises long term information, especially speaker identity information or speaking style information; and/or
wherein the second data stream (Q2) comprises short term information, especially phonetic content.
Encoder (10') according to one of the previous claims, wherein the first data stream (Q1), is transmitted less frequently or with longer transmission intervals than the second data stream (Q2).
Encoder (10') according to one of the previous claims, wherein the first data stream (Q1), is quantized or transmitted or stored disjointly from the second data stream (Q2).
Encoder (10') according to one of the previous claims, wherein the input audio signal (IS) is the speech signal; or
wherein the input audio signal (IS) is a speech signal of a person, the person being characterized by the first data stream (Q1) with regard to a characteristic speech style, e.g. male speech style or another speech style.
Encoder (10') according to one of the previous claims, wherein the style encoder (10s) and/or the content encoder (10c) use neural codecs like NESC or other network codecs for encoding.
Encoder (10') according to one of the previous claims, wherein the style encoder (10s) and/or the content encoder (10c) are trained or trainable using machine learning techniques; and/or
wherein a layer of the style encoder and/or content encoder comprise convolutional layers within each frame and/or recurrent layers along different frames and/or a convolutional layer operating with each frame.
Encoder (10') according to one of the previous claims, wherein the style encoder (10s) is configured to employ layers of a recurrent neural network or layers representing long term information; and/or
wherein the style encoder and/or content encoder comprises a dual path convolutional recurrent neural network and/or a convolutional network block and/or a residual vector quantizer or scalar quantizer and/or configured for rolling window frontend .
Encoder (10') according to one of the previous claims, wherein the style encoder (10s) or the content encoder (10c) comprises:
a format definer configured to define a first multi-dimensional audio signal representation of the input audio signal (IS), the first multi-dimensional audio signal representation of the input audio signal including at least:
a first dimension so that a plurality of mutually subsequent frames is ordered according to the first dimension; and

a second dimension so that a plurality of samples of at least one frame are ordered according to the second dimension,

and at least one learnable layer configured to process the first multidimensional audio signal representation of the input audio signal (IS), or processed version of the first multi-dimensional audio signal representation.
Encoder (10') according to one of the previous claims, wherein the content encoder (10c) is configured to use normalization layers enforcing temporal whiteness or input perturbations or vocal tract length perturbations on short term signal characteristics.
Decoder (20') for decoding an encoded input audio signal, comprising:
a conditioning entity (20c) receiving a first data stream (Q1) of the encoded input audio signal and configured to output a control signal (C) based on the first data stream (Q1), the first data stream (Q1) comprising an encoded style information;

a content decoder (20d) configured to decode a second data stream (Q2) of the encoded input audio signal comprising content information, wherein the content decoder (20d) is controlled and/or adapted by the control signal (C).
Decoder (20') according to claim 12, wherein the content decoder (20d) is configured to obtain a reconstructed signal (IS') of the encoded input audio signal based on the first (Q1) and second data stream (Q2).
Decoder according to claim 12 or 13, wherein the content decoder (20d) is configured to use a default control signal or initial control signal before the conditioning entity provides the control signal (C) based on the first data stream (Q1).
Decoder (20') according to claim 14, wherein the default control signal or initial control signal represents an average style and/or an average style for a male speaker and/or an average style for a female speaker; or
wherein the default control signal or initial control signal represents an average style selected out of a plurality of average styles, wherein the selection is taken based on an external information, or

wherein the default control signal or initial control signal represents a given style selected out of a plurality of available styles, wherein the selection is taken based on an external information or control.
Decoder (20') according to claims from 12 to 15, wherein the first data stream (Q1), is transmitted less frequently or with longer transmission intervals than the second data stream (Q2).
Decoder according to one of claims 12 to 16, wherein the content decoder comprises an entity performing decoding by streamwise style mel GAN and/or a decoder and/or a decoder including a temporal adaptive DE normalization layer and/or a decoder comprising dequantizer.
Decoder according to one of claims 12 to 17, wherein the content decoder comprises a styling element of the content information from the first data stream or a processed version of it, the styling element being controlled by the control signal (C).
Decoder according to one of claims 12 to 18, wherein the content decoder comprises at least a styling element to obtain a reconstructed signal (IS') of the encoded input audio signal based on the first (Q1) and second data stream (Q2).
Decoder according to claim 18 or 19, wherein at least one of the styling elements comprises an adaptive element-wise modulation achieved by at least an element-wise multiplication or addition, or wherein at least one of the styling elements comprises a learnable affine transform, or wherein at least one of the styling elements is part of a weight layer in a residual block of a neural network.
Method for encoding an input audio signal (IS), a method comprises the steps:
encoding style information of the input audio signal to obtain a first data stream (Q1);

encoding content information to obtain a second data stream (Q2).
Method for decoding an encoded input audio signal, the method comprising the following steps:
receiving a first data stream (Q1) of the encoded input audio signal and outputting a control signal based on the first data stream (Q1), the first data stream (Q1) comprising an encoded style information;

decoding a second data stream (Q2) of the encoded input audio signal comprising content information, wherein the content decoder is controlled and/or adapted by the control signal.
Computer program for performing, when running a computer one of the methods according to claim 21 or 22.