FR3136884A1 - Ultra-low bit rate audio compression - Google Patents

Abstract

Very low bit rate audio compression. The invention relates to a method and a device for generating textual data representative of an audio signal comprising at least one voice segment. It comprises, on a terminal (DE), the following steps: - identify in the audio signal (S1) at least one voice segment containing at least one voice information; - extract (S4) from the voice segment at least one voice context data (LEM), representative of an emotion associated with said vocal information; - prepare (S6) a textual description of said vocal segment, comprising at least said vocal information and said vocal context data. fig. 4

Description

Ultra-low bitrate audio compression Field of the invention

The invention relates generally to telecommunications, and more specifically to communications involving the transmission of audio streams comprising voice information. It applies more particularly to terminals equipped with physical and software resources including a microprocessor and a speech recognition and synthesis module.

Prior Art

Voice compression generally uses traditional audio compression techniques, e.g. MPEG3, AAC, etc. These techniques are capable of compressing audio signals, including speech, effectively. However, they take little account of the content of the signal, in particular the presence of speech information and the emotions associated with such speech, which allow a faithful reconstruction of the speech signal.

Recently, new techniques have appeared in this field for associating emotions with a speech signal. They are known as “Emotional Speech Synthesis”, later abbreviated to “ESS”. For example, patent application US 2022/0122580 A1 describes a system capable of associating an emotion with a human voice and of synthesizing it so as to reflect this emotion.

However, none of these techniques address the coding of such voice and emotion information, and the transmission of such information in a very compact manner in order to transmit it over a very low bandwidth network between a transmitter and a receiver. .

An additional difficulty arises when such a speech signal is accompanied by sound signals such as music, background noise, etc.

There are therefore needs to encode and transmit an audio signal containing speech at very low bit rate.

The invention improves the state of the art.

To this end, it proposes a method for generating textual data representative of an audio signal comprising at least one voice segment, the method being characterized in that it comprises, on a terminal called a transmitter terminal, the following steps:
- identify in the audio signal said at least one voice segment containing at least one voice information;
- extract from the vocal segment at least one piece of vocal context data, representative of an emotion associated with the vocal information;
- prepare a textual description of said voice segment, comprising at least said voice information and said voice context data;

Advantageously according to the invention, a textual description of the audio signal containing speech is carried out by taking into account emotion information associated with the vocal context. Speech and emotion information are represented and optionally encoded as text. It is thus possible to store or transmit this textual description at very low speed, since the representation of a text, or sequence of ASCII characters, occupies, in a known manner, very little space compared to audio data, even compressed, on a storage medium or on a transmission channel.

By “audio signal” we mean the representation of any sound, in analog or digital form, for a certain duration.

By “speech segment” we mean a portion of the audio signal which contains speech information. Such a segment can for example correspond to a fixed duration, or to a range of speech ending in silence, etc. It can include a syllable, a word, a sentence or a set of sentences pronounced by one or more speakers. It may also contain, in addition, other sound information which is not considered as speech (music, song, ambient noise, etc.)

By “emotion”, we mean a state of the speaker which translates into variation in the characteristics of the audio signal which carries his voice, for example anger, joy, etc.

By “textual description”, we mean a description of the audio signal in text form, that is to say a series of characters (letters, numbers, special characters, symbols, etc.) which can be represented for example by ASCII codes.

Other non-specifically emotional context data (such as voice intensity) may also be part of the textual description.

According to a particular mode of implementation of the invention, the method as described above further comprises the steps of extracting from said at least one vocal segment at least one piece of sound context data, representative of the sound content of the vocal segment , and in which said textual description further comprises said sound context data.

Advantageously according to this mode, the voice segment further comprises a sound context, with which sound context data is associated. This sound context data is also represented in textual form. It may be a description of the sound content (for example a song or music title) or an address of the sound content, said address being represented in text form (for example an http address). Thus, advantageously, the sound content also takes up very little space on the transmission channel.

According to a particular mode of implementation of the invention, the audio signal further comprises at least one non-vocal segment, and the method as described above further comprises the following steps:
- identify said at least one non-vocal segment comprising at least one sound context data, representative of the non-vocal content;
- prepare a textual description of said non-vocal segment, comprising at least said sound context data.

Advantageously, according to this mode, the signal includes non-vocal segments in addition to the vocal segments. By “non-voice segment” is meant a portion of the audio signal which does not contain speech information within the meaning of this description. Such a segment can for example correspond to a fixed duration, or to a range of music ending in silence, etc. It is also prepared before transmission in the form of textual data. As explained before, it can be a description of the sound content (such as a song title) or an address of the sound content, said address being represented in text form (for example an http address). Thus advantageously the audio signal is represented in the form of a succession of textual representations of vocal and non-vocal segments, which take up very little space on the disk or the transmission channel.

According to a particular mode of implementation of the invention, the textual description is a set of data in XML or JSON format.

Advantageously, in this mode, a standardized text format is used to describe the vocal and non-vocal segments.

The XML language (from the English “eXtended Markup Language”) includes a list of data in the form of fragments classically described between an opening tag (< >) and a closing tag </ >. It has many advantages, including being easy to read by both a person and a machine. XML is a standard, and it is structured (the structure of an XML document is defined and can be validated by a schema), hierarchical, etc.

The JSON (JavaScript Object Notation) language is a lightweight language for exchanging textual data. It also has many advantages. It also offers greater compactness than XML.

According to a particular mode of implementation of the invention, the vocal context data is generated by a classification module.

Advantageously according to this mode, a classification module such as for example a neural network can be trained to recognize the emotions conveyed by the voice of a speaker, and to subsequently carry out automatic recognition. So when a vocal segment is presented to the classifier, it automatically detects associated emotion data.

According to a particular mode of implementation of the invention, the textual description is transmitted over a network.

Advantageously according to this mode, the textual description comprising only text is very well adapted to a transmission network since, occupying very little space, it limits the load of the network. This is particularly important in the case of a very low bandwidth network, for example a LORA technology network, making it possible to structure an extended network at low consumption and low cost.

According to one variant, the transmission is carried out in the form of an SMS message.

Advantageously, according to this mode, the vocal and non-vocal segments, in other words the entire text stream, can be transmitted directly on a radio channel in the form of SMS, which are text messages. This offers numerous advantages, including that of offloading a mobile network between two devices, since the voice of the interlocutors uses the SMS channel with a very low speed and no longer the traditional voice channel.

SMS (Short Message Service) means a text sent from a device such as a mobile phone to another device. The terminologies of small messages or text messages are also used. The SMS messaging service allows you to transmit short text messages. It is one of the mobile telephony services. It was introduced by the GSM standard.

According to a particular mode of implementation of the invention, the method further comprises a learning phase, comprising the steps of:
- receive training audio segments comprising at least one vocal information;
- receive labeling data comprising at least one vocal context data representative of an emotion associated with the vocal information of the segment;
- adapt a voice context detection system based on the received training audio segments and labeling data.

Advantageously, according to this mode, a classifier is trained prior to the request to extract an emotion from an audio segment. The classifier, which can be for example a neural network, is thus well trained on the voice of the speaker.

Correlatively, the invention also proposes a method for generating an audio signal from a textual description comprising at least one vocal segment, comprising the steps of:
- extract from the textual description of the vocal segment at least one vocal information and one vocal context data representative of an emotion associated with the vocal information;
- synthesize the voice segment from at least said voice information and voice context data.

According to one embodiment, the method of generating an audio signal further comprises a step of extracting from the textual description of the vocal segment at least one sound context data representative of the sound content of the vocal segment, and in which synthesizing the voice segment also uses sound context data.

According to a hardware aspect, the invention also relates to a device for generating textual data representative of an audio signal comprising at least one vocal segment, the device comprising a memory and a processor configured to:
- identify in the audio signal at least one voice segment containing at least one voice information;
- extract from the vocal segment at least one piece of vocal context data, representative of an emotion associated with the vocal information;
- prepare a textual description of said voice segment, comprising at least said voice information and said voice context data.

According to a hardware aspect, the invention also relates to a device for generating an audio signal from a textual description comprising at least one vocal segment, comprising a memory and a processor configured to:
- extract from the textual description of the vocal segment at least one vocal information and one vocal context data representative of an emotion associated with the vocal information;
- synthesize the voice segment from at least said voice information and voice context data.

The invention also relates to a system comprising a device for generating textual data representative of an audio signal and a device for generating an audio signal from a textual description as described previously, the system being characterized in that the textual data representative of the audio signal is transmitted over a network between the device for generating textual data and the device for generating an audio signal.

The invention also relates to a computer program comprising instructions for implementing one of the above methods according to any of the particular embodiments described above, when said program is executed by a processor. The method can be implemented in various ways, notably in hardwired form or in software form. This program may use any programming language, and be in the form of source code, object code, or intermediate code between source code and object code, such as in a partially compiled form, or in any other desirable shape.

The invention also relates to a recording medium or information medium readable by a computer, and comprising instructions for a computer program as mentioned above. The recording media mentioned above can be any entity or device capable of storing the program. For example, the support may comprise a storage means, such as a ROM, for example a CD-ROM or a microelectronic circuit ROM, or even a magnetic recording means, for example a hard disk. On the other hand, the recording media may correspond to a transmissible medium such as an electrical or optical signal, which may be conveyed via an electrical or optical cable, by radio or by other means. The programs according to the invention can in particular be downloaded on an Internet type network.

Alternatively, the recording media may correspond to an integrated circuit in which the program is incorporated, the circuit being adapted to execute or to be used in the execution of the method in question.

These devices and these computer programs have characteristics and advantages similar to those described above in relation to the corresponding methods.

List of Figures

Other characteristics and advantages of the invention will appear more clearly on reading the following description of particular embodiments, given as simple illustrative and non-limiting examples, and the appended drawings, among which:

There illustrates an example of an environment for implementing the invention according to a particular embodiment;

There illustrates an architecture of a textual data generator device (or “encoder”) according to embodiments of the invention;

There illustrates an architecture of a DR audio signal generator device (or “decoder”) according to embodiments of the invention;

There illustrates steps of the method for generating textual data according to one embodiment of the invention;

There illustrates steps of the audio signal generation method according to one embodiment of the invention;

There schematically represents an example of an emotion prediction device for implementing the method of according to one embodiment;

There represents, in flowchart form, the main steps of a method of training an artificial neural network for such an emotion prediction device;

There represents in schematic form an emotion database according to one embodiment of the invention.

Description of an embodiment of the invention

General principle of the invention

The general principle of the invention consists of transmitting, instead of a portion of an audio stream comprising a speech signal, which would be compressed by a conventional technique, a textual representation of the content of this signal, for example in the form of ASCII characters, including in particular the user's voice, as well as associated emotions, and optionally information relating to a portion of non-voice audio signal.

Particular modes of carrying out the invention.

There illustrates an example of an environment for implementing the invention according to a particular embodiment.

According to this embodiment, a transmitter device, denoted DE, captures, analyzes and transmits the voice of a user to a receiver device, denoted DR, through a RES network. Optionally, the transmitted video stream may include, in addition to the user's voice, audio sound data, or sounds, such as background noise (vehicle traffic, birdsong, etc.) or music.

According to the embodiment presented, the transmitter device is a mobile terminal (smartphone). According to other embodiments, it can take the form of another terminal, for example a PC, a tablet, etc.

According to the embodiment presented, the receiving device is a portable computer (PC). According to other embodiments, it can take the form of another terminal, for example a smartphone, a tablet, etc.

The RES network can conventionally be a mobile type network (for example UMTS, 4G, 5G, etc.), Internet, or other. It can be local or extended.

At the transmitter terminal, responsible for generating the flow, the user's voice is captured. This involves identifying the user’s voice, but also their emotions, their tone, etc. The incoming stream is divided into voiced or non-voice segments. A speech to text conversion engine (in English, “Speech To Text”, subsequently abbreviated to “STT”) is used, as well as an emotion analysis engine. Then the terminal transmits the audio stream, which can contain only speech, or optionally a mixture of voice and traditional audio. Voice segments are encoded as text. The associated emotion information is encoded in the form of specific data, called voice context data, reflecting the emotions associated with the speech segment. This context data is also represented in text form. Optionally, traditional data corresponding to non-vocal segments (such as music, ambient noise, etc.), juxtaposed or superimposed on speech, are also encoded in text form.

At the receiving terminal, the signal is received and analyzed, then a voice segment is synthesized from the received text and a model capable of reconstructing the voice of the interlocutor with the emotions of the original signal. Such methods exist and are known as " Emotional Speech Synthesis", later abbreviated to "ESS". Non-speech sound information can be added to the synthesized signal

There illustrates an architecture of a textual data generator device (or “encoder”) according to embodiments of the invention.

The device DE has the classic architecture of a computer and notably comprises a memory MEM, a processing unit equipped for example with a processor PROC, and controlled by the computer program PGM stored in memory MEM. At initialization, the code instructions of the PGM computer program are, for example, loaded into a memory before being executed by the PROC processor. The processor PROC of the processing unit implements in particular the steps of the method for generating textual data according to any one of the particular embodiments described in relation to Figures 4 and 6-8, according to the instructions of the program d PGM computer.

• Un module d’acquisition de signaux audio (AUDIO IN) apte à séparer une entrée audio en segments vocaux (contenant de la parole) et en segments non vocaux (son d’ambiance, musique, etc.).
• Un module de traitement de segments vocaux (VOICE) travaillant en association avec un module STT de conversion de parole en texte ;
• Un module EMOT de traitement des émotions de l’utilisateur sur un segment vocal, apte à générer un ou plusieurs labels d’émotion à partir du segment vocal ;
• Un module ENTR d’apprentissage des émotions, par exemple un réseau de neurones, apte à apprendre à associer une émotion à un segment vocal ; le module EMOT correspond à la mise en œuvre de ce module, une fois l’apprentissage terminé.
• Un module de sortie de signaux audio sous forme de texte (TEXT OUT) comportant des données de parole, des données de contexte vocal (représentatives d’une émotion), et optionnellement des données de contexte sonore.
• Un module SOUND de traitement des informations sonores ;
• Une base de données BD comportant par exemple un index, ou un label, associé à chaque émotion (par exemple, « 1 » pour joie, « 2 » pour colère, etc.) connue. Optionnellement, elle peut aussi comporter des caractéristiques prosodiques des signaux associés à une émotion particulière (variations de fréquence, d’intensité, de prononciation, etc. liées à cette émotion).

The DE system also includes:

• An audio signal acquisition module (AUDIO IN) capable of separating an audio input into vocal segments (containing speech) and non-vocal segments (ambient sound, music, etc.).
• A voice segment processing module (VOICE) working in association with an STT speech-to-text conversion module;
• An EMOT module for processing the user's emotions on a vocal segment, capable of generating one or more emotion labels from the vocal segment;
• An ENTR module for learning emotions, for example a neural network, capable of learning to associate an emotion with a vocal segment; the EMOT module corresponds to the implementation of this module, once the learning is completed.
• An audio signal output module in text form (TEXT OUT) comprising speech data, vocal context data (representative of an emotion), and optionally sound context data.
• A SOUND module for processing sound information;
• A BD database comprising, for example, an index, or a label, associated with each known emotion (for example, “1” for joy, “2” for anger, etc.). Optionally, it can also include prosodic characteristics of the signals associated with a particular emotion (variations in frequency, intensity, pronunciation, etc. linked to this emotion).

There illustrates an architecture of a DR device generating an audio signal from textual data (or “decoder”) according to embodiments of the invention.

The device DR has the classic architecture of a computer and notably comprises a memory MEM', a processing unit UT', equipped for example with a processor PROC', and controlled by the computer program PGM' stored in memory SAME'. At initialization, the code instructions of the computer program PGM’ are for example loaded into a memory before being executed by the processor PROC’. The processor PROC' of the processing unit UT' notably implements the steps of the method for generating an audio signal from a textual description according to any one of the particular embodiments described in relation to the figures 5 or 8, according to the instructions of the PGM' computer program.

• Un module d’acquisition de signaux audio textuels (TEXT IN) apte à séparer une entrée audio textuelle en segments vocaux (comportant de la parole et optionnellement des informations non vocales) et en segments non vocaux (son d’ambiance, musique, etc.).
• Un module de traitement de segments vocaux (VOICE’) travaillant en association avec un module TTS de conversion de texte en parole ;
• Un module EMOT’ de traitement des émotions de l’utilisateur sur un segment vocal, apte à modifier la restitution audio d’un segment vocal par la prise en compte des labels d’émotion associés au segment vocal ;
• Une base de données d’émotions BD’, comprenant des données pour le module EMOT’, par exemple des paramètres prosodiques caractéristiques des signaux associés à une émotion particulière (variations de fréquence, d’intensité, de prononciation, etc.), ou un pointeur sur un réseau de neurones spécifique.
• Un module SOUND’ de traitement des informations sonores ;
• Un module de sortie de signaux audio sous forme audio (AUDIO OUT) apte à restituer des segments audio de parole associés à des émotions et optionnellement des segments audio non vocaux.

The DR system also includes:

• A textual audio signal acquisition module (TEXT IN) capable of separating a textual audio input into vocal segments (comprising speech and optionally non-vocal information) and non-vocal segments (ambient sound, music, etc. ).
• A voice segment processing module (VOICE') working in association with a TTS text-to-speech conversion module;
• An EMOT' module for processing the user's emotions on a vocal segment, capable of modifying the audio restitution of a vocal segment by taking into account the emotion labels associated with the vocal segment;
• An emotion database BD', including data for the EMOT' module, for example prosodic parameters characteristic of signals associated with a particular emotion (variations in frequency, intensity, pronunciation, etc.), or a pointer to a specific neural network.
• A SOUND' module for processing sound information;
• An audio signal output module in audio form (AUDIO OUT) capable of restoring speech audio segments associated with emotions and optionally non-vocal audio segments.

There illustrates steps of the method for generating textual data according to one embodiment of the invention.

The method according to this embodiment receives voice input in the form of an audio stream; it separates it into voice audio segments and non-voice audio segments. The voice segments are processed to provide a textual representation composed of character strings corresponding to the user's words and associated emotion data. In this way, it encodes a voice adapted to the emotions of labeling data. The speech segments may additionally include sound (non-speech) information. Non-vocal information (music, background noise, ambient noise, etc.) can also be associated with a textual description.

During an initial S0 step, all the modules necessary for implementing the method are initialized, in particular the training of the EMOT module if it is produced in the form of a neural network. , or the appropriate population of a base of emotions associated with recordings, etc.

In a step S1, an audio stream is acquired by the audio acquisition module (AUDIO IN of the ). This audio stream is broken down into different segments corresponding to vocal segments comprising speech (SVC) or non-vocal segments not containing speech to be processed within the meaning of the invention (SNVC). It should be noted that a song can be non-vocal information within the meaning of the invention. There are different methods well known to those skilled in the art for extracting speech segments from an audio signal while ignoring noise, music, silence, etc. For example, it is possible to use over a time window (of a given duration, for example 10 seconds) thresholding on the energy of the signal or other acoustic parameters, segmentation using models of Hidden Markov, or neural networks, etc. According to another example, the segment stops as soon as 3 seconds of silence are exceeded and a new segment begins.

Generally speaking, a speech segment corresponds to acoustic properties linked to the mechanism of the speaker's vocal production. In particular, speech is characterized by a formant and non-stationary structure which reflects the resonance of the vocal tract. The alternation of voiced, unvoiced and silence sounds also allows it to distinguish itself from the properties of other sounds. A non-vocal segment, on the other hand, includes sounds with their own characteristics, such as silence, noise, music, etc. Music, for example, is characterized by a harmonic and stationary structure, a repetitive rhythm, an absence of silence.

These notions can naturally be expanded or reduced without loss of generality: we can retain or not the non-vocal segments of silence, or music, etc. Likewise, a vocal segment may contain music superimposed on speech, or ambient noise (bird songs, mower noises, etc.), which we may choose to process (i.e. to include in the textual description) or not.

During an optional step S2, non-voice information is extracted from a voice or non-voice segment. According to one embodiment, a segment can in fact be considered vocal (since it includes speech) but also include sound. In this case, step S2 can be applied to this segment. In one embodiment, a voice segment has no sound, step S2 is skipped. In one embodiment, only the non-vocal sound segments are processed in step S2. In one embodiment, the sound is not processed, that is to say step S2 is not implemented. When step S2 is implemented, it results in the extraction of information, or data, from sound context, associated with the segment, for example the description of a background noise (bird songs) or the address of a sound recording. Step S3 processes its textual representation. Steps S4, S5, however, concern the speech information (VOICE) of the vocal segments, with or without its associated sound.

In a step S4, the encoder submits the vocal segment to an emotion detection module (EMOT module of the ). For example, it uses an artificial neural network for this purpose, as shown in the figure. , previously trained on a corpus of vocal segments, as shown in support of the . In this case, the data is presented at the input of the neural network previously trained according to an expected format, for example a normalized vector of input data from the neural network, which is called “input data” DEi . This data is of the same nature as that used for training.

According to another embodiment, a simple signal analysis technique, for example the analysis of fundamental frequency (pitch), intensity, flow rate, articulation, etc. of speech, in other words prosody, can be considered: a given vocal segment is compared to the characteristics which are recorded in the BD database, and a decision is made on the nature of the emotion conveyed.

According to another embodiment, the vocal segment includes a key word making it possible to deduce the emotion label (the speaker says: "I am sad and angry"; two emotion labels corresponding to "sadness" and "are deduced"). annoyance").

At the end of this step, one or more LEMi emotion labels are assigned to the vocal segment, such as 1 for “joy”, N for “anger”, etc. The emotion label corresponds to vocal context data.

Note that “emotion” is to be taken here in the broad sense. It can be characteristic of the intensity of the speech segment, the speed of speech, etc. More generally, this is context data representative of the vocal content.

In a step S5, the speech segment is converted into text using a speech to text conversion module (STT module of the ). Such a module is well known to those skilled in the art and will not be described further. For example, we can use DeepSpeech software for this purpose, a speech recognition engine from the company Mozilla©.

Steps S4 and S5 can be parallel, as shown in the , or successive. In particular, we can consider “riding” the speech segment of its emotional component, before transforming the speech into text, for better speech recognition.

At the end of steps S4 and S5, the speech segment is prepared for coding, during a step S6 . According to one embodiment, the preparation consists of generating a text in XML language form. According to one example, a known markup language is used which provides a standard means of annotating text for the generation of synthetic speeches, proposed by the W3C organization, named SSML ( Speech Synthesis Markup Language (SSML) Version 1.1 - W3C Recommendation 7 September 2010 ), available at https://www.w3.org/TR/speech-synthesis11/.

An example of the use of such a language, applied to the Alexa© voice synthesizer from the Amazon company, is reproduced below:

<speak>

<amazon:emotion name="sadness" intensity="medium">

I want to tell you a secret.

</amazon:emotion>

<voice name="Kendra"> I am not a real human </voice>

<amazon:emotion name="anger" intensity="high"> No way!</amazon:emotion>

</speak>

In the example above, the emotion label “sadness”, or vocal context data, is associated with the text that follows it (“I want to tell you a secret”) and allows this emotion to be reflected during restitution. sound of the segment by the decoder. The “medium” label is another piece of vocal context data, not specifically associated with an emotion, which makes it possible to adjust the sound level during the sound reproduction of the segment.

During an optional step S3, a textual representation of a sound (non-vocal) is generated. An example of representation of sound information is provided below. The “src” tag gives an address in the form of a URI (Uniform Resource Identifier), in http format, of an audio stream comprising, in this example, a musical sound recording and a recording of bird songs. The “descr” tag provides a description that can correspond to alternative content in the case where the source cannot be played (synthesis of birdsong, or “the birds are singing” type utterance). The “clipBegin” tag indicates the moment of insertion into the segment. Non-vocal sound can be inserted over the vocals from the beginning of the segment, or later. It can also have an insertion end time (clipEnd), a sound volume to play the sound (soundLevel), etc. We can refer to the aforementioned W3C specification which offers a certain number of possible parameters. Naturally, the list specified in this specification is not exhaustive.

<audio>

<descr="Ninth Symphony">

<src="https://www.mybd.com/fr/9th" clipBegin="10s" clipEnd="20s" soundLevel="-6dB"

</ audio >

< audio > < descr="birds"> <src="https://www.mybd.com/birds7th"> </ audio >

According to another example, the address can consist of an index pointing to a database known on the transmitter and receiver side. According to one example, it may be a private musical library of the user of the transmitting device, known to the receiving device. According to an example, the formulated address may be intended to be recognized by a musical recognition tool (for example an entry recognizable by Shazam©):

< audio > < src ="seven nation army" > </ audio >

< audio > < src ="4567345" > </ audio >

…

Naturally, any other textual description within the reach of those skilled in the art can be considered. According to another embodiment, a Json type format is used.

During a step S7, the segments are prepared for coding and possibly concatenated into a single file or textual stream. If, during the optional step S3, non-voice (sound) information has been extracted from a voice or non-voice segment, it is also inserted into the textual representation. During a step S7, it is possible to carry out a concatenation of the vocal and non-vocal (sound) context data in the same textual segment. Multiple segments can also be concatenated into the same description (the same “file”, intended to be stored or transmitted) XML. For example, you will end up with a markup file describing three segments, as shown below: the first contains only speech; the second contains speech and sound (the ninth symphony, superimposed on speech); the third contains only sound (bird songs). Each segment is introduced by the “segment” tag according to this example.

<segment=1>

<speak>

<amazon:emotion name="sad" intensity="medium">

I want to tell you a secret.

</amazon:emotion>

</speak>

</segment>

<segment=2>

<voice name="Kendra">

<amazon:emotion name="anger" > I am not a real human. </amazon:emotion>

</voice>

<audio>

<descr="Ninth Symphony">

<src="https://www.mybd.com/fr/9th" clipBegin="2s">

</ audio >

</segment>

<segment=3>

< audio > < descr="birds"> <src="https://www.mybd.com/birds7th"> </ audio >

</segment>

Finally, during a step S8, the data is coded and transmitted on the communication channel. According to one embodiment, the data is in XML format and encoded in ASCII. According to one embodiment, the data is compressed using a standard tool adapted to the compression of such data, such as for example W3C EXI (Efficient Extensible Interchange) coding.

According to a preferred embodiment, the data is transmitted over a mobile network in the form of SMS type messages, adapted to text format.

According to one embodiment, the data is transmitted over a LORA type network, at very low speed and long range.

According to one embodiment, the data is stored in a file.

There illustrates steps of the method for generating an audio signal from textual data (or decoding) according to one embodiment of the invention.

The method according to this embodiment receives input in the form of text and labeling data including emotional information associated with a speech segment, as well as optionally textual inputs associated with non-speech sounds. On this basis, it generates an audio stream including a voice adapted to the emotions of the labeling data and optionally sound data such as ambient noises, music, etc. Other non-specifically emotional data (such as voice intensity) may also be part of the textual description.

During an initial step S10, all the modules necessary for implementing the process are initialized.

In a step S11, a stream of textual data is acquired by the textual acquisition module (TEXT IN of the ). This textual stream is broken down into different segments corresponding to vocal segments comprising speech (SVC) or non-vocal segments (SNVC). Any implementation within the reach of those skilled in the art, making it possible to identify the start and end of a text segment, is possible. According to the example given in support of the , the flow is in XML format and each start and end of segment is identified by an XML tag. According to the example provided in support of step S7, the extraction method receives a stream of three segments identified by appropriate tags <segment>: the first segment contains only speech; the second contains speech and sound; the third contains only sound.

During an optional step S12, non-voice information is extracted from a voice or non-voice segment. According to one embodiment, a segment can in fact be considered vocal (since it includes speech) but also include sound. In this case, step S12 can be applied to this segment. This is the case for segment number 2 of the example. In one embodiment, if a voice segment has no sound, step S12 is skipped. In one embodiment, the sound is not processed, that is to say, step S12 is not implemented. When step S12 is implemented, it results in the extraction of sound context information associated with the segment, for example the description of background noise (bird songs) or the sound reproduction of a recording. Step S3 deals with its audio synthesis. Steps S14, S15, however, concern the speech information (VOICE) of the vocal segments.

In a step S15, the extracted speech segment (“I want to tell you a secret”) is converted into sound (or synthesized) using a text-to-speech conversion module (TTS module of the ). Such a module is well known to those skilled in the art and will not be described further. For example, we can use DeepSpeech software for this purpose, a speech synthesis engine from the company Mozilla©.

In a step S14, if it detects an emotion label in the segment, the decoder submits the vocal segment to an emotion synthesis module (EMOT' module of the ). According to the embodiment proposed in support of the , the emotion label is extracted from the segment by reading the “emotion” tag (for example, “sad”). Alternatively, we can read an index (“1”) pointing into a database on parameters associated with the emotion.

To take emotion into account, according to one embodiment, the decoder uses a signal synthesis technique based on prosody, as presented in the article by Shroeder: “ Emotional Speech Synthesis: A Review, EUROSPEECH 2001 Scandinavia , 7th European Conference on Speech Communication and Technology, 2nd INTERSPEECH Event, Aalborg, Denmark, September 3-7, 2001 ”. Such modeling of emotion in speech is based on a certain number of prosodic parameters such as fundamental frequency (pitch), intensity, flow, articulation, etc. of speech: a given vocal segment is modified (or synthesized) by taking into account the parameters characteristic of the emotion which are recorded in the BD' database of the (or comics of my , or BD_1, BD_2 of the ). Such a set of parameters is illustrated in support of the .

According to another embodiment, the decoder uses an artificial neural network, which has been previously trained to modify a speech segment by adding an emotion. In this case, the emotion label as well as the synthesized speech data from step S15 are presented at the input of the neural network previously trained according to an expected format, for example a normalized vector of input data from the neural network. This data is of the same nature as that used for training. An example of such a neural network is detailed in the article " Efficiently Trainable Text-to-Speech System Based on Deep Convolutional Networks with Guided Attention," 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2018 , pp. 4784-4788 ", by H. Tachibana et al. This network can be adapted to take into account the emotion label(s) as input. presents an example of the use of neural networks specialized by emotion, or class of emotions.

Steps S14 and S15 can be parallel, as shown in the , or successive. In one embodiment, step S15 is performed first, then the speech is presented with the emotion at the input of step S14, in order to synthesize the speech with the emotion.

At the end of steps S14 and S15, the speech segment is prepared for its sound reproduction, during a step S16. According to one embodiment, the preparation consists of generating an audio segment of standardized format (WAV, MP3, AAC, etc.)

During an optional step S13, the non-voice sound information (which may belong to a voice or non-voice segment) is processed. In the example given above, the “audio” tag makes it possible to detect such sound information in segments 2 and 3. It can take, as discussed previously, the value of a URI of a sound file, or of an index pointing to a database known on the transmitter and receiver side, private database or music recognition engine, etc. Optionally, a tag (“descr”) provides a description that can correspond to alternative content in the event that the source cannot be played; there can then be a synthesis of birdsong, or an utterance such as “the birds are singing”.

During a step S17, the segments are prepared for sound reproduction and possibly concatenated into a single audio file or stream. If, during the optional step S13, non-voice (sound) information has been extracted from a voice or non-voice segment, it is also inserted into the audio stream. During a step S17, it is possible to superimpose the audio and vocal data (possibly including an emotion) in the same audio segment. According to the example given above, we will end up with an audio file comprising three segments: the first contains only speech; the second contains speech and sound (the ninth symphony, superimposed on speech); the third contains only sound (bird songs).

Finally, during a step S18, the data is returned to a sound device (AUDIO OUT on the ), for example a loudspeaker.

There represents, schematically, an emotion prediction device for the implementation of the user's emotion processing module on a vocal segment (EMOT module of the ) according to one embodiment.

It includes an artificial neural network, for example of the convolutional type.

The artificial neural network shown has been previously trained to identify an emotion label from a set of emotion labels, according to a detailed method supporting the .

Once the network has been trained, it is able to output an index, or label, LEMi, or vocal context data, which allows access to an emotion (for example: “joy”), or a set of labels of emotions (for example: “joy” and “hurried”), in the database which contains the emotion labels. If the network has been correctly trained, this index corresponds to that of the emotion (or set of emotions) closest to the emotion conveyed by the user's voice in this segment.

• le réseau de neurones comprend une couche d’entrée à laquelle sont présentées les données d’entraînement DEi correspondant au segment vocal à classifier, une couche de sortie et un ou plusieurs couches cachées, chaque couche comprenant une pluralité de nœuds, aussi appelés neurones. Chaque couche cachée, ainsi que la couche de sortie, est associée à une opération mathématique, aussi appelée fonction d’activation, réalisée au niveau de chaque nœud de ladite couche. De plus, chaque nœud d’une couche cachée et de la couche de sortie est « relié » à chaque nœud de la couche précédente par un poids, chaque nœud prenant ainsi en entrée le résultat (ou la valeur, pour la couche d’entrée) de chaque nœud de la couche précédente multiplié par le poids associé, c.-à-d. le poids liant ledit nœud audit nœud de la couche précédente. Dans un tel cas, l’étape S0 de la / S20 de la permet de définir le nombre de couches, ainsi que les fonctions d’activation et le nombre de nœuds des couches cachées et de sortie.
• Les données d’entraînement DEi (DE₁, DE₂… DE_N) peuvent correspondre à des données audio brutes échantillonnées ou à un ensemble de paramètres du signal audio (paramètres de prosodie comme les variations de hauteur, d’intensité, de durée, etc. du signal de parole) ou encore à une image spectrale du signal (par exemple une image spectrale logarithmique, ou spectrogramme de type MEL, dont les fréquences sont adaptées à un auditeur humain).
• Le réseau de neurones délivre en chaque nœud de la couche de sortie un score de prédiction SCi (SC₁, SC₂… SC_N) correspondant au score d’appartenance à l’émotion i, repérée par son index LEMi (LEM1, LEM2 … LEMN).
• Le réseau de neurones comprend aussi une couche notée « softmax » dont l’un des objectifs, de manière connue, est de normaliser les scores obtenus pour les faire correspondre à des probabilités Pi (P₁, P₂… P_N) d’appartenance au label i, la somme des probabilités d’appartenance étant par exemple égale à 1.
• Lors de l’entraînement, l’index du nœud de sortie associé à l’émotion d’entraînement (qui est connue) doit être le plus proche possible de la probabilité la plus élevée. L’entraînement consiste à raffiner les différents poids associés aux différents neurones des différentes couches pour obtenir une telle probabilité en sortie ; une boucle de retour (aussi appelée rétropropagation) notée ici RETRO, connectée à une fonction de type « softmax », est utilisée à cet effet.
• Enfin le réseau de neurones selon cet exemple comprend un module OHE (pour One Hot Encoding ) qui permet de choisir un index LEMi unique en sortie (typiquement, celui dont la probabilité est la plus élevée).

According to the example shown:

• the neural network comprises an input layer to which the training data DEi corresponding to the speech segment to be classified are presented, an output layer and one or more hidden layers, each layer comprising a plurality of nodes, also called neurons. Each hidden layer, as well as the output layer, is associated with a mathematical operation, also called activation function, carried out at each node of said layer. In addition, each node of a hidden layer and the output layer is “connected” to each node of the previous layer by a weight, each node thus taking as input the result (or the value, for the input layer ) of each node of the previous layer multiplied by the associated weight, i.e. the weight linking said node to said node of the previous layer. In such a case, step S0 of the / S20 of the allows you to define the number of layers, as well as the activation functions and the number of nodes of the hidden and output layers.
• The training data DEi (DE ₁ , DE ₂ … DE _N ) can correspond to sampled raw audio data or to a set of parameters of the audio signal (prosody parameters such as variations in pitch, intensity, duration, etc. of the speech signal) or to a spectral image of the signal (for example a logarithmic spectral image, or MEL type spectrogram, whose frequencies are adapted to a human listener).
• The neural network delivers at each node of the output layer a prediction score SCi (SC ₁ , SC ₂ … SC _N ) corresponding to the score of belonging to emotion i, identified by its index LEMi (LEM1, LEM2 … LEMN).
• The neural network also includes a layer denoted “softmax” one of whose objectives, in a known manner, is to normalize the scores obtained to make them correspond to probabilities Pi (P ₁ , P ₂ … P _N ) of membership to label i, the sum of the membership probabilities being for example equal to 1.
• When training, the index of the output node associated with the training emotion (which is known) should be as close as possible to the highest probability. The training consists of refining the different weights associated with the different neurons of the different layers to obtain such a probability as output; a feedback loop (also called backpropagation) denoted here RETRO, connected to a “softmax” type function, is used for this purpose.
• Finally, the neural network according to this example includes an OHE module (for One Hot Encoding) which makes it possible to choose a unique LEMi index at the output (typically, the one with the highest probability).

There represents, in flowchart form, the main steps of a method of training an artificial neural network for such an emotion prediction device, which can be part of the step S0 of initializations of the . This process is implemented by an ENTR training module ( ) which is used to configure the neural network as shown by way of example in .

The artificial neural network is trained to identify an emotion label (or several) from a set of possible emotions, more specifically the label is an emotion index from a set of emotion indexes. The emotion label corresponds to vocal context data.

In a step S20, the structure of the artificial RN neural network is defined. In accordance with the example given in , it is a classification network based on the use of a CNN network, a “softmax” function (SM), a backpropagation module (CE), and a one-hot encoding (OHE). Alternatively, the structure used to define the neural network could be for example that of a forward propagation neural network, a multi-layer perceptron, etc. Any other example within the reach of those skilled in the art can be used.

During a step S21, the method obtains training data DEj of an audio segment, from a set of training audio segments SEGj. For example, the training set includes N training segments, where N takes the value 10000, numbered from 1 to 10000, and corresponding to the emotions EMi of label LEMi with the index j between 1 and 10000 and the index i between 1 and 10 (i.e. we have 10 emotion labels). Audio segments may or may not be recorded in association with LEMi labels. Indeed, once the network is trained, according to this embodiment, it is no longer necessary to have the training audio segments. However, it can be useful to be able to access it to calculate distortions, etc. According to one variant, they are deleted from the base. According to another variation, they are removed after training. According to another variant, they are kept in base. These training audio segments can be obtained, without loss of generality, from a database, a hard disk, network access, the output of a decoder, or live from the microphone of a speaker, etc. According to one embodiment, a smartphone type terminal is capable of training the network on the fly on the voice of its user and that of their interlocutors to extract the emotions and synthesize them at the same time as the voice. As mentioned previously, the training segments can correspond to audio sequences of variable duration including speech. They each include a set of audio samples.

For example, a SEGi audio segment includes 20,000 samples. Training audio segments may or may not have the same number of samples. During this step S21 the audio segment SEGi is processed so as to obtain the first set of training data DEi. For example, the audio segment SEGi is converted into a vector so each index includes an audio sample value encoded on a predefined number of bits. According to another example, the audio segment SEGi is converted into a vector of (normalized) characteristic values such as its prosodic characteristics (pitch, pitch, speech rate, etc.). According to another example, the input is a MEL type spectogram.

During a step S22, the training data set DEi is applied to the first layer of the neural network. According to the first previous example, an audio sample value is applied to each neuron in the input layer.

During a step S23, the different layers of the neural network are conventionally implemented (in particular by a sequence of mathematical operations at the different layers, etc.). Such an implementation corresponds to the general knowledge of those skilled in the art, and will not be detailed here.

During a step S24, the artificial RN neural network outputs a prediction score for each output neuron, corresponding to an LEM emotion label. The number of output nodes can be any, and in all cases less than the number of training segments.

During a step S25, the “softmax” (SM) function is used to generate a set of probabilities which are applied as input to the CE module.

During step S26 the CE module imposes the modification of at least one weight of the neural network, for example according to a gradient backpropagation method, so that the maximum probability corresponds to the expected value, this is i.e. the value of the LEMi index of the audio segment used for training.

This cycle can be repeated with other or the same training audio segments until the end of the training of the neural network, for example when the accuracy of the classification (association of an audio segment to its index LEMi) no longer increases. This test can be carried out during an optional step S27.

The artificial RN neural network trained following the implementation of the method is typically stored in the memory of a device such as the transmitting device itself, but can alternatively be stored externally, for example on a network gateway (“gateway”, in Anglo-Saxon terminology), a server or a fixed or mobile terminal such as a personal computer, a tablet, a television or a smartphone

There represents, in schematic form, an emotion database according to two embodiments of the invention.

The BD/BD’ database is preferably populated by the encoder and used by the decoder. According to another embodiment, the two bases may be different.

The schematic database includes emotions (EMi) accessible by an index, or key, or label, denoted LEMi, obtained at the level of the encoder via the output of the EMOT module, and at the level of the decoder via reading the textual description received . A label corresponds to vocal context data. Access to the database can be done for example via an HTTP type request. According to another example, the database corresponds to storage on a hard disk. According to another example, it is done by a digital index, which can take any known form (address, encrypted or unencrypted key, index, etc.). Generally speaking, the element that allows access to the emotion is called an index, and if necessary a set of parameters associated with the emotion. Two embodiments are proposed on the .

• FREQ, qui peut représenter par exemple un écart à la fréquence moyenne, ou pitch, du locuteur sur le segment,
• TEMPO, qui peut représenter une variation de la vitesse d’élocution sur le segment,
• LOUDNESS qui correspond à une atténuation de la voix du locuteur sur le segment.

According to a first embodiment, a first database, BD_1, is intended to be used for emotional speech synthesis of the prosodic type. As described in support of the , step S14, a certain number of prosodic parameters representative of an emotion are stored. They are schematized in the figure by the columns noted:

• FREQ, which can represent for example a deviation from the average frequency, or pitch, of the speaker on the segment,
• TEMPO, which can represent a variation in speaking speed over the segment,
• LOUDNESS which corresponds to an attenuation of the speaker's voice over the segment.

Thus, according to the example, the emotion “joy” is characterized by a 50% increase in frequency, 30% in tempo and zero variation in attenuation.

Naturally, many other parameters are possible and easily accessible to those skilled in the art to characterize an emotion.

According to another embodiment, an emotional synthesis is used based on a neural network as described in the aforementioned paper by Tachibana et al. The neural network is selected from a plurality of networks each trained by a corpus of different emotions (“RNi” column of Table 2). For example, a neural network was trained with the emotion label “joy” and corresponds to a specific neural network pointed to by the LEM1 index of the emotion “joy”. A reference of the RNi neural network is stored in the BD_2 database.

According to another embodiment, a single neural network trained for all types of emotion is used, in this case the base is no longer useful.

At the receiver (DR), the emotion device receives as input a synthesized voice segment and an emotion label (1), or an emotion (“joy”). It searches for the parameters (FREQ, TEMPO, etc.) or the neural network (RNi) associated with the emotion and generates as output a vocal segment corrected by the use of these parameters.

At the transmitter (DE), the base can be populated using any technique accessible to those skilled in the art: neural network, statistical analysis of a corpus of test sequences, manual entry of characteristics from information obtained from the literature, etc.

Naturally, this base can be external to the devices, for example it can be located in the network (cloud) or on an external hard drive, it can be distributed or not, etc. as long as the encoder-transmitter (DE) can access it for writing, and the decoder-receiver (DR) for reading.

It goes without saying that the embodiment which has been described above has been given for purely indicative purposes and is in no way limiting, and that numerous modifications can easily be made by those skilled in the art without departing from the scope. of the invention.

In particular, in the case of a two-way conversation, each of the terminals DE and DR may include a module for generating textual data and a module for generating an audio signal from textual data.

Claims (15)

Method for generating textual data representative of an audio signal comprising at least one voice segment (SVC), the method being characterized in that it comprises, on a terminal (DE), the following steps:
- identify in the audio signal (S1) said at least one voice segment containing at least one voice information;
- extract (S4) from the vocal segment at least one vocal context data (LEMi), representative of an emotion associated with the vocal information;
- prepare (S6, S7) a textual description of said voice segment, comprising at least said voice information and said voice context data. Method for generating textual data according to claim 1, further comprising a step of extracting (S2) from said at least one vocal segment at least one piece of sound context data, representative of the sound content of the vocal segment, and in which said textual description further comprises said sound context data. Method for generating textual data according to claim 1, in which the audio signal further comprises at least one non-voice segment (SNVC) and the method further comprises:
- identify (S1) said at least one non-vocal segment comprising at least one sound context data, representative of the non-vocal content;
- prepare (S2, S7) a textual description of said non-vocal segment, comprising at least said sound context data: Method for generating textual data according to one of the preceding claims, in which the textual description is a set of data in XML or JSON format. Method for generating textual data according to claim 1, in which the vocal context data is generated by a classification module. Method for generating textual data according to one of the preceding claims, in which the textual description is transmitted (S8) over a network. Method for generating textual data according to the preceding claim, in which the transmission (S8) is carried out in the form of an SMS message. Method for generating textual data according to claim 1, further comprising a learning phase (S0) comprising the steps of:
- receive training audio segments (DEi) comprising at least one voice information;
- receive labeling data comprising at least one vocal context data (LEMi) representative of an emotion associated with the vocal information of the segment;
- adapt a voice context detection system based on the received training audio segments and labeling data. Device for generating textual data representative of an audio signal comprising at least one vocal segment, the device comprising a memory (MEM) and a processor (PROC) configured to:
- identify in the audio signal said at least one voice segment containing at least one voice information;
- extract from the vocal segment at least one piece of vocal context data, representative of an emotion associated with said vocal information;
- prepare a textual description of said voice segment, comprising at least said voice information and said voice context data. Method for generating an audio signal from a textual description comprising at least one vocal segment, comprising the steps of:
- extract (S14) from the textual description of the vocal segment at least one vocal information and one vocal context data representative of an emotion associated with the vocal information;
- synthesize (S15, S16, S17) the voice segment from at least said voice information and voice context data. Method for generating an audio signal according to claim 10, further comprising a step of extracting (S12) from the textual description of the vocal segment at least one piece of sound context data, representative of the sound content of the vocal segment, and in which synthesizing the vocal segment also uses the sound context data. Device for generating an audio signal from a textual description comprising at least one voice segment, comprising a memory (MEM') and a processor (PROC') configured to:
- extract from the textual description of the vocal segment at least one vocal information and one vocal context data representative of an emotion associated with the vocal information;
- synthesize the audio segment from at least said vocal information and vocal context data. System comprising:
- a device for generating textual data representative of an audio signal according to claim 9;
- a device for generating an audio signal from a textual description of a voice segment according to claim 12;
characterized in that the textual data representative of an audio segment are transmitted over a network between the device for generating textual data and the device for generating an audio signal. Computer program capable of being implemented on a device as defined in claim 9, the program comprising code instructions which, when the program is executed by a processor, carries out the steps of the method defined according to one of the claims 1 to 8. Computer program capable of being implemented on a device as defined in claim 12, the program comprising code instructions which, when the program is executed by a processor, carries out the steps of the method defined according to one of the claims 10 to 11.