ES2728708T3 - Audiovisual dialogue system and procedure - Google Patents

Abstract

Audiovisual dialogue system (1) comprising: an audio input device (21); an audio output device (31); a visual output device (32); and a processor (5), the processor being arranged to: receive (72) an audio input signal representing a source voice from the audio input device; performing (73) a substantially real-time conversion of the voice conversion into the audio input signal to produce an audio output signal representing a voice of interest, in which the audio output signal is provided to the audio output device, and in which the real-time voice conversion process includes: i) decomposing the audio input signal into a set of time-varying filtering characteristics and a residual excitation signal; ii) spectrally transform the filtering characteristics that vary over time, and / or modify the tone of the residual excitation signal; and iii) synthesize the audio output signal depending on the filtered filtering characteristics with time transformed and / or the residual excitation signal with the modified tone; generate (71) an avatar, in which the avatar is presented visually on the visual output device; and facially animate the generated avatar (74), in which the animation is synchronized with the audio output signal; in which the processor is further arranged to customize the conversion of the voice in real time, including customization: 1) select one of a plurality of predefined voices of interest, in which the predefined voices of interest are represented by a set of respective linear transformations that include a set of time-varying filtering characteristics and a pitch scaling factor; and 2) adjust (82) the filtering characteristics variable with time and / or the scaling factor of the predefined voice tone of interest, selected to provide the customized parameters to the voice of interest, in which the filtering characteristics variables with time and / or the scaling factor of the tone of the selected predefined voice of interest, are adjusted using a plurality of sliders (A, B, C, D) arranged in the user interface, which when activated by the user they establish a magnitude of change by which the filtering characteristics that vary with time and / or the pitch scaling factor are adjusted.

Description

DESCRIPTION

Audiovisual dialogue system and procedure

Technical sector

The present invention relates to a system and a procedure for audiovisual dialogue. In particular, but not exclusively, aspects of the present invention relate to an audiovisual dialogue system and a procedure that include real-time voice conversion, facial animation and voice customization. Background of the invention and prior art

Spoken dialogue systems are used regularly in a wide range of technical sectors, for example, in mobile telephony devices to process user speech and transmit it to a communications receiving device. Spoken dialogue systems usually employ a combination of speech synthesis and voice recognition techniques. Speech synthesis techniques are often used in fun productions such as video games and animated presentations, but it can also be a useful tool in assistive technology for people with a variety of disabilities, for example, visual and speech impairments, and dyslexia

A common technique of speech synthesis is a text-to-speech system (TTS), in which plain text is analyzed, converted into phonetic expressions and then converted into speech, by generating shape waves. However, the TTS system may have trouble identifying the correct pronunciation of certain aspects of the text, for example, numbers, abbreviations and spellings that have different pronunciations depending on the context of the word. As a result, TTS can be a long and complex process and, therefore, it is often necessary to know the text long before speech synthesis. In view of this, TTS systems are not suitable for real-time voice conversion.

The treatment of auditory hallucinations has been disclosed by a number of means of publication. News articles published by the BBC (http: Zwww.bbc.co.uk/news/health-22691718) and the newspaper "The Guardian" (http: Zwww.guardian.co.uk/society/2013/may/29/ schizophrenia-mental-health) both comment on the possibility of using virtual reality (VR) techniques to produce an "avatar" that represents the "voice" heard by schizophrenic patients in order to help them control their hallucinations. Another BBC news article (http: /www.bbc.co.uk/news/science-environment-23144208) reports how the technology used in a clinical trial is able to adjust the voice of a virtual avatar provided by the therapeutic professional, so that it resembles the voice of the patient's auditory hallucination.

Other prior techniques include: "Computer-assisted therapy for medication resistant auditory hallucinations: proof-of-concept study" by Leff et al., (Computer-assisted therapies for auditory drug-resistant hallucinations: conceptual demonstration study), British Journal of Psychiatry, vol. 202 n ° 6, February 21, 2013, pages 428-433, as well as “Continous Probabilistic Transform for Voice Conversion” by Stylianou et al., IEEE Proceedings on Speech and Audio processing, Vol. 6, No. 2, March 1, 1998.

Summary of the Invention

The present invention discloses an audiovisual dialogue system that allows a user to create an "avatar" that can be customized to look and sound in a certain way. The avatar can be created to resemble, for example, a person, animal or mythical creature, and can be generated to have a variable voice that can be feminine or masculine. The system then uses a real-time voice conversion in order to transform an audio input, for example, a spoken word, into a voice of interest that is selected and customized by the user. The system is arranged to facially animate the avatar using a real-time lip synchronization algorithm, so that the generated avatar and the voice of interest are synchronized. That is, the avatar seems to say the words of an audio input with the voice of interest. The present invention can be used in multiple applications, for example, for game software and for animated films.

A particular application of the present invention is the treatment of schizophrenia, in which patients suffering from voice hearing can create a visual and audible representation of their auditory hallucination in the form of the avatar. The patient can customize the visual appearance of the avatar and customize the voice of the avatar, so that it closely resembles his auditory hallucination. Once the patient has designed his avatar, a therapeutic professional can speak to the patient through the avatar so that the patient can face his hallucination and enter into conversation with her, thus allowing control of the disease. In view of the foregoing, one aspect of the present invention discloses an audiovisual dialogue system, comprising: an audio input device; an audio output device; a visual output device and a processor, the processor being arranged to: receive an audio input signal representing a voice source from the audio input device; perform a conversion of the voice, substantially in real time, of the audio input signal to produce an audio output signal representing a voice of interest, in which the audio output signal is provided to the output device of audio, and in which the real-time voice conversion process includes: i) decomposing the audio input signal into a set of time-varying filtering characteristics and a residual excitation signal; ii) spectrally transform the filtering characteristics that vary over time, and / or modify the tone of the residual excitation signal; and iii) synthesize the audio output signal depending on the transformed filtering characteristics that vary with time and / or the residual excitation signal of the modified tone; generate an avatar, in which the avatar is presented visually on the visual output device; and facially animate the generated avatar, in which the animation is synchronized with the audio output signal; in which the processor is further arranged to customize the conversion of the voice in real time, including customization: 1) select one of a plurality of predefined voices of interest, in which the predefined voices of interest are represented by means of a set of the respective linear transformations that include a set of time-varying filtering characteristics and a pitch scaling factor; and 2) adjust the filtering characteristics that vary with time and / or the scaling factor of the tone of the predefined voice of interest selected to provide customized parameters to the voice of interest, in which the filtering characteristics vary with time and / or the scaling factor of the tone of the selected predefined voice of interest, are adjusted using a plurality of sliders displayed on the user interface which, when activated by the user, establish a magnitude of change by which the Filtering characteristics that vary with time and / or pitch scaling factor.

In one embodiment, the conversion of the voice substantially in real time includes the decomposition of the audio input signal into a set of time-varying filtering characteristics and a residual excitation signal, spectrally transforming the variable filtering characteristics with the time, and / or modifying the tone of the residual excitation signal, and synthesizing the audio output signal depending on the transformed filtered characteristics with time and / or the residual excitation signal of the modified tone.

Preferably, the decomposition of the audio input signal is performed by means of a linear prediction coding process. It should be borne in mind that the filtering characteristics that vary over time refer to the prediction coefficients of the linear speech prediction processing.

In another embodiment, the time-varying filtering characteristics are estimated in short sections with windows of the input audio signal. Preferably, the short sections with windows of the input audio signal are of a duration of 20 to 40 ms and are superimposed of 5 to 15 ms. Additionally, the prediction error of the linear prediction coding can be calculated separately for the short window sections, the prediction errors being subsequently superimposed and added to produce the residual excitation signal.

In a preferred embodiment, the time-varying filtering characteristics are transformed from Fourier into a multiple point, for example, 256 response amplitude points before the spectral transformation of the time-varying filtering characteristics. Preferably, the spectrally transformed response amplitude is then an inverse Fourier transform transformed back into time-varying filtering characteristics.

In another embodiment, the pitch modification includes a new sampling of the residual excitation signal and the change in the fundamental frequency of the residual excitation signal sampled again by means of a constant tone scaling factor. In addition, the change in the duration of the residual excitation signal produced by the new sampling can be corrected by compression or extension of the residual excitation signal over time to restore the original duration.

According to one embodiment, the spectral transformation of the filtering characteristics variable over time is carried out by means of a set of linear transformations. The sets of linear transformations can then be generated between the audio input signal and a plurality of predefined voices of interest. In one embodiment, more than 50 predefined voices of interest are used to generate the sets of linear transformations. Preferably, at least 82 voices of interest are used, including at least 40 male broadcasters of interest and at least 42 female broadcasters of interest. In particular, in one embodiment, a set of 8 linear transformations between the audio input signal and a predefined voice of interest can be generated. In other embodiments, 16, 24, or other amounts may be used.

According to another embodiment, a series of phrases can be pronounced both by means of the pluralities of predefined voices of interest and by the audio input signal and, preferably, in one embodiment, at least 20 sentences are pronounced. The plurality of phrases pronounced by the input audio signal and the plurality of phrases pronounced by the plurality of predefined voices of interest are temporarily aligned. The plurality of phrases may be temporarily aligned using Mel cepstral coefficients in combination with a dynamic programming algorithm. In one embodiment, in the case of a plurality of Signal sections within the pairs of temporarily aligned phrases, the prediction coefficients and the linear prediction coding spectrum are calculated, and the optimal frequency assignment is found using a dynamic programming algorithm.

Preferably, the frequency assignments are used for the plurality of signal sections in the case of the audio input signal to find the set of linear transformations that minimizes the transformation error of the quadratic mean. In a specific embodiment, a set of 8 linear transformations can be found. Other numbers of transformations may be used in other embodiments.

The selected predefined voice of interest may be associated with a set of linear transformations. Preferably, the transformation vectors of the set of linear transformations are reduced to a mean transformation vector and a plurality of orthogonal change vectors, in which a slider can be used to adjust the magnitude by which a vector is added change to the average vector, so that the filtering characteristics that vary over time are adjusted.

According to another embodiment, the processor is further arranged to facially generate the generated avatar, in which the facial customization of the generated avatar may include a visual arrangement of different faces for selection. Preferably, the visual arrangement of the different faces includes at least 250 different faces. In addition, the visual arrangement of the different faces can vary in gender, age, ethnicity and hair type and a series of accessories and additional hair types can be arranged for your selection.

Other aspects of the present invention are apparent from the appended claims,

Brief description of the drawings

The present invention will be described below only by way of example and with reference to the accompanying drawings, in which:

Figure 1 shows a set of avatars chosen by the patients;

Figure 2 shows the front page of the online face selector;

Figure 3 shows the voice registration user interface;

Figure 4 shows the execution of the system in an embodiment of the present invention;

Figure 5a is an analysis that schematically shows the spoken signal;

Figure 5b is a reconstruction of the spoken signal shown schematically;

Figure 6 is a block diagram showing an apparatus according to an embodiment of the present invention; Figure 7 is a flow chart showing a method according to an embodiment of the present invention; Figure 8 is a flow chart showing a voice conversion method according to an embodiment of the present invention;

Figure 9 is a flow chart showing the spectral distortion method according to an embodiment of the present invention;

Fig. 10 is a flow chart showing the process of changing the residual tone according to an embodiment of the present invention;

Figure 11 is a flow chart showing the procedure for generating the spectral distortion function according to an embodiment of the present invention.

Detailed description of the preferred embodiments

Summary

The present invention proposes a highly novel system of speech technology for the execution of CBT-based therapies for auditory hallucinations, called "Avatars Therapy." The basis of the system is a series of components for the creation and operation of an individualized avatar that "speaks" in the voice the patient hears, and virtually resembles the face the patient perceives. In cases where the patient does not clearly perceive a face, he or she is asked to choose a face with which they feel comfortable talking to her. The characteristics of the avatar are chosen by prior consultation with each patient using a series of computerized tools In figure 1, a set of images of avatars chosen by the patients is shown.

During therapy sessions, the therapeutic professional and the patient are seated in front of a computer screen in separate rooms and communicate via a dual audio connection. The expressions of the avatar are pronounced by the therapeutic professional and the patient's responses to the avatar are fed back (audio only) so that the patient can interact and question the avatar. The voice of the avatar is produced by modifying the speech of the therapeutic professional in real time, so that the words of the therapeutic professional are pronounced by the avatar in the simulated voice on the client's computer with lip sync. In the same channel, the therapeutic professional can also communicate instructions or advice and encourage the patient in his own voice, as would be the case in a normal therapy session. The small-scale results of an NIHR-based RCT study are disclosed in [9] and also discussed in [10].

introduction

The phenomenon of auditory hallucinations ("hearing voices") is a permanent problem in the treatment of serious mental illnesses such as schizophrenia. Approximately 30% of people with this diagnosis continue to experience hallucinations and delusions despite treatment with anti-psychotic medication [1]. Hearing voices is not only distressing for those who suffer from it, but also has a serious impact on their caregivers and other people they come into contact with. Auditory hallucinations manifest themselves in a certain number of ways, including voices that say out loud what the patient is thinking, voices that express a comment about the patient's actions or about imagined external acts, two or more people chatting, referring to often to the patient in the third person and orders that force the patient to perform certain actions (often violent). Persistent voices severely limit the patient's ability to concentrate on tasks, and thus make rehabilitation attempts difficult. The direct costs of treatment in the United Kingdom are estimated at 2 billion pounds annually, while the indirect costs, including the loss of employment of patients and caregivers, amount to another 2 billion pounds [2].

Approximately in the last 15 years, a number of randomized controlled trials (RCT) have been carried out in Britain to assess the value of cognitive behavioral therapy (CBT) in the case of persistent medication-resistant psychosis symptoms [3, 4, 5, 6]. While the tests have shown some effect in reducing auditory hallucinations, they have been criticized based on their experimental design. A more recent CBT of the CBT, although it does not affect the frequency or intensity of auditory hallucinations, was successful in reducing the power of the dominant voice as perceived by the patients, and in their distress.

[7].

When asked about the worst aspect of hearing persecutory voices, many patients explain "the feeling of helplessness" to which they induce. Those who are able to establish a dialogue with their voice feel that they control it much more and consequently their suffering is reduced. Many people who hear voices also visualize a face associated with the voice. This face may be that of a person known to them, that of a well-known personality or an imaginary figure, or perhaps representing an angel, a demon or other religious or mythical figure. One means by which a patient could be helped to take control of his voice is by creating a virtual avatar that represents the person they think speaks to them, and then allows the avatar to become progressively under control. of the patient. Therefore, the incorporation of the voice within the context of an avatar is a natural step to perform from a clinical point of view.

Virtual reality (VR) techniques have been explored previously for modeling psychotic episodes involved in schizophrenia. Banks and others [8] report a sophisticated VR environment to simulate auditory and visual hallucinations and motivate their use in medical education, but they have not tested their system in a clinical setting, nor have they attempted to individualize voice characteristics.

Design System Considerations

Avatar expression generation

An important parameter in the design of a dialogue system with a synthetic voice is the procedure of generating synthetic expressions, either directly synthesizing speech with the desired characteristics, or transforming natural speech to produce a quality different from that of the original speaker . Since the content of the expressions of the therapeutic professional to be pronounced by the avatar cannot be known in advance, the standard text-to-speech (TTS) system is not usable in the present invention. Typing the text during therapy sessions could frequently introduce long unacceptable delays, interrupting the flow of dialogue. Therefore, to allow an interaction as close to real time as possible, a system based on voice conversion is used instead of speech synthesis.

The design parameters of the experimental study require that the voice of interest and the face be generated in a Initial registration session with each patient, and not lasting more than one hour. Theoretically, this would be followed by the first series of therapy sessions, which means that the avatar must be obtained in its final form during the registration session. Topics were recruited continuously during the study and therefore voices of interest and voices were not available prior to technological development.

In conventional voice conversion applications in which the set of speakers of interest is known in advance, a learning process is applied to produce the assignment, or transformation, from the source to the voice of interest. This requires the availability to learn speech data from both the source and the speakers of interest. In the development of the present invention the main technical problem was the fact that the voices of interest are not known until the patients have registered them and, of course, real samples cannot be obtained in any case. This means that assignments cannot be learned individually and that, on the contrary, a predetermined set of transformed voices must be developed using the voice of the therapeutic professional as a source. Although this limits the field of voices that can be produced with good quality, a variety of different voices can still be produced, provided some additional means of fine tuning of the transformations is available.

System components for registration and therapy

The system comprises a combination of online and offline procedures, and includes a real-time voice conversion system, voice and face personalization systems (registration), a customizable facial animation system with synchronization of real-time lips and a two-way audio channel with switching between the voices of the therapeutic professional and the avatar.

Offline, or in registration procedures, the patient chooses the appropriate voice and face using a set of computer-based tools, assisted by an operator trained and supervised by the therapeutic professional. It should be noted that some of these tools are commercially available products or software toolkits that have been customized to meet the needs of the present invention.

Real-time voice conversion system

Linear speech prediction processing

The spoken signals are produced by means of excitation signals produced in the throat that are modified depending on the shape of the vocal tract. In order to analyze and synthesize speech, speech signals must be broken down into their separate components and must be considered independently. One way to achieve this is the linear speech prediction processing [19, 20] in which the speech signal is modeled as a weighted sum of p previous speech samples, plus a residual excitation waveform that can be also referred to as a prediction error. The speech signal can be expressed as:

y a k y jn - k ) Cuando x(n) es la señal de habla, e(rí) es el error de predicción, akson los coeficientes de predicción y k —\

yaky jn - k) When x (n) is the speech signal, e (rí) is the prediction error, akson the prediction coefficients and k - \

It is the weighted sum of the previous speech samples.

Consequently, in the case of a particular speech signal, a transfer function is applied that breaks down the speech signal into the two components, as shown by means of Figure 5a, in which the transfer function is expressed how:

Conversely, for a given prediction error, the inverse of the transfer function can be applied which allows the speech signal to be reconstructed, as shown by means of Figure 5b. That is, it is possible to model the excitation signal that leads to the speech signal.

The speech signal is sampled in short sections of windows, and in order to decompose or reconstruct the speech signal, the prediction coefficients ak for the transfer function must be calculated, so as to minimize the quadratic mean of the error of prediction e (n). The prediction error of the linear prediction model is essentially the difference between the real speech signal and the predicted speech signal, and the value of its quadratic mean is indicative of the energy in the residual excitation waveform. Two current techniques for minimizing the prediction error of the quadratic mean and the calculation of the prediction coefficients include the Covariance procedure and autocorrelation procedure. Both techniques result in an array of equations that can be solved by means of a universal algorithm. Once these prediction coefficients have been calculated, it is possible to decompose the speech signal, manipulate the speech signal components and then reconstruct the speech signal.

Additionally, the frequency response of the linear prediction model corresponds to the spectral envelope of the speech signal and is free of tone harmonics. It is the spectral envelope that provides the characteristics of the vocal tract. The shape of the spectral envelope depends on the number of parameters that are used in the linear prediction model, that is, the number of p previous samples used to model the speech signal. As p increases, more tone harmonics are captured. Therefore, if it is chosen that p is very high, the spectral peaks capture these harmonics of the tone. However, if a p that is too low is chosen, the key resonance zones will be absent from the spectrum. As a general rule, p should be chosen as double the number of resonances of the vocal tract (shapers) in the signal bandwidth. Also, it should be taken into account that the number of linear prediction parameters (p) has no appreciable effect on the error of the quadratic mean of the linear prediction model.

Technology

Voice conversion technology is broadly based on the approach described by Stylianou [11]. In this approach, as shown in Figure 8, speech is broken down into two elements: a filter that varies with time and a waveform of residual excitation (step 81). The spectral transformations of the filter combined with the modification of the tone (step 82) of the residue allow the synthesis (step 83) of versions of a pronunciation in which the characteristics of the speaker appear to have changed.

The speech decomposition is carried out by means of linear prediction, as described above, the prediction filter being estimated in short sections of windows of the signal with a duration of 30 ms and an overlap of 10 ms. In other embodiments, other window and overlay lengths can be used. Next, the prediction error of the filter is calculated separately for each window and then they are superimposed and added to create the residual excitation waveform. To make prediction coefficients more susceptible to spectral assignment, prediction coefficients are transformed from Fourier into a multipoint, for example, 256 points of amplitude of response. This transformation is simply a mathematical comfort, allowing the response of the filter to be adjusted by means of a spectral distortion function for the transformation of the characteristics of the voice. In the case of synthesis, the distorted spectrum is converted back into prediction coefficients. With uniform spectral distortion, conversion to and from the amplitude response does not introduce any appreciable distortion of the signal.

Spectral manipulation of the filter response is carried out by means of a set of linear transformations (step 82). In this work, a set of 8 transformations is used for each voice of interest, whose exact forms are found during the training and customization procedure described below. To select how the transformations are used for a given elongation of the speech signal, the spectral signal envelope for the source speaker is modeled using a Gaussian Mixture Model (GMM) of 8 mixtures. The behavior can be improved with 64 mixes [11, 12] but since the voice conversion system of the present invention is designed for operation with flexibility and in real time, without any concept of “precision” that recreates a specific voice , 8 mixtures constitute the best compromise between the required quality and a minimum calculation load. Each mixture is then associated with a linear transformation and the final transformation applied to the filter is based on the sum of transformations weighted according to the probability of the mixture for the corresponding envelope of the speech signal source. See [11] for a mathematical description.

In the case of a tone scaling, the residual signal of the source announcer is first sampled again to change the fundamental frequency by means of a constant factor. To correct the change in duration produced by the new sampling, the residue is then compressed or lengthened in time to restore the original duration using a Waveform-Similarity Overlap-Add (WSOLA) (Addition-overlap by similarity of waveforms ) (see [13]).

Figures 9 to 11 show, by way of example, procedures for spectral distortion, residual pitch change and the generation of the spectral distortion function that should be used in voice conversion. The voice conversion begins with the sampling of the speech source signal in sections of the 30 ms signal (9-11a), superimposed on 10 ms. These sections of the signal are then arranged in windows using the Hamming window function (9-11 b) in order to determine all values outside 0 <n <N, to zero. From this window signal, the spectral distortion procedure shown in Figure 9 can be performed, and the tone change procedure shown in Figure 10. The coefficients of the window signal prediction filter are calculated by means of the autocorrelation procedure to first find a set of autocorrelation coefficients (9-10c), and the recurring Levinson-Durbin procedure to find the prediction coefficients (9-10d) below. Once they have been found, the prediction coefficients are transformed by Fourier to obtain the amplitude of response (9e), and said amplitude of response can be distorted then using a spectral distortion function (9f) calculated for said window signal.

To obtain the spectral distortion function, as shown by Figure 11, the Melted Scale Cepstral Coefficients (MFCC) (11c) are calculated for the window signal. Additionally, the spectrum involved for the window signal is modeled using a Gaussian mixing model (Gaussian Mixture Model, (GMM)) of 8 mixtures and the probabilities that the MFCC data could have been generated by means of each mixture of the GMM (11 d). Each mixture is associated with a linear transformation (corresponding to the voices of interest) and the sum of these linear transformations weighted by the above probabilities provides the spectral distortion function to be applied to the amplitude response (11 e). Once the spectral distortion function has been applied to the amplitude response, the distorted resulting amplitude response is an inverse Fourier transform to obtain the corresponding set of autocorrelation coefficients (9g). Next, using the recurring Levinson-Durbin procedure, the correlation coefficients are used to obtain a set of distorted prediction coefficients (9h).

The second stage of voice conversion is the modification of the tone of the speech signal. The prediction coefficients of each window signal, as described above, are used to calculate the prediction error of the window signals, and then the prediction errors are superimposed and added to obtain the residual signal of the signal from Speak (10e). This residual signal is sampled again by a factor of 4, (10f), so that it increases the sampling rate and the sampled signal is sampled again to obtain the change of tone of the residual signal (10g). The residual signal with the changed tone is superimposed and then added to a historical accumulator of window signals of 3 x 30 ms, (10h), superimposed on 10 ms. In order to restore the changed tone signal to the original duration of the residual signal, the addition-superposition by similarity of waveforms is used to modify the time scale without affecting the tone, (10i). Finally, the residue changed in tone is sampled, then reducing it by a factor of 4, (10j). The distorted prediction coefficients and the residual signal with pitch change can be used to synthesize the output signal (9i) that refers to the voice of interest.

Training

The entrance to the voice personalization process is a large number of transformations of the voice formed, generated between the only source speaker (the therapeutic professional) and a certain number of speakers trained. In the case of the present invention, 55 speakers of the body of British English accents were taken [14], and 27 speakers of the UCL body of speakers of variability were taken [15]. In total, 40 male broadcasters and 42 female broadcasters of interest were used and in each case, a selection of 20 phrases pronounced by each speaker and a matching set of 20 sentences pronounced by the source speaker was obtained.

The materials spoken by the source announcer were used to construct a mixture of 8 GMM based on the MFCC characteristics of the signal [16]. This GMM was then used to form all of the 82 sets of linear transformations using the Stylianou procedure. This, in the first place, implied, in the case of the source speaker and each speaker of interest, a temporal alignment between each pair of matching phrases. The MFCC vectors for this alignment were used in combination with a dynamic programming algorithm. Next, for each matching pair of signal sections, the prediction coefficients and the LPC spectrum were calculated and the optimal frequency of assignment for said pair of structures was found using a dynamic programming algorithm. Having obtained all the assignments for all the structures matched in all the sentences, together with the set of GMM mixing probabilities for each source structure, it is possible to find the average set of 8 transformations that minimize the quadratic mean of the transformation error. Additionally, the fundamental average frequency of each speaker and the source speaker was measured. Thus, for each speaker of interest, the training procedure generates a set of 8 linear frequency transformations and a pitch scaling factor, which taken together constitute the best attempt to obtain the morphology of the characteristics of the speaker's voice source in the speaker of interest (step 83).

Registration Procedures

Selection and customization of faces

The face selection procedure is based on the FaceGen © Modeller software (face generation modeler) developed by the Singular Inversions firm, as shown in Figure 2. This program allows the creation of a virtually infinite range of faces in three dimensions and save them in various file formats. As a starting point of the selection, a virtual layout or “palette” of about 250 different faces was created covering a wide range of features. The faces were chosen to cover a wide variety of face types that vary in gender, age and ethnicity as well as various current types of hair.

A limited number of hair types and accessories are available. Each of these files was saved as both a FaceGen model file and a JPEG image file. Image files were compiled on web pages that connected to each other to form a website to select the initial palette of faces.

Each of the images in the image selection site has an associated FaceGen model file, stored in a parallel directory structure. Once the client has chosen a suitable initial face, the corresponding face model file is easily located for future improvement in the FaceGen according to the client's wishes. The image of the resulting face is exported as a set of animation targets in the .OBJ 3D graphics format. These consist essentially of a set of image and texture files for the preselected set of “visemas” that are used in the facial animation system described below.

Voice record

In the pilot study, voice customization was performed in two stages. However, the process of the present invention is integrated into a single application. First, a sample of 20-voice transformations for the appropriate genre is chosen from the training set and the client is asked to choose which of these voices has the locution characteristics closest to the required voice. On the selection screen each speech bubble represents a different voice and when it is punctured it interprets an audio file with synchronization of the lips in the voice it represents, as shown in Figure 3.

In a second stage, the client manipulates the whole of the chosen transformation using a small set of sliders that alter the pitch scaling factor and the transformation coefficients that can be integrated in the screen of figure 3. To allow manipulation of the transformations using a small number of sliders, an analysis of the main components in the 82 transformations estimated during the formation is carried out. This reduces the transformation vector chosen to a medium transformation vector plus a certain number of orthogonal change vectors. Each slider manipulates the amount by which each change vector is added to the middle vector, and a different slider manipulates the pitch scaling factor. The user interface is designed to allow the client to experiment with multiple variants of the chosen voice and keep copies of promising alternatives, so that at the end of the process the best possible transformation can be saved.

Performance issues

During the development phase of the present invention, voice and face selection procedures were tested with three voice listeners who did not take part in the subsequent study. These listeners reported a similarity accuracy of approximately 60-70% with respect to the target face. A similar figure was obtained with the patients who took part in the study. However, in a few cases, the effect was more spectacular. A small number of patients, when they were shown the avatar for the first time, found that the realism was so great that they were unable to see it for a long time.

Given the technical limitations and the requirement that all voices can be derived from a single source (a male speaker in his 70s and with a low voice), it was sometimes difficult to match the voice exactly. Not surprisingly, the greatest difficulty arose in making female voices fit together. In general, patients reported that the accuracy of the match was within a range of 60-90%. The duration of the process depends on factors such as the client's ability to communicate, the level of detail desired and the degree of customization required. It usually takes about 20 minutes, rarely less than 15 or more than 30 minutes.

Execution system

The real-time execution of the converted voice is carried out using personalized software that works on two personal computers connected to the network: a “server” process that works on the medical side and a “client” process that works on the patient side. A schematic illustration of the experimental arrangement is shown in Figure 4.

The server process captures the audio from the therapeutic professional's microphone and performs the entire signal processing and speech conversion. The broadcast speech is then transferred via the network to the client's computer. The client process interprets the converted speech again, and captures the patient's speech, loading it into the therapeutic professional's computer. The therapeutic professional uses two “push-to-talk” buttons that control whether the natural voice or the converted voice is issued to the client. The therapeutic professional listens with the headphones, so that the client's responses are not fed back through the microphone of the therapeutic professional. Similarly, the client's microphone is muted while the audio is being sent from the server, and also to avoid feedback through the system. This has a limited but acceptable effect. in the dialogue

It is decisive to create the separation between the therapeutic professional and the avatar to ensure that only the expressions of the avatar pass through the talking head. Since both signals are conducted on the same channel, this is achieved through the capture of speech in stereo on the server side (therapeutic professional) and by passing the raw audio and converting through the left and right channels respectively . Only the right channel passes to the recognition device and the lip sync engine in the client animation, thereby achieving the desired effect.

Finally, the client side is equipped with a panic button that disconnects the avatar immediately when pressed, presents an image of a landscape and sounds soothing music. In the study, few clients found it necessary to use this device but their presence alone proved to help relieve some clients' anxiety.

Additional comments

The strengths of the audio system have been achieved, which are the personalization of the voice and the real-time execution of a personalized voice. When the voice of interest is masculine and the transformation is not too great, the quality of the transformation is good and relatively undistorted. The main weak point is that the range of good quality voices available is somewhat limited, and when the required transformations are too large, there is also a loss of quality. Female voices generated from the voice of a male therapeutic professional may sound somewhat distorted and unconvincing to some patients.

In addition, because all audio processing is performed using sound cards and standard sound drivers, the processing system introduces about 50 ms of delay in the dialogue. Additionally, the variable delays introduced by the network connection can cause more serious problems that interfere with the conduct of the session. For this reason, the system must be redeployed to work with a direct audio connection, with restructured audio processing to use the “Windows core” audio platform with the result of much lower latencies.

It should be borne in mind that the term "real-time voice conversion" should be interpreted as meaning that the voice conversion is carried out live, so that a dialogue can be exchanged between two individuals, for For example, a therapeutic professional and a patient for the treatment of auditory hallucinations. It should be clear to the expert that any delay added by the processing system is acceptable as long as it does not have an appreciable effect on the flow of the conversation between the two individuals. As mentioned above, an acceptable delay can be a delay of about 50 ms or, for example, a delay within a range of 20 ms to 250 ms.

Face animation platform - Implementation

The animation platform is based on real-time lip sync SDK from Annosoft LLC [17]. Usually used for 3D animation of characters who speak in video games or movies, this set of tools allows 3D characters designed in other applications to be imported in a variety of formats and converted with a time-phonetic phonetic recognition device. real that produces movements of the lips. The system can work with both direct audio inputs and previously recorded conversations. Information about the characters and objectives of the animation is loaded from a simple text configuration file connected to a set of graphic files, and can be compiled beforehand in binary format for faster loading and viewing. In the present invention a set of 12 "visemes" (see, for example, [18]), including one for silence, can be used. It has been found that this is the best compromise between the precision of mouth movements and uniformity and the speed of response. The configuration file for each character simply specifies which 3D model animation target should be used for each “visema” and what audio transformation should be used. The creation of the configuration files is incorporated into the registry programs based on a predetermined list of phoneme assignments and “visemas”.

The estimation of the shapes of the lips improves if the recognition device has access to longer fragments of speech, resulting in a compromise between the precision of the lip synchronization and the latency of the response. The best compromise obtained has been a latency value of 150 ms.

An embodiment of the present invention that implements the techniques described above will be described below. In this way, the techniques will not be described again and their details will be incorporated as a reference. Figure 6 shows an example of a general computer system 1 that can form the platform for embodiments of the invention. The computer system 1 comprises a central processing unit (CPU) 5, a working memory 6 and an input interface 2 arranged to receive control inputs from a user to through an input device 22, such as a keyboard, mouse, or other control device, and an output hardware 3 arranged to provide the issuance of information to a user. The input interface 2 is further arranged to receive audio inputs from the user through an audio input device 21, such as a microphone or other device capable of capturing sounds. The output equipment 3 includes a visual display unit 32 such as a computer screen, a television, or other visual display means, and an audio output device 31 such as a speaker or other device capable of producing sound.

The computer system 1 is also provided with a storage medium 4 that can be read by a computer, such as a hard disk (HDD), a USB memory, a solid state drive, or any other form of storage of usage data. In general, in which the stored data 43, 46 and various control programs are arranged to control the computer system 1 to operate in accordance with the embodiments of the present invention. For example, the control program 40 is arranged and prepared to provide a global control of the system to carry out the embodiments of the present invention. This control program 40 may, for example, receive user inputs and launch other programs to perform specific tasks. The other programs under the control of the control program 40 include an avatar generation program 42 and a facial customization program 44 that are prepared to generate an avatar according to the specifications of the user inputs, as has been described above in the section of selection and customization of faces. A visual arrangement of different faces 46 is also arranged, from which the user can choose one and customize it. A set of predefined voices of interest 43 are arranged from which the user can choose and then select one, using a voice customization program 41 as described in the voice recording section. A real-time voice conversion program 45 according to the voice conversion technique described above is also arranged, so that the audio inputs received through the audio input device 21 are transformed into the voice of interest, such as has been customized by the user. Finally, a facial animation program 47 is arranged and prepared to animate the facial movements of the generated avatar and synchronize the movements of the avatar's lips according to the output of the voice conversion program 45 in real time. The facial animation procedure has been described above.

It should be understood that, of course, several other components and systems are known to one skilled in the art to allow the computer system 1 to function.

An application of the embodiments of the present invention, as previously described in detail, is the treatment of auditory hallucinations. The computer system, as shown in Figure 6, is capable of carrying out the audiovisual dialogue system described in the present invention and illustrated by the steps of Figure 7. A patient affected with auditory hallucinations can provide control inputs. through the input device 22 to instruct the computer system 1 to generate an avatar (step 71). The computer system 1 may comprise a storage medium 4 that can be read, including a control program 40 arranged to provide overall control of the system. The control program 40 receives the patient inputs and launches an avatar generation program 42 that is ready to generate an avatar (step 71). Next, the control program 40 can launch a face customization program 44 that allows the patient to personalize the generated avatar using a visual arrangement of different faces 46 stored in the medium 4 that can be read by the computer. Once the patient has completed facial customization, the completed avatar is presented in a visual presentation unit 32 (step 71). The control program 40 may also launch a voice personalization program 41 that is arranged to allow the patient to choose the voice of interest of the generated avatar, so that it resembles as much as possible his auditory hallucination (step 70). The voice customization program 41 uses a set 43 of predefined voices of interest that includes a range of male and female voices from which the patient can select one. Then, the patient is able to customize the parameters of the selected voice of interest, as shown in Figure 3, using a selection of sliders that appear on the input device 22 until a close resemblance to the auditory hallucination is achieved.

Once the facial and voice personalization processes are completed, the therapeutic professional and the patient can enter into conversation through the generated avatar. The therapeutic professional provides an audio input (step 72) through an audio input device 21 and the control program 40 launches a real-time voice conversion program 45 to transform the audio input, that is, the voice of the therapeutic professional in the voice of interest, using the personalized parameters chosen by the patient (step 73), in which the voice conversion program employs the voice conversion technique described above. The output of the real-time voice conversion program 45 is then provided to the patient through an audio output device 31. Simultaneously, the control program 40 launches a facial animation program 47 that is arranged to animate the facial movements of the generated avatar, namely, synchronizing the lips of the avatar in real time with the output of the voice conversion program 45 in real time (stage 74). As a result, when the therapeutic professional speaks on the audio input device 21, the avatar seems to say the words pronounced by the therapeutic professional in the voice of the patient's auditory hallucination.

Conclusions

There seems to be no other study for the treatment of auditory hallucinations using computer generated avatars. The technology used in the original pilot study was developed in a short period of time and is currently being further developed as part of a 3-year clinical trial involving a larger group of patients (140), including its extension into other languages. All the basic components are well known in the speech sector, but their combination and adaptation to suit the specific application is novel. In practice, it is relatively simple and low cost to be implemented in a clinical setting, requiring only two standard desktops connected by an audio cable. Patients quickly grasp the idea of the system and are motivated by it. From the psychiatric point of view, the introduction of an avatar allows researchers to study the relationship between patients and their voices, first hand and for the first time.

Clinical results have so far shown promise, with spectacular results in a small number of patients [9, 10]. The budgetary limitations initially allowed only a single therapeutic professional (male), however, the current study includes both men and women, thus expanding the set of available voices. The main purpose of the new study is to repeat avatar therapy in a new environment with different therapeutic professionals to establish whether therapeutic benefits can be obtained independently by any trained therapeutic professional. Other research questions include testing the effect of accuracy on the coincidence of the voice and face, the level of realism required to obtain the best therapeutic effect, if the creation of the avatar itself is therapeutic, and which subgroups of patients are more likely to benefit from therapy.

Claims (12)

1. Audiovisual dialogue system (1) comprising: an audio input device (21); an audio output device (31); a visual output device (32); Y a processor (5), the processor being arranged to: receiving (72) an audio input signal representing a source voice from the audio input device; performing (73) a substantially real-time conversion of the voice conversion into the audio input signal to produce an audio output signal representing a voice of interest, in which the audio output signal is provided to the audio output device, and in which the real-time voice conversion process includes: i) decompose the audio input signal into a set of time-varying filtering characteristics and a residual excitation signal; ii) spectrally transform the filtering characteristics that vary over time, and / or modify the tone of the residual excitation signal; Y iii) synthesize the audio output signal depending on the filtered filtering characteristics with time transformed and / or the residual excitation signal with the modified tone; generate (71) an avatar, in which the avatar is presented visually on the visual output device; and facially animate the generated avatar (74), in which the animation is synchronized with the audio output signal; in which the processor is also arranged to customize voice conversion in real time, including customization: 1) select one of a plurality of predefined voices of interest, in which the predefined voices of interest are represented by a set of respective linear transformations that include a set of time-varying filtering characteristics and a pitch scaling factor; and 2) adjust (82) the filtering characteristics variable with time and / or the scaling factor of the predefined voice tone of interest, selected to provide the customized parameters to the voice of interest, in which the filtering characteristics that vary with time and / or the scaling factor of the selected predefined voice tone of interest, are adjusted using a plurality of sliders (A, B, C, D) arranged in the interface of the user, which when activated by the user establishes a magnitude of change by which the filtering characteristics are adjusted with time and / or the pitch scaling factor. 2. System according to claim 1, wherein the time-varying filtering characteristics are calculated in short window sections of the audio input signal, wherein the short window sections of the audio input signal they last 20 to 40 ms and are superimposed from 5 to 15 ms. 3. System according to any preceding claim, wherein the time-varying filtering characteristics are transformed from Fourier with a multipoint response amplitude before the spectral transformation of the time-varying filtering characteristics, and in the that the spectrally transformed response amplitude is an inverse Fourier transform again with the filtering characteristics variable over time. 4. System, according to any of the preceding claims, in which a set of at least 8 linear transforms is generated between the audio input signal and a voice of predefined interest, and in which more than 50 voices of Predefined interests are used to generate sets of linear transformations. 5. System according to any of the preceding claims, wherein a plurality of phrases are pronounced both by the plurality of predefined voices of interest and by the audio input signal, and in which at least 20 sentences are pronounced. 6. System according to claim 5, wherein the plurality of phrases pronounced by the audio input signal and the plurality of phrases pronounced by the plurality of predefined voices of interest are temporarily aligned, in which the plurality of phrases are Temporarily aligned using Mel cepstral frequency coefficients of the audio input signal in combination with a dynamic programming algorithm. 7. System according to claim 6, wherein in the case of a plurality of signal sections within the plurality of temporarily aligned phrases: a) the prediction coefficients and the coding spectrum of the linear prediction are calculated; and b) the optimal frequency assignment is found using a dynamic programming algorithm. 8. System according to any of the preceding claims, wherein the transformation vectors of the set of linear transformations are reduced to a medium transformation vector and a plurality of orthogonal change vectors. 9. System according to claim 8, wherein a slider is used to adjust the magnitude by which a change vector is added to the average transformation vector, such that the variable filtering characteristics are adjusted with the weather. 10. System according to any of the preceding claims, wherein the processor is further arranged to facially generate the generated avatar, in which the facial customization of the generated avatar includes providing a visual arrangement with different faces for selection. 11. System according to any of the preceding claims, wherein the audio input device and the audio output device can be connected to form a two-way audio channel. 12. Audiovisual dialogue procedure comprising: receiving (72) an audio input signal representing a source voice, from an audio input device; perform (73) the conversion of the voice substantially in real time into the audio input signal to produce an audio output signal representing a voice of interest, in which the audio signal output is provided to a device audio output, and in which the conversion of voice substantially in real time includes: i) decompose (81) the audio input signal into a set of time-varying filtering characteristics and a residual excitation signal; ii) spectrally transform (82) the time-varying filtering characteristics, and / or modify the tone of the residual excitation signal; Y iii) synthesize (83) the audio output signal depending on the time-varying filtering characteristics and / or the residual excitation signal with the modified tone; generate an avatar (71), in which the avatar is displayed visually on a visual output device; and facially animate (74) the generated avatar, in which the animation is synchronized with the output audio signal; The procedure also includes the personalization of voice conversion in real time, including customization: 1) select one of a plurality of predefined voices of interest, in which the predefined voices of interest are represented by means of a set of the respective linear transformations that include a set of time-varying filtering characteristics and a scaling factor of the tone; Y 2) adjust (82) the time-varying filtering characteristics and / or the scaling factor of the predefined voice tone of interest selected to provide custom parameters to the voice of interest, in which the filtering characteristics variable in the time and / or the scaling factor of the selected predefined voice tone of interest are adjusted using a plurality of sliders (A, B, C, D) presented in a user interface that, when activated by the user, sets a magnitude of change by which the time-varying filtering characteristics and the scaling factor of the voice tone are adjusted.