GB2516965A - Synthetic audiovisual storyteller - Google Patents

Abstract

A method of animating a head and displaying text of an electronic book, whereby the head has a mouth which moves in synch with an audio sound of the text being displayed thus lip-syncing the words, the method comprises steps of inputting the text of said book; dividing said input text into a sequence of acoustic units; determining expression characteristics for the inputted text; calculating a duration for each acoustic unit using a duration model; converting said acoustic units to image vectors using a statistical model, which define a face of said head; converting said acoustic units into a sequence of text display indicators using a text display model; outputting said sequence of image vectors as video; wherein said selected expression is expressed as a weighted sum of parameters of the same type, and wherein the weighting used is expression dependent; and outputting 105 said sequence of text display indicators as video which is synchronised with the lip movement of the mouth and movement of the head and face. Sound effects may also be included

Description

Synthetic audâovisual storyteller

FIELD

Embodiments of the present invention as generally described herein relate to an synthetic audiovisual storytelling system and method.

BACKGROUND

Synthetic audiovisual storytelling systems are systems where audio speech or audio speech files are outputted along with visual images or visual image files, usually in response to the reception of a text file.

There is a continuing need to make systems more realistic and engaging.

BRIEF DESCRIPTION OF THE FIGURES

Systems and methods in accordance with non-limiting embodiments will now be described with reference to the accompanying figures in which: Figure 1 is a schematic of an audio-visual storytelling system according to an embodiment; Figure 2 is a schematic of a system showing how expression characteristics can be selected; Figure 3 is a schematic of a system showing how sound effects can be selected; Figure 4 is an image of a user interface enabling a user to select expression characteristics and sound effects; Figure 5 is an image of a further user interface enabling a user to select expression characleristics and sound elibets; Figure 6 is a schematic of a system showing how sound effects and expression characteristics can be selected for display on the user interface of figure 5; Figure? is a schematic of a system showing how text display style can be selected; Figure 8 is an image of the audio-visual storyteller display according to an embodiment; Figures 9(a) and 9(b) are images of the audio-visual storyteller display according to embodiments; Figure lOis a schematic of an audio-visual storytelling system; Figure 11 is a flow diagram showing the basic steps for rendering an animated head, text display and speech audio according to an embodiment; Figure 12(a) is an image of the generated head with user interface and 12(b) is a line 1 5 drawing of the interface; Figure 13 is a variation on the system of figure 2; Figure 14 is a further variation on the system of figure 2; Figure 15 is a schematic of a Gaussian probability distribution; Figure 16 is a schematic of the clustering data arrangement used in a method in accordance with an embodiment of the present invention; Figure 17 is a flow diagram showing the basic steps for rendering an animated head, text display, sound effects audio and speech audio according to an embodiment; Figure 18 is a variation on the system of figure 3; Figure 19 is a flow diagram denionstrating a method of training a head generation system in accordance with an embodiment of the present invenlion; Figure 20 is a schematic of decision trees used by embodiments in accordance with the present invention; Figure 21 is a flow diagram showing the adapting of a system in accordance with an embodiment of the present invention; Figure 22 is a flow diagram showing the adapting of a system in accordance with a further embodiment of the present invention; Figure 23 is a flow diagram showing the training of a system for a head generation system where the weightings are factorised; Figure 24 is a flow diagram showing in detail the sub-steps of one of the steps of the flow diagram of figure 23; Figure 25 is a flow diagram showing in detail the sub-steps of one of the steps of the flow diagram of figure 23; Figure 26 is a flow diagram showing the adaptation of the system described with reference to figure 23; Figure 27 is an image model which can be used with method and systems in accordance with embodiments; Figure 28(a) is a variation on the model of figure 27; Figure 28(b) is a variation on the model of figure 28(a); Figure 29 is a flow diagram showing the training of the model of figures 28(a) and (b); Figure 30 is a schematic showing the basics of the training described with referenec to figure 29; Figure 31 (a) is a plot of the error against the number of modes used in the image models described with reference to figures 27, 28(a) and (b) and figure 31(b) is a plot of the number of sentences used for training against the errors measured in the trained model; Figure 32(a) to (d) are confusion matrices for the emotions displayed in test data; and Figure 33 is a table showing preferences for the variations of the image model.

DETAILED DESCRIP'I'ION an embodiment, a method of animating a computer generation of a head and displaying the text of an electronic book is provided, the head having a mouth which moves in accordance with the speech of the text of the electronic book to be output by 1 5 the head and a word or group of words from the text being displayed while simultaneously being mimed by the mouth, said method comprising: inputting the text of said book.; dividing said input text into a sequence ol'acoustic units; determining expression characteristics for the inputted text; calculating a duration for each acoustic unit using a duration model; converting said sequence of acoustic units to a sequence of image vectors using a statistical model, wherein said model has a plurality of model parameters describing probability distributions which relate an acoustic unit to an image vector, said image vector comprising a plurality of parameters which define a face of said head; converting said sequence of acoustic units into a sequence of text display indicators using an text display model, wherein converting said sequence of acoustic units to said sequence of text display indicators comprises using the calculated duration of each acoustic unit to determine the timing and duration of (he display of each section of text; outputting said sequence of image vectors as video such that the mouth of said head moves to mime the speech associated with the input text with the selected expression, wherein a parameter of a predetermined type of each probability distribution in said selected expression is expressed as a weighted sum of parameters of the same type, and wherein the weighting used is expression dependent, such that converting said sequence of acoustic units to a sequence of image vectors comprises retrieving the expression dependent weights for said selected expression, wherein the parameters are provided in clusters, and each cluster comprises at least one sub-cluster, wherein said expression dependent weights are retrieved for each cluster such that there is one weight per sub-cluster; and outputting said sequence of text display indicators as video which is synchronised with the lip movement of the head.

It should be noted that the mouth means any part of the mouth, for example, the tips, jaw, tongue etc. In a further embodiment, the lips move to mime said input speech.

It should be noted that one or more computer generated heads may be animated.

1 5 It should be further noted that the text display indicators may comprise single words of the text or longer passages of the text such as sentences or paragraphs or mulfiple paragraphs.

In an embodiment, the text display indicators comprise timed subtitles. In a further embodiment, the timed subtides comprise highlighting or indicating individual words within said text, the text highlighters/indicators may comprise red dots or dots of any other colour, altering the colour of the text, altering the font or font size of the text, changing the colour of the area surrounding the text, or any other method of indicating the text which is being read. The text display indicators may comprise both text and text bighlighters/indicators.

Outputting the text display indicators as video may or may not comprise mixing a video of text and/or texL highlighters/indicators with background illustrations or background video. Outputting the text display indicators as video may comprise adjusting (he Liming of the display of background illustrations/background video according to Ihe calculated duration such that the illustrations/contents of the video is displayed in accordance with the lip movement of the head. In an embodiment, the background illustrations or

background video may or may not comprise text.

Outputting the text display indicators may comprise outputting a video simulating the pages of a book being turned as they are read.

The above head can output speech visually from the movement of the lips of the head.

In a further embodiment, said model is further configured to convert said acoustic units into speech vectors, wherein said model has a plurality of model parameters describing probability distributions which relate an acoustic unit to a speech vector, the method further comprising outputting said sequence of speech vectors as audio which is synchronised with the lip movement of the head, Thus the head can output both audio andvideo.

In an embodiment, sound effects are added to the displaying of said book. Adding sound effects comprises determining sound effects to be added to the displaying of said book; converting said sequence of acoustic units into a sequence of sound effects using a sound effects model, wherein converting said sequence of acoustic units to said sequence of sound effects comprises using the calculated duration of each acoustic unit to determine the timing and duration of the output of each sound effect; and outputting said sequence of sound effects as audio which is synehronised with the lip movement of the head. Thus, in this embodiment, the displaying of the book comprises timed text display indicators, timed sound effects and an animated head.

the input may be a text input which is divided into a sequence of acoustic units. In a further embodiment, the input is a speech input which is an audio input, the speech input being divided into a sequence of acoustic units and output as audio with the video of the head. Once divided into acoustic units the model can be run to associate the acoustic units derived from the speech input with image vectors such that the head can be generated to visually output the speech signal along with the audio speech signal.

In an embodiment, each sub-cluster may comprise at least one decision tree, said decision tree being based on questions relating to at least one of linguistic, phonetic or prosodic differences. There may be difference in the structure between the decision trees of the clusters and between trees in the sub-clusters. I'he probability distributions may be selected from a Gaussian disnibution, Poisson distribution, Gamma distribution, Student-t distribution or Laplacian distribution.

The expression characteristics may be selected from at least one of different emotions, accents or speaking styles. Variations to the speech.vill often cause subtle variations to the expression displayed on a speaker's face when speaking and the above method can be used to capture these variations to allow the head to appear natural. Variations to the speech may cause variations to the duration of each acoustic unit. The timing of the display of the text indicators and output of the sound effect audio varies in accordance with the expression characteristics.

1 0 In one embodiment, determining expression characteristics comprises providhg an input to allow the weightings to be determined via the input. Also, determining expression characteristics comprises predicting from the speech to be outputted the weightrngs which should be used. In a yet further embodiment, selecting expression characteristic comprises predicting from external information about the speech to be 1 5 output, the weightings which should be used.

It is also possible for the method to adapt to a new expression characteristic. For example, determining expression comprises receiving a video input containing a face and varying the weightings to simulate the expression characteristics of the face of the video input.

Where the input data is an audio tile containing speech, the weightings which are to be used for controlling the head can be obtained from the audio speech input.

In a further embodiment, selecting an expression characteristic comprises randomly selecting a set of weightings from a plurality of pre-stored sets of weightings, wherein each set of weightings comprises the weightings for all sub-clusters.

the image vector comprises parameters which allow a lhce to be reconstructed from these parameters. In one embodiment, said image vector comprises parameters which allow the face to be constructed from a weighted sum of modes, and wherein the modes represent reconstructions of a face or part thereoL In a further embodiment, the modes comprise modes to represent shape and appearance of the face. the same weighting parameter may be used for a shape mode and its corresponding appearance mode.

The modes may be used to represent pose of the face, deformation of regions of the face, blinking etc. Static features of the head may be modelled with a fixed shape and textuTe.

In one embodiment, determining sound effects comprises providing an input to allow the sound effects to be selected via the input. In another embodiment, determining sound effects may comprise predicting from the speech to be outputted the sound effects which should be used. In another embodiment, determining sound effects may comprise predicting from the input text the sound elTects which should be used.

In an embodiment the determining of expression characteristics for the inputted text is performed by a user via a user interface and comprises selecting expression characteristics for one or more acoustic units of said text, wherein said one or more acoustic units are selected from phones, diphones, syllables, moras, words, accent feet, international phrases and sentences. In another embodiment the determining of sound effects for the inputted text is performed by a user via a user interface and comprises selecting sound effects for one or more acoustic units of the text, wherein the one or more acoustic units are selected from phones, diphones, syllables, moras, words, accent feet, international phrases and sentences.

The expression characteristics and/or sound effects are assigned to different portions of the input text by the user via the user interface. In an embodiment, the user interface is a text-based interface. In another embodiment, the user interface is a timcline interface. In an embodiment, the expression characteristics and/or sound effects for each portion of text are predicted from the input text. In an embodiment, the predictions are displayed to the user via the user interface. In a further embodiment, the user may manually adjust the expression characterislics and/or sound effects predicted froni the input text via the user interface.

The user interface enables a human editor to define the expression and characteristics of the synthetic speech and/or the synthetic face or faces for each part of the text of the e-book at any desired level ranging from phone to the whole sentence; sound effects which help the listener to locate the speaker in a specific scenario and position, such as panning, equalization, etc; and sound props. The user interface may allow a fine time control of each audio source. The interface may provide an estimation of characters and/or expressions to be associated with each character on receipt of a standard format input text. The interface may be translated into a standard theatre script with annotations.

In an embodiment, the text is displayed in accordance with a determined text display style. Examples of different styles include reproducing the effect of turning the pages of a book, timed subtitles, using word highlighters as described above or any other style of displaying the text of a book. In one embodiment, determining the text display style comprises providing an input to allow the text display style to be selected via the input.

In an embodiment, the sequence of image vectors may be converted into a set of image parameters. In a further embodiment, the sequence of text display indicators may be converted into a set of text display parameters. In yet a further embodiment, the set of image parameters and said set of text display parameters may be output as a computer readable code configured to cause a computer to output said sequence of image vectors as video such that the mouth of said head moves to mime the speech associated with the input text with the selected expression and further to output said sequence of text display indicators as video which is synchronised with the lip movement of the head. In an embodiment, the sequence of speech vectors may be converted into a set of speech parameters which are output as a computer readable code configured to cause a computer to output said sequence of speech vectors as audio. In yet another embodiment, the sequence of sound effects may be converted into a set of sound effects parameters which are output as a computer readabJe code configured to cause a computer to output said sequence of sound effects as audio. In an embodiment, the sequence of image vectors, the sequence of text display vectors, the sequence of speech vectors and the sequence of sound effects are all converted into a set of parameters.

The set of parameters may be reduced. For example, they may be reduced according to the VPEG4 compression algorithm or any other compression algorithm. Ihe reduced number of parameters may be retrievable by users via a communication system. The reduced number of parameters may enable a portable device to generate the hill audio-visual book.

In an embodiment, a system for animating a computer generation of a head and displaying the text of an dectronic book is provided, such that the lips of the computer generated head move to mime the speaking of the text of the book, the display of said text being synchronized with said miming of the speaking of the text of the book by the lips of the computer generated head such that the display changes in accordance with the text being spoken, the system comprising: an input for receiving text of said book; a processor configured to: divide said input into a sequence of acoustic units; determine expression characteristics for the inputted text; calculate a duration for each acoustic unit using a duration model; convert said sequence of acoustic units to a sequence of image vectors using a statistical model, wherein said model has a plurality of model parameters describing 1 5 probability distributions which relate an acoustic unit to an image vector, said image vector comprising a plurality of parameters which define a face of said head; convert said sequence of acoustic units into a sequence of text display indicators using an text display model, wherein converting said sequence of acoustic units to said sequence of text display indicators comprises using the calculated duration of each acoustic unit to determine the timing and duration of the display of each section of text; output said sequence of image vectors as video such that the mouth of said head moves to mime the speech associated with the input text with the selected expression; and output said sequence of text display indicators as video which is synchronised with the lip movement of the head, wherein a parameter of a predetermined type of each probability distribution in said selected expression is expressed as a weighted sum of parameters of the same type, and wherein the weighting used is expression dependent, such that converting said sequence of acoustic units to a sequence of image vectors comprises retrieving the expression dependent weights for said selecled expression, wherein the parameters are provided in clusters, and each cluster comprises at least one sub-cluster, wherein said expression dependent weights are retrieved for each cluster such that there is one weight per sub-cluster.

In an embodiment, an adaptable system for animating a computer generated head and displaying the text of an electronic book is provided, such that the lips of the computer generated head move to mime the speaking of the text of the book, the display of said text being synchronized with said miming of the speaking of the text of the book by the lips of the computer generated head such that the display changes in accordance with the text being spoken, the system comprising: an input for receiving text of said book; a processor configured to: divide said input into a sequence of acoustic units; determine expression characteristics for the inputted text; calculate a duration for each acoustic unit using a duration model; convert said sequence of acoustic units to a sequence of image vectors using a statistical model, wherein said model has a plurality of model parameters describing probability distributions which relate an acoustic unit to an image vector, said image 1 5 vector comprising a plurality of parameters which define a face of said head; convert said sequence of acoustic units into a sequence of text display indicators using an text display model, wherein converting said sequence of acoustic units to said sequence of text display indicators comprises using the calculated duration of each acoustic unit to determine the timing and duration of the display of each section of text; output said sequence of image vectors as video such that the mouth of said head moves to mime the speech associated with the input text with the selected expression; and output said sequence of text display indicators as video which is synchronised with the lip movement of the head, wherein a parameter of a predetermined type of each probability distribution in said selected expression is expressed as a weighted sum of parameters of the same type, and wherein the weighting used is expression dependent, such that converting said sequence of acoustic units to a sequence of image vectors comprises retrieving the expression dependent weighis for said selecled expression, wherein the parameters are provided in clusters, and each cluster comprises at least one sub-cluster, wherein said expression dependent weights are retrieved for each cluster such that there is one weight per sub-cluster; the system further comprising a memory configured to store the said parameters provided in clusters and sub-clusters arid the weights for said sub-clusters, the system being ftirther configured to receive a new input video file; the processor being configured to re-calculate the weights applied to the sub-clusters to maximise the similarity between the generated image and the new video file.

The above generated head may be rendered in 2D or 3D. For 3D, the image vectors define the head in 3 dimensions. In 3D, variations in pose are compensated for in the 3D data. However, blinking and static features may be treated as explained above.

Since some methods in accordance with embodiments can be implemented by software, 1 0 some embodiments encompass computer code provided to a general purpose computer on any suitable carrier medium. The carrier medium can comprise any storage medium such as a floppy disk, a CD ROM, a magnetic device or a programmable memory device, or any transient medium such as any signal e.g. an electrical, optical or microwave signal.

In an embodiment, a system to read c-books is provided that combines illustrations as fixed background image; synthetic speech; a video of one or more synthetic faces talking in synchronization with the synthetic speech; and the reading of the text, in synchronization with both the speech and lip movement of the synthetic face or faces.

The system may or may not additionally provide sound effects.

Figure 1 is a schematic of a system for audio-visual storytelling according to an embodiment. In this embodiment, the audio-visual storytelling system produces a video comprising text display, audio of the text spoken with expression and one or more computer generated heads each with a mouth which moves to mime speaking of the text with expression.

The system comprises an expressive taking head video and speech generator 1 and a video mixer 103.

The input to the expressive talking head video and speech generator 1 comprises text, and expression/style parameters. In an embodiment, the input to the expressive talking head video and speech generator 1 further comprises sound effect parameters.

The expression/style parameters comprise data relating to the expression, if any, of the speech and talking head. The expression can be selected from happy, sad, neutral, angry, afraid, tender etc, or a combination thereof. In further embodiments, the expression can relate to the speaking style of the user, for example, whispering, shouting etc., or the accent of the user. In yet further embodiments, the expression can relate to different characters of the book.

In one embodiment, the expression/style parameters may or may not comprise expression weights. Expression weights will be described in more detail with relation figure 17.

In an embodiment, the expression/style parameters comprising data relating to the expression of the speech and talking head are determined by the user via a user interface. A simple schematic system in accordance with such an embodiment is shown 1 5 in Figure 2. Here, the display 253 shows different expressions which may be selected by selecting unit 251.

The sound effect parameters comprise data relating to sound effects. The sound ellects may help the listener to locate the speaker in a specific scenario and position by using techniques such as panning and equalization. Ihese techniques are well known in the art and will not be discussed here, the sound effects may comprise sound props. The sound effects may comprise music, sound effects relevant to the text of the book or any other type of sound. In an embodiment, the output of the sound is timed so as to occur in accordance with the speaking and/or miming of a relevant portion of text. For example, if part of the story takes place on a river bank, the accompanying sound effect might comprise the sound of a flowing river, In an embodiment, sound effect parameters comprising data relating to the sound effects are determined by the user via a user interface. Figure 3 shows a simple schenialic system in accordance with such an embodiment. Here, the display 553 shows different styles of sound effects which may be selected by selecting unit 551.

In an embodiment, a single user interface is used to select parameters relating to expression and sound effects. In an embodiment the user interface comprises a text-based editor. An example of such an interface according to an embodiment is shown in Figure 4. In the embodiment of figure 4. the user interface comprises a table of options 601 available to the editor and the text editor itself 613. In this embodiment, the table of options comprises options for the character speaking the text 615, emotions with which the selected character is speaking 603, sound effects related to particular events 607 and background sound effects 605. The text editor comprises the text 617 itself to which the user can add options from the table of options 601. In the example of figure 4, the user has selected to add sound effects to the first paragraph 609. The background sound effect of water has been selected for the entire paragraph, indicated by the shading of the text. Further, at the end of the paragraph, the user has selected to add an event sound effect of shaking wet clothes. This is indicated in the text editor by the tilde symbol 611.

In another embodiment, the user interface comprises a timeline editor, for example a sequencer-like editor interface. An example of a timeline editor according to an embodiment is shown in Figure 5. The user interface comprises options table 707 and timeline 701. In this embodiment, the options table 707 comprises options for the character speaking the text 715, emotions with which the selected character is speaking 709, sound efTecis related to particular events 717 and background sound effects 719.

The timeline 701 comprises components for each type of audio output. For example, the timeline of the embodiment of figure 5 comprises two components 703 relating to sound effects and three components 705 corresponding to characters in the story. The user is able to divide the text 713 between the characters according to the desired output. In this embodiment, the user is provided with three pairs of opposite expressions: happy/sad, stressed/relaxed and aggressive/peaceful, as indicated in table 709. The user is able to control the balance within each pair of emotions by adjusting graphs 711 corresponding to each segment of text. Further, the left/right equalization of the audio output may be adjusted by the user for each segment of text or each sound effect via a second set of aphs 713.

I'hus, in this embodiment, the user interface is configured such that an editor may define the expression and characteristics of the speech and/or of the computer generated head for any and/or each part of the text at any desired acoustic level ranging from a phone to a whole sentence. In an embodiment, the user interface is further configured to allow an editor to choose sound effects for any and/or each part of the text at any desired acoustic level ranging from a phone to a whole sentence. The sound effects may be adjusted to help the listener to locate the speaker in a specific scenario and position by using techniques such as panning and equalization. These techniques are well known in the art and will not be discussed here. The sound effects may further comprise sound props.

By enabling the user to select expression characteristics for the voice and the sound effects at any acoustic level, the interface according to the above embodiment enables a tine time control of each audio source.

In an embodiment, the user interface is provided with a memory which saves predetermined rules for adding expression and speech characteristics and sound effects to the text. Such a system is shown in figure 6. For example, the user interface may recognise from the text when something is being spoken by a character in the book as 1 5 opposed to the narrator, for example from quotation marks, and indicate this on the timeline 701 of the user interface. Further, the system may be configured to recognise if the text refers to a happy moment, or an anxious moment and indicate this on the timeline.

In the above system as shown in tigure 6, a memory 861 is provided which stores the attributes and rules to be checked in the text. Ihe input text is provided by unit 863 to memory 861, The rules for the text are checked and information concerning the type of expression or sound effect are then passed to display unit 865. Display unit 865 then displays the expression or sound effect on the timeline.

In an embodiment, the user may optionally adjust the expression or sound effects added to the timeline by the system. The user may also add to the expression or sound effects added to the tilneline by the system.

In an embodiment, the interface according to the above embodiment is able, upon input of a standard formal input text, to automalically provide a rough eslimation of characters and/or expressions to be associated with each character in a story.

Returning now to figure 1, in addition to the parameters described above, the total time may be input into the expressive taking head video and speech generator 1. This enables the user to determine the total length of the duration of the talking head video so that it could, for example be synchronized with an external event, such as an audio effect, a video sequence, etc. In an embodiment, the expressive taking head video and speech generator 1 uses these inputs to generate duration data, face video and spcech audio and, in an embodiment, sound effects audio. In another embodiment, the expressive talking head video and 1 0 speech generator I uses these inputs to generate text display video, face video, speech audio and, in an embodiment, sound effects audio. The process by which this is carried out will be described in more detail below with relation to figure 11.

The face video, speech audio, text display video or duration data and sound effects audio which are output from the talking head video and speech generator 1 are subsequently input into the video mixer 103. In addition to the face video, speech audio, text display video or duration data and sound effects audio, the input to the video mixer 103 comprises fixed illustration or background video. The fixed illusiration or background video comprises the background image for the electronic book. For example, the background may comprise the original illustrations from the physical book. The video mixer 103 combines the inputs into a single video of the audio-visual book.

Connected to the video mixer 103 is output is audiovisual output 105. The output 105 comprises a display 107 which displays the generated video of the audio-visual book.

A still from a video of an audio-visual book according to one embodiment is shown in figure 8. In this embodiment, the video comprises background illustration 31, computcr generaled, animated, expressive head 33 and text display 37. The head reads the text 37 in an expressive matter and the speech is output as audio. When the head finishes reading the passage comprising the text 37, the text display will change. In an embodiment, the background illustration 31 also changes when the head finishes reading the passage comprising the text 37. thus the text display comprises timed subtitles for the animated head.

In the embodiment, of figure 8, the text display additionally comprises red dots 35.

These red dots are text highlighters which indicate the individual word being spoken by the computer animated head at any one time. As each word of the text 37 is output as audio andlor mimed by the talking head, a red dot above the corresponding word in the text is highlighted or becomes visible. For example, as the talking head 33 mines the word "home", red dot 39 appears. Again, the text display comprises timed subtitles for the animated head but in this embodiment, single words are indicated as opposed to entire passage. In this embodiment, the text indicators are red dots. However, the text indication could also comprise altering the colour of the text, altering the font or font size of the text, changing the colour of the area surrounding the text or any other method of indicating the text that is being read.

The head when animated moves its lips in accordance with the speech to be output and 1 5 displays the desired expression. The head also has an audio output that outputs an audio signal containing the speech. Additional sound effects may be output by the audio output.

[he text is displayed in accordance with the speech to be output and the lip movement of the animated head. The audio speech, lip movement of the head, text display and audio effects are synchronized.

In an embodiment, the expressive talking head and speech generator 1 outputs a duration data file to the video mixer 103. The duration data is used by the video mixer to ensure that the illustrationlbackground video is synchronised with the reading of the text by the computer animated head.

In a further embodiment, the illustrationlbackground video comprises text. In this embodiment, the duration data is used by the video mixer to ensure that the text display as part of the illustration/background video is synchronised with the reading of the text by the computer animated head..

In another embodiment, the expressive talking head and speech generator I outputs text display video to the video mixer 103.

In one embodiment, the text display video comprises text indicators or highlighters such Figure 8 above. The video mixer combines the text display video and the background illustration/video such that the final video comprises text indicators or highlighters indicating the text synchronised with the illustration/background video and reading of the text by the computer animated head.

In another embodiment, the text display video comprises text. The video mixer combines the text display video with the background illustration/video such that the 1 0 final video comprises the display of text synchronised with the backwound illustration/video and reading of the text by the computer animated head.

In yet another embodiment, the text display video comprises both text and text indicators/highlighters. The video mixer combines the text display video with the 1 5 background illustration/video such that the final video comprises text and text highlighters/indicators synchronised with the background illustration/video and reading of the text by the computer animated head.

in an embodiment, the style of text display may be determined by the user via a user interface. Figure 7 shows a schematic system in accordance with such an embodiment.

Here the display 1053 shows different styles of text display which may be selected by selecting unit 1051. The text display parameters may then be input into the expressive talking head video and speech generator I. In an embodiment, face location parameters may be optionally entered as inputs into the video mixer in order to control the position of the computer generated talking head.

Two examples of possible positioning of the talking head are shown in figures 9(a) and 9(b). In another embodiment, the face location parameters are pre-programmed into the video mixer.

Suitable video mixers are well known in the an and will not be described here.

A schematic of an expressive talking head video and speech generator system 1 according to an embodiment is shown in Figure 10. The system 1 comprises a processor 3 which executes a program 5. System I further comprises storage or memory 7. The storage 7 stores data which is used by program S to convert text to convert speech and/or render a talking head and render the text display or produce the duration data file. In an embodiment, the storage 7 also stores data which is used by program S to convert text to sound effects. The storage 7 may further comprise data which is used by program S to convert text to speech and/or render a talking head and render the display of text or produce the duration data file. In an embodiment, the storage 7 further comprises data which is used by program 5 to convert text to sound effects. System 1 further comprises an input module 11 and an output module 13. The input module 11 is connected to an input for data relating to text, emotion or expression with which the text is to be output and style with which the text is to be displayed. The type of data which is input may take many forms which will be described in more detail later. The data which is input may comprise text. The input 15 may be an interface which allows a user to directly input data such as those described above in relation to figures 4 and 5.

Alternatively, the input may be a receiver for receiving data from an external storage medium or a network.

Connected to the output module 13 is output 17.

In use, the system 1 receives data through data input 15. The program 5 executed on processor 3 converts inputted data into text display or duration data, speech to be output by the head, and the expression which the head is to display. In an embodiment, the program 5, executed on processor 3, also converts inputted data into sound effects to be output. The program accesses the storage to select parameters on the basis of the input data. The program renders the head, converts the data to speech and produces the duration data file. In an embodiment, the program also renders the sound effects. The head when animated moves its lips in accordance with thc speech to be output and displays the desired expression. the head also has an audio output which outputs an audio signal containing the speech. The audio speech is synchronised with the lip movement of the head. The text is displayed in accordance wiih the speech to be output.

The text display is synchronised with the lip movement of the head and the output of speech such that, as a portion of speech is output or mimed by the lips of the head, the corresponding portion of text is displayed. In an embodiment, sound effects are synchronised with speech being output or mimed by the lips of the head such that sound effects appropriate to the contents of the speech are output as that portion of speech is output.

In an embodiment, the output 17 comprises speech audio, head video and duration data.

In another embodiment, the output 17 comprises speech audio, head video and text display video. In yet another embodiment, the output 17 comprises sound effect audio.

In another embodiment, the output 17 does not comprise visual or audio output but 1 0 instead comprises data relating to the speech, head, and text display or duration which are output to a data file which may be sent to storage, networked. etc. In an embodiment, the output 17 further comprises data relating to sound effects which are output to a data file which may be sent to storage, networked, etc. 1 5 Figure 11 is a schematic of the basic process for animating and rendering the head, producing the speech audio and duration data file or text display video according to an embodiment. In step S201. an input is received which relates to the speech to be output by the talking head and will also contain information re'ating to the expression that the head should exhibit while speaking the text.

In this specific embodiment, the input will be text. In figure 11 the text is separated from the expression input.

However, the input does not need to be a text input, it can be any type of signal which allows the output of speech. For example, the input could be selected from speech input, video input, combined speech and video input. Another possible input would be any form of index that relates to a set of face/speech already produced, or to a predefined Lcxt/cxprcssion, e.g. an icon to makc the system say "please" or "I'm sorry".

For the avoidance of doubt, it should be noted that by outputting speech, the lips of the head move in accordance with the speech to be outputted. However, the volume of the audio output may be silent. In an embodiment, there is just a visual representation of the head miming the words where the speech is output visually by the movement of the lips. In further embodiments, this may or may not be accompanied by an audio output of the speech.

For the avoidance of doubt, it should be further noted that the timing of the text display is in accordance with the speech to be outputted and the movement of lips of the head which mime the speech to be outputted. For example, the display of a word of the text may be timed to occur as the word is output as speech and'or mimed by the lips of the head. However, the volume of the audio output may be silent or the lips of the head may be still.

When text is received as an input, it is then converted into a sequence of acoustic units which may be phonemes, graphemes, context dependent phonemes or graphemes and In one embodiment, additional information is given in the input to allow expression to be selected in step S205. In one embodiment, this information is output from one of the user interface system described in relation to figure 2, 4 or S above. This then allows the expression weights which will be described in more detail with relation to figure 16 to be derived in step S207.

In some embodiments, steps S205 and S207 are combined. This may be achieved in a number of different ways. For example, Figure 12 shows an interface for selecting the expression. Here, a user directly selects the weighting using, for example, a mouse to drag and drop a point on the screen, a keyboard to input a figure etc. In figure 12(b), a selection unit 1251 which comprises a mouse, keyboard or the like selects the weightings using display 1253, Display 1253, in this example has a radar chart which shows the weightings. The user can use the selecting unit 1251 in order to change the dominance of the various clusters via the radar chart. It will be appreciated by those skilled in the art that other display methods may be used in the interface, in sonic embodiments, the user can directly enter text, weights for emotions, weights for pitch, speed and depth. In an embodiment, weight selection such as that described above is integrated into one of the user interfaces described in relation to Figure 2, 4 or 5.

Pitch and depth can affect the movement of the face since that the movement of the face is different when the pitch goes too high or too low and in a similar way varying the depth varies the sound of the voice between that of a big person and a little person.

Speed can be controlled as an extra parameter by modifying the number of frames assigned to each model via the duration distributions.

Figure 12(a) shows the overall unit with the generated head. The head is partially shown with as a mesh without texture. in normal use, the head will be fully textured.

In a further embodiment, the system is provided with a memory which saves predetermined sets of weightings vectors. Each vector may be designed to allow the text to be outputted via the head using a different expression. The expression is indicated by the head and also is manifested in the audio output.

Such a system is sho in figure 13. For example, the system may need to output speech via the talking head corresponding to text which it recognises as being a command or a question. The system may recognise from the text when something is being spoken by a character in the book as opposed to the narrator, for example from quotation marks, and change the weighting to introduce a new expression to be used in the output. Similarly, the system may be configured to recognise if the text is repeated.

In such a situation, the voice characteristics may change for the second output. Further the system may be configured to recognise if the text refers to a happy moment, or an anxious moment and the text outputted with the appropriate expression. This is shown schemafically in step S911 where the expression weights are predicted directly from the text.

In the above system as shown in figure 13, a memory 261 is provided which stores the attributes and rules to be checked in the text. The input text is provided by unit 263 to memory 261. The rules for the text are checked and information concerning the type of expression are then passed to selector unit 265. Selection unit 265 then looks up the weightings for the selected expression.

In a further embodiment, the system receives information about how the head should output speech from a further source. An example of such a system is shown in figure 14. For example, the system may receive inputs indicaling how certain parts of the text should be outputted.

In the system of figure 14, the further information on how the head should output speech is received from unit 271. Unit 271 then sends this information to memory 273.

Memory 273 then retrieves information concerning how the voice should be output and send this to unit 275. Unit 275 then retrieves the weightings for the desired output from the head.

In a further embodiment, speech is directly input at step S209. Here, step S209 may comprise three sub-blocks: an automatic speech recognizer (ASR) that detects the text from the speech, and aligner that synchronize text and speech, and automatic expression recognizer. The recognised expression is converted to expression weights in S207. The recognised text then flows to text input 203. This arrangement allows an audio input to 1 5 the talking head system which produces an audio-visual output. This allows for example to have real expressive speech and from there synthesize the appropriate face for it.

In a further embodiment, input text that corresponds to the speech could be used to improve the performance of module S209 by removing or simplifying the job of the ASR sub-module.

In step S213, the text and expression weights are input into an acoustic model which in this embodiment is a cluster adaptive trained HMM or CAT-HMM.

The text is then converted into a sequence of acoustic units. These acoustic units may be phonemes or graphemes. The units may be context dependent e.g. triphones, quinphones etc. which take into account not only the phoneme which has been selected but the proceeding and following phonemes, the position of the phone in the word, the number of syllables in the word the phone belongs to, etc. Ihe text is converted into the sequence of acoustic units using teelmiques which are well-known in the art and will not be explained further here.

There are many models available for generating a face. Some of these rely on a parameterisation of the face in terms of, for example, key points/features, muscle structure etc. Thus, a face can be defined in terms of a "face" vector of the parameters used in such a face model to generate a face. This is analogous to the situation in speech synthesis where output speech is generated from a speech vector. In speech synthesis, a speech vector has a probability of being related to an acoustic unit, there is not a one-to-one correspondence. Similarly, a face vector only has a probability of being related to an acoustic unit. Thus, a face vector can be manipulated in a similar manner to a speech vector to produce a talking head which can output both speech and a visual representation of a character speaking. Thus, it is possible to treat the face vector in the same way as the speech vector and train it from the same data.

The probability distributions are looked up which relate acoustic units to image parameters. In this embodiment, the probability distributions will be Gaussian distributions which are defined by means and variances. Although it is possible to use other distributions such as the Poisson. Student-t, Laplacian or Gamma distributions some of which are defined by variables other than the mean and variance.

Considering just the image processing at first, in this embodiment, each acoustic unit does not have a definitive one-to-one correspondence to a "face vector" or "observation" to use the terminology of the art. Said face vector consisting of a vector of parameters that define the gesture of the face at a given frame. Many acoustic units are pronounced in a similar manner, are affected by surrounding acoustic units, their location in a word or sentence, or are pronounced differently depending on the expression, emotional state, accent, speaking style etc of the speaker. Thus, each acoustic unit only has a probability of being related to a face vector and text-to-speech systems calculate many probabilities and choose the most likely sequence of observations given a sequence of acoustic units.

A Gaussian distribution is shown in figure 15. Figure 15 can be thought of as being the probability distribution of an acoustic unit relating to a face vector. For example, the speech vector shown as X has a probability Pt of corresponding to the phoneme or other acoustic unit which has the distribution shown in figure 15.

The shape and position of the Gaussian is defined by its mean and variance. These parameters are determined during the training of the system.

These parameters are then used in a model in step S2 13 which will be termed a "head model". The "head model" is a visual or audio visual version of the acoustic models which arc used in speech synthesis. in this description, the head model is a Hidden N larkov Model (HMM). However, other models could also be used.

The memory of the talking head system will store many probability density functions relating to an acoustic unit i.e. phoneme, grapheme, word or part thereof to speech parameters. As the Gaussian distribution is generally used, these are generally referred to as Gaussians or components.

In a Hidden Markov Model or other type of head model, the probability of all potential face vectors relating to a specific acoustic unit must be considered. Then the sequence of face vectors which most likely corresponds to the sequence of acoustic units will be taken into account. This implies a global optimization over all the acoustic units of the sequence taking into account the way in which two units affect to each other. As a result, it is possible that the most likely face vector for a specific acoustic unit is not the best face vector when a sequence of acoustic units is considered.

In the flow chart of figure ii, a single stream is shown for modelling the image vector as a "compressed expressive video model". In some embodiments, there will be a plurality of different states which will each be modelled using a Gaussian. For example, in an embodiment, the talking head system comprises multiple streams. Such streams might represent parameters for only the mouth, or only the tongue or the eyes, etc. The streams may also be further divided into classes such as silence (sil), short pause (pau) and speech (spe) etc. in an embodiment, the data from each of the streams and classes will be modelled using a HMM. The 11MM may comprise different numbers of states, for example, in an embodiment, 5 state HMMs may be used to model the data from some of the above streams and classes. A Gaussian component is determined for each HMM state.

The above has concentrated on the head outputting speech visually. However, the head may also output audio in addition to the visual output. Returning to figure 12, the "head model" is used to produce the image vector via one or more steams and in addition produce speech vectors via one or more streams. In figure 11, two audio streams are shown which are spectrum and Log FO.

Cluster adaptive training is an extension to hidden Markov model text-to-speech (HMM-TTS). HMM-flS is a parametric approach to speech synthesis which models context dependent speech units (CDSU) using HMMs with a finite number of emitting states, usually five. Concatenating the HIMIVs and sampling from them produces a set of parameters which can then be re-synthesized into synthetic speech. Typically, a decision tree is used to cluster the CDSU to handle sparseness in the training data. For any given CDSU the means and variances to be used in the HMMs may be looked up using the decision tree.

CAT uscs multiple dccision trccs to capture style-or emotion-dependent information.

This is done by expressing each parameter in terms of a sum of weighted pararnctcrs where the weighting k is derived h-nm step S207. The parameters arc combined as shown in figure 16.

Thus, in an embodiment, the mean of a GalLssian with a selected expression (for either speech or face parameters) is expressed as a weighted sum of independent means of the (Jaussians. =

Eqn. 1 where is the mean of component in in with a selected expressions, {i p} is the index for a cluster with P the total number of clusters, 25S) is the expression dependent interpolation weight of the 11h oJuster for the expression s; is the mean for component in in cluster /. In an embodiment, one of the clusters, for example, cluster i1, all the weights are always set to 1.0. This cluster is c1ed the bias cluster'. Each cluster comprises at least one decision tree. There will be a decision tree for each component in the cluster. In order to simplify the expression, c (rn,i) {i N} indicates the general leaf node index for the component iii in the mean vectors decision tree for cluster /1, with N the total number of leaf nodes across the decision trees of all the clusters. The details of the decision trees will he explained later.

For the head model, the system looks up the means and variances which will be stored in an accessible manner. The head model also receives the expression weightings from step S207. It will be appreciated by those skilled in the art that the voice characteristic dependent weightings may be looked up before or after the means are looked up.

The expression dependent means i.e. using the means and applying the weightings, are then used in a head model in step.S2 13.

The face characteristic independent means are clustered. In an embodiment, each cluster comprises at least one decision tree, the decisions used in said trees are based on linguistic, phonetic and prosodic variations. In an embodiment, there is a decision tree for each component which is a member of a cluster. Prosodic, phonetic, and linguistic contexts affect the facial gesture. Phonetic contexts typically affects the position and movement of the mouth, arid prosodic (e.g. syllable) and linguistic (e.g., part of speech of words) contexts affects prosody such as duration (rhythm) and other parts of the face, e.g., the blinking of the eyes. Each cluster may comprise one or more sub-clusters where each sub-cluster comprises at least one of the said decision trees.

The above can either be considered to retrieve a weight for each sub-cluster or a weight vector for each cluster, the components of the weight vector being the weightings for each sub-cluster.

l'he following configuration may be used in accordance with an embodiment of the present invention. To model this data, in this embodiment, 5 state HMMs are used. The data is separated into three classes for this example: silence, short pause, and speech. In this particular embodiment, the allocation of decision trees and weights per sub-cluster are as follows.

In this particular embodiment the following streams are used per cluster: Spectrum: 1 stream, 5 states, 1 tree per state x 3 classes LogFO: 3 streams, S states per stream, 1 tree per state and stream x 3 classes BAP: 1 stream, 5 states, 1 tree per state x 3 classes VID: 1 stream, 5 states, 1 tree per state x 3 classes Duration: I stream, 5 states, I tree x 3 classes (each tree is shared across all states) Total: 3x3 I = 93 decision trees For the above, the following weights are applied to each stream per expression characteristic: Spectrum: 1 stream, 5 states, I weight per stream x 3 classes LogFO: 3 streams, 5 states per stream, 1 weight per stream x 3 classes BAP: 1 stream, 5 states, 1 weight per stream x 3 classes \TJD: 1 stream. 5 states, 1 weight per stream x 3 classes Duration: 1 stream, S states, 1 weight per state and stream x 3 classes Total: 3x1 I = 33 weights.

As shown in this example, it is possible to allocate the same weight to different decision trees (VID) or more than one weight to the same decision tree (duration) or any other combination. As used herein, decision trees to which the same weighting is to be applied are considered to form a sub-cluster.

In one embodiment, the audio streams (spectrum, logF0) are not used to generate the video of the talking head during synthesis but are needed during training to align the audio-visual stream with the text.

The following table shows which streams are used for alignment, video and audio in accordance with an embodiment of the present invention.

Stream Used for Used for Used for l..Jsed for Used for alignment face video speech audio text sound effect synthesis synthesis display a.udi.o synthesis synthesis Spectrum Yes No Yes No No 1.c)gFO Yes No Yes No No BA.P No!o Yes (but: No No may he omitted) \ID No Yes No No No Duration Yes Yes Yes Yes Yes In an embodiment, the mean of a Gaussian distribution with a selected voice characteristic is expressed as a weighted sum of the means of a Gaussian component, where the summation uses one mean from each cluster, the mean being selected on the basis of the prosodic, linguistic and phonetic context of the acoustic unit which is currently being processed.

The training of the model used in step S213 will be explained in detail with reference to figures 19 to 21. Figure 11 shows a simplified model with four streams, 3 related to producing the speech vector (1 spectrum, 1 Log FO and 1 duration) and one related to the face/VU) parameters.. The duration stream is also related to producing the text display indicators. (However, it should he noted from above, that many embodiments will use additional streams and multiple streams may be used to model each speech or video parameter. For example. in this figure BAP stream has been removed for simplicity. This corresponds to a simple pulse/noise type of excitation. However the mechanism to include it or any other video or audio stream is the same as for represented streams.) These produce a sequence of speech vectors, a sequence of face vectors and a set of duration data which are output at step 5215.

The speech vectors (including the corresponding duration data) are then fed into the speech generation unit in step S2l 7 which converts these into a speech sound file at step S219. The face vectors are then fed into face image generation unit at step S221 which converts these parameters to video in step S223.

In one embodiment, the duration data is converted into a duration data file in step S23 1.

The face video, speech audio and duration data are combined with illustrationibackground video by a video mixer, as described above in relation to figure I. The video mixer subsequently outputs a video to a display of the electronic book. The duration data input into the video mixer ensures that the transition between background illustrations or duration of the background video is synchronised with the face video and speech audio such that when a particular passage is spoken, the corresponding section of the background video or corresponding background illustration is displayed. In an embodiment, this background video/illustration comprises the text of the book itself In another embodiment, the duration data file is not output to the video mixer but is fimrther fed into the text display generation unit in step S233. In the text display generation unit, the duration data is converted into a text display video in step S24 1. As described above, in this embodiment, the text display video comprises the text itself and/or text indicators, In this embodiment, the face video, speech audio and text display video are combined with ilIust.rationIbackound video by a video mixer. The video mixer subsequently outputs a video to a display of the electronic book.

In anothcr embodiment. the video, specch and duration parameters are output to a data file. This file may be sent to storage, networked, outpuL to a compuLer readable device, etc. In an embodiment, the file may be used to cause a portable device to generate a display of the electronic book. In an embodiment, the file may comprise a reduced number of parameters relative to the number of parameters output by the video mixer, for example the parameters produced by the compression algorithm described in the MIPEG4 standard. In an embodiment, the file may be uploaded onto a server from which a user could download it in order to use to allow a portable device to generate the full audiovisual book from the reduced number of parameters.

Figure 17 is a schematic of the basic process according to another embodiment for animating and rendering the head, producing the speech audio, text display and sound effects. The process comprises the same steps as those of the embodiment described in relation to Figure 11 and we refer the reader to that description. The process additionally comprises steps for producing sound effect audio.

In one embodiment, additional information is given in the input to allow sound effects to be selected in step S235. In one embodiment, this information is output from the user interface system described above. In an embodiment, the sound effect selection is performed via a user interface such as those described in relation to Figures 3, 4 and 5.

[n a further embodiment, the system is provided with a memory which saves predetermined sets of sound effects vecLors. Each veetur may be designed to allow the output of a different sound effect. The sound effects can be selected from music or other sound effects, in further embodiments the sound effects can relaie to the location of the action in the story, for example, if the story takes place near a river, or in a busy sn'eet etc. In yet further embodiments, the sound effects can relate to specific events in a story, for example, the ringing of a phone, etc. Such a system is shown in figure 18. For example, the system may need to output sound effects corresponding to text which it recognises as corresponding to a particular location, etc. The system may recognise from the text when a particular action is taking place. For example, the system may know when to introduce a ringing sound effect, for example from the position of the word "ringing" in *the text, and introduce the sound effect to be included in the output. Further the system may be configured to recognise if Ihe text refers to a happy moment, or an anxious moment and an appropriate musical sound effect outputted. This is shown schematically in step S227 where the sound effects are predicted directly from the text.

In the above system as shown in figure 18, a memory 1461 is provided which stores the attributes and rules to be checked in the text. The input text is provided by unit 1463 to memory 1461. The rules for the text are checked and information concerning the type of sound effect is then passed to selector unit 1465. Selection unit 1465 then looks up the vectors for the selected sound effect.

In step S237, the duration data calculated in step 5213 are fed into the sound effect generation unit. The duration data are combined by the sound effect generation unit with the sound effects selected in step 8235. The sound effects generation unit converts 1 0 these parameters into audio in step S239. Combining the duration data with the sound effects vectors ensures that the sound effect corresponding to a particular segment of text has the same duration as the speaking of that portion text and/or the miming of that portion of text by the talking head and will be output simultaneously with it.

1 5 In this embodiment, the face video, speech audio and text display video or duration data are generated in the same way as described in relation to Figure 11.

In one embodiment, the face video, speech audio, text display video or duration data and sound effects audio are combined with illustration/background video by a video mixer, as described above in relation to figure 1. The video mixer subsequently outputs a video to a display of the electronic book.

hi another embodiment, the video, speech sound, duration data and sound effects parameters are output to a data file. This file may be sent to storage, networked, output to a computer readable device, etc. In an embodiment, the file may be used to cause a portable device to generate a display of the electronic book. In an embodiment, the file may comprise a reduced number of parameters relative to the number of parameters output by the video mixer, for example Ihe reduced number of parameters produced by the compression algorithm described in the MPEG4 standard.

Next, the training of a system in accordance with an embodiment of the present invention will be described with reference to figure 19.

In image processing systems which are based on Hidden Markov Models O-IMMs), the HMIM is often expressed as: M=(A.B,n) Eqn.2 where A = and is the state transition probability distribution, B = $. o)} is the state output probability distribution and U = f,r}" is the initial state probahi I.ity distribution and where N is the number of states in the 11MM.

As noted above, the face vector parameters call be derived from a HMM in the same way as the speech vector parameters.

In the current embodiment, the sLate transition probability distribution A and the initial state probability distribution are determined in accordance with procedures well known in the art. Therefore, the remainder of this description will be concerned with the slate output probability distribution.

Generally in talking head systems the state output vector or image vector 0(1) from an inth Gaussian component in a model set: M is 3W)= N (oQ); [) Eqn.3 where, and £m are the mean and covarianee of the mth Gaussian component for speaker s.

The aim when training a conventional talking head system is to estimate the Model parameter set M which inaximises likelihood for a given observation sequence. in the conventional model, there is one single speaker from which data is collected and the emotion is neutral, therefore the model parameter set is g = 11,,, and = E for the all components m.

As it is not possible to obtain the above model set based on so called Maximum Likelihood (ML) criteria purely analytically, the problem is conventionally addressed by using an iterative approach known as the expectation maximisation (EM) algorithm which is often referred to as the Baum-Welch algorithm. Here, an auxiliary function (the "Q" function) is derived: y,Q)iog p(o(t),tn'M) Eqn4 1 0 wliei-e Yrn (t) is the posterior probability ol component in goncrating the obscrvation 0(t) given the current model parameters M and M is the new parameter set. Aft**er each iteration, the parameter set M' is replaced by the new parameter set M which maximiscs Q(M, M'). p(o(/), in AO is a generative model such as a 0MM, 11MM etc. 1 5 In the present embodiment a 11MM is used which has a state output vector of: P(o(frn,s,JW) = o(t)4i, Lv(m)) Eqn. 5 Where m MN} , T} and ye s} are indices for component, time and expression respectively and where MN, T, and S are the total number of components, frames, and speaker expression respectively. Here data is collected from one speaker, but the speaker will exhibit different expressions.

The exact form of j) and depends on the type of expression dependent transforms that are applied. In the most genera! way the expression dependent transforms includes: -a set of expression dcpcndcnt weights A') -a exprcssion-dependent cluster I1c(m) -a set of linear transforms [Ail) , After applying all the possible expression dependent transforms in step S213 the mean vector and covariance matrix ±; of the probability distribution m for expression s become = c.i) (s -. Eqn 6

±(s) (, ()T -I () In \ r(..) i'(u) i,r) / Eqn. 7 where tare th.e n.eans of cluster I for compon ent in as described in Eqa. 1, is the mean vector for component ne of the additional cluster for the expiession s, which will be described later, and A)and b) arc tile linear transformation mairix and the bias vector associated with regression class r(m) for the expression s.

R is the total number of regression classes and r (in) R} denotes the regression class to which the component in belongs.

If no linear transformation is applied A.)and b) become an identity matrix and zero 1 5 vector respectively.

For reasons which will be explained later, in this embodiment, the covariances are c].ustere.d and arranged into decision trees where (,) [1, . v} denotes the leaf node in a covariance decision tree to which the co-variance matrix of the component m belongs and V is the total number of variance decision tree leaf nodes.

Using the above, (he auxiliary function can he expressed as: Q(M,MO= -f Ym(1)OLv()H (0(1)-Eqn 8 where C is a constant independent of M Thus, using the above and substituting equations 6 and 7 in equation 8, the auxiliary function shows that the model parameters may be split into four distinct parts.

The first part are the parameters of the canonical model i.e. expression independent means {p} and the expression independent covariance { E k} the above indices ii and k indicate leaf nodes of the mean and variance decision trees which will be described later. The second part are the expression dependent weights where s indicates expression and i the ellLster index parameter, The third part arc the means of the expression dependent cluster j-tcOn,x) and the fourth part arc the CMLLR constrained maximum likelihood].i.nea.r regression transforms IA,bT)}V4 where s indicates expression and d indicates component or expression regression class to which component in belongs.

In detail, for determining the ML estimate of tile mean, the following procedure is performed.

To simplify the following equations it is assumed that no linear transform is applied.

If a linear transform is applied, the original observation vectors {Or(t)} have to be substituted by the transformed vectors = A..)o(t) } Eqn.9 Similarly, it will be assumed that there is no additional cluster. The inclusion of that extra cluster during the training is just equivalent to adding a linear transform on which A,)is the identity matrix and = First, the auxiliary function of equation 4 is differentiated with respect to 11n as follows: = k, -C, p, - G.m, tjj U ii Eqn. 10 Where = y = mi c(rn,i)=n c(mni)=ri in =L' Eqn. 11 with G° and k s'") accumulated statistics = (t, 8)AY. E' 112,qnn) v ( rn) , q(-m) t,s = 7,r (t, S)A]m) (in) o(t) Eqn.12 By maximizing the equation in the normal way by setting the derivaLive Lu zero, the following formula is achieved for the ML estimate of p i.e. ,: Rn = c 71.

Eqn. 13 It should bc noted, that the ML estimate of s also depends on PLc where k does not equal n. The index n is used o represent leaf nodes of decisions trees of mean vectors, whereas the index k represents leaf modes of covariance decision trees. Therefore, it is necessary to perform the optimization by iterating over all until convergence.

Thi.s can he performed by optimizing all J-tn simultaneously by solving the followin.g equations. r

LN

Eqn. 14 However, if the training data is small or N is quite large, the coefficient maLrix of equation 14 cannot have full rank. This problem can be avoided by using singular value decomposition or other well-known matrix lhctorization techniques.

The same process is then performed in order to perform an ML estimate of the covariances i.e. the auxiliary function shown in equation 8 is differentiated with respect to ZktO give:

-

t1sm 7m(t, s)o(t) o(t -V(?fl)-k ts,m Y.rntt, s) v(in)=k Fqn. 15 Where a(t) = o(t) -Eqn. 16 The ML esümate for expression dependent weights and the expression dependent linear transform can also be obtained in the same maimer i.e. differentiating the auxiliary function with respect to the parameter for which the ML estimate is required and then setting the value of the differential to 0.

For the expression dependent weights this yields = ( v71(t, s)IVJ _1Mrn) -l t,ra q(m)=q )JJT_lo(t) fli q(ni)=q Eqn. 17 In a preferred embodiment, the process is perfonned in an iterative manner. This basic system is explained with reference to the flow diagram of figure 1.

Fri step S301, a plurality of inputs of video ima.ge are received. In this illustrative example, 1 speaker is used. hut the speaker exhibits 3 different emotions when speaking and also speaks with a neutral expression. The data both audio and video is collected so that there is one set of data for the neutral expression and three further sets of data, one for each of the three expressions.

Next, in step 5303, an audiovisual model is trained and produced for each of the 4 data sets. The input visual data is parameterised to produce training data. Possible methods are explained in relation to the training for the image model with respect to figure 29.

The training data is collected so that there is an acoustic unit which is related to both a speech vector and an image vector. In this embodiment, each of the 4 models is only trained using data from one face.

A cluster adaptive model is initialised and trained as follows: In step S305, the number of clusters P is set to V + I, where V is the number of expressions (4).

In step S 307, one cluster (cluster 1), is determined as the bias cluster. In an embodiment, this will be the cluster for neutral expression. The decision trees for the bias cluster and the associated cluster mean vectors are initialised using the expression which in step 5303 produced the best model. In this example, each face is given a tag Expression A (neutral)", "Expression B", "Expression C" and "Expression D". here The covariance matrices, space weights for multi-space probability distributions (MSD) and their parameter sharing structure are also initialised to those of the Expression A (neutral) model.

Each binary decision tree is constructed in a locally optimal fashion starting with a single root node representing all contexts. In this embodiment, by context, the following bases are used, phonetic, linguistic and prosodic. As each node is created, the Llcxt optimal question about the context is selected. The question is selected on die basis of which question causes the maximum increase in likelihood and the terminal nodes generated in the training examples.

Then, the set of terminal nodes is searched to find the one which can be split using its optimum question to provide the largest increase in the total likelihood to the training data. Providing that this increase exceeds a threshold, the node is divided using the optimal question and two new terminal nodes are created. The process stops when no new terminal nodes can be formed since any further splitting will not exceed the threshold applied to the likelihood split.

This process is shown for example in figure 20. The nth terminal node in a mean decision trec is dividcd into two ncw tcrminal nodes;and fl by a. question q. The likelihood gain achieved by this split can be calculated as follows: f(n) = ( c.r)) jim i7iEES(i) + y (k.r -mCS(? Ji Fqft 18 Where 8(r) denotes a set of components associated with node n. Note that thc terms which are constant with respect to jt are not included.

Where C is a constant term independent of i,. The maximum. likelihood of Ri is given 1 5 by equation 1 3. Thus, the above can he written as: = ( rJ?CS(H: Eqn. 19 Thus, the likelihood gained by splitting node n into,t'and ti. is given by; At(:; q) = ) + tn) -Eqn. 20 Using the above, it is possible to construct a decision tree for each cluster where the tree is arranged so that the optimal question is asked first in the tree and the decisions are arranged in hierarchical order according to the likelihood of splitting. A weighting is then applied to each cluster.

Decision trees might be also constructed for variance. The covariance decision trees are constructed as follows: If the case terminal node in a covariance decision tree is divided into two new terminal nodes kf and k by question q, the cluster covariance matrix and the gain by the split are expressed as follows: (t) V (nfl in t S

E

5T iJt) v(-m=k Eqn.2i = (t) log + D

-

) (n-i) = k Eqn. 22 whcrc D is constant independent of {). therefore the increment iii likelihood is LLU, q) = E(W.) + L(k) -L(k) Eqn.23 In step S309, a specific expression tag is assigned to each of 2,..,,P clusters e.g. cluslers 2, 3. 4, and S are for expressions B, C, D and A respectively. Note, because expression A (neutral) was used to initialise the bias cluster it is assigned to thc last cluster to he initialised.

In step S311, a set of CAT ititerpolation weights are simply set to I or 0 according to the assigned expression (referred to as "voieeta.g" below) as: 1.0 if 1=1 = 1.0 ?f voicetag(s) = 0.0 otherwise In this embodiment, there are global weights per expression, per stream. For each expression/stream combination 3 sets of weights are set: for silence, image and pause.

In step S3 13, for each cluster 2 (P-i) in turn the clusters are initialised as follows.

The face data for the associated expression, e.g. expression B for cluster 2, is aligned using the mono-speaker model for the associated face trained in step S3 03. Given these alignments, the statistics are computed and the decision tree and mean values for the 1 0 cluster are estimated. The mean values for the cluster are computed as the normalised weighted sum of the cluster means using the weights set in step S3 11 i.e. in practice this results in the mean values for a given context being the weighted sum (weight 1 in both cases) of the bias cluster mean for that context and the expression B model mean for that context in cluster 2.

In step S3 15, the decision trees are then rebuilt for the bias cluster using all the data from all 4 faces, and associated means and variance parameters re-estimated.

Afler adding the clusters for expressions B, C and D the bias cluster is re-estimated using all 4 expressions at the same time In step S3 17, Cluster P (Expression A) is now initialised as for the other clusters, described in step S3 13, using data only from Expression A. Once the clusters have been initialised as above, the CAT model is then updated/trained as follows.

In step S3 19 the decision trees are re-constructed cluster-by-cluster from cluster I to P. keeping the CAT weights fixed. In step S321, new means and variances are estimated in the CM model. Next in step S323, new CAT weights are estimated for each cluster.

In an embodiment, the process loops back to S321 until convergence. The parameters and weights are estimated using maximum lilcelthood calculations performed by using the auxiliary function of the Baum-Welch algorithm to obtain a better estimate of said parameters.

As previously described, the parameters are estimated via an iterative process.

In a further embodiment, at step S323, the process ioops back to step S319 so that the decision trees are reconstructed during each iteration until convergence.

In a further embodiment, expression dependent transforms as previously described are 1 0 used, Here, the expression dependent transforms are inserted after step S323 such that the transforms are applied and the transformed model is then iterated until convergence.

In an embodiment, the transforms would be updated on each iteration.

Figure 20 shows clusters 1 to P which are in the forms of decision trees. In this simplified example, there are just four terminal nodes in cluster 1 and three terminal nodes in cluster P. It is important to note that the decision trees need not be symmetric i.e. each decision tree can have a different number of terminal nodes. The number of Lermina! nodes and the number of branches in the tree is determined purely by the log likelihood splitting which achieves the maximum split at the first decision and then the quesLions are asked in order of the question which causes the larger split. Once the split achieved is below a threshold, the splitting of a node terminates.

The above produces a canonical model which allows the following synthesis to be performed: 1. Any of the 4 expressions can be synthesised using the final set of weight vectors corresponding to that expression 2. A random expression can be synthesised from the audiovisual space spanned by the CA'l' model by setting thc weight vectors to ai'bitrary positions.

In a further example, the assistant is used to synthesise an expression characteristic where the system is given an input of a target expression with the same characLeristic.

In a further example, the assistant is used to synthesise an expression where the system is given an input of the speaker exhibiting the expression.

Figure 21 shows one example. First, the input target expression is received at step 501.

Next, the w'eightings of the canonical model i.e. the weightings of the clusters which have been previously trained, are adjusted to match the target expression in step 503.

The face video and duration data is then outputted using the new weightings derived in step S503.

In a further embodiment. a more complex method is used there a new cluster is provided for the new expression. This will be described with reference to figure 22.

As in figure 21, first, data of the speaker speaking exhibiting the target expression is received in step 8501. The weightings are then adjusted to best match the target expression in step 8503.

Thea, a new cluster is added to the model for the target expression in step 8507. Next, the decision tree is built for the new expression cluster in the same manner as described with reference to figure 19.

Then, the model paramelers i.e. in this example, the means are computed for the new cluster in step 8511.

Next, in step 85 13, the weights are updated for all clusters. Then, in step 8515, the structure of the new cluster is updated.

As before, the speech vector and face vector with the new target expression and duration data for that expression is outputted using the new weightings with the new cluster in step 8505.

Nole, that iii this embodiment, in step 8515. the other clusters are not updated 4 Ihis tinie as this would require the training data to be available at synthesis time.

In a further embodiment the clusters are updated after step 8515 and thus the flow diagram loops back to step 8509 until convergence.

Finally, in an embodiment, a linear transform such as C}VILLR can be applied on top of the model to further improve the similarity to the target expression. The regression classes of this transform can be global or be expression dependent.

In the second case the tying structure of the regression classes can be derived from the decision tree of the expression dependent cluster or from a clustering of the distributions obtained after applying the expression dependent weights to the canonical model and adding the extra cluster.

At the start, the bias cluster represents expression independent characteristics, whereas the other clusters represent their associated voice data set. As the training progresses the precise assignment of cluster to expression becomes less precise. The clusters and CAT weights now represent a broad acousfic space.

The above embodiments refer to the clustering using just one attribute i.e. expression.

However, it is also possible to factorise voice and facial attributes to obtain further control. In the following embodiment, expression is subdivided into speaking style(s) and eniotion(e) and the model is factorised for these two types or expressions or aLtributes. Here, [he state output vector or vector comprised of the model pairameters o(O from an mth Gaussian component in a model set M is P(o(t)tn,s,e,JV! )= N(o(t)t',E') Eqn. 24 where and are the mean and covariance of the mth Gaussian component for speaking style s and emotion e.

In this embodiment, s will refer to speaking style/voice, Speaking style can be used to represent styles such as whispering, shouting etc. It can also be used to refer to accents etc. Similarly, in this embodiment only two factors are considered but the method could be extended to other speech factors or these factors could be subdivided further and factorisation is performed for each subdivision.

The aim when training a conventional text-to-speech system is to estimate the Model parameter set M which maximises likelihood for a given observation sequence. In the conventional model, there is one style and expressionlemotion, therefore the model parameter set is p = p, and s(se) = 1'm for the all components in, 1 0 As it is not possible to obtain the above model set based on so called Maximum Likelihood (NIL) criteria purely analytically, the problem is conventionally addressed by using an iterative approach known as the expectation maximisation (P4) algorithm which is often referred to as the Baum-Welch algorithm. Here, an auxiliary function (the "Q" function) is derived: Q(SM \4)= y (t)log p(o(s),4M) ill F.qn2S whcrc Ym (t) is thc postcrior probability of component in generating the obscrvation o(t) given the current model parameters mi and M is the new parameter set. After each iteration, the parameter set M' is replaced by the new parameter set M which maximises Q(M, M'). p(o(t), in Al) is a generative model such as a (1MM, 11MM etc. In the present embodiment a HMM is used which has a state output vector of: P(oQrn,s,e,M)= N(oQ);i.L c) Eqn. 26 Where me{1, ,Ahll\r} ,re{1 T} , se{I,. siandeei,., E}areindicesfor component, time, speaking style and expression/emotion respectively and where MJ T, S and E are the total number of components, frames, speaking styles and expressions respectively.

The exact form of and depends on the pe of speahng style and emotion dependent transforms that are applied. In the most general way the style dependent transforms includes: -a set of style-emotion dependent weights x a style-eniotion-dependenl. cjtLsLer -a set of linear transforms [A;,bj whereby these transform could depend just on the style, just on Ihe emolion or on both.

Alter applying all the possible slyle dependent transforms, the mean vector;and covariance matrix,e) of the probability distribution in for style s and emotion e become = + -Eqn27 (s.e -A e)Ty'.I A(s.e1 k C,,;) (i:-/ Eqn. 28 whereg(fl 1are the means of cluster I for component in, is the mean vector for componcnt in of the additional cluster for style s emotion e. which will be described later, and and are the linear transformation matrix and the bias vector associated with regression class r(m) for the style s, expression e.

R is the total number of regression classes and r (in) R} denotes the regression class to which the component m belongs.

If 110 linear iransformation is applied Aand b3 become an identily matrix and zero vector respectively.

For reasons which will be explained later, in this embodiment, the covariances are clustered and arranged into decision trees where vQi2) e denotes the leaf node in a covariance decision tree to which the co-variance matrix of the component m belongs and V is the total number of variance decision tree leaf nodes.

Using the above, the auxiliary function can be expressed as: Qç; ,MO= r@)o VV()H ((t)-çjc.e)f (o(i)_ ())}+ c F.qn29 where C is a constant independent of M Thus, using the above and substituting equations 27 and 28 in equation 29, the auxiliary function shows that the model pa.ran.ieters may he split into Four distinct parts.

The first pail are tile parameters of the canonical model i.e. style and expression independent means {p} and the style and expression independent covariance { E k] Ge above indices it and k indicate leaf nodes of the mean and variance decision trees which will be described later. The second part are the sLyle-expression dependent weights (se) }se.J where s indicates speaking style, e indicates expression and / die cluster index parameter. l'he third part are the means of' the style-expression dependent cluster p(mx;L and the fourth part are the CMLLR constrained maximum likelihood linear regression transforms jA "), b t)}ved where s indicates style, e expression and d indicates component or style-emotion regression class to which component m belongs.

Once the auxiliary function is expressed in the above manner, it is then maximized with respect to each of the variables in turn in order to obtain the ML values of the style and emotion/expression characteristic parameters, the style dependent parameters and the expression/emotion dependent parameters.

In detail, for determining the ML estimate of the mean, the following procedure is performed: To simplify the following equations it is assumed that no linear transform is applied.

If a linear transform is applied, the original observation vectors {or(t)} have to be = A,o(t) + Bqn. 19 Similarly, it will be assumed that there is no additional cluster. The inclusion of that extra cluster during the training is just equiva]ent to adding a linear transform on which A.is the identity matrix and = 1c{m.x)} First, the auxiliary function of equation 29 is differentiated with respect to as follows: __________ = -Fqn.31 Where = = ?U 2 c(rnA)=n.

c(inj)=jj Equ. 32 with and kt accumulated statistics = JF[ (t. s. =

Eqn.33 By maximizing the equation in the normal way by setting the derivative to zero, the following formula is achieved for the ML estimate of i.e. : fin = GJ (k?l - GflJ/bt) xi n Eqn. 34 It should be noted, that the ML esfimate of also depends on k where k does not equal n. The index n is used to represent leaf nodes of decisions trees of mean vectors, whereas the index k represents leaf modes of covariance decision trees, Therefore, it is necessary to perform the optimization by iterafing over all t,1 until convergence.

This can be performed by optimizing all.t1 simultaneously by solving the following equations.

FGii. GIN [h c1 L1Nj LLJ\ Eqn. 35 F-Towcvcr, if the training data is small or N is quite large, the coefficient matrix of equation 35 cannot have frill rank. This problem can he avoided by using singular value decomposition or other Well-kIlown matrix lactorization tecliniqILes.

The same process is then performed in order to perform an ML eslimate of the covariances La the auxiliary function shown in equation 29 is differentiated with respect to E kto give: tsm v1(t. ,g, e)O. (*() -i i=k ,s,e.ni in) = k Eqn. 36 Where o1 (t) = 0(t) -Rqn. 37 The Mi. estimate For style dependent weights and the style dependent linear transform an also be obtaincd in the same manner i.e. diFFerentiating the auxiliary Function with respect to the parameter for which tile ML estimate is required and then setting the value of the diffcrential to 0.

For the expression/emotion dependent weights this yields A = ( 7m (t, s. e) 1(eT -i Ai4.) q (iii) =q y (t. s. e)MT E) ) U) (t) (m) = q Eqn38

S

\Vhere = 0(t) -Itc(ml) -And similarly, for the style-dependent weights A = ( , 7(t, s. e)M) E1)). (t) t,m,e q[n==q Where LI., (c) (c) 0q(;) Lt) = a{t) --Ag [n a preferred embodiment, the process is pcrformccl iii an iterative manner. Tins basic system is explained with reference to the flow diagrams of figures 23 to 25.

[n step S40 1, a. plurality of inputs of audio and video are received. In this illustrative example, 4 styles are used.

Next, in step S403, an acoustic model is trained and produced for each of the 4 voices/styles, each speaking with n.eutra.l emotion. in this embodiment, each of the 4 models is only trained using data with one speaking style. S403 will he explained in more detail with, reference to the flow chart of figure 24.

In step S805 of figure 24, the number of clusters P is set to V + 1, where V is the number of voices (4).

In step S807, one cluster (cluster 1), is determined as the bias cluster. The decision trees for the bias cluster and the associated cluster mean vectors are initialised using the voice which in step S303 produced the best model. In this example, each voice is given a tag "Style A", "Style B", "Style C" and "Style D", here Style A is assumed to have produced the best model. The covariance matrices, space weights for multi-space probability distributions (MSD) and their parameter sharing structure are also initialised to those of the Style A model.

Each binary decision tree is constructed in a locally optimal fashion starting with a single root node representing all contexts. In this embodiment, by context, the following bases are used, phonetic, linguistic and prosodic. As each node is created, the 1 5 next optimal question about the context is selected. The question is selected on the basis of which question causes the maximum increase in likelihood and the terminal nodes generated in the training examples.

Then, the set of terminal nodes is searched to lind the one wInch can be split using its optimum question to provide the largest increase in the total likelihood to the training data as explained above with reference to figures 19 to 22.

Decision trees might be also constructed for variance as explained above.

In step S809, a specific voice tag is assigned to each of 2,.. .,P clusters e.g. clusters 2, 3, 4, and 5 are for styles B, C, D and A respectively. Note, because Style A was used to initialise the bias cluster it is assigned to the last cluster to be initialised.

In step 8811, a set of CAT interpolation weights are simply set to 1 or 0 according to the assigned voice Lag as: 1.0 if 1=1 = 1.0 f oice1ag(s) = 0.0 othenvise In this embodiment, there are global weights per style, per stream.

In step S8 13, for each cluster 2 (P-I) in turn the clusters are initialised as follows.

The voice data for the associated style, e.g. style B for cluster 2, is aligned using the mono-style model for the associated style trained in step S303. Given these alignments, the statistics are computed and the decision tree and mean values for the cluster are estimated. The mean values for the cluster are computed as the normalised weighted sum of the cluster means using the weights set in step S8 11 i.e. in practice this results in the mean values for a given context being the weighted sum (weight 1 in both cases) of the bias cluster mean for that context and the style B model mean for that context in cluster 2.

In step S8 15, the decision trees are then rebuilt for the bias cluster using all the data from all 4 styles, and associated means and variance parameters re-estimated.

After adding the clusters for styles B, C and D the bias cluster is re-estimated using all 4 styles at the same time.

in step S817, Cluster P (style A) is now initialised as for the other clusters, described in step S8l3, using data only from style A. Once the clusters have been initialised as above, the CA'l' model is then updated/trained as follows: In step S819 the decision trees are re-constructed cluster-by-cluster from cluster I to P. keeping the CAT weights fixed. In step S821, new means and variances are estimated in the CAT model. Next in step S823, new CAT weights are estimated for each cluster. In an eniboditnent, the proccss loops back to Sg21 until convergence. The paramelers and weights are estimated using maximum likelihood calculations performed by using tiic auxilia -y function of the Baum-Welch algorithm to obtain a better estimate of said parameters.

As previously described, the parameters are estimated via an iterative process.

In a further embodiment, at step S823, the process loops back to step S8l9 so that the decision trees are reconstructed during each iteration until convergence.

The process then returns to step S405 of figure 23 where the model is then trained for different emotion both vocal and facial.

In this embodiment, emotion is modelled using cluster adaptive training in the same maimer as described for modelling the speaking style in step S403. First, "emotion clusters" are initialised in step S405. This will be explained in more detail with reference to figure 25.

Data is then collected for at least one of the styles where in addition the input data is emotional either in terms of the facial expression or the voice. It is possible to collect data from just one style, where the speaker provides a number of data samples in that 1 5 style, each exhibiting a different emotions or the speaker providing a plurality of styles and data samples with different emotions. In this embodiment, it will be presumed that the speech samples provided to train the system to exhibit emotion come from the style used Lu collect the data to train the initial CAT model in step S403. However, the system can also train to exhibit emotion using data collected with different speaking styles for which data was not used in S403.

In step S45 1, the non-Neutral emotion data is then grouped into Ne groups. In step S'453, Ne additional clusters are added to model emotion. A cluster is associated with each emotion group. For example, a cluster is associated with "Happy", etc. These emotion clusters are provided in addition to the neutral style clusters formed in step S403.

In step S455, initialise a binary vector for the emotion cluster weighting such that jf speech data is to be used for training exhibiting one emotion, the cluster is associated with that emotion is set to "F' arid all other emotion clusters are weighted at "0".

During this initialisation phase Ihe neutral emotion speaking style clusters are set Lu the weightings associated with the speaking style for the data.

Next, the decision trees are built for each emotion cluster in step S457. Finally, the weights are re-estimated based on all of the data in step 8459.

After the emotion clusters have been initialised as explained above, the Gaussian means and variances are re-estimated for all clusters, bias, style and emotion in step S407.

Next, the weights for the emotion clusters are re-estimated as described above in step 8409. The decision trees are then re-computed in step 8411. Next, the process loops back to step 5407 and the model parameters, followed by the weightings in step 5409, tollowed by reconstructing the decision trees in step 5411 are performed until convergence. In an embodiment, the loop S407-S409 is repeated several times.

Next, in step S413, the model variance and means are re-estimated for all clusters, bias, styles and emotion. In step 8415 the weights are re-estimated for the speaking style clusters and the decision trees are rebuilt in step 8417. The process then loops back to step 8413 and this loop is repeated until convergence. Then the process loops back to step 8407 and the loop concerning emotions is repeated until converge. the process continues until convergence is reached for both loops jointly.

In a further embodiment, the system is used to adapt to a new attribute such as a new emotion. This will be described with reference to figure 26.

First, a target voice is received in step Soul, the data is collected for the voice speaking with the new attribute. First, the weightings for the neutral style clusters are adjusted to best match the target voice in step S603.

then, a new emotion cluster is added to the existing emotion clusters for the new emotion in step 8607. Next, the decision Lree for the new cluster is initialised as described with rclatior to figure 25 from step 5455 onwards. The weightings, model parameters and trees are then re-estimated and rebuilt for all clusters as described with reference to figure 23.

The above methods demonstrate a system which allows a computer generated head to output speech in a natural manner while simultaneously displaying the text being spoken as the head and timing of the text display can adopt and adapt to different expressions. The clustered form of the data allows a system to be built with a small footprint as the data to run the system is stored in a very efficient manner, also the system can easily adapt to new expressions as described above while requiring a relatively small amount of data.

The above has explained in detail how CAT-HMM is applied to render and animate the head. As explained above, the face vector is comprised of a plurality of face parameters. One suitable model for supporting a vector is an active appearance model (AAM.). Although other statistical models may be used.

An AAIvI is defined on a mesh of V vertices. The shape of the model, s (xj;yj; x3;y2; defines the 2D posilion (x1; y,.) of each mesh vertex and is a linear model given by:

M

i=1 Eqn.2.l where s0 is the mean shape of the model, s is the 1th mode of Ad' linear shape modes and c1 is ils corresponding parameter which can be considered to be a "weighting parameter". The shape modes and how they are trained will be described in more detail with reference to figure 27. However, the shape modes can be thought of as a set of facial expressions. A shape for the face may be generated by a weighted sum of the shape modes where the weighting is provided by parameter c1.

By defining the outputted expression iii Lhis manner it is possible for the lace to express a continuum of expressions.

Colour values are then included in the appearance of the model, by a = (ri,' gj; bi; r2; g; h2; rp; gp; hp)'; where (ri; g1; h,) is the RGB representation of the jm of the p pixels which project into the mean shape s. Analogous to the shape model, the appearance is given by:

M

a= a0 + Eqn. 2.2 where a0 is the mean appearance vector of the model, and a is the 1th appearance mode.

In this embodiment, a combined appearance model is used and the parameters c in equations 2.1 and 2.1 are the same and control both shape and appearance.

Figure 27 shows a schematic of such an AAM. Input into the model arc the pararnctcrs in step S 1001. These weights arc then directed into both the shapc model 1003 and the appearance model 1005.

Figure 27 demonstrates the modes s, si SM of the shape model 1003 and the modes a, ai aM of the appearance model. The output 1007 of the shape model 1003 and the output 1009 of the appearance iodel are combined in step SIOlI to produce the desired face image.

Th.e parameters which are input into this model can he used as the face vector referred to above in the description accompanying figures 11 and 1 7 above.

The global nature of AAMs leads to some of the modes handling variations which are due to both 3D pose change as well as local deformation.

In this embodiment AAM modes are used which correspond purely to head rotation or to other physically meaningful motions. This can he expressed mathematically as: s = so + ct1sf?°8° + 5deforrn i=K+i Eqn.2.3 In this embodiment, a similar expression is also derived for appearance. However, the coupling of shape and appearance in AAMs makes this a difficult problem. To address this, during training, first the shape components are derived which model}K by recording a short training sequence of head rotation with a fixed ncutral cxpression and applying PCA to the observed mean normalized, shapes § = s -s0. Next g is projected into the pose variation space spanned by {si1e} to estimate the parameters {c1}I in equation 2.3 above: AT pose (1= Eqn. 2.4 Having found these parameters the pose component is removed from each training shape to obtain a pose normalized training shape s*:

---

Bqn. 2.5 If shape and appearance were indeed independent then the deformation components could be found using principal component analysis (PCA) of a training set of shape samples normalized as in equation 2.5, ensuring Lhat only modes orthogonal to the pose modes arc found.

However, there is no guarantee that the parameters calculated using equation 2.4 are the same for the shape and appearance modes, which means that it may not be possible to reconstruct training examples using the model derived from them.

Jo overcome this problem die mean of each {, } of the appearance and shape parameters is computed using: 1 TsPose = (sp0p + Hafl2)' Eqn. 2.6 The model is then constructed by using these parameters in equation 2,3 and finding the deformation modes from samples of the complete training set.

In lurtlier embodiments, the model is adapted (hr accommodate local deformations such as eye blinking. This can be achieved by a niodificcl version of the method described in which model blinking are learned from a video containing blinking with no other head titoti On.

Directly applying the method taught above for isolating pose to remove these blinking modes from the training set may introduce artifacts. The reason for this is apparent when considering the shape mode associated with blinking in which the majority of the movement is in the eyelid. This means that if the eyes are in a different position relative to the eentroid of the face (for example if the mouth is open., lowering the centroid) then the eyelid is movcd toward the mean eyelid position, even if this artificially opens or closes the eye. Instead of computing the parameters of absolute coordinates in equation 1 5 2.6, relative shape coordinates are implemented using a Laplacian operator: (blI11k -1 (L(?' L(hth) + -2 2 2 Eqn. 2.7 The Laplacian operator LQ is defined on a shape sample such that the relative position.

ä of each vertex i within the shape can be calculated from its original position Pj using / . . 2 jeN Eqn.2.B where N is a one-neighbourhood defined on the AAM mesh and is the distance between vertices i and j in the mean shape. This approach correctly normalizes the training samples for blinking, as relative motion within the eye is modelled instead of the position of the eye within the face.

Further embodiments also accommodate for the fact that different regions of the face can be moved nearly independently. It has been explained above that the modes arc decomposed into pose and deformalion components. This allows further separation of Ihe deformation components according to the local region they affect. Ihe model can he split into R regions and its shape can be modelled according to:

IC P

pose -, ." j S o.+.

I:1-I icf Eqn.2.9 where J is the set of component indices associated with region j. In one embodiment, modes for each region are learned by only considering a subset of the model's vertices according to manually selected boundaries marked in the mean shape. Modes are iteratively included up to a maximum number, by greedily adding the mode corresponding to the region which allows die model to represent the greatest proportion of Ihe observed variance in the training set.

1 5 An analogous model is used for appearance. Linearly blending is applied locally near the region boundaries. This approach is used to split the face into an uppcr and Lower hail. The advantage of this is that changes in mouth shape during synthcsis cannot lead to artifacts i.n the upper half of the face. Since global modes are used to model pose there is no risk oftle tipper md lower halves of Ue face having a different pose.

Figure 28 demonstrates the enhanced AAM as described above. As for the AAM of figure 27, the input weighlings For the AAM oF ligure 28(a) can Fonn a Face vector to be used in the algorithms described with reference to figures 11 and 17.

However, here the input parameters ci. are divided into parameters for pose which are input at S 1051, parameters for blinking SI 053 and parameters to model deformation in each region a.s input at S 1055. In figure 28, regions Ito Rare shown.

Next, these parameters are fed into the shape model 1057 and appearance model 1059. Here:

the pose parameters are used to weight the pose modes 1061 of the shape model 1057 and the pose modes 1063 of the appearance model; the blink parameters are used to weight the blink mode 1065 of the shape model 1057 and the blink mode 1067 of the appearance model; and the regional deformation parameters are used to weight the regional deformation modes 1069 of the shape model 1057 and the regional deformation modes 1071 of the appearance model.

As for figure 28, a generated shape is output in step S 1073 and a generated appearance is output in step S 1075. The generated shape and generated appearance are then combined in step S1077 to produce the generated image.

Since the teeth and tongue are occluded in many of the training examples, the synthesis 1 5 of these regions may cause significant artifacts. To reduce these artifacts a fixed shape and texture for the upper and lower teeth is used. 11w displacements of these static textures are given by the displacement of a vertex at the centre of the upper and lower teeth respectively. Th.e teeth are rendered before the rest of the face, ensuring that the correct occlusions occur.

Figure 28(b) shows an amendment to figure 28(a) where the static artifacts are rendered first. After the shape and appearance have been generated in steps Si 073 and S 1075 respectively, the position of the teeth arc determined in step 51081. In an embodiment, the teeth are determined to be at a. position which is relative to a. fixed visible pcint on Ihe lace. The teeth are then rendered by assuming a lixed shape and texture for the teeth in step S1083. Next the rest of the face is rendered in step S 1085.

Figure 29 is a flow diagram showing the training of the system in accordance with an embodiment of the present invention. Training images are collected in step S 1301. In one embodim exit, the training images are collected covering a range of expressions. For example, audio and visual data may he collected by using cameras arranged to collect the speaker's facial expression and microphones to collect audio. The speaker can read out sentences and will receive instructions on the emotion or expression which needs to be used when reading a particular sentence.

The data is selected so that it is possible to select a set of frames from the training images which correspond to a set of common phonemes in each of the emotions. In some embodiments, about 7000 training sentences are used. However, much of this data is used to train the speech model to produce the speech vector as previously described.

In addition to the training data described above, further training data is captured to isolate the modes due to pose change. For example, video of the speaker rotating their head may be captured while keeping a fixed neutral expression.

Also, video is captured of the speaker blinking while keeping the rest of their face still.

In step 51303, the images for building the AAM are selected. In an embodiment, only about 100 frames are required to build the AAM. The images are selected which allow data to be collected over a range of frames where the speaker exhibits a wide range of emotions. For example, frames may be selected where the speaker demonstrates difierenL expressions such as different mouth shapes, eyes open, closed, wide open etc. In one embodiment, frames are selected which correspond to a set of common phonemes in each of the emotions to be indicated by the head.

In further embodiments, a larger number of frames could be use, for example, all of the frames in a long video sequence. In a yet further embodiment frames may be selected where the speaker has performed a set of facial expressions which roughly correspond to separate groups of muscles being activated.

In step SI 305, the points of interest on the frames selected in step 31303 arc labelled.

In an embodiment this is done by visually identifying key points on the face, for example eye corners, mouth corners and moles or blemishes. Some contours may also be labelled (for example, face and hair silhouette and lips) and key points may be generated automatically from these conLours by equidistant subdivision of the contours into points.

In other embodiments, the key points are found automatically using trained key point detectors. In a yet further embodiment, key points are found by aligning multiple face images automatically. In a yet further embodiment, two or more of the above methods can be combined with hand labelling so that a semi-automatic process is provided by inferring some of the missing information from labels supplied by a user during the process In step Si 307, the frames which were captured to model pose change are selected and an AAM is built to model posc alone.

Next, in step 81309, the frames which were captured to model blinking are selected AAM modes are constructed to mode blinking alone.

Next, a further AAM is built using all of the frames selected including the ones used to model pose and blink, but before building the model, the effect of k modes was removed from the data as described above.

Frames where the AAM has performed poorly are selected. I'hese frames are then hand labelled and added to the training set. The process is repeated until there is hide further improvement adding new images.

the AAM has been trained once all AAM parameters for the modes -pose, blinking and deformation have been established.

Figure 30 is a schematic of how the AAM is constructed, The training images 1361 are bbelled and a shape moded 1363 is derived. The texture 1365 is also extracted for each face modcl. Once thc AAM modes and parameters arc calculated as explained above, the shape model 1363 and the texture model 365 are combined to generate the face 1367.

In one embodiment, the AAM parameters and their first time derivates are used at the input for a CA'l'-l-IMM training algorithm as previously described.

In a further embodiment, the spatial domain of a previously trained AAM is extended to further domains without affecting the existing model. For example, it may be employed to extend a model that was trained only on the face region to include hair and ear regions in order to add more realism.

A set of N training images for an existing AAM are knon, as are the original model coefficient vectors cf} c C R' for these images. The regions to be included in the model are then labelled, resulting in a new set of N training shapes {17} and appearances {a7' }1 Given the onginal model with Al modes, the new shape modes s, }ff, should satis' the following constraint: ciT siT (ext)T CNT sA,fT (;t)T Eqn. 2.10 which states that the new modes can he combined, using the original model coefficients.

to reconstruct the extended training shapes r' . Assuniing that the number of training samples N is larger than the number of modes A'!, the new shape modes can be obtained as the least-squares solution. New appearance modes are found analogously.

To illustrate the above, an experiment was conducted. Here, a corpus of 6925 sentences divided between 6 emotions; neutral, tender. angry, afraid, happy and sad was used.

From the data 300 sentences were held out as a test set and the remaining data was used to train the speech model. The speech data was parameterized using a standard feature set consisting o145 dimensional Mel-frequency cepsIra coefficients, log-FO (pitch) and band aperiodicities, together with the first and second time derivatives of these features. The visual data was paraincterized using the different AAMs described below.

Sonic AAMs were trained in order to evaluate the iinprovcnnents obtained with the proposed extensions. In each ease the AAM was controlled by 17 parameters and the parameter values and their first time derivatives were used in the CAl' model.

The first model used, AAMbase, was built from 71 training images in which 47 facial key points were labelled by hand. Additionally, contours around both eyes, the inner and outer lips, and the edge of the face were labelled and points vere sampled at uniform intervals along their length. The second model, AAMdecomp, separates both 3D head rotation (modelled by two modes) and blinking (modelled by one mode) from the deformation modes. The third model, AAMregions, is built in the same way as AAMdecomp expect that 8 modes are used to model the lower half of the face and 6 to model the upper half The final model, AAMfull, is identical to AAMregions except for the mouth region which is modified to handle static shapes differently. in the first experiment the reconstruction error of each AAM was quantitatively evaluated on the complete data set of 6925 sentences which contains approximately I million frames.

The reconstruction error was measured as the L2 norm of the per-pixel difference between an input image warped onto the mean shape of each AAM and the generated appearance.

Figure 31(a) shows how reconstruction errors vary with the number of AAM modes. It can be seen that while with few modes, AAMbase has the lowest reconstruction error, as the number of modes increases the difference in error decreases. In other words. the flexibility that semantically meaningful modes provide does not come at the expense of reduced tracking accuracy. In fact the modified models were found to be more robust than the base model, having a lower worst case error on average, as shown in figure 3 1(b). This is likely due to AAMregions and AAMdecomp being better able to generalize to unseen examples as they do not over fit the training data by learning spurious correlations between different face regions.

A number of large-scale user studies were performed in order to evaluate the perceptual quality of the synthesized videos. The experiments were distributed via a crowd sourcing website, presenting users with videos generated by the proposed system.

In the first study the ability of the proposed VII'S system to express a range of emotions was evaluated. Users were presented either with video or audio clips of a single sentence from the test set and were asked to identify the emotion expressed by the speaker, selecting from a list of six emotions. The synthetic video data for this evaluation was generated using the AAMregions model. It is also compared with versions of synthetic video only and synthetic audio only, as well as cropped versions of the actual video footage. In each case 10 sentences in each of the six emotions were evaluated by 20 people, resulting in a total sample size of 1200.

The average recognition rates are 73% for the captured footage, 77% for our generated video (with audio), 52% for the synthetic video only and 68% for the synthetic audio only. These results indicate that the recognition rates for synthetically generated results are comparable, even slightly higher than for the real footage. this may be due to the stylization of the expression in the synthesis. Confusion matrices between the different expressions are shown in figure 32. Fender and neutral expressions are most easily confused in all cases. While some emotions are better recognized from audio only, the overall recognition rate is higher when using both cues.

To determine the qualitative effect of the AAM on the final system preference tests were performed on systems built using the different AAMs. For each preference test 10 sentences in each of the six emotions were generated with two models rendered side by side. Each pair of AAMs was evaluated by 10 users who were asked to select between the left model, right model or having no preference (the order of our model renderings was switched beiween experiments to avoid bias), resulting in a total of 600 pair wise comparisons per preference test.

In this experiment the videos were shown without audio in order to focus on the quality of the face model. From table 1 shown in figure 33 it can be seen that AAMfull achieved the highest score, and that AAMregions is also preferred over the standard AAM. This preference is most pronounced for expressions such as angry, where there is a large amount of head motion and less so for emotions such as neutral and tender which do not involve significant movement of the head.

The systems and methods according to the embodiments described above enable the pro*visiox1 of clcctronie books comprising text display, sound effects and an cxpressive face image which reads the text of the book. Such electronic books are advantageous because naturalistic, expressive speaking of the text is achieved without employing expensive real speakers. F urther, the expression adds interest to the reading of the book compared with synthetic voices without expression. The addition of the expressive face image with lips which mime the text renders the expression easily identifiable and the speech easily understandable. The expressive face image may also increase the engagement of the user. The synchronization between the text display and the text being spoken may be used as a teaching aid for those learning to read. Further, as described above, the expression with which the text is read may be manually confrolled by the user. This provides users with a tool to create their own animations or versions of existing books.

Systems and methods according to the embodiments described above may be used in e-book reading systems for children, computer systems to assist children in learning how to read, as tools for people with dyslexia and as tools for helping autistic children recognize and understand human expressions. Adding an expressive face to the speech makes the expression more easily identifiable and the speech more understandable. The systems and methods according to embodiments described above may also be used for 1 5 comic strips.

The user interface according to the embodiment described in relation to Figure 5 may be used to generate standard theatre scripts with annotations in order to guide actors regarding expression or style with which they should speak cacti passage. Further, the computer generated speech itself may be used to guide an actor on how to deliver the text according to the director's instructions by providing concise information about the director's intended pitch. rhythm, expression, intensity, etc for a given passage.

Alternatively, a number of actors may provide their version of the text so that a director may select his preferred reading. Further, the direction may directly integrate his preferred reading of the text into the c-book as a speech file. A server may be provided to enable multiple persons to provide a version of the text.

\Vhile certain embodiments have been described, these embodiments have been prcscntcd by way of example only. and are not intended to limit the scope of the inventions. Indeed the novel methods and apparatus described herein may be embodied in a variety of other Ibims; furthermore, various omissions, substitutions and changes in the form of methods and apparatus described herein may be made without departing from the spirit of the inventions. The accompanying claims and their equivalents are intended to cover such forms of modifications as would fall.vithin the scope and spirit of the inventions.