FR2749420A1 - METHOD AND DEVICE FOR FORMING MOVING IMAGES OF A CONTACT PERSON - Google Patents

Abstract

The invention discloses a method for forming animated and sound images representing the face of a speaker consisting in effecting the following steps: initially: receiving an image of the said speaker's face; modeling a face said to be "modelled" representing the said face and adapted to be animated; and memorising the said modelled face; then after receiving a message likely to be spoken by the said speaker, analysing this message to determine a series of facial expressions to match the speaking out of the said message, and: transmitting a voice speaking out the said message, and animating the modelled face corresponding all the time to the delivery of the message.

Description

 The present invention relates to a method and a device for forming animated images of an interlocutor.

 It applies equally to the formation of images at a distance, for example for viewers, as to the formation of images locally for insertion into a video game.

 The currently known moving image transmission devices require the use of a photosensitive sensor which supplies an electrical signal representing perceived luminosities. The amount of information representative of the image is then very high and, although image compression technologies exist, the transmission of images requires a transmission medium capable of transferring large amounts of information.

 In addition, the use of the photosensitive sensor implies controlling the shooting conditions. Finally, the capture and transmission of moving images is then of high cost because of the quantities of information to be transmitted.

 In general, the invention intends to remedy these drawbacks by proposing to successively transmit a fixed image of a face, to model an image of this face which can be animated, to analyze a voice to determine a facial expression and then to animate the image model to give it said facial expression.

 Thus, the image of the interlocutor is captured by capturing not light rays reflected by the face but by sounds emitted by said face, and more particularly by the mouth, itself representative of the expression of the face. The cost of the device of the invention is thus limited to that of producing a still image, for example with a camera or by computer, and to that of capturing the voice, for example by a telephone.

 The present invention provides an image forming device, characterized in that it comprises: - a means for memorizing an image representative of a real face; - means for receiving voice representative signals - means for analyzing said signals adapted to determine facial expressions according to predetermined rules taking into account said signals; a means of animating the image by said facial expressions.

 Thanks to these arrangements, the moving image transmission medium is a voice transmission medium and it is therefore not necessary for it to be capable of transmitting more than the voice frequencies. The complexity and the cost of this transmission and of the formation of moving images are therefore very limited.

 According to a preferred embodiment, the device, object of the present invention, comprises an image modeling means suitable for providing an image intended to be animated, as a function of an image taken by a photosensitive sensor and the storage means stores said image.

 Thanks to these provisions, the transmission of a still image intended to be animated is carried out by transmission of a photograph or of a video image, and the cost of capture and transmission is very limited.

 According to a particular embodiment, this modeling means is adapted to perform a partial "morphing", that is to say a partial metamorphosis taking into account said image taken by a photosensitive sensor, and a predetermined model adapted to be animated .

 Thanks to these provisions, the predetermined model is adapted to be deformed to present the facial expressions transmitted and this model is combined with a stored image representative of the real face.

 According to a first aspect of the invention, the means for receiving voice representative signals is connected to a telephone line.

 Thanks to these provisions, the transmission of images takes place remotely via any type of telephone network, whether switched or digital.

 An example of this first aspect concerns the transmission in a television program of an animated image of a viewer who participates in the program.

 According to a second aspect of the invention, the means for receiving voice representative signals is connected to a soundtrack of an audiovisual program.

 Thanks to these provisions, the audiovisual program incorporates an animated image of the face used.

 As an example of this second aspect of the present invention, the face is that of the spectator of a film and the voice is that of a character from said film. Thus said spectator sees his image in the film, animated with the expressions of one of the characters in the film.

 According to a third aspect of the present invention, the means for receiving voice representative signals is connected to a radio antenna.

 An exemplary implementation of this third aspect relates to the transmission of a television program by broadcasting an animated image of journalists, presenters, well-known or media people who participate in the program.

 The invention also relates to a game console, a computer and an audiovisual editing bench, characterized in that they comprise a device as succinctly presented above.

The process for forming animated images of a contact, according to the invention, is characterized in that it consists in successively carrying out the following steps - firstly
* to transmit a fixed image of a face of the interlocutor; and
* to model an image representative of said face and adapted
to be animated; - then upon receiving a voice:
* to analyze this voice to determine an expression of
face;
* to animate said image so that it presents said expression
of face.

 Thanks to these arrangements, the step carried out initially can be done preliminary to the reception of the voice, possibly quite slowly.

 The computing power required is thus lower than for the transmission of an animated image captured by a photosensitive sensor.

 According to a first aspect of the invention, the voice and the face correspond to the same interlocutor.

 Thanks to these provisions, it is the face of this interlocutor which is transmitted remotely.

 An example of this first aspect concerns the transmission in a television program of an animated image of a viewer who participates in the program.

 According to a second aspect of the present invention, the voice and the face come from different sources of information.

 Thanks to these arrangements, the image is transmitted with expressions which are superimposed.

 As an example of this second aspect of the present invention, the face is that of the spectator of a film and the voice is that of a character from said film. Thus said viewer sees his image in the film animated with the expressions of one of the characters in the film.

 According to a particular implementation of this second aspect, the voice comes from a predetermined audiovisual medium and the face comes from a spectator.

 Thanks to these provisions, the spectator sees his face in an audiovisual program with expressions which come from said program.

 According to a particular implementation of the method of the invention, to model an image representative of the face and to adapt this image to be animated, the interlocutor who emits the voice intended to be analyzed positions himself marks on certain points of said picture.

 Thanks to these provisions, simple and well-known computer systems are sufficient to implement a method of television transmission (also called videophone) without the user needing professional assistance.

 The invention also relates to a game console, a computer and an audiovisual editing bench, characterized in that they implement an image transmission method as succinctly presented above.

Other advantages, aims and characteristics of the present invention will emerge from the description which follows, given with reference to the appended drawings, in which:
- Figure 1 shows a block diagram of image transmission and reception devices according to the first aspect of the present invention
- Figure 2A shows the successive operations performed according to the first aspect of the present invention
- Figure 2B shows a hardware layout of a device shown in Figure 1;
- Figure 3 shows a block diagram of image transmission and reception devices according to the second aspect of the present invention
- Figure 4 shows the successive operations performed according to the second aspect of the present invention
- Figure 5 shows a photograph of a face;
- Figure 6 shows the realization of a modeled image of the face presented in Figure 5
- Figure 7 shows the spectrum of a sound signal representative of a voice
- Figure 8 represents the expression corresponds to the sound signal presented in Figure 7
- Figure 9 shows a model animated by voice whose spectrum is shown in Figure 7, intended to be matched with the face shown in the photograph presented in Figure 5 to provide the image of the interlocutor.

 In the block diagram of devices for transmitting and receiving images according to the first aspect of the present invention (FIG. 1), are represented - a telephone apparatus 1 comprising a handset 2 and a keyboard 3 and adapted to transmit an electrical signal representative of the sounds reaching his microphone and of the presses made on the keys of his keyboard 3, according to techniques known in telephone apparatus with vocal frequencies; - a telephone network 4, of known type, represented in the form of two rectangles diagramming telephone sockets - a voice analysis means 5, the operation of which is explained with reference to FIG. 2 and which provides data representative of expressions oral to face animation means 9; a means of analysis of synthetic voice frequencies 6 of known type, adapted to transmit digital data representative of the voice frequencies emitted on the telephone network by the telephone apparatus 1, and representative of a possible pressure applied to one of the keys his keyboard 3; an image digitizing means 7, adapted to transmit data representative of an image of a face, of a type known in office scanners; a modeling means 8 adapted to combine data representative of an image received from the image digitizing means with a model and to provide the resulting combined still image by means of image animation 9 - the face animation means 9 adapted to animate the model combined with facial expressions corresponding to the oral expressions coming from the voice analysis means 5; a person animation means 10, of known type, and adapted to provide information representative of animated images of a person having the face produced by the face animation means 9, as a function of data coming from the means synthetic voice frequency analysis 6; a means of person and scene combination 11 adapted to insert the image of the person represented by the information leaving the person animation means with information on a scene comprising, for example mobiles, decorations and characters , according to known techniques; a display means 12 of known type, for example consisting of a television set, adapted to display the image emerging from the combination means II; a video transmitter 13 of known type, for example consisting of a radio transmitter or a transmitter on a cable television network; and - a recording means 14 of known type, for example consisting of a video recorder.

The voice analysis means 5 which provides data representative of oral expressions consists, for example, of a computer, a sound acquisition card, of known type, and a so-called detection software " FV ", initials of the words" Vocal Frequencies ", which determines animation keys, according to the vocal frequencies used, bass, treble, levels and changes in these frequencies used, aggressiveness, pronounced letters, laughter, loud or silent expectations,
and other attitudes. Known suppliers of these types of software are SILICLONE and SOFTIMAGE VIEWER.

The modeling means 8 is adapted to combine data representative of an image received from the image scanning means with a model. It is, for example, composed of a computer and image processing software suitable for carrying out image processing known by the English name of "morphing", sometimes translated into French under the name of "metamorphosis" and which matches - characteristic points of two images (corners of the mouth, nose and face contour points, corners of the eyes) to determine the distance between these points on the two images (here one of the images is a model face and the other is a digital photograph) and - the other points, by calculations on triangles which surround these points and whose vertices are characteristic points, by assigning to the displacement of the point of the photograph, a barycenter of the vectors displacement vertices, barycenter whose weights are decreasing as a function of the distances between said point and said vertex. Some technical reference books, such as the user guide and the software user manual
SOFTIMAGE "EDDIE", printed in Canada, defines morphing or metamorphosis as a crossfade between two states. They are incorporated into the description by reference.

 It is noted that, according to the invention, the modeled person resulting from the operation of the modeling means 8 corresponds to an intermediate state of metamorphosis between a photograph of a real face and of a model intended to be animated.

 Thus, the resulting modeled person has a face representative of the photograph, the features of this face being able to be animated by displacement of particular points or lines of the model with which the photograph has been associated to form the modeled person.

 The face animation means 9 adapted to animate the model combined with facial expressions corresponding to the oral expressions coming from the voice analysis means 5 is of known type in the animation of cartoon characters.

The reader will be able, for a better understanding, to refer to the reference works cited above as well as to the "Tool Book" and the user manual of the corresponding software from the company ALIAS WAVE.
FRONT and the user guide of the "Morph" software from GRYPHON
SOFTWARE CORPORATION, these five documents being incorporated by reference into the description of the invention.

All the image processing functions described with regard to FIGS. 1 and 2 can be carried out by computers using software such as those mentioned above as well as software
SOFTIMAGE VIEWER, SILICLONE and software from the National Audiovisual Institute performing morphings or metamorphoses in three dimensions automatically.

The operation of the device presented in Figure 1 is simple
to transmit an animated image, the device uses on the one hand a processing of a fixed image in order to make it suitable for being animated (by means of the image digitization means 7 and the modeling means 8) and d 'other hand a voice processing to determine oral expressions (performed by the voice analysis means 5) and facial expressions that animate the still image (animation performed by the face animation means 9).

 According to a variant, intended for the transmission of images between telephone interlocutors, the telephone apparatus may not be at voice frequency, the voice frequency analysis means 6, the person animation means 10, the combination means person and scene 11, The video transmitter 13 and the recording means 14 are deleted, the display means 12 directly displaying the image coming out of the face animation means.

 FIG. 2 represents the successive operations carried out according to the first aspect of the present invention, by following a program preserved in the read-only memory 17 of the computer 15.

 The first two operations, referenced 200 and 201, are carried out preliminary to the reception of the sound signal which carries the voice. During operation 200, the face is digitized, by the digitizing means 7 and stored in the random access memory 16 of the computer 15 (FIG. 1).

 During operation 201, the face digitized during operation 200 is combined, by metamorphosis, with a model intended to be animated. The resulting facial features are therefore those of the digitized face, that is to say those of the interlocutor, but the elements of this face are set in motion according to procedures depending on the digitized face, but also linked to the model intended to be animated. .

 The resulting face is therefore hybrid, its appearance being that of the digitized face and its movements being those of the model. As described with reference to FIGS. 1 and 2, the movements are controlled, by means of the analysis of the interlocutor's voice, by the movements of the interlocutor's face.

 Operation 202 corresponds to the reception of telephone sound on a signal input of the device presented in FIG. 1.

 Operation 203 corresponds to the spectral analysis of the sound received during operation 202 in order to provide a frequency spectrum of this signal. In said spectrum, with each frequency, on the abscissa, is associated the amplitude of the signal having this frequency, on the ordinate, in the received sound signal.

 Operation 204 consists in extracting the synthetic voice frequencies to determine whether the keyboard 3 of the telephone apparatus 1 has been used. It is noted that the vocal frequencies used always correspond to a combination of at least two non-harmonic primary frequencies, so that the risks of detection error are limited.

 The result of the extraction of the vocal frequencies, is the setting in random access memory 16, in the register freqvoc of the value of the key of the keyboard on which a pressure was detected.

 During operation 205, the voice analysis means determines, by analysis of the spectrum carried out during operation 203, the phonemes and other oral expressions used by the interlocutor. To perform this operation, the spectrum of the sound signal is compared to characteristic spectra of phonemes and oral expressions (such as laughter), said characteristic spectra being stored in the read-only memory 17 of the computer 15. It is noted, however, that this analysis is carried out dynamically, which means that it is not always a single instantaneous spectrum which makes it possible to determine the oral expression but also sometimes a succession of spectra, said succession being able to be characteristic of an oral expression. For a better understanding of this operation, we will refer to the instructions for use of the software mentioned above.

 During operation 206, each oral expression is related to a facial expression, for example the phoneme pronounced on reading the letter "O" corresponds to a facial expression in which the jaws are slightly apart and the lips form a small round opening.

 Each of these facial expressions is matched, during operation 207, with a succession of movements of points characteristic of the face model which supports the face of the interlocutor. For example if the pronunciation of the letter "O" is carried out for a period of one second, between two rests, the successive movements of the model and therefore of the modeled person, correspond: - first, for a quarter of a second to the spacing of the jaws and putting the lips in circles - then, maintaining this posture for the duration of the pronunciation of the letter "O" - then, at the end of this pronunciation (and therefore here on the return of silence), the face resumes its initial appearance in a quarter of a second.

 If the refresh rate of the image is 25 Hertz, that is to say that 25 different images are displayed during each second (frequency corresponding to television), the sequence described above corresponds to approximately 37 images therefore to 37 successions of movements made between two images.

 Operation 208 consists in putting the face in motion according to the successive values kept in the freqvoc register (FIG. 3) to take account of the pressing of the keys of the keyboard 3. This operation 208 corresponds for example to a game which the l 'interlocutor.

 Operation 209 consists in inserting the face resulting from operations 207 and 208 into a scene whose characteristics also depend on the oral expressions and the keyboard keys used.

 For example, during a game, oral expressions can be used to distort an object, and pressing keys can be used to move the object.

 Operation 210 corresponds to the display of the scene comprising the face, to its memorization and to the emission of the image. In the example of the televised game considered here, the interlocutor sees his image on a television set and he uses, on the one hand the keyboard 3 of his telephone apparatus 1, and on the other hand his voice, to play the game considered.

 Following operation 210, the system returns to operation 202.

 It should be noted that the sequence of operations described here is carried out, in fact, in real time, that is to say that, after the display of each image (operation 210), all of the operations 202 to 209 are performed in a period shorter than that of the image refresh, the face and the scene being in fact gradually modified during these operations.

 FIG. 2B represents a hardware layout of a device presented in FIG. 1. This device is, here, organized according to architectures known in the field of computers, around a computer communication bus 20 to which are connected - a central processing unit calculation 21, of known type, comprising in particular a processor; - a random access memory 16 which includes memory registers and in particular the freqvoc register; - a read only memory 17 which stores the operating program of the device; - a video output port 18 - an input and sound processing port 19; and - an input and image processing port 22.

 The video output port 18 is of known type, it provides, according to variants, either a signal adapted to television standards, for example the CCIR standard in Europe, or a signal adapted to standards relating to screen displays. computer, for example the super VGA standard.

 The sound input and processing port 19 is of known type, on the one hand it digitizes the sound, and on the other hand it extracts an instantaneous spectrum.

The resulting information is stored in the RAM 16.

 The input and image processing port 22 is of known type, it makes it possible to store in random access memory 16 digital data representative of a visual scene.

 FIG. 3 represents a block diagram of devices for transmitting and receiving images according to the second aspect of the present invention.

 We find in FIG. 3 certain elements of FIG. 1, the digitization means 7, the modeling means 8, the face animation means 9, the voice analysis means 5, the display means 12, the recording means 14 and the video transmitter 13.

 To these different elements, which provide the same functions as in the embodiment presented in FIG. 1, are added: - a reading means 33 of an audiovisual medium comprising an audio medium 30, for example a memory which stores signals sound, combined and synchronized with a visual medium 31, for example a memory which stores successive scenes from a show; this reading means provides a signal representative of the information carried by each of the two supports; and a means for combining the face and the scene, which combines the functions of the person animation means 10 and the person and scene combination means 11, presented in FIG. 1.

 During the operation of this embodiment of the device, the audio support 30 is played and it provides an audible signal by means of voice analysis 5. At the same time, the visual support 31 and the face animation means 9 each provide image representative signals by means of face and scene combinations 32.

 FIG. 4 represents the successive operations carried out according to the second aspect of the present invention.

 We find in Figure 4 operations 200, 201, 203, 205, 206 and 210, as well as two additional operations - operation 400 which follows operation 201, which precedes operation 203 and which consists of reception and storing the sound from the audio medium 30; The operation which follows operation 206, which precedes operation 210 and which carries out the insertion of the facial expressions determined during operation 206 on the face of a model, or character, forming part of the program visual from visual support 31.

 Figure 5 shows a photograph of a face.

 Figure 6 shows a model of the face shown in Figure 5, to which a swing arm microphone has been added. The modeled face created by the modeling means 8 during operation 201 provides an image which is intermediate in the metamorphosis of the images presented in FIGS. 5 and 6. We observe (FIG. 8) that the faces of FIGS. 5 and 6 can be associated by characteristic points 60, on the forehead, 61, on the cheekbones, 62, at the corners of the lips and 63 on the chin. These points called "primary" are those which are the sources or markers of facial deformation, that is to say that the other points of the face are displaced as a function of the displacement of these primary points.

 Although as of now, the automatic positioning of the primary points on the image shown in FIG. 5 is possible, by shape recognition techniques applied to these points, it is more economical and safer to position manually, using means known pointing devices, such as a mouse or joystick, associated according to known techniques, to the central unit 21 implementing the method of the invention.

 It is the displacement of the primary points which is put in relation with facial expressions, themselves in relation with oral expressions.

 FIG. 7 represents the spectrum of a sound signal representative of a voice.

 As indicated above, on the abscissa of this spectrum are the frequencies and on the ordinate the corresponding amplitudes. It is the spectral distribution and the distortion of this distribution which make it possible to determine an oral expression.

 FIG. 8 represents the expression which corresponds to the sound signal presented in FIG. 7.

 This expression of laughter involves, compared to the same expressionless face - a downward displacement of the primary point of the chin 63; - A spacing and a lateral rise of the primary points of the lips 62; - a rise in the primary points of the cheekbones 61 - a bringing together of the primary front points 60.

 FIG. 9 represents the animated image corresponding to the expression carried by the voice presented in FIG. 7. Each point of the modeled face, represented in FIG. 6, is associated with three primary points, vertices of a triangle which surrounds the point considered. . This point is moved in proportion to the deformation and the displacement of the vertices of this triangle, so that the displacements are continuous on all the surface of the triangle.

 We observe that to implement the invention, two procedures can be followed indifferently: either, as described above, we constitute by partial metamorphosis, a modeled person, certain points of which, called primary, serve as benchmarks for the animation, or we animate the model then we constitute a partial metamorphosis of the animated model and the real image.

 According to a variant not shown, the modeler 8 no longer works from a photograph, but from information transmitted by the interlocutor using the telephone 2. By way of example, this user indicates the characteristics of his face, hair, glasses, braces, color

 In this respect, it should be noted that the invention thus allows the creation of news or entertainment television channels using only the bandwidth corresponding to human voices, possibly supplemented by digital information processed on reception so as not to be audible by the viewers. Thus, a broadcasting station can add to its normal programs information which, processed on reception, will allow the device according to the invention to determine the modeled person to be animated by the voice transmitted by said station (this modeled person being able, in addition be animated by non-audible digital information transmitted by the broadcast signal).

 Likewise, users of communication networks, such as the INTERNET network, will not only be able to transmit their image and animate it with their voice, but also to transmit images and voices of other people.

 According to a last variant, the interlocutor whose image is transmitted transmits his electronic image, captured by a known photoelectric sensor, by positioning himself the primary points or landmarks allowing to animate this image by using the oral expressions transmitted by the voice, as described above. To position said marks, a computer system comprising a computer and a pointing device, such as for example a mouse, may be used according to techniques known to those skilled in the art.

 The invention is not limited to the embodiments described and shown but extends, quite the contrary to the embodiments, improvements, modifications and improvements within the reach of the skilled person.

Claims (17)

1. A device for forming animated images, characterized in that it comprises: - a means for memorizing an image representative of a real face; - means for receiving voice representative signals; a means of analysis of said signals adapted to determine facial expressions according to predetermined rules taking into account said signals; a means of animating the image by said facial expressions. 2. Device according to claim 1, characterized in that it comprises an image modeling means adapted to provide an image intended to be animated, as a function of an image taken by a photosensitive sensor of one and in that the storage means stores said image. 3. Device according to claim 2, characterized in that said modeling means is adapted to perform a partial "morphing", that is to say a partial metamorphosis taking into account said image taken by a photosensitive sensor, and a model predetermined suitable to be animated. 4. Device according to any one of claims 1 to 3 characterized in that the means for receiving voice representative signals is connected to a telephone line. 5. Device according to any one of claims 1 to 3, characterized in that the means for receiving voice representative signals is connected to a soundtrack of an audiovisual program. 6. Device according to any one of claims 1 to 3, characterized in that the means for receiving signals representative of voice is connected to a radio antenna. 7. Game console, characterized in that it comprises a device according to any one of claims 1 to 6. 8. Computer, characterized in that it comprises a device according to any one of claims 1 to 6. 9. Audiovisual assembly bench, characterized in that it comprises a device according to any one of claims 1 to 6. 10. A method of forming animated images of a contact, characterized in that it consists in successively carrying out the following steps - firstly:  * to transmit a fixed image of a face of the interlocutor; and  * to model an image representative of said face and adapted  to be animated; - then upon receiving a voice:  * analyze this voice to determine a facial expression  * to animate said image so that it presents said expression  of face. 11. Method according to claim 10, characterized in that, the voice and the face correspond to the same interlocutor. 12. Method according to claim 10, characterized in that the voice and the face come from different sources of information. 13. Method according to claim 12, characterized in that the voice comes from a predetermined audiovisual medium and the face comes from a spectator. 14. Method according to any one of claims 10 to 13, characterized in that to model an image representative of the face and adapt this image to be animated, the interlocutor who emits the voice intended to be analyzed positions himself landmarks on certain points of said image. 15. Game console, characterized in that it implements a method according to any one of claims 10 to 14. 16. Computer, characterized in that it implements a method according to any one of claims 10 to 14. 17. Audiovisual editing bench, characterized in that it implements a method according to any one of claims 10 to 14.