WO2010037956A1

WO2010037956A1 - Method and system for generating an interface for controlling the facial expressions of an avatar

Info

Publication number: WO2010037956A1
Application number: PCT/FR2009/051832
Authority: WO
Inventors: Nicolas Stoiber; Gaspard Breton; Renaud Seguier
Original assignee: France Telecom
Priority date: 2008-10-02
Filing date: 2009-09-28
Publication date: 2010-04-08

Abstract

The invention relates to a method for generating an interface for controlling the facial expressions of an avatar, said method including the following steps: obtaining (E1) a model of the deformations of a real face using images from a database (10) of images of the real face, said model including a set of aspect parameter vectors defining a real face aspect space (11); extracting (E2) a subset of parameter vectors of the real face aspect space based on the influence of the parameter vectors on the constitution of the real face aspect space and determining so-called characteristic expressions corresponding to the extracted parameter vectors; generating (E2) a graphic control interface (14) for navigating the real face aspect space and for controlling (CM1) the synthesis of any expression on the real face; creating (E3) a database (12) of images of an avatar face, referred to as an avatar database, by applying said real characteristic expressions to an avatar face; obtaining (E4) a model of the deformations of the avatar face from the images of the avatar database, said model including a set of aspect parameter vectors defining an avatar face aspect space (13); applying said control interface (14) to the navigation of the avatar face aspect space (13) in order to control (CM2) the synthesis of any expression on the avatar face.

Description

PROCESS AND SYSTEM FOR GENERATING AN INTERFACE FOR CONTROLLING FACIAL EXPRESSIONS OF AN AVATAR

DESCRIPTION The present invention relates to the techniques of analysis and synthesis of facial expressions, including the synthesis of facial expressions for virtual characters. In particular, the invention relates to a method and a system for generating an interface for controlling the facial expressions of an avatar. An avatar or virtual character is here defined as a digital character resulting from a graphic production, for example a character representing a user in a virtual meeting software, or a game, involving the user, or a synthetic character communicating with the user. Such a virtual character can be a character of computer-generated images or a human person resulting from real images then processed by image synthesis.

In the field of video game production, the creation of a virtual universe is almost always accompanied by its population by characters (whether humanly controlled or by an artificial intelligence) commonly called "avatars". In addition, the widespread democratization of computers, both as a work tool and as a fun medium, has boosted research efforts on the modes of interaction between men and machines. The latest advances favor natural multimodal communication by offering the user the possibility of interacting in real time with an avatar meeting his expectations.

In any case, the virtual characters strongly focus the attention of the user, and must therefore demonstrate maximum credibility. This credibility goes through the plastic quality of the character, but also and above all, by the naturalness of his movements and his attitudes. In addition to the labial movements associated with dialogue, the presence of natural and realistic facial expressions is of paramount importance, an expression that is inexpressive or even slightly incoherent, immediately leading to the loss of interest. the user, or even his dislike vis-à-vis the avatar. It is therefore essential to provide virtual characters with a realistic facial expression management system.

Several processes for creating facial expressions have emerged over the years. The best known (parametric, pseudomuscular or muscular systems) reduce the control of a facial expression when adjusting a set of parameters. Parametric and pseudomuscular systems are based on the parameterization of the variations on the elements of the face whereas the muscular systems model more rigorously the anatomy responsible for the facial expressivity. These systems nonetheless offer the common possibility of finely controlling virtual faces by acting on several parameters. However, the number of parameters to be adjusted makes the task of the animators long and tedious and, because of the unintuitive nature of these controls, mastering only one of these systems generally requires a long training. In addition, the size and complexity of the parameter set provided is not suitable for use in interactive animation engines (for virtual character dialog applications, for example) where facial animation must be generated on the fly. and in adequacy with the context. Finally, these systems offer, for the most part, only the control of the geometry of the face (position of the remarkable elements such as the mouth, the nose, the eyes), but not the management of effects related to the texture of the skin ( appearance of hollows, bumps, wrinkles). These effects, however important for the visual credibility of the character, are, if necessary, added by hand, thus increasing the duration and the complexity of the task.

Several previous approaches have already addressed the creation of intuitive interfaces for controlling a virtual face. For example, in the document titled: "Emotion Disc and Emotion Squares: Tools to Explore the Facial Expression Space" by Z. Ruttkay et al. - Computer Graphics Forum, vol. 22, pp. 49-53, 2003, the authors establish a bilinear interpolation between expressive faces distributed on a circle inspired by the disc of emotion of H. Schlosberg described by the latter, for example in the document: "The This paper is based on theoretical models describing the emotional states developed by social scientists as described for example in the document entitled "The Nature of Emotions" by R. Plutchik, published in American Scientist, vol 89, pp. 344, 2001. These theoretical models have the advantage of being simple and intuitive and the The purpose of these methods is to establish a link between these theoretical models and the "physical" parameters associated with the effective control of facial expressions, so that the physical parameters are "forcibly" adjusted to the theoretical models, which may be inconsistent because Theoretical emotional models were not designed based on physical criteria of facial movements, and facial movements are highly variable in an individual. from one to another, peculiarity that emotional models do not take into account.

The present invention aims to improve the existing techniques relating to the synthesis of facial expressions for a virtual character, proposing, in particular, a system for generating a control interface for a synthetic face, allowing easy generation and fast realistic facial expressions for a virtual character, while being accessible for use by professional animators as amateurs.

To this end, according to a first aspect, the present invention relates to a method for generating an interface for controlling the facial expressions of an avatar. According to the invention, this method comprises the following steps:

obtaining a model of deformations of a real face from the images of a database of images of the real face, this model comprising a set of vectors of appearance parameters forming a face appearance space real; extracting a subset of vectors of parameters from the real face appearance space, as a function of the influence of the parameter vectors on the constitution of the real face appearance space, and determining expressions, called characteristic expressions, corresponding to the extracted parameter vectors;

- Generation of a graphical control interface adapted to navigation in the actual face appearance space, and to control the synthesis of any expressions on the real face;

creating an image database of an avatar face, called an avatar database, by applying the characteristic real expressions to an avatar face;

obtaining an avatar face deformation pattern from the images of the avatar database, which model comprises a set of appearance parameter vectors forming an avatar face appearance space;

- application of the control interface to the navigation in the avatar face's appearance space, to allow the control of the synthesis of any expressions on the avatar's face.

Such a method for generating an interface for controlling the expressions of a virtual character has the advantage of not being based on a predefined emotion parameter space, on the contrary the starting parameter space that is to say, the real face appearance space, proceeds from the analysis of modes of deformation of the face of a real person, for example an actor. This real face appearance space is then used, through the determination of real face feature expressions, to create an avatar image database that will then be the basis for expression modeling for the face. avatar. Thus the control interface initially designed from an image database of a real face (human), can then be applied to such an avatar face database, to provide realism up to 'then unknown in the continuous synthesis of facial expressions on an avatar face.

According to a particular embodiment, the aforementioned model of real face deformations consists of a set of appearance parameter vectors combining shape and texture information, these appearance parameter vectors constituting a vector space. in which each parameter vector is associated with a single facial expression corresponding to a single image of the real face image database.

By choosing to use such a type of modeling, describing a facial expression not only in terms of geometric shape, but also in terms of texture information, the model is much more realistic than most known models, thus capturing effects of skin movements accompanying changes in facial expression. In particular, the movements of the wrinkles of the skin can thus be taken into account in the modeling.

According to a particular embodiment of the invention, this graphic interface for controlling the synthesis of any expressions on the avatar face is a graphical interface having the form of a disk in which:

a given type of facial expression is determined by a given radius of the graphic disk, and is selected as a function of the displacement of a marker on any one of the concentric circles centered on the center of the disk, and having as radius one radius between zero and the radius of the disk; and

an intensity value relative to a type of expression considered is determined by the displacement of the marker on the radius of the disc corresponding to the type of expression considered.

Such a graphical control interface is easy and ergonomic for use, both for a professional animator and an amateur animator, and offers the ability to navigate continuously in the appearance space and thus generate any type of facial expressions according to the desired intensity of the expression.

According to a second aspect, the subject of the invention is an interface for controlling the facial expressions of an avatar face, obtained by implementing a method of generating a control interface as briefly defined above.

According to a particular embodiment of the invention, the step of generating a control graphical interface comprises: a step of identifying, from the scatterplot constituting the actual face appearance space, a predetermined number of dominant directions of the appearance space, said dominant directions being determined using an algorithm of convex hull detection; and a step of applying a mapping of each point of the real-face appearance space, carried by an identified dominant direction, to a point in the space constituting the control graphical interface.

The characteristic expressions used to design the avatar image database come from these dominant directions, so this is a kind of transfer of knowledge about the nature and organization of dominant directions from the human face to a synthetic face. The use of the control interface to control the synthesis of expressions of a virtual face thus provides a very realistic expressivity.

According to a third aspect, the invention relates to the use of an interface for controlling the facial expressions of an avatar face obtained by an interface generation method according to the invention, for the synthesis of facial expressions on an avatar face. According to another aspect, the invention relates to the use of the aforementioned control interface, for the synthesis of facial expressions on a human face.

Indeed, such a facial expression control interface generated by a method according to the invention can be applied to any synthetic character face from a reduced database of images (pseudo database ), but also to synthesize facial expressions directly on a scanned human face, in the latter case the control interface controls the navigation directly in the real face appearance space.

The invention also concerns a system for generating an interface for controlling the facial expressions of an avatar, this system comprising:

means for modeling the deformations of a real face from the images of a database of images of the real face, the model obtained comprising a set of appearance parameter vectors forming a real face appearance space;

means for extracting a subset of vectors of parameters of the real face appearance space, as a function of the influence of the parameter vectors on the constitution of the real face appearance space , and determining the expressions, called characteristic expressions, corresponding to the extracted parameter vectors;

means for creating an image database of an avatar face, called an avatar database, by applying the characteristic real expressions to an avatar face;

means for modeling the deformations of the avatar face from the images of the avatar database, the model obtained comprising a set of vectors of appearance parameters forming an avatar face appearance space; means for generating a control interface suitable for navigation in the real-face appearance space, and for controlling the synthesis of any expressions on the real face, the control interface being applicable to the navigation in the avatar face's appearance space, to allow automatic control of the synthesis of any expressions on the avatar's face.

The invention also relates to a computer program, comprising instructions adapted to the implementation of all or part of the steps of a method of generating a control interface as described above when it is executed on a computer. The advantages provided by such a computer program or such a system for generating an interface for controlling the facial expressions of an avatar are identical to those mentioned above in relation to the method for generating an interface, according to the invention.

The invention will be better understood on reading the detailed description which follows, given solely by way of example and with reference to the drawings in which: FIG. 1 illustrates a global method for generating an interface for controlling the facial expressions of a virtual character, according to the invention;

FIG. 2 illustrates the step of modeling the deformations of a real face according to one embodiment of the invention;

3 represents two different views of the distribution of the samples (parameter vectors) of the real face database in the real face appearance space, obtained according to the invention;

FIG. 4 illustrates the characteristics of the real face appearance space, as well as the process of reducing the appearance space to a 2-dimensional space, according to an embodiment of the invention;

FIG. 5 illustrates the step of generating a 2-dimensional control interface, according to an embodiment of the invention, from the real face appearance space; FIG. 6 illustrates the use of a 2-dimensional control interface according to one embodiment of the invention, for controlling the facial expressions of a virtual character;

FIG. 7 illustrates the result of the synthesis of facial expressions obtained on both the avatar face and the real face, using a 2D control interface according to the invention; and

FIG. 8 illustrates the emotional interpretation of the control space covered by the 2D control interface according to the invention.

Figure 1 illustrates a global method for generating an interface for controlling the facial expressions of a virtual character, according to the invention.

As represented in FIG. 1, the method according to the invention comprises: An initial step E1 of modeling the deformations of a real face from the images of a database of images 10 obtained from a human face real. The model of deformations obtained at the end of the modeling step comprises a set of vectors of parameters of appearance of dimension Na and which form, when graphically represented in a space with Na dimensions, a cloud of points, 11, called "appearance space" of real face (human). In practice, according to a choice of embodiment, this modeling step E1 of the deformations or expressions of a human face, is implemented by a particular software module. This modeling step (E1) will be detailed below in connection with FIG. 2.

A step E2 for generating a control graphical interface 14 obtained from the real face appearance space (11), this interface being able to allow the control of the synthesis of any expressions on the real face; . This control function is represented by the dashed arrow CM1 in FIG. 1. The generation step E2 of the graphical interface 14 which will be detailed below in relation to FIGS. 3 to 5, includes a step of extraction of FIG. a subset of vectors of parameters of the real face appearance space, depending on the influence of the parameter vectors on the constitution of the real-face appearance space, and the determination of the expressions, so-called characteristic expressions, corresponding to the extracted parameter vectors. In practice, according to an embodiment, the step E2 for generating an interface is implemented by a particular software module.

A step E3 of creating a database (12) of images of an avatar face (or virtual character), called the avatar database, by applying the real characteristic expressions obtained during the step E2, to an avatar face.

An E4 step of modeling the deformations of the avatar face from the images of the avatar database 12, this model comprising a set of vectors of appearance parameters forming an avatar face appearance space 13. In practice, the modeling step E4 can be implemented either by the modeling module used for the aforementioned step E1, or by a separate software module.

A control function CM2 for the application of the control interface 14 to the navigation in the avatar face appearance space (13), in order to allow the control of the synthesis of any expressions on the avatar face. Modeling of real facial deformities

Figure 2 illustrates the step (E1) of modeling the deformations of a real face according to one embodiment of the invention. Firstly, a database of images is created from sequences of images of the face of a real person, for example an actor, adopting facial expressions. The goal is to obtain a database containing a large quantity of varied natural expressions, types (anger, joy, surprise, etc.) and intensity (extreme, subtle, stereotyped, etc.). It is important that captured expressions are not labeled. The database 10 contains a high number of images, in practice of the order of 5000 images. In practice, the required image sequences are obtained by means of, for example, a video camera.

The images of the database 10 are then annotated in order to identify the striking elements of the face such as the eyes, the mouth, the nose thus making it possible to extract the "shape" of the face. This annotation process is known per se, and is done automatically using an automatic tracking process (tracking) of the shape of the face. At the end of the annotation step of the images, each expression recorded in the database, via an image, is associated, on the one hand, with a geometrical shape (201), and on the other hand, with the set of pixels of variable intensity located inside the shape, this set of pixels representing "texture" information (202).

The shape and texture information associated with each of the images of the database 10 are the input data for the modeling (E1) of the deformations of the real face. According to the embodiment presented here, the modeling step is implemented using an Active Appearance Models (AAM) technique. For more information on MAAs, see the following document: "Active Appearance Models" by TF Cootes, G. Edwards, and CJ. Taylor, Lecture Notes in Computer Science, Vol. 1407, p. 484, 1998. In summary, the modeling step E1 using the MAAs identifies the main modes of shape and texture variations of the face when displaying a facial expression. At the output of the AAM modeling step, a set of parameters is obtained that represent the contribution of the variation modes in form and texture in a combined manner. Each of the facial expressions can then be projected onto the AAM parameter space, to keep only the relevant information regarding facial movements and texture changes. In the AAM parameter space, called "appearance space", each facial configuration from the database (10) is represented by a point of dimension Na (Na is typically of the order of 30).

It will be noted here that if the mapping of the shape and texture space to the appearance space is reversed, then each point of dimension Na belonging to the appearance space can be associated with a unique pair of shape and texture, and therefore synthesize the corresponding facial expression. In addition, an essential property of the appearance space is that each point within the space covered by the facial expressions of the images of the database (10), corresponds to a valid facial expression. Navigating anywhere in the appearance space thus makes it possible to synthesize a continuum of natural facial expressions within limits defined by the examples of the real face image database (10).

In more detail, in FIG. 2, after the aforementioned annotation step, there is available for each image of the base (10) a shape information (step 203), as well as a texture information (step 204). The purpose of modeling is to extract a set of parameters that is adapted to the variation modes observed in the images of the database. In order to detect the main modes of variation, a technique known as Principal Component Analysis (PCA) is used, both for shape information (PCA 205) and for texture information (PCA 206). A shape is a collection of points of dimension N ₈ (N ₃ is 2 or 3), and a texture is usually a collection of pixel values, and both can be considered as vectors that can feed into a PCA process.

At the end of the PCA (205) form analysis step, intermediate shape parameters B _s are obtained. Similarly, all the textures are analyzed by PCA (206) and intermediate texture parameters B ₁ are obtained. The two PCA operations (205, 206) are intended to identify the main modes of variation observed in the database and to filter to keep the most significant. The vector pairs B ₃ and B _t are then combined in a concatenation step 207 to obtain for each image a vector of combined parameters of shape and texture. The combined parameter vector then undergoes another Principal Component Analysis (PCA) operation (208), whose role is to identify the correlations between shape and texture variations and take advantage of these correlations to reduce the size of the parameter set (dimensional reduction). At the output of the PCA block 208, a coherent set (20) of reduced vectors, C, of mixed parameters (shape and texture) is thus obtained.

The modeling operation of the AAM type (step E1) used in the context of the invention, which makes it possible to obtain an appearance space (from the set of vectors C of mixed parameters), offers the advantage can be reversed and thus allow the synthesis of a facial expression from any vector C. Such appearance spaces have already been used successfully for the synthesis of facial expressions, but only for real faces. Regarding this last point, we can refer for example to the document: "Facial expression recognition and synthesis based on an appearance modet", of B. Abboud, F. Davoine and M. Dang - Signal Processing: Image Communication, vol. No. 8, pp. 723-740, 2004.

Generating a control interface from the real face appearance space

• Structure of the appearance space As mentioned before, the appearance spaces offer interesting possibilities concerning the navigation in these spaces and the animation of real or synthetic faces. However, the dimension (Na) of the mixed shape-texture vectors (Q vectors) is too large (of the order of 30) to allow easy navigation in the appearance space, and consequently to create a graphical control interface to drive the synthesis of facial expressions from this high-dimensional space is a difficult problem.

In order to be able to automatically create a control interface, it is sought to simplify the navigation in the appearance space, by observing the way in which the samples (images) of the image database 10 are distributed in their representation. in the appearance space.

Figure 3 shows two different views of the sample distribution (parameter vectors) of the actual face database in the real face appearance space, obtained according to the invention. For simplification of representation, only the first three dimensions are represented.

Following the second PCA operation (208), it can be considered that the first coordinates of the parameter vectors C are the most significant vis-à-vis the variance of the image database. Thus, the two point clouds shown in FIG. 3 correspond to the points associated only with the first three coordinates (di, d _Σ , dβ) of the parameter vectors. The coloration of the points (shades of gray) is a function of the type (Ei) of emotion associated with each facial expression represented by an image of the real face image database (10). This staining was carried out so as to distinguish the basic expressions (groups Ei to Ee) defined according to P. Ekman et al., For example in "Facial expressions of emotion: An old controversy and new findings" - Philosophical Transactions: Biological Sciences, flight. 335, no. 1273, pp. 63-69, 1992. On a given scatter plot, it can be determined that neutral facial expressions (with no particular expression) correspond to the center of the cloud, whereas very expressive faces correspond to points in the cloud. at the outer limits of the cloud. Intermediate expressions are located in the continuous space between neutral and extreme expressions. On this subject, for example, the document "Appearance Manifesto of Facial Expressions" by C. Shan, S. Gong, and PW McOwan - Computer Vision in Human Computer Interaction, Berlin, Spring 2005, pp. 221 -230.

It can be seen that a given point cloud (right or left cloud, Fig. 3) has some dominant directions that can be identified as segments that connect points corresponding to neutral expressions (center) to points corresponding to extreme expressions. . Most expressions are distributed, either along these segments forming the dominant directions to thus constitute a given expression according to different levels of intensity, or between these dominant directions to constitute transitions of expressions between the dominant types of expressions. According to the invention, these dominant directions of the appearance space have been rearranged in a space of reduced size, but according to a comparable topology. In the embodiment chosen and exposed here, it is a space of reduced appearance of dimension 2. In this way, it is then possible to create a graphical interface of control adapted in 2 dimensions (2D), by example in the form of a 2D disk. The process of reducing the initial appearance space (real face) to a dimension space 2 will now be detailed in relation to Figure 4.

Figure 4 illustrates the characteristics of the real face appearance space, as well as the process of reducing the appearance space to a 2-dimensional space, according to one embodiment of the invention. As shown in Figure 4 - upper part, the central part of the appearance space corresponds to the neutral expressions (see lowest image). An extreme expression (top image) corresponds to a point on the outer boundary of the cloud on a dominant direction (top image), and the variable intensity expressions are focused on the dominant directions of the appearance space between neutral and extreme positions (middle image). Thus, the three images presented in Figure 4 (upper part) correspond to an expression of type "anger" (type E ₁ ) varying between a neutral expression (bottom image) and an extreme expression (top image).

To transform the space of initial appearance (of real face) into a space of reduced dimension (2D for example), it is first necessary to determine the points, that is to say the vectors of parameters (vectors C), which are the most significant in terms of variance, so that these vectors are taken into account in the reduced space. It is therefore, according to the invention, to extract a subset of vectors of parameters of the appearance space of the real face, as a function of the influence of the parameter vectors on the constitution of the space appearance of the real face.

It is considered here that the most significant parameter vectors are those corresponding to the points on the convex hull (in English "convex eight") of the point cloud, so that, according to the invention, a subset of vectors of real face appearance space parameters, by selecting the parameter vectors corresponding to the points forming the convex hull in the real face appearance space.

In practice, the vectors are selected automatically by using a convex hull detection algorithm, for example that described in the document "The quickhull algorithm for convex hulls", by C. B. Barber et al. - ACM Transactions on Mathematical Software (TOMS), vol. 22, no. 4, pp. 469-483, 1996. The result of the convex hull detection contains an important redundancy since the intersection between the convex envelope and the scatterplot contains several neighboring points, these neighboring points representing however only one dominant direction. In order to eliminate this redundancy, we apply a filtering algorithm known as the "Mean Shiff" (which could be translated into French as "average offset") to the selected vectors of the convex envelope, in order to merge the points neighbors in a single representative of the convex hull.

Usually, convex hull detection and Mean Shift algorithm filtering consumes a lot of processor resources for large dimensions. However, in the context of the invention, since the dimensions of the appearance space are determined by a PCA treatment (principal components analysis), these dimensions are ordered according to their percentage of participation in the overall variance of the database. In other words, the first dimension of the appearance space has the highest variance, and the variance decreases when moving from a given dimension to the larger dimension. It is determined that above a certain dimensionality, the result of the convex hull detection corresponds to insignificant variations. In practice, it suffices to consider only the first 8 dimensions to obtain a sufficiently stable envelope detection; in fact, the first 8 dimensions represent 75% of the variance of the image database (base 10).

In Figure 4, bottom left corner, the extreme expressions detected as the points of the convex hull are represented by the large black dots on the periphery of the cloud. The lower right part of Figure 4 highlights the dominant directions (black line segments) that extend between the central points of neutral expressions to a few points selected at the periphery and corresponding to extreme expressions.

• Generation of a 2-dimensional control interface

The objective here is to adapt all the dominant directions of the appearance space (11) to a 2-dimensional space, in particular a 2D disc (14). For this purpose, the only information considered is the angle between the dominant directions in the appearance space, the dominant directions being viewed as unit length segments of a Na dimensionality hypersphere centered on the sample of neutral expression. Therefore, each of the dominant directions of the appearance space must correspond to a 2D direction on the disc and the distortion between the direction distributions must be minimized. Specifically, the 2D implementation of the directions must be such that the distribution of the angular distances in the 2D implementation must be similar to that existing in the appearance space.

The problem is then to search for a path in the hypersphere of the space of appearance that passes through each of the dominant directions only once and at the same time minimizes the angular distance accumulated during the course of the path, which can be defined as the cost of the path. By analogy to the problem mentioned above, the well-known problem of the traveling salesman problem (TSP) can be applied. As a reminder, the problem of the commercial traveler is an NP-complete problem whose aim is to determine the shortest way to cross a set of cities whose distance between any pair of cities is known. Thus, by replacing the cities by the principal directions of dimension Na and the distance between cities by the angular distance between directions, one can solve the problem of the optimal path in the space of appearance by using a generic method of solving the problem of the traveling salesman. Of course, given the NP character of the TSP problem, the optimization task becomes excessively expensive in computing time when the number of detected directions increases. In the context of the invention, a stochastic resolution method based on a so-called "simulated annealing" process is used, and the results show that with a reasonable number of directions, the optimization of the problem TSP always reaches a relatively optimal solution in a reasonable time (less than 2 minutes for 30 directions). The directions are then positioned on the circle (perimeter of the disk) according to the sequence of directions obtained by the optimization algorithm of the TSP problem. In addition, the angle values between the directions implemented on the circle are calculated proportionally to the angular distribution observed in the path through the directions in the Na-dimensional appearance space.

Figure 5 illustrates the step of generating a 2-dimensional control interface from the real face appearance space. In Figure 5 as in Figure 4, to simplify the representation, only the first 3 dimensions (di, d ₂ , d ₃ ) are drawn. Figure 5 illustrates more precisely the transformation of the appearance space into a 2D disk.

The upper and lower left portions of Figure 5 represent two views of the dominant directions detected in the appearance space. The directions appear with distinct shades of gray to highlight the type of emotions worn by each direction.

The central parts of Figure 5 (high and low) illustrate the paths obtained by the business traveler optimization algorithm to obtain the main direction sequence in which the directions are to be implemented on a 2D disk (right). The right part of the

Figure 5 illustrates the implementation of dominant directions on the 2D disk.

Above, a 2-dimensional distribution of the dominant directions of the Na-dimensional appearance space has been described. Next, we must define a link or correspondence between each point of the real face appearance space, carried by a selected dominant direction, to a point in the 2D space carried by the corresponding direction implemented in the control graphical interface. (2D disc according to the embodiment presented).

In order to achieve this correspondence, various methods can be used, for example regression methods (linear, polynomial, etc.). However, in the context of the present invention, another approach based on the approximation method known as "Thin-Plate Spline" (TPS) is used. This method has the advantage of being a compromise between the precision of the interpolation of the known samples to be matched, and the smoothing of the mapping. Now, the criterion of smoothing is important in the present case, since the space of appearance must be able to be the object of a continuous navigation to realize the animation of a real or synthetic face. In the foregoing, we have just described the generation operation (E2,

Fig. 1) a control interface which, in summary, first involves the identification, from the cloud of points constituting the real-face appearance space, of a predetermined number of dominant directions of space of appearance, using a convex hull detection algorithm. Next, a mapping of each point in the real-world appearance space, carried by a selected dominant direction, to a point in the space constituting the control graphical interface (2D) is applied. According to the embodiment presented here, the graphical control interface of the expression synthesis is a graphic interface having the form of a disk in which:

In practice, the graphical interface according to the invention can be represented on a computer screen, and the aforementioned marker, operated on the screen by means of a pointing device such as a mouse. The 2D control interface according to the invention has been designed from the real face appearance space (11) itself based on the image samples of the real face database. (10). Thus, the control interface (14) can be used to "navigate" in the reduced appearance space of the interface and determine a corresponding vector in the appearance space (11) at Na dimensions, in order to synthesize (arrow CM1 in Fig. 1) any expression applied to the real human face object of the images of the database 10.

Using the real face expression control interface to synthesize facial expressions of an avatar face

According to the invention, it has been determined that in order to be able to apply the interface of control of the expressions of the real face, to the synthesis of expressions of a synthetic face (face of a virtual character or avatar), it was necessary to use a database of avatar images designed by applying the characteristic expressions of the real-life space. Thus, it is a question of extracting a subset of vectors of parameters of the real-face appearance space, according to the influence of the vectors of parameters on the constitution of the space of appearance of the real face, and to determine the corresponding expressions, called characteristic expressions. According to the invention, the above-mentioned characteristic expressions are those which correspond to the parameter vectors forming the convex envelope of the real-appearance space. These characteristic expressions have already been determined during the generation of the control interface (see above, description with reference to FIG. 4).

• Creating the avatar image database

Thus, according to the invention, a database (12) of images of an avatar face, called the avatar database, is created (E3) by applying the actual real expressions, mentioned above, to a face. avatar. In practice, a human operator designs, using computer graphics tools, as many synthetic faces of a chosen avatar as there are characteristic expressions to apply. A database of avatar faces is thus designed, here called pseudo database, because the number of images is limited, in practice of the order of 25 to 30 images.

• Modeling deformations of the avatar face

Once the avatar image database (12) is available, model (E4) the deformations of the avatar face from the images of the avatar database. For this purpose, the same modeling process of the AAM type is used as that implemented in the context of the modeling of the real face deformations (see description above in relation to FIG. 2). We then obtain a model comprising a set of vectors of appearance parameters forming an avatar face appearance space (13). Each point of the appearance space (13) of the avatar face thus corresponds to a vector of shape-texture mixed parameters corresponding to a characteristic expression obtained from the real face appearance space (11). In practice, the appearance space (13) of the virtual face is smaller or equal in size (e.g., in the order of 15) to that of the actual face appearance space (11) (of order of 30), depending on the visual complexity of the avatar face. • Using the control interface (14) to navigate the avatar face appearance space

Since the facial expressions that correspond to the dominant directions of the real face image database have been integrated into the avatar image database to form the avatar face appearance space, the directions Dominants implemented in the reduced space of the 2D graphical control interface can be mapped to both the actual face appearance space (1 1) (see above) and the appearance space (13) avatar face. It is therefore possible to apply the control interface (14) to the navigation in the avatar face's appearance space (13), in order to allow the control (arrow CM2,

Fig. 1) the synthesis of any expressions on the avatar face.

Figure 6 illustrates the use of a 2-dimensional control interface according to an embodiment of the invention for controlling the facial expressions of a virtual character. As shown in FIG. 6, from the real face image database, a real face appearance space (left in the figure) is obtained, from which, thanks to the generation method of interface according to the invention, one can generate a control tool in the form of disk (in the center in the figure) to control the change of facial expressions of a virtual character (right in the figure). As can be seen on the disk in the center of the figure, a user

(not shown) of the interface positioned a marker (square) on the sector

"joy" of the disc, at a distance close to the periphery of the disc, which corresponds to an intensity of joy close to the maximum intensity.

Figure 7 illustrates the result of facial expression synthesis obtained on both the avatar face and the actual face, using a 2D control interface according to the invention. In Figure 7, three examples (EX1, EX2, EX3) of the synthesis results are shown. For each example, the synthesis is controlled by the disk-shaped 2D interface, again designated by 2D control space, by selecting a point on the control space, as represented by the black dot shown on the disks, middle column, of each of the EX1-EX3 examples. As said above, this selected point of the control space is for example chosen via a computer mouse. For each example, the left column shows the result obtained with the real face and the right column the result obtained with the face of the avatar.

Example EX1 illustrates three synthesized expressions on both the human face and the avatar face: a neutral expression (dot in the middle of the disc) (at the top), an expression of maximum intensity (the point being located on the edge of the disc) between surprise and fear (in the middle), and an expression of maximum intensity expressing surprise (below).

Example EX2 also illustrates three synthesized expressions on both the human face and the avatar face, these expressions corresponding to disgust (top), anger (middle), and an intermediate expression between anger. and surprise (below).

Example EX3 illustrates three other expressions synthesized on both the human face and the avatar face: an indefinite expression of joy and sadness (above), an expression clearly of joy (in the middle) but not of maximum intensity (the point is inside the disc), and an expression of maximum intensity of joy mingled with surprise (bottom).

As can be seen in Figure 7, the expressions obtained by the function CM2 (see Fig. 1) of synthesis applied to the avatar face are very close to those obtained by the function CM1 (see Fig. 1). of synthesis applied to the real face. One can even observe that wrinkles are present around the eyes on the face of the avatar, in particular in the example EX2 (faces of the top and the bottom). This is a good illustration of the fact that, according to the invention, the "knowledge" of the behavior of the human face has been transferred successfully to the synthetic face. In addition, expressions initially present in the database of images of the real face, as well as new expressions (intermediate or mixed), can be generated successfully on both faces. Figure 8 illustrates the emotional interpretation of the control space covered by the 2D control interface according to the invention. In this embodiment, the control space is partitioned into sectors of emotions corresponding to the categories of emotions defined by P. Ekman (see reference cited above in the description). As previously expressed, the type (or category) of expressions is determined by the angular position of a marker (a dot in Fig. 7) on the disc, while the radial position of the marker on the disc determines the intensity of expression.

It can also be observed that in the presented example of a disk-based 2D control interface, the color of the disk varies angularly continuously as a function of the expression change (from green to red), and varies in luminous intensity from continuously, radially, that is to say from the center to the edge of the disc, correlated with the intensity of the corresponding expression. This color scheme associated with the control interface disk, significantly increases the ergonomics of use of such a control interface according to the invention.

It will be noted here that in the context of the disclosed embodiment, the expression control interface of a real or synthetic face corresponds to a 2-dimensional space (in particular a disk). However, the invention can also be implemented by means of a more sophisticated control interface, for example 3-dimensional, such as a sphere and achieve a "ball of facial expressions", allowing, while remaining intuitive of use, to synthesize more diverse facial expressions. From a hardware implementation point of view, the invention is implemented in the form of software means, that is to say computer program modules performing the operations relating to the creation of a control interface of the computer. expressions of an avatar as described in the present description. Thus, it is possible to use a program module for implementing the modeling operations (E1, E4) and a software module for automatically generating the control interface (14) from the set of vectors of parameters (C vectors) obtained at the output of the modeling operation (E1) of the deformations of the real face.

In practice, the aforementioned program modules are implemented when they are loaded and executed in a computer or computer device. Thus, a computer, such as a personal computer, or a workstation, in which such programs are installed according to the invention is therefore an image processing system equipped with software and / or hardware capable of implementing a method for generating a facial expression control interface, and / or a corresponding facial expression synthesis method, according to the invention.

It will also be noted that a computer program according to the invention, the purpose of which is the implementation of the invention when it is executed by an appropriate computer system, can be stored on an information carrier of various types. . Indeed, such an information carrier may be constituted by any entity or device capable of storing a program according to the invention.

For example, the support in question may comprise a hardware storage means, such as a memory, for example a CD ROM or a memory of the ROM or RAM microelectronic circuit type, in particular incorporated in a USB key, or a means magnetic recording, for example a hard disk.

From a design point of view, a computer program according to the invention can use any programming language and be in the form of source code, object code, or intermediate code between source code and object code (for example eg, a partially compiled form), or in any other form desirable for implementing a method according to the invention.

Claims

1. A method for generating an interface for controlling the facial expressions of an avatar, characterized in that it comprises the following steps:

obtaining (E1) a model of deformations of a real face from the images of a database (10) of images of the real face, this model comprising a set of vectors of appearance parameters forming a real face appearance space (11); extracting (E2) a subset of vectors of parameters of the real face appearance space, as a function of the influence of the parameter vectors on the constitution of the real face appearance space, and determining expressions, called characteristic expressions, corresponding to the extracted parameter vectors; - generating (E2) a control graphical interface (14) adapted to navigation in the real-face appearance space, and for controlling (CM1) the synthesis of any expressions on the real face;

creating (E3) a database (12) of images of an avatar face, called avatar database, by applying said characteristic real expressions to an avatar face;

obtaining (E4) a model of avatar face deformations from the images of the avatar database, this model comprising a set of vectors of appearance parameters forming a face appearance space; avatar (13); - applying said control interface (14) to the navigation in the avatar face's appearance space (13), to allow control (CM2) of the synthesis of any expressions on the avatar's face .

The method of claim 1, wherein the real face deformation pattern is comprised of a set of appearance parameter vectors combining shape and texture information, said appearance parameter vectors constituting a space. vector in which each parameter vector is associated with a single facial expression corresponding to a single image of the real face image database.

The method of claim 2, wherein the appearance parameters of the real face deformation pattern are obtained using an Active Assumption Templates technique.

The method of any of the preceding claims, wherein the step of retrieving a subset of parameter vectors from the real face appearance space includes a step of selecting parameter vectors. corresponding to the points forming the convex hull in the real face appearance space.

The method of any one of the preceding claims, wherein the step of generating a control graphical interface comprises the steps of:

identifying, from the cloud of points constituting the actual face appearance space, a predetermined number of dominant directions of the appearance space, said dominant directions being determined using an envelope detection algorithm convex;

applying a mapping of each point of the real-face appearance space, carried by an identified dominant direction, to a point in the space constituting the control graphical interface.

6. Method according to any one of the preceding claims, wherein said graphical interface for controlling the synthesis of any expressions on the avatar face, is a graphic interface having the form of a disk in which: a type considered facial expression is determined by a given ray of the graphic disk, and is chosen according to the displacement of a marker on any circle among the concentric circles having for centers the center of the disc, and having as radius a radius between zero and the radius of the disc; and

A method according to any one of the preceding claims, wherein the avatar face deformation pattern is comprised of a set of appearance parameter vectors combining shape and texture information, said parameter vectors. Appearance constituting an appearance vector space, wherein each parameter vector is associated with a single facial expression corresponding to a single image of the image database of the avatar face.

The method of claim 7, wherein the appearance parameters of the avatar face deformation pattern are obtained using an Active Assumption Template technique.

9. Facial control interface of an avatar face, characterized in that it is obtained by implementing a method of generating a control interface according to any one of claims 1 to 8.

10. Use of a facial expression control interface of an avatar face obtained by an interface generation method, according to any one of claims 1 to 8, for the synthesis of facial expressions on a face avatar.

11. Use of a facial expression control interface of an avatar face obtained by an interface generation method, according to any one of claims 1 to 8, for the synthesis of facial expressions on a face human.

12. System for generating an interface for controlling the facial expressions of an avatar, characterized in that it comprises:

means for modeling the deformations of a real face from the images of a database of images of the real face, the model obtained comprising a set of vectors of appearance parameters forming a face appearance space real; means for extracting a subset of vectors of parameters of the real face appearance space, as a function of the influence of the parameter vectors on the constitution of the real face appearance space, and determining expressions, called characteristic expressions, corresponding to the extracted parameter vectors;

means for creating an image database of an avatar face, called an avatar database, by applying said characteristic real expressions to an avatar face;

means for modeling the deformations of the avatar face from the images of the avatar database, the model obtained comprising a set of vectors of appearance parameters forming an avatar face appearance space; means for generating a control interface suitable for navigation in the real-face appearance space, and for controlling the synthesis of any expressions on the real face, said control interface being applicable to the navigation in the appearance space of the avatar face, to allow automatic control of the synthesis of any expressions on the avatar face.

Interface generating system according to claim 12, comprising means for implementing a method for generating a control interface according to any one of claims 2 to 8.

14. Computer program, comprising instructions adapted to the implementation of all or part of the steps of a method of generating a control interface as claimed in any one of claims 1 to 8, when the program is run on a computer.