JP2004509391A - Avatar video conversion method and device using expressionless facial image - Google Patents

Abstract

The present invention provides a method and an apparatus for creating a moving image conversion product using an expressionless facial image. The avatar editor creates an avatar using the front image and the side image of the expressionless face model. The avatar uses the elastic bunch graph matching method to automatically find the location of the facial features on the front and side images. Significant time savings can be achieved by creating animated conversions by associating avatar meshes obtained from expressionless facial features with general avatar meshes. It is possible to improve the quality of an avatar obtained by applying a moving image conversion product for an expressionless work to another expressional avatar mesh. Animationless face-based video conversion does not require the long editing time required to create a specific video conversion for each specific facial expression (and / or pose) feature and greatly improves the facial expression head model.

Description

が感知され、得られたパラメータは

によって対応する

にマッピング処理できる。種々な表情に対応するいくつかのアバターメッシュは感知された表情のさらに正確な描写を提供する。異なる表情のためのメッシュは形態ターゲット（ｍｏｒｐｈ ｔａｒｇｅｔ）と呼称できよう。例えば、

は笑顔画像からの

を使用して創出できる。別な

は驚きや感嘆を示す顔面画像からの

を使用して創出できる。同様に、

は

に対応する。感知された

は１次回帰を利用して対応する

にマップ処理できる。
【００１５】
さらに写真的効果を生み出すには、それぞれの表情のノードポジションを手動で調べ、非自然物と歪みを補正すべきである。しかしながら、経験によれば、それぞれのアバター頭部モデルの補正には数分を要する。写真的アバターには１４から１８もの表情に基くアバターメッシュを必要とする。
【００１６】
大幅な時間の節約は

を使用した動画変換ρを創出させることで達成できる（ブロック２０、図１）。得られた

が式１に示されるようにアバター変換によって一般的

に関連付けられる。

得られるアバターの質を高めるために無表情造作の動画変換を他の表情アバターメッシュに適用することができる（ブロック２２）。例えば、笑顔に関連するアバターメッシュを式２に示すように無表情動画変換ρによって変換できる。

無表情顔面に基いた動画変換は、それぞれの特定表情（及び／又はポーズ）に対する特定動画変換を創出することで要する長い編集時間をかけずに顔面表情頭部モデルを大幅に改善する。
前述の説明で本発明の好適実施例を開示しているが、当業界技術者であれば本発明の範囲内でそれら実施例を幾様にも変更できよう。
【図面の簡単な説明】
【図１】本発明による無表情顔面画像を利用したアバター動画変換物の創出のための方法を表すフロー図である。
【図２】本発明によるアバターの創出アバターエディター（編集装置）を図示する。
【図３】図２のアバターエディターにより提供されたアンカーポイント（ａｎｃｈｏｒ ｐｏｉｎｔ）を利用して創出されたアバターの背面画像である。
【図４】本発明によるアバター画像から非自然物と歪みとを除去するように補正されたアンカーポイントポジションを利用したアバター創出用のアバターエディターを図示する。
【図５】本発明による図４の補正されたアンカーポイントポジションを利用して創出されたアバターの背面画像である。
【図６】アバターメッシュに対する感知された顔面造作部分の１次回帰マッピング（ｌｉｎｅａｒ ｒｅｇｒｅｓｓｉｏｎ ｍａｐｐｉｎｇ）のための顔面表情造作とアバターメッシュとの関係を示すグラフである。[0001]
[Related application]
This application claims the priority of US Provisional Patent Application No. 60/220330, filed July 24, 2000, entitled "Method and Apparatus for Converting Avatar Video Using Expressionless Facial Images," Claim the priority of Partial Continuation Patent Application Ser. No. 09 / 188,079 “Wavelet-based facial motion imaging for avatar video”.
[0002]
BACKGROUND OF THE INVENTION
The present invention relates to avatar moving images, and more particularly to moving image conversion using expressionless facial images.
[0003]
Avatar-filled virtual spaces are an attractive means of providing a shared environment experience. However, manually creating a photo-like avatar takes a long time, and automatic avatar creation involves intervening non-natural objects and deforming facial features.
[0004]
Therefore, an avatar editing means for quickly and faithfully providing an avatar head model is desired. The present invention fulfills that need.
[0005]
Summary of the Invention
The present invention provides a method of performing avatar moving image conversion using an expressionless facial image and an apparatus related thereto. This method provides front and side images of the expressionless face, and creates an avatar on the front and side images of the head using elastic bunch graph matching, and automatically detects the position of the facial features. Do. Nodes (nodes: basic points) are automatically allocated to the feature positions of the front image and the side image. The node position is scrutinized with the naked eye, non-natural objects are removed from the avatar created based on the node position, and the distorted features are minimized and corrected.
[0006]
The method can also include the creation of an animation transform based on the correction node position of the expressionless face. The method further includes applying the video transform to an expressive facial avatar mesh that is used to create the avatar.
[0007]
Other features and advantages of the present invention will become apparent from the following detailed description, taken in conjunction with the accompanying drawings.
[0008]
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
The present invention provides the method and apparatus shown in FIG. 1 for creating a moving image conversion using an expressionless facial image. The avatar editor uses the front and side images of the expressionless facial model to create an avatar (block 12). The avatar is created by automatically detecting the position of the facial feature on the front image and the side image using the elastic bunch graph matching method (block 14). A method for determining the position of a facial feature of an image using an elastic bunch graph matching method is described in U.S. Patent Application No. 09 / 188,079. In the expansion / contraction graph matching technique, an image is converted into a Gabor space using a wavelet transform based on a Gabor wavelet. The transformed image is represented by a complex wavelet component value associated with each pixel of the original image. The elastic bunch graph matching method automatically arranges a node graph having an anchor point on the front image and the side image, respectively. These anchor points are scored at the general locations of the facial features found using the matching technique (block 16).
[0009]
The avatar editor window 26 shown in FIG. 2 allows the user to create an avatar that appears similar to the model. A new avatar 28 is created based on the front image 30 and the side image 2 of the model. Alternatively, you can freely edit your existing avatar. The front image and the side image are mapped on the avatar mesh. You can animate the avatar on the mesh or move and move the drive control points. The movement of the drive control point can be controlled by tracking processing of the facial features.
[0010]
Initially, avatar editor window 26 includes a wizard (not shown) that guides the user through steps to improve avatar tracking accuracy. The avatar wizard includes a tutor face that allows the user to express some facial expressions and change the head pose. An image is formed for each facial expression or pose, and features are automatically located for each facial image. However, a specific non-natural object of an image may lead to the placement of a facial feature partial node at an incorrect position during the facial feature processing. In addition, accurate node locations can create non-natural objects that degrade the photographic avatar. Thus, the user has the opportunity to manually correct the automatically located feature position (block 18).
[0011]
For example, the front image 30 and the side image 32 shown in FIG. 2 have a shadow outline detected by an error as a profile of the side image 32. Also, features such as the ears of the model have a number of patterns that would cause node scoring errors. Of particular importance is the proper placement of eye and mouth nodes. Avatar 28 may have artificial eye or tooth inserts that are "exposed" when the eyes and / or mouth are open. Therefore, even if the matching process results in proper node scoring, the resulting avatar will contain distorted features.
[0012]
More photographic avatars can be provided by empirical adjustment of node scoring. For example, the rear view of the avatar 28 shown in FIG. 3 was created utilizing the node positions shown in the avatar editor window 26 of FIG. A particularly problematic non-natural object is the white patch 34 behind the head. This white patch is based on the fact that the node position automatically pointed patches a part of the white background of the side image 32 on the back of the avatar.
[0013]
Improperly scored nodes can be manually adjusted as shown in FIG. 4 to more precisely node the corresponding features. The general head models 36 and 38 display the node positions so that the user can accurately assign the node positions to the front image and the side images. A node is moved by clicking on the node with a mouse-like pointer and pulling the node to the desired position. Avatars based on exact node locations, as in the front view of avatar 28 ', also provide a photographic avatar. Further, the position of the node of the occiput on the side image is adjusted as shown in FIG. 5 so as to eliminate an undesirable white patch.
[0014]
The model images shown in FIGS. 2 to 5 are expressionless faces. As described above, images of various expressions and poses can be obtained by using training expressions. As shown in FIG.

Is sensed and the resulting parameter is

Corresponding to

Can be mapped. Some avatar meshes corresponding to different facial expressions provide a more accurate depiction of the sensed facial expressions. Meshes for different facial expressions could be referred to as morph targets. For example,

Is from the smile image

Can be created using Another

From a facial image showing surprise or exclamation

Can be created using Similarly,

Is

Corresponding to Perceived

Correspond using linear regression

Can be mapped.
[0015]
To create more photographic effects, you should manually examine each expression's node position to correct for unnatural objects and distortion. However, experience has shown that correcting each avatar head model can take several minutes. Photographic avatars require 14 to 18 facial expressions based avatar meshes.
[0016]
The significant time savings

(Block 20, FIG. 1). Got

Is generalized by the avatar transformation as shown in Equation 1.

Is associated with

An expressionless animation conversion can be applied to other expression avatar meshes to enhance the quality of the resulting avatar (block 22). For example, an avatar mesh related to a smile can be converted by the expressionless moving image conversion ρ as shown in Expression 2.

Expressionless face-based video conversion greatly improves the facial expression head model without the long editing time required to create a specific video conversion for each specific expression (and / or pose).
While the foregoing description discloses preferred embodiments of the present invention, those skilled in the art will be able to modify those embodiments in many ways within the scope of the present invention.
[Brief description of the drawings]
FIG. 1 is a flowchart illustrating a method for creating an avatar moving image conversion using an expressionless facial image according to the present invention.
FIG. 2 illustrates an avatar editor (editing device) according to the present invention.
FIG. 3 is a rear view image of an avatar created using an anchor point provided by the avatar editor of FIG. 2;
FIG. 4 illustrates an avatar editor for avatar creation using an anchor point position corrected to remove non-natural objects and distortion from an avatar image according to the present invention.
FIG. 5 is a rear view image of an avatar created using the corrected anchor point position of FIG. 4 according to the present invention.
FIG. 6 is a graph illustrating a relationship between a facial expression feature and an avatar mesh for linear regression mapping of a sensed facial feature portion with respect to an avatar mesh;

Claims (13)

A method of creating an avatar video conversion product,
Providing an expressionless face front image and a side image for avatar creation;
Automatically finding a head feature position on the front image and the side image utilizing a telescopic bunch graph matching method,
Automatically arranging a node at a feature position on the front image and the side image;
Manually checking and correcting the node position to remove non-natural objects in the avatar created based on the node position and minimize distorted features;
A method characterized by comprising: The method of claim 1, further comprising the step of creating a moving image transform based on the node positions corrected for the expressionless face. The method of claim 2, further comprising applying a video transform to the facial expression avatar mesh to create an avatar. The method of claim 2, further comprising applying a video conversion to the morphological target. An avatar video conversion product creation device,
Means for providing an expressionless face front image and a side image for avatar creation;
Means for automatically finding a head feature position on the front image and the side image using a telescopic bunch graph matching method,
Means for automatically arranging nodes at feature positions on the front image and the side images;
Means for manually checking and correcting said node position to remove non-natural objects in the avatar created based on said node position and to minimize distorted features;
An apparatus characterized by comprising: The apparatus of claim 5, further comprising means for creating a moving image conversion based on the node position corrected for the expressionless face. The apparatus of claim 6, further comprising: means for applying a moving image transform to the facial expression avatar mesh to create an avatar. The apparatus of claim 6, further comprising means for applying a video conversion to the morphological target. A method of creating an avatar video conversion product,
Providing an expressionless face front image and a side image for avatar creation;
Automatically finding a head feature position on the front image and the side image using an image analysis method based on a wavelet component value created by a wavelet transform of each expressionless face front image and side image; ,
Automatically arranging a node at a feature position on the front image and the side image;
Manually checking and correcting the node position to remove non-natural objects in the avatar created based on the node position and minimize distorted features;
A method characterized by comprising: The method of claim 9, further comprising the step of creating a moving image transform based on the node positions corrected for the expressionless face. The method of claim 10, further comprising applying a video transform to the facial expression avatar mesh to create an avatar. The method of claim 10, further comprising applying a video transform to the morphological target. The method of claim 9, wherein the wavelet transform utilizes Gabor wavelets.

Images