JP2002092626A - Image processor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To reduce data quantity of an animation so as to efficiently form and manage parts data and so as to dynamically change expression formation. SOLUTION: An image processor is provided with a motion type designating means 101 designating one or more information about parts data, an expression parameter storing means 104 storing parameter data for giving motion to the parts, a data linking means 106 relating the parts data and the parameter data to each other, a parameter data extracting means 105 extracting the parameter data from the expression parameter storing means 104 on the basis of information about the motion designated by means of the motion type designating means 101, and an animation composing means 107 forming a part animation image with a motion from the parts data and the parameter data related mutually by means of the data linking means 106 and synthesizing other unrelated parts data into the same line for forming a single moving image.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention is used in personal computers, word processors, workstations, portable information tools, copiers, scanners, facsimiles, televisions, videos, video cameras, etc., and uses techniques such as computer graphics. Processing method for performing processing such as storage, transmission, drawing, and the like of animation data created in advance, and an apparatus for performing the same. In particular, by combining partial images, a moving image with a cartoon-like The present invention relates to an image processing apparatus capable of generating an expression moving image by combining images, for example, parts of a person's face.

[0002]

2. Description of the Related Art In recent years, with the spread of personal computers, word processors, digital cameras, and the like, it has become possible to easily acquire images and perform various processes on the acquired images to enhance practicality and entertainment. It is becoming. As one of the image processing, specific features related to the input image, for example, the position, size, and shape of eyes and mouths in a person image are extracted, and a cartoon-like portrait is extracted based on the extracted information. There are techniques for generating images.

Further, when the relationship between the position and the size of an object arranged in an image satisfies certain constraint conditions, a method of extracting a feature amount related to the object in the image and using the extracted characteristic amount for reconstructing the object image. For example, Japanese Patent Application No. 11-1210
No. 97, when the image is a human image, eyes, nose,
A method of extracting features such as a mouth and a face contour and generating a cartoon-like portrait is disclosed. Also, JP-A-10-2409
No. 19 discloses a technique of preparing rules of component arrangement and deformation together with component images as a library in advance and selecting an arbitrary rule to generate a different portrait image from a single image. I have.

In these caricature techniques, each part of the face prepared in advance is synthesized and displayed. Therefore, a plurality of continuous still images are assigned to each part to form a partial moving image, so that the face of the caricature is obtained. It is also possible to perform facial expression animation by moving parts.

On the other hand, as one of animation production techniques in computer graphics, there is a technique called key frame animation. In key frame animation, instead of creating all frame images in advance, objects within a key frame for movement (hereinafter referred to as key frames) are created, and objects to be drawn in frames other than key frames are key frames. Automatically generated by interpolating and calculating frame objects.

If this key frame animation technique is applied to the face part image, the facial expression animation can be performed in the same manner as when the part image is converted into a partial moving image by assigning continuous still images. is there. Since the method of applying the key frame animation does not need to record all the frame information, the amount of data to be recorded as a whole can be greatly reduced.

[0007]

However, in the case where an animation is created by combining partial moving images, such as in the case of facial expression animation, there is not much problem if only a single facial expression animation is drawn.
When performing animation drawing in which a plurality of facial expressions can be arbitrarily specified, animation part data for the number of facial expressions must be prepared. Furthermore, in the above portrait technology, each part constituting the image is not a single part, but a large number of different part data corresponding to different feature amounts of the face to be a portrait, for example, features such as thick and thin eyebrows. I have to prepare it. That is, (number of facial expressions) x (number of parts) x (number of parts) video data is required,
A huge amount of video data must be created in advance,
Even if moving image data is created by key frame animation, there is a problem that an increase in data creation cost is inevitable.

[0008] In addition, when an expression animation is to be used for the purpose of real-time communication, etc., a function of continuously reproducing various expressions along a time axis, an animation corresponding to a user to use and a situation at that time. It is necessary to efficiently manage animation playback, such as the ability to change and modify the playback time and degree of movement in real time, but simply playing back a video created in advance is such a dynamic response Was difficult.

SUMMARY OF THE INVENTION The present invention has been made to solve the above problems, and its object is to reduce the amount of data for a moving image, to efficiently create and manage part data, and to dynamically change expression generation. It is an object of the present invention to provide an image processing apparatus capable of performing such operations.

[0010]

In order to solve the above-mentioned problems, an image processing apparatus according to the present invention has a plurality of types of parts having a certain constraint on a shape and a configuration thereof, and a plurality of stationary parts for each type of parts. Component data storage means for storing image data, component selection instruction means for selecting and instructing an arbitrary component, component data extraction means for extracting appropriate component data from the component data storage means, and component data extraction means Means for combining one or more images by combining component data extracted by
A movement type designating unit for designating one or more pieces of information relating to the movement of the component data, a parameter data storage unit for storing parameter data for giving a motion to the component, and a data link unit for associating the component data with the parameter data A parameter data extracting unit that extracts parameter data from the parameter storage unit based on information on the motion specified by the motion type specifying unit; and a component data and a parameter data associated by the data link unit. It is characterized in that it comprises a moving image generating means for generating a partial moving image with a motion and combining other unrelated component data in the same row to generate a single moving image.

According to the present invention having such features,
The part data and the motion parameter data are clearly separated and handled, and are associated by the data link means. Accordingly, it is not necessary to create a series of partial moving images in advance, and a partial moving image can be dynamically generated according to the association. In other words, the component data and the motion parameter data can be separately corrected, and efficient creation and management of animation data can be performed. In addition, it is possible to dynamically change the creation of the partial moving image according to the situation, and it is possible to change or correct the motion in real time. Furthermore, by generating a moving image for each part, a part that does not require a moving image, for example, a nose part in the case of a close-up expression, can use a still image part as it is,
Since it is not necessary to generate useless partial moving images in advance, the production amount can be reduced.

Here, as a method of separating the component data and the motion parameter data, any method may be used as long as the effect can be expected. For example, a key frame animation method may be used. By using, efficient management can be performed. That is,
If the still image component data is the first key frame, the above-described separation method can be realized by associating the parameter data representing the corresponding motion as the next key frame. The intermediate frame between the key frames is calculated by interpolation. The interpolation method is disclosed in, for example, Japanese Patent Application Laid-Open No. 60-191366, and the calculation method itself includes linear interpolation and spline. A method using curve approximation is known as a known technique.

[0013] In addition, in the key frame animation, in order to perform interpolation calculation, information of association such as which object is associated between key frames is important. In the key frame animation, the components of an object are usually defined as lines or surfaces as coordinate points or a group thereof, but the association of these components is defined by separately defining an association table and recording / transmitting it. In a normal animation, there is only one association table for one key frame. However, when the shape and the configuration of the object to be animated have a certain constraint condition, that is, the object to be animated can be specified in advance like the face part of the portrait, and the type of the face part is determined in advance. If certain constraints can be set for the components and their shapes of parts, if the same part type, even if the association of the objects is dynamically changed, the Can be assigned. That is, even if another key frame is dynamically selected as a target of association, the key frames can be connected to generate an intermediate frame by interpolation.
For example, if the eye part is divided into white-eye parts, black-eye parts, eyelashes, and eye corners, and the coordinate data of each element is defined in advance and the part data is created, the movement parameter as the next key frame is obtained. The same association table can be applied even if the data changes dynamically.

Further, the image processing apparatus of the present invention, in the above configuration, further comprises a parameter sharing means for sharing parameter data by associating the same parameter data with a plurality of component data. Features.

That is, by sharing the parameter data, it becomes possible to reduce the parameter data prepared in advance and to compress the data amount. Normally, if the shape of a part is different, a moving image of the number of parts is necessary.However, if the parts are of the same category, such as eyes and mouth, like a facial part of a portrait, the part The components of the object (such as the configuration / arrangement of the lines and surfaces and the color scheme) are similar. If the purpose is to generate a moving image that changes in a short time like an expression, as long as it returns to the original still image component state, the same parameter data can be used regardless of the different components. Even if the moving images are the same, it is said that there is not much effect on the meaning of the expression, for example, whether the expression is "laughing" or "anger".

As described above, when it is assumed that the shape of an object to be animated and its configuration have certain constraint conditions, if the same type of part is used, even if the association of the object is dynamically changed, It is possible to establish a correspondence between objects without making any modifications. For example, if eye parts are divided into white-eye parts, black-eye parts, eyelashes, and corners of the eyes, and the number of coordinate points for each element is defined in advance and part data is created, the shape will be similar to a drooping eye and a seam. The same association table can be applied to parts having slightly different features. As a result, it is possible to prepare only one type of parameter data representing the movement of "laugh" and "anger" without preparing moving images of "laugh" and "anger" separately for the drooping eyes and the stitches. The moving image can be dynamically created by connecting to the component images of the sag and the seam, respectively. As a result, the required amount of moving image data can be reduced.

Further, in the image processing apparatus of the present invention, in each of the above configurations, the position of the component image to be finally synthesized in the image, the size in the image, the inclination, the color information, the order between the images,
It is characterized by further comprising means for changing the transmittance.

That is, after a partial moving image or a still image is created for each component type by the above configuration, the position of the partial image in the image to be finally synthesized can be determined. Whether to enlarge or reduce the image size, add a tilt (rotation / verse) to the partial image, change the color information itself of each pixel (for example, uniformly change the color to red, etc.) A function to change the image information at the stage of synthesizing the partial image, such as changing the value of the blending ratio such as the pixel luminance between the images if it overlaps with the background image or other partial images With this, it is possible to dynamically generate various moving images. For example, in the case of animation processing of a portrait, by increasing the magnification of the partial image of the eyes to be synthesized in proportion to the moving image with a surprised expression,
Multiple expressions can be generated from the same partial data to enrich the expression.

Further, the image processing apparatus according to the present invention further comprises a data relinking means for associating the parameter data with the generated partial moving image again in each of the above structures, Is characterized in that another moving image is continuously generated in accordance with

In each of the above-described configurations, the association is performed between the component data and the motion parameter data, and the partial moving image is finally generated by this. In this configuration, the partial moving image is finally generated. A mechanism for reconnecting motion parameter data to a partial moving image is provided, and another moving image is continuously generated.

In the case of a portrait animation, in the case of continuous moving image reproduction of facial expressions, the face must be in a true face state (still image).
Assuming that the process returns to the above, continuous reproduction can be realized by generating a partial moving image again from the component data and the motion parameters and performing overwrite drawing of the still image being displayed. However, in this method, since the user always returns to the true face, the user may feel uncomfortable. For example,
When performing continuous facial expression playback such as getting angry immediately after being surprised,
It becomes more natural expression to shift from surprised expression to angry expression continuously. The above configuration enables such continuous reproduction by reapplying the motion parameter data to the created partial image.

In the case of key frame animation, it is necessary to extract the middle frame as a key frame in order to reattach the parameter data to the middle frame and to continuously reproduce the data. For this reason, for example, the parameter data is composed of a plurality of key frames in advance, and the reproduction interruption part is limited to the key frame in the middle, or the suspended frame in the reproduction is reconstructed as the key frame. Thus, the state during reproduction is extracted as a key frame. Also, the reproduction of the moving image is temporarily ended up to the key frame. If the extracted key frame is used as the first frame and the next frame is used as a new key frame extracted from the motion parameter data, a new partial moving image is generated, and the reproduction is resumed by overwriting and drawing again. It is possible to carry out continuous reproduction without any problem.

Further, in the image processing apparatus of the present invention, in the above-described configuration, the motion type specified in advance is specified every predetermined period, and the data relinking unit associates the parameter data again. And a means for intermittently generating a moving image.

Intermittent moving image reproduction can be realized by performing explicit designation from the motion type designation means in order along the time axis. On the other hand, the above-described configuration has a means for autonomously reproducing a moving image without explicitly instructing continuous reproduction.

That is, at predetermined time intervals, data re-set so that the previously set motion parameter data is dynamically linked to the currently displayed partial image regardless of the designation of the motion designation type means. It has means for designating the link means. Thus, for example, in the case of facial expression animation, more physiological expressions other than facial expressions such as "laughing" and "angry", for example, facial expression such as "blink", regardless of whether the facial expression is being reproduced by explicit designation , It is possible to play back autonomously. In other words, by using the data re-linking means, even when the expression "angry" is reproduced, for example, it is possible to automatically connect and reproduce the part of the blinking motion to the eye part. Therefore, a more natural expression reproduction can be performed without any explicit designation.

[0026]

Embodiments of the present invention will be described below with reference to the drawings.

The image processing apparatus of the present invention is an apparatus for generating an arbitrary moving image by synthesizing partial images (objects constituting the image). Here, as one embodiment, an image processing for generating an animation of a human face expression as a final moving image using an image of each part in a human face as a partial image will be described. .

[First Embodiment] FIG. 1 is a functional block diagram showing the configuration of an image processing apparatus according to the first embodiment.

The part selection designating means 100 is a means for designating a part image of each part of the face, and is used by a user to create a face image by arbitrarily combining the part images.
The motion type specifying means 101 is a means for specifying the type of the generated facial expression animation (laughing, angry, crying, etc.), and allows the user to input an arbitrary facial expression. The face part data storage unit 102 stores a plurality of vector image data having different shapes, colors, and the like for each part of the face (such as eyes, mouth, and nose). The component data extraction means 103
This is a means for extracting and transmitting vector image data (part data) of an arbitrary part from the face part data storage means 102. The expression parameter storage unit 104 stores expression parameter data obtained by parameterizing the movement (change in expression) of the component data.

The parameter data extracting means 105 is a means for extracting and transmitting arbitrary expression parameter data from the expression parameter storage means 104. The data link means 106 is based on the component data of each part of the face specified by the component selection instructing means 100 and the information of the facial expression type specified by the motion type specifying means 101, respectively, Data extraction means 105
Is used to dynamically select component data and parameter data as appropriate, combine the two data as component data having a facial expression movement, and transmit the combined data to the dynamic synthesizing unit 107.

The dynamic synthesizing unit 107 creates a partial moving image of each part of the face based on the combination of the transmitted part data and the expression parameters, and extracts a partial still image of an unlinked part from the part data extracting unit 103. Is a means for generating a single moving image by combining the respective partial images after the extraction using. The image output unit 108 is a unit that outputs the combined moving image generated by the dynamic combining unit 107 to a display device such as a CRT.

In the above configuration, the component selection designating means 10
In the first embodiment, 0 is provided with means for arbitrarily specifying parts of each part of the face. Also, a list of selectable parts may be displayed on a display device such as a CRT so that the operator of the image processing apparatus can arbitrarily select the parts with a mouse or a cursor key, or select parts to be separately selected. The information may be described as a list, and the list may be used as an input.
Alternatively, the feature amount of each part is extracted from the photograph of the face of the person,
Each part of the face may be automatically determined based on the feature amount, and the result may be input as a list. As described above, for example, Japanese Patent Application No. 11-121097 discloses a method of extracting a feature amount from a portrait of a person and selecting a part.

The motion type specifying means 101 is a means for arbitrarily specifying a facial expression. For example, it is realized by displaying an icon that abstractly represents a type of expression that can be selected by a display device such as a CRT, and arbitrarily selecting the icon with a mouse or a cursor key or the like. As described above, the input from the motion type specifying unit 101 may be realized interactively, or a list specifying one or more facial expressions along a time axis may be described in advance, and the list may be described by the motion type specifying unit 101. It may be implemented as an input, and display designation is sequentially performed internally along the time axis.

Further, the face part data storage means 102, the part data extraction means 103, the expression parameter storage means 10
4. Parameter data extracting means 105, data link means 106, and dynamic synthesizing means 107 are constituted by an arithmetic unit and a storage device (not shown), and perform arithmetic processing on data stored in the storage device. By doing so, the purpose of each means shall be achieved. These arithmetic processings may be uniformly performed by a processing program stored in advance on the same storage device.

The image output means 108 is a means for outputting a moving image generated by the image processing apparatus on a display device such as a CRT. If the generated moving image is composed of image information in a raster format, the pixel information is sequentially displayed for each unit time specified according to the display device. Alternatively, if the moving image is composed of image information in a vector format, the image is drawn from the vector information as needed, and is displayed sequentially for each unit time.

Next, an image processing operation performed by the image processing apparatus having the above configuration will be described with reference to a flowchart shown in FIG.

First, in step S11, the operator selects and specifies a desired part image of each part of the face using the part selection specifying means 100. As described above, the component selection designation unit may be interactively input and designated by the operator, or may be designated by a list. In addition, in order to avoid the need for the operator to specify parts for all parts, parts to be used as standard may be determined in advance as parts for parts not specified by the operator.

Next, in step S12, a desired facial expression is designated by using the motion type designation means 101. As described above, the motion type designation unit 101 may be interactively input and designated by the operator, or may be designated by a list. In particular, when there is no designation by the operator, a standard expression type may be determined in advance, and that expression may be selected.

Next, in step S13, the data linking means 106 extracts component data using the component extracting means 103 based on the information on the component specified in step S11. The component data in the first embodiment is image information processing in a vector format, and includes a coordinate sequence representing the shape of the object, color information of the object, and the like. Further, link information of the component is simultaneously extracted as information accompanying the component data. The link information is information relating to parameter data to be connected (linked) to the extracted component, and specifies a group of parameter data necessary for recreating the component as a moving image. Details of the link information will be described later.

Next, in step S14, the data linking means 106 determines parameter data to be extracted from the information on the expression designation specified in step S12 and the link information extracted in step S13. From the parameter data belonging to the extracted parameter data group, necessary parameter data is specified based on the expression specification.

Next, in step S15, the data link means 106 uses the parameter data extraction means 105 to convert the parameter data specified in step S14.
Extract parameter data.

Next, in step S16, the data link means 106 connects (links) the parameter data extracted in step S15 to the component data extracted in step S13, and connects the component data (motion expression component) having motion information. Data). The expression part data has a key frame animation configuration. The details of the method of creating expression part data will be described later.

Next, in step S17, the dynamic synthesizing means 107 creates a partial moving image for each part of the face. Among the parts of the face, parts that need to be moved, such as eyes and mouth, create partial moving images based on the expression part data created in step S16. In the first embodiment, since the expression part data is created as a configuration of the key frame animation, it can be used as it is as a moving image format.

On the other hand, with respect to other parts that do not need to move, for example, partial moving images such as a nose and a face contour, the part data extracting means 103 is used based on the part information specified by the part selection specifying means 100. After extracting the part data, the part data is connected to the parameter data having the motion information of "no change over time" to generate a partial moving image in the same format as the expression part data. By creating partial moving images of all parts in the same format as described above, time management and the like can be uniformly processed when combining partial moving images into one moving image.

Next, in step S18, the moving image synthesizing means 107 converts the partial moving image created in step S17 into one.
Are combined into one moving image. Partial moving images are created in the same moving image format, and time management is also defined internally, so the format of the composite moving image defines each partial moving image format and defines the arrangement information of each partial moving image do it. Details of the composite moving image format will be described later.

Next, in step S19, the combined moving image created in step S17 is displayed on a display device such as a CRT by the image output means 108. In the first embodiment, since the synthesized moving image is created as a moving image format obtained by synthesizing the key frame animation, the moving image format is analyzed, decomposed into partial moving images, and interpolation is calculated between the key frames. It is necessary to generate an image every unit time. A method of generating an image by interpolating between key frames is disclosed in, for example, Japanese Patent Laid-Open No. 60-191366, as described above.

The link information extracted in step S13, the link processing performed by the data link means 106 in steps S14 to S16, and the method of creating the generated expression part data will be described in more detail below.

In the first embodiment, the component data storage unit 1
02 stores link information together with each component data. The link information may be stored in a storage device separate from the component data. However, the link information must maintain a one-to-one relationship with each component data and must be able to be extracted in pairs with the data.
The link information is information for specifying parameter data for animating each component, and in the first embodiment,
Parameter data for the number of facial expressions of each part is managed as a group, and a unique number is assigned to each group. Then, this group number is defined as link information.

The data link means 106 determines in step S1
In step 3, component data is extracted, and this link information is also extracted. Next, in step S14, a group of parameter data is specified based on the extracted link information, and the parameter data within the group is uniquely specified based on the expression specifying information specified by the motion type specifying unit 101. I do. Then, step S
At 15, the specified parameter data is extracted from the facial expression parameter storage unit 104.

FIG. 3 is a view schematically showing how parameter data is specified by specifying link information and expression, using a mouth part as an example. That is, regarding the mouth part, the parameter data group “mouth part A” is pointed as the link destination, and the parameter data is specified when the operator specifies “laugh” as the display specification. I have.

Here, component data and parameter data according to the first embodiment will be described.

The component data is still image data in a vector format, and is composed of a coordinate sequence representing the shape of an object in the image, color information, and the like. Usually, even if the data is vector data, the shape of the object is various, and the number of coordinates and the like cannot be limited with respect to the coordinate sequence of the shape of the object.
However, if an object has a constant feature amount, its shape can be defined to some extent in advance. In the first embodiment, there is a limitation of a face part, and the shape within each part type does not greatly differ if each part type such as an eye, a nose, and a mouth is considered separately. Therefore, each component data can be created after the number of coordinates of the coordinate sequence representing the shape is defined in advance.

That is, if each component data and parameter data are of the same component type, management can be performed independently without being aware of each other's combination.
Thus, as long as the rule of the shape is followed, even if the component data and the parameter data are separately corrected, one of them is not affected, so that efficient animation management can be performed. Also, it is possible to easily correct and change the parameter data dynamically before connecting.

For example, processing such as flexibly changing the reproduction time of a moving image can be easily performed. Furthermore, with respect to component data that does not need to be animated, if parameter data indicating "no change over time" is internally connected, it can be easily converted to the same format as other partial moving image formats. As a result, "moving images that do not change over time"
Can be reduced compared to the case where the number of components is prepared in advance for the number of components.

FIG. 5 shows an example of a specific configuration of the component data and parameter data. This shows a format example of the component data of the left eye shown in FIG. 4 and a format example of parameter data connected to the component data. As shown in FIG. 4, this part data is composed of a total of nine graphic elements of white eye, black eye 1, black eye 2, iris, hole, lower eyelid, upper eyelid, line 1, line 2, and the information thereof. The storage format is the outline information P (X (n), Y (n)) of the figure.
And the color C (R, G, B) of the figure. Here, the coordinates (X (n), Y (n)) represent coordinate points on the figure outline, and information corresponding to the number of coordinate points is stored. Also,
(R, G, B) represents a color for filling the figure,
The luminance values of red (R), green (G), and blue (B) are stored.

On the other hand, the parameter data defines, for each graphic element of the component data, a shape contour coordinate sequence after a certain time has elapsed. The information storage format is
Time designation T (t) and contour information Q (X (n), Y
(N)). Here, t represents the elapsed time, and is specified, for example, by a multiple of the unit time. Also, (X
(N), Y (n)) represent coordinate points on the contour of the figure as in the case of P, and information corresponding to the number of coordinate points is stored.
The outline information P and Q and the color information C are set for each graphic element, and are sequentially stored in the component data and the parameter data according to the drawing order of each graphic. Since the storage order of these graphic elements is the same for the same part type, they can be easily specified.

As described above, the extracted component data and parameter data are sequentially connected (linked) to be converted into a moving image format (expression component data) by key frame animation. FIG. 6A shows a connection example 1 of the above-described left eye component data and parameter data.

In connection example 1 in FIG. 6A, first, key frames P and Q are defined. First, the frame time from P to Q is described by T (P, Q, 200). Next, for each graphic element, the outline information for the key frame P is P (X (n), Y (n)), and the outline information for the key frame Q is Q (X (n), Y ( n)). The color information is specified for each element by C (R, G, B). In connection example 1, the change in the color information is not considered, and the figures in the key frames P and Q keep the same color.

Also, the component data that does not need to be animated can be converted to a moving image format by connecting parameter data that does not change over time. There is no problem simply re-specifying the shape information of the part data, but if you define an information format that does not change over time, you can connect more easily and reduce the amount of data. Is also effective. For example, in connection example 2 shown in FIG. 6B, although the same graphic element as connection example 1 is used,
The outline information Q at the time of the key frame Q is described as Q (P). This indicates that the outline information is the same as the outline information P for the key frame P.

By the above processing, the expression part data of each part of the face can be dynamically created. By changing the parameter data to be connected, arbitrary expression part data can be dynamically created.

Further, these expression part data are combined to form one moving image. FIG. 7 shows an example in which the expression part data is combined into a single moving image format. In this example, arrangement information and expression part data are stored in order for each partial image. First, the arrangement information L on the partial moving image of the eyes
Is described. The arrangement information L includes the vertical and horizontal sizes (w, h) of the partial images and the coordinate positions (x, y) in the composite image. Next, the eye expression part data generated through the above process is described as it is. Other parts types such as mouths are similarly enumerated and described for the number of part types.

When outputting an image, the coordinate position (x, y) in the composite image is set at the center, and a frame of size (w, h) is
It is rendered as a key frame animation along with the time designation in the facial expression part data. In this way, various expression animations can be easily created by combining various expression part data. FIG. 8 shows an example of the created facial expression animation.

Here, the image processing apparatus of the first embodiment has parameter sharing means for sharing parameter data among a plurality of component data. For example, in FIG. 3, the parameter data connected to the mouth part data is referred to as “mouth part A” in the parameter data group, and this “mouth part A” has a one-to-one relationship with the part data. It does not have to be.

That is, the parameter data belonging to the “part A of mouth” is common to a plurality of part data, and the same parameter data can be connected to a plurality of part data. In other words, there is a limit to a part type having a constant feature amount, a face part in the first embodiment, and if it is considered separately for each part type such as an eye, a nose, and a mouth, the shape within each part type is considered. Does not greatly differ, so that if the component types are the same, the configuration of the graphic elements of the component data does not change, meaning that the same parameter data can be connected.

Further, it changes in a short time like an expression,
Also, in the case of a moving image that returns to the original image, by using the same parameter data, even if the key frame for changing the expression of the moving image becomes the same, the meaning of the expression, for example, "laughing" It doesn't have much effect on telling whether it's an anger. Even with different component data, an intermediate image is created by interpolation calculation by key frame animation, so that a moving image without discomfort can be created. As described above, it is possible to significantly compress the information for a moving image as compared to creating a moving image for each component.

In the image processing apparatus according to the first embodiment,
When synthesizing the partial moving image by the dynamic synthesizing unit 107, a unit for changing the position of the partial moving image in the image, the size in the image, the inclination, the color information, the order between the images, and the transmittance is provided.

In the first embodiment, when synthesizing a partial moving image, the synthesized moving image is described by sequentially storing the arrangement information L and the facial expression component data as shown in FIG. By having means for dynamically changing the arrangement information L, the position in the image and the size in the image can be easily changed.

For example, suppose that the motion type specifying means 101 further specifies the expression "very surprised" when the expression "surprise" is specified. If the expression designation is added, and if this change means is not provided, it is necessary to add expression parameter data at the same time. But,
In the case of having this change means, when "very surprised" is specified, the facial expression parameter data is created using the normal "surprise" parameter data to create facial expression part data, Image size (w,
h) for the eye image size specification that normally uses
Specify a large value. As a result, another moving image can be generated using the same expression part data, and the expression can be easily added without the trouble of creating the data.

Note that among the designations other than the position of the partial moving image with respect to the composite image and the size in the image, the inclination, color information, and transmittance of the image, in addition to the arrangement information L, each pixel of the partial moving image This can be realized by adding information that uniformly changes the information. Further, the order between images can be realized only by changing the storage order of the expression part data. The instruction to change the method of synthesizing these component moving images may be implemented so that the instruction is included in the expression type designation of the motion type designation unit 101 as in the above example, or the designation unit is separately provided. It may be provided.

[Second Embodiment] FIG. 9 is a functional block diagram showing the configuration of an image processing apparatus according to the second embodiment.

Among the functional blocks shown in FIG. 9, component selection designation means 200, motion type designation means 201, face part data storage means 202, component data extraction means 203, facial expression parameter storage means 204, parameter data extraction means 205, Data link means 206, dynamic synthesis means 20
7. The image output unit 208 is a component selection / designation unit 100, a motion type designation unit 101, a face component data storage unit 102, a component data extraction unit 103, and a facial expression parameter storage unit, which are components of the first embodiment shown in FIG. 10
4. It has the same functions as the parameter data extracting means 105, the data linking means 106, the dynamic synthesizing means 107, and the image output means 108, so that the functions to be added and changed are described each time. Omitted.

The means newly added in the second embodiment is the data relinking means 209. This is a means for providing a function for performing continuous moving image generation, a function for converting an intermediate image of a partial moving image once created into component data again, and connecting parameter data to this component data. Having. Further, the dynamic synthesizing unit 207 shown in FIG. 9 has a function of extracting an intermediate image of a partial moving image and transmitting the intermediate image to the data relinking unit 209 in addition to the function described in the first embodiment.

The data relinking means 209 is composed of an arithmetic unit and a storage device, although not shown.
The purpose of each means is achieved by performing arithmetic processing on data stored on the storage device. These calculation processes include a face part data storage unit 202, a part data extraction unit 203, an expression parameter storage unit 204,
Parameter data extraction means 205, data link means 2
06 may be uniformly processed by a processing program stored in advance on the same storage device as that of the storage device 06.

Next, an image processing operation performed by the image processing apparatus having the above configuration will be described with reference to a flowchart shown in FIG. Since the second embodiment is realized by adding a function to the first embodiment, the image processing operation is partly the same as that of the first embodiment. Therefore, the description of the processing of the same operation is omitted here.

In steps S 21 and S 22, the designation of a face part and the designation of a facial expression are performed using the part selection designation means 200 and the movement type designation means 201. The specific processing is the same as step S11 and step S12 in the flowchart (FIG. 2) of the first embodiment, and thus the description is omitted.

Next, in step S23, it is determined whether or not a display animation is currently being generated, that is, whether or not the dynamic synthesizing means 207 is currently processing. This determination mechanism is incorporated in the data link unit 206, for example. It is determined whether to proceed to step S24 or to step S27 according to the determination result.

If it is determined that the display animation is being reproduced, the process proceeds to step S24. Step S24
In the case where the intermediate image of each partial moving image currently being reproduced is a dynamic image, this intermediate image is calculated by interpolation calculation. In the key frame animation, it is possible to create this intermediate image as image data in a vector format.

Next, in step S25, the data linking unit 206 converts the intermediate image acquired in step S24 into a component data format. As described above, in the second embodiment, an intermediate image in a vector format can be obtained, and it is easy to convert this to a component data format as shown in the example of FIG.

Next, in step S26, the data relinking means 209 sends the parameter data extracting means 205 based on the expression designation designated by the movement type designation means 201.
Is used to extract parameter data. As described with reference to FIG. 3 in the first embodiment, normally, it is not possible to specify parameter data only by specifying display. However, if the facial expression animation is being reproduced, it means that the link information has already been extracted based on the component designation and the parameter group number has been once specified. Since the link information is the same as long as the selection of parts does not change, by using the link information again, the parameter data can be specified only by specifying the expression.

By the above processing, the component data and the expression parameter data are prepared. Then, in step S30, the connection of the component data and the parameter data is performed, and the component data (expression component data) having the motion information is generated. .

After step S30, the process proceeds to step S31. In step S31, the dynamic composition unit 207 creates a partial moving image for each part of the face. Here, the dynamic synthesizing unit 207 generates a moving image, performs interpolation calculation of each partial moving image, and generates an intermediate image for each unit time as moving image data in vector format. Embodiment 1
In the above, the dynamic synthesizing unit 107 merely generates the key frame, that is, stores the expression part data as it is. To generate an intermediate image result created for each unit time.

Next, at step S32, at step S
The intermediate synthesizing means 20 converts the intermediate image result generated at any time
7 or a storage means provided separately and stored in its memory area. It is not necessary to store all intermediate image results, and the storage area may be overwritten each time it is generated.

The stored intermediate image is stored in step S24,
It is used for data relinking performed in step S25. Here, as conditions necessary for the generated intermediate image,
Currently, the image must be substantially the same as the frame image of the partial moving image in the composite image displayed by the image output means 208. In other words, in order to continuously play back expressions without a sense of discomfort, it is necessary that at least the image in the first frame does not change significantly even if a new partial moving image is overwritten on the currently displayed image. It is. For this purpose, it is necessary to always create and hold an intermediate image every unit time.

If the amount of interpolation calculation is so large that intermediate frames cannot be reproduced for each unit time, or if there is a deviation in the drawing time from the display device, the display parameter data is composed of a plurality of key frames in advance. With such a definition, the intermediate key frame can be stored as an intermediate image and used for data relinking. Thereby, the interpolation calculation can be omitted and the processing can be simplified. However, there is a problem that the data amount of the expression parameter increases and that the expression cannot be switched in real time unless key frame data is prepared for each unit time.

After step S32, step S33,
Subsequently, the process proceeds to step S34. In steps S33 and S34, partial moving images are arranged and combined, and a combined moving image is output. This process is the same as step S18 and step S19 in the flowchart (FIG. 2) of the first embodiment, and thus the description is omitted.

According to the above-described processing, when performing continuous moving image reproduction of facial expressions, continuous reproduction without discomfort can be performed. FIG. 11 is a diagram schematically illustrating a case where continuous reproduction is performed from “laughing” to “angry”.

In the image processing apparatus according to the second embodiment,
The data relinking means 209 has a function of automatically performing a preset motion type designation internally for each predetermined period and automatically performing data relinking for each predetermined period. In order to automatically perform the data relinking, the motion type specifying unit 201 monitors the input of the operator and uses a timer (not shown) installed inside to
What is necessary is just to realize a function of automatically specifying the expression set in advance at a certain time interval to the data relinking unit 209.

As described above, the data relinking means 209 holds the currently displayed intermediate image. For this intermediate image, at predetermined time intervals, preset motion parameter data is
Regardless of the specification, the expression parameter data is dynamically connected. This connection may be made individually instead of to all face parts. FIG. 12 shows a state in which “blink” is autonomously connected when the “angry” facial expression animation is reproduced. By using the data relinking means 209, even when the expression "angry" is being reproduced, a more natural expression reproduction can be performed by connecting the animation of blinking only the eyes individually and continuously.

[0089]

As described above, according to the image processing apparatus of the present invention, an image processing apparatus for generating a combined moving image by combining an object having a certain constraint condition with a shape and a configuration thereof, particularly, In an image processing device that can generate facial expression moving images by combining facial parts of a person, the amount of data for moving images can be reduced, efficient creation management of part data, dynamic change of facial expression generation, etc. It can be performed. Further, at the time of dynamic moving image generation, continuous expression reproduction without a sense of discomfort can be performed.

[Brief description of the drawings]

FIG. 1 is a functional block diagram illustrating a configuration of an image processing apparatus according to a first embodiment of the present invention.

FIG. 2 is a flowchart for explaining an image processing operation in the image processing apparatus shown in FIG. 1;

FIG. 3 is an explanatory diagram schematically showing how parameter data is specified by specifying link information and expression, using a mouth part as an example.

FIG. 4 is a diagram for explaining that a facial expression changes according to a figure of a component, a composite diagram, and parameter data, taking an example of a left eye portion;

FIG. 5 is an explanatory diagram showing a format sample of component data and parameter data for the component shown in FIG. 4;

6A is an explanatory diagram showing a format example (connection example 1) in a case where the component data and parameter data shown in FIG. 5 are connected to form expression component data.

FIG. 7 is an explanatory diagram showing a format example in a case where expression part data is combined into a single moving image format.

FIG. 8 is an explanatory diagram illustrating an example of a facial expression animation created in the first embodiment.

FIG. 9 is a functional block diagram illustrating a configuration of an image processing apparatus according to a second embodiment of the present invention.

FIG. 10 is a flowchart for explaining an image processing operation in the image processing apparatus shown in FIG. 9;

FIG. 11 is an explanatory diagram showing, as an example of connection of parameter data to an intermediate expression, a pattern in which a continuous expression is reproduced from an “laughing” expression to an “angry” expression.

FIG. 12 is an explanatory diagram schematically showing an example in which “blinking” is autonomously connected to an intermediate image during the reproduction of an “angry” expression as an example of autonomously connecting internal parameter data.

[Explanation of symbols]

100, 200 part selection designation means 101, 201 motion type designation means 102, 202 face part data storage means 103, 203 part data extraction means 104, 204 facial expression parameter storage means (parameter data storage means) 105, 205 parameter data extraction means 106 , 206 Data link means 107, 207 Dynamic synthesis means 108, 208 Image output means 209 Data relink means

 ────────────────────────────────────────────────── ─── Continuing from the front page (72) Inventor Minehiro Konya 22-22, Nagaikecho, Abeno-ku, Osaka-shi, Osaka F-term (reference) 5B050 BA07 BA08 BA12 CA07 DA02 DA04 EA19 EA24 FA12 GA08

Claims (5)

[Claims] 1. A part data storage means having a plurality of types of parts having a certain constraint condition in shape and configuration thereof, and storing a plurality of still image data for each type of parts, and selecting an arbitrary part Means for instructing selection of parts, means for extracting appropriate part data from the means for storing part data, means for combining the part data extracted by the means for extracting part data, and means for synthesizing one or more images. A motion type designating means for designating one or more pieces of information relating to the movement of the component data; a parameter data storing means for storing parameter data for giving a motion to the component; Data link means for associating; and information on the motion specified by the motion type specifying means. Parameter data extraction means for extracting parameter data from the meter storage means, and generating a partial moving image in which motion is added to the component data from the component data and parameter data associated by the data link means, An image processing apparatus comprising: dynamic synthesis means for generating a single moving image by synthesizing component data in the same row. 2. The image processing apparatus according to claim 1, further comprising parameter sharing means for sharing parameter data by associating the same parameter data with a plurality of component data. . 3. The apparatus according to claim 1, further comprising means for changing the position in the image, the size in the image, the inclination, the color information, the order between the images, and the transmittance of the component image to be finally synthesized. The image processing device according to claim 1. 4. A data relinking means for associating parameter data again with a generated partial moving image, and continuously generating another moving image in accordance with the association by the data relinking means. The image processing device according to claim 1, 2 or 3, wherein 5. A method in which a motion type preset in the apparatus is specified every predetermined period, and the data relinking means re-associates the parameter data with each other.
The image processing apparatus according to claim 4, further comprising a unit that generates a moving image intermittently.