JP2007257508A - Animation generation system and method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To smoothly perform an editing operation of a feature point extracted from a video image obtained by photographing a moving subject. <P>SOLUTION: The animation creation system 1 extracts a motion of the subject by the feature point from the video image 19 obtained by photographing the moving subject to crate animation by using the extracted feature point. The system includes: a video image input part 3; a feature point extraction part 5; a feature point storage part 7; a synthesizing part 9; a feature point editing part 11; an action data creation part 13; an animation creation part 15; and an output part 17. The synthesizing part 9 superimposes the feature point on an image corresponding to the feature point by using position coordinates of the feature point when the position coordinates of the feature point are stored in the feature point storage part 7. Thus, to what part of the image the feature point corresponds, is grasped at a glance, and the editing operation of the feature point is smoothly proceeded. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a system and method for generating animation based on feature points extracted from an input video.

  In recent years, for example, animation using computer graphics (CG) has been frequently used in television, movies, games, and the like. In addition, such animations are increasingly applied to characters such as avatars in the communication field, and in particular, there is an increasing need for animations that express facial movements.

  However, creating an animation requires a great deal of money and time. Therefore, there has been a need for a method and system for creating an animation that looks natural in an easier way.

  Patent Document 1 discloses a method of detecting an individual's head, eyes, mouth, and the like from an input video, and creating an animation in which the movement of the character is controlled using CG based on the information. However, in Patent Document 1, since it is assumed to be used in a communication system such as a videophone, for example, when detection of a person's head, eyes, mouth, or the like fails, other parts are mistakenly placed on the head or There has been a problem that there is no correction means for correcting these detection results when they are detected as eyes or mouths, and as a result, a precise animation cannot be created.

In Patent Document 2, a subject with a marker is photographed from a plurality of directions to obtain a moving image of the subject, feature points are extracted from the image, and animation is performed based on the extracted feature points. A method of editing such as correcting the position of the extracted feature point at the time of creation, or interpolating a feature point that is hidden behind something and not extracted is disclosed. However, since Patent Document 2 does not include means for assisting these editing operations, there is a problem that the amount of work actually becomes enormous.
JP 2003-85583 A JP-A-10-222668

  The present invention eliminates the drawbacks of the prior art, can efficiently edit the feature points extracted to create an animation, and can express the natural movement of the character. An object of the present invention is to provide an animation generation method and system.

  In order to solve the above-described problem, in the present invention, a superimposed image in which the extracted feature points are superimposed on a corresponding image is created and displayed on a display unit such as a display. Therefore, it is possible to perform editing work while viewing this superimposed image. For example, it is possible to grasp which feature point is extracted from which part of the subject, and the feature point editing work can be performed efficiently. It becomes possible to do.

  Also, feature points can be extracted for each frame, not only the feature points corresponding to the image of a certain frame are superimposed, but also the feature points in other frames and the images corresponding to this feature point are also superimposed. It is possible to create a superimposed image. Further, when feature points corresponding to other frames are superimposed, the feature points may be connected by a line. By superimposing the feature points in other frames on the superimposed image in this way, it becomes possible to grasp the temporal change of the feature points, that is, the movement of the character generated from the feature points. It becomes possible to express movement.

  In the present invention, it is determined whether or not the change is appropriate by grasping how the feature point has changed with time. Specifically, a distance is calculated between temporally adjacent feature points, and then it is determined whether or not this distance is within a normally conceivable range. In this way, for example, it is possible to easily identify a frame in which a feature point exists in a position that does not exist originally due to an extraction mistake or the like, and the feature point editing work can be performed efficiently. Is possible. Furthermore, in the present invention, a feature point whose distance is not within the normally conceivable range is corrected using a preset correction formula. Therefore, editing work can be automatically performed, and a plurality of editing work can be performed at a time.

  According to the present invention, feature points can be edited efficiently. Further, it becomes possible to express the natural movement of the character.

  Embodiments of the present invention will now be described in detail with reference to the accompanying drawings. FIG. 1 is a block diagram conceptually showing the structure of an embodiment of an animation creating system according to the present invention. In FIG. 1, an animation creation system 1 includes a video input unit 3, a feature point extraction unit 5, a feature point storage unit 7, a synthesis unit 9, a feature point editing unit 11, a motion data creation unit 13, an animation creation unit 15, and an output. This is a system that includes a unit 17, extracts feature points from a video 19 obtained by shooting a moving subject, creates an animation based on the extracted feature points, and outputs the animation as an output unit 17.

  More specifically, the video input unit 3 is input with a video 19 obtained by photographing a subject that is moving in an animation. When such a video 19 is input to the video input unit 3, the feature point extraction unit 5 extracts feature points from the video 19 for each frame. The feature point is a point necessary for controlling the movement of the character by animation, that is, a main point necessary for expressing the movement.

  The extracted feature points are stored in the feature point storage unit 7. When the feature point is stored in the feature point storage unit 7, the feature point editing unit 11 edits the feature point. Thereafter, an animation is created using the edited feature points in the motion data creation unit 13 and the animation creation unit 15. Portions that are not directly related to the understanding of the present invention are not shown and redundant description is avoided. In the present invention, the configuration of each unit is not limited to the present embodiment, and any configuration can be adopted according to the animation creation system 1.

  In FIG. 1, the video input unit 3 receives a video 19 obtained by shooting a moving subject. This movement of the subject is a movement that is desired to be expressed by animation, that is, a movement that is desired to be made to be represented by the animation. In this embodiment, since the movement of the face, i.e., the change in facial expression, is to be expressed by animation, an image 19 of a human face changing the facial expression by moving each part of the face, for example, the eyes, nose, mouth, eyebrows, Entered.

  When such a video 19 is input, the video input unit 3 supplies the input video 19 to the feature point extraction unit 5. The feature point extraction unit 5 extracts feature points from the input video 19 for each frame, and calculates the position of the extracted feature points. In this embodiment, feature points are extracted for each frame, but the present invention is not limited to this.

  Here, the feature point is a point necessary for controlling the movement of the character by animation as described above, and is a main point necessary for expressing the movement. An arbitrary point can be used as a feature point in accordance with the movement. For example, in this embodiment, an animation in which the facial expression of the character changes is created. Is a feature point. Specifically, as shown by a square in FIG. 2, the right eyebrow has a substantially right end point, feature point A, the right eyebrow substantially left end feature point B, and the left eyebrow substantially right end feature point. C, feature point D at the substantially left end of the left eyebrows, feature point E at the pupil of the right eye, feature point F at the pupil of the left eye, feature point G at the substantially right end of the mouth, and feature point G at the substantially right end of the mouth The feature point H, the feature point I is the substantial center of the upper lip, and the feature point J is the substantial center of the lower lip.

  FIG. 2 is a diagram conceptually showing the process of extracting feature points, and is a diagram conceptually showing a certain frame 25 in the video 19 input to the animation creation system 1. Although only one frame 25 is shown in FIG. 2, this is for ease of explanation, and the video 19 can include any number of frames. Further, the present invention is not limited to the image 19 in which the facial expression is photographed, but it is possible to adopt the image 19 in which any motion is photographed according to the motion desired to be made to the character, and the subject is also animated. It is possible to employ any subject according to the character represented by.

  In FIG. 2, these feature points A to J can be extracted by adopting an arbitrary method. For example, the video 19 may be divided into frames as shown in FIG. 2 and the still images of each frame may be extracted by image processing, or a marker may be added in advance to a portion determined to be extracted as a feature point. Thus, when a subject with such a marker is photographed and feature points A to J are extracted, the corresponding marker portion may be extracted for each frame. The present invention is not limited to these. When the feature points A to J are extracted in this way, the feature point extraction unit 5 calculates the positions of the extracted feature points A to J, respectively.

  In this embodiment, as shown in FIG. 2, the lower left end point 51 of the frame is the origin, the horizontal direction is the x axis, and the vertical direction is the y axis, and the positions of the feature points A to J are calculated as coordinates for each frame. To do. An example of the coordinates indicating the calculated position of each feature point, that is, an example of the position coordinates is shown in FIG. In FIG. 3, n is a positive integer. In the present embodiment, the feature points A to J are extracted from all the frames of the video 19, and the position coordinates are calculated. This is because the subsequent operation data creation unit 13 creates the operation data from the feature points A to J in all frames included in the video 19. However, the present invention is not limited to this, and it is only necessary to extract the feature points and calculate the position coordinates as required by the operation data creation unit 13 in the subsequent stage. In this embodiment, two dimensions are used for the sake of easy understanding. However, the present invention is not limited to this. For example, the x axis and the z axis perpendicular to the y axis are further set to three dimensions. It is also possible to do.

  The position coordinates of the feature points A to J calculated by the feature point extraction unit 5 are sent to the feature point storage unit 7. The feature point storage unit 7 includes storage means such as a memory, for example, and stores the position coordinates of the feature points A to J sent thereto. The storage can be performed by adopting an arbitrary method. For example, a table for each of the feature points A to J can be created and stored as shown in FIG. In addition to the feature points, any feature points can be stored in the feature point storage unit 7. For example, in this embodiment, when the feature points A to J are extracted in the feature point storage unit 7. The image used in the above, that is, the image of each frame is also stored.

  When the position coordinates of the feature points A to J are stored in the feature point storage unit 7, the motion data creation unit 13 uses the known method to determine the movement of the character from the stored position coordinates of the feature points A to J. Operation data, which is data defining temporal changes, is created. The motion data is supplied to the animation creating unit 15, and the animation creating unit 15 creates an animation in which the character moves according to the motion data using a known method. The created animation is supplied to the output unit 17. The output unit 17 includes an output unit such as a display or a printer, for example, and displays the animation supplied from the animation creation unit 15 on a display or the like, or outputs the animation from the printer to paper.

  A motion is extracted from the video 19 in which the moving subject is photographed as described above, and an animation of a character that moves based on the extracted motion is created. However, if motion data is created using the position coordinates of the feature points A to J extracted by the feature point extraction unit 5 as they are, the movement of the subject itself is strange, the video is not photographed well, the feature points A to In some cases, such as incorrect extraction of J, motion data that does not make the character move naturally is created, and an animation with strange character motion is created, so the position coordinates can be changed or extracted. Editing operations such as complementing the feature points that have not been performed are performed by the feature point editing unit 11.

  The feature point editing unit 11 is connected to the feature point storage unit 7 and corrects the position coordinates of the feature points A to J stored in the feature point storage unit 7 or is not extracted as a shadow of something. Editing operations such as complementing feature points are performed. For example, the editing operation is performed by a user operation signal input from an operation unit (not shown) such as a mouse or a keyboard, or is automatically performed by a preset editing program. The edited position coordinates are supplied again to the feature point storage unit 7 and stored in the feature point storage unit 7. Thereafter, the motion data creation unit 13 creates motion data from the position coordinates of the feature points A to J after correction and interpolation, and the animation creation unit 15 creates the motion data created by the motion data creation unit 13. Create an animation based on

  Conventionally, when an editing operation is performed by a user operating an operation unit (not shown), for example, the feature points extracted by looking only at the feature points are corrected, or the features that are hidden behind something are not extracted. Since the points are interpolated, for example, it is unclear which part of the subject each feature point represents, and there is a problem that the amount of work becomes enormous.

  Therefore, in the present embodiment, the animation creation system 1 includes a synthesis unit 9. In this synthesis unit 9, the feature points stored in the feature point storage unit 7 are superimposed on the corresponding frame image, and the superimposed image is displayed. create. Thereafter, the generated superimposed image is supplied to the output unit 17 and output to, for example, a display included in the output unit 17. Therefore, the editing work in the feature point editing unit 7 can be performed while looking at the superimposed image output to the output unit 17, so that the editing work can be performed efficiently. Details will be described below.

  In FIG. 1, the synthesis unit 9 is connected to the feature point storage unit 5 and the output unit 17, and each feature point corresponds to this feature point based on the position coordinates stored in the feature point storage unit 5. A superimposed image superimposed on the image is created.

FIG. 4 is a diagram conceptually illustrating an example of the superimposed image created by the synthesis unit 9. 4, the same reference numerals as those in FIG. 2 denote the same components. In FIG. 4, each frame corresponds to each frame shown in FIG. For example, the position coordinates of the feature point A ₁ of the first frame shown in FIG. 4 are the position coordinates of the feature point A ₁ of the first frame 61 shown in FIG. 3, that is, (X _A1, Y _A1 ). The same applies to other feature points. In FIG. 4, superimposed images 61, 63, and 65 for three frames are shown. This is for ease of explanation, and the synthesis unit can create superimposed images for arbitrary frames. It is.

  When the position coordinates of the feature points A to J are stored in the feature point storage unit 7 in order to create the superimposed images 61, 63, and 65 as shown in FIG. Further, when the images 67, 69, 71 corresponding to the read position coordinates are stored together with the feature points in the feature point storage unit 7, for example, in a place where the images are stored, for example, in this embodiment, For example, when it is stored in the video input unit 3, it is read out from the feature point storage unit 7. Thereafter, the read position coordinates and the images 67, 69, 71 corresponding to the position coordinates are superimposed to create superimposed images 61, 63, 65.

In the case of the superimposed image 61 in FIG. 4, for example, the first frame of the image 67 that correspond from the feature point storage unit 7 and the position coordinates of each feature point A ₁ through J ₁ in the first frame, and the feature point A ₁ through J ₁ And the feature points A to J are drawn at positions specified by the position coordinates in the image 67 of the first frame, that is, superimposed to create a superimposed image 61. Note that the present invention is not limited to superimposing all the extracted feature points A to J as in this embodiment, but only a certain feature point, for example, a mouth feature point G to J. It is also possible to superimpose only. Note that the image used in this case may be an image of one entire frame, or an image corresponding to a certain feature point, for example, an image of only the corresponding mouth if mouth feature points G to J are used. It is possible to set arbitrarily.

  Arbitrary methods can be adopted for superimposing feature points. For example, the image and the feature point may be superimposed by adding the pixel value of the feature point to the pixel value of the portion where the feature point in the corresponding image overlaps, or for example, the layer and the feature point of the corresponding image These layers may be formed and overlapped by overlapping them. Note that the present invention is not limited to this.

  The superimposed images 61, 63, 65 created in this way by the synthesizing unit 9 are supplied to the output unit 17, for example, output to a display included in the output unit 17, or included in, for example, the output unit 17 Printed on a printer and output on paper. Therefore, it is possible to perform editing work in the feature point editing unit 11 while using the superimposed images 61, 63, 65, and specifically, watching the superimposed images 61, 63, 65 output to the output unit 17. Therefore, it is possible to grasp at a glance which part in the images 61, 63, and 65 the feature points A to J correspond to, and the editing work can be performed efficiently. In addition, since the feature points A to J are superimposed, it is possible to assume the movement of the character created by the animation creating unit 15 in the subsequent stage, and it is possible to efficiently create an animation in which the character moves naturally. It becomes possible. Note that the present invention is not limited to the example shown in FIG. 4, and the synthesis unit 9 performs the time of feature points A to J as shown in FIGS. 5, 6, 7, 8, and 9, for example. A superimposed image may be created so that a change in the image can be grasped.

  FIG. 5 is a diagram conceptually illustrating another example of the superimposed image created by the synthesis unit 9. In FIG. 5, the same reference numerals as those in FIG. In the example shown in FIG. 5, the synthesizing unit 9 creates a superimposed image by superimposing a feature point corresponding to this image on an image of one frame and also superimposing a feature point in another frame. This other frame can be a frame located in front of or behind the frame of the superimposed image.

For example, in the example shown in FIG. 5, the feature point A _{1 to} J ₁ in the first frame and the feature point A _{2 to} J ₂ in the second frame are superimposed on the image 67 of the first frame shown in FIG. 91 has been created. Note that the present invention is not limited to this, and it is possible to superimpose feature points in an arbitrary frame, and it is possible to arbitrarily set the number of superimpositions. For example, 20 frames are selected for every 10 frames from 100 frames located at the front and back of a certain frame, and the feature points of each selected frame are superimposed on the image of a certain frame. A superimposed image may be created. The present invention is not limited to this.

  When superimposing a feature point of another frame on a superimposed image of a frame, for example, as shown in FIG. 5, the form of the feature point to be displayed, for example, the color, pattern, or shape of the feature point is displayed for each frame. Although it is preferable to change how the feature points change by changing the frame, that is, it is possible to grasp the temporal change of the feature points, the present invention is not limited to this. Absent. In the example shown in FIG. 5, all feature points in other frames are superimposed, but the present invention is not limited to this, and for example, it is arbitrarily set to superimpose feature points in other frames. It is also possible to use only the feature points.

  By superimposing the feature points of other frames in this way, it is possible to grasp the temporal change of the feature points, so edit the feature points so that the character moves more naturally. Becomes easier. As a result, it is possible to express more natural movement of the character.

  FIG. 6 is a diagram conceptually illustrating another example of the superimposed image created by the synthesis unit 9. In FIG. 6, the same reference numerals as those in FIGS. 4 and 5 indicate the same components, and thus the description thereof is omitted. In the example shown in FIG. 6, the combining unit 9 superimposes feature points corresponding to this image and feature points in another frame on an image of one frame, and further connects the feature points of the same type with lines. A superimposed image 95 is created.

Here, the same type of feature points are feature points extracted from the same portion of the image. For example, the feature point H1 of the first frame and the feature point H2 of the second frame have the same kind of feature point relationship. Therefore, the synthesis unit connects the feature point H1 and the feature point H2 with a line 97. The same applies to other feature points. For example, when the position hardly changes between the first frame and the second frame as in the feature point A ₁ and the feature point A ₂ , they may be connected by a line, or conversely, they are connected by a line as in this embodiment. It may not be possible.

  When there are many types of feature points in the frame, for example, as in this embodiment, it is possible to grasp at a glance which feature points change by changing the type of line for each type of feature points. It may be possible to do this. For example, in this embodiment, the type of line to be connected is changed for each of the feature points G, H, I, and J connected by a line. In addition, this invention is not necessarily limited to this, The kind of line to connect may be the same.

  By connecting the same kind of feature points with a line in this way, it becomes possible to further grasp the temporal change of the feature points. Note that the present invention is not limited to connecting with lines. For example, as shown in FIG. 7, it is possible to connect with arrows. For example, as shown in FIG. In the same manner, it may be possible to grasp temporal changes in feature points. 7 and 8 are diagrams conceptually illustrating another example of the superimposed image created by the synthesizing unit 9. FIG. 7 and FIG. 8, the same reference numerals as those in FIG. 4, FIG. 5, and FIG.

  In the superimposed image 101 shown in FIG. 7, feature points of the same type are connected by arrows. In the example shown in FIG. 7, the type of arrow is not changed for each type of feature point, but the present invention is not limited to this. For example, the type of arrow may be changed for each type of feature point. Good. In the superimposed image 103 shown in FIG. 8, the feature point form, that is, the color, pattern, shape, or the like is changed for each type of feature point, and the temporal change of the feature point is shown. Specifically, in FIG. 8, the feature point E of the left eye is represented by a square with a diagonal pattern diagonally to the right, and the feature point F of the right eye is represented by a square with a diagonal pattern diagonally to the left. . The same applies to other feature points. Therefore, it is possible to grasp the temporal change of the feature points by following the feature points of the same form.

Further, in FIG. 8, the size of the feature point of the frame positioned in front of the time is made larger than that of the frame positioned behind, so that the temporal context can be seen at a glance even if they are not connected by arrows. I understand it. Specifically, in FIG. 8, the size of the feature points A _{1 to} J ₁ in the first frame is made larger than the size of A _{2 to} J ₂ in the second frame, and which is the time in the previous frame in time. The feature points can be grasped without being connected with arrows.

  Note that the present invention is not limited to the examples shown in FIGS. 5 to 8, and any method can be adopted as long as it is possible to grasp temporal changes in feature points. For example, feature points, lines, and arrows, for example, blink from a frame that is temporally forward to a frame that is temporally backward, or have a lighter or darker color It is also possible to indicate the direction in which the points change.

  For example, the methods shown in FIGS. 5 to 8 can be used in combination. For example, when there are many types of feature points and when the number of superimposed features is large, the form is changed for each feature point of the same type, and the feature points of the same type are connected with lines or arrows. It becomes possible to make it easier to grasp temporal changes of feature points. The present invention is not limited to these.

  In this way, the feature point corresponding to this image and the feature point of another frame are superimposed on the image of a certain frame, and the feature points of the same type are connected by lines and arrows, or the shape of the feature point of the same type By unifying, it becomes possible to make it easier to grasp temporal changes in feature points, so that editing work can be performed smoothly and natural movements can be formed. . Note that the superimposed image created by the combining unit 9 is not limited to the examples illustrated in FIGS. 5 to 8. For example, as illustrated in FIG. 9, the combining unit displays images corresponding to feature points of other superimposed frames. It is also possible to create a superimposed image.

  FIG. 9 is a diagram conceptually illustrating another example of the superimposed image created by the synthesis unit. In FIG. 9, the same reference numerals as those in FIGS. In the example illustrated in FIG. 9, the synthesis unit 9 creates a superimposed image 111 by superimposing an image of a certain frame, a feature point corresponding to this frame, a feature point of another frame, and an image corresponding to another frame.

Specifically, the synthesizing unit 9 uses the first frame image 67 as a reference, and the first frame image 67 includes the first frame feature points A _{1 to} J ₁ , the second frame image 69, and the second frame image 67. 2 feature points of the frame a ₂ through J _2, and the image 71 of the third frame, and superimposes the characteristic point a ₃ through J ₃ of the third frame to create a superimposed image 111. The reference frame here, for example, the first frame in this embodiment, is a frame that becomes a starting point when grasping a temporal change. That is, in this embodiment, the movement in the second frame and the third frame is grasped from the first frame as a starting point. The reference frame can be arbitrarily set according to the movement to be grasped.

  In this embodiment, all feature points A to J in each frame and images corresponding to the feature points A to J are superimposed. However, the present invention is not limited to this, and only arbitrary feature points are included. It is possible to superimpose, and it is also possible to superimpose only images corresponding to the superimposed feature points.

  For example, using the present embodiment, since only the mouth feature point in each frame changes, this mouth feature point, i.e., feature points G, H, I, J are superimposed, and this mouth feature point G, It is also possible to superimpose a portion corresponding to H, I, and J, that is, an image of the mouth only for three frames. When superimposing an image of another frame only on a certain part in this way, the image of the reference frame, for example, the image of the first frame in this embodiment, for example, uses the entire image, not the part of the mouth alone. However, the present invention is not limited to this because it is possible to make a comprehensive judgment.

  As described above, when superimposing images of different frames, it is possible to superimpose by adopting an arbitrary method in the same manner as superimposing feature points. For example, pixel values may be added, or an image of another frame may be formed as another layer, and the layers may be overlapped to overlap. When superimposing, the image of the frame other than the reference frame may be the original, for example, the color is made light, semi-transparent, dotted, or an image represented by only one color shade, for example For example, the displayed form may be changed. For example, when there are a large number of images to be superimposed, it is possible to perform editing work smoothly by making the superimposed images easy to see by thinning the color in this way, but the present invention is not limited to this. It is possible to adopt any method.

  Since the image of another frame is also superimposed as described above, it is possible to determine at a glance whether each feature point corresponds to a desired position in each frame while grasping the temporal change of the feature point. Thus, the editing operation can be performed smoothly, and the feature points can be edited so that the movement becomes natural. Even when images of other frames are superimposed, the same kind of feature points may be connected by lines or arrows as shown in FIGS. 6 and 7, or the same kind of feature points as shown in FIG. The form may be divided into any of these, and can be arbitrarily selected.

  FIG. 10 is a block diagram conceptually showing the structure of another embodiment of the animation creating system according to the present invention. 10, the same reference numerals as those in FIG. 1 indicate the same components. In FIG. 10, the animation creation system 201 includes a distance calculation unit 203 and a determination unit 205. The distance calculation unit 203 calculates a distance between feature points that are temporally adjacent, that is, in a front-rear relationship. The amount of change over time is obtained. Thereafter, the determination unit 205 determines whether or not the amount of change is normal, extracts feature points with a large amount of change, and the feature point editing unit 11 edits feature points with a large amount of change. Therefore, it is possible to automatically extract feature points having a large amount of change, that is, feature points that do not form a natural motion, and it is possible to easily edit such feature points. This will be specifically described below.

  In FIG. 10, the distance calculation unit 203 is connected to the feature point storage unit 7, and when the position coordinates of the feature point are stored in the feature point storage unit 7, the distance calculation unit 203 is temporally adjacent using the position coordinates, that is, the time In other words, the distance between feature points of the same type is calculated. For example, in this embodiment, since the feature points are calculated for each frame, the distance between the same types of feature points between adjacent frames is calculated.

  The determination unit 205 is connected to the distance calculation unit 203 and determines whether or not the position of the feature point is normal using the distance calculated by the distance calculation unit 203. In this embodiment, the determination unit 205 stores a preset value, and the determination unit 205 determines whether or not the distance calculated using this value is equal to or greater than this value, thereby determining the position of the feature point. It is determined whether or not is normal. The present invention is not limited to this. For example, every time the determination unit 205 makes a determination, a value used for the determination may be input from an operation unit (not shown) or the like.

  Such a distance calculation unit 203 and determination unit 205 can be provided in any part of the animation creation system 201. For example, the distance calculation unit 203 and the determination unit 205 may be provided separately from the feature point editing unit 11 as in the present embodiment. For example, it can be provided in the feature point editing unit 11. Note that the present invention is not limited to this.

  As described above, in the example shown in FIG. 10, since the distance calculation unit 203 and the determination unit 205 are included, it is possible to easily identify a feature point having a large temporal change. As a result, for example, a feature point extraction unit In 5 it is possible to easily detect frames that have been extracted as non-feature points, and frames that have feature points that do not move naturally. It becomes possible to do. This will be described in detail below.

  FIG. 11 is a flowchart showing an example of a processing procedure of editing processing in the animation creating system 201 shown in FIG. In FIG. 11, for example, when an instruction for calculating a distance is input from an operation unit (not shown), the distance calculation unit 203 is temporally related to calculate the distance between feature points of the same type. In this embodiment, the position coordinates of the feature points in two adjacent frames are read from the feature point storage unit 7 (step S1).

For example, when each distance between the feature points A _{1 to} J ₁ and the feature points A _{2 to} J ₂ shown in FIG. 3 is calculated, the distance effect unit 203 uses the position coordinates of the feature points A _{1 to} J ₁ in the first frame. , And the position coordinates of the feature points A _{2 to} J ₂ in the second frame are read from the feature point storage unit 7. Note that the feature points to be read can be arbitrarily set. For example, the position coordinates of all the feature points can be read, or for example, arbitrarily selected feature points. The reading range can also be set arbitrarily. For example, in the present embodiment, the distance between feature points in the frames from the first frame to the nth frame shown in FIG. 3 is calculated. The present invention is not limited to this.

  When the distance calculation unit 203 reads the position coordinates of each feature point in two adjacent frames, the distance calculation unit 203 determines whether or not there is a feature point whose position coordinates have not been calculated (step S2). The determination is made in this way when the point set as the feature point is not extracted as a shadow, for example, because the position coordinates have not been calculated and the distance cannot be calculated. Because.

If the position coordinates of the feature points are calculated as a result of the determination in step S2, the distance calculation unit 203 proceeds to step S3, and calculates the distance between the same type of feature points in adjacent frames using the position coordinates ( Step S3). For example, the distance between the position coordinates of the feature point A ₁ in the first frame and the position coordinates of the feature point A ₂ in the second frame is calculated. The same applies to other feature points.

Here, the distance can be calculated by adopting an arbitrary method. For example, when the position of a feature point is considered on a coordinate plane with an arbitrary point as the origin as in the present embodiment, the distance may be calculated using, for example, the three-square theorem, or a certain axis, for example It is also possible to use the amount of displacement at as a distance. Specifically, using the feature point A ₁ and the feature point A ₂ , the position coordinates of the feature point A ₁ (X _A1 , Y _A1 ) and the position coordinates of the feature point A ₂ (X _A2 , Y _A2 ) It is also possible to calculate the distance using the three-square theorem, and the displacement of the feature points A ₁ and A _{2 on} the x axis, that is, the value obtained by subtracting X _A2 from X _A1 and the feature points A ₁ and It is possible to consider the displacement of A _{2 in} the y-axis, that is, the value obtained by subtracting Y _A2 from Y _A1 as the distance. In addition, this invention is not necessarily limited to these, It is possible to employ | adopt arbitrary methods.

  When the distance is calculated, the calculated distance is supplied to the determination unit 205. In this embodiment, the determination unit 205 stores a preset setting value. When the distance is supplied from the distance calculation unit 203, the determination unit 205 compares the set value with the supplied distance. Thus, it is determined whether or not the temporal change of the feature point is appropriate (step S4). Here, an arbitrary value can be adopted as the set value. For example, the set value may be a value set for each movement of the subject, or may be a value determined by experience by the user. Note that the present invention is not limited to this.

  If the result of determination in step S4 is that the distance is smaller than the set value, the process proceeds to step S6 to determine whether there is a remaining frame for calculating the distance, and if there is no remaining frame, the process ends. To do. Conversely, if there is a frame for calculating the distance, the process returns to step S1. On the other hand, as a result of the determination in step S4, if the distance is equal to or greater than the set value, the process proceeds to step S5, where the feature point is equal to or greater than the value set for the distance in the combining unit 9, that is, the feature point having a strange temporal change. , And tell them to create a superimposed image using the image.

  Any method can be adopted for the processing to be transmitted to the synthesis unit 9. For example, the information may be transmitted directly from the determination unit 205 to the synthesis unit 9, or may be transmitted via the feature point storage unit 7, for example. For example, in the present embodiment, by adding preset information for notifying that the temporal change is strange to the position coordinates of the characteristic point having a strange temporal change in the characteristic point storage unit 7, 9 tells us about the existence of strange feature points that change over time. Note that the present invention is not limited to this. The synthesizing unit 9 creates a superimposed image and outputs it to the output unit 17 to notify the user that there is a feature point having a strange position in the frame (step S5).

  Since the distance between the two frames is calculated, the feature point at an abnormal position is a feature point in one of the frames used for the calculation. Therefore, the determination unit 205 further determines which of the two frames used for the distance calculation is a feature point that is not in a normal position, and informs the synthesis unit 9 of only one specified frame. Alternatively, both frames may be notified to the synthesizing unit 9 as being out of position without making a determination in reverse, and any method can be adopted.

Specifically, for example, when the distance between the first frame and the second frame is calculated and the determination unit 205 determines that the distance of the feature point H is longer than the set value, the determination is made. The unit may further determine which of the feature point H ₁ and the feature point H ₂ is wrong, or both may have a wrong position as in the present embodiment, for example, and the user may be allowed to determine.

  FIG. 12 is a diagram conceptually illustrating an example of the superimposed image, and is a diagram conceptually illustrating an output example in the output unit 17. 12, the same reference numerals as those in FIG. 4 denote the same components. In this embodiment, as shown in FIG. 12, both are output as being in a wrong position, and the user is made to judge which is wrong, and the stranger is corrected. Further, in this embodiment, the outline of the first frame including the feature point H1 that is determined to be incorrect and the second frame including the feature point H2 are thickened so that the user has a feature point with an incorrect position in the frame. I know.

  Note that the present invention is not limited to the example shown in FIG. 12. For example, although not shown, the background color of the superimposed image is changed, the shape of the feature point having a wrong position is changed, or the feature point having a strange outline, background, or position is used. Etc. may be notified to the user by blinking. Also, for example, instead of creating and outputting a superimposed image for a frame other than the frame in which the feature point determined to be out of position exists as shown in FIG. 12, that is, the third frame in FIG. The user may be notified by creating and outputting a superimposed image only for frames in which strange feature points exist.

  Furthermore, for example, as shown in FIG. 13, a superimposed image 209 in which feature points in the third frame are also superimposed may be created, and can be arbitrarily selected. Note that the present invention is not limited to the examples shown in FIGS. 12 and 13, and any method can be adopted to notify the user of the presence of a feature point having a wrong position.

  FIG. 13 is a diagram conceptually illustrating another example of the output screen of the output unit when notifying the user. 13, the same reference numerals as those in FIGS. 5 to 8 denote the same components. In FIG. 13, the feature points A to J in the first to third frames are superimposed on the image 67 in the second frame to create a superimposed image. In the example shown in FIG. 13, the feature points A to J are superimposed on the image 67 of the second frame, but the present invention is not limited to this, and the distance is calculated between the first frame and the second frame. Therefore, it is sufficient that the feature points A to J of the first frame and the second frame are superimposed on at least one of the images, and it is possible to arbitrarily set which one is adopted.

After informing the user, the process proceeds to step S6, and it is determined whether there are any remaining frames (step S6). As a result, if there are remaining frames, the process returns to step S1. If not, the process ends. For example, in this embodiment, the feature points A _{3 to} J ₃ in the third frame, which is the next frame, are read, and the distance is calculated between the second frame and the third frame.

  On the other hand, if there is a feature point whose position coordinates are not calculated as a result of the determination in step S2, the distance calculation unit 203 proceeds to step S5 and uses the feature points and images in the frame in which no position coordinates exist in the combining unit 9. Tell them to create a superimposed image. As for the processing to be transmitted to the synthesizing unit 9, an arbitrary method can be adopted as in the processing to be transmitted from the determining unit 205 to the synthesizing unit 9. The synthesizing unit 9 creates a superimposed image using the feature points and images in this frame and outputs them to the output unit 17 to inform the user that there are feature points that have not been extracted. After the notification, the process proceeds to step S6, where it is determined whether or not there are remaining frames. If there is, the process returns to step S1, and if not, the process ends.

  In this way, in the example shown in FIG. 11, since the feature point with the wrong position is detected by calculating the distance, the feature point with the wrong position can be easily found. If it is determined that the position is incorrect, the feature point and the image corresponding to the feature point are superimposed to create a superimposed image. The position can be corrected, and the editing operation can be performed smoothly. Further, when calculating the distance, since it is determined whether or not the position coordinates exist, it is possible to easily detect the feature points that have not been correctly extracted.

  FIG. 14 is a block diagram conceptually showing the structure of another embodiment of the animation creating system according to the present invention. In FIG. 14, the same reference numerals as those in FIGS. 1 and 10 denote the same components. In the animation creation system shown in FIG. 14, the feature point editing unit 303 includes a correction formula storage unit 305. In FIG. 14, the feature point editing unit 303 includes only the correction formula storage unit 305, and is the same as the feature point editing unit 11 shown in FIG. The correction formula storage unit 305 stores a correction formula, and the feature point editing unit 303 corrects the position coordinates of the feature points using the correction formula.

  The correction formula is used to automatically edit these feature points when feature points that are not extracted by the distance calculation unit 203 are found or when feature points having an incorrect position are found by the determination unit 205. In this embodiment, the correction formula is a rule or a formula that defines how a feature point that has not been extracted or a feature point having a wrong position should be edited.

  More specifically, the rules define the way of editing. For example, in this embodiment, there is a rule that the position coordinates of a feature point having a strange position are set to the midpoints of feature points before and after the frame in which the feature point exists, and there is a frame from which no feature point has been extracted. Uses rules such as the same position as the feature point in the previous frame. In addition, this invention is not necessarily limited to the rule mentioned above, It is possible to employ | adopt arbitrary rules.

  The mathematical formula defines a temporal change in the position coordinates of the feature points. For example, in this embodiment, a function or the like using the frame (t) as an independent variable and the dependent variables as position coordinates (x, y) as shown in Equation 1 is used. Note that the present invention is not limited to the present embodiment, and any mathematical formula or the like can be adopted.

y =-at ² + b (1)
Such a correction formula can be created by adopting an arbitrary method. For example, it can be created based on experience, or can be created from the position coordinates of feature points used for motion recognized as smooth motion. Is possible. In addition, the correction formula can be created in any kind, for example, it can be created for each frequently appearing motion, and for example, a formula showing a linear variation of a feature point And several formulas showing curved fluctuations are created, and if the feature points fluctuate linearly from the frame before and after the frame where the feature points with strange positions exist, the fluctuations It is also possible to correct using a mathematical expression that shows a linear variation that fits the curve. Is possible. The present invention is not limited to this.

  By creating a correction formula in this way and storing it in the correction formula storage unit 305, if a feature point with a wrong position is found in the distance calculation unit 203 or the determination unit 205, it is automatically selected simply by selecting the correction formula. The feature points can be edited, and the feature points can be edited so that the movement automatically becomes natural. In addition, since a plurality of frames can be edited at a time by using the correction formula, the editing operation can be performed more smoothly.

  FIG. 15 is a flowchart showing an example of a processing procedure for performing an editing operation using a correction formula in the animation creating system shown in FIG. In the example shown in FIG. 15, a feature point that has not been successfully extracted by the distance calculation unit or the determination unit, or a feature point having a long distance between adjacent frames, that is, a feature point having a large temporal change is represented by a correction formula. Automatically correct using.

  14 and 15, the animation creation system 301 identifies feature points that have not been successfully extracted by the distance calculation unit 203 and the determination unit 205 or feature points that have a large temporal change, for example, by the processing illustrated in FIG. 11. Then, a superimposed image of the frame including such feature points is created and output to the output unit (step S1).

For example, the position of the feature point of H ₂ second frame is determined to be inappropriate, the superimposed image is outputted to the output section as shown in FIG. 13. When the superimposed image is output in this way, the feature point editing unit 303 selects a correction formula from the correction formula storage unit 305 (step S2). This selection may be automatically performed by the feature point editing unit 303 when, for example, information about feature points that have not been successfully extracted is transmitted from the distance calculation unit 203 or the determination unit 205 to the feature point editing unit 303. In addition, for example, the user may select by operating an operation unit (not shown) while viewing the superimposed images 61, 63, 65 output to the output unit 17, and any method can be adopted. . For example, in the present embodiment, the above-described rule that the midpoint of the feature point of the frame located in the front and rear is used as the position coordinate is selected by the user from an operation unit (not shown).

  When the correction formula is selected in this way, the feature point editing unit 303 corrects the position coordinates of the feature point using this correction formula (step S3). More specifically, in this embodiment, correction is performed using a rule that the midpoint of the frame before and after the frame to be corrected is used, so that the position coordinates of the feature point H1 of the first frame and the feature point of the third frame are used. A midpoint 309 of these position coordinates is obtained from the position coordinates of H3, and the feature point H2 of the second frame is moved to the obtained midpoint 309. The midpoint 309 can be calculated using a known method.

  When editing is completed as described above, it is determined whether or not editing has been performed on all frames whose position is determined to be inappropriate (step S4). If there are remaining frames, step S1 is performed. Return to and edit the remaining frames in the same way. When the editing work is completed for all frames, the editing work is finished.

  In this way, in the examples shown in FIGS. 14 and 15, since the editing operation is performed using the correction formula, it is possible to automatically edit feature points extracted in error or feature points with incorrect positions. Become. Further, since the correction formula is used, it is possible to edit a plurality of frames at a time, and it is possible to efficiently perform the editing work. Further, by using a correction formula created so as to achieve natural movement, it becomes possible to efficiently edit feature points so as to achieve natural movement. In addition, when a feature point or the like whose position is strange is specified by the distance calculation unit 203 or the determination unit 205, editing is performed using a correction formula as shown in FIGS. 14 and 15 or without using the correction formula, for example, Whether the position or the like is directly input from the operation unit or the like not to be performed can be arbitrarily selected as necessary.

It is a block diagram which shows notionally the structure of the Example of the animation production system by this invention. It is the figure which showed notionally the process which extracts a feature point. It is the figure which showed notionally the calculated position coordinate of each feature point. It is the figure which showed notionally an example of the superimposed image produced in the synthetic | combination part. It is the figure which showed notionally another example of the superimposed image. It is the figure which showed notionally another example of the superimposed image. It is the figure which showed notionally another example of the superimposed image. It is the figure which showed notionally another example of the superimposed image. It is the figure which showed notionally another example of the superimposed image. It is a block diagram which shows notionally the structure of another Example of the animation production system by this invention. 11 is a flowchart illustrating an example of a processing procedure of editing processing in the animation creation system illustrated in FIG. 10. It is the figure which showed notionally the example of an output in the output part shown in FIG. It is the figure which showed notionally another output example in the output part shown in FIG. It is a block diagram which shows notionally the structure of another Example of the animation production system by this invention. It is the flowchart which showed an example of the process sequence which performs an edit operation | work using a correction formula.

Explanation of symbols

1 Animation creation system
3 Video input section
5 Feature point extraction unit
7 Feature point storage
9 Synthesis section

Claims (20)

A feature point extracting means for inputting a video including at least two frames obtained by photographing a moving subject and extracting a feature point for controlling the movement of the character for each frame from the video;
A feature point editing means for editing the feature points;
An animation creating system including an animation creating means for creating an animation in which the character moves based on the edited feature point,
Combining means for superimposing the first feature point on a first image corresponding to the first feature point to create a superimposed image;
And an output unit for outputting the superimposed image.   2. The animation creating system according to claim 1, wherein the synthesizing unit further superimposes a second feature point extracted from a second frame other than the first frame from which the first feature point is extracted to superimpose the superimposition. An animation creation system characterized by creating images.   3. The animation creation system according to claim 2, wherein the synthesizing unit further creates the superimposed image by superimposing a second image corresponding to the second feature point.   4. The system according to claim 2, wherein the synthesizing unit connects feature points of the same type in the superimposed image with lines.   4. The system according to claim 2, wherein the synthesizing unit connects the same kind of feature points in the superimposed image with arrows.   6. The system according to claim 2, wherein the synthesizing unit represents the same type of feature points in the superimposed image in the same form.   7. The system according to claim 2, wherein the second frame is a frame located before and / or after the first frame. The system according to any of claims 1 to 7, further comprising:
Distance calculating means for calculating the distance between the feature points adjacent in time;
Judging means for judging whether the temporal change of the feature point is appropriate using the distance;
The synthesizing unit creates the superimposed image on which the feature point determined to be inappropriate by the determining unit and the image corresponding to the feature point are superimposed,
The animation creating system, wherein the output unit outputs the superimposed image. 9. The system according to claim 8, further comprising a correction formula storage means for storing a correction formula indicating the position of the feature point.
The animation creation system, wherein the feature point editing unit edits the feature point determined by the determining unit that the temporal change of the feature point is not appropriate using the correction formula.   The system according to claim 1, wherein the video is a video obtained by photographing a facial motion. A feature point extracting step of inputting a video including at least two frames obtained by photographing a moving subject and extracting a feature point for controlling the movement of the character for each frame from the video;
A feature point editing step for editing the feature points;
An animation creating step including creating an animation in which the character moves based on the edited feature points, the method comprising:
A synthesis step of superimposing the first feature point on a first image corresponding to the first feature point to create a superimposed image;
An output step of outputting the superimposed image,
The animation creating method, wherein the editing step performs a variation work while viewing the outputted superimposed image.   12. The animation creation method according to claim 11, wherein the synthesis step further superimposes a second feature point extracted from a second frame other than the first frame from which the first feature point has been extracted to superimpose the superposition. An animation creation method characterized by creating an image.   13. The animation creation method according to claim 12, wherein the synthesis step further creates the superimposed image by superimposing a second image corresponding to the second feature point.   The method according to claim 12 or 13, wherein in the synthesis step, feature points of the same type in the superimposed image are connected by a line.   The method according to claim 12 or 13, wherein in the combining step, feature points of the same type in the superimposed image are connected by an arrow.   16. The method according to claim 12, wherein the synthesizing step represents the same kind of feature points in the superimposed image in the same form.   17. The method according to claim 12, wherein the second frame is a frame located before and / or after the first frame. 18. A method according to any of claims 11 to 17, further comprising:
A distance calculation step of calculating a distance between the feature points adjacent in time;
A determination step of determining whether the temporal change of the feature point is appropriate using the distance,
The synthesis step creates the superimposed image in which the feature point determined to be inappropriate in temporal change by the determination step and the image corresponding to the feature point are superimposed,
The output step outputs the superimposed image,
The animation creating method characterized in that the editing step edits a feature point that is determined to have an inappropriate temporal change while viewing the output superimposed image.   19. The method according to claim 18, wherein the feature point editing step edits the feature point determined by the determination step that the temporal change is not appropriate using a correction formula indicating a position of the feature point. An animation creation method characterized by:   The method according to claim 11, wherein the video is a video obtained by photographing a facial motion.

Images