JP2008003673A - Animation creation system and method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To smoothly perform editing work of a feature point which is extracted from a video obtained by photographing a moving subject. <P>SOLUTION: The animation creation system 1 is a system for extracting movement by extracting a feature point from a video 19 which has photographed a moving subject and using the feature point to create an animation, and includes: a video input part 3; a feature point extraction part 5; a retrieval part 7; a feature point storage part 9; a composition part 11; a feature point editing part 13; a movement data creation part 15; an animation creation part 17; and an output part 21. The retrieval part 7 identifies a frame which includes feature points that are not extracted. The composition part 11 creates a composed image of the identified frame so that it differs from a composed image of unidentified frames in display configuration. Thus, an editor can grasp frames which include feature points which are not extracted at a glance and perform editing work smoothly. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a system and method for creating an animation based on feature points extracted from an input video.

  In recent years, for example, animation using computer graphics (CG) has been frequently used in television, movies, games, and the like. In addition, since such animations are increasingly applied to characters such as avatars in the communication field, the need for animations that express facial movements is increasing. However, creating an animation requires a great deal of money and time. Therefore, there has been a need for a method and system for creating an animation that looks natural in an easier way.

For example, Patent Document 1 discloses a method of extracting a person's head, eyes, mouth, and the like from an input video, and creating an animation in which the movement of the character is controlled using CG based on the information. In Patent Document 2, a subject with a marker is photographed from a plurality of directions to obtain an image in which the subject is moving, feature points are extracted from the image, and animation is performed based on the extracted feature points. A method of editing such as correcting the position of an extracted feature point or interpolating a feature point that is hidden behind something and not extracted at the time of creation is disclosed. Further, Patent Document 3 discloses an apparatus for extracting a part of a face from a video and extracting a part corresponding to each part of the face, for example, eyes, nose, mouth, etc. as a feature point from the face image. Yes.
JP 2003-85583 A Japanese Patent Laid-Open No. 10-222668 JP 2003-281539 A

  However, in Patent Document 1, since it is assumed to be used in a communication system such as a video phone, for example, when extraction of a person's head, eyes, mouth, or the like fails, other parts are mistakenly placed on the head or There has been a problem that there is no correction means for correcting these extraction mistakes when they are extracted as eyes, mouths, etc., and as a result, a precise animation cannot be created. Further, Patent Document 2 and Patent Document 3 do not include means for assisting editing work, and thus have a problem that the amount of work actually becomes enormous.

  The present invention eliminates the drawbacks of the prior art, can efficiently edit the feature points extracted to create an animation, and can express the natural movement of the character. An object of the present invention is to provide an animation generation method and system.

  In order to solve the above-described problems, the present invention automatically specifies feature points that have not been extracted and frames that include such feature points. Therefore, it is possible to efficiently perform a complementing operation that compensates for feature points that have not been extracted. It is possible to adopt an arbitrary method for this specification. For example, when extracting facial feature points, feature points are extracted after identifying each part of the face, such as the mouth, eyes, and eyebrows. Therefore, identifying feature points that have not been extracted can identify each part of the face. This can be done by determining whether or not.

  Further, in the present invention, preset position coordinates and position coordinates of extracted feature points are stored, and feature points that have not been extracted are automatically complemented using these. Therefore, since the editor only needs to change the position of the feature point, the editing operation of the feature point can be performed more efficiently.

  According to the present invention, feature points can be edited efficiently.

  Embodiments of the present invention will now be described in detail with reference to the accompanying drawings. FIG. 1 is a block diagram conceptually showing the structure of an embodiment of an animation creating system according to the present invention. In FIG. 1, an animation creation system 1 includes a video input unit 3, a feature point extraction unit 5, a search unit 7, a feature point storage unit 9, a synthesis unit 11, a feature point editing unit 13, an action data creation unit 15, and an animation creation unit. 17 and an output unit 21, extract feature points from the video 23 obtained by shooting the moving subject, extract the movement of the subject, create an animation based on the extracted feature points, and output unit 21 It is a system that outputs to.

  The animation creation system 1 will be described more specifically. The video input unit 3 is input with a video 23 of a subject that is moving as a character. When such a video 23 is input to the video input unit 3, the feature point extraction unit 5 extracts feature points from the video 23 for each frame. The feature point is a point necessary for controlling the movement of the character by animation, that is, a main point necessary for expressing the movement. The extracted feature points are stored in the feature point storage unit 9. When feature points are stored in the feature point storage unit 9, the feature point editing unit 13 edits these feature points, and then the edited feature points are used by the motion data creation unit 15 and the animation creation unit 17. An animation is created. Portions that are not directly related to the understanding of the present invention are not shown and redundant description is avoided.

  In the present invention, the configuration of each unit is not limited to the present embodiment, and any configuration can be adopted according to the animation creation system 1. The animation creation system 1 may be realized by, for example, a single device or a plurality of devices. For example, when the animation creation system 1 is realized by a plurality of devices, the video input unit 3 and the output unit 21 are included in the terminal device 31 on the animation receiving side as shown by the dotted lines in FIG. The extraction unit 5, the search unit 7, the feature point storage unit 9, the synthesis unit 11, the feature point editing unit 13, the motion data creation unit 15, and the animation creation unit 17 may be included in the terminal device 33 on the animation providing side. Good.

  When the animation creation system 1 is configured by the two terminal devices 31 and 33 as described above, the terminal device 31 sends the video 23 to the terminal device 33 and receives the animation created from the terminal device 33. Further, the terminal device 33 extracts feature points from the received video 23 and stores them in the feature point storage unit 9, edits the stored feature points to create an animation, and provides it to the terminal device 31.

  Further, for example, although not shown, the terminal device 33 on the animation providing side can be configured by a plurality of terminal devices. For example, the feature point extraction unit 5, the search unit 7, and the feature point storage unit 9 are included in the first terminal device on the animation providing side, the synthesis unit 11, the feature point editing unit 13, the motion data creation unit 15, and the animation creation unit It is also possible to include 17 in the second terminal device on the animation providing side. Note that the present invention is not limited to being realized by a plurality of terminal devices, and the animation creation system 1 may be realized by a single terminal device. In such a case, each unit is controlled and integrated by a control unit (not shown). The present invention is not limited to these examples.

  Each part will be described in more detail with reference to FIG. The video input unit 3 is a part to which the video 23 is input. This video 23 is an image of a moving subject. In this embodiment, the video 23 includes at least two frames. The movement of the subject in the video 23 is a movement to be expressed by an animation created by the animation creating system 1, that is, a movement to be made to be a character represented by the animation. For example, in this embodiment, since it is desired to express the movement of the face, that is, the change of the expression, by animation, the image 23 is obtained by moving each part of the face, for example, eyes, nose, mouth, eyebrows, etc. It is an image of the face.

  When such a video 23 is input, the video input unit 3 supplies the input video 23 to the feature point extraction unit 5. The feature point extraction unit 5 is a part that extracts feature points for each frame from the input video 23. For example, in the present embodiment, the feature point extraction unit 5 divides the input video for each frame, that is, for each still image, and points set as feature points from this still image, for example, the center of the eye, the upper and lower left and right end points of the mouth, etc. Next, the position coordinates of the identified point are calculated, and the feature point is extracted.

  As described above, the feature points are points that are necessary when controlling the movement of the character by animation, and are the main points when expressing the movement. For example, in this embodiment, in order to create an animation in which the facial expression of a character changes, points necessary for expressing facial movement are used as feature points as shown in FIG. FIG. 2 is a diagram conceptually showing a process of extracting feature points, and is a diagram conceptually showing a certain frame 37 in the video 23 inputted to the animation creating apparatus 1. In FIG. 2, only one frame 37 is shown, but this is for ease of explanation, and the video 23 can include any number of frames.

  In the present embodiment, as shown by a square in FIG. 2, the right eyebrow has a substantially right end point, the right eyebrow has a substantially left end feature point B, and the left eyebrow has a substantially right end feature point. C, feature point D at the substantially left end of the left eyebrows, feature point E at the pupil of the right eye, feature point F at the pupil of the left eye, feature point G at the substantially right end of the mouth, and feature point G at the substantially right end of the mouth The feature point H, the feature point I is the substantial center of the upper lip, and the feature point J is the substantial center of the lower lip. Note that the present invention is not limited to the example shown in FIG. 2, and an arbitrary point can be used as a feature point in accordance with movement.

  Therefore, the feature point extraction unit 5 identifies points set as these feature points from each still image divided for each frame, and from position information indicating the position of these points, for example, position coordinates or certain predetermined points The feature point is extracted by calculating the amount of change of. An arbitrary method can be adopted for extracting the feature points. For example, in this embodiment, as shown in FIG. 2, the video 23 is divided into still images for each frame, and the still images are processed to identify points A to J set as feature points. After that, the coordinates indicating the position of points A to J, that is, the position coordinates are calculated with the lower left end point 39 as the origin, the horizontal direction as the x axis, and the vertical direction as the y axis, and the feature points A to J are extracted. Yes.

  In addition, this invention is not necessarily limited to this example, It is possible to employ | adopt arbitrary methods. For example, when photographing a subject with a marker attached to a portion determined as a feature point and extracting feature points A to J, the corresponding marker portion is specified and the position coordinates of the specified marker are obtained. It is also possible. It is not limited to extracting feature points for each frame, and can be extracted for each feature point as long as a feature point corresponding to a predetermined position of a still image of each frame can be finally obtained. is there. The present invention is not limited to this.

  FIG. 3 shows an example of the calculated position coordinates of each feature point. In FIG. 3, n is a positive integer. In this embodiment, feature points A to J are extracted from all frames of the video 23. This is because the subsequent operation data creation unit 13 creates motion data from the feature points A to J in all frames included in the video 23. However, the present invention is not limited to this, and it is possible to extract feature points as much as required by the subsequent operation data creation unit 13. In the present embodiment, two dimensions are used for ease of explanation, but the present invention is not limited to this. For example, when the animation to be created is three dimensions, the x axis and the y axis are orthogonal. It is also possible to further set the z-axis to be three-dimensional.

  The position coordinates of the feature points A to J calculated by the feature point extraction unit 5 are sent to the feature point storage unit 9. The feature point storage unit 9 includes storage means such as a memory, for example, and stores the position coordinates of the sent feature points A to J. The storage can be performed by adopting an arbitrary method. For example, as shown in FIG. 3, a table for each feature point A to J may be created and stored. The present invention is not limited to this. In addition to the feature points, any feature points can be stored in the feature point storage unit 9. For example, in this embodiment, when the feature points A to J are extracted in the feature point storage unit 9. The images used in the above, that is, still images of each frame are also stored.

  When the position coordinates of the feature points A to J are stored in the feature point storage unit 9, the combining unit 11 uses the position coordinates of the feature points A to J stored in the feature point storage unit 9, for example in FIG. As shown, composite images 43, 45, and 47 are created by superimposing each feature point on the image used when these are extracted. The created composite images 43 to 47 are output to a display or the like included in the output unit 21. FIG. 4 is a diagram conceptually showing a composite image. Although only three frames are shown in FIG. 4 for easy understanding, the synthesis unit 21 can create an arbitrary number of synthesized images.

  In the present embodiment, since the feature point extraction unit 5 extracts the feature points A to J for each frame, the synthesis unit creates the composite images 43 to 47 for each frame and displays them continuously in the horizontal direction. Note that the present invention is not limited to this, and for example, it is possible to display a composite image continuously to form a video in which feature points are combined with the original video 23. By creating the composite images 43 to 47 in this way, it is possible to determine whether or not the extracted feature points A to J are extracted from the set points when extracted, and the frame Since it becomes possible to grasp how the feature points A to J move each time it changes, it is possible to assume the movement of the character created by the animation creating unit 15 at the subsequent stage.

  Further, when the position coordinates of the feature points A to J are stored in the feature point storage unit 9, the motion data creating unit 15 uses a known method to store the position coordinates of the feature points A to J on the character. Motion data, which is data defining temporal changes in motion to be generated, is created. Thereafter, the motion data is supplied to the animation creating unit 17, and the animation creating unit 17 creates an animation in which the character moves according to the motion data using a known method. The created animation is supplied to the output unit 21. The output unit 21 includes output means such as a display or a printer, and displays the animation supplied from the animation creating unit 17 on, for example, a display or outputs the paper from the printer to paper.

  A motion is extracted from the video 23 obtained by shooting the moving subject as described above, and an animation of a character that moves based on the extracted motion is created. However, if motion data is created using the position coordinates of the feature points A to J extracted by the feature point extraction unit 5 as they are, the movement of the subject itself may be strange, or the subject may not be captured well, The point extraction unit 5 may not be able to recognize the feature points, and these may not have been extracted, creating motion data that does not make the character's movement natural, and creating an animation with strange character movement. . Therefore, in order to create an animation that is recognized as a natural movement, the feature point editing unit 13 performs an editing operation such as changing the position of the feature point or complementing the feature point that has not been extracted.

  The feature point editing unit 13 is connected to the feature point storage unit 9, corrects the position coordinates of each feature point A to J stored in the feature point storage unit 9, or is not extracted as a shadow of something Editing operations such as complementing feature points are performed. For example, the editing operation is performed by an operation signal of an editor input from an operation unit such as a mouse or a keyboard (not shown), or is automatically performed by a preset position change program. After editing, the position coordinates newly formed by the editing operation are supplied again to the feature point storage unit 9 and stored in the feature point storage unit 9. Thereafter, the motion data creation unit 15 creates motion data from the position coordinates of the feature points A to J after correction and interpolation, and the animation creation unit 17 creates the motion data created by the motion data creation unit 15. Create an animation based on

  Conventionally, however, feature points that have not been extracted have been supplemented manually by the editor, which has a problem of poor work efficiency. More specifically, in the past, the editor determines whether or not feature points have been extracted while viewing the composite video output to the output unit 21, and as a result, there are feature points that have not been extracted. In this case, for example, a new feature point is formed using a mouse or a keyboard, or an appropriate position coordinate is input, so that the work efficiency is low, and an unextracted feature point is overlooked. There was a problem such as. Such a problem is more likely to occur as the number of feature points increases. Therefore, in the present embodiment, the search unit 7 is included, and the search unit 7 automatically specifies the feature points that have not been extracted by the feature point extraction unit 5, and informs the editor to complement the feature points. Details will be described below.

  In FIG. 1, the search unit 7 is a part that identifies feature points that have not been extracted. In the example shown in FIG. 1, the search unit 7 is provided in the feature point extraction unit 5 and identifies feature points that have not been extracted using information formed during the extraction process in the feature point extraction unit 5. The information formed at the time of this extraction process is information created during the process of extracting feature points. For example, in the example shown in FIG. 1, the information is set as feature points as shown in FIG. Information formed when it is determined whether or not to specify a point is used. Note that the present invention is not limited to this, and it is possible to determine whether or not feature points have been extracted using arbitrary information created during the process of extracting feature points.

  FIG. 5 is a diagram showing an example of the processing procedure of the feature point extraction process in the feature point extraction unit 5 shown in FIG. 1 and the feature point specification process not extracted in the search unit 7. In FIG. 5, when the video 23 is sent from the video input unit 3, the feature point extraction unit 5 divides the video 23 into still images for each frame (step S1). Thereafter, the feature point extraction unit 5 extracts a face area, that is, a face area, from the still image (step S2). A known method can be adopted for extracting the face area. For example, when the still image is a color image, a predetermined flesh-colored part is extracted from the still image, and the facial appearance is determined from the extracted flesh-colored part, for example, whether or not the extracted part is substantially circular is determined. By doing so, it is possible to extract the face area. The present invention is not limited to this method.

  Next, the feature point extraction unit 5 determines whether or not a face area has been extracted by the process of step S2, for example, from information such as how circular the area is (step S3). As a result of the determination, if the face area is extracted, the process proceeds to step S4, and from the face area, for example, eyebrows, eyes, nose, mouth, etc., a smaller area for each part, that is, the face part area Is further extracted (step S4). It is possible to adopt a known method for extracting the face part region.

  For example, when the face area is expressed by a monochrome image with gradation from white to black, the eyebrows, eyes, and mouth become darker gray than other parts such as cheeks. Therefore, it is possible to extract a region including the eyebrows, a region including the eyes, and a region including the mouth by specifying a portion having a high density. Similarly, for example, it is possible to identify the boundary between the small nose and the cheek and extract the region including the nose.

  Thereafter, the feature point extraction unit 5 proceeds to step S5, and determines whether or not a face part region has been extracted by the process of step S4, for example, based on whether or not there is a high density portion (step S5). As a result, when the face part area is extracted, the process proceeds to step S6, and the face part included in the area is further extracted from the face part area and set as the feature point in the extracted face part. A point is specified, and the position coordinates of this point are calculated (step S6). A well-known method can also be adopted for extracting the facial parts, specifying the feature points, and calculating the position coordinates.

  For example, a standard image of each face part and data indicating the position of a point set as a feature point in the standard image are prepared in advance, and the standard image and the extracted region are prepared. In comparison, facial parts, such as eyes and mouth, included in this area are extracted. After that, in the extracted face part, a point corresponding to the point set as the feature point in the standard image is specified as the feature point, and the position coordinates in the frame of the feature point are calculated to identify the feature point, and the feature The position coordinates of the points can be calculated. The present invention is not limited to this.

  If the face area cannot be extracted as a result of the determination in step S3, or if the face part area cannot be extracted as a result of the determination in step S5, the process proceeds to step S7, and the search unit 7 extracts the face area. A feature point that has not been extracted or a frame that includes such a feature point is identified from the face region or face part region that has not been created (step S7). For example, when the face area cannot be extracted, the search unit 7 specifies this frame as a frame including the feature points that have not been extracted. When the face part area cannot be extracted, the search unit 7 identifies the feature point corresponding to the face part for which this area could not be extracted as the feature point that has not been extracted.

  Thereafter, the feature point extraction unit 5 causes the feature point storage unit 9 to store the position coordinates of each feature point calculated in step S6. In addition, the search unit 7 adds information indicating that the feature points have not been extracted to the feature points not extracted in step S7 and the frame numbers including the feature points not extracted (step S8). This is done by notifying the feature point that was not extracted and the number of the frame that includes such a feature point to the later synthesis unit 11, and including the feature point that was not extracted or the feature point that was not extracted. This is because the composite image of the frame is generated in a display form different from the extracted feature point or the composite image of the frame including only the extracted feature point. As long as it can be transmitted to the combining unit 11, the notification is not limited to the notification via the feature point storage unit 9. For example, it is also possible to transmit directly from the search unit 7 to the combining unit 11.

  Note that the information indicating that the information has not been extracted can be arbitrarily set. For example, in this embodiment, as shown in FIG. 6, a preset underline is added to a feature point that has not been extracted or a frame number that includes a feature point that has not been extracted. The present invention is not limited to this.

FIG. 6 is a diagram conceptually showing the position coordinates of the feature points A to J, as in FIG. 3. For example the feature point of the mouth of the second frame 51, that is, the feature point G ₂ through J ₂ is not extracted, the search unit, the second frame 51, and the position coordinates of these characteristic points 53,55,57,59 Is underlined. Note that the feature points G _{2 to} J ₂ are not extracted, that is, the position coordinates are not calculated, and therefore no data exists in the position coordinates 53 to 59.

  Returning to FIG. 5, the feature point extraction unit 5 then determines whether or not there are remaining frame images from which feature points are extracted (step S9). If there are remaining frames, the process returns to step S2, and the remaining frames are extracted. If there is no frame, the process of extracting feature points is terminated. As described above, the feature point extraction unit 5 extracts the feature points, and the search unit 7 determines whether or not the feature points have been extracted based on information formed during the feature point extraction process. Therefore, feature points that have not been extracted can be identified with a simple configuration, and feature point extraction processing and feature point identification processing that has not been extracted can be performed at a time. The processing can be performed quickly.

  Note that the present invention is not limited to this, and any method can be adopted. For example, the search unit 7 determines whether or not position coordinates are stored in the feature point storage unit 9, and, as a result of the determination, specifies a feature point that does not store position coordinates as a feature point that has not been extracted. Also good. The present invention is not limited to this.

  Returning to FIG. 1, when it is determined whether or not feature points have been extracted by the search unit 7 as described above, information on which frame feature points have not been extracted, that is, search results are edited. In order to inform the user, the composition unit 11 creates a composite image reflecting the search result. FIG. 7 is a diagram conceptually showing an example of a composite image created by the synthesis unit 11 and reflecting the search result, and is a diagram conceptually showing an output example in the output unit 21. 7, the same reference numerals as those in FIG. 4 denote the same components.

In FIG. 7, since the feature points G _{2 to} J ₂ of the second frame are not extracted in this embodiment, the synthesis unit 11 creates a synthesized image 45 by making the outline of the second frame thicker than the other frames. The editor is informed that there are feature points not extracted in the second frame. Note that the present invention is not limited to the example shown in FIG. 12, and an arbitrary method can be adopted to notify the presence of feature points that have not been extracted. For example, although not shown, instead of creating a composite image reflecting the search results, a list or the like showing only the search results may be created and notified to the editor. For example, although not shown, the background color of the synthesized image may be changed, or the frame or background may be blinked to notify the editor. Further, for example, as shown in FIG. 8, a column indicating search results may be added outside the composite image, and the search results may be described there. The present invention is not limited to these examples.

  FIG. 8 is a diagram conceptually showing another example of a composite image created by the synthesis unit and reflecting the search result, and is a diagram conceptually showing another output example in the output unit 21. . In the example shown in FIG. 8, in addition to the column indicating the composite image, columns 71 and 73 indicating search results are separately provided, and whether or not there is a feature point not extracted in the frame is displayed in the column 71. In the column 73, feature points that have not been extracted are displayed.

  In this way, in the present embodiment, feature points that have not been extracted are automatically specified, and the result is reflected in, for example, a composite image or displayed in a column indicating the result to notify the editor. Therefore, it is possible to immediately find a feature point that has not been extracted by the editor or a frame including such a feature point, and it is possible to smoothly perform a work for complementing such a feature point.

  As shown in FIG. 1, the present invention is not limited to providing the search unit 7 in the feature point extraction unit 5, but can be provided in an arbitrary part according to the animation creation system 1. For example, as shown in FIG. 9, the feature point editing unit 83 can be provided. FIG. 9 is a block diagram conceptually showing the structure of another embodiment of the animation creating system according to the present invention. 9, the same reference numerals as those in FIG. 1 denote the same components.

  In the animation creation system 81 shown in FIG. 9, the feature point editing unit 83 includes a search unit 85. Note that the feature point editing unit 83 includes only the search unit 85 and is the same as the feature point editing unit 13. The search unit 85 exists in the feature point editing unit 83, and the number of feature points stored in the feature point storage unit 9 and the feature points not extracted using the position coordinates of the feature points, and such feature points Identify frames that contain.

  In the search unit 85 shown in FIG. 9, a rule 91 as shown in FIG. 10, for example, defining the number of feature points A to J and the positions of the feature points A to J is stored in advance. Using this rule 90, a feature point that has not been extracted or a frame including such a feature point is specified. In addition, when the feature point that has not been extracted is specified, the search unit 85 notifies the synthesis unit 11 via the feature point storage unit 7 as in the search unit shown in FIG.

  FIG. 10 is a diagram conceptually illustrating an example of rules used in the search unit 85. In FIG. In FIG. 10, m is a positive integer. In FIG. 10, the first rule 93 defines the number of feature points A to J, “10 feature points in one frame”, and specifies a frame including feature points that have not been extracted. It is a rule to do. Therefore, the search unit 85 uses the first rule 93 to determine whether or not the number of position coordinates of each feature point stored in the feature point storage unit conforms to the first rule 93, that is, the feature whose position coordinate is calculated. It is determined whether or not there are 10 points, and a frame that does not comply with the first rule 93 is specified as a frame including a feature point that has not been extracted.

  In FIG. 10, the second rule 95 defines the positional relationship between the feature points E and F that “the feature point E of the right eye is on the right side of the feature point F of the left eye”. Similarly, the third rule 97 defines the positional relationship between the feature points E to H. In the example shown in FIG. 9, by using a rule that defines a positional relationship such as the second rule 95 or the third rule 97, the feature point that has not been extracted is specified, and the position where the position is desired even if it has been extracted. The feature point which is not in is specified.

  For example, when the second rule 95 is violated, it indicates that the feature point E or feature point F does not exist because it has not been extracted, or that the feature point E does not exist at a desired position. By using rules that define the positional relationship in this way, in addition to specifying feature points that have not been extracted, it is possible to specify feature points that have been extracted but do not have the desired position. However, the present invention is not limited to the example shown in FIG. 10, and the rule to be used is at least a feature point that is not extracted or a rule that can specify a frame including such a feature point. Good.

  FIG. 11 is a flowchart showing an example of a processing procedure of search processing using the rule 91 in the search unit 85 shown in FIG. 9 and 11, when the feature point extraction unit 5 extracts the feature points A to J and the position coordinates are stored in the feature point storage unit 9, the search unit 85 performs the feature points A to J for each frame. Are read from the feature point storage unit 9 (step S1). Next, the search unit 85 applies the rule 91 shown in FIG. 10 to the frame and the feature points A to J in the frame, and whether the number of feature points in the frame and the position of the feature points comply with each rule. Is determined (step S2).

  If it is determined in step S2 that the feature points comply with the rules, it is determined whether there are any remaining frames (step S3). If there are no remaining frames, the process ends. Conversely, if there are remaining frames, the process returns to step S1 to read the position coordinates of each feature point in the next frame.

  On the other hand, if there is a feature point that does not conform to the rules as a result of the determination in step S2, information indicating that the feature point has not been extracted is added to the feature point storage unit 9 as in step S9 in FIG. (Step S4). In the example shown in FIG. 9, the information is transmitted to the feature point storage unit 9 via the information to the synthesis unit 11, but the present invention is not limited to this, and may be directly transmitted to the synthesis unit, for example. After that, the synthesizing unit 11 creates a composite image as shown in FIGS. 7 and 8, for example, to the editor that there is a feature point that has not been extracted or that there is a feature point that has not been extracted from a desired position. Inform.

  As described above, in the example shown in FIG. 9, the feature points that are not extracted using the rule 91 are specified, and the result is reflected in, for example, the composite image or displayed in the column indicating the result to the editor. Inform. Therefore, it becomes possible for the editor to find feature points that have not been immediately extracted and a frame including such feature points, and it is possible to smoothly perform the work of complementing such feature points. Also, in the case of specifying using a rule, by preparing a rule regarding the position of each feature point as in this embodiment, it is natural to create a feature point that has not been extracted from a desired position or an animation. It is possible to automatically specify a feature point or the like at a position that is not recognized as a simple movement.

  Note that the present invention is not limited to the examples shown in FIGS. 1 and 9, and it is possible to identify feature points that have not been extracted by adopting an arbitrary method and frames including such feature points. is there. For example, although not shown, when FIG. 1 and FIG. 9 are combined, feature points that are not extracted both at the time of extraction and at the timing after extraction, and frames that include such feature points are specified. For example, although not shown, a search unit can be provided separately from the feature point extraction unit and the feature point editing unit. When the search unit is provided separately from the feature point extraction unit and the feature point editing unit, the feature points that are not extracted are identified using, for example, a rule or whether or not the feature points are stored in the feature point storage unit Is possible. The present invention is not limited to this.

  FIG. 12 is a block diagram conceptually showing the structure of another embodiment of the animation creating system according to the present invention. 12, the same reference numerals as those in FIG. 1 denote the same components. In the animation creation system 101 shown in FIG. 12, the feature point editing unit 103 includes a complementing unit 105, and the complementing unit 105 automatically complements feature points that have not been extracted using preset position coordinates. . Therefore, the burden of editing work can be further reduced, and the editing work can be performed more smoothly. This will be described in detail below.

  In FIG. 12, a feature point editing unit 103 includes a complementing unit 105, and automatically complements feature points that have not been extracted by the complementing unit. Note that the feature point editing unit 103 includes only a complementing unit 105 and is the same as the feature point editing unit 13. The complementing unit 105 exists in the feature point editing unit 103 and automatically complements the feature points that have not been extracted using preset position coordinates. It is automatically formed using the set position coordinates. For example, in the example shown in FIG. 12, the complementing unit 105 automatically selects feature points that have not been extracted using the same type of feature points in the previous frame, that is, the position coordinates of feature points extracted from the same part in the video. Complementary.

  FIG. 13 is a diagram conceptually showing the complementing process in complementing section 105 shown in FIG. In FIG. 13, when the position coordinates of the feature points are stored in the feature point storage unit 9, the complementing unit 105 reads the position coordinates of the feature points stored in the feature point storage unit 9 (step S1). This reading may be performed, for example, for each frame, or may be performed for each feature point, for example, for each feature point A, and any method can be adopted. Further, the present invention is not limited to reading all the frames included in the video 23 and all the extracted feature points. For example, it is possible to read frames and feature points in a range designated by the editor.

  Next, the complementing unit 105 determines whether there is a feature point that has not been extracted (step S2). Any method can be adopted for this determination. For example, in this embodiment, since information such as underlining is added to feature points that have not been extracted, the complementing unit 105 determines whether or not information is added. Note that the present invention is not limited to this, and the search result may be directly transmitted from the search unit 7 to the complement unit 105, for example.

  As a result of this determination, if there are no feature points that have not been extracted, that is, if all feature points have been extracted, the process proceeds to step S5.For example, if complement processing is performed for each frame, the remaining frames For example, if complement processing is performed for each feature point, it is determined whether there is a remaining feature point (step S5). If there is, the process returns to step S1 and remains If no frame or feature point is read, the process ends.

  If there is a feature point that has not been extracted as a result of the determination in step S2, the coordinate position of this feature point is complemented using the coordinate position of the same type of feature point in the previous frame (step S3). . In this embodiment, the position coordinates of the same type of feature points in the previous frame are used as they are for complementation. That is, the position coordinates of feature points that have not been extracted are set to the same type of position coordinates in the previous frame. The present invention is not limited to this example. For example, the intermediate point is calculated using the position coordinates of the same type of feature point in the previous frame and the next frame, and the intermediate point is complemented. It is also possible to set the position coordinates after, and for example, the position coordinates after complementing the position coordinates of the same type of feature point in any frame several minutes before or after, plus a preset amount of change. It is also possible. Further, the present invention is not limited to using the coordinate position of the extracted feature point. For example, the coordinate position stored in advance in the feature point storage unit 9 or the complement unit 105 can be used. For example, the position coordinates set for natural movement may be stored in the feature point storage unit 9, the complement unit 105, and the like, and the complement unit 105 may be supplemented using the position coordinates. The present invention is not limited to these examples.

  Note that, if the replacement unit 105 can form at least the position coordinates of feature points that have not been extracted, for example, when performing complement processing for each frame, the position of the feature points extracted in that frame The coordinates can also be changed to the position coordinates calculated using the position coordinates of the feature points of other frames or the position coordinates of the feature points of other frames, that is, the feature points can be replaced. In the case of replacing the extracted feature points in this way, for example, in the frame including the feature points that have not been extracted, the position coordinates of all the extracted feature points may be replaced. Only the feature points selected by the person while viewing the composite image or the like may be replaced. The present invention is not limited to these examples.

  Thereafter, the position coordinates created by complementation and the replaced position coordinates when replacement is performed are stored again in the feature point storage unit 9 (step S4). When storing in the feature point storage unit 9, for example, by adding information indicating that the position coordinates are created by complementation or replacement, feature points that have been complemented or replaced, and feature points that have not been performed It is also possible to create a composite image by changing the display form, so that these can be distinguished. For example, in this embodiment, the display forms are distinguished as shown in FIG.

FIG. 14 is a diagram conceptually illustrating an example of a composite image when complementation is performed. 14, the same reference numerals as those in FIG. 7 denote the same components. For example, in the case where the position coordinates at the mouth feature points G _{2 to} J ₂ of the second frame are formed by the complementing process in the complementing unit 105, the combining unit 11 makes the frame of the second frame thicker than the other frames. , mouth star feature point G ₂ through J ₂ of which is formed by complementary, also supplement, or a feature point which is unsubstituted create a composite image 45 which is the square, to allow differentiation of both editorial ing. Although not shown in FIG. 14, the display form of feature points whose position coordinates have been changed by replacement may be changed in the same manner as feature points that have been complemented and feature points that have not been complemented or replaced.

  Therefore, the editor can grasp which feature points in which frames have been complemented or replaced by the complementing unit 105 only by looking at the composite images 43 to 47 at a glance. Note that the present invention is not limited to this example, and an arbitrary method can be adopted. For example, it is possible not to notify the editor.

  Returning to FIG. 13, when the complementing unit 105 stores the position coordinates formed by the complementing or replacement in the storage unit 9, the process proceeds to step S <b> 5, and determines whether or not there are remaining feature points or frames. If so, the process returns to step S1, and the remaining frames or feature points are read. If not, the complement process is terminated. Since the feature points that have not been extracted as described above are automatically complemented by the complementing unit 105, the editor only needs to perform the remaining editing work, for example, to move the feature points to positions where natural movement is achieved. It will be over. Therefore, it is possible to reduce the burden of editing work on the editor, and it is possible to perform editing work quickly.

  The present invention is not limited to providing the complementing unit 105 in the feature point editing unit 103, and can be provided in any part depending on the animation creation system. For example, although not shown, the feature point editing unit may include a search unit and a complement unit. In this case, for example, the complementing unit can perform the complementing process by directly receiving the search result from the retrieving unit and reading out the position coordinates used for performing the complementing process from the position coordinate storage unit or the like. Further, for example, as shown in FIG. 15, it can be provided in the feature point extraction unit.

  FIG. 15 is a block diagram conceptually showing the structure of another embodiment of the animation creating system according to the present invention. 15, the same reference numerals as those in FIG. 1 denote the same components. The feature point extraction unit 123 includes only a search unit 125 and a complement unit 127, and is the same as the feature point extraction unit 5. In addition, the search unit 125 only communicates the search result to the complement unit 127, and is the same as the search unit 7, and the complement unit 127 also only exists in the feature point extraction unit 123. Thus, it is the same as the complement unit 105.

  When the complementing unit 123 is provided in the feature point extracting unit 125 as in the animation creating system 121 shown in FIG. 15, for example, the complementing unit 127 temporarily stores the position coordinates of the feature point in the previous frame. In addition, when a feature point that has not been extracted in the next frame is specified by the search unit 125, it is possible to perform complementation processing using the position coordinates of the feature point in the previous frame for complementation. is there. Further, for example, the position coordinates used for complementation may be stored in advance in the feature point storage unit 7, and when a feature point that has not been extracted is specified, the complement unit 127 may read out the position coordinates and perform the complement processing. .

  Further, for example, the position coordinates of the same type of feature point in the next frame may be used for complementation. For example, if complementation is performed with the same type of feature points extracted next, it is not necessary to temporarily store the feature points used for complementation. In addition, this invention is not necessarily limited to these examples, Complementary processing can be performed by adopting an arbitrary method. Further, the complementing unit is not limited to being provided in the feature point editing unit and the feature point extracting unit, but can be provided in any place according to the animation creation system.

It is a block diagram which shows notionally the structure of the Example of the animation production system by this invention. It is the figure which showed notionally the process which extracts a feature point. It is the figure which showed notionally the position coordinate of each feature point. It is the figure which showed notionally an example of the synthesized image produced in the synthetic | combination part. It is the figure which showed an example of the process sequence of the feature point extraction process in a feature point extraction part, and the identification process of the feature point which was not extracted in the search part. It is the figure which showed notionally the position coordinate of each feature point. It is the figure which showed notionally an example of the synthesized image in which the search result was reflected. It is the figure which showed notionally another example of the synthesized image in which the search result was reflected. It is a block diagram which shows notionally the structure of another Example of the animation production system by this invention. It is the figure which showed an example of the rule notionally. It is the flowchart which showed an example of the process sequence of the search process using the rule in a search part. It is a block diagram which shows notionally the structure of another Example of the animation production system by this invention. It is the figure which showed notionally the complement process in a complement part. It is the figure which showed notionally an example of the synthesized image at the time of complementing. It is a block diagram which shows notionally the structure of another Example of the animation production system by this invention.

Explanation of symbols

1 Animation creation system
3 Video input section
5 Feature point extraction unit
7 Search part
9 Feature point storage

Claims (8)

A feature point extracting means for extracting a feature point for controlling the movement of the character from an image of a moving subject,
A feature point editing means for editing the feature points;
An animation creating system including an animation creating unit that creates an animation in which the character moves based on the feature point edited by the feature point editing unit.
An animation creation system comprising: a search unit for specifying a feature point that has not been extracted by the feature point extraction unit and / or a frame including the feature point that has not been extracted.   2. The system according to claim 1, further comprising feature point complementing means for complementing the feature points that have not been extracted. The system of claim 2, wherein the video includes at least two frames;
The feature point extracting means extracts the feature points for each frame;
The feature point complementing means complements the feature points that have not been extracted by the feature point extracting means by using the position information of the feature points of the frames that are not the frames including the feature points that have not been extracted. Animation creation system.   The system according to any one of claims 1 to 3, wherein the video is a video obtained by photographing a movement of a face. A feature point extracting step for extracting feature points for controlling the movement of the character from the video of the moving subject,
A feature point editing step for editing the feature points;
An animation creation method including an animation creation step of creating an animation in which the character moves based on the feature points edited by the feature point editing means.
An animation creating method comprising: a search step of identifying a feature point not extracted in the feature point extraction step and / or a frame including the feature point not extracted.   6. The method according to claim 5, further comprising a feature point complementing step of complementing the feature points that have not been extracted. The method of claim 6, wherein the video comprises at least two frames;
The feature point extraction step extracts the feature points for each frame;
The feature point complementing step complements a feature point that has not been extracted by the feature point extraction unit using position information of a feature point of the frame that is not a frame including the feature point that has not been extracted. How to create animation.   8. The method according to claim 5, wherein the video is a video obtained by photographing a facial motion.

Images