JP2002008057A - Device and method for compositing animation image - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a device and a method of efficiently compositing the face of a substitute person in response to the face pose and the expression of a specified person rendered by animation cartoon, shortening the processing time for image compositing, and improving the quality of the composited image. SOLUTION: A target image is taken out from an animation image frame 10, and the image information of a first set and the image information of a second set in relation to the taken target image are decided, and a digital reference image is taken out from a predetermined static image (namely, a substituent person image frame 20), and the digital reference image is composited on the basis of the image information of the second set in relation to the target image so as to generate the composited image, and this composited image is placed in the target image in the animation image frame 10. The image information of the first set in the area data surrounding the target image and a center coordinate, and the image information of the second set is the element area data surrounding a main structural element in the target image and the feature data of each structural element.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an animation image synthesizing technique, and more particularly to an animation image synthesizing technique which is capable of efficiently synthesizing a desired person's face according to the facial expression and pose of a characteristic person in a moving image frame such as an animation. And a method for synthesizing an animation image.

[0002]

2. Description of the Related Art Conventionally, for extracting a face area from an ID photograph, for example, there are a technique using color distribution information and a technique using a face outline. In the conventional face area extraction method using the color distribution information, a face color histogram is created using a large number of face samples, and the pixel value of a photograph is calculated using the face color histogram, so that the face area is made elliptical. Take out. In addition, the positional relationship between the eyes and the mouth is converted into a mathematical expression based on information on a variety of facial states, and the regions of the eyes and the mouth are obtained using these expressions. However, such a conventional method uses a fixed elliptical mask at the time of extracting a face area, so that if the size of the face is various, it is difficult to extract the face area accurately, and If the face is tilted left or right or has a complicated background, it is difficult to accurately extract the face area.

On the other hand, in the technique using a contour line, it is checked whether each area of the initial image has an inverted U shape, contains a small area inside, or is very small, and determines the most appropriate area. Is selected as the head, and the face area is extracted by approximating the contour of the face with an ellipse using the size of the area. However, such a face area extracting technique of a contour has a disadvantage that it is difficult to accurately extract the positions of the eyes and mouth with high accuracy because the face area is simply extracted as the head area. Therefore, the conventional face region extraction technique using the color distribution information or the contour line information cannot find not only an accurate face region but also various expressions such as delicate changes in eyes and mouth, so that the facial expressions and poses are relatively low. When applied to moving images such as animations using simple colors and exaggerated, it is difficult to realize natural image synthesis.

[0004] Further, even if it is possible to detect constituent elements such as a face outline, eyes and mouth using a conventional technique, it is necessary to extract a specific face for each frame in an animation moving image and replace it with a desired face. It takes a long time, and there is a limit in appropriately reconstructing a desired facial feature portion and replacing it with a specific face in an animation video frame.

[0005]

SUMMARY OF THE INVENTION Accordingly, it is an object of the present invention to provide an image synthesizing method by efficiently synthesizing a substitute person's face so as to match a specific person's face pose and facial expression produced by an animation moving image. An object of the present invention is to provide an improved animation video synthesizing apparatus and method capable of shortening the processing time and improving the image quality of a synthesized animation video.

Another object of the present invention is to provide a storage medium with a built-in program for use in an animation video synthesizing apparatus.

[0007]

According to a first preferred embodiment of the present invention, there is provided a video synthesizing method for synthesizing a predetermined stop video and a target video in a video frame. Wherein the predetermined stop image and the target image are each composed of a plurality of pixels, extract the target image from an image medium having the image frame, and set a first set of image information for the extracted target image. And a step of determining a second set of image information, and extracting a digital reference image from the predetermined stop image,
B. Synthesizing the digital reference video based on the second set of video information with respect to the target video to provide a synthesized video, and replacing the target video in the video frame with the synthesized video. and c) providing a video composing method.

According to another second preferred embodiment of the present invention, there is provided a video synthesizing apparatus for synthesizing a predetermined still image and a target image in an image frame, wherein the predetermined still image and the target image are combined. Video extracting means for extracting the target image from an image medium having the image frame, wherein the target image is composed of a plurality of pixels, and determining a first set of image information and a second set of image information for the extracted target image A first synthesizing unit for extracting a digital reference image from the predetermined stop image, and synthesizing the digital reference image based on the second set of image information through coordinate movement and pixel interpolation to generate a synthesized image. Means for replacing the target image in the image frame with the synthesized image, the first set of images Wherein the information has region data surrounding the target image and center coordinates, and the second set of image information has element region data surrounding main components in the target image and characteristic data of each component. Is provided.

[0009]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of the present invention will be described below in detail with reference to the drawings.

FIG. 1 is a schematic block diagram of an animation image synthesizing apparatus according to a first preferred embodiment of the present invention. The animation video synthesizing apparatus 100 of the present invention includes an animation face extracting unit 30, an alternative face synthesizing unit 40, an image synthesizing unit 50, and a post-processing unit 60.

The animation face extracting unit 30 receives, for example, a plurality of animation video frames 10 written on a storage medium such as a hard disk, and extracts and masks, for example, the main character's face in each video frame, and performs masking. Is stored in an internal memory (not shown), and outputs information of the animation image frame 10, face area information related to the hero's face, and face change information. Here, the face area information indicates the entire area of the face including the main components such as eyes and mouth, and the face change information indicates the degree of rotation or the facial expression of the protagonist's face. These face area information and face change information are obtained for each frame.

An alternative face synthesizing unit 40 according to a first preferred embodiment of the present invention receives an alternative person image frame 20 which is a still image such as a photograph taken by a camera or the like, and receives a face change from an animation face extracting unit 30. Based on the information, the alternative face is synthesized, and the synthesized reference video is output.

An image synthesizing unit 50 masks the synthesized reference image input from the alternative synthesizing unit 40 based on the information of the animation image frame 10 from the animation face extracting unit 30 and the face area information of the hero. This is combined with the face area that has already been output.

Next, the post-processing unit 60 processes an error portion in the animation video frame synthesized by the video synthesis unit 50, and outputs the finally synthesized animation video frame to the disc play 70.

The display 70 marks the input animation video frame, that is, the synthesized animation video frame in real time. Optionally, the post-processing unit 50
Animation video frames output from
(Not shown), and can be reproduced by a video reproducing device as needed.

Hereinafter, the animation face extracting unit 30 according to the preferred embodiment of the present invention will be described in detail with reference to FIGS.

[0017]Removing the box adjacent to the face  As shown in FIG. 3A, the user determines the position of the mouse or the like.
First loaded on a marking device (not shown) using a
Of the face in the animation video frame 10
The appropriate reference position Pref is specified (step S31). face's
If the reference position Pref is designated, the first area extraction
The adjacent box surrounding the face is sought through Gorism 31
Is performed (step S32). More specifically, step S32
By specifying the reference position Pref of the face with the mouse,
A predetermined value in the radial direction with the pixel value at the reference position
An adjacent pixel having similarity is searched for a face area (the actual area in FIG. 3).
Line) and expand / decrease the maximum / minimum pixels in the expanded face area
Determine pixel coordinates A, B, C, and D having values, and
An adjacent box (dotted line in FIG. 3) of the connected face is obtained. Face
Face adjacent box information including area is animated face extraction
The information is output as face area information by the selector 30.

Note that the reference position and reference pixel value of the subsequently input animation video frame are set as the reference position and reference pixel value specified in the previously input and processed video frame. I want to be. In addition, when the outline of the face is divided into two or more in the animation video frame 10, that is, when the outline is divided into two areas by the lightness and darkness of the face, the user determines the approximate position of each light and dark area. Designate as reference position,
Through the above process, the box adjacent to the face can be extracted.

[0019]Removing the box adjacent to the eye  Next, in step S33, the second region extraction algorithm
32 removes the adjacent box surrounding the eye
You. When the user moves roughly inside each eye area in the face-adjacent box
If the heart is specified using a positioning device such as a mouse, the face
Approximate eye position range with a fixed ratio of adjacent box sizes
Is automatically determined. Next, the user uses the mouse to
If the iris and iris regions of the eye are sequentially specified, the corresponding pixel
A secondary box image represented by the value is generated
You. Then, for the binary image of the rough eye box,
Many pixels are concentrated in any row and column
By performing a projection to determine whether
A substantial eye area including an eye area and a white eye area is detected.
It is.

Specifying the iris region of each eye is for searching for the substantial center position of each eye, and the center position of the eye specified in the current video frame is the center position of the eye in the succeeding video frame. Is set as The adjacent box of the eyes determines four pixel coordinates having the maximum / minimum pixel value among the searched pixel values in the same manner as the face adjacent box extraction method, and connects the respective coordinates to obtain the coordinates of FIG. ) Are obtained as shown in FIG.

[0021]Removing the box next to the mouth  The box adjacent to the mouth is the third region extraction algorithm 33
And is taken out in step S34 using. The user is
Use a mouse to set the reference position on the upper lip or lower lip.
Specify one point. Once the reference position is specified, the process
Is the image of the reference position in the same way as
Neighbors that have a certain degree of similarity in the radial direction around the prime value
The pixel is searched and the box adjacent to the mouth as shown in FIG.
Take out MX.

In the present invention, only the eyes and the mouth are described as the main feature parts of the face. However, for different feature parts such as the ears and the nose, the area can be expanded by the above-described method. Description is omitted.

[0023]Extracting facial expression information  FIGS. 4 and 5 show a table according to a first preferred embodiment of the present invention.
2 using the information extraction algorithm 34
To explain the facial expression information extracting method performed in S35
FIG.

As shown in FIG. 4, the process measures the eye width We and the height He in the box adjacent to the eye retrieved in step S33. The eye width We is determined as the distance between the maximum and minimum pixel coordinates among the pixel coordinates located on the X-axis, and the height He is between the maximum and minimum pixel coordinates among the pixel coordinates located on the Y-axis. Is calculated as the distance of The expression (open / close) of the eyes can be produced by moving the eyelids up and down. In order to clearly detect the degree of opening and closing of the eyes, it is preferable to select a video frame in which the face of the animation hero has a normal expression such as an ID photo as the first video frame.

For example, if no black and white eye regions are detected in the extracted eye adjacent box, it is determined that the eyes are closed, and the degree of opening and closing of the eyes can be calculated as 0. That is, the height He is zero. FIG. 4 (a)
Shows a state where the eyes are fully opened, and FIG. 4B shows a state where the eyes are half-opened.

Therefore, the expression data related to the eyes is obtained by calculating the degree of opening and closing of the eyes using the expression information extracting algorithm 34 as described above. Similarly, the facial expression (opening / closing) of the mouth can be produced by moving the upper lip and the lower lip up, down, left, and right. In order to accurately detect the degree of opening and closing of the mouth, it is preferable to select a video frame in which the face of the animation hero has a normal expression such as an ID photo as the first video frame.

As shown in FIG. 5, the mouth, unlike the eye, can be represented as two adjacent boxes, an outer box Ebox and an inner box Ibox. As previously mentioned,
The outer box Ebox is obtained by connecting the maximum and minimum pixel points located on the X axis and the Y axis among the searched pixels, and the inner box Ibox has a pixel value different from the lip pixel value in the outer box Ebox. Is obtained by connecting the maximum and minimum pixel coordinate points located on the X axis and the Y axis among the pixel coordinates having

By calculating the ratio of the heights of the two adjacent boxes determined in this way, ie the height M0 of the outer box Ebox and the height Mi of the inner box Ibox, it is possible to associate the height with the mouth. The extracted facial expression data is extracted.

Thereafter, the eye-related expression data and the mouth-related expression data extracted in step S35 are transmitted to the alternative face synthesizing unit 40 in FIG.

[0030]Extracting face pose information  Hereinafter, a first preferred embodiment of the present invention will be described with reference to FIGS.
A method of extracting face pose information according to an embodiment will be described.

In general, face poses can be identified by various rotation directions of the face, and can be roughly classified into three types.
As shown in FIG. 6A, the first pose is a pose of tilting the neck (hereinafter, referred to as “saw rotation”), and the second pose is a pose of swinging the neck left and right (hereinafter, “negative rotation”). The third pose is a pose of shaking the head straight (hereinafter, referred to as “positive rotation”).

In step S61 of FIG. 7, the process determines whether the center of the animation protagonist's face has changed. Here, the center of the face indicates a line segment perpendicular to the line segment connecting the centers of both eyes. If the center of the face has not changed in step S61, the process determines that the current face is the front, and outputs the front information, that is, 0 rotation data (step S62).

On the other hand, if the center of the face has changed in step S61, the process checks whether the center of the face has changed to the left or right (step S63). It is determined that the state is "", and the corresponding rotation degree is calculated (step S64). FIG.
(b) is an example showing a “negative rotation” state, in which the distance from the center of the left and right eyes to the face box boundary is different, and the distance from the center of one eye to the face box is shorter, That is, it is a form rotated to the left. The degree of the negative rotation is determined by calculating the ratio of a long distance b to a short distance a from the center of the eye to the face box boundary. The rotation ratio obtained in step S64 is output as negative rotation data.

At step S63, if the center of the face has not changed to the left or right, the process checks whether the center of the face has moved up and down (step S65). It is determined that the state is "positive pause", and the corresponding rotation degree is calculated (step S66). FIG.
(c) shows the “positive rotation” state, in which the maximum and minimum pixel coordinates on the y-axis of the face area have moved up and down. For example, when the main character's face faces downward, the maximum pixel coordinate Pmax of the Y-axis of the main character's face area moves downward, and the minimum pixel coordinate Pmin of the y-axis.
Will not change. On the other hand, when the main character's face is facing upward, the minimum pixel coordinate Pmin of the Y-axis of the main character's face area moves upward, and the maximum pixel coordinate P of the Y-axis changes.
max will not change. That is, the positive rotation data is Y
It is determined by calculating the displacement of the maximum and minimum pixel coordinates with respect to the axis. The displacement amount obtained in step S66 is output as positive rotation data.

If the center of the face has not changed in the upper or lower direction in step S65, the process checks whether the center of the face has been tilted left and right (step S6).
7) If tilted, it is determined that the current face is in the “Surpose” state, and the degree of rotation is calculated (step S6).
8). FIG. 6 (d) shows a "Surpose" state, in which the line connecting the centers of the eyes in each adjacent box is inclined with respect to the horizontal line. The degree of inclination of the eyes is determined by the horizontal distance Lh from the center of the line connecting the centers of both eyes to the center of the left eye (or right eye) and the vertical distance Lv from the center of the left eye (or right eye) to the horizontal line. Is calculated by calculating the ratio Dy. The ratio obtained in step S68 is output as the saur rotation data.

In step S70, the face pose data is output to the alternative face synthesizing section 40 shown in FIG. 1 based on the negative rotation data, the positive rotation data, and the sad rotation data obtained through the above process.

[0037]Synthesis of facial expressions  FIG. 8 shows an alternative face synthesizing unit 4 according to the first preferred embodiment of the present invention.
FIG. 9 is a diagram for explaining the process performed in FIG.
10 and 10 show the method of synthesizing faces based on facial expression data, respectively.
It is a schematic diagram and a flowchart for explaining.

In step S41, the user obtains an area of a substitute face from the substitute person image frame 20 by a normal method using a positioning device such as a mouse.

In step S42, the user designates main feature points of the face on the acquired face area and generates a mesh connecting the feature points. Each feature point is the main boundary of the face (for example,
Eyebrow, eyes, mouth, etc. must be located at the upper and lower left and right boundaries) or at the bent part (for example, nose tip, cheekbones), the mesh is a component of the face (i.e., eyes, mouth, etc.) by a mesh grid You must be surrounded.

More specifically, as shown in FIGS. 9A and 9B, an adjacent box including an eye mesh and an adjacent box including a mouth mesh are set. In FIG. 9, a mark “□” located at an adjacent box boundary or the like indicates a fixed point, and a mark “■” located on the mesh indicates a moving point to be moved according to expression data from the animation face extracting unit 30. The mark “□” is a target point indicating the degree of opening and closing of the actual eyes (or mouth) of the animation hero. As shown in FIG. 9A, the eye opening / closing effect can be produced by controlling the moving point す る located at the eyelid mesh point. FIG.
The predetermined space provided between the mesh and the adjacent box, that is, between the fixed point and the moving point is a pixel set used as a reference pixel at the time of pixel interpolation (described later).

As shown in FIG. 9B, the opening and closing effect of the mouth can be produced by moving each moving point 下 located at the boundary between the lower lip and the upper lip up and down. The moving points "" and "" provided on the left and right sides with respect to the moving point provided at the vertical center of the lips move up and down in half proportion to the horizontal distance from the moving point "". By providing, the expression of the mouth can be produced more naturally. In fact, in the alternative face, the mesh of the mouth may be distorted due to the increase in the height of the facial expression when the facial expression is synthesized, and the form of the synthesized animation hero face may be distorted. It is preferable to create a mesh by setting to.

In step S43, the process reconstructs (synthesizes) the facial expression associated with the eyes and mouth of the substitute person using the first synthetic algorithm 40a based on the facial expression data input from the animation face extracting unit 30. . FIG. 10 is a flowchart for explaining the first combination algorithm 40a.

As shown in FIG. 10, in step S11, the process determines whether or not facial expression data has been input from the animation face extracting unit 30. If facial expression data is input, the process determines in step S12 whether the data is eye-related expression data. If it is determined in step S12 that the data is the eye-related expression data, the process moves the mesh control point as described with reference to FIG. Synthesize the eyes.

On the other hand, if the data input from the animation face extracting unit 30 is not the eye-related expression data in step S12, the process proceeds to step S14 to determine whether the data is the mouth-related expression data.

If it is determined in step S14 that the input data is the mouth-related expression data, the process moves the mesh control point in step S15 as described with reference to FIG. The substitute person's open mouth expression is synthesized. That is, as shown in FIG. 9B, the opening point is moved by moving each moving point located at the upper part in the box adjacent to the mouth upward and moving each moving point located at the lower part downward. Expressions can be combined. At this time, it is assumed that the maximum size or height of the upper lip and the lower lip is fixed.

[0046]Synthesis of face poses  Hereinafter, using the second synthesis algorithm 40b according to the present invention,
Explain how to compose the alternative person's face pose.
You.

In step S44, the process is based on a series of pose data input from the animation face extracting unit 30 of FIG. 1, ie, front data, negative rotation data, positive rotation data, and sad rotation data, and refer to FIG. As described above, by rotating the center of the face of the substitute person, the face of the substitute person is combined with the face area of the animation hero.

In the synthesizing process in the alternative face synthesizing section 40 of the present invention, the facial expression of the eyes and the mouth are synthesized first and the pose of the face is synthesized later.

According to the first preferred embodiment of the present invention,
After each moving point is moved (or expanded) to the target point in the box adjacent to the eye or mouth, a pixel interpolation technique is used to fill an appropriate pixel value between the moving point and the target point. FIG.
FIG. 3 is a schematic diagram for explaining a pixel interpolation technique according to the present invention.

More specifically, when the second moving point ■ “” located in the upper part of the box adjacent to the eye shown in FIG. 10 is moved (expanded) to the second target point □ “”, 2 If the space between the second moving point ■ and the second target point □ is filled based on the pixel value corresponding to the second moving point ■, the filled pixel value and the pixel value in the adjacent box do not match. Become natural.
Therefore, in the present invention, as shown in FIG. 11A, a pixel value to be inserted into the expansion area is obtained by using a pixel value between each fixed point and each moving point located on the adjacent box.

That is, the input pixel value between the fixed point and the moving point is, for example, “90, 80, 60, 65, 71, 81”.
, The output pixel value to be inserted into the expansion area is, for example, “90, 83.3, 73.2, 60, 65, 69.
By interpolating and outputting as in the case of 08, 74.5, 81.18 ", the expression of the eyes is processed more naturally. FIG. 11B shows the pixel value when the mesh moving point is contracted. Is thus interpolated. In this way, the face image of the alternative person in which the facial expressions of the eyes and the mouth are synthesized is input to the image synthesizing unit 50 of FIG.

FIG. 12 is a schematic block diagram showing an animation video synthesizing apparatus 200 according to a second preferred embodiment of the present invention.

The animation image synthesizing apparatus 200 according to the second preferred embodiment of the present invention includes a plurality of alternative human image frames 2 which are still images such as photographs taken by a camera or the like.
This embodiment is the same as the first embodiment except that corresponding reference videos are extracted from 0 ′ and these reference videos are used at the time of synthesis. Here, the reference image means an image that is taken in order to optimally obtain a target composite image.

As shown in FIG. 13, the plurality of substitute person image frames 20 'have seven reference images: front, upper, lower, half left, half right, left, and right.

As described with reference to FIG. 8, the alternative face synthesizing unit 40 'extracts the above-mentioned seven reference images from the alternative person image frame 20', and applies the extracted reference images as shown in FIG. Create a simple face mesh and store it
(Not shown).

Prior to the description, a case where only a front reference image is referred to when a desired face is synthesized based on the expression and pose of the protagonist of the animation is considered. When the face of the animation hero is rotated violently right and left, if the reference image of the front is used as the source mesh, the y coordinate of each mesh point changes and the image whose side is distorted as shown in FIG. Bring. Similarly, when an intermediate image is generated during vertical rotation, the x-coordinate of each mesh point changes, and although not shown, an image in which the upper and lower portions are distorted is generated. In other words, when the direction change occurs violently on the left and right sides, the image quality of the synthesized video may be degraded. Therefore, in the second preferred embodiment of the present invention, the source mesh is set as the front reference image, and the half-right or half-left reference image is set as the source mesh when rotated to the side more than the half right or half left. To generate an intermediate frame (intermediate video) for an arbitrary pose. Here, the intermediate frame is an image between the front reference image and the half-left and right reference images, an image between the front reference image and the upper and lower reference images, or a half-left and right reference image and the left and right reference images. It means an image between the reference image.

More specifically, as shown in FIG. 15, the face pose data input from the animation face extracting unit 30 is shifted from the front reference image 20 "to the right by a predetermined reference value 0.
When rotated further than 5, that is, when it is located at the position R, the alternative face synthesizing unit 40 'extracts the previously written half-right reference image and sets the image as the source mesh.
After that, the alternative face synthesis unit 40 'generates an intermediate image mesh based on the half-right reference image, and then performs the above-described linear interpolation and pixel interpolation on the intermediate image mesh to generate a desired intermediate image. I do. At this time, the linear interpolation is x
The coordinate component and the y-coordinate component are individually obtained and performed.

According to a second preferred embodiment of the present invention,
Provided is an algorithm for implementing an effect that a face is tilted left and right and rotated clockwise or counterclockwise. 180 ° and 2 which are multiples of 90 ° are set around the rotation axis of the face image.
In the case of 70 °, it can be realized by simple coordinate offset, and in the case of other rotation angles, it can be realized by applying the normal bilinear interpolation method and the pixel interpolation method of the present invention.

Finally, the image synthesizing section 50 receives an input from the alternative face synthesizing section 40 based on the face area information of the animation hero provided from the animation face extracting section 30, that is, the image information in which the hero's face is masked. The synthesized face image thus synthesized is synthesized with the masked face area of the animation hero.

In the post-processing unit 60, when a space occurs between the face area of the animation hero and the alternative face area due to a process error, the pixel located at the boundary portion of the alternative face using the pixel interpolation technique of the present invention. Fill the space based on the values or calculate the average pixel value between the pixel value close to the outside of the face area of the animation hero and the pixel value located at the boundary of the alternative face, and calculate the average pixel value. Can be buried in space.

As a result, the animated hero's face is replaced with the face of the substitute person, or the synthesized animation video frame is stored in a data storage device (not shown) such as a hard disk of a computer or a video cassette recorder. Since the animation video synthesizing process for the subsequent frame is performed by repeating the above-described series of processes, description thereof will be omitted.

Although the present invention has been described above with respect to a two-dimensional animation image, it is needless to say that the present invention can be applied to a three-dimensional image medium by using the pose / expression synthesis technique and the pixel interpolation technique of the present invention. is there. Therefore, there is an advantage that the image of the two-dimensional product provided by the shopping company during home shopping using a computer can be reproduced and felt as a three-dimensional image on a personal computer.

Although the preferred embodiment of the present invention has been described above, those skilled in the art will be able to make various modifications without departing from the scope of the present invention.

[0064]

Therefore, according to the present invention, the following effects can be obtained. 1. ADVANTAGE OF THE INVENTION According to this invention, the main character's face produced by an animation moving image using one photograph can be replaced with his / her own face, can also be used for education, and becomes the main character at a low production cost. A moving image can be produced. 2. According to the present invention, by acquiring a plurality of reference videos and using a reference video most suitable for the rotation direction of the face of the animation hero, it is possible to produce an animation moving image with more natural and improved image quality. 3. According to the video synthesis technique of the present invention, a video of a product can be reproduced as a three-dimensional video during home shopping using a computer. 4. By using the video synthesis technique according to the present invention, the game main character can be replaced with the user's face during the game, so that the game can be more interesting.

[Brief description of the drawings]

FIG. 1 is a schematic block diagram of an animation image synthesizing apparatus according to a first preferred embodiment of the present invention;

FIG. 2 is a flowchart illustrating an operation of an animation face extracting unit according to the present invention.

FIG. 3 is a schematic diagram for explaining a method of extracting an adjacent box from a main face image according to the present invention;

FIG. 4 is a schematic diagram illustrating a method for extracting facial expression information according to a first preferred embodiment of the present invention;

FIG. 5 is a schematic view for explaining a method for extracting facial expression information according to the first preferred embodiment of the present invention;

FIG. 6 is a schematic diagram for explaining a method of extracting face pose information according to the first preferred embodiment of the present invention.

FIG. 7 is a flowchart illustrating a method for extracting face pose information according to a first preferred embodiment of the present invention;

FIG. 8 is a flowchart illustrating an operation of the alternative face extracting unit according to the first preferred embodiment of the present invention;

FIG. 9 is a schematic diagram for explaining a method of synthesizing a face of a substitute person based on facial expression data according to the present invention.

FIG. 10 is a flowchart illustrating a method of synthesizing a substitute person's face based on facial expression data according to the present invention.

FIG. 11 is a schematic diagram illustrating a pixel interpolation technique according to a preferred embodiment of the present invention.

FIG. 12 is a schematic block diagram of an animation image synthesizing apparatus according to a second preferred embodiment of the present invention;

FIG. 13 is a schematic diagram showing a plurality of reference videos.

FIG. 14 is a schematic diagram showing a composite video image distorted by horizontal rotation.

FIG. 15 is a schematic diagram showing a case where a face pose is tilted rightward from a front reference image.

FIG. 16 is a schematic diagram illustrating a process of performing a series of facial synthesis in the order of facial expressions and poses according to the present invention.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 10 Animation video frame 20 Alternative person video frame 30 Animation face extraction part 40 Alternative face synthesis part 50 Video synthesis part 60 Post-processing part 70 Display

Continuing on the front page (72) Inventor Zen Byung-Hwan South Korea, Chungcheongnam-do, Cheonan-si, Sanggyong-dong, Wolfeng 1-star, Part 509-501 (72) Inventor Choi Jun Shin 270-75 Kaifeng 3-dong, Guro-gu, Seoul, Korea. No. 102 (72) Inventor Xu ▲ Jun ▼ Source 398-10, 26/1 F-term (Reference) 5B050 AA06 AA08 AA09 BA08 BA12 CA07 DA02 DA04 DA07 EA07 EA09 EA12 EA18 EA19 EA24 FA02 FA05 GA08 5B057 AA20 BA02 BA29 CA01 CA08 CA13 CA16 CB01 CB08 CB13 CB16 CC03 CD03 CE08 DA08 DB03 DB06 DB09 DC03 DC25 DC32

Claims (29)

[Claims] 1. A video synthesizing method for synthesizing a predetermined stop video and a target video in a video frame, wherein the predetermined stop video and the target video each include a plurality of pixels, Extracting a target image from an image medium having a frame, determining a first set of video information and a second set of video information for the extracted target image, and digitally referring to the predetermined stop image. Extracting a video, synthesizing the digital reference video based on the second set of video information with respect to the target video to provide a synthesized video, and combining the target video in the video frame with the synthesized video. C) replacing the image with an image. 2. The method according to claim 1, wherein the image medium is an animation image. 3. The method according to claim 2, wherein the target image is a face of a specific person in the animation image. 4. The method according to claim 1, wherein the step (a) comprises: extracting a first boundary area surrounding the face of the specific person in the animation image; and enclosing a main component of the face within the first boundary area. 4. The method according to claim 3, further comprising the step of extracting a plurality of second boundary areas. 5. The method according to claim 1, wherein, in the step a1, when a predetermined position in the first boundary area is designated by a user, adjacent pixels having a predetermined similarity based on a pixel value of the designated position are used as a reference. A111 step of determining the first boundary area by searching for; and determining pixel coordinates having maximum and minimum pixel values in horizontal and vertical directions among the searched neighboring pixel values, A first adjacent area is obtained by connecting the coordinates so as to surround one boundary area, and a coordinate corresponding to the center of the face of the specific person is selected from the pixel coordinates, and the first adjacent area and the face are selected. Outputting the center coordinates of the first set as the first set of image information. 6. The main component of the face of the specific person includes a pair of eyes and a mouth, and the a2 step includes the step of: when the user specifies a general center position of each eye, the first boundary. A21 step of searching for a pixel value belonging to a predetermined range in the area; and determining a row and a column in which the searched pixel value is concentrated for each eye, and based on the determination, A22 determining a second boundary region including a binary image representing a black eye and a white eye; and a maximum and a minimum pixel value in the horizontal and vertical directions among pixel values located in the second boundary region for each eye. 5. The method according to claim 4, further comprising the step of: a23 determining a second adjacent area for each eye by determining pixel coordinates having the following and connecting the respective coordinates. 7. The step a2, when two approximate center positions of the mouth are designated by a user, adjacent pixels having a predetermined similarity based on a pixel value of each center position are determined. A24 step of searching to determine a second boundary region for the mouth, and determining pixel coordinates having maximum and minimum pixel values in the horizontal and vertical directions among the pixel values searched in the step a21. 7. The method according to claim 6, further comprising the step of: a25 determining a second adjacent area to the mouth by connecting the coordinates. 8. The a25 step: connecting maximum and minimum pixel coordinates on the horizontal and vertical sides among pixel coordinates having a pixel value different from the lip pixel value in the second boundary region with respect to the mouth. The method according to claim 7, further comprising: a26 obtaining a second internal boundary area adjacent to the mouth formed inside the second boundary area. 9. The step a) after the step a23, calculating a width and a height of the second boundary region with respect to each eye, and calculating a ratio of the calculated width and height to a first expression. A27 step of outputting as data, calculating the height of the second boundary area and the height of the second internal boundary area with respect to the mouth, and outputting the calculated ratio of each height as second expression data A28 stage to be performed,
The method according to claim 8, further comprising: 10. The method according to claim 1, wherein the step (a) is performed after the step (a23).
Each distance to the boundary of the adjacent area is calculated, and if the calculated distances are the same, first rotation data is output. If the distances are not the same, a smaller distance among the distances is output. A29 step of outputting the value of as the second rotation data; determining whether the maximum and minimum pixel coordinates with respect to the vertical axis have moved up and down in the first adjacent area, and assigning a value corresponding to the degree of movement to the A3 output as 3 rotation data
Step a31, determining whether a line segment connecting the centers of the second adjacent regions to the respective eyes is inclined with respect to the horizontal line segment, and outputting a value corresponding to the degree of inclination as fourth rotation data. 9. The video synthesizing method according to claim 8, further comprising: 11. The center position and reference pixel value of each eye specified in the animation video frame,
7. The method according to claim 6, wherein the method is used as a reference position and a reference pixel value of a subsequent animation video frame. 12. The method according to claim 1, wherein the step (b) comprises: extracting the digital reference image from the predetermined stop image, and designating a feature point including a fixed point and a control point on the main component in the reference image. 10. The video compositing method according to claim 9, comprising: b1 step; and b2 step of generating the mesh by connecting each feature point so that a main component of the face of the specific person is surrounded by a mesh grid. Method. 13. The method according to claim 12, wherein the control point is located on each eyelid of the reference image. 14. The method according to claim 12, wherein the control point is located on a second boundary area and a second internal boundary area for the mouth of the reference image. 15. The method according to claim 12, wherein the second adjacent area is formed at a predetermined distance from the second boundary area. 16. The method according to claim 1, wherein the step (b2) includes moving the control point designated in a second boundary area with respect to the eye by a distance corresponding to the first expression data.
The method further comprises the following three steps: and a step b4 of moving the control point specified in the second boundary area or the second inner boundary area with respect to the mouth by a distance corresponding to the second facial expression data. The video composing method according to claim 12. 17. The method of claim 10, wherein the b2 step includes a b5 step of rotating a center of the reference image based on at least one of the first to fourth rotation data. Video synthesis method. 18. In the b4th step, each control point adjacent to a control point having a maximum and minimum vertical axis pixel value in a second internal boundary region with respect to the mouth is referred to as a reference control point. 17. The video composing method according to claim 16, wherein the video is vertically moved in half proportion to the horizontal distance. 19. The control point of the mesh for the mouth,
19. The method according to claim 18, wherein the image is extended to the outside from a control point of the mesh for another component. 20. After moving the control point by a distance corresponding to the first and second expression data, the fixed point is determined based on a series of pixel values located between the control point and the fixed point. 2. The method according to claim 1, further comprising: obtaining a pixel value corresponding to the moved distance from the image data; and interpolating the pixel value between the moved distances to generate the mesh.
9. The video synthesizing method according to 9. 21. A video synthesizing method for synthesizing a predetermined stop video and a target video in a video frame, wherein the predetermined stop video and the target video each include a plurality of pixels, Extracting the target image from the image medium having a, and determining a first set of image information and a second set of image information for the extracted target image, a; Fetching and storing the digital reference image, and searching for and synthesizing the digital reference image to be optimally referred to the second set of image information from the plurality of digital reference images to generate an optimum synthesized image B. Replacing the target video in the video frame with the synthesized video. C. Method. 22. The method according to claim 21, wherein the video medium is an animation video. 23. The target image is a specific person's face in the animation image, the first set of image information is area data surrounding the specific person's face and center coordinates of the face, 23. The video synthesizing method according to claim 22, wherein the video information of the set is expression data for main components in the face of the specific person and rotation data indicating a degree of rotation of the face. 24. The method of claim 21, wherein the plurality of digital reference images are front, upper, lower, half-left, half-right, left, and right images of the face. 25. The step of b, wherein the rotation data in the second set of image information indicates that the specific face of the animation image is located in a first direction. 24. The video synthesizing method according to claim 23, wherein the searched reference video is searched from the plurality of digital reference videos, and the optimum synthesized video is output using the searched reference video. 26. An image synthesizing apparatus for synthesizing a predetermined still image and a target image in a video frame, wherein the predetermined still image and the target image each include a plurality of pixels, Video extracting means for extracting the target image from an image medium having a frame and determining a first set of image information and a second set of image information for the extracted target image; and a digital reference from the predetermined stop image. First synthesizing means for extracting an image, synthesizing the digital reference image based on the second set of image information through coordinate movement and pixel interpolation to generate a synthesized image, and the target image in the image frame. Second combining means for replacing the combined image with the combined image, wherein the first set of image information surrounds the target image. It has data and the center coordinates, the image synthesizing apparatus according to claim video information of the second set to have the characteristic data of the principal components surrounding element regions data and each component in the target image. 27. The video synthesizing apparatus according to claim 26, wherein the video medium is an animation video, and the target video is a face of a characteristic person in the animation video. 28. An image synthesizing apparatus for synthesizing a predetermined still image and a target image in a video frame, wherein the predetermined still image and the target image each include a plurality of pixels, Video extracting means for extracting a desired target image from an image medium having a frame, extracting a first set of image information and a second set of image information for the extracted target image, and from a plurality of the predetermined stop areas Optimum synthesized video by retrieving and storing a plurality of said digital reference videos, and searching and synthesizing from said plurality of digital reference videos a digital reference video to be optimally referenced to said second set of video information And a second synthesizing unit that replaces the target video in the video frame with the synthesized video. Wherein the first set of image information has area data and center coordinates surrounding the target image, and the second set of image information has element area data surrounding main components in the target image and characteristics of each element. A video synthesizing device having data. 29. The apparatus according to claim 27, wherein the image medium is an animation image, and the target image is a face of a characteristic person in the animation image.