KR20000074633A - Real-time virtual character system - Google Patents

Abstract

PURPOSE: A real-time imaginary character system is to smoothly interlock a motion and a facial expression of a performer with an imaginary character and a background graphic set, and control the motion and an expression of the imaginary character in real time using a facial image of the performer. CONSTITUTION: A real-time imaginary character system comprises a body motion controller(1) for controlling a joint motion of an imaginary character according to the joint motion of a performer, a hand motion controller(2) for extracting bending information of a finger joint of the performer to control a hand motion of the imaginary character, a facial expression controller(3) for analyzing a facial expression of the performer input from a camera to produce the facial expression of the imaginary character, a camera system(4) for generating motion data of the camera and then transmitting the data to a graphic workstation, a camera motion interlocking portion(5) for interlocking a motion of the imaginary character and a background graphic set with the motion of the camera, and an image synthesizing portion(6) for synthesizing the imaginary character and the background graphic set with an image output from the camera.

Description

Real-time virtual character system

The present invention relates to a real-time virtual character system, and more particularly, to control the motion of the virtual character in real time using motion information read through the three-dimensional sensor attached to the body joint of the performer, using a camera for capturing facial expressions Reads the facial image of the performer's face and uses it to control the facial expression of the virtual character in real time, as well as linking the virtual character and the set of background graphics to the camera movement, and using a special screen called chroma key panel. It was invented so that it could be naturally synthesized with the presenter (mostly blue).

The conventional virtual character system generates the virtual character's motion and facial expression by pre-production, so it is very difficult for real-time applications where the virtual character appears on TV live broadcasts to talk or dance with other performers and only operate according to a predetermined scenario. Only passive virtual characters can be produced.

The pre-production method can invest a lot of production time to produce a virtual character very similar to a real human being, but it requires a lot of animation time for each frame, making it difficult to apply to TV live broadcasting.

In addition, the conventional virtual character system does not have the function of linking the virtual character and the background graphic set with the camera movement (pan, tilt, zoom, position movement), so that the virtual character and the background graphic set are naturally connected to the presenter's movement according to the camera movement. Cannot match.

That is, when photographing the presenter using the camera, the motion data of the camera cannot be applied to the motion of the virtual character and the background graphic set.

However, in order to solve the above problems, foreign countries have recently developed a real-time virtual character system, and a product which integrates all or partly functions of virtual character gesture control, hand gesture control, facial expression control, and camera interworking functions is being developed. It is announced.

Accordingly, the present invention has been made to solve the above-mentioned conventional problem, and controls the movement of the virtual character in real time by capturing and substituting the motion and expression of the actual actor, and interlocking the virtual character and the background graphic set with the camera movement. The purpose is to provide a real-time virtual character system that can be synthesized naturally with the host.

The object of the present invention, the motion control unit for naturally moving the motion of the virtual character in accordance with the body movement of the performer; A hand motion control unit which moves a hand of the virtual character in accordance with the hand movement of the motion actor; A facial expression control unit for expressing a facial expression of the virtual character by using the facial image of the facial expression performer input using the facial expression capture camera; A camera system for generating camera motion data and transmitting it to a graphics workstation; A camera movement linker for linking the virtual character and the background graphic set with the camera movement; In the real-time virtual character system consisting of a video synthesizer that can naturally synthesize the virtual character and the background graphic set with the presenter located in front of the chroma key panel,

The gesture control unit may include: a gesture capture unit which detects joint position and rotation information by using a three-dimensional sensor of the movement capture device attached to the joint of the performer; Hand motion capture unit which is composed of a body motion insertion unit that moves the joints of the virtual character naturally by substituting the captured joint motion data, and the hand motion control unit detects the bending information of the finger node from the glove-shaped hand motion capture device mounted on the motion actor's hand. Wow; A hand gesture substitution unit for moving the finger joint of the virtual character by substituting captured finger information, the facial expression control unit comprising: an expression capturing camera for inputting an actor's facial expression; An expression capture unit for obtaining displacements of feature points implicitly including the shape of each face part from the input face image; It consists of facial expression substitution that transforms the face control point of the virtual character from the facial feature points, and the image synthesizing unit delays the camera output as much as the graphics processing speed of the graphics workstation when composing the virtual character and the background graphic set with the camera output. Wow; This can be achieved by removing the blue portion of the camera output image and then configuring the chromakey portion to create a natural composite screen by filling the output portion of the graphics workstation in place.

Therefore, by capturing the body movements, hand movements and facial expressions of the actors in real time and connecting them with the body movements and facial expressions of the virtual characters, the virtual characters appear live and can interact with the performers. Naturally, virtual characters can be synthesized with performers.

1 is an overall block diagram of an apparatus of the present invention

2 is a block diagram illustrating a gesture control unit and a hand control unit in FIG. 1;

FIG. 3 is a block diagram illustrating a face expression control unit and an image synthesis unit in FIG. 1.

Figure 4 is an exemplary view showing a screen showing the result of the substitution of motion for two virtual characters having different joint sizes by the method of the present invention

5 is a flowchart illustrating five steps of capturing a facial expression by the method of the present invention.

6 is an exemplary view showing a screen showing a result of substitution of facial expression information by the method of the present invention.

Explanation of symbols for main parts in the drawings

1: gesture controller 2: gesture controller

3: facial expression control unit 4: camera system

5: camera movement linkage 6: image synthesis section

11: gesture capture 12: gesture assignment

21: hand gesture capture portion 22: hand gesture assignment

31: camera for capturing facial expressions 32: expression capturing unit

33: Expression assignment part 61: Frame delay part

62: chroma key portion 121: center position generating portion

122: regional inverse kinematics 123: interaction possibility extraction unit

124: motion mixing unit 321: facial expression extracting unit

322: head tracker

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram showing the overall block diagram of the apparatus of the present invention, FIG. 2 is a gesture control unit and a hand control unit in FIG. 1, FIG. 3 is a face expression control unit and an image synthesizer in FIG. 1, and FIG. 4 is a joint. FIG. 5 illustrates an example of operation substitution for two virtual characters having different sizes, FIG. 5 illustrates an example diagram illustrating five steps of capturing facial expressions, and FIG. 6 illustrates an example diagram of results of substituting facial expression information. It is shown.

According to this, the motion control unit (1) for controlling the joint movement of the virtual character in accordance with the movement of the actor joint; A hand motion control unit 2 for controlling the movement of the hand by extracting bending information of the finger node of the performer; A facial expression control unit 3 for generating a virtual character expression by analyzing a facial expression of an expression performer input from an expression capturing camera; A camera system 4 for generating and transferring camera motion data to a graphics workstation; A camera motion linking unit 5 for linking the motion of the virtual character and the background set to the camera movement; In the real-time virtual character system composed of a virtual character and an image synthesizing unit (6) for combining a set of background graphics with a camera image,

The gesture control unit 1 includes a gesture capture unit 11 for finding out the position and rotation information of the joint using a three-dimensional sensor of the movement capture device attached to the joint of the performer; Substituting the captured joint motion data is composed of the motion substitution part 12 which naturally moves the joint of the virtual character.

When the size or shape of the virtual character is different from the actual performer, the motion substitution part 12 may distort the position of the hand and leg, which are the distal positions of the virtual character, so that the hands and legs of the virtual character are distal to the actual performer. A center position generating unit 121 for determining a position of the pelvis, which is the center of the virtual character, to be placed at the same position as that of the virtual character; A local inverse kinematics unit 122 that determines joint angles of the shoulders, elbows, hips, and knees so that the ends of the virtual characters reach a designated position; An interaction possibility extraction unit 123 for determining mutual possibilities between the terminal of the virtual character and the surrounding environment; By combining the results of the central position generating unit 121, the regional inverse kinematics unit 122 and the interaction possibility extracting unit 123, the smaller the mutual probability, the greater the weight of the joint angle-based motion and the closer the distance is. It is characterized by consisting of a motion mixing unit 124 for generating a movement of the body center and joint of the virtual character by assigning a large weight to the motion based on the distal position.

The hand motion control unit 2 includes a hand motion capture unit 21 for detecting bending information of a finger node from a glove-shaped hand motion capture device equipped with a sensor; Characterized in that it consists of a hand motion substitution unit 22 for moving the finger joint of the virtual character by substituting the captured finger information.

The facial expression control unit 3 includes an expression capturing camera 31 for inputting an actor's facial expression; An expression capture unit 32 which obtains displacements of feature points in real time including the shape of each face part from an input face image in real time; It consists of the expression substitution part 33 which deforms the face control point of a virtual character from facial feature points.

In this case, the expression capture unit 32 includes an expression extractor 321 for finding the positions of the feature points formed by the mouth, eyes, eyebrows, and jaws, which are the feature points of the face, using the input face image; As the actual actor's head moves, the shape of the face shown in the image changes, so the head tracking unit 322 tracks the movement of the head.

The image synthesizing unit 6 includes a frame delay unit 61 for delaying the camera output image by the graphics processing speed of the graphic workstation when synthesizing the virtual character and the background graphic set with the camera output image; After removing the blue portion of the camera output, it fills in the output portion of the graphics workstation in place and constitutes a chroma key portion 62 that produces a natural composite screen.

Referring to the effect of the device of the present invention configured as described in detail as follows.

First, the motion controller 1 of the virtual character generates motion motion by substituting motion information obtained from motion capture equipment into the virtual character model.

The motion capture device uses a magnetic field to send the rotation angle and position for each of the 14 sensors to the graphics workstation up to 144 times per second via Ethernet.

Simply inserting information about the direction of the joint provided by the motion capture device into the virtual character may maintain the actor's natural motion, but the distal position of the virtual character may be distorted.

That is, as the position of the hand and the foot changes, the virtual character's foot, which should be on the floor, slides or penetrates the floor, and when the object is picked up by hand, the hand does not touch the object or enters into the object frequently.

This is especially true if the size or shape of the virtual character is different from the actual actor.

On the other hand, if the information about the position of the joint provided by the motion capturing device, that is, each end position of the actual performer is applied to the mapping character as it is, it is possible to generate the movement with less distortion of the end position, but cannot maintain the natural movement.

Therefore, the gesture substitution part 12 determines whether each end of the actual actor interacts with the predefined environment, and if the possibility of interaction is high, the weight of the motion that substitutes the position of the end is increased, otherwise the actual actor's Mixing the substituted motion of each joint angle to a greater specific gravity can produce relatively natural motion in both cases.

To this end, the motion substitution unit 12 includes a center position generating unit 121 for adjusting the center position of the virtual character; A local inverse kinematics unit 122 for calculating joint angles using inverse kinematics by substituting the position of the terminal; An interaction possibility extraction unit 123 for determining whether each terminal interacts with the environment; It consists of a motion mixing unit 124 for mixing the results to generate a motion.

The virtual character center position adjustment of the center position generating unit 121 is performed to bring the end of the virtual character to the same position as the end of the actual actor.

In determining the virtual position of the virtual character, that is, the position of the pelvis center, the first consideration is to keep the position of the pelvic center of the actual actor as much as possible, and the range that each end can reach to obtain the virtual character's pelvic center position. You should know

When the position of the shoulder or hip in the arm or leg is determined, the range of reach of the hands and feet is shown in the form of a sphere, each of which is centered around the shoulder or hip, or pivot, and when the arms and legs are fully extended. Means a sphere whose radius is, which is called the range sphere at the end.

Given the position of each end, placing the pelvic center within the sphere of the sphere allows the end to reach the specified position, so in order to have all the ends at the specified place, the pelvic center is the intersection of the four spheres obtained by each end. Should be located.

If there is no intersection of four spheres, the range spheres obtained at relatively insignificant ends are removed to satisfy the remaining end conditions.

If the position of the terminal must be maintained, the terminal is likely to interact with an external object, so the range sphere by the terminal that is less likely to interact is removed, and the range sphere is removed until the intersection appears. Finally, the range of presence of the pelvis can be obtained.

The joint angle calculation of the regional inverse kinematics 122 calculates the joint angle with respect to the distal position using the inverse kinematics.

Applying inverse kinematics to the human body takes a relatively long time to calculate, the present invention limits the scope of application to the arms and legs and locally calculates the joint angle to generate motion based on the position of the distal end more quickly.

Since there are many kinds of joint angles obtained by inverse kinematics, the joint angles and positions of actual performers are referred to to generate motions similar to those of actual performers.

Whether or not the interaction possibility extraction unit 123 interacts calculates the distance from all environments defined for each end and calculates the possibility of interaction using the same.

In order to find out whether there is an interaction in the actual actor's motion, the location of the environment to be interacted with must first be specified, and since the objects of the environment can be represented by a relatively simple geometric structure, The distance can be calculated.

Once the distance between the end of each object and the end is obtained, the possibility of interaction can be calculated, and the largest value of the possibilities of interaction becomes the possibility of the end interacting.

The mixed motion generation of the motion mixing unit 124 gives priority to the motion generated around the actual actor's joint angle when the possibility of interaction with the external environment is small, and when the motion mixing part 124 has a large value, By mixing with specific gravity, proper motion mixing results in a relatively natural motion in both cases.

Before mixing the joint angles, the joint angles of the pelvis and the trunk, neck, head, etc. are substituted for the actual actor's joint angles, and each joint angle of the remaining limbs mixes the two movements.

4 is an exemplary view showing the result of substituting the ball throwing motion to two virtual characters having significantly different joint ratios, as well as removing the distorted distal position caused by the joint difference and adjusting the virtual character center position. The joint angle is properly changed to generate a free motion.

On the other hand, the facial expression control unit 3 analyzes the facial expression of the performer input by using a facial expression capture camera to find the displacement of the feature points including the shape of the face part, and the face of the virtual character from the modified face feature points Create control expressions by modifying control points.

To this end, the expression capture unit 32 obtains displacements of the feature points representing the shape of each part of the face, and tracks the head movement to obtain the facial expression accurately because the face shape shown in the image is changed when the actor's head moves.

In order to obtain the displacement of the feature points, the expression extractor 321 converts the input image into an image that can easily obtain an expression, obtains a curve representing the shape of each face from the converted image, and then extracts the feature points extracted from the curves. Get displacement.

Even if the environment of the actor or lighting changes, the color information of the image is modified to easily find each part of the face from the image to obtain a gray-scale image having a large difference in brightness between each part and the surrounding skin. The shape of each part of the face is identified from the image.

Then, we find a simple parametric curve representing the outline of each face and extract the feature points.

Expression targets include mouth, eyes, eyebrows, and chin, mouth lip and skin borders, eye shape of eyelids formed by eyelashes, eyebrows curves under the eyebrows, and chin curves around the outline of the chin. By finding the shape of each part of the face can be obtained.

If the curve representing the outline of the image has energy, and the energy decreases as it approaches the boundary of each part of the face, a curve representing the shape of the outline may be obtained using energy minimization.

However, if you use only the energy of the rate of change of brightness, you can find something similar to the outline you are looking for, but it is completely different from the outline you are looking for, for example, when you look for the outer line of the lips, you can find the inner line of the lips.

Therefore, in order to position the curve at the correct part, the inner and outer subtraction curves of the curve generated by using the brightness information of each part in addition to the energy due to the brightness change rate are used together.

The head tracking unit 322 sets the points which are hardly influenced by the change of the facial expression as reference points, and tracks the movement by obtaining each three-dimensional coordinates using these correlations.

In order to obtain a stable head tracking result, three of the reference points are selected and the process of obtaining the tracking result is repeated, and a stable value for each tracking result is obtained.

The three reference points were defined as the end points of the eyes and the nostrils, and the tracking using these three points does not uniquely determine the point that satisfies the positional relationship. Using the property of not happening together, we select the solution that has the least change and get the correct three-dimensional coordinates.

5 illustrates a process of capturing the actual actor's expression in the facial expression capture unit 32.

On the other hand, in the expression substitution unit 33 controls the facial expression of the virtual character using the displacement of the feature points captured on the actor's face, paying attention to the fact that the displacements of the feature points and the control points are linear with each other, to quickly and flexibly match them. Use the corresponding function.

The linear correspondence function can be used to correct for the difference in magnitude between the feature point and the control point, where the domain is the displacement of the feature points and the range is the displacement of the control points. The linear correspondence function is expressed as a matrix having the weights of the feature points, and when the captured feature point information is input, the control point information may be calculated by multiplying the matrix representing the linear correspondence function.

Feature Point Information Is a linear correspondence function Control point information multiplied by = A x Can be calculated as

This correspondence can be used even when the number of feature points and control points is different because the domain and the range are not limited. on Adding an axial displacement can also artificially create three-dimensional movement of the control point.

At this point, a training process is required to sample some important facial expressions. This training process selects the displacement of the control point so that the virtual character best represents the facial expressions of the characteristic facial expressions of the actual actor.

Since the expressions of the actual performer and the expression of the virtual character are determined only by the corresponding function defined by the matrix, the training function is appropriately configured to exaggerate or alleviate the expression.

Deformation of facial expression using displacement of control point is using curve transformation method. Given that most eyes, lips and chin on the face are composed of curves, it can be seen that curve is a very suitable means to control the deformation of face. have.

When the facial expression control point is applied to a parametric curve, direct control is not possible because the control point is not placed on the curve by the parametric curve characteristic.

Therefore, the B-spline curve interpolation method was used to make all the expression control points pass through the curve, and the curve was directly controlled by the control point.

6 shows the result of substituting the captured actual actor's expression into the virtual character, and it can be seen that the expression of the actual actor's expression matches that of the virtual character.

As described above, according to the present invention, the motion information and the facial expression of the virtual character are not only controlled in real time by using the motion information of the actor and the facial image of the facial expression actor, but the virtual character and the background graphic set are controlled by the camera movement. By developing a real-time virtual character system that naturally combines with a host located in front of a chroma key panel while interlocking, unlike a system that creates a virtual character using a pre-production technique, a virtual character appears on TV live and communicates with other performers. It is possible to apply the new virtual character model easily by using the real-time application that can perform the dance or dance together, and the body substitution technique that overcomes the difference in body type between the actor and the virtual character.

Claims (3)

A gesture controller 1 for controlling joint movement of the virtual character according to the movement of the actor joint; A hand motion control unit 2 for controlling the movement of the hand by extracting bending information of the finger node of the performer; A facial expression control unit 3 for analyzing a facial expression of an expression performer input from a camera to generate a virtual character facial expression; A camera system 4 for generating and transferring camera motion data to a graphics workstation; A camera motion linking unit 5 for linking the motion of the virtual character and the background graphic set to the camera motion; In the real-time virtual character system composed of a virtual character and an image synthesizing unit (6) for combining a set of background graphics with the camera output image, The gesture control unit 1 includes a gesture capture unit 11 for finding out the position and rotation information of the joint using a three-dimensional sensor of the movement capture device attached to the joint of the performer; Consists of the captured joint motion data to the motion substitution part 12 to naturally move the joints of the virtual character, the hand motion control unit 2 obtains the bending information of the fingertips from the glove-shaped hand motion capture device with a sensor A hand gesture catcher 21 for emitting; And a hand gesture substitution unit 22 for moving the finger joint of the virtual character by substituting captured finger information, and the facial expression control unit 3 includes a facial expression capture camera 31 for inputting an actor's facial expression; An expression capture unit 32 for obtaining displacements of feature points implicitly including the shape of each face part from the input face image; A real-time virtual character system comprising a facial expression substitution unit (33) for deforming a face control point of a virtual character from facial feature points. The method according to claim 1, wherein the motion substitution part 12 is the position of the virtual character's hand and leg is distorted when the size or shape of the virtual character is different from the actual actor, so the hands and legs of the virtual character A center position generating unit 121 for determining a position of the pelvis that is the center of the virtual character so as to be located at the same position as the end of the actual performer; A local inverse kinematics unit 122 that determines joint angles of the shoulders, elbows, hips, and knees so that the end of the virtual character can reach a designated position; An interaction possibility extraction unit 123 for determining mutual possibilities between the terminal of the virtual character and the surrounding environment; By combining the results of the central position generating unit 121, the regional inverse kinematics unit 122 and the interaction possibility extracting unit 123, the smaller the mutual probability, the greater the weight of the joint angle-based motion and the closer the distance is. Real-time virtual character system comprising a motion mixing unit 124 for generating a movement of the body and the joint of the virtual character by assigning a large weight to the motion based on the end position. The method of claim 1, wherein the expression capture unit 32 comprises: an expression extracting unit 321 for finding the position of the feature points formed by the mouth, eyes, eyebrows and chin which are the feature points of the face using the input face image; Real-time virtual character system, characterized in that consisting of a head tracking unit 322 for tracking the movement of the head because the face shape shown in the image changes when the head of the actual performer moves.