KR20220062461A - Device and method for generating animation based on motion of object - Google Patents

Abstract

The present invention relates to a device for generating an object motion-based animation to enable general users to easily generate object motion-based animation files. According to the present invention, the device comprises: an image receiving unit receiving an image of an object photographed through a camera; a recognition unit recognizing a motion of the object from the received image and recognizing joint position information from the recognized motion of the object; an extraction unit extracting a value of at least one joint on the basis of the position information of the recognized joint; a correction unit correcting the value of the at least one extracted joint; and an animation generation unit generating a skeleton-based data structure on the basis of the at least one corrected joint value and generating an object motion-based animation on the basis of the at least one corrected joint value and the generated skeleton-based data structure. The correction unit corrects the value of the joint corresponding to an ideal point among the values of the at least one joint on the basis of the degree of tilt displacement between the current frame and the previous frame of the image so that the joint value of the current frame and the joint value of the previous frame are the same.

Description

DEVICE AND METHOD FOR GENERATING ANIMATION BASED ON MOTION OF OBJECT

The present invention relates to an apparatus and method for generating an animation based on object motion.

The gesture recognition technology refers to a technology for controlling a computer by recognizing the movement of the whole or part of the user's body, hands, face, and the like. The gesture recognition technology is divided into a contact method and a non-contact method.

The contact method refers to a method in which a device equipped with a gyro sensor, an acceleration sensor, etc. is worn on the user's body, thereby recognizing the user's motion. The contact method has the advantage of recognizing the user's motion relatively accurately, but there is an inconvenience that the user has to wear a separate device.

The non-contact method refers to a method in which a user's motion can be recognized by recognizing a user from an image captured by a camera. The non-contact method is divided into a marker method and a markerless method. In the marker method, after defining the marker properties in advance, the user's motion can be recognized by tracking the marker. The markerless method has an advantage of recognizing a user's motion without a marker, but has a disadvantage of low accuracy.

In relation to such a gesture recognition technology, Korean Patent Registration No. 10-1963694, which is a precedent, discloses a wearable device for gesture recognition control and a gesture recognition control method using the same.

Recently, 3D content based on the user's motion, such as augmented reality content and virtual reality content, as well as controlling the device by recognizing the user's motion has been provided. However, when trying to recognize a user's motion using a camera, like the non-contact method, there is a problem in that the accuracy of the user's motion recognition is low, making it difficult to produce 3D content.

To provide an apparatus and method for recognizing the motion of an object from an image captured by a camera, and recognizing the position information of the joint of the object from the motion of the object, thereby generating an animation based on the position information of the joint of the object do.

An object motion-based animation generating apparatus and method for extracting a joint value based on joint position information of an object and correcting the extracted joint value are provided.

An object is to provide an apparatus and method for generating an object motion-based animation based on a data structure based on a joint value and a skeleton of an object.

However, the technical problems to be achieved by the present embodiment are not limited to the technical problems described above, and other technical problems may exist.

As a means for achieving the above-described technical problem, an embodiment of the present invention is an image receiving unit for receiving an image of an object through a camera, recognizing the motion of the object from the received image, and the recognized object A recognition unit for recognizing the position information of the joint from the operation of, an extraction unit for extracting the value of at least one joint based on the recognized position information of the joint, a correction unit for correcting the extracted value of the at least one joint, and A skeleton-based data structure is generated based on the corrected value of the at least one joint, and the object motion-based animation is generated based on the corrected value of the at least one joint and the generated skeleton-based data structure. and an animation generating unit, wherein the correcting unit compares the joint value corresponding to the outlier to the joint value of the current frame based on the degree of inclination displacement between the current frame and the previous frame of the image among the values of the at least one joint. An animation generating apparatus for compensating so that the joint values of the frames are the same may be provided.

The above-described problem solving means are merely exemplary, and should not be construed as limiting the present invention. In addition to the exemplary embodiments described above, there may be additional embodiments described in the drawings and detailed description.

According to any one of the above-described problem solving means of the present invention, the motion of the object is recognized from the image captured by the low-cost RGB-D camera, and the joint value of the extracted object is naturally calculated based on the recognized motion of the object. It is possible to provide an object motion-based animation apparatus and method for correction.

It is possible to provide an animation apparatus and method that allows non-expert users to easily generate an animation file based on object motion by correcting the joint value of an object.

1 is a block diagram of an animation generating apparatus according to an embodiment of the present invention.
2A and 2B are exemplary views showing before and after the joint value is corrected based on the joint range of motion according to an embodiment of the present invention.
3 is an exemplary diagram illustrating joint values corrected using a Kalman filter according to an embodiment of the present invention.
4 is an exemplary diagram illustrating joint values in which joint values are corrected based on a degree of inclination displacement and a Kalman filter according to an embodiment of the present invention.
5 is an exemplary diagram illustrating an animation generated based on a skeleton-based data structure according to an embodiment of the present invention.
6 is a flowchart of a method for generating an animation based on an object motion in the animation generating apparatus according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art can easily implement them. However, the present invention may be embodied in several different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part is "connected" with another part, this includes not only the case of being "directly connected" but also the case of being "electrically connected" with another element interposed therebetween. . Also, when a part "includes" a component, it means that other components may be further included, rather than excluding other components, unless otherwise stated, and one or more other features However, it is to be understood that the existence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded in advance.

In this specification, a "part" includes a unit realized by hardware, a unit realized by software, and a unit realized using both. In addition, one unit may be implemented using two or more hardware, and two or more units may be implemented by one hardware.

Some of the operations or functions described as being performed by the terminal or device in this specification may be instead performed by a server connected to the terminal or device. Similarly, some of the operations or functions described as being performed by the server may also be performed in a terminal or device connected to the server.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram of an animation generating apparatus according to an embodiment of the present invention. Referring to FIG. 1 , the animation generating apparatus 100 may include an image receiving unit 110 , a recognizing unit 120 , an extracting unit 130 , a correcting unit 140 , and an animation generating unit 150 .

The image receiving unit 110 may receive an image obtained by photographing an object through a camera. The camera may be, for example, an RGB-D (Depth) camera. The RGB-D camera has the advantage of providing RGB color image information and depth information. By using such an RGB-D camera, it is possible to obtain a three-dimensional model of an object.

The recognition unit 120 may recognize the motion of the object from the received image, and recognize the position information of the joint from the recognized motion of the object. For example, the recognizer 120 may recognize various motions of the object, such as waving hands, spreading arms, crossing arms, sitting, walking, etc. from the image. Thereafter, the recognition unit 120 may recognize the position information of joints such as the face, shoulder, wrist, forearm, waist, thigh, calf, and ankle of the object based on the recognized motion of the object.

The extractor 130 may extract a value of at least one joint based on the recognized joint position information. Here, the value of the joint may be extracted based on the position coordinates and angle of the joint.

The correction unit 140 may correct the extracted values of at least one joint. To this end, the correction unit 140 may use a previously measured joint motion range, a Kalman filter, an inclination displacement algorithm, etc. alone or a combination thereof to correct the joint value, but is not limited thereto.

The compensator 140 may correct a value of at least one joint based on a range of motion (ROM). Here, the joint range of motion refers to the range of motion when the joints create motion. The joint range of motion is largely divided into a joint range and a muscle range. The joint range is a goniometer, for each joint part, Flexion, Extension, Abduction, Addition, Rotation, Circumduction. ), supination, pronation, inversion, and abduction (Eversion) are measured and recorded as angles. The muscle range is the maximum functional amplitude of the muscle. It means the distance that can be increased and shortened again.

For example, the compensator 140 sets a constraint condition for the range of motion of the joint based on a range of motion (ROM, Range Of Motion) measured previously from the object, and at least one extracted based on the set constraint condition. can correct the value of the joint. The joint values corrected based on the joint range of motion will be described in detail with reference to FIGS. 2A and 2B .

2A and 2B are exemplary views showing before and after the joint value is corrected based on the joint motion range according to an embodiment of the present invention.

2A is an exemplary view showing the joint value before correction according to an embodiment of the present invention. Referring to FIG. 2A , it can be seen that the joint values extracted through the extraction unit 130 show a continuous line shape, and then show a discontinuous shape at a specific point 200 .

2B is an exemplary view showing the joint values after correction according to an embodiment of the present invention. Referring to Figure 2b, the value of the joint is corrected based on the previously measured range of motion (ROM, Range Of Motion), so that the specific point 210, which had a discontinuous shape, is corrected to show a continuous shape again. can be checked

That is, the correcting unit 140 sets a restriction on the range of motion of the joint, thereby correcting the value of the joint that exceeds the movable range of the joint, thereby providing the effect of naturally correcting the inaccurate posture and motion of the object. there is.

Returning to FIG. 1 again, the correction unit 140 may correct the value of at least one joint by using a Kalman filter. Here, the Kalman filter refers to a recursive signal processing method of estimating an original signal generated by the observed data value based on the observed data value including noise.

The corrector 140, for example, applies a Kalman filter to the extracted values of at least one joint to derive an estimate, and uses the derived estimate to determine the value of at least one joint. can be corrected.

Through this, the compensator 140 derives an estimate closest to the original signal by performing linear transformation and linear combining it based on the currently known joint value (observed data value), and extracting it using the derived estimate. It is possible to correct the value of the joint. The joint values corrected using the Kalman filter will be described in detail with reference to FIG. 3 .

3 is an exemplary diagram illustrating joint values corrected using a Kalman filter according to an embodiment of the present invention. Referring to FIG. 3 , the left diagram 300 is a diagram illustrating the joint values extracted by the extraction unit 130 . Here, the extracted joint value may be original data. The right figure 310 is a diagram showing the joint values corrected by applying the Kalman filter to the extracted joint values.

Comparing the before and after correction by applying the Kalman filter, the specific part 301 of the left figure 300 shows a discontinuous shape and the interval between the values of each joint is wide, but correction is performed through the Kalman filter As a result, it can be confirmed that a lot has been improved through the specific part 311 of the right drawing 310 .

Returning to FIG. 1 again, the correction unit 140 may correct the value of at least one joint based on a gradient correction. If you want to express the joint position value (x∈T, R) for any one 'joint j' as a graph on the discrete time (t) axis, it will ideally have a continuous shape. However, in reality, it has a position value x in the form of an outlier at an arbitrary time t due to a position recognition error or the like.

In this case, the displacement of the slope between consecutive inliers in the graph may have a small value. This can be explained through Equation (1).

)(

)

Referring to Equation 1, the displacements of AB, BC, and CD, which are slopes between points A, B, C, and D, which are inliers, may have small values.

Alternatively, the displacement of the slope with the discontinuous outliers in the graph may have a high value. This can be explained through Equation (2).

)(

)

Referring to Equation 2, displacements of EF, FG, and GH, which are slopes between the inlier-outlier points E, F, G, and H, may have large values.

However, when the displacement between the inlier and the outlier has a large value, the motion of the object becomes unnatural.

To improve this, the correction unit 140 may correct the value of the joint corresponding to the outlier by using Equation (3).

)(here,

)

Referring to Equation 3, when the extracted values of at least one joint are expressed in a first shape of a continuous shape and a second shape of a discontinuous shape through a graph, the corrector 140 may The degree of inclination displacement of the value of at least one joint extracted for the previous frame and the subsequent frame is derived, and when the derived degree of inclination displacement is equal to or greater than a threshold value, the value of the joint corresponding to the second shape may be corrected. The value of the joint corrected based on the degree of inclination displacement will be described in detail with reference to FIG. 4 .

4 is an exemplary view showing before and after the joint value is corrected based on the degree of inclination displacement and the Kalman filter according to an embodiment of the present invention. Referring to FIGS. 3 and 4 , the left drawing 400 is a diagram showing the joint values corrected based on the degree of inclination displacement from the original data corresponding to the left drawing 300 of FIG. 3 .

First, comparing the left drawing 400 of FIG. 4 and the left drawing 300 of FIG. 3 , in the case of a specific part 401 of the left drawing 400 of FIG. 4 , it is included in the left drawing 300 of FIG. 3 . It can be seen that the corrected outliers are corrected to be included in the inlier.

The right figure 410 is a figure showing the joint values corrected by applying the Kalman filter to the joint values corrected based on the degree of inclination displacement in the left figure 400 . When the specific part 401 of the left figure 400 and the specific part 411 of the right figure 410 are compared, the joint value corrected in the left figure 400 is further corrected through the Kalman filter, so that the joint value It can be seen that they have been calibrated so that they are connected smoothly.

The animation generator 150 may generate an animation based on an object motion. The process of generating the animation will be described in detail with reference to FIG. 5 .

5 is an exemplary diagram illustrating an animation generated based on a skeleton-based data structure according to an embodiment of the present invention. Referring to FIG. 5 , the animation generator 150 generates a skeleton-based data structure based on the corrected values of at least one joint, and stores the corrected values of the at least one joint and the generated skeleton-based data structure. Based on this, you can create an animation based on the motion of an object.

The animation generator 150 may generate an animation based on object motion by applying the value of at least one joint corrected in real time from the image to the generated skeleton-based data structure.

Through this process, by correcting the joint values of the object in various ways, and creating a skeleton-based data structure using the corrected joint values, post-processing of the object's motion to create an animation with natural motion. can do.

6 is a flowchart of a method for generating an animation based on an object motion in the animation generating apparatus according to an embodiment of the present invention. The method for generating an animation based on an object motion shown in FIG. 6 includes steps that are time-series processed by the animation generating apparatus 100 according to the embodiment shown in FIGS. 1 to 5 . Therefore, even if omitted below, it is also applied to the method of generating an animation based on an object motion in the animation generating apparatus 100 according to the embodiment shown in FIGS. 1 to 5 .

In step S610, the animation generating apparatus 100 may receive an image obtained by photographing an object through a camera.

In operation S620, the animation generating apparatus 100 may recognize the motion of the object from the received image, and recognize the position information of the joint from the recognized motion of the object.

In operation S630, the animation generating apparatus 100 may extract a value of at least one joint based on the recognized joint position information.

In operation S640, the animation generating apparatus 100 may correct the extracted values of at least one joint.

In step S650, the animation generating apparatus 100 generates a skeleton-based data structure based on the corrected value of at least one joint, and an object based on the corrected value of the at least one joint and the generated skeleton-based data structure. You can create motion-based animations.

In the above description, steps S610 to S650 may be further divided into additional steps or combined into fewer steps, according to an embodiment of the present invention. In addition, some steps may be omitted as needed, and the order between the steps may be switched.

The method for generating an object motion-based animation in the animation generating apparatus described with reference to FIGS. 1 to 6 is also implemented in the form of a computer program stored in a medium executed by a computer or a recording medium including instructions executable by the computer. can be Also, the method of generating an animation based on an object motion in the animation generating apparatus described with reference to FIGS. 1 to 6 may be implemented in the form of a computer program stored in a medium executed by a computer.

Computer-readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. Also, computer-readable media may include computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

The above description of the present invention is for illustration, and those of ordinary skill in the art to which the present invention pertains can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a dispersed form, and likewise components described as distributed may be implemented in a combined form.

The scope of the present invention is indicated by the following claims rather than the above detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be interpreted as being included in the scope of the present invention. do.

100: animation generating device
110: video receiver
120: recognition unit
130: extraction unit
140: correction unit
150: animation generator

Claims (5)

An apparatus for generating an object motion-based animation, comprising:
an image receiving unit for receiving an image obtained by photographing an object through a camera;
a recognition unit for recognizing the motion of the object from the received image, and recognizing the position information of the joint from the recognized motion of the object;
an extraction unit for extracting a value of at least one joint based on the recognized joint position information;
a correction unit for correcting the extracted at least one joint value; and
A skeleton-based data structure is generated based on the corrected value of the at least one joint, and the object motion-based animation is generated based on the corrected value of the at least one joint and the generated skeleton-based data structure. including an animation generating unit that
The correction unit calculates the joint value corresponding to the outlier based on the degree of tilt displacement between the current frame and the previous frame of the image among the values of the at least one joint according to the following equation between the joint value of the current frame and the previous frame An animation generating device that corrects the values of the joints to be the same.

here,

is the current frame,

is the previous frame,

denotes the value of the joint corresponding to the above-mentioned outlier.
The method of claim 1,
The compensator sets a constraint condition for the range of motion of the joint based on a joint motion range (ROM, Range Of Motion) measured in advance from the object, and based on the set constraint condition, An animation generating device that corrects a value.
The method of claim 1,
The correction unit derives an estimate by applying a Kalman filter to the extracted at least one joint value, and corrects the extracted value of the at least one joint by using the derived estimate, animation generator.
The method of claim 1,
When the value of the extracted at least one joint is expressed in a first shape of a continuous shape and a second shape of a discontinuous shape through a graph,
The correction unit derives the degree of inclination displacement of the value of the at least one joint extracted from the current frame of the image with respect to the previous frame and the subsequent frame,
When the derived degree of inclination displacement is equal to or greater than a threshold value, the value of the joint corresponding to the second shape is corrected.
The method of claim 1,
The animation generating unit generates the animation based on the object motion by applying the value of at least one joint corrected in real time from the image to the generated skeleton-based data structure.

Images