KR20100025048A - Image analysis apparatus and method for motion capture - Google Patents

Abstract

PURPOSE: An image analysis apparatus for motion capture and a method thereof are provided to easily and accurately track a human body area by a smaller amount of calculation. CONSTITUTION: A human body area detector(210) differentiates a background area of an image. The human body area detector extracts a skin color. The human body area detector detects a plurality of human body areas. A coordinate correcting unit(220) corrects the coordinate of the detected human body area using an image correction parameter. A human body area tracking unit(240) applies the Kalman filter to the coordinate of the corrected human body area. The human body area tracking unit tracks motion of the human body area.

Description

Image analysis apparatus and method for motion capture

The present invention relates to an image analysis method and apparatus, and more particularly, to an image analysis method that performs motion capture based on an image, so that each human region can be easily and accurately tracked with a small amount of calculation in real time. The present invention relates to an image analysis method and apparatus for motion capture capable of capturing motion.

With the development of computer vision and computer graphics technology today, the natural motion and realistic facial expressions of humans are reproduced in 3D virtual characters through motion capture technology. This motion capture technology can be used to examine the special effects of movies, the interface of games through HCI (Human Communication Interface) system, the automatic diagnosis system of orthopedic patients, the system for improving athletes' achievements, and the behavior of crowds in public places. It is being utilized and researched in various fields such as surveillance system.

As such, the conventional motion capture technology for restoring human motion to three-dimensional digital data includes a mechanical method of attaching a mechanical device to each joint of the human body, a magnetic method of attaching a sensor to generate a magnetic field, and measuring the change thereof. There is an optical method for estimating three-dimensional coordinates from multiple images obtained by attaching a marker to a specific part of the human body. However, such a method using a mechanism, a sensor, a marker, and the like have various disadvantages such as expensive hardware equipment, a limitation of user's movement, and a user's complexity. Therefore, research on image-based motion capture that is independent from hardware devices and does not use machinery, sensors, and markers for free motion capture of users is being conducted.

For image-based motion capture, images are acquired from an image input device such as one camera, and the training data are used to limit the feature space through temporal and spatial constraints in the video sequence, and to project the 3D human model onto the image. There are many methods such as the mechanism of detecting each part of the human body, and it is impossible or impossible to estimate human posture in real time. In addition, there is a problem that it is difficult to estimate the posture for various movements due to the movement constraints of the operator.

On the other hand, a method of estimating the posture of an operator from an image acquired from multiple angles using a plurality of image input apparatuses generally has difficulty in acquiring an accurate silhouette or outline and is sensitive to noise. In addition, as a plurality of image input devices are used, a lot of hardware is required and motions must be synchronized for each image from each image input device.

Such an image-based motion capture system extracts features such as detecting an object's silhouette or outline, pixel-based feature points, a human face, hands, and feet. However, in order to extract an accurate silhouette or outline of an object from an image, it takes a lot of time to perform calculations for environmental constraints and optimization. In addition, when many pixel-based feature points are used, it is difficult to match the corresponding points and motion capture cannot be performed in real time as a large amount of computation is required in order to find an accurate matching point.

An object of the present invention is to provide an image analysis method and apparatus for motion capture that can easily and accurately track each human region in a method of performing motion capture based on an image.

It is another object of the present invention to provide an image analysis method and apparatus capable of performing motion capture in real time by tracking each human region with a smaller amount of calculation.

The image analysis method according to the present invention is an image analysis method for tracking a human body movement using an image received from at least one image input device, and detects a plurality of human regions by extracting a skin color and subtracting a background region of the image. Making; Correcting the coordinates of the detected human body area using image correction parameters; And tracking the movement of the human body region using the corrected coordinates of the human body region.

The human body region may be five regions corresponding to both hands, feet, and face of the human body.

The detecting may include detecting the human body region from each of the two images input from the two image input apparatuses.

The detecting may include down sampling the input image; And subtracting a background area and extracting skin color using at least one of a Hue Saturation Intensity (HSI) and a Red Green Blue (RGB) color space in the down-sampled input image.

After the correcting, the method may further include removing noise included in the human body region by matching the human body region from each of the images based on a preset initial posture.

The tracking may be a step of tracking the movement of the human body region by applying a Kalman filter to the corrected coordinates of the human body region.

Meanwhile, the image analyzing apparatus according to the present invention is an image analyzing apparatus for tracking a human body movement using an image received from at least one image input apparatus. A human body detector for detecting; A coordinate corrector configured to correct detected coordinates of the human body region using image correction parameters; And a human body tracking unit that tracks the movement of the human body region by applying a Kalman filter to the corrected coordinates of the human body region.

The human body area detector detects five human body areas corresponding to both hands, feet, and face of the human body.

The human body detection unit may detect the human body region from each of the two images input from the two image input apparatuses.

The human body detection unit down-samples the input image, subtracts a background area from the down-sampled input image by using at least one of a hues saturation intensity (HSI) and a red green blue (RGB) color space, and colors the skin. The human body region may be extracted by detecting the.

According to the present invention, the image analysis method and apparatus detects a human body area corresponding to the head, both hands and feet by using the skin color of the human body from the input image, and tracks the movement of each human body area in the same way, thereby reducing the amount of computation. In addition, there is an effect that can perform motion capture.

In addition, according to the present invention, the image analysis method and apparatus can perform motion capture in real time as it can be tracked with a smaller amount of calculation by the movement of the human body region.

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings and preferred embodiments. For reference, in the following description, detailed descriptions of well-known functions and configurations that may unnecessarily obscure the subject matter of the present invention are omitted.

1 is a block diagram illustrating an image analyzing apparatus according to the present invention.

Referring to the drawings, the image analyzing apparatus 200 according to the present invention includes a human body detector 210, a coordinate corrector 220, a human body matcher 230, and a human body tracker 240.

The human body detector 210 detects the human body region by receiving an input image from the image input apparatus, and preferably receives the first image and the second image from the two image input apparatuses 100_1 and 100_2.

The human body detector 210 subtracts the background region from each image and extracts skin color. More specifically, the human body detector 210 first extracts the human body region from the first image and the second image input from the first image input apparatus 100_1 and the second image input apparatus 100_2 in real time. The image can be reduced by down sampling the image of the image to 1/2 size. Subsequently, the human body detection unit 210 subtracts the background area from the down-sampled input image by using a Hue Saturation Intensity (HSI) and / or a red green blue (RGB) color space and detects a skin color by detecting a skin color. Extract the area. In this case, the human body detection unit 210 may combine the HSI color space and the RGB color space or use an adaptive algorithm to extract an accurate human area. Preferably, the human body detection unit 210 is insensitive to a change in illumination and performs fast. You can use the normalized RGB color space.

Therefore, the human body detection unit 210 according to the preferred embodiment of the present invention detects the human body candidate area by extracting the skin color, which is a common feature of the human body, and thus detects the human body area using respective features such as the face, hands, and feet. Compared with the conventional Youngsan analysis device, since it has a smaller amount of calculation, the time required for detecting a human body region is advantageous.

The coordinate corrector 220 corrects the coordinates of the human candidate region detected in the first image and the second image by using image correction parameters.

The image correction parameter may include an internal parameter such as a focus of a camera and an external parameter such as a position and a movement of a camera. The parameter may include a first image through Z. Zhang's robust camera correction method using a two-dimensional plane pattern. It can be calculated by tracking the projection matrix P of the input device and the second image input device 100_1, 100_2. This will be described below.

The projection matrix P expresses a relationship in which one point in the three-dimensional space is projected onto the two-dimensional image plane, which is represented by Equation 1 below.

Where K is an internal parameter and R and T are internal parameters.

Assuming that the two-dimensional plane Z of the real world is Z = 0, the homography H between the coordinates in the image and the two-dimensional plane coordinates may be calculated through Equation 2 below.

Here, (x, y) is a two-dimensional plane coordinate, and (X, Y) is a coordinate in the image.

Among the components of the homography (H), the unique parameter is estimated using the properties of the rotation matrix (R) and the properties of the internal parameter (K).

On the other hand, since the image received from the image input apparatuses 100_1 and 100_2 increases in the radial direction, the straight line appears in the form of a curve as the image moves away from the principal point. Therefore, the radial distortion is corrected by modifying the coordinate system of the image using Equation 3.

Here, (u, y) is the coordinate of the ideal image without distortion, (x, y) is the coordinate of the normalized image without distortion, k ₁ , k ₂ is the distortion coefficient.

In addition, even if the radial distortion is corrected through Equation 3, since the center of curvature of the lens surface is not always coplanar, tangential distortion occurs. The tangential distortion is corrected using Equation 4.

Here, p ₁ and p ₂ are tangential distortion coefficients.

The coordinate corrector 220 calculates the coordinates of the image through Equation 5 after correcting the radial and tangential distortions.

는 취득된 영상의 좌표이다.here,

Is the coordinate of the acquired image.

Since the coordinates of the image acquired through the above process are distorted due to the radiation distortion of the lenses included in the image input apparatuses 100_1 and 100_2, the estimated internal and external parameters also need correction. Accordingly, the coordinate corrector 220 may optimize internal and external parameters, radiation distortion coefficients, and corrected coordinates of an image by using a maximum likelihood estimation method. The coordinate corrector 220 may estimate the optimal parameter and coordinates by minimizing the difference between the coordinates of the pattern extracted from the image and the coordinates of the spatial pattern projected onto the image through Equation 6.

는 패턴의 각 좌표들이 추정된 투영행렬에 의해 영상에 투영되는 좌표이다.Where m _ij is the coordinate of the j-th pattern in the i-th image, R _i and t _i are the rotation and shift matrices that are external parameters of the i-th image, and M _j is the coordinate in the three-dimensional space of the j-th pattern,

Is the coordinate at which the respective coordinates of the pattern are projected onto the image by the estimated projection matrix.

Using the image correction parameter calculated through the above process, the coordinate corrector 220 adjusts coordinates of five human candidate regions corresponding to H, LH, RH, LF, and RF received from the human body detector 210. Correct each.

The human body matching unit 230 matches the human candidate region corrected by the coordinate corrector 220 based on a preset initial posture.

2A and 2B are diagrams illustrating respective images input through two image input apparatuses, and FIGS. 3A and 3B are diagrams illustrating a human candidate region detected from the images of FIGS. 2A and 2B using an image analyzing apparatus according to the present invention. Inverted drawing.

First, referring to the drawing, the human body detector 210 may receive a first image as shown in FIG. 2A and a second image as shown in FIG. 2B from the first image input apparatus 100_1. In this case, the human body detection unit 210 subtracts the background area using the RGB color space from the first image and the second image, and detects the skin color. As shown in FIGS. 3A and 3B, the head (H) and the left hand (LH) are detected. : Human candidate regions corresponding to Left Hand, Right Hand (RH), Left Foot (LF: Left Foot) and Right Foot (RF) are extracted. The coordinate correcting unit 220 corrects the extracted coordinates of the human candidate region through Equations 1 to 6.

However, since the extracted human candidate regions may include noises such as 310, 320, and 330 due to lighting and shadow regions, the human candidate region thus extracted is based on an initial posture (eg, a T-shaped posture) of a predetermined operator. The noises 310, 320, and 330 are removed by matching the human candidate region extracted from the first image (FIG. 3A) and the human candidate region extracted from the second image (FIG. 3B).

Hereinafter, a process of matching coordinates of a human body region in the image analyzing apparatus according to the present invention will be described in detail with reference to FIG. 4.

The epipolar geometry between two image input devices 100_1 and 100_2 means that each image plane intersects with a plurality of planes including a baseline connecting the center of each image input device.

Fig. 4 shows the epipolar geometry, where O and O 'are the centers of the respective image input devices, X represents a point in space, and x and x' are the points in space where X is projected onto each image. E and e 'are points at which the baseline intersects each image plane.

In each image, a line connecting the epipole (e, e ': epipole) and the feature points (x, x') of the image is called an epipolar line. If the exact correspondence is extracted, each correspondence point is the epipolar line. Will be above. That is, the corresponding point corresponding to a point x of the first image is present on the epipolar line 1 'corresponding to the second image. A fundamental matrix (F) representing this relationship is shown in Equation 7.

By the above properties, the error of matching point matching is calculated through Equation 8 below. For the corresponding points of N, the average of the distance between each corresponding point and the corresponding epipolar line is defined as the average epipolar distance error.

The human body matching unit 230 detects each human region by matching the human candidate regions extracted from two images more accurately by correcting the corresponding point matching error using epipolar geometry.

The human body tracking unit 240 tracks the movement of the human body region by applying a Kalman filter to the coordinates of the human body region calculated and matched by the human body matching unit 230.

The human body tracking unit 240 may use the Kalman filter and information on the area and the average color of the region found in the previous frame among the human candidate regions for stable tracking of the human body region.

More specifically, the body region tracking unit 240 is a three-dimensional image of each body region matched by the body region matching unit 230 by using coordinates corrected by the coordinate corrector 220 and image correction parameters. Real coordinates in space may be calculated through a linear triangulation method.

In addition, the human body tracking unit 240 captures the motion capture of the operator using the following Equation 9 for the five human regions H, LH, RH, LF, and RF acquired from the first image and the second image. Calculate.

Where A is a 4X4 matrix, p is a projection matrix, and X is a coordinate in space.

Table 1 shows the experimental results for the image analysis device according to the present invention. The execution time is an average time per frame after performing each step for 1,500 frames.

algorithm Execution time Background difference 7.43 ms Skin color extraction 2.57 ms Labeling    1 ms Coordinate correction    0 ms Human Area Tracking    5 ms Motion capture    0 ms

Since coordinate correction and motion capture are problems of solving linear equations for five corresponding points, the processing time can be obtained in near 0ms because the processing can be performed in a very short time. The execution time of skin color extraction algorithm is 3.0ms, 2.57ms, and 2.79ms, respectively, as the result of YCbCr, normalized RGB and Peer method, and skin color extracted through YCbCr and normalized RGB shows similar results. . Peer uses the correlation between R, G, and B in the RGB color space to define skin color. In addition, for the labeling algorithm, the run length encoding method takes 1ms, while the GrassFire method takes 1.89ms.

Accordingly, as shown in Table 1, the image analysis apparatus according to the present invention may process two images input from two image input devices simultaneously in real time as the time required to perform motion capture is small.

Hereinafter, the image analysis method according to the present invention will be described in detail with reference to FIG. 5.

First, the human body detector 210 of the image analyzing apparatus down-samples the first image and the second image received from the first image input apparatus 100_1 and the second image input apparatus 100_2, respectively (S510). The downsampled first and second images are subtracted from the background area using HSI (Hue Saturation Intensity) and / or Red Green Blue (RGB) color space, respectively, and skin color is extracted to correspond to the head, both hands, and both feet. Each of the five first human candidate regions and the second human candidate regions is extracted (S520).

Thereafter, the coordinate corrector 220 uses image guarantee parameters such as internal parameters and external parameters, which calculate the coordinates of the extracted first human candidate region and the second human candidate region through Equations 1 to 5, respectively. Correct with correct coordinates (S530).

The human body matching unit 230 detects the human body region by matching the first human candidate region and the second human candidate region corrected by the coordinate corrector 220 based on a preset initial posture (S540). In addition, the human body tracking unit 240 captures motion by tracking each human body area corresponding to the head, both hands, and both feet using the Kalman filter (S550).

Therefore, the image analysis method and apparatus according to the present invention detects the human body region using the skin color of the human body in the image and uses the same tracking method for the head, both hands, and both feet, so that the amount of calculation is small when detecting the human body region. According to this, the motion can be captured more accurately and the movement of the operator can be tracked in real time.

All of the above functions may be performed by a processor such as a microprocessor, a controller, a microcontroller, an application specific integrated circuit (ASIC), or the like according to software or program code coded to perform the function. The design, development and implementation of the code will be apparent to those skilled in the art based on the description of the present invention.

Although the present invention has been described above with reference to the embodiments, it will be apparent to those skilled in the art that the present invention may be modified and changed in various ways without departing from the spirit and scope of the present invention. I can understand. Therefore, the present invention is not limited to the above-described embodiment, and the present invention will include all embodiments within the scope of the following claims.

1 is a block diagram illustrating an image analyzing apparatus according to the present invention.

2A and 2B are diagrams illustrating respective images input through two image input apparatuses.

3A and 3B are views inverting a human candidate region detected from the images of FIGS. 2A and 2B by using an image analyzing apparatus according to the present invention.

4 is a view for explaining a process of matching the coordinates of the human area in the image analysis device according to the present invention.

5 is a flowchart illustrating an image analysis method according to the present invention.

Explanation of symbols on the main parts of the drawings

100_1 and 100_2: video input device 200: video analysis device

210: human body detection unit 220: coordinate correction unit

230: human body area matching unit 240: human body area tracking unit

Claims (11)

An image analysis method for tracking a movement of a human body using an image received from at least one image input device, Detecting a plurality of human regions by calming the background region of the image and extracting skin color; Correcting the coordinates of the detected human body region using image correction parameters; And And tracking the movement of the human body region using the corrected coordinates of the human body region. The method of claim 1, wherein the human body region, Image analysis method characterized in that the five areas corresponding to both hands, feet and face of the human body. The method of claim 1, wherein the detecting comprises: And detecting the human body region from the two images inputted from the two image input apparatuses, respectively. The method of claim 3, wherein the detecting comprises: Down sampling the input image; And And subtracting a background area and extracting skin color using at least one of a hues saturation intensity (HSI) and a red green blue (RGB) color space in the down-sampled input image. The method of claim 4, wherein after the correcting step, And removing the noise included in the human body area by matching the human body area from each of the images based on a preset initial posture. The method of claim 1, wherein the tracking comprises: And tracking a movement of the human body region by applying a Kalman filter to the corrected coordinates of the human body region. An image analysis apparatus for tracking a movement of a human body using an image received from at least one image input apparatus, A human body detector detecting a plurality of human regions by calming the background region of the image and extracting skin color; A coordinate corrector configured to correct detected coordinates of the human body region using image correction parameters; And And a human body tracking unit for tracking movement of the human body region by applying a Kalman filter to the corrected coordinates of the human body region. The method of claim 7, wherein the human body area detection unit, 5. An image analyzing apparatus comprising detecting five human regions corresponding to both hands, feet, and face of a human body. The method of claim 7, wherein the human body area detection unit, And the human body region is detected from the two images input from the two image input apparatuses. The method of claim 9, wherein the human body detection unit, Down sampling the input image and subtracting the background region using at least one of a Hue Saturation Intensity (HSI) and a red green blue (RGB) color space in the down sampled input image to extract the skin color The image analysis device, characterized in that for detecting. The method of claim 10, And a human body matching unit configured to remove noise included in the human body area by matching the human body area from each of the images based on a preset initial posture.

Images