JP2001005973A - Method and device for estimating three-dimensional posture of person by color image - Google Patents

Abstract

PROBLEM TO BE SOLVED: To make detectable the feature point of a part even when the part of a human body overlaps another part against a camera. SOLUTION: A binary image is prepared from color images picked up by a plurality of cameras in respectively different directions, and a feature point is detected on the basis of the binary image (S13). In the case the feature point can not be detected or predicted from the binary image, a two-dimensional flesh color area is extracted from the color images on the basis of a flesh color index. The extracted flesh color area is subjected to volume restoration in a three-dimensional manner (S23), a volume in which the volume is maximum is defined as a vicinity area, and its center of gravity is defined as the three- dimensional coordinates of the feature point (S31 and S43).

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method and apparatus for estimating a three-dimensional posture of a person, and more particularly to estimating the three-dimensional posture of a person based on, for example, color images input from a plurality of color cameras. The present invention relates to an estimation method and an apparatus.

[0002]

2. Description of the Related Art An example of this kind of prior art is disclosed in Japanese Patent Application Laid-Open No. Hei 10-25804, which was filed on Sep. 29, 1998.
No. 4 [A61B 5/10]. In this conventional technology, a thermal image is subjected to threshold processing to extract a silhouette image of a human body, and feature points of the body are detected from the contour shape of the silhouette image to estimate a three-dimensional posture of the person.

[0003]

In the prior art, a so-called non-contact three-dimensional posture can be estimated without attaching a special device to the subject's body, so that physical or psychological pressure is applied to the subject. Although there is an advantage that no thermal image is used, information that can be used for posture estimation is only a silhouette image of a human body and its contour shape because a thermal image is used. Therefore, part of the human body,
For example, when the hand overlaps with another part, for example, the body, the part cannot be detected, and there is a limit in improving the estimation accuracy.

[0004] Therefore, a main object of the present invention is to provide a novel three-dimensional posture estimation method and apparatus which can improve the estimation accuracy.

Another object of the present invention is to provide a three-dimensional posture estimation method and apparatus capable of detecting a characteristic point of a body even when a part of a human body overlaps another part. .

[0006]

An estimating method according to the present invention detects a feature point of a body based on a plurality of color images taken from a plurality of viewpoints using a plurality of color cameras, and detects the feature point from the feature point. A method for estimating a three-dimensional posture of a person, comprising the steps of (a) binarizing a color image to obtain a binary image, (b) detecting feature points from the binary image, and (c) feature points. Determining the validity of
If not, a method of estimating a three-dimensional posture of a person, including processing a color image to detect feature points.

The estimating apparatus according to the present invention includes a plurality of color cameras for photographing a person from different directions and outputting a color image, binarizing means for binarizing the color image and outputting a binary image, First detecting means for detecting the feature points of the body from the value image, validity determining means for determining the validity of the feature points, and, if the feature points are not appropriate, processing the color image to detect the feature points. It is an apparatus for estimating a three-dimensional posture of a person, comprising two detecting means.

[0008]

A color image of a person taken by a plurality of cameras from different directions is binarized, for example, by setting the person image to "1" and the background image to "0". Detect points. Specifically, the center of gravity, the main axis, and the outline, that is, the boundary of the person in the binary image are detected from the binary image.

Preferably, the plurality of color cameras are arranged so as to photograph a person from the front, side, and elevation of the person and at least one facing direction, or at least one of the front, side, and elevation of the person. Are arranged so as to photograph a person from two intermediate positions. By doing so, blind spots can be eliminated.

In order to detect a characteristic point of the body based on the center of gravity, the main axis and the contour, a contour point above the upper body center of gravity and the shortest distance from the upper body main axis is set as a temporary head vertex, and further, a toe point, a hand point point, etc. Detect feature points. At this time, if no feature point is detected, the position of the feature point is predicted by the prediction filter.

When detecting a feature point, it is necessary to determine the most suitable color image, that is, the shooting direction or viewpoint. In other words, a feature point candidate is detected for each of the plurality of color images, and the viewpoint is determined by evaluating the feature point candidates. By selecting a viewpoint in this way, a feature point can be detected or estimated more accurately.

Then, the validity of the detected or predicted feature point is determined. Specifically, when the three-dimensional straight-line distance between the feature points is a criterion and the straight-line distance is impossible, the corresponding feature point is determined to be invalid. At this time, for example, a joint angle of a shoulder, an elbow, a crotch, a knee, or the like may be added as a criterion. The bending direction and angle of these joints are determined, and when they deviate from those directions and angles, it may be determined that the feature points are not appropriate.

If it is determined that the color image is not appropriate, the color image is processed, and a feature point is detected from a flesh color region extracted from the color image. Specifically, for each color image, a three-dimensional volume is restored from the two-dimensional coordinate group of the flesh-colored area, the three-dimensional volume having the maximum volume is selected as the flesh-colored area, and, for example, a hand point is determined from the flesh-colored area.

The three-dimensional posture of the subject can be estimated from the feature points determined to be appropriate and / or the feature points obtained by color processing.

In another embodiment of the present invention, a method of extracting feature points from a silhouette image and a method of extracting feature points based on a time difference image are used together, and each feature point is processed by a Kalman filter. . The method for obtaining the feature points from the time difference image is, specifically, (a) a step of forming a time difference image from the color image data, (b) a step of forming an image obtained by extracting a skin color region from the color image data, c) extracting a feature point of the person based on the difference image data and the image data from which the flesh color region is extracted, and (d) performing a filtering process using a Kalman filter and a future position prediction on the feature point.

According to this embodiment, a feature point, that is, a three-dimensional posture can be accurately estimated even in a case where self-shielding occurs.

[0017]

According to the present invention, when a feature point cannot be detected or predicted by a binary image, the feature point is detected by color image processing. Even when it overlaps with another part, that part can be detected accurately. Therefore, according to the present invention, the accuracy of the estimated three-dimensional posture can be improved.

The above objects, other objects, features and advantages of the present invention will become more apparent from the following detailed description of embodiments with reference to the drawings.

[0019]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A virtual transformation device 10 of this embodiment shown in FIG.
Means five color CCD cameras (hereinafter simply referred to as "cameras").
That. ) 12a to 12e, each camera 1
2a to 12e photograph the target person from different directions. For example, the camera 12a is arranged so as to photograph a person from the front, the camera 12b is arranged from the side, the camera 12c is arranged from the upper side, the camera 12d is arranged diagonally from the left and the camera 12e is arranged diagonally from the rear. It should be noted that the cameras 12d and 12e may be arranged not from such an intermediate viewpoint but from the front or side facing surface, the back surface or the right or left side surface. May be changed as appropriate. However, using at least four cameras is desirable in that it eliminates blind spots. For example,
3 in case of only three directions of front, side and elevation (top)
Only three quarters of the body area is captured by the camera, and the remaining quarter is completely blind spot. In order to eliminate the optical blind spot, it is effective to install more cameras.

Color images are output from the cameras 12a to 12e, and each color image is output from each of the cameras 12a to 12e.
12e individually assigned to the image processing apparatuses 14a to 14e
4e. Each of the image processing devices 14a to 14e is a computer having a processing capability of a personal computer, but replaces an individual computer by sharing one or more higher-performance computers by performing time-division processing. be able to. And
As will be described in detail later, the image processing devices 14a to 14e process the input color images and detect the two-dimensional coordinates of the feature points of the body on each image.

The two-dimensional coordinate information of the characteristic points obtained by the image processing devices 14a to 14e is
It is sent to the dimensional coordinate calculation device 16. The three-dimensional coordinate calculation device 16 is also one computer,
The dimensional coordinates are three-dimensionally restored based on the principle of triangulation, and the validity of the restored data is verified based on, for example, the initially set body data of the person.

Then, the three-dimensional coordinate information of the feature points obtained by the three-dimensional coordinate calculating device 16 is sent to the body motion synthesizing device 18. Based on the three-dimensional coordinates of the feature points, the body motion synthesizer 18 uses a computer graphic technique to create a virtual image of the subject's alter ego called “avatar”.
Reproduce or reproduce in a dimensional environment.

The virtual transformation device 10 shown in FIG. 1 is a device that can be used as the “Shall We Dance?” System previously announced by the inventors and the like. Therefore, the present invention can be used not only for such a system but also for any system that needs to estimate the three-dimensional posture of the subject, for example, a training system, a monitoring system, a non-contact manipulation system, and the like.

Referring to FIG. 2, in virtual transformation apparatus 10 of the embodiment of FIG. 1, first, in step S1, camera 12
Color images of a person (subject) photographed by a to 12e are input to the corresponding image processing devices 14a to 14e.

Next, in step S3, the image processing apparatus 1
4a to 14e extract a person image from the input color image by the background difference method. The background subtraction method is to extract only a person image by subtracting an image without a person (background image) from an image with a person (input image). Note that the background image has been obtained in advance by preprocessing described later.

Then, in step S5, the input image is binarized by setting the person image to "1" and the background image to "0", and as shown in FIG.
Extract the value image.

In FIGS. 3 to 7, (a) shows a front view,
(B) shows an image of the side surface, (c) shows an image of the elevation surface,
The intermediate position image (or the opposing position image) based on the cameras 12d and 12e is omitted. However, it should be pointed out in advance that the same result can be obtained for the intermediate position image or the opposing position image by the same treatment as described below.

In step S7, the center of gravity (A) of the body shown in FIG. 4 is detected. When the body center of gravity is directly obtained from the silhouette image of FIG. 3 which is a binary image, the position of the center of gravity significantly moves due to the posture of the arm or foot and is unstable. Reduce the effects of posture due to limbs. The body center of gravity (IC, JC) in the distance conversion image dij is obtained by the following equation.

[0029]

(Equation 1)

In step S9, the main shaft (b) of the upper body shown in FIG. 5 is detected. The upper body main axis is the distance conversion image
The upper body distance image gij is obtained from dij, and the inclination and angle of the upper body are detected from the angle θ of the principal axis of inertia passing through the center of gravity of the upper body distance image gij.

The upper body distance image gij is a distance conversion image dij
For images above the horizontal position of the body center of gravity at
For removing the arm region, a Gaussian distribution with the upper body main axis of the previous frame as a symmetry axis is obtained by multiplying the distance value. The reason for removing the arm region is to eliminate an error in detecting the upper body main axis based on the arm region.

The angle θ of the principal axis of inertia passing through the center of gravity of the upper body distance image gij can be obtained by the following equation.

[0033]

(Equation 2)

When the hand is extended in the horizontal direction, the main axis may not be determined along the spinal cord, but this problem can be solved by removing the arm region.

In step S11, the outline of the person shown in FIG. 6 is extracted from the silhouette image of FIG. The contour line of the person is raster-scanned from the center of gravity of the body in the silhouette image, the boundary line is traced counterclockwise starting from the first found contour (boundary) pixel, "1" on the boundary line, It is obtained by extracting a binary image that becomes “0”. Here, the contour point at the shortest distance from the upper body main axis is defined as a temporary head vertex.

In step S13, the two-dimensional coordinate position of the characteristic point (c) of the human body on each of the front, side, and elevation images is detected as shown by the small black rectangle in FIG. The feature point detection method will be described with reference to the flowchart shown in FIG.

Referring to FIG. 8, first, in step S51, the toe point of the human body is detected. A skeleton image is created by calculating the maximum value of the distance conversion image dij used for calculating the body center of gravity, and the lower body (below the vertical position of the body center of gravity) and the left (right) half body (left centering on the horizontal position of the body center of gravity) Alternatively, an end point that satisfies the condition “the distance from the center of gravity of the body is the largest” is selected from the end points of the skeleton existing on the right side) as the toe point.

Next, in step S53, a hand point is detected. As shown in FIG. 9, the number of pixels included in the contour corresponding to the temporary head vertex and the toe obtained in step S11 is
It is divided into the ratio of h: mh: nh, and the contour point included in the center portion is set as a candidate section of the hand point. Note that lh: mh: n
The ratio of h is an empirically determined constant, for example, 2:
4: 4 is set.

As shown in FIG. 9, the point having the highest vertical position in the candidate section is the hand candidate point A, the point having the lowest vertical position is the hand candidate point B, and the point having the longest horizontal distance is the hand candidate point C. , Let the point on the toe side at the end point of the candidate section be the hand candidate point D,
Evaluation is performed in order according to the following conditions. The evaluation method for each hand candidate point will be described with reference to the flowchart shown in FIG.

Referring to FIG. 10, first, at step S61.
Then, when the vertical position of the hand candidate point A is larger than the value obtained by subtracting T1 from the vertical position of the temporary head vertex, the score 3 is given to the hand candidate point A because it is most likely.

Next, in step S63, the hand candidate point B
If the vertical position is smaller than the value obtained by adding T1 to the vertical position of the body center of gravity, there is a high possibility that score 2 is given to the hand candidate point B.

In step S65, when the horizontal position of the hand candidate point C is smaller than the value obtained by adding T2 to the horizontal position of the body center of gravity, the score is given to the hand candidate point C because the possibility is the lowest.

In step S67, if the hand candidate points A, B, and C do not satisfy any of the above conditions, a score of 0 is given to each of the hand candidate points A, B, and C.

Note that T1 and T2 are constants determined by the height of the circumscribed rectangle of each silhouette image.

In this way, the hand candidate points A, B and C are given a score of 0 to 3 according to the possible possibility, and two hand candidate points are assigned in descending order of the score. It is selected as a hand candidate. That is, a camera, that is, a shooting direction or a viewpoint is selected by this evaluation. Then, as described later with reference to step S21 in FIG. 2, three-dimensional restoration is performed from the image including the two selected hand candidate points.

If the scores given to the hand candidate points A, B and C are all 0, the hand candidate point D is regarded as the hand point.

Returning to FIG. 8, in step S55, the top of the head is detected. The number of pixels of the contour corresponding to both hands from the provisional head vertex is divided into the ratio of lP: mP: nP shown in FIG. And
The point where the distance to the upper body main axis is the shortest is set as the neck position,
The bisection point of the contour sandwiched between the left and right neck positions is defined as the top vertex. Note that the ratio of lP: mP: nP is an empirically obtained constant, as in the case of detecting the hand point.

In step S57, the elbow position is estimated.
As shown in FIG. 12, a straight line HT connecting the head vertex HP and the hand point TP is drawn, and a group of contour points between the head vertex HP and the hand point TP is defined as CP. Then, a straight line H in the contour point group CP
The point farthest from T is the provisional elbow point TE. Also, from general experience, the distance from the hand point to the elbow is set as HL.

If the difference between the distance from the hand point TP to the provisional elbow point TE and the distance HL from the hand point to the elbow known from experience is smaller than a predetermined threshold MD, the provisional elbow point TE is the position of the elbow.

Conversely, a difference MD between the two is determined in advance by a threshold MD.
If it is larger than the elbow position, the middle point of the curve composed of the contour points CP from the head vertex HP to the hand point TP is set as the position of the elbow.

Returning to FIG. 2, in step S15, it is determined whether or not the feature point can be detected. For example, when the hand point overlaps the torso, a hand point candidate section cannot be set in the contour image of the person in FIG. 6, so that the hand point cannot be detected. Also, when the toe point is located only on one side from the center of gravity, the current method assumes that the foot is on the left or right of the center of gravity, and therefore cannot detect the toe point.

If it is determined that the feature point can be detected, the process proceeds to step S17, and the two-dimensional coordinate information of the feature point detected in the previous step S13 is input to the three-dimensional coordinate calculation device 16. Conversely, when it is determined that the feature point cannot be detected, the process proceeds to step S19.

In step S19, the position of a feature point is predicted by a prediction filter. Here, the positions of the feature points in the current frame are predicted by the Kalman filter or the linear prediction filter from the detection results for the past several frames. The two-dimensional coordinate information of the predicted feature point is input to the three-dimensional coordinate calculation device 16.

Note that the past detection result is not the detection result of the image processing devices 14a to 14e but the two-dimensional coordinate value calculated by the three-dimensional coordinate calculation device 16 in step S27 and step S41 described later. The value is fed back to the image processing device 14 every time one frame is detected.

Next, in step S21, an image including the two hand candidate points selected in the previous step S67 (FIG. 10) is selected as the line of sight of the camera. That is, the two cameras that have captured the image are selected.

In step S23, the two-dimensional coordinates of the feature point are three-dimensionally restored. If the relative position or angle of the two cameras selected in step S21 is obtained, three-dimensional coordinate values are obtained from the two-dimensional coordinate information of the feature points input in step S17 according to the principle of triangulation. be able to.

In step S25, the validity of the three-dimensional coordinate values of the feature points is determined. The validity of the three-dimensional coordinate values of the feature points is determined based on, for example, the straight-line distance from the hand point to the hand point or the head vertex to the hand point in the three-dimensional space. That is, the validity of the linear distances is determined based on the initial limb length. In addition, not only the distance but also the angles of the joints (for example, the shoulder joint, the elbow joint, the hip joint, the knee joint, and the like) serve as the determination criterion. This is because the direction or angle at which the human joint can bend is determined. Therefore, the validity can be determined from the direction or angle at which the joint is bent.

Here, if it is determined that it is appropriate, the process proceeds to step S27. If it is determined that it is not appropriate, the process proceeds to step S29, and the feature point is detected again.

In step S27, the three-dimensional coordinate values of the feature points calculated by the three-dimensional coordinate calculation device 16 are inversely mapped two-dimensionally. That is, each camera 12a-
The inverse transformation is performed to the two-dimensional coordinates in 12e. Then, the coordinates of the two-dimensionally replaced feature points are transmitted to the image processing device 14 and used for predicting the motion of the person by the prediction filter in the next step S19, that is, predicting the movement position of the feature points. Thereby, it is possible to prevent the prediction accuracy from deteriorating.

In step S31, the three-dimensional coordinate values of the feature points three-dimensionally restored in step S23 are transmitted to the body movement synthesizing device 18.

If it is determined in step S25 that the image is not valid, the color image is processed in step S29, and the feature points are detected again (second detecting means). In this embodiment, in order to obtain a skin color index used in color image processing, preprocessing is performed before normal image processing. The preprocessing method will be described with reference to the flowchart shown in FIG.

Referring to FIG. 13, first, in step S71,
Then, the user learns a background image without a person. next,
In step S73, an image in which a person appears in the background is photographed. In step S75, only the person image is extracted by the background subtraction method as in step S3.

In step S77, referring to the "skin color space" in a generally known RGB space (a space in which RGB is represented by, for example, 256 gradations), all skin color portions including hands and faces of a person image are referred to. Is extracted.

Then, in step S79, a flesh color index unique to the person (subject) or the lighting is created from the flesh color portion. The created skin color index is used to extract a skin color region of a person from a color image in color image processing in step S29 described below.

Next, the color image processing in step S29 will be described with reference to the flowchart shown in FIG.

Referring to FIG. 14, first, in step S81,
In, color images of a person taken by the cameras 12a to 12e are input to the image processing devices 14a to 14e.

Next, in step S83, from the detection result of the previous frame, a range in which a hand in the frame can be detected is predicted by the same method as the prediction filter in the previous step S19. This is because the background image is removed as much as possible by reducing the detection area, and the processing speed and accuracy are improved.

In step S85, hand candidates are detected. As the hand candidate, an area that matches the person-specific skin color index created in step S79 of the preprocessing is extracted from the hand area predicted in step S83, and the area is set as a hand candidate area.

In step S87, a two-dimensional noise in the hand candidate area is removed by a low-pass filter or the like, and a flesh color area of the human body is extracted. That is, when there are a plurality of detected flesh-colored areas, only the widened area of the flesh-colored area is left, and the narrowed area is removed.

Then, in step S89, the information of the skin color area extracted in the previous step S87 is transmitted to the three-dimensional coordinate calculation device 16.

Returning to FIG. 2, the volume of the skin color area extracted in the previous step S87 is restored in step S33. That is, a two-dimensional area obtained by combining all the skin color areas extracted from the respective cameras 12 is restored as an area (volume) having a three-dimensional volume.

Next, in step S35, the three-dimensional noise of the three-dimensionally restored skin color region is removed by the same method as the two-dimensional noise removal in step S87. That is, an area volume having a small volume is removed.

Then, in step S37, of the restored three-dimensional skin color volume, only one set having the largest volume is selected and determined as the hand area.

In step S39, the previous step S37
, The center of gravity of the selected volume, that is, the hand area is detected. The detected value of the center of gravity is set as a representative value of the three-dimensional coordinates in the volume, and the value is set as the hand point.

In step S41, as in step S27, the three-dimensional coordinate values of the feature points calculated by the three-dimensional coordinate calculation device 16 are inversely mapped two-dimensionally. Then, the coordinates of the two-dimensionally replaced feature points are transmitted to the image processing device 14 and used for the prediction filter processing in the next step S19.

Then, in step S43, the three-dimensional coordinate value of the center of gravity, that is, the hand point detected in the previous step S39 is transmitted to the body movement synthesizing device 18.

As described above, in this embodiment, even when a feature point of a human body cannot be detected from a silhouette image obtained by binarizing an input color image, the feature point is detected from the skin color region by processing the color image again. can do. Therefore, even when a part of a human body, for example, a hand overlaps another part, for example, a torso, a feature point can be accurately detected or estimated, and an improvement in the accuracy of posture estimation can be expected.

In the above-described embodiment, if a feature point cannot be detected due to, for example, self-shielding (for example, a posture in which the body and hand overlap), step S1 in FIG.
In 9, the Kalman filter is used to estimate feature points. However, the feature points are the parietal position, the left and right hand positions, the left and right elbow joint positions, and the left and right foot positions.

However, the method of estimating a feature point based on a so-called silhouette image cannot fundamentally solve the problem of self-shielding. Therefore, a new method that can reliably execute the feature point estimation by the Kalman filter in step S19 will be described below.

FIG. 15 shows a part of the operation of another embodiment of the present invention. In this embodiment, steps S3-S1 of FIG.
Feature point detection processing according to procedures up to 3 (step S2)
And a feature point detection process using an inter-frame difference are executed in parallel. If the reliability of the feature point detection in step S2 is high, specifically, if the score in FIG. 10 is high, the coordinates obtained by this processing are adopted as the observation result of each feature point. Using the observation result, the Kalman filter is updated in step S101, the observation data is filtered, and the position of each feature point in the future frame is predicted. Conversely, if the reliability of the feature point, that is, the score is low in step S2, the coordinates detected by the difference image are used as the observation result of the feature point, and the Kalman filter is updated using the result, and the filtering process of the observation data is performed. Predict the position of each feature point in the future frame.

More specifically, step S9 in FIG.
In step 1, the difference between temporally adjacent frame images is calculated. In this case, it takes an enormous amount of time to calculate the difference for all the image areas. Therefore, as shown in FIG. 16, a window area for the difference calculation is set. Step S101 executed in the previous frame for setting this window area
The prediction result using the Kalman filter is used. That is, when the predicted position shown in FIG. 16 is estimated as a feature point in step S101 of the k-th frame, a difference calculation area (window) of a predetermined size is set around the predicted position in the (k + 1) -th frame. In this way, the efficiency of calculation is improved.

In step S91, the difference between the image in the window area of the k-th frame and the image in the window area of the (k + 1) -th frame is calculated. In the actual calculation,
As shown in FIG. 17, the window area is enlarged and displayed, and a difference is calculated for the enlarged and displayed window area. After calculating the inter-frame difference, the difference image is binarized to obtain a binarized inter-frame difference image as shown in FIG.

Thereafter, in step S95, the binarized difference image is approximated to a quadrangle. For example, in particular, the hand is in front of the torso and self-shielding occurs, and the hand and the torso move simultaneously. In such a case, since it is difficult to extract the difference image of only the hand even by the difference calculation, in step S93, the skin color area is masked. Specifically, as shown in FIG. 18, a skin color region is extracted based on the skin color information obtained in advance from the color image, and a skin color mask is applied to the difference image obtained in step S91. As shown in FIG.
Extract the hand image.

Next, in step S95, the difference image binarized or extracted as described above is approximated to a quadrangle, for example, as shown in FIG. Specifically, a point where the coordinates in the vertical direction and the horizontal direction are maximum or minimum is determined, and a quadrangular area is determined at that point.

Then, in step S97, the distance between all the vertices of the quadrangle approximated in step S95 and the feature point specified in the k-th frame is calculated, and the vertex of the quadrangle having the smallest distance is calculated. Step S
At 99, it is detected as a feature point in the (k + 1) th frame. Then, the process proceeds to the Kalman filter process in step S101. The method of estimating feature points using the Kalman filter is described in detail in, for example, "Kalman Filter" written by Taku Arimoto and published in Sangyo Tosho in 1977. explain. 1. Modeling of feature point time history by AR model Position coordinates [xPs (k) yPs (k) of feature points Ps (s = top, right hand, left hand, right foot, left foot, right elbow joint and left elbow joint) ] Has a time history of 3
Assume that given by the R model.

[0086]

(Equation 3)

However, the variable ζ (k) in Expression 3 is xPs (k) when the x coordinate is targeted, and yP when the y coordinate is used.
Ps (k). ε (k) is Gaussian white noise of N (0, R), and ai is an unknown coefficient. Assuming that the unknown coefficient is a state vector, the state equation can be expressed by Equation 4.

[0088]

(Equation 4)

The observation equation is given by Expression 5 from Expression 3.

[0090]

(Equation 5)

2. Estimation of State Vector by Kalman Filter An unknown coefficient is sequentially estimated by applying a Kalman filter to the systems of Equations 4 and 5. However, since c (k) is time-varying, the Kalman filter is as shown in Equation 6.

[0092]

(Equation 6)

The calculation procedure of the Kalman filter will be described below.

(1) First, an initial value of Expression 7 is set.

[0095]

(Equation 7)

(2) Subsequently, the observed value xPs (k) (yPs
(K), zPs (k)) is taken and used as ψ on the right side of equation (4) of equation (6).

(3) Then, a Kalman filter is calculated. That is, M (k) (Equation (7)), the covariance P (k) of the estimation error (Equation (6)), and the Kalman gain K
(K) (equation (5)) and state vector 状態
^順次 (k) (Equation 6 (4)) is sequentially calculated. 3. The filtering and the estimation result coefficient ai at the future predicted time k are substituted into Equation 3 to perform filtering on the observation result ζ (k). By substituting the filtering result by time by one step and substituting it into Equation 3, d-step future (d
= 1, 2, ...).

As described above, in step S101, the Kalman filter process is executed, and finally the feature points are estimated. Therefore, even when self-occlusion occurs, each feature point of the person image can be obtained in a personality, and the posture of the person can be grasped in a non-contact and personal manner.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a virtual transformation device according to an embodiment of the present invention.

FIG. 2 is a flowchart for explaining the overall operation of this embodiment.

FIG. 3 is an illustrative view showing a binary image of a body;

FIG. 4 is an illustrative view showing a center of gravity of a body;

FIG. 5 is an illustrative view showing a main body main axis of the body;

FIG. 6 is an illustrative view showing a contour line of a body;

FIG. 7 is an illustrative view showing characteristic points of a body;

FIG. 8 is a flowchart illustrating a method for detecting a feature point of a body.

FIG. 9 is an illustrative view for explaining a method of detecting a hand point;

FIG. 10 is a flowchart illustrating a method for evaluating a hand candidate point.

FIG. 11 is an illustrative view for explaining a method of detecting a top vertex;

FIG. 12 is an illustrative view for explaining a method of estimating an elbow position;

FIG. 13 is a flowchart illustrating a pre-processing method.

FIG. 14 is a flowchart illustrating a color image processing method.

FIG. 15 is a flowchart showing the operation of another embodiment of the present invention.

FIG. 16 is an illustrative view showing a window area when calculating an inter-frame difference image;

FIG. 17 is an illustrative view showing a procedure until a feature point is specified from an inter-frame difference image;

FIG. 18 is an illustrative view showing a skin color mask process;

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 10 ... Virtual transformation apparatus 12a-12e ... Color camera 14a-14e ... Image processing apparatus 16 ... 3D coordinate calculation apparatus 18 ... Body motion synthesis apparatus

 ──────────────────────────────────────────────────の Continuing on the front page (72) Inventor Tatsumi Sakaguchi 5th Sanraya, Inaya small character, Seika-cho, Soraku-gun, Kyoto Pref. 5F, Sanraya, Seiya-cho, Seika-cho, Soraku-gun F-term (reference) 2F065 AA17 AA37 CC16 FF04 JJ03 JJ05 JJ26 QQ04 QQ13 QQ31 QQ32 QQ33 QQ45 4C038 VA04 VA20 V57B02 5 BA19 CA01 CA08 CA12 CA16 CE06 CE12 CE16 DA07 DB02 DB03 DB06 DB09 DC06 DC08 DC16 DC25 DC32 5L096 AA02 CA05 EA43 FA06 FA15 FA60 FA63 FA66 FA67 GA08 GA38 GA55 HA01 9A001 GG03 HH20 HH25 HH27 HH28 HH29 HH31 KK37

Claims (15)

[Claims] 1. A method for detecting a feature point of a body based on a plurality of color images photographed from a plurality of viewpoints using a plurality of color cameras, and estimating a three-dimensional posture of a person from the feature points. (a) binarizing the color image to obtain a binary image, (b) detecting the feature point from the binary image, (c) determining the validity of the feature point, and (d) A method for estimating the three-dimensional posture of a person, comprising processing the color image if it is not appropriate to detect a feature point. 2. The method according to claim 1, wherein said step (b) includes a step (b1) of predicting a feature point by a prediction filter. 3. The method according to claim 1, wherein the step (b) comprises detecting feature point candidates for each of the plurality of color images.
The method according to claim 1 or 2, comprising: (b) selecting a viewpoint by evaluating each feature point candidate. 4. The method according to claim 1, wherein the step (c) includes a step (c1) of determining validity based on at least a distance between the feature points. 5. The method according to claim 4, wherein in said step (c1), said validity is determined by further considering a joint angle. 6. The method according to claim 1, wherein the step (d) includes a step (d1) of extracting a flesh color area from the color image and a step (d2) of detecting a feature point from the flesh color area. The method described in Crab. 7. The method according to claim 6, wherein in the step (d1), a two-dimensional coordinate group is restored to a three-dimensional volume, and a three-dimensional volume having a maximum volume is selected as a skin color area. 8. A step of (a) binarizing a plurality of color images photographed from a plurality of viewpoints using a plurality of color cameras to obtain a binary image, and (b) obtaining the feature points from the binary image. Detecting, (c) determining the validity of the feature point, (d) if not, processing the color image to detect the feature point, and (e) according to the detected feature point A recording medium storing a program for causing a computer to execute a step of estimating a three-dimensional posture. 9. A plurality of color cameras for photographing a person from different directions and outputting a color image, binarizing means for binarizing the color image and outputting a binary image, First detection means for detecting a feature point; validity determination means for determining the validity of the feature point; and second detection for processing the color image and detecting the feature point when the feature point is not valid. An apparatus for estimating a three-dimensional posture of a person, comprising: 10. The apparatus of claim 9, wherein the plurality of color cameras are arranged to photograph a person from the front, side, and elevation of the person and at least one facing direction. 11. The apparatus of claim 9, wherein the plurality of color cameras are arranged to photograph the person from a front, side, elevation, and at least one intermediate position of the person. 12. The apparatus according to claim 9, wherein said second detecting means includes a flesh color area extracting means for extracting a flesh color area from said color image, and wherein said characteristic point is detected based on said flesh color area. An apparatus according to claim 1. 13. A method for extracting a feature point from color image data measured by a color camera and estimating a three-dimensional posture of a person according to the feature point, comprising: (a) forming a time difference image from the color image data; (B) forming an image in which a flesh color region is extracted from the color image data; (c) extracting a feature point of a person based on the difference image data and the image data in which the flesh color region is extracted. ,
And (d) performing a filtering process using a Kalman filter and a future position prediction on the feature points,
3D pose estimation method. 14. The method according to claim 14, further comprising the step of: (e) forming a silhouette image of a person from the color image data to extract feature points of the person;
(c) or processing the feature points obtained in step (e),
The three-dimensional posture estimation method according to claim 13. 15. A step of: (a) forming a time difference image from the color image data; (b) forming an image in which a skin color region is extracted from the color image data; and (c) forming the difference image data and the skin color. Extracting a feature point of the person based on the image data from which the region is extracted, (d) performing a filtering process using a Kalman filter and a future position prediction on the feature point, and (e) detecting the feature point. A recording medium recording a program for executing a step of estimating a three-dimensional posture of a person.