JP3400961B2 - Apparatus for estimating posture of human image and recording medium storing program for estimating posture of human image - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a posture estimation device for estimating the posture of a human image captured by a camera, and a recording medium recording a posture estimation program for estimating the posture of a human image captured by the camera.

[0002]

2. Description of the Related Art An example of this type of prior art is disclosed in Japanese Patent Application Laid-Open No. 10-2580 filed on September 29, 1998.
No. 44 publication [A61B5 / 10]. This prior art estimates the posture of a person by extracting the silhouette of the person from a thermal image taken by an infrared camera and analyzing the contour of the extracted silhouette image.

[0003]

However, in this prior art, there are a number of conditions (for example, the right foot is below the center of gravity and to the right of the body; the hand is above the center of gravity and below the head). , A certain distance from the main axis.) Has to be set in advance, which poses a problem that the process for posture estimation becomes complicated. In addition, in this conventional technology,
It was impossible to detect the feature points when the limbs crossed each other or when the limbs and the body completely overlapped each other.

Therefore, a main object of the present invention is to provide a posture estimating apparatus capable of estimating the posture of a person by a simple process.

Another object of the present invention is to provide a recording medium recording a posture estimation program capable of estimating the posture of a person by a simple process.

Another object of the present invention is to provide a posture estimation device capable of accurately detecting feature points even when the posture of a person moves. Still another object of the present invention is to provide a recording medium recording a posture estimation program capable of accurately detecting feature points even when the posture of a person moves.

[0007]

According to a first aspect of the present invention, there is provided a posture estimating device for estimating a posture of a human image captured by a camera, a contour detecting means for detecting a contour of the human image, and a person from each position on the contour. A posture estimation apparatus comprising: a distance detection unit that detects a distance to each of a plurality of reference positions on an image, and a feature point detection unit that detects a feature point of a human image based on the distance. is there.

[0008]

According to a second aspect of the present invention, in a recording medium recording a posture estimating program for estimating the posture of a human image captured by a camera, the posture estimating program includes a contour detecting step for detecting the contour of the human image, and a contour detecting step. A distance detection step of detecting a distance from each position to each of the plurality of reference positions on the human image, and a feature point detection step of detecting a feature point of the human image based on the distance, It is a recording medium.

[0010]

[0011]

According to the first invention, the contour of the human image is detected by the contour detecting means, and the distance from each position on the contour to each of the plurality of reference positions on the human image is detected by the distance detecting means. To be detected. The feature point detecting means detects the feature points of the person image based on the distance thus detected.

The plurality of reference positions preferably include a top position and a barycentric position of the person image.

In one aspect of the present invention, the characteristic points are detected as follows. First, the calculating means squares the plurality of distances detected at a certain position on the contour individually, and then the adding means adds the plurality of square values obtained by the calculating means to each other. Then, the position on the contour where the added value of the addition means is maximum is detected by the maximum position detection means. The detected position is a feature point.

In another aspect of the present invention, the camera captures a human image whose posture changes with time at predetermined intervals, and the feature point detecting means detects a feature point of each captured human image. . Here, the initial posture of the person image is preferably a posture in which the limbs do not overlap the body and the limbs do not intersect with respect to the camera.

In one embodiment of the present invention, the Kalman filter process is applied to the feature points of the current person image detected by the feature point detecting unit by the filter unit. The feature point determination means determines the feature point of the current human figure based on the processing result of the filter means and the first autoregressive model formula.

Further, when the feature point predicting means predicts the feature point of the next person image to be photographed next time based on the processing result of the filter means and the second autoregressive model formula, the partial image area setting means A partial image area including the predicted feature point is set on the next person image. At this time, the distance detecting means
The distance from each position on the contour included in the partial image area to each of the plurality of reference positions is detected.

[0017]

[0018]

[0019]

According to the second aspect of the invention, a posture estimation program for estimating the posture of the human image taken by the camera is recorded in the recording medium. In this posture estimation program, the contour of the human image is detected in the contour detecting step, and the distance from each position on the contour to each of the plurality of reference positions on the human image is detected in the distance detecting step. In the feature point detection step, the feature point of the human image is detected based on the distance.

[0021]

[0022]

According to the first and second inventions,
The distance from each position on the contour to each of the multiple reference positions on the human image is detected, and the feature points of the human image are detected based on the detected distances. Can be estimated.

[0023]

The above-mentioned objects, other objects, features and advantages of the present invention will become more apparent from the following detailed description of embodiments with reference to the drawings.

[0025]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Referring to FIG. 1, a posture estimation apparatus 10 of this embodiment includes a color camera (hereinafter, simply referred to as "camera") 12. The camera 12 captures an entire image of a person (subject) whose posture changes over time, for example, every 1/30 seconds, and inputs the captured entire image and background image data to the image processing device 14. The posture estimation program recorded on the optical disc 16 is pre-installed in the image processing device 14. The image processing device 14 includes the camera 1.
The posture of the subject is estimated based on the image data input from 2 and the installed posture estimation program, and the posture of the avatar is controlled. That is, the posture of the avatar is made to follow the change in the posture of the subject. Here, "avatar" means
It is a two-dimensional CG model reproduced in a virtual environment. The image data of the avatar taking a desired posture is the model synthesizing device 1
8, the model synthesizer 18 synthesizes the image data of the given avatar with the background image data. The composite image data generated in this way is displayed on the screen by the image display device 20.

The above posture estimation program is constructed as shown in FIGS. 2 and 3. The CPU 14a provided in the image processing device 14 follows the flow chart as described above.
The image data input from the camera 12 is processed.

The CPU 14a first sets the count value K of the frame counter 14b to "0" in step S1, and initializes the search window (search area) in step S3. The size of the initialized search window is equal to the size of the screen, and all the captured images (person image and background) are included in the search window. The CPU 14a then proceeds to step S
An image is captured from the camera 12 in 4 and a human image is extracted from the input image by the background subtraction method in step S5. The background subtraction method is a method of extracting only the human image by subtracting the background image from the image containing the human image. It should be noted that the subject needs to take a posture toward the camera 12 at least at the start of photographing so that the four limbs do not overlap with the body and the four limbs do not cross each other.

In step S7, the CPU 14a further binarizes the input image by setting the person image to "1" and the background image to "0". Thereby, the silhouette image of the person is obtained. Subsequently, the CPU 14a detects the center of gravity of this silhouette image in step S9, and detects the main axis of the upper half of the same silhouette image in step S11. That is, when the silhouette image has the posture shown in FIG. 4, the center of gravity G shown in FIG. 4 and the principal axis of inertia extending upward from the center of gravity G are detected.

In step S13, a contour line is detected from the silhouette image inside the search window. That is, the boundary line that is first discovered by raster scanning from the center of gravity G is used as a starting point, and the boundary line is traced counterclockwise. The contour line of the silhouette image can be obtained by extracting a binary image that has "1" on the boundary line and "0" at other positions. In a succeeding step S15, an intersection point between the principal axis of inertia and the contour line is detected, and this is set as a vertex. That is, the intersection H shown in FIG.
Is the top point.

In this way, the contour line of the human figure and a plurality of reference positions (the center of gravity and the apex of the head) on the human figure.
Is detected.

After that, the CPU 14a proceeds to step S16, and determines the position on the contour line moved from the apex by a predetermined amount as the measurement position. The moving direction is the counterclockwise direction. In step S17, the vectors from the current measurement position toward the vertex and the center of gravity are obtained, and the norm of these two vectors is detected. That is, the distances from the current measurement position to each of the top vertex and the center of gravity are obtained. In the following step S19, the sum of squares of the two detected distances is calculated according to the equation 1.

[0032]

[Equation 1] In step S21, it is determined whether or not the current measurement position is the apex, and if NO, the measurement position is updated in step S22. That is, the position that is moved in the counterclockwise direction by a predetermined amount from the current measurement position on the contour line is set as the next measurement position. Then, the process returns to step S17. As a result, the sum of squares of the distance from each measurement position on the contour line to the center of gravity and the apex of the head is obtained. If the current measurement position is the position P shown in FIG. 4, the distance a (s) from the position P to the center of gravity G and the distance b (s) from the position P to the apex H are obtained in step S17, and in step S19. The sum of squares of such distances a (s) and b (s) is obtained.

When the sum of squares at each position on the contour line is obtained, the CPU 14a detects the position on the contour where the sum of squares has a maximum value in step S23. Figure 4
When the posture shown in is taken, the sum of squares changes as shown in FIG. In the initial posture, the subject faces the camera 12, and the limbs do not overlap the body with respect to the camera 12 and do not cross each other. Further, the moving direction of the measurement position is predetermined to be the counterclockwise direction. Therefore, the maximum position A shown in FIG.
Is the right hand tip, the maximum position B is the right foot tip, the maximum position C is the left foot tip, and the maximum position D is the left hand tip.
That is, by the process of step S23, the coordinates of the four feature points on the outline of the human image are detected, and
It is determined which of the four limbs the detected feature point coordinates are.

The CPU 14a determines 2 based on the distance from the current measurement position to the two reference positions (center of gravity and top vertex).
I am looking for the sum of squares. Therefore, as long as the human image is captured in a posture in which the limbs do not overlap the body and do not intersect with each other, the characteristic points of the human image can be accurately detected. When the subject takes the posture shown in FIG. 6 at the start of photographing, the sum of squares shows a larger value at each feature point. Therefore, the feature point can be detected most reliably.

The head apex detected in step S15 is also a feature point located on the contour line, and the coordinates of a total of five feature points have been detected by the processing up to this point. Each detected feature point coordinate is written in the table shown in FIG. 7 formed in the memory 14c. The X and Y coordinates of the characteristic point detected in the Kth frame are defined as X (K) and Y (K). Further, the feature point coordinates of the right hand tip are associated with M = 1, and the feature point coordinates of the right foot tip are M = 2.
, The feature point coordinate of the left foot is associated with M = 3, the feature point coordinate of the left hand is associated with M = 4,
Then, the feature point coordinates of the crown are associated with M = 5. Such a table is created for each frame.

It should be noted that X shown in this table
_S (K), Y _S (K), X _S (K + 1), Y _S (K + 1),
The details of a _{1 to} a _p and b _{1 to} b _p will be described later. In addition, in this embodiment, a total of five feature points are detected, but the number "5" is determined in consideration of the simplicity of the processing, and when it is desired to improve the accuracy of posture estimation. May increase the number of feature points to be detected.

The CPU 14a subsequently sets the count value M of the feature point counter 14d to "1" in step S24. In step S25, the characteristic point coordinates X (K) and Y (K) corresponding to the current count value M are read from the table shown in FIG. 7, and each coordinate is subjected to the Kalman filter processing according to the equations 2 and 3. The coefficients a ₁ , a ₂ ... _Ap and b ₁ , b ₂ ... B _p obtained by this are written in the positions corresponding to the current count value M in the table shown in FIG. Note that X (K-1), X (K-2), ... X (K-
p) and Y (K-1), Y (K-2), ... Y (K-
p) is the feature point coordinates detected before the previous frame, and these data are read from another table created in the same manner as in FIG. 7.

[0038]

[Equation 2]

[0039]

[Equation 3] In the following step S27, the coefficients a ₁ , a ₂ ... _Ap and b ₁ , b ₂ ... B _p obtained by the equations 2 and 3 are applied to the autoregressive model formula represented by the equation 4 to calculate the characteristic points. Coordinates X _S (K) and Y _S (K) are calculated.

[0040]

[Equation 4] _{Here, X S (K-1)} , X S (K-2), ... X S (K-
p) and Y _S (K-1), Y _S (K-2), ... Y
_S (K-p) is also a feature point coordinate calculated before the previous frame, and is read from another table. In addition,
e (K) indicates an error.

Coordinates X of feature points read from the table
(K) and Y (K) are only temporary coordinates. For this reason, the Kalman filter processing is applied to the coordinates X (K) and Y (K), and the coefficients a ₁ and a ₂ obtained by this are applied.
The coordinates X _S (K) and Y _S (K) are calculated based on a _p and b ₁ , b _2, ... B _p and the autoregressive model formula. As a result, the coordinates of the feature points are fixed. Confirmed coordinate X
_S (K) and Y _S (K) are written in the position corresponding to the current count value M in the table shown in FIG.

In step S29, the feature point counter 14
It is determined whether or not the count value K of d indicates "5", and if NO, the count value M is incremented in step S31 and the process returns to step S25. Therefore, the Kalman filtering process and the calculation using the autoregressive model formula are performed on all the five feature points stored in the memory 14c, and the coordinates of all the feature points are determined.

When the count value M reaches "5", the CPU
Step 14a advances from step S29 to step S33, and controls the posture of the avatar based on the determined five feature point coordinates. The image data of the avatar whose posture is controlled is given from the image processing circuit 14 to the model synthesizer 18. As a result, an avatar that takes the same posture as the subject is displayed on the image display device 20. That is, an avatar that takes a desired posture is reproduced in the virtual environment.

Subsequently, the CPU 14a sets the count value M of the feature point counter 14d to "1" again in step S35. Then, in step S37, the autoregressive model formula represented by the equation 5 is calculated.

[0045]

[Equation 5] Here, coefficients a ₁ calculated in step S25 immediately before, a ₂ ... a _p and _{b 1, b 2 ... b p} , feature point coordinates calculated at step S27 immediately before X _S (K) and Y _S ( K) and step S27 before the previous frame
Feature point coordinates X _S (K-1), ... X _S (K + 1)
-P) and Y _S (K-1), ... Y _S (K + 1-p) are used for the calculation. As a result, the characteristic point coordinates X _S (K + 1) and Y _S (K + 1) of the human image in the next frame are predicted. That is, the posture of the subject in the next frame is predicted.

At step S39, the count value M is "5".
If NO, step S4
After incrementing the count value M by 1, step S
Return to 37. As a result, the coordinates of all feature points in the next frame are predicted. Note that the predicted feature point coordinates X
_S (K + 1) and Y _S (K + 1) are also stored in the table shown in FIG.

When M = 5, the CPU 14a proceeds to step S43 and sets a search window centered on the characteristic point coordinates X _S (K + 1) and Y _S (K + 1) predicted by the processing of step S37 on the screen. . When the subject is predicted to assume the posture shown in FIG. 6 in the next frame, the search window is set to the position shown by the dotted line in the figure. The number of search windows set is “5”, which is the same as the number of feature points, and all the search windows are rectangular. Furthermore, the size of each search window is significantly smaller than the size of the screen. That is, although the number of initial search windows is one and the size is equal to the screen size, the number of search windows is increased to five by updating the search windows in step S43, and the size is significantly larger than the screen size. Get smaller. When such a search window is set, the CPU 14a increments the frame counter 14b in step S45. And step S
From 4 onward, similar processing is executed for the input image of the next frame.

As a result of updating the number and size of the search windows, only the contours around the feature points are extracted in the next step S13, and the feature points (head vertex and maximum are extracted based on such partial contour lines. The coordinates of (point) are detected. Then, the detected feature point coordinates are subjected to Kalman filtering, the feature point coordinates of the next frame are determined based on the coefficient obtained thereby, and the feature point coordinates of the next frame are predicted. Further, the search window is updated based on the predicted feature point coordinates.

According to this embodiment, in detecting the characteristic points (hand point and toe point) of the human image, the distances from the respective positions on the contour line of the human image to the center of gravity and the apex of the head are detected, Furthermore, the sum of squares of the two distances detected at each position is calculated. The position where the calculated sum of squares is maximum is set as the characteristic point. Therefore, as long as the image of the person takes a posture in which the limbs do not overlap the body and the limbs do not intersect with each other, each feature point can be easily detected, and the detected feature point is which part of the person. It can be easily determined. That is, the posture of the person image can be easily estimated.

Further, in this embodiment, the detected characteristic points (including the apexes) are subjected to the Kalman filter processing, and the characteristic points of the human image are determined based on the coefficients and the autoregressive model formula obtained thereby. I am trying. Therefore, the feature points can be accurately specified.

Further, in this embodiment, the characteristic points of the human image captured in the next frame are predicted based on the coefficient obtained by the Kalman filter processing and another autoregressive model formula, and the search including the predicted characteristic points is performed. Set the window above the figure in the next frame. Then, the distances from the respective positions on the contour belonging to the set search window to the center of gravity and the head apex are obtained, and the feature points are detected in the same manner as described above based on the obtained distances. Therefore, in the second and subsequent frames, it is not necessary to scan the entire range of the contour line to detect the characteristic points.

[Brief description of drawings]

FIG. 1 is a block diagram showing an embodiment of the present invention.

FIG. 2 is a flowchart showing a part of the operation of the embodiment in FIG.

FIG. 3 is a flowchart showing another portion of the operation of FIG. 1 embodiment.

FIG. 4 is an illustrative view showing a silhouette image of a subject.

FIG. 5 is a graph showing the relationship between each measurement position on the contour line and the sum of squared distances.

FIG. 6 is an illustrative view showing a person image and a search window set thereon.

FIG. 7 is an illustrative view showing a table related to a Kth frame.

[Explanation of symbols]

10 ... Attitude estimation device 12 ... Color camera 14 ... Image processing device 16 ... Optical disc 18 ... Model synthesizer 20 ... Image display device

Continuation of the front page (56) References Shoichiro Iwasawa and 4 others, Construction of "Shall We Dance?", IEICE Technical Report (PRMU98-112-122), November 12, 1998, Vol. 98, No. 394, pp. 15-22 (58) Fields surveyed (Int.Cl. ⁷ , DB name) G06T ⁷ /00-7/60 A61B 5/107 G06T 1/00 JISST file (JOIS)

Claims (8)

(57) [Claims] 1. A posture estimating apparatus for estimating the posture of a human image captured by a camera, comprising: a contour detecting means for detecting a contour of the human image; Distance detecting means for detecting the distance to each of the reference positions,
And a posture estimation device comprising feature point detection means for detecting a feature point of the person image based on the distance. 2. The posture estimating apparatus according to claim 1, wherein the plurality of reference positions include a top position of the human figure and a barycentric position of the human figure. 3. The feature point detecting means is a computing means for individually squaring a plurality of distances detected at a certain position on the contour, and an adding means for adding together a plurality of squared values obtained by the computing means. And the maximum position detection means for detecting the position on the contour where the added value by the addition means is maximum, and the posture estimation apparatus according to claim 1 or 2. 4. The camera according to claim 1, wherein the camera photographs a human image whose posture changes with time at predetermined intervals, and the feature point detecting means detects a feature point of each photographed human image. The posture estimation device according to any one of 1. 5. The posture estimating apparatus according to claim 4, wherein the initial posture of the person image is a posture in which the limbs do not overlap the body and the limbs do not intersect with respect to the camera. 6. A filter means for performing a Kalman filter process on the feature points of the current person image detected by the feature point detecting means, and the current person based on the processing result of the filter means and a first autoregressive model formula. The posture estimation apparatus according to claim 4, further comprising a feature point determination unit that determines a feature point of the image. 7. A feature point prediction unit that predicts a feature point of a next person image to be captured next time based on the processing result of the filter unit and a second autoregressive model formula, and the feature point prediction unit. Further comprising a partial image area setting means for setting a partial image area including the feature points on the next person image, wherein the distance detecting means is configured to detect the plurality of positions from each position on the contour included in the partial image area. The posture estimation device according to claim 6, which detects a distance to each of the reference positions. 8. A recording medium in which a posture estimation program for estimating the posture of a human image captured by a camera is recorded, wherein the posture estimation program includes a contour detecting step of detecting a contour of the human image, A distance detection step of detecting a distance from a position to each of a plurality of reference positions on the human image, and a characteristic point detection step of detecting a characteristic point of the human image based on the distance. Recording medium.