JP5767078B2 - Posture estimation apparatus, posture estimation method, and posture estimation program - Google Patents

Description

  The present invention relates to a posture estimation apparatus, a posture estimation method, and a posture estimation program that perform posture estimation of a person from image data captured by a camera.

  In recent years, research that performs posture estimation / motion analysis of a human body from image data without using a motion capture system or a distance image has been widely performed (for example, see Non-Patent Document 1). These studies are expected to be applied to a wide range of fields such as video surveillance, human interaction, robot motion control, CG animation production, and medical fields. In particular, if a three-dimensional posture can be estimated, the range of applications will be further expanded.

  However, when estimating the posture of the human body in the three-dimensional virtual space based on the image data, it is difficult to estimate the three-dimensional rotation angle of the joint of the human body at an arbitrary time. In particular, when estimating a three-dimensional posture from image data captured by a monocular camera, since the human body is expressed in two dimensions in the image data, information for expressing in three dimensions is lacking. As a result, problems such as inability to deal with the ambiguity of posture and inability to deal with shielding by limbs and obstacles arise.

  In order to cope with these problems, attention has been paid to a technique using prior knowledge about the movement of the human body when estimating. This prior knowledge is information for supplementing information that is lacking in order to express in three dimensions based on two-dimensional image data. The two-dimensional image data which is information for supplementation is composed of joint position coordinates on the two-dimensional image for estimating a three-dimensional posture. The joint position coordinates are obtained by tracking the image feature amount of the target location across the frames. As prior knowledge, for example, as in Non-Patent Document 2, one using a motion model obtained by extracting low-dimensional features from highly accurate three-dimensional motion data measured by an optical motion capture system is known. Using this motion model as prior knowledge, it is possible to estimate a three-dimensional posture from a joint position on a two-dimensional image.

Shimada, Nobutaka, Arita, Daisaku, Tamaki, Toru. "Model fitting of joint objects" IPSJ SIG CVIM Vol.154. Page.375-392. 2006. R. Urtasun, D. J. Fleet, and P. Fua. "3d people tracking with gaussian process dynam-ical models." CVPR, 2006.

  However, when using a motion model to supplement a three-dimensional posture from a two-dimensional joint position, the uncertainty of the three-dimensional posture increases as the uncertainty of the information on the two-dimensional joint position increases. In general, since a two-dimensional joint position is estimated based on information on colors and edges that can be acquired on an image, there is a problem that stable estimation is difficult. Such a problem is not limited to the human body and joints, but is a general problem regarding an object whose posture changes (for example, a human body or a robot) and a tracking target (for example, a joint or a movable part) of the object. is there.

  The present invention has been made in view of such circumstances, and performs a stable estimation of a two-dimensional tracking target, and accordingly, an improvement in the accuracy of the three-dimensional posture estimation, An object is to provide a posture estimation method and a posture estimation program.

  The present invention includes an image input unit that inputs an image of a posture estimation target, a template storage unit that stores a tracking target feature amount of the posture estimation target as template data, and a three-dimensional operation of the tracking target. By means of a 3D motion model storage means storing model data, a position candidate initial setting means for setting a plurality of initial position candidates for the tracking target with reference to the 3D motion model data, and the image input means Based on the weight of similarity obtained as a result of calculating the feature amount in the position candidate of the tracking target from the input image and comparing the feature amount with the template data stored in the template storage unit, the tracking target Position estimation means for estimating the position of the tracking target and outputting estimated information of the position of the tracking target, the weight of similarity, and the three-dimensional Position candidate setting means for newly setting the position candidate of the tracking target based on the cropping model data; estimation of the position of the tracking target by the position estimating means; and the position candidate of the tracking target by the position candidate setting means The tracking processing means for tracking the position of the tracking target, the estimation information of the position of the tracking target, and the three-dimensional motion model data are referred to by repeating the setting of the plurality of times. And a posture estimation means for estimating a three-dimensional posture.

  The present invention is characterized in that the posture estimation target is a human body and the tracking target is a joint.

  The present invention includes an image input unit that inputs an image of a posture estimation target, a template storage unit that stores a tracking target feature amount of the posture estimation target as template data, and a three-dimensional operation of the tracking target. A posture estimation method in a posture estimation apparatus including a three-dimensional motion model storage unit storing model data, wherein a plurality of initial position candidates are set for the tracking target with reference to the three-dimensional motion model data. A result of comparing the feature amount with the template data stored in the template storage unit by calculating the feature amount in the position candidate to be tracked from the position input initial setting step and the image input by the image input unit; Based on the obtained similarity weight, the position of the tracking target is estimated, and estimation information of the tracking target position is output. A position candidate setting step for setting again the position candidate of the tracking target based on the weight of the similarity and the three-dimensional motion model data, and the position of the tracking target by the position estimation step The tracking processing step of tracking the position of the tracking target by repeating the estimation and the setting of the position candidate of the tracking target by the position candidate setting step a plurality of times, the estimation information of the position of the tracking target, And a posture estimation step of estimating a three-dimensional posture of the posture estimation target with reference to three-dimensional motion model data.

  The present invention is characterized by causing a computer to execute the posture estimation method.

  According to the present invention, in addition to being able to stably estimate the tracking target (joint position), it is possible to estimate the three-dimensional posture of the posture estimation target simultaneously with the position estimation of the tracking target. The effect that the improvement of the accuracy can be realized is obtained.

It is a block diagram which shows the structure of one Embodiment of this invention. It is a flowchart which shows operation | movement of the two-dimensional joint tracking part 103 shown in FIG. It is a block diagram which shows the structure of the modification of one Embodiment of this invention. 4 is a flowchart illustrating an operation of a two-dimensional joint tracking unit 109 illustrated in FIG. 3. It is explanatory drawing which shows the general concept of an object tracking method.

  Hereinafter, an attitude estimation apparatus according to an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing the configuration of the embodiment. In this figure, reference numeral 101 denotes an analysis video storage unit in which video data to be analyzed for posture is stored. As an example of the video data, the video data is composed of video data captured by one or more uncalibrated cameras. Reference numeral 102 denotes an image input unit that inputs video data stored in the analysis video storage unit 101, and converts the input video data into a format for performing an estimation process. This conversion is, for example, hue conversion or frame interpolation. Reference numeral 103 denotes a two-dimensional joint tracking unit that extracts the coordinates of the joint positions of the human body from the video data input by the image input unit 102 using image feature amounts and the like, and performs tracking processing for all frames to be analyzed. . Reference numeral 104 denotes a three-dimensional posture estimation unit that estimates a three-dimensional posture from the two-dimensional joint position coordinates of the human body tracked by the two-dimensional joint tracking unit 103.

  Reference numeral 105 denotes a template storage unit that stores information such as characteristics and positions of regions to be tracked as templates. The template format depends on the method used by the two-dimensional joint tracking unit 103. The two-dimensional joint tracking unit 103 performs tracking processing from the image information of each frame of the video data input by the image input unit 102 and the template stored in the template storage unit 105. Reference numeral 106 denotes a posture information storage unit that holds the three-dimensional posture information estimated by the three-dimensional posture estimation unit 104. Reference numeral 107 denotes a prior 3D motion model storage unit that stores prior 3D motion model information used when the 3D posture estimation unit 104 estimates a 3D posture. As the motion model, for example, as shown in Non-Patent Document 2, a representation of three-dimensional motion with low-dimensional features can be applied. Reference numeral 108 denotes an estimated information storage unit that stores estimated state information of the tracking target estimated by the two-dimensional indirect tracking unit 103. The three-dimensional posture estimation unit 104 estimates the three-dimensional posture with reference to the two-dimensional joint position information in the estimation information storage unit 108 and the motion model information in the prior three-dimensional motion model storage unit 107. The three-dimensional posture estimation processing in the three-dimensional posture estimation unit 104 is performed using a known method.

  Here, a processing operation for tracking the two-dimensional joint position using a particle filter will be described with reference to FIG. When using a particle filter, as shown in FIG. 5, a large number (N) of position candidates to be tracked are generated as hypotheses. First, the hypothesis position is initialized (STEP 0). At each time, the feature quantity at the hypothesis position is compared with the template feature quantity to be tracked, and the similarity is calculated as a weight (STEP 1). Next, a plausible one is selected from N hypotheses and set as an estimated position (state) at that time (STEP 2). Subsequently, as a preparation for the next time, a hypothesis having a large weight is duplicated and a small hypothesis is eliminated (STEP 3). Then, the hypothesis is moved in accordance with the state transition model to be tracked (STEP 4), the process returns to STEP 1 and the process is repeated. Although it is desirable to define the state transition model based on the motion model of the object, constant velocity linear motion or random walk is assumed when it is difficult to formulate the motion model.

Next, the operation of the two-dimensional joint tracking unit 103 shown in FIG. 1 will be described with reference to FIG. FIG. 2 is a flowchart showing the operation of the two-dimensional joint tracking unit 103 shown in FIG. First, the two-dimensional joint tracking unit 103 sets time t to 0 (step S1), extracts a frame at time t from the analysis target video data stored in the analysis video storage unit 101, and sets the joint to be tracked. The initial position [x ₀ , y ₀ ] is designated (step S2). For this designation, for example, image processing such as manual designation, skin color detection, and edge detection can be used.

Next, the two-dimensional joint tracking unit 103 calculates the feature value I _temp of the joint position designated in step S2, and stores this in the template storage unit 105 as a template (step S3). As this feature amount, for example, an arbitrary one such as a color histogram or an HOG feature amount can be used. Subsequently, the two-dimensional joint tracking unit 103 sets the position of the tracking target to the state Φ (t) = [x (t), y (t)], generates N hypotheses, and specifies the initial specified in step S2 Initialization is performed such that the distribution is centered on the position [x ₀ , y ₀ ] (step S5). At this time, for example, a normal distribution having an average [x ₀ , y ₀ ] can be used.

Next, the two-dimensional joint tracking unit 103 determines whether or not the time t is less than or equal to the number of analysis frames T (step S6). If the time t is not less than or equal to the number of analysis frames T, the process ends. On the other hand, if the time t is equal to or less than the number of analysis frames T, a frame image with t = T is extracted from the analysis target video data stored in the analysis video storage unit 101 (step S7). Then, the two-dimensional joint tracking unit 103 performs a hypothesis Φ ⁱ (t), i = 1, 2,. . , N, the feature quantity I ⁱ (t) is calculated (step S10).

Next, the two-dimensional joint tracking unit 103 compares the obtained I ⁱ (t) with the template feature value I _temp to calculate the similarity weight w ⁱ (t) (step S11). As the weight, for example, a distance between I ⁱ (t) and I _temp can be used. Subsequently, the two-dimensional joint tracking unit 103 estimates the state Φ ^* (t) at the time t from the calculated weight w ⁱ (t) (step S12). This estimation can be performed by selecting a hypothesis having the largest weight or taking a weighted average. Then, the estimated state information is stored in the estimated information storage unit 108.

Next, the two-dimensional joint tracking unit 103 replicates hypotheses with large weights according to the weights w ⁱ (t) and deletes small hypotheses so that hypotheses are gathered around the estimated state. Sampling is performed (step S13). Subsequently, as a preparation for the next time, the two-dimensional joint tracking unit 103 samples a hypothesis based on the state transition model to be tracked (step S14). This sampling is performed by, for example, a constant velocity straight line or random walk model. Then, the two-dimensional joint tracking unit 103 adds 1 to the time t (step S15), returns to step S6, and repeats the process.

  Next, a modification of the posture estimation apparatus shown in FIG. 1 will be described with reference to FIG. FIG. 3 is a block diagram showing a configuration of a modification of the posture estimation apparatus shown in FIG. In this figure, the same parts as those in the posture estimation apparatus shown in FIG. The posture estimation apparatus shown in FIG. 3 is different from the posture estimation apparatus shown in FIG. 1 in that a two-dimensional joint tracking unit 109 is provided instead of the two-dimensional joint tracking unit 103. The two-dimensional joint tracking unit 109 refers to the video data input by the image input unit 102, the template information stored in the template storage unit 105, and the motion model information stored in the prior three-dimensional motion model storage unit 107. Tracking process.

  Next, the operation of the two-dimensional joint tracking unit 109 shown in FIG. 3 will be described with reference to FIG. FIG. 4 is a flowchart showing the operation of the two-dimensional joint tracking unit 109 shown in FIG. 4, parts that are the same as the operations shown in FIG. 2 are given the same reference numerals, and descriptions thereof are omitted. The operation shown in FIG. 4 is different from the operation shown in FIG. 2 in that steps S4, S51, S8, S9, S121, and S141 are provided.

First, the two-dimensional joint tracking unit 109 sets time t to 0 (step S1). A frame whose time t is 0 is extracted from the analysis target video data stored in the analysis video storage unit 101, and the initial position [x ₀ , y ₀ ] of the joint to be tracked is designated (step S2).

Next, the two-dimensional joint tracking unit 109 calculates the feature value I _temp of the joint position designated in step S2, and stores this in the template storage unit 105 as a template (step S3). Subsequently, the two-dimensional joint tracking unit 109 specifies the initial viewpoint V _{0 of the} video to be analyzed. This initial viewpoint V ₀ is used when a three-dimensional posture is projected in two dimensions.

Next, the two-dimensional joint tracking unit 109 displays the three-dimensional posture Y (t) to be tracked and the projection viewpoint V (t) projected onto the two-dimensional image in the state Φ (t) = [Y (t), V (t) ], N hypotheses are generated, and Y (t) is stored in the prior three-dimensional motion model storage unit 107 so that V (t) is distributed around the initial viewpoint V ₀ specified earlier. Initialization is performed so that the distribution is centered on the current motion information (step S51).

Next, the two-dimensional joint tracking unit 109 determines whether or not the time t is less than or equal to the number of analysis frames T (step S6). If the time t is not less than or equal to the number of analysis frames T, the process ends. On the other hand, if the time t is equal to or less than the number of analysis frames T, a frame image with t = T is extracted from the analysis target video data stored in the analysis video storage unit 101 (step S7). Then, the position of the center of gravity of the human body is calculated from the frame image at time t (step S8). This can be expressed by, for example, the position of the waist, and can be performed by using the center of gravity of the silhouette area of the human body region. Subsequently, the two-dimensional joint tracking unit 109 applies the hypothesis Φ ⁱ (t), i = 1, 2,. . , N two-dimensional joint positions [x ⁱ (t), y ⁱ (t)] are calculated from the three-dimensional posture Y ⁱ (t) and the projection viewpoint V ⁱ (t) (step S9).

Next, the two-dimensional joint tracking unit 109 performs a hypothesis Φ ⁱ (t), i = 1, 2,. . Calculates the feature amount ^I i in the case of N (t) (step S10), and calculates the resulting ^I i (t) and its similarity by comparing the template feature amount _{I temp} weights ^w i (t) (Step S11). Then, the two-dimensional joint tracking unit 109 determines the state Φ ^* (t) = [Y ^* (t), V ^* (t)] and the two-dimensional joint position [x at time t from the calculated weight w ⁱ (t). ^* (T), y ^* (t)] is estimated (step S121). Here, the estimated state information and two-dimensional joint position information are stored in the estimated information storage unit 108.

このとき、

である。 Next, the two-dimensional joint tracking unit 109 copies the hypothesis having a large weight according to the size of the weight w ⁱ (t) and deletes the small hypothesis so that the hypothesis is gathered around the estimated state. Sampling is performed (step S13). Subsequently, as a preparation for the next time, the two-dimensional joint tracking unit 109 samples a hypothesis based on the state transition model to be tracked (step S141). Here, the state transition model is defined from motion information held in the prior three-dimensional motion model storage unit 107. For example, when the motion model of Non-Patent Document 2 is used, the state transition of the three-dimensional posture Y ⁱ is given by Equations (1) and (2).

At this time,

It is.

このとき、α，βは学習により求めるパラメータである。 Here, X (n) is a low-dimensional representation of the three-dimensional posture Y (n). K _X (X ^* ) and k _Y (X ^* ) are vectors having k _X (X ^* , X _i ) and k _Y (X ^* , X _i ) as the i-th element, for example, (5) It is defined by an appropriate function as shown in Equation (6).

At this time, α and β are parameters obtained by learning.

The state transition of the projection viewpoint V ^i, for example, defined using the translation amount of change in the rotation of the pre-learned behavior information from the waist position. More specifically, when the projection viewpoint is expressed as V = [θ, φ], the viewpoint changes dθ (n) and dφ (n) are preliminarily obtained from the motion data closest to the three-dimensional motion data Y at the same time. From the three-dimensional motion model storage unit 107, the viewpoint change amount dθ _t (n) = t (n) −t (n−1), from the translation t (n−1) and rotation r (n−1), respectively. dθ _r (n) = r (n) −r (n−1) is obtained. The viewpoint change as a whole is defined by dθ (n) = dθ _t (n) −dθ _r (n). The same applies to φ.

  Next, the two-dimensional joint tracking unit 109 adds 1 to the time t (step S15), returns to step S6, and repeats the processing.

  As described above, when estimating the two-dimensional joint position, the tracking of the two-dimensional joint position is performed using not only the observed image information but also the previously learned three-dimensional motion information. In addition, since it is possible to perform the three-dimensional motion that has been estimated from the tracking result of the two-dimensional joint position at the same time as the tracking of the two-dimensional joint position, the accuracy of the three-dimensional posture estimation can be improved. it can.

  The program for realizing the functions of the processing unit in FIGS. 1 and 3 is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read by the computer system and executed. An estimation process may be performed. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer system” includes a WWW system having a homepage providing environment (or display environment). The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Further, the “computer-readable recording medium” refers to a volatile memory (RAM) in a computer system that becomes a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. In addition, those holding programs for a certain period of time are also included.

  The program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The program may be for realizing a part of the functions described above. Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, what is called a difference file (difference program) may be sufficient.

  As mentioned above, although embodiment of this invention has been described with reference to drawings, the said embodiment is only the illustration of this invention, and it is clear that this invention is not limited to the said embodiment. is there. Accordingly, additions, omissions, substitutions, and other modifications of components may be made without departing from the spirit and scope of the present invention.

  It can be applied to applications where it is essential to estimate the posture of a person from image data captured by a camera.

  DESCRIPTION OF SYMBOLS 101 ... Analysis video memory | storage part, 102 ... Image input part, 103 ... Two-dimensional joint tracking part, 104 ... Three-dimensional attitude | position estimation part, 105 ... Template memory | storage part, 106 ... Attitude Information storage unit, 107 ... Pre-three-dimensional motion model storage unit, 108 ... Estimated information storage unit

Claims (4)

An image input means for inputting an image in which the posture estimation target is photographed;
A template storage means in which the feature quantity of the tracking target possessed by the posture estimation target is stored as template data;
3D motion model storage means storing the 3D motion model data to be tracked;
Position candidate initial setting means for setting a plurality of initial position candidates for the tracking target with reference to the three-dimensional motion model data;
From the image input by the image input means, a feature amount in the position candidate to be tracked is calculated, and the feature amount is compared with the template data stored in the template storage means to obtain a similarity weight obtained as a result. A position estimating means for estimating the position of the tracking target and outputting estimation information of the position of the tracking target;
Based on the weight of similarity and the three-dimensional motion model data, the position candidate of the tracking target is set again, and is set again based on a state transition model defined based on the three-dimensional motion model data Position candidate setting means for moving the position candidate;
Tracking processing means for tracking the position of the tracking target by repeating the estimation of the position of the tracking target by the position estimating means and the setting of the position candidate of the tracking target by the position candidate setting means a plurality of times;
An attitude estimation apparatus comprising: estimation information of the position of the tracking object and attitude estimation means for estimating the 3D attitude of the attitude estimation object with reference to the 3D motion model data.   The posture estimation apparatus according to claim 1, wherein the posture estimation target is a human body and the tracking target is a joint. Image input means for inputting an image in which the posture estimation target is photographed, template storage means for storing the tracking target feature quantity of the posture estimation target as template data, and the tracking target three-dimensional motion model data stored A posture estimation method in a posture estimation device comprising the three-dimensional motion model storage means,
A position candidate initial setting step for setting a plurality of initial position candidates for the tracking target with reference to the three-dimensional motion model data;
From the image input by the image input means, a feature amount in the position candidate to be tracked is calculated, and the feature amount is compared with the template data stored in the template storage means to obtain a similarity weight obtained as a result. A position estimating step for estimating the position of the tracking target and outputting estimation information of the position of the tracking target;
Based on the weight of similarity and the three-dimensional motion model data, the position candidate of the tracking target is set again, and is set again based on a state transition model defined based on the three-dimensional motion model data A position candidate setting step for moving the position candidate;
A tracking processing step of tracking the position of the tracking target by repeating the estimation of the position of the tracking target by the position estimation step and the setting of the position candidate of the tracking target by the position candidate setting step a plurality of times;
A posture estimation method, comprising: estimation information of the position of the tracking target, and a posture estimation step of estimating a three-dimensional posture of the posture estimation target with reference to the three-dimensional motion model data.   A posture estimation program for causing a computer to execute the posture estimation method according to claim 3.

Images