WO2004094943A1

WO2004094943A1 - Motion capturing method, motion capturing device, and motion capturing marker

Info

Publication number: WO2004094943A1
Application number: PCT/JP2003/016080
Authority: WO
Inventors: Hiroshi Arisawa; Kazunori Sakaki
Original assignee: Hiroshi Arisawa; Kazunori Sakaki
Priority date: 2003-04-22
Filing date: 2003-12-16
Publication date: 2004-11-04
Also published as: AU2003289108A1; JPWO2004094943A1

Abstract

In order to acquire the positional information and the angular information of the components of an articulated body, e.g. a human body, using a small number of markers, restrictive conditions of each component of the articulated body are utilized. A candidate attitude of any other component satisfying the restrictive conditions is extracted from the positional information of some component obtained from a small number of markers and then the angular information is acquired by superposing the images of the candidate attitude and the articulated body.

Description

Specification of motion capture method, motion capture device and marker for motion capture

The present invention relates to a motion canopy that captures the motion of an object in the real world on a computer. Background art

In various fields such as industry, medicine, and sports, it is attempted to take objects in the real world onto a computer and perform various processing on the computer. For example, information on the movement of a person or object or the shape of an object is acquired and used for analysis of movement of a person or object, formation of a virtual space, and the like.

However, because people and objects that want to be evaluated actually work in various environments, they are not necessarily suitable places for acquiring such information. In addition, in order to directly capture events that are taking place in the real world on a computer, it is necessary not to give time to an object such as a person or object and its surrounding environment, and to cause no trouble in work.

Heretofore, as a method of capturing such an object in the real world on a computer, there is known a motion canopy. This motion capture simulates the motion of a person or other moving body.

For example, mechanical, magnetic and optical types are known as motion caps. In the mechanical motion canopy, an angle detector or pressure sensor is attached to the performer's body, and the bending angle of the joint is detected. The motion of a performer is detected, and in the case of a magnetic motion cap, a magnetic sensor is attached to each part of the performer's body, the performer is moved in an artificially generated magnetic field, and the density and angle of magnetic lines of force are determined. By detecting it with a magnetic sensor, the movement of the performer is detected by deriving the absolute position where the magnetic sensor is present.

In addition, in the case of an optical motion canopy, a marker is attached to the place where it is desired to measure the movement of the performer's body, and the movement of each part is measured from the position of the marker by imaging this marker with a camera.

In either method, it is necessary to attach a detector, sensor or marker to the subject, which is a burden on the subject. Even in optical motion cameras that can obtain high accuracy, it is necessary to attach dozens of markers in order to obtain the motion of the whole human body, and its application is limited.

On the other hand, there is also proposed a motion canopy that does not burden the subject. This motion capture captures the motion of the human body in a non-contact manner by correlating with a virtual three-dimensional human body model using images of a multi-viewpoint camera. There are, for example, documents 1 and 2 as a method for matching such multi-view images with models. In Document 1, the posture is estimated by overlaying and evaluating a three-dimensional model and a silhouette image obtained by extracting only the subject in each image. In addition, '2' is a method of determining the difference between the current image and the next image, and using this difference to determine the posture.

Literature: Transactions of the Institute of Electronics, Information and Communication Engineers D— II Vol. J 8 2 — D— I I N o. 10 p. 1 7 3 9-1 7 4 9 1 9 9 9 1 9

"Attitude determination of person by movement and formation model"

Reference 2: Transactions of the Institute of Electronics, Information and Communication Engineers D— II Vol. J 8 0 — D— IIN 0. 6 pp. 1 5 8 1-1 5 8 9 1 9 9 June 7 Disclosure of the invention

The above-mentioned non-contact type motion capture that does not use a marker needs to acquire an element for determining the motion from the image, and therefore has not reached a practical level in terms of motion extraction accuracy and analysis processing time.

This is because the number of postures that can be taken by the human body is huge, so the amount of data handled in image processing is large, and it is difficult to extract accurate postures at high speed. For example, when the human body is represented by a three-dimensional human body model as shown in FIG. 13, it is composed of components having degrees of freedom in two or three axis directions, and the posture of the human body model rotates these components. Is represented by The components form a hierarchical structure with each other, and the rotational movement of each component is reflected in the lower components. Therefore, the number M of cases that each component can take is represented by M = P ^n, where ⁿ is the number of degrees of freedom and P is the number to be evaluated in each degree of freedom.

Even in the case of the simple human body model shown in Fig. 13, since all the degrees of freedom are 2 9, the evaluation number M is M = 1 0 0 ^{2 when} it is evaluated by rotating 100 degrees for each degree of freedom. It will be ^nine .

As described above, in order to obtain the posture of an articulated body in which a plurality of components such as the human body are connected with each other by a non-contact motion capture without using a marker, the number of postures is large. This requires a huge amount of arithmetic processing, making it difficult to process accurately and at high speed. Furthermore, it is difficult to process moving articulated bodies in real time.

Therefore, there is a problem that in the case of the conventionally proposed optical motion canopy, the motion burden on the subject is large by attaching a large number of markers in the motion canopy using a force. Although there is no burden due to marker attachment in motion markers without markers, the number of possible postures is large, so analysis processing time and extraction accuracy There is a problem in point.

Therefore, in order to determine the posture of an articulated body such as a human body in real time, there is a need for a motion canopy that has a small burden on the subject, has a short analysis processing time, and has high extraction accuracy.

Therefore, the present invention is intended to solve the above-mentioned conventional problems and to reduce the burden of attaching a marker in motion capture, shorten the analysis processing time, and obtain high extraction accuracy.

The present invention reduces the load due to marker attachment by reducing the number of markers attached, and can adopt an attitude that can be taken by using positional information obtained by the markers and constraints that relate the components. The number is reduced, which shortens the analysis processing time and obtains high extraction accuracy.

The present invention includes a motion capture method, each aspect of a motion capture device, and a marker suitable for the motion capture.

In the determination of the posture and operation of an articulated body in which a plurality of components are jointed with each other, the posture of the articulated body can be determined by the positions of the components constituting the articulated body and the angle with each other. The movement of the articulated body can be regarded as a temporal change of this posture. Therefore, the information that motion canopy wants to acquire is the position information and angle information of the components that make up the articulated body.

The motion capture of the present invention obtains positional information and angular information of components of an articulated body such as a human body using a small number of markers.

Here, each component of the articulated body has constraints on the distance (length) of the components and the angular relationship that can be taken with each other. For example, the upper arm and the forearm constituting the human body are connected by an elbow joint, and the length of the upper arm and the forearm, the distance between the upper arm and the forearm, and the angle that the forearm can take with the upper arm are limited due to human body structure. You can not take positions or angles beyond this limit. Therefore, these distances and angles become possible constraints of the component. Usually, in order to obtain position information and angle information of components of an articulated body such as a human body using markers, it is necessary to attach a large number of markers to each component. Sufficient position information and angle information can not be obtained, and the posture of the articulated body can not be determined. The present invention uses the constraints of each component of the articulated body in order to obtain position information and angle information of the component of the articulated body such as a human body using a small amount of force. The motion capture of the present invention extracts candidate postures of other components that satisfy this constraint condition from position information of certain components obtained from a small number of markers, and generates images of the posture candidates and the articulated body. Angle information is acquired by superimposing with. Thus, the number of postures to be evaluated is reduced by narrowing down the postures satisfying the constraint condition determined by the position information from among a large number of postures. In this way, by reducing the number of postures to be evaluated in posture determination, the analysis time is shortened and the accuracy of posture extraction is increased.

Therefore, according to the first aspect of the motion capture method of the present invention, in the determination of the posture and motion of an articulated body in which a plurality of components are jointed with each other, a specific portion of the component constituting the articulated body is determined. Position information is determined, and angle information between each component that satisfies a constraint condition determined by the position information is determined based on the position information and a constraint that relates the plurality of components to each other. Determine body posture and movement.

The position information is acquired from the specific part of the component, and the other angle information is acquired using the constraint condition, whereby the articulated body is obtained without acquiring the position information for all the components provided in the articulated body. You can get your attitude.

The second aspect of the motion capture method of the present invention comprises a plurality of components In the determination of the posture and motion of an articulated body in which the 's are connected to each other, distance and angle constraints that relate a plurality of components to each other are determined in advance, and at least one component is connected A marker which can be individually identified is provided at a specific site of the component to be obtained, and an image of an articulated body including this marker is obtained. The position information of the specific part is obtained from the position of the marker in the obtained image. In addition, the component angle information is a component that satisfies the constraints defined by the line position information based on the image of the articulated body in the image, the position information determined, and the constraints that relate the plurality of components to each other. The angle information corresponding to the image of the articulated body is obtained by finding the angle information among them. From the position information and the angle information, the posture and motion of the articulated body are determined.

Each marker can be individually identified, and the marker attachment position can be identified by identifying the marker. Also, the component to which the force is attached should be placed with the other component in between. As a result, the force to be attached to an articulated body such as a human body can be disposed by thinning out, and the number of markers required for posture acquisition can be reduced.

According to a third aspect of the motion capture method of the present invention, there is provided a constraint that relates a plurality of components to each other in determining the posture and motion of an articulated body in which the plurality of components are articulated to each other, and A plurality of models representing postures of body components are predetermined, and an articulated body including individually distinguishable markers provided at specific portions of the components connected with at least one component interposed therebetween. Ask for an image. The position information of the specific part is obtained from the position of the marker in the obtained image. In addition, component angle information is extracted from a plurality of models that conform to constraints determined by position information, and a model that is most approximate to the image of the articulated body in the image is extracted from the extracted models, The angle information of the components of the articulated body is obtained from the angles between the components of the extracted model. Posture of the articulated body by this position information and angle information And determine the action.

The motion capture without using a conventional marker is to evaluate the superposition with the image acquired for all models without using this constraint condition, and the number of postures that hierarchically connected components can take. Is an enormous number because it increases with the number of components and the multiplier of their degrees of freedom. If a more detailed model is set by increasing the number of components constituting an articulated body, the number of possible postures will increase.

On the other hand, in the motion canopy of the present invention, the number of models as attitudes to be evaluated can be reduced, so that the analysis time can be shortened, and it becomes closer to real-time (real-time) motion analysis. Is possible.

In the fourth aspect of the motion canopy according to the present invention, an image of a marker attached to a human body and a specific part of the human body is acquired, and the individual force is individually identified and extracted from this image to detect the specific part of the human body. The position information is determined, and the angle information of the component of the human body is determined by image matching between the model extracted based on the position information of the specific part from among the prepared models and the acquired image, and the determined position information and angle information Determine the posture of the human body.

Models are extracted by selecting at least one model that meets the human posture constraints determined by the position of a specific part. Image alignment between the image and the model is performed by selecting from among the extracted models the model that most closely approximates the human body image in the image. For the selection of an approximate model, for example, with respect to the vertexes provided on the component, the distance between the image and the model is obtained for each posture, and a model that can obtain the posture with the shortest distance is selected. Angle information can be obtained from the angle of the component in the selected model.

In the motion capture of the present invention, the marker emits light to identify the marker. 03 016080

8

It is performed by the combination of at least two different colors, and the position of a single force is determined from the position of one of the color combinations. Thus, individual identification of each force can be performed and position information can be obtained.

The specific site can be the waist, elbow, and knee in the human body. Even when the left and right sides of each part are provided with an equal force, the number of markers is only six, and a small amount of force can reduce the burden on the subject. A first aspect of the motion capture device of the present invention is a motion capture device for determining the posture and operation of an articulated body in which a plurality of components are connected to each other by articulation, the motion capture device comprising the components constituting the articulated body. Based on position detection means for obtaining position information of a specific part, position information, and constraints that relate the plurality of components to each other, angle information between each component that satisfies the constraint defined by the position information is obtained. An attitude of the articulated body is determined based on the detected position information and angle information.

A second aspect of the motion capture device of the present invention is a motion capture device for determining the posture and motion of an articulated body in which a plurality of components are articulated to one another, wherein a constraint relating the plurality of components to one another A storage means for pre-storing a plurality of models representing the conditions and postures of the components of the articulated body, and individually distinguishable at specific parts of the components connected across at least one component Image acquisition means for obtaining an image of an articulated body including various markers, position information detection means for obtaining position information of a specific site from the position of the marker in the image, multiple models determined by the position of the specific site The model selection means for selecting a model that meets the constraint conditions of the joint body, and the model closest to the image of the articulated body in the image is extracted from the selected models, Configuration from the angle between the components of the out model of the multi-joint body It is configured to have matching means for obtaining angle information of the element, and the posture and motion of the articulated body are determined by the position information and the angle information.

In addition, a marker suitable for the motion spacer of the present invention is a marker for identifying a specific part of the articulated body in the motion capsule, which determines the posture and operation of the articulated body in which a plurality of components are connected to each other. A first light emitting diode disposed at the center, and a plurality of second light emitting diodes disposed equiangularly spaced around the first light emitting diode; The light emitting diode and the second light emitting diode have different light emitting colors.

When light emission from the first and second light emitting diodes is imaged from above, an annular image consisting of the light emitting color of the first light emitting diode at the center and the light emitting color of the second light emitting diode at the outer periphery By combining these two colors, it is possible to identify each marker individually, and to identify the site on the articulated body where the marker is provided.

In addition, it is possible to prevent change in color information due to changes in shadows and lighting by using the power of the light emitter, and by using a combination of two colors, it is possible to use the same color information other than the object. Mistakes caused by the presence of backgrounds or objects can prevent degradation in extraction accuracy.

By disposing a plurality of second light emitting diodes at equal angular intervals around the first light emitting diode, it is possible to reduce the image dropout regardless of the posture of the articulated body.

In addition, a light shield can be provided between the first light emitting diode and the second light emitting diode. This light shield can prevent the mixing of the two colors on the image. When two colors are mixed, they are identified as different colors, which makes it difficult to identify markers and causes false recognition. The light shield is on the image Prevents mixing of two colors and makes marker identification easier. Brief description of the drawings

FIG. 1 is a view for explaining an outline of a motion canopy according to the present invention, FIG. 2 is a flow chart for explaining a procedure of attitude determination by the motion canopy according to the present invention, and FIG. FIG. 4 is a schematic view of components for explaining a procedure of posture determination by motion capture, FIG. 4 is a diagram for explaining components of an articulated body, and FIG. FIG. 6 is a flow chart for explaining an example of a calculation procedure of position information, FIG. 6 is a color model of a bi-hexagonal pyramid of HSI color system, and FIG. 7 is an extraction of a specific part and position information FIG. 8 is a view for explaining an example of the calculation procedure of the present invention, FIG. 8 is a view for explaining a motion capture device of the present invention, and FIG. 9 is a posture of a human body by the motion capture of the present invention. , Work Figure 10 is a hierarchy diagram for explaining the mounting position of the force when the posture and motion of the human body are determined by the motion canopy of the present invention. FIG. 11 is a diagram for explaining a marker suitable for the motion of the present invention, and FIG. 12 is a diagram for explaining a marker suitable for the motion capture of the present invention. Thus, FIG. 13 is a view showing an example of a human body model. BEST MODE FOR CARRYING OUT THE INVENTION

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

FIG. 1 is a view for explaining the outline of the motion chart of the present invention. First, multi-view video is acquired in Fig.1. Multi-view images are acquired by arranging multiple cameras. The images taken by each camera are For example, synchronization is acquired in frame units (A in the figure). The acquired multi-view video is processed almost in real time, and the position information and angle information of the articulated object as the object are obtained, and the posture and motion are obtained.

The position information is obtained by extracting a specific part of the multi-joint body that is the object from each image of the multi-view video, and from the position (B in the figure). The specific site is any site defined on the component in the articulated body.

Since the position information obtained by the motion canopy of the present invention relates to a specific part defined on a component in a multi-segment body, it is not possible to determine the posture of the other component by this position information alone. .

The present invention uses angle information of each component in addition to position information as an element for determining the posture of the object.

In the motion capture of the present invention, each image of the multi-view video is superimposed on a predetermined model (D in the figure), a model matching the image is extracted, and an angle from the angle of each component of the model is extracted. Finding information (C in the figure), Extraction of the model that most closely matches the image by superimposing this image on the model is called matching.

The present invention uses the constraint that determines the relationship between the constituent elements of the articulated body in extracting the angle information. In an articulated body, there is a relationship between the specific part and each component according to the characteristics of the articulated body, and once the position of the specific part is determined, the other components are defined with the relationship as the constraint and the range It is not possible to take an internal attitude and take an attitude that deviates from this constraint. The motion capture of the present invention reduces the time required for matching by narrowing down the model to be matched using the position information of the specific part and the above-mentioned restriction condition.

Hereinafter, the procedure of the attitude determination by the motion canopy according to the present invention will be described using the flowchart shown in FIG. 2 and the schematic view of the components shown in FIG. Do.

A multi-viewpoint image of the articulated body is acquired by a plurality of cameras arranged around the object, the articulated body. Multi-viewpoint images can be acquired as synchronized frame images. In the following processing, a three-dimensional position and angle are obtained from a plurality of images acquired by these multi-viewpoint images, but in FIG.

Fig. 3 (a) simply shows an image acquired in multi-view video. The articulated body shown in the figure shows a configuration example in which three components are connected so as to be able to change the angular relationship of each other by joints.

FIG. 4 is a view for explaining the components of this articulated body. Here, the articulated body 10 is composed of the components 1 1, 1 2 and 1 3, and the component 1 1 and the component 1 2 are rotatably articulated at the joint 1 4, and the component 1 2 and the component 1 2 The component 1 3 is pivotably articulated at the joint 1 5. In this multi-segment body 10, one end of the component 11 is designated as a specific site 16 and one end of the component 13 is designated as a specific site 17. These specific parts are provided in the components 1 1 and 1 3 sandwiching the component 1 2, but if they are connected components, there may be one or more components placed in between. it can. Here, the lengths of the respective components 1 1, 1 2, and 13 or the distances between the respective components are determined from the structure constituting an articulated body such as a human body.

Also, because these lengths and distances are determined, for example,

Once the spatial positions of 16, 17 are determined, the angular relationships among the component houses 1, 1, 2, and 13 are limited within a certain range. For example, the range of angles 0 1 0 that can be taken by component 1 1 at a specific site 1 6 is 1 0 1 1 to θ 1 2 2, and the angle 0 4 0 at which components 1 3 can take at a specific site 1 7 The range is from 1 0 4 1 to 0 4 2. Also, the range of angles 0 2 0 that component 1 2 can take with respect to component 1 1 at joint 1 4 is 1 0 2:! ~ 0 2 2 The range of the angle 0 3 0 that the component 1 3 can take on the component 1 2 at the joint 1 5 is − Θ 3 1 to 0 3 2. Here, the clockwise angle is shown as a negative angle. In addition, in FIG. 4, in order to extract a specific site | part from an image, marker M1 shall be provided in specific site | part 16 and marker M2 shall be provided in specific site | part 17. FIG.

As described above, each component constituting an articulated body is limited in possible postures when the position of a part is determined, and these lengths and angles may be regarded as possible posture constraints. it can. Although the relationship between the components shown in Fig. 4 is expressed in two dimensions, these relationships and constraints are set in three dimensions by the image obtained from the multi-viewpoint image (step S l).

First, position information is obtained from the image acquired in the process of step S 1. In order to obtain position information, a specific part is extracted from the image (step S 2), and position information of the extracted specific part is obtained. Figure 3 (b) simply shows the state in which the specific part is extracted from the image of the articulated body in Figure 3 (a). The position information (X1, y1, z1) is obtained by extracting the force M 1 provided at the specific portion 16, and the force M 2 provided at the specific portion 17 is extracted. The position information (x2, y2, ζ2) is obtained.

Here, an example of the extraction procedure of the specific part and the calculation procedure of the position information will be described using the flowchart of FIG. 5 and the explanatory diagram of FIG. 7.

An image obtained by an imaging means such as a CCD camera is usually obtained as an RGB signal. This RGB signal represents the signal strength by, for example, a gradation value of 0 to 25 steps, but since this gradation value includes elements such as brightness and color tone in a mixed form, Markers provided can not be identified and extracted by color.

The motion capture of the present invention identifies and extracts markers provided at each specific site by color-coding the markers. Identify the rank. Therefore, markers in the image need to be identified by color.

Therefore, the R G B signal of the image data is converted to the H S I signal. The H S I color system has three attributes: hue (H: hue), saturation (S: saturation), and lightness (I: intensity).

The following equation is known as a method of converting the R G B color system to the H S I color system.

V = max (R, G, B), X = min <R, G, B)

S = (V-X) / V

And when

Hue H is

If R = V, then Η = (π / 3) * (b-g)

When G = V, H = (C / 3) * (2 + r-b)

When B = V, H = (7t / 3) * (4 + g-r)

However,

r = (V-R) / (V-X)

g = (V-G) / (V-X)

b = (V-B) / (V-X)

Figure 6 is a color model of the double hexagonal pyramid in the H S I color system. For example, hue (H) can be represented by angle values in the order of yellow, green, cyan, lue and magenta, where red is 0 °. The number of hues is an example and may be another number of hues. In addition, lightness (V) can also be expressed numerically.

Therefore, for hue (H) and lightness (V), in order to identify hue (H) and lightness (V) in advance according to the color emitted by the predetermined power, threshold values are determined and converted. The obtained hue signal and lightness signal are 0

15

By sorting by value, it is possible to identify individual values grouped by color. Here, the saturation is not used to identify the power (step S 1 1).

Therefore, the lightness signal (V) obtained by the conversion is selected by comparing it with a preset threshold value, and the high luminance part in the image is evaluated. Fig. 7 (a) schematically shows an example of extracting a high-intensity part from the image (step S12).

Next, based on the color information of the marker of the specific part, the hue signal (H) is selected by comparing with the threshold of the hue set in advance, and the specific part (marker from the high brightness part in the image Extract the region where) exists. Fig. 7 (b) schematically shows an example in which a specific part is extracted from the high-intensity part in the image (step S13).

In extraction by hue, the color part of a specific part such as a marker has an area, and therefore, it is detected as an area across multiple pixels on the image. Therefore, in order to determine the position of the specific part, the position of the specific part is calculated from the area of the specific part detected in step S13. For calculating the position of the specific part, any calculation method such as calculation of peak position of luminance, calculation of barycentric position of region, calculation of barycentric position weighted using luminance can be used.

A light emitting body such as a light emitting diode does not necessarily have a uniform light emitting state, and even within the same marker, there may be a difference in color between the light emitting portion and the periphery thereof. Due to this color change, the originally connected area may be divided. In order to compensate for this, processing to expand or contract the area on the image may be performed. A plurality of position information of these specific parts are obtained by cameras of different arrangement positions by multi-viewpoint images, and are performed by each camera and each frame. Figure 7 (c) schematically shows an example of the position of a specific part extracted in the image (step S14). Since the obtained position information is a force camera coordinate system, in order to obtain the position coordinate in real space, it is necessary to convert the position coordinate on the image to the position coordinate in real space. The camera coordinate system and the real space coordinate system can be transformed by the combination of the rotation matrix and the translation matrix. At this time, non-linearity caused by the optical system between the position on the camera coordinate system and the position projected on the real space coordinate system may be corrected by parameters such as focal length and lens distortion coefficient. Yes (step S1 5).

Next, three-dimensional position data is calculated using the position of the specific part obtained by each camera. The method of Ts a i is known as a method of converting markers in an image from a camera coordinate system to a real coordinate system in consideration of weights.

The calculation of the three-dimensional position of the marker position in the real coordinate system corresponds to, for example, finding the intersection of a plurality of straight lines extending from each camera image in FIG. 7 (d). Note that in practice, errors in the camera parameters may cause a shift in the angle of the straight line, and therefore may not intersect at one point. In such a case, the middle point on the straight line at which the distance between the straight lines is minimum may be determined as the point of intersection of the straight lines. Further, the midpoint is determined in the combination of n straight lines, and the average of these midpoints is set as the three-dimensional position of the marker (step S 16). Thereby, position information is obtained for the specific part (step S 3). Next, in steps S4 to S8, the component angle information is determined.

Based on the constraints determined by the position information acquired at step S3, a possible model is selected. The model is obtained in advance for various postures of the articulated body as the object, and the sizes and tolerances of the lengths, the distance relationships, and the angular relationships of each component of the articulated body are also determined. Set as a constraint condition. This restriction condition may, for example, define the range of mutual angle tolerance of each component with respect to the position of a specific part. it can. The tolerance range can be expressed numerically or as a function representing a mutual relationship.

When sorting by constraints is not performed, in order to select a model that matches the image, it is necessary to evaluate the superposition state while changing the positional relationship between the model and the image for all prepared models. It does not. On the other hand, by applying constraints, the number of models to be superimposed on the image can be reduced, and the amount of calculation of the matching process can be reduced.

Figure 3 (c) schematically shows the model selection. For example, if there are c-1, c-1, c-3 as models of component, choose a model that satisfies the constraints from among these models. Here, for example, a case is shown where a model is selected with the possible angular range of each component as a constraint.

Among the models c−l, c−12, c−3, c−1, c−3 models satisfy the constraints, and c−12 do not satisfy the constraints. , c Choose one of the three models (step S 4).

The model selected in step S4 is superimposed on the image to model the image and the model. For matching between an image and a model, for example, a three-dimensional model is placed in a virtual space, and the distribution of lightness and saturation on the model is determined as a histogram while changing the posture of the model. The histogram of the distribution of lightness and saturation can be obtained and the approximation of this histogram can be evaluated (step S 5). Perform the matching of the above image with the model for all selected models. At this time, in addition to a method of evaluating all possible postures, a nonlinear least squares method, a genetic algorithm or the like can be used to obtain an evaluation function so as to minimize it. 6080

18

For example, it is possible to obtain an evaluation value by assuming a certain parameter value (0 or position), changing the parameter and repeating evaluation based on the result, and finding the minimum evaluation value. it can. This evaluation function represents the degree of deviation determined by comparing the result of the silled image and the result of the optical flow with the model (step S 6). Find the model closest to the image in matching the image with all selected models. Figure 3 (d) schematically shows this matching process (step S7).

From the model obtained by the optimal matching found in step S7, the component angle information is found. Figure 3 (e) shows how to obtain angle information (Θ1 to Θ4) from the obtained model (step S8).

The posture of the articulated body is determined from the position information determined in step S3 and the angle information determined in step S9. By performing this posture for each frame acquisition, the motion of the articulated body can be determined in almost real time.

Next, the motion capture device of the present invention will be described with reference to FIG.

The motion capture device 1 is an image acquisition unit 2 that acquires multi-viewpoint images of multiple joints such as a human body and acquires an image, and an operation unit 3 that obtains position information and angle information of components from this image. The position and movement of the articulated body are determined by the position information and the angle information.

The image acquisition means 2 obtains an image of an articulated body including individually distinguishable markers provided at specific parts of constituent elements connected across at least one constituent element. A plurality of cameras are arranged to acquire multi-view images, and each image frame is synchronized.

The calculating means 3 is a detecting means 3a for detecting a specific part (marker) in the image. 6080

19

A position information detecting means 3 b for obtaining position information of a specific part (marker), a constraint that relates a plurality of components to one another, and a memory for storing in advance a plurality of models representing the postures of components of an articulated body Means 3 c, model selecting means 3 d for selecting a model that fits the constraints of the articulated body determined by the position of the specific part from a plurality of models, and an image of the articulated body in the image from among the selected models A matching unit 3 e for extracting the most approximate model and obtaining angle information of the component of the articulated body from the angle between the components of the extracted model.

Each means provided in the computing means 3 is for describing each function performed by the computing means, and is not necessarily provided with hardware for executing each function, and can be implemented by software.

Next, the attachment position of the marker in the case where the posture and motion of the human body are determined by the motion canopy of the present invention will be described with reference to FIGS. The human body can be regarded as an articulated body in which a plurality of components are connected by joints, and these components can be expressed hierarchically. The components can be set arbitrarily. Figures 9 and 10 show one model, which can be set by combining other components.

In the model shown in Fig. 9, the lumbar region is the upper layer, and the upper body is the chest, upper arm (upper left arm, upper right arm) and head, forearm (left forearm, upper arm) hand (left hand) The lower body is connected to the lower layer in the order of the thigh (left thigh and right thigh), the shin (left shin and right shin), and the legs (left foot and right foot). Be done.

This model is prepared for the human body to be measured, and each layer represented by the model corresponds to the component of the articulated body described above. For each component, set constraints on the length, distance between components, and angle. In motion chap- ters, multi-view images of the human body are acquired and obtained. 16080

20

The posture and motion of the human body are acquired by obtaining position information and angle information of a specific part of each component from the captured image according to the above-described process.

Here, a marker is attached to a specific part of the human body in order to obtain position information of the specific part. The markers are not attached to all components of the human body, but are attached to selected ones of the components of the model using a small amount of force. The number of forces to be attached is such that the subject is not burdened, and the position of force attachment is such that the force is always reflected on the image even when the human body is in various postures. Choose a similar site.

In addition, at the marker installation position, an appropriate hierarchical spacing is provided between the components to which the markers are attached, in consideration of the operation complexity in the model-image matching process. While matching between models and images reduces the number of models that can be obtained by the constraints determined by the position information of a specific part, the number of possible models also increases as the number of components between specific parts increases. Do. Therefore, in determining the posture and motion of the human body, place a force on specific parts at appropriate intervals that reduce the total amount of computation.

In the model shown in Fig. 9 and 10, a force is attached to the upper waist of the hierarchy, a marker is attached to the joints of the upper arm and forearm in the upper body, and a marker is attached to the joints of the thigh and shin in the lower body. The waist is attached to both sides, and the joints of the upper and lower arms and the joints of the thigh and leg are attached to the left and right.

In this way, a total of six forces are attached to the human body, and the position information of the waist, the position information of both left and right elbows, and the position information of both knees are calculated. For angle information, for example, the chest and upper arm are obtained by using the chest and upper arm constraints determined by the position information of the waist and the position of the elbow, for example, the angle information of the chest and upper arm. The posture to be evaluated is extracted from among the postures to be obtained, the number of processing is reduced, and the model is found by matching the image. Next, the force suitable for the motion canopy of the present invention will be described with reference to FIGS. Figure 1 1 (a) is a cross-sectional view of the marker, and Figure 1 1 () is a view looking downward from the position shown by E-E in Figure 1 1 (a).

The marker 20 comprises a plurality of light emitting diodes (L E D) including one first light emitting diode 21 and a plurality of second light emitting diodes 22. The first light emitting diode 21 is used to obtain position information of a specific site, and the second light emitting diode 22 is used to identify each marker and distinguish the specific site from other specific sites. Use. In the identification of the power, the first light emitting diode 21 and the second light emitting diode 22 have different light emitting colors, and the power identification is performed by a combination of the light emitting colors. The combinations of light emission colors are selected so as to make the hue angles symmetrical with each other in the H S I color system, and to be easily identified by the threshold value. For example, the combination of the central light emitting diode as red and the surrounding light emitting diode as blue, the central light emitting diode as red and the surrounding light emitting diode as green, or Select a combination such that the color of the central light emitting diode is yellow and the color of the surrounding light emitting diodes is blue.

By combining different colors, it is possible to easily distinguish them from each other even when there is a portion with the same color as the background in the image.

In the arrangement 20, the first light emitting diode 21 is disposed above the base 23, and the plurality of second light emitting diodes 22 are disposed in a ring around the lower periphery thereof. Also, on the base 21 is provided a power supply 25 of each light emitting diode. The power source 25 can use, for example, a battery. Reference numeral 26 in the figure is a switch for controlling the connection between the power supply 25 and the light emitting diode. For example, a removable insulator is used to remove this insulator. Can cause the light emitting diode to emit light.

As shown in FIG. 11 (b), the second light emitting diodes 22 are arranged at equal angular intervals around the first light emitting diode 21 such that the light emitting directions are equal angular intervals. A light emitting diode usually has directivity. Since the human body takes various postures, it is desirable that the light emission of the marker be non-directional in order to make the marker well reflected to the camera. Therefore, the light emitting directions of a plurality of light emitting diodes are arranged at equal angular intervals. In FIG. 1 1 (b), the five second light emitting diodes 2 2 are arranged in the direction of, for example, 72 °. Although the number of the second light emitting diodes 22 can be set arbitrarily, the relationship between the physical size of the light emitting diodes and the size of the small force of the light load, and the detection condition in the acquired image Set in consideration of various conditions such as.

The force sensor 20 includes a light transmitting cover 2'4 for internally housing the light emitting diodes 21 and 22 and the power source 25 and the like. The light of the light emitting diodes 2 1 and 2 2 is emitted to the outside through the cover 2 4. The inner surface or outer surface of the cover 24 may be a scattering surface, or the material constituting the cover 24 may be a light scatterer. By making the cover 24 scatter, the light emitted from the light emitting diode is scattered by the cover 24 and the reflection to the camera can be improved.

A light shielding body 2 8 is provided between the first light emitting diode 2 1 and the second light emitting diode 2 2 so that the installation position can be changed. The light shield 2 8 is a mixture of the light emission of the first light emitting diode 2 1 and the light emission of the second light emitting diode 2 2 on the image, and a color different from the light emission color of the light emitting diode This is to ensure that it does not occur. If different colors occur in the marker portion on the image, it may be difficult to identify the force, or it may be a factor in detecting the position of the force incorrectly. The light shield 2 8 is positioned between the light emitted from the first light emitting diode 21 and the light emitted from the second light emitting diode 22, whereby the first light emitting diode projected onto the image is formed. Separate the images of diode 2 1 and the second light emitting diode 2 2 to prevent mixing of both lights.

The light shielding body 2 8 is an annular body having an opening at the central portion,

It can be attached by passing the protruding part of 2 1 force par 2 4. In addition, by using the material of the light shield 2 8 as the inertia material, mounting to the cover 14 can be facilitated. Alternatively, an annular recessed portion 2 7 may be provided around the outer peripheral surface of the cover 24, and the cover 24 may be fitted into the recessed portion 2 7. The recessed portion 27 may be formed in multiple stages in the vertical direction of the cover 24 so that the mounting position on the cover 24 can be changed.

Figures 1 2 (a) and (c) show a state in which the light shield 2 8 is attached to the lower part of the depressed portion 27. Figures 1 2 (b) and (d) show the light shield 2 8 in the depressed portion 2 It is shown attached to the upper part of 7. By changing the mounting position of the light shield 2 8, the area of the annular portion of the second light emitting diode 2 2 displayed in the image can be increased or decreased. By adjusting the light emitting area of this second light emitting diode 22, the marker image is adjusted according to the imaging environment such as the distance between the human body and the camera, the state of the background color, the state of illumination, etc. It can be done well.

According to the aspect of the present invention, the number of markers attached can be reduced, the burden on the subject can be reduced, and position information and angle state can be acquired with a small burden.

According to the aspect of the present invention, since the number of models to be evaluated can be reduced in matching between an image and a model, processing time can be shortened, and real-time (real-time) processing of posture and motion can be approached. about Can. Also, if the processing time is the same, the measurement accuracy can be improved.

As described above, according to the present invention, in the motion capture, the burden of attaching the marker can be reduced, the analysis processing time can be shortened, and high extraction accuracy can be obtained. Industrial Applicability

The present invention can be used for analysis of mobile objects such as people and objects, and formation of a virtual space, and can be applied to the fields of industry, medicine, sports and the like.

Claims

The scope of the claims

1. In the determination of the posture and movement of an articulated body in which a plurality of components are articulated with one another,

The position information of a specific part of a component constituting the articulated body is determined, and each component satisfying a constraint defined by the position information is determined based on the position information and a constraint that relates the plurality of components to each other. Angle information between 3⁄4 ί,

A motion capture method characterized in that an attitude and an action of an articulated body are determined based on the position information and the angle information.

2. In the determination of the posture and movement of an articulated body in which a plurality of components are articulated with one another,

The distance and angle constraints relating the plurality of components to one another are determined in advance,

Providing individually identifiable markers at specific parts of the components to be linked across at least one component,

Find an image of a multi-segmented body including the said power,

The position information of the specific part is determined from the position of the force in the image,

Among the angle information between each component that satisfies the constraint defined by the position information, based on the image of the articulated body in the image, the position information, and the constraint that relates the plurality of components to each other. Find angle information that matches the image of the articulated body from

A motion capture method characterized in that the posture and motion of an articulated body are determined based on the position information and the angle information.

3. In the determination of the posture and movement of an articulated body in which several components are articulated to one another,

Constraint which relates the plurality of components to each other, and configuration of articulated body Predetermine multiple models that represent the attitude of the element,

An image of an articulated body including individually distinguishable markers provided at a specific site of a component to be connected across at least one component is obtained, and a specific site is obtained from the position of the force in the image. Find location information of

From the plurality of models, a model that meets constraints defined by the position information is extracted, and from among the extracted models, a model that is most approximate to the image of the articulated body in the image is extracted.

The angle information of the component of the articulated body is determined from the angle between the components of the extracted model,

A posture capture method characterized in that an attitude and an action of an articulated body are determined based on the position information and the angle information.

4. Acquire an image of a marker attached to the human body and a specific part of the human body, individually identify and extract the markers from the image, and obtain position information of the specific part of the human body,

Angle information of a component of the human body is determined by image alignment of the image and a model extracted based on position information of the specific part from among the models prepared in advance,

The posture of the human body is determined from the position information and the angle information. Motion capture method.

5. The extraction of the model is performed by selecting at least one model that meets the constraint condition of the human body posture determined by the position of the specific part. Image alignment of the image and the model is among the extracted models. The motion capture method according to claim 4, wherein the motion capture method is performed by selecting a model that most closely matches a human image in the image.

6. Identify the marker by the combination of at least two different colors that the marker emits, The method according to any one of claims 2 to 5, wherein the marker position is determined from the position of any one color of the color combination.

7. The motion capture method according to any one of claims 1 to 6, wherein the specific part is a waist, an elbow, and a knee in a human body.

8. A motion capture device for determining the posture and motion of an articulated body in which a plurality of components are articulated to one another,

Position detection means for obtaining position information of a specific part of a component constituting an articulated body;

And angle detection means for obtaining angle information between each component that satisfies a constraint defined by the position information based on the position information and a constraint that relates the plurality of components to each other,

A motion capture device characterized in that the posture and motion of an articulated body are determined based on the detected position information and angle information.

9. A motion capture device for determining the posture and motion of an articulated body in which a plurality of components are articulated to one another,

A constraint that relates the plurality of components to each other; and storage means for pre-storing a plurality of models representing the postures of the components of the articulated body.

An image acquisition means for obtaining an image of an articulated body including individually identifiable markers provided at specific parts of the components connected across at least one component;

Position information detecting means for obtaining position information of a specific part from positions of markers in the image;

Model selecting means for selecting from the plurality of models, a model that meets the constraint condition of the articulated body determined by the position of the specific part; A matching unit that extracts a model that most approximates the image of the articulated body in the image from the selected models, and obtains angle information of the component of the articulated body from the angle between the components of the extracted model. And

A motion capture device characterized in that the posture and motion of an articulated body are determined based on the position information and the angle information.

1 0. A marker that identifies a specific part of an articulated body in motion masks that determine the posture and operation of an articulated body in which a plurality of components are articulated to each other,

The marker includes one first light emitting diode disposed at the center, and a plurality of second light emitting diodes disposed at equal angular intervals around the first light emitting diode, and the first light emitting diode A marker for motion capture, wherein the second light emitting diode is different in light emitting color.

11. The motion capture marker according to claim 10, wherein a light shield is provided between the first light emitting diode and the second light emitting diode.