JP4878330B2 - Method and apparatus for acquiring joint structure of object - Google Patents

Description

The present invention relates to a method and apparatus for acquiring a joint structure of an object such as a human being or other living things.

Human motion measurement and model estimation are research subjects that are actively used in a wide range of fields such as robotics, computer vision, and body motion science.

Motion capture is generally used in the field of motion measurement. Optical motion capture using retro-reflective markers enables high-precision measurement and real-time measurement. However, since optical motion capture is a technique for measuring the coordinates of a point, inverse kinematic calculation using a joint model is required to obtain the posture and shape of a human body.

On the other hand, in applications that require a three-dimensional shape of an object, motion capture that does not use a marker is used. For such motion capture, a method using the visual volume intersection method (Non-patent Document 1) is generally used. Iiyama et al. (Non-Patent Document 2) arranged cameras in all directions to enable measurement of the three-dimensional shape of the entire circumference of the object including the lower surface of the object. In addition, the conventional visual volume intersection method often deals with voxel data, and has a drawback that the calculation amount is large.

In addition, there are studies that thin the three-dimensional shape and estimate the joint model and posture of the human body. Chi-Wei et al. (Non-Patent Document 3) estimated a joint model from markerless motion capture data. In conventional posture estimation and model estimation using markerless motion capture, thinning is often performed. However, the joint that can be expressed by thinning is a two-degree-of-freedom joint obtained by removing the degree of freedom around the roll axis from the spherical joint. In order to represent a general spherical joint, only the thinning process is incomplete.

In addition, a method using a Reeb Graph (non-patent document 4) in which topological information of three-dimensional shape matching is expressed in a graph has been studied. The leave graph is a graph in which a curved surface is divided into several parts by the value of a continuous function μ defined on the curved surface or an object, and the connection relation of each part is represented by a graph. It is possible to obtain characteristic information such as topological information and skeletal information from 3D shape data by using a leave graph, and from the viewpoint of the amount of information, it is possible to compress information and reduce calculation costs. Has brought benefits. Hiraga et al. (Non-Patent Document 5) and Tung et al. (Non-Patent Document 6) performed shape matching by determining the similarity of three-dimensional shapes using a multi-resolution leave graph. Miyamoto et al. (Non-Patent Document 7) perform posture estimation of a human body using a leave graph.
WNMartin and JK Aggarwal.Volumetric Descriptions of Objects from MultipleViews.IEEE Trans.on Pattern Analysis and MachineIntelligence, Vol. 5, No. 2, 1983. Masaaki Iiyama, Yoshinari Kameda, and Michihiko Mino. 4π measurement system: acomplete volume reconstruction system for freely-moving objects.MultisensorFusion and Integration for Intelligent Systems, MFI2003.Proceedings of IEEE International Conference on, pp. 119.124, 2003. Chi-Wei Chu, Odest Chadwicke Jenkins, and Maja J Mataric.Markerless KinematicModel and Motion Capture from Volume Sequences.In Proceedings of IEEE ComputerVision and Pattern Recognition (CVPR 2003), June 2003. G.Reeb.On thesingular Points of a Completely Integrable Pfaff Form or of a Numerical Function.Comptes Randus Acad.Sciences Paris, Vol. 222, pp. 847.849, 1946. Masaki Hilaga, Yoshihisa Shinagawa, Taku Kohmura, and Tosiyuki L. Kunii. Topology Matching forFully Automatic Similarity Estimation of 3D Shapes. SIGGRAPH 2001 ConferenceProceedings LA, CA, USA, pp. 203.212, August 2001. T. TUNG and F. SCHMITT. THE AUGMENTED MULTIRESOLUTION REEB GRAPH APPROACH FOR CONTENT-BASEDRETRIEVAL OF 3D SHAPES.International Journal of Shape Modeling, Vol. 11, No.1, pp. 91.120, 2005. Shin Miyamoto, Kazuhiko Sumi, Takashi Matsuyama. Posture Estimation of Human Body Using Skeletal Line Model and Cylindrical Articulated Model. IPSJ SIG Notes, Vol. 2006, No. 51, 2006.

The present invention is to acquire the joint structure of an object without prior knowledge about the object. Another object of the present invention is to model a joint structure including a spherical joint.

The method for acquiring a joint structure of an object adopted by the present invention uses a view volume intersection method to acquire a three-dimensional shape of an object from images obtained from a plurality of cameras, and a leave graph. The average geodesic distance μ is defined on the surface of the three-dimensional shape of the obtained object, the contour line of the value of μ is obtained, and virtual joints that are candidate joints are arranged based on the contour lines, and the object The step of acquiring a virtual joint model and a step of estimating a virtual joint with a lot of movement as an actual joint by comparing corresponding virtual joints among a plurality of virtual joint models acquired based on different postures of the object And a step of acquiring a joint model of an object excluding joints other than virtual joints estimated as actual joints.

The apparatus for acquiring a joint structure of an object adopted by the present invention uses a view volume intersection method, a means for acquiring a three-dimensional shape of an object from images obtained from a plurality of cameras, and a leave graph. The average geodesic distance μ is defined on the surface of the three-dimensional shape of the obtained object, the contour line of the value of μ is obtained, and virtual joints that are candidate joints are arranged based on the contour lines, and the object A means for acquiring a virtual joint model of a subject and a means for estimating a virtual joint with a lot of movement as an actual joint by comparing corresponding virtual joints between a plurality of virtual joint models acquired based on different postures of the object And means for obtaining a joint model of an object excluding joints other than virtual joints estimated as actual joints.

In one preferred aspect, in the step of acquiring the three-dimensional shape, the three-dimensional shape of the object is expressed as a polygon mesh, and in the step of acquiring the virtual joint model, an average geodetic distance μ is set on the surface of the polygon. Define. In the present invention, the three-dimensional shape can be expressed by voxel data, but a polygon mesh is advantageous in consideration of the amount of calculation of the μ function. In one preferred aspect, the object visual volume is represented by a polygon mesh; the visual volume polygon mesh is converted into a convex polygon mesh; and the intersection processing of the convex polygon meshes obtained from each camera is performed. In one preferred embodiment, the intersection process includes reducing the number of polygons by reconstructing the polygon when cutting out the polygons of the convex polygon mesh in the intersection process. In one preferred embodiment, the intersection processing includes combining the convex polygon meshes after all the intersection processing is completed, or in addition, after each intersection processing is completed.

The virtual joint model is obtained by placing one virtual joint on one abstract node of the leave graph obtained based on the average geodesic distance μ, which is a function independent of posture. In one embodiment, the part where μ is minimum is the root of the leave graph (virtual joint model), and the direction in which μ increases is the child joint. The virtual joint model can have a graph structure composed of virtual joints constituting nodes and edges connecting the virtual joints. In this case, the direction of the parent joint is the direction of the edge connecting the parent joint and the child joint. In one preferred embodiment, a virtual joint that is a candidate for a joint is arranged at the center of gravity of each contour line. As another aspect, any point on the contour line can be set as the position of the virtual joint. In a preferred aspect, the apparatus includes means for giving texture information of an object to the three-dimensional surface. It is well known to those skilled in the art to attach a texture to polygon data.

In obtaining the virtual joint model, in one preferred embodiment, the obtained virtual joint model includes correcting the obtained virtual joint model based on a phase structure of a known joint of the object. In a more specific aspect, the modification includes severing any of the branches that make up the loop in the joint structure. In another aspect, the modification includes dropping a short branch after a loop break or otherwise.

In one preferred aspect, estimating a virtual joint with a lot of motion as an actual joint is obtained by obtaining a joint motion variance among the virtual joint models for each virtual joint and performing a motion based on the obtained variance. Determine the virtual joints with many. In one preferred embodiment, the joint motion dispersion is a joint angle dispersion of a virtual joint. In one preferable aspect, the joint angle is an angle between two adjacent line segments (edges) connecting the virtual joints. In one preferable aspect, the joint motion variance is a variance of inner products of two adjacent line segments (edges) connecting the virtual joint. In one preferred embodiment, a virtual joint whose motion variance exceeds a threshold is estimated as an actual joint. In one preferred embodiment, when the number N of actual joints is given, N virtual joints are selected in descending order of motion variance and estimated as actual joints.

The joint model of the object finally obtained is a link mechanism model, and a coordinate system fixed to the joint is required. Therefore, in order to leave a virtual joint with a large change and create a link mechanism model, it is necessary to set a local coordinate system for the remaining virtual joint. The “joint angle” in the link mechanism model is calculated as a relative posture of the local coordinate system. Therefore, in acquiring the joint structure of the object according to the present invention, the local coordinate system of the virtual joint having three axes is set before or after the actual joint is estimated from the virtual joint.

In one preferable aspect, the setting of the local coordinate system of the virtual joint is performed by principal component analysis of the coordinates of the contour line, and the obtained three eigenvectors are set as the local coordinates of the virtual joint. In a more specific aspect, in the principal component analysis, the axis of the third principal component is the normal of the contour plane; the major axis direction of the circumscribed rectangle of the contour line in which the axis of the first principal component is projected onto the contour plane; The axis of the two principal components is defined as the minor axis direction of the circumscribed rectangle of the contour line projected onto the contour plane.

In one preferred embodiment, texture information of the object is used when setting the local coordinate system of the virtual joint. If the cross section of the object is close to a circle and the distinction between the major and minor axes is not clear, texture is used to identify the rotation around the cylinder axis. In a more specific preferred embodiment, using the texture information of the object, one axis from the centroid of the contour line to the texture information is specified as the texture axis, and the texture axis is replaced with one axis in the contour plane. Use. The texture axis extending in the contour plane can be set based on the contour line projected onto the contour plane, or it can be set by projecting the axis obtained from the relationship between the point on the contour line and the texture onto the contour plane. May be. Further, the normal axis of the contour plane may be replaced with the direction of the parent joint (in this case, correction is necessary so that the three axes are orthogonal to each other).

The setting of the local coordinate system is not essential for the calculation of “joint angle” in “estimating an actual joint from a virtual joint”. However, by setting the local coordinate system before estimating the actual joint, the range of selection of the joint angle calculation method can be expanded. For example, the following methods can be taken. The relative posture of the parent joint with respect to the local coordinate system is calculated, and any component of the Euler angle is used. Similarly, the relative orientation is calculated and the equivalent rotation angle (rotation angle when the coordinate transformation is expressed by one rotation around a certain axis) is used. In addition, assuming that there is a virtual joint that only twists and turns, the angle of the line connecting the virtual joints does not change, so in estimating a virtual joint with many movements as an actual joint, using the texture information, the twist angle Can be considered.

In the present invention, virtual joints are arranged using a leave graph from a three-dimensional shape (polygon data) obtained by markerless motion capture using the visual volume intersection method, and data of a plurality of postures (frames) are obtained. By comparing, unnecessary virtual joints are removed to estimate the joint structure of the object. According to the present invention, it is possible to acquire a joint structure of an object from shape data by using images obtained from a plurality of cameras, and therefore to estimate a joint model without prior knowledge of the joint structure There is an advantage that can be. This makes it possible to estimate a joint model of a target (such as a mollusk) that does not normally define a joint.

In addition, incorrect skeleton information may be obtained due to the touch of the hand and the body. In such a case, if the phase information of the object is known in advance, the skeleton information is corrected and a highly accurate model estimation can be performed. The prior knowledge is also used only for phase information (topology), and it is not necessary to use knowledge about the number, length, and diameter of the cylinder.

In the present invention, a spherical joint can be modeled by setting a local coordinate system having three axes for each joint of the joint model of the object. In the present invention, the twist of the joint is taken into account by the long and short axes of the contour lines or the texture axis.

[A] Outline of the Present Invention An apparatus for acquiring a joint structure of an object according to an embodiment of the present invention uses a visual volume intersection method to obtain an object composed of polygon meshes from images obtained from a plurality of cameras. Using means for obtaining a three-dimensional shape and a leave graph, an average geodetic distance μ is defined on the surface of the obtained polygon, a contour line with a value of μ is obtained, and a joint candidate is added to the center of gravity of each contour line. By comparing the virtual joint model between a plurality of virtual joint models acquired based on different postures of the target object and a means for acquiring the virtual joint model of the target object by arranging the virtual joints to be virtual It comprises means for estimating a joint as an actual joint, and means for obtaining a joint model of an object excluding joints other than virtual joints estimated as actual joints. And the hardware of the acquisition apparatus of the joint structure of a target object is comprised from the some camera which image | photographs a test subject, and one or several computer apparatus. The computer device includes an arithmetic processing unit, an input unit, an output unit, a display unit, and a storage unit, and the functions of the computer are executed according to a predetermined computer program. Specifically, the calculation in the present invention includes image processing, visual volume polygon intersection processing, μ calculation, joint angle calculation, local coordinate system setting, texture setting, and the like. Can be executed.

An overview of the present invention is shown in FIGS. An outline of acquisition of a three-dimensional shape by the visual volume intersection method will be described. First, an object is photographed with a plurality of cameras (FIG. 1). Using a camera image and camera parameters, a three-dimensional shape composed of a polygon mesh is obtained by the visual volume intersection method (left figure in FIG. 2). More specifically, the three-dimensional shape procedure of the visual volume intersection method obtains an object region from a camera image; obtains an object visual volume using camera parameters; The three-dimensional shape is obtained by extraction. Then, polygon texture information is acquired (right diagram in FIG. 2).

Next, an outline of joint estimation using a leave graph will be described. First, a function μ is defined on a three-dimensional shape. The function μ is a function that does not depend on the posture. In the left figure of FIG. 3, μ values in different postures are shown by shading. Next, a contour line of μ is obtained, and a virtual joint model is obtained by arranging a virtual joint at the center of gravity of the contour line (FIG. 3). A virtual joint is a candidate for a joint. Finally, virtual joint models of a plurality of frames (different postures) are compared, and joint positions are estimated by removing unnecessary virtual joints (FIG. 4).

The procedure of this method is illustrated below. The order of these steps is not limited to the order described, and the order may be partially changed.
step 1: Capture images of objects using multiple color cameras;
step 2: Extract objects in the image;
step 3: Using the object region in the image and the external and internal parameters of the camera, obtain a three-dimensional region (view volume polygon mesh) where the object can exist;
step 4: Divide the visual volume polygon mesh into convex polygon meshes;
step 5: Obtain the three-dimensional shape by intersecting (product) the viewing volume polygon meshes obtained from each camera;
step 6: Give texture information to the 3D shape;
step 7: Combine the obtained convex polygon meshes to obtain polygon connection information;
step 8: find the function μ of the polygon on the 3D shape;
step 9: Topology calculation by Reeve graph based on μ;
step 10: Take a contour line with a value of μ and place the virtual joint corresponding to the node of the leave graph at the center of gravity of the contour line;
step 11: Find the local coordinate system of the virtual joint;
step 12: By removing a plurality of virtual joint models, unnecessary virtual joints are removed to obtain a joint model of the target object;
Consists of.
Hereinafter, embodiments of the present invention will be described in detail.

[B] Acquisition of three-dimensional shape by visual volume intersection method The visual volume intersection method is to obtain a visual volume obtained by converting an object region of a camera image into three dimensions, and to obtain a three-dimensional shape by obtaining intersections of the visual volumes of a plurality of cameras It is a technique to obtain. A three-dimensional shape is acquired by the following procedure. First, the projection area of the target object is acquired from the camera image. A cone-shaped open space consisting of a set of half lines connecting the center of the camera lens and an arbitrary point on the projection area is called a viewing volume. In other words, the object visual volume is obtained by converting the object area in the camera image into a three-dimensional space, and is a polygonal weight having the camera position as the apex. The target object is always inscribed in this visual volume. The viewing volume exists for each camera, and the three-dimensional surface shape of the target object is obtained by taking the intersection (product) of the viewing volumes of all the cameras.

[B-1] Extraction of Target Object Region in Image As shown in FIG. 5, the object region extraction method extracts an object region by performing background difference and feature color extraction twice. First, a background image that does not include the target object is prepared. Paying attention to the corresponding pixel in the image including the target object and the background image, and if the difference in value exceeds a certain threshold, it is determined that the pixel is a part of the target object. However, the above method extracts even the shadow of the object. Therefore, in addition to the background difference, the object region is extracted using the color information of the object. Finally, the outline of the region obtained by the above method is taken and polygon approximation is performed. It should be understood by those skilled in the art that other means used in image processing can be employed for extracting the target object region in the image.

[B-2] Obtaining a Visual Volume Polygon Mesh By calibrating the camera, external parameters of the camera position and orientation and internal parameters such as focal length and distortion coefficient are obtained. By using these values, it is possible to obtain a three-dimensional half line on which a point can exist from one point in the image. However, the starting point of the half line is the absolute position of the camera.

By obtaining this half line along the boundary line of the object projection area, the viewing volume of the polygonal pyramid is obtained. FIG. 6 shows the view volume. First, the visual volume is expressed by a polygon mesh. A polygon mesh is a set of polygons (polygons) and is used to express a three-dimensional surface shape. Further, the polygon mesh has connection information between polygons. Polygon meshes have the advantage that the amount of calculation is small compared to voxels. In the present embodiment, description will be made based on a triangular polygon. Since polygons of quadrangular or higher can be divided into triangular polygons, polygons other than triangular can be used.

[B-3] Polygon mesh intersection process First, a polygonal mesh with a visual volume is divided into convex polygon meshes. A convex polygon mesh is a polygon mesh composed only of polygons whose angle between two adjacent polygons is 180 degrees or less. Next, the product operation of the convex polygons is performed, and the set of results becomes the product of the viewing volume. In the case of the intersection of the viewing volumes A and B, the equation (1) is obtained. Here, ai and bj are convex polygon meshes that constitute A and B.
Further, information on two polygons constituting the side is given to the side of the polygon mesh. Thereby, polygon connection information can be obtained.

Next, the intersection process of the convex polygon mesh will be described. Polygons have a distinction between outside and inside. By defining normal vectors, the outside and inside are expressed. In order to perform the intersection processing of the convex polygon mesh, first, attention is paid to one polygon Ti in the convex polygon mesh. A portion outside the other polygon Tj is cut out from the polygon Ti. Cut processing is performed for all Tj. This is shown in the upper part of FIG. Further, the same processing is performed on all Ti, and the set of remaining portions is the result of the intersection processing. An example of the intersection processing of two convex polygon meshes (triangular pyramids) is shown in the lower diagram of FIG. After the intersection processing of the convex polygon meshes, the convex polygon meshes are combined. The process of combining convex polygon meshes is a process of combining a plurality of polygon meshes by deleting polygons that enter the interior of the finally obtained polygon mesh.

[B-4] Obtaining Texture of 3D Shape Data Finally, a texture is pasted on the obtained 3D shape data. A texture is attached to each polygon using the pixel value of the camera closest to the normal direction of the polygon. However, when another polygon exists between the camera and the polygon when the above method is used, an image from another camera is used.

Conditions for viewing the polygon P from the camera C are as follows (see FIG. 8). First, P must be in the field of view of C; second, P's normal should be in the direction of C; and finally, no other polygon exists between P and C That is. The second condition is that the inner product of the P to C vector PC and the P normal vector is positive. Finally, the determination is made by performing contact determination between the line segment PC and all other polygons. If the above conditions are satisfied, texture information in the C image is given to P.

Next, a method for reducing the number of polygons when performing a polygon mesh intersection process will be described. In this method, since the polygon is expressed as a triangle, the triangle is divided when the polygon is cut out by the intersection process. For this reason, the number of polygons increases more than necessary while the intersection process is repeated. Therefore, the number of polygons is reduced by performing polygon reconstruction when cutting the polygons. FIG. 9 shows the flow of gradually cutting a triangle. However, in the polygon reconstruction method, the number of polygons is reduced by canceling the triangle division once and performing a new triangle division after cutting. ing. As another method for reducing the number of polygons, the convex polygon meshes are joined not only after all the intersection processes have been completed, but also after a single intersection process has been completed. In this process, only those that can maintain the convex state even if they are combined are combined. FIG. 10 is an example in which the left side can be combined to maintain the convex state, and the right side is an example in which the convex shape is lost.

[C] Arrangement of virtual joints using a leave graph A leave graph is a three-dimensional shape phase information obtained from a μ value defined on the surface of a three-dimensional shape. By using a leave graph, the object can be divided into several parts by the value of the continuous function μ defined on the object surface, and the connection relation of each part can be represented by a graph. In this method, a joint position of a three-dimensional shape is estimated using a leave graph. Although a leave graph is considered to refer to a graph obtained in a narrow sense, in this specification, a leave graph is used in a broad sense including a method of obtaining the graph from μ values. The leave graph is used to obtain the topology from the μ value distribution on the polygon. More specifically, the virtual joint placement step includes three steps: calculation of μ; calculation of topology using a leave graph; placement of virtual joints based on contour lines of μ. The leave graph is formed from a plurality of nodes and edges connecting the nodes. It is known to obtain a leave graph from the calculated μ value, and is described in Non-Patent Document 6, for example. A node in a leave graph is an abstract thing that only represents the topology, not a physical point. On the other hand, the virtual joint in the present invention corresponds to the node of the Ree graph, but the “virtual joint model” and the “Leave graph” are equivalent in that the attribute of the position based on the contour line of μ is provided. is not. The procedure for virtual joint placement using a leave graph is as follows. First, the average geodesic distance μ is defined. Next, μ is converted into a continuous function. Next, the contour line of the value of μ is acquired, and the virtual joint corresponding to the node of the leave graph is arranged at the center of gravity of the contour line.

[C-1] The definition function μ of the average geodesic distance μ is a function of the polygon T. μ is defined as shown in Equation (2). Here, Ti is a polygon, a (Tj) is the area of the polygon Tj, and g (Ti, Tj) is the shortest distance (shortest geodetic distance) when tracing the surface from the polygon Ti to the polygon Tj according to the connection information. g can be calculated by the Dijkstra method.
Μ (Ti) defined in this way is a function that represents the average of the shortest distances from one polygon Ti to all other polygons. The value of μ decreases as it approaches the center of the body, and increases as it approaches the tip. In addition, since μ is based on the distance along the surface, μ is a function independent of the posture.

In order to define the contour line, μ needs to be a continuous function on the surface. In the above definition, μ is set for each polygon, so μ is a discrete value on the surface. Therefore, μ is defined at the vertex, and μ in the polygon is obtained by linear interpolation of μ at the vertex. This makes it a continuous function on the surface. The method of taking the value of μ at the vertex v is as shown in the following equation.
The value of μ at the vertex v is the average of the values of μ of Ti (polygon including v). The method of linear interpolation is as shown in the following equation, and the value of μ at the point p (s, t) in the polygon can be expressed using the value of μ at each vertex and the parameter st.
s, t are parameters in the polygon, and va, vb, vc are the vertices of the polygon.

[C-2] Arrangement of Virtual Joint Next, a contour line of μ is acquired. The contour lines connect points with equal values of μ, and are always closed curves. Here, the contour line CP satisfying the following two conditions is set as a parent contour of the contour line C. The first is that there is a path that does not cross other contour lines when tracing on the surface from an arbitrary point of C to an arbitrary point of Cp. The second is that μ (C) = μ (Cp) + d. Where d is the contour line interval. Then, a virtual joint is arranged at the center of gravity of the contour line. Further, the parent-child relationship of the contour lines is set as the parent-child relationship of the virtual joint. d is a value determined in advance so that an appropriate number of virtual joints are set, and a specific value of d can be appropriately set by those skilled in the art.

In later joint angle calculations, if the reference joint (parent joint) is on the same side for all frames, the joint angle becomes unstable. In one embodiment, the portion where μ is minimum is used as the root of a leave graph (virtual joint model), and the direction in which μ is increased is used as a child joint, thereby preventing the parent-child relationship from changing from frame to frame. As shown in FIG. 12, there is a route on the side close to the center of the body (μ is small), and child joints are sequentially formed from the root toward the end.

[D] Setting of Local Coordinate System of Virtual Joint Next, a method of setting the local coordinate system of the virtual joint will be described. The local coordinate system of the virtual joint is obtained by principal component analysis of points constituting the contour line. Three eigenvectors are obtained as a result of the principal component analysis, and the three axes are used as the local coordinate system of the virtual joint.
Pi is the coordinate of the midpoint of the contour line segment, and li is the length of the contour line segment. Thereby, a spherical joint can be expressed. The three axes are shown in FIG. Each main component (three axes) can be given the following meaning.

Third principal component: a principal component having a minimum eigenvalue λ3. The eigenvector x of the third principal component represents a normal vector of the contour plane. The contour plane is a plane that can represent the contour line shape with the least loss when projected onto the plane.

First principal component: The principal component having the maximum eigenvalue λ1. The eigenvector y of the first principal component represents the major axis direction of the minimum circumscribed rectangle of the contour line projected onto the contour plane.

Second principal component: The eigenvector z of the second principal component represents the minor axis direction of the minimum circumscribed rectangle of the contour line projected onto the contour plane.

Finally, setting of coordinate axes using texture will be described. As described above, the eigenvectors of the first principal component and the second principal component represent the major axis direction and the minor axis direction of the smallest rectangle surrounding the contour line projected on the contour plane, but when the contour line is close to a circle, the principal component analysis The first principal component and the second principal component are close to each other. The major axis direction and the minor axis direction of the axes of the first main component and the second main component are meaningless. In such a case, an axis that changes to the two axes is defined using texture information. A feature color is extracted from the texture of the contour line, the direction of the feature color is defined as the texture axis from the center of gravity of the contour line, and is used instead of the eigenvector (major axis direction) of the first principal component.

In the experimental example to be described later, a band-like pattern extending substantially along the x-axis is used as the texture. However, the y and z axes of each virtual joint do not have to be in a specific direction. The pattern constituting the texture is not limited as long as the y and z axes of the virtual joint can be determined. That is, it is a condition required for the texture that one axis from the center of gravity of the contour line to the contour line can be specified. Therefore, most patterns can be used as texture information except in a limited case where the same pattern is a pattern that repeats in the circumferential direction.

[E] When a part of the phase information correcting body touches another part, phase information different from the actual one can be obtained by the leave graph. When the phase information of the target is known, it is possible to correct the phase information by providing the information, and the estimation accuracy can be improved. Conversely, when the phase information is unknown, it may be difficult to associate virtual joints between frames.

The phase information loop correction will be specifically described. For example, if the hand touches the waist, the joint structure will loop. In such a case, one of the branches constituting the loop is cut. After the branches constituting the loop are cut, the branches that match the known phase information or the phase information of other frames are cut.

FIG. 12 shows the deletion of the loop portion. When the leave graph has a loop, a branch having phase information that matches the model is selected and separated from the branches constituting the loop. The case where a model and a leave graph have a loop on the upper side of FIG. 12 is shown. The loop is formed of a branch A, a branch B, a branch C, and a branch D. On the lower side of FIG. 12, a branch A is cut, a branch B is cut, a branch C is cut, and a branch D is cut. Each is shown separated. Since the branch coincides with the model because the branch C is cut off, the branch C is selected and cut off.

Actually, the three-dimensional shape is not cut, but the polygon connection information at the cut end is deleted. Recalculate μ for the three-dimensional shape. Based on the recalculated μ, a virtual joint model is constructed, and the variance is calculated in comparison with the virtual joint models of other frames.

[F] Joint model estimation When phase information is obtained in all frames, unnecessary virtual joints are removed to obtain joint structures.

The removal of unnecessary virtual joints will be described. Since μ is a posture-independent function, as long as the phase information matches, virtual joints can be easily associated between a plurality of frames. By comparing the virtual joint models of a plurality of frames, the virtual joints are associated between the plurality of frames. When the association is completed, for each virtual joint, the distribution of the motion of the virtual joint between frames is obtained. A virtual joint estimated to be an actual joint is selected based on the obtained dispersion, and unnecessary virtual joints are removed.

In a specific example, joint angle dispersion is calculated between corresponding virtual joints, and a virtual joint having a large dispersion is estimated to be an actual joint. In one aspect, a dispersion threshold is set, and a virtual joint that is equal to or greater than the threshold is estimated to be an actual joint. In another aspect, when the number of joints is given as N, N virtual joints are selected in descending order of variance and are estimated as actual joints. Several methods are conceivable by those skilled in the art for calculating the joint angle. Typically, an angle between line segments connecting virtual joints is a joint angle. In addition, Euler angles, equivalent rotation angles, and the like can be used.

[G] Human joint structure estimation Joint structure estimation was performed using a human as an object. The experimental conditions are as follows. The test subject is a human. As shown in FIG. 13, a red garment was worn to distinguish it from the background, and blue lines were attached to both arms, legs, and torso to use texture information (texture axis). As shown in FIG. 14, the camera is eight color cameras (resolution: 1628 pixels × 1236 pixels). In FIG. 14, the numbers beside the camera indicate the height of the camera.

FIG. 15 shows the result of image processing. The upper left is the background image, the upper right is the background and the object image, the lower left is the object area obtained, and the lower right is the triangle of the outline of the object area.

FIG. 16 shows the results of obtaining a three-dimensional shape, μ distribution, μ contour lines, and virtual joints. The left is a three-dimensional shape, the center is a μ distribution, and the right is a μ contour and a virtual joint. In the center diagram, the μ distribution is represented by shading. In the right figure, white points (nodes) indicate virtual joints, and white lines (edges) indicate parent-child relationships of joints.

FIG. 17 shows the distribution of the motion of the virtual joint in a plurality of postures (a plurality of frames). This graph relates to the left foot. The horizontal axis represents the distance from the toe, the horizontal axis represents the distance from the toe, and the vertical axis represents the inner product of two adjacent line segments connecting the virtual joints. The inner product of two adjacent line segments connecting the virtual joints can be practically treated as a value representing the joint angle of the virtual joint. From FIG. 17, it can be read that the variance is large between the knee and the ankle.

As described above, a method for obtaining a three-dimensional surface shape having texture information by the visual volume intersection method using images from a plurality of cameras has been proposed. We proposed a method to estimate the actual joint position by comparing virtual joint models of multiple postures obtained using a leave graph. A virtual joint was placed using a reeve graph, and a local coordinate system setting method based on contour line shape and texture information was proposed to enable modeling of spherical joints.

The present invention can be used to model a living organism such as a human being as a link structure in CG and biological simulation. In the current motion capture system, since the joint structure is assumed in advance, it is necessary to determine the joint structure by some method even when targeting an animal or the like. In addition, it is difficult to measure organisms that do not have a clear joint structure, such as mollusks. Even if the joint structure is known to some extent like a human being, there are individual differences in the distance between the joints, and separate measurement is required to construct an accurate model. From these things, it can be judged that this invention has the high utility value in CG and biological simulation. The necessary hardware is only a high-resolution camera, and other data processing / calculation is realized by software, so the possibility of practical use is high.

It is a figure which shows imaging | photography of the target object with a some camera. The left figure shows the three-dimensional shape of the object consisting of a polygon mesh, and the right figure shows the acquisition of polygon texture information. Acquire a contour line of μ and place a virtual joint at the center of gravity of the contour line. The virtual joint models of a plurality of frames are compared, and the joint position is estimated by removing unnecessary virtual joints. It is a figure which shows the extraction method of a target object area | region. The appearance volume is shown. It is a figure explaining the intersection process of a convex polygon mesh. FIG. 5 is a diagram for explaining conditions for viewing a polygon P from a camera C. It is the figure which showed the flow which cuts off a triangle gradually. This is an example in which the left side can be joined to maintain the convex state, and the right side is not convex when the right side is joined. Three axes that can represent a spherical joint are shown. Indicates deletion of the loop part. The object used in the experiment is shown. The arrangement of cameras in the experimental apparatus is shown. The result of image processing is shown. The upper left is the background image, the upper right is the background and the object image, the lower left is the object area obtained, and the lower right is the triangle of the outline of the object area. The left is a three-dimensional shape, the center is a μ distribution, and the right is a μ contour and a virtual joint. It is a graph of the dispersion | variation in the motion of the virtual joint in a several attitude | position. The horizontal axis represents the distance from the toe, and the vertical axis represents the inner product of two adjacent line segments connecting the virtual joint.

Claims (40)

Obtaining a three-dimensional shape of an object from images obtained from a plurality of cameras using a view volume intersection method;
Using a leave graph, define the average geodesic distance μ on the surface of the three-dimensional shape of the obtained object, acquire contour lines of the value of μ, and select virtual joints that are candidate joints based on each contour line Placing to obtain a virtual joint model of the object;
Estimating a virtual joint with a lot of movement as an actual joint by comparing corresponding virtual joints between a plurality of virtual joint models acquired based on different postures of the object;
Obtaining a joint model of an object excluding joints other than virtual joints estimated as real joints;
A method for obtaining a joint structure of an object. In the step of acquiring the three-dimensional shape, the three-dimensional shape of the object is expressed as a polygon mesh,
In the step of acquiring the virtual joint model, an average geodesic distance μ is defined on the surface of the polygon.
The method for acquiring a joint structure of an object according to claim 1. The step of acquiring the three-dimensional shape includes
Expressing the object visual volume with a polygon mesh;
Converting the view volume polygon mesh to a convex polygon mesh;
A step of performing intersection processing between convex polygon meshes obtained from each camera;
The method for acquiring a joint structure of an object according to claim 2.   The method for acquiring a joint structure of an object according to claim 3, wherein the intersection processing step includes reconstructing polygons to reduce the number of polygons when cutting out polygons of the convex polygon mesh in the intersection processing step.   5. The method according to claim 3, wherein the intersection processing step includes combining the convex polygon meshes after all the intersection processing is completed, or in addition to it, after each one intersection processing is completed. A method for obtaining a joint structure of an object according to claim 1.   The method for acquiring a joint structure of an object according to claim 1, wherein in the step of acquiring the virtual joint model, a virtual joint that is a candidate for a joint is arranged at the center of gravity of each contour line.   The target according to claim 1, wherein the step of acquiring the virtual joint model includes a step of correcting the obtained virtual joint model based on a phase structure of a known joint of the target object. How to get the joint structure of an object.   The method for acquiring a joint structure of an object according to claim 7, wherein the step of correcting the virtual joint model includes cutting off any of branches constituting a loop in the joint structure.   The step of estimating a virtual joint with a lot of motion as an actual joint is to determine the variance of joint motion among the virtual joint models for each virtual joint and determine a virtual joint with a lot of motion based on the obtained variance The method for acquiring a joint structure of an object according to any one of claims 1 to 8.   The joint structure acquisition method according to claim 9, wherein the joint motion variance is variance of joint angles of virtual joints.   The method according to claim 10, wherein the joint angle is an angle between two adjacent line segments connecting virtual joints.   10. The method for acquiring a joint structure of an object according to claim 9, wherein the joint motion variance is variance of inner products of two adjacent line segments connecting virtual joints.   The method for acquiring a joint structure of an object according to any one of claims 9 to 12, wherein a virtual joint whose variance exceeds a threshold value is estimated to be an actual joint.   The joint structure of an object according to any one of claims 9 to 12, wherein when an actual number N of joints is given, N virtual joints are selected in descending order of variance and are estimated to be actual joints. How to get   The method for acquiring a joint structure of an object according to claim 1, wherein the virtual joint model includes a graph structure including virtual joints constituting nodes and edges connecting the virtual joints.   The method for acquiring a joint structure of an object according to claim 1, wherein a local coordinate system of a virtual joint having three axes is set before or after an actual joint is estimated from the virtual joint.   17. The method for acquiring a joint structure of an object according to claim 16, wherein the setting of the local coordinate system of the virtual joint is performed by performing principal component analysis on the coordinates of contour lines and using the obtained three eigenvectors as local coordinates of the virtual joint.   18. The method for acquiring a joint structure of an object according to claim 16, wherein texture information of the object is used when setting a local coordinate system of the virtual joint.   19. The object according to claim 18, wherein the texture information of the object is used to identify one axis from the centroid of the contour line to the texture information as a texture axis, and the texture axis is replaced with one axis in the contour plane. How to get the joint structure of an object.   The method for estimating a joint structure of an object according to any one of claims 1 to 19, comprising a step of providing texture information of the object on the three-dimensional surface. Means for obtaining a three-dimensional shape of an object from images obtained from a plurality of cameras using a view volume intersection method;
Using a leave graph, define the average geodesic distance μ on the surface of the three-dimensional shape of the obtained object, acquire contour lines of the value of μ, and select virtual joints that are candidate joints based on each contour line Means for placing and obtaining a virtual joint model of the object;
Means for estimating a virtual joint with a lot of movement as an actual joint by comparing corresponding virtual joints between a plurality of virtual joint models acquired based on different postures of the object;
Means for obtaining a joint model of an object excluding joints other than virtual joints estimated as actual joints;
An apparatus for acquiring a joint structure of an object. In the means for obtaining the three-dimensional shape, the three-dimensional shape of the object is expressed as a polygon mesh,
In the means for acquiring the virtual joint model, an average geodesic distance μ is defined on the surface of the polygon.
The apparatus for acquiring a joint structure of an object according to claim 21. The means for acquiring the three-dimensional shape is:
Means for expressing the object visual volume with a polygon mesh;
Means for converting a visual volume polygon mesh into a convex polygon mesh;
Means for performing intersection processing between convex polygon meshes obtained from each camera;
The apparatus for acquiring a joint structure of an object according to claim 22.   24. The method for obtaining a joint structure of an object according to claim 23, wherein the intersection processing means includes reducing the number of polygons by reconstructing polygons when cutting out polygons of a convex polygon mesh in the intersection processing.   25. Any one of claims 23 and 24, wherein the intersection processing means includes combining convex polygon meshes after all the intersection processes are completed, or in addition, after each intersection process is completed. An apparatus for acquiring a joint structure of an object according to claim 1.   26. The apparatus for acquiring a joint structure of an object according to claim 21, wherein in the means for acquiring the virtual joint model, a virtual joint that is a candidate for a joint is arranged at the center of gravity of each contour line.   27. A target according to any one of claims 21 to 26, wherein the means for obtaining the virtual joint model comprises means for correcting the obtained virtual joint model based on a known joint phase structure of the object. An apparatus for acquiring the joint structure of an object.   28. The apparatus for acquiring a joint structure of an object according to claim 27, wherein the means for correcting the virtual joint model includes cutting off any one of branches constituting a loop in the joint structure.   The means for estimating a virtual joint with a lot of motion as an actual joint calculates the joint motion variance among the virtual joint models for each virtual joint, and determines a virtual joint with a lot of motion based on the obtained variance. The apparatus for acquiring a joint structure of an object according to any one of claims 21 to 28.   30. The apparatus for acquiring a joint structure of an object according to claim 29, wherein the joint motion variance is a variance of a joint angle of a virtual joint.   The said joint angle is an acquisition apparatus of the joint structure of the target object of Claim 30 which is an angle of two adjacent line segments which connect a virtual joint.   The joint structure acquisition device according to claim 30, wherein the joint motion variance is variance of inner products of two adjacent line segments connecting virtual joints.   The apparatus for acquiring a joint structure of an object according to any one of claims 30 to 32, wherein a virtual joint whose variance exceeds a threshold value is estimated to be an actual joint.   33. The joint structure of an object according to any one of claims 30 to 32, wherein when the actual number of joints N is given, N virtual joints are selected in descending order of estimation and are estimated to be actual joints. Acquisition device.   The apparatus for acquiring a joint structure of an object according to any one of claims 21 to 34, wherein the virtual joint model includes a graph structure including virtual joints constituting nodes and edges connecting the virtual joints.   36. The apparatus for obtaining a joint structure of an object according to any one of claims 21 to 35, comprising means for setting a local coordinate system of a virtual joint having three axes.   37. The apparatus for acquiring a joint structure of an object according to claim 36, wherein the setting unit of the local coordinate system of the virtual joint performs principal component analysis on the coordinates of the contour line and uses the obtained three eigenvectors as local coordinates of the virtual joint. .   38. The apparatus for acquiring a joint structure of an object according to claim 36 or 37, wherein the setting means of the local coordinate system of the virtual joint uses texture information of the object.   The setting means of the local coordinate system of the virtual joint uses the texture information of the target object to specify one axis from the center of gravity of the contour line to the texture information as a texture axis, and the texture axis is set as one axis in the contour plane. 40. The apparatus for acquiring a joint structure of an object according to claim 39, which is used as a replacement.   40. The apparatus for estimating a joint structure of an object according to any one of claims 21 to 39, comprising means for providing texture information of the object on the three-dimensional surface.