JP2019507934A - 3D motion evaluation apparatus, 3D motion evaluation method, and program - Google Patents

Abstract

The three-dimensional motion evaluation apparatus 100 includes a two-dimensional image corresponding point search unit 101 and a three-dimensional motion optimization unit 102. The two-dimensional image corresponding point search unit 101 searches for a dense two-dimensional corresponding point between subsequent frames and outputs a two-dimensional motion in pixel units between frame images. The 3D motion optimization unit 102 uses the depth of a single point observed in a single frame to obtain accurate 3D motion in the real world. By optimizing the error and optimizing the error of the two-dimensional motion in this pixel unit, the three-dimensional real world motion of the object is calculated.

Description

  The present invention relates to a three-dimensional motion evaluation apparatus, a three-dimensional motion evaluation method, and a program for realizing them, and further to the field of video three-dimensional reconstruction, more specifically, non-rigid 3 from a monocular image sequence. Related to dimensional motion evaluation.

  The field of 3D reconstruction from monocular image sequences has been the field of active research in the computer vision community for nearly 20 years. Three-dimensional reconstruction from images has found various applications in various fields such as animation, three-dimensional printing, video and image editing. Most conventional systems in this field operate in a camera-based manner in which the camera takes images of the desired object from various viewpoints. At this time, the image is used to calculate the structure of the object and the movement of the camera at the same time. The structure of an object is classified as a class in a three-dimensional reconstruction method widely used in this field based on the movement of a camera. Also, an image sequence is acquired, and the structure and camera motion are calculated using rank constraints on the image data. This stage is usually followed by a bundle adjustment stage that simultaneously optimizes the camera pose and object structure.

  The challenge in this area is to calculate the structure of a non-rigid object from only corresponding points in two dimensions efficiently and accurately. The structure of the high-density non-rigid body is solved by using a decomposition method, assuming a affine camera model (Non-Patent Document 1) based on a motion technique, applying rank constraints. However, the affine camera model assumes that the image formation is independent of the depth of the point from the optical center of the camera, so the transformation cannot be recovered along the optical axis.

  In order to solve this depth ambiguity problem, it is necessary to model a three-dimensional to two-dimensional motion mapping as perspective projection. A study conducted by Triggs et al. (Non-Patent Document 2) provides a formulation by factorization for rigid body structure and camera pose estimation under perspective projection. In recent years, the density of rigid structures due to movement has been successfully implemented by Ondruska et al. (Non-Patent Document 3) on portable platforms such as mobile phones and tablets using common stock cameras available on these platforms. .

  The above study deals with the example of rigid body structure by motion, but the final solution of structure and pose is on a very complex manifold, and the solution depends heavily on the initial species, so perspective projection The problem of non-rigid structure density from motion at is very difficult. Such optimization problems are often cumbersome and difficult to solve in real time. In many cases, prior knowledge is applied to the solution to constrain the space in which the solution exists. In the field of perspective projection based non-rigid body reconstruction by Vidal et al. (Non-Patent Document 4), research has been done for some time, but these methods are mainly based on reconstruction of sparse points. Yes, it cannot scale well for high density reconstructions.

  Newcombe et al. (Non-Patent Document 5) tried to calculate a standard rigid body model of an object in RGB-D-based input data, and calculated a three-dimensional motion between frames to obtain a standard rigid body model. , And generate an actual non-rigid deformation. This work is closest to the proposed invention, but uses only RGB information to calculate the 3D flow under a fixed perspective camera, making the problem much more difficult.

Garg, R .; Roussos, A .; Agapito, L., "Dense Variational Reconstruction of Non-rigid Surfaces from Monocular Video," in Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on, vol., No., pp.1272-1279, 23-28 June 2013 Triggs, B., "Factorizationmethods for projective structure and motion," in Computer Vision and Pattern Recognition, 1996. Proceedings CVPR '96, 1996 IEEE Computer Society Conference on, vol., No., Pp.845-851, 18-20 Jun 1996 Ondruska, P .; Kohli, P .; Izadi, S., "MobileFusion: Real-Time Volumetric Surface Reconstruction and Dense Tracking on Mobile Phones," in Visualization and Computer Graphics, IEEE Transactions on, vol.21, no.11, pp.1251-1258, Nov. 15 2015 Ren´e Vidal and Daniel Abretske, “Nonrigid Shape and Motion from Multiple Perspective Views”, ECCV 2006-European Conference on Computer Vision, 2014 Newcombe, RA; Fox, D .; Seitz, SM, "DynamicFusion: Reconstruction and tracking of non-rigid scenes inreal-time," in Computer Vision and Pattern Recognition (CVPR), 2015 IEEE Conference on, vol., No., pp.343-352, 7-12 June 2015

  Despite the progress described above, three-dimensional reconstruction of high density non-rigid bodies is a difficult problem as before. Therefore, the present invention has been made to solve the above-described problems.

  An example of an object of the present invention is to evaluate 3D motion, which can solve 3D motion from 2D corresponding points and warp 3D motion of the initial model for 3D reconstruction of non-rigid bodies An apparatus, a three-dimensional motion evaluation method, and a program are provided.

In order to achieve the above object, a three-dimensional motion evaluation apparatus according to one aspect of the present invention is a three-dimensional motion evaluation apparatus for calculating high-density non-rigid three-dimensional motion of an object from a monocular image,
A two-dimensional image corresponding point search unit that searches for dense two-dimensional corresponding points between subsequent frames and outputs a two-dimensional motion in pixel units between frame images;
In order to obtain accurate real-world 3D motion, the single-point depth observed in a single frame is used to optimize the 2D motion error in pixel units, A three-dimensional motion optimization unit that calculates the three-dimensional real-world motion of the object by optimizing the two-dimensional motion error of
It is characterized by having.

In order to achieve the above object, a three-dimensional motion evaluation method according to one aspect of the present invention is a three-dimensional motion evaluation method for calculating a high-density non-rigid three-dimensional motion of an object from a monocular image,
(A) searching for dense two-dimensional corresponding points between subsequent frames and outputting a two-dimensional motion in pixel units between frame images;
(B) In order to obtain accurate real-world 3D motion, the single point depth observed in a single frame is used to optimize the 2D motion error on a pixel-by-pixel basis. Calculating a three-dimensional real-world motion of the object by optimizing a two-dimensional motion error in pixel units;
It is characterized by having.

In order to achieve the above object, a program according to one aspect of the present invention is a program for calculating a high-density non-rigid three-dimensional motion of an object from a monocular image by a computer,
In the computer,
(A) searching for dense two-dimensional corresponding points between subsequent frames and outputting a two-dimensional motion in pixel units between frame images;
(B) In order to obtain accurate real-world 3D motion, the single point depth observed in a single frame is used to optimize the 2D motion error on a pixel-by-pixel basis. Calculating a three-dimensional real-world motion of the object by optimizing a two-dimensional motion error in pixel units;
Is executed.

  As described above, according to the present invention, the three-dimensional motion can be solved from the two-dimensional corresponding points, and the three-dimensional motion of the initial model can be warped for the three-dimensional reconstruction of the non-rigid body. .

The drawings together with the detailed description serve to explain the principle of the three-dimensional motion evaluation method of the present invention. The drawings are for illustration purposes and do not limit the application of the technology.
FIG. 1 is a schematic block diagram showing the configuration of the three-dimensional motion evaluation apparatus of the present invention. FIG. 2 is a block diagram showing a specific configuration of the three-dimensional motion evaluation apparatus according to the embodiment of the present invention. FIG. 3 is a workflow showing detailed steps in calculating a high-density three-dimensional motor area from two-dimensional corresponding points. FIG. 4 is a diagram illustrating a 2D to 3D motion mapping process under a perspective camera model. FIG. 5 is a diagram showing a comparison between an estimated shape of an object using the present invention and a three-dimensional shape of an actual object. FIG. 6 is a block diagram illustrating an example of a computer platform on which the three-dimensional motion evaluation apparatus is implemented.

(Summary of Invention)
[Technical issues]
The process of 3D reconstruction of non-rigid surfaces from 2D image sequences is a very difficult problem and depends on various conditions of the camera and object structure for reliable 3D pose and structure estimation . Motion modeling plays an important role in determining whether the object and camera configuration is recoverable. For example, in many high density structures resulting from motion problems (eg, as in Non-Patent Document 4), the affine camera model cannot recover the transformation along the optical axis. Therefore, when the camera is fixed and the object is moving along the optical axis, it is impossible to obtain an accurate solution using only the affine model. To solve this problem, one must assume a projection camera model that is inherently ambiguous in depth, resulting in deliberate problems and a computational scale for high-density reconstruction. It becomes difficult to up.

There are existing approaches that attempt to solve this problem, but these also have the following disadvantages: Most methods for processing projected non-rigid structures from motion assume low rank constraints and attempt to solve structures and shapes using decomposition methods. Such a method is suitable for sparse reconstruction, but in high-density reconstruction of objects, it cannot adjust well for the evolution over time of structure and pose.

  In addition to the entities described above, other obvious and obvious disadvantages that the present invention can overcome will become apparent from the detailed description and drawings. The outline for solving these problems is as follows.

[Problem explaining the problem]
In order to solve the technical problem mentioned above, the overall approach is outlined below. The process of 3D non-rigid reconstruction is treated as an incremental process and the shape in the current frame is composed of a combination of the previous frame shape and the 3D motion in the current frame. Such processing of the 3D reconstruction process reduces the above problem to the lower two problems, (i) identifies a reliable initial rigid body model of the object, and (ii) animates the rigid body model 3D motion is calculated for each frame, and as a result, 3D reconstruction of the non-rigid body in the scene is performed. Many techniques that can be used to generate a reliable initial model of an object exist in the literature (eg, Non-Patent Document 3).

  The technical advantage of the present invention is that, according to this method, a two-dimensional corresponding point is assumed, assuming a perspective camera projection model that is temporally and spatially coherent and robust to image and motion noise. It is possible to generate a high-density three-dimensional motor area. The framework of the proposed invention is incremental in nature. That is, the elucidation of the three-dimensional motor area is performed for each frame based on the previous frame. Also, since the proposed model uses a known absolute depth at least at one point, it can calculate 3D motion on an absolute scale in any direction of motion relative to the camera.

  Accordingly, the present invention relates to a number of steps, the relationship between one or more of these steps, an apparatus that embodies the features of the arrangement, a combination of elements, and such steps. The arrangement of parts optimized to influence is configured, all of which are illustrated in the following detailed disclosure, ie, the description of the drawings and the detailed description. The scope of the invention is indicated in the claims.

(Embodiment)
Hereinafter, a network system, a three-dimensional motion evaluation apparatus, a three-dimensional motion evaluation method, and a program according to an embodiment of the present invention will be described with reference to FIGS.

  Hereinafter, an example of an embodiment of the present invention will be described in detail. Implementations of the invention are described in full detail. The description provided herein along with the exemplary drawings provides a solid guide to those skilled in the art practicing the present invention.

  The present invention relates to calculating high density three-dimensional non-linear motion from corresponding points in a high density two-dimensional image using a perspective camera model. The present invention is broadly divided into calculation of high-density two-dimensional corresponding points and calculation of three-dimensional motions from these two-dimensional corresponding points. The latter calculation is also performed using a perspective projection model that constrains the solution by constraints that are consistent in time and space. This 3D motion is warped with the previous 3D shape to obtain the current 3D shape.

[Device configuration]
First, the configuration of the three-dimensional motion evaluation apparatus of the present invention will be described with reference to FIG. FIG. 1 is a schematic block diagram showing the configuration of the three-dimensional motion evaluation apparatus of the present invention.

As shown in FIG. 1, the three-dimensional motion evaluation apparatus 100 performs the above-described task. The three-dimensional motion evaluation apparatus can be further divided into various units. Here, the function of each unit will be described with reference to FIG.

  As shown in FIG. 1, the three-dimensional motion evaluation apparatus 100 includes a two-dimensional image corresponding point search unit 101 and a three-dimensional motion optimization unit 102. The two-dimensional image corresponding point search unit 101 searches for a dense two-dimensional corresponding point between subsequent frames and outputs a two-dimensional motion in pixel units between frame images. The 3D motion optimization unit 102 uses the depth of a single point observed in a single frame to obtain accurate 3D motion in the real world. By optimizing the error and optimizing the error of the two-dimensional motion in this pixel unit, the three-dimensional real world motion of the object is calculated.

  As described above, in this embodiment, the two-dimensional motion error in pixel units is optimized using the depth of a single point. Therefore, it becomes possible to solve the three-dimensional motion from the two-dimensional corresponding points, and the three-dimensional motion of the initial model is warped for the three-dimensional reconstruction of the non-rigid body.

  Subsequently, the three-dimensional motion evaluation apparatus 100 in the present embodiment will be described in more detail with reference to FIG. FIG. 2 is a block diagram showing a specific configuration of the three-dimensional motion evaluation apparatus according to the embodiment of the present invention.

  As shown in FIG. 2, the three-dimensional motion evaluation apparatus 100 includes a three-dimensional motion warping unit 103 in addition to a two-dimensional image corresponding point search unit 101 and a three-dimensional motion optimization unit 102.

  In the proposed three-dimensional motion evaluation apparatus 100, the first unit is a two-dimensional image corresponding point search unit 101. The process of 3D motion evaluation begins by first evaluating 2D corresponding points between image frames. As can be seen from FIG. 1, the image sequence 200 is input to the two-dimensional image corresponding point search unit 101, and the two-dimensional image corresponding point search unit 101 calculates a two-dimensional motion in units of pixels between subsequent frames. The two-dimensional image corresponding point search unit 101 calculates a high-density two-dimensional corresponding point by comparing image patches in consecutive frames having the same image intensity for each image pair. One method for finding two-dimensional corresponding points between images is optical flow. The optical flow is determined for each pixel in the reference frame using the luminance constant assumption, and the motion vector is calculated densely using local optimization or global optimization. Another way to find the two-dimensional image corresponding points is by a feature tracking technique. In the feature tracking technique, a feature descriptor is calculated around each pixel in the reference frame that matches the target frame, and a two-dimensional motion vector is calculated. One of two methods, one of optical flow method and one of feature tracking based method, or similar method is used for two-dimensional motion search.

  The next unit is a three-dimensional optimization unit 102 that finds a corresponding three-dimensional motion from a two-dimensional corresponding point. The high-density two-dimensional corresponding points obtained from the two-dimensional image search point search unit 101 are used for calculation of three-dimensional motion by assuming a perspective camera model. When the absolute depth 201 at a single point in the object is available, the 3D motion optimization unit 102 projects the current constant value of the 3D motion onto the image plane using the perspective projection model.

The absolute depth at a single point can be obtained using a commercially available laser depth sensor. Image-based methods such as triangulation of known patterns on the object can also be used to obtain absolute depth. This projected two-dimensional motion is compared with the observed two-dimensional motion, and an optimization algorithm minimizes the error between them. Variational optimization techniques are used to resolve motion updates at each step. This minimizes the error between the projected 2D motion and the observed 2D motion. The energy function used in variational optimization also ensures that spatial and temporal consistency is maintained in the 3D motion solution. This ensures the robustness of the final solution against outliers. When the optimization converges, the three-dimensional motion optimization unit 102 provides an output as the optimal three-dimensional motion in the current frame.

  The three-dimensional motion warping unit 103 projects a three-dimensional motion into a two-dimensional motion using a perspective projection model that assumes only relative conversion between the camera and the object. The three-dimensional motion warping unit 103 calculates an error between a two-dimensional motion in a pixel unit in the current frame and a modeled absolute three-dimensional motion to a single parameter with a camera intrinsic parameter. Calculation is performed using the depth value of a single point in the frame. The three-dimensional motion warping unit 103 minimizes the above-described error. Also, the three-dimensional motion warping 103 preserves the spatial smoothness of each frame and the temporal smoothness between frames for optimal three-dimensional motion, and minimizes the above-mentioned error in a three-dimensional real-world motion. Is calculated. In addition, the 3D motion warping unit 103 may receive a previous non-rigid 3D shape as an input, and add the calculated 3D motion of the current frame to update the current non-rigid 3D model. it can. Then, the three-dimensional motion warping unit 103 outputs a three-dimensional shape in the current frame 203.

In the next step, the current 3D motion is warped by the 3D motion warping unit 103 together with the previous shape 202 of the object obtained in the previous step, and the final shape in the current frame 107 is obtained. This is achieved by expressing the three-dimensional shape as a two-dimensional mesh. In this case, connection information of several edges is included in each vertex of the three-dimensional position. When the optimal motion is found in each frame, the 3D position of each vertex in the mesh is updated using the calculated current 3D motion and the mesh vertex position in the previous frame, and At this time, the edge connection between the vertices is not changed and is maintained as it is.
This achieves a final non-rigid shape reconstruction of the object from the corresponding image sequence.

[Device operation]
Next, the entire calculation process of the high-density three-dimensional motion from the high-density two-dimensional corresponding points and the search of the three-dimensional structure based on the result will be described.

  The operation of the three-dimensional motion evaluation apparatus 100 according to the embodiment of the present invention will be described with reference to FIG. FIG. 3 is a flowchart showing the operation of the three-dimensional motion evaluation apparatus in the embodiment of the present invention. In the following description, FIGS. 1 and 2 are referred to as appropriate. In the present embodiment, the three-dimensional motion evaluation method is executed by operating the three-dimensional motion evaluation device 100. Therefore, the following description of the operation of the three-dimensional motion evaluation apparatus 100 is replaced with the description of the three-dimensional motion evaluation method in the present embodiment.

  The two-dimensional image corresponding point search unit 101 of the system acquires the previous image frame and the current image frame as inputs (step 301). Next, the two-dimensional image corresponding point search unit 101 calculates a two-dimensional density corresponding point from these images (step 302).

  Next, the two-dimensional image correspondence search unit 101 determines whether or not the two-dimensional motion frame that is the current input is the first frame in the sequence (step 303). In the case of the first frame, the two-dimensional image corresponding point search unit 101 calculates an initial value for three-dimensional motion assuming that the object is a rigid body in units of blocks, and for each block. The movement is calculated (step 304).

  If it is not the first frame, the two-dimensional image corresponding point search unit 101 assumes a constant velocity model using the previous three-dimensional motion field, and calculates an initial value for the three-dimensional motion in the current frame (step). 305).

  Next, the iterative process starts. The two-dimensional image corresponding point search unit 101 projects a three-dimensional motion on the image plane using the known absolute depth (step 306). Next, the two-dimensional image corresponding point search unit 101 calculates an error between the projected two-dimensional motion and the observed two-dimensional motion (step 307).

  Next, the three-dimensional motion optimization unit 102 determines whether the error is less than a specific threshold (step 308). If the error is not less than a specific threshold, the three-dimensional motion optimization unit 102 performs an optimization step, thereby reducing a two-dimensional motion error between the projected motion and the observed motion. Thus, the three-dimensional motion is updated (step 309).

  When the error is less than a specific threshold, the 3D motion warping unit 103 warps the shape of the object in the previous frame 310 together with the optimal 3D motion obtained from the 3D motion optimization unit 102. As a result, the final output of the algorithm is a three-dimensional shape in the current frame (step 311).

  Here, the mathematical details of the present invention will be described with reference to FIG. The motion mapping of the perspective camera will be described with reference to FIG.

  In FIG. 4, the non-rigid object is shown in time frame t (401) and time frame t + 1 (402). As shown in FIG. 4, the shape of the object is deformed. The specific point Xt (403) moves to the point Xt + 1 (404) together with T = (Tx, Ty, Tz) T (408) which is a three-dimensional motion between the points. These points are arranged at a point mt (406) and a point mt + 1 (407) on the image plane 405. The three-dimensional motion is registered as a two-dimensional motion, ot = mt + 1−mt, on the image plane. The object of the present invention is to search for a three-dimensional motion (T) from a corresponding two-dimensional motion (ot) using the perspective projection model shown in FIG. The camera is assumed to be fixed in space. It is also assumed that the three-dimensional motion between frames can be accurately modeled only by translational motion.

  From the above discussion, if the two components are represented as ot = (mxt, myt) T and the two-dimensional motion in the image is represented as ot = mt + 1-mt, the three-dimensional motion T = (Tx, Ty, Tz) T And the two-dimensional motion (mxt, myt) T at any given pixel is given as:

  Here, u and v are zero-centered image coordinate systems, f is a focal length, and Z is an absolute distance from the camera optical center. The absolute depth at a point can be obtained using a commercially available laser depth sensor. Image-based methods such as triangulation of known patterns on the object can also be used to obtain absolute depth. The present invention tries to calculate the three-dimensional motion T from the two-dimensional motion for each pixel.

Assuming that the camera is fixed and the object is in a nonlinear translation in free space, we aim to find a dense 3d motion between subsequent frames from a dense 2d motion correspondence. Assuming that the image sequence is given by the set {It, t = 1,2, ... N}, N is the number of images in the sequence. Between the image It and the image It-1, the two-dimensional motor area is represented by mt. Image It and Image
Assume that the three-dimensional motion field between It-1 is Mt, and the three-dimensional motion of each previous frame, Mt-1, is stored in the buffer during the calculation of the three-dimensional motion in the current frame Mt. Next, the number 2 is framed as an energy function to calculate the three-dimensional motion in the current frame. Equation 2 minimizes the value of the optimal 3D motion Mt *.

  Equation 2 above gives the global energy function to be reduced. The data term Ed depends on the observed two-dimensional motion and the estimated value of the three-dimensional motion. The data term uses the observed 2D to measure the error of the 2D motion projected onto the image plane, and penalizes the solution if the error is large. The data term is generally given by Equation 3.

  Ψ (.) Is a robust weight function that penalizes outlier errors that are less critical than the conventional L2 norm. A sum of errors for each pixel of interest is obtained over the entire image.

  The second term of the energy function is a smooth term that plays a role in maintaining the spatial and temporal smoothness of the solution. Spatial smoothness refers to the final three-dimensional motion solution being smooth along the XY axis of the image and without steep discontinuities. That is, there should be a strong correlation between the three-dimensional motions of adjacent pixels. This is expressed as a spatial gradient of the three-dimensional motion in the XY axis in image coordinates. Temporal coherence refers to the fact that for a given image, the speed should not change abruptly with time, and the temporal change in the three-dimensional speed of the pixel should be smooth. This appears in the energy term as a gradient along the time axis. Therefore, the smooth term Es is given as in Equation 4.

  Here, ∇ = (δ / δx, δ / δy, δ / δt) represents a three-dimensional motion vector along the X and Y axes of the image. The weight function φ (.) Is a robust kernel similar to Ψ (.). The sum of smoothness for each pixel is obtained over the entire image. The 3D motion in the previous frame is necessary to calculate the gradient of the motion in time and needs to be buffered during the calculation of the 3D motion in the current frame.

In order to obtain an appropriate three-dimensional motion from the above equation, various techniques for searching for an overall optimization solution are employed. Since the energy function is the sum of the errors of all terms, the resulting solution is an overall optimal solution that is robust to outliers and errors. The weight functions φ (.) And Ψ (.) Are selected as convex functions that ensure that the optimization converges overall. Optimization depends on initialization of three-dimensional motion, and initialization close to global optimization can result in faster convergence. If the motion frame is the first frame in the sequence, there is no past information about the motion. In this case, the initialization is performed and calculated assuming that the motion is a block-based rigid body and the three-dimensional motion is each block. For the other frames, consider a constant velocity model that assumes that the motion between successive frames changes very slowly with time. With this assumption, the 3D motion is initialized as the 3D motion in the previous frame.

  In order to express the three-dimensional structure of the object to be processed, a two-dimensional mesh structure represented as a graph Gt = (Vt, E) that evolves with time is used. Here, Vt is a set of all vertices that evolve in time, and E is a set of unchanging edges that includes information on how the vertices are connected. Each vertex in the vertex set includes the three-dimensional position of each point. Point cloud data can be obtained from only the vertices, and edge information can be used to construct a three-dimensional surface. The output of the 3D motion from the variational optimization step is warped to the previous shape and generates the current 3D shape of the object. This is achieved by updating the three-dimensional position of the vertices as Vt + 1 = Vt + mt without changing the edge connectivity of the graph. In this way, a mesh that progresses with time is obtained. A mesh is used to represent the three-dimensional structure of an object that evolves over time. In the first frame, it is necessary to estimate the shape of the object. There are many methods described in the literature that can be used for this purpose. To estimate the initial shape of the object, it is possible to create a rigid structure from motion using the first few frames. Photometric approaches such as shape from shading can also be used to obtain the initial object shape.

  Experiments were performed to verify the effectiveness of the present invention. An image sequence of surface bending under sinusoidal motion along the x axis was simulated. The present invention was used to estimate the time evolution of the surface under motion using the monocular image sequence described above. The experimental results are as shown in FIG. FIG. 5 shows the estimated three-dimensional motion 602 of the object and the actual three-dimensional motion 601 of the object. This plot shows a particular frame among all frames in the image sequence. As can be seen from this comparison, the present invention performs a reconstruction of a three-dimensional shape that closely matches the actual ground data. In this way, the proposed scheme is verified.

FIG. 5 shows a block diagram illustrating an embodiment of a computer and network system in which an embodiment of the methodology of the present invention is implemented. The system shows a computer with an input device and a network connection. The computer platform 502 is an EEPROM (Electrically Erasable and Programmable Read Only) for storing data and instructions.
To connect to a host or client system using a memory (RAM) and a RAM (Random Access Memory), a CPU (Central Processing Unit) and a data bus for processing information and executing instructions, and a local network or the Internet 504 Network card. The computer platform may be connected to the basic / input / output device 501. The computer platform may include, for example, a keyboard, a mouse, a display, and an external storage device.

Finally, the processes, techniques, and methodologies described and illustrated herein are not limited to or related to a particular device, and this is clear. Implementation of the present invention is possible by a combination of components. In addition, various types of general-purpose devices can be used in accordance with the instructions in this specification. Furthermore, the present invention has been described using a specific set of examples. However, these are merely examples and are not limiting. For example, the described software may be implemented in a wide variety of languages such as C ++, Java, Python, Perl. Furthermore, other implementations of the technology of the present invention will be apparent to those skilled in the art.

DESCRIPTION OF SYMBOLS 100 3D motion evaluation apparatus 101 2D corresponding point search part 102 3D motion optimization part 103 3D motion warping part 200 Image sequence 201 Single point depth 202 Previous shape 203 3D shape in present frame 401 Time frame t
402 Time frame t + 1
403 Specific point Xt
404 points Xt + 1
405 Image plane 406 points mt
407 points mt + 1
408 T = (Tx, Ty, Tz) T
501 Actual three-dimensional shape of object 502 Estimated three-dimensional shape of object 601 Basic input / output device 602 Computer platform 603 Client system 604 Local network or Internet 605 Host system

Claims (9)

A three-dimensional motion evaluation apparatus for calculating a high-density non-rigid three-dimensional motion of an object from a monocular image,
A two-dimensional image corresponding point search unit that searches for dense two-dimensional corresponding points between subsequent frames and outputs a two-dimensional motion in pixel units between frame images;
In order to obtain accurate real-world 3D motion, the single-point depth observed in a single frame is used to optimize the 2D motion error in pixel units, A three-dimensional motion optimization unit that calculates the three-dimensional real-world motion of the object by optimizing the two-dimensional motion error of
A three-dimensional motion evaluation apparatus comprising: The three-dimensional motion optimization unit includes:
Projecting 3D motion to 2D using a perspective projection model that only assumes relative transformation between the camera and the object;
The error between the 2D motion in pixels in the current frame and the modeled absolute 3D to 2D motion is expressed as the camera intrinsic parameter and the depth of a single point in a single frame. Using the value and
Storing the spatial smoothness for each frame and temporal smoothness between frames for the optimal three-dimensional motion, minimizing the error, and calculating a three-dimensional real-world motion that minimizes the error;
The three-dimensional motion evaluation apparatus according to claim 1. A 3D motion warping unit that receives a previous non-rigid 3D shape as input, adds the calculated 3D motion of the current frame, and updates the current non-rigid 3D model;
The three-dimensional motion evaluation apparatus according to claim 1 or 2. A three-dimensional motion evaluation method for calculating a high-density non-rigid three-dimensional motion of an object from a monocular image,
(A) searching for dense two-dimensional corresponding points between subsequent frames and outputting a two-dimensional motion in pixel units between frame images;
(B) In order to obtain accurate real-world 3D motion, the single point depth observed in a single frame is used to optimize the 2D motion error on a pixel-by-pixel basis. Calculating a three-dimensional real-world motion of the object by optimizing a two-dimensional motion error in pixel units;
A three-dimensional motion evaluation method characterized by comprising: In step (b),
Projecting 3D motion to 2D using a perspective projection model that only assumes relative transformation between the camera and the object;
The error between the 2D motion in pixels in the current frame and the modeled absolute 3D to 2D motion is expressed as the camera intrinsic parameter and the depth of a single point in a single frame. Using the value and
Storing the spatial smoothness for each frame and temporal smoothness between frames for the optimal three-dimensional motion, minimizing the error, and calculating a three-dimensional real-world motion that minimizes the error;
The three-dimensional movement evaluation method according to claim 4. (C) receiving a previous non-rigid 3D shape as input, adding the calculated 3D motion of the current frame and updating the current non-rigid 3D model;
In addition,
The three-dimensional motion evaluation method according to claim 5 or 6. A program for calculating a high-density non-rigid three-dimensional motion of an object from a monocular image by a computer,
In the computer,
(A) searching for dense two-dimensional corresponding points between subsequent frames and outputting a two-dimensional motion in pixel units between frame images;
(B) In order to obtain accurate real-world 3D motion, the single point depth observed in a single frame is used to optimize the 2D motion error on a pixel-by-pixel basis. Calculating a three-dimensional real-world motion of the object by optimizing a two-dimensional motion error in pixel units;
A program characterized by having executed. In step (b),
Projecting 3D motion to 2D using a perspective projection model that only assumes relative transformation between the camera and the object;
The error between the 2D motion in pixels in the current frame and the modeled absolute 3D to 2D motion is expressed as the camera intrinsic parameter and the depth of a single point in a single frame. Using the value and
Storing the spatial smoothness for each frame and temporal smoothness between frames for the optimal three-dimensional motion, minimizing the error, and calculating a three-dimensional real-world motion that minimizes the error;
The program according to claim 8. In the computer,
(C) receiving a previous non-rigid 3D shape as input, adding the calculated 3D motion of the current frame, and updating the current non-rigid 3D model;
The program according to claim 7 or 8.