JP2017123087A - Program, device and method for calculating normal vector of planar object reflected in continuous photographic images - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a program or the like capable of calculating a normal vector of a planar object reflected in continuous photographic images in a drastically small amount of calculation without being greatly restricted by camera work on precondition that a target object is a planar object (or an approximatable object in an almost plane surface).SOLUTION: The program has normal vector calculation means for calculating camera attitude parameters Rand t, and a normal vector nfor minimizing a reprojection error function by using Nc frames i of continuous photographic images, three-dimensional coordinates Xof Np pieces of registration points pdetected from a registered image of a planar object, and the tracking coordinates mof each of the Np pieces of registration points preflected in each frame. The normal vector calculation means causes a computer to function so as to express the three-dimensional coordinates Xof the registration points pof the registered image by the registration points p, a reference point Xgoing through an object plane, and the normal vector nand calculate the three-dimensional coordinates Xby back projection of the registration points pto the object plane.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a technique for calculating a normal vector (normal direction) of a planar object from an image captured by a camera.

  According to the technology of computer vision or robot vision, it is possible to estimate and track the position and orientation of the target object reflected in the video by analyzing the video taken by the camera. For example, there is a technique for tracking a vehicle shown in continuous captured images of a surveillance camera (see, for example, Patent Document 1). According to this technique, a target object in an image is tracked by template matching for each frame. However, only the position of the vehicle shown in the image is tracked, and the direction (direction) cannot be tracked.

On the other hand, according to the augmented reality technology that superimposes and displays virtual information on the target object, it is possible to estimate and track the 6-DOF position and orientation of the target object with respect to the camera and display augmented reality with a high sense of reality. . For example, by placing a virtual object on the target object specified in the image and estimating / tracking the position and orientation of the target object, the virtual object can be displayed as if it exists in the specified area. (See, for example, Patent Document 2).
There is also a technique for acquiring a three-dimensional structure of a target object in real time with a 3D sensor and estimating / tracking the position and orientation (see, for example, Patent Document 3).
Furthermore, there is a technique for estimating a three-dimensional structure of a captured image using only a monocular camera (see, for example, Patent Document 4).

Japanese Patent No. 36551745 JP 2013-164597 A JP 2014-515991 A JP 2014-149582 A Japanese Patent Application Laid-Open No. 2015-069354

A. Ruiz et al., "Practical planar metric rectification," In Proc. Of British Machine Vision Conference, 2006. A Mulloni et al., "User friendly SLAM initialization," in Proc. Of IEEE International Symposium on Mixed and Augmented Reality, 2013. F Yu et al., "3D Reconstruction from Accidental Motion," in Proc. Of IEEE Conference on Computer Vision and Pattern Recognition, 2014.

  However, according to the above-described prior art, when the position and orientation of the target object are estimated and tracked using continuous captured images of a monocular camera, if the three-dimensional structure of the target object is unknown, Unable to predict changes in appearance due to changes. For this reason, robustness is significantly impaired.

  Further, when only a monocular camera is used, there are high restrictions on processing load, estimation accuracy, and camera work (how to move the camera during shooting). For example, it is necessary to move the camera so as to acquire an image group including a large viewpoint change.

On the other hand, under the precondition that the target object shown in the captured image is a planar object (or an object that can be approximated by a plane), the amount of calculation can be reduced and the estimation accuracy can be increased. By expressing the plane of the three-dimensional structure of the target object with a “normal vector”, the number of unknown parameters that affect the amount of calculation can be greatly reduced.
If the target object is a plane, the direction of the target object can be estimated from, for example, a homography matrix between images (see, for example, Non-Patent Document 1).
There is also a technique for improving estimation accuracy when a target object includes a plurality of planes (see, for example, Patent Document 5).
However, these techniques also have restrictions on camera work.

Furthermore, as a technique for eliminating the restrictions of the existing camera work, there is a technique for estimating a three-dimensional structure of a shooting scene using bundle adjustment from continuous images shot with a small camera work (for example, Non-Patent Document 2, 3). According to these techniques, by assuming a small camera work, the parameters to be estimated are restricted and the accuracy of the initial parameters is improved.
However, according to this technique, there is a restriction on the pattern of the target object to be specified, and practical accuracy can be obtained only when feature points (image features) are detected from the target object. Moreover, the robustness with respect to the feature point tracking failure is poor.

  Therefore, the present invention provides a method for a planar object that is reflected in a continuous photographed image under the precondition that the target object is a planar object (or an object that can be approximated by a plane) without significant restrictions on camera work. Provided are a program, an apparatus, and a method capable of calculating a line vector with as little calculation as possible.

According to the present invention, in a program that causes a computer to function to calculate a normal vector of a planar object reflected in a captured image,
Nc frames i of continuous shot images;
Three-dimensional coordinates X _j (= [x _j , y _j , z _j ) of Np registration points p _j (= [u _j , v _j ] ^T , j = 1 to Np) detected from the registered image of the planar object. ] ^T ),
Using tracking coordinates m _ij (= [u _ij , v _ij ] ^T , i = 1 to Nc) for each of Np registration points p _j reflected in each frame,
Camera orientation parameters R _i (= [r _ix , r _iy , r _iz ] ^T ) and t _i (= [t _ix , t _iy , t _iz ] ^T ) that minimize the reprojection error function, and normal vectors normal vector calculation means for calculating n _t (= [x _n , y _n , z _n ] ^T , x _n ² + y _n ² + z _n ² = 1),
Normal vector calculating means, the three-dimensional coordinates X _j of registration points p _j of the reference image, a registration point p _j, the reference point _X 0 through the object plane _{(= [x 0, y 0} , z 0] T) And a normal vector n _t, and the computer is made to function so as to be calculated by back projection of the registration point p _j onto the object plane.

According to another embodiment of the program of the present invention,
The normal vector calculation means uses the reference point X ₀ as a back projection (X ₀ = 1 / w ₀ [u ₀ , v ₀ , 1) of the center of gravity p ₀ (= [u ₀ , v ₀ ] ^T of the registration point p _j. It is also preferred to have the computer function so as to calculate according to ^T ).

According to another embodiment of the program of the present invention,
For normal vector calculating means, the reprojection error function, R _i represented by the formula _{', t i', n t} '= arg min Ri, ti, nt Σ i = 1 Nc Σ j = 1 Np ( m _ij −proj (R _i , t _i , X _j )) ²
X _j = (n _t · X ₀ / n _t · p _j ') p _j '
= (X _n x ₀ + y _n y ₀ + z _n z ₀ ) / (x _n u _j + y _n v _j + z _n ) [u _j , v _j , 1] ^T
x _n ² + y _n ² + z _n ² = 1
z _n = √ (1−x _n ² −y _n ² )
p _j ′ (= [u _j , v _j , 1] ^T ): homogeneous coordinate expression of the registration point p _j
i: Number of Nc frames of the captured image m _ij : Tracking coordinates for each of Np registration points p _j reflected in frame i R _i and t _i : Camera posture parameters of frame i n _t : Normal of plane object Vector X ₀ : Reference point through which the object plane passes
proj (R _i , t _i , X _j ): projection function [R _i | t _i ] X _j 'of the three-dimensional coordinate X _j
X _j ': _Homogeneous coordinate representation of X _j R _i ' and t _i ': Estimated values of camera posture parameters R _i and t _i of frame i n _t ': Estimated value of normal vector n _t of planar object It is also preferable to make the computer function.

According to another embodiment of the program of the present invention,
Regarding the normal vector calculation means, the number of unknown parameters in the reprojection error function is the camera posture parameters R _i (= [r _ix , r _iy , r _iz ] ^T ) and t _i (= [t _ix , t _iy , t _iz ] ^T ) and normal vectors n _t (= [x _n , y _n , z _n ] ^T , x _n ² + y _n ² + z _n ²⁾ = 1) It is also preferable to make the computer function so that 6Nc + 2 is obtained by adding the two.

According to another embodiment of the program of the present invention,
The normal vector calculation means, under the precondition that the camera work of the photographed image is minute, sets the computer to calculate the normal vector with the camera orientation parameter Ri as the unit matrix and t _i as the zero vector. It is also preferable to make it function.

According to another embodiment of the program of the present invention,
Regarding the normal vector calculation means, the reprojection error function causes the computer to function so that the initial value of the normal vector in bundle adjustment is a direction n _t = [0,0,1] ^T parallel to the optical axis. Is also preferable.

According to another embodiment of the program of the present invention,
It is also preferable that the computer further function as an image feature tracking unit that excludes mistracked tracking coordinates using a homography matrix between the registered points of the registered image and the tracking coordinates of the captured image.

According to another embodiment of the program of the present invention,
It is also preferable to specify a target area in which a planar object appears in accordance with a user operation from the captured images, and further cause the computer to function as registered image storage means for storing the target area as a registered image.

According to another embodiment of the program of the present invention,
The registered image storage means preferably causes the computer to function so as to geometrically transform the registered image into a frontalized image using the normal vector calculated by the normal vector calculating means, and store the frontaled image as a registered image. .

According to the present invention, in an image processing apparatus that calculates a normal vector of a planar object reflected in a captured image,
Nc frames i of continuous shot images;
Three-dimensional coordinates X _j (= [x _j , y _j , z _j ) of Np registration points p _j (= [u _j , v _j ] ^T , j = 1 to Np) detected from the registered image of the planar object. ] ^T ),
Using tracking coordinates m _ij (= [u _ij , v _ij ] ^T , i = 1 to Nc) for each of Np registration points p _j reflected in each frame,
Camera orientation parameters R _i (= [r _ix , r _iy , r _iz ] ^T ) and t _i (= [t _ix , t _iy , t _iz ] ^T ) that minimize the reprojection error function, and normal vectors normal vector calculation means for calculating n _t (= [x _n , y _n , z _n ] ^T , x _n ² + y _n ² + z _n ² = 1),
Normal vector calculating means, the three-dimensional coordinates X _j of registration points p _j of the reference image, a registration point p _j, the reference point _X 0 through the object plane _{(= [x 0, y 0} , z 0] T) And a normal vector n _t and calculated by back projection of the registration point p _j onto the object plane.

According to the present invention, in the normal vector calculation method of the apparatus for calculating the normal vector of the planar object reflected in the captured image,
Nc frames i of continuous shot images;
Three-dimensional coordinates X _j (= [x _j , y _j , z _j ) of Np registration points p _j (= [u _j , v _j ] ^T , j = 1 to Np) detected from the registered image of the planar object. ] ^T ),
Using tracking coordinates m _ij (= [u _ij , v _ij ] ^T , i = 1 to Nc) for each of Np registration points p _j reflected in each frame,
Camera orientation parameters R _i (= [r _ix , r _iy , r _iz ] ^T ) and t _i (= [t _ix , t _iy , t _iz ] ^T ) that minimize the reprojection error function, and normal vectors n _t (= [x _n , y _n , z _n ] ^T , x _n ² + y _n ² + z _n ² = 1)
This step, the three-dimensional coordinates X _j of registration points p _j of the reference image, a registration point p _j, the reference point _X 0 through the object plane and _{(= [x 0, y 0} , z 0] T), Law It is expressed by the line vector n _t and is calculated by back projection of the registration point p _j onto the object plane.

  According to the program, the apparatus, and the method of the present invention, it is possible to continuously capture captured images without a large limitation of camera work under the precondition that the target object is a planar object (or an object that can be approximated by a plane). The normal vector of the planar object to be reflected can be calculated with a significantly small amount of calculation. Specifically, since the number of unknown parameters in bundle adjustment for calculating a normal vector is reduced, the processing load can be reduced and the estimation accuracy can be increased. Further, by using the homography matrix, it is possible to improve the robustness against the tracking failure of the feature points.

It is a functional block diagram of the image processing apparatus in this invention. It is explanatory drawing showing the normal vector of a registration image and a front-facing image. It is explanatory drawing showing the relationship between a normal vector and a front-facing rotation matrix. It is explanatory drawing showing the correspondence between the registration point of a registration image, and the tracking coordinate of a picked-up image. It is an image figure showing a response | compatibility with a registration point and a tracking coordinate. It is an image figure showing the feature point (tracking coordinate) expressed by the number of different unknown parameters. It is explanatory drawing showing the geometric relationship of a registration point, a reference point, a normal vector, and an object plane. It is a graph showing the geometric relationship between the optical axis center, the registration point of the registration image on the image plane, and the normal vector of the captured image. It is a graph showing the convergence and processing time of the normal vector with respect to the number of frames according to bundle adjustment.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a functional configuration diagram of an image processing apparatus according to the present invention.

  The image processing apparatus of FIG. 1 includes an image acquisition unit 10, a registered image storage unit 11, an image feature tracking unit 12, and a normal vector calculation unit 13. These functional components are realized by executing a program that causes a computer installed in the apparatus to function. Further, the processing flow of these functional components can be understood as a normal vector calculation method of the apparatus.

According to the present invention, when calculating the normal vector of a planar object reflected in continuous captured images, the number of unknown parameters of the three-dimensional structure is reduced by assuming the following two conditions: Run with as little computation as possible.
(Precondition 1) The target object is a plane object (or an object that can be approximated by a plane) (Precondition 2) The camera work of the captured image is very small.

  The “target object” reflected in the captured image is a planar “rigid” “planar object” and does not target an object that deforms over time. Further, in order to estimate the normal vector by feature point tracking, it is assumed that a relatively large number of feature points are detected from the target object shown in the captured image. However, in the case of a single color plane, it may be difficult to detect feature points. As target objects, for example, magazines, posters, advertisements, trading cards, and building walls are assumed.

  “Camera work” means how the user moves the camera. “The camera work is very small” means a state in which the user holds the camera so as not to move as much as possible and photographs the target object. On the other hand, “the camera work is large” means a state in which the user moves the camera greatly to photograph the target object from various positions and directions.

[Image acquisition unit 10]
The image acquisition unit 10 acquires continuous captured images from the camera. The captured image may be recorded in advance, or may be input in a time series from the outside via an interface (for example, live video). The interface may be a communication interface connected to a network or an input interface from a camera. The acquired captured image is output to the image feature tracking unit 12. Here, when the registered image is designated by the user, the acquired captured image is also output to the registered image storage unit 11.

[Registered image storage unit 11]
When starting the posture estimation of a planar object in a captured image, the user sets a registered image via the user interface and instructs the start of posture estimation. The registered image storage unit 11 stores a registered image serving as a target area from among the captured images input from the image acquisition unit 10.

  The registered image storage unit 11 can accept an instruction of a rough image range in which a planar object is reflected from the user via the user interface. For example, the user instructs a rectangular region or contour that includes a planar object in accordance with a pointing operation. Here, it is preferable that the registration object has a relatively large image of the planar object. This registered image is tracked in the captured images that are continuously input.

  The registered image storage unit 11 can improve the robustness of image feature tracking with respect to the background by trimming the image range of the registered image and applying image processing to fill the background outside the image range with a single color. . In addition, a mask image representing the target object (an image expressed by setting the luminance value of the target region to 255 and the other luminance values to 0 within the same range as the registered image) is held, and an image is generated using the mask image. The feature tracking unit 12 may limit the range in which image features are detected.

  FIG. 2 is an explanatory diagram showing normal vectors of the registered image and the front-facing image.

According to FIG. 2, the registered image is geometrically transformed using the normal vector n _t = [ _x n, yn, z n] ^T calculated by the normal vector calculator 13. As a result, the registered image in which the planar object is reflected can be converted into a frontalized image (a converted image simulating an image when the planar object is imaged from the front). When a planar object is photographed from the front, the normal vector is perpendicular to the image plane and parallel to the optical axis. On the other hand, when a planar object is photographed from the side, the normal vector has a relationship perpendicular to the optical axis and parallel to the image plane.

  The registered image storage unit 11 can store the frontal image. By converting the registered image into a frontalized image, the accuracy of image recognition and posture tracking can be improved.

  FIG. 3 is an explanatory diagram showing the relationship between the normal vector and the front-facing rotation matrix.

According to FIG. 3, the rotation matrix Rrec is transformed so that the normal vector n _t of the planar object coincides with a vector n _Z parallel to the optical axis (Z axis).
Rrec = R _Y (θ _Y ) R _X (θ _X )
R _X : rotation matrix around the X axis
R _Y : rotation matrix around Y axis
theta _X: angle along the X-axis of the n _t
theta _Y: angle along the Y-axis of the n _t

Using this rotation matrix Rrec, the registered image can be fronted. For example, the registered image can be converted into a front-facing image by the following homography matrix Hrec.
Hrec = [p ₁ , p ₂ , p ₄ ]
Prec = [p ₁ , p ₂ , p ₃ , p ₄ ] = A [Rrec | t]
t: Translation vector for adjusting the position of the planar object in the registered image Here, t is preferably adjusted and set so that the planar object in the frontalized image comes to the center of the image.

The internal parameter A of the camera can be expressed as follows when image distortion is ignored.
f _{x, f y:} focal length, c _x, c _y: displacement focal length f _x of the optical axis, f _y and shift c _x of the optical axis, c _y may be kept calculated by prior calibration . Two-dimensional pixel coordinates [u, v] ^T on the image plane can be converted into normalized coordinates [u ′, v ′] ^T by the following formula using internal parameters.
u ′ = (u−c _x ) / f _x , v ′ = (v−c _y ) / f _y
Thus, by expressing the two-dimensional coordinates with the normalized coordinates, the formula can be described simply. Therefore, it should be noted that the two-dimensional coordinates of the registration points and feature points of the present invention are described in normalized coordinates.

[Image Feature Tracking Unit 12]
The image feature tracking unit 12 tracks a registration point (image feature) of a registered image in continuous captured images.

  FIG. 4 is an explanatory diagram showing the correspondence between the registration points of the registered image and the tracking coordinates of the captured image.

There are generally two techniques for tracking image features between images:
<Feature point tracking base>
<Local feature matching base>
The image feature tracking unit 12 uses one or both of the two at the registration point p _j (= [u _j , v _j ] ^T , j = 1 to Np) of the registered image (normal vector n _t ). Acquire tracking coordinates m _ij (= [u _ij , v _ij ] ^T , i = 1 to Nc) of the corresponding captured image.

<Feature point tracking base>
According to the feature point tracking-based technique, tracking coordinates in a captured image corresponding to a registration point of a registered image are detected by a Harris corner detector or a FAST corner detector. In general, a KLT (Kanade-Lucas-Tomasi) algorithm or a template matching technique for patches obtained by cutting out local regions around feature points is used. As a template matching similarity calculation method, NCC (Normalized Cross Correlation) or SSD (Sum of Squared Difference) can be used.

<Local feature matching base>
According to local feature matching-based technology, matching between features such as SIFT (Scale-Invariant Feature Transform) and SURF (Speeded Up Robust Features) is robust to changes in position, rotation, and distortion. Get point correspondence. SIFT extracts a set of 128-dimensional feature vectors from one image, analyzes a characteristic local region using a scale space, and describes a feature vector that is invariant to scale change and rotation. SURF can perform higher-speed processing than SIFT by using an integral image, and extracts a set of 64-dimensional feature vectors from one image. Further, in the case of FAST (Features from Accelerated Segment Test) and FRAK (Fast Retina Keypoint) which are binary feature vector extraction algorithms, feature vectors that are faster and more compact than SIFT or SURF can be extracted.

  Here, there is a case where a background image is captured between the registration point and the tracking coordinates, or an erroneous tracking (outlier) is caused due to a change in illumination, a pattern pattern, or the like. It is preferable that a corresponding point of the outlier is not input to the registration point and tracking coordinates input to the normal vector calculation unit 13 at the subsequent stage. Here, the image feature tracking unit 12 excludes mistracked tracking coordinates using a homography matrix between the registration points of the registered image and the tracking coordinates of the captured image. That is, the tracking coordinates of the outlier are excluded, and only the tracking coordinates of the inlier are output to the normal vector calculation unit 13.

  FIG. 5 is an image diagram showing correspondence between registered points and tracking coordinates.

According to FIG. 5, assuming that the target object is a plane, mistracking can be excluded by geometric verification. The change in the position of the registration point on the planar object can be expressed by homography conversion. Therefore, the image feature tracking unit 12, by estimating the homography matrix H _RC between the captured image and the registered image, excluding the tracking coordinates do not correspond to the H _RC, to improve the robustness of the tracking Can do.

Specifically, for each registered point p _j , the distance d _ij between the conversion position using _HRC and the actual tracking coordinate m _ij is calculated by the following equation.
d _ij = | m _ij −p ′ _ij |
P ′ _ij ￣ = H _RC p _ij ￣
P _ij '￣: homogeneous representation of p _ij
d _ij: By deviance d _ij from the homography matrix H _RC tracking coordinates m _ij is excluded tracking coordinates m _ij above a certain threshold, it is possible to improve the robustness of the tracking.

In general, the estimated value of the homography matrix is obtained by solving the minimization problem of the projection error function of the registration points. Here, further, by combining the robust estimation such as RANSAC (RANdom SAmple Consensus), the inliers and outliers tracking coordinates corresponding to H _RC, it can be robustly separated.

  When a planar object appears on the entire registered image, the homography matrix between the images can be estimated with high accuracy. On the other hand, when a planar object appears only in a part of the registered image (when the background appears), or when the target object partially includes a three-dimensional structure, the homography matrix should be uniquely estimated from the correspondence of the image features I can't. At this time, even if the tracking coordinates are correct, outliers may occur due to homography estimation. Since these points do not match the planar structure, if they are included in the tracking result, the normal vector estimation accuracy deteriorates. Therefore, it is preferable to be excluded by robust estimation.

  If there are few such points, they can be correctly excluded by robust estimation. It should be noted that if the area of the planar object is relatively small with respect to the background, outlier exclusion fails and an incorrect homography matrix is calculated.

<Estimating the image range of a registered image using a homography matrix>
The background region can be excluded to some extent from the registered image in which the planar object is reflected according to the user's instruction. However, it is difficult for the user to strictly specify a planar object for the entire registered image, and the registration image specification includes an error. Note that, from the viewpoint of convenience, it is originally preferable that the user does not need to designate the target area.

  Here, the image feature tracking unit 12 estimates the target region in the registered image and restricts the image features used for calculating the homography matrix, thereby improving the robustness of the homography estimation. Also, robustness of image feature tracking is improved by verifying whether image features outside the target region match the homography matrix. Eventually, the target area can be updated using only the registration points that have become inliers.

<Estimation of the position of a planar object in the registered image>
It is generally expected that the planar object appears in the vicinity of the center of the registered image, and it is desirable to designate an area near the center of the registered image as the initial value of the target area. After obtaining the inlier that matches the homography matrix, the region including the inlier is updated as the target region.

<Interruption of processing for camera work that is too large>
If the number of tracking coordinates (image features) that have been successfully tracked is significantly small, there are multiple causes. For example, there may be a case where the camera work is too large, an image whose tracking of image features is extremely difficult due to a light source change or a focus change is input, or noise is added to the registered image. In these cases, the normal vector cannot be calculated accurately. Therefore, the image feature tracking unit 12 interrupts the image processing and starts again from the image registration when the number of image features that have been successfully tracked is equal to or less than the first predetermined threshold value τ _np .

<Interruption of processing for too few tracking coordinates>
If a sufficient number of tracking coordinates (image features) can be tracked but the number of tracking coordinates that match the homography matrix (becomes inliers) is extremely small, the target area may not be a plane. . In this case, the image feature tracking unit 12 interrupts the image processing when the number of image features that have been successfully tracked is equal to or less than a second predetermined threshold (<first predetermined threshold), and “the target object is a plane “None” is clearly indicated to the user.

<Repetitive processing for polyhedral target object>
When the target object is composed of a plurality of planes, the first H _RC is used as the second target area by using robust estimation such as RANSAC again from only the tracking coordinates that are outliers for the first target area. It is also preferable to calculate a second homography matrix H _RC different from, and extract the tracking coordinates that become the inlier. This can be repeated for three or more target regions.

  The normal vectors after the second target area are processed by the normal vector calculation unit 13 after the normal vector of the first target area is estimated, or the same process is repeated, or each target area However, the processing of the normal vector calculation unit 13 can be executed in parallel. Executing in parallel can estimate the normal vector at a higher speed, but the processing load becomes higher. It is preferable to select a processing procedure according to the processing resource.

[Normal vector calculation unit 13]
The normal vector calculation unit 13 receives a group of continuous registration points p corresponding to each captured image from the image feature tracking unit 12. The normal vector calculation unit 13 uses the following elements in order to calculate the normal vector of a planar object reflected in the captured image.
i: Number of Nc frames of consecutive captured images X _j (= [x _j , y _j , z _j ] ^T ): Np registration points p _j (= [u _j , v _j ] ^T , j = 1 to Np) three-dimensional coordinates m _ij (= [u _ij , v _ij ] ^T , i = 1 to Nc): Np registration points p _j reflected in each frame Then, the normal vector calculation unit 13 minimizes the reprojection error function (bundle adjustment), and camera orientation parameters R _i (= [r _ix , r _iy , r _iz ] ^T ) and t _i ( = [T _ix , t _iy , t _iz ] ^T ) and normal vector n _t (= [x _n , y _n , z _n ] ^T , x _n ² + y _n ² + z _n ² = 1) calculate.
At this time, the vector generator 13 of the present invention, the three-dimensional coordinates X _j of registration points p _j of the reference image, a registration point p _j, the reference point X ₀ through the object plane ₍₌ [x _0, y ₀ , z ₀ ] ^T ) and the normal vector n _t, and is calculated by back projection of the registration point p _j onto the object plane.
Thus, two parameters of the normal vector n _t can be used as unknown parameters instead of using X _j (= [x _j , y _j , z _j ] ^T ) as unknown parameters.

Further, the normal vector calculation unit 13 uses the reference point X ₀ as a back projection (X ₀ = 1 / w ₀ [u ₀ , v] of the center of gravity p ₀ (= [u ₀ , v ₀ ] ^T of the registration point p _{j. 0} , 1] ^T ).
The reference point X ₀ (= [x ₀ , y ₀ , z ₀ ] ^T ) through which the object plane passes can be set near the center of the object plane from the viewpoint of the stability of the reprojection error function minimization calculation. preferable. The camera pose is estimated for scale indefinite depth of X ₀ may be set to any value. For _{example, X 0 = [0,0, f} ] T (f is the focal length _f x, close to _{f y)} may be set to.
However, since the object does not always appear in the center of the image, the center of gravity p ₀ (= [u ₀ , v ₀ ] ^T ) of the registration point p _j is calculated, and the back projection point X ₀ (= 1) of p ₀ / W ₀ [u ₀ , v ₀ , 1] ^T ) may be calculated. This can improve the stability of normal vector calculation when the object is reflected toward the edge of the screen. In this case the well, the depth 1 / w ₀ may be set to any value (for example, the focal length f _x, close to f _y). In addition, when the distance to the object plane is roughly known in advance, it is also preferable to set it within the known range.

Regarding the normal vector calculation unit 13, the reprojection error function (bundle adjustment) is expressed by the following equation.
_{R i ', t i',} n t '= arg min Ri, ti, nt Σ i = 1 Nc Σ j = 1 Np (m ij -proj (R i, t i, X j)) 2
X _j = (n _t · X ₀ / n _t · p _j ') p _j '
= (X _n x ₀ + y _n y ₀ + z _n z ₀ ) / (x _n u _j + y _n v _j + z _n ) [u _j , v _j , 1] ^T
x _n ² + y _n ² + z _n ² = 1
z _n = √ (1−x _n ² −y _n ² )
p _j ′ (= [u _j , v _j , 1] ^T ): homogeneous coordinate expression of the registration point p _j
i: Number of Nc frames of the captured image m _ij : Tracking coordinates for each of Np registration points p _j reflected in frame i R _i and t _i : Camera posture parameters of frame i n _t : Normal of plane object Vector X ₀ : Reference point through which the object plane passes
proj (R _i , t _i , X _j ): projection function [R _i | t _i ] X _j 'of the three-dimensional coordinate X _j
X _j ′: homogeneous coordinate representation of X _j R _i ′ and t _i ′: estimated values of camera posture parameters R _i and t _i of frame i n _t ′: estimated values of normal vector n _t of the planar object

  FIG. 6 is an image diagram showing feature points (tracking coordinates) expressed by the number of different unknown parameters.

In the case of bundle adjustment used in three-dimensional restoration based on the existing technology, X _j (= [x _j , y _j , z _j ] ^T ) is an unknown parameter. For this purpose, the number of unknown parameters in the reprojection error function is determined by the camera posture parameters R _i (= [r _ix , r _iy , r _iz ] ^T ) and t _i (= [t _ix , t _iy , t _iz ] ^T ) and Np three-dimensional coordinate registration points p _j , and the number of unknown parameters is 6Nc + 3Np.
_{R i ', t i',} X j '= arg min Ri, ti, xj Σ i = 1 Nc Σ j = 1 Np (m ij -proj (R i, t i, X j)) 2

On the other hand, according to the present invention, instead of X _j (= [x _j , y _j , z _j ] ^T ), two parameters of the normal vector n _t are set as unknown parameters n _t ′. For this purpose, the number of unknown parameters in the reprojection error function is determined by the camera posture parameters R _i (= [r _ix , r _iy , r _iz ] ^T ) and t _i (= [t _ix , t _iy , t _iz ] ^T ) and the normal vector n _t (= [x _n , y _n , z _n ] ^T , x _n ² + y _n ² + z _n ² = 1) for the registration point p _j ), And a total of 6Nc + 2.
_{R i ', t i',} n t '= arg min Ri, ti, nt Σ i = 1 Nc Σ j = 1 Np (m ij -proj (R i, t i, X j)) 2
Therefore, according to the present invention, the number of parameters can be greatly reduced, and the effects of reducing the processing load and improving the accuracy can be obtained.

  FIG. 7 is an explanatory diagram showing a geometric relationship between a registration point, a reference point, a normal vector, and an object plane.

  If the normal vector can be calculated, the three-dimensional structure of the planar object can be calculated. Here, it is assumed that no information regarding the appearance and size is given in advance to the planar object to be estimated for the normal vector. The normal vector of the planar object estimated by the normal vector calculation unit 13 is information on the normal vector of the planar object with respect to the image plane.

  In the normal vector calculation unit 13, the reprojection error function preferably sets the initial value of the normal vector in bundle adjustment to a direction parallel to the optical axis.

  The minimization of the reprojection error function can use a solution of a nonlinear minimization problem represented by the Gauss-Newton method.

Under the precondition that the camera work of the captured image is very small, the normal vector calculation unit 13 sets the camera posture parameter R _i as a unit matrix, t _i as a zero vector, and n _t = [ [0,0,1] ^T is calculated as a normal vector. Since the rotation matrix R _i can be decomposed by the following equation, the parameters r _x , r _y , r _z are used as parameters.
θ = || r || _L2
r = [r _x , r _y , r _z ] ^T

In particular, as the camera work (viewpoint change) is smaller, the rotation matrix Ri can be approximated by the following expression, and θ _i ^x , θ _i ^y , and θ _i ^z may be used as parameters.
Finally, the normal vector calculation unit 13 can output the estimated normal vector n _t to a predetermined application. Further, the normal vector n _t can be output to the registered image storage unit 11 to store a frontal image with respect to the registered image.

  FIG. 8 is a graph showing the geometric relationship among the optical axis center, the registration point of the registered image on the image plane, and the normal vector of the captured image.

According to FIG. 8, a 400 × 400 pixel image plane is sampled at 20 pixel intervals to obtain 441 initial tracking points p _j (Np = 441).
A normal n _{t of the} target plane is randomly generated within an angle of 45 degrees from the z-axis direction, and the registration point p _j is back-projected to obtain a three-dimensional coordinate X _j . The focal length f is set to 500 mm.
Then, Gaussian noise is added to the initial camera posture r ₀ = [0 0 0] ^T and t ₀ = [0 0 0] ^T to generate camera postures R _i and t _i (i = 1 to 100).
X _j is projected by camera postures R _i and t _i , and Gaussian noise is added to generate tracking points m _ij (j = 1 to 441).
Using m _ij as an input, the normal direction n _t ′ is estimated.
The number of frames used for estimation is gradually increased from 2 to 100, and accuracy and processing time are evaluated (Nc = 2 to 100).
100 sets of _nt are prepared, and the average of accuracy and processing time is calculated.

  FIG. 9 is a graph showing the accuracy of the normal vector and the processing time with respect to the number of frames according to bundle adjustment.

According to FIG. 9, different bundle adjustments are compared.
DE: A normal vector n _t ′ is calculated by performing principal component analysis on the three-dimensional coordinate X _j ′ by conventional bundle adjustment (number of parameters: 6Nc + Np = 453 to 1041).
OE: A normal vector is calculated by the bundle adjustment (parameter number 6Nc + 2 = 14 to 602) of the present invention.

  FIG. 9A is a graph showing the normal vector estimation accuracy according to the number of frames. Here, comparing the bundle adjustment DE of the prior art and the bundle adjustment OE of the present invention, the accuracy of the normal vector is the same. That is, the normal vector can be calculated with high accuracy (angle error of about 2 degrees) in about 20 frames.

  FIG. 9B is a graph showing the processing time of the normal vector according to the number of frames. Here, comparing the bundle adjustment DE of the prior art with the bundle adjustment OE of the present invention, the normal vector estimation processing time is greatly different. That is, it can be understood that the processing time of the bundle adjustment OE of the present invention is extremely fast (short time). Specifically, it can be confirmed that Nc = 2 is 60 times faster, Nc = 20 is about 12 times faster, and Nc = 100 is about 3 times faster.

  According to the above-described embodiment, the planar object constituted by one plane has been described. Of course, according to the present invention, even a target object composed of a plurality of main planes can be applied to each of a plurality of planes. That is, the present invention does not limit the number of main planes constituting the target object to one. By increasing the number of planes, it can be applied to a target object having an arbitrary shape.

  As described above in detail, according to the program, the apparatus, and the method of the present invention, there is a significant limitation of camera work under the precondition that the target object is a plane object (or an object that can be approximated by a plane). Without any problem, the normal vector of a planar object reflected in a continuous captured image can be calculated with a significantly small amount of calculation. Specifically, since the number of unknown parameters in bundle adjustment for calculating a normal vector is reduced, the processing load can be reduced and the estimation accuracy can be increased. Further, by using the homography matrix, it is possible to improve the robustness against the tracking failure of the feature points.

  Various changes, modifications, and omissions of the above-described various embodiments of the present invention can be easily made by those skilled in the art. The above description is merely an example, and is not intended to be restrictive. The invention is limited only as defined in the following claims and the equivalents thereto.

DESCRIPTION OF SYMBOLS 1 Image processing apparatus 10 Image acquisition part 11 Registered image memory | storage part 12 Image feature tracking part 13 Normal vector calculation part

Claims (13)

In a program characterized by operating a computer to calculate a normal vector of a planar object reflected in a captured image,
Nc frames i of continuous shot images;
Three-dimensional coordinates X _j (= [x _j , y _j , z _j ) of Np registration points p _j (= [u _j , v _j ] ^T , j = 1 to Np) detected from the registered image of the planar object. ] ^T ),
Using tracking coordinates m _ij (= [u _ij , v _ij ] ^T , i = 1 to Nc) for each of Np registration points p _j reflected in each frame,
Camera orientation parameters R _i (= [r _ix , r _iy , r _iz ] ^T ) and t _i (= [t _ix , t _iy , t _iz ] ^T ) that minimize the reprojection error function, and normal vectors normal vector calculation means for calculating n _t (= [x _n , y _n , z _n ] ^T , x _n ² + y _n ² + z _n ² = 1),
The normal vector calculating means, the three-dimensional coordinates X _j of registration points p _j of the reference image, a registration point p _j, the reference point _X 0 through the object plane _{(= [x 0, y 0} , z 0] T ) And a normal vector n _t, and a computer functioning to calculate by back projection of the registration point p _j onto the object plane. The normal vector calculation means uses the reference point X ₀ as the back projection (X ₀ = 1 / w ₀ [u ₀ , v ₀ , V ₀ , ^T centroid p ₀ (= [u ₀ , v ₀ ] ^T) of the registration point p _j . 1] The program according to claim 1, which causes the computer to function as calculated by ^T ). With respect to the normal vector calculation means, the reprojection error function is represented by the following equation: R _i ′, t _i ′, n _t ′ = arg min _{Ri, ti, nt} Σ _{i = 1} ^Nc Σ _{j = 1} ^Np (m _ij −proj (R _i , t _i , X _j )) ²
X _j = (n _t · X ₀ / n _t · p _j ') p _j '
= (X _n x ₀ + y _n y ₀ + z _n z ₀ ) / (x _n u _j + y _n v _j + z _n ) [u _j , v _j , 1] ^T
x _n ² + y _n ² + z _n ² = 1
z _n = √ (1−x _n ² −y _n ² )
p _j ′ (= [u _j , v _j , 1] ^T ): homogeneous coordinate expression of the registration point p _j
i: Number of Nc frames of the captured image m _ij : Tracking coordinates for each of Np registration points p _j reflected in frame i R _i and t _i : Camera posture parameters of frame i n _t : Normal of plane object Vector X ₀ : Reference point through which the object plane passes
_{proj (R i, t i,} X j): projection function of the three-dimensional coordinates _{X j [R i | t i} ] X j
X _j ': _Homogeneous coordinate representation of X _j R _i ' and t _i ': Estimated values of camera posture parameters R _i and t _i of frame i n _t ': Estimated value of normal vector n _t of planar object The program according to claim 1 or 2, which causes a computer to function. With respect to the normal vector calculation means, the number of unknown parameters in the reprojection error function is the camera orientation parameters R _i (= [r _ix , r _iy , r _iz ] ^T ) and t _i that occur every Nc frames _i. (= [T _ix , t _iy , t _iz ] ^T ) and normal vectors n _t (= [x _n , y _n , z _n ] ^T , x _n ² + y _n ² + z) for registered points _{n 2} ^{= 1)} the sum of two and a program according to claim 3, characterized in that causes a computer to function so that 6 nC + 2. The normal vector calculating means calculates the normal vector under the precondition that the camera work of the photographed image is minute, with Ri of the camera posture parameter as a unit matrix and t _i as a zero vector. The program according to any one of claims 1 to 4, which causes a computer to function. With respect to the normal vector calculation means, the reprojection error function uses a computer so that the initial value of the normal vector in bundle adjustment is in a direction parallel to the optical axis (n _t = [0,0,1] ^T ). 6. The program according to claim 1, wherein the program is made to function. The computer is further caused to function as an image feature tracking unit that excludes mistracked tracking coordinates using a homography matrix between a registration point of a registered image and a tracking coordinate of a captured image. The program according to any one of 6 above. 8. The computer further functions as registered image storage means for specifying a target area in which a planar object appears in accordance with a user operation from the photographed image and storing the target area as a registered image. The program according to any one of the above. The registered image storage means causes the computer to function to geometrically transform the registered image into a frontalized image using the normal vector calculated by the normal vector calculating means, and to store the frontaled image as a registered image. The program according to claim 8. The image feature tracking means, when the number of image features that have been successfully tracked is equal to or less than a first predetermined threshold, causes the computer to function so as to interrupt image processing and start again from image registration. The program according to any one of 1 to 9. The image feature tracking unit interrupts image processing when the number of image features that have been successfully tracked is equal to or less than a second predetermined threshold (<first predetermined threshold), and states that “the target object is not a plane”. The program according to claim 10, wherein the computer is caused to function so as to clearly indicate to a user. In an image processing apparatus that calculates a normal vector of a planar object reflected in a captured image,
Nc frames i of continuous shot images;
Three-dimensional coordinates X _j (= [x _j , y _j , z _j ) of Np registration points p _j (= [u _j , v _j ] ^T , j = 1 to Np) detected from the registered image of the planar object. ] ^T ),
Using tracking coordinates m _ij (= [u _ij , v _ij ] ^T , i = 1 to Nc) for each of Np registration points p _j reflected in each frame,
Camera orientation parameters R _i (= [r _ix , r _iy , r _iz ] ^T ) and t _i (= [t _ix , t _iy , t _iz ] ^T ) that minimize the reprojection error function, and normal vectors normal vector calculation means for calculating n _t (= [x _n , y _n , z _n ] ^T , x _n ² + y _n ² + z _n ² = 1),
The normal vector calculating means, the three-dimensional coordinates X _j of registration points p _j of the reference image, a registration point p _j, the reference point _X 0 through the object plane _{(= [x 0, y 0} , z 0] T ) And a normal vector n _t and calculated by back projection of the registration point p _j onto the object plane. In the normal vector calculation method of the apparatus for calculating the normal vector of a planar object reflected in a captured image,
Nc frames i of continuous shot images;
Three-dimensional coordinates X _j (= [x _j , y _j , z _j ) of Np registration points p _j (= [u _j , v _j ] ^T , j = 1 to Np) detected from the registered image of the planar object. ] ^T ),
Using tracking coordinates m _ij (= [u _ij , v _ij ] ^T , i = 1 to Nc) for each of Np registration points p _j reflected in each frame,
Camera orientation parameters R _i (= [r _ix , r _iy , r _iz ] ^T ) and t _i (= [t _ix , t _iy , t _iz ] ^T ) that minimize the reprojection error function, and normal vectors n _t (= [x _n , y _n , z _n ] ^T , x _n ² + y _n ² + z _n ² = 1)
In the above step, the three-dimensional coordinates X _j of the registration point p _j of the registration image, the registration point p _j , the reference point X ₀ (= [x ₀ , y ₀ , z ₀ ] ^T ) passing through the object plane, and the modulus A normal vector calculation method for an apparatus, which is expressed by a line vector n _t and is calculated by back projection of a registration point p _j onto an object plane.