JP2011008687A - Image processor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processor which superposes a virtual object over an image in an actual space.SOLUTION: The image processor includes a camera for obtaining moving image data. The image processor detects feature points for long-term tracking at intervals of the prescribed number of frames of moving image data, detects feature points for short-term tracking with respect to all frames of the moving image data, performs inter-frame feature point tracking on the basis of respective feature quantities of feature points for long-term tracking detected from two different frames, performs inter-frame feature point tracking by block matching based on neighboring brightness values of feature points for long-term tracking detected from two different frames or feature points for short-term tracking, estimates the position and posture of the camera in a three-dimensional space on the basis of tracked feature points, estimates three-dimensional positions of feature points on the basis of the tracked feature points and the estimated position and posture of the camera in the three-dimensional space, and superimposes virtual object data input from the outside on the moving image data to output the data on the basis of estimated camera position information and posture information in the three-dimensional space and three-dimensional position information of feature points.

Description

  The present invention relates to an image processing apparatus that synthesizes computer graphics data with photographed image data.

  Conventionally, HMD (Head Mount Display) or the like, which is a small display device that is mounted on the head, is captured as if there is a virtual object generated by CG or the like by capturing a real space as moving image data with a camera. A virtual reality system that realizes Augmented Reality by displaying with (for example, see Non-Patent Document 1). In this system, a marker for specifying the display position (three-dimensional position) of a virtual object is arranged in the real space, and the display position of the virtual object to be superimposed is determined by detecting and tracking the marker from the moving image data. Display. However, in a system using a marker, it is necessary to install a marker in the real space for each position where a virtual object is displayed, so that there is a problem that the usage environment and usage of the system are limited.

  In order to solve such a problem, a technique for detecting and tracking a feature point having a specific image feature from an image taken instead of a marker and specifying a display position of a virtual object has been proposed (for example, Non-Patent Document 2). In the technology for detecting and tracking feature points described in Non-Patent Document 2, a corner feature in each image is detected by using a technology called FAST feature detector (see Non-Patent Document 3), and peripheral pixels of the detected corner feature are detected. Can be used as a template, corner features can be tracked by template matching between images, and the camera position and orientation and the three-dimensional positions of feature points can be estimated from the results to determine the display position of the virtual object. In the FAST feature detector, the difference between the luminance value of a pixel located at the center of a circle and a pixel having a predetermined radius R with respect to each pixel existing on the circumference of an arbitrary pixel in the image is a predetermined threshold. When the number of pixels having a value TH1 or more continues for a predetermined number N or more, or when the difference from the luminance value of the pixel located at the center of the circle is −N1 or less continues for N or more, the center of the circle In this method, the detection position is the corner feature of the luminance value of each pixel on the center and circumference of the circle.

Hirokazu Kato, MarkBillinghurst, Koichi Asano, Keihachiro Tachibana: Augmented reality system based on marker tracking and its calibration, Transactions of the Virtual Reality Society of Japan, Vlo. 4, no. 4, pp. 607-616 (1999) Parallel Tracking and Mapping for Small AR Workspaces In Proc.International Symposium on Mixed and Augmented Reality (ISMAR'07, Nara) E. Rosten and T. Drummond. Machine learning for high-speed corner detection. In Proc. 9th European Conference on Computer Vision (ECCV'06), Graz, May 2006.

  However, the method using image features is more susceptible to changes in the situation during shooting than the method using markers, and the output camera position and orientation and the estimation result of the three-dimensional position of feature points are likely to be unstable. There is a problem. For example, according to the FAST feature detector method described in Non-Patent Document 3, when the contrast of a captured image is low and the influence of noise at the time of shooting is large, or when illumination conditions change during long-time shooting, Regardless of whether the camera itself moves, it is difficult to stably detect the same position as a feature point in a plurality of images. Even when the camera moves during shooting and the shape of the subject being photographed changes significantly, it is difficult to continue to detect the same position on the subject as a feature point. This is an element in which the estimation result of the three-dimensional position becomes unstable. Since there is a high probability that these situation changes during shooting will occur as the shooting time increases, the stability of the estimation result during long-time shooting becomes a problem.

  As a means for solving this problem, a method using image features having robust performance in image rotation, enlargement / reduction, affine deformation, illumination change, and noise addition as compared with the FAST feature detector method can be considered. Such image features include, for example, SIFT (Scale-invariant feature transform) features. In order to detect a feature point, the SIFT feature first performs DoG (Difference-of-Gaussian) processing on the target image, and sets a point that is an extreme value in the generated DoG image as a candidate for the feature point. Next, feature points are detected by removing from the candidates points whose main curvature, which is easily affected by noise, is greater than or equal to a predetermined threshold value, and points whose contrast is less than or equal to a predetermined threshold value. In addition, using the brightness gradient direction of the detected feature point and the brightness gradient histogram in the neighboring region as the feature amount of the feature point, the feature amount of the detected feature point is compared between images, and a combination with high similarity is obtained. Thus, the feature in the image can be tracked. Further, as other image features, there are GLOH (Gradient Location and Orientation Histogram) features and the like.

  However, the image feature such as SIFT feature and GLOH feature has a very large amount of calculation required for feature detection compared to the case where the FAST feature detector method is used, and real-time processing is difficult. For this reason, Augmented Reality is obtained by capturing real space as moving image data with a camera and displaying in real time while superimposing a virtual object generated by computer graphics or the like on the moving image data. There is a problem that it is difficult to realize.

  The present invention has been made in view of such circumstances, and is stable against changes in shooting conditions when realizing augmented reality in which a virtual object generated by computer graphics or the like is superimposed on real space. An object of the present invention is to provide an image processing apparatus capable of realizing feature point tracking.

  The present invention is an image processing apparatus that superimposes and outputs image data of a disguise object on captured moving image data, the camera that captures the surrounding environment to obtain moving image data, and the moving image data A first feature detection unit for detecting feature points for long-term tracking based on a first image feature at every predetermined frame interval, and features for short-term tracking based on second image features for all frames of the moving image data A second feature detection unit for detecting points; a first feature tracking unit for tracking feature points between frames based on the feature quantities of the long-term tracking feature points detected from two different frames; and A second feature tracking unit for tracking feature points between frames by performing block matching based on long-term tracking feature points detected from two different frames or neighborhood luminance values of short-term tracking feature points; Special A camera position / orientation estimation unit that estimates the position and orientation of the camera in a three-dimensional space based on the feature points tracked by the tracking unit, the feature points tracked by the first feature tracking unit, and the estimated A feature point 3D position estimator for estimating the 3D position of the feature point based on the position and orientation of the camera in the 3D space; camera position information and orientation information in the estimated 3D space; An image composition unit that superimposes and outputs image data of a virtual object input from the outside on the moving image data captured by the camera based on the three-dimensional position information of the feature points. And

  In the present invention, the camera position / orientation estimation unit estimates camera position information and attitude information in a three-dimensional space with respect to all frames taken, and the feature point three-dimensional position estimation unit It is characterized in that the three-dimensional position information of feature points is estimated for each frame interval.

  In the present invention, the camera position and orientation estimation unit performs initial estimation of the camera position and orientation of the frame using the tracked short-term tracking feature points, and the tracked short-term tracking feature points and long-term tracking feature points. The camera position and orientation of the frame are determined using the feature points.

  The present invention is characterized in that the feature point three-dimensional position estimation unit estimates a three-dimensional position with respect to the tracked long-term tracking feature point.

  According to the present invention, image feature detection processing with a large amount of computation for feature point detection is performed at predetermined frame intervals to reduce the computation amount, and image feature detection processing with a large computation amount is performed. For short-time camera position and orientation changes in frames that do not perform image processing, feature point detection using image feature detection processing with a small amount of computation and block matching can be used to achieve high-precision smooth camera position and orientation. The effect that it becomes possible to estimate the change is obtained. For this reason, stable camera position and orientation estimation and feature point 3D position estimation by tracking feature points in an arbitrary image can be realized in real time, and a virtual object generated by computer graphics or the like is superimposed on real space Processing that realizes augmented reality can be executed stably.

1 is a block diagram illustrating a configuration of an image processing apparatus according to an embodiment of the present invention. It is explanatory drawing of the coordinate system handled with the image processing apparatus of this invention. It is an operation | movement flow of the camera position and orientation estimation process in the image processing apparatus of this invention. It is explanatory drawing for demonstrating the existence range of the feature point in an adjacent frame. It is an operation | movement flow of the three-dimensional position estimation process of the feature point in the image processing apparatus of this invention.

  Hereinafter, an image processing apparatus according to an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing the configuration of the embodiment. In this figure, reference numeral 1 denotes a camera that captures the surrounding environment and outputs moving image data. Reference numeral 2 denotes a first feature detection unit that detects a feature point based on a predetermined first image feature as a long-term tracking feature point from a predetermined frame in the moving image data output from the camera 1. Reference numeral 3 denotes a second feature detection unit that detects a feature point based on a predetermined second image special report as a feature point for short-term tracking from each frame in the moving image data output from the camera 1. Reference numeral 4 denotes a first feature tracking unit that performs feature point tracking between frames based on the feature amounts of long-term tracking feature points detected from two different frames. Reference numeral 5 denotes a second feature tracking unit that performs block matching on the basis of luminance values in the vicinity of long-term tracking feature points or short-term tracking feature points detected from two different frames and tracks feature points between frames.

  Reference numeral 6 denotes a camera position and orientation estimation unit that estimates the position and orientation of the camera 1 based on the short-term tracking feature points tracked by the second feature tracking unit 5. Reference numeral 7 denotes a feature point three-dimensional position estimation unit that estimates a three-dimensional position of a feature point based on long-term tracking feature points tracked by the first feature tracking unit 4 and estimated camera position and orientation information. is there. Reference numeral 8 denotes a storage unit that stores feature point map information including an observation position of a long-term tracking feature point on a frame, an estimated three-dimensional position, and a camera position and orientation of each frame. Reference numeral 9 denotes a CG data storage unit in which computer graphics (hereinafter referred to as CG) data of a virtual object to be superimposed is stored in advance. Reference numeral 10 denotes a moving image that outputs the CG data stored in the CG data storage unit 9 from the camera 1 based on the estimated camera position and orientation information stored in the storage unit 8 and the three-dimensional position information of the feature points. An image composition unit that outputs a composite image superimposed on data.

  In the image processing apparatus shown in FIG. 1, each of two different image features is used as an image feature for long-term tracking and an image feature for short-term tracking. In the following description, the first image feature detected by the first feature detection unit 2 is the aforementioned SIFT feature, and the second image feature detected by the second feature detection unit 3 is the aforementioned FAST feature detector method. The first image feature is an image feature that has robust performance in image rotation, enlargement / reduction, affine deformation, illumination change, noise addition, and the like compared to the second image feature. Any image feature may be used, and instead of the SIFT feature, for example, the above-described GLOH feature, SURF (Speeded Up Robust Features) feature, LESH (Local Energy based Shape Histogram) feature, or the like may be used. The other second image feature may be any image feature as long as the image feature has a smaller amount of calculation than the first image feature. For example, instead of the corner feature detected by the FAST feature detector method, for example, A configuration using corner features detected by a SUSAN corner detector or a Harris operator may also be used.

  Next, in order to explain the operation of the image processing apparatus shown in FIG. 1, the coordinate system, symbols, and mathematical expressions related to the three-dimensional space and the two-dimensional space handled by the image processing apparatus shown in FIG. 1 will be described with reference to FIG. . In FIG. 2, a world coordinate system (Xw, Yw, Zw) is a coordinate system representing a three-dimensional space in which a subject photographed by the camera 1 exists, and is a coordinate system defined by using the XwYwZw coordinate axis in the figure. is there. The camera coordinate system (Xc, Yc, Zc) is a local coordinate system that represents a three-dimensional space with the viewpoint position of the camera 1 as the origin, and is a coordinate system that is defined using the XcYcZc coordinate axis in the figure. The Zc axis in the system represents the direction of the optical axis (line of sight) of the camera.

  The projected coordinate system (u, v) is a coordinate system in an image plane (frame) onto which a three-dimensional space photographed by the camera 1 is projected, and is a coordinate system defined by using UV coordinates in the drawing. . The relationship between these three coordinate systems is that the transformation matrix Ecw from the world coordinate system (Xw, Yw, Zw) to the camera coordinate system (Xc, Yc, Zc), and the camera coordinate system (Xc, Yc, Zc) to the image plane. (1) and (2) can be expressed using the projection model CamProj (P).

Pc = EcwPw (1)
Pv = CamProj (Pc) = CamProj (EcwPw) (2)

Here, Pw, Pc, and Pv are the position Pw = (Xp, Yp, Zp) ^{t of} the feature point P, the position Pc = (Xcp, Ycp, Zcp) ^t of the feature point P, and the feature point P, respectively. Represents a position Pv = (Up, Vp) ^t in the projected coordinate system on the image plane (frame) on which is projected. The subscript t indicates a transposed matrix. The transformation matrix Ecw represents the position and orientation of the camera 1 in the world coordinate system, and the transformation matrix Ecw changes with the movement of the camera 1. In the following description, a transformation matrix Ec _n that indicates the position and orientation of the camera 1 in the nth frame (hereinafter referred to as frame n) taken by the camera 1 is used as appropriate.

  Next, the operation of the image processing apparatus shown in FIG. 1 will be described. The operation of the image processing apparatus shown in FIG. 1 is roughly divided into a camera position / posture estimation process and a feature point three-dimensional position estimation process, and two processing operations are executed in parallel. First, the camera position / posture estimation processing operation in the image processing apparatus shown in FIG. 1 will be described with reference to FIG. FIG. 3 is a flowchart showing the camera position / orientation estimation processing operation in the image processing apparatus shown in FIG.

  First, the second feature detection unit 3 receives one frame of image data output from the camera 1, and detects feature points for short-term tracking in this frame using processing of the FAST feature detector method (step S1). ). The second feature detection unit 3 outputs the projected coordinates of the detected short-term tracking feature points on the image plane to the second feature tracking unit 5. Note that an image pyramid (consisting of a set of images represented by multi-resolution images) for the input image data may be generated, and feature points may be detected from low-resolution image data.

  Next, the second feature tracking unit 5 sets a region of N × N (N is a natural number) pixels in the vicinity of the feature point for the short-term tracking feature point in two consecutive frames (frame n−1, frame n). Feature points for short-term tracking are tracked by performing block matching as a target (step S2). However, when the first frame (frame 0) is input, this step is omitted because there is no tracking target frame, and the subsequent steps S3 to S6 are similarly omitted. As an example of the block matching evaluation criterion for the N × N neighboring pixel region of the feature point, SSD (Sum of Squared Difference) shown in Equation (3) is used.

In the equation (3), F _n (u, v) represents the luminance value at the coordinates (u, v) of the frame n, and the coordinates (u _j , v _j ) and (u _i , v _i ) are the frame n−. The coordinates of the feature point j in 1 and the feature point i in the frame n are shown. For the feature point j in the frame n−1, the second feature tracking unit 5 sets the feature point i = that minimizes the SSD value obtained by the expression (3) among all the feature points detected in the frame n. When k, a set of feature point j and feature point i = k is output as a tracking result. However, if TH <SSD using a predetermined threshold value TH, the tracking of the feature point for the feature point j fails and no output is performed.

  Note that here, the matching candidates for the feature point j in the frame n−1 are all the feature points detected in the frame n. However, in normal shooting, the feature points between two consecutive frames are displayed. Since the movement is likely to be limited to a narrow range, only the feature point satisfying the equation (4) is matched with the feature point j using the threshold value d0 regarding the upper limit of the moving distance of the feature point. Also good.

  By adopting such a configuration, it is possible to prevent a wrong set of feature points from being output as a tracking result due to erroneous matching when there are a wide range of similar textures, and It is possible to reduce the amount of calculation required for tracking.

  As a block matching evaluation method, SAD (Sum of Absolute Difference) or ZNCC (Zero-mean Normalized Cross-Correlation) may be used instead of SSD. However, the configuration using ZNCC is different in that the feature point having the maximum evaluation value is selected as the tracking result.

Next, the camera position / orientation estimation unit 6 performs initial estimation of the camera position / orientation (conversion matrix Ecw _n ) using information on the feature point tracking result output from the second feature tracking unit 5 (step S3). For the estimation of the transformation matrix Ecw _n , a predetermined number N1 of feature point pairs are randomly selected and estimated. Here, estimation is performed by minimizing M ₁ in equation (5) using M-estimator, which is one of robust estimation methods, as in equation (5).

In equation (5), e _i is a measurement error of the feature point i (0 ≦ i <N1), and f ₁ (e) is an evaluation function of the measurement error used for suppressing the influence of the exceptional value. When a common feature point is observed in consecutive frames n-1 and n, the feature point observed in frame n as shown in FIG. 4 is ideally observed on the epipole line shown in FIG. Therefore, the measurement error ei of the feature point is a distance between the feature point i = (u _i , v _i ) observed in the frame n and the epipole line a _i U + b _i V + c _i = 0 on the frame n ( 6) It is defined according to the formula.

The epipole line a _i U + b _i V + c _i = 0 is a feature point j = (u _j , obtained as an output of the second feature point tracking unit 5 corresponding to the feature point i = (u _i , v _i ). v _j ) and the camera position and orientation (conversion matrix Ecw _n−1 , conversion matrix Ecw _n ). Therefore, the initial estimation of the camera position and orientation is completed by obtaining the transformation matrix Ecw _n that minimizes M ₁ in Equation (5). The conversion matrix Ecw _n indicating the estimated camera pose is temporarily stored in the storage unit 8 as the camera position and orientation estimation result of the frame.

In the above description has described how to estimate a transformation matrix Ecw _n indicating the camera position and orientation using the M-estimator as an example of the estimation method, RANSAC (RANdom SAmple Consensus) and LMedS (Least Median of Squares) A known robust estimation method such as estimation can be used. Further, when M _{1 in the} expression (5) is equal to or less than the predetermined threshold value TH _M , the processing in steps S4 and S5 described later is omitted, and the estimation result obtained in step S3 is used as the camera position / posture of frame n. The conversion matrix Ecw _n may be used.

Next, the first feature tracking unit 4 performs the same processing as in the case of the short-term tracking feature points in step S2 on the long-term tracking feature points in the two consecutive frames (frame n-1, frame n) (3) Long-term tracking feature points are tracked by performing block matching using the formula as an evaluation criterion (step S4). However, the feature point to be tracked here is a point that is targeted for a three-dimensional position estimated feature point Pw _i (0 ≦ i <number of stored feature points) stored in the storage unit 8 by a processing operation to be described later. And different. Using the formula (2), the projection positions Pv _{n−1, i} and Pv _{n, i} on the frame n−1 and the frame n of the feature point Pw _i are obtained by the formulas (7) and (8), respectively.

Pv _{n−1, i} = CamProj (Ecw _n−1 Pw _i ) (7)
Pv _{n, i} = CamProj (Ecw _n Pw _i ) (8)

Unlike the case of the short-term tracking feature point, the matching candidate for the long-term tracking feature point Pv _{n−1, i} in the frame n−1 exists within the distance d1 with the long-term tracking feature point Pv _{n, i} as the center. Block matching is performed using all points as candidate points, and the rest is the same as the processing operation in step S2.

Next, the camera position / orientation estimation unit 6 estimates a transformation matrix Ecw _n indicating the camera position / orientation using all the feature point tracking results output from the second feature tracking unit 5 in steps S2 and S4 ( Step S5). For estimation of the transformation matrix Ecw _n , a set of feature points of predetermined N2 (N2 <N1) is selected at random. Since the estimation processing operation of the transformation matrix Ecw _n are the same as step S3 a detailed description thereof is omitted here. Then, the camera position and orientation estimation unit 6 stores a camera position and orientation estimation result of the finally obtained frame n transformation matrix Ecw _n to the feature point map in the storage unit 8.

Next, the image composition unit 10 uses the three-dimensional position information of the feature point obtained by the processing operations to be described later transformation matrix Ecw _n is a camera position and orientation estimation result of the frame n stored in the storage unit 8, a predetermined In accordance with the above condition, the CG data representing the virtual object stored in the CG data storage unit 9 is superimposed on the image data of the captured frame n and output (step S6). The predetermined condition here may be, for example, a configuration in which a region in which many feature points are distributed is regarded as a marker region generated by an arbitrary image, and CG data representing a virtual object is displayed on the marker region. A configuration in which a region where feature points are distributed is regarded as an obstacle and CG data representing a virtual object is displayed in a space where no feature points exist may be considered according to usage.

  By repeatedly performing the processing operations of steps S1 to S6 shown in FIG. 3 described above for each frame input from the camera 1, image data obtained by synthesizing CG data with real image data is generated and output in real time. It becomes possible.

Next, the feature point three-dimensional position estimation processing operation in the image processing apparatus shown in FIG. 1 will be described with reference to FIG. First, at the start of shooting by the camera 1, the feature point map held in the storage unit 8 is initialized to an empty state (step S11). The transformation matrix Ecw ₀ indicating the camera position and orientation of the first frame is set so that the camera coordinate system (Xc, Yc, Zc) and the world coordinate system (Xw, Yw, Zw) coincide with each other. Save as a map.

  Next, the first feature detection unit 2 inputs image data for every predetermined frame interval N out of the image data output from the camera 1, and performs long-term tracking feature point detection based on SIFT features (step S12). ). Hereinafter, a frame for each frame interval N input to the first feature detection unit 2 is referred to as a key frame. The first feature detection unit 2 outputs the detected position of the long-term tracking feature point and the SIFT feature quantity to the first feature tracking unit 4.

  Next, the first feature tracking unit 4 performs SIFT feature value matching on feature points for long-term tracking in two consecutive key frames (key frame n−1, key frame n), thereby performing long-term tracking. The feature points are tracked (step S13). However, when the first key frame (key frame 0) is input, since there is no key frame to be tracked as a pair, this processing operation is omitted, and the processing returns to step S12 without performing the subsequent processing operation. In addition, as a method for obtaining feature points to be paired, for example, an ANN (Approximate Nearest Neighbor) algorithm (S. Arya, DMMount, R. Silverman, AYWu, “An optimal algorithm for approximate nearest neighbor searching”, Journal of the ACM, Vol. 45, No. 6, pp. 891-923, 1998). The ANN algorithm is a local search method using a k-d tree structure, and is an algorithm that approximately obtains a set of feature points having the highest degree of similarity, and can obtain feature points that form a set at high speed.

Next, the feature point 3D position estimation unit 7 determines whether the feature point set obtained by the process of step S13 is based on whether or not the 3D position is already stored in the feature point map on the storage unit 8. It is determined whether or not there is a new feature point (step S14). If there is a feature point that is not stored in the feature point map on the storage unit 8 as a result of this determination, the feature point three-dimensional position estimation unit 7 determines the estimated camera position and orientation for each of the key frame n-1 and the key frame n. Transformation matrices Ecw _{(n−1) N} and Ecw _nN indicating the projected coordinates (u _i , v _i ) and (u _i , v _i ) of the feature points observed in the key frame n−1 and the key frame n, respectively. The initial value of the three-dimensional position of the feature point is estimated by triangulation, and the three-dimensional position of the feature point and the projected coordinates (u of the feature point observed in the key frame n−1 and the key frame n, respectively) _i , v _i ), (u _i , v _i ) are stored in the feature point map on the storage unit 8 (step S15).

Next, the feature point three-dimensional position estimation unit 7 updates the three-dimensional position of the feature point by minimizing M ₂ in Equation (9) using M-estimator which is one of robust estimation methods. (Step S16).

In the equation (9), e _{i, j} indicates an observation error of the feature point j in the key frame i, and f ₂ (e) is an evaluation function of a measurement error used for suppressing the influence of the exceptional value. The camera position and orientation from key frame 0 to key frame n have been estimated, and the feature point map on the storage unit 8 is updated with the three-dimensional position of each feature point obtained by minimizing M ₂ (step S17). Return to step S12.

  As described above, image feature detection processing with a large amount of computation for feature point detection is performed at predetermined frame intervals to reduce the computation amount, and image feature detection processing with a large computation amount For short-time camera position and orientation changes in frames that do not perform image processing, feature point detection using image feature detection processing with a small amount of computation and block matching can be used to achieve high-precision smooth camera position and orientation. The effect that it becomes possible to estimate the change is obtained. For this reason, stable camera position and orientation estimation by tracking feature points in an arbitrary image and three-dimensional position estimation of feature points can be realized in real time, and CG or the like can be performed on moving image data obtained by photographing a real space with a camera. It is possible to stably execute the processing of the virtual reality system that realizes augmented reality by superimposing the virtual object generated in (1) and displaying it on an HMD (Head Mount Display) or the like.

  1, the first feature detection unit 2, the second feature detection unit 3, the first feature tracking unit 4, the second feature tracking unit 5, the camera position and orientation estimation unit 6, the feature point three-dimensional position estimation unit 7 and A program for realizing the functions of the image composition unit 10 is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read into a computer system and executed, whereby real image data is processed. You may perform the process which produces | generates and outputs the image data which synthesize | combined CG data. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer system” includes a WWW system having a homepage providing environment (or display environment). The “computer-readable recording medium” refers to a portable medium such as a flexible disk, a magneto-optical disk, a ROM, and a CD-ROM, and a storage device such as a hard disk built in the computer system. Further, the “computer-readable recording medium” refers to a volatile memory (RAM) in a computer system that becomes a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. In addition, those holding programs for a certain period of time are also included.

  The program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The program may be for realizing a part of the functions described above. Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, what is called a difference file (difference program) may be sufficient.

  Virtual reality system that realizes Augmented Reality by shooting a real space as moving image data with a camera and superimposing a virtual object generated by computer graphics on this moving image data. Applicable to.

DESCRIPTION OF SYMBOLS 1 ... Camera, 2 ... 1st feature detection part, 3 ... 2nd feature detection part, 4 ... 1st feature tracking part, 5 ... 2nd feature tracking part, 6 ... Camera position and orientation estimation unit, 7... Feature point three-dimensional position estimation unit, 8... Storage unit, 9... CG data storage unit, 10.

Claims (4)

An image processing apparatus that superimposes and outputs image data of a virtual object on captured moving image data,
A camera that captures moving image data by photographing the surrounding environment,
A first feature detector for detecting feature points for long-term tracking based on a first image feature at predetermined frame intervals of the moving image data;
A second feature detection unit for detecting feature points for short-term tracking based on a second image feature for all frames of the moving image data;
A first feature tracking unit for tracking feature points between frames based on each feature amount of the long-term tracking feature points detected from two different frames;
A second feature tracking unit for tracking feature points between frames by performing block matching on the basis of the long-term tracking feature points detected from the two different frames or the near luminance values of the short-term tracking feature points;
A camera position and orientation estimation unit that estimates the position and orientation of the camera in a three-dimensional space based on the feature points tracked by the second feature tracking unit;
A feature point 3D position estimation unit that estimates a 3D position of the feature point based on the feature point tracked by the first feature tracking unit and the estimated position and orientation of the camera in the estimated 3D space When,
Based on the estimated camera position information and posture information in the three-dimensional space and the three-dimensional position information of the feature point, the image of the virtual object input from the outside with respect to the moving image data captured by the camera And an image composition unit that superimposes and outputs the data. The camera position and orientation estimation unit estimates camera position information and orientation information in a three-dimensional space for all captured frames,
The image processing apparatus according to claim 1, wherein the feature point three-dimensional position estimation unit estimates three-dimensional position information of feature points at every predetermined frame interval. The camera position and orientation estimation unit
Using the tracked short-term tracking feature points, initial estimation of the camera position and orientation of the frame is performed,
The image processing apparatus according to claim 1, wherein the camera position and orientation of the frame are determined using the tracked short-term tracking feature points and long-term tracking feature points. The feature point three-dimensional position estimation unit includes:
The image processing apparatus according to claim 1, wherein a three-dimensional position is estimated with respect to the tracked long-term tracking feature point.

Images