JP2011118724A - Apparatus and program for estimating posture of camera - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To improve accuracy, robustness, and flexibility in shooting environment in estimating a posture of a camera. <P>SOLUTION: An apparatus 1 for estimating a posture of camera includes: a tracking state measuring part 40 which measures a tracking state including an error err<SB>1</SB>corresponding to edges and an error err<SB>2</SB>corresponding to feature points, on the basis of a three-dimensional model 11 of a subject, a feature point database 21, and an input photographic image; a reliability calculation part 50 which calculates reliability f of integrated tracking obtained by integrating edge-base tracking and feature point-base tracking, as an index for prorating the error err<SB>1</SB>corresponding to edges and the error err<SB>2</SB>corresponding to feature points in accordance with the tracking state; a reliability correction part 60 which generates a correction reliability η when necessary; and a camera posture estimation means 70 which varies a rate of prorating the error err<SB>1</SB>corresponding to edges and the error err<SB>2</SB>corresponding to feature points, in accordance with the correction reliability η to generate an integrated error err, and estimates a camera posture E<SP>k+1</SP>so as to minimize the integrated error. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a camera posture estimation device and a camera posture estimation program used in virtual reality, augmented reality, robot control, video composition, and the like, and more particularly to a model-based camera posture estimation device and a camera posture estimation program.

  A technique for estimating camera parameters (camera posture) is a technique required in an apparatus that expresses video synthesis, virtual reality (VR), and augmented reality (AR). In recent years, methods have been proposed to estimate the camera posture from captured images without using special sensors, but there are issues with the stability and accuracy of camera tracking and the flexibility of changing shooting environments, and improvements are desired. ing.

  Among these, for example, there are many prior arts for marker-based methods for calculating camera parameters based on marker coordinates in a captured image, and some methods have already been put into practical use. However, installing a marker in the shooting direction places many restrictions on operation. For example, the marker may be a visual obstacle, or may be difficult to install in an outdoor environment, or may require a corresponding size in consideration of installation in a distant view. For this reason, many methods that do not use markers have been proposed. Among them, several methods based on model-based methods have been proposed because of their robustness and flexibility.

  In the model-based camera posture estimation method (model-based camera posture estimation method), it is assumed that a system that realizes this method has a three-dimensional model of a shooting scene as shown in FIG. It is said. In this example, the three-dimensional model is, for example, a rectangular parallelepiped 301, a cylinder 302, a floor or a wall around them, and information on the three-dimensional coordinates thereof.

  3A is a camera image generated by projecting information indicating the characteristics of the subject in the region indicated by reference numeral 303 in FIG. 3A onto the camera coordinate space with a camera posture obtained in the past or a roughly (roughly) obtained camera posture. A projected image) is shown in FIG. Here, the projection image includes edges 304, 305, and 306. Here, it is assumed that the three-dimensional model and the rough camera posture are assumed to be known, for example, by measuring in advance.

  Then, as shown in FIG. 3C, visual cues such as feature points (for example, corners of a pattern included in the subject surface pattern) and edges are extracted from the current video (camera image). Here, the current video includes edges 314, 315, and 316. Then, as shown in FIG. 3 (d), visual cues such as feature points and edges extracted from the current video, and a camera posture roughly obtained from a three-dimensional model of a shooting scene that the system has in advance. Find the positional deviation from the projected camera image. In this example, the positional deviation between the edges 304, 305, and 306 and the edges 314, 315, and 316 is obtained by camera tracking. Then, a function for converting the structure (translational movement amount Δt and rotational movement amount ΔR) is obtained so that the positional deviation is minimized. With this series of procedures, the model-based camera posture estimation method can estimate a highly accurate camera posture.

  Conventionally, a model-based camera posture estimation method using either a feature point or an edge on a video is known. A model-based camera posture estimation method (see Non-Patent Documents 1 to 5) that uses both feature points and edges on a video is also known. The techniques described in Non-Patent Documents 1 to 4 relate to an estimation method for minimizing the sum of deviations between edges and feature points. The technique described in Non-Patent Document 5 relates to a method based on theoretical analysis of both edges and feature points and continuity of features of both edges and feature points.

Vacchetti L, Lepetit V, Fua P. Combining edge and texture information for real-time accurate 3D camera tracking.In Proc. Of IEEE and ACM Intl.Sym. On Mixed and Augmented Reality, 2004; 48-57. Dornaika F, Garcia C. Pose estimation using point and line correspondences.Real-Time Imaging 1999; 5: 215-230. Pressigout M, Marchand E. Real-time 3D model-based tracking: Combining edge and texture information.In Proc. Of Intl. Conf. On Robotics and Automation, 2006: 2726-2731. Park H, 0h J, Seo BK, Park JI. Object-adaptive tracking for AR guidance system.In Proc. Of Intl. Conf. On Virtual-Reality Continuum and Its Applications in Industry, 2008. Rosten E, Drummond T. Fusing points and lines for high performance tracking.In Proc. Of Intl. Conf on Computer Vision, 2005; 2: 1508-1511.

However, existing model-based camera pose estimation methods that use either feature points or edges on the video are restricted to a limited environment or environmental conditions in order to perform reliable estimation. It is necessary to restrain to some extent. Specifically, the following restrictions (1) to (3) are necessary.
(1) A clear pattern is included in the shooting scene.
(2) The shooting scene includes a plurality of strong edges.
(3) An environment such as illumination does not change as a photographing condition.

  Further, in the model-based camera posture estimation method using both feature points and edges on the video, the techniques described in Non-Patent Documents 1 to 4 are errors that integrate errors using feature points and errors using edges. In consideration of this, weighting is not performed on both sides. Even if it is assumed that such integration is performed, the weight is determined heuristically (not simply but simply or approximately). In other words, the techniques described in Non-Patent Documents 1 to 4 do not use the feature points and the contributions from the edges in an analytically fused manner, and must constrain the environment.

  The technique described in Non-Patent Document 5 theoretically analyzes both feature points and edges, but does not consider the dependence of both feature points and edges on environmental conditions. Need to be restrained.

  Furthermore, in a broadcast video, the shooting scene is rarely static, and the lighting conditions are not constant, so it is difficult to constrain the environment.

  The present invention has been made in view of the above problems, and it is an object of the present invention to improve the accuracy and robustness of camera tracking and the degree of freedom of the shooting environment in estimating the camera posture.

  In order to solve the above problems, a camera posture estimation apparatus according to claim 1 of the present invention is a model-based camera posture estimation apparatus that estimates a camera posture using edges and feature points in a captured image of a subject. In addition, a three-dimensional model storage unit, a feature point database storage unit, a tracking state measurement unit, a reliability calculation unit, and a camera posture estimation unit are provided.

  According to this configuration, the camera posture estimation device stores a three-dimensional model indicating information on the characteristics of the subject existing in the shooting direction of the camera in the three-dimensional model storage unit. Note that the position information of the three-dimensional model is described by, for example, three-dimensional coordinates in the world coordinate space. In addition, the camera posture estimation apparatus stores a feature point database storing feature point descriptors and feature point information including three-dimensional information in the feature point database storage unit. Note that the feature point position information in the feature point database is described by, for example, three-dimensional coordinates in the camera coordinate space.

  Then, the camera posture estimation device obtains the edge based on the three-dimensional model and the feature point database created in advance by the tracking state measuring unit and the captured image including the input subject. The tracking state including at least the first tracking error and the second tracking error determined based on the feature point is measured.

  Then, the camera posture estimation device uses edge-based tracking and feature point-based tracking as indices for dividing the first error and the second error according to the measured tracking state by the reliability calculation means. Calculate the reliability of integrated tracking that integrates. Here, a function that determines the proportion of the first error and the second error according to the shooting environment can be obtained in advance. This function can be a function that returns a value at an arbitrary position in the section [a, b] on the x-axis, for example, according to the measured tracking state. In this case, the case where only feature point base tracking is used may be represented by the position x = a, and the case where only edge base tracking is used may be represented by the position x = b. In general, edge-based tracking is more accurate than feature-point-based tracking in an organized environment where there is no environmental change. Therefore, in the simplest case, the section [a, b] can be represented by the section [0, 1]. In this case, for example, if the reliability of integrated tracking is 0.4, the contribution of the first error is larger than the second error, so that the feature point-based tracking contributes more than the edge-based tracking. become.

  Then, the camera posture estimation device generates an integrated error by changing a proportion of dividing the first error and the second error according to the reliability of the integrated tracking by the camera posture estimation unit, The current camera posture is estimated so that the integration error is minimized. For example, the following operation is performed for each frame of the input video. That is, the first error, the second error, etc. are measured as the tracking state, and an integrated error is generated using these results. Then, the error included in the objective function prepared in advance for camera tracking for estimating the camera attitude is replaced with the generated integrated error, and the value of the camera attitude when the error in the objective function is minimized is Obtained as an estimated value of the camera posture.

  The camera posture estimation apparatus according to a second aspect of the present invention is the camera posture estimation apparatus according to the first aspect, wherein the tracking state measuring means sets the first error and the second as the tracking state. An error, the number of edges, the number of feature points, a motion blur intensity indicating the number of edges blurred according to the motion of the camera, and a reliability of an initial camera posture indicating a reliability with respect to a value before estimation of the camera posture; The number of edge corresponding point candidates existing around the edge in the captured image corresponding to the model edge of the three-dimensional model is measured.

According to this horizontal composition, in the camera posture estimation device, when the reliability calculation unit has a value of the motion blur intensity measured as the tracking state larger than a predetermined first threshold, The reliability of the integrated tracking is calculated so that only the error is used.
Further, the reliability calculation means is configured such that when the value of the motion blur intensity is equal to or less than the first threshold, and the reliability value of the initial camera posture measured as the tracking state is based on a predetermined second threshold. Is larger than the motion blur intensity and is proportional to the reliability of the initial camera posture and the number of feature points, the reliability of the integrated tracking is calculated.
In addition, the reliability calculation means, when the value of the motion blur intensity is less than or equal to the first threshold and when the reliability of the initial camera posture is less than or equal to the second threshold, The integrated tracking reliability is calculated by proportionally proportional to the motion blur intensity and inversely proportional to the reliability of the initial camera posture, the number of feature points, and the number of edge corresponding point candidates.

  Here, the first threshold value of the motion blur intensity and the second threshold value of the reliability of the initial camera posture are different values depending on the shooting environment. These threshold values can be determined based on experiments performed in advance on the actual environment of the shooting scene. In addition, this experiment can analyze how the two features of the edge and the feature point on the video should be evaluated in a unified manner. Based on this analysis, different feature quantities such as edges and feature points were classified for each condition in consideration of the influence of changes in the shooting environment, and a suitable weighting derivation formula was obtained. The reliability calculation means uses these weighting derivations. Therefore, the reliability calculation means can use the edge and the feature point cooperatively. Thereby, the model-based camera posture estimation method can be made robust. In this way, by changing the weighting derivation formula according to the conditions and using it, different features such as edges and feature points can be handled and evaluated uniformly. Therefore, it is possible to reduce the influence from changes in the shooting environment and to estimate the camera posture stably and with high accuracy.

  The camera posture estimation apparatus according to claim 3 of the present invention is the camera posture estimation apparatus according to claim 2, further comprising a reliability correction unit, wherein the camera posture estimation unit is responsive to the correction reliability. Thus, the proportion of dividing the first error and the second error is changed, and the current camera posture is estimated.

  According to this horizontal composition, the camera posture estimation device generates the correction reliability in which the reliability of the integrated tracking is corrected by the reliability correction unit. Then, the reliability correction means determines the reliability of the integrated tracking when the reliability of the integrated tracking is larger than 0.5 and the sample ratio indicating the ratio of the number of edges to the number of feature points is smaller than 1. The correction reliability is calculated by a first correction formula that is proportional to the degree and inversely proportional to the sample ratio. Here, the case where the reliability of the integrated tracking is larger than 0.5 indicates a case where the photographing environment of the edge-based tracking is good. A case where the sample ratio is smaller than 1 indicates a case where the number of edges is relatively small. As described above, even if the shooting environment of edge-based tracking is good, the reliability calculation result may not be good when the number of edges used is relatively small. However, by calculating the correction reliability using the first correction formula, it is possible to prevent erroneous correspondence between the model edge and the edge detected from the captured image.

  In addition, the reliability correction unit is proportional to the reliability of the integrated tracking when the reliability of the integrated tracking is smaller than 0.5 and the sample ratio is larger than 1, and The correction reliability is calculated using a second correction formula that is inversely proportional. Here, the case where the reliability of the integrated tracking is smaller than 0.5 indicates a case where the photographing environment of the feature point base tracking is good. The case where the sample ratio is greater than 1 indicates a case where the number of feature points is relatively small. Thus, even if the shooting environment for feature point-based tracking is good, if the number of feature points used is relatively small, the reliability calculation result may not be good. However, by calculating the correction reliability using the second correction formula, it is possible to prevent erroneous correspondence between the feature points in the feature point database and the feature points detected from the captured image.

  According to a fourth aspect of the present invention, in the camera posture estimation apparatus according to the second or third aspect, the tracking state measurement unit includes an edge detection unit, an edge matching unit, and motion blur calculation. Means, feature point detection matching means, and initial camera attitude calculation means.

  According to this horizontal composition, in the camera posture estimation device, the tracking state measuring means detects the edge from the inputted photographed video by the edge detecting means. Then, the tracking state measuring unit calculates the number of edge corresponding point candidates and the number of edges by the matching process between the model edge included in the three-dimensional model and the detected edge by the edge matching unit. To do. In addition, the tracking state measuring unit calculates the motion blur intensity based on the motion blur of the captured video input by the motion blur calculation unit. Further, the tracking state measuring unit detects a feature point from the input captured image by the feature point detection matching unit, and performs a matching process between the feature point stored in the feature point database and the detected feature point To calculate the number of feature points. Then, the tracking state measuring means obtains the initial camera attitude from the result of the feature point matching process by the initial camera attitude calculating means, and calculates the reliability of the initial camera attitude with respect to the obtained initial camera attitude. . Here, the initial camera posture indicates the camera posture before estimation.

  According to a fifth aspect of the present invention, there is provided a camera posture estimation program for estimating a camera posture based on a model using an edge and a feature point in a photographed image of a subject. Three-dimensional model storage means for storing a three-dimensional model indicating a feature point; and feature point database storage means for storing a feature point database storing feature point information including feature point descriptors and three-dimensional information of the three-dimensional model. This is a program for causing a computer provided to function as tracking state measuring means, reliability calculating means, and camera posture estimating means.

  According to such a configuration, the camera posture estimation program detects the edge based on the three-dimensional model and the feature point database created in advance by the tracking state measurement unit and the captured image including the input subject. A tracking state including at least a first tracking error determined based on the base and a second tracking error determined based on the feature point is measured.

  Then, the camera posture estimation program uses edge-based tracking and feature point-based tracking as indices for dividing the first error and the second error according to the measured tracking state by the reliability calculation means. Calculate the reliability of integrated tracking that integrates.

  The camera posture estimation program generates an integrated error by changing a proportion of the first error and the second error according to the reliability of the integrated tracking by the camera posture estimation means, The current camera posture is estimated so that the integration error is minimized.

  According to the first aspect of the present invention, the camera posture estimation device is a tracking obtained based on an edge according to a tracking state measured based on a model created in advance, a feature point database, and a captured image. An integrated error is generated by varying the proportion of the first error and the second tracking error obtained based on the feature points, and the current camera posture is estimated. Therefore, the accuracy of the camera tracking, the robustness, and the degree of freedom of the photographing environment can be improved as compared with the case where either one of the errors is used or when both errors are fixedly distributed.

  According to the second aspect of the present invention, the camera posture estimation device uses, as the tracking state, the first tracking error obtained based on the edge, the second tracking error obtained based on the feature point, and motion blur. The strength, the number of edges, the number of feature points, the reliability of the initial camera posture, and the number of edge corresponding point candidates are measured, and the reliability is calculated with an optimum conditional expression according to the measurement result. Therefore, the camera posture can be flexibly applied even if the shooting environment changes.

  According to the third aspect of the invention, the camera posture estimation device corrects the reliability of the integrated tracking according to the sample ratio indicating the ratio of the number of edges to the number of feature points and the value of the reliability of the integrated tracking. Generate confidence. Accordingly, it is possible to prevent erroneous correspondence between edges or feature points.

  According to the fourth aspect of the present invention, the camera posture estimation device calculates the number of edge corresponding point candidates, the number of edges, and the number of feature points by matching processing, and uses the motion blur intensity and the initial camera from the captured video. The posture reliability can be calculated. Therefore, the camera posture can be estimated stably.

  According to the fifth aspect of the present invention, the camera posture estimation program obtains the tracking based on the edge according to the tracking state measured based on the model, the feature point database, and the photographed image created in advance. The current camera posture can be estimated by generating an integrated error by varying the proportion of the first error and the tracking second error obtained based on the feature points.

It is a block diagram which shows typically the structure of the camera attitude | position estimation apparatus which concerns on embodiment of this invention. BRIEF DESCRIPTION OF THE DRAWINGS It is explanatory drawing which shows typically the video composition system containing the camera attitude | position estimation apparatus which concerns on embodiment of this invention, Comprising: (a) is a block diagram which shows the structure of a video composition system, (b) is the state with which the person mounted | wore An example of each is shown. It is explanatory drawing which shows the outline | summary of the camera tracking by the model base of this invention, Comprising: (a) is a three-dimensional model, (b) is the projection image using the past camera attitude | position, (c) is the present camera image, ( d) shows the amount of movement, respectively. It is a graph which shows the simulation result of the camera attitude | position estimation error at the time of adding Gaussian noise to the initial value of a camera attitude | position using the conventional method, Comprising: (a) is a point-based tracking method, (b) is an edge-based tracking method. Show. It is a graph which shows the simulation result of the relationship between the amount of visual cues equivalent to the complexity of a scene and the camera posture estimation error using a conventional method, (a) is a point-based tracking method, (b) is edge-based tracking The tracking method is shown. It is a graph which shows the simulation result of the camera attitude | position estimation error regarding the edge search performance in which the complexity of a scene influences in the conventional edge-based tracking method. It is a graph which shows the simulation result of the camera attitude | position estimation error at the time of adding a motion blur to a camera image | video using the conventional method, (a) is a point-based tracking method, (b) has shown the edge-based tracking method. . It is a graph which shows the simulation result of the camera attitude | position estimation at the time of adding Gaussian noise to a camera image | video using the conventional method, Comprising: (a) has shown the point-based tracking method, (b) has shown the edge-based tracking method. It is a flowchart which shows an example of the procedure of the reliability calculation process of the camera attitude | position estimation apparatus which concerns on embodiment of this invention, and a reliability correction process. FIG. 7 is a graph showing the estimation result of the translation term t1 of the camera posture, where (a) is a conventional point-based method, (b) is a conventional edge-based method, (c) is when η is fixed, and (d) is When (eta) is changed in this invention, (e) has each shown the change of (eta). FIG. 6 is a graph showing the estimation result of the translation term t2 of the camera posture, where (a) is a conventional point-based method, (b) is a conventional edge-based method, (c) is when η is fixed, (d) is The case where η is changed in the present invention is shown. FIG. 6 is a graph showing the estimation result of the translation term t3 of the camera posture, where (a) is a conventional point-based method, (b) is a conventional edge-based method, (c) is when η is fixed, (d) is The case where η is changed in the present invention is shown. In this invention, it is a figure which shows typically the image composition result when a camera attitude | position is estimated when illumination changes with a motion of a camera, Comprising: (a) is normal illumination, (b) is illumination. When it becomes dark, (c) has shown the case where illumination becomes bright, respectively.

  Hereinafter, a mode for carrying out a camera posture estimation apparatus of the present invention (hereinafter referred to as “embodiment”) will be described in detail with reference to the drawings. In the following, 1. 1. Outline of camera posture estimation device 2. video composition system; 3. Configuration of camera posture estimation device 4. Model-based camera tracking by linear iterative method, 5. Analysis method of shooting environment The operation of the camera posture estimation device will be described in order in each chapter.

[1. Overview of camera posture estimation device]
A camera posture estimation device 1 shown in FIG. 1 is a model-based camera posture estimation device that estimates a camera posture using edges and feature points in a captured image of a subject. Here, the camera posture is a so-called external parameter of the camera. Here, for example, the camera lens imaging angle, the three-dimensional coordinates indicating the installation location and height of the camera lens, the axis movement indicating the camera posture reflecting the operations of the camera pan axis, tilt axis, zoom axis, focus axis, etc. It may include an angle, an axial movement distance, a focal length of a camera lens, a pixel pitch of an image sensor of the camera, and the like.

  In this camera posture estimation device 1, as shown in FIG. 3, the camera posture is minimized by minimizing the deviation between the projected image obtained by projecting the three-dimensional model of the subject on a virtual screen and the feature on the subject of the photographed video. Is used. In the present embodiment, as the characteristics of the subject on the video, information on both edges and feature points is weighted corresponding to the video and used. Further, as will be described later, the influence of both the edge and feature point information from the shooting environment is analyzed from the shot video in advance. The camera pose estimation device 1 uses the weighting derivation formula derived based on the analysis result to evaluate the deviation between the camera image and the projected image and reflect it in the weighting. This makes it possible to have robustness against changes in illumination conditions, generally assumed shooting conditions such as fast camera movement, and environmental changes.

  The camera posture estimation apparatus 1 shown in FIG. 1 shows only the components of the video composition system 100 shown in FIG. 2, and may include the camera 2 and the like omitted in FIG. Here, the video composition system 100 will be described.

[2. Video composition system]
A video composition system 100 shown in FIG. 2A is a system for a user to enjoy augmented reality, and shows one application example of the camera posture estimation apparatus 1 shown in FIG.
The video composition system 100 includes a camera posture estimation device 1, a camera 2, a virtual three-dimensional object model storage unit 3, a rendering device 4, and a video composition device 5. Here, the video composition system 100 is worn by the user as shown in FIG. A wearer P shown in FIG. 2B wears a camera 2 and an HMD (Head Mounted Display) 7 on his head. The wearer P wears a wearable PC (Personal Computer) 8 on the waist. The wearable PC 8 includes a camera posture estimation device 1, a virtual three-dimensional object model storage unit 3, a rendering device 4, and a video composition device 5.

In this video composition system 100, the user can enjoy augmented reality without having to wear a special sensor or the like other than the camera as the visual sensor.
The camera 2 is a small camera mounted on the user's head, and outputs the captured video to the camera posture estimation device 1 and the video composition device 5.
The camera posture estimation device 1 estimates the camera posture by analytically integrating visual cues such as edges and feature points in the video, and outputs the estimated camera posture to the rendering device 4. Details of the camera posture estimation device 1 will be described later.

The virtual three-dimensional object model storage means 3 stores CG (Computer Graphics) data 6 and is composed of a general memory or the like.
The rendering device 4 generates virtual three-dimensional spatial data based on the CG data 6, renders a CG object (CG image) and an alpha plane based on the input camera posture, and renders the rendered CG object together with the alpha plane. The image is output to the video composition device 5. The alpha plane is an image having information for distinguishing the subject area of the CG object from the non-object area.

  The video synthesizing device 5 synthesizes a video image of the CG object rendered by the rendering device 4 and the captured image output from the camera 2 using the alpha plane. The composite image (composite video) output from the video composition device 5 is displayed on, for example, the HMD 7 shown in FIG. 2B and presented to the user. The video composition device 5 may be realized by a known virtual studio CG composition device.

According to the video composition system 100, for example, the user can act while visually feeling as if a virtual object that is not actually arranged there is present. That is, the user can act while enjoying augmented reality.
In addition, the video composition system 100 does not control the environment. For example, in the environment where the wearer P acts, the environment is not controlled from the outside so as to artificially realize the following conditions (1) to (3).
(1) A clear pattern is included in the shooting scene.
(2) The shooting scene includes a plurality of strong edges.
(3) An environment such as illumination does not change as a photographing condition.
That is, in the video composition system 100, when there is no clear pattern or edge in the direction in which the wearer P points the line of sight (the shooting direction of the camera 2), in the direction in which the line of sight is detected by detecting it. It does not automatically place clear patterns and edges. Further, when the wearer P turns his / her line of sight in a dark direction, the environment is not controlled so as to detect this and brighten the illumination in the direction of the line of sight.
However, since the video composition system 100 includes the camera posture estimation device 1 of the present embodiment, it can robustly estimate the camera posture. Further, it is possible to robustly estimate the camera posture even in an environment with temporal changes.

[3. Configuration of camera posture estimation device]
The camera posture estimation device 1 shown in FIG. 1 includes, for example, a calculation device such as a CPU, a storage device (storage means) such as a memory and a hard disk, and an interface device that transmits and receives various types of information to and from the outside. It consists of a computer and a program installed on this computer.

  The camera posture estimation device 1 is realized by the hardware device and software cooperating to control the above hardware resources by a program. As shown in FIG. , Feature point database storage means 20, video input means 30, tracking state measurement section (tracking state measurement means) 40, reliability calculation section (reliability calculation means) 50, reliability correction section (reliability correction means) ) 60, camera posture estimation means 70, and output means 80. The block diagram of FIG. 1 shows a diagram that directly reflects the processing flow of the camera parameter estimation algorithm.

<Three-dimensional model storage means>
The three-dimensional model storage means 10 includes a memory device such as a RAM (Random Access Memory) and a ROM (Read Only Memory), and a storage device such as a hard disk, and stores a three-dimensional model 11 created in advance. The three-dimensional model 11 shows information on the characteristics of a subject existing in the shooting direction of the camera. The subject feature information is, for example, information indicating the position of a feature point such as a corner of a picture included in the shape or surface pattern of the subject. The position information of the three-dimensional model 11 is described by, for example, three-dimensional coordinates in the world coordinate space.

<Feature point database storage means>
The feature point database storage means 20 is composed of a storage device such as a memory or a hard disk, and stores a feature point database 21 created in advance. The feature point database 21 is a database storing feature point descriptors including feature point descriptors and three-dimensional information of the three-dimensional model 11 of the subject. Here, the feature point descriptor of the 3D model describes what kind of feature point exists in the 3D model of the subject, and a name or identifier that can identify the feature point. Indicates. The three-dimensional information indicates the x coordinate, y coordinate, and z coordinate of the feature point of the three-dimensional model. Note that the three-dimensional information of feature points in the feature point database is described by coordinates (camera coordinates) projected from the world coordinate space of the three-dimensional model onto the camera coordinate space. Further, the feature point database 21 stores a camera posture assumed based on the information of the three-dimensional model 11, a projection of the information on a virtual screen, and a camera parameter when the projection is performed. .

<Video input means>
The video input means 30 inputs a captured video including a subject, and is configured from a predetermined input interface or the like. Note that the video input unit 30 may be configured by a communication interface or the like for inputting a captured video from a communication network (not shown). The captured image input here is output to the edge detection matching unit 41, the motion blur calculation unit 42, and the feature point detection matching unit 43 of the tracking state measurement unit 40.

<Tracking state measurement unit>
The tracking state measurement unit (tracking state measurement means) 40 includes an edge correspondence error err ₁ and a feature point correspondence error err ₂ based on the three-dimensional model 11 and the feature point database 21 and the input captured image. The tracking state is measured.

Here, the edge correspondence error err ₁ is a tracking error (first error) obtained based on the edge, and indicates how many corresponding edges are present.
The feature point correspondence error err ₂ is a tracking error (second error) obtained based on the feature point, and indicates how many corresponding feature points are present.

In the present embodiment, the tracking state measurement unit 40 further includes the number of edges a _e , the number of edge corresponding point candidates c _c , the number of feature points a _p , the reliability v of the initial camera posture, and the motion as the tracking state. The blur intensity b is measured.
The edge number a _e represents the number of edges in the captured image corresponding to the model edge of the three-dimensional model 11.
The number c _c edge corresponding point candidate represents the number of edges corresponding point candidate existing around the edges in the captured image corresponding to the model edge. That is, the number c _c of edge corresponding point candidates indicates the number of different edges existing around the target edge.
The feature point number _ap represents the number of feature points in the captured image corresponding to the feature points in the feature point database 21.
The reliability v of the initial camera posture indicates the reliability of the integrated error with respect to the value before the camera posture is estimated (initial value of the camera posture). This is a parameter that takes into account that the error will increase depending on the initial value of optimization.
The motion blur intensity b indicates the number of edges that blur according to the motion of the camera 2. Here, the blurring of the edge means that the edge widens when the camera is shaken in terms of video.

In other words, in the present embodiment, the tracking state measuring unit 40 uses five parameters (v, b, c _c , a) from the input image as the tracking state, in addition to the edge correspondence error err ₁ and the feature point correspondence error err _{2. e} , a _p ) are measured. These five types of parameters are used by the reliability calculation unit 50 in the subsequent stage to calculate the reliability.
In order to measure these parameters, the tracking state measurement unit 40, as shown in FIG. 1, includes an edge detection matching unit 41, a motion blur calculation unit (motion blur calculation unit) 42, and a feature point detection matching unit (feature inspection). Output matching means) 43 and an initial camera attitude calculation unit (initial camera attitude calculation means) 44. Note that the measurement includes both direct measurement of the target amount and indirect measurement calculated from the result of direct measurement from the related amount.

The edge detection matching unit 41 detects an edge from an input captured video and performs a matching process between the model edge stored in the three-dimensional model 11 and the detected edge. Here, the edge detection matching unit 41 includes an edge detection unit (edge detection unit) 45 and an edge matching unit (edge matching unit) 46.
The edge detection unit 45 detects an edge from the photographed image to be input and outputs it to the edge matching unit 46.

The edge matching unit 46 performs a matching process between the model edge included in the three-dimensional model 11 of the subject and the edge detected by the edge detection unit 45, and calculates the number of edge corresponding point candidates c _c and the number of edges a _e . Count to calculate. The operator used in the edge matching unit 46 is not particularly limited. For example, the edge number a _e can be calculated by the edge detection operator. Here, the edge detection operator may be a Sobel operator, an edge detection method called a differential type such as a Prewitt operator or a Roberts operator, or a template type such as an edge detection operator of Robinson or an edge detection operator of Kirsch. Various edge detection methods such as an edge detection method can be used. The calculated number of edge corresponding point candidates c _c and the number of edges a _e are output to the reliability calculation unit 50. Note that the value of the obtained edge number a _e is also used in the reliability correction unit 60.

Further, the edge matching unit 46 obtains the edge correspondence error err ₁ by matching processing, and outputs the obtained edge correspondence error err ₁ to the posture movement amount calculation unit 71 of the camera posture estimation means 70. The edge matching unit 46 uses the camera posture (initial camera posture E ^(k) ) obtained from the initial camera posture calculation unit 44 in the state before estimation (this is referred to as state k) as the camera of the camera posture estimation unit 70. Output to the posture calculation unit 72.

The motion blur calculation unit (motion blur calculation means) 42 calculates a motion blur intensity b indicating the number of edges blurred in the video according to the motion of the camera, based on the motion blur of the input captured video. Note that the motion blur intensity b is not directly measured, but is calculated from the average edge width w _avg . A conventionally known method can be used for such conversion. The calculated motion blur intensity b is output to the reliability calculation unit 50.

A feature point detection matching unit (feature point detection matching means) 43 detects a feature point from the input captured video, and performs a matching process between the feature point stored in the feature point database 21 and the detected feature point. The number of feature points a _p is counted and calculated. The operator used in the feature point detection matching unit 43 is not particularly limited as long as it can count the number of feature points _ap . The calculated value of the number of feature points a _p is output to the reliability calculation unit 50. Note that the obtained value of the number of feature points a _p is also used in the reliability correction unit 60.

The initial camera posture calculation unit (initial camera posture calculation means) 44 obtains the result of the feature point matching process and information such as camera parameters stored in the feature point database 21 from the feature point detection matching unit 43, The camera posture (initial camera posture E ^(k) ) in the state before estimation (state k) is obtained by calculation and output to the edge matching unit 46. The state k corresponds to, for example, the frame number of the input video.
Further, the initial camera posture calculation unit 44 calculates the reliability v of the initial camera posture with respect to the obtained initial camera posture E ^(k) . The calculated value of the reliability v of the initial camera posture is output to the reliability calculation unit 50.

<Reliability calculator>
The reliability calculation unit (reliability calculation means) 50 uses the edge correspondence error err ₁ obtained from the edge in the captured image and the feature point in the captured image according to the tracking state measured by the tracking state measurement unit 40. Using the obtained feature point correspondence error err ₂ as an index, the reliability f of integrated tracking obtained by integrating edge-based tracking and feature point-based tracking is calculated.

In the present embodiment, the reliability calculation unit 50 uses the integrated tracking so that only the edge correspondence error err _{1 is} used when the value of the measured motion blur intensity b is larger than a predetermined first threshold th _b . The reliability f is calculated.
In addition, the reliability calculation unit 50 is configured such that the measured value of the motion blur intensity b is equal to or less than the first threshold th _b , and the measured value of the reliability v of the initial camera posture is a predetermined second threshold. If it is larger than th _v, the integrated tracking reliability f is calculated by proportionally proportional to the motion blur intensity b and inversely proportional to the reliability v of the initial camera posture and the number of feature points _ap. .

In addition, the reliability calculation unit 50, when the value of the measured motion blur intensity b is less than or equal to the first threshold th _b and the value of the reliability v of the measured initial camera posture is less than or equal to the second threshold th _v In this case, the distribution is proportional to the number of edges a _e and the motion blur intensity b, and inversely proportional to the reliability v of the initial camera posture, the number of feature points a _p, and the number of edge corresponding point candidates c _c. Then, the reliability f of the integrated tracking is calculated.

The reliability f of integrated tracking is described, for example, as in Expression (1). Here, the reliability of f the consolidated tracking, b is the motion blur intensity, v is the reliability of the initial camera position, a _e is the number of edges, a _p is the number of feature points, c _c is the number of edges corresponding point candidates, kappa ₀ , Κ ₁ is a constant for adjustment, th _b is a motion blur intensity b threshold value (first threshold value), and th _v is a reliability threshold value (second threshold value) of the initial camera posture. Κ ₀ , Κ ₁ , th _b , th _v can be specified by the user.

As will be described later, it is possible to apply only the necessary parameters by applying the if statement of formula (1) in order from the top. Here, as an example, since the tracking state measurement unit 40 calculates all five types of parameters (v, b, c _c , a _e , a _p ), Expression (1) is replaced with Expression (1a), Rewrite as (1b) and equation (1c).

<Reliability correction unit>
The reliability correction unit (reliability correction unit) 60 generates a correction reliability η obtained by correcting the integrated tracking reliability f calculated by the reliability calculation unit 50. Since the above-described equations (1b) and (1c) are derived on the assumption that the number of feature points is the same as the number of edges, the number of feature points is not equal to the number of edges. Can cause inconvenience. Therefore, the reliability f of the integrated tracking is corrected when the number of feature points and the number of edges are not equal. In this embodiment, the reliability correction unit 60 obtains a sample ratio γ indicating the ratio of the number of edges a _{e to} the number of feature points a _p , and corrects according to the sample ratio γ and the reliability f of integrated tracking at that time. A reliability η is generated. Thereby, even when the number of feature points and the number of edges are not equal, the estimation accuracy of the camera posture can be improved. The correction reliability η generated here is output to the camera posture estimation means 70.

In the present embodiment, the reliability correction unit 60 is proportional to the reliability f of integrated tracking when the reliability f of integrated tracking is greater than 0.5 and the sample ratio γ is less than 1, and The correction reliability η is calculated by the first correction formula that is inversely proportional to the sample ratio γ.
Further, when the integrated tracking reliability f is smaller than 0.5 and the sample ratio γ is larger than 1, the reliability correction unit 60 is proportional to the integrated tracking reliability f and the sample ratio γ. The correction reliability η is calculated by a second correction formula that is inversely proportional to.

  However, when correction is not necessary, that is, when the above two conditions are not satisfied, the reliability correction unit 60 directly uses the acquired integrated tracking reliability f as the correction reliability η (η = f ). That is, in this case, the integrated tracking reliability f (correction reliability η) is output to the camera posture estimation means 70.

  For example, the reliability correction unit 60 calculates the correction reliability η by the following equation (2), and calculates the sample ratio γ by the equation (3).

  Equation (2) can be decomposed into the first correction equation shown in the following equation (2a), the second correction equation shown in equation (2b), equation (2c), and equation (2d).

<Camera posture estimation means>
The camera pose estimation means 70 generates an integrated error err that is an apportionment of the edge correspondence error err ₁ and the feature point correspondence error err ₂ according to the integrated tracking reliability f calculated by the reliability calculation unit 50. The current camera posture is estimated so that the integrated error err is minimized. Here, the current camera posture is expressed as E ^{(k + 1)} with respect to the camera posture in the state k before estimation (initial camera posture E ^(k) ). The state k indicates the state of each frame of the input video, for example. In the present embodiment, the camera posture estimation means 70 obtains the integrated error err according to the correction reliability η generated by the reliability correction unit 60 and estimates the current camera posture E ^{(k + 1)} . .

The camera posture estimation means 70 estimates the camera posture when the camera tracking error is minimized by, for example, a linear iterative solution method described later, and includes a posture movement amount calculation unit 71 and a camera posture calculation unit 72.
The posture movement amount calculation unit 71 corrects the edge correspondence error err ₁ acquired from the edge matching unit 46 and the feature point correspondence error err ₂ acquired from the feature point detection matching unit 43 from the reliability correction unit 60. Weighted summation is performed according to the degree η, and the camera attitude movement amount ΔE is calculated so that the integration error err is minimized. The posture movement amount calculation unit 71 obtains the integrated error err using, for example, Expression (4). The integrated error err is a single weight value obtained by the error (reliability) of both the edge and the feature point in the error function to be minimized. The weight value is obtained analytically by analysis as will be described later.

err = ηerr ₁ + (1−η) err ₂ ... (4)

Camera orientation calculation unit 72, a camera attitude change amount ΔE calculated in attitude change amount calculation unit 71, the initial camera pose before estimates obtained from the edge matching unit 46 E ^(k) (the state k of the camera posture E ^{(k )} )), The current camera posture E ^{(k + 1)} is estimated by equation (5). The camera postures E ^(k) , E ^{(k + 1) and the} like are represented by a matrix as will be described later, and “·” in the equation (5) indicates matrix multiplication.

E ^{(k + 1)} = ΔE · E ^(k) (5)

  Note that the tracking state measurement unit 40, the reliability calculation unit 50, the reliability correction unit 60, and the camera posture estimation unit illustrated in FIG. 1 are stored in, for example, a predetermined program stored in a ROM or the like of a storage unit by the CPU. It is realized by expanding and executing the above.

<Output means>
The output unit 80 outputs the estimated current camera posture E ^{(k + 1)} to the rendering device 4 (see FIG. 2) or an output device (not shown), and is configured by a predetermined output interface or the like. The output device (not shown) includes, for example, an HMD, a CRT (Cathode Ray Tube), a liquid crystal display (LCD), a PDP (Plasma Display Panel), an EL (Electronic Luminescence), and the like. Note that an output device (not shown) can also switch and display information output from the rendering device 4 (see FIG. 2) and the video composition device 5 (see FIG. 2).

[4. Model-based camera tracking by linear iterative method]
<4.0. >
Here, the principle of the linear iterative solution performed by the camera posture estimation means 70 will be described. The camera pose estimation process is based on Drummond's method based on Lie group and Lie algebra. The Drummond method is described in `` Drummond T. and Cipolla R .: Real-time visual tracking of complex structures, IEEE Trans. On Pattern Analysis and Machine Intelligence 2002; 24 (7): 932-946 ''. Yes. The specific method is described below.

ここで、Ｒおよびｔはそれぞれカメラの回転マトリクスと並進マトリクスである。 <4.1. Camera projection matrix>
The camera projection matrix P includes three-dimensional coordinates (X, Y, Z) on the scene (such as feature points on the subject) and two-dimensional coordinates (u / w, v / w) on the photographed image obtained by projecting them. From this relationship, the camera internal matrix is defined as K and the external parameter is multiplied by E, and is expressed as the following equation (6). The internal matrix indicates optically unknown internal parameters including, for example, lens distortion.

Here, R and t are a rotation matrix and a translation matrix of the camera, respectively.

<4.2. Estimation by repetitive update of camera posture>
When the camera internal parameters and the 3D-2D correspondence of visual cues are given, the camera external parameter E can be calculated by solving the equation (6). In this embodiment, the following equation (7) is repeated to update the camera posture obtained before or roughly from the current frame to the camera posture matrix of the current frame. That is, the motion matrix for the current frame is optimized by repetition.

Here, M is a 4 × 4 motion matrix, which is a minute translation with respect to the x, y, z axis directions and a minute rotation with respect to the x, y, z axes, and is represented by the following equation (8).

In order to reduce the calculation cost, the motion matrix M is approximated by the following equation.

Therefore, the estimation of the camera posture is equivalent to estimating α _i (i = 0, 1,..., 5), which can be performed by repeating the above calculation.

<4.3. Estimation of α _i by iterative calculation>
The square sum of errors between the projected model (coordinates of visual cues) and the corresponding visual cues (corresponding points) on the captured image can be formulated as follows.

Here, N is the number of visual cues such as edges and feature points, and d is the distance between the coordinates of the visual cues and the corresponding points. f _i is the motion component between the coordinates of the visual cue and the corresponding point (related to the displacement when the point is projected with G _i : see Drummond's method).
If α _i is equal to the correct α _i ^GT , the partial differential equation of equation (11) is zero.

However, since α _i usually includes an error term ε in a real environment, equation (11) does not become zero.
That is, α _i = α _i ^GT + ε. Therefore, equation (11) can be modified as follows.

From the equation (12), the following linear equation for obtaining the error term ε can be obtained.

α _i ^Ext can be obtained by subtracting the obtained error term ε from α _i . Also, it converges near the correct answer by iterative processing. That is, the motion matrix M has a loop that solves the linear equation in Section 4.3 inside the loop that repeats the expression (7) described in Section 4.2, and is optimized by these iterations.

<4.4. Analytical fusion of visual cues>
In order to use the visual cues of both the edge and the feature point in a complementary manner, the above equation (10) is modified as follows.

Here, N _e + N _f is the number of visual cues (= N), N _e is the number of edges (a _e ), and N _f is the number of feature points (a _f ). Further, η indicates a correction reliability. The above formula (11), formula (12), and formula (13) are similarly modified. In order to establish these, it is necessary to determine the correction reliability η. In the equation (14), the correction reliability η may be replaced with the integrated tracking reliability f.

[5. Analysis method of shooting environment]
<5.0. Premise>
In this chapter, we analyze the desired conditions to robustly track each visual cue (edge or feature point) and dynamically adjust the integrated tracking reliability f or correction reliability η in any environment An example of a method for obtaining the evaluation formula will be described. In the following, the unique properties of edges and feature points are analyzed and analyzed, but this analysis focuses on the dependency of both edges and feature points on environmental conditions.

  For this analysis, CG images were produced and used with known camera work. Here, the camera is set to be separated from the CG object so that as many CG objects existing in the scene as possible can be photographed. Then, using the created video, the camera posture was estimated by each of the method using only the feature points and only the edges. To perform a thorough analysis, a known (correct) posture was used as the initial (or just prior) posture. Here, four types of analyzes (Analysis 1 to Analysis 4) were performed.

<5.1. Analysis 1: When Gaussian noise is added to the initial camera posture estimation value>
In order to analyze the robustness of feature points (hereinafter also simply referred to as points) or the edge-based camera tracking method with respect to the accuracy (reliability v) of the initial value of camera posture estimation (value before posture estimation), camera posture estimation A simulation was performed by adding different levels of Gaussian noise to the initial translation term. The result at this time is shown in FIG.

FIG. 4A shows the result of point-based tracking, and FIG. 4B shows the result of edge-based tracking. The horizontal axis of each graph indicates the camera translations t ₁ , t ₂ , t ₃ measured as the camera posture, the camera rotations r ₁ , r ₂ , r _3, and the reprojection error. The vertical axis of each graph indicates an error. The error is shown by standard deviation. The unit of rotation angle of the camera posture is radian, and the unit of reprojection error is a pixel on the camera screen.

  In FIG. 4, “v” represents not the value (0 to 1) of the reliability “v” of the initial camera posture but the “added noise” corresponding thereto. In this analysis 1, the estimation error was obtained by changing the value of v in the range of integer values from 0 to 30.

  When the added noise is small, as shown in FIG. 4B, there is no significant effect on edge-based tracking. Next, when the added noise was large, there was no significant effect on point-based tracking, as shown in FIG. However, in this case, as shown in FIG. 4B, the performance of edge-based tracking has decreased exponentially. For this reason, the point-based tracking is more advantageous for the posture accuracy used for the initial value of the estimation. Therefore, in order to determine the reliability of edge-based tracking, it is considered that the accuracy of the posture used for the initial value of the estimation is an effective condition.

<5.2. Analysis 2: Scene complexity>
≪5.2.1≫
The complexity of the scene is the texture and the fineness of the edges contained in the scene itself, and is directly related to the amount of two visual cues (the number of edges a _e and the number of feature points a _p ). Therefore, we analyzed the performance of point-based tracking and edge-based tracking with respect to the amount of two visual cues. The result at this time is shown in FIG. The graphs shown in FIGS. 5A and 5B have the horizontal and vertical axes similar to those in FIGS. 4A and 4B. In FIG. 5A, a corresponds to the number of feature points a _p , and in FIG. 5B, a corresponds to the number of edges a _e . Each graph shows experimental results when the parameter a is changed.

The amount of direct visual cues (number of edges a _e and number of feature points a _p ) can be obtained by changing the sample interval on the edge of the model (three-dimensional model of the object) and sub-reference feature points (feature point database). The sum of visual cues is adjusted by an operation that is evenly distributed according to the degree of the sample. However, the amount of visual cues (the number of edges a _e and the number of feature points a _p ) needs to be kept above a certain minimum amount in order to estimate the appropriate camera posture. In this example, the number of feature points a _p is kept at 15 or more, and the number of edges a _e is kept at 50 or more. When the amount of these visual cues is sufficient, it has been found that performance is not significantly affected by changes in the amount of visual cues, as shown in FIG. However, when the amount of visual cues decreases, performance decreases linearly.

≪5.2.2≫
The complexity of the scene is related to the distribution of edges that easily affect the reliability in the process of searching for corresponding points with the model edges in the captured image. For example, if there are many erroneous edges (edges that do not correspond to model edges) in the captured image, the possibility of increasing the number of erroneous correspondences (number of edge correspondence point candidates c _c ) increases. Therefore, we analyzed the performance of edge-based tracking related to reliability in the process of searching for corresponding points of model edges. Therefore, l red lines arranged at random are additionally drawn on the camera image so as to interfere. As a result of the experiment, as shown in FIG. 6, it has been found that an increase in l significantly reduces the performance of edge-based tracking. Note that the graph of FIG. 6 has the same axis as the graph of FIG.

<5.3. Analysis 3: Add motion blur to camera image>
Different levels of horizontal motion blur were added to the camera image and simulated. To simulate motion blur, a moving average filter [1 / b... 1 / b] was used (the number of 1 / b in [] is b). FIG. 7 shows the estimated posture difference with respect to the correct camera posture. Note that the graph of FIG. 7 has the same axis as the graph of FIG. In FIG. 7, b represents the motion blur intensity. In this analysis, the estimation error was obtained by changing the value of b in the range of integer values from 0 to 11.

  As the motion blur (motion blur intensity b) increases, the performance of both the point-based tracking in FIG. 7A and the edge-based tracking in FIG. 7B decreases in proportion. However, it can be seen that point-based tracking is relatively heavily affected. Therefore, it is considered that the integrated tracking reliability f or the correction reliability η should increase in proportion to the amount of motion blur. It was also noted that the increase in motion blur has led to a decrease in the number of feature points found in camera images. Therefore, it is considered that the motion blur must be equal to or less than a predetermined threshold (b = 11 according to this experiment) in order to be able to extract feature points sufficient for the corresponding camera posture estimation.

<5.4. Analysis 4: Add noise to camera image>
A simulation was performed by adding different levels of Gaussian noise N (0, n ² ) to the pixel values (R, G, B) of the camera image. The difference between the estimated posture and the correct answer is shown in FIG. Note that the graph of FIG. 8 has the same axis as the graph of FIG. In this analysis, the estimation error was obtained by changing the value of n in the range of integer values from 0 to 100.

  In both the point-based tracking shown in FIG. 8A and the edge-based tracking shown in FIG. 8B, changing the value of n did not significantly affect the estimation error. Similar results were obtained even when uniform noise U (-n, n) was used instead of Gaussian noise. These experimental results are considered to be due to SURF (Speeded Up Robust Features) and robustness against the noise of the Canny operator.

<5.5. Summary of analysis>
As a result of the analysis, the integrated tracking reliability f or the correction reliability η (hereinafter simply referred to as f) is proportional to the number of edges a _e and the motion blur intensity b, and v and the number of feature points a _p shown in FIG. Is in inverse proportion to l shown in FIG. 6, and is not related to n shown in FIG. Here, v shown in FIG. 4 corresponds to the reliability v of the initial camera posture.

Here, for the sake of simplicity, it is assumed that the relationship between the integrated tracking reliability f and each parameter is linear. In practice, v shown in FIG. 4 is not directly measured, but v is calculated from the average distance (d _avg ) between the feature points and their corresponding points. Also, instead of directly measuring the l shown in FIG. 6, assuming that the number c _c edge corresponding point candidate is proportional to l shown in FIG. 6, the l from a few c _c edge corresponding point candidate calculate. Further, instead of directly measuring the motion blur intensity b, b is calculated from the average edge width ( _wavg ). Such a conversion formula is known.

  From the above analysis, the reliability f of the integrated tracking calculated by the reliability calculation unit 50 is described as the above-described equation (1). As described above, since the properties specific to edges and feature points are analyzed based on experiments in consideration of the shooting environment, the derived equations (1) and (2) are optimally and flexibly applied to any environment. be able to.

[6. Operation of camera posture estimation device]
Next, the operation of the camera posture estimation apparatus 1 shown in FIG. 1 will be described. The block diagram of FIG. 1 shows the camera parameter estimation algorithm of the camera posture estimation apparatus 1 as it is. That is, the tracking state measurement unit 40 calculates all five types of parameters (v, b, c _c , a _e , a _p ) for the input video, and the above-described equations (1a), (1b), and (1c) was used. The description of the typical processing flow of the camera posture estimation device 1 is omitted, and instead, the tracking state measurement unit 40 uses the above-described equation (1) and is required among the five types for the input video. With reference to FIG. 9 (refer to FIG. 1 as appropriate), an example of the procedure of the reliability calculation process and the reliability correction process when calculating only the parameters will be described.

In this case, first, the camera posture estimation device 1 measures the motion blur intensity b based on the motion blur of the input captured video by the motion blur calculation unit 42 of the tracking state measurement unit 40 (step S1). Then, the camera posture estimation apparatus 1 determines whether or not the value of the motion blur intensity b is larger than the first threshold th _b by using the reliability calculation unit 50 (step S2). When the value of the motion blur intensity b is equal to or less than the first threshold th _b (step S2: No), the camera posture estimation device 1 measures the number of feature points _ap by the feature point detection matching unit 43 of the tracking state measurement unit 40. The initial camera posture calculation unit 44 measures the reliability v of the initial camera posture (step S3).

Then, the camera posture estimation device 1 determines whether or not the reliability v of the initial camera posture is larger than the second threshold th _v by the reliability calculation unit 50 (step S4). When the reliability v of the initial camera posture is equal to or smaller than the second threshold th _v (step S4: No), the camera posture estimation device 1 uses the edge matching unit 46 of the tracking state measurement unit 40 to determine the number of edges a _e and the edge. measuring the number c _c corresponding point candidate (step S5).

Then, the camera posture estimation apparatus 1 uses the reliability calculation unit 50 to calculate the reliability of integrated tracking according to the above-described equation (1c) using the five types of parameters (b, v, a _p , a _e , and c _c ). In addition to calculating the degree f, the reliability correction unit 60 calculates the sample ratio γ according to the above-described equation (3) using the two types of parameters (a _p , a _e ) (step S6).

  The camera posture estimation apparatus 1 determines the relationship between the integrated tracking reliability f and the sample ratio γ by the reliability correction unit 60 (step S7). Then, when the reliability f of integrated tracking is larger than 0.5 and the sample ratio γ is smaller than 1 (f> 0.5 and γ <1), the reliability correction unit 60 described above. The correction reliability η is calculated from the equation (2a) (step S8).

In step S7, the reliability correction unit 60 determines that the integrated tracking reliability f is smaller than 0.5 and the sample ratio γ is larger than 1 (f <0.5 and γ > 1), the correction reliability η is calculated by the above equation (2b) (step S9).
Further, in step S7 described above, the reliability correction unit 60 sets the integrated tracking reliability f to the correction reliability η (η = f) as it is in other cases (step S10). That is, when f ≦ 0.5 and γ ≧ 1, or f ≧ 0.5, and γ ≦ 1, the correction reliability η is calculated according to the above equation (2c) or equation (2d).

On the other hand, when the value of the reliability v of the initial camera posture is larger than the second threshold th _{v in} step S4 described above (step S4: Yes), the camera posture estimation device 1 uses the reliability calculation unit 50 until then. Using the three parameters (b, v, a _p ) obtained in (1), the integrated tracking reliability f is calculated according to the above-described equation (1b) (step S11). Subsequently, in step S10 described above, the reliability correction unit 60 sets the integrated tracking reliability f as it is as the correction reliability η (η = f).

Thus, when the value of the reliability v of the initial camera posture is larger than the second threshold th _v (step S4: Yes), the number of edges a _e and the number of edge corresponding point candidates c _c are measured (or calculated). ) And the reliability f of the integrated tracking need not be corrected, so that the processing load can be reduced and the processing speed can be increased. In virtual reality (VR) and the like, real-time performance is an important factor, so the lower the computational load, the better. Therefore, the process of calculating only the necessary parameters as in the flow of FIG. 9 is effective for application to VR.

If the value of the motion blur intensity b is larger than the first threshold th _{b in} step S2 described above (step S2: Yes), the camera posture estimation device 1 uses the above-described formula (1a) by the reliability calculation unit 50. ), The integrated tracking reliability f is set to "1" (step S12). Subsequently, in step S10 described above, the reliability correction unit 60 sets the integrated tracking reliability f as it is as the correction reliability η (η = f = 1).

Thus, when the value of the motion blur intensity b is larger than the first threshold th _b (step S2: Yes), it is not necessary to measure (or calculate) parameters other than the motion blur intensity b, and integrated tracking Since it is not necessary to correct the reliability f, the processing load can be reduced and the processing speed can be increased.

In addition, according to the present embodiment, the camera posture estimation device 1 includes the edge correspondence error err ₁ according to the tracking state measured based on the three-dimensional model 11 of the subject, the feature point database 21, and the captured image. An integrated error err is generated by varying the proportion of the feature point correspondence error err ₂ and the current camera posture is estimated. Therefore, as shown in the examples described later, the accuracy of the camera tracking, the robustness, and the freedom of the shooting environment are improved as compared with the case where either one of the errors is used or when both errors are fixedly distributed. Can be improved.

  In addition, according to the present embodiment, in order to use edges and feature points in a cooperative manner, analysis of the environment and how to evaluate two features of edges and feature points in a unified manner are considered. Furthermore, the properties unique to edges and feature points were analyzed based on experiments. Thereby, the derived equations (1) and (2) can be applied optimally and flexibly to any environment. Therefore, there is an effect that the model-based camera posture estimation method can be made robust.

  As mentioned above, although this invention was demonstrated based on this embodiment, this invention is not limited to this. For example, the camera posture estimation apparatus shown in FIG. 1 can be realized by causing a general computer to operate according to a program that functions as each of the means described above. This program (camera posture estimation program) can be distributed via a communication line, or can be distributed by writing on a recording medium such as a CD-ROM.

  In the present embodiment, the camera posture estimation means 70 estimates the camera posture by, for example, a linear iterative solution method. However, for solving the minimum value problem, for example, a linear programming method such as a Lagrange undetermined multiplier method is used. The calculation may be performed using a nonlinear programming method such as a penalty method.

  In order to confirm the effect of the present invention, a computer simulation experiment was conducted to verify the performance of the camera posture estimation apparatus of the present invention. Specifically, an experiment (experiment 1) comparing the camera posture estimated by the camera posture estimation apparatus of the present invention with a camera posture estimated by using an edge or a feature point alone, and application to augmented reality An experiment (Experiment 2) was conducted to confirm the adaptability to environmental changes in the system.

[Experiment 1]
(Experimental conditions)
In the above equation (1), the values of user-specified constants of th _b , th _v , Κ ₀ , and の₁ are th _b = 95, th _v = 6, Κ ₀ = 0.8, and Κ ₁ = 0.12, respectively. It was specified as follows.

(Comparative example)
The estimated camera posture was obtained by using the feature points independently (point-based method: Comparative Example 1).
In addition, the camera posture estimated by using the edge alone was obtained (edge-based method: Comparative Example 2). Further, when both edge tracking and feature point tracking are used, an estimated camera posture is obtained using an error weighted with a fixed value of 0.5 regardless of the situation (reliability is η = Method fixed at 0.5: Comparative Example 3).

(Experimental result)
The camera orientation was measured for camera translations t ₁ , t ₂ , t ₃ and camera rotations r ₁ , r ₂ , r ₃ . Table 1 shows the average values and standard deviations as measurement results.

The experimental results in the case of camera translation t ₁ are shown in FIG. The experimental results in the case of camera translation t ₂ are shown in FIG. The experimental results in the case of camera translation t ₃ are shown in FIG. 10 to 12, (a) shows Comparative Example 1, (b) shows Comparative Example 2, (c) shows Comparative Example 3, and (d) shows an example of the present invention. FIG. 10E shows a change in the correction reliability η in the embodiment of the present invention in the case of the translation t ₁ of the camera. In each graph of FIGS. 10 to 12, the horizontal axis indicates the frame number (time axis). The vertical axis represents the estimated camera position error (difference from the correct answer). In addition, the vertical axis | shaft of the graph of FIG.10 (e) has shown correction | amendment reliability (eta).

L (el) shown in the graph of FIG. 10 (e) is the parameter described in the above section 5.2.2, and is the number of red lines randomly arranged in the camera image by additional drawing so as to interfere. _Yes , this corresponds to the number c _c of edge corresponding point candidates. Here, l = 300 throughout each frame. Further, an average distance d _avg between the feature points and their corresponding points was obtained as the reliability v of the initial camera posture, and v was calculated _therefrom . In frame numbers 101 to 151, v = 5, but in frame numbers 151 to 201, v = 10. Further, b was calculated from the average edge width w _avg as the motion blur intensity b. In frame numbers 126 to 176, b = 7 (b was smaller than 11). As a result, the correction reliability η changed as shown in the graph of FIG.

  As shown in Table 1 and FIGS. 10 to 12, the example of the present invention had better accuracy than Comparative Example 1 and Comparative Example 2 that were estimated using edges or feature points alone. Also, compared with Comparative Example 3 in which both the edges and feature points are used and their contributions are simply averaged, the embodiment of the present invention has good accuracy and is temporally Stable accuracy was also obtained. Therefore, it can be said that the camera posture estimation apparatus of the present invention has improved accuracy and improved robustness compared to the prior art.

[Experiment 2]
The video synthesis output results of the video synthesis system 100 shown in FIG. 2 were compared. In the shooting environment, the lighting and the direction of the camera change over time. In this video composition system 100, FIG. 13 schematically shows a video composition result when the camera posture is estimated when the illumination changes with the movement of the camera. FIG. 13A, FIG. 13B, and FIG. 13C show video composition output results when time elapses in this order. Among the video composition output results, only the pot indicates a CG object, and the stand and the wall surface on which the pot is placed indicate a live-action video. In FIG. 13, the change in illumination is exaggerated. FIG. 13A shows a case where the illumination is normal, FIG. 13B shows a case where the illumination becomes dark, and FIG. 13C shows a case where the illumination becomes bright.

  Even when the illumination suddenly becomes dark as shown in FIG. 13B, the camera posture can be stably estimated by detecting both edges and feature points, and the CG object and the live-action image can be accurately obtained. I was able to synthesize. Further, as shown in FIG. 13C, even when the lighting suddenly becomes brighter, it is possible to stably estimate the camera posture by detecting both the edge and the feature point, and to obtain the CG object and the live-action image. It was possible to synthesize with high accuracy. Since the conventional model-based camera estimation method is constructed on the assumption that the illumination and the direction of the camera do not change with time, the conventional method synthesizes the CG object and the live-action image with high accuracy in this way. I couldn't. Therefore, it can be said that the video composition system using the camera posture estimation apparatus of the present invention can improve the adaptation to the change of environment and the flexibility more than before.

  The present invention can be used for industrial applications of virtual reality and augmented reality and video production using video composition. Also, it can be used as a sensor for various robots that use posture information.

DESCRIPTION OF SYMBOLS 100 Image composition system 1 Camera posture estimation apparatus 2 Camera 3 Virtual 3D object model storage means 4 Rendering apparatus 5 Image composition apparatus 6 CG data 10 3D model storage means 11 3D model 20 Feature point database storage means 21 Feature point database 30 Video input means 40 Tracking state measuring unit (tracking state measuring means)
41 Edge detection matching unit 42 Motion blur calculation unit (motion blur calculation means)
43 Feature Point Detection Matching Unit (Feature Point Detection Matching Unit)
44 Initial camera posture calculation unit (initial camera posture calculation means)
45 Edge detection unit (edge detection means)
46 Edge matching part (edge matching means)
50 Reliability calculation part (Reliability calculation means)
60 reliability correction unit (reliability correction means)
70 Camera Posture Estimation Unit 71 Posture Movement Amount Calculation Unit 72 Camera Posture Calculation Unit 80 Output Unit

Claims (5)

A model-based camera posture estimation device for estimating a camera posture using edges and feature points in a captured image of a subject,
Three-dimensional model storage means for storing a three-dimensional model indicating information on characteristics of a subject existing in the shooting direction of the camera;
Feature point database storage means for storing a feature point database storing feature point information including feature point descriptors and three-dimensional information of the three-dimensional model;
Based on the first three-dimensional model and the feature point database created in advance and the first tracking error determined based on the edge based on the input photographed image including the subject, and the feature point Tracking state measuring means for measuring a tracking state including at least the second tracking error determined in the above;
Reliability calculation for calculating the reliability of integrated tracking in which edge-based tracking and feature point-based tracking are integrated as an index for apportioning the first error and the second error according to the measured tracking state Means,
According to the reliability of the integrated tracking, the ratio of dividing the first error and the second error is changed to generate an integrated error, and the current camera posture is estimated so that the integrated error is minimized. Camera posture estimation means for
A camera posture estimation apparatus comprising: The tracking state measuring means includes
As the tracking state, the first error, the second error, the number of edges, the number of feature points, the motion blur intensity indicating the number of edges blurred according to the motion of the camera, and before the estimation of the camera posture Measuring the reliability of the initial camera posture indicating the reliability of the value of the value and the number of edge corresponding point candidates existing around the edge in the captured image corresponding to the model edge of the three-dimensional model;
The reliability calculation means includes:
When the value of the motion blur intensity measured as the tracking state is larger than a predetermined first threshold value, the reliability of the integrated tracking is calculated so that only the first error is used,
When the value of the motion blur intensity is equal to or less than the first threshold value and the reliability value of the initial camera posture measured as the tracking state is larger than a predetermined second threshold value, the motion blur value is determined. Proportionally proportional to the intensity and inversely proportional to the reliability of the initial camera posture and the number of feature points, respectively, to calculate the reliability of the integrated tracking,
When the value of the motion blur intensity is less than or equal to the first threshold, and when the reliability value of the initial camera posture is less than or equal to the second threshold, the number of edges and the motion blur intensity are proportional to each other, And, the reliability of the initial tracking is calculated, and the reliability of the integrated tracking is calculated so as to be inversely proportional to the number of feature points and the number of edge corresponding point candidates.
The camera posture estimation apparatus according to claim 1. A reliability correction means for generating a correction reliability that corrects the reliability of the integrated tracking;
The camera posture estimation means varies a proportion of the first error and the second error according to the correction reliability, estimates a current camera posture,
The reliability correction means includes
If the integrated tracking reliability is greater than 0.5 and the sample ratio indicating the ratio of the number of edges to the number of feature points is less than 1, the sample is proportional to the integrated tracking reliability and the sample Calculating the correction reliability according to the first correction formula inversely proportional to the ratio;
When the integrated tracking reliability is smaller than 0.5 and the sample ratio is larger than 1, the second correction formula is proportional to the integrated tracking reliability and inversely proportional to the sample ratio. To calculate the correction reliability,
The camera posture estimation apparatus according to claim 2. The tracking state measuring means includes
Edge detection means for detecting an edge from the input captured video;
Edge matching means for calculating the number of edge corresponding point candidates and the number of edges by matching processing between model edges included in the three-dimensional model and the detected edges;
Motion blur calculation means for calculating the motion blur intensity based on the motion blur of the photographed video input;
A feature point detection matching unit that detects a feature point from the input captured video and calculates the number of feature points by a matching process between the feature point stored in the feature point database and the detected feature point;
An initial camera posture calculating means for calculating the initial camera posture from the result of the feature point matching process and calculating a reliability of the initial camera posture with respect to the obtained initial camera posture;
The camera posture estimation apparatus according to claim 2 or claim 3, wherein Three-dimensional model storage means for storing a three-dimensional model indicating information on the characteristics of the subject existing in the shooting direction of the camera in order to estimate the camera posture based on the model using the edges and feature points in the captured image of the subject; A computer comprising a feature point database storing means for storing a feature point database storing feature point information including feature point descriptors and three-dimensional information of the three-dimensional model,
Based on the first three-dimensional model and the feature point database created in advance and the first tracking error determined based on the edge based on the input photographed image including the subject, and the feature point Tracking state measuring means for measuring a tracking state including at least the second tracking error determined in the above;
Reliability calculation for calculating the reliability of integrated tracking in which edge-based tracking and feature point-based tracking are integrated as an index for apportioning the first error and the second error according to the measured tracking state means,
According to the reliability of the integrated tracking, the ratio of dividing the first error and the second error is changed to generate an integrated error, and the current camera posture is estimated so that the integrated error is minimized. Camera posture estimation means,
Camera posture estimation program to function as

Images