JP6061770B2 - Camera posture estimation apparatus and program thereof - Google Patents

Description

  The present invention relates to a camera posture estimation device for estimating the posture of a photographing camera and a program thereof.

  In video production, CG (computer graphics) objects are synthesized using a camera posture so as to be in contact with a subject in a captured video. For example, this combining process is a process in which a desk is included in the captured video and a CG bag is combined on the desk.

  Conventionally, a method of measuring a camera posture using a photographed video has been proposed. In general, in a camera posture estimation method using a photographed video, the positions of feature points and feature vectors on the photographed video are extracted by video analysis. In this camera posture estimation method, the feature point position is tracked in the captured video using the feature vector similarity. Further, in this camera posture estimation method, the camera posture is estimated by an optimization method such as bundle adjustment using the tracking result (see, for example, Patent Document 1).

  For example, the feature point is a portion where the pattern is a corner on the photographed video, and indicates a specific part of the pattern appearing on the subject. Therefore, in the camera posture estimation method using the photographed video, when the CG object is virtually in contact with the subject in the photographed video as described above, it is difficult for the subject and the CG object to shift. It has the advantage of being suitable for video production. On the other hand, in this camera posture estimation method, when a feature point cannot be extracted or a feature point cannot be tracked, the accuracy of camera posture estimation decreases or the estimation is broken.

  In addition, a method for measuring the camera posture using posture measuring means such as a gyro sensor and an acceleration sensor has been proposed (see, for example, Patent Document 2). In this camera posture measurement method, the detection method of the sensor is dead reckoning (relative self-position estimation), so that an error of the drift component occurs. Also, with this camera attitude measurement method, the accuracy required for video production is not satisfied even if the error of the drift component is ignored.

  Furthermore, in the camera posture measurement method using the sensor, the posture of the sensor itself or the posture of the camera housing on which the sensor is mounted is measured, and measurement is not performed on the subject. For this reason, with this camera orientation measurement method, when the camera is moved or changed in direction, a measurement error of the subject due to structural distortion occurs in the camera housing or lens, and this measurement error is calibrated. Thus, it is difficult to obtain an accurate camera posture.

  As described above, the camera posture estimation method using a captured image and the camera posture measurement method using a sensor have different advantages and disadvantages, respectively, and obtain a highly accurate camera posture that selects a shooting environment. Is difficult.

Therefore, it has been conventionally proposed to use a method using a captured image and a sensor (hereinafter referred to as “conventional combination method”). In the conventional combined method, a total average of reprojection errors is obtained as a scale for evaluating the estimation accuracy of the camera posture.
The reprojection error indicates an error between the two-dimensional coordinates obtained by reprojecting the feature points on the subject represented by the three-dimensional coordinates on the imaging surface of the photographing camera and the two-dimensional coordinates of the extracted feature points.

  In the conventional combination method, either the camera posture estimated from the captured image or the camera posture measured by the sensor is selectively used based on the total average of the reprojection errors. At this time, in the conventional combined method, when the corresponding points exist in the captured image without deviation, the reprojection error of each feature point is uniform, so the total average of the reprojection errors falls within a certain range, and the camera posture is estimated. Accuracy can be correctly evaluated.

JP2009-237845A JP2007-14993

  However, in the conventional combined method, when the corresponding points are unevenly distributed in the captured image, even if the reprojection error of each feature point is not uniform, the total average of the reprojection errors converges to a certain range, and the camera It is difficult to correctly evaluate posture estimation accuracy.

  Accordingly, an object of the present invention is to solve the above-described problem and provide a camera posture estimation device and a program thereof that can estimate the camera posture with high accuracy.

  In order to solve the above-described problem, a camera posture estimation device according to the first invention of the present application is a camera posture measurement in which a photographed image photographed by a photographing camera and a measuring means attached to the photographing camera measure the posture of the photographing camera. A camera posture estimation device for estimating the posture of a photographing camera using information, a camera posture estimation means, a non-uniformity calculation means, an uneven distribution degree calculation means, an evaluation value determination means, and a high-precision section estimation Means, medium accuracy interval estimation means, and accuracy interval estimation means.

  According to this configuration, the camera posture estimation device extracts the feature points from the frame images constituting the captured video by the camera posture estimation means, and finds corresponding points in which the same feature points are associated between different frame images. The camera position estimation information obtained by estimating the attitude of the photographing camera and the spatial position of the corresponding point included in the frame image are obtained by searching and performing outlier exclusion processing and optimization processing.

  Further, the camera posture estimation device uses the camera posture estimation information for each divided region of the frame image by the non-uniformity calculation unit, and re-projects the corresponding point represented by the spatial position on the frame image, A reprojection error average value with the position in the image of the corresponding point searched by the camera posture estimation means is calculated, and a non-uniformity that is a maximum and minimum ratio of the reprojection error average value is calculated.

In addition, the camera posture estimation device calculates the uneven distribution degree of the corresponding points based on the number of corresponding points included in each divided region of the frame image by the uneven distribution degree calculation unit.
In addition, the camera posture estimation device uses the evaluation value determination unit based on the evaluation value, and based on the evaluation value, for the captured video, a high-accuracy section in which the estimation accuracy of the camera posture estimation information is high, and an estimation accuracy in the high-precision section. A medium accuracy interval lower than the medium accuracy interval and a low accuracy interval whose estimation accuracy is lower than the medium accuracy interval.

  This evaluation value is represented by a ratio between the uneven distribution degree and the nonuniformity degree. For this reason, the camera posture estimation apparatus can correctly evaluate the posture estimation accuracy of the photographing camera even when the corresponding points are unevenly distributed in the photographed image and the reprojection error of each corresponding point becomes non-uniform.

  Further, the camera posture estimation device outputs camera posture estimation information as the posture estimation result of the photographing camera in the high accuracy section by the high accuracy section estimation means. Then, the camera posture estimation device re-estimates the posture of the photographing camera using the camera posture estimation information by the optimization processing using the camera posture measurement information as a constraint condition by the medium accuracy interval estimation unit, and in the medium accuracy interval Output as the camera posture estimation result. Further, the camera posture estimation device outputs camera posture measurement information as a posture estimation result of the photographing camera in the low accuracy section by the low accuracy section estimation means.

Further, in the camera posture estimation device according to the second invention of this application, the non-uniformity calculation means calculates a reprojection error average value for each of the four divided regions, and the reprojection error is calculated as shown in Equation (1) described later. A non-uniformity degree D that is a ratio of the maximum E _reproMax of the average value and the minimum E _{reproMin of the} average reprojection error is calculated, and the uneven distribution degree calculating means calculates the number of corresponding points for each of the four divided regions, As shown in the following formula (2), the sum of the minimum number of corresponding points N _{Min in the} divided areas and the second smallest number of corresponding points N _{Min2nd in} the divided areas is determined for each divided area. by dividing the sum of the number N _i of the point, and calculates the uneven distribution of S.
According to such a configuration, the camera posture estimation device can accurately obtain the non-uniformity and the uneven distribution.

Further, in the camera posture estimation device according to the third invention of the present application, the medium accuracy interval estimation means obtains rotation information indicating the direction of the photographing camera from the camera posture measurement information as an optimization process using the camera posture measurement information as a constraint. The spatial position of the corresponding point included in the frame image of the high-accuracy section corresponding to the corresponding point included in the frame image of the medium-accuracy section is extracted and expressed in the spatial position in the direction of the shooting camera indicated by the rotation information Camera orientation estimation information is projected so that the corresponding points are projected onto the imaging surface of the camera, and the extension line between the corresponding points represented by the spatial position and the corresponding points projected onto the imaging surface converges to the optical principal point of the camera. The position of the photographing camera indicated by is optimized.
According to such a configuration, the camera posture estimation apparatus can perform interpolation so that the camera posture does not change suddenly at the boundary of the medium accuracy section.

  Here, the camera posture estimation apparatus according to the first invention of the present application is for causing hardware resources such as a CPU (Central Processing Unit), a memory, and an HDD (Hard Disk Drive) included in a computer to cooperate with each other as described above. It can also be realized as a camera posture estimation program. This camera posture estimation program may be distributed via a communication line, or may be distributed by writing in a recording medium such as a CD-ROM or flash memory (the fourth invention of the present application).

According to the present invention, the following excellent effects can be obtained.
According to the first and fourth inventions of the present application, even when the corresponding points are unevenly distributed in the photographed image and the reprojection error of each corresponding point becomes nonuniform, the evaluation value is represented by the ratio between the uneven distribution degree and the nonuniformity degree. Therefore, the posture estimation accuracy of the photographing camera can be correctly evaluated. Thus, according to the first and fourth inventions of the present application, the posture of the photographing camera can be estimated with high accuracy.

According to the second aspect of the present invention, since the nonuniformity and the uneven distribution are accurately obtained, the posture of the photographing camera can be estimated with higher accuracy.
According to the third aspect of the present invention, since the posture of the photographing camera does not change abruptly at the boundary of the medium accuracy section, it is possible to produce a smooth CG image using the posture estimation result of the photographing camera.

It is a block diagram which shows the structure of the camera attitude | position estimation apparatus which concerns on embodiment of this invention. In embodiment of this invention, it is explanatory drawing explaining a corresponding point. In embodiment of this invention, it is explanatory drawing explaining the principle which estimates a camera attitude | position from a picked-up image. In embodiment of this invention, it is explanatory drawing explaining a reprojection error. It is explanatory drawing explaining the block which the nonuniformity calculation means of FIG. 1 divided | segmented. It is explanatory drawing explaining the re-estimation by the re-estimation means of FIG. In embodiment of this invention, it is explanatory drawing explaining a coincidence of rotation information. It is explanatory drawing explaining the connection of the rotation information by the re-estimation means of FIG. In embodiment of this invention, it is explanatory drawing explaining the coincidence of position information. It is explanatory drawing explaining the shear by the re-estimation means of FIG. It is a flowchart which shows operation | movement of the camera attitude | position estimation apparatus of FIG. In the modification of this invention, it is explanatory drawing explaining the relationship between distribution of a corresponding point, and uneven distribution degree.

(Embodiment)
[Outline of camera posture estimation system]
An outline of the camera posture estimation system 100 will be described with reference to FIG.
As shown in FIG. 1, the camera posture estimation system 100 estimates the posture of the photographing camera C, and includes the photographing camera C and the camera posture estimation device 1.

The photographing camera C is a general camera that photographs a video. In the present embodiment, the photographing camera C includes posture measuring means Ca and lens state measuring means Cb.
The posture measuring means Ca measures the posture of the photographing camera C, and is, for example, a gyroscope and an acceleration sensor attached to the photographing camera C. Then, the posture measurement means Ca outputs sensor information (camera posture measurement information) indicating the posture of the photographing camera C to the camera posture estimation device 1.
This sensor information includes rotation information (rotation matrix) indicating the direction of the photographing camera C and position information (parallel progression) indicating the position of the photographing camera C.

  The lens state measuring means Cb measures the lens state (for example, zoom value) of the photographing camera C. For example, the lens state measuring means Cb is a rotary encoder that measures the amount of rotation of a zoom ring (not shown). Then, the lens state measurement unit Cb outputs lens information indicating the lens state to the camera posture estimation device 1.

  As long as the photographing camera C can measure the lens state and the camera posture, the posture measuring unit Ca and the lens state measuring unit Cb are not particularly limited. For example, the posture measuring means Ca may separately attach a sensor camera (not shown), analyze the video of this sensor camera, and generate position information of the photographing camera C.

The camera posture estimation device 1 estimates the posture of the photographing camera C by using both the photographed video input from the photographing camera C and the sensor information.
In the camera posture estimation method using the captured video, the higher the estimation accuracy, the better the quality of the CG video. For this reason, the camera posture estimation method using a photographed video has a high affinity for video production. Therefore, the camera posture estimation apparatus 1 is based on a camera posture estimation method that uses a captured video.
Hereinafter, “camera posture estimation method using captured video” may be abbreviated as “image analysis”.

  That is, the camera posture estimation device 1 evaluates an estimation result by image analysis, and shoots a captured image with a high-precision section with high estimation accuracy, a medium-precision section with medium estimation accuracy, and a low-precision section with low estimation accuracy. Divide into

Here, the camera posture estimation device 1 outputs the estimation result by the image analysis as it is for the high accuracy section.
In addition, the camera posture estimation apparatus 1 interpolates the estimation result obtained by image analysis for the medium accuracy interval using sensor information. In other words, the camera posture estimation device 1 can improve the accuracy of the estimation result by performing re-estimation using both the feature points and sensor information from the posture measurement means Ca in the medium accuracy section in which the feature points can be traced slightly. Improve as much as possible.
Furthermore, the camera posture estimation device 1 outputs sensor information for the low-accuracy section.

[Configuration of camera posture estimation device]
Hereinafter, the configuration of the camera posture estimation device 1 will be described.
As shown in FIG. 1, the camera posture estimation apparatus 1 includes a storage unit 10, an image analysis unit (camera posture estimation unit) 20, an estimation result determination unit 30, and a first interpolation processing unit (high accuracy section estimation unit). 40, a re-estimation unit (medium accuracy interval estimation unit) 50, and a second interpolation processing unit (low accuracy interval estimation unit) 60.

  The storage unit 10 is, for example, a semiconductor recording device that stores a captured video input from the camera C, sensor information, and lens information. At this time, the storage unit 10 stores the sensor information and the lens information in association with the frame image constituting the captured video.

The image analysis unit 20 obtains posture information (camera posture estimation information) indicating the posture of the photographing camera C by performing image analysis processing on the captured video stored in the storage unit 10. Specifically, the image analysis means 20 performs a feature point extraction process, a corresponding point search process, an outlier exclusion process, and an optimization process as the image analysis process.
This posture information includes rotation information (rotation matrix) indicating the direction of the photographing camera C and position information (parallel progression) indicating the position of the photographing camera C.

  First, the image analysis unit 20 extracts feature points such as SIFT (Scale Invariant Feature Transform) and SURF (Speeded Up Robust Features) from the standard image and the reference image.

As shown in FIG. 2, the reference image 90 is a frame image that is subjected to image analysis processing among the frame images that constitute the captured video.
The reference image 91 is a frame image that is different from the standard image 90.

Next, the image analysis unit 20 tracks the feature points 92 extracted from the standard image 90 and searches for corresponding points 92 between the standard image 90 and the reference image 91. This corresponding point 92 is obtained by associating the same feature point 92 between the base image 90 and the reference image 91. In the example of FIG. 2, the image analysis unit 20 searches the corresponding point 92 ₁ of the foreground part, the corresponding point 92 ₂ in the corner of the table, and a corresponding point 92 ₃ in the corner of the box.

Next, the image analysis unit 20 performs outlier exclusion processing such as RANSAC (RANdom SAmple Consensus), LmedS (the Least Median of Squares), and the like. Then, the image analysis means 20 performs optimization processing such as bundle adjustment, and obtains the posture information and the three-dimensional coordinates (spatial position) of the corresponding points 92. In the example of FIG. 3, the two-dimensional coordinates (image positions) of the corresponding points 92 ₁ , 92 ₂ , and 92 ₃ are known in the base image 90 and the reference images 91 and 91 ′. Further, the feature points 92 ₁ , 92 ₂ , 92 _{3 of} the subject Obj correspond to the corresponding points 92 ₁ , 92 ₂ , 92 ₃ of the base image 90 and the reference images 91 ₁ , 91 ′. Therefore, the image analysis means 20 uses the correspondence relationship between the corresponding points 92 ₁ , 92 ₂ , and 92 ₃ and optimizes the three-dimensional coordinates of the feature points 92 ₁ , 92 ₂ , and 92 ₃ of the subject Obj. , Posture information can be obtained.

  Thereafter, the image analysis unit 20 outputs the captured video, sensor information, and lens information stored in the storage unit 10 to the estimation result determination unit 30. Further, the image analysis means 20 estimates the posture information obtained by the image analysis processing, the three-dimensional coordinates of the corresponding points on the subject Obj, and the two-dimensional coordinates of the corresponding points obtained in the course of the image analysis processing. It outputs to the determination means 30.

  Note that the camera posture estimation device 1 can arbitrarily set the reference image 90 and the reference image 91 as long as the reference image 90 and the reference image 91 are different frame images. For example, when the captured video is short, the camera posture estimation device 1 performs image analysis processing by combining all of the standard image 90 and the reference image 91. In addition, since the processing load increases when the captured video is long, the camera posture estimation device 1 divides the captured video every predetermined time (for example, 30 seconds), and performs image analysis processing for each of the divided captured videos. Also good. Further, when the captured video is long, the camera posture estimation device 1 may perform a scene extraction process on the captured video and perform an image analysis process for each captured video of the same scene.

Returning to FIG. 1, the description of the configuration of the camera posture estimation apparatus 1 will be continued.
The estimation result determination unit 30 determines how accurate the estimation result of the image analysis unit 20 is. The non-uniformity calculation unit 31, the uneven distribution degree calculation unit 33, the evaluation value determination unit 35, and the like. Is provided.

The non-uniformity calculation means 31 calculates the average value of the reprojection error using posture information for each block (divided region) of the reference image 90, and calculates the non-uniformity D.
This reprojection error is a two-dimensional coordinate (reprojection position) obtained by reprojecting a corresponding point on the subject Obj onto the reference image 90, and a two-dimensional coordinate (intra-image coordinate) of the corresponding point extracted by the image analysis means 20. The error of is shown.

The uneven distribution degree calculating means 33 calculates the uneven distribution degree S based on the number of corresponding points included in each block of the reference image 90.
The evaluation value determining means 35 determines a high-precision section, a medium-precision section, and a low-precision section for the captured video based on the evaluation value C that is the ratio of the uneven distribution degree S and the non-uniformity degree D. is there.

<Determination by estimation result judging means>
The determination by the estimation result determination unit 30 will be described with reference to FIGS. 4 and 5 (see FIG. 1 as appropriate).
The camera posture estimation device 1 does not have a correct camera posture. For this reason, the estimation result determination unit 30 indirectly determines the accuracy of the estimation result based on whether the estimation condition is good or bad. Here, this inventor has obtained the following knowledge (A)-(D).

  (A) In a moving image, when the estimation target frame image (reference image) and other frame images (reference images) have the same feature points, that is, there are few corresponding points, camera posture estimation tends to fail. is there. Of course, if the number of corresponding points necessary for camera posture estimation is not satisfied, camera posture estimation itself cannot be performed.

  (B) The corresponding points are obtained from the similarity of the feature vectors of the feature points, but are not necessarily correct and include incorrect corresponding points. In the estimation target frame image, when the error rate at corresponding points, that is, the ratio of erroneous correspondence is large, the estimation accuracy is degraded or the estimation is broken.

(C) By the optimization, the posture information and the three-dimensional position of the corresponding point are obtained. Using the obtained posture information and lens information, the three-dimensional coordinates (spatial position) of the obtained corresponding points are reprojected on the imaging surface (reference image 90) of the photographing camera C. The two-dimensional coordinates (reprojection position) of the corresponding points and the two-dimensional coordinates (position in the image) obtained by the image analysis process are originally the same position. This is because the optimization process uses the sum of the distances of both positions as the cost function for the reprojection position where the corresponding point is reprojected on the imaging surface and the corresponding point obtained by the image analysis process. This is because of the processing.
The reprojection method will be described later.

Therefore, when there is an erroneous correspondence, as shown in FIG. 4, the corresponding point 92 ₁ is the same as the position 92 ₁ ′ where the corresponding point 92 ₁ is reprojected on the imaging surface (reference image 90) of the photographing camera C. It will not be in position. In other words, re-projection error Δ1 of corresponding points 92 ₁ indicates the misalignment between the corresponding points 92 ₁ and reprojection position 92 ₁ '.
Since the corresponding points 92 ₂ and 92 ₃ are the same as the corresponding point 92 ₁ , the description thereof is omitted.

  In general, a conventional average of reprojection errors is used as a scale for determining whether or not the estimation is successful. With respect to the total average of the reprojection errors, if the corresponding points are uniformly distributed in the frame image and the reprojection error at each corresponding point is uniformly reduced, the estimation accuracy can be correctly evaluated. However, when the reprojection error becomes large, there is a high possibility that the estimation accuracy is lowered or the estimation is broken. In the optimization process, since the camera posture is statistically estimated, for example, when the corresponding points are unevenly distributed, the unevenly distributed portion is not appropriately optimized and converges to a value deviating from the true value as a whole. . In this case, since the number of corresponding points having a large reprojection error is small, it does not contribute much to the optimization, or is excluded as an outlier. Therefore, by evaluating whether or not the reprojection error is uneven for each block of the reference image, it is possible to determine whether or not the estimation accuracy is low even if the reprojection error is within a certain range.

  (D) When the corresponding points are unevenly distributed in the frame image to be estimated, overfitting (overlearning) is performed at the unevenly distributed portion, and the estimation accuracy decreases.

  Here, it is possible to determine (A) without using the evaluation value C. Regarding (B), it is difficult to determine whether or not it is a mishandling, but it can be included in the evaluation of (C) and (D). Therefore, from the knowledge of (C) and (D), the evaluation value C can be obtained as follows.

Inhomogeneity D, as shown in equation (1), the maximum _{E ReproMax} reprojection error average value, expressed as the ratio between the minimum _{E ReproMin} reprojection error average value.

In this case, the non-uniformity calculation means 31 divides the reference image 90 into four blocks as shown by the broken lines in FIG. 5, and obtains a reprojection error at each corresponding point. Then, the nonuniformity calculation means 31 averages the reprojection errors of the corresponding points included in each block, and obtains a reprojection error average value for each block. Furthermore, the non-uniformity calculation means 31 uses the equation (1), where E _reproMax is the reprojection error average value of the block that is the maximum error, and E _reproMin is the reprojection error average value of the block that is the minimum error, The non-uniformity D is calculated.

As shown in the equation (2), the uneven distribution degree S is _obtained by adding an addition value of the minimum number of corresponding points N _{Min in the} block and the second smallest number of corresponding points N _{Min2nd in} the block. It is defined by the formula and dividing by the sum of the number N _i of the point.

In this case, the uneven distribution degree calculating means 33 divides the reference image 90 into four blocks as shown in FIG. 5 and counts the number of corresponding points included in each block. The uneven distribution calculation means 33, the corresponding number of blocks corresponding points is minimized and N _Min, the corresponding number of blocks corresponding points is small the second as N _Min2nd, using equation (2), uneven distribution of S is calculated.

  The evaluation value C is represented by the ratio between the uneven distribution degree S and the nonuniformity degree D, as shown in Expression (3). The evaluation value C indicates that the estimation accuracy is higher as the value is larger, and the estimation accuracy is lower as the value is smaller.

The evaluation value determination means 35 calculates an evaluation value C from each reference image, and determines whether or not the evaluation value C is less than the threshold value _Cth using the equation (4). Specifically, the evaluation value determination unit 35 determines that the reference image belongs to the high-accuracy section when Expression (4) is not satisfied. On the other hand, the evaluation value determination means 35 determines that this reference image belongs to a section other than the high-precision section (either the medium-precision section or the low-precision section) when Expression (4) is satisfied.

  As described above, when the number of corresponding points is small, the estimation accuracy is lowered or the estimation is broken. Further, even if a certain amount of corresponding points remain at the time of the corresponding point search process, the number of corresponding points may be reduced by the outlier exclusion process. For this reason, the evaluation value determination means 35 determines whether or not re-estimation as a medium accuracy interval can be performed for the interval determined to belong to other than the high accuracy interval, using the number of corresponding points after the optimization process.

Here, let us consider a case where the corresponding points remaining after the optimization processing are included in the reference image in the high accuracy section and the reference image other than the high accuracy section. In this case, the number of corresponding points and P _C included in the reference image other than the high-precision interval. The evaluation value determining means 35, as shown in Equation (5), a decision is made as to whether the corresponding points P _C exceeds the threshold value P _th.

  The evaluation value determination unit 35 determines that the reference image belongs to the medium accuracy section when Expression (5) is satisfied. On the other hand, the evaluation value determination unit 35 determines that the reference image belongs to the low-precision section when Expression (5) is not satisfied.

Thereafter, the estimation result determination unit 30 outputs posture information to the first interpolation processing unit 40 for the high-precision section. Further, the estimation result determination unit 30 outputs posture information, sensor information, lens information, and corresponding point information to the re-estimation unit 50 for the medium accuracy interval. Further, the estimation result determination unit 30 outputs the sensor information to the second interpolation processing unit 60 for the low accuracy section.
Note that the corresponding point information indicates two-dimensional coordinates of corresponding points in the reference image and three-dimensional coordinates of corresponding points on the subject.

The threshold value C _th and the threshold value P _th are preferably set manually based on empirically obtained values, and are preferably adjusted according to the size and quality of the captured video. In this embodiment, for example, the threshold value C _th = 0.05 and the threshold value P _th = 100 are set.

<Reprojection method>
Hereinafter, the reprojection method will be described.
Projective transformation for performing reprojection can be defined by equations (6) and (7).

In the equations (6) and (7), the three-dimensional coordinates of the corresponding points are (X _world , Y _world , Z _world ), the rotation matrix of the imaging camera C is R, the parallel progression sequence of the imaging camera C is T, and the imaging camera The center of the imaging surface of C is (o _x , o _y ), the pixel size of the photographing camera C is (k _x , k _y ), the focal length of the lens of the photographing camera C is f, and the two-dimensional coordinates of the photographing surface of the photographing camera C Let (x _image , y _image ) be

Furthermore, since the lens of the photographing camera C is distorted, the distortion is reflected in the two-dimensional coordinates (x _image , y _image ) of the imaging surface of the photographic camera C using the equations (8) and (9). Let

In the equations (8) and (9), the two-dimensional coordinates of the imaging surface after distortion correction are (x _u , _yu ), the two-dimensional coordinates of the imaging surface before distortion correction are (x _d , y _d ), The center of distortion that is the lens center of the photographing camera C is (x _c , y _c ), the n-th order bending distortion coefficient is K _n , the n-th order tangential distortion coefficient is P _n , and the lens center (x _c , y _{c) of the} photographing camera C. ) To the two-dimensional coordinates (x _d , y _d ).

That is, the two-dimensional coordinates (x _image , y _image ) of the imaging surface represented by Expression (7) are the two-dimensional coordinates (x _d , y _d ) of the imaging surface represented by Expression (8) and Expression (9). ). In the present embodiment, only the primary bending distortion coefficient K ₁ and the secondary bending distortion coefficient K ₂ are considered (n = 1, 2), and the lens center (x _c , y _c ) and tangential distortion are considered. Not.

In addition, since the method of reflecting distortion is described in the following reference, further description is abbreviate | omitted.
Reference "de Villiers, JP; Leuschner, FW; Geldenhuys, R. (17-19 November 2008)." Centi-pixel accurate real-time inverse distortion correction ". 2008 International Symposium on Optomechatronic Technologies. SPIE. Doi: 10.1117 / 12.804771 "

In addition, this invention is not limited to the said lens model, You may apply another lens model.
In the present invention, re-projection may be performed without taking lens distortion into consideration.

Returning to FIG. 1, the description of the configuration of the camera posture estimation apparatus 1 will be continued.
The first interpolation processing unit 40 outputs the posture information from the estimation result determining unit 30 as the posture estimation result of the photographing camera C in the high accuracy section. That is, in the high accuracy section, the posture information obtained by the image analysis process is output as it is.

  The re-estimating unit 50 re-estimates the posture of the photographing camera C using the posture information, the lens information, and the corresponding point information by an optimization process (for example, bundle adjustment) using the sensor information as a constraint condition. Is. Then, the re-estimator 50 outputs the re-estimated posture of the photographing camera C as a posture estimation result of the medium accuracy section.

<Re-estimation by re-estimation means>
With reference to FIG. 6, the re-estimation by the re-estimator 50 will be described (see FIG. 1 as appropriate).
Consider a case where the corresponding points remaining after the optimization processing are included in the frame image belonging to the medium accuracy interval and the frame image belonging to the high accuracy interval. Corresponding points included in the frame image of this high-accuracy section may be referred to as “residual corresponding points”.

  The re-estimating means 50 performs re-estimation using the corresponding point information in addition to the posture information and the sensor information in the medium accuracy section. The medium accuracy interval is a state in which the optimization process has converged to a value other than the true value or the camera posture estimation has failed. For this reason, it is difficult to estimate the camera posture with high accuracy even if the two-dimensional coordinates of the corresponding points included in the frame image in the medium accuracy section are used. On the other hand, corresponding points (remaining corresponding points) remaining after the optimization processing in the high-precision section have three-dimensional coordinates. Accordingly, in the medium accuracy section, the corresponding point having a corresponding relationship with the remaining corresponding point can be associated with the three-dimensional coordinate. Further, rotation information indicating the direction of the photographing camera C can be used in the sensor information.

First, the re-estimating means 50 can obtain the three-dimensional coordinates of the remaining corresponding points corresponding to the corresponding points of the frame image belonging to the medium accuracy section from the corresponding point information. Further, the re-estimation unit 50 extracts rotation information from the sensor information. At this time, the re-estimation means 50 performs a rotation information connection process described later so that the rotation information matches at the connection point.
The connection point is a portion where the frame image belonging to the high accuracy section and the frame image belonging to the medium accuracy section or the low accuracy section are in contact with each other in the time direction.

Next, the re-estimating means 50 uses the rotation information and lens information as constraint conditions, and the distance between the position where the three-dimensional coordinates of each corresponding point are reprojected on the imaging surface and the two-dimensional position of the corresponding point obtained by image analysis. The optimization process is performed so that the total average value of is minimized. Specifically, the re-estimation unit 50 projects a corresponding point 92 of the subject Obj represented by three-dimensional coordinates on the imaging surface 93 of the imaging camera C as shown in FIG. At this time, the re-estimation unit 50 directs the imaging surface 93 in the direction indicated by the rotation information. Then, the re-estimation unit 50 obtains an extension line 94 between the corresponding point 92 of the subject Obj and the feature point 92 projected on the imaging surface 93. Furthermore, the re-estimating unit 50 optimizes the position of the photographing camera C (lens optical principal point) indicated by the posture information so that all the extended lines 94 converge on the optical principal point of the photographing camera C.
In FIG. 6, only a part of the reference numerals are shown for easy viewing of the drawing.
Further, the imaging surface 93 of the photographing camera C is a frame image (reference image 90) that constitutes a photographed video.

Returning to FIG. 1, the description of the configuration of the camera posture estimation apparatus 1 will be continued.
The second interpolation processing means 60 outputs the sensor information from the estimation result determination means 30 as the posture estimation result of the photographing camera C in the low accuracy section.
Here, posture information cannot be used in a low-precision section. Therefore, the second interpolation processing means 60 extracts rotation information and position information from the sensor information. Further, the second interpolation processing means 60 performs rotation information connection processing and position information connection processing on the connection points of the low accuracy section, the medium accuracy section, and the high accuracy section, and performs interpolation.

<Rotation information connection processing>
The rotation information connection process performed by the re-estimator 50 will be described with reference to FIGS. 7 and 8 (see FIG. 1 as appropriate).

  As shown in FIG. 7, the world coordinate system α of the sensor information measured by the posture measuring means Ca and the world coordinate system β of the posture information obtained by the image analysis process are different. For this reason, the re-estimation means 50 performs a connection process for matching the world coordinate systems α and β.

Here, in order to suppress the drift component of the sensor information, the connection process does not target the entire captured video, but targets a short section (time) around the connection point.
Specifically, the re-estimation unit 50 matches the rotation matrix R _s included in the sensor information with reference to the rotation matrix R _i included in the posture information, as shown in Expression (10). Namely, re-estimating means 50, and the rotation matrix _{R s,} the inverse matrix _R ^{s -1} of rotation matrix _{R s,} multiplying the rotation matrix _{R i.}

As shown in FIG. 8, there is a possibility that a drift component error has occurred on the front side F and the rear side B of the medium accuracy interval 95 (reference A in FIG. 8). Therefore, the re-estimation unit 50 matches the rotation matrices R _i and R _s with the inverse matrix R _s ^{−1 on} the front side F of the medium accuracy interval 95. Further, the re-estimation means 50 represents the error component of the rotation matrix R _s generated when the rear side B of the medium accuracy interval 95 is matched with a quaternion, and uses the interframe distance to calculate the error of the rotation matrix R _s . Distribute the components linearly. As a result, the re-estimation means 50 can reduce the shock of the rotation component at the connection point between the medium accuracy section 95 and the high accuracy section 96 and prevent the rotation information from changing suddenly (reference A ′ in FIG. 8). ).
Note that the second interpolation processing unit 60 performs the rotation information connection process at the connection point between the low-accuracy section and the medium-accuracy section or the high-accuracy section, as in the re-estimation section 50, and thus the description thereof is omitted.

<Location information connection processing>
The position information connection process performed by the second interpolation processing means 60 will be described with reference to FIGS. 9 and 10 (see FIG. 1 as appropriate).

Before and after this low accuracy section, the translation component has a larger error than the rotation component.
First, the second interpolation processing unit 60 matches the position of the photographing camera C included in the posture information with the position of the photographing camera C ′ included in the sensor information by parallel movement on the front side of the low-accuracy section. Specifically, as shown in FIG. 9, the second interpolation processing unit 60 translates the position of the photographing camera C ′ indicated by the sensor information to the position of the photographing camera C indicated by the position information, so that the photographing camera C , C ′ are matched.

  Next, the second interpolation processing means 60 performs either spatial shearing processing or scaling processing on the rear side of the low-accuracy section, and the position information included in the posture information and the position included in the sensor information. Match information. Thereby, the second interpolation processing means 60 can reduce the shock of the translation component at the connection point and prevent the position information from changing suddenly.

  Spatial shearing is to perform affine transformation shearing on the X, Y, and Z axes in a three-dimensional space. For example, in the case of shear in the X-axis direction, as shown in FIG. 10A, the displacement in the X-axis direction is proportional to the Y coordinate value. In shearing in the X-axis direction, as shown in FIG. 10B, the displacement in the X-axis direction may be proportional to the Z coordinate value. In FIG. 10, the affine transformation is shown by a solid line, and the affine transformation is shown by a broken line.

The scaling process is a process of performing rotation and scaling so that the front side of the low precision section is the rotation axis and the rear side of the low precision section is coincident.
In the present embodiment, the second interpolation processing unit 60 shears the space.

[Operation of camera posture estimation device]
The operation of the camera posture estimation device 1 will be described with reference to FIG. 11 (see FIG. 1 as appropriate).
Here, it is assumed that the storage unit 10 stores captured images, sensor information, and lens information.

The camera posture estimation apparatus 1 obtains posture information by performing image analysis processing on the captured video stored in the storage unit 10 by the image analysis unit 20 (step S1).
In the camera posture estimation apparatus 1, the nonuniformity calculation unit 31 calculates the nonuniformity D using the above-described equation (1) (step S2).
The camera posture estimation device 1 calculates the uneven distribution degree S using the above-described equation (2) by the uneven distribution degree calculation means 33 (step S3).

The camera posture estimation device 1 calculates the evaluation value C by the evaluation value determination means 35 using the above-described equation (3) (step S4).
The camera posture estimation apparatus 1 determines whether or not the evaluation value C is less than the threshold value _Cth by the evaluation value determination unit 35 (step S5).
If the evaluation value C is less than the threshold _{C th} (Yes in step S5), and the camera posture estimation apparatus 1, the evaluation value determination unit 35 determines whether the corresponding points _{P C} exceeds the threshold value _{P th} (step S6 ).

If the corresponding points P _C exceeds the threshold value P _th (Yes in step S6), and camera posture estimation apparatus 1, the re-estimation means 50, the optimization process as a constraint sensor information, and pose information, lens information Then, re-estimation is performed using the corresponding point information (step S7).

Corresponding points P if _C is less than the threshold value P _th (No in step S6), and the camera posture estimation apparatus 1, by the second interpolation processing unit 60, as the posture estimation result of the photographing camera C in low precision interval, outputs the sensor information (Step S8).

When the evaluation value C is equal to or greater than the threshold value _Cth (No in step S5), the camera posture estimation device 1 outputs posture information as the posture estimation result of the photographing camera C in the high accuracy section (step S9).

As described above, the camera posture estimation apparatus 1 uses the equations (1) to (5) even if the corresponding points are unevenly distributed in the captured video and the reprojection error of each feature point is not uniform. The degree of uniformity D and the degree of uneven distribution S can be accurately calculated. For this reason, the camera posture estimation apparatus 1 can correctly evaluate the posture estimation accuracy of the photographing camera C and estimate the posture of the photographing camera with high accuracy.
Furthermore, since the camera posture estimation apparatus 1 performs re-estimation by the re-estimation unit 50 so that the posture of the photographing camera C does not change suddenly at the connection point, a smooth CG image can be produced.

  The camera posture estimation apparatus 1 can be realized by hardware using discrete logic, FPGA (Field Programmable Gate Array), or the like. The camera posture estimation apparatus 1 can also be realized by software (program) that causes a general computer to cooperate with each other as the above-described means.

(Modification)
In addition, this invention is not limited to above-described embodiment, A deformation | transformation can be added in the range which does not deviate from the meaning.
The non-uniformity calculating unit 31 and the uneven distribution calculating unit 33 may divide the reference image 90 into blocks other than four.
Further, the non-uniformity calculating unit 31 and the uneven distribution calculating unit 33 may divide the reference image 90 into edge regions or color regions instead of dividing the reference image 90 into rectangular blocks.

Hereinafter, a method for calculating the evaluation value C will be described for the case where the reference image 90 is divided into other than four blocks (see FIG. 1 as appropriate).
Basically, the estimation result determination unit 30 calculates the evaluation value C in consideration of the variance and correlation, and the reprojection error.

First, the non-uniformity calculation means 31 extracts corresponding points where the reprojection error is below a certain level. Further, the non-uniformity calculating unit 31 calculates the variance (position on the image) of corresponding points in the reference image. This variance is not the covariance of the two-dimensional coordinate system but the variance with respect to the distance from the center of gravity of the corresponding point. As a result, the non-uniformity calculation means 31 determines whether or not corresponding points with small reprojection errors are concentrated at a specific location in the reference image (the square of the non-uniformity D), as shown in Equation (11). ) Can be obtained.
In this equation (11), the barycentric position of the corresponding point is x￣, the number of corresponding points is n, and the distance between the corresponding points is x _j .

The uneven distribution degree calculating means 33 calculates the correlation between corresponding points in the reference image as shown in the equation (12). Thereby, the uneven distribution degree calculating means 33 can obtain the determination index (the uneven distribution degree S) of the uneven distribution tendency of the corresponding points having a small reprojection error.
In this equation (12), the two-dimensional coordinates of the centroid position of the corresponding point are (x￣, y￣), and the two-dimensional coordinates of the corresponding point are (x _j , y _j ).

  In the example of FIG. 12, the closer the uneven distribution degree S is to “0”, the more uneven the corresponding points are, and the closer the uneven distribution degree S is to “1”, the more uneven the corresponding points are. In the example of FIG. 12, the distribution of corresponding points in the reference image is illustrated by dots, and the degree of unevenness S is illustrated at the top of the reference image. That is, FIG. 12 illustrates 20 patterns of the relationship between the uneven distribution degree S and the distribution of corresponding points.

  Here, consider a case where the reference image is divided into other than four blocks. In this case, the evaluation value determination means 35 can use a technique for dividing blocks and a technique for not dividing blocks.

  When the block is divided, the evaluation value determination unit 35 counts the number of corresponding points with a reprojection error equal to or less than a certain value for each block. Then, the evaluation value determining means 35 sets the flag of the block having the corresponding point number equal to or greater than “1” and the flag of the block having the corresponding point number less than the constant to “0”. Further, the evaluation value determination means 35 applies the calculation method (replace corresponding points to the coordinates of one block) of the above-described equations (11) and (12) for the block whose flag is set to “1”. Thus, the evaluation value C can be calculated using (3) described above.

  On the other hand, the evaluation value determination unit 35 uses the coordinates of the corresponding points when the corresponding points having a reprojection error of a certain value or less are used in units of points without dividing the blocks, using the coordinates of the corresponding points. By applying the calculation method (12) (replace corresponding points with the coordinates of one block), the evaluation value C can be calculated using (3) described above.

  The present invention can be used for video production such as CG synthesis and augmented reality (AR).

DESCRIPTION OF SYMBOLS 1 Camera attitude | position estimation apparatus 10 Memory | storage means 20 Image analysis means (camera attitude | position estimation means)
30 Estimation result determining means 31 Non-uniformity calculating means 33 Unevenness calculating means 35 Evaluation value determining means 40 First interpolation processing means (high-precision section estimating means)
50 Re-estimation means (medium precision interval estimation means)
60 Second interpolation processing means (low precision interval estimation means)
100 camera posture estimation system C photographing camera Ca posture measuring means Cb lens state measuring means

Claims (4)

A camera posture estimation device that estimates the posture of the photographing camera using a photographed image photographed by the photographing camera and camera posture measurement information obtained by posture measurement means mounted on the photographing camera measuring the posture of the photographing camera. Because
Extracting feature points from frame images constituting the photographed video, searching for corresponding points in which the same feature points are associated between different frame images, and performing the outlier removal processing and optimization processing, the photographing camera Camera posture estimation information for estimating the posture of the camera, and camera posture estimation means for obtaining a spatial position of the corresponding point included in the frame image;
For each divided region of the frame image, using the camera posture estimation information, a reprojection position obtained by reprojecting the corresponding point represented by the spatial position on the frame image, and a corresponding point searched by the camera posture estimation unit A non-uniformity calculating means for calculating a re-projection error average value with a position in the image and calculating a non-uniformity that is a maximum and minimum ratio of the re-projection error average value;
An uneven distribution degree calculating means for calculating an uneven distribution degree of the corresponding points based on the number of corresponding points included in each divided region of the frame image;
Based on an evaluation value that is a ratio between the degree of uneven distribution and the non-uniformity, a high-accuracy section in which the estimation accuracy of the camera posture estimation information is high for the captured video, and the estimation accuracy is lower than the high-precision section Evaluation value determination means for determining a medium accuracy interval and a low accuracy interval in which the estimation accuracy is lower than the medium accuracy interval;
High-precision section estimation means for outputting the camera posture estimation information as a posture estimation result of the photographing camera in the high-precision section;
Medium accuracy that re-estimates the posture of the photographing camera using the camera posture estimation information and outputs it as a posture estimation result of the photographing camera in the medium accuracy section by an optimization process using the camera posture measurement information as a constraint condition Interval estimation means;
As a posture estimation result of the photographing camera in the low accuracy section, a low accuracy section estimation means for outputting the camera posture measurement information,
A camera posture estimation apparatus comprising: The non-uniformity calculating means calculates a reprojection error average value for each of the four divided areas, and represents the maximum reprojection error average value E _reproMax and the reprojection error average as shown in Equation (1). Calculating the non-uniformity D, which is the ratio of the value to the minimum E _reproMin ,
The uneven distribution degree calculating means calculates the number of corresponding points for each of the four divided areas, and, as shown in Expression (2), the minimum number of corresponding points N _{Min in the} divided areas and the number of the divided areas The uneven distribution degree S is calculated by dividing the addition value of the second smallest number of corresponding points N _{Min2nd by} the sum of the number N _i of corresponding points for each of the divided regions. The camera posture estimation apparatus according to 1.

The medium accuracy section estimation means is an optimization process using the camera posture measurement information as a constraint condition.
Extracting rotation information indicating the direction of the photographing camera from the camera posture measurement information,
Finding the spatial position of the corresponding point included in the frame image of the high-precision section corresponding to the corresponding point included in the frame image of the medium-precision section,
Projecting the corresponding point represented by the spatial position onto the imaging surface of the imaging camera pointing in the direction indicated by the rotation information,
The position of the photographing camera indicated by the camera posture estimation information is optimized so that an extension line between the corresponding point represented by the spatial position and the corresponding point projected on the imaging surface converges on the optical principal point of the photographing camera. The camera posture estimation apparatus according to claim 1 or 2, characterized by comprising:   A camera posture estimation program for causing a computer to function as the camera posture estimation device according to claim 1.

Images