JP2016003930A - Image processing apparatus, image processing method, and image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing apparatus configured to easily estimate a camera parameter.SOLUTION: An image processing apparatus configured to estimate a camera parameter of a camera which has captured refocus-able images from the two refocus-able images having an overlapping area and variable focal depth includes: input means which inputs the two refocus-able images, a refocused-image division number which is the number of divisions of an image having a predetermined focal depth, and an internal parameter of the camera; image processing means which detects features points of refocused images and a feature-point corresponding group between the refocused images from the two refocus-able images, and outputs them; and geometric processing means which estimates a camera parameter from the feature-point corresponding group and the positions of the feature points.

Description

  The present invention relates to an image processing device, an image processing method, and an image processing program that are used for highly accurate estimation of a three-dimensional shape of a subject and generation of a high-quality virtual viewpoint video.

  With the widespread use of light field cameras and the like, there is an increased chance that a focus-able image (an image with variable depth of focus) is shot in various everyday scenes, and the same subject is shot from multiple viewpoints. It is not unusual. Images taken from a plurality of viewpoints can be used in various applications such as three-dimensional measurement of an object and image generation at a virtual viewpoint. In general, in order to use images captured from a plurality of viewpoints for these applications, it is necessary to estimate the camera parameters of the camera that captured each image.

  A technique called Structure from Motion (SfM) that estimates a camera parameter using a plurality of images having overlapping regions as an input has been proposed (for example, see Non-Patent Document 1). These methods take correspondence between feature points detected from images among a plurality of images, and obtain camera parameters from the correspondence relationship. It is known that the camera parameter estimation accuracy largely depends on the feature point correspondence accuracy. One of the causes affecting the accuracy of the feature point correspondence is that the depths in focus are different in a plurality of images. This is because the position and feature amount of the detected feature point are different between when the focus is achieved and when the focus is not achieved.

Noah Snavely, Steven M Seitz, Richard Szeliski, “Photo Tourism: Exploring image collections in 3D”, ACM Transaction on Graphics (Proceedings of SIGGRAPH 2006), 2006.

  By the way, since a plurality of focused images (images focused at a certain depth) can be generated from a single focused-able image, when performing SfM using two focused-able images, It is necessary to consider all the combinations of focused images in the focused-able image.

  However, many of these combinations are focused images whose depths do not match, and there is a problem that it is difficult to automatically select the result of performing SfM on a combination of focused images having appropriate depths.

  The present invention has been made in view of such circumstances, and an object thereof is to provide an image processing apparatus, an image processing method, and an image processing program capable of easily estimating camera parameters.

  The present invention is an image processing apparatus for estimating camera parameters of a camera that has photographed each of the focus-able images from two focus-ble images that are overlapping images having overlapping regions and variable in-focus depths. Input means for inputting the two refocus-able images, the number of refocused image divisions that are the number of divisions of an image focused on a predetermined depth, and the internal parameters of the camera, and the two refocus-ables image processing means for detecting and outputting each feature point of the focused image and the feature point correspondence group between the focused images from the image; and a geometry for estimating the camera parameters from the feature point correspondence group and the position of the feature point processing Characterized by comprising a stage.

  According to the present invention, the image processing unit inputs two pieces of the referenced-able image and the number of pieces of the referenced image, and creates the referenced image according to the number of pieces of the referenced image from the reference-able image. A means for inputting each of the focused images, detecting feature points of each focused image, a means for inputting the feature points of the focused image, obtaining a combination of all the feature points, and a combination of the feature points Means for obtaining a feature point correspondence group based on the feature amount of each detected feature point, and inputting the obtained feature points and the correspondence of the feature points, and epipolar constraints on the correspondence of the feature points R using And a means for performing processing by the ANSAC method and outputting a combination of feature point correspondence groups and feature point groups as inliers.

  In the present invention, the image processing means inputs a feature point correspondence group that becomes an inlier in all combinations, and outputs a combination of feature points that has the largest number of feature point correspondences included and a feature point correspondence group that becomes an inlier. It is characterized by doing.

  According to the present invention, the geometric processing means inputs the feature point correspondence group and the feature points, calculates a basic matrix, and inputs the basic matrix and the internal parameters of the camera, and calculates the basic matrix A means for inputting the basic matrix and outputting a rotation matrix and a translation vector; and inputting the rotation matrix, the translation vector, the internal parameter, the feature point, and the feature point correspondence group, and minimizing a reprojection error. And a means for outputting the rotation matrix, the translation vector, the optimal solution of the internal parameters, the number of iterations in the optimization, and the reprojection error.

  According to the present invention, the geometric processing means inputs bundle adjustment results for all combinations, and outputs bundle adjustment results for a combination of feature points having the smallest number of iterations.

  In the present invention, the geometric processing means inputs the result of bundle adjustment performed for all combinations of feature points for all combinations, and bundles for combinations of feature points having the smallest reprojection error. The result of the adjustment is output.

  The present invention is an image processing method for estimating camera parameters of a camera that has photographed each of the focus-able images from two focus-ble images that are images with variable overlapping depths that have overlapping areas. An input step for inputting the two refocus-able images, the number of the refocused image that is the number of divisions of the image focused on a predetermined depth, and the internal parameters of the camera, and the two refocus-ables An image processing step of detecting and outputting each feature point of the focused image and a feature point correspondence group between the focused images from the image, and estimating the camera parameter from the position of the feature point correspondence group and the feature point And having a what processing steps.

  The present invention is an image processing program for causing a computer to function as the image processing apparatus.

  According to the present invention, when calibrating the camera that acquired each image using the two refocus-able images, the camera parameter estimation result using the refocused image at an appropriate in-focus position is automatically calculated. The effect that it becomes possible to output to is obtained.

It is a block diagram which shows the structure of 1st Embodiment of this invention. It is a flowchart which shows operation | movement of the image processing apparatus by 1st Embodiment. It is a flowchart which shows operation | movement of the image processing apparatus by 2nd Embodiment. It is a flowchart which shows operation | movement of the image processing apparatus by 3rd Embodiment.

<First Embodiment>
Hereinafter, an image processing apparatus according to a first embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing the configuration of the embodiment. In this figure, reference numeral 1 denotes an input unit for inputting the number of the focused-able image and the focused image and the internal parameter of the camera. Reference numeral 2 denotes an image processing unit that receives two reflected-able images as input and outputs a feature point of each focused image (an image focused on a certain depth) and a feature point correspondence group between the focused images. Reference numeral 3 denotes a geometric processing unit that estimates and outputs a camera parameter from the feature point correspondence group and the position of the feature point.

  The feature point correspondence group between the refocused images is all feature point correspondences obtained in a certain pair of feature points. The feature point group refers to all feature points detected from one image. Also, the feature point correspondence represents the relationship between the feature point in one image and which feature point in the other image. The image processing apparatus shown in FIG. 1 automatically outputs a plurality of feature point correspondence group candidates and camera parameter candidates when using a focused image at an appropriate in-focus position.

  Next, the operation of the image processing apparatus shown in FIG. 1 will be described with reference to FIG. FIG. 2 is a flowchart showing the operation of the image processing apparatus shown in FIG. The image processing apparatus shown in FIG. 1 inputs the focused image (two images), the number of divisions of the focused image, the internal parameters of the camera that captured each focused image, and outputs the external parameters (rotation matrix, translation vector) and the internal parameters. To do.

  First, the input unit 1 inputs two references-able images (step S1). Subsequently, the input unit 1 inputs camera internal parameters (step S2). Then, the input unit 1 inputs the number of refocused images to be created (step S3).

  Next, the image processing unit 2 creates the number of refocused images input in step S3 from the refocus-able image input in step S1 (step S4).

  Next, the image processing unit 2 detects a feature point (the position of the feature point and the feature amount) for the focused image created in step S4 (step S5).

  Next, the image processing unit 2 prepares all combinations of the feature point group (all feature points detected from a single focused image) (step S6). However, one of the combinations is a feature point group of the focused image created from the first focused image, and the other is a group of feature points of the focused image created from the second focused-able image. .

  Next, the image processing unit 2 obtains feature point correspondence groups for all combinations of feature point groups (step S7).

  Next, the geometric processing unit 3 performs RANSAC (RANDdom Sample Consensus) on the feature point correspondence group obtained in step S7 to obtain a feature point correspondence group to be an inlier (step S8). Then, the geometric processing unit 3 outputs the one corresponding to the largest number of feature points among the results of step S8 (step S9).

  Next, the geometric processing unit 3 obtains a basic matrix from the feature point correspondence group that is an inlier obtained in step S9 (step S10).

  Next, the geometric processing unit 3 obtains a basic matrix from the internal parameters input in step S2 and the basic matrix obtained in step S10 (step S11).

  Next, the geometric processing unit 3 calculates a rotation matrix and a translation vector from the basic matrix obtained in step S11 (step S12).

  Next, the geometric processing unit 3 uses the rotation matrix obtained in step S12, the translation vector, and the internal parameters input in step S2 as initial values, and bundles using the feature point correspondence group that is an inlier obtained in step S9. Adjustment is performed to obtain an optimal solution of the rotation matrix, translation vector, and internal parameters (step S13).

  Finally, the geometric processing unit 3 outputs the result obtained in step S13 (step S14).

  As described above, it is possible to automatically output a plurality of feature point correspondence group candidates and camera parameter candidates when a focused image at an appropriate focus position is used.

Second Embodiment
Next, an image processing apparatus according to a second embodiment of the present invention will be described. Since the device configuration in the second embodiment is the same as the configuration shown in FIG. 1, detailed description thereof is omitted here.

  Next, the operation of the image processing apparatus in the second embodiment will be described with reference to FIG. FIG. 3 is a flowchart showing the operation of the image processing apparatus according to the second embodiment. In this figure, the same operations as those shown in FIG. Similar to the image processing apparatus in the first embodiment, the image processing apparatus in the second embodiment inputs the refocused image (2 images), the number of divisions of the refocused image, and the internal parameters of the camera that has captured each refocused image. Output parameters (rotation matrix, translation vector) and internal parameters.

  First, the input unit 1 inputs two references-able images (step S1). Subsequently, the input unit 1 inputs camera internal parameters (step S2). Then, the input unit 1 inputs the number of refocused images to be created (step S3).

  Next, the image processing unit 2 creates the number of refocused images input in step S3 from the refocus-able image input in step S1 (step S4).

  Next, the image processing unit 2 detects a feature point (the position of the feature point and the feature amount) for the focused image created in step S4 (step S5).

  Next, the image processing unit 2 prepares all combinations of the feature point group (all feature points detected from a single focused image) (step S6).

  Next, the image processing unit 2 selects one combination of feature points, and obtains a feature point correspondence group for this combination of feature points (step S15).

  Next, the geometric processing unit 3 performs RANSAC on the feature point correspondence group obtained in step S15 to obtain a feature point correspondence group to be an inlier (step S8).

  Next, the geometric processing unit 3 obtains a basic matrix from the feature point correspondence group that is an inlier obtained in step S8 (step S10).

  Next, the geometric processing unit 3 obtains a basic matrix from the internal parameters input in step S2 and the basic matrix obtained in step S10 (step S11).

  Next, the geometric processing unit 3 calculates a rotation matrix and a translation vector from the basic matrix obtained in step S11 (step S12).

  Next, the geometric processing unit 3 sets the rotation matrix obtained in step S12, the translation vector, and the internal parameters input in step S2 as initial values, and bundles using the feature point correspondence group that is an inlier obtained in step S8. Adjustment is performed to find the optimal solution for the rotation matrix, translation vector, and internal parameters (step S17).

  Next, the geometric processing unit 3 determines whether or not processing has been completed for all combinations of feature points, and if not, the process returns to step S15, where all combinations of feature points are prepared. A process is performed for this (step S18).

  Finally, the geometric processing unit 3 outputs a result with the smallest number of iterations of the obtained nonlinear optimization among the results of all combinations (step S19).

  As described above, it is possible to automatically output a plurality of feature point correspondence group candidates and camera parameter candidates when a focused image at an appropriate focus position is used.

<Third Embodiment>
Next, an image processing apparatus according to a third embodiment of the present invention will be described. Since the apparatus configuration in the third embodiment is the same as the configuration shown in FIG. 1, detailed description thereof is omitted here.

  Next, the operation of the image processing apparatus in the third embodiment will be described with reference to FIG. FIG. 4 is a flowchart illustrating the operation of the image processing apparatus according to the third embodiment. In this figure, the same operations as those shown in FIG. 3 are denoted by the same reference numerals, and description thereof will be briefly made. Similar to the image processing apparatuses in the first and second embodiments, the image processing apparatus according to the third embodiment inputs the number of refocused images (two images), the number of divisions of the refocused image, and the internal parameters of the camera that has captured each refocused image. External parameters (rotation matrix, translation vector) and internal parameters are output.

  First, the input unit 1 inputs two references-able images (step S1). Subsequently, the input unit 1 inputs camera internal parameters (step S2). Then, the input unit 1 inputs the number of refocused images to be created (step S3).

  Next, the image processing unit 2 creates the number of refocused images input in step S3 from the refocus-able image input in step S1 (step S4).

  Next, the image processing unit 2 detects a feature point (the position of the feature point and the feature amount) for the focused image created in step S4 (step S5).

  Next, the image processing unit 2 prepares all combinations of the feature point group (all feature points detected from a single focused image) (step S6).

  Next, the image processing unit 2 selects one combination of feature points, and obtains a feature point correspondence group for this combination of feature points (step S15).

  Next, the geometric processing unit 3 performs RANSAC on the feature point correspondence group obtained in step S15 to obtain a feature point correspondence group to be an inlier (step S8).

  Next, the geometric processing unit 3 obtains a basic matrix from the feature point correspondence group that is an inlier obtained in step S8 (step S10).

  Next, the geometric processing unit 3 obtains a basic matrix from the internal parameters input in step S2 and the basic matrix obtained in step S10 (step S11).

  Next, the geometric processing unit 3 calculates a rotation matrix and a translation vector from the basic matrix obtained in step S11 (step S12).

  Next, the geometric processing unit 3 sets the rotation matrix obtained in step S12, the translation vector, and the internal parameters input in step S2 as initial values, and bundles using the feature point correspondence group that is an inlier obtained in step S8. Adjustment is performed to find the optimal solution for the rotation matrix, translation vector, and internal parameters (step S17).

  Next, the geometric processing unit 3 determines whether or not processing has been completed for all combinations of feature points, and if not, the process returns to step S15, where all combinations of feature points are prepared. A process is performed for this (step S18).

  Finally, the geometric processing unit 3 outputs a result having the smallest reprojection error after the obtained nonlinear optimization among the results of all combinations (step S20).

  As described above, it is possible to automatically output a plurality of feature point correspondence group candidates and camera parameter candidates when a focused image at an appropriate focus position is used.

Note that in step S1 described above, the two pieces of reference-able image to be input may be either (1) or (2) below.
(1) An image taken from two viewpoints by one camera.
(2) Images taken by two cameras from two viewpoints.
However, the two cameras do not necessarily have to be synchronized, but need to have overlapping areas.

In step S2 described above, the internal parameters of the camera to be input may be obtained using the internal parameters described in Reference Document 1.
Reference 1: Zhang, Zhengyou. "A flexible new technique for camera calibration." Pattern Analysis and Machine Intelligence, IEEE Transactions on 22.11 (2000): 1330-1334.

Further, in step S3 described above, the number of inputted focused images may be estimated by estimating the depth from the focused-able image and setting an appropriate number based on the depth value. As a technique for estimating the depth from the Focus-able image, a known technique (see, for example, Reference 2) is used.
Reference 2: Tao, Michael W., et al. "Depth from Combining Defocus and Correspondence Using Light-Field Cameras." ICCV, (2013).

  As a specific method of determining the number, for example, clustering is performed on the depth, and the number of clusters is set as the number of reflected images. In this case, however, the number of iterations of clustering and the threshold value must be determined in advance.

In step S4 described above, a known method (for example, see Reference 3) is used as a method for creating a focused image from a Focused-able image.
Reference 3: Ng, Ren, et al. "Light field photography with a hand-held plenoptic camera." Computer Science Technical Report CSTR 2.11 (2005).

However, the depth of the created focused image may be either (3) or (4) below.
(3) Step = (largest depth) / (number of focused images to be created), and a focused image is created at a depth that is an integral multiple of step from the closest optical center of the camera.
(4) Cluster the depth with k-means or the like (k is the number of focused images to be created), and create a focused image with the depth of the center of gravity of each cluster.

  Moreover, in step S5 mentioned above, well-known methods, such as SIFT, SURF, and CARD, can be used as a method of detecting a feature point.

  In step S7 described above, as a technique for associating feature points, a technique based on the similarity of feature quantities of feature points as proposed in SIFT or the like can be used.

  In step S8 described above, epipolar constraints may be used as a model used for RANSAC.

  Moreover, in step S10 mentioned above, as a method for obtaining the basic matrix, an eight-point algorithm may be used.

  Further, in Steps S13 and S17 described above, the Levenberg-Marquardt method may be used as a nonlinear optimization method used for bundle adjustment.

  As described above, when SfM is performed on two refocus-able images, camera parameters can be easily output by automatically outputting the result of applying SfM to a combination of refocused images having appropriate depths. Can be estimated. These can be used for highly accurate estimation of a three-dimensional shape of a subject and generation of a high-quality virtual viewpoint video.

  You may make it implement | achieve the image processing apparatus in embodiment mentioned above with a computer. In that case, a program for realizing this function may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be read into a computer system and executed. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” dynamically holds a program for a short time like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory inside a computer system serving as a server or a client in that case may be included and a program held for a certain period of time. Further, the program may be for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in the computer system. It may be realized using hardware such as PLD (Programmable Logic Device) or FPGA (Field Programmable Gate Array).

  As mentioned above, although embodiment of this invention has been described with reference to drawings, the said embodiment is only the illustration of this invention, and it is clear that this invention is not limited to the said embodiment. is there. Therefore, additions, omissions, substitutions, and other modifications of the components may be made without departing from the technical idea and scope of the present invention.

  When SfM is performed on two refocus-able images, it can be applied to an indispensable use in which it is indispensable to automatically output a result of applying SfM to a combination of refocused images having appropriate depths.

  DESCRIPTION OF SYMBOLS 1 ... Input part, 2 ... Image processing part, 3 ... Geometric processing part

Claims (8)

An image processing apparatus that estimates camera parameters of a camera that has captured each of the focus-able images from two focus-ble images that are images with varying depths to be focused and having overlapping regions.
Input means for inputting the two refocus-able images, the number of refocused image divisions which are the number of divisions of an image focused on a predetermined depth, and the internal parameters of the camera;
Image processing means for detecting and outputting each feature point of the focused image and a feature point correspondence group between the focused image from the two focused-able images;
An image processing apparatus comprising: a geometric processing unit that estimates the camera parameter from the feature point correspondence group and the position of the feature point. The image processing unit
Means for inputting the two pieces of the referenced-able image and the number of the focused image, and generating the focused image from the reflected-able image according to the number of the divided pieces of the recommended image;
Means for inputting each of the focused images and detecting a feature point of each focused image;
Means for inputting feature points of the focused image and obtaining a combination of all feature points;
Means for inputting a combination of the feature points and obtaining a feature point correspondence group based on the feature amount of each detected feature point;
Each of the obtained feature points and the correspondence between the feature points are inputted, the processing by the RANSAC method using the epipolar constraint is performed on the correspondence of the feature points, and the feature point correspondence group and the feature point group which are inliers are combined. The image processing apparatus according to claim 1, further comprising:   The image processing means inputs feature point correspondence groups that become inliers in all combinations, and outputs a combination of feature point groups with the largest number of feature point correspondences included and feature point correspondence groups that become inliers. The image processing apparatus according to claim 2. The geometric processing means includes:
Means for inputting the feature point correspondence group and the feature points and calculating a basic matrix;
Means for inputting the basic matrix and internal parameters of the camera and calculating the basic matrix;
Means for inputting the basic matrix and outputting a rotation matrix and a translation vector;
The rotation matrix, the translation vector, the internal parameter, the feature point, and the feature point correspondence group are input, and bundle adjustment is performed so that a reprojection error is minimized, and the rotation matrix, the translation vector, the internal point The image processing apparatus according to any one of claims 1 to 3, further comprising: an optimum parameter solution, an iteration count in optimization, and a means for outputting a reprojection error. The geometric processing means includes:
5. The image processing apparatus according to claim 4, wherein a bundle adjustment result for all combinations is input, and a bundle adjustment result for a combination of feature points having the smallest number of iterations is output. The geometric processing means includes:
Input the result of bundle adjustment performed for all combinations of feature points for all combinations, and output the result of bundle adjustment for the combination of feature points with the smallest reprojection error The image processing apparatus according to claim 4. An image processing method for estimating camera parameters of a camera that has photographed each of the focus-able images from two focus-ble images that are images with variable depths to be focused and having overlapping regions,
An input step for inputting the two refocus-able images, the number of the refocused image that is the number of divisions of the image focused on a predetermined depth, and the internal parameters of the camera;
An image processing step of detecting and outputting each of the feature points of the focused image and the feature point correspondence group between the focused images from the two focused-able images;
And a geometric processing step of estimating the camera parameter from the feature point correspondence group and the position of the feature point.   An image processing program for causing a computer to function as the image processing apparatus according to claim 1.

Images