JP2015179426A - Information processing apparatus, parameter determination method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To accurately calculate parameters for a transformation model for use in the alignment of images.SOLUTION: There is provided an information processing apparatus 10 comprising: a storage unit 11 for storing a first feature point q1 contained in a first image P1, a second feature point q2 contained in a second image P2, and a transformation model TM which transforms a point on the first image P1 to a corresponding point on the second image P2; and an arithmetic unit 12 for calculating an estimated value A1 of a parameter A contained in the transformation model TM using a set of the first and second feature points q1 and q2 greater in feature quantity than a set first threshold Th1, selecting a pair of the first feature point q1 and the second feature point q2 greater in feature quantity than a second threshold Th2 smaller than the first threshold Th1 and falling within a range set by an error of the transformation model TM to which the estimated value A1 is applied, calculating the parameter A and an evaluation value of the parameter A per selected pair, and determining the parameter A based on the evaluation value.

Description

  The present invention relates to an information processing apparatus, a parameter determination method, and a program.

  Examples of familiar portable devices that can be used as imaging devices include digital cameras, smartphones, and mobile phones. The processing capability of such an imaging device is improving year by year, and recently it has become possible to perform advanced image processing inside the imaging device. For example, an electronic camera shake correction function, an HDR (High Dynamic Range) imaging function, and the like are realized by advanced image processing on a plurality of images captured continuously.

  The electronic camera shake correction applied at the time of moving image shooting is a technique for displaying the subject in alignment so that the position of the subject does not shift between consecutive moving image frames. Electronic camera shake correction applied during still image shooting is a technique for aligning and synthesizing a plurality of images continuously shot at a high shutter speed. HDR photography is a technique for generating an image with less whiteout or blackout by combining a plurality of images taken with different exposures. In HDR shooting, the images are aligned.

  The alignment between images corresponds to a process of converting one image so that a plurality of feature points included in one image and a plurality of feature points included in the other image are best matched. For example, the translation movement, rotation, enlargement / reduction, etc. of an image caused by camera shake are expressed by conversion parameters of affine transformation, conversion parameters are determined using feature points of multiple images, and the entire image is converted by applying the conversion parameters. By doing so, alignment between images is realized.

  As a method for calculating the conversion parameter, for example, there is a RANSAC (RANdom SAmple Consensus) method. However, calculation of the conversion parameter by the RANSAC method is excellent in robustness but has a large amount of calculation. For this reason, if the above-described imaging device is caused to execute conversion parameter calculation based on the RANSAC method, it takes time for processing. In addition, the higher the degree of freedom of conversion to consider and the greater the number of types of conversion parameters, the longer it takes to calculate conversion parameters based on the RANSAC method.

  Under such circumstances, a technique has been proposed in which conversion parameter calculation based on the RANSAC method is executed in two stages. In the first stage of this technique, images are aligned by the RANSAC method using a transformation model that includes only transformation parameters corresponding to translational movement, and combinations of feature points that match within an allowable range are extracted. In the second stage, a conversion model including a conversion parameter corresponding to translation and rotation is used, and the conversion parameter is calculated by the RANSAC method using the feature points extracted in the first stage.

JP 2009-122843 A

  When the above technique is applied, there is a possibility that the calculation load can be reduced. However, when the technique for dividing the calculation of the conversion parameter into two stages is applied, there is a possibility that the combination of feature points cannot be correctly extracted at the first stage when applied to an image having a large rotation or enlargement / reduction component. . In this case, a correct conversion parameter cannot be obtained even if the second stage process is executed. Also, if the types of conversion parameters applied to the first stage processing are increased, or the allowable range is expanded to increase the number of feature points extracted in the first stage, the effect of reducing the amount of computation is suppressed. Is done. In other words, in order to speed up the calculation of the conversion parameter, it becomes a problem to accurately calculate the conversion parameter in a realistic shooting situation.

  Therefore, according to one aspect, an object of the present invention is to provide an information processing apparatus, a parameter determination method, and a program capable of accurately calculating parameters of a conversion model used for registration between images. It is in.

  According to one aspect of the present disclosure, a first feature point included in the first image, a second feature point included in the second image, and a point on the first image are converted into corresponding points on the second image. Calculating a parameter estimated value included in the conversion model by using a storage unit that stores the conversion model to be used, and a set of first and second feature points having a feature amount larger than the set first threshold, A pair of first feature points and second feature points having a feature amount larger than a smaller second threshold and having an error with respect to the conversion model to which the estimated value is applied is set. An information processing apparatus is provided that includes a calculation unit that calculates a parameter and an evaluation value of the parameter, and that determines a parameter based on the evaluation value.

  According to another aspect of the present disclosure, a first feature point included in the first image, a second feature point included in the second image, and a point on the first image correspond to each other on the second image. A computer capable of acquiring information about the first feature point, the second feature point, and the conversion model from a storage unit that stores a conversion model to be converted into a point has a feature amount larger than a set first threshold value. And an estimated value of a parameter included in the conversion model using the set of the second feature points, and an error with respect to the conversion model in which the feature amount is larger than the second threshold smaller than the first threshold and the estimated value is applied. Select a set of first feature points and second feature points within a set range, calculate a parameter and an evaluation value of the parameter for each selected set, and determine a parameter for determining the parameter based on the evaluation value A method is provided.

  According to another aspect of the present disclosure, a first feature point included in the first image, a second feature point included in the second image, and a point on the first image correspond to each other on the second image. A computer that can acquire information of a first feature point, a second feature point, and a conversion model from a storage unit that stores a conversion model to be converted into a point. And an estimated value of a parameter included in the conversion model using the set of the second feature points, and an error with respect to the conversion model in which the feature amount is larger than the second threshold smaller than the first threshold and the estimated value is applied. Select a set of first feature points and second feature points within the set range, calculate parameters and evaluation values of the parameters for each selected set, and execute a process of determining parameters based on the evaluation values A program is provided.

  According to the present invention, it is possible to accurately calculate parameters of a conversion model used for alignment between images.

It is the figure which showed an example of the information processing apparatus which concerns on 1st Embodiment. It is a figure for demonstrating the RANSAC method. It is the flowchart which showed the flow of a process about calculation of the parameter by a RANSAC method. It is the figure which showed the example which applies the calculation of the parameter by a RANSAC method to image processing. It is the figure which showed an example of the hardware which can implement | achieve the function which the information processing apparatus which concerns on 2nd Embodiment has. It is a block diagram for demonstrating the function of the information processing apparatus which concerns on 2nd Embodiment. It is the figure which showed an example of the feature point information which concerns on 2nd Embodiment. It is the figure which showed an example of the vote number table which concerns on 2nd Embodiment. It is the figure which showed an example of the counter table which concerns on 2nd Embodiment. It is a figure for demonstrating the front | former process (extraction of a feature point pair) which concerns on 2nd Embodiment. It is a figure for demonstrating the front | former process (voting process) which concerns on 2nd Embodiment. It is a figure for demonstrating the front | former process (determination of an estimated conversion matrix) which concerns on 2nd Embodiment. It is a figure for demonstrating the back | latter stage process (calculation of the maximum vote number) which concerns on 2nd Embodiment. It is a figure for demonstrating the back | latter stage process (extraction of a feature point pair) which concerns on 2nd Embodiment. It is a figure for demonstrating the back | latter stage process (comparison of a conversion matrix) which concerns on 2nd Embodiment. It is a figure for demonstrating the back | latter stage process (voting process) which concerns on 2nd Embodiment. It is a figure for demonstrating the back | latter stage process (determination of a conversion matrix) which concerns on 2nd Embodiment. It is the flowchart which showed the whole flow of the process which the information processing apparatus which concerns on 2nd Embodiment performs. It is the 1st flowchart which showed the flow of the front | former stage process which the information processing apparatus which concerns on 2nd Embodiment performs. It is the 2nd flowchart which showed the flow of the front | former stage process which the information processing apparatus which concerns on 2nd Embodiment performs. It is the 1st flowchart which showed the flow of the back | latter stage process which the information processing apparatus which concerns on 2nd Embodiment performs. It is the 2nd flowchart which showed the flow of the back | latter stage process which the information processing apparatus which concerns on 2nd Embodiment performs. It is the 3rd flowchart which showed the flow of the back | latter stage process which the information processing apparatus which concerns on 2nd Embodiment performs. It is the figure which showed the outline | summary of the back | latter stage process which concerns on 3rd Embodiment. It is a block diagram for demonstrating the function of the information processing apparatus which concerns on 3rd Embodiment. It is the figure which showed an example of the tolerance | permissible_range information which concerns on 3rd Embodiment. It is the figure which showed an example of the pair map list | wrist which concerns on 3rd Embodiment. It is a 1st figure for demonstrating the generation method of the pair map list which concerns on 3rd Embodiment. It is a 2nd figure for demonstrating the generation method of the pair map list which concerns on 3rd Embodiment. It is a figure for demonstrating the back | latter stage process (determination of a conversion matrix) which concerns on 3rd Embodiment. It is the 1st flowchart which showed the flow of the back | latter stage process which the information processing apparatus which concerns on 3rd Embodiment performs. It is the 2nd flowchart which showed the flow of the back | latter stage process which the information processing apparatus which concerns on 3rd Embodiment performs. It is the flowchart which showed the flow of the process regarding the production | generation of a pair map list among the back | latter stage processes which the information processing apparatus which concerns on 3rd Embodiment performs. It is a 1st figure for demonstrating the combination of 2nd and 3rd embodiment. It is a 2nd figure for demonstrating the combination of 2nd and 3rd embodiment. It is the flowchart which showed the flow of the back | latter stage process which concerns on the combination of 2nd and 3rd embodiment.

  Embodiments of the present invention will be described below with reference to the accompanying drawings. In addition, about the element which has the substantially same function in this specification and drawing, duplication description may be abbreviate | omitted by attaching | subjecting the same code | symbol.

<1. First Embodiment>
The first embodiment will be described with reference to FIG. FIG. 1 is a diagram illustrating an example of an information processing apparatus according to the first embodiment. Note that the information processing apparatus 10 illustrated in FIG. 1 is an example of an information processing apparatus according to the first embodiment.

As illustrated in FIG. 1, the information processing apparatus 10 includes a storage unit 11 and a calculation unit 12.
The storage unit 11 is a volatile storage device such as a RAM (Random Access Memory) or a non-volatile storage device such as an HDD (Hard Disk Drive) or a flash memory. The arithmetic unit 12 is a processor such as a CPU (Central Processing Unit) or a DSP (Digital Signal Processor). However, the arithmetic unit 12 may be an electronic circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA). Moreover, the calculating part 12 performs the program memorize | stored in the memory | storage part 11 or another memory, for example.

  The storage unit 11 converts the first feature point q1 included in the first image P1, the second feature point q2 included in the second image P2, and the point on the first image P1 into a corresponding point on the second image P2. The conversion model TM to be converted is stored. For example, the first image P1 and the second image P2 are images that are continuously captured. The first feature point q1 is a point where the feature amount is larger than the set threshold value. For example, the first feature point q1 is obtained by detecting the feature amount for each pixel of the first image P1 and extracting pixels whose detected feature amount is larger than the set threshold value.

  Similarly, the second feature point q2 is a point where the feature amount is larger than the set threshold value. For example, the feature amount is detected for each pixel of the second image P2, and the second feature point q2 is obtained by extracting pixels whose detected feature amount is larger than a set threshold value. Examples of the feature amount include edge strength (gradient magnitude) obtained by edge detection and edge direction (gradient direction). When the gradient direction is used, the number of pixels having a gradient direction different from the pixel of interest among the pixels around the pixel of interest is detected, and pixels of interest whose number is larger than the set threshold value are extracted as feature points.

  In the example of FIG. 1, first feature points q11, q12,..., Q16 are shown as the first feature points q1. The first feature points q11, q12, and q13 are feature points that indicate a feature portion of the object C11. In addition, the first feature points q14, q15, and q16 are feature points that indicate feature portions of the object C12.

  In the example of FIG. 1, second feature points q21, q22,..., Q26 are shown as the second feature points q2. The second feature points q21, q22, and q23 are feature points that indicate a feature portion of the object C21. In addition, the second feature points q24, q25, and q26 are feature points that indicate feature portions of the object C22. The objects C11 and C21 represent the same subject. The objects C12 and C22 represent the same subject. However, in the example of FIG. 1, since the shooting situation (such as a change in exposure or a shift in the imaging range due to camera shake of the photographer) differs when shooting the first image P1 and when shooting the second image P2. The point q1 and the second feature point q2 do not match.

  The calculation unit 12 calculates an estimated value A1 of the parameter A included in the conversion model TM using a set of the first feature point q1 and the second feature point q2 having a feature amount larger than the set first threshold Th1. As the conversion model TM, for example, a conversion model of affine conversion can be applied. In this case, the parameter A is a conversion matrix including a plurality of parameters representing translation, rotation, and enlargement / reduction.

  Note that by setting the first threshold value Th1 higher than the threshold value used when extracting the first feature point q1 and the second feature point q2, the reliability as the feature point representing the features of the objects C11, C12, C21, and C22. A set of the first feature point q1 and the second feature point q2 having a high degree is obtained. Therefore, if the above-described method of narrowing down feature points using the first threshold Th1 is applied, higher estimation accuracy can be obtained compared to a method of randomly selecting feature points used for calculation of the estimated value A1.

  The computing unit 12 that has calculated the estimated value A1 has a feature amount larger than the second threshold Th2 that is smaller than the first threshold Th1, and is within a set range of an error with respect to the conversion model TM to which the estimated value A1 is applied. A set of the feature point q1 and the second feature point q2 is selected. In addition, the calculation unit 12 calculates the parameter A and the evaluation value of the parameter A for each selected set. Then, the calculation unit 12 determines the parameter A based on the evaluation value.

  As described above, since the second threshold Th2 is smaller than the first threshold Th1, more feature points than the first feature point q1 and the second feature point q2 used for calculating the estimated value A1 are selected. . However, calculation of an evaluation value is avoided for a set having a large error with respect to the conversion model TM to which the estimated value A1 is applied.

  As described above, the estimated value A1 is calculated using feature points having a large feature amount, and is close to the value of the parameter A calculated using all the first feature points q1 and the second feature points q2. it is conceivable that. Therefore, even if the parameter A is calculated using a group having a large error, the evaluation value of the parameter A becomes small and does not contribute to the determination of the parameter A. In this way, by omitting the set that does not contribute to the determination of the parameter A from the candidates and avoiding the calculation of the evaluation value, the calculation amount can be reduced while maintaining the calculation accuracy of the parameter A.

  Note that the above error is obtained by using the difference between the parameter A calculated from the set of the first feature point q1 and the second feature point q2 and the estimated value A1, or the conversion model TM to which the estimated value A1 is applied. It is possible to evaluate based on the distance between the point obtained by converting the one feature point q1 and the second feature point q2. As a method for calculating the evaluation value, for example, a voting process used for calculation based on the RANSAC method can be applied. In this case, the evaluation value is the number of votes, and the parameter A that maximizes the number of votes is determined. In the example of FIG. 1, parameters A21, A22,... Are shown as candidate parameters A, and the parameter A21 is determined by voting.

The first embodiment has been described above.
<2. Second Embodiment>
Next, a second embodiment will be described. The second embodiment relates to a method of executing conversion parameter calculation based on the RANSAC method in two stages, and proposes a mechanism for realizing a reduction in the amount of calculation while maintaining calculation accuracy.

[2-1. About RANSAC method]
Here, the RANSAC method will be described with reference to FIGS. FIG. 2 is a diagram for explaining the RANSAC method. FIG. 3 is a flowchart showing a processing flow for parameter calculation by the RANSAC method.

  The RANSAC method is one of methods for estimating a mathematical model that well represents a large number of data. In the RANSAC method, a process for calculating a parameter of a mathematical model using a set of randomly extracted data, and whether or not an error between the mathematical model to which the parameter is applied and other data are included in an allowable range. The determination process is repeatedly executed.

  When the data is included in the allowable range is called an inlier, and when the data is not included in the allowable range, it is called an outlier. In the RANSAC method, the number of inlier data is counted for each parameter, and the parameter with the largest count is determined as the optimum parameter. That is, the mathematical model to which the determined parameter is applied is the estimation result of the mathematical model that well represents the data.

  FIG. 2 illustrates a method for estimating the parameter (a) of the mathematical model (y = ax) from a plurality of data (coordinate points) arranged in the xy coordinate system. In this example, since the mathematical model is a linear expression, the parameters can be calculated if there are two data. For example, when two pieces of data (x1, y1) and (x2, y2) are extracted at random, the parameter a is given by (y2-y1) / (x2-x1).

  When the parameter a is calculated from two randomly extracted data, the distance d between the mathematical model (y = ax) to which the calculated parameter a is applied and other data is calculated. Then, the number of data included in the permissible range (the region between the two chain lines 11 and 12 in the example of FIG. 2) (the number of inliers) is counted. Similarly, extraction of data, calculation of parameter a, and count of the number of data serving as inliers are repeatedly executed, and the parameter a having the maximum count is determined as an optimum parameter.

  Note that the number of repetitions of data extraction, parameter a calculation, and counting of the number of data serving as inliers is set to be equal to or greater than the number of sampling times T given by the following equation (1). Here, F represents the degree of freedom of the mathematical model, E represents the proportion of data that becomes an outlier, and P represents the probability that a correct mathematical model will be obtained. That is, if the number of repetitions is set to T or more, a correct mathematical model can be obtained with a probability P or more.

パラメータを推定する方法の一つである最小二乗法の場合、アウトライヤとなるデータがパラメータの推定結果に影響を与えてしまう。一方、ＲＡＮＳＡＣ法の場合、インライヤとなるデータを用いてパラメータが推定されるため、アウトライヤとなるデータがパラメータの推定結果に影響を及ぼさないという利点がある。

In the case of the least square method, which is one of the methods for estimating the parameter, the outlier data affects the parameter estimation result. On the other hand, in the case of the RANSAC method, parameters are estimated using data serving as inliers, and therefore there is an advantage that data serving as outliers does not affect the parameter estimation results.

Here, with reference to FIG. 3, the flow of processing for parameter calculation by the RANSAC method will be described.
As shown in FIG. 3, first, two pieces of data are selected at random (S11). Next, parameters are calculated using the data selected in S11 (S12). Next, one data is extracted from the remaining data excluding the data selected in S11 (S13). Next, it is determined whether or not the data extracted in S13 is an inlier (S14). If so, the process proceeds to S15. On the other hand, if it is an outlier, the process proceeds to S16.

  When the process proceeds to S15, voting is performed for the parameter calculated in S12 (S15). That is, the number of votes is set for each parameter, and the number of votes for the parameter calculated in S12 increases by one. When the process of S15 is completed, the process proceeds to S16.

  When the process proceeds to S16, it is determined whether all the remaining data except for the data selected in S11 has been extracted in S13 (S16). If all the remaining data has been extracted in S13, the process proceeds to S17. On the other hand, if there is data that has not been extracted in S13, the process proceeds to S13.

  When the process proceeds to S17, it is determined whether a series of processes from S11 to S17 has been executed T times (whether T has been looped) (S17). T indicating the number of loops is given by, for example, the above equation (1). If the process loops T times, the process proceeds to S18. On the other hand, if it has not looped T times, the process proceeds to S11.

  When the process proceeds to S18, the parameter with the largest number of votes is output (S18). That is, the parameter that maximizes the number of inliers is output. When the process of S18 is completed, the series of processes shown in FIG.

  As described above, the calculation process by the RANSAC method includes the outer loop process from S11 to S17 and the inner loop process from S13 to S16. The inner loop process is a process (hereinafter referred to as a voting process) in which voting is performed by determining whether or not to become an inlier for all data not used for parameter calculation. Therefore, when the number of data increases, the number of times that the voting process is executed increases dramatically. In the second embodiment, a mechanism for reducing the number of times the voting process is executed while maintaining calculation accuracy is proposed.

(Example of application to image processing)
Here, an example of applying the parameter calculation by the RANSAC method to image processing will be described with reference to FIG. FIG. 4 is a diagram showing an example in which parameter calculation by the RANSAC method is applied to image processing.

  Image processing techniques include a technique for improving image quality using a plurality of images and a technique for performing authentication by collating a plurality of images. For example, electronic camera shake correction applied at the time of moving image shooting is a technology that suppresses disturbance of a display image due to camera shake by positioning between moving image frames and cropping and displaying an overlapping region. In addition, electronic image stabilization applied during still image shooting captures multiple images at a fast shutter speed, combines the images and combines them, and crops the overlapping area, thereby reducing camera shake and subject shake. This is a technology that suppresses image disturbance caused by.

  In HDR shooting, multiple images with different exposures are shot in succession, the images are aligned and combined, and the overlapping areas are cropped, so that overexposure and overexposure cause overexposure and overexposure. This is a technique for generating a suppressed dynamic range image. Authentication techniques such as vein authentication, glow authentication, and face authentication are techniques for verifying the identity by comparing a template image and a captured image.

  All of the image processing techniques as described above use registration between images. Note that the registration of images referred to here is correction of translation, rotation, enlargement / reduction, and the like between target images. For example, even in the case of continuously captured images, various shifts occur between the images due to the effects of subject shake and camera shake. Even if the images are overlapped with each other in a state where a shift has occurred, effects such as an improvement in image quality and an improvement in authentication accuracy cannot be obtained. Therefore, various processes such as display, composition, and authentication are performed after the registration of the images.

  FIG. 4 illustrates a flow of processing for combining two images P1 and P2 captured in succession. In the example of FIG. 4, the image P1 includes four feature points q10,..., Q13, and the image P2 includes four feature points q20,. The relationship between the coordinates of the image P1 and the coordinates of the image P2 can be expressed by, for example, a conversion model of affine transformation (see the following formula (2)). However, (x1, y1) are coordinates on the image P1. Further, (x2, y2) are coordinates on the image P2. W11, w12,..., W23 are conversion parameters.

上記の変換モデルにより画像Ｐ１に含まれる各画素の座標変換を行えば、画像Ｐ２との位置合わせが実現される。例えば、画像Ｐ１を座標変換した画像を画像Ｐ３とすると、２つの画像Ｐ２、Ｐ３を合成することで高画質な画像Ｐ４が得られる。このような画像の合成を精度良く行うには、上記の変換パラメータｗ１１、ｗ１２、…、ｗ２３を精度良く求めることが大事になる。そのため、変換パラメータｗ１１、ｗ１２、…、ｗ２３の計算に対してロバスト性に優れるＲＡＮＳＡＣ法を適用することは好適である。

If coordinate conversion of each pixel included in the image P1 is performed using the above conversion model, alignment with the image P2 is realized. For example, if an image obtained by coordinate-transforming the image P1 is an image P3, a high-quality image P4 can be obtained by combining the two images P2 and P3. In order to accurately synthesize such an image, it is important to obtain the above conversion parameters w11, w12,. Therefore, it is preferable to apply the RANSAC method having excellent robustness to the calculation of the conversion parameters w11, w12,..., W23.

  When the RANSAC method is applied, conversion parameters w11, w12,..., W23 are calculated using a combination of feature points randomly extracted from the feature points q10,..., Q22, and the voting process described above is executed. In addition, the conversion parameters w11, w12,..., W23 are repeatedly calculated and voted, and the conversion parameters w11, w12,..., W23 obtained the maximum number of votes are used for the coordinate conversion of the image P1. However, since the number of feature points actually included in the image is large, the second embodiment proposes a mechanism for reducing the amount of calculation.

(Other application examples)
The parameter calculation by the RANSAC method can be used for purposes other than image processing. For example, it can be used to match feature points of voice data when performing voiceprint authentication or voice analysis. In this case, feature points included in the two audio data are extracted, and parameters of the conversion model expressing the relationship between the feature points are calculated by the RANSAC method. In the following, for the sake of explanation, the description will be made assuming that the images are aligned. However, the scope of application of the technique according to the second embodiment is not limited to this, and extends to any field to which the RANSAC method can be applied. .

The RANSAC method has been described above.
[2-2. hardware]
Next, the hardware of the information processing apparatus 100 capable of realizing the conversion parameter calculation method according to the second embodiment will be described with reference to FIG. The information processing apparatus 100 is an example of an information processing apparatus according to the second embodiment. FIG. 5 is a diagram illustrating an example of hardware capable of realizing the functions of the information processing apparatus according to the second embodiment.

  The functions of the information processing apparatus 100 to be described later can be realized by using, for example, hardware resources of the information processing apparatus illustrated in FIG. That is, the functions of the information processing apparatus 100 are realized by controlling the hardware shown in FIG. 5 using a computer program.

  As shown in FIG. 5, this hardware mainly includes a CPU 902, a ROM (Read Only Memory) 904, a RAM 906, a host bus 908, and a bridge 910. Further, this hardware includes an external bus 912, an interface 914, an input unit 916, an output unit 918, a storage unit 920, a drive 922, a connection port 924, and a communication unit 926.

  The CPU 902 functions as, for example, an arithmetic processing unit or a control unit, and controls the overall operation of each component or a part thereof based on various programs recorded in the ROM 904, the RAM 906, the storage unit 920, or the removable recording medium 928. . The ROM 904 is an example of a storage device that stores a program read by the CPU 902, data used for calculation, and the like. The RAM 906 temporarily or permanently stores, for example, a program read by the CPU 902 and various parameters that change when the program is executed.

  These elements are connected to each other via, for example, a host bus 908 capable of high-speed data transmission. On the other hand, the host bus 908 is connected to an external bus 912 having a relatively low data transmission speed via a bridge 910, for example. As the input unit 916, for example, a mouse, a keyboard, a touch panel, a touch pad, a button, a switch, a lever, or the like is used. Furthermore, as the input unit 916, a remote controller capable of transmitting a control signal using infrared rays or other radio waves may be used.

  As the output unit 918, for example, a display device such as a CRT (Cathode Ray Tube), an LCD (Liquid Crystal Display), a PDP (Plasma Display Panel), or an ELD (Electro-Luminescence Display) is used. As the output unit 918, an audio output device such as a speaker or headphones, or a printer may be used. In other words, the output unit 918 is a device that can output information visually or audibly.

  The storage unit 920 is a device for storing various data. As the storage unit 920, for example, a magnetic storage device such as an HDD is used. Further, as the storage unit 920, a semiconductor storage device such as an SSD (Solid State Drive) or a RAM disk, an optical storage device, a magneto-optical storage device, or the like may be used.

  The drive 922 is a device that reads information recorded on a removable recording medium 928 that is a removable recording medium or writes information on the removable recording medium 928. As the removable recording medium 928, for example, a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory is used.

  The connection port 924 is a port for connecting an external connection device 930 such as a USB (Universal Serial Bus) port, an IEEE 1394 port, a SCSI (Small Computer System Interface), an RS-232C port, or an optical audio terminal. For example, a printer or the like is used as the external connection device 930.

  The communication unit 926 is a communication device for connecting to the network 932. As the communication unit 926, for example, a communication circuit for wired or wireless LAN (Local Area Network), a communication circuit for WUSB (Wireless USB), a communication circuit or router for optical communication, an ADSL (Asymmetric Digital Subscriber Line) Communication circuits, routers, communication circuits for mobile phone networks, and the like are used. A network 932 connected to the communication unit 926 is a wired or wireless network, and includes, for example, the Internet, a LAN, a broadcast network, a satellite communication line, and the like.

The hardware of the information processing apparatus 100 has been described above.
[2-3. Function of information processing apparatus]
Next, functions of the information processing apparatus 100 will be described with reference to FIG. FIG. 6 is a block diagram for explaining functions of the information processing apparatus according to the second embodiment. In the following description, FIGS. 7 to 17 are referred to as appropriate.

As illustrated in FIG. 6, the information processing apparatus 100 includes a storage unit 101, an imaging unit 102, a feature point extraction unit 103, a pre-processing unit 104, a post-processing unit 105, and an image processing unit 106.
Note that the function of the storage unit 101 can be realized by using the above-described RAM 906, the storage unit 920, or the like. The functions of the feature point extraction unit 103, the pre-processing unit 104, the post-processing unit 105, and the image processing unit 106 can be realized using the above-described CPU 902 or the like.

(Storage unit 101)
The storage unit 101 stores feature point information 101a, model information 101b, a vote count table 101c, and a counter table 101d. The feature point information 101a is information related to feature points extracted from the image. The model information 101b is information related to a conversion model used for image alignment. The vote count table 101c and the counter table 101d are tables used for managing the vote count in the voting process.

Here, the feature point information 101a will be further described with reference to FIG. FIG. 7 is a diagram illustrating an example of feature point information according to the second embodiment.
In the example of FIG. 7, information (coordinates and feature amounts) regarding feature points extracted from each of the two images P1 and P2 to be aligned is included in the feature point information 101a. Note that the feature points of the image P1 and the feature points of the image P2 are not associated with each other. Examples of the feature amount include edge strength (gradient magnitude) obtained by edge detection and edge direction (gradient direction). The feature quantity indicating the direction of the edge is expressed by an index based on the number of pixels having a gradient direction different from that of the pixel of interest among the pixels around the pixel of interest.

Here, the model information 101b will be further described.
The model information 101b includes information on a mathematical expression expressing the conversion model and conversion parameters. As the conversion model, for example, a conversion model of affine transformation expressed by the above formula (1) can be applied. In the case of affine transformation, the transformation parameters are expressed in the form of a transformation matrix. This transformation matrix includes six transformation parameters. Therefore, if three pairs of feature points of the image P1 and feature points of the image P2 (hereinafter referred to as feature point pairs) are prepared, a transformation matrix can be calculated.

Here, the vote number table 101c will be further described with reference to FIG. FIG. 8 is a diagram showing an example of the vote number table according to the second embodiment.
As shown in FIG. 8, the vote number table 101c includes the feature point pairs used for calculating the transformation matrix, the parameters included in the calculated transformation matrix, and the number of votes voted for the parameters in the voting process. It is a table to associate. For example, in the example of FIG. 8, conversion parameters (0.31, 0.26, 0.61, 0.18) for three feature point pairs (q14, q22), (q13, q22), and (q11, q27). , 0.53, 0.12). The number of votes (10) is associated with this conversion parameter.

Here, the counter table 101d will be further described with reference to FIG. FIG. 9 is a diagram illustrating an example of a counter table according to the second embodiment.
As illustrated in FIG. 9, the counter table 101 d is a table that associates feature points included in one image with a counter value. In the example of FIG. 9, each of the feature points q10, q11,..., Q1n of the image P1 is associated with a counter value. The counter value is a value obtained by counting feature points included in a feature point pair determined as an inlier in the voting process. The counter table 101d is used when controlling the selection order of feature points for determining whether or not an inlier is performed when executing a voting process.

(Imaging unit 102, feature point extracting unit 103)
Refer to FIG. 6 again. The imaging unit 102 is an element that provides a camera function and captures an image. For example, the imaging unit 102 includes an optical system, an imaging device, an analog / digital conversion circuit, a control circuit, and the like. Images captured by the imaging unit 102 (hereinafter, images P1 and P2) are stored in the storage unit 101 and input to the feature point extraction unit 103. The feature point extraction unit 103 extracts feature points from each of the input images P1 and P2, and generates feature point information 101a. The feature point information 101 a generated by the feature point extraction unit 103 is stored in the storage unit 101.

(Pre-processing unit 104)
The pre-processing unit 104 calculates an estimated value of a transformation matrix (hereinafter, an estimated transformation matrix) using a feature point having a large feature amount. In the following, the process executed by the pre-processing unit 104 may be referred to as “pre-process”. In order to execute the pre-stage processing, the pre-stage processing section 104 includes a parameter calculation section 141 and a voting processing section 142 as shown in FIG.

  The parameter calculation unit 141 refers to the feature point information 101a stored in the storage unit 101, and among the feature points extracted from the image P1, sets a predetermined number (X) of feature points in descending order of feature amount. Extract. Further, the parameter calculation unit 141 extracts a preset number (X) of feature points in descending order of the feature amount from the feature points extracted from the image P2. Then, the parameter calculation unit 141 selects a feature point pair from the extracted feature points, and calculates a transformation matrix included in the transformation model.

  The parameter calculation unit 141 records the selected feature point pair information and the calculated conversion parameter of the conversion matrix in the vote count table 101c. In addition, the transformation matrix calculated by the parameter calculation unit 141 is input to the voting processing unit 142. The voting processing unit 142 sequentially selects feature points that are not used in the calculation of the input transformation matrix among the X feature points extracted for the image P1 by the parameter calculation unit 141, and the selected feature points are inlined. It is determined whether or not.

  For example, the voting processing unit 142 converts the feature point of the selected image P1 using the input conversion matrix, and whether or not the feature point of the image P2 exists within the allowable range with the converted point as a reference. Determine whether. When the feature point of the image P2 exists within the allowable range, the voting processing unit 142 determines that the feature point of the selected image P1 is an inlier. Note that the voting processing unit 142 determines whether or not the shortest distance between the converted point and the feature point of the image P2 is within the allowable range, and determines that it is an inlier when the shortest distance is within the allowable range. May be.

  Each time the voting processing unit 142 determines to be an inlier, it votes for the conversion parameter of the input conversion matrix. That is, when it is determined that the voter is an inlier, the voting processing unit 142 increases the number of votes set in the vote number table 101c corresponding to the conversion parameter of the input conversion matrix by a set number (for example, 1). When the above determination is completed for all X feature points, the voting processing unit 142 notifies the parameter calculating unit 141 of the end of the voting process for the input transformation matrix. Upon receiving this notification, the parameter calculation unit 141 selects an unselected feature point pair and calculates a transformation matrix included in the transformation model.

  The parameter calculation unit 141 records the selected feature point pair information and the calculated conversion parameter of the conversion matrix in the vote count table 101c. In addition, the transformation matrix calculated by the parameter calculation unit 141 is input to the voting processing unit 142. The voting processing unit 142 sequentially selects feature points that are not used in the calculation of the input transformation matrix among the X feature points extracted for the image P1 by the parameter calculation unit 141, and the selected feature points are inlined. It is determined whether or not. Each time the voting processing unit 142 determines to be an inlier, it votes for the conversion parameter of the input conversion matrix.

  Similarly to the above, the parameter calculation unit 141 calculates a transformation matrix by sequentially selecting feature point pairs from X feature points extracted from the images P1 and P2. And the voting process part 142 performs a voting process with respect to the calculated conversion matrix. When all combinations that can be extracted as feature point pairs have been selected, the pre-processing unit 104 refers to the vote number table 101c and extracts a conversion matrix (estimated conversion matrix) having the maximum number of votes. Information on the estimated transformation matrix extracted by the upstream processing unit 104 is stored in the storage unit 101 and input to the downstream processing unit 105.

(Post-processing unit 105)
The post-processing unit 105 uses the more feature points than the number of feature points used for calculation of the estimated transformation matrix among the feature points included in the feature point information 101a stored in the storage unit 101, and uses the image P1, A transformation matrix used for the alignment of P2 is calculated. At this time, the post-processing unit 105 narrows down feature point pairs used for calculation of the transformation matrix based on the estimated transformation matrix. In the following, the processing executed by the post-processing unit 105 may be referred to as “post-processing”. In order to execute the post-stage processing, the post-stage processing unit 105 includes a parameter calculation unit 151, a candidate extraction unit 152, and a voting processing unit 153 as shown in FIG.

  The parameter calculation unit 151 refers to the feature point information 101a stored in the storage unit 101, and randomly selects a feature point pair that is a pair of the feature point extracted from the image P1 and the feature point extracted from the image P2. Select and calculate the transformation matrix included in the transformation model. At this time, the parameter calculation unit 151 may refer to the counter table 101d stored in the storage unit 101 and select feature points in order from the feature point with the smallest number of votes. The transformation matrix calculated by the parameter calculation unit 151 is input to the candidate extraction unit 152.

  Also, the parameter calculation unit 151 records the selected feature point pair information and the calculated conversion parameter of the conversion matrix in the vote count table 101c. At this time, the parameter calculation unit 151 prepares a vote number table 101c different from the vote number table 101c used by the pre-processing unit 104 at the time of calculating the estimated conversion matrix, and includes information on feature point pairs in the prepared vote number table 101c. Record the conversion parameters.

  The candidate extraction unit 152 calculates an error between the input conversion matrix and the estimated conversion matrix, and determines whether or not the error is within an allowable range. For example, for each type of conversion parameter (rotation, translational movement, enlargement / reduction, etc.), the candidate extraction unit 152 determines the difference between the conversion parameter included in the input conversion matrix and the conversion parameter included in the estimated conversion matrix ( Difference absolute value). Then, the candidate extraction unit 152 determines whether or not the calculated difference is smaller than a preset allowable value. However, the allowable value is set for each type of conversion parameter.

  If the error is within the allowable range, the candidate extraction unit 152 inputs the input conversion matrix to the voting processing unit 153. In this case, a voting process by the voting processing unit 153 described later is executed for the transformation matrix determined that the error is within the allowable range. On the other hand, when the error is outside the allowable range, the candidate extraction unit 152 notifies the parameter calculation unit 151 that the error is outside the allowable range. In this case, voting processing by the voting processing unit 153, which will be described later, is not executed for the transformation matrix for which the error is determined to be outside the allowable range.

  The voting processing unit 153 to which the transformation matrix is input sequentially selects the feature points that are not used in the calculation of the inputted transformation matrix among the feature points extracted from the image P1, and the selected feature point is an inlier. It is determined whether or not.

  For example, the voting processing unit 153 converts the feature point of the selected image P1 using the input conversion matrix, and whether or not the feature point of the image P2 exists within the allowable range with the converted point as a reference. Determine whether. When the feature point of the image P2 exists within the allowable range, the voting processing unit 153 determines that the feature point of the selected image P1 is an inlier. The voting processing unit 153 determines whether or not the shortest distance between the converted point and the feature point of the image P2 is within the allowable range, and determines that the inlier is in the case where the shortest distance is within the allowable range. May be.

  Each time the voting processing unit 153 determines to be an inlier, it votes for the conversion parameter of the input conversion matrix. That is, when it is determined as an inlier, the voting processing unit 153 increases the number of votes set in the vote number table 101c corresponding to the conversion parameter of the input conversion matrix (for example, 1). Further, when determining that the voter is an inlier, the voting processing unit 153 increases the count value of the counter table 101d for the feature point of the image P1 included in the feature point pair used for the calculation of the input transformation matrix.

  When the above determination is completed for all the feature points, the voting processing unit 153 notifies the parameter calculating unit 151 of the end of the voting process regarding the input transformation matrix. Upon receiving this notification, the parameter calculation unit 151 selects an unselected feature point pair and calculates a transformation matrix included in the transformation model.

  The parameter calculation unit 151 records the selected feature point pair information and the calculated conversion parameter of the conversion matrix in the vote count table 101c. In addition, the transformation matrix calculated by the parameter calculation unit 151 is input to the candidate extraction unit 152. The candidate extraction unit 152 performs a determination process on the input transformation matrix. When a conversion matrix is input, the voting processing unit 153 sequentially selects feature points that are not used for calculation of the input conversion matrix, and determines whether or not the selected feature points are inliers. The voting processing unit 153 votes for the conversion parameter of the input conversion matrix every time it is determined as an inlier.

  Similarly to the above, the parameter calculation unit 151 sequentially selects feature point pairs from the feature points extracted from the images P1 and P2, and calculates a transformation matrix. Then, the candidate extraction unit 152 executes a determination process for the selected transformation matrix. Further, the voting processing unit 153 performs voting processing on the calculated conversion matrix when the conversion matrix is input. When these processes are executed a predetermined number of times (for example, T given by the above equation (1)), the post-processing unit 105 refers to the vote number table 101c and selects a conversion matrix having the largest vote number. Extract.

  Information on the transformation matrix extracted by the post-processing unit 105 is input to the image processing unit 106. The image processing unit 106 performs coordinate conversion of the pixels included in the image P1 using the input conversion matrix. Then, the image processing unit 106 performs preset image processing using the converted image P1 and the image P2. For example, the image processing unit 106 performs electronic camera shake correction, HDR image generation, various authentication processes, and the like.

(Pre-processing: Calculation of estimated transformation matrix)
Here, the pre-processing will be further described with reference to FIGS.
FIG. 10 is a diagram for explaining pre-processing (extraction of feature point pairs) according to the second embodiment. FIG. 11 is a diagram for explaining the pre-stage process (voting process) according to the second embodiment. FIG. 12 is a diagram for explaining pre-stage processing (determination of an estimated transformation matrix) according to the second embodiment.

  As shown in FIG. 10, n (for example, n = 100) feature points q10, q11,..., Q1n are detected from the image P1, and m (for example, m = 80) feature points q20 are detected from the image P2. Consider the case where q21,..., q2m are detected. In this case, the parameter calculation unit 141 extracts X feature points (X = 10 in the example of FIG. 10) in descending order of the feature amount from the feature points detected from the image P1. Also, the parameter calculation unit 141 extracts X feature points (X = 10 in the example of FIG. 10) in descending order of the feature amount from the feature points detected from the image P2.

  A feature point having a large feature amount is a feature point having high reliability as a feature point. Further, extracting feature points having a large feature amount corresponds to increasing a threshold value that serves as a selection criterion for feature points. As described above, by reducing the number of feature points, it is possible to significantly reduce the number of combinations when generating feature point pairs by combining the feature points of the image P1 and the feature points of the image P2. If the number of possible feature point pairs decreases, the number of times the voting process is executed by the voting processing unit 142 decreases, and the processing load is reduced.

  The parameter calculation unit 141 selects a combination of feature points from the extracted feature points, and generates a number of feature point pairs sufficient to calculate the transformation matrix. When the transformation model of affine transformation given by the above equation (2) is used, a transformation matrix can be calculated by using at least three feature point pairs. Therefore, FIG. 10 shows an example in which three feature point pairs are generated. Note that the distance between feature points may be taken into account when extracting feature points. A method for considering the distance between feature points will be described later.

  After generating the feature point pair as described above, the parameter calculation unit 141 calculates a transformation matrix using the generated feature point pair as shown in FIG. Note that a transformation matrix corresponding to the i-th generated feature point pair is denoted Ri (i = 1, 2,...). That is, the parameter calculation unit 141 calculates the conversion parameters r11, r12,..., R23 of the conversion matrix Ri. The calculated transformation matrix Ri is used for voting processing by the voting processing unit 142.

  The voting processing unit 142 selects feature points one by one from the feature points of the image P1 extracted by the parameter calculation unit 141, and converts the coordinates of the selected feature points using the transformation matrix Ri. Then, the voting processing unit 142 determines whether the converted coordinates and the feature points of the image P2 are within the allowable range.

  In the example of FIG. 11, the converted coordinates a corresponding to the feature point q15 of the image P1 and the feature point q27 of the image P2 are within the allowable range. The same applies to the feature point q17 of the image P1. On the other hand, the feature point of the image P2 does not exist within the allowable range based on the coordinate b after conversion corresponding to the feature point q16 of the image P1. In this case, the feature points q15 and q17 are determined to be inliers, and the feature point q16 is determined to be an outlier. If it is determined as an inlier, the voting processing unit 142 increases the number of votes in the transformation matrix Ri.

  When all feature points (X) of the image P1 extracted by the parameter calculation unit 141 have been selected, the voting processing unit 142 notifies the parameter calculation unit 141 of the completion of the voting process for the transformation matrix Ri. Upon receiving this notification, the parameter calculation unit 141 generates another feature point pair. Then, the voting process by the voting processing unit 142 is executed again. As shown in FIG. 12, feature point pair extraction and voting processing are repeatedly executed. For example, the process is repeatedly executed until the process is completed for all possible feature point pairs. However, the number of repetitions may be set in advance.

  When the iterative process is completed, a vote number table 101c as shown in FIG. 12 is obtained. In the example of FIG. 12, information on the conversion matrix corresponding to the conversion parameter is also described. For example, the voting processing unit 142 sorts the records of the vote number table 101c in descending order of the number of votes, and extracts a conversion matrix corresponding to the record positioned at the top. In the example of FIG. 12, the transformation matrix R3 corresponds to the maximum number of votes. Then, the voting processing unit 142 sets the extracted transformation matrix (R3 in the example of FIG. 12) as the estimated transformation matrix R. When the setting of the estimated transformation matrix R is completed, the pre-processing is finished.

(Post-processing: Determination of transformation matrix)
Next, the subsequent processing will be further described with reference to FIGS.
FIG. 13 is a diagram for explaining post-processing (calculation of the maximum number of votes) according to the second embodiment. FIG. 14 is a diagram for explaining post-processing (extraction of feature point pairs) according to the second embodiment. FIG. 15 is a diagram for explaining post-stage processing (comparison of transformation matrices) according to the second embodiment. FIG. 16 is a diagram for explaining post-processing (voting processing) according to the second embodiment. FIG. 17 is a diagram for explaining post-processing (conversion matrix determination) according to the second embodiment.

  As illustrated in FIG. 13, the post-processing unit 105 calculates an initial value of an index (maximum number of votes) that is referred to in the process. This index is used to determine a voting process for a transformation matrix that is not finally adopted. For example, if there is a transformation matrix with the largest number of votes at the current time, and it is determined that the number of feature points that are inliers in the voting process for other transformation matrices is less than the largest number of votes at the moment, The transformation matrix is not finally adopted. In this case, it is useless to continue the voting process for another transformation matrix. Therefore, the above index (maximum number of votes) is introduced in order to eliminate useless processing.

  The voting processing unit 153 selects feature points one by one from the feature points of the image P1, and converts the coordinates of the selected feature points using the estimated conversion matrix R. At this time, the feature points selected by the voting processing unit 153 are not limited to the X feature points used in the previous process, but all the feature points extracted from the image P1 or X or more feature points are used. Is done. The voting processing unit 153 determines whether the converted coordinates and the feature points of the image P2 are within the allowable range.

  In the example of FIG. 13, the converted coordinates a corresponding to the feature point q15 of the image P1 and the feature point q27 of the image P2 are within the allowable range. The same applies to the feature point q17 of the image P1. On the other hand, the feature point of the image P2 does not exist within the allowable range based on the coordinate b after conversion corresponding to the feature point q16 of the image P1. In this case, the feature points q15 and q17 are determined to be inliers, and the feature point q16 is determined to be an outlier. When it is determined as an inlier, the voting processing unit 153 increases the maximum number of votes.

  After the maximum number of votes is calculated, the parameter calculation unit 151 selects a combination of feature points randomly from the feature points of the images P1 and P2, as shown in FIG. 14, and has a number sufficient to calculate the transformation matrix. Generate feature point pairs. However, the parameter calculation unit 151 generates a feature point pair different from the feature point pair used for calculating the estimated transformation matrix R.

  After generating the feature point pair, the parameter calculation unit 151 calculates a transformation matrix using the generated feature point pair as shown in FIG. Note that the transformation matrix corresponding to the i-th generated feature point pair is denoted as Hi (i = 1, 2,...). That is, the parameter calculation unit 151 calculates the conversion parameters h11, h12,..., H23 of the conversion matrix Hi. The calculated transformation matrix Hi is used by the candidate extraction unit 152.

  The candidate extraction unit 152 determines the difference between the conversion parameters r11, r12, ..., r23 of the estimated conversion matrix R and the conversion parameters h11, h12, ..., h23 of the conversion matrix Hi | rk-hk | (where k = 11, 12,..., 23) are calculated.

  Further, the candidate extraction unit 152 determines whether or not the calculated difference | rk−hk | is smaller than a preset threshold value Thk. When the calculated difference | rk−hk | is smaller than the threshold Thk, the candidate extraction unit 152 inputs the transformation matrix Hi as a candidate to the voting processing unit 153. In this case, the transformation matrix Hi is used for voting processing by the voting processing unit 153. On the other hand, when the calculated difference | rk−hk | is not smaller than the threshold value Thk, the candidate extraction unit 152 excludes the transformation matrix Hi from the candidates and does not input it to the voting processing unit 153. In this case, the transformation matrix Hi is not used for voting processing by the voting processing unit 153.

  When the transformation matrix Hi is input to the voting processing unit 153, the voting processing unit 153 selects feature points one by one from the feature points of the image P1, and transforms the coordinates of the selected feature points using the transformation matrix Hi. At this time, the feature points selected by the voting processing unit 153 are not limited to the X feature points used in the previous process, but all the feature points extracted from the image P1 or X or more feature points are used. Is done. At this time, the voting processing unit 153 refers to the counter table 101d and selects feature points in ascending order of the number of votes. Also, the voting processing unit 153 determines whether or not the converted coordinates and the feature points of the image P2 are within an allowable range.

  In the example of FIG. 16, the converted coordinates a corresponding to the feature point q15 of the image P1 and the feature point q27 of the image P2 are within the allowable range. The same applies to the feature point q17 of the image P1. On the other hand, the feature point of the image P2 does not exist within the allowable range based on the coordinate b after conversion corresponding to the feature point q16 of the image P1. In this case, the feature points q15 and q17 are determined to be inliers, and the feature point q16 is determined to be an outlier.

  If it is determined as an inlier, the voting processing unit 153 increases the number of votes for the transformation matrix Hi. Further, the voting processing unit 153 increases the count value of the counter table 101d for the feature points of the image P1 included in the feature point pair used for the calculation of the transformation matrix Hi, and updates the counter table 101d.

  In addition, the voting processing unit 153 counts the number of feature points determined to be outliers at the present time. When the current maximum number of votes is Nmax and the number of feature points of the image P1 used for calculation of the estimated transformation matrix R is d, the voting processing unit 153 determines that the number of feature points determined to be outliers is (n−d -Nmax) is determined. When the number of feature points determined to be outliers is larger than (nd−Nmax), the voting processing unit 153 notifies the parameter calculation unit 151 that the voting processing for the transformation matrix Hi is stopped.

  When all the feature points of the image P1 have been selected, the voting processing unit 153 notifies the parameter calculating unit 151 of the completion of the voting process for the transformation matrix Hi. Upon receiving the notification of completion or cancellation, the parameter calculation unit 151 generates another feature point pair. And the process by the candidate extraction part 152 and the voting process part 153 is performed again. As shown in FIG. 17, feature point pair extraction, transformation matrix comparison, and voting are repeatedly performed. For example, the process is executed for a preset number of repetitions (for example, T given by the above equation (1)).

  When the iterative process is completed, a vote number table 101c as shown in FIG. 17 is obtained. In the example of FIG. 17, information on the conversion matrix corresponding to the conversion parameter is also described. For example, the voting processing unit 153 sorts the records of the vote number table 101c in descending order of the number of votes, and extracts a conversion matrix corresponding to the record positioned at the highest level. In the example of FIG. 17, the conversion matrix H9 corresponds to the maximum number of votes. Then, the voting processing unit 153 sets the extracted conversion matrix (H9 in the example of FIG. 17) as the conversion matrix H used for the conversion of the image P1. When the setting of the transformation matrix H is completed, the subsequent process is terminated.

(Modification: A method that considers the distance between feature points in the previous process)
Here, a method (variation example) of extracting a feature point pair used for the pre-stage processing in consideration of the distance between the feature points will be further described.

  The method of extracting X feature points in descending order of the feature amount in the preprocessing has already been described. Here, a method for preferentially extracting not only the size of the feature amount but also a distance between feature points will be described.

  For example, the pre-processing unit 104 extracts Y (Y> X) feature points in descending order of the feature amount, and sequentially selects the feature points with the longest shortest distance from other feature points. Select and extract X feature points. As another example, after weighting the feature amount of the pixel of interest using a weight value that increases as the distance between the pixel of interest and the surrounding pixels increases, the X weighted feature amount increases in descending order. A method of extracting the feature points of can also be considered. As another example, a method may be considered in which an image is divided into a plurality of divided regions, and Z feature points (where X = the number of divided regions × Z) are selected in descending order of the feature amount for each divided region.

  If the above method is applied, feature points that are widely dispersed throughout the entire image are preferentially extracted, so that quantum errors can be obtained even when targeting an image in which feature points with large feature quantities are gathered locally. The effect of reducing the amount can be expected. Furthermore, when a subject including a feature point having a large feature amount is a moving object, an effect of preventing the entire screen from being dragged by the moving object can be expected.

The function of the information processing apparatus 100 has been described above.
[2-4. Process flow]
Next, the flow of processing executed by the information processing apparatus 100 will be described with reference to FIGS.

(Overall processing flow)
First, the overall flow of processing will be described with reference to FIG. FIG. 18 is a flowchart showing an overall flow of processing executed by the information processing apparatus according to the second embodiment.

  As illustrated in FIG. 18, the information processing apparatus 100 acquires images P1 and P2 (S101). For example, the information processing apparatus 100 acquires images P <b> 1 and P <b> 2 by continuously capturing images of the subject using the function of the imaging unit 102. However, the information processing apparatus 100 may acquire the images P1 and P2 from an external storage device connected to the network 932 or the connection port 924. Further, the information processing apparatus 100 may acquire the images P1 and P2 recorded on the removable recording medium 928 via the drive 922.

  Next, the information processing apparatus 100 extracts feature points (for example, 100 pieces each) from the images P1 and P2 by the function of the feature point extraction unit 103 (S102). Next, the information processing apparatus 100 executes pre-processing by the function of the pre-processing unit 104 (S103). In the process of S103, an estimated transformation matrix is calculated.

  Next, the information processing apparatus 100 executes post-processing by the function of the post-processing unit 105 (S104). In the process of S104, a conversion matrix used for conversion of the image P1 is determined. Next, the information processing apparatus 100 performs image processing using the transformation matrix determined in S104 by the function of the image processing unit 106 (S105), and outputs an image after image processing (S106). When the process of S106 is completed, the series of processes illustrated in FIG.

(Flow of pre-processing)
Next, the flow of the pre-processing will be further described with reference to FIGS. FIG. 19 is a first flowchart illustrating a flow of pre-processing performed by the information processing apparatus according to the second embodiment. FIG. 20 is a second flowchart showing the flow of pre-processing performed by the information processing apparatus according to the second embodiment. The processes shown in FIGS. 19 and 20 are mainly executed by the pre-processing unit 104.

  As illustrated in FIG. 19, the pre-processing unit 104 extracts X feature points (for example, X = 10) from the feature points of the image P1 in descending order by the function of the parameter calculation unit 141 (S111). . Next, the pre-processing unit 104 extracts X feature points in descending order of the feature amount from the feature points of the image P2 by the function of the parameter calculation unit 141 (S112).

  Next, the pre-processing unit 104 uses the function of the parameter calculation unit 141 to combine d feature point pairs obtained by combining the feature points extracted from the image P1 in the process of S111 and the feature points extracted from the image P2 in the process of S112. Is generated (S113). Next, the pre-processing unit 104 calculates a transformation matrix Ri using the d feature point pairs generated in the process of S113 (S114). However, the transformation matrix Ri represents a transformation matrix calculated using the i-th feature point pair generated in the process of S113.

  Next, the pre-processing unit 104 selects one feature point that is not used in the calculation of the transformation matrix Ri from the feature points of the image P1 by the function of the voting processing unit 142 (S115). Next, the pre-processing unit 104 converts the coordinates of the feature point selected in the process of S115 with the conversion matrix Ri (S116). When the process of S116 is completed, the process proceeds to S117 of FIG.

  When the process proceeds to S117, the pre-processing unit 104 uses the function of the voting processing unit 142 to display a feature point whose distance from the point converted by the process of S116 is equal to or less than a threshold ThD (for example, 5 pixels). It is determined whether or not it exists in P2 (S117). If a feature point whose distance from the converted point is equal to or less than the threshold ThD exists in the image P2, the process proceeds to S118. On the other hand, if there is no feature point in the image P2 whose distance from the converted point by the process of S116 is equal to or less than the threshold ThD, the process proceeds to S119.

  When the process proceeds to S118, the pre-processing unit 104 votes for the transformation matrix Ri by the function of the voting processing unit 142 (S118). When the process of S118 is completed, the process proceeds to S122.

  When the process proceeds to S119, the pre-stage processing unit 104 calculates (Xd-Nmax) based on the maximum number of votes Nmax at the current time by the function of the voting processing unit 142 (S119). Next, the pre-processing unit 104 calculates the number of feature points that have been determined as outliers by the function of the voting processing unit 142 (S120).

  Next, the pre-processing unit 104 determines whether the calculation result in the processing of S120 is greater than (Xd-Nmax) by the function of the voting processing unit 142 (S121). If the calculation result is greater than (Xd-Nmax), the process proceeds to S113 in FIG. On the other hand, if the calculation result is not greater than (X−d−Nmax), the process proceeds to S122.

  When the process proceeds to S122, the pre-processing unit 104 determines whether all (Xd) feature points have been selected in the process of S115 (S122). If all (Xd) feature points have been selected, the process proceeds to S123. On the other hand, when there is an unselected feature point among (Xd) feature points, the process proceeds to S115 in FIG.

  When the process proceeds to S123, the pre-processing unit 104 determines all the feature points that can be generated by combining the feature points extracted from the image P1 in the process of S111 and the feature points extracted from the image P2 in the process of S112. It is determined whether or not a pair has been generated in the process of S113 (S123). If all the feature point pairs have been generated in S113, the process proceeds to S124. On the other hand, if there is a feature point pair that has not been generated in the process of S113, the process proceeds to S113 in FIG.

  When the process proceeds to S124, the pre-processing unit 104 sets the conversion matrix Ri having the maximum number of votes as the estimated conversion matrix R (S124). When the process of S124 is completed, the series of processes shown in FIGS. 19 and 20 ends. In the above description, the method for generating all possible feature point pairs has been described. However, the process is modified so that feature point pairs are generated by repeating the process a preset number of times. Is also possible.

(Flow of subsequent processing)
Next, the flow of the subsequent process will be further described with reference to FIGS. FIG. 21 is a first flowchart illustrating a flow of subsequent processing executed by the information processing apparatus according to the second embodiment. FIG. 22 is a second flowchart showing the flow of subsequent processing executed by the information processing apparatus according to the second embodiment. FIG. 23 is a third flowchart showing the flow of subsequent processing executed by the information processing apparatus according to the second embodiment. The series of processing illustrated in FIGS. 21 to 23 is mainly executed by the post-processing unit 105.

  As illustrated in FIG. 21, the post-processing unit 105 selects one feature point that is not used in the calculation of the estimated transformation matrix R from the feature points of the image P1 by the function of the voting processing unit 153 (S131). Next, the post-processing unit 105 uses the function of the voting processing unit 153 to convert the coordinates of the feature points selected in the process of S131 with the estimated conversion matrix R (S132).

  Next, the post-processing unit 105 uses the function of the voting processing unit 153 to determine the feature points of the image P2 whose distance from the point after the conversion in the process of S132 is equal to or less than the threshold ThD and not used for calculating the estimated conversion matrix R It is determined whether or not there is. (S133). If the distance from the converted point is equal to or less than the threshold ThD and there is a feature point of the image P2 that is not used for calculation of the estimated conversion matrix R, the process proceeds to S134. On the other hand, if there is no feature point of the image P2 that is not used for calculation of the estimated transformation matrix R, the distance from the transformed point is equal to or less than the threshold ThD, the process proceeds to S131.

  When the process proceeds to S134, the subsequent processing unit 105 votes for the estimated transformation matrix R by the function of the voting processing unit 153 (S134). Next, the post-processing unit 105 determines whether or not all (nd) feature points have been selected in the processing of S131 by the function of the voting processing unit 153 (S135). If all (n−d) feature points have been selected, the process proceeds to S136. On the other hand, if there is an unselected feature point among the (n−d) feature points, the process proceeds to S131.

  When the process proceeds to S136, the subsequent stage processing unit 105 sets the number of votes (total number) of the estimated transformation matrix R voted in the process of S134 to Nmax by the function of the voting processing unit 153 (S136). Next, the post-processing unit 105 initializes the counter table 101d (sets the counter value to 0) (S137).

  Next, the post-processing unit 105 randomly extracts the feature points of the image P1 and the feature points of the image P2 by the function of the parameter calculation unit 151, and generates d feature point pairs (S138). Next, the post-processing unit 105 uses the function of the parameter calculation unit 151 to calculate the transformation matrix Hi using the d feature point pairs generated in the process of S138 (S139). However, the transformation matrix Hi represents a transformation matrix calculated using the i th feature point pair generated in the process of S138. When the process of S139 is completed, the process proceeds to S140 of FIG.

  When the process proceeds to S140, the post-processing unit 105 determines whether the difference between the transformation matrix Hi and the estimated transformation matrix R is within an allowable range by the function of the candidate extraction unit 152 (S140). If the difference between the transformation matrix Hi and the estimated transformation matrix R is within the allowable range, the process proceeds to S141. On the other hand, if the difference between the transformation matrix Hi and the estimated transformation matrix R is not within the allowable range, the process proceeds to S138 in FIG.

  When the process proceeds to S141, the post-processing unit 105 selects, from the feature points of the image P1, feature points that are not used for calculating the transformation matrix Hi by using the function of the voting processing unit 153 (S141). At this time, feature points are selected in ascending order of counter values in the counter table 101d. Next, the post-processing unit 105 uses the function of the voting processing unit 153 to convert the coordinates of the feature points selected in the processing of S141 using the conversion matrix Hi (S142).

  Next, the post-processing unit 105 uses the function of the voting processing unit 153 to determine the feature points of the image P2 that are not used in the calculation of the conversion matrix Hi, the distance from the point after conversion by the process of S142 is equal to or less than the threshold ThD It is determined whether or not there is (S143). If there is a feature point of the image P2 that is not used for the calculation of the transformation matrix Hi, the process proceeds to S144 when the distance from the transformed point is equal to or less than the threshold ThD. On the other hand, if there is no feature point of the image P2 that is not used for calculation of the transformation matrix Hi, the process proceeds to S146 when the distance from the transformed point is equal to or less than the threshold ThD.

  When the process proceeds to S144, the post-processing unit 105 votes for the transformation matrix Hi by the function of the voting processing unit 153 (S144). Next, the post-processing unit 105 increases the counter value of the feature point selected in the processing of S141 by the function of the voting processing unit 153 (S145). When the process of S145 is completed, the process proceeds to S149 of FIG.

  As a result of the determination in the process of S143, when the process proceeds to S146, the post-processing unit 105 calculates (nd−Nmax) by the function of the voting processing unit 153 (S146). Next, the post-processing unit 105 calculates the number of feature points that have been determined as outliers by the function of the voting processing unit 153 (S147).

  Next, the post-processing unit 105 determines whether or not the calculation result in the processing of S147 is greater than (nd−Nmax) by the function of the voting processing unit 153 (S148). If the calculation result is greater than (nd−Nmax), the process proceeds to S138 in FIG. On the other hand, if the calculation result is not greater than (nd−Nmax), the process proceeds to S149 in FIG.

  When the process proceeds to S149, the post-processing unit 105 determines whether all (Xd) feature points have been selected in the process of S141 (S149). If all (Xd) feature points have been selected, the process proceeds to S150. On the other hand, when there is an unselected feature point among (Xd) feature points, the process proceeds to S141 in FIG.

  When the process proceeds to S150, the post-processing unit 105 determines whether or not the number of votes in the transformation matrix Hi exceeds the maximum number of votes Nmax (S150). When the number of votes in the conversion matrix Hi exceeds the maximum number of votes Nmax, the process proceeds to S151. On the other hand, if the number of votes in the transformation matrix Hi does not exceed the maximum number of votes Nmax, the process proceeds to S152.

  When the process proceeds to S151, the post-processing unit 105 updates the maximum vote number Nmax with the vote number of the conversion matrix Hi (S151). When the process of S151 is completed, the process proceeds to S152. When the process proceeds to S152, the post-processing unit 105 sorts the counter table 101d in order of the number of votes (S152).

  Next, the post-processing unit 105 determines whether or not the processing after S138 has been executed T times (S153). However, T is given by the above equation (1). If the process has been executed T times, the process proceeds to S154. On the other hand, if the process has not been executed T times, the process proceeds to S138 in FIG. When the process proceeds to S154, the post-processing unit 105 determines the conversion matrix Hi having the largest number of votes as the conversion matrix H used for image processing (S154). When the process of S154 is completed, the series of processes shown in FIGS.

  The flow of processing executed by the information processing apparatus 100 has been described above. As described above, the processing load is reduced by evaluating the difference between the transformation matrix Hi and the estimated transformation matrix R in the subsequent processing and avoiding the voting process for the transformation matrix Hi whose difference is outside the allowable range. Furthermore, the processing is reduced by avoiding the voting process for the transformation matrix Hi that is unlikely to update the maximum vote count Nmax at an early stage by executing the processes in order from the feature point with the smallest count value. As a result, the transformation matrix H can be determined at high speed.

The second embodiment has been described above.
<3. Third Embodiment>
Next, a third embodiment will be described.

  In the second embodiment, the estimated transformation matrix R is calculated in the former stage process, and the useless voting process is avoided based on the difference between the estimated transformation matrix R and the transformation matrix Hi calculated for each feature point pair in the latter stage process. Was the way. On the other hand, as shown in FIG. 24, the third embodiment converts the feature points of the image P1 using the estimated conversion matrix R, and falls within an allowable range (hereinafter referred to as an inlier range) based on the converted points. A method for generating a feature point pair using the feature points of the image P2 is proposed. FIG. 24 is a diagram showing an outline of the post-stage processing according to the third embodiment.

  In the example of FIG. 24, the feature points q11, q12, and q13 of the image P1 are converted by the estimated conversion matrix R, and the coordinates of the converted points a, b, and c are obtained, respectively. In this case, a pair of each feature point of the image P2 included in the inlier range Da with the point a as a reference and the feature point q11 of the image P1 is a candidate feature point pair. In addition, a pair of each feature point of the image P2 included in the inlier range Db with the point b as a reference and the feature point q12 of the image P1 is a candidate feature point pair. Further, a pair of each feature point of the image P2 included in the inlier range Dc with the point c as a reference and the feature point q13 of the image P1 is a candidate feature point pair.

  From the feature point pair candidates extracted as described above, the feature point pairs to be subjected to the voting process are sequentially selected, and the voting process is executed. That is, after the candidates are narrowed down, a feature point pair is selected from the candidates. For this reason, the number of times the voting process is executed is reduced, and the load of the subsequent process is reduced. Hereinafter, the information processing apparatus 200 capable of executing the subsequent processing as described above will be described. The information processing apparatus 200 is an example of an information processing apparatus according to the third embodiment. Moreover, in order to avoid duplication description about the element substantially the same as the information processing apparatus 100 which concerns on said 2nd Embodiment, the same code | symbol may be attached | subjected and detailed description may be abbreviate | omitted.

[3-1. Function of information processing apparatus]
The function of the information processing apparatus 200 will be described with reference to FIG. FIG. 25 is a block diagram for explaining functions of the information processing apparatus according to the third embodiment. In the following description, FIGS. 26 to 30 will be referred to as appropriate.

As illustrated in FIG. 25, the information processing apparatus 200 includes an imaging unit 102, a feature point extraction unit 103, a pre-processing unit 104, an image processing unit 106, a storage unit 201, and a post-processing unit 205.
Note that the function of the storage unit 201 can be realized by using the above-described RAM 906, the storage unit 920, or the like. The functions of the feature point extraction unit 103, the pre-processing unit 104, the post-processing unit 205, and the image processing unit 106 can be realized using the above-described CPU 902 or the like.

(Storage unit 201)
The storage unit 201 stores feature point information 101a, model information 101b, a vote count table 101c, allowable range information 201a, and a pair map list 201b. The storage unit 201 may store a counter table 101d. The allowable range information 201a is information for determining an inlier range used when extracting feature amount pair candidates. The pair map list 201b is a list in which feature point pair candidates are described.

Here, the allowable range information 201a will be further described with reference to FIG. FIG. 26 is a diagram illustrating an example of allowable range information according to the third embodiment.
As shown in FIG. 26, the allowable range information 201a includes the type of conversion parameter and the allowable range of the conversion parameter. The example of FIG. 26 relates to the transformation parameters w11, w12,..., W23 of the affine transformation given by the above equation (2). For example, the allowable range of the conversion parameter w11 is set to [w11−t11, w11 + t11] (that is, the allowable range of w11−t11 ≦ w11 ≦ w11 + t11). Similarly, the allowable ranges of the conversion parameters w12, w13,..., W23 are also set. However, t11, t12,..., T23 are preset setting values that determine the width of the allowable range.

  For example, when the feature point q11 of the image P1 is converted by the estimated conversion matrix R, when the conversion parameters r11, r12,..., R23 of the estimated conversion matrix R are moved by the above width, the area (inlier range) taken by the converted points is taken. ) Is decided. In the example of FIG. 24, the inlier range Da corresponding to the feature point q11 is determined by moving the conversion parameters w11, w12,..., W23 of the estimated conversion matrix R with the above width. Note that it is possible to transform the circular area or rectangular area having a radius set in advance with reference to the converted point as an inlier range. In this case, information indicating the radius and the side length is included as the allowable range information 201a. It is set in advance.

Next, the pair map list 201b will be further described with reference to FIG. FIG. 27 is a diagram showing an example of a pair map list according to the third embodiment.
As shown in FIG. 27, the pair map list 201b is a list in which information for specifying candidate feature point pairs and the feature points of the images P1 and P2 included in the feature point pairs are described. In the example of FIG. 27, candidate feature point pairs L1, L2,..., L8 are shown. For example, the feature point pair L1 includes a feature point q11 of the image P1 and a feature point q21 of the image P2. This means that the feature point q21 is a feature point included in the inlier range based on the point obtained by transforming the feature point q11 with the estimated transformation matrix R. The same applies to the other feature point pairs L2, L3,.

(Imaging unit 102, feature point extraction unit 103, pre-processing unit 104, image processing unit 106)
Refer to FIG. 25 again. The imaging unit 102, the feature point extraction unit 103, the pre-processing unit 104, and the image processing unit 106 are substantially the same as those in the second embodiment. Information on the estimated transformation matrix R calculated by the upstream processing unit 104 is stored in the storage unit 201 and input to the downstream processing unit 205. Further, information on the transformation matrix determined by the post-processing unit 205 is input to the image processing unit 106.

(Post-processing unit 205)
The post-processing unit 205 uses the feature points more than the number of feature points used for calculation of the estimated transformation matrix among the feature points included in the feature point information 101a stored in the storage unit 201, and the image P1, A transformation matrix used for the alignment of P2 is calculated. At this time, the post-processing unit 205 narrows down feature point pairs used for calculation of the transformation matrix based on the estimated transformation matrix. In order to execute the post-stage processing, the post-stage processing unit 205 includes a candidate extraction unit 251, a parameter calculation unit 252, and a voting processing unit 253 as illustrated in FIG.

  The candidate extraction unit 251 refers to the feature point information 101a stored in the storage unit 201, selects one feature point extracted from the image P1, and converts the coordinates of the selected feature point using the estimated conversion matrix R. Further, the candidate extraction unit 251 calculates an inlier range based on the converted coordinates based on the allowable range information 201a. Further, the candidate extraction unit 251 extracts feature points of the image P2 included in the inlier range.

  Then, the candidate extraction unit 251 generates feature point pair candidates by combining the feature points of the selected image P1 and the extracted feature points of the image P2. Further, the candidate extraction unit 251 records the generated feature point pair candidate information in the pair map list 201b. In addition, the candidate extraction unit 251 selects one feature point extracted from the image P1, and generates a feature point pair candidate corresponding to the selected feature point. By repeatedly executing these processes, the candidate extraction unit 251 generates a pair map list 201b.

  The parameter calculation unit 252 selects d feature point pairs from the feature point pair candidates recorded in the pair map list 201b, and calculates a transformation matrix included in the transformation model. Also, the parameter calculation unit 252 records the selected feature point pair information and the calculated conversion parameter of the conversion matrix in the vote count table 101c. The voting processing unit 253 sequentially selects feature points that are not used for calculation of the transformation matrix from the feature points extracted from the image P1, and determines whether or not the selected feature points are inliers. The transformation matrix calculated by the parameter calculation unit 252 is input to the voting processing unit 253.

  For example, the voting processing unit 253 converts the feature point of the selected image P1 using the input conversion matrix, and an allowable range based on the converted point (a range different from the above inlier range may be used). It is determined whether or not the feature point of the image P2 exists in the image. When the feature point of the image P2 exists within the allowable range, the voting processing unit 253 determines that the feature point of the selected image P1 is an inlier.

  Each time the voting processing unit 253 determines to be an inlier, it votes for the conversion parameter of the input conversion matrix. That is, when it is determined as an inlier, the voting processing unit 253 increases the number of votes set in the vote number table 101c corresponding to the conversion parameter of the input conversion matrix by a set number (for example, 1). When the above determination is completed for all feature points, the voting processing unit 253 notifies the parameter calculation unit 252 of the end of the voting processing for the input transformation matrix. Upon receiving this notification, the parameter calculation unit 252 selects an unselected feature point pair and calculates a transformation matrix included in the transformation model.

  The parameter calculation unit 252 records the selected feature point pair information and the calculated conversion parameter of the conversion matrix in the vote count table 101c. Further, the conversion matrix calculated by the parameter calculation unit 252 is input to the voting processing unit 253. The voting processing unit 253 sequentially selects feature points that are not used for calculation of the input transformation matrix, and determines whether or not the selected feature points are inliers. Each time the voting processing unit 253 determines to be an inlier, the voting processing unit 253 votes for the conversion parameters of the input conversion matrix.

  Similarly to the above, the parameter calculation unit 252 sequentially selects feature point pairs from the pair map list 201b and calculates a transformation matrix. In addition, the voting processing unit 253 performs voting processing on the input conversion matrix. When these processes are executed a preset number of times (for example, T given by the above equation (1)), or when the processes are executed for all feature point pairs in the pair map list 201b, the post-processing unit 205 Extracts the transformation matrix with the largest number of votes.

(Post-processing: Determination of transformation matrix)
Here, the subsequent processing will be further described with reference to FIGS. 28 to 30. FIG. 28 is a first diagram for explaining a pair map list generation method according to the third embodiment. FIG. 29 is a second diagram for explaining a pair map list generation method according to the third embodiment. FIG. 30 is a diagram for explaining post-processing (conversion matrix determination) according to the third embodiment.

  Consider a method of generating a pair map list 201b of feature point pairs including feature points q11, q12,..., Q16 of the image P1. Hereinafter, points obtained by transforming the feature points q11, q12,..., Q16 by the estimated transformation matrix R are denoted as Q11, Q12,.

  In the example of FIG. 28, the feature point q21 of the image P2 is included in the inlier range of the point Q11. Therefore, a pair of the feature point q11 of the image P1 and the feature point q21 of the image P2 is recorded in the pair map list 201b as a feature point pair candidate. Similarly, since the feature point q22 of the image P2 is included in the inlier range of the point Q12, a pair of the feature point q12 of the image P1 and the feature point q22 of the image P2 is included in the pair map list 201b as a candidate feature point pair. To be recorded.

  The inlier range of the point Q13 includes the feature point q23 of the image P2. This feature point q23 is also included in the inlier range of points Q15 and Q16. In this case, a pair of feature points q13 and q23, a pair of feature points q15 and q23, and a pair of feature points q16 and q23 are recorded in the pair map list 201b. Similarly, since the feature point q26 of the image P2 is included in the inlier range of the points Q15 and Q16, the pair of feature points q15 and q26 and the pair of feature points q16 and q26 are recorded in the pair map list 201b.

  In the example of FIG. 28, the feature points q24 and q27 of the image P2 are not included in any inlier range. Therefore, a pair related to the feature points q24 and q27 is not recorded in the pair map list 201b. In addition, the feature point of the image P2 is not included in the inlier range of the point Q14. Therefore, a pair related to the feature point q14 is not recorded in the pair map list 201b. By executing the above processing, the correspondence between feature points as shown in FIG. 29 is obtained.

  As shown in FIG. 29, the feature point q11 (corresponding to the point Q11) of the image P1 and the feature point q21 of the image 2 form a pair (feature point pair L1). Further, the feature point q12 (corresponding to the point Q12) of the image P1 and the feature point q22 of the image 2 form a pair (feature point pair L2). Further, the feature point q13 (corresponding to the point Q13) of the image P1 and the feature point q23 of the image 2 form a pair (feature point pair L3). On the other hand, the feature point q14 (corresponding to the point Q14) of the image P1 is not paired.

  Further, the feature point q15 (corresponding to the point Q15) of the image P1 and the feature point q23 of the image 2 form a pair (feature point pair L4). Further, the feature point q15 (corresponding to the point Q15) of the image P1 forms a pair (feature point pair L5) with the feature point q25 of the image 2. The feature point q15 (corresponding to the point Q15) of the image P1 also forms a pair (feature point pair L6) with the feature point q26 of the image 2.

  Also, the feature point q16 (corresponding to the point Q16) of the image P1 and the feature point q23 of the image 2 are paired (feature point pair L7). Further, the feature point q16 (corresponding to the point Q16) of the image P1 also forms a pair (feature point pair L8) with the feature point q26 of the image 2. On the other hand, the feature points q24 and q27 of the image P2 are not paired. When the correspondences between these feature points are collected, a pair map list 201b as shown in FIG. 30 is obtained.

  When the pair map list 201b is obtained, the parameter calculation unit 252 uses d feature point pairs (L1, L2, and L3 in the example of FIG. 30) extracted from the pair map list 201b as shown in FIG. Calculate the transformation matrix. Then, the voting processing unit 253 performs voting processing on the transformation matrix calculated by the parameter calculation unit 252. These processes are repeatedly executed, and a conversion matrix having the maximum number of votes is finally determined as a conversion matrix H used for image processing. As described above, by narrowing down the number of feature point pairs, the number of voting processes is reduced, and the processing load is reduced.

The function of the information processing apparatus 200 has been described above.
[3-2. Process flow]
Next, the flow of processing executed by the information processing apparatus 200 will be described with reference to FIGS. Of the processes executed by the information processing apparatus 200, the pre-stage process is substantially the same as in the second embodiment. Therefore, a detailed description of the flow of the upstream processing is omitted, and the flow of the downstream processing is described.

(Flow of subsequent processing)
With reference to FIG. 31 and FIG. 32, the flow of the post-stage processing will be described. FIG. 31 is a first flowchart illustrating a flow of subsequent processing executed by the information processing apparatus according to the third embodiment. FIG. 32 is a second flowchart showing the flow of subsequent processing executed by the information processing apparatus according to the third embodiment. The series of processing shown in FIGS. 31 and 32 is mainly executed by the post-processing unit 205.

  As illustrated in FIG. 31, the post-processing unit 205 calculates the maximum number of votes Nmax for the estimated conversion matrix R by the function of the voting processing unit 253 (S201). The process of S201 corresponds to the processes of S131 to S136 shown in FIG. Next, the post-processing unit 205 generates the pair map list 201b by the function of the candidate extraction unit 251 (S202).

  Next, the post-processing unit 205 uses the function of the parameter calculation unit 252 to select d feature point pairs from the pair map list 201b (S203). Next, the post-processing unit 205 uses the function of the parameter calculation unit 252 to calculate the transformation matrix Hi using the d feature point pairs selected in the processing of S203 (S204). However, the transformation matrix Hi represents a transformation matrix calculated using the i-th feature point pair generated in the process of S203.

  Next, the post-processing unit 205 uses the function of the voting processing unit 253 to select feature points that are not used in the calculation of the transformation matrix Hi from the feature points of the image P1 (S205). Next, the post-processing unit 205 uses the function of the voting processing unit 253 to convert the coordinates of the feature points selected in the processing of S205 with the conversion matrix Hi (S206). When the process of S206 is completed, the process proceeds to S207 of FIG.

  When the process proceeds to S207, the post-processing unit 205 does not use the calculation of the conversion matrix Hi in which the distance from the point after conversion by the process of S206 is equal to or less than the threshold ThD by the function of the voting processing unit 253. It is determined whether there is a feature point of the image P2 (S207). If there is a feature point of the image P2 that is not used for the calculation of the transformation matrix Hi, the process proceeds to S208 when the distance from the transformed point is equal to or less than the threshold ThD. On the other hand, if there is no feature point of the image P2 that is not used for calculation of the transformation matrix Hi, the distance from the transformed point is equal to or less than the threshold ThD, the process proceeds to S209.

  When the process proceeds to S208, the post-processing unit 205 votes for the transformation matrix Hi by the function of the voting processing unit 253 (S208). When the process of S208 is completed, the process proceeds to S212.

  When the process proceeds to S209, the post-processing unit 205 calculates (nd−Nmax) by the function of the voting processing unit 253 (S209). Note that n is the number of feature points extracted from the image P1. Next, the post-processing unit 205 calculates the number of feature points that have been determined as outliers up to the present time by the function of the voting processing unit 253 (S210).

  Next, the post-processing unit 205 determines whether or not the calculation result in the processing of S210 is greater than (nd−Nmax) by the function of the voting processing unit 253 (S211). If the calculation result is greater than (nd−Nmax), the process proceeds to S203 in FIG. On the other hand, if the calculation result is not greater than (nd−Nmax), the process proceeds to S212.

  When the process proceeds to S212, the post-processing unit 205 determines whether all (n−d) feature points have been selected in the process of S205 by the function of the voting processing unit 253 (S212). If all (n−d) feature points have been selected, the process proceeds to S213. On the other hand, when there is an unselected feature point among the (nd) feature points, the process proceeds to S205 in FIG.

  When the process proceeds to S213, the post-processing unit 205 determines whether or not the number of votes in the conversion matrix Hi exceeds the maximum number of votes Nmax by the function of the vote processing unit 253 (S213). If the number of votes in the conversion matrix Hi exceeds the maximum number of votes Nmax, the process proceeds to S214. On the other hand, if the number of votes in the transformation matrix Hi does not exceed the maximum number of votes Nmax, the process proceeds to S215.

  When the process proceeds to S214, the post-processing unit 205 updates the maximum vote number Nmax with the vote number of the conversion matrix Hi (S214). When the process of S214 is completed, the process proceeds to S215. When the process proceeds to S215, the post-processing unit 205 determines whether or not the process after S203 has been performed T times (for example, T given by the above equation (1)) (S215). If the process has been executed T times, the process proceeds to S216. On the other hand, if the process has not been executed T times, the process proceeds to S203 in FIG.

  When the process proceeds to S216, the post-processing unit 205 outputs the conversion matrix Hi having the maximum number of votes as the conversion matrix H used for image processing (S216). When the process of S216 is completed, the series of processes shown in FIGS. 31 and 32 is finished. In the above description, an example in which the processing after S203 is executed T times is shown, but it is also possible to modify the processing after S203 until the feature point pairs in the pair map list 201b disappear.

(Processing flow for generating a pair map list)
Here, with reference to FIG. 33, a flow of processing relating to generation of the pair map list 201b in the latter-stage processing described above will be described. FIG. 33 is a flowchart showing a flow of processing relating to generation of a pair map list among subsequent processing executed by the information processing apparatus according to the third embodiment. The series of processing illustrated in FIG. 33 corresponds to the processing of S202 described above, and is mainly executed by the candidate extraction unit 251.

  As shown in FIG. 33, the candidate extraction unit 251 selects one feature point of the image P1 (S221). Next, the candidate extraction unit 251 converts the feature points selected in the process of S221 with the estimated conversion matrix R (S222). Next, the candidate extraction unit 251 extracts all the feature points of the image P2 included in the inlier range of the points after the conversion in the process of S222 (S223).

  Next, the candidate extraction unit 251 describes a pair of the feature point extracted in the process of S223 and the feature point selected in the process of S221 in the pair map list 201b (S224). Next, the candidate extraction unit 251 determines whether all feature points of the image P1 have been selected (S225). When all the feature points of the image P1 have been selected, the series of processes shown in FIG. 33 ends. On the other hand, when there is an unselected feature point among the feature points of the image P1, the process proceeds to S221.

  The flow of processing executed by the information processing apparatus 200 has been described above. As described above, by narrowing down feature point pairs based on the inlier range, it is possible to reduce the number of times the voting process is executed, and to reduce the calculation load of the transformation matrix H used for image processing.

The third embodiment has been described above.
<4. Combination of second and third embodiments>
Here, a mechanism for combining the method according to the second embodiment and the method according to the third embodiment will be described with reference to FIGS. 34 to 36.

  FIG. 34 is a first diagram for explaining a combination of the second and third embodiments. FIG. 35 is a second diagram for explaining a combination of the second and third embodiments. FIG. 36 is a flowchart showing the flow of subsequent processing according to the combination of the second and third embodiments.

  As already described, according to the method according to the third embodiment, the processing load is reduced by using the pair map list 201b. For example, when there are feature points q21 and q22 included in the inlier range of the point Q11, a pair map list 201b as shown in FIG. 34 is obtained. As in this example, when the number of feature points included in the inlier range is small, the number of feature point pairs described in the pair map list 201b is small, so the effect of reducing the processing load is great.

  On the other hand, as shown in FIG. 35, when the number of feature points included in the inlier range is large (in the example of FIG. 35, feature points q21, q22,..., Q25 are included in the inlier range), the pair map list 201b The number of feature point pairs to be described increases. That is, when the feature points are locally dense, the processing load reduction effect obtained by using the pair map list 201b is reduced. In such a case, there is a possibility that a larger processing load reduction effect can be obtained by applying the method according to the second embodiment.

  Therefore, in the situation shown in FIG. 34, the method according to the third embodiment is applied, and in the situation shown in FIG. 35, the method according to the second embodiment is applied to improve the processing load reduction effect. Propose. When this combination method is applied, the processing flow is as shown in FIG. Here, the execution subject of the process illustrated in FIG. 36 is referred to as a post-processing unit 300.

  As illustrated in FIG. 36, the post-processing unit 300 converts the feature points of the image P1 using the estimated conversion matrix R (S301). Next, the post-processing unit 300 calculates the number Z of feature points included in the inlier range based on the point converted by the process of S301 (S302).

  Next, the post-processing unit 300 determines whether or not the number Z calculated in the process of S302 is greater than a preset threshold ThZ (S303). If Z ≧ ThZ, the process proceeds to S304. On the other hand, if Z <ThZ, the process proceeds to S305.

  When the process proceeds to S304, the post-processing unit 300 executes the post-process of the second embodiment (S304). On the other hand, when the process proceeds to S305, the post-processing unit 300 performs the post-process of the third embodiment (S305). When the process of S304 or S305 is completed, the series of processes illustrated in FIG. 36 ends.

  As described above, a combination of the post-stage process of the second embodiment and the post-stage process of the third embodiment according to the number of feature points included in the inlier range realizes a suitable process according to the situation, which is larger. An effect of reducing the processing load can be obtained.

The combination of the second and third embodiments has been described above.
<5. Addendum>
The following additional notes are disclosed with respect to the embodiment described above.

(Supplementary Note 1) A first feature point included in the first image, a second feature point included in the second image, and a conversion model for converting a point on the first image into a corresponding point on the second image; A storage unit for storing
An estimated value of a parameter included in the conversion model is calculated using a set of the first and second feature points having a feature amount larger than the set first threshold, and the feature is smaller than the second threshold smaller than the first threshold. A set of the first feature point and the second feature point that is large and has an error with respect to the conversion model to which the estimated value is applied is within a set range, and for each selected set, the parameter and An information processing apparatus comprising: an arithmetic unit that calculates an evaluation value of the parameter and determines the parameter based on the evaluation value.

(Additional remark 2) The said calculating part is
Selecting a set of the first feature point and the second feature point from a set of the first and second feature points having a feature amount larger than the second threshold, and calculating the parameter for each selected set; The information processing apparatus according to claim 1, wherein the error is determined to be within the set range when a difference between the parameter and the estimated value is smaller than a set allowable value.

(Supplementary note 3)
From the set of the first and second feature points extracted so that the distance between the extracted first feature points and the distance between the extracted second feature points are larger than the first set value, the second feature points are extracted. The information processing apparatus according to appendix 1 or 2, wherein a set of the first and second feature points having a feature amount larger than a threshold is specified.

(Supplementary Note 4)
In the process of calculating the evaluation value of the parameter, the first feature point that is not used in the calculation of the parameter is selected, the first feature point is converted by the conversion model to which the parameter is applied, and the converted point When the distance between the second feature point and the second feature point is smaller than the second set value, the first vote number for the parameter is increased, and the cumulative value of the first vote number for all the selected first feature points is the parameter. The information processing apparatus according to any one of supplementary notes 1 to 3, wherein the evaluation value is set as the evaluation value.

(Supplementary Note 5)
In the process of calculating the evaluation value of the parameter, when increasing the first vote number for the parameter, the second vote number for the first feature point used for calculating the parameter is increased, and the first feature point The information processing apparatus according to claim 4, wherein the first feature point is selected in ascending order of the second number of votes when selecting.

(Additional remark 6) The said calculating part is
The first feature point that is not used for calculation of the estimated value is selected, the first feature point is converted by the conversion model to which the estimated value is applied, and the distance between the converted point and the second feature point is converted When the value is smaller than the second set value, the third vote number for the first feature point is increased, and the cumulative value of the third vote number for all the selected first feature points is set as an upper limit value, When the second vote number is larger than the upper limit value, the upper limit value is updated with the second vote number,
In the process of calculating the evaluation value of the parameter, the number of the first feature points at which the distance between the converted point by the parameter and the second feature point is larger than the second set value reaches the upper limit value. The information processing apparatus according to appendix 5, wherein the calculation of the evaluation value for the parameter is avoided when the parameter is calculated.

(Supplementary note 7)
A set of the first feature point and the second feature point is selected from the set of the first and second feature points having a feature amount larger than the second threshold value, and the estimated value is applied to each selected set. The first feature point is converted by the converted model, and it is determined whether or not the second feature point is included in a region set based on the converted point, and the second feature point is included in the region. The information processing apparatus according to any one of appendices 1 to 4, wherein the information is determined to be within the set range.

(Supplementary Note 8) A first feature point included in the first image, a second feature point included in the second image, and a conversion model for converting a point on the first image into a corresponding point on the second image A computer capable of acquiring information on the first feature point, the second feature point, and the conversion model from a storage unit that stores
An estimated value of a parameter included in the conversion model is calculated using a set of the first and second feature points having a feature amount larger than the set first threshold, and the feature is smaller than the second threshold smaller than the first threshold. A set of the first feature point and the second feature point that is large and has an error with respect to the conversion model to which the estimated value is applied is within a set range, and for each selected set, the parameter and A parameter determination method that calculates an evaluation value of the parameter and determines the parameter based on the evaluation value.

(Supplementary Note 9) A first feature point included in the first image, a second feature point included in the second image, and a conversion model for converting a point on the first image into a corresponding point on the second image; A computer capable of acquiring information on the first feature point, the second feature point, and the conversion model from a storage unit storing
An estimated value of a parameter included in the conversion model is calculated using a set of the first and second feature points having a feature amount larger than the set first threshold, and the feature is smaller than the second threshold smaller than the first threshold. A set of the first feature point and the second feature point that is large and has an error with respect to the conversion model to which the estimated value is applied is within a set range, and for each selected set, the parameter and A program that calculates an evaluation value of the parameter and determines the parameter based on the evaluation value.

(Supplementary Note 10) A first feature point included in the first image, a second feature point included in the second image, and a conversion model for converting a point on the first image into a corresponding point on the second image; A computer capable of acquiring information on the first feature point, the second feature point, and the conversion model from a storage unit storing
An estimated value of a parameter included in the conversion model is calculated using a set of the first and second feature points having a feature amount larger than the set first threshold, and the feature is smaller than the second threshold smaller than the first threshold. A set of the first feature point and the second feature point that is large and has an error with respect to the conversion model to which the estimated value is applied is within a set range, and for each selected set, the parameter and A computer-readable recording medium storing a program for executing a process of calculating an evaluation value of the parameter and determining the parameter based on the evaluation value.

(Supplementary Note 11) A first feature point included in the first image, a second feature point included in the second image, and a conversion model for converting a point on the first image into a corresponding point on the second image; A storage unit for storing
An estimated value of a parameter included in the conversion model is calculated using a part of the first and second feature points, and the first feature point and the second feature point are targeted for all the first and second feature points. A set of feature points is sequentially selected and the parameter is calculated for each set. If the difference between the calculated parameter and the estimated value is within a set range, an evaluation value of the parameter is calculated, and the evaluation An information processing apparatus comprising: an arithmetic unit that identifies the parameter having a maximum value.

(Supplementary Note 12) The computer selects a set of the first feature point and the second feature point from a set of the first and second feature points having a feature amount larger than the second threshold, and the selected set The parameter determination method according to claim 8, wherein the parameter is calculated every time, and the error is determined to be within the set range when a difference between the parameter and the estimated value is smaller than a set allowable value.

(Supplementary Note 13) The first and second features extracted by the computer so that the distance between the extracted first feature points and the distance between the extracted second feature points are larger than a first set value. The parameter determination method according to appendix 8 or 12, wherein a set of the first and second feature points having a feature amount larger than the second threshold is specified from a set of points.

(Additional remark 14) In the process which calculates the evaluation value of the said parameter, the said computer selects the said 1st feature point which is not used for the calculation of the said parameter, The said 1st feature point by the said conversion model which applied the said parameter And when the distance between the converted point and the second feature point is smaller than the second set value, the first vote number for the parameter is increased, and the first feature points for all the selected first feature points are The parameter determination method according to any one of appendices 8, 12, and 13, wherein a cumulative value of the number of votes is the evaluation value for the parameter.

(Supplementary Note 15) In the process of calculating the evaluation value of the parameter, when the computer increases the first vote number for the parameter, the second vote number for the first feature point used for calculating the parameter is calculated. The parameter determination method according to claim 14, wherein when the first feature point is selected, the first feature point is selected in ascending order of the second vote number.

(Supplementary Note 16) The computer
The first feature point that is not used for calculation of the estimated value is selected, the first feature point is converted by the conversion model to which the estimated value is applied, and the distance between the converted point and the second feature point is converted When the value is smaller than the second set value, the third vote number for the first feature point is increased, and the cumulative value of the third vote number for all the selected first feature points is set as an upper limit value, When the second vote number is larger than the upper limit value, the upper limit value is updated with the second vote number,
In the process of calculating the evaluation value of the parameter, the number of the first feature points at which the distance between the converted point by the parameter and the second feature point is larger than the second set value reaches the upper limit value. The parameter determination method according to supplementary note 15, wherein calculation of the evaluation value for the parameter is avoided when the parameter is calculated.

(Supplementary Note 17) The computer selects a set of the first feature point and the second feature point from a set of the first and second feature points having a feature amount larger than the second threshold, and the selected set Each time, the first feature point is converted by the conversion model to which the estimated value is applied, and it is determined whether or not the second feature point is included in an area set based on the converted point. The parameter determination method according to any one of appendices 8 and 12 to 14, wherein the error is determined to be within the set range when the second feature point is included.

DESCRIPTION OF SYMBOLS 10 Information processing apparatus 11 Memory | storage part 12 Calculation part q1, q11, q12, q13, q14, q15, q16 1st feature point q2, q21, q22, q23, q24, q25, q26 2nd feature point A, A21, A22 Parameter A1 Estimated value C11, C12, C21, C22 Object P1 First image P2 Second image TM Conversion model Th1 First threshold Th2 Second threshold

Claims (9)

A memory for storing a first feature point of the first image, a second feature point of the second image, and a conversion model for converting a point on the first image into a corresponding point on the second image And
An estimated value of a parameter included in the conversion model is calculated using a set of the first and second feature points having a feature amount larger than the set first threshold, and the feature is smaller than the second threshold smaller than the first threshold. A set of the first feature point and the second feature point that is large and has an error with respect to the conversion model to which the estimated value is applied is within a set range, and for each selected set, the parameter and An information processing apparatus comprising: an arithmetic unit that calculates an evaluation value of the parameter and determines the parameter based on the evaluation value. The computing unit is
Selecting a set of the first feature point and the second feature point from a set of the first and second feature points having a feature amount larger than the second threshold, and calculating the parameter for each selected set; The information processing apparatus according to claim 1, wherein the error is determined to be within the set range when a difference between the parameter and the estimated value is smaller than a set allowable value. The computing unit is
From the set of the first and second feature points extracted so that the distance between the extracted first feature points and the distance between the extracted second feature points are larger than the first set value, the second feature points are extracted. The information processing apparatus according to claim 1 or 2, wherein a set of the first and second feature points having a feature amount larger than a threshold value is specified. The computing unit is
In the process of calculating the evaluation value of the parameter, the first feature point that is not used in the calculation of the parameter is selected, the first feature point is converted by the conversion model to which the parameter is applied, and the converted point When the distance between the second feature point and the second feature point is smaller than the second set value, the first vote number for the parameter is increased, and the cumulative value of the first vote number for all the selected first feature points is the parameter. The information processing apparatus according to any one of claims 1 to 3, wherein the information value is an evaluation value of the information. The computing unit is
In the process of calculating the evaluation value of the parameter, when increasing the first vote number for the parameter, the second vote number for the first feature point used for calculating the parameter is increased, and the first feature point The information processing apparatus according to claim 4, wherein the first feature point is selected in ascending order of the second vote number. The computing unit is
The first feature point that is not used for calculation of the estimated value is selected, the first feature point is converted by the conversion model to which the estimated value is applied, and the distance between the converted point and the second feature point is converted When the value is smaller than the second set value, the third vote number for the first feature point is increased, and the cumulative value of the third vote number for all the selected first feature points is set as an upper limit value, When the second vote number is larger than the upper limit value, the upper limit value is updated with the second vote number,
In the process of calculating the evaluation value of the parameter, the number of the first feature points at which the distance between the converted point by the parameter and the second feature point is larger than the second set value reaches the upper limit value. The information processing apparatus according to claim 5, wherein calculation of the evaluation value with respect to the parameter is avoided when the parameter is set. The computing unit is
A set of the first feature point and the second feature point is selected from the set of the first and second feature points having a feature amount larger than the second threshold value, and the estimated value is applied to each selected set. The first feature point is converted by the converted model, and it is determined whether or not the second feature point is included in a region set based on the converted point, and the second feature point is included in the region. The information processing apparatus according to claim 1, wherein the error is determined to be within the set range. A memory for storing a first feature point of the first image, a second feature point of the second image, and a conversion model for converting a point on the first image into a corresponding point on the second image A computer capable of acquiring information on the first feature point, the second feature point, and the conversion model from
An estimated value of a parameter included in the conversion model is calculated using a set of the first and second feature points having a feature amount larger than the set first threshold, and the feature is smaller than the second threshold smaller than the first threshold. A set of the first feature point and the second feature point that is large and has an error with respect to the conversion model to which the estimated value is applied is within a set range, and for each selected set, the parameter and A parameter determination method that calculates an evaluation value of the parameter and determines the parameter based on the evaluation value. A memory for storing a first feature point of the first image, a second feature point of the second image, and a conversion model for converting a point on the first image into a corresponding point on the second image A computer capable of acquiring information on the first feature point, the second feature point, and the conversion model from
An estimated value of a parameter included in the conversion model is calculated using a set of the first and second feature points having a feature amount larger than the set first threshold, and the feature is smaller than the second threshold smaller than the first threshold. A set of the first feature point and the second feature point that is large and has an error with respect to the conversion model to which the estimated value is applied is within a set range, and for each selected set, the parameter and A program that calculates an evaluation value of the parameter and determines the parameter based on the evaluation value.

Images