JP6665488B2 - Image processing method and image processing apparatus - Google Patents

Description

  The present invention relates to an image processing method and an image processing apparatus for detecting an object corresponding to a template image in an input image by comparing the input image with a predetermined template image.

  Template matching for detecting an image (template image) of an object registered in advance from images (query images) captured by a camera is a basic technology of image processing, and is widely used in industry. For example, rotation matching for estimating an angle parameter consisting of in-plane rotation using template matching is used for many applications such as bin picking and appearance inspection by a robot in a factory. For this reason, making the rotation matching fast and robust is an important issue in practical applications.

  Rotational matching can be performed at high speed using a technique such as SIFT (Scale-Invariant Feature Transform), but there is a problem that it is difficult to apply the technique to a textureless target having few patterns on the surface. On the other hand, several methods based on the correlation between a template image and a query image effective for a textureless target have been proposed.

  For example, NCC (Normalized Cross-Correlation) is widely used because it is robust against constant brightness fluctuation and noise. In the NCC, robustness can be improved with respect to a method in which an inner product value is set to a similarity by setting a value normalized by a norm of a query image in a convolution area as a similarity. However, when applying the NCC to the rotation matching, it is necessary to calculate the similarity between the position and angle posture parameters by repeating the correlation calculation between the query image and the template image rotated at a fixed rotation pitch a plurality of times. In order to increase the accuracy of estimating the detection rate and the angle parameter, it is necessary to reduce the rotation pitch. Therefore, there is a problem that the number of necessary convolution operations increases and the processing time increases.

  RIPOC (Rotation Invariant Phase Only Correlation) using Log-Polar transformation is known as a technique for speeding up rotation matching. RIPOC utilizes the fact that the power spectra of the template image and the query image do not depend on translation, and uses the POC to determine the rotational deviation of the log-polar image of the power spectrum. Estimate the translation by POC. This method is very efficient because rotation matching can be performed by two matchings, one for estimating rotation and the other for estimating translation. However, since the RIPOC discards the phase component that characterizes the image when estimating the rotation angle, there is a problem that the detection tends to fail if the query image contains many frequency components similar to the template image.

  An eigenvalue template method has been proposed as a method that can solve the problem of the above-described method (for example, see Patent Document 1). In the eigenvalue template method, a new template image (eigenvalue template) obtained by compressing information of a template image group by singular value decomposition (SVD) instead of matching a plurality of template images one by one with respect to a query image is used. Image) to efficiently calculate the similarity. Also, unlike the RIPOC, the eigenvalue template method is robust to the influence of the background because it calculates the similarity without discarding important information.

Patent No. 5317250

  The eigenvalue template method described above is roughly divided into a process of obtaining a correlation value (response) between a query image and each eigenvalue template image, and a similarity of posture parameters by a product-sum operation of the correlation value and an eigenfunction (basis function). Is calculated. However, there is room for improvement because the load of the calculation amount of the product-sum operation for calculating the similarity from the response of the query image to the eigenvalue template image is relatively high and the processing time tends to be long.

  The present invention has been made in view of the above points, and an object of the present invention is to provide an image processing method and an image processing apparatus capable of improving the eigenvalue template method and performing a similarity calculation at a higher speed. I do.

In order to achieve the above object, an image processing method according to a first aspect of the present invention is a method for detecting an object corresponding to a template image in an input image by comparing an input image with a predetermined template image. So,
A first step of frequency-decomposing the template image in a rotational direction based on a predetermined number of frequency components from a low frequency to a high frequency belonging to a predetermined frequency range and calculating the same number of eigenvalue template images as the predetermined number of frequency components (S100, S110),
A second step (S210) of calculating a correlation value with the input image for each eigenvalue template image by performing a convolution operation on the input image and the eigenvalue template image;
A third step (S240) of performing an FFT operation on the calculated correlation value based on a predetermined rotation direction to calculate a similarity for each rotation angle;
A fourth step (S250, S260) of calculating the rotation angle of the target object in the input image from the similarity for each rotation angle.

Further, the image processing apparatus according to the second invention is an apparatus for detecting a target object corresponding to the template image in the input image by comparing the input image with a predetermined template image,
The same number of eigenvalue template images as the predetermined number of frequency components are calculated by performing frequency decomposition on the template image based on a predetermined number of frequency components from a low frequency to a high frequency belonging to a predetermined frequency range in a rotational direction. A storage unit (21);
A correlation value calculation unit (S210) for calculating a correlation value with the input image for each eigenvalue template image by performing a convolution operation on the input image and the eigenvalue template image stored in the storage unit;
A similarity calculation unit (S240) that performs an FFT calculation process on the calculated correlation value based on a predetermined rotation direction to calculate a similarity for each rotation angle;
A rotation angle calculation unit (S250, S260) for calculating the rotation angle of the target object in the input image from the similarity for each rotation angle.

  According to the above-described image processing method and image processing apparatus, a predetermined number of template images are frequency-decomposed in the rotational direction based on a predetermined number of frequency components from a low frequency to a high frequency belonging to a predetermined frequency range. As many eigenvalue template images as frequency components are calculated. For this reason, in order to calculate the similarity, the sum of the correlation value calculated by the convolution operation of the eigenvalue template image and the input image is calculated by performing the product-sum operation with the frequency basis function having a predetermined number of frequency components as elements. , FFT operation processing, it is possible to obtain an equivalent result. The FFT operation processing can significantly reduce the operation time as compared with the product-sum operation of the correlation value and the frequency basis function. Therefore, it is possible to calculate the similarity at a higher speed than the conventional eigenvalue template method while maintaining the detection rate equivalent to the conventional eigenvalue template method.

  The reference numbers in the parentheses are merely an example of a correspondence relationship with a specific configuration in the embodiment described later in order to facilitate understanding of the present invention, and do not limit the scope of the present invention. Not intended.

  Further, technical features described in each claim of the claims other than the above-described features will be apparent from the description of the embodiments and the accompanying drawings described later.

FIG. 1 is a configuration diagram illustrating an overall configuration of a system for performing bin picking by a robot in a factory, including an image processing apparatus according to a first embodiment. 9 is a graph showing eigenvalues for each frequency component when an L-shaped workpiece is used as a template image. FIG. 9 is an explanatory diagram for explaining that a similarity obtained as a result of an N-dimensional FFT can be approximated using an M-dimensional (M <N) FFT. 5 is a flowchart illustrating a learning process including a method of generating an eigenvalue template image according to the first embodiment. It is a flowchart which shows the collation process with respect to the input image for detecting a target object. 9 is a flowchart illustrating a learning process including a technique for creating an eigenvalue template image according to the second embodiment. The figure which shows the result of having compared the method in the image processing apparatus which concerns on 1st Embodiment, the image processing apparatus which concerns on 2nd Embodiment, and the conventional eigenvalue template method from a viewpoint of collation time, learning time, and memory usage. It is. 9 is a sample image used in an experiment for verifying the superiority of the technique according to each embodiment. FIG. 11 is a diagram illustrating an example of a difference in processing time between the method according to the embodiment and various conventional methods. FIG. 7 is a diagram illustrating an example of a difference in detection rate between the technique according to the embodiment and various conventional techniques.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

(1st Embodiment)
FIG. 1 shows a configuration of an entire system for bin picking by a robot in a factory, including an image processing apparatus 20 according to the first embodiment.

  As shown in FIG. 1, the system includes a camera 10. The camera 10 captures an image of the inside of a component box that accommodates an object and flows on a line, for example. The image captured by the camera 10 is provided to the image processing device 20 as an input image.

  The image processing device 20 detects a target object corresponding to the template image in the input image by performing matching using the input image and an eigenvalue template image prepared in advance and stored in the internal memory 21. The learning process including the method of creating the eigenvalue template image will be described later in detail with reference to the flowchart of FIG.

  The input image usually has a larger image area than the template image. Therefore, the image processing device 20 determines a matching area to be matched with the template image in the input image and sets the matching area as a query image. Note that the query image means an image used for calculating the similarity in the image processing device 20.

  At this time, the image processing device 20 sets the matching area a plurality of times while shifting the matching area by a predetermined pixel unit (for example, one pixel unit) so that the matching area covers the entire input image. Further, the image processing apparatus 20 performs an FFT operation on the correlation value with the eigenvalue template image for each collation region set at a different position of the input image, that is, for each query image, and performs similarity for each rotation angle. Is calculated, and a rotation angle at which the similarity is maximized is obtained. When the image areas of the input image and the template image are equivalent, the input image may be directly used as the query image.

  Then, based on the similarity calculated for each query image, the image processing apparatus 20 sets the posture parameter related to the position and the rotation angle at which the similarity is maximum, or the posture parameter related to the position and the rotation angle at which the similarity is equal to or more than the threshold. , The information about the posture of the finally required object is obtained. For example, the image processing device 20 compares the similarities calculated for each query image with each other, and detects the orientation of the target object from the position and rotation angle of the query image that has the maximum similarity. The details of the above-described collation processing in the image processing apparatus 20 will be described later in detail with reference to the flowchart of FIG.

  The robot 30 acquires posture information on the position and rotation angle of the target object from the image processing device 20 and, based on the posture information, for example, grasps the target object and performs arm assembly so as to perform assembly to a product. Control.

  The image processing apparatus 20 according to the present embodiment can be applied to, for example, a system for inspecting the appearance of components and products, in addition to a system for performing bin picking. When applied to a system for performing a visual inspection, for example, information about the similarity is output from the image processing apparatus 20 to a display device. If so, a warning display may be performed.

  Next, a method of speeding up the rotation matching based on the correlation in the image processing device 20 according to the present embodiment will be described.

Conventional rotation matching based on correlation can be formulated as Equation 1 below. In other words, by performing convolution operation N times between the query image f and the template image Tθ rotated in-plane at a constant rotation pitch (Δθ = 2π / N), the similarity for each of the posture parameters x, y, and θ is calculated. calculate. R indicates an area where the template image Tθ overlaps the query image.

  If the angle pitch Δθ is reduced in order to increase the detection rate and the accuracy of estimating the angle parameter θ, the number of necessary convolution operations increases, so that the processing time increases. An eigenvalue template method has been proposed as a method that can solve this problem.

  In the eigenvalue template method, similarity is efficiently calculated by using a new template image (eigenvalue template image) obtained by compressing information of a template image group by singular value decomposition for matching. Hereinafter, an example of a similarity calculation method using the eigenvalue template method will be described.

For example, the template image group T is defined as in the following Expression 2.

Then, as shown in the following Expression 3, the template image group T is composed of M eigenvalue template images E = [E ₀ ,..., E _M−1 ] smaller than the number N of images of the template image group T, and M固有_k (θ) = [φ ₀ ^T (θ),..., Φ _M-1 ^T (θ)] ^T.

The approximate expression of Expression 3 is substituted into Expression 1 to obtain an approximate expression of similarity shown in Expression 4 below.

  By calculating the similarity using Expression 4, N convolution operations can be reduced to M times as compared with Expression 2, and the operation process can be made more efficient.

In the image processing apparatus 20 according to the present embodiment, Expression 4 is considered in two steps in order to further speed up the arithmetic processing for calculating the similarity. That is, as shown in the following Expression 5, a process of calculating a correlation value vector (response vector) r (x, y) by a convolution operation of the query image f and the M eigenvalue template images E is a first process. Step. Then, as shown in the following Expression 6, the process of calculating the similarity g from the sum of the products of the elements of the correlation value vector r (x, y) and the eigenfunction φ _k (θ) is a second step and I do.

The calculation of the correlation value vector r in Expression 5 can be efficiently performed in a fixed processing time if the number M of the eigenvalue template images E is fixed. However, the calculation of the sum of the products of the correlation value vector r and the eigenfunction φ _k (θ) in Equation 6 increases the processing load in proportion to the number of elements included in each. In other words, if the calculation of the sum of products of the correlation value vector r and the eigenfunction φ _k (θ) can be made more efficient, the calculation processing of the similarity g (x, y, θ) can be further speeded up. Can be.

  Therefore, the image processing apparatus 20 according to the present embodiment pays attention to the fact that the template image group T can be singular value-decomposed using a DFT (Discrete Fourier Transform) matrix, and that large singular values are concentrated in low frequency components. The eigenvalue template image E is created using a frequency basis function using a low-frequency part of the DFT matrix as the eigenfunction φ. In this way, in order to calculate the similarity, the DFT matrix (frequency basis function) of the low frequency part is used as the eigenfunction for the correlation value vector r calculated by the convolution operation of the eigenvalue template image E and the query image f. An equivalent result can be obtained when the product-sum operation is performed and when the FFT operation is performed. The FFT operation processing can greatly reduce the operation time as compared with the product-sum operation of the correlation value vector r and the DFT matrix. Therefore, rotation matching, that is, calculation of similarity, can be performed at a higher speed than the conventional eigenvalue template method, while maintaining the detection rate equivalent to the conventional eigenvalue template method.

  Hereinafter, the theoretical background of the above method will be described.

  First, it will be shown that the template image group T can be subjected to singular value decomposition using a DFT matrix.

Like the template image group T, the variance-covariance matrix of a matrix created by in-plane rotation at a constant rotation pitch becomes a circular matrix, and the circular matrix is subjected to eigenvalue decomposition into a DFT matrix F and an eigenvalue matrix Σ. Can be. Therefore, the following equation 7 is established. Note that the subscript “T” indicates a transposed matrix, and “†” indicates a conjugate transposed matrix.

From Expression 7, the template image group T can be subjected to singular value decomposition using the DFT matrix F as in Expression 8 below.

Here, when the eigenvalue template image E is defined as in Expression 9, the eigenvalue template image E can be obtained from the template image group T and the DFT matrix F as in Expression 10.

  Next, a method of calculating the similarity using the eigenvalue template image E created by performing frequency decomposition on the template image group T as described above will be described.

From Equation 8, since the N eigenvalue template images E are orthogonal to each other, the similarity of Equation 1 can be expressed in the form of a Fourier series as in Equation 11 below.

The definition of the correlation value vector r _k is the same as Equation 5.

  Here, since the luminance change in the in-plane rotation direction is gradual in many images, when the template image group T is subjected to Fourier series expansion, high-valued eigenvalues tend to concentrate in the low-frequency region. For example, FIG. 2 shows eigenvalues for each frequency component when an L-shaped work is used as a template image and decomposed into eigenvalues and frequency components. FIG. 2 also shows that the eigenvalue corresponding to the low frequency component shows a relatively high value.

Therefore, using a truncated DFT matrix F ^{(N × M)} composed of M low-frequency components of a discrete Fourier basis, the template image group T can be approximated as in the following Expression 12. As the M low frequency components, for example, M frequency components having a DC component as a lower limit and a predetermined reference frequency as an upper limit can be selected. Each eigenvalue template image E is assumed to be composed of N elements.

By applying Expression 12 to Expression 11, an approximate expression of the similarity based on the M eigenvalue template images E is obtained as in Expression 13 below.

Next, it will be shown that the similarity of Expression 13 can be calculated by the N-dimensional FFT of the correlation value vector r obtained as a result of convolution of the eigenvalue template image E and the query image f. As a preparation, the similarity of Expression 13 is rewritten using a truncated DFT matrix as in Expression 14 below.

Note that the definition of the correlation value vector r in Expression 14 is as shown in Expression 15.

The similarity obtained by the product-sum operation of the correlation value vector r of Expression 14 and the truncated DFT matrix F (N × M) is the correlation value vector r (= [r ^T (x, y), 0,..., 0]) and an N-dimensional DFT matrix.

Therefore, the similarity of Expression 14 can be modified as in Expression 16 below, and can be calculated by an N-dimensional FFT operation.

  It is shown that the similarity obtained as a result of the N-dimensional FFT of Expression 16 can be approximated using a faster M (<N) -dimensional FFT so that the similarity can be calculated faster.

The projection of the DFT matrix F ^{(N × M)} onto the M-dimensional DFT matrix has a shape in which a diagonal matrix is crushed vertically, as shown in FIG. From this, it can be seen that the similarity of Expression 17 can be well approximated by extending the similarity obtained by performing the M-dimensional FFT on the correlation value vector r without performing zero padding in the rotation angle direction.

  Therefore, the initial value of the angle parameter θmax (x, y) corresponding to the peak value g (x, y, θmax) of the similarity in the rotation direction of each matching region can be estimated from the result of the M-dimensional FFT by the peak search. Then, in order to further increase the accuracy of the angle parameter θmax (x, y), the rotation angle is estimated in subpixel units using the Newton method with θmax (x, y) as an initial value. This is because the similarity can be described by an analytic function (a linear sum of Fourier bases), and its second derivative can also be analytically calculated. Alternatively, the rotation angle may be estimated by performing parabola fitting using three points around the peak.

  Hereinafter, the learning process in the image processing device 20 will be described with reference to the flowchart of FIG.

  In this learning process, first, in step S100, a template image group T including a plurality of template images having different rotation angles is generated. In the following step S110, the template image group T is approximately decomposed into a truncated DFT matrix F having M frequency components as elements and M eigenvalue template images E as shown by the above-described Expression 12. In this way, M unique value template images E are created. The created M unique value template images E are stored in the memory 21.

  The learning process described above may be performed by another computer, and the created M unique value template images E may be stored in the memory 21 of the image processing device 20.

  Next, with reference to the flowchart of FIG. 5, the collation processing executed in the image processing apparatus 20 will be described.

  First, in step S200, a matching area to be matched with the template image is set in the input image. The set collation area is a query image and is a processing target of each processing described below.

  In the following step S210, a convolution operation of the query image f and the M eigenvalue template images is performed in accordance with the above-described Expression 5 to calculate a correlation value vector r including M elements shown in Expression 15.

In step S220, the sum of the absolute values of each element of the calculated correlation value vector r is calculated. The reason for performing this processing is that the upper limit of the calculated similarity is simply calculated by focusing on the structure of the DFT matrix F. If the upper limit is less than the threshold, the detailed similarity is calculated. This is to further increase the processing speed by omitting. In DFT matrix F, l ₂ norm of the row vector is constant regardless of the corresponding theta. Therefore, the upper limit of the similarity can be calculated from the sum of the absolute values of the elements of the correlation value vector r.

  In step S230, the sum of the calculated absolute values is compared with a predetermined threshold. When it is determined that the sum of the calculated absolute values is less than the predetermined threshold, the process for calculating the similarity is skipped, and the process proceeds to step S270. On the other hand, if it is determined that the difference is equal to or larger than the threshold, the process proceeds to step S240.

  In step S240, an M-dimensional FFT operation is performed on the correlation value vector r. At this time, the M-dimensional FFT is performed based on a predetermined rotation angle θ as shown in Expression 13. Thereby, the correspondence between each frequency component obtained by the M-dimensional FFT operation and the rotation angle is uniquely determined. Therefore, in step S250, the initial value of the angle parameter θmax (x, y) at which the similarity peaks can be obtained from the result of the M-dimensional FFT operation. Then, in step S260, the rotation angle is finally estimated in subpixel units using the Newton method, with θmax (x, y) calculated in step S250 as an initial value.

  In step S270, it is determined whether or not the matching has been performed over the entire area of the input image. If it is determined that the matching in the entire area has not been completed yet, the process returns to step S200, and the processing from step S210 is performed by shifting the matching area. On the other hand, if it is determined in step S270 that the matching in the entire area has been completed, the process proceeds to step S280, and the position and rotation angle of the target object are determined based on the matching results in all the matching areas.

(2nd Embodiment)
Next, an image processing device 20 according to a second embodiment of the present invention will be described.

  In the image processing device 20 according to the first embodiment described above, in order to create the eigenvalue template image E, the template image group T including a plurality of template images having different rotation angles is generated. When such a template image group T is used, the process of rotating the template image must be performed a plurality of times, which complicates the learning process. Also, a large amount of memory is required to store the rotated template image.

  Therefore, the image processing apparatus 20 according to the present embodiment creates an eigenvalue template image by expressing one template image in polar coordinates, and performing Fourier transform on the template image expressed in polar coordinates in the rotation axis direction. , Approximate calculation of similarity is performed using the eigenvalue template image.

  Hereinafter, the theoretical background of the method according to the present embodiment will be described.

First, the similarity between the template image T (r, τ−θ) rotated by the angle θ and the query image f (r, τ) is defined as the following Expression 17 using the polar coordinate parameters (r, θ). I do.

Further, the template image T (r, τ) subjected to the polar coordinate conversion can be expressed as the following Expression 18 by Fourier series expansion in the rotation axis direction.

Even when the template image is subjected to the polar coordinate conversion, as in the case of the template image group T including a plurality of template images having different rotation angles, the luminance change in the rotation direction concentrates on the low frequency components. Therefore, Expression 18 can be approximated by a linear sum of the M low-frequency components based on M low-frequency components, as in Expression 19 below.

By substituting Equation 19 into Equation 17 and rearranging for kθ, the similarity g (θ) is expressed by the linear sum of the Fourier bases using the above-described M low-frequency components as in Equation 20 below. can do.

The eigenvalue template image E _k (r, τ) in Expression 20 is defined as in Expression 21 below.

As described above, the eigenvalue template image E _k (r, τ) is obtained by developing the M low-frequency components and the coefficients φ _k (r ) Can be calculated.

  In the case where Expression 21 is used as it is, it is necessary to polarize the query image f when performing collation with the query image f, so that the calculation load increases, and the efficiency of the inner product operation using the convolution theorem cannot be improved. The problem arises.

Therefore, in equation 21, φ _k (r), by the inverse polar coordinate conversion to ψ _k (τ), as shown in Equation 22, the eigenvalues template image _E k (r, τ) the calculated on Euclidean coordinates.

Thus, as in the first embodiment, the convolution operation between the query image f and eigenvalues template image E _k can be calculated correlation value vector r. Therefore, similarly to the image processing apparatus 20 according to the first embodiment, the image processing apparatus 20 according to the present embodiment calculates the similarity by the matching process illustrated in the flowchart of FIG. be able to.

  Hereinafter, a learning process performed in the image processing apparatus 20 according to the present embodiment will be described with reference to a flowchart of FIG.

First, in step S300, the template image is converted into polar coordinates based on the center of the template image. In the subsequent step S310, the template image converted into the polar coordinates is subjected to Fourier series expansion in the direction of the rotation axis, thereby performing frequency decomposition. In this Fourier series expansion, as shown in Expression 19, M low-frequency components are used as a basis, and the low-frequency components are approximated by a linear sum of the M low-frequency components. As a result, M low frequency components and the coefficient φ _k (r) of each low frequency component are determined.

In the following step S320, the function _{ｋ k} (τ) having M frequency components as elements and the coefficient φ _k (r) of each frequency component are converted into Euclidean coordinates. Then, in step S330, using the function _{ｋ k} (τ) having M low frequency components converted into Euclidean coordinates as elements according to Expression 22, and the coefficient φ _k (r) of each low frequency component, The eigenvalue template image E _k (r, τ) is calculated.

  In the above description, a method of performing Fourier transform after the template image is subjected to polar coordinate conversion has been described. However, it is also possible to perform Fourier transform in polar coordinates directly from a template image on Euclidean coordinates. Further, similarly to the first embodiment, the learning process described above is performed by another computer, and the created M eigenvalue template images E are stored in the memory 21 of the image processing device 20. Is also good.

  Here, the method in the image processing apparatus according to the first embodiment and the method in the image processing apparatus according to the second embodiment described above is compared with the conventional eigenvalue template method in terms of the matching time, the learning time, and the memory usage. FIG. 7 shows the results. 7A to 7C, “actual 1” indicates a result when the method described in the first embodiment is used, and “actual 2” indicates a method described in the second embodiment. The result when adopting is shown. The values shown in FIGS. 7A to 7C are examples of theoretical values.

  FIG. 7A shows a comparison result regarding a matching time required for a matching process with a query image. In the methods according to the first embodiment and the second embodiment, since the product-sum operation of the correlation value vector and the frequency basis function is speeded up by the FFT operation, the matching time can be reduced to about 1/5 of the conventional method. I have.

  FIG. 7B shows a comparison result regarding learning time including creation of an eigenvalue template image. In the method according to the first embodiment, singular value decomposition (SVD) having a high processing load is replaced with FFT in the learning algorithm, so that the learning time can be reduced to about 1/4 compared to the conventional method. .

  FIG. 7C shows a comparison result regarding the memory usage in the learning process. In the method according to the second embodiment, the use of the polar coordinate transformation of the template image at the time of learning eliminates the need to store the in-plane rotated image in the memory, which is required in the conventional method. Thus, according to the method of the second embodiment, the memory usage during the learning process can be reduced to about 1/120 of the conventional method.

(Third embodiment)
Next, an image processing device 20 according to a third embodiment of the present invention will be described.

  The image processing device 20 according to the present embodiment includes a matching method using phase information that is robust against background noise and a luminance gradient that is robust against illumination variation in order to enhance robustness against illumination and background variation. The matching using the direction is applied to the matching based on the information compression of the template image according to the first embodiment and the second embodiment described above.

  The direction of the brightness gradient is robust against a change in the brightness of the image due to a change in the brightness of the environment. Further, even when there is a pattern similar to the template image in the background of the query image, the POC using the phase component can calculate the similarity robustly. By executing the learning process and the matching process described in the first embodiment and the second embodiment using an image to which these ideas are applied, it is possible to enhance robustness against variations in lighting and background.

Therefore, in the present embodiment, a complex gradient image composed of the intensity and direction of the luminance gradient is calculated for each of the query image and the template image as shown in Expression 23 below.

In Expression 23, T _x and T _y are differential images in the horizontal and vertical directions obtained by applying a Gaussian filter to the original image T.

  Further, an image (an image having an amplitude of 1) is extracted from the phase component in the frequency space obtained by subjecting the complex gradient image to two-dimensional Fourier transform.

  The variance-covariance matrix of the template group created from the image generated by the above method from the template image rotated in the plane becomes a circular matrix. For this reason, it can be applied to the method described in the first embodiment and the second embodiment.

  Also, when the luminance of the background greatly changes between the template image and the query image, the direction of the luminance gradient is reversed (π rotation). Therefore, in Expression 23, even when the direction of the luminance gradient is reversed, the rotation component is multiplied by 2 so that the value of the inner product becomes the same, thereby making such fluctuations robust.

(Fourth embodiment)
Next, an image processing device 20 according to a fourth embodiment of the present invention will be described.

  For example, a false peak may easily occur depending on a query image, for example, when a correlation between a query image and an eigenvalue template image is calculated using only the phase component of the complex gradient image described in the third embodiment. . False peaks generated by the background of the query image and the like tend to show high values over a wide area.

Therefore, in the present embodiment, the false peak is suppressed by evaluating the result of the FFT operation, that is, the relationship between the peak value peak of the similarity and the average value mean and the variance σ around the peak value peak. Specifically, a PSR (Peak Signal Ratio) filter process is performed on the maximum similarity in the rotation direction to obtain a final similarity. The PSR filter processing is defined by the following equation (24).

  Note that the peak value peak, average value mean, and variance σ in each query image can be calculated at high speed by using an integral image.

(Evaluation experiment)
Experiments by simulation were performed to evaluate the usefulness of the technique for speeding up and robustness against background / brightness according to each of the above-described embodiments. The Proposed Method 1 (Proposed-Phase) described below corresponds to a method in which an image generated from the complex gradient image according to the third embodiment is applied to the method according to the first embodiment, and is referred to as Proposed Method 2 (Proposed-Phase). -Naive) corresponds to a method that directly uses the method described in the first embodiment.

(1) Evaluation items The detection success rate and the processing speed were evaluated. The detection success rate was determined to be within ± 3 [pix] from the correct position for the XY position and within ± 3 [deg] for the rotation angle θ. The processing speed was determined by measuring the time required to search for a parameter that maximizes the similarity g in each method.

  Note that the determination using the threshold is not performed. The reason is that the same threshold value cannot be set in each method because the index of the correlation value differs in each method. Therefore, in Proposed Methods 1 and 2, no truncation based on the sum of the absolute values of the elements of the correlation value vector r is performed.

(2) Methods Compared The following four methods are compared with the proposed method 1 (Proposed-Phase) and the proposed method 2 (Proposed-Naive).
・ Integral normalized edge eigenvalue template method (Eigen-Edge)
・ Eigenvalue template method (Eigen-Naive)
・ Rotation-invariant phase-only correlation (RIPOC)
・ Rotation matching (NCC) using normalized cross-correlation
In this evaluation experiment, since the accuracy is not evaluated, the sub-pixel estimation processing is disabled.

  In addition, with respect to each method, including the Proposed Method 1 (Proposed-Phase) and the Proposed Method 2 (Proposed-Naive), the processing speed was also evaluated for implementations in which processing by the GPU was parallelized.

  In the proposed method (Proposed-Phase), the proposed method 2 (Proposed-Naive), the integral-type normalized edge eigenvalue template method (Eigen-Edge), and the eigenvalue template method (Eigen-Naive), the template image is represented by an angular pitch Δθ. A group of 512 template images generated by rotation at an interval of = 360/512 [deg] was approximated by M = 32 eigenvalue template images.

  For the rotation-invariant phase-only correlation method (RIPOC), a low-pass filter was applied in POC processing to improve robustness to noise.

  Rotational matching (NCC) using normalized cross-correlation was also performed with the number of angular steps N = 512.

(3) Target Work and Query Image The target work was assumed to be a textureless industrial part, and an L-shaped part was used as shown in FIGS. 8 (a) to 8 (d). Also, as for the query image, four types of images having different characteristics, such as a mounting surface of a circuit board taken separately, are used as background images, and are combined with an image of the target work, so as to be shown in FIGS. 8A to 8D. Four types of query images were created. Further, in order to increase the number of query images, the target work is rotated at random with respect to the background image based on each of the query images in FIGS. 8A to 8D and then combined with the background image. 1000 query images were created from each query image. Also, in order to reproduce the noise in the real environment, a Gaussian noise with a variance of 15 is added to the query image, and shading such that I (x, y) ← I (x, y) × (0.5 × x / 512) is obtained. Was performed.

(4) Processing Time FIG. 9 shows the processing time of each method. The processing time shown in FIG. 9 is an average value for four types of query images × 1000 sheets.

  Rotation matching (NCC) using normalized cross-correlation requires a convolution operation with each of a large number of in-plane rotated template images, and the processing time is extremely long. Also, the improvement effect by using the GPU is not so much seen. One possible reason is that the transfer time to the GPU is a bottleneck.

  The proposed method 1 (Proposed-Phase), the proposed method 2 (Proposed-Naive), the integral-type normalized edge eigenvalue template method (Eigen-Edge) and the eigenvalue template method (Eigen-Naive) have 32 sheets of convolution. Since it is only necessary to perform convolution operation with the eigenvalue template image, the processing time can be greatly reduced for rotation matching (NCC) using normalized cross-correlation.

  Furthermore, the proposed method 1 (Proposed-Phase) and the proposed method 2 (Proposed-Naive) replace the product-sum operation of the correlation value and the eigenfunction with the FFT operation. About 5.5 times in comparison with the shape-normalized edge eigenvalue template method (Eigen-Edge), and about 3.5 times in comparison with the proposed method 2 (Proposed-Naive) and the eigenvalue template method (Eigen-Naive). It was confirmed that the processing speed was increased.

In addition, the proposed method 1 (Proposed-Phase) and the proposed method 2 (Proposed-Naive) achieve approximately three times speedup when using a GPU, as compared with RICOP, which is known to be able to perform high-speed rotation matching. I realized it.
(5) Detection Success Rate FIG. 10 shows the detection success rate of each method. In FIG. 10, the detection success rate is shown for each of the query images shown in FIGS. 8 (a) to 8 (d).

  In the proposed method 2 (Proposed-Naive) and the eigenvalue template method (Eigen-Naive), the same detection success rates are obtained for the query images shown in FIGS. 8A and 8B, and the textures used in this experiment were used. It was confirmed that, for a work with a small number, the technique for increasing the speed described in the first embodiment did not adversely affect the detection success rate.

  However, both the proposed method 2 (Proposed-Naive) and the eigenvalue template method (Eigen-Naive) use other methods that have improved robustness for images with complex backgrounds as shown in FIGS. 8C and 8D. The detection success rate was lower than that.

  On the other hand, the proposed method 1 (Proposed-Phase) applying the image generated from the complex gradient image described in the third embodiment is based on the rotation matching (NCC) using the normalized cross-correlation and the integral-type normalized edge eigenvalue. A higher detection success rate was obtained compared to the conventional method that improved robustness such as the template method (Eigen-Edge). In particular, in the integration type normalized edge eigenvalue template method (Eigen-Edge), the success rate of detecting the query image in FIG. In Proposed-Phase), the posture of the target work could be robustly estimated even for such a query image.

  Each of the above-described embodiments is a preferred embodiment of the image processing method and the image processing apparatus of the present invention. However, the image processing method and the image processing apparatus of the present invention are not limited to the above-described embodiments, and the present invention is not limited thereto. Various modifications can be made without departing from the spirit of the invention.

  For example, in the first and second embodiments described above, M low-frequency components having a DC component as a lower limit and a predetermined reference frequency as an upper limit correspond to relatively large eigenvalues among all eigenvalues. It is assumed that things are included. However, in the case of a template in which a frequency component corresponding to a large eigenvalue is in a high frequency region, it is necessary to select a DFT matrix so as to include a frequency component corresponding to the large eigenvalue. For example, the magnitude of the eigenvalue corresponding to each frequency component may be checked for each template, and the frequency component of the DFT matrix may be determined based on the result.

  Further, by applying a pre-processing that cuts a high-frequency component in the rotation direction of the image, a frequency component corresponding to a large eigenvalue may be concentrated in a low-frequency region.

Reference Signs List 10 camera 20 image processing device 21 memory 30 robot

Claims (14)

An image processing method for detecting an object corresponding to the template image in the input image by comparing the input image with a predetermined template image,
A first step of performing frequency decomposition on the template image in a rotational direction based on a predetermined number of frequency components from a low frequency to a high frequency belonging to a predetermined frequency range as a basis, and calculating the same number of eigenvalue template images as the frequency components ( S100, S110),
A second step (S210) of calculating a correlation value with the input image for each of the eigenvalue template images by performing a convolution operation on the input image and the eigenvalue template image;
A third step (S240) of performing an FFT calculation process on the calculated correlation value based on a predetermined rotation direction to calculate a similarity for each rotation angle;
A fourth step (S250, S260) of calculating a rotation angle of the object in the input image from the similarity for each rotation angle.   The image processing method according to claim 1, wherein the predetermined number of frequency components from a low frequency to a high frequency belonging to the predetermined frequency range has a DC component as a lower limit and a predetermined reference frequency as an upper limit. The first step includes:
A generation step (S100) of generating an image group including a plurality of the template images having different rotation angles;
The image group is frequency-decomposed in the rotational direction, and a frequency basis function including the predetermined number of frequency components equal to or lower than a predetermined reference frequency, and the frequency components are decomposed into the same number of eigenvalue template images as the eigenvalue template. The image processing method according to claim 1, further comprising a decomposition step of calculating an image. The first step includes:
A conversion step (S300) of converting the template image into polar coordinates;
The template image converted to the polar coordinates, the frequency component of a predetermined number or less of a predetermined reference frequency or less as a basis, frequency decomposition, using the coefficients of each frequency component when the frequency decomposition, using each frequency component, The image processing method according to claim 1, further comprising a calculating step of calculating the eigenvalue template image (S <b> 310 to S <b> 340). For the input image and the template image, respectively, a fifth step of calculating a complex gradient image composed of the intensity and direction of a luminance gradient, and converting the complex gradient image into an image obtained by extracting a phase component,
The image processing method according to any one of claims 1 to 4, wherein the processing from the first step to the fourth step is performed using the converted image. The image processing method according to claim 5, wherein in the fifth step, a complex gradient image is created by using the following Expression 1.
Here, Tx and Ty are horizontal and vertical differential images of the original image T. The input image has an image area wider than the template image, and includes a sixth step (S200, S270) of defining a collation region in the input image to be collated with the template image.
In the sixth step, the collation area is defined a plurality of times while shifting the collation area by a predetermined pixel unit so that the collation area covers the entire area of the input image,
In the second step, the correlation value is calculated by performing a convolution operation with the eigenvalue template image using the matching area determined in the sixth step as the input image,
Furthermore, comparing the sum of the absolute values of the correlation values calculated for each of the eigenvalue template images with a predetermined threshold value, and if the sum of the absolute values of the correlation values is less than the predetermined threshold value, the third step The image processing method according to claim 1, further comprising a seventh step (S <b> 230) of excluding from subsequent processing. The third step is
The maximum degree of similarity among the similarity for each rotation angle calculated by the FFT arithmetic processing, saw including a filter step of performing a filtering process shown by the following equation 2, the similarity of the filtered, final The image processing method according to claim 1, wherein the similarity is a maximum similarity .
Here, peak is the maximum similarity, mean is the average value of the similarities, and σ is the variance of the similarities. An image processing device that detects an object corresponding to the template image in the input image by performing collation between the input image and a predetermined template image,
The template image is stored by storing the same number of eigenvalue template images as the number of the frequency components, which are calculated by performing frequency decomposition in a rotational direction based on a predetermined number of frequency components from a low frequency to a high frequency belonging to a predetermined frequency range as a base. Part (21),
A correlation value calculation unit (S210) for calculating a correlation value with the input image for each of the eigenvalue template images by performing a convolution operation on the input image and the eigenvalue template images stored in the storage unit;
A similarity calculation unit (S240) that performs an FFT calculation process based on a predetermined rotation direction on the calculated correlation value to calculate a similarity for each rotation angle;
A rotation angle calculation unit (S250, S260) for calculating a rotation angle of the object in the input image from the similarity for each rotation angle.   The image processing apparatus according to claim 9, wherein a predetermined number of frequency components from a low frequency to a high frequency belonging to the predetermined frequency range has a DC component as a lower limit and a predetermined reference frequency as an upper limit.   The image processing apparatus according to claim 9, further comprising: a conversion unit configured to calculate a complex gradient image including the intensity and direction of a luminance gradient for the input image and the template image, and to convert a phase component of the complex gradient image into an extracted image. The image processing apparatus according to any one of the preceding claims. The image processing device according to claim 11, wherein the conversion unit creates a complex gradient image using Expression 3 below.
Here, Tx and Ty are horizontal and vertical differential images of the original image T. The input image has an image area wider than the template image, and includes a collation area setting unit (S200, S270) that determines a collation area to be collated with the template image in the input image.
The collation area setting unit, while shifting the collation area by a predetermined pixel unit, such that the collation area covers the entire area of the input image, defines the collation area a plurality of times,
The correlation value calculating unit, as the input image, using a matching area determined by the matching area setting unit, performs a convolution operation with the eigenvalue template image, calculates the correlation value,
Further, the sum of the absolute values of the correlation values calculated for each of the eigenvalue template images is compared with a predetermined threshold, and when the sum of the absolute values of the correlation values is less than the predetermined threshold, the similarity calculation is performed. The image processing apparatus according to any one of claims 9 to 12, further comprising a correlation value determining unit (S230) for excluding the correlation value from being calculated by the unit. The similarity calculating section,
The maximum degree of similarity among the similarity for each rotation angle calculated by the FFT arithmetic processing, saw including a filter unit that performs a filtering process shown in Equation 4 below, the similarity of the filtered, final 14. The image processing apparatus according to claim 9 , wherein the similarity is a maximum similarity .
Here, peak is the maximum similarity, mean is the average value of the similarities, and σ is the variance of the similarities.