JP6225722B2 - Image processing method and image processing apparatus - Google Patents

Description

  The present invention relates to an image processing method and an image processing apparatus for detecting an angle of a dominant straight line portion in a color or monochrome gradation image.

    For example, in the manufacturing process and inspection process of industrial products, for the delivery of intermediate products between the previous and subsequent processes, or when inspecting the presence or absence of defects compared with a pre-recorded good product image, In many cases, it is necessary to obtain the angle of a straight line portion dominant in an image captured by a camera with a certain level of accuracy in a short time. There is also a need for a high-performance image processing algorithm for correcting the inclination of an image obtained by scanning a document or chart with a scanner, or for detecting the angle of a road or railway from an aerial photograph.

  In the angle detection image processing apparatus described in Patent Document 1, angle detection is performed by performing Hough transform in two stages. In the image processing method described in Patent Document 2, a histogram in the edge direction is created, and an angle corresponding to the maximum peak value on the histogram is set as the angle of the object. In Patent Document 3, the edge strength and edge direction of a target pixel of an image are calculated, and if the edge strength is equal to or greater than a certain threshold value, the target pixel is determined to be on a straight line, and determined to be on a straight line. An angle detection method is disclosed in which an edge direction histogram is created for the image thus obtained, and an angle at which the frequency is maximized is extracted.

JP-A-6-119447 JP 11-96372 A Japanese Patent No. 4327654

  However, the methods using the angle histograms disclosed in Patent Documents 2 and 3 have a problem that the maximum frequency angle cannot be obtained with an accuracy finer than the step size. If the step size is increased, the accuracy is improved. There is a dilemma that if it is reduced, the reliability decreases because the frequency of falling into each segment decreases. Also, even for pixels whose edge strength is above a certain threshold, there is a relative variation in edge strength, and pixels with relatively low edge strength may be affected by noise. I don't want to focus on it. However, when using the method of taking an angle histogram, the same applies to pixels that are highly reliable in angle because of high edge strength and pixels that may be affected by noise because of low edge strength. There is a problem that it is counted by weight. This problem also causes a decrease in the accuracy of the detected angle.

  In fact, in Patent Document 3, the angle obtained as the mode value of the angle histogram is set as the “first angle”, and a provisional line is determined based on the angle, and a regression line is obtained based on the provisional line, and the inclination is finally determined. Output angle. That is, it is described that the first angle requires the second step because the accuracy is not sufficient. It can also be pointed out that no method for self-diagnosis of the degree of reliability is presented for the obtained angle.

  The present invention has been made in view of such problems, and eliminates two factors of accuracy degradation due to angle increments and accuracy degradation due to not using edge strength information, and allows a single statistical process. It is an object of the present invention to provide an image processing method and an image processing apparatus for calculating a highly accurate angle.

  A first invention for solving the problem is an image processing method for detecting an angle of a straight line dominant in a target image, wherein a gradient vector relating to a pixel value of each pixel of the image is obtained, Calculating a covariance matrix around the origin for a set of gent vectors, performing principal component analysis based on the covariance matrix, and obtaining a direction orthogonal to the first principal component to determine a straight line angle dominant in the target image It is an image processing method characterized by comprising the step of determining.

Here, the “covariance matrix around the origin” is as follows. Now, assume that n pieces of data represented by a set of two variables (x _i , y _i ) are given as in Equation 1. i is a subscript indicating the number of data. For convenience, x _i will be referred to as a first component and y _i as a second component. The average value of the first component and the second component is expressed by Equation 2.

Using this, the normal covariance matrix C can be expressed by Equation 3.

The covariance matrix around the origin is obtained by replacing the average value of the first component x _i and the second component y _i with 0 in Equation 3. That is, the covariance matrix C ′ around the origin is expressed by Equation 4.

By the way, obtaining the first principal component axis by principal component analysis has the following meaning.
Obtaining the first principal component axis by principal component analysis is drawn into the space for the given data plotted on the xy plane (space) by the first component xi and the second component yi. This corresponds to finding the closest straight line to the data distribution among the arbitrary straight lines L. As a criterion for “close” at that time, the length of the perpendicular line from each data to the straight line L (distance from the straight line L of the data) is regarded as an error, and the closer the sum of squares of the error, the “closer” It is said. (See Figure 19)
In actual calculation of principal component analysis, a covariance matrix is usually obtained based on data, and its eigenvalue and eigenvector are obtained. The eigenvector corresponding to the maximum eigenvalue is the first principal component. Then, a straight line that passes through the average value of the data and whose direction vector is the first principal component direction is the best approximate straight line to be obtained.

On the other hand, in the present application, a covariance matrix around the origin is used instead of the covariance matrix. This corresponds to limiting the approximate straight line of data to that passing through the origin. In other words, if the eigenvector corresponding to the maximum eigenvalue is regarded as the first principal component (around the origin) with respect to the covariance matrix around the origin, it passes through the origin and the direction vector is A straight line that is one main component is obtained.
In the gradient vector, since the direction of each vector centered on the origin represents the normal of the edge on the image, it is essential to look at the origin, and each vector is centered on the average value of the data. The direction seen is meaningless. In view of this, a covariance matrix around the origin is used.

  The present invention is a simple method in which if a gradient vector for each pixel is obtained, it is considered as a set of sample values and principal component analysis is performed. Conventional steps such as setting the step size required for histogram processing suppresses the upper limit of measurement accuracy, and handling the direction information equivalently without considering edge strength information makes it easier for noise to affect the measurement result. It has an excellent feature that it can avoid the disadvantages of the technology and can obtain the result obtained by a single statistical process. Furthermore, the contribution ratio of the first principal component obtained as a result of the principal component analysis may be used as an index for determining whether the target image is an image suitable for the processing method of the present invention. There is no feature in the prior art that such a reliability index can be output together.

In the first aspect of the invention, the step of obtaining the gradient vector includes sequentially setting each pixel as a pixel of interest, defining a predetermined neighborhood region centered on the pixel of interest, performing multiple regression analysis in the neighborhood region, and obtaining an approximation obtained It is preferable to determine the gradient vector based on a mathematical expression representing a plane. A plane obtained by multiple regression analysis by appropriately determining the vicinity region of each pixel of interest is sufficiently approximated to the tangent plane of the pixel of interest.
Furthermore, in the preferable aspect of the first invention, it is more preferable that the neighboring area is an area defined by pixels within a circle having a constant radius centered on the pixel of interest. By taking a circular neighborhood, it is more preferable because the target image is not biased by the dominant straight line direction.

A second invention that solves the problem is an image processing device that detects an angle of a dominant straight line in processing target image data, and calculates a gradient vector of pixel values in each pixel;
A means for calculating a covariance matrix around the origin with respect to the gradient vectors of all pixels, and principal component analysis is performed based on the obtained covariance matrix to determine an angle for obtaining a direction orthogonal to the first principal component. And an image processing apparatus.

  A third invention for solving the problem is a computer program describing a series of instructions for causing a computer to operate as the image processing apparatus according to the second invention.

The present invention employs a revolutionary and innovative processing algorithm that does not exist in the prior art, so that the complexity of calculation, the setting of the step size essential for histogram processing determines the upper limit of the measurement accuracy, the edge strength information It overcomes the drawbacks of the prior art, such as being susceptible to noise due to equivalent handling of direction information without consideration. Furthermore, there is a remarkable effect that it is possible to give a measure of reliability as to whether or not the target image is an image suitable for the present image processing method.

It is a flowchart explaining the process sequence of this invention. It is a figure explaining the vicinity area | region of an attention pixel. Sample image 1 Processing result by square neighborhood of sample image 1 (neighbor radius r = 8) Processing result by circular neighborhood of sample image 1 (neighbor radius r = 8) Sample image 2 Processing result by square neighborhood of sample image 2 (neighbor radius r = 8) Processing result by circular neighborhood of sample image 2 (neighbor radius r = 8) Sample image 3 Processing result by square neighborhood of sample image 3 (neighbor radius r = 8) Processing result of sample image 3 by circular neighborhood (neighbor radius r = 8) Processing result by circular neighborhood of sample image 1 (neighbor radius r = 6) Processing result of sample image 2 by circular neighborhood (neighbor radius r = 6) Processing result of sample image 3 by circular neighborhood (neighbor radius r = 6) It is a figure which shows the near variation. It is a graph showing the result of the test by 24 variations. FIG. 6 is a graph plotted with the vertical axis representing the contribution ratio and the horizontal axis representing the neighborhood radius (pixel) for each of the 6 sets of (3 types of images) × (2 neighborhood shapes). It is a schematic block diagram of the image processing apparatus 100 which concerns on one Embodiment of this invention 2nd invention. It is a figure explaining the meaning of principal component analysis. Diagram explaining that the gradient vector tends to be biased toward 45 ° near the square

  From here, the image processing method as one of the suitable embodiment of this invention is demonstrated. This image processing method is a processing method keeping in mind that the image processing is performed on a general-purpose computer having an image memory or an image processing apparatus installed in a factory production line. In the following description, it is assumed that the processing target image has already been given (already captured or input and temporarily stored in the image memory). Then, the processing target image data is read into a general-purpose computer or a central processing unit of the image processing apparatus in order to perform image processing. FIG. 1 is a flowchart for explaining this image processing method. This will be described below with reference to the flowchart of FIG.

    The image data is obtained by recording the value of the pixel P for each pixel P constituting the image. The pixel P is specified by coordinates (x, y) with any one of the four corners of the image plane as the origin. Therefore, the image data can be mathematically viewed as a two-variable function in the form of f (x, y), 1 <= x <= n, 1 <= y <= m. Hereinafter, the pixel P at the coordinates (x, y) is represented by P (x, y), and the pixel value thereof is represented by f (x, y).

    First, the first (next) target pixel P (x, y) is set from the image data. (Step S110)

Next, the multiple regression analysis is performed only in the vicinity region of the target pixel P, and the gradient vectors ((（f / ∂x) _P , (∂f / ∂y) _P ) in _P are determined (step S120). A Sobel filter or a Prewitt filter can be used to calculate the gradient vector at each pixel of interest. Both of these consist of 3 x 3 pixels. However, in a noisy image, it is more stable to obtain a gradient vector statistically from a neighboring region wider than 3 × 3.
Therefore, in this embodiment, a neighboring region V having an appropriate size centered on the pixel of interest P (x, y) is set, and the pixel P _j (x _j , y _j ) in the region V in the three-dimensional space is set. A plane (approximate plane) equation z = ax + by + c that best fits the distribution {z _j = f (x _j , y _j )} is obtained by performing multiple regression analysis. The gradient vectors at P are determined as (∂f / グ ラ x) _P = a, (（f / ∂y) _P = b from the coefficients a and b of the obtained plane equation. However, here, the neighborhood region V is, for example, an 8-pixel neighborhood rectangular region (a) of the pixel of interest P as shown in FIG. 2 or a circular neighborhood region (b) having a distance from P of 8 pixels or less. This step eventually determines the tangent plane (or direction) of the pixel of interest P.

What is desired is the angle of the straight line portion that is dominant in the target image. In this image processing method, the straight line portion is not directly obtained and its angle is determined, but the target image is regarded as a curved surface defined by the image of the two-variable function f and the distribution of gradient vectors determined for each pixel is considered. To do. In this case, for example, a bright area and a dark area that occupy a certain area in the image can be imaged as a high (or low) plateau, and a straight line portion that appears in the target image is usually a cliff region that appears at the boundary between the high plateau and the low plateau I can imagine. The direction of the gradient vector points to the steepest climb of this cliff, which is orthogonal to the contour lines. The magnitude of the gradient vector indicates its steepest slope. The gradient vector distribution of the cliff determines the dominant straight angle in the image.
Therefore, the obtained partial differential coefficients (∂f / ∂x) _P and (∂f / ∂y) _P are integrated as shown in the following equation 5 (step S130).

Repeat S110 to S13010 for all pixels. When the summation of S1, S2, and S3 for all pixels is completed, the covariance matrix C ′ around the origin is determined by the following equation 6 (step S140).

    Principal component analysis based on the covariance matrix C ′ around the origin is performed. Specifically, an eigenvalue and an eigenvector are obtained, and an eigenvector (hereinafter referred to as the first eigenvector) corresponding to the larger eigenvalue (hereinafter referred to as the first eigenvalue) is obtained. (Step S150) The linear direction of the image is determined based on this eigenvector. Specifically, when this eigenvector is (x, y), a vector orthogonal to this vector in the xy plane, that is, a vector that gives an angle to be obtained (step S160).

In the flowchart of FIG. 1 described above, step S120 corresponds to “a step of obtaining a gradient vector related to the pixel value of each pixel of the image”. The loop from step S110 to step S130 and step S140 correspond to “a step of calculating a covariance matrix around the origin regarding the collection of gradient vectors”. Steps S150 and S160 correspond to “the step of performing principal component analysis based on the covariance matrix and determining the angle of a straight line dominant in the target image by obtaining a direction orthogonal to the first principal component”.

(Verification of experimental results)
An experimental result for verifying the effect of the image processing method shown in the flowchart of FIG. 1 is shown below. The images used in the experiment are the three monochrome gradation images shown in FIGS. FIG. 3 shows a sample image 1 composed of two regions with a diagonal line as a boundary. FIG. 6 shows a sample image 2 in which a diagonal line is a band-like region having a thickness. FIG. 9 shows a sample image 3 composed of a plurality of band-like regions having various thicknesses. In both cases, an oblique straight line runs and the angle is 12.345 °. Every image is blurred and noisy throughout. Since the angle is as described above, the correct answer in the direction of the gradient vector is 102.345 ° plus 90 °.

(3 types of images) × (Near square, Near circle) × (Near radius = 2, 4, 6, 8 pixels)
A test was conducted to calculate the angles for the 24 combinations. FIG. 15 is a diagram showing variations according to the radius of the neighborhood radius. The upper part shows the case of a square and the lower part shows a case of a circle.

  FIG. 4 is a graph plotting the distribution of gradient vectors as a result of processing in the vicinity of a square (radius 8 pixels) with respect to the sample image 1. The long line segment passing through the origin represents the first eigenvector obtained from the covariance matrix around the origin. FIG. 5 is a graph plotting the distribution of the gradient vector as a result of processing the sample image 1 in the vicinity of a circle (radius 8 pixels). The long line segment passing through the origin represents the first eigenvector obtained from the covariance matrix around the origin. In FIG. 5, the gradient vector distribution is almost plotted on the line segment representing the first eigenvector.

  FIG. 7 is a graph plotting the distribution of the gradient vector as a result of processing in the vicinity of the square (radius 8 pixels) with respect to the sample image 2. The long straight line passing through the origin represents the first eigenvector obtained from the covariance matrix around the origin. FIG. 8 is a graph plotting the distribution of gradient vectors as a result of processing the sample image 2 in the vicinity of a circle (radius 8 pixels). The long line segment passing through the origin represents the first eigenvector obtained from the covariance matrix around the origin. In FIG. 8, the gradient vector distribution is almost plotted on a straight line representing the first eigenvector.

  FIG. 10 is a graph plotting the distribution of the gradient vector as a result of processing in the vicinity of the square (radius 8 pixels) with respect to the sample image 3. Since the sample image 3 has a plurality of band-like regions and has different widths, the distribution of the gradient vector is more complicated in curvature than the results of the sample images 1 and 2 (FIGS. 4 and 7). You can see that. FIG. 11 is a graph plotting the distribution of gradient vectors as a result of processing in the vicinity of a circle (radius 8 pixels) with respect to the sample image 3. Since there are a plurality of band-like regions, the gradient vector distribution is on the straight line representing the first eigenvector, but the amplitude of the fluctuation is slightly larger than in the case of the sample images 1 and 2 (FIGS. 5 and 8). . However, since the principal component analysis is performed, an appropriate direction is obtained as the direction of the first eigenvector.

  4 and 5, FIG. 7 and FIG. 8, and FIG. 10 and FIG.

    FIG. 12 shows the distribution of the gradient vector obtained in the processing flow shown in FIG. 1 with the size of the neighboring region being a circle having a radius of 6 with respect to the sample image 1 of FIG. Since the neighborhood region has become smaller, the gradient vector distribution has a slightly larger fluctuation width compared to FIG.

  FIG. 13 is a distribution of gradient vectors obtained by the processing flow shown in FIG. 1 in which the size of the neighboring region is a circle having a radius of 6 with respect to the sample image 2 of FIG. Since the neighborhood area has become smaller, the gradient vector distribution has a slightly larger fluctuation width than that in FIG.

    FIG. 14 is a distribution of gradient vectors obtained by the processing flow shown in FIG. 1 in which the size of the neighboring region is a circle having a radius of 6 with respect to the sample image 3 of FIG. Since the neighborhood area has become smaller, the gradient vector distribution has a slightly larger fluctuation width than that in FIG.

FIG. 16 is a graph showing the results of tests using 24 variations. The vertical axis represents the calculated angle (°), the horizontal axis represents the radius of the neighborhood (pixel size), and 6 line graphs showing any one of (3 types of images) × (2 neighborhood shapes) are shown. Yes. The relationship is
gr-s-ta80r2n4: Sample image 1, near square
gr-s-tl80w20r2n4: Sample image 2, near square
gr-s-tls80r2n4: Sample image 3, near square
gr-c-ta80r2n4: Sample image 1, circular neighborhood
gr-c-tl80w20r2n4: Sample image 2, near a circle
gr-c-tls80r2n4: Sample image 3, near a circle

(In the drawing, the hyphen "-" is displayed as an underscore). The correct answer is 102.345 ° as described above. Each graph must lie on a horizontal line showing 102.345 ° as the neighborhood radius increases. gr-c-ta80r2n4 and gr-c-tl80w20r2n4 indicate the vicinity of 102.3 ° which can be said to be almost correct when the radius is 6 or more. The gr-c-tls80r2n4 also has a slight error compared to the former two, but with a neighborhood radius of 6 or more and 102.0 °, a value very close to the correct answer. On the other hand, in the case of a square neighborhood, the neighborhood is closest to the correct answer when the radius is 2 pixels, and the deviation from the correct answer increases as the radius increases. The shift is remarkable in gr-s-tls80r2n4, but the effect that the calculated value of the gradient is pulled in the 45 ° direction when the contour line is applied to the corner near the square becomes stronger as the neighborhood radius increases. Because it goes. This will be described in more detail with reference to FIG. 20. In the case where the neighboring area slightly covers a steep outline near the outer periphery thereof, when the shape of the neighboring area is square (FIG. 20A), the actual contour is obtained. Regardless of the slope of the line, the calculated gradient vector tends to deviate in the 45 ° direction. On the other hand, when the shape of the neighborhood region is circular (FIG. 20B), such a bias does not occur. In any case, at least in the case of the three sample images used in this experiment, it can be seen that the angle can be detected accurately when the neighborhood radius is about 5 to 8 pixels in the vicinity of the circle.

FIG. 17 is a plot of the contribution rate (contribution rate of the first principal component) on the vertical axis and the neighborhood radius (pixel) on the horizontal axis for each of the 6 sets of (3 types of images) × (2 types of neighborhood shape). It is a thing. The contribution ratio represents the goodness of fitting when a straight line is applied to data by principal component analysis. As shown in the graph of FIG. 17, the three broken lines gr-c-ta80r2n4, gr-c-tl80w20r2n4, and gr-c-tls80r2n4 using the circular neighborhood have a radius of 4 or more and the contribution ratio exceeds 99.9%. It can be said that the image is suitable for the method of the present invention. That is, it can be said that the reliability of the angle value obtained when processing under the condition of a radius of 4 or more near the circle is extremely high.

  According to the applicant's experiment, the value of the contribution rate is often less than 70% for an original image in which the angles of the edges are not uniform and the dominant straight line angle is unclear. Therefore, when the contribution ratio of the first principal component is less than 70%, it is assumed that there is no dominant straight line or straight line group having a certain angle in the original image, and the calculated angle value is adopted. It is possible to take measures such as making an error without doing so. That is, the contribution ratio of the first principal component is useful as an index of the reliability of the result.

  Next, an image processing apparatus equipped with an angle detection processing function based on the image processing method described so far will be described. FIG. 18 is a schematic configuration diagram of an image processing apparatus 100 according to an embodiment of the second invention of the present application. The image processing apparatus 100 can be realized by connecting a display unit 120, an input unit 130, and an image input unit 140 to a general-purpose computer main body and mounting a program for performing angle detection processing. The display unit 120 is a monitor display that displays display data in the display memory of the display interface 102 in color. The input unit 130 is a keyboard and / or a mouse. In addition, you may comprise by the touchscreen with which the display part and the input part were united.

  The image input unit 140 includes an optical lens system, an image sensor, and an AD conversion circuit. The image processing apparatus 100 includes a central processing unit (CPU) 101, a display interface 102, an input interface 103, an image input interface 104, and a storage device 106. These are connected by a bus 109. The storage device 106 is connected to an OS (operating system) code that provides a user with the basic functions of the display unit 120 and the input unit 130 connected to the image processing apparatus 100, image data of a processing target image, and an image input device. The program code for realizing the image acquisition function for controlling the image input interface 104 to input the image data, the program code for detecting the angle from the image data, the main menu of the image processing apparatus 100 are displayed, and the operator's instruction is displayed. Various control program codes that are received and call necessary functions are recorded. The program code for angle detection is called and executed by the CPU 101 to obtain means for calculating a gradient vector for each pixel, means for calculating a covariance matrix around the origin for the gradient vectors of all pixels. The principal component analysis is performed based on the covariance matrix and functions as a means for determining an angle for obtaining a direction orthogonal to the first principal component.

  As described above, when an image processing method according to an embodiment of the present invention or an image processing apparatus in which the image processing method is installed as software is obtained, a gradient vector for each pixel is obtained and considered as a set of sample values. It is a simple method that only performs component analysis. Conventional steps such as setting the step size required for histogram processing suppresses the upper limit of measurement accuracy, and handling the direction information equivalently without considering edge strength information makes it easier for noise to affect the measurement result. It has an excellent feature that it can avoid the disadvantages of the technology and can obtain the result obtained by a single statistical process. Furthermore, the contribution ratio of the first principal component obtained as a result of the principal component analysis can also be used as an index for determining whether the target image is an image suitable for the present processing method. The fact that such a reliability index can be confirmed together is also an advantageous feature of the present image processing method that is not available in the prior art.

As described above, the image processing method described in the embodiment of the present invention and the image processing apparatus equipped with this algorithm are used in an aerial photograph process as an angle correction function of an image reading apparatus in an industrial product manufacturing process or inspection process. It can be used in various industrial fields. In particular, it is suitable for processing for handling the following images.
1) It consists of a high brightness area (white area) and a low brightness area (black area), and the boundary between the two areas is an oblique straight line. 2) White area has a black thin line or a certain width. 3) An image in which the black belt region is drawn obliquely or its inverted image 3) An image in which parallel lines with various black thicknesses are obliquely drawn in the white background region, or its inverted image

100 Image processing apparatus 101 Central processing unit (CPU)
DESCRIPTION OF SYMBOLS 102 Display interface 103 Input interface 104 Image input interface 106 Storage apparatus 120 Display part 130 Input part 140 Image input part

Claims (5)

An image processing method for detecting an angle of a straight line dominant in a target image,
Obtaining a gradient vector for the pixel value of each pixel of the image;
Calculating a covariance matrix around the origin for the set of gradient vectors;
An image processing method comprising: performing principal component analysis based on the covariance matrix and determining an angle of a straight line dominant in the target image by obtaining a direction orthogonal to the first principal component. The step of obtaining the gradient vector includes:
Each pixel is set as a pixel of interest sequentially,
Define a predetermined neighborhood area centered on the pixel of interest,
Perform multiple regression analysis in the neighborhood,
The image processing method according to claim 1, wherein a gradient vector is determined based on a mathematical expression representing the obtained approximate plane. The neighborhood region is
The image processing method according to claim 2, wherein the image processing method includes pixels within a circle having a fixed radius centered on the pixel of interest. Whether the target image is an image suitable for the image processing method according to any one of claims 1 to 3 is determined based on a contribution ratio of a first principal component obtained as a result of principal component analysis based on the covariance matrix. The image processing method according to claim 1. An image processing apparatus that detects an angle of a dominant straight line in image data to be processed,
Means for calculating a gradient vector of pixel values in each pixel;
Means for calculating a covariance matrix around the origin with respect to a gradient vector of all pixels;
An image processing apparatus comprising: a component that performs principal component analysis based on the obtained covariance matrix and determines a direction orthogonal to the first principal component and a desired angle.