EP3314573A1

EP3314573A1 - Method of image segmentation

Info

Publication number: EP3314573A1
Application number: EP16742345.8A
Authority: EP
Inventors: Vincent ARVIS; Michael CHAUSSARD; Michel Bilodeau
Original assignee: Compagnie Generale des Etablissements Michelin SCA
Current assignee: Compagnie Generale des Etablissements Michelin SCA
Priority date: 2015-06-29
Filing date: 2016-06-28
Publication date: 2018-05-02
Also published as: CN107851317A; US20180197293A1; FR3038110B1; FR3038110A1; WO2017001765A1

Abstract

The invention relates to a method of segmenting an image, representative of an object whose external surface exhibits relief patterns, into a first zone comprising patterns and a second zone not comprising any, the method comprising the following steps: a step of thresholding in the course of which the grey level image is transformed into a binary image, a step of evaluating the number of patterns on each line of the image, and a step in the course of which, as a function of the results of the previous steps which make it possible to obtain a number of patterns in the image, a first set of pixels of the image representing patterns is determined.

Description

Image segmentation method

FIELD OF THE INVENTION

The invention relates to the field of visual control of manufactured products whose outer surface has elements in relief. This invention can be applied in particular, but not exclusively to products having periodic patterns, for example tires.

This visual check allows, for example, to detect any surface defects resulting from manufacture. At present, the visual inspection and is still most often called upon the dexterity of the operators responsible for detecting any visible imperfections on the surface of the objects. However, with the advancement of processing power of computer resources, manufacturers now see the possibility of making these automatic control tasks

For this purpose, different means of illumination and digital imaging are used to acquire images, for subsequent digital processing to detect imperfections previously detected visually by operators. These imaging means make it possible to take different images, whether in two dimensions or in three dimensions, of the inner and / or outer surface of the tire to be inspected.

In the case of objects having certain areas on which relief patterns are present, it is useful to apply different treatments to the patterned areas, and areas without patterns. For this purpose, it is useful to be able to differentiate, on an image of a tire, the different zones in the presence.

[0005] Various techniques are known which aim at making such a differentiation, but none of them has sufficient robustness to be applied to a wide range of manufactured products. In addition, the known solutions have treatment times too long to be acceptable in an industrial environment.

The present invention is therefore to provide a segmentation solution to overcome the aforementioned drawbacks. GENERAL DESCRIPTION OF THE INVENTION

The present invention is therefore to provide a method for segmenting an image of the outer surface of a manufactured product so as to distinguish the areas of the image having patterns in relief, other areas.

Thus, the present invention relates to a method of segmenting an image, representative of a manufactured product whose outer surface has patterns in relief, in a first zone having patterns and a second zone not comprising the method comprising the following steps:

A thresholding step during which the image in gray level is converted into a binary image,

A step of evaluating the number of patterns on each line of the image, and

A step in which, according to the results of the preceding steps which make it possible to obtain a number of patterns in the image, a first set of pixels of the image representing patterns is determined.

In the rest of the application, we will sometimes designate an image to which a method according to the invention is applied by the expression "input image".

By convention, throughout the application, we will use the notations shown in Figure 1. Thus, we use the notation L and H respectively for the width and height of an input image, and the point I (x, y) will be marked in the (x, y) mark shown in this figure.

It has been found that some input images have a curvature along the rows and columns. This curvature appears by measuring the average of the rows and columns of the image: each line, and each column of the image, has a different mean, correlated with its position in the image. This effect, due to the natural curvature of the tire (which differs depending on the type of tire) and the mechanical stress experienced by the tire during its acquisition, must be corrected before any other treatment, so that all the elements of the tire have a comparable height regardless of their position in the tire.

For this purpose, in a particular embodiment, the method comprises an initial step in which it performs a flattening of the image.

In an exemplary embodiment, this step of flattening the image comprises a step of detecting a carrier signal on which the patterns are based. To do this, we realize a simple moving average along the lines: at each pixel of the cleaned image we calculates the average of the neighboring pixels (located within a certain distance) and located on the same line; we then subtract this value from the pixel.

These operations can be defined in the following way: Let a two-dimensional image I, and a positive integer r, define the operation AvgSub as a function taking as input a two-dimensional image I and outputting an image of the same size

The operation consists in calculating, in a horizontal window of size 2r +1 centered on the pixel (x; y), (considering the specificity of recollement of the right and left edges, expressed by the minimum), the average pixels, and to subtract the latter from the value of the pixel (and this repeated for all the pixels of the image).

The segmentation method of the present invention could also be used on images representing all types of objects, and not necessarily tires. In this case, the flattening step would not necessarily prove useful, since it is made necessary in the case of a tire due to the rounded shape of the object.

Preferably, we obtain the flat image, which we will now call "flat image", subtracting the minimum of the image to this average value to obtain only positive values. For this last operation, it is advantageous to choose a radius to retain the patterns while removing the carrier. In addition, it turns out that, although performed only on the lines of the image, this operation also makes it possible to overcome the curvature along the columns of the image.

Once the input image is flat, the next step of the method consists in performing a thresholding operation, in order to transform the flat image, which is in gray levels, into a binary threshold image. This makes it possible to create an output mask comprising a first set of pixels containing the patterns, and a second set of pixels comprising the other elements.

To decide whether a pixel belongs to the output mask, three criteria are verified:

The first two consist in calculating on the one hand the average of the gray levels of all the pixels of the row (respectively of the column) on which is a pixel, on the other hand the standard deviation, and of add up these two values.

- The third criterion consists in verifying that the pixel belongs to a valid line, that is to say a line that is not too close to the top or the bottom of the image; or a line which does not contain too many outliers, or a line that is not too close to a line containing a large number of outliers.

Depending on the output mask thus defined, it is then possible to detect, in the flat image, lines in which patterns are present. At the end of this detection, a one-dimensional image is obtained, of the same size as the height of the flat image.

This detection step is performed as follows:

for each line of the flat image, the variance of the gray levels of the pixels not belonging to the binary threshold image is calculated, which amounts to excluding the possible patterns,

the calculation is carried out a second time by horizontally expanding the threshold image, which amounts to excluding more pixels from the calculation.

The lines on which patterns are present in the flat image will present a result significantly different from other lines. Indeed, the variance calculation performed excludes the cast shadow phenomenon caused by the patterns, which leads to low values compared to the average. Thus lines with patterns will have a big difference between the two variance values, whereas streaked lines do not.

For each line, the ratio between the two variance values calculated in this way is then carried out. If this ratio becomes greater than a predetermined threshold, it will be considered that the line has a streak.

It is noted here that we use a specificity of the patterns in relief, namely the shadow phenomenon, to detect them. Other methods could be considered to achieve this detection step, however it was found that the method described here provided the best results.

It should be noted once again that this pattern detection step is only useful for certain types of manufactured objects. Indeed, in the case where the patterns are present only on a portion of the product, it is necessary to detect, on an image representing the product, the lines that actually contain patterns.

The next step in a method according to the invention is to evaluate the number of patterns on each line. To do this, we implement the following steps: first, we calculate the variance along each of the columns of an input image, to form a one-dimensional image with a pattern-like regularity. According to the examples, the input image will be chosen as the flat image or the thresholded image.

two successive Fourier transforms are applied to this one-dimensional image to obtain a decomposition of the image in the frequency space,

we then look for one or more maximums of the decomposed image. Indeed, if a value of abscissa x is high, it means that there is in the image a pattern that repeats every x pixels. Therefore a maximum of the image can correspond to a period of patterns.

However, it has been found that in some cases, a maximum of the decomposed image does not correspond to a desired period of the patterns, but to a harmonic, that is to say a value due to a group of patterns. which is repeated regularly in the image. Therefore it is useful in a particular embodiment to look at the fractions of the determined maximum to detect possible candidates for the pattern period.

In a final step of a method according to the invention, the best components that can be patterns are detected in the thresholded image and they are retained in a result set. To do this, we go through the related components of the thresholded image by decreasing size, and we retain them if two conditions are met:

First, the related component, if it was added to the result set, must not result in a row of the thresholded image having a number of elements in the result set. greater than the number of patterns detected in the previous step, and

- The related component must also belong to a valid line as defined in paragraph [0019]

At the end of this last step, we then obtain a first set of pixels, corresponding to the result set, comprising all the patterns of the image.

However, it was found that the result set could, in some cases, include supernumerary items that it would be useful to delete. For this purpose, in one embodiment, a method according to the invention also comprises the following steps:

a step of reassessing the number of patterns in the image, and a step of filtering the determined set of pixels, as a function of the number of reevaluated patterns, to obtain a second set of pixels of the image.

In another embodiment, a method according to the invention further comprises a step during which one completes empty spaces of the image to obtain a third set of pixels of the image.

In one embodiment, a method according to the invention comprises a step in which is eliminated, the third set of pixels, supernumerary components to obtain a fourth set of pixels of the image representing patterns.

It has been found that a slight noise present in the image on which the method is applied could disturb the detection of the patterns, and thus lead to a bad segmentation. To remedy this, in one embodiment, it is useful to provide a preliminary step of cleaning the image with morphological filters. Consider, in a particular example, an image where the topography of a tire is represented, that is to say that the value of each pixel of the image represents the height of the vicinity of the corresponding point in the tire. On such an image, the high gray level values indicate high altitude pixels, while the low gray level values indicate low altitude pixels. Thus, the patterns present on a tire resemble, in a topographic image, extensive mountain ranges, without necessarily being very high, while the noise present in the image appears in the form of peak of very (mountains) or very low (canyons) altitude but small.

The objective of this step is to remove these peaks value. For this purpose, a morphological opening is used, which consists of removing all the narrow mountains, whatever their altitude, followed by a morphological closure which consists of removing all the narrow canyons, whatever their depth.

The opening operation consists firstly in replacing the value of each pixel of the image by the minimum value of the pixels located in a certain neighborhood, then in starting the operation again, this time taking the value Max. The closing operation consists in performing the same two operations, but in the opposite direction (first the maximum value, then the minimum value). The chosen neighborhood consists of the set of pixels located on the same line, that the pixel studied (it is called opening and closing by a linear structuring element) and at a distance less than a certain threshold. Preferably a threshold value is chosen to eliminate, on each line, mountains and canyons of small size. However, this choice of radius must represent a compromise between a value too low that would not allow a proper cleaning, and too high a value that could lead to the removal of some elements of interest reasons.

DESCRIPTION OF THE BEST MODE OF PRODUCTION

We will describe below the detail of each step of the method in the case of a specific product, namely a tire. In the description of this embodiment, the raised patterns will be called streaks. In this example, the cleaning step is performed beforehand. Thus, if we call CEA the starting image, the cleaned image will be:

As described previously in this text, the flattening step is performed using an AvgSub operation.

The following calculation is then carried out: in a horizontal window of size 2r +1 centered on the pixel (x; y), the average of the pixels is calculated, and the latter is subtracted from the value of the pixel (and this is repeated for all pixels in the image). Preferably, we obtain the flat image, which we will now call "flat image", subtracting the minimum of the image from this average value in order to obtain only positive values:

The calculation of the thresholded image is performed as follows:

A function is built that allows to assign a label to each line y of the input image. If we consider an input mask PNM, representing the outliers of the input image (the outlier pixels are at A, and the others are at 0, and we proceed in two steps: first, defining a first temporary unidimensional image, of the same size as the height of the input images, and such that:

The first condition makes it possible to mark as invalid the first ten and the last lines of the image, and the second condition makes it possible to mark as invalid all the lines having more than 5% of pixels marked as aberrant in the PNM image.

One puts Line_NOTOK_PNM = 0 and Ligne_OK = 2. The Line image, which will give us the label of each line, is obtained by propagating the labels of invalid lines, thanks to an erosion, as follows:

- We can then define the output mask by realizing a threshold based on our previously defined criteria and on the labeling of the lines, starting from the previously flattened image (see equation 2)

This formula outputs a mask of size equal to the size of the input images, and where a pixel will be present if it is on a valid line (first condition), if its value is greater than the average plus the standard deviation pixels in the same line as him

(second condition), and if its value is greater than the average plus the standard deviation of the pixels of the same column as it (third condition).

The step of detecting lines with streaks is performed as follows:

- We start by calculating, for each line y of the flat image (see equation

2), the variance of the pixels not belonging to the thresholded image, and the variance of the pixels not belonging to the dilated thresholded image of 60 pixels:

As previously explained, the ratio between these two values is calculated in a new unidimensional Image image of size equal to the height of the input images. We also calculate, in the unidimensional image Report, the relationship between the two values but by cleaning with openings and closures, as follows:

The image Report will serve us later in this section, while the Score image will be used in the following sections. For each valid line of the image (using the Line image defined in Equation 3), we look for the two extrema of Report values.

After several experiments, we have established that the variance ratio threshold which acts as a limit is 1.5: if the min value Ratio is greater than this threshold, then we consider that the streaks are present on all the lines of the image, if the max ratio is less than this ratio, so the image has no streaks.

What is more complicated to determine happens when these two extrema are located on both sides of the threshold. In this case, the value of 1, 5 no longer plays the role of a satisfactory threshold, and it is necessary to find another different threshold according to the images. Our technique consists in choosing a threshold value s so as to divide the lines of the image into two classes (with streak, we suppose, and without streaking, we suppose) so that the variances of the two classes are very close (this procedure is discussed in the following):

If several thresholds are candidates to be the minimum, we will take the lowest threshold. You can build the Ligne2_tmp image which assigns a label to each line of the input image by assigning, for any

We put Ligne_NOTOK_NOTSTRIE = l.

The final Line2 image, which assigns the final label of each line of the input image, is a mix between Line and a cleaned version of Ligne2_tmp.

For everything we ask

As previously explained, the Line2 image that assigns a label to each line of the input image is composed by mixing the line and

Ligne2_tmp. If all lines have a satisfactory variance ratio (greater than 1.5), then the streaks are present over the entire height of the image and Line2 will be a copy of Line. Otherwise, if only certain lines have a satisfactory variance ratio, then Line2 is equal to a cleaned version of Ligne2_tmp except for lines with too many outliers, where the NOTOK PNM Line label is copied (this operation is carried out thanks to the use of the minimum);

The cleaning of Ligne2_tmp is done thanks to an opening, followed by a closing. However, we notice in images with streaks that they do not all stop on the same line: they disappear gradually, and do not disappear in the same lines. For this reason, erosion of

Ligne2_tmp to widen the labels of the strapless lines and to include, as a precaution, these "fuzzy" areas as strapless lines.

The step of evaluating the number of streaks on each line is preferably carried out as follows:

The variance of each of the columns of an input input image which is, according to the embodiment, the flat flat image or the threshold threshold image, is calculated. This calculation is performed excluding, thanks to Ligne2, the elements located on lines of no streaks. In addition, erosion of the Line2 elements is carried out in advance to remove valid line tags from these areas:

The image Var_col thus obtained is a one-dimensional image of the same size as the width of the input images. We see that this image has a repeating pattern as many times as there are streaks in the image. We will then perform a Fourier analysis of this image Var_col to find the number of striations present on each line of the image. This calculation is as follows:

The size of the image F is equal to the largest power of two strictly less than L (Input) plus 1. Thus, for images 40,000 pixels wide, F is 32769 pixels. The image F is such that a peak on F (1000) means that there is, in the image, a pattern locating every 1000 pixels. Therefore, it is useful to look for the peaks of F. However, beforehand, we clean the image with an opening and closing as follows:

It can be seen that the structure of F2 is often the same regardless of the input image: in a first third of the image, interesting oscillations are observed placed on a carrier signal, then the signal remains flat, collapse to half of the picture, and go up a bit to stay flat in the last half. We will thus look for the minimum of the image after the first third, and place 0 after this minimum, to obtain an image F3 as follows:

A geodesic reconstruction of an image D in F3 is then carried out in order to recover the carrier signal which can then be suppressed. The image D is an image of the same size as the image F3, having the value in all points except the abscissa 0 where it is F3 (0). We then look for the position of the maximum of F 4: ^" $ ■

It has been found that this maximum does not always represent the spatial period of the streaks in the image. Indeed, it is necessary to take into account the phenomenon of harmonics in the image. For this purpose, fractions of the previously determined maximum will be tested, and a maximum p _n of F4 is sought in a certain neighborhood Rn as follows:

We then construct the set Nb_strie which contains all the candidates to the number of streaks in the image:

The step of detecting the best candidates of the binary image is performed as follows:

Let C be the set of related components of Threshold; C the set of elements of C that appear on at least 20 lines labeled as valid, and let S be the sequence of elements of C sorted by decreasing size. We then have:

I! ½ "· kj i: ^' _: , i ^: € {$, i <*;ί¾> ≠ ¾

The candidate set is then constructed by adding the elements of S if they do not conflict with the elements already added in Candidate. For this, we build a sequence of sets R: Finally, the present invention provides a method implementing a number of original features compared to known solutions of the state of the art.

Thus, the means for detecting the lines of the image where patterns are present are different from the known solutions, since the principle of taking a mask of candidate pixels to belong to patterns, and to observe how the variance (calculated excluding the elements of this mask) evolves according to the expansion of this mask, is original. Indeed, in the present invention, looking for relief elements that cause a shadow projected on the image, and detecting the lines of the image having patterns by trying to detect the lines having a drop shadow.

Moreover, the present invention aims to propose a method for dividing the lines of the image into two categories: those where patterns are present, and those that do not. It has been found that the known solutions, namely the conventional approach of minimizing intra class variance or maximizing inter-class variance, do not work (especially since classes can have strong variances). In the present invention, means are used to equalize the class variances using a linear time algorithm, which overcomes the disadvantage of known solutions.

In addition, the method of counting patterns, to know how many patterns are normally present on the image in the absence of defects has two inventive elements:

The first lies in the fact of making a Fourier transform not on each line of the image, as presented in the known solutions, but on a signal in one dimension, representative of the lines of the image. This signal is obtained by calculating the variance of each column of the image: thanks to the relief of the patterns and their drop shadow, we obtain a signal with the same period as the patterns of the image. This solution makes it possible to reduce the calculation times implemented.

The second element comes from the fact of carrying out morphological operations on the results of the Fourier transform in order to clean it of parasitic elements which could distort the result obtained.

Finally, a method according to the invention implements, for the selection of the best candidate components that may belong to a streak, a series of placement operations then removal of the candidates by decreasing as and when constraints on their position. This pattern of decreasing constraints as it goes is the opposite of all the solutions of the state of the art which generally consist in increasing the constraints over time.

Claims

A method of segmenting an image, representative of a manufactured product whose outer surface has patterns in relief, in a first zone having patterns and a second zone not comprising them, the method comprising the following steps:

A step of evaluating the number of patterns on each line of the image, and

The method of claim 1, including an initial step in which the image is flattened.

The method of claim 1 wherein the step of flattening the image comprises a step of detecting a carrier signal on which the patterns are based.

Method according to one of the preceding claims, comprising a step, before the step of evaluating the number of patterns, of detecting the lines of the image comprising patterns.

The method of claim 1 or 2, further comprising the steps of:

a step of re-evaluating the number of patterns in the image, and

a step of filtering the determined set of pixels, as a function of the number of reevaluated patterns, to obtain a second set of pixels of the image.

The method of claim 3, further comprising a step of completing blank spaces of the image to obtain a third set of pixels of the image.

A method as claimed in claim 4, including a step of eliminating from the third set of pixels supernumerary components to obtain a fourth set of pixels of the image representing patterns. Method according to one of the preceding claims comprising the prior step of cleaning the image with morphological filters.