WO2018000731A1

WO2018000731A1 - Method for automatically detecting curved surface defect and device thereof

Info

Publication number: WO2018000731A1
Application number: PCT/CN2016/108866
Authority: WO
Inventors: 黄茜; 黄梓淳; 周洲
Original assignee: 华南理工大学
Priority date: 2016-06-28
Filing date: 2016-12-07
Publication date: 2018-01-04
Also published as: CN106127780B; CN106127780A; SG11201811381UA

Abstract

A method for automatically detecting curved surface defect and a device thereof, the method comprising: (1) a training phase: acquiring sample images, constructing a training set, performing manual defect identification on the images in the training set, and marking all areas where defects appear; executing a defect pre-targeting step for each image in the training set, so as to obtain all areas R where the defects may appear; comparing R with all of the manually marked areas where the defects appear, dividing into negative samples and positive samples according to an overlap degree between the two; performing offline training of a deep neural network model according to the positive samples and the negative samples, and outputting types and specific coordinates of defect areas; (2) online detection phase: acquiring current curved surface images to be detected, executing the defect pre-targeting step, obtaining a set R, and inputting the set R into the network model to obtain the types and the specific coordinates of the defect areas. The method and device have the advantages of being highly adaptable and real-time, as well as having a high identification accuracy rate.

Description

Surface surface defect automatic detecting method and device thereof

Technical field

The invention relates to the field of image processing and deep learning research, in particular to a method and device for automatically detecting a curved surface defect.

Background technique

At present, the automatic detection of surface defects on small curved surfaces in the industrial field mainly adopts the following algorithm:

First, extract image features and perform image processing. This method generally uses an industry expert to manually design an image feature extraction method based on the defect characteristics, and then matches the real-time image;

Secondly, the back propagation neural network method is adopted. The image feature is extracted by artificially designed feature extraction method and then sent to the input end of the neural network. The neural network is used to classify whether the image is defective or not.

Third, the use of support vector machine and other classifiers to classify the manually extracted features.

The above methods have the following disadvantages:

1. All of them are manually selected, which may result in the loss of useful information of the image due to inaccurate subjective judgment, resulting in a decrease in recognition accuracy;

2, the method of extracting features depends too much on the setting of parameters, and the applicability is not strong. For different types of defects, it is often necessary to reset the parameters;

3. The use of neural network and other classifiers can only classify the entire image, and it is impossible to accurately locate the position of the defect, so that the requirements for online detection cannot be met.

Therefore, it is of great practical value to find a method and device for automatically detecting curved surface defects with high adaptability, high real-time performance and high recognition accuracy.

Summary of the invention

The main object of the present invention is to overcome the shortcomings and shortcomings of the prior art, and to provide an automatic surface defect detection method, which has the advantages of high adaptability, high real-time performance and high recognition accuracy.

Another object of the present invention is to provide an apparatus for realizing the above-described automatic surface defect detection method, which is easy to operate, simple and stable in structure.

The object of the present invention is achieved by the following technical solutions: an automatic surface defect detection method, Including steps:

(1) Training stage: collecting sample images, constructing a training set, performing artificial defect recognition on the images in the training set, and marking the areas where all the defects appear;

Perform defect pre-positioning steps for each image in the training set:

(1-1) Calculate the Euclidean distance of the RGB space 3 channels between each pixel in the image and its adjacent 8 directions, and use the Euclidean distance as the weight, and each weight represents the two connected to the edge. The degree of dissimilarity of the region, the following merge operation is performed for each edge to obtain the initial partition region set R: as long as the inter-class difference between the two regions is greater than the intra-class difference of any one region, it is merged into a new region, otherwise Not merged;

(1-2) Initializing the inter-area similarity set S is an empty set;

(1-3) calculating the similarity of each two adjacent regions in the segmentation region set R, and adding to the set S;

(1-4) Find the highest similar value s _max in the set S, merge the two regions corresponding to the value, add the newly merged region to the set R, and delete the set S related to the two regions. For all similarities, recalculate the similarity of the new segmentation region set R; repeat the above operations until the set S becomes an empty set, and finally the resulting set R is the region where all defects may occur;

Comparing the set R obtained from each image in the training set with the area where all the defects are manually marked, and dividing the negative sample and the positive sample according to the coincidence degree of the two;

Taking the positive and negative samples as input, the offline training of the deep neural network model is carried out. The training process adopts the stochastic gradient descent method, and the output is the category and specific coordinates of the defect area;

Obtained a trained deep neural network model;

(2) Online detection stage: collecting the surface image of the current surface to be detected, performing the defect pre-positioning step, obtaining the set R, and inputting the set R into the trained deep neural network model to obtain the category and specific coordinates of the defect area.

Preferably, when constructing the training set according to the sample image, at least one of the following methods is used for performing amplification: performing a random angle rotation, a single translation, a zoom, a flip, a stretch, and a crop. Thereby, the collected sample image can be expanded by a maximum of 6 times to effectively prevent over-fitting of the training.

Further, the rotation randomly selects an angle from 0° to 360°, the translation is a random offset of -8 to 8 pixels, and the scaling factor of the scaling is randomly selected between 1/1.5 and 1.5, The flipping includes two directions, horizontal and vertical. The stretching refers to stretching the short side, and the stretching factor is randomly selected between 1/1.2 and 1.2, and the cutting is manually selecting the target region portion. Amplifying the sample using the above parameters, after The model obtained from the continuous training will be more accurate.

Preferably, before performing the defect pre-positioning, the average value of the three-channel pixels of all the images in the training set is calculated, and then the average value of each channel is subtracted from the three-channel pixel value of each image to obtain a new sample image. Through the above processing, the convergence speed can be improved.

Further, the order of the new sample images in the training set is randomly disturbed before the defect pre-positioning is performed. Thereby making the established training network more accurate.

Specifically, the difference between the classes is defined as the smallest weight among all the edges connecting the two regions, and the intra-class difference is defined as the largest weight of all the edges in the region plus the limiting coefficient.

Where s is the number of all pixels in the region, and k is the segmentation threshold.

Preferably, in the step (1-2), the similarity adopted by the inter-region similarity set S includes three types of color similarity, texture similarity and size similarity, wherein:

Color similarity

The range of color gradation are divided into M sub-intervals, called M bins, obtaining Mbins each color channel histogram of the region a and b, a and b are each region to obtain a 3M-dimensional vector C _a, C _b;

Texture similarity

First, the image is converted into a grayscale image, and the gradient histogram of Nbins is obtained for the region a and the region b, and then the region a and the region b obtain an N-dimensional vector T _a , T _b ;

Size similarity

s(a) is that the area a contains the number of pixels, s(b) is that the area b contains the number of pixels, and s(im) is the number of pixels in the entire image.

Preferably, the steps of dividing the negative sample and the positive sample according to the degree of coincidence are:

Coincidence degree

Where S _p is the area of the predetermined bit area, S _t is the real target frame area, S _o is the area of overlap of S _p and S _{t , the} area of coincidence is 0 to 50% as the negative sample, and the area of coincidence is 70% or more. As a positive sample, the rest are ignored.

Preferably, after the category and the specific coordinates of the defect area are obtained in the step (2), the non-maximum suppression algorithm is used to eliminate the redundant target frame, and the optimal position of the defect area is determined.

The device for realizing the above automatic surface surface defect detection method comprises a plurality of cameras, and a sliding table, a bottom plate base, a motor, a motor controller, a light source, a light source controller and a host computer, and each camera is fixed in a three On the degree of freedom camera mount, the three-degree-of-freedom camera bracket, the sliding table, the light source, and the light source controller are all fixed on the bottom plate base, and the motor is fixedly connected to the sliding table through the coupling, and the camera passes The RJ45 network interface is connected with the host computer; during the detection, the workpiece is horizontally placed on the sliding table, and the upper computer sends a motion command to the motor controller, and the motor controller controls the motor to drive the workpiece on the sliding table to move in the area where the camera views the field of view. A light source with a light source controller illuminates the camera to obtain a clear image of the entire surface of the workpiece to be inspected.

Preferably, the device comprises two industrial cameras, the first industrial camera is mounted at a height of 20 cm to 30 cm above the workpiece, and is at an angle of 30 to 45 degrees downward from the horizontal plane, and the second industrial camera is mounted below the level of the workpiece. 20cm ~ 30cm, and upward angle of 30 ° ~ 45 ° with the horizontal plane. The workpiece is photographed by the first and second industrial cameras of different angles to ensure that the shooting field completely covers the surface to be inspected.

Compared with the prior art, the present invention has the following advantages and beneficial effects:

1. The metal small curved surface defect automatic detecting device used in the invention can capture high-quality curved defect images clearly and quickly, and ensures high definition of the captured image.

2. The method of the present invention amplifies the amount of data by performing various transformation processes on the defect sample in the sample preparation process, and at the same time prevents the over-fitting of the training to a certain extent.

3. The method of the present invention pre-predicts the defect area based on the idea of graph theory, and then uses the offline training network model to identify all the defect areas of the predetermined position, which can improve the recognition accuracy compared with the whole image; While identifying the type of defect, the network structure also more accurately locates the location of defects in the area.

4. The defect detection process of the invention is automatically performed without manual parameter setting, and has strong applicability to different image collection environments and different types of defects.

DRAWINGS

1 is a schematic diagram showing the hardware structure of the apparatus of this embodiment;

Figure 2 is a skeleton diagram of the apparatus of the embodiment;

3 is a schematic flow chart of the method of the embodiment;

4 is a flow chart of sample image preparation in a training phase in the method of the embodiment;

5 is a flow chart of a defect pre-positioning algorithm in the method of the embodiment;

6 is a flowchart of a network training algorithm in the method of this embodiment;

FIG. 7 is a flow chart of the online detection of the method of the embodiment.

detailed description

The present invention will be further described in detail below with reference to the embodiments and drawings, but the embodiments of the present invention are not limited thereto.

Example 1

As shown in FIG. 1 , an automatic surface defect detecting device of the present embodiment includes two industrial cameras with micro focal length lenses, two three-degree-of-freedom camera brackets and two light sources, a sliding table, a bottom plate base and A motor. The three-degree-of-freedom camera mount 3 and the slide table 6 are fixed to the bottom plate base 9, and the motor 7 is fixedly coupled to the slide table 6 via a coupling. The workpiece 5 is placed horizontally on the slide table 6, and when the workpiece 5 is in the AB position, an image is taken by the industrial camera 1 fixed on the camera holder 3, and when the workpiece 5 is moved to the BC position, the image is taken by the industrial camera 2, The surface light source 4 of the light source controller 10 provides illumination to the camera. The industrial cameras 1 and 2 are connected to a PC (host computer) via an RJ45 network interface, and the motor 7 is connected to a PC. The shaded portion of the workpiece 5 is a curved surface having a small curvature. In order to cover the arc of the semicircle with the field of view, the industrial camera 1 is mounted 20 cm above the horizontal plane of the workpiece 5, and is at an angle of 30° to the horizontal. Industrial The camera 2 is mounted 20 cm below the horizontal plane of the workpiece 5 and is at an angle of 30° to the horizontal plane. The two-angle industrial camera photographs the workpiece at the position AB and the position BC to ensure that the shooting field completely covers the surface to be inspected.

When the length of the AB segment is larger than the radius r of the sharp focus area of the lens, n shots are taken in the AB segment, taking n=AB/r, and the workpiece has a still shooting time of 100 ms.

The process of detection is that the PC sends a motion command to the motor controller, and the control motor drives the workpiece on the slide table to move in the AB and BC regions. When the workpiece is detected to stop moving in the AB region, the camera 1 collects an image and transmits it to the computer through the RJ45 interface. When it is detected that the workpiece stops moving in the BC area, the camera 2 collects an image and transmits it back to the computer through the RJ45 interface until the AB and BC arc surfaces of the workpiece are detected.

2 shows that the apparatus of the present embodiment can be divided into an illumination and imaging component, a mechanical transmission component, and a PC according to functions, wherein the illumination and imaging components include a surface light source, a light source controller, an industrial camera, and a three-degree-of-freedom camera bracket, each of which The camera is fixed on a three-degree-of-freedom camera bracket, and the three-degree-of-freedom camera bracket, the sliding table, the light source, and the light source controller are all fixed on the bottom plate base, and the light source controller is connected to the surface light source through an electrical connection line for controlling the light source. strength. Mechanical transmission components, including motor, motor controller, sliding table and motion control card, the motor is fixedly connected with the sliding table through the coupling, the motor controller is connected to the motor through the electrical connection line, the motion control card is connected with the motor controller, and the motion The control card is connected to the PC through the PCI interface.

The PC is equipped with an automatic defect detection system, which includes two offline modules, sample preparation and network training, and four online modules of motion control, image acquisition, defect pre-positioning and defect detection. Industrial cameras in lighting and imaging components are connected to a PC via an RJ45 network interface to capture online images; PC The machine is connected to the motion control card in the mechanical transmission component through the motion control module to control the start and stop of the motor, speed adjustment, direction change, etc., to drive the workpiece movement on the sliding table.

FIG. 3 is a main working flow chart of the method according to the embodiment, which includes two main steps of offline deep neural network model training and online automatic defect detection. During the training phase, including sample preparation, performing a defect pre-positioning algorithm, simultaneously inputting a set of image samples having obtained a plurality of predetermined bit frames to the deep neural network, performing offline training of the deep neural network model, and then using the trained network model Online detection. The online detection process is to first collect the image of the small arc surface of the workpiece online, perform the same defect pre-positioning algorithm as the offline training, and then input a single image of the obtained predetermined position frame to the deep neural network, and sequentially image the image. Each predetermined bit area is identified and classified, the coordinates and defect categories of the final defect area are obtained, and the detection result is displayed and output.

4 is a flowchart of a sample preparation algorithm according to the embodiment, which includes the following steps:

S1. Acquiring 20,000 images; of course, how many images are collected, and in practical applications, those skilled in the art can adjust themselves.

S2. The following method is used to amplify the image data set, artificially increasing the number of training samples, including performing a random angle rotation (randomly selecting angles from 0° to 360°), and one translation (random offset -8 to 8) Pixels), one zoom (scaling factor is randomly selected from 1/1.5 to 1.5 times), one flip (horizontal and vertical), one stretch (stretching short side, stretching factor from 1/1.2 to 1.2) Random selection (multiple times), one cropping (manual selection of the target area), the image data set is expanded by six times to effectively prevent over-fitting of the training;

S3. Label the image data set, manually select the location of all defects on each image, and save the coordinates of the location and the type of the defect to a text file, and the text file name is consistent with the image name;

S4. Calculating the average value of the three-channel pixels of all the images, and subtracting the average value of the above three-channel pixels from all the image dataset training samples to improve the convergence speed;

S5. Randomly scramble the image order of all data sets without repetition.

FIG. 5 is a flowchart of a defect pre-positioning algorithm according to the embodiment, including the following steps:

S1. Consider each pixel in the image as a single region, and calculate the Euclidean distance of each channel of the RGB space for each pixel and its adjacent 8 pixels as a weight, and each weight represents the The dissimilarity of the two regions connected by the edge, the following merge operation is performed for each edge to obtain the initial segmentation region set R: as long as the inter-class difference between the two regions is greater than the intra-class difference of any one region, they are merged into one New areas, otherwise not merged. The difference between classes is defined as the smallest weight among all the edges connecting the two regions. The intra-class difference is defined as the largest weight among all the edges in the region plus the limiting coefficient.

Where s is the number of all pixels in the area, and k is the segmentation threshold, set to 500;

S2. Initialization area similarity set S is an empty set, and the similarity definition includes three types of color similarity, texture similarity and size similarity, wherein: color similarity

For the regions a and b to obtain a histogram of each color channel 25bins, then regions a and b each get a 75-dimensional vector C _a , C _b ; texture similarity

First convert the image into a grayscale image, and obtain a gradient histogram of 8bins for the region a and the region b, then the region a and the region b get an 8-dimensional vector T _a , T _b ; size similarity

s(a) is that the area a contains the number of pixels, s(b) is that the area b contains the number of pixels, and s(im) is the number of pixels of the entire image;

S3. Calculating the similarity of each two adjacent regions in the region set R, added to the set S;

S4. Adopted in the set S the highest similarity value s _max, two combined values corresponding to the regions, add new regions combined to set R, S delete all related to the similarity of the two regions Then, the similarity of the new region set R is recalculated, and the above operation is repeated until the set S becomes an empty set, and the obtained set R is an area where all defects may occur.

FIG. 6 is a flowchart of an offline network training algorithm in this embodiment, including the following steps:

S1. Calling the above-mentioned defect pre-positioning algorithm on the prepared image data set to perform a defect pre-positioning bit to obtain a plurality of defect pre-positioning regions in each image sample;

S2. Calculate the coincidence degree and coincidence degree of the real target frame of these areas and manually marked defects

Where S _p is the area of the predetermined bit area, S _t is the real target frame area, S _o is the area of overlap of S _p and S _{t , the} area of coincidence is 0 to 50% as the negative sample, and the area of coincidence is 70% or more. As a positive sample, the rest are ignored;

S3. Set the maximum number of iterations to 40000, the learning rate is 0.001, and the number of images to be learned per batch is 128. The positive and negative samples are taken as input, and the deep neural network model is trained offline. The output is the category and specificity of the defect area. coordinate;

S4. The training process uses a stochastic gradient descent method to continuously reduce the gap between network output and expectations.

Figure 7 is a flow chart of the online detection system of the embodiment, comprising the following steps:

S1. Initialize the camera, mechanical transmission components, calibrate the light intensity, and adjust the camera;

S2. Control the motor movement to a fixed position to stop and collect images;

S3. Calling the same defect pre-positioning algorithm as in the offline execution, performing the defect pre-positioning bit to obtain a defect pre-positioning area in each of the images to be detected;

S4. Calling the deep neural network model to classify and identify the pre-location area, and output the defect area Category and specific coordinates;

S5. Use a non-maximum suppression algorithm to eliminate redundant target frames and determine the optimal location of the defect area.

The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments, and any other changes, modifications, substitutions, combinations, and combinations thereof may be made without departing from the spirit and scope of the invention. Simplifications should all be equivalent replacements and are included in the scope of the present invention.

Claims

A method for automatically detecting a surface defect of a curved surface, comprising the steps of:

(1) Training stage: collecting sample images, constructing a training set, performing artificial defect recognition on the images in the training set, and marking the areas where all the defects appear;

Perform defect pre-positioning steps for each image in the training set:

(1-1) Calculate the Euclidean distance of the RGB space 3 channels between each pixel in the image and its adjacent 8 directions, and use the Euclidean distance as the weight, and each weight represents the two connected to the edge. The degree of dissimilarity of the region, the following merge operation is performed for each edge to obtain the initial partition region set R: as long as the inter-class difference between the two regions is greater than the intra-class difference of any one region, it is merged into a new region, otherwise Not merged;

(1-2) Initializing the inter-area similarity set S is an empty set;

(1-3) calculating the similarity of each two adjacent regions in the segmentation region set R, and adding to the set S;

(1-4) Find the highest similar value s max in the set S, merge the two regions corresponding to the value, add the newly merged region to the set R, and delete the set S related to the two regions. For all similarities, recalculate the similarity of the new segmentation region set R; repeat the above operations until the set S becomes an empty set, and finally the resulting set R is the region where all defects may occur;

Comparing the set R obtained from each image in the training set with the area where all the defects are manually marked, and dividing the negative sample and the positive sample according to the coincidence degree of the two;

Taking the positive and negative samples as input, the offline training of the deep neural network model is carried out. The training process adopts the stochastic gradient descent method, and the output is the category and specific coordinates of the defect area;

Obtained a trained deep neural network model;

(2) Online detection stage: collecting the surface image of the current surface to be detected, performing the defect pre-positioning step, obtaining the set R, and inputting the set R into the trained deep neural network model to obtain the category and specific coordinates of the defect area.
The method for automatically detecting a curved surface defect according to claim 1, wherein when constructing the training set according to the sample image, at least one of the following methods is used for performing amplification: performing a random angle rotation, one translation, and one time. Zoom, one flip, one stretch, one crop.
The method for automatically detecting a curved surface defect according to claim 2, wherein the rotation randomly selects an angle from 0° to 360°, and the translation is a random offset of -8 to 8 pixels, and the scaling is performed. The factor is randomly selected between 1/1.5 and 1.5. The inversion includes two directions, horizontal and vertical. The stretching refers to stretching the short side, and the stretching factor is randomly selected between 1/1.2 and 1.2. The cropping is the manual selection of the target area.
The method for automatically detecting a curved surface defect according to claim 1, wherein the average value of the three-channel pixels of all the images in the training set is calculated before the defect pre-positioning, and then the three-channel pixel value of each image is subtracted. The average value of each of the above channels is obtained as a new sample image;

The order of the new sample images in the training set is randomly disturbed before the defect pre-position is performed.
The method for automatically detecting a curved surface defect according to claim 1, wherein the difference between the classes is defined as a minimum weight among all sides connecting the two regions, and the intra-class difference is defined as the largest weight among all the edges in the region. Value plus limit factor
Where s is the number of all pixels in the region, and k is the segmentation threshold.
The method for automatically detecting a curved surface defect according to claim 1, wherein in the step (1-2), the similarity adopted by the inter-region similarity set S includes color similarity, texture similarity, and size similarity. 3 kinds, of which:

Color similarity
The range of color gradation is divided into M sub-intervals, called M bins, and the histograms of each color channel Mbins are obtained for regions a and b, and then regions a and b each obtain a 3M-dimensional vector C a , C b ;

Texture similarity
First, the image is converted into a grayscale image, and the gradient histogram of Nbins is obtained for the region a and the region b, and then the region a and the region b obtain an N-dimensional vector T a , T b ;

Size similarity
s(a) is that the area a contains the number of pixels, s(b) is that the area b contains the number of pixels, and s(im) is the number of pixels in the entire image.
The method for automatically detecting a curved surface defect according to claim 1, wherein the step of dividing the negative sample and the positive sample according to the coincidence degree is:

Coincidence degree
Where S p is the area of the predetermined bit area, S t is the real target frame area, S o is the area of overlap of S p and S t , the area of coincidence is 0 to 50% as the negative sample, and the area of coincidence is 70% or more. As a positive sample, the rest are ignored.
The automatic surface defect detection method according to claim 1, wherein after the category and the specific coordinates of the defect area are obtained in the step (2), the non-maximum suppression algorithm is used to eliminate the redundant target frame, and the defect area is determined. Best location.
The device for realizing the automatic surface defect detection method according to any one of claims 1-8, comprising: a plurality of cameras, and a sliding table, a bottom plate seat, a motor, a motor controller, a light source, and a light source The controller and the upper computer, each camera is fixed on a three-degree-of-freedom camera bracket, The three-degree-of-freedom camera bracket, the sliding table, the light source, and the light source controller are all fixed on the bottom plate base, and the motor is fixedly connected to the sliding table through the coupling, and the camera is connected to the upper computer through the RJ45 network interface; when detecting, the workpiece is horizontally placed On the sliding table, the upper computer sends a motion command to the motor controller. The motor controller controls the motor to drive the workpiece on the sliding table to move in the area of the camera's field of view, and the light source with the light source controller provides illumination for the camera, thereby obtaining a clear workpiece. All images of the surface to be inspected.
The apparatus according to claim 9, wherein said apparatus comprises two industrial cameras, and the first industrial camera is mounted at a height of 20 cm to 30 cm higher than the horizontal plane of the workpiece, and is at an angle of 30 to 45 degrees downward from the horizontal plane. The second industrial camera is mounted 20 cm to 30 cm below the horizontal plane of the workpiece and is at an angle of 30 to 45 degrees from the horizontal.