CN113920075A

CN113920075A - Simple defect detection method and system based on object identification

Info

Publication number: CN113920075A
Application number: CN202111152247.1A
Authority: CN
Inventors: 招嘉焕; 陶洋; 陈小军; 黄章良
Original assignee: Guangzhou Lubangtong IoT Co Ltd
Current assignee: Guangzhou Lubangtong IoT Co Ltd
Priority date: 2021-09-29
Filing date: 2021-09-29
Publication date: 2022-01-11

Abstract

The invention belongs to the field of image recognition, and discloses a simple defect detection method based on object recognition, which comprises the following steps: step 1: obtaining a total image through a camera; the total image has an object to be identified; step 2: training and identifying the type and the boundary of an object in the total image through an AI algorithm, and separating the image of the object from the total image according to the boundary; and step 3: processing the image of the object by a semantic segmentation algorithm to obtain the outline of the object; and 4, step 4: rotating the angle of the image of the object according to the outline of the object, so that the angle of the object in the image is arranged according to a preset angle; and 5: and identifying the parts containing the characteristics from the image of the object according to the characteristics of each part on the object to obtain a detection result. The method combines the traditional computer graphics and AI algorithm, gives full play to the advantages of 2 modes, complements the respective disadvantages, and rapidly obtains the detection result of the defect through smaller calculation power.

Description

Simple defect detection method and system based on object identification

Technical Field

The invention relates to the field of image recognition, in particular to a simple defect detection method and system based on object recognition.

Background

At present, defect detection is performed in an image mode, and there are mainly 2 modes.

Computer graphics

The characteristics on the surface of the object are obtained through common computer graphics operations such as gray scale, expansion, scaling, cutting, deformation and the like, and whether defects exist is judged according to the characteristics.

The following disadvantages and drawbacks exist:

1. the boundaries of the entire object cannot be determined.

If the boundary of the object cannot be determined, it is impossible to distinguish whether all the features can be captured, and it is impossible to locate where on the picture the image operation and defect detection are required.

If the fixture is fixed, the cost is high, and the operation time per time is increased. Nor suitable for simultaneous detection of bulk objects.

If the object is positioned by labeling and finding the characteristics. The operability is not strong and many devices do not have tags themselves. And if the similarity degree of the 2 types of objects is higher, the 2 types of objects cannot be effectively distinguished.

2. It is impossible to distinguish what is an object, and only a simple shape or color can be used for judgment.

If other objects have certain similarity and enter the lens by mistake, the judgment is misjudged.

3. Since the object class is not distinguished, the result cannot be automatically associated with the object, only by manual selection. Not conducive to different types of object reuse systems. For example, the object 1 needs to detect whether the USB interface is defective, the object 2 needs to detect whether the LED lamp is well soldered, and the object 3 needs to detect whether the antenna is installed. In these cases, it is necessary to select an object type in advance and then perform detection.

However, the advantages of computer graphics are computational power for the operating environment and have no requirements.

4. The offset angle of the object cannot be determined and if the rotation angle is large, all will fail for the reference coordinate.

AI algorithm identification

The detection of object defects by AI is emerging in large numbers, effectively improving the shortcomings of graphics.

The following disadvantages and drawbacks exist:

1. has certain calculation force requirement on the operating environment

At present, classification and image recognition algorithms are various, and all algorithms need to carry out operations such as convolution, pooling, activation and the like. And the required calculation power is higher corresponding to the algorithm with higher precision, otherwise, the real-time detection cannot be carried out.

2. The image input size is required

The AI algorithm requires the size of the input, which is typically determined prior to training. The size is too large, and although a tiny object can be found, the training and reasoning time is very long, and the real-time operation can be realized by using equipment with high calculation capacity.

The size is too small, and although the performance is improved, the object with small volume cannot be identified. At present, the resolution is generally in the range of 200X 200-600X 600, and even if the resolution reaches 600X600, the defects on the surface of an object are reduced after the resolution is reduced by one shot picture. If the fine defect needs to be judged, great difficulty is faced.

3. The object to be identified being determined beforehand

Such as LED lamps, vary in shape and have different solder connections around them. Therefore, a great deal of work is required to be spent when the data set is labeled, other devices are not selected, and only the LED devices are reserved.

If the LED is replaced with a different shaped device, it must be retrained to fit.

Different defect types also need to be introduced for training. For example, in the case of welding problems, problems such as cold joint, no welding, welding errors and the like exist, and each type of welding needs to be photographed for training. And the welding points with different shapes also need to be agreed to operate once. It will take a lot of time to do data processing work such as shooting and marking.

Such algorithms are based on training of the AI algorithm, which requires extensive training if accurate information is to be calculated.

For example, CN201811634845.0 discloses an AI algorithm for labeling and identifying automobiles based on two-dimensional labels, which constructs two-dimensional automobile labels, where the two-dimensional labels include a first-dimensional automobile brand label and a second-dimensional automobile part label; constructing a tag set of an automobile brand; when the model generated after the automobile brand labeling training based on the two-dimensional label is used for identifying the automobile brand of the picture in the movie, the original identification result comprises the automobile brand and the similarity in the one-dimensional label and also comprises the automobile part similarity in the two-dimensional label, and the similarity with the automobile brand is recalculated according to the similarity of two dimensions and a fitting algorithm according to the weight, so that the automobile brand and the automobile area appearing in each frame of the video are judged. The problems that a conventional AI method cannot effectively identify a far lens in a film and a shield exists and an automobile brand is difficult to identify are effectively solved.

The problem with using AI algorithms alone is that once matching of parts is involved, the training set is very large, the computational effort is also large, and in addition, the standard requirement for images is high.

The technical problem to be solved by the scheme is as follows: how to quickly identify the common defects occurring in the generation process with less calculation power.

Disclosure of Invention

The invention aims to provide a simple defect detection method based on object identification, which fully exerts the advantages of 2 modes by combining the traditional computer graphics and AI algorithm, complements the respective defects and quickly obtains the detection result of the defect by less calculation power.

The invention focuses on the maximum requirements of practical application scenes, such as whether an indicator lamp works normally, whether an antenna is installed, whether an interface is welded, whether a screw is screwed and the like in common conventional detection scenes. Excessively small defects, such as surface scratches, component soldering, etc., are not considered to be included in the design considerations of the system. This allows a maximum percentage of defect coverage to be achieved at the lowest cost.

Meanwhile, the system is designed to be compatible with various types and objects, the type selection is carried out without manual intervention, and after the type is automatically obtained through an AI algorithm, the corresponding algorithm plug-in is selected for screening, so that the automation degree is improved.

Finally, the system design supports simultaneous detection of multiple types of articles, and the overall detection efficiency is improved.

In order to achieve the purpose, the invention provides the following technical scheme: a simple defect detection method based on object identification comprises the following steps:

step 1: obtaining a total image through a camera; the total image has an object to be identified;

step 2: training and identifying the type and the boundary of an object in the total image through an AI algorithm, and separating the image of the object from the total image according to the boundary;

and step 3: processing the image of the object by a semantic segmentation algorithm to obtain the outline of the object;

and 4, step 4: rotating the image of the object according to the outline of the object, so that the angles of the object in the image are arranged according to a preset angle;

and 5: and identifying the parts containing the characteristics from the image of the object according to the characteristics of each part on the object to obtain a detection result.

In the above simple defect detection method based on object identification, in step 2, a specific method for identifying the type of the object is as follows: and pre-shooting pictures of various types of objects for training through an object recognition algorithm, and carrying out self-defined object recognition by using the trained model.

In the simple defect detection method based on object identification, the step 3 specifically includes:

step 31: classifying pixels in the image of the object through semantic segmentation to obtain a filling color block of each object in the image;

step 32: and drawing the outline of the object according to the filling color blocks.

In the above simple defect detection method based on object identification, the preset angle in step 4 is an angle at which the object is in a horizontal state.

In the above simple defect detection method based on object identification, the step 4 further includes an integrity identification step of identifying the integrity of the object, and the step 5 is performed after the integrity identification step is passed;

the integrity identification step comprises the following steps: calculating the size of the object according to the outline of the object, comparing the size of the object with a preset size, if the size of the object is matched with the preset size, the comparison is passed, and if the size of the object is not matched with the preset size, the comparison is not passed; the preset size is the actual size of the object.

In the above simple defect detection method based on object identification, in step 5, the component is one or more of an interface, a lamp, and a label.

In the above simple defect detection method based on object identification, in the identification process of the interface, if the colors of the interface and other main positions of the object are different, the interface is identified according to the color characteristics.

In the above simple defect detection method based on object identification, in the identification process of the interface, if the colors of the interface and other main positions of the object are different, the interface is identified according to the color characteristics and the size of the area where the color is located.

In the simple defect detection method based on object recognition, in the recognition process of the lamp, the lamp is in a lighting state, and the lamp is recognized according to the brightness characteristic of the lamp.

Finally, the invention also discloses a simple defect detection system based on object identification, which comprises a camera, an AI identification module and an image identification module;

the camera is used for acquiring a total image; the total image has an object to be identified;

an AI identification module: the system comprises a camera, a semantic segmentation algorithm, a camera and a camera, wherein the camera is used for acquiring a total image from the camera, identifying the type and the boundary of an object in the total image, separating the image of the object from the total image according to the boundary, processing the image of the object through the semantic segmentation algorithm, obtaining the outline of the object and adjusting the angle of the image;

an image recognition module: the part recognition device is used for recognizing parts containing the characteristics from the image of the object according to the characteristics of all parts on the object to obtain a detection result.

Compared with the prior art, the invention has the beneficial effects that:

the outstanding effect of the invention is that the method of the invention fully exerts the advantages of 2 modes by combining the traditional computer graphics and AI algorithm and complements the respective disadvantages. The focus is focused on the maximum requirements of practical application scenes, such as whether an indicator lamp works normally, whether an antenna is installed, whether an interface is welded, whether a screw is screwed and the like in common conventional detection scenes. Excessively small defects, such as surface scratches, component soldering, etc., are not considered to be included in the design considerations of the system. This allows a maximum percentage of defect coverage to be achieved at the lowest cost.

Meanwhile, the system is designed to be compatible with various types and objects. The category selection is carried out without manual intervention, and after the category is automatically obtained through an AI algorithm, the corresponding algorithm plug-in is selected for screening, so that the automation degree is improved.

Drawings

Fig. 1 is a total image of an actual case of embodiment 1 of the present invention;

FIGS. 2 and 3 are cases of semantic recognition of embodiment 1 of the present invention;

fig. 4 is a case of angular adjustment of an image of an object of embodiment 1 of the present invention;

FIG. 5 is a block flow diagram of the method of embodiment 1 of the present invention;

fig. 6 is a block diagram showing the system according to embodiment 1 of the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

Example 1

Referring to fig. 5, a simple defect detection method based on object identification includes the following steps:

taking fig. 1 as an example, there are 3 devices and 2 PCB boards in fig. 1, and the devices and the PCB boards are objects.

In the production process, a camera is generally arranged above the production line, and a total image is acquired through the camera. The specifications of the camera are not so high, and generally, the camera can hold low pixels, for example, about 1920 × 1080 or 1024 × 720 pixels.

in the prior art, a plurality of AI algorithms exist, and the AI algorithm is mainly used for detecting in real time because the AI algorithm of the invention does not relate to the identification of parts.

At present, the open-source SSD-MobileNet V2-lite and YoloV4-Tiny are both good in performance and suitable for embedded algorithms, for example, the real-time performance of YoloV4-Tiny on Jetson Nano reaches 36fps, the real-time performance of YoloV4-Tiny on MX8Plus reaches 50fps, and the requirements are already exceeded. Even can detect 4 production lines simultaneously, every probably is 9 ~ 12fps, can both satisfy the real-time requirement.

Html records a specific method for predicting a sliding window and a Bounding Box, and the convolution of an AI algorithm-sliding window method is adopted to realize the method, so that a more accurate frame is obtained.

The step meets the requirement of real-time detection, and has very important significance for improving the detection efficiency.

Referring to fig. 1, the boundary is a box in fig. 1, and this box is only a fuzzy range of the bounding object, and is mainly used for subsequent semantic analysis for contour analysis and class identification.

the semantic segmentation algorithm is to divide different contours through the sharpness and transition conditions of pixels, and then to confirm whether the partial contours belong to the same object according to training data. It is mainly distinguished according to color blocks + boundary lines.

The semantic segmentation algorithm can refer to:

https:// blog.csdn.net/qq _31820761/article/details/82628873, which discloses a method using a Bounding box (Bounding box) as auxiliary information; another specific processing method of the semantic segmentation algorithm takes the Scribbled line (scribed line) as auxiliary information.

The method comprises the following steps:

taking fig. 2 and 3 as examples, fig. 2 is a general image containing an object, and fig. 3 is a filled color block obtained through semantic segmentation.

And classifying each pixel point so as to obtain the accurate outline of an object. Because the category of each pixel point is to be determined, for example, whether a certain pixel point belongs to a car, a person, a background, or the like, there is no object, and only pixels. After the pixel point set is obtained, the algorithm calculation of the rotation angle of the object can be carried out, and finally the object is rotated to a horizontal angle.

Step 32: and drawing the outline of the object according to the filling color blocks. Referring to fig. 4, fig. 4 shows an example of semantic segmentation according to step 3 resulting in an outline of an object.

And 4, step 4: rotating the object in the image to be horizontal according to the angle of the image of the object in the outline of the object;

in order to accurately perform the step 5, problems in the shooting process, such as how to treat the object by half, are eliminated.

For this case, before step 5, an integrity recognition step of recognizing the integrity of the object is further included, which specifically includes:

calculating the size of the object according to the outline of the object, comparing the size of the object with a preset size, if the size of the object is matched with the preset size, the comparison is passed, and if the size of the object is not matched with the preset size, the comparison is not passed; the preset size is the actual size of the object.

Specifically, one problem with the AI algorithm output is that it does not tell whether an object can be completely captured. The algorithm can judge the object by matching certain characteristics, and if partial images are lacked, the object cannot be known.

Therefore, before being sent to the next-stage graphic algorithm, image integrity judgment is carried out. Because the application scene of the system is on the production line, the cameras shoot from the top vertically downwards, only a picture of a certain face of an object is obtained, and judgment is carried out according to the length-width ratio of the face.

For example, dev1 and dev2 devices in the above figures have different length-width ratios. Dev1 assumes the length to width ratio of the surface to be detected is 2.3:1, then from the Dev1 boundary coordinates obtained from AI algorithm, it is determined whether the length to width ratio of the bounding box is within the range of 2.3: 1.

The parts are one or more of interfaces, lamps and labels.

Taking fig. 1 as an example, each type of device, or PCB, has its own features and interfaces. Therefore, different algorithm designs are required according to the identified types. The algorithms are mainly designed aiming at the installation of an antenna and a screw, whether the LED lamp works normally, whether each interface has welding and whether label printing is correct, and the directions. The detection of the demand directions can be completed, and more than 90% of demands can be basically met.

Specifically, if the color of the interface is different from the color of the other main positions of the object in the interface identification process, the interface is identified according to the color characteristics.

Power source interface and plug wire interface are green shell, can obtain whether the profile of 2 interfaces exists through screening green.

The usb interface has silver and white characteristics, and the net port is characterized by a net port lamp and also by color filtering.

In a specific case, the portal lamp is green, and the power supply and the plug-in are also green, and in order to distinguish the two in the pattern recognition process, the size of the area of the green pattern is judged by the size of each returned area through the contour detection function, and the green pattern is distinguished.

In the process of identifying the label, an OCR algorithm (character recognition) is more specific, and the algorithm can be open source or charged, and the effect is good. It is only necessary to intercept the correct label position and send it to the OCR algorithm, and in order not to be affected by other printed characters, the label application position and orientation of each device are clearly required. In the actual operation process, only the currently shot picture needs to be judged to be 0 degree or 180 degrees. Since there are only 2 possibilities, image cropping can be done in turn, to the OCR algorithm.

In the lamp identification process, the lamp is in an illuminated state, and the lamp is identified according to the brightness characteristics of the lamp.

The specific process comprises the following steps:

whether the whole equipment or the PCB is adopted, whether the welding and the installation of the LED lamp are normal or not is judged by whether the LED lamp is electrified or not. When the LED is on, the brightness difference is very larger than that when the LED is off; the filtering can be done by color to get how many lamps are working properly.

Firstly, filtering colors on an original image;

after the color is filtered, only the content of the corresponding color is reserved, and other areas are blackened;

then carrying out gray level processing and binarization processing, and reserving a region with highly matched colors;

finally, through expansion and corrosion, the capillary of the image is removed, and a relatively complete graph can be obtained;

finally, obtaining a box of the LED lamp through a contour detection function; the box herein refers to the coordinate description of the object outline, and is a general term for coordinates x, y, dimensions width, height, etc.

After all the lamps are processed by the round, the number of the lamps can be obtained to judge whether the defects exist or not.

In summary, the type of each device and PCB determines what algorithm should be invoked, how many lamps qualify, what interfaces qualify, or crop that relative position for OCR determination. There is a need for a data structure design that is well compatible with and extends across different object types.

Taking the specific practical application scenario in fig. 1 as an example, the following codes comprehensively determine and output the result.

As shown in the above structure, the length-width ratio of the object, what operation needs to be performed, the color and format of the interface, the LED, etc., and the keywords and coordinates of OCR are defined.

The invention has the following advantages compared with the prior art according to the implementation task:

1. and the simple AI algorithm can be selected for the object identification at the front end, and the overall class characteristics of the production are obvious, so that high-computing-power equipment is not required, the deployment cost is reduced, and the instantaneity is improved.

2. The method supports the simultaneous detection of multiple paths, multiple paths and multiple types of objects, and determines what graphic algorithm calculation is needed subsequently for each type of object through the configuration files of the object types, so that the detection result is obtained.

3. After the object is identified, the contour coordinate of the object in the horizontal direction is obtained through semantic analysis and contour angle calculation. And then, whether the object is completely shot is judged through the length-width ratio configuration of the object.

4. The simple system adopts a graphic algorithm, and focuses on the identification functions of a plurality of commonly used scenes, such as interfaces, LED lamps, label characters and the like. And fine images such as surface scratches, components or wires are not analyzed, so that the requirements of most scenes are met.

5. By combining the AI algorithm and the computer graphic algorithm, the defects that the object type cannot be identified, the boundary contour of the object cannot be obtained, whether all the objects are shot cannot be judged by a single computer graphic algorithm and the like are overcome. The AI algorithm is overcome, if detection judgment such as an interface, an LED lamp and an OCR is needed, a large number of pictures and time are needed for training and correcting, and meanwhile, a large amount of computing power is consumed.

6. By a simple system focusing several scenes, the requirements on the size and definition of the picture are reduced, so that the deployment cost is reduced on the premise of meeting most scenes, and multi-path real-time detection is provided to improve the efficiency.

Through the design of the system, the system can be closer to most of actual requirements of the current factory production, the detection efficiency is improved, the detection cost is reduced, and automation and intellectualization are realized.

Referring to fig. 6, fig. 6 shows a system for implementing the above method, comprising: the device comprises a camera 1, an AI identification module 2 and an image identification module 3;

The number of the image recognition modules can be multiple, and each image recognition module is respectively responsible for recognizing different parts. For example, the device 1 in fig. 1 has a lamp and an interface, and then the image of the segmented object may be sent to two independent image recognition modules for recognition, and the recognition results are obtained and summarized.

It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential attributes thereof. The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference sign in a claim should not be construed as limiting the claim concerned.

Claims

1. A simple defect detection method based on object identification is characterized by comprising the following steps:

2. The simple defect detection method based on object identification as claimed in claim 1, wherein in the step 2, the specific method for identifying the type of the object is as follows: and pre-shooting pictures of various types of objects for training through an object recognition algorithm, and carrying out self-defined object recognition by using the trained model.

3. The simple defect detection method based on object identification according to claim 1, wherein the step 3 specifically comprises:

4. The simple defect detection method based on object identification as claimed in claim 1, wherein the preset angle in step 4 is an angle at which the object is in a horizontal state.

5. The simple defect detection method based on object identification as claimed in claim 1, wherein the step 4 further comprises an integrity identification step of identifying the integrity of the object, and the step 5 is performed after the integrity identification step is passed;

6. The simple defect detection method based on object identification as claimed in claim 1, wherein in step 5, the component is one or more of an interface, a lamp and a label.

7. The simple defect detection method based on object identification as claimed in claim 6, wherein in the identification process of the interface, if the color of the interface is different from that of other main positions of the object, the interface is identified according to the color characteristics.

8. The simple defect detection method based on object identification according to claim 7, wherein in the identification process of the interface, if the color of the interface is different from the color of other main positions of the object, the interface is identified according to the color feature and the size of the area where the color is located.

9. The simple defect detection method based on object identification as claimed in claim 7, wherein in the identification process of the lamp, the lamp is in an on state, and the lamp is identified according to the brightness characteristic of the lamp.

10. A simple defect detection system based on object identification is characterized by comprising a camera, an AI identification module and an image identification module;