CN115170497A

CN115170497A - PCBA online detection platform based on AI visual detection technology

Info

Publication number: CN115170497A
Application number: CN202210755121.1A
Authority: CN
Inventors: 乔景全; 盛颜开
Original assignee: Wujiang Sigma Electronic Technology Co ltd
Current assignee: Wujiang Sigma Electronic Technology Co ltd
Priority date: 2022-06-30
Filing date: 2022-06-30
Publication date: 2022-10-11

Abstract

The invention discloses a PCBA online detection platform based on an AI visual detection technology, which comprises a PCBA intelligent AI real-time detection system, a target detection algorithm model system, a PCBA detection data system and a detection product image storage system.

Description

PCBA online detection platform based on AI visual detection technology

Technical Field

The invention relates to the field of PCBA online detection, in particular to a PCBA online detection platform based on an AI visual detection technology.

Background

The PCBA online detection platform is a supporting system for PCBA element detection, in the industrial production process, visual detection is a key ring, defective parts need to be accurately and quickly identified, the step of analyzing and checking the products usually needs human eye identification for a long time, system training needs to be carried out on inspectors, qualified visual inspectors comprehensively use various identification knowledge and skills, and with the continuous development of artificial intelligence technology, the ability of the artificial intelligence to reliably execute tasks according to programs is continuously developed. In face recognition, artificial intelligence determines the position and size of a face and the positions of eyes, nose, ears, and other organs by analyzing an input image and outputting a bounding box. However, in the image detection of Printed Circuit Board Assembly (PCBA) in the electronic manufacturing industry, the environment is far more complicated than the human face recognition, and for example, electronic components of various packaging types, different circuit line shapes, different identification marks, silk-screen characters, welding points of components, solder masks of different colors and the like all affect the detection of the PCBA. Fortunately, the PCBA is produced according to the designed assembly document, so the environmental influence factors are easy to filter unnecessary factors through the document to lock a detection target, the intelligent detection equipment has high efficiency of replacing manual detection, is accurate and stable, main detection items comprise part missing detection, surface defect characteristic detection, pin packaging integrity detection, component damage detection, terminal pin detection and the like, non-contact measurement cannot cause any damage to an observer and an observed person, so that the reliability of the system is improved, the system works stably for a long time, a human is difficult to observe the same object for a long time, machine vision can perform measurement, analysis and identification tasks for a long time, a machine vision solution is utilized, a large amount of labor resources can be saved, and the requirement of people on the PCBA online detection platform is higher and higher along with the continuous development of science and technology.

The existing PCBA online detection platform has certain disadvantages when in use, firstly, the error rate of manual visual inspection is usually 20% -30%, some of the errors are caused by misoperation, and some errors cannot be avoided due to limited operation space, so that the use of people is not facilitated.

Disclosure of Invention

Technical problem to be solved

Aiming at the defects of the prior art, the invention provides a PCBA online detection platform based on an AI visual detection technology, which is based on an AI detection mode, is introduced into common and obvious element fault detection, greatly improves the detection precision, enhances the generalization capability of an algorithm, greatly improves the detection efficiency, constructs a quality tracking feedback system based on an intelligent detection result, has higher integration level, richer information and more accurate and rapid tracing quality problem, integrates the result of the intelligent algorithm and information such as PCBA bar codes, can rapidly trace the quality problem, and can effectively solve the problem in the background technology.

(II) technical scheme

In order to achieve the purpose, the invention adopts the technical scheme that: the utility model provides a PCBA on-line measuring platform based on AI visual detection technique, includes PCBA intelligence AI real-time detection system, target detection algorithm model system, PCBA detection data system and detection product image storage system, its characterized in that: based on AI detection mode, the detection target and element key area are quickly locked and labeled by means of PCBA related design or manufacturing data, then a data set of the detection element is obtained through modern neural network learning and training, and finally, the existing learning data set is adopted to detect a new PCBA product through learning migration.

By adopting the scheme, the detection efficiency and the detection precision are improved, the generalization capability of the algorithm is enhanced, the production period is shortened, the cost is reduced, the competitiveness of enterprises is enhanced, the PCBA data can be used for constructing a quality database by utilizing intelligent detection, a tracking feedback system is realized, a closed loop is formed by relevant stations in production and continuously improved, the quality monitoring and continuous improvement of a production line are more automatic and systematized, once more, during intelligent detection, PCBA bar codes are read and photographed, finally, the detection result and product photos are associated and stored in a database through bar code IDs (identity), and the quality problem can be quickly traced back, finally, intelligent detection equipment is integrated into a mechanical arm, the automation of a full detection process is realized, manual detection is reduced, meanwhile, the opportunity that personnel contact with products is avoided due to unmanned operation, the risk of ESD damage of chips on the products is reduced, a target detection algorithm model system is constructed and then outputs a detection platform, and the data can be automatically marked according to PCBM and CAD data and can be applied to the following tasks: efficiency Net for sorting the plug-in components; the twin network is used for metric learning of small samples of the plug-in component and the SMT component, wherein all models are compiled by using a PyTorch tool, NVIDIA Tesla P100 GPU is trained, and a data enhancement scheme comprises filtering denoising, contrast increasing, graying, sharpening and the like;

the PCBA intelligent AI real-time detection system comprises the following operation steps:

first, the target of detection is locked by reprocessing the design or manufacturing related data of the PCBA, such as CAD or ODB + + data of the PCBA, the components are classified according to component type and package according to the BOM, and regions for AI detection components and regions of Interest (ROI) of the components are determined and then labeled. Secondly, the AI learns and trains the labeled element product through a modern neural network, and saves the characteristic parameters of the element detection in the AI detection model. Through the use of the data to lock the PCBA target, the BOM is used to classify and label the target, and the AI learning and training are performed, so that the PCBA data set is converted into the PCBA element and welding spot data set classified according to the element type and the package. Finally, by utilizing the element and welding spot data set obtained through the related detection, when a new PCBA product is detected, only the learned element detection characteristic parameters are required to be migrated to the new PCBA detection, the convolution layer parameters are kept unchanged during training, the full connection layer is finely adjusted (such as the size or the direction of the element, and the like, and the fine adjustment can be realized), and the accurate detection effect can be achieved through a small amount of sample learning. The process is transfer learning, the new PCBA product can be detected by using the learned component characteristic parameters, and the difference of different PCBA detections from the AI detection learning point of view is the component type, size, direction and the number of the assembled components.

The method comprises the following steps of obtaining element positions and a boundary box according to CAD or ODB + + files, classifying elements according to the types and packages of BOM elements, and dividing key regions of interest (ROI) in the boundary box for the elements, wherein the method comprises the following steps:

A. the central area of the bounding box is used for identifying whether the element is missing or not and the contents of the inscription;

the inspection system needs to identify the following elements:

(a) Whether component position mounting is appropriate: no missing pieces or component offsets;

(b) Correct inscriptions of component models;

B. the bonding pads or bonding areas in the bounding box are used for detecting the quality of the bonding points;

C. detecting adjacent pads or plug-in holes at the periphery of the boundary frame to intercept bad solder bridges or short circuits and the like;

D. inspiring the concept of ROI, and adopting a spatial attention mechanism in a detection model to give more weight to the ROI;

the method comprises the following steps of performing element and welding spot detection model transfer learning, wherein the following contents are included:

A. once the predefined elements are labeled based on the positions and the packaging types, the AI can perform detection training on any PCBA sample labeled with the elements, and the parameters detected by the elements are stored in the AI detection model, based on the capability of learning the element types, the whole PCBA product does not need to be learned, only the learned element detection parameters are transferred to different PCBA products, the convolution layer parameters are kept unchanged during training, the full connection layer is finely adjusted, along with the training of the AI model and the detection of more PCBA products, bad data accumulated according to the element types can be continuously stored, and when the learning data of different element types can be transferred, any new PCBA product can be learned through a small amount of samples, so that an accurate detection effect is achieved;

B. the transfer learning can ensure that when each new PCBA is detected, excessive repeated training learning is not required, and different PCBAs can be detected by utilizing a large amount of labeled and learned learning data through the transfer learning function, so that a large amount of training learning time is saved, and the labeled data can be associated with element position numbers;

C. the effective transfer learning is that the work of detecting the similarity is carried out by using artificial intelligence, from the aspect of PCBA detection standard, an inspector has certain professional skills or adopts the detection standard IPC-A-610 of the PCBA industry to carry out detection, AI only needs to carry out similar detection tasks, namely, the detection and judgment are carried out on the detected object by referring to the industry standard, and from the aspect of AI detection learning, the differences of different PCBA detections are the types, sizes, directions and the quantity of the assembled elements.

As a preferred technical solution of the present application, in an offset Net for sorting of plug-in components, the offset Net is 3*3 convolution layer +7 Mobile Net volume blocks, which include ordinary 1*1 convolution, deep separable convolution, different channel weights given by SE layer +1*1 convolution + pooling + full connection, and finally output categories, which can easily mark positions and categories of electronic components on the PCBA, and automatically generate training and test data sets, and the marking effect is better than human because the obtained image data has less redundant information, applying these data to the offset Net can reduce 1-2 Mobile volume blocks, and still can achieve the effect of classifying various electronic components, the trained offset Net can be directly used for detection of the PCBA, and the electronic component pictures on the PCBA are sequentially input into the offset Net according to the BOM, and the offset Net position of the PCBA should not appear when the offset category is found.

As a preferred technical scheme of the application, the method is used for a twin network for plug-in components and SMT (surface Mount technology) component small sample metric learning, and has the following basic ideas: inputting two pictures, respectively extracting features by using the same convolutional neural network, and calculating the distance between two feature vectors to determine whether the contents of the two pictures belong to the same category; the network structure is as follows: firstly, a ResBlock structure, namely batch standardization, reLU activation, convolutional layer, maximum pooling downsampling and residual error connected convolutional neural network is used for extracting the characteristics of two input pictures, then the distance between two characteristic vectors is calculated, then the distance vectors are input into another convolutional neural network, namely convolutional layer, reLU activation and maximum pooling downsampling to extract the characteristics of a higher dimension between the two input pictures, finally the two input pictures enter a full connection layer to obtain a scalar, then a Sigmoid activation function is used for outputting a final result, if the two input pictures are of the same type, the result is 1, and if the two input pictures are of different types, the result is 0.

As a preferred technical scheme of this application, PCBA intelligence AI real-time detection system adopts AI technique, AI technique utilizes component type and encapsulation to establish detection model and detect the component, and AI technique detects every component, utilizes prior art data to obtain the relevant information of detecting the component fast, and utilizes these known data to predefine the position and the classification of equipment component, combines PCBA assembly document to mark the detecting element, turns into the data set of PCBA component with the PCBA data set, discerns and contrasts, AI technique focus detects the bad defect that above test procedure can't be intercepted.

As a preferred technical scheme of the application, the PCBA intelligent AI real-time detection system comprises the following operation steps:

s1: the method comprises the steps that the PCBA is detected by using an AI technology, the PCBA detection is mainly aimed at the bad phenomenon that the circuit test cannot be covered, the detection is an important link in the whole test system, and the AI detection value is that the bad condition that the existing detection mode cannot be completely covered can be detected;

s2: classifying the elements in the boundary frame according to the known element types, acquiring the boundary frame by using element position information, defining the element types by using the element types, greatly reducing the learning process of AI, not only predefining the size of the boundary frame of the element in the PCBA image by combining the element shape in CAD data and a bonding pad thereof, but also determining the types of the boundary frame through the types and the encapsulation of the elements;

s3: predefining a component classification according to the type and packaging of the component;

s4: dividing a key attention area in the boundary frame, predefining the boundary frame and the key attention area according to the element type and the encapsulation thereof, and then quickly acquiring element characteristics to remove unnecessary information in the picture;

s5: the element and welding spot detection model is subjected to transfer learning, different PCBAs can be detected by utilizing a large amount of marked and learned learning data through a transfer learning function, a large amount of training learning time is saved, the marked data can be related to element position numbers, and the work of detecting similarity is carried out by utilizing artificial intelligence;

s6: the simulation inspector carries out online detection, and online AI detects not only has the bad interception of reliability, can avoid personnel to contact the product simultaneously through using integrated arm, and then reduces PCBA and receives ESD or the risk that manual operation probably causes the harm when improper.

(III) advantageous effects

Compared with the prior art, the invention provides a PCBA online detection platform based on an AI visual detection technology, which has the following beneficial effects: the PCBA online detection platform based on the AI visual detection technology is introduced into common and obvious element fault detection based on an AI detection mode, so that the detection precision is greatly improved, the generalization capability of an algorithm is enhanced, the detection efficiency is greatly improved, a quality tracking feedback system is constructed based on an intelligent detection result, the integration level is higher, the information is richer, the quality problem can be traced more accurately and rapidly, the system integrates the result of the intelligent algorithm and information such as PCBA bar codes, and the like, the quality problem can be traced rapidly, batch detection targets on a production line are continuously and accurately measured in the AI visual detection mode, the error rate and the cost are greatly reduced, and the efficiency is improved;

1. AI-based detection mode for PCBA intelligent detection

The invention relates to a PCBA online detection platform based on an AI visual detection technology, which is based on an AI detection mode, realizes the quick locking and marking of a detection target and an element key area by means of related design or manufacturing data of the PCBA, then obtains a data set of a detection element through modern neural network learning and training, and finally realizes the detection of a new PCBA product by adopting the existing learning data set through learning migration. And the data of intelligent detection is utilized to construct a quality database, a tracking feedback system is realized, a closed loop is formed with relevant stations in production for continuous improvement, quality monitoring and continuous improvement of a production line are more automatic and systematic, again, during intelligent detection, a PCBA bar code is read and photographed, finally, a detection result and a product photo are associated and stored in a database through a bar code ID, and the quality problem can be quickly traced back, finally, intelligent detection equipment is integrated into a mechanical arm, full detection process automation is realized, manual detection is reduced, meanwhile, due to unmanned operation, the opportunity that personnel contact with a product is avoided, the risk of ESD damage of a chip on the product is reduced, after the target detection algorithm model system is constructed, a PCBA detection platform is output, and according to a material list (Bill of materials, BOM) and CAD data can be automatically labeled and applied to several tasks: efficiency Net for sorting the plug-in components; twin networks for small sample metric learning for socket and SMT.

2. Product database construction and higher quality tracking feedback management system

The basic contents of the product database comprise storage, searching of a detected product, component image and detection information, intelligent identification of a product bar code, material number information management and the like, one key point of the product database is that the product bar code is processed and is associated with a product real object, whether a reserved area on the circuit board lacks the bar code is also a problem that important detection is needed in the project, whether the bar code is lacked or not is detected and accurately identified in the project under the complex production line environment, the bar code is associated with the product material number information, the detection precision information and the like to form a high-quality retrospective product detection information database, the intelligent detection system has high detection speed, the labor cost is saved, and fatigue misjudgment caused by human eyes is thoroughly solved, the detection can be carried out continuously for 24 hours, the fine defects can be accurately identified, the detection efficiency is improved, the problems of fine defects and easy missed detection and misjudgment of visual detection are solved, different sorting conditions can be set for different types of defects according to a work order so as to meet the quality requirements of different batches of products, detection data and picture information are stored in a database server, defect detection results are summarized and analyzed, product quality tracing is carried out in real time, an intelligent detection technology is introduced to detect the appearance defects of the products, the labor cost is reduced, the detection precision and efficiency are greatly improved, better reputation and greater income are brought to enterprises, the detection effect of artificial intelligent detection equipment on the appearance defects of the products is remarkable, compared with manual work, the detection efficiency of the leakage test of PCBA components reaches over 90 percent, each PCBA bar code is accurately read and relevant information is stored, PCBA detects the steady operation of data system, whole PCBA on-line measuring platform simple structure, convenient operation, the effect of use is better for traditional mode.

Drawings

Fig. 1 is a schematic overall structure diagram of a PCBA online detection platform based on an AI visual detection technology according to the present invention.

FIG. 2 is a schematic structural diagram of a target detection model building module in the PCBA online detection platform based on the AI visual detection technology.

FIG. 3 is a schematic diagram of a CAD of a PCBA product in an AI-based online detection platform according to the present invention, covering information such as component positions.

Fig. 4 is an ODB + + diagram of a PCBA product in the PCBA online detection platform based on the AI visual detection technology according to the present invention, covering information such as component position and component contour.

Fig. 5 is a schematic structural diagram of one of the important attention areas in the bounding box divided in the PCBA online detection platform based on the AI visual detection technology according to the present invention.

Fig. 6 is a schematic structural diagram of a second area of interest within a bounding box divided in the PCBA online detection platform based on the AI visual detection technology according to the present invention.

Detailed Description

The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings and the detailed description, but those skilled in the art will understand that the following described embodiments are some, not all, of the embodiments of the present invention, and are only used for illustrating the present invention, and should not be construed as limiting the scope of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention. The examples, in which specific conditions are not specified, were conducted under conventional conditions or conditions recommended by the manufacturer. The reagents or instruments used are not indicated by the manufacturer, and are all conventional products available commercially.

In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicate orientations or positional relationships based on the orientations or positional relationships shown in the drawings, and are only for convenience of description and simplicity of description, but do not indicate or imply that the device or element being referred to must have a particular orientation, be constructed and operated in a particular orientation, and thus, should not be construed as limiting the present invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance.

In the description of the present invention, it should be noted that, unless otherwise explicitly specified or limited, the terms "mounted," "connected," and "connected" are to be construed broadly, e.g., as meaning either a fixed connection, a removable connection, or an integral connection; can be mechanically or electrically connected; they may be connected directly or indirectly through intervening media, or they may be interconnected between two elements. The specific meanings of the above terms in the present invention can be understood in specific cases to those skilled in the art.

The first embodiment is as follows:

as shown in fig. 1-6, a PCBA on-line detection platform based on AI visual detection technology includes a PCBA intelligent AI real-time detection system, a target detection algorithm model system, a PCBA detection data system and a detection product image storage system, and is characterized in that: based on AI detection mode, the detection target and element key area are quickly locked and labeled by means of PCBA related design or manufacturing data, then a data set of the detection element is obtained through modern neural network learning and training, and finally, the existing learning data set is adopted to detect a new PCBA product through learning migration.

By adopting the scheme, the detection efficiency and the detection precision are improved, the generalization capability of the algorithm is enhanced, the production period is shortened, the cost is reduced, the competitiveness of enterprises is enhanced, the intelligent detection data is utilized, a quality database can be constructed, a tracking feedback system is realized, closed loop is formed by relevant stations in production and continuously improved, the quality monitoring and continuous improvement of a production line are more automatic and systematized, once more, during intelligent detection, PCBA bar codes are read and photographed, finally, the detection result and product photos are associated and stored in a database through bar code IDs (identity), and the quality problem can be quickly traced back, finally, the intelligent detection equipment is integrated into a mechanical arm, the automation of a full detection process is realized, manual detection is reduced, meanwhile, the opportunity that personnel contact with products is avoided due to unmanned operation, the risk of ESD damage of chips on the products is reduced, a PCBA detection platform is output after a target detection algorithm model system is constructed, data can be automatically marked according to BOM and CAD data, and the method is applied to the following tasks: an efficiency Net for sorting the jack pieces; a twin network for metric learning of small samples of the socket piece and the SMT piece; all models are written by a PyTorch tool, NVIDIA Tesla P100 GPU training is carried out, and a data enhancement scheme comprises filtering and denoising, contrast increasing, graying, sharpening and the like;

first, the target of detection is locked by reprocessing the design or manufacturing related data of the PCBA, such as CAD or ODB + + data of the PCBA, the components are classified according to component type and package according to the BOM, and regions for AI detection components and regions of Interest (ROI) of the components are determined and then labeled. Secondly, the AI learns and trains the labeled element products through a modern neural network, and stores the characteristic parameters of the element detection in an AI detection model. Through the adoption of the data, the target of the PCBA is locked, the target is classified and marked by the BOM, and the data set of the PCBA is converted into the data set of the PCBA elements and welding spots classified according to the element types and the packages through AI learning and training. Finally, by utilizing the element and welding spot data set obtained through the related detection, when a new PCBA product is detected, only the learned element detection characteristic parameters are required to be migrated to the new PCBA detection, the convolution layer parameters are kept unchanged during training, the full connection layer is finely adjusted (such as the size or the direction of the element, and the like, and the fine adjustment can be realized), and the accurate detection effect can be achieved through a small amount of sample learning. The process is transfer learning, the new PCBA product can be detected by using the learned element characteristic parameters, and the difference of different PCBA detections from the AI detection learning point of view is the type, size, direction and the number of the assembled elements.

The method comprises the following steps of obtaining element positions and a boundary box according to CAD or ODB + + files, classifying elements through the types and packages of BOM elements, and dividing key regions of interest (ROI) in the boundary box for the elements, wherein the method comprises the following steps:

the inspection system needs to identify the following elements:

(b) Correct inscriptions of component models;

B. the welding pads or welding areas in the boundary frame are used for detecting the quality of welding points;

D. inspiring the idea of ROI, and adopting a spatial attention mechanism in a detection model to give more weight to the ROI;

Furthermore, the PCBA intelligent AI real-time detection system adopts AI technology, the AI technology utilizes element types and packaging to create detection models and detect the elements, the AI technology detects each element, relevant information of the detected elements is quickly obtained by utilizing existing technical data, the positions and the classifications of the assembled elements are predefined by utilizing known data, the detected elements are labeled by combining with PCBA assembly documents, the PCBA data set is converted into the data set of the PCBA elements to be identified and compared, and the AI technology mainly detects the bad defects which cannot be intercepted by the testing procedures.

Further, the PCBA intelligent AI real-time detection system comprises the following operation steps:

1. PCBA lacks bypass capacitance, which is typically found upon visual inspection with a stencil;

2. a lack of components, which can be found at ICT or visual inspection;

3. bad damaged parts which cannot be intercepted are tested and can be found through visual detection;

4. polarity error capacitive elements which cannot be intercepted in an ICT/FCT test can be found through visual detection;

5. if a passive component with the wrong packaging size is assembled, the passive component can be found through visual detection;

6. poor component assembly offset which cannot be intercepted in a testing link can be found through visual detection;

7. bad phenomena such as tin balls, tin slag, tin bridges and the like in the welding process can be found through visual detection;

8. aiming at the risk that the ESD is bad due to artificial contact in the detection process, the integrated mechanical arm is used for assisting the detection process, and the risk is reduced.

S2: classifying the components in the boundary frame according to the known component types, acquiring the boundary frame by utilizing the component position information, defining the component types by adopting the component types, greatly reducing the learning process of AI (artificial intelligence) and not only predefining the size of the boundary frame of the components in the PCBA image by combining the component shapes and the pads thereof in the CAD data, but also determining the types of the boundary frames through the types and the encapsulation of the components;

1. the capacitors are classified by package as follows

a.through hole/jack class

i.electric/electrolytic capacitor

Ceramic/ceramic capacitor

SMT/Patch class

Aluminum/aluminum capacitor

Tantalum/tantalum capacitor

iii.Chip type per package size such as 0201,0402 …

The chip capacitor has 0201,0402 … according to the packaging size

2. The resistors are classified as follows according to the package

a.through hole/jack class

Class i.through hole/jack

SIP/Single inline

SMT/Patch class

Chip type per package size Such as 0201,0402 … chip resistor, if 0201,0402 … according to package size

3. Inductors are classified as follows according to package

a.through hole/jack class

SMT/Patch class

Chip type package size subcch as 0603,0402 … chip inductor, if 0603,0402 … according to package size

Power/Power inductor

4. The diodes are classified by package as follows

a.through hole/jack class

SMT/Patch class

i.SOT23/Small form factor transistor

SOD/small outline diode

5. The transistors are classified by package as follows

a.through hole/jack class

SMT/Patch class

SOT23/small outline transistor

SOT4/Small form factor transistor

6. The connectors are classified by package as follows

a.through hole/jack class

SMT/Patch class

7. The chips are classified according to the packages as follows

a.through hole/jack class

DIP/dual in-line package

SMT/Patch class

i.SOIC/small-outline integrated circuit

SOP/small outline package

QFP/quad flat package

BGA/ball grid array packages.

Example two:

on the basis of the first embodiment, as shown in fig. 1 to 6, a PCBA online detection platform based on an AI visual detection technology includes a PCBA intelligent AI real-time detection system, a target detection algorithm model system, a PCBA detection data system, and a detection product image storage system, and is characterized in that: based on AI detection mode, the method realizes fast locking and labeling of detection targets and element key areas by means of PCBA related design or manufacturing data, then obtains data sets of detection elements through modern neural network learning and training, and finally realizes detection of new PCBA products by adopting the existing learning data sets through learning migration.

By adopting the scheme, the detection efficiency and the detection precision are improved, the generalization capability of the algorithm is enhanced, the production period is shortened, the cost is reduced, the competitiveness of an enterprise is enhanced, the intelligent detection data is utilized to construct a quality database, a tracking feedback system is realized, a closed loop is formed with relevant stations in production to continuously improve, the quality monitoring and continuous improvement of a production line are more automatic and systematized, once more, when the intelligent detection is carried out, a PCBA bar code is read and taken, finally, a detection result and a product photo are associated and stored in a database through a bar code ID, the quality problem can be quickly traced, finally, the intelligent detection equipment is integrated into a mechanical arm, the automation of a full detection process is realized, the manual detection is reduced, meanwhile, the opportunity that personnel contact with the product is avoided due to unmanned operation, the risk of ESD damage to chips on the product is reduced, a PCBA detection platform is output after a target detection algorithm model system is constructed, and the target detection algorithm model system can automatically label data according to BOM and CAD data and is applied to the following tasks: an efficiency Net for sorting the jack pieces; the twin network is used for plug-in hole piece and SMT piece small sample metric learning; all models are written by a PyTorch tool, NVIDIA Tesla P100 GPU training is carried out, and a data enhancement scheme comprises filtering denoising, contrast increasing, graying, sharpening and the like;

first, the target of detection is locked by reprocessing the PCBA related design or manufacturing related data, such as the CAD or ODB + + data of the PCBA, classifying the components by component type and package according to the BOM, and determining the area of AI detection component and the region of Interest (ROI) of the component, which are then labeled. Secondly, the AI learns and trains the labeled element products through a modern neural network, and stores the characteristic parameters of the element detection in an AI detection model. Through the use of the data to lock the PCBA target, the BOM is used to classify and label the target, and the AI learning and training are performed, so that the PCBA data set is converted into the PCBA element and welding spot data set classified according to the element type and the package. Finally, by utilizing the element and welding spot data set obtained through the related detection, when a new PCBA product is detected, only the learned element detection characteristic parameters are required to be migrated to the new PCBA detection, the convolution layer parameters are kept unchanged during training, the full connection layer is finely adjusted (such as the size or the direction of the element, and the like, and the fine adjustment can be realized), and the accurate detection effect can be achieved through a small amount of sample learning. The process is transfer learning, the new PCBA product can be detected by using the learned element characteristic parameters, and the difference of different PCBA detections from the AI detection learning point of view is the type, size, direction and the number of the assembled elements.

the inspection system needs to identify the following elements:

(b) Correct inscriptions of component models;

B. the transfer learning can ensure that when each new PCBA is detected, excessive repeated training learning is not needed, and different PCBAs can be detected by utilizing a large amount of labeled and learned learning data through the transfer learning function, so that a large amount of training learning time is saved, and the labeled data can be associated with element position numbers;

Furthermore, in the offset Net for sorting the plug-in components, the offset Net is 3*3 convolution layer +7 Mobile Net convolution blocks, wherein the offset Net comprises ordinary 1*1 convolution, depth separable convolution, SE layer endowing different channels with different weights +1*1 convolution + pooling + full connection, and finally outputting categories, and by virtue of BOM and CAD, the positions and categories of electronic components on the PCBA can be easily marked, training and test data sets are automatically generated, and the marking effect is better than that of manual marking, because the obtained image data has less redundant information, the data are applied to the training offset Net, 1-2 Mobile Net convolution blocks can be reduced, the effect of sorting various electronic components can be still achieved, the trained offset Net can be directly used for detecting the PCBA, the electronic component pictures on the PCBA are sequentially input into the offset Net according to the BOM, and the positions of the electronic components on the offset Net are compared with the categories of the components, and the categories of the components are wrong.

Further, in the twin network for plug-in and SMT small sample metric learning, the basic idea is as follows: inputting two pictures, respectively extracting features by using the same convolutional neural network, and calculating the distance between two feature vectors to determine whether the contents of the two pictures belong to the same category; the network structure is as follows: firstly, a ResBlock structure, namely batch standardization, reLU activation, convolutional layer, maximum pooling downsampling and residual error connected convolutional neural network is used for extracting the characteristics of two input pictures, then the distance between two characteristic vectors is calculated, then the distance vectors are input into another convolutional neural network, namely convolutional layer, reLU activation and maximum pooling downsampling to extract the characteristics of a higher dimension between the two input pictures, finally the two input pictures enter a full connection layer to obtain a scalar, then a Sigmoid activation function is used for outputting a final result, if the two input pictures are of the same type, the result is 1, and if the two input pictures are of different types, the result is 0.

The working principle is as follows: the invention relates to a PCBA online detection platform based on an AI visual detection technology, which is based on an AI detection mode, realizes the quick locking and marking of a detection target and an element key area by means of related design or manufacturing data of the PCBA, then obtains a data set of a detection element through modern neural network learning and training, and finally realizes the detection of a new PCBA product by adopting the existing learning data set through learning migration. And the data of intelligent detection is utilized to construct a quality database, a tracking feedback system is realized, a closed loop is formed with relevant stations in production for continuous improvement, quality monitoring and continuous improvement of a production line are more automatic and systematic, again, during intelligent detection, a PCBA bar code is read and photographed, finally, a detection result and a product photo are associated and stored in a database through a bar code ID, and the quality problem can be quickly traced back, finally, intelligent detection equipment is integrated into a mechanical arm, full detection process automation is realized, manual detection is reduced, meanwhile, due to unmanned operation, the opportunity that personnel contact with a product is avoided, the risk of ESD damage of a chip on the product is reduced, after the target detection algorithm model system is constructed, a PCBA detection platform is output, and according to a material list (Bill of materials, BOM) and CAD data can be automatically labeled and applied to several tasks: efficiency Net for sorting the plug-in components; the twin network is used for plug-in hole piece and SMT piece small sample metric learning; whether or not the expected result can be obtained subsequently. In image-based data analysis, the quality of image quality directly affects the accuracy of the design and effect of the recognition algorithm, and therefore, data preprocessing is performed before image analysis. The main purposes of image preprocessing are to eliminate irrelevant information in an image, recover useful real information, enhance the detectability of relevant information and simplify data to the maximum extent, so that the reliability of feature extraction, image segmentation, matching and identification is improved, and image filtering is to suppress the noise of a target image under the condition of keeping detailed features of the image as much as possible. In an image, the high frequency part represents the edge information, i.e. sharpness, of this image; the low frequency part is just the opposite and represents the grey scale change information, i.e. the content, of this image. Filtering is to filter out a part of the noise wave through a filter operator to highlight the detail information. When the color image is processed, RGB three channels need to be processed in sequence, and the time consumption is long. In order to increase the processing speed of the whole application system, the data amount required to be processed needs to be reduced, and the gray-scaled image can effectively reduce the data operation amount. The obtained grayscale image is subjected to binarization processing. The purpose is to perform background classification on the PCBA image and prepare the PCBA image for subsequent electronic component identification. The image sharpening and the image smoothing are the opposite operations, and the sharpening is to reduce the blurring in the image by enhancing high-frequency components, enhance the detail edges and the contours of the image, enhance the gray contrast and facilitate the identification and the processing of the target in the later period. The sharpening process also increases the noise of the image while enhancing the edges of the image,

the method comprises the steps of developing and using a deep learning-based mode based on project requirements to carry out real-time quality intelligent detection on industrial products on an automatic production line, wherein a data set used by an experimental project is a PCBA image on the production line, acquiring a large number of existing component images in the actual PCBA production process, carrying out intelligent detection on the images, classifying and establishing the data set, establishing a computer vision target detection model by taking the data set as input and the PCBA packaging intelligent detection type as output, realizing detection and identification of the intelligent detection type and carrying out corresponding marking on the PCBA image, storing a pre-training model at a PC end of a factory production line, connecting a model prediction interface with an image detection button in an interface, clicking an image input button by a user, reading the latest PCBA image and displaying the latest PCBA image in the center of the interface, clicking the image detection button to carry out intelligent detection, cutting out bar codes, and identifying character strings corresponding to the bar codes. The method can call a network, a database and other means to quickly process the character strings, the characteristics of the bar codes are considered in bar code identification, and the items aim at that the positions of the bar codes in the picture are relatively vertical and have no various inclinations. Firstly, morphological gradient operation is carried out, the characteristics in the X direction are reserved, and the interference in the Y direction is removed. And (4) image blurring is performed, so that the image connection in the later period is facilitated. And (4) solving a threshold value of the image, accelerating algorithm processing, and reasonably removing black holes by using fuzzification effect morphology. And (5) performing closed operation expansion corrosion to further determine the position of the bar code. And searching the outline of the bar code, calculating the maximum area of the outline, and fitting an outline rectangle to obtain a final result. And cutting the image by taking the fitted rectangle as the identification boundary of the bar code, transmitting the cut bar code into a server or a network, and reading the corresponding character string. And storing the character string in a PC (personal computer) end or a server, displaying the character string in an integrated interface, and further associating the character string information with the relevant information of the detection system. The information contained in the PCBA is then stored in the PC side or a remote server. The method mainly comprises the information of the material number of the PCBA component; PCBA bar code stores information; whether the information is returned to the factory, repaired and the like. The database can help the project to trace the quality problem quickly, provide more abundant and accurate related information to strengthen the PCBA maintenance mechanism, integrate the detection functions into a software system, and the PCBA image detection flow system to be designed comprises main functional modules including an image input module, an image detection module, a detection result display module and a manual confirmation module. The intelligent detection conditions of the system main interface layout are respectively ten PCBA boards from left to right, and are displayed on the left side in a tree menu mode; PCBA is displayed in the center of the interface; the intelligent detection position information, the intelligent detection type and the material information are displayed on the right side. The manual confirmation button position and the detail display window are integrated at the lower right. The image input button, the image detection button and the like are used as the top of a menu bar integration and interface, and PCBA intelligent detection, PCBA result display and PCBA bar code information reading are further integrated in a unified integration system. The main functions of the system are as follows: user account login, PCBA intelligent detection and PCBA detection information statistics. And setting a user login function, and endowing different accounts with different user permissions. After the detection process is completed, the detection result is stored and counted, in the implementation process of the project, the system platform construction is further adjusted and perfected according to the implementation of specific functions, the debugging result and the field operation condition, the main point of the AI on the PCBA detection is to classify elements in a boundary frame according to the known element types aiming at the bad phenomenon which cannot be covered in the circuit test, the element classification is predefined according to the element types and the encapsulation, after the predefined elements are labeled based on the positions and the encapsulation types, the AI can carry out detection training on any PCBA sample labeled with the elements, and the parameters detected by the elements are stored in an AI detection model. Based on the ability to learn the component type, the entire PCBA product need not be learned, and only the learned component detection parameters need to be transferred to different PCBA products. We only need to keep the convolutional layer parameters unchanged during training and fine-tune the fully-connected layer. As the AI model trains and tests more PCBA products, bad data accumulated by component type can be stored on a continuous basis. When learning data of different element types can be transferred, any new PCBA product can be learned through a small amount of samples, and accurate detection effect can be achieved. The transfer learning can enable each new PCBA to be detected without performing too many repeated training studies. Through the transfer learning function, a large amount of labeled and learned learning data can be used for detecting different PCBAs, so that a large amount of training learning time is saved. The noted data may be associated with a component location number. The effective transfer learning is the work of detecting the similarity by using artificial intelligence, and from the PCBA detection standard, an inspector has certain professional skills or detects by adopting the detection standard IPC-A-610 of the PCBA industry, and AI only needs to carry out similar detection tasks, namely, the detected object is detected and judged by referring to the industry standard. From AI detection study's angle, the difference that different PCBA detected is exactly PCBA size and the component number of its equipment, and online AI detects not only has the bad interception of reliability, can avoid personnel to contact the product simultaneously through using integrated arm, and then reduces PCBA and receives ESD or the risk that manual operation probably causes the harm at that time, in SigmaTron company's manufacturing process, artificial intelligence is used for visual detection, and this visual detection has integrateed machine vision and AI deep learning's function. In a manufacturing environment, the AI can utilize prior art data to quickly obtain information about the sensing elements, such as the location and classification of the elements. Once the components of the PCBA board are identified, the characteristic parameters of the components can be learning trained by component type and migrated to a new PCBA board, which can reduce the number of learning samples and speed up the learning process. Therefore, by the above method, the existing AI technology can be improved and applied to the PCBA online detection, and the function of the AI detection model is not used for detecting the whole PCBA, but used for detecting each element. AI is to use the component type and package to create a detection model and detect the component, such as missing component, displacement, tombstone, solder joint or short circuit.

According to BOM and CAD data, data can be automatically marked and applied to the following tasks: ( All models were written using a PyTorch tool, NVIDIA Tesla P100 GPU trained. The data enhancement scheme comprises filtering denoising, increasing contrast, graying, sharpening and the like )

1. Efficiency Net for plug-in sorting:

3*3 convolutional layer +7 Mobile Net convolutional blocks (including 1*1 convolution, depth separable convolution, SE layer with different channel weights) +1*1 convolution + pooling + full connection, and finally outputting the category.

Thanks to BOM and CAD, we can easily mark the position and type of electronic components on the PCBA, automatically generate training and testing data sets, and the marking effect is better than that of human because the obtained image data has less redundant information. The data are applied to training Efficient Net, 1-2 Mobile Net volume blocks can be reduced, and the effect of classifying various electronic elements can be achieved.

The trained Efficient Net can be directly used for PCBA detection, electronic element pictures on the PCBA are sequentially input into the Efficient Net according to the BOM, then the classes of the elements at the position on the BOM are compared, and if the classes are not the classes of the elements which should appear at the position, an error is reported.

2. Twin network for plug-in and SMT piece small sample metric learning:

the basic idea is as follows: inputting two pictures, respectively extracting features by using the same convolutional neural network (shared weight), and calculating the distance between two feature vectors to determine whether the contents of the two pictures belong to the same category.

The network structure is as follows: firstly, extracting the characteristics of two input pictures by using a convolutional neural network with a ResBlock structure (batch standardization + ReLU activation + convolutional layer + maximum pooling downsampling + residual error connection), then calculating the distance between two characteristic vectors, then inputting the distance vector into another convolutional neural network (convolutional layer + ReLU activation + maximum pooling downsampling) to extract the characteristics of a higher dimension between the two input pictures, finally entering a full connection layer to obtain a scalar, and then outputting a final result by using a Sigmoid activation function, wherein if the input pictures are similar, the final result is 1, and if the input pictures are heterogeneous, the output picture is 0.

For our, only need input two electronic component pictures basis while training, input two components belong to the same classification.

In our production environment, the twin network can be used for identification of missing or missing pieces of PCBA. For the detection of a PCBA, firstly, there is a 'gold standard' PCBA which is manually compared without errors, the elements on the PCBA are taken as one of the inputs of the twin network according to the BOM and the CAD data, then the elements at the same position of the PCBA to be detected are also input into the network, and the network judges whether the elements belong to the same category.

Such a network structure can be used for small sample learning, and for a well-trained twin network, even if the input is the electronic component class which is not seen during training, the network structure can be used for comparing with the electronic component in the same position on the gold standard PCBA to output whether the two electronic components belong to the same class.

It is noted that, herein, relational terms such as first and second (first, second, and the like) and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising a … …" does not exclude the presence of another identical element in a process, method, article, or apparatus that comprises the element.

The foregoing shows and describes the general principles and broad features of the present invention and advantages thereof. It will be understood by those skilled in the art that the present invention is not limited to the embodiments described above, which are described in the specification and illustrated only to illustrate the principle of the present invention, but that various changes and modifications may be made therein without departing from the spirit and scope of the present invention, which fall within the scope of the invention as claimed.

Claims

1. The utility model provides a PCBA on-line measuring platform based on AI visual detection technique, includes PCBA intelligence AI real-time detection system, target detection algorithm model system, PCBA detection data system and detection product image storage system, its characterized in that: based on AI detection mode, rapidly locking and marking detection targets and element key areas by means of PCBA related design or manufacturing data, then obtaining a data set of detection elements through modern neural network learning and training, and finally detecting new PCBA products by adopting the existing learning data set through learning migration;

by adopting the scheme, the detection efficiency and the detection precision are improved, the generalization capability of the algorithm is enhanced, the production period is shortened, the cost is reduced, the competitiveness of enterprises is enhanced, the PCBA data can be used for constructing a quality database by utilizing intelligent detection, a tracking feedback system is realized, a closed loop is formed by relevant stations in production and continuously improved, the quality monitoring and continuous improvement of a production line are more automatic and systematized, once more, during intelligent detection, PCBA bar codes are read and photographed, finally, the detection result and product photos are associated and stored in a database through bar code IDs (identity), and the quality problem can be quickly traced back, finally, intelligent detection equipment is integrated into a mechanical arm, the automation of a full detection process is realized, manual detection is reduced, meanwhile, the opportunity that personnel contact with products is avoided due to unmanned operation, the risk of ESD damage of chips on the products is reduced, a target detection algorithm model system is constructed and then outputs a detection platform, and the data can be automatically marked according to PCBM and CAD data and can be applied to the following tasks: efficiency Net for sorting the plug-in components; the method is used in a twin network for plug-in and SMT small sample metric learning, wherein all models are compiled by using a PyTorch tool, NVIDIA Tesla P100 GPU training is carried out, and a data enhancement scheme comprises filtering denoising, contrast increasing, graying, sharpening and the like;

firstly, the design or manufacture related data of the PCBA is utilized, for example, the CAD or ODB + + data of the PCBA is utilized to carry out reprocessing to lock the detection target, the components are classified according to the component types and the packaging according to the BOM (Bill of Material), the area of the AI detection component and the region of Interest (ROI) of the component are determined, and then the marking is carried out;

secondly, the AI performs modern neural network learning and training on the labeled element product, and saves the characteristic parameters of element detection in an AI detection model;

through the data, the PCBA is subjected to target locking, the BOM is used for classifying and marking targets, and AI learning and training are performed, so that the PCBA data set is converted into the PCBA element and welding spot data set classified according to element types and packages;

finally, by utilizing the element and welding spot data set obtained through the related detection, when a new PCBA product is detected, only the learned element detection characteristic parameters are transferred to the new PCBA detection, the convolution layer parameters are kept unchanged during training, the full connection layer is finely adjusted (such as the size or the direction of the element and the like, and can be achieved through fine adjustment), and the accurate detection effect can be achieved through a small amount of sample learning;

the process is transfer learning, the new PCBA product can be detected by using the learned element characteristic parameters, and the difference of different PCBA detections from the AI detection learning perspective is the element type, size, direction and the number of the assembled elements;

the inspection system needs to identify the following elements:

(b) Correct inscriptions of component models;

2. The AI visual inspection technology-based PCBA online inspection platform as recited in claim 1, wherein: in the Efficient Net used for classifying the plug-in components, the Efficient Net is 3*3 convolution layers and +7 Mobile Net convolution blocks, wherein the Efficient Net comprises a common 1*1 convolution, depth separable convolution and SE layers which are endowed with different channels and different weights +1*1 convolution + pooling + full connection, and finally, a category is output, so that the positions and the categories of electronic components on the PCBA can be easily marked, a training and testing data set is automatically generated, the marking effect is better than that of human, because the obtained image data redundant information is less, the data are applied to the training Efficient Net, 1-2 Mobile Net convolution blocks can be reduced, the effect of classifying various electronic components can be still achieved, the well-trained Efficient Net can be directly used for detecting the PCBA, the electronic component pictures on the A are sequentially input into the Efficient Net according to the BOM, the components at the positions of the positions on the BOM are compared, and the category of the electronic components does not appear in wrong positions, and the PCBNet appears.

3. The AI visual inspection technology-based PCBA online inspection platform as recited in claim 1, wherein: in the twin network for plug-in and SMT small sample metric learning, the basic idea is as follows: inputting two pictures, respectively extracting features by using the same convolutional neural network, and calculating the distance between two feature vectors to determine whether the contents of the two pictures belong to the same category; the network structure is as follows: firstly, a ResBlock structure, namely batch standardization, reLU activation, convolutional layer, maximum pooling downsampling and residual error connected convolutional neural network is used for extracting the characteristics of two input pictures, then the distance between two characteristic vectors is calculated, then the distance vectors are input into another convolutional neural network, namely convolutional layer, reLU activation and maximum pooling downsampling to extract the characteristics of a higher dimension between the two input pictures, finally the two input pictures enter a full connection layer to obtain a scalar, then a Sigmoid activation function is used for outputting a final result, if the two input pictures are of the same type, the result is 1, and if the two input pictures are of different types, the result is 0.

4. The AI visual inspection technology-based PCBA online inspection platform as recited in claim 1, wherein: the PCBA intelligent AI real-time detection system adopts AI technology, the AI technology utilizes element types and encapsulation to establish a detection model and detect elements, the AI technology detects each element, utilizes prior technical data to quickly obtain relevant information of the detected element, utilizes the known data to predefine the position and classification of an assembled element, combines a PCBA assembly document to mark the detected element, converts a PCBA data set into a PCBA element data set, and identifies and compares the PCBA data set with the PCBA element data set, and the AI technology mainly detects the defect that the test procedures cannot be intercepted.

5. The AI visual inspection technology-based PCBA online inspection platform in accordance with claim 4, wherein: the PCBA intelligent AI real-time detection system comprises the following operation steps:

s6: the simulation inspector carries out online detection, and online AI detects not only has the bad interception of reliability, can avoid personnel to contact the product through using integrated arm simultaneously, and then reduces PCBA and receives ESD or the risk that manual operation probably caused the harm when improper.