CN117710310A - System and method for detecting appearance defects of chip based on artificial intelligence - Google Patents
System and method for detecting appearance defects of chip based on artificial intelligence Download PDFInfo
- Publication number
- CN117710310A CN117710310A CN202311718714.1A CN202311718714A CN117710310A CN 117710310 A CN117710310 A CN 117710310A CN 202311718714 A CN202311718714 A CN 202311718714A CN 117710310 A CN117710310 A CN 117710310A
- Authority
- CN
- China
- Prior art keywords
- chip
- defect
- training
- defects
- detection
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000007547 defect Effects 0.000 title claims abstract description 173
- 238000000034 method Methods 0.000 title claims abstract description 56
- 238000013473 artificial intelligence Methods 0.000 title claims abstract description 37
- 238000001514 detection method Methods 0.000 claims abstract description 84
- 238000012549 training Methods 0.000 claims abstract description 71
- 238000013135 deep learning Methods 0.000 claims abstract description 34
- 238000002372 labelling Methods 0.000 claims abstract description 24
- 238000003860 storage Methods 0.000 claims abstract description 15
- 230000000694 effects Effects 0.000 claims description 16
- 230000002950 deficient Effects 0.000 claims description 11
- 230000002159 abnormal effect Effects 0.000 claims description 7
- 238000012986 modification Methods 0.000 claims description 5
- 230000004048 modification Effects 0.000 claims description 5
- 230000000007 visual effect Effects 0.000 claims description 5
- 238000004590 computer program Methods 0.000 claims description 4
- 230000001960 triggered effect Effects 0.000 claims description 4
- 230000005856 abnormality Effects 0.000 claims description 3
- 238000012545 processing Methods 0.000 claims description 3
- 238000010586 diagram Methods 0.000 description 17
- 238000004422 calculation algorithm Methods 0.000 description 16
- 238000010422 painting Methods 0.000 description 6
- 238000000605 extraction Methods 0.000 description 5
- 230000006870 function Effects 0.000 description 5
- 230000003287 optical effect Effects 0.000 description 5
- 230000004927 fusion Effects 0.000 description 4
- 239000004065 semiconductor Substances 0.000 description 4
- 239000003086 colorant Substances 0.000 description 3
- 230000011218 segmentation Effects 0.000 description 3
- 238000012795 verification Methods 0.000 description 3
- 238000011179 visual inspection Methods 0.000 description 3
- 241000519995 Stachys sylvatica Species 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 238000012790 confirmation Methods 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000009826 distribution Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 238000009825 accumulation Methods 0.000 description 1
- 239000000470 constituent Substances 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 230000008676 import Effects 0.000 description 1
- 230000001788 irregular Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 230000000630 rising effect Effects 0.000 description 1
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Abstract
The invention relates to a system for detecting appearance defects of a chip based on artificial intelligence, which comprises: the image acquisition module is used for setting double strip-shaped light sources to polish two sides of the chip to be detected, and the telecentric lens of the industrial camera is used for acquiring images of the chip which is controlled by the motion platform; the deep learning module is used for automatically labeling and automatically training the acquired images; and the data statistics module is used for comparing and analyzing the feedback result which is actually output with the detection demand data, classifying and counting the identified defect types, counting and summarizing the identification rate and the reject ratio of different defects, and writing the identification rate and the reject ratio of different defects into the local database. The invention also relates to a corresponding method, device, processor and storage medium thereof. The system, the method, the device, the processor and the storage medium thereof for realizing the chip appearance defect detection based on the artificial intelligence can save detection time, thereby improving detection efficiency.
Description
Technical Field
The invention relates to the technical field of artificial intelligence, in particular to the technical field of image detection, and specifically relates to a system, a method, a device, a processor and a computer readable storage medium for realizing chip appearance defect detection based on artificial intelligence.
Background
Since the surface defects of the semiconductor chip may affect the service performance and efficiency of the chip, and more serious defects may cause chip failure, for the appearance defect detection of the semiconductor (defects mainly include dirty points, scratches and breakage), high efficiency and accuracy are generally required, and the defects can be captured quickly and effectively. At the earliest, the manual visual inspection mode of a microscope is adopted, along with the development of a semiconductor process technology, the process of a semiconductor chip is developed from 90nm to 5/7nm at present, the visual inspection mode is used for detecting the appearance defects, the requirement of precision cannot be met, and the manual visual inspection method has the conditions of low detection efficiency and detection accuracy. In recent years, due to the rising technology of artificial intelligence, chip appearance defects can be detected by an automated detection method based on vision and deep learning.
Disclosure of Invention
It is an object of the present invention to overcome the above-mentioned drawbacks of the prior art and to provide a system, method, device, processor and computer readable storage medium thereof for implementing chip appearance defect detection based on artificial intelligence.
To achieve the above object, the system, method, device, processor and computer readable storage medium for implementing chip appearance defect detection based on artificial intelligence of the present invention are as follows:
the system for detecting the appearance defects of the chip based on the artificial intelligence is mainly characterized by comprising the following components:
the image acquisition module is used for setting double strip-shaped light sources to polish two sides of the chip to be detected, and using a telecentric lens of the industrial camera to acquire and process images of the chip which is controlled by the motion platform;
the deep learning module is connected with the image acquisition module and is used for automatically labeling and training the acquired image, outputting a sample image with defect information and carrying out image prediction processing on the sample image to obtain a prediction feedback result;
the data statistics module is connected with the deep learning module, compares and analyzes the feedback result which is actually output with the detection demand data, classifies and counts the identified defect types, counts and gathers the identification rate and the reject ratio of different defects, and writes the identification rate and the reject ratio into the local database.
Preferably, when the sensor arranged on the motion platform detects that the chip reaches the detection area, a trigger signal is sent to the industrial camera so that the industrial camera can start exposure and acquisition, the double-bar light source is synchronously triggered to strobe, and finally, the acquired image is sent to the computer end of the deep learning module through the network interface.
Preferably, the deep learning module includes a prediction sub-module, the prediction sub-module is configured to output detected defect information, including coordinates of a rectangular frame circumscribed by the chip, width and height, and names of defects thereof, and transmit a prediction feedback result to a computer end for local storage, and when an image with defect abnormality is detected, correct information is input again for modification after detection and confirmation are performed manually.
Preferably, the deep learning module further comprises a training sub-module connected with the prediction sub-module for automatically labeling and training the received defect information, wherein,
the automatic labeling process specifically comprises the following steps: using an automatic labeling tool to automatically label the image with the defect, outputting a sample image with labeling information, and importing the sample image into a training tool to perform iterative training treatment;
the automatic training process specifically comprises the following steps: selecting a small number of samples with balanced categories for labeling training, adjusting, modifying and increasing training samples in a small amplitude according to the identification result, re-marking the training samples which are missed to be detected and are misidentified, inheriting the training again until the identification effect of the total sample set is good, and inputting the finally output improved model into the prediction submodule for iterative training.
Preferably, the data statistics module specifically includes:
when the size of the outputted image defect is larger than the size of the detection requirement, or when the defect is defective, or when the size of the defect does not meet the size of the detection requirement, and a plurality of defects exist on the chip, when the sum of the areas exceeds the area of the detection requirement, judging whether the appearance of the chip has defects currently, classifying and counting the identified defect types, counting and summarizing the identification rate and the defective rate of different defects, and finally writing the identification rate and the defective rate into a local database in a data form.
The method for detecting the chip appearance defects based on artificial intelligence by using the system is mainly characterized by comprising the following steps of:
(1) When the workpiece positioning detector detects that the chip moves to be close to the center of the visual field of the industrial camera, a trigger pulse signal is sent to the industrial camera;
(2) The industrial camera respectively sends stroboscopic trigger signals to the double-bar-shaped light source according to a preset program and delay, and performs image acquisition;
(3) After the industrial camera starts exposure and completes image acquisition, image information is transmitted to a computer end for memory;
(4) The prediction submodule processes, analyzes and identifies the acquired image and obtains a prediction result;
(5) The training sub-module automatically marks the stored images, carries out iterative training, and finally outputs an improved model to the prediction sub-module so as to improve the detection rate and the accuracy;
(6) The prediction module feeds back a prediction result to the computer end, and the computer end controls the motion of the motion platform to classify the product defects;
(7) And counting the identification rate, the reject ratio of different defects and the abnormal images through a data counting module and summarizing the identification rate, the reject ratio of different defects and the abnormal images to a local database.
The device for detecting the appearance defects of the chip based on artificial intelligence is mainly characterized by comprising the following components:
a processor configured to execute computer-executable instructions;
and a memory storing one or more computer-executable instructions which, when executed by the processor, perform the steps of the artificial intelligence based chip appearance defect detection method described above.
The processor for realizing the chip appearance defect detection based on the artificial intelligence is mainly characterized in that the processor is configured to execute computer executable instructions, and when the computer executable instructions are executed by the processor, the steps of the method for detecting the chip appearance defect based on the artificial intelligence are realized.
The computer readable storage medium is characterized in that the computer program is stored thereon, and the computer program can be executed by a processor to implement the steps of the method for detecting chip appearance defects based on artificial intelligence.
The system, the method, the device, the processor and the computer readable storage medium thereof for realizing the chip appearance defect detection based on the artificial intelligence can realize hundred millisecond detection, the detection precision can reach 20um, the current shooting precision is 5um of one pixel, the minimum defect size acceptable by the deep learning module is 3 pixels, namely when the defect is more than 3 pixels, the corresponding defect can be better found by the deep learning algorithm; if the set defect precision is smaller than 3 pixels, the training time is greatly increased and more erroneous judgment is possible, so that hardware selection is required according to the characteristics of the algorithm.
The optical scheme and the model sample are not required to be changed aiming at products of different models, so that the compatibility is high; the current visual field size is 22.5mm×16.8mm, and the appearance detection of chips with different sizes can be satisfied. The method can realize simultaneous identification and output aiming at different types of defects without additional classification steps, and can see that the scratches of the chip are marked by the painting brushes from the predicted result display diagram of the technical scheme, the dirty defects are marked by the black point painting brushes, and in practical application, different defects can be marked by the painting brushes with different colors and the defect names, so that different defects of the appearance of the chip are simultaneously output, and classification operation is not needed on the recognized defect results, thereby saving the detection time and improving the detection efficiency.
Drawings
FIG. 1 is a schematic diagram of a system for detecting chip appearance defects based on artificial intelligence according to the present invention.
FIG. 2 is a flow chart of the process of the present invention for automatic labeling.
FIG. 3 is a flow chart of the process of the present invention for automatic training.
FIG. 4 is a schematic diagram of the results of model training according to the present invention.
FIG. 5 is a flow chart of a process for performing deep learning pixel segmentation in accordance with the present invention.
Fig. 6 is a flow chart of the process of feature extraction according to the present invention.
FIG. 7 is a schematic diagram of an interface for software control according to the present invention.
Fig. 8 is an interface diagram of AI initialization setup according to the present invention.
FIG. 9 is a diagram of the data statistics performed in accordance with the present invention.
FIG. 10 is a diagram illustrating a log query according to the present invention.
FIG. 11 is a schematic diagram showing the result reporting error when the log query is performed in the present invention.
Fig. 12 is an effect diagram of the present invention using coaxial light photographing to appear white spots.
Fig. 13 is a graph showing the effect of scratch occurrence using coaxial light photographing in the present invention.
Fig. 14 is a graph showing the effect of breakage using coaxial light photographing in the present invention.
Fig. 15 is a schematic diagram of the present invention when performing object detection in practical application.
Fig. 16 is a schematic diagram of the present invention for performing object recognition in practical application.
Fig. 17 is a schematic diagram of obtaining a target recognition result in practical application of the present invention.
Detailed Description
In order to more clearly describe the technical contents of the present invention, a further description will be made below in connection with specific embodiments.
Before describing in detail embodiments that are in accordance with the present invention, it should be observed that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus.
Referring to fig. 1, the system for detecting chip appearance defects based on artificial intelligence includes:
the image acquisition module is used for setting double strip-shaped light sources to polish two sides of the chip to be detected, and using a telecentric lens of the industrial camera to acquire and process images of the chip which is controlled by the motion platform;
the deep learning module is connected with the image acquisition module and is used for automatically labeling and training the acquired image, outputting a sample image with defect information and carrying out image prediction processing on the sample image to obtain a prediction feedback result;
the data statistics module is connected with the deep learning module, compares and analyzes the feedback result which is actually output with the detection demand data, classifies and counts the identified defect types, counts and gathers the identification rate and the reject ratio of different defects, and writes the identification rate and the reject ratio into the local database.
As a preferred embodiment of the invention, when the sensor arranged on the motion platform detects that the chip reaches the detection area, a trigger signal is sent to the industrial camera so that the industrial camera can start exposure and acquisition, the double-bar light source is synchronously triggered to strobe, and finally, the acquired image is sent to the computer end of the deep learning module through a network interface.
As a preferred embodiment of the invention, the deep learning module comprises a prediction sub-module, wherein the prediction sub-module is used for outputting detected defect information, including coordinates, width and height of a rectangular frame externally connected with a chip and names of defects, transmitting a prediction feedback result to a computer end for local storage, and inputting correct information again for modification when an image with defect abnormality is detected after detection and confirmation are carried out manually.
As a preferred embodiment of the invention, the deep learning module further comprises a training sub-module, which is connected with the prediction sub-module and is used for automatically marking and training the received defect information, wherein,
the automatic labeling process specifically comprises the following steps: using an automatic labeling tool to automatically label the image with the defect, outputting a sample image with labeling information, and importing the sample image into a training tool to perform iterative training treatment;
the automatic training process specifically comprises the following steps: selecting a small number of samples with balanced categories for labeling training, adjusting, modifying and increasing training samples in a small amplitude according to the identification result, re-marking the training samples which are missed to be detected and are misidentified, inheriting the training again until the identification effect of the total sample set is good, and inputting the finally output improved model into the prediction submodule for iterative training.
As a preferred embodiment of the present invention, the data statistics module specifically includes:
when the size of the outputted image defect is larger than the size of the detection requirement, or when the defect is defective, or when the size of the defect does not meet the size of the detection requirement, and a plurality of defects exist on the chip, when the sum of the areas exceeds the area of the detection requirement, judging whether the appearance of the chip has defects currently, classifying and counting the identified defect types, counting and summarizing the identification rate and the defective rate of different defects, and finally writing the identification rate and the defective rate into a local database in a data form.
The individual constituent modules of the system for implementing chip appearance defect detection based on artificial intelligence will be described in further detail below.
An image detection module:
because the defects to be detected are scratches, defects, white spots and the like, the defects have the characteristics of irregular shape, low contrast of depth and the like, the edges of the defects on the plane are highlighted by a double-strip-shaped light source low-angle side-striking mode, and the interference caused by reflection is reduced. In a precise optical detection system, a common optical lens has certain constraint factors such as edge distortion of an image, so that the measurement precision is influenced, and therefore, a telecentric lens is required to be selected, and the situation of parallax influence of a traditional industrial lens is mainly corrected, so that the obtained image magnification cannot be changed within a certain object distance range.
When the sensor detects that the chip reaches the detection area, the motion platform can send a trigger signal to the camera end, the camera end starts to expose and start to collect after receiving the signal, the light source is triggered synchronously to perform stroboscopic operation, after the collection is completed, the collected image can be transmitted to the computer through a network interface, the prediction module can acquire image information from the memory of the computer and predict the image information, and the training module can automatically label the image stored in the local computer, so that the model is continuously perfected after iterative training.
And the deep learning module is used for:
the traditional algorithm is easy to misdetect, and for defects with lower contrast, such as scratches and scratches, the period required to develop the traditional algorithm can be long, and the algorithm can not meet the representation of all defect characteristics, and a good effect can be achieved by adjusting parameters on site for many times.
Through the deep learning algorithm, the conditions of false detection and missed detection can be effectively reduced, in addition, a certain amount of samples are needed for deep learning, and along with the accumulation of the samples, the recognition rate of the deep learning is better.
a. Training module
The whole training module mainly comprises two parts of automatic labeling and automatic training:
the flow of automatic labeling is shown in fig. 2:
the prediction end outputs defect information including coordinates, width and height of a defect circumscribed rectangular frame, names of the defects and the like, the computer end stores collected images to a local memory, after the images are confirmed manually, if the images with abnormal defects are encountered, the coordinate information and the width and height of the defects can be input, the defect names are modified, after the automatic marking tool is imported, the tool finally outputs a sample image with marking information to import the training tool for iterative training.
The flow of the automatic training is as shown in fig. 3:
in defect detection, the phenomenon of unbalanced sample distribution is often encountered, a certain class of defect objects in a picture set are far more than other classes, and unbalanced samples can influence the learning classification and recognition of a network model, so that the recognition effect of the network model deviates towards more classes, and generalization and recognition accuracy of the model are influenced.
Therefore, it is suggested to label the training samples according to the principle of 'a small number of times', so that a better training effect is obtained. It can be summarized in general as: and selecting a small amount of representative sample labeling training with balanced categories, adjusting, modifying and increasing training samples in a small amplitude according to the recognition result, inheriting the training again, and repeating until the recognition effect of the total sample set is good.
Combining the currently acquired pictures and the detection demands of clients, 2-3 pictures with different defect types can be marked each time as a training set for training according to different defects, after the round training is finished, the overall detection effect of the remaining verification set is manually confirmed, and then 2-3 pictures with different defect types are randomly selected as the training set for continuous iterative training; thus, a model with excellent effect is finally produced.
As shown in fig. 4, the posaccs is the accuracy of the positive sample (the marked defect area is the positive sample), the NegAcc is the accuracy of the negative sample (the background area except the defect area is the negative sample), both parameters need to be stabilized above 0.9, the Loss rate (the parameters determine whether the defect area is found accurately during prediction or not, good padding is made for the size calculation of the defect to be done later) is the Loss rate (the Loss parameters need to be smaller and better, and training can be suspended when the Loss parameters need to be smaller and better and lower than 0.04, and the obtained training model can basically meet the current sample verification requirement.
b. Prediction module
Under the prediction module, the deep learning algorithm firstly generates a picture with the same size as the predicted image, probability threshold judgment is carried out on each pixel point in the image through algorithm prediction, namely the similarity degree of each pixel point and defect characteristics, finally, a defect distribution heat map with the same size as the original image resolution is obtained according to the judgment of the probability threshold, the pixel points with the larger probability threshold are classified as defects, and the coordinates of the pixel points are recorded. The coordinates satisfying the defect index (defect similarity score, namely probability of the defect) are expressed by 1-255, and are expressed as final defect output, and the final effect presented to the customer is information such as the position size and the size of the defect.
The flow frame diagram for the deep learning pixel segmentation is shown in fig. 5 and 6:
the algorithm comprises the following steps:
and inputting the feature maps with different resolutions into a multi-scale fusion enhancement network.
In the first feature extraction and exchange unit (unit 1), each branch performs independent feature extraction, and then information exchange of different feature graphs is realized through feature fusion. In the process of information fusion, the feature map of the first branch can obtain feature maps 1A1 and 1A2,1A2 through downsampling with a step length of 1 or 2, then obtain corresponding feature maps 1A3 through downsampling with a step length of 2, the feature maps of the second branch and the third branch obtain feature maps A21 and A31 through bilinear interpolation upsampling, and the output feature map of the first branch of the unit1 can be obtained by adding and fusing the channel numbers of the A21 and the A31 with the 1A1 after the channel numbers are adjusted. The output characteristic diagrams of the second branch and the third branch are the same. Thus, three feature maps of different resolutions can be acquired by unit1 and used as inputs to the second feature extraction and exchange unit (unit 2).
The calculation process of the unit2 is the same as that of the unit1, the output of three branches in the unit2 is the final output characteristic of the multi-scale fusion enhancement network, and the characteristic diagram of each branch contains high-resolution detail information, so that the accuracy of pixel segmentation is improved.
The convolution process replaces the traditional algorithm to find the judgment basis of the current pixel positioning, and theoretically, as long as the network capacity is enough to design, a sample is enough to find a network meeting the feature mapping, wherein in the deep learning process, the convolution process of feature extraction and the design of a loss function are designed to finally influence the quality of the obtained network.
And a data statistics module: according to the detection requirement of a customer, proper training parameters are adjusted to train, including detection precision, detection size and the like, after a prediction model is generated, images of chips on a production line are acquired in real time, defect detection analysis is carried out on the images, and coordinates and size of defects are output. Through carrying out the contrast analysis to the data of detection demand and the result data of actual output, when the defect size of output is greater than the size of customer's detection demand, or when the priority of defect is higher (appear this kind of defect, this chip can judge as the defective products promptly), or when defect size does not satisfy the size of customer's detection demand, but there are many defects on this chip, when the area sum exceeds the detection demand area, thereby judge whether the chip outward appearance has the defect, and classify and count the defect type that has been discerned, carry out statistics and summarize the defective rate of recognition rate, different defects finally write into local database with the form of data, finally make things convenient for the terminal to improve and perfect to the technological process.
The method for detecting the chip appearance defects based on artificial intelligence by using the system comprises the following steps:
(1) When the workpiece positioning detector detects that the chip moves to be close to the center of the visual field of the industrial camera, a trigger pulse signal is sent to the industrial camera;
(2) The industrial camera respectively sends stroboscopic trigger signals to the double-bar-shaped light source according to a preset program and delay, and performs image acquisition;
(3) After the industrial camera starts exposure and completes image acquisition, image information is transmitted to a computer end for memory;
(4) The prediction submodule processes, analyzes and identifies the acquired image and obtains a prediction result;
(5) The training sub-module automatically marks the stored images, carries out iterative training, and finally outputs an improved model to the prediction sub-module so as to improve the detection rate and the accuracy;
(6) The prediction module feeds back a prediction result to the computer end, and the computer end controls the motion of the motion platform to classify the product defects;
(7) And counting the identification rate, the reject ratio of different defects and the abnormal images through a data counting module and summarizing the identification rate, the reject ratio of different defects and the abnormal images to a local database.
In a specific embodiment of the present invention, the actual detection steps of the present technical solution are as follows:
1. device scanning
Clicking the 'open device' to scan and connect the device, reading the camera parameters in the xml file, modifying the trigger mode and trigger source, and controlling the camera to collect images in a hard trigger or soft trigger mode.
Ai initialization
Clicking the AI path to set the engineering path and the model path can be automatically saved in the ini file after setting, and the software is opened next time without repeated setting unless the path needs to be modified. According to the actual condition of the site and different detection standards, the threshold value and the detection area are modified, so that the recognition rate and the accuracy rate can be effectively improved.
When the detected areas are set to be different, fine defects can be effectively filtered, as shown in fig. 7, when the detected areas are set to be 30 pixels, the fine defects are filtered, and the prediction result is OK.
3. Algorithmic prediction
Clicking on "start acquisition", the camera is in an acquisition mode, single frame acquisition is carried out through a soft trigger button, the single frame acquisition is transmitted to a deep learning module for prediction, the identified defects can be displayed in an interface image, different defects can be marked by using brushes with different colors and defect names, and statistics can be conveniently confirmed. The output result has prediction time, prediction result and defect number.
4. Data statistics
Clicking on "prediction statistics" can see that statistics of the predicted output result at this time are displayed in the form of a histogram and a pie chart, respectively.
5. Log query
Clicking the log inquiry can monitor the utilization rate and occupancy rate of the CPU and the GPU of the industrial personal computer at the moment, and the time stamp of each operation and some results are displayed in a wrong way.
When the color of red appears, the error reporting prompt information is indicated to be of higher level, which may affect some subsequent operations, and some functions may be lost or skipped; green indicates that all the current operations are normal; yellow indicates that the operation is wrong but does not affect the normal operation of the main program, and mainly plays a role of prompting.
The effect diagrams of the coaxial light shooting are shown in fig. 12 to 17, and since the deep learning algorithm needs a certain amount of samples for learning, a certain amount of samples need to be collected on site for training and labeling in the process of preparing the project in advance. The software has the function of storing pictures, saves a certain amount of sample pictures, and then carries out iterative training through a training tool to finally obtain a better model file. The contrast of the sample can be improved in an on-axis light mode, recognition of a deep learning algorithm is facilitated, and the detection effect is better. Firstly, a detected part is required to be positioned (a place affecting the detection effect is removed, so that the detection accuracy can be improved and the detection time can be shortened) to mark the place considered as the defect, then training is carried out (an artificial intelligent deep learning algorithm is enabled to learn defect characteristics, and thus the defect characteristics can be automatically identified when similar defects are detected), the artificial intelligent deep learning algorithm is used for verification, and the defect with lower contrast can be effectively found by marking obvious scratch white point defects and carrying out iterative training.
Device for realizing chip appearance defect detection based on artificial intelligence, wherein the device comprises:
a processor configured to execute computer-executable instructions;
and a memory storing one or more computer-executable instructions which, when executed by the processor, perform the steps of the artificial intelligence based chip appearance defect detection method described above.
The implementation is based on the processor of the chip appearance defect detection of the artificial intelligence, wherein the processor is configured to execute the computer executable instructions, and when the computer executable instructions are executed by the processor, the steps of the method of the chip appearance defect detection based on the artificial intelligence are implemented.
The computer readable storage medium having stored thereon a computer program executable by a processor to perform the steps of the method for artificial intelligence based chip appearance defect detection described above.
Any process or method descriptions in flow charts or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps of the process, and further implementations are included within the scope of the preferred embodiment of the present invention in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art of the present invention.
It is to be understood that portions of the present invention may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the various steps or methods may be implemented in software or firmware stored in a memory and executed by a suitable instruction execution device.
Those of ordinary skill in the art will appreciate that all or a portion of the steps carried out in the method of the above-described embodiments may be implemented by a program to instruct related hardware, and the program may be stored in a computer readable storage medium, where the program when executed includes one or a combination of the steps of the method embodiments.
The above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, or the like.
In the description of the present specification, reference to the terms "one embodiment," "some embodiments," "examples," "specific examples," or "embodiments," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiments or examples. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
While embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and not to be construed as limiting the invention, and that variations, modifications, alternatives and variations may be made to the above embodiments by one of ordinary skill in the art within the scope of the invention.
The system, the method, the device, the processor and the computer readable storage medium thereof for realizing the chip appearance defect detection based on the artificial intelligence can realize hundred millisecond detection, the detection precision can reach 20um, the current shooting precision is 5um of one pixel, the minimum defect size acceptable by the deep learning module is 3 pixels, namely when the defect is more than 3 pixels, the corresponding defect can be better found by the deep learning algorithm; if the set defect precision is smaller than 3 pixels, the training time is greatly increased and more erroneous judgment is possible, so that hardware selection is required according to the characteristics of the algorithm.
The optical scheme and the model sample are not required to be changed aiming at products of different models, so that the compatibility is high; the current visual field size is 22.5mm×16.8mm, and the appearance detection of chips with different sizes can be satisfied. The method can realize simultaneous identification and output aiming at different types of defects without additional classification steps, and can see that the scratches of the chip are marked by the painting brushes from the predicted result display diagram of the technical scheme, the dirty defects are marked by the black point painting brushes, and in practical application, different defects can be marked by the painting brushes with different colors and the defect names, so that different defects of the appearance of the chip are simultaneously output, and classification operation is not needed on the recognized defect results, thereby saving the detection time and improving the detection efficiency.
In this specification, the invention has been described with reference to specific embodiments thereof. It will be apparent, however, that various modifications and changes may be made without departing from the spirit and scope of the invention. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense.
Claims (8)
1. A system for implementing chip appearance defect detection based on artificial intelligence, the system comprising:
the image acquisition module is used for setting double strip-shaped light sources to polish two sides of the chip to be detected, and using a telecentric lens of the industrial camera to acquire and process images of the chip which is controlled by the motion platform;
the deep learning module is connected with the image acquisition module and is used for automatically labeling and training the acquired image, outputting a sample image with defect information and carrying out image prediction processing on the sample image to obtain a prediction feedback result;
the data statistics module is connected with the deep learning module, compares and analyzes the feedback result which is actually output with the detection demand data, classifies and counts the identified defect types, counts and gathers the identification rate and the reject ratio of different defects, and writes the identification rate and the reject ratio of different defects into a local database;
the deep learning module comprises a training submodule which is used for automatically marking and automatically training the received defect information, wherein,
the automatic labeling process specifically comprises the following steps: using an automatic labeling tool to automatically label the image with the defect, outputting a sample image with labeling information, and importing the sample image into a training tool to perform iterative training treatment;
the automatic training process specifically comprises the following steps: selecting a small amount of samples with balanced categories for labeling training, adjusting, modifying and increasing training samples in a small amplitude manner according to the identification result, re-marking the training samples which are missed and misidentified and inheriting the training again until the identification effect of the total sample set is good, and inputting the finally output improved model into the prediction submodule for iterative training, wherein the automatic training parameters comprise the accuracy PosAcc for labeling the defect area as a positive sample, the accuracy NegAcc for labeling the background area except the defect area as a negative sample and the sample Loss rate Loss.
2. The system for detecting the appearance defects of the chip based on the artificial intelligence according to claim 1, wherein when the sensor arranged on the motion platform detects that the chip reaches the detection area, a trigger signal is sent to the industrial camera so that the industrial camera can start exposure and acquisition, the double-bar light source is triggered synchronously to strobe, and finally the acquired image is sent to the computer end of the deep learning module through a network interface.
3. The system for detecting the appearance defects of the chip based on the artificial intelligence according to claim 2, wherein the deep learning module comprises a prediction sub-module, the prediction sub-module is used for outputting detected defect information including coordinates, width and height of a rectangular frame externally connected with the chip and names of defects of the rectangular frame, transmitting a prediction feedback result to a computer end for local storage, and inputting correct information again for modification when an image with defect abnormality is detected after the detection is confirmed by the artificial.
4. The system for detecting appearance defects of a chip based on artificial intelligence according to claim 3, wherein the data statistics module is specifically:
when the size of the outputted image defect is larger than the size of the detection requirement, or when the defect is defective, or when the size of the defect does not meet the size of the detection requirement, and a plurality of defects exist on the chip, when the sum of the areas exceeds the area of the detection requirement, judging whether the appearance of the chip has defects currently, classifying and counting the identified defect types, counting and summarizing the identification rate and the defective rate of different defects, and finally writing the identification rate and the defective rate into a local database in a data form.
5. A method for implementing artificial intelligence based chip appearance defect detection using the system of claim 4, said method comprising the steps of:
(1) When the workpiece positioning detector detects that the chip moves to be close to the center of the visual field of the industrial camera, a trigger pulse signal is sent to the industrial camera;
(2) The industrial camera respectively sends stroboscopic trigger signals to the double-bar-shaped light source according to a preset program and delay, and performs image acquisition;
(3) After the industrial camera starts exposure and completes image acquisition, image information is transmitted to a computer end for memory;
(4) The prediction submodule processes, analyzes and identifies the acquired image and obtains a prediction result;
(5) The training sub-module automatically marks the stored images, carries out iterative training, and finally outputs an improved model to the prediction sub-module so as to improve the detection rate and the accuracy;
(6) The prediction module feeds back a prediction result to the computer end, and the computer end controls the motion of the motion platform to classify the product defects;
(7) And counting the identification rate, the reject ratio of different defects and the abnormal images through a data counting module and summarizing the identification rate, the reject ratio of different defects and the abnormal images to a local database.
6. An apparatus for implementing chip appearance defect detection based on artificial intelligence, the apparatus comprising:
a processor configured to execute computer-executable instructions;
a memory storing one or more computer-executable instructions which, when executed by said processor, perform the steps of the artificial intelligence based chip appearance defect detection method of claim 5.
7. A processor for implementing artificial intelligence based chip appearance defect detection, wherein the processor is configured to execute computer executable instructions that, when executed by the processor, implement the steps of the method for artificial intelligence based chip appearance defect detection of claim 5.
8. A computer readable storage medium having stored thereon a computer program executable by a processor to perform the steps of the artificial intelligence based chip appearance defect detection method of claim 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311718714.1A CN117710310A (en) | 2023-12-14 | 2023-12-14 | System and method for detecting appearance defects of chip based on artificial intelligence |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311718714.1A CN117710310A (en) | 2023-12-14 | 2023-12-14 | System and method for detecting appearance defects of chip based on artificial intelligence |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117710310A true CN117710310A (en) | 2024-03-15 |
Family
ID=90147392
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311718714.1A Pending CN117710310A (en) | 2023-12-14 | 2023-12-14 | System and method for detecting appearance defects of chip based on artificial intelligence |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117710310A (en) |
-
2023
- 2023-12-14 CN CN202311718714.1A patent/CN117710310A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109550712B (en) | Chemical fiber filament tail fiber appearance defect detection system and method | |
| US11774735B2 (en) | System and method for performing automated analysis of air samples | |
| CN109829914B (en) | Method and device for detecting product defects | |
| US20170277979A1 (en) | Identifying defect on specular surfaces | |
| TWI669519B (en) | Board defect filtering method and device thereof and computer-readabel recording medium | |
| US8824823B1 (en) | Increased quality of image objects based on depth in scene | |
| CN114280075A (en) | Online visual inspection system and method for surface defects of pipe parts | |
| KR20220164124A (en) | System for inspecting product defects by type based on a deep learning model | |
| CN111426693A (en) | Quality defect detection system and detection method thereof | |
| CN113095445B (en) | Target identification method and device | |
| CN114674830A (en) | Bottle cap flaw detection module on high-speed production line | |
| KR20180092033A (en) | Component registration device | |
| CN112161984B (en) | Wine positioning method, wine information management method, device, equipment and storage medium | |
| JP7379478B2 (en) | Milk analyzer for classifying milk | |
| CN113145473A (en) | Intelligent fruit sorting system and method | |
| CN117710310A (en) | System and method for detecting appearance defects of chip based on artificial intelligence | |
| CN111275756A (en) | Spool positioning method and apparatus | |
| CN116681677A (en) | Lithium battery defect detection method, device and system | |
| CN116245882A (en) | Circuit board electronic element detection method and device and computer equipment | |
| CN116256366A (en) | Chip defect detection method, detection system and storage medium | |
| CN115672778A (en) | Intelligent visual recognition system | |
| CN114219758A (en) | Defect detection method, system, electronic device and computer readable storage medium | |
| CN108362708A (en) | Pen core scribing line standardized detection system based on machine vision and method | |
| CN113642473A (en) | Mining coal machine state identification method based on computer vision | |
| CN114596243A (en) | Defect detection method, device, equipment and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |