TW202120917A - The automatic marking method and device of intelligent optical detection for sample features and defects - Google Patents

Abstract

The invention disclosures an automatic marking method and device of intelligent optical detection for sample features and defects, which is implemented in three stages, the first stage is to generate a small number of data sets by artificial marking and to train the intelligent recognizer in a deep learning manner. The second stage then treats the characteristics and defects intended in the image with the intelligent recognizer of deep learning training to make a large number of data sets, but some programs still perform three implementable processes manually and retrain the data in a deep learning manner. Through a number of manual checks, and did not find that the automatic marking method has misjudged, then enter the third stage of fully artificial intelligent and fully automatic marking processing, no longer through the manual processing process.

Description

Method and device for automatically marking sample features and defects for intelligent optical inspection

The present invention relates to a method and device for automatic marking of sample features and defects for intelligent optical inspection; applied to optical inspection, especially a method that can effectively integrate deep learning, can greatly reduce manpower consumption and improve efficiency.

In the field of automated testing, there is a basic shortcoming that the detection effect is often limited due to the inability to obtain enough defective samples, or it takes a very long time to adjust after the actual production line to stabilize the results.

Moreover, if it is to be used in the field of deep learning, the marking of features and defects will cost a lot of human resources and there are many difficulties and problems that must be overcome.

Therefore, the inventor of the present case has, after years of painstaking research and development, in view of the deficiencies of Xi’s technology, proposes a method and device for automatic marking of sample features and defects for intelligent optical inspection and its device as described below.

In view of the shortcomings caused by the above-mentioned conventional technologies, the purpose of an automatic marking method and device for sample features and defects in intelligent optical inspection of the present invention is to use deep learning technology to target the desired marked features and defects in the image to be tested. Automatic marking can greatly reduce manpower consumption and improve efficiency.

The present invention is an intelligent optical inspection method for automatic marking of sample features and defects and its device. The implementation method is divided into three stages. The first stage is the manual marking stage, the second stage is the semi-automatic marking manual assistance stage, and the third stage is Automatically mark the stage.

The first stage of the implementation method of the present invention is the manual marking stage. The process includes sample acquisition, which means taking a camera to take an image of the object to be measured; artificial marking of features or defects means manually taking the sample in the image Select and mark the features or defects of the image; manual marking classification refers to manually classifying and recording the captured sample images and marking results; training refers to the classification of the captured sample images and marking results The data is transferred to the feature and defect intelligent automatic marking model, and the neural network based on the object detection model is used for training in the way of deep learning. Because the features or defects are manually marked and the marking results are classified, It consumes more manpower and time, so the data set generated at this stage is small; exporting model training results refers to exporting the model training results to the database unit for later intelligent identification, feature and defect intelligent automatic marking model or Used by other applications.

The second stage of the implementation method of the present invention is the semi-automatic marking and manual assistance stage. In this stage, there are three processes that can be implemented accordingly. The first process includes sample acquisition, which refers to capturing the object to be measured with a camera; manual marking features or Defects refers to manually selecting and marking the characteristics or defects of the sample in the captured image; automatic marking classification refers to the automatic classification and recording of the captured sample image and the marking result by the intelligent recognizer; Manual inspection means to prevent the intelligent recognizer from automatically selecting and marking sample features or defects in the captured image, or when the intelligent recognizer automatically classifies the marking results, because the number of data sets used as the basis for discrimination is not large enough As a result, the marking or classification results have missed or overkilled misjudgments, so the marking or classification results are manually checked, and if a misjudgment occurs, the marking or classification results will be corrected; training means that the image will be taken The data of the sample image and the marking result are transferred to the feature or defect intelligent automatic marking model, and the neural network based on the object detection model is used for training in the way of deep learning, because the intelligent recognizer automatically performs Marking results classification can save human resources and have faster classification efficiency. Therefore, the generated data set is also greatly increased compared with the previous manual marking stage; exporting model training results refers to exporting model training results to the database unit for After that, the smart recognizer, the feature or defect smart automatic marking model, or other applications are used.

The second stage of the implementation method of the present invention is the semi-automatic marking manual assistance stage. The second process can be implemented at the same time as the first process, including sample capturing, which means capturing the object to be tested with a camera; automatically marking features or defects , Refers to the automatic frame selection and marking of sample features or defects in the captured image by the intelligent recognizer based on the model training result data transmitted by the database unit; manual label classification refers to the result of marking the captured sample image Classification is performed manually; manual inspection refers to preventing the intelligent recognizer from automatically frame selection and marking of sample features or defects in the captured image, or when the intelligent recognizer automatically classifies the marking results, it is used as the basis for discrimination The data set is not large enough to cause misjudgment of missing or overkill in the classification results, so the marking and classification results are manually checked. If there is a misjudgment, the marking and classification results will be corrected, but the process is therefore It is a manual method to classify the sample image marking results, which already contains the meaning of manual inspection, so the manual inspection procedure in this process is not a necessary implementation step; training refers to the sample image taken and the marking result The data is transferred to the feature and defect intelligent automatic marking model, and the neural network based on the object detection model is used for training in the way of deep learning. Because the intelligent recognizer automatically frames and marks the sample features or defects in the captured image, it can save human resources and have a faster classification efficiency, so the generated data set is also greatly increased compared with the previous manual marking stage; export model The training result refers to exporting the model training result to the database unit for later use by the intelligent recognizer or the feature and defect intelligent automatic marking model or other applications.

The second stage of the implementation method of the present invention is the semi-automatic marking manual assistance stage. The third process is implemented after the first process and the second process and includes sample acquisition, which means that the object to be measured is captured by a camera; automatic marking features Or defect refers to the automatic frame selection and marking of the sample features or defects in the captured image using the model training result data transmitted from the database unit with the intelligent identifier; automatic marking classification refers to the sample image and The marking result is automatically classified and recorded by the result of selecting and marking the sample features or defects with the model training result data transmitted by the database unit by the intelligent identifier; manual inspection means to prevent the intelligent identifier from automatically capturing the image When selecting and marking the features or defects of the medium sample, or automatically classifying the marking results, because the number of data sets used as the basis for the discrimination is not large enough, the marking or classification results have missed or overkilled misjudgments, so the manual method Check the marking or classification results, and correct them if there is a misjudgment. Training means to transfer the acquired sample image and the data of the marking result to the feature or defect intelligent automatic marking model, and use the object detection The neural network based on the model (object detection) is trained by deep learning. Because the intelligent recognizer automatically frames and marks the sample features or defects in the captured image, and automatically classifies the marking results, it can save human resources and have a faster classification efficiency, so the generated data set is also marked manually than before The number of stages has increased greatly; exporting model training results refers to exporting model training results to a database unit for later use by smart identifiers or feature and defect smart automatic marking models or other applications.

When the second stage, the semi-automatic marking manual assistance stage, has accumulated a large enough amount of generated data sets, so that the smart recognizer has enough data sets as the basis for discrimination, and the classification results have no more misjudgments after repeated manual inspection procedures. , You no longer need manual assistance, and enter the third stage, which is the automatic marking stage.

The third stage of the implementation method of the present invention, which is the automatic marking stage, includes sample acquisition, which refers to capturing the object to be tested with a camera; automatic marking of features or defects is model training transmitted by the database unit with a smart identifier Result data, automatically frame and mark the sample features or defects in the captured image; automatic label classification refers to the use of the captured sample image and the marked result with the intelligent identifier based on the model training result transmitted by the database unit The data is automatically classified as a result of frame selection and marking of features or defects. Exporting results refers to exporting the mark and classification result data to the database unit for later use by smart recognizers or other applications.

Therefore, the present invention is an intelligent optical inspection method and device for automatically marking sample features and defects. The first stage is to manually mark the desired features and defects in the image under test and classify the marking results. , The classification results are generated in a small amount of data sets by means of deep learning. After that, in the second stage, the smart identifier semi-automatic marking method is used, but some of the procedures still need to be manually assisted by three implementable processes to generate a large number of data sets. After many manual inspections and no misjudgment is found, you can enter the third stage to completely automatically mark the features and defects you want to mark in the test picture with a smart recognizer and classify the marking results, and finally export and save the classification results In the database unit, for use by smart identifiers or other applications. There is no need to go through manual processing procedures, which can greatly reduce manpower consumption and improve efficiency.

The present invention is an intelligent optical inspection method and device for automatically marking sample features and defects. The collector mainly includes a loading and collecting platform for placing the object to be tested; a light source device, which is set on the loading and collecting platform. The upper periphery of the platform is equipped with various sets of light source fixtures such as coaxial light source, line light source, backlight source, ring light source, spherical light source, etc., combined with movable mechanisms such as mechanical arms and light source photography control modules to provide multiple lighting methods In order to provide different light source control methods to switch each group of light sources, and edit the intensity and angle of the light sources; the camera unit, the camera unit and the light source device are staggered on the upper periphery of the loading and collection platform, including camera fixtures, line camera fixtures Camera fixtures and other applications combined with the camera’s repeat shooting function and the function of quickly collecting data; the smart identifier is connected to the database unit, and the model training result data transmitted by the database unit can automatically take pictures of the sample The features and flaws in the frame are selected and marked, and the marking results are classified; the light source photography control module is connected to the light source device and the camera unit and the movable mechanism to control the light source device and the camera unit for intelligence Lighting photography process; movable mechanism, the movable mechanism can be a lifting mechanism conveying mechanism or a mechanical arm; features and defects intelligent automatic marking model, linked to the intelligent identifier and database unit, can capture the sample The data of the image and labeling results are trained by a neural network based on the object detection model, and the training model results are exported to the database unit; the database unit, respectively, is linked to the news In the feature and defect intelligent automatic marking model and intelligent recognizer, the captured sample image frame selection and marking results and classified data can be stored in the database unit, and the data will be transferred to the feature and defect intelligent automatic marking The model allows it to be trained by deep learning and the model training results are imported to the database unit for storage. The data can also be sent to the smart recognizer so that it can automatically frame and select the features and defects in the sample image. Mark or classify the marking results.

The following is a specific example to illustrate the implementation of the present invention. Those familiar with the art can easily understand the other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied by other different specific examples. The details in this specification can also be based on different viewpoints and applications, and various modifications and changes can be made without departing from the spirit of the present invention.

First, please refer to the first figure. The present invention is an intelligent optical inspection method and device for automatically marking sample features and defects. The implementation method is mainly divided into three stages. The first stage is manual marking stage, and the second stage is semi-automatic marking. Manual assistance stage, the third stage is the automatic marking stage.

Please refer to the second figure again. The first stage of the implementation method of the present invention is the manual marking stage. The process includes sample acquisition, which means that the camera unit 3 is combined with the light source device 2 to acquire the image of the object to be tested; artificial marking of features or defects, Refers to manually frame selection and marking of the features or defects of the sample in the captured image; manual marking classification refers to the manual classification and recording of the captured sample image and marking results; training refers to The acquired sample image and the data of the marking result are transferred to the feature and defect intelligent automatic marking model 6, and the neural network based on the object detection model is used for training in the way of deep learning. Because manually marking features or defects and categorizing the marking results takes more manpower and time, the number of data sets generated at this stage is small; exporting model training results refers to exporting model training results to database unit 7. For later use by the smart identifier 8 or the feature and defect smart automatic marking model 6 or other applications.

Please refer to the third figure again. The second stage of the implementation method of the present invention is the semi-automatic labeling manual assistance stage. There are three processes in this stage that can be implemented accordingly. The first process includes sample acquisition, which refers to the combination of the camera unit 3 and the light source. Device 2 takes an image of the object to be measured; manual marking of features or defects refers to manually frame selection and marking of the features or defects of the sample in the taken picture; automatic marking classification refers to the image of the sample taken And the marking results are automatically classified and recorded by the smart recognizer 8; manual inspection means to prevent the smart recognizer 8 from automatically selecting and marking the sample features or defects in the captured image, or classifying the marking results At the time, due to the insufficient number of data sets used as the basis for the discrimination, the marking or classification results have missed or overkilled misjudgments, so the classification results are manually checked, and if misjudgments occur, the classification results will be corrected; Training refers to the transfer of the acquired sample image and the data of the marking result to the feature and defect intelligent automatic marking model 6, using the neural network based on the object detection model to perform in-depth learning training. Because the smart recognizer 8 automatically classifies the labeling results, it can save human resources and have faster classification efficiency, so the generated data set is also greatly increased compared with the previous manual labeling stage; exporting the model training results refers to the model training results Export to the database unit 7 for later use by the smart identifier 8 or the feature and defect smart automatic marking model 6 or other applications.

Please refer to the third figure again. The second stage of the implementation method of the present invention is a semi-automatic labeling manual assistance stage. The second process can be implemented simultaneously with the first process. The second process includes sample acquisition, which refers to the combination of the camera unit 3 The light source device 2 takes an image of the object to be measured; automatic marking of features or defects means that the intelligent recognizer 8 automatically frames and marks the sample features or defects in the captured image by virtue of the model training result data transmitted by the database unit 7 ; Manual marking classification refers to manually classifying and recording the captured sample images and marking results; manual inspection refers to preventing automatic frame selection of sample features or defects in the captured images by the intelligent recognizer 8 and When marking or categorizing the marking results, because the number of data sets used as the basis for discrimination is not large enough, the classification results have missed or overkilled misjudgments, so the classification results are manually checked. If there is a misjudgment situation, correct The classification results have been revised, but the manual marking and classification procedures of the process have manually classified the marking results of the samples in the captured images, which already contains the meaning of manual inspection, so the manual inspection procedures in this process are not necessary Implementation steps; training is to transfer the acquired sample image and the data of the marking results to the feature and defect intelligent automatic marking model 6, using the neural network based on the object detection model to use depth Training in the way of learning. Because the smart recognizer 8 automatically frames and marks the sample features or defects in the captured image, it can save human resources and have a faster classification efficiency, so the generated data set is also greatly increased compared with the previous manual marking stage; The model training result refers to exporting the model training result to the database unit 7 for later use by the intelligent recognizer 8 or the feature and defect intelligent automatic marking model 6 or other application terminals.

Please refer to the third figure again. The second stage of the implementation method of the present invention is the semi-automatic labeling manual assistance stage. The third process is implemented after the first and second processes, including sample acquisition, which refers to the combination of the camera unit 3 and the light source. The device 2 takes an image of the object to be measured; automatic marking of features or defects means that the model training result data transmitted by the database unit 7 from the intelligent identifier 8 automatically frames and marks the features or defects of the sample in the image taken; Automatic marking result classification refers to the automatic classification and recording of the results of the model training result data transmitted by the database unit 7 from the database unit 7 of the captured sample image and the marking result using the intelligent identifier 8, and the result of frame selection and marking of features or defects; Manual inspection refers to the prevention of automatic frame selection and marking of sample features or defects in the image taken by the intelligent recognizer 8, or automatic classification of the marking results, because the data set used as the basis for discrimination is insufficient due to insufficient accumulated data In the judgment, the classification result is misjudged by missing or overkill. Therefore, the classification result is manually checked. If a misjudgment situation occurs, the classification result will be corrected; training means taking the sample image And the data of the marking result is transferred to the feature and defect intelligent automatic marking model 6, and the neural network based on the object detection model is used for training in the way of deep learning. Because the smart recognizer 8 automatically frames and marks the sample features or defects in the captured image and classifies the marking results, it can save human resources and have a faster classification efficiency, so the generated data set is also compared with the previous manual marking stage Large increase; Exporting model training results refers to exporting the model training results to the database unit 7 for later use by the intelligent recognizer 8 or the feature and defect intelligent automatic marking model 6 or other application terminals.

When the second stage, the semi-automatic marking manual assistance stage, has accumulated a large enough amount of generated data sets, so that the smart recognizer has enough data sets as the basis for discrimination, and the classification results have no more misjudgments after repeated manual inspection procedures. , You no longer need manual assistance, and enter the third stage, which is the automatic marking stage.

Please refer to the fourth figure again, the third stage of the implementation method of the present invention is the automatic marking stage Section, this process includes sample acquisition, which means taking the camera unit 3 combined with the light source device 2 to take an image of the object to be measured; automatic marking of features or defects means the model training result data transmitted by the smart identifier 8 using the database unit 7 Automatically frame and mark the sample features or defects in the captured image; automatic marking result classification refers to the model training result data transmitted by the intelligent identifier 8 and the database unit 7 of the captured sample image and the marking result , The marking result is automatically classified and recorded. Because the smart recognizer 8 automatically frames and marks the sample features or defects in the captured image, and classifies the marking results, and the data accumulated in the database unit 7 at this stage is sufficient, there is no need to manually check the smart recognition If there is any misjudgment of the device 8 occurs, it can save more human resources and have a faster classification efficiency; exporting results refers to the automatic marking and classification of the sample image features or defects of the captured image to the database Unit 7.

Please refer to the fifth and sixth figures with the remaining figures; the present invention is an intelligent optical inspection method and device for automatically marking sample features and defects, which is based on the concept of this method to provide:

Load the collection platform 1 for the samples to be tested;

The light source device 2, which is arranged on the upper periphery of the loading and collection platform 1, is equipped with a coaxial light source 20, a line light source 21, a backlight source, a ring light source, a spherical light source and other sets of light source fixtures, combined with a movable mechanism 5 For example, the robotic arm and light source photography control module 4 can provide different light source control methods to switch each group of light sources, and edit the intensity and angle of the light source;

Camera unit 3, the camera unit 3 and the light source device 2 are interlaced on the upper periphery of the loading and collecting platform 1, including camera fixtures, line camera fixtures and other application camera fixtures combined with the camera's repeat shooting function and achieve rapid The function of collecting data;

The smart identifier 8 is connected to the database unit 7. The model training result data transmitted by the database unit 7 can automatically frame and mark the features and defects in the sample image, and classify the marking results;

The light source photography control module 4 is connected to the light source device 2 and the camera unit 3 and the movable mechanism 5 to control the light source device 2 and the camera unit 3 to perform smart lighting photography processes;

Movable mechanism 5, the movable mechanism 5 may be a lifting mechanism conveying mechanism or a robotic arm, etc.;

Feature and defect intelligent automatic marking model 6, linked to the intelligent identifier 8 and database unit 7, can use the object detection model (object detection)-based neural network to use the acquired sample image and the data of the marking result Road, train by means of deep learning, and export the model training results to database unit 7;

The database unit 7 is connected to the feature and defect intelligent automatic marking model 6 and the intelligent identifier 8 respectively. The sample image frame selection and marking results and classification data of the sample can be stored in the database unit 7, or Pass the data to the feature and defect intelligent automatic labeling model 6 for deep learning training, and import the model training results to the database unit 7 for storage, or pass the model training result data to the intelligent recognizer 8 to make it available To automatically frame and mark the features and flaws in the sample acquisition picture or classify the marking results.

Therefore, the present invention is an intelligent optical inspection method and device for automatically marking sample features and defects. The implementation method is mainly divided into three stages. In the first stage, the features and defects to be marked in the test image are marked by manual marking and the marking results are classified. The classification results are generated by the feature and defect intelligent automatic labeling model 6 by deep learning to generate a small number of data sets and merge Export to database unit 7. After that, the second stage uses a semi-automatic marking method, where the smart recognizer 8 marks the features and defects to be marked in the test picture or automatically classifies the marking results, but some of the procedures still need to be manually assisted in three possible implementations After that, the classification results are generated by the feature and defect intelligent automatic labeling model 6 in the way of deep learning to generate a large number of data sets. When no misjudgment by the smart recognizer 8 is found in the process of multiple manual inspections at this stage, the third stage can be entered and the smart recognizer 8 will automatically mark the desired features and defects in the image to be tested and mark the result. Classification, and finally export and store the classification results in database unit 7. No more manual processing procedures can save more human resources and have faster identification and classification efficiency. The application of the present invention to the article feature and defect detection process has the advantages of automation, intelligence and data at the same time, and thus becomes an effective creator of the present invention.

1: Loading the collection platform

2: Light source device

20: Coaxial light source

21: Line light source

3: camera unit

4: Light source photography control module

5: Movable mechanism

6: Intelligent automatic marking model of features and defects

7: Database unit

8: Smart recognizer

The first figure is a three-stage flow chart of the present invention.

The second figure is a flowchart of the first stage of the present invention.

The third figure is a flowchart of the second stage of the present invention.

The fourth figure is a flowchart of the third stage of the present invention.

The fifth figure is a side view of the structure of the device of the present invention.

The sixth figure is a block diagram of the present invention.

Claims (7)

A method and device for automatically marking sample features and defects for intelligent optical inspection. The implementation method is mainly divided into three stages according to the sequence. The first stage is the manual marking stage, the second stage is the semi-automatic marking artificial assist stage, and the third stage is Automatically mark the stage. As described in item 1 of the scope of patent application, a method and device for automatic marking of sample features and defects for intelligent optical inspection. The first-stage implementation method includes sample acquisition, which refers to the use of a camera unit and a light source device for the object to be tested Take an image; manual marking of features or defects refers to manually selecting and marking the features or defects of the sample in the image taken; manual marking classification refers to manually taking the sample image and marking results Method to classify and record; training means to transfer the acquired sample image and marking result data to the feature and defect intelligent automatic marking model, using the neural network based on the object detection model to Deep learning is used for training; exporting model training results refers to exporting model training results to the database unit. As described in item 1 of the scope of patent application, a method and device for automatic marking of sample features and defects for intelligent optical inspection, the second-stage implementation method has three processes that can be implemented accordingly. The first process includes sample acquisition. Refers to the use of a camera unit combined with a light source device to capture images of the object to be tested; manual marking of features or defects refers to manually selecting and marking the features or defects of the sample in the captured image; automatic marking classification refers to The captured sample images and marking results are automatically classified and recorded by the smart recognizer using the model training result data transmitted by the database unit; manual inspection means to prevent the smart recognizer from automatically checking the characteristics of the sample in the captured image or When the defects are framed and marked, or when the marking results are classified, the marking or classification results have missed or overkilled misjudgments, so the classification results are manually checked, and if misjudgments occur, the classification results will be checked. Correction; training means to transfer the data of the captured sample image and the marking result to the feature and defect intelligent automatic marking model, using the neural network based on the object detection model, and deep learning Method for training; exporting model training results refers to exporting model training results to the database unit. As described in the first item of the scope of patent application, a method and device for automatically marking sample features and defects for intelligent optical inspection. The second process of the second-stage implementation method includes sample acquisition, which refers to the combination of a camera unit and a light source device Take an image of the object to be tested; automatic marking of features or defects means that the intelligent recognizer uses the model training result data transmitted by the database unit to automatically frame and mark the sample features or defects in the captured image; manual marking and classification, Refers to manually classifying and recording the captured sample images and marking results; manual inspection refers to preventing automatic frame selection and marking or marking of sample features or defects in the captured image by the intelligent recognizer 8 When the results are classified, there are false judgments of missed or overkill. Therefore, the classification results are manually checked. If a false judgment occurs, the classification results will be corrected; training means taking the sample image and The data of the marking result is transferred to the feature and defect intelligent automatic marking model, and the neural network based on the object detection model is used for training in the way of deep learning; exporting the model training result means training the model The result is exported to the database unit. As described in the first item of the scope of patent application, a method and device for automatic marking of sample features and defects for intelligent optical inspection and its device. The third process of the second-stage implementation method includes sample acquisition, which refers to the combination of a camera unit and a light source device Take an image of the object to be tested; automatic marking of features or defects means that the intelligent recognizer uses the model training result data transmitted by the database unit to automatically frame and mark the sample features or defects in the image taken; automatic marking classification is Refers to the automatic classification and recording of the marked results by the smart recognizer using the model training result data transmitted by the database unit; manual inspection refers to prevent the smart recognizer from automatically checking the samples in the captured image The features or defects are framed and marked, or there is a misjudgment of missed or overkill in the automatic classification of the marking results, so the classification results are manually checked, and the classification results should be corrected if any misjudgments occur; Training refers to the transfer of the acquired sample image and the data of the marking results to the feature and defect intelligent automatic marking model, and the use of the object detection model (object detection)-based neural network for training by means of deep learning ; Exporting model training results refers to exporting model training results to the database unit. As described in the first item of the scope of patent application, a method and device for automatically marking sample features and defects for intelligent optical inspection. The third-stage implementation method includes sample acquisition, which refers to the use of a camera unit and a light source device to perform the test Acquisition; automatic marking of features or defects means that the smart recognizer uses the model training result data to automatically frame and mark the sample features or defects in the acquired image; automatic marking classification means the sample image and The marking result is automatically classified and recorded by the smart recognizer using the model training result data; the export result refers to the result of automatic marking and classification of the captured sample image features or defects to the database unit. For example, the method and device for automatic marking of sample features and defects for intelligent optical inspection described in item 4 of the scope of patent application, manual inspection procedures may not be implemented.

Images