CN113838015B - Electrical product appearance defect detection method based on network cooperation - Google Patents

Abstract

The invention relates to a method for detecting appearance defects of electrical appliances based on network collaboration. The present invention utilizes human-machine collaboration technology to collect detection samples in the appearance detection of electrical products and superimposes a deep learning mechanism to dynamically optimize the electrical appearance detection model, and finally forms an expert database of electrical appearance detection models. At present, the traditional appearance inspection system in the electrical appliance industry has high requirements and requires professionally trained personnel to operate. This system uses automatic learning of inspection data to implement an intelligent quality management analysis system, achieve inspection standardization, and reduce the impact of human factors on inspection results. , improve the quality management level, while reducing personnel deployment and saving production costs.

Description

A method for detecting appearance defects of electrical appliances based on network collaboration

Technical field

The invention relates to a method for detecting appearance defects of electrical appliances based on network collaboration, and belongs to the technical fields of artificial intelligence and industrial control automation.

Background technique

With the in-depth development and advancement of intelligent manufacturing, industrial Internet, and artificial intelligence in our country, the application of various types of electrical equipment is increasing day by day, and the corresponding product quality and use safety issues cannot be ignored. At present, the intelligence of inspection methods and tools in the manufacturing process of user-end electrical equipment is still in its infancy in my country and cannot meet the needs of the rapid development of the industry.

Currently, product appearance defect detection systems and devices based on deep learning generally include image display equipment, servers, controllers, etc. Most detection systems with servers include a deep learning module on the server side, and iterative calculations are performed on the collected image data to obtain an updated detection model. However, most of the defect types of the detection model are fixed and do not have the extension to new categories. At the same time, for the model being inspected and the product inspection results it detects, users on the edge cannot inform the machine learning module of new defect types and product characteristics, and thus cannot form a new inspection mechanism and update products for this new situation. Detection expert database

The invention patent application with application number 202010107000.7 discloses an intelligent control system for the assembly production line based on an expert database, including an expert database, a host computer, a robot, a screw machine, a detection unit, a drive unit, a visual recognition unit, a nail feeding machine, and intelligent assembly. Unit etc. This patent application does not use deep machine learning to iteratively optimize the detection model. At the same time, the generated expert library cannot be updated in a similar distributed system from edge to remote, which is not conducive to the formation of new detection mechanisms.

The invention patent application with application number 202010888429.4 discloses a deep learning device and a deep learning application method, including: a model library, an operating device, and an execution device. Users select and operate page visualization components according to application requirements, and call the corresponding deep learning model to process the input data, thereby achieving the required deep learning tasks on the input data. The method disclosed in this patent application does not reflect the iterative training of the model, which limits the breadth of deep learning and the accuracy of the output model.

The invention patent application with application number 202010829978.4 discloses an intelligent steel strip visual inspection equipment. This patent application is more about using a model that has been trained and implemented for inspection. There is no step based on network collaboration and machine learning, and it is traditional. Visual recognition detection solutions.

Contents of the invention

The purpose of this invention is to introduce a human-machine collaboration mechanism into a product appearance defect detection method based on deep learning.

In order to achieve the above objectives, the technical solution of the present invention is to provide a network-based collaboration-based appearance defect detection method for electrical products, which is characterized by including the following steps:

Step 1. Establish an expert model library for appearance detection in the field of electrical appliances in the remote training server. The remote training server uses the training data sets corresponding to different electrical equipment models uploaded by the edge-side human-computer interaction device to train the machine deep learning algorithm. Multiple appearance detection deep learning models corresponding to different electrical equipment models are formed, and all appearance detection deep learning models are stored in the appearance detection expert model library in the field of electrical appliances, and each appearance detection deep learning model is associated with the corresponding electrical equipment. The model establishes a mapping relationship;

When the edge-side production equipment needs to produce electrical products of new electrical equipment models, the remote training server trains a new appearance detection deep learning model and stores it in the appearance detection expert model library in the electrical field. Then the remote training server will add the new appearance detection model. The model information of the appearance detection deep learning model notifies the edge-side human-computer interaction device, so that the user can select the newly added appearance detection deep learning model in the appearance detection expert model library in the electrical appliance field based on the electrical equipment model through the edge-side human-computer interaction device;

When the appearance detection deep learning model detects appearance defects of electrical products, it first performs a standardization operation on the received image data of any size, and unifies the received image data into the input size of the CNN convolutional neural network to obtain standardized image data; then Input the standardized image data into the CNN convolutional neural network, and perform convolution operations on the standardized image data with convolution kernels of different dimensions and types to generate small target feature maps, medium target feature maps, and large target feature maps respectively. Among them, small target feature maps The sampling receptive field of the image is smaller than that of the medium target feature map, and the sampling receptive field of the medium target feature map is smaller than the sampling receptive field of the large target feature map; the appearance detection deep learning model detects small target defects based on the small target feature map, and detects small target defects based on the medium target feature map. The target feature map detects medium target defects and detects large target defects based on the large target feature map. During detection, the appearance detection deep learning model uses the box regression algorithm and the multi-classification algorithm to detect small target feature maps, medium target feature maps and large target feature maps. The corresponding vector group performs frame regression prediction of the target detection object and classification of the target object under the frame. If it is determined that there is a defect, it is obtained to mark the location of the small target defect and/or the location of the medium target defect and/or the large target. The small target defect border and/or the medium target defect border and/or the large target defect border and the defect category at the location of the defect are obtained by obtaining the output parameters x, y, w, h and confidence, where x and y represent small target defects. The X-axis offset and Y-axis offset of the center position of the border or the middle target defect border or the large target defect border relative to the upper left corner of the current defect. w and h represent the small target defect border or the medium target defect border or the large target defect border. The width and height of the target defect border are proportional to the width and height of the entire image respectively, and the confidence value is not greater than 1;

Step 2: Based on the electrical equipment model of the electrical product actually produced by the edge-side production equipment, the user uses the edge-side human-computer interaction device to generate model calling information related to the electrical equipment model; the human-computer interaction device uploads the model calling information to the remote After the remote training server is installed, the remote training server calls the appearance detection deep learning model corresponding to the current electrical equipment model stored in the appearance detection expert model library in the electrical field based on the model calling information;

Step 3. After the edge-side production equipment obtains the real-time appearance picture of the current electrical product through the image acquisition device, it uploads the real-time appearance picture to the remote training server. The remote training server inputs the received real-time appearance picture into the called appearance detection The deep learning model uses the appearance detection deep learning model to judge whether the current electrical product has defects using real-time appearance pictures. If it is determined that the current electrical product has defects, it will output the predicted small target defect border and/or medium target defect border and/ Or the large target defect border and defect category, output the corresponding output parameters x, y, w, h, confidence and defect category;

Step 4. The edge-side human-computer interaction device uses the received output parameters x, y, w, h to frame the corresponding defect area on the real-time appearance image, and displays the corresponding confidence level and defect category;

Step 5. If the confidence level displayed in step 4 is lower than the preset threshold, or a new defect category appears based on the real-time appearance picture, use the edge-side human-computer interaction device to draw the defect border on the real-time appearance picture and Enter the corresponding defect category, and the edge-side human-computer interaction device uses the drawn defect border to obtain the corresponding defect border parameters x, y, w, h. At the same time, the edge-side human-computer interaction device sets the confidence level to 1; the edge-side human-computer interaction device The interactive device saves the previously obtained defect border parameters x, y, w, h as well as the corresponding defect category, confidence value and real-time appearance image as a new piece of training data;

If the confidence level displayed in step 4 is not lower than the preset threshold and no new defect category appears, the edge-side human-computer interaction device will obtain the defect frame parameters x, y, w, h and the corresponding defects obtained in step 4. Categories, confidence values, and real-time appearance images are saved as a new piece of training data;

Step 6: The edge-side human-computer interaction device uploads all new training data collected at each cycle step to the training server according to the set cycle. The training server uses all new training data collected at each cycle step to form a Create a new training data set, and retrain the appearance detection deep learning model used in steps 2 to 5 based on this training data set to obtain an updated and optimized appearance detection deep learning model, and replace the existing appearance detection deep learning model. The model is stored in the model library of appearance inspection experts in the field of electrical appliances.

Preferably, in step 1, the training data set of the current electrical equipment model is obtained using the following method:

The edge-side human-computer interaction device obtains the appearance picture of the electrical product of the current electrical equipment model through the image acquisition device, and then determines whether the appearance picture of the electrical product has appearance defects, and manually annotates the appearance pictures of the electrical product with appearance defects; when performing manual annotation , based on the standards for appearance inspection in the field of electrical appliances, the sampled pictures are annotated, and the defect borders and defect categories are marked on the appearance pictures of the electrical appliances; the edge-side human-computer interaction device will mark the appearance pictures of the electrical appliances that have been manually labeled and do not require manual The annotated appearance pictures of electrical appliances are uploaded to the remote training server. The remote training server constructs a training data set based on the appearance pictures of electrical appliances received within a certain period of time. The training data set is used to train the machine deep learning algorithm to obtain Appearance detection deep learning model corresponding to current electrical equipment models.

This invention studies the combination of human-machine collaboration and deep learning in visual inspection technology to jointly form an expert database of product inspection models and improve inspection quality. The present invention collects on-site data through visual detection technology and human-computer interaction technology, and realizes deep learning of traditional electrical appliance appearance detection models on the server side through cloud platform technology. Finally, the present invention stores these optimized models based on a large number of electrical appliance appearance sample data on the edge side into a proprietary database to form an expert model library for electrical appliance appearance detection. In practical applications, these models stored in the expert library form a high-level model in the electrical appliance industry. Intelligent testing standards.

The present invention utilizes human-computer collaboration technology to collect detection samples in the appearance detection of electrical products and superimposes a deep learning mechanism to dynamically optimize the electrical appearance detection model, and finally forms an expert database of electrical appearance detection models. At present, the traditional appearance inspection system in the electrical appliance industry has high requirements and requires professionally trained personnel to operate. This system uses automatic learning of inspection data to implement an intelligent quality management analysis system, achieve inspection standardization, and reduce the impact of human factors on inspection results. , improve the quality management level, while reducing personnel deployment and saving production costs.

Description of the drawings

Figure 1 is a schematic diagram of the product appearance defect detection method based on network collaboration;

Figure 2 is a schematic diagram of the application of the product appearance defect detection method based on network collaboration;

Figure 3 shows the model storage flow chart after the deep machine learning algorithm training is completed;

Figure 4 is a flow chart of the human-computer collaborative detection process on the human-computer interaction interface;

Figure 5 is a visual WEB interface control deep machine learning service flow chart.

Detailed ways

The present invention will be further described below in conjunction with specific embodiments. It should be understood that these examples are only used to illustrate the invention and are not intended to limit the scope of the invention. In addition, it should be understood that after reading the teachings of the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms also fall within the scope defined by the appended claims of this application.

The invention provides a method for detecting appearance defects of electrical products based on network collaboration, which includes the following steps:

Step 1. Establish an expert model library for appearance detection in the field of electrical appliances in the remote training server. The remote training server uses the training data sets corresponding to different electrical equipment models uploaded by the edge-side human-computer interaction device to train the machine deep learning algorithm. Multiple appearance detection deep learning models corresponding to different electrical equipment models are formed, and all appearance detection deep learning models are stored in the appearance detection expert model library in the field of electrical appliances, and each appearance detection deep learning model is associated with the corresponding electrical equipment. The model establishes a mapping relationship.

In this step, the training data set of the current electrical equipment model is obtained using the following method:

The edge-side human-computer interaction device obtains the appearance picture of the electrical product of the current electrical equipment model through the image acquisition device, and then determines whether the appearance picture of the electrical product has appearance defects, and manually annotates the appearance pictures of the electrical product with appearance defects. When performing manual annotation, the sampled images are annotated based on the standards for appearance inspection in the electrical appliance industry, and the defect borders and defect categories are marked on the appearance images of the electrical product. The edge-side human-computer interaction device uploads the appearance images of electrical products that have been manually annotated and the appearance images of electrical products that do not require manual annotation to the remote training server. The remote training server is based on the appearance images of electrical products received within a certain period of time. Construct a training data set and use the training data set to train the machine deep learning algorithm to obtain an appearance detection deep learning model corresponding to the current electrical equipment model.

When the edge-side production equipment needs to produce electrical products of new electrical equipment models, the remote training server trains based on the above method to obtain a new appearance detection deep learning model and stores it in the appearance detection expert model library in the field of electrical appliances. Then the remote training server will The model information of the new appearance detection deep learning model is notified to the edge-side human-computer interaction device, so that users can choose the newly added appearance detection deep learning in the appearance detection expert model library in the field of electrical appliances through the edge-side human-computer interaction device based on the electrical equipment model. Model.

In this embodiment, when the appearance detection deep learning model detects appearance defects of electrical products, it first performs standardization operations on the received image data of any size, and unifies the received image data into a CNN through scaling and pad (extended pixel value 0) The input size of the convolutional neural network to obtain normalized image data. Then the standardized image data is input into the CNN convolutional neural network, and convolution operations are performed on the standardized image data with convolution kernels of different dimensions and types to generate small target feature maps with a receptive field of 13×13pixels with 32 times downsampling and 16 times downsampling respectively. The sampling receptive field is a medium target feature map of 26×26pixels, and the 8x downsampling receptive field is a large target feature map of 52×52pixels. Detect targets of different sizes under multi-size conditions through feature maps of three different sizes, such as: detecting small target defects, such as scratches, blemishes, and peeling paint, based on small target feature maps; detecting medium target defects based on medium target feature maps, For example, nameplate defects and knob defects on the surface of electrical appliances; detect large target defects based on large target feature maps, such as abnormal wiring terminals. Finally, the appearance detection deep learning model uses the box regression algorithm (BBoxReg) and the multi-classification algorithm (SVMs) to perform target detection on the vector groups corresponding to the small target feature map, the medium target feature map and the large target feature map at three scales. Frame regression prediction and classification of target objects under the frame. If it is determined that there is a defect, then the small target defect border and/or the location of the large target defect are obtained to respectively mark the location of the small target defect and/or the location of the medium target defect. /or medium target defect border and/or large target defect border and defect category, obtain the output parameters x, y, w, h and confidence, where x, y represent small target defect border or medium target defect border or large target The X-axis offset and Y-axis offset of the center position of the defect border relative to the upper left corner of the current defect. w and h represent the width and height of the small target defect border, medium target defect border, or large target defect border respectively. The ratio of the width and height of the entire image, and the confidence value is not greater than 1. The appearance detection deep learning model normalizes the input image, and the identified parameters x, y, w, h are output according to the center point coordinates x, y, w, h of the normalized defect border. The output is the proportion of x, y, w, h occupying the pixels of the image. Finally, when the terminal device displays the defect frame, the restoration operation needs to be performed accordingly.

Step 2: The user uses the edge-side human-computer interaction device to generate model calling information related to the electrical equipment model based on the electrical equipment model of the electrical product actually produced by the edge-side production equipment. After the human-computer interaction device uploads the model calling information to the remote training server, the remote training server calls the appearance detection deep learning corresponding to the current electrical equipment model stored in the appearance detection expert model library in the electrical field based on the model calling information. Model.

Step 3. After the edge-side production equipment obtains the real-time appearance picture of the current electrical product through the image acquisition device, it uploads the real-time appearance picture to the remote training server. The remote training server inputs the received real-time appearance picture into the called appearance detection The deep learning model uses the appearance detection deep learning model to judge whether the current electrical product has defects using real-time appearance pictures. If it is determined that the current electrical product has defects, it will output the predicted small target defect border and/or medium target defect border and/ Or the large target defect border and defect category, and output the corresponding output parameters x, y, w, h, confidence and defect category.

Step 4. The edge-side human-computer interaction device uses the received output parameters x, y, w, h to frame the corresponding defect area on the real-time appearance image, and displays the corresponding confidence level and defect category.

Step 5. If the confidence level displayed in step 4 is lower than the preset threshold, or a new defect category appears based on the real-time appearance picture, use the edge-side human-computer interaction device to draw the defect border on the real-time appearance picture and Input the corresponding defect category, and the edge-side human-computer interaction device uses the drawn defect border to obtain the corresponding defect border parameters x, y, w, and h. At the same time, the edge-side human-computer interaction device sets the confidence level to 1. The edge-side human-computer interaction device saves the previously obtained defect frame parameters x, y, w, h as well as the corresponding defect category, confidence value and real-time appearance image as a new piece of training data.

If the confidence level displayed in step 4 is not lower than the preset threshold and no new defect category appears, the edge-side human-computer interaction device will obtain the defect frame parameters x, y, w, h and the corresponding defects obtained in step 4. Categories, confidence values, and real-time appearance images are saved as a new piece of training data.

Step 6: The edge-side human-computer interaction device uploads all new training data collected at each cycle step to the training server according to the set cycle. The training server uses all new training data collected at each cycle step to form a Create a new training data set, and retrain the appearance detection deep learning model used in steps 2 to 5 based on this training data set to obtain an updated and optimized appearance detection deep learning model, and replace the existing appearance detection deep learning model. The model is stored in the model library of appearance inspection experts in the field of electrical appliances.

In this embodiment, the built-in hardware standard of the human-computer interaction device needs to be equipped with Intel Celeron J1800 processor as the main chip, built-in 2G DDR3 memory, SSD hard drive, pre-installed Ubuntu16.04 and above operating system, and equipped with touch function display. As shown in Figure 2, the human-computer interaction device needs to have a configuration function interface to complete the necessary configuration options, and a display interface to display the detected images, upload data sets, and update the detection model.

The built-in hardware standard of the training server must be equipped with Intel Xeon 5128 processor as the main chip, built-in 32GDDR3 memory, SSD hard disk, must be configured with a GPU unit, and pre-installed Ubuntu16.04 and above operating systems. In order to achieve model optimization, the operating system must also be pre-installed with relational databases, deep machine learning frameworks, and web server frameworks. training server

This embodiment uses a visual WEB operation interface to operate the training server. The operation process is as follows:

1. The visual operation interface is a visual operation interface provided by the WEB service built into the server that can be deployed in the cloud.

2. Connect to the human-computer interaction device through the visual operation interface to view the parameters of the human-computer interaction equipment, connect to the server through the visual operation interface to specify the training parameters, start the electrical appliance appearance detection model training, and download the electrical appliance appearance detection model model library information to the user Computer interaction equipment.

In this system, the Web interface is developed based on the Flask framework and is responsible for communicating with human-computer interaction devices and servers. As shown in Figure 1, the Web interface needs to be connected to the server to be able to issue training instructions from the visual interface to the training server to start training.

As shown in Figure 1, the Web interface needs to be connected to the human-computer interaction device, so that users can query the parameters of the detected product and change the settings of the human-computer interaction device in the visual interface, thereby configuring and monitoring the entire system in the cloud to facilitate management of system operation. state.

Claims (2)

1. A method for detecting appearance defects of electrical appliances based on network collaboration, which is characterized by including the following steps: Step 1. Establish an expert model library for appearance detection in the field of electrical appliances in the remote training server. The remote training server uses the training data sets corresponding to different electrical equipment models uploaded by the edge-side human-computer interaction device to train the machine deep learning algorithm. Multiple appearance detection deep learning models corresponding to different electrical equipment models are formed, and all appearance detection deep learning models are stored in the appearance detection expert model library in the field of electrical appliances, and each appearance detection deep learning model is associated with the corresponding electrical equipment. The model establishes a mapping relationship; When the edge-side production equipment needs to produce electrical products of new electrical equipment models, the remote training server trains a new appearance detection deep learning model and stores it in the appearance detection expert model library in the electrical field. Then the remote training server will add the new appearance detection model. The model information of the appearance detection deep learning model notifies the edge-side human-computer interaction device, so that the user can select the newly added appearance detection deep learning model in the appearance detection expert model library in the electrical appliance field based on the electrical equipment model through the edge-side human-computer interaction device; When the appearance detection deep learning model detects appearance defects of electrical products, it first performs a standardization operation on the received image data of any size, and unifies the received image data into the input size of the CNN convolutional neural network to obtain standardized image data; then Input the standardized image data into the CNN convolutional neural network, and perform convolution operations on the standardized image data with convolution kernels of different dimensions and types to generate small target feature maps, medium target feature maps, and large target feature maps respectively. Among them, small target feature maps The sampling receptive field of the image is smaller than that of the medium target feature map, and the sampling receptive field of the medium target feature map is smaller than the sampling receptive field of the large target feature map; the appearance detection deep learning model detects small target defects based on the small target feature map, and detects small target defects based on the medium target feature map. The target feature map detects medium target defects and detects large target defects based on the large target feature map. During detection, the appearance detection deep learning model uses the box regression algorithm and the multi-classification algorithm to detect small target feature maps, medium target feature maps and large target feature maps. The corresponding vector group performs frame regression prediction of the target detection object and classification of the target object under the frame. If it is determined that there is a defect, it is obtained to mark the location of the small target defect and/or the location of the medium target defect and/or the large target. The small target defect border and/or the medium target defect border and/or the large target defect border and the defect category at the location of the defect are obtained by obtaining the output parameters x, y, w, h and confidence, where x and y represent small target defects. The X-axis offset and Y-axis offset of the center position of the border or the middle target defect border or the large target defect border relative to the upper left corner of the current defect. w and h represent the small target defect border or the medium target defect border or the large target defect border. The width and height of the target defect border are proportional to the width and height of the entire image respectively, and the confidence value is not greater than 1; Step 2: Based on the electrical equipment model of the electrical product actually produced by the edge-side production equipment, the user uses the edge-side human-computer interaction device to generate model calling information related to the electrical equipment model; the human-computer interaction device uploads the model calling information to the remote After the remote training server is installed, the remote training server calls the appearance detection deep learning model corresponding to the current electrical equipment model stored in the appearance detection expert model library in the electrical field based on the model calling information; Step 3. After the edge-side production equipment obtains the real-time appearance picture of the current electrical product through the image acquisition device, it uploads the real-time appearance picture to the remote training server. The remote training server inputs the received real-time appearance picture into the called appearance detection The deep learning model uses the appearance detection deep learning model to judge whether the current electrical product has defects using real-time appearance pictures. If it is determined that the current electrical product has defects, it will output the predicted small target defect border and/or medium target defect border and/ Or the large target defect border and defect category, output the corresponding output parameters x, y, w, h, confidence and defect category; Step 4. The edge-side human-computer interaction device uses the received output parameters x, y, w, h to frame the corresponding defect area on the real-time appearance image, and displays the corresponding confidence level and defect category; Step 5. If the confidence level displayed in step 4 is lower than the preset threshold, or a new defect category appears based on the real-time appearance picture, use the edge-side human-computer interaction device to draw the defect border on the real-time appearance picture and Enter the corresponding defect category, and the edge-side human-computer interaction device uses the drawn defect border to obtain the corresponding defect border parameters x, y, w, h. At the same time, the edge-side human-computer interaction device sets the confidence level to 1; the edge-side human-computer interaction device The interactive device saves the previously obtained defect border parameters x, y, w, h as well as the corresponding defect category, confidence value and real-time appearance image as a new piece of training data; If the confidence level displayed in step 4 is not lower than the preset threshold and no new defect category appears, the edge-side human-computer interaction device will obtain the defect frame parameters x, y, w, h and the corresponding defects obtained in step 4. Categories, confidence values, and real-time appearance images are saved as a new piece of training data; Step 6: The edge-side human-computer interaction device uploads all new training data collected at each cycle step to the training server according to the set cycle. The training server uses all new training data collected at each cycle step to form a Create a new training data set, and retrain the appearance detection deep learning model used in steps 2 to 5 based on this training data set to obtain an updated and optimized appearance detection deep learning model, and replace the existing appearance detection deep learning model. The model is stored in the model library of appearance inspection experts in the field of electrical appliances. 2. A method for detecting appearance defects of electrical products based on network collaboration as claimed in claim 1, characterized in that, in step 1, the training data set of the current electrical equipment model is obtained by the following method: The edge-side human-computer interaction device obtains the appearance picture of the electrical product of the current electrical equipment model through the image acquisition device, and then determines whether the appearance picture of the electrical product has appearance defects, and manually annotates the appearance pictures of the electrical product with appearance defects; when performing manual annotation , based on the standards for appearance inspection in the field of electrical appliances, the sampled pictures are annotated, and the defect borders and defect categories are marked on the appearance pictures of the electrical appliances; the edge-side human-computer interaction device will mark the appearance pictures of the electrical appliances that have been manually labeled and do not require manual The annotated appearance pictures of electrical appliances are uploaded to the remote training server. The remote training server constructs a training data set based on the appearance pictures of electrical appliances received within a certain period of time. The training data set is used to train the machine deep learning algorithm to obtain Appearance detection deep learning model corresponding to current electrical equipment models.