CN112497219B - Columnar workpiece classifying and positioning method based on target detection and machine vision - Google Patents

Abstract

The invention discloses a columnar workpiece classification high-precision positioning method based on target detection and machine vision, which comprises two parts of target detection, defect detection, coarse positioning and high-precision positioning of the machine vision of yolov 3. The yolov3 part comprises the steps of making a data set, improving a network structure, adjusting candidate frame parameters, carrying out real-time positioning identification and defect detection. And acquiring a workpiece image by an eye-to-hand camera, fusing an image enhancement algorithm, and improving the candidate frame parameters by adopting a vector similarity measurement method. And a machine vision part, wherein the yolov3 algorithm coarse positioning position guides an eye-in-hand camera to obtain an image, image features are extracted, maximum constraint is adopted to reject abnormal features, and finally workpiece contour features are fitted to obtain the high-precision position of the target workpiece.

Description

A method for classifying and positioning columnar workpieces based on target detection and machine vision

Technical field

The present invention relates to industrial robots and machine vision applications, and specifically to a high-precision positioning method for columnar workpiece classification based on target detection and machine vision.

Background technique

With the development of intelligent manufacturing, industrial robots have the advantages of good versatility and high repeated positioning accuracy. In some industrial automation fields, most of them use robot teaching methods. There is still a long way to go to realize true intelligent manufacturing, and traditional teaching cannot meet the needs of intelligent manufacturing. Machine vision technology can well solve the needs of robot position control, but at the same time there is the problem of incompatible recognition flexibility and accuracy. Target detection technology based on deep learning can better meet the flexibility requirements of multi-target recognition, but there is a problem of insufficient positioning accuracy. Traditional machine vision inspection technology has high recognition accuracy, but the recognition features are single.

The patent with publication number CN111238450A discloses a visual positioning method and device. This method collects multiple frames of images for a single target workpiece, and uses the visual positioning information corresponding to each frame of image to satisfy the acquisition pose transformation relationship corresponding to each frame of image. For multi-target workpieces Unable to recognize location. The patent with publication number CN106272416A discloses a precision assembly system and method for a robot's slender axis based on force sense and vision. This invention uses various types of sensors such as vision, position, and force sense to achieve precision assembly, and has certain limitations. .

Target detection based on deep learning can be achieved but the accuracy is poor; traditional machine vision recognition and positioning accuracy is high but the detection target is too single. Therefore, the classification and high-precision positioning of multi-target workpieces is an urgent problem that needs to be solved in the field of industrial robots and machine vision applications.

Contents of the invention

The invention provides a high-precision positioning method for columnar workpiece classification based on target detection and machine vision. The target workpiece is detected through deep learning to complete the workpiece classification and rough positioning of the target. The rough positioning of the target position guides the manipulator to the top of the workpiece, and high-precision positioning of the target is completed through machine vision. Thus, the classification, identification and high-precision positioning of multi-target workpieces are achieved.

Therefore, the present invention provides a high-precision positioning method for columnar workpiece classification based on target detection and machine vision, which includes the following steps:

Multi-target recognition, rough positioning and defect identification process based on yolov3 target detection algorithm:

The Eye-To-Hand camera of the experimental platform is used to collect images of multi-target workpieces. The experimental platform includes a manipulator, a visual control system, an Eye-To-Hand camera and an Eye-In-Hand camera. The Eye-To-Hand camera is fixed directly above the test platform, and the camera has a high working distance from the experimental platform to image different types of multi-target workpieces on the field of view. The Eye-To-Hand camera has low rough recognition and positioning accuracy due to its large working distance.

S1: The Eye-To-Hand camera collects images of multi-target workpieces on the test bench, inputs the collected images into the yolov3 algorithm that improves the network structure, and trains the yolov3 algorithm model that improves the network structure. Use the training The good yolov3 algorithm multi-target detection model performs target detection and obtains various categories and coarse-precision image coordinates of multi-target workpieces.

S2: Based on coordinate transformation, hand-eye calibration is performed on the Eye-To-Hand camera and the end of the manipulator through the calibration plate calibration method. The obtained rough-precision image coordinates of the multi-target workpiece are combined with the hand-eye calibration parameters to calculate the world coordinates of each target workpiece. Also returns the category of each target artifact.

S3: The yolov3 algorithm model with improved network structure is trained on multi-target workpiece types during training, and the typical defects of each workpiece are trained at the same time. When using the yolov3 algorithm that has been trained to improve the network structure for target detection, it can identify key defects such as scratches and missing corners.

High-precision positioning process of target workpiece based on machine vision:

S4: The rough positioning workpiece coordinates are obtained by identifying the yolov3 algorithm model with an improved network structure, and the position coordinates are transmitted to the visual control system based on the communication protocol, and the control system sends them to the manipulator. The vision control system is played by an industrial computer; the Eye-In-Hand camera is connected at the end of the manipulator. The Eye-In-Hand camera moves along with the manipulator to the target workpiece.

S5: The Eye-In-Hand camera moves above the workpiece to collect images of the workpiece. The workpiece is placed above the test bench. The system performs image processing and feature extraction on the acquired images to obtain the key feature coordinates of the workpiece. Combined with the hand-eye calibration parameters of the Eye-In-Hand camera, the high-precision world coordinates of the workpiece are obtained and sent to the vision system.

S6: The system processor guides the manipulator to carry out clamping, transportation or assembly based on high-precision coordinates.

S7: Repeat the above steps S4-S6 to perform high-precision positioning of target workpieces of different categories to achieve high-precision positioning of multiple target workpieces.

The workpiece is a shaft part; the multi-target workpiece includes four different types of workpieces; the camera and the vision system communicate based on the GigE protocol for image transmission; the vision system and the manipulator are based on TCP/IP protocol communication for position coordinate transmission.

Further, the above-mentioned S1 step is specifically:

S11: Use the Eye-To-Hand camera on the test workbench to collect images of the target to be detected. After collecting, mark and classify different types of workpieces to create a training data set. The workpiece marking classification is divided into five categories, including four different types of shaft parts and four different types of workpieces with defects.

S12: Enhance the training data set, input the enhanced data set into the improved yolov3 algorithm model for training, and obtain a parameter model.

S13: Input the original multi-target workpiece image to be identified into the yolov3 model of the trained improved network, and output the corresponding defect detection, classification and identification rough positioning results.

S14: Use the vector similarity measurement method to measure the parameters of the candidate boxes in the training set, perform statistical analysis on them based on the standardized Euclidean distance, perform statistical analysis on the parameters of the candidate boxes based on the standardized Euclidean distance, and write the parameter with the smallest error Enter the configuration file to improve the yolov3 target detection frame.

The yolov3 model with improved network structure is improved based on the network results of darknet53 to meet the requirements of multi-target workpiece target detection. In the target detection and defect identification method provided by the present invention, the yolov3 algorithm network structure model is optimized and improved, specifically including:

The original network model of the Yolov3 target detection algorithm obtains detection results at three scales: 13×13×75, 26×26×75, and 52×52×75 through a series of downsampling processes, where 13, 26, and 52 represent the sampling scales. 75 is split into 3×(4+1+20), 3 represents the detection box of three scales, 4 represents the position information of each detection box, which includes the width and height of the detection box and the center position coordinates of the detection box, 1 represents recognition probability, 20 represents the type of target that can be detected. The yolov3 algorithm that improves the network structure is that the modified network structure can meet four different types of multi-target workpiece target detection, and can also identify different types of defective workpieces, obtaining 13×13×39, 26×26×39, Output of three different scales: 52×52×39.

Further, the above-mentioned S2 step is specifically:

S21: Hand-eye calibration of Eye-To-Hand camera based on the calibration method of halcon calibration plate;

S22: Hand-eye calibration obtains the external parameters of the Eye-To-Hand camera, and normalizes the parameters into matrix form;

S23: Combine the image coordinates obtained by the yolov3 target detection model with the external parameter matrix, and convert the obtained image coordinates into the world coordinates of the robot.

Further, the above-mentioned S5 step is specifically:

S51: After the Eye-In-Hand camera takes a picture of a single target workpiece, it performs image preprocessing, noise reduction and other operations; the preprocessed image is adaptively binarized to obtain edge feature information of the columnar workpiece.

S52: According to the edge feature information of the circle, the method based on outlier detection is used to fit the circular contour of the cylindrical workpiece, and the select_max_length_contour method with maximum value constraints is used to obtain the maximum outer circle contour of the cylindrical workpiece to achieve high accuracy in visual positioning.

The select_max_length_contour method performs maximum value constraints on the concentric circle contour of the cylindrical workpiece obtained after fitting the key information of the workpiece, and returns the contour feature information of the cylindrical workpiece. This method first initializes the longest length and longest length index, then traverses the obtained contour feature length, saves the length and index of the longest contour, and finally returns the index of the longest contour length.

The method of the present invention can classify and position columnar workpieces with an accuracy of micron level, and can identify multiple different types of columnar workpieces at the same time. The recognition accuracy of many different types of columnar workpieces can reach more than 90%, and the recognition speed can reach more than 50fps.

Compared with the prior art, the present invention has the following advantages:

1. This method can perform high-precision positioning of multi-target workpieces on the test workbench, and at the same time cooperate with the manipulator to achieve full automation in the process of clamping, handling and assembly of multi-target workpieces, without manual intervention in the entire process, and significantly improve Increase productivity.

2. Through the Eye-To-Hand camera fixed on the test workbench based on the yolov3 model of the improved network, multi-target workpieces are automatically detected and rough positioned, and defective workpieces are simultaneously detected.

3. The coordinate position returned by rough positioning is passed to the manipulator, and the manipulator takes the Eye-In-Hand camera to move above the target workpiece to position it with high precision. This method can overcome the problem that target detection technology based on deep learning can better meet the flexibility requirements of multi-target recognition, but has insufficient positioning accuracy and traditional machine vision detection technology has high recognition accuracy but single recognition features.

Description of drawings

Figure 1 is a schematic diagram of the camera layout of the present invention.

Figure 2 is a schematic flow chart of a high-precision positioning method for columnar workpiece classification based on target detection and machine vision provided by the present invention.

Figure 3 is a schematic diagram of the structure of the improved yolov3 target detection model of the present invention.

Figure 4 is a flow chart of the select_max_length_contour algorithm used in the present invention.

Figure 5 is a rendering of the high-precision positioning method for columnar workpiece classification based on target detection and machine vision.

Detailed ways

The invention provides a high-precision positioning method for columnar workpiece classification based on target detection and machine vision. The target workpiece is detected through deep learning to complete the workpiece classification and rough positioning of the target. The rough positioning of the target position guides the manipulator to the top of the workpiece, and high-precision positioning of the target is completed through machine vision. Thus, the classification, identification and high-precision positioning of multi-target workpieces are achieved.

Therefore, the present invention provides a high-precision positioning method for columnar workpiece classification based on target detection and machine vision, which includes the following steps:

Multi-target recognition, rough positioning and defect identification process based on yolov3 target detection algorithm:

The Eye-To-Hand camera of the experimental platform is used to collect images of multi-target workpieces. The experimental platform includes a manipulator, a visual control system, an Eye-To-Hand camera and an Eye-In-Hand camera. The Eye-To-Hand camera is fixed directly above the test platform, and the camera has a high working distance from the experimental platform to image different types of multi-target workpieces on the field of view. The Eye-To-Hand camera has low rough recognition and positioning accuracy due to its large working distance.

S1: The Eye-To-Hand camera collects images of multi-target workpieces on the test bench, inputs the collected images into the yolov3 algorithm that improves the network structure, and trains the yolov3 algorithm model that improves the network structure. Use the training The good yolov3 algorithm multi-target detection model performs target detection and obtains various categories and coarse-precision image coordinates of multi-target workpieces.

S2: Based on coordinate transformation, hand-eye calibration is performed on the Eye-To-Hand camera and the end of the manipulator through the calibration plate calibration method. The obtained rough-precision image coordinates of the multi-target workpiece are combined with the hand-eye calibration parameters to calculate the world coordinates of each target workpiece. Also returns the category of each target artifact.

S3: The yolov3 algorithm model with improved network structure is trained on multi-target workpiece types during training, and the typical defects of each workpiece are trained at the same time. When using the yolov3 algorithm that has been trained to improve the network structure for target detection, it can identify key defects such as scratches and missing corners.

High-precision positioning process of target workpiece based on machine vision:

S4: The rough positioning workpiece coordinates are obtained by identifying the yolov3 algorithm model with an improved network structure, and the position coordinates are transmitted to the visual control system based on the communication protocol, and the control system sends them to the manipulator. The vision control system is played by an industrial computer; the Eye-In-Hand camera is connected at the end of the manipulator. The Eye-In-Hand camera moves along with the manipulator to the target workpiece.

S5: The Eye-In-Hand camera moves above the workpiece to collect images of the workpiece. The workpiece is placed above the test bench. The system performs image processing and feature extraction on the acquired images to obtain the key feature coordinates of the workpiece. Combined with the hand-eye calibration parameters of the Eye-In-Hand camera, the high-precision world coordinates of the workpiece are obtained and sent to the vision system.

S6: The system processor guides the manipulator to carry out clamping, transportation or assembly based on high-precision coordinates.

S7: Repeat the above steps S4-S6 to perform high-precision positioning of target workpieces of different categories to achieve high-precision positioning of multiple target workpieces.

The workpiece is a shaft part; the multi-target workpiece includes four different types of workpieces; the camera and the vision system communicate based on the GigE protocol for image transmission; the vision system and the manipulator are based on TCP/IP protocol communication for position coordinate transmission.

Further, the above-mentioned S1 step is specifically:

S11: Use the Eye-To-Hand camera on the test workbench to collect images of the target to be detected. After collecting, mark and classify different types of workpieces to create a training data set. The workpiece marking classification is divided into five categories, including four different types of shaft parts and four different types of workpieces with defects.

S12: Enhance the training data set, input the enhanced data set into the improved yolov3 algorithm model for training, and obtain a parameter model.

S13: Input the original multi-target workpiece image to be identified into the yolov3 model of the trained improved network, and output the corresponding defect detection, classification and identification rough positioning results.

S14: Use the vector similarity measurement method to measure the parameters of the candidate boxes in the training set, perform statistical analysis on them based on the standardized Euclidean distance, perform statistical analysis on the parameters of the candidate boxes based on the standardized Euclidean distance, and write the parameter with the smallest error Enter the configuration file to improve the yolov3 target detection frame.

The yolov3 model with improved network structure is improved based on the network results of darknet53 to meet the requirements of multi-target workpiece target detection. In the target detection and defect identification method provided by the present invention, the yolov3 algorithm network structure model is optimized and improved, specifically including:

The original network model of the Yolov3 target detection algorithm obtains detection results at three scales: 13×13×75, 26×26×75, and 52×52×75 through a series of downsampling processes, where 13, 26, and 52 represent the sampling scales. 75 is split into 3×(4+1+20), 3 represents the detection box of three scales, 4 represents the position information of each detection box, which includes the width and height of the detection box and the center position coordinates of the detection box, 1 represents recognition probability, 20 represents the type of target that can be detected. The yolov3 algorithm that improves the network structure is that the modified network structure can meet four different types of multi-target workpiece target detection, and can also identify different types of defective workpieces, obtaining 13×13×39, 26×26×39, Output of three different scales: 52×52×39.

Further, the above-mentioned S2 step is specifically:

S21: Hand-eye calibration of Eye-To-Hand camera based on the calibration method of halcon calibration plate;

S22: Hand-eye calibration obtains the external parameters of the Eye-To-Hand camera, and normalizes the parameters into matrix form;

S23: Combine the image coordinates obtained by the yolov3 target detection model with the external parameter matrix, and convert the obtained image coordinates into the world coordinates of the robot.

Further, the above-mentioned S5 step is specifically:

S51: After the Eye-In-Hand camera takes a picture of a single target workpiece, it performs image preprocessing, noise reduction and other operations; the preprocessed image is adaptively binarized to obtain edge feature information of the columnar workpiece.

S52: According to the edge feature information of the circle, the method based on outlier detection is used to fit the circular contour of the cylindrical workpiece, and the select_max_length_contour method with maximum value constraints is used to obtain the maximum outer circle contour of the cylindrical workpiece to achieve high accuracy in visual positioning.

The select_max_length_contour method performs maximum value constraints on the concentric circle contour of the cylindrical workpiece obtained after fitting the key information of the workpiece, and returns the contour feature information of the cylindrical workpiece. This method first initializes the longest length and longest length index, then traverses the obtained contour feature length, saves the length and index of the longest contour, and finally returns the index of the longest contour length.

The method of the present invention can classify and position columnar workpieces with an accuracy of micron level, and can identify multiple different types of columnar workpieces at the same time. The recognition accuracy of many different types of columnar workpieces can reach more than 90%, and the recognition speed can reach more than 50fps.

Claims (8)

1. A columnar workpiece classification high-precision positioning method based on target detection and machine vision is characterized by comprising the following steps:

acquiring images of the multi-target workpiece by using an Eye-To-Hand camera of the experimental platform; the experimental platform comprises a manipulator, a visual control system, a Eye-To-Hand camera and an Eye-In-Hand camera; the Eye-To-Hand camera is fixed above the experimental platform, and images different types of multi-target workpieces on the visual field surface; the multi-target workpiece includes four types of columnar workpieces;

s1: the Eye-To-Hand camera acquires images of the multi-target workpieces on the experimental platform, inputs the acquired images into a yolov3 algorithm model with an improved network structure, trains the yolov3 algorithm model with the improved network structure, and performs target detection by using the trained yolov3 algorithm model with the improved network structure To obtain image coordinates of various categories and coarse precision of the multi-target workpieces;

s2: based on coordinate transformation, performing Hand-Eye calibration on the Eye-To-Hand camera and the tail end of the manipulator through a calibration plate calibration method, combining the obtained image coordinates of the multi-target workpiece with Hand-Eye calibration parameters, calculating the world coordinates of each target workpiece, and returning To the category of each target workpiece;

s3: the yolov3 algorithm model with the improved network structure trains multiple target workpiece types during training, and trains scratch and unfilled corner target defects of all workpieces; when the yolov3 algorithm model with the trained improved network structure is used for target detection, identifying scratch and unfilled corner target defects;

s4: the image coordinates of each category and coarse precision of the multi-target workpiece are identified and obtained by a yolov3 algorithm model with an improved network structure, the image coordinates are transmitted to a vision control system based on a communication protocol, and the vision control system sends the image coordinates to a manipulator; the visual control system is acted by the industrial personal computer; the Eye-In-Hand camera is connected with the tail end of the manipulator; the Eye-In-Hand camera moves to the position above the target workpiece along with the manipulator;

s5: the Eye-In-Hand camera moves to the upper part of the workpiece to collect images of the workpiece, the workpiece is placed above the experimental platform, the vision control system performs image processing and feature extraction on the collected images to obtain key feature coordinates of the workpiece, and the Eye-Hand calibration parameters of the Eye-In-Hand camera are combined to obtain high-precision world coordinates of the workpiece and send the world coordinates to the vision control system;

s6: the system processor guides the manipulator to clamp, carry or assemble according to the high-precision world coordinates;

s7: and repeating the steps S4-S6, and carrying out high-precision positioning on different types of target workpieces to realize high-precision positioning of multiple target workpieces.

2. The high-precision positioning method for classifying cylindrical workpieces based on target detection and machine vision according To claim 1, wherein the Eye-To-Hand camera, the Eye-In-Hand camera and the vision control system are communicated based on a GigE protocol for image transmission; and the vision control system and the manipulator are communicated based on TCP/IP protocol to carry out position coordinate transmission.

3. The method for classifying and positioning columnar workpieces with high precision based on target detection and machine vision according to claim 1, wherein the step S1 specifically comprises the following steps:

s11: acquiring images of target workpieces To be detected by using an Eye-To-Hand camera on an experimental platform, and performing marking classification on different types of workpieces after acquisition To manufacture a training data set;

s12: performing enhancement processing on the training data set, and inputting the data set subjected to the enhancement processing into a yolov3 algorithm model with an improved network structure for training to obtain a parameter model;

s13: inputting an original multi-target workpiece image to be identified into a yolov3 algorithm model of a trained improved network structure, and outputting corresponding defect detection and classification identification coarse positioning results;

s14: and measuring the candidate frame parameters in the training set by adopting a vector similarity measurement method, carrying out statistical analysis on the candidate frame parameters according to the standardized Euclidean distance, writing the parameters with the minimum error into a configuration file, and improving the yolov3 target detection frame.

4. The columnar workpiece classification high-precision positioning method based on target detection and machine vision according to claim 1, wherein the yolov3 algorithm model with an improved network structure is improved based on a network result of a dark net53, and the requirement of multi-target workpiece target detection is met.

5. The method for classifying and positioning columnar workpieces with high precision based on target detection and machine vision according to claim 1, wherein an original network model of a yolov3 algorithm model obtains detection results under three scales of 13×13×75, 26×26×75 and 52×52×75 by a series of downsampling processes, wherein 13, 26 and 52 represent sampling scales; 75 is split into 3× (4+1+20), 3 represents three dimensions of the detection boxes, 4 represents the position information of each detection box, which includes the width and height of the detection box and the central position coordinates of the detection box, 1 represents the probability of identification, and 20 represents the detected target species; the yolov3 algorithm model with improved network structure is a modified network structure, can meet the target detection of four different types of multi-target workpieces, and can identify different types of defective workpieces to obtain three different-scale outputs of 13×13×39, 26×26×39 and 52×52×39.

6. The method for classifying and positioning columnar workpieces with high precision based on object detection and machine vision according to claim 1, wherein the step S2 is specifically as follows:

s21: the Eye-Eye calibration is carried out on the Eye-To-Hand camera by a calibration method of a calibration plate based on the halcon;

s22: the Hand-Eye calibration is carried out To obtain the external parameters of the Eye-To-Hand camera, and the parameters are standardized into a matrix form;

s23: and combining the image coordinates obtained by the yolov3 algorithm model with an external parameter matrix, and converting the obtained image coordinates into world coordinates of the manipulator.

7. The method for classifying and positioning columnar workpieces with high precision based on object detection and machine vision according to claim 1, wherein the step S5 specifically comprises:

s51: after photographing a single target workpiece by the Eye-In-Hand camera, performing image preprocessing and noise reduction operation; performing self-adaptive binarization on the preprocessed image to obtain edge characteristic information of the columnar workpiece;

s52: fitting the circular outline of the columnar workpiece according to the edge characteristic information of the circle based on an abnormal value detection method, and obtaining the maximum excircle outline of the columnar workpiece by adopting a maximum value constraint selection_max_length_contour method to realize high precision of visual positioning.

8. The high-precision positioning method for classifying cylindrical workpieces based on target detection and machine vision according to claim 1, wherein the classification positioning precision of the cylindrical workpieces is in a micron level, and a plurality of different types of cylindrical workpieces can be identified.