CN112560704B - Visual identification method and system for multi-feature fusion - Google Patents
Visual identification method and system for multi-feature fusion Download PDFInfo
- Publication number
- CN112560704B CN112560704B CN202011504074.0A CN202011504074A CN112560704B CN 112560704 B CN112560704 B CN 112560704B CN 202011504074 A CN202011504074 A CN 202011504074A CN 112560704 B CN112560704 B CN 112560704B
- Authority
- CN
- China
- Prior art keywords
- image
- images
- coordinate system
- parts
- rectangular frame
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/20—Scenes; Scene-specific elements in augmented reality scenes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/253—Fusion techniques of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/254—Fusion techniques of classification results, e.g. of results related to same input data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V2201/00—Indexing scheme relating to image or video recognition or understanding
- G06V2201/06—Recognition of objects for industrial automation
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Abstract
The invention provides a visual identification method for multi-feature fusion, which comprises the steps of receiving a part source diagram; dividing the part source diagram to obtain each part image; performing two-stage classification and identification on each part image to determine the type of the part in each part image and the attribution of the image classification; determining a template image and a rotation image of each part based on part images in the similar images, and extracting and matching key points of the template image and the rotation image to calculate an affine transformation matrix between the rotation angles of each part and the images; determining a coordinate transformation matrix, and combining the affine transformation matrix to enable the coordinates of the grabbing points to be transformed to the coordinates under the actual robot coordinate system; and sending the types and the rotation angles of the parts and the coordinates of the grabbing points to the actual robot. By implementing the invention, the types and the rotation angles of scattered parts in an actual scene and the coordinate information of the grabbing points can be identified and obtained, and the correct guidance of the robot for adjusting the gesture of the parts and clamping and assembling work is realized.
Description
Technical Field
The invention relates to the technical field of flexible assembly systems of miniature circuit breakers, in particular to a visual identification method and a visual identification system for multi-feature fusion.
Background
With the increasing demand for automation and intellectualization of machines in industrial production, machine vision has been studied in depth as one of the important technologies for the development of industrial production in the intellectualization direction, and more technologies for visual inspection, measurement, positioning, etc. are applied to practical production.
The machine vision technology is applied to the research in industrial manufacturing, so that the mechanical degree of manufacturing equipment can be reduced, the flexibility and the flexibility of industrial production can be improved, and the quality and the production efficiency of products can be improved. However, the development of intelligent manufacturing technology based on machine vision is supported by the technologies of high-precision image classification, identification, positioning tracking and the like, and the feasibility and reliability of the intelligent manufacturing technology are supported by the different functional requirements of various production objects with various shapes and colors, complex industrial environments and the like in production, so that great difficulties and challenges are brought to visual identification processing work.
In the aspect of industrial assembly, when the recognition objects with relatively low complexity are faced, modeling methods such as a vector machine and the like can be supported for classification, and once the recognition objects are more and more complex, the data volume required by model training is larger, so that the development period is greatly prolonged. In addition, when the complexity of the identification object is low, the information such as the outline, the area and the color of the identification object can be utilized to classify, and the coordinates and the gesture of the grabbing point can be estimated through methods such as feature matching, template matching and the like, but when the complexity of the identification object is high, especially in the automatic assembly application of a miniature circuit breaker, the appearance and the variety of the parts of the circuit breaker are complex and various, the feature point distinction of part of the parts is not obvious, the rotation angle and the coordinates of the grabbing point can not be accurately estimated through the methods such as feature matching and template matching, namely, the identified errors can not be controlled within the allowable error range of a system, and thus, the robot can not be guided to finish the subsequent work of adjusting the gesture and clamping assembly of the parts correctly.
Therefore, a machine vision identification method for automatic assembly of a miniature circuit breaker is needed, which can identify and acquire the types and rotation angles of scattered parts in an actual scene and the coordinate information of the grabbing points, ensure that the identification error is within the allowable range of a system and realize correct guidance of the robot for adjusting the gesture of the parts and clamping and assembling work.
Disclosure of Invention
The technical problem to be solved by the embodiment of the invention is to provide a visual identification method and a visual identification system for multi-feature fusion, which can identify and acquire the types and rotation angles of scattered parts and the coordinate information of the grabbing points in an actual scene, ensure that the identification error is within the allowable range of the system and realize the correct guidance of the robot for adjusting the gesture of the parts and clamping and assembling work.
In order to solve the technical problems, the embodiment of the invention provides a visual identification method for multi-feature fusion, which is used in a flexible assembly system of a miniature circuit breaker and comprises the following steps:
s1, receiving a part source diagram of the whole shooting of a field part;
s2, carrying out segmentation processing on the part source diagram to obtain part images in all independent states;
step S3, carrying out two-stage classification and identification on the part images in each independent state based on the types and the arrangement postures of the parts so as to determine the types of the parts in each part image and the attribution of the image classification; wherein, a plurality of part images with different placing postures of the same part type are all classified into the same type of image;
s4, determining a template image and a rotation image of each part based on the part images in the same type of images, and extracting and matching key points of the template image of each part and the corresponding rotation image to calculate affine transformation matrixes among the rotation angles of each part and the images;
s5, determining a coordinate transformation matrix for transforming a pixel coordinate system into a robot coordinate system, and transforming the grabbing point coordinates in the part source diagram into coordinates under an actual robot coordinate system by combining the affine transformation matrix;
and S6, transmitting the types and the rotation angles of the parts and the transformed coordinates of the grabbing points to a controller of the actual robot so as to realize the guidance of the robot for adjusting the posture of the parts and clamping assembly work.
Wherein, between the step S1 and the step S2, the method further comprises the following steps:
preprocessing the part source diagram; the preprocessing comprises image graying, part edge extraction, binarization, morphological operation, part contour extraction and maximum and minimum rectangular frame fitting.
In the step S2, the "part image in each independent state" is obtained by dividing the plurality of parts included in the part source map into the parts independent from each other by re-drawing the part pixels in the minimum rectangular frame into the blank image having the size of the maximum circumscribed rectangular frame.
The step S3 specifically includes:
and carrying out primary classification and identification on the part images in each independent state based on the part types to determine the types of the parts in the part images in each independent state, and further carrying out secondary classification and identification on the part images belonging to the same part type based on the arrangement postures of the part types to determine the image classification attribution of the part images.
The types of the parts in the part images in the independent states are realized by analyzing and comparing the outline area, the minimum circumscribed rectangular frame area and the color of the parts in the part images in the independent states, and then comparing the actual characteristic data obtained after the comparison and analysis with a first set threshold value.
The image classification attribution of the part images is realized by analyzing and comparing the colors, the minimum rectangular frame area, the imaging structure and the inherent specific features of the parts under each placing posture after the first-level classification recognition is carried out on the part images in each independent state, and then comparing the actual feature data obtained after the comparison and analysis with a second set threshold value.
The step S4 specifically includes:
extracting two part images with different placing postures in the same type of images to be respectively used as template images and rotation images of corresponding parts;
analyzing and selecting interested sides and interested vertex positions in the minimum rectangular frame of each part on a template image and a rotating image of each part, redefining storage sequence of each vertex of the rectangular frame according to requirements, and further sequentially searching part contour points closest or farthest to each vertex as key points;
matching key points on the template image of each part with key points on the corresponding rotating image, then calculating vectors formed by the template image of each part and the corresponding key points in the rotating image, and further setting the average value of included angles among the calculated vectors of each part as the rotating angle of each part;
and calculating affine transformation matrixes between the template image and the rotation image of each part through three or more key point pairs.
The step S5 specifically includes:
obtaining camera internal parameters and external parameters matrix through camera calibration, sequentially converting a pixel coordinate system into an image coordinate system, converting an image into a camera coordinate system and converting a camera into a world coordinate system, and combining translation and rotation relations between the world coordinate system and a robot coordinate system to obtain a coordinate conversion matrix capable of converting the pixel coordinate from the pixel coordinate system to the robot coordinate system;
carrying out affine transformation on the designated grabbing points in the part source diagram through the affine transformation matrix to obtain pixel coordinates of the grabbing points in an actual image, and then carrying out space transformation on the pixel coordinates of the grabbing points through the coordinate transformation matrix to obtain coordinates of the grabbing points under a robot coordinate system.
The embodiment of the invention also provides a visual identification system for multi-feature fusion, which comprises the following steps:
the image receiving unit is used for receiving a part source diagram shot by the whole field part;
the image segmentation unit is used for carrying out segmentation processing on the part source image to obtain part images in all independent states;
the image classification recognition unit is used for carrying out two-stage classification recognition on the part images in each independent state based on the type and the placement posture of the part so as to determine the type of the part in each part image and the attribution of the image classification; wherein, a plurality of part images with different placing postures of the same part type are all classified into the same type of image;
the part rotation angle calculation unit is used for determining a template image and a rotation image of each part based on part images in the same type of images, and extracting and matching key points of the template image of each part and the corresponding rotation image so as to calculate affine transformation matrixes among the rotation angles and the images of each part;
the grabbing point coordinate transformation unit is used for determining a coordinate transformation matrix for transforming a pixel coordinate system into a robot coordinate system, and transforming grabbing point coordinates in the part source diagram into coordinates under an actual robot coordinate system by combining the affine transformation matrix;
and the information sending unit is used for sending the types and the rotation angles of the parts and the transformed grabbing point coordinates to a controller of the actual robot so as to realize the guidance of the robot for adjusting the posture of the parts and clamping assembly work.
Wherein, still include:
the image preprocessing unit is used for preprocessing the part source diagram; the preprocessing comprises image graying, part edge extraction, binarization, morphological operation, part contour extraction and maximum and minimum rectangular frame fitting.
The embodiment of the invention has the following beneficial effects:
1. compared with the prior art, the method and the device have the advantages that the part image segmentation is carried out in a mode that the part pixel points in the minimum rectangular frame are drawn into the blank image with the maximum external rectangular frame size, so that the problem that the non-target part is segmented into one image at the same time when the parts are relatively close to each other can be effectively avoided; by analyzing the features of the outline area, the minimum circumscribed rectangular frame area, the color, the imaging structure and the like of the parts, two-stage classification recognition is performed in a threshold comparison mode, the classification accuracy is ensured, and meanwhile, the development period of a system can be shortened;
2. the invention obtains the key point pairs by searching the part contour points which are closest or farthest from the vertex of the smallest rectangle frame after the rearrangement, calculates the rotation angle and the actual grabbing point coordinates of the parts by the key point pairs, and finally sends the types and the rotation angles of the parts and the transformed grabbing point coordinates to the controller of the actual robot, thereby identifying and obtaining the types and the rotation angles of the scattered parts in the actual scene and the grabbing point coordinate information, ensuring that the identification error is within the allowable range of the system and realizing the correct guidance of the robot for adjusting the gesture of the parts and clamping and assembling work;
3. compared with the modeling classification mode, the invention is more convenient for the maintenance and adjustment of a later system, and can meet the requirement of the system on the recognition precision more than the modes of feature matching, template matching and the like.
Drawings
In order to more clearly illustrate the embodiments of the invention or the technical solutions of the prior art, the drawings which are required in the description of the embodiments or the prior art will be briefly described, it being obvious that the drawings in the description below are only some embodiments of the invention, and that it is within the scope of the invention to one skilled in the art to obtain other drawings from these drawings without inventive faculty.
FIG. 1 is a flowchart of a visual recognition method for multi-feature fusion provided by an embodiment of the present invention;
FIG. 2 is a graph showing a segmentation comparison of the source map of the part in the application scenario of step S2 of FIG. 1;
FIG. 3 is a flowchart illustrating the step S3 of FIG. 1 when the first class classification of the part image is performed;
FIGS. 4-11 are schematic and algorithmic flow diagrams of part operations during secondary classification and identification of part images in the application scenario of step S3 of FIG. 1;
fig. 12 is a schematic view of angle calculation in the application scenario of step S4 in fig. 1;
FIG. 13 is a schematic diagram of a calibration required coordinate system in the application scenario of step S5 in FIG. 1;
fig. 14 is a schematic structural diagram of a multi-feature fusion visual recognition system according to an embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings, for the purpose of making the objects, technical solutions and advantages of the present invention more apparent.
As shown in fig. 1, in an embodiment of the present invention, a visual recognition method for multi-feature fusion is provided, which is used in a flexible assembly system of a miniature circuit breaker, and includes the following steps:
s1, receiving a part source diagram of the whole shooting of a field part;
the specific process is that after receiving the image acquisition signal, a camera is called to take a picture, so that a part source diagram of the whole shooting of the camera field part is obtained.
S2, carrying out segmentation processing on the part source diagram to obtain part images in all independent states;
the method comprises the specific steps that on a part source diagram, part pixels in a minimum rectangular frame are redrawn into blank images with the size of the maximum circumscribed rectangular frame, and a plurality of parts contained in the part source diagram are divided into independent parts, so that part images in independent states are obtained.
In one embodiment, as shown in fig. 2 (a), the image segmentation cannot be directly performed by using a green minimum rectangular frame, when the parts are closely spaced, the problem of simultaneously segmenting the non-target part into one image as shown in fig. 2 (b) occurs by using a blue maximum circumscribed rectangular frame, and according to the provided method, the image segmentation is performed in such a way that the pixel points of the part in the minimum rectangular frame are drawn into a blank image with the size of the maximum circumscribed rectangular frame, as shown in fig. 2 (c), so that the problem can be effectively avoided.
It should be noted that, before the segmentation processing is performed on the part source diagram, the method further comprises the step of preprocessing the part source diagram; the preprocessing comprises the steps of image graying, part edge extraction, binarization, morphological operation, part contour extraction, maximum and minimum rectangular frame fitting and the like.
Step S3, carrying out two-stage classification and identification on the part images in each independent state based on the types and the arrangement postures of the parts so as to determine the types of the parts in each part image and the attribution of the image classification; wherein, a plurality of part images with different placing postures of the same part type are all classified into the same type of image;
the specific process is that based on the types of the parts, the first-level classification and identification are carried out on the part images in each independent state so as to determine the types of the parts in the part images in each independent state, and further based on the placement postures of the types of the parts, the second-level classification and identification are carried out on the part images belonging to the same part type so as to determine the image classification attribution of the part images.
The types of the parts in the part images in the independent states are realized by analyzing and comparing the outline area, the minimum circumscribed rectangular frame area and the color of the parts in the part images in the independent states, and then comparing the actual characteristic data obtained after the comparison and analysis with a first set threshold value.
The image classification attribution of each part image is realized by analyzing and comparing the color, the minimum rectangular frame area, the imaging structure and the inherent specific characteristics of the part under each placing posture after the first-level classification recognition is carried out on the part image in each independent state, and then comparing the actual characteristic data obtained after the comparison and analysis with a second set threshold value.
In one embodiment, a first class identification is performed on the part image. As shown in FIG. 3, S 0 For initializing variables max_S and min_S, S for the minimum rectangular frame area of the first part after division n Then the minimum rectangular frame area of the other four parts. Th (Th) 1 、Th 2 The reference threshold for the number of green and red pixels, respectively. Sr (Sr) 1 、Sr 2 The minimum rectangular frame area of the part represented by pictures 3 and 4, respectively.
In another embodiment, the part images are subjected to secondary classification recognition.
Part 0: comparing the minimum rectangular box area max_s with a given threshold Th 3 The size of the classification result lab 0 The following formula (1):
as shown in (a) and (c) of FIG. 4, the embodiment of the invention uses P for two possible cases of rectangular frames with postures 01 and 02 1 The point is taken as a basic point to judge P 1 P 2 And P 1 P 3 And redefine the order of the four vertices as shown in fig. 4 (b) and (d) by the following formula (2).
As shown in the shaded areas (b) and (d) of FIG. 4, embodiments of the present invention are illustrated as Q 1 Q 2 The long side where it is located delimits the region of interest (ROI). Assuming that the set of pixel points representing the part is S, the set of pixel points in the region of interest is S, and selecting the long side of the rectangle of interest by the following formula (3), denoted as L:
the selection is also made in an interesting way for the two end points, i.e. vertices, of the selected edge L. As shown in fig. 5, in actual operation, point a is the final selected point of interest, and two points B, C forming the short side and the long side of the rectangle are recorded. Let the included angle between the vectors AB and AC be Φ, since the clockwise rotation angle in the image is negative and vice versa, a class determination is made according to the following formula (4), which is written as:
part 1: as shown in FIG. 6, part 1 two-stage classification algorithm flow, th 4 Threshold value Th is referenced for green pixel number of part 1 in different self-postures 1 Differing from it in Th 1 A reference that itself is compared to other parts; th (Th) 5 A reference threshold value for self contour area; small circles are as yellow marked circles in figure 7 (a) and are in contrast to (b). The steps of the sub-process 'pose 10or11' are substantially identical to the distinguishing steps of part 0 poses 01 and 02. The side of interest selected by part 1 is the short side of the rectangular frame and is based on the distance between the two short sides and the centroid point m. In actual operation, as shown in fig. 7 (c) and (d), the selected edge is AB edge, and the point is point a. The classification result is shown as the following formula (5) and is denoted as lab 1 ：
Part 2: minimum rectangular frame surface of comparison part 2Integrating min_S with a given threshold Th 6 The size of (2) is determined by the following formula (6) and is denoted as lab 2 ：
As shown in fig. 8, the contour of the part 2 is subjected to convex hull detection to find a bending point O at the middle bending position. The convex hull is a convex polygon like the blue surrounding lines in (a) and (c) in fig. 8, and the white area surrounded by the convex hull is a convex "defect". The AB edge is found in the mode of selecting the edge in the part 0, and the vertex A of the rectangular frame is closer to the point O under both postures, so that the point A is determined. Assuming that the angle between the vectors OA and OB is phi, the classification result is expressed as the following formula (7) and is denoted as lab 2 ：
Part 3: as shown in fig. 9, the flow of the two-stage classification algorithm of the part 3 makes further judgment according to whether it passes the red extraction judgment in the one-stage classification. s3 is the minimum rectangular frame area, th 7 Th is the discrimination threshold for poses 30 and 32 8 Is a discrimination threshold for poses 31 and 33.
Part 4: comparing the minimum rectangular frame area s4 of the part 4 with a given threshold Th 9 And Th (Th) 10 Is of the size of Th 9 <Th 10 . The classification result is shown in the following formula (8) and is denoted as lab 4 ：
For poses 41, 42, and 43, part profile area S is compared cnt And a given threshold Th 11 Is classified into two subclasses, and is denoted as lab as shown in the following formula (9) 4 ：
As shown in fig. 10, the poses 41 and 42 have great similarity in terms of gray scale, shape, edge gradient, etc., and are difficult to distinguish accurately, so that they are mechanically changed into the poses 40 or 43 and then angle recognition is performed.
For poses 44 and 45: and (3) sequentially defining the interested areas for the vertexes of the rectangular frame, and meeting the condition expressed by the following formula (10), namely determining the vertexes of interest. S is a part pixel point set, and S is a pixel point set in the region of interest.
Retracting the rectangular frame to obtain part contour points in the new rectangle; the obtained contour points are compared with the minimum distance between the sides of the two original rectangular frames which do not take the vertex of interest as an endpoint to determine the positions of the two sides. In practice, as shown in fig. 11, the vertex of interest is a, the side with the smallest distance from the outline point in the retracted rectangular frame is BD, and the other side is CD. Assuming that the included angle between the vectors AB and AC is phi, making a class judgment according to the following formula (11), and recording as lab 4 ：
S4, determining a template image and a rotation image of each part based on the part images in the same type of images, and extracting and matching key points of the template image of each part and the corresponding rotation image to calculate affine transformation matrixes among the rotation angles of each part and the images;
extracting two part images with different placing postures in the same type of images to be respectively used as a template image and a rotation image of a corresponding part;
analyzing and selecting interested sides and interested vertex positions in the minimum rectangular frame of each part on a template image and a rotating image of each part, redefining storage sequence of each vertex of the rectangular frame according to requirements, and further sequentially searching part contour points closest or farthest to each vertex as key points;
matching key points on the template image of each part with key points on the corresponding rotating image, then calculating vectors formed by the template image of each part and the corresponding key points in the rotating image, and further setting the average value of included angles among the calculated vectors of each part as the rotating angle of each part;
and calculating affine transformation matrixes between the template image and the rotation image of each part through three or more key point pairs.
In one embodiment, as shown in FIG. 12, a schematic view of angle calculation is shown, and the vectors ac of two key points in the actual image and the vector a in the corresponding template are calculated 1 c 1 The included angle θ is the rotation angle. To reduce the error, as many values as possible are averaged.
S5, determining a coordinate transformation matrix for transforming a pixel coordinate system into a robot coordinate system, and transforming the grabbing point coordinates in the part source diagram into coordinates under an actual robot coordinate system by combining the affine transformation matrix;
the method comprises the steps of obtaining camera internal parameters and external parameters through camera calibration, sequentially converting a pixel coordinate system into an image coordinate system, converting an image into a camera coordinate system and converting a camera into a world coordinate system, and combining translation and rotation relations between the world coordinate system and a robot coordinate system to obtain a coordinate conversion matrix capable of converting the pixel coordinate from the pixel coordinate system to the robot coordinate system;
carrying out affine transformation on the designated grabbing points in the part source diagram through the affine transformation matrix to obtain pixel coordinates of the grabbing points in an actual image, and then carrying out space transformation on the pixel coordinates of the grabbing points through the coordinate transformation matrix to obtain coordinates of the grabbing points under a robot coordinate system.
In one embodiment, first, pixel coordinates of a grabbing point in an actual image are obtained by affine transformation of pixel coordinates of the grabbing point in a template image corresponding to a current posture part, which is stored in advance, as shown in the following formula (12):
wherein M is 1 Is affine transformation matrix, (u, v) is pixel coordinate of grabbing point in actual image, (u) 0 ,v 0 ) Pixel coordinates in a template image for a capture point
Next, the actual coordinates of the pixels of the grabbing points are transformed by 4 times of coordinate systems required by camera calibration and hand-eye calibration as shown in fig. 13, so as to obtain the coordinates of the grabbing point robot in the coordinate system, as shown in the following formula (13):
wherein M is 2 Is a coordinate transformation matrix, (x, y, z) is the coordinates of the grabbing point in the robot coordinate system, (x) 0 ,y 0 ,z 0 ) The offset between the robot coordinate system and the world coordinate system.
And S6, transmitting the types and the rotation angles of the parts and the transformed coordinates of the grabbing points to a controller of the actual robot so as to realize the guidance of the robot for adjusting the posture of the parts and clamping assembly work.
The specific process is that a controller of the robot is used as a main controller, and the received information comprises the type of the part, the rotation angle and the coordinates of the grabbing points under the robot coordinate system, so that the correct guidance of the robot for adjusting the gesture of the part and clamping and assembling work is ensured. The controller of the robot plays roles of receiving a part arrival signal input by the power-off switch, outputting a camera shooting starting signal, controlling the movement of the robot and the action of the cylinder clamping jaw and the like.
As shown in fig. 14, in an embodiment of the present invention, a multi-feature fusion visual recognition system is provided, including:
an image receiving unit 110 for receiving a part source map of the whole shot field part;
an image segmentation unit 120, configured to perform segmentation processing on the part source map, so as to obtain part images in each independent state;
an image classification recognition unit 130, configured to perform two-stage classification recognition on the part images in the independent states based on the type and the placement posture of the part, so as to determine the type of the part in each part image and the attribution of the image classification; wherein, a plurality of part images with different placing postures of the same part type are all classified into the same type of image;
a part rotation angle calculating unit 140, configured to determine a template image and a rotation image of each part based on part images in the same type of image, and perform key point extraction and matching on the template image of each part and the corresponding rotation image, so as to calculate affine transformation matrices between each part rotation angle and the images;
a grabbing point coordinate transformation unit 150, configured to determine a coordinate transformation matrix for transforming a pixel coordinate system into a robot coordinate system, and transform grabbing point coordinates in the part source diagram into coordinates in an actual robot coordinate system in combination with the affine transformation matrix;
and the information sending unit 160 is used for sending the types and the rotation angles of the parts and the transformed coordinates of the grabbing points to the controller of the actual robot so as to realize the guidance of the robot for adjusting the postures of the parts and clamping assembly work.
Wherein, still include:
the image preprocessing unit is used for preprocessing the part source diagram; the preprocessing comprises image graying, part edge extraction, binarization, morphological operation, part contour extraction and maximum and minimum rectangular frame fitting.
The embodiment of the invention has the following beneficial effects:
1. compared with the prior art, the method and the device have the advantages that the part image segmentation is carried out in a mode that the part pixel points in the minimum rectangular frame are drawn into the blank image with the maximum external rectangular frame size, so that the problem that the non-target part is segmented into one image at the same time when the parts are relatively close to each other can be effectively avoided; by analyzing the features of the outline area, the minimum circumscribed rectangular frame area, the color, the imaging structure and the like of the parts, two-stage classification recognition is performed in a threshold comparison mode, the classification accuracy is ensured, and meanwhile, the development period of a system can be shortened;
2. the invention obtains the key point pairs by searching the part contour points which are closest or farthest from the vertex of the smallest rectangle frame after the rearrangement, calculates the rotation angle and the actual grabbing point coordinates of the parts by the key point pairs, and finally sends the types and the rotation angles of the parts and the transformed grabbing point coordinates to the controller of the actual robot, thereby identifying and obtaining the types and the rotation angles of the scattered parts in the actual scene and the grabbing point coordinate information, ensuring that the identification error is within the allowable range of the system and realizing the correct guidance of the robot for adjusting the gesture of the parts and clamping and assembling work;
3. compared with the modeling classification mode, the invention is more convenient for the maintenance and adjustment of a later system, and can meet the requirement of the system on the recognition precision more than the modes of feature matching, template matching and the like.
It should be noted that, in the above system embodiment, each unit included is only divided according to the functional logic, but not limited to the above division, so long as the corresponding function can be implemented; in addition, the specific names of the functional units are also only for distinguishing from each other, and are not used to limit the protection scope of the present invention.
Those of ordinary skill in the art will appreciate that all or a portion of the steps in implementing the methods of the above embodiments may be implemented by a program to instruct related hardware, where the program may be stored in a computer readable storage medium, such as ROM/RAM, a magnetic disk, an optical disk, etc.
The above disclosure is only a preferred embodiment of the present invention, and it is needless to say that the scope of the invention is not limited thereto, and therefore, the equivalent changes according to the claims of the present invention still fall within the scope of the present invention.
Claims (4)
1. A visual identification method for multi-feature fusion is used in a flexible assembly system of a miniature circuit breaker, and is characterized by comprising the following steps:
s1, receiving a part source diagram of the whole shooting of a field part;
s2, carrying out segmentation processing on the part source diagram to obtain part images in all independent states;
step S3, carrying out two-stage classification and identification on the part images in each independent state based on the types and the arrangement postures of the parts so as to determine the types of the parts in each part image and the attribution of the image classification; wherein, a plurality of part images with different placing postures of the same part type are all classified into the same type of image;
s4, determining a template image and a rotation image of each part based on the part images in the same type of images, and extracting and matching key points of the template image of each part and the corresponding rotation image to calculate affine transformation matrixes among the rotation angles of each part and the images;
s5, determining a coordinate transformation matrix for transforming a pixel coordinate system into a robot coordinate system, and transforming the grabbing point coordinates in the part source diagram into coordinates under an actual robot coordinate system by combining the affine transformation matrix;
s6, the types and the rotation angles of the parts and the transformed coordinates of the grabbing points are sent to a controller of an actual robot so as to realize the guidance of the robot for adjusting the posture of the parts and clamping assembly work;
the "part image in each independent state" in the step S2 is obtained by dividing the plurality of parts included in the part source map into each independent part by re-drawing the part pixels in the minimum rectangular frame into a blank image having the size of the maximum circumscribed rectangular frame;
the step S3 specifically includes:
performing primary classification and identification on the part images in each independent state based on the part types to determine the types of the parts in the part images in each independent state, and further performing secondary classification and identification on the part images belonging to the same part type based on the placement posture of each part type to determine the image classification attribution of each part image;
the types of the parts in the part images in the independent states are realized by analyzing and comparing the outline area, the minimum circumscribed rectangular frame area and the color of the parts in the part images in the independent states, and then comparing the actual characteristic data obtained after the comparison and analysis with a first set threshold value;
the image classification attribution of each part image is realized by analyzing and comparing the color, the minimum rectangular frame area, the imaging structure and the inherent specific characteristics of the part under each placement posture of each part after the first-level classification recognition is carried out on the part images in each independent state, and then comparing the actual characteristic data obtained after the comparison and analysis with a second set threshold value;
the step S4 specifically includes:
extracting two part images with different placing postures in the same type of images to be respectively used as template images and rotation images of corresponding parts;
analyzing and selecting interested sides and interested vertex positions in the minimum rectangular frame of each part on a template image and a rotating image of each part, redefining storage sequence of each vertex of the rectangular frame according to requirements, and further sequentially searching part contour points closest or farthest to each vertex as key points;
matching key points on the template image of each part with key points on the corresponding rotating image, then calculating vectors formed by the template image of each part and the corresponding key points in the rotating image, and further setting the average value of included angles among the calculated vectors of each part as the rotating angle of each part;
obtaining affine transformation matrixes between the template images and the rotating images of all parts through calculation through three or more key point pairs;
the step S5 specifically includes:
obtaining camera internal parameters and external parameters matrix through camera calibration, sequentially converting a pixel coordinate system into an image coordinate system, converting an image into a camera coordinate system and converting a camera into a world coordinate system, and combining translation and rotation relations between the world coordinate system and a robot coordinate system to obtain a coordinate conversion matrix capable of converting the pixel coordinate from the pixel coordinate system to the robot coordinate system;
carrying out affine transformation on the designated grabbing points in the part source diagram through the affine transformation matrix to obtain pixel coordinates of the grabbing points in an actual image, and then carrying out space transformation on the pixel coordinates of the grabbing points through the coordinate transformation matrix to obtain coordinates of the grabbing points under a robot coordinate system.
2. The visual recognition method of multi-feature fusion according to claim 1, further comprising, between the step S1 and the step S2, the steps of:
preprocessing the part source diagram; the preprocessing comprises image graying, part edge extraction, binarization, morphological operation, part contour extraction and maximum and minimum rectangular frame fitting.
3. A multi-feature fusion visual recognition system, comprising:
the image receiving unit is used for receiving a part source diagram shot by the whole field part;
the image segmentation unit is used for carrying out segmentation processing on the part source image to obtain part images in all independent states;
the image classification recognition unit is used for carrying out two-stage classification recognition on the part images in each independent state based on the type and the placement posture of the part so as to determine the type of the part in each part image and the attribution of the image classification; wherein, a plurality of part images with different placing postures of the same part type are all classified into the same type of image;
the part rotation angle calculation unit is used for determining a template image and a rotation image of each part based on part images in the same type of images, and extracting and matching key points of the template image of each part and the corresponding rotation image so as to calculate affine transformation matrixes among the rotation angles and the images of each part;
the grabbing point coordinate transformation unit is used for determining a coordinate transformation matrix for transforming a pixel coordinate system into a robot coordinate system, and transforming grabbing point coordinates in the part source diagram into coordinates under an actual robot coordinate system by combining the affine transformation matrix;
the information sending unit is used for sending the types and the rotation angles of the parts and the transformed coordinates of the grabbing points to a controller of the actual robot so as to realize the guidance of the robot for adjusting the posture of the parts and clamping assembly work;
the "part image in each independent state" in the image dividing unit is obtained by dividing a plurality of parts contained in the part source map into each independent part by re-drawing part pixels in a minimum rectangular frame into a blank image having the size of the maximum circumscribed rectangular frame;
the image grading identification unit specifically comprises:
performing primary classification and identification on the part images in each independent state based on the part types to determine the types of the parts in the part images in each independent state, and further performing secondary classification and identification on the part images belonging to the same part type based on the placement posture of each part type to determine the image classification attribution of each part image;
the types of the parts in the part images in the independent states are realized by analyzing and comparing the outline area, the minimum circumscribed rectangular frame area and the color of the parts in the part images in the independent states, and then comparing the actual characteristic data obtained after the comparison and analysis with a first set threshold value;
the image classification attribution of each part image is realized by analyzing and comparing the color, the minimum rectangular frame area, the imaging structure and the inherent specific characteristics of the part under each placement posture of each part after the first-level classification recognition is carried out on the part images in each independent state, and then comparing the actual characteristic data obtained after the comparison and analysis with a second set threshold value;
the part rotation angle calculating unit specifically includes:
extracting two part images with different placing postures in the same type of images to be respectively used as template images and rotation images of corresponding parts;
analyzing and selecting interested sides and interested vertex positions in the minimum rectangular frame of each part on a template image and a rotating image of each part, redefining storage sequence of each vertex of the rectangular frame according to requirements, and further sequentially searching part contour points closest or farthest to each vertex as key points;
matching key points on the template image of each part with key points on the corresponding rotating image, then calculating vectors formed by the template image of each part and the corresponding key points in the rotating image, and further setting the average value of included angles among the calculated vectors of each part as the rotating angle of each part;
obtaining affine transformation matrixes between the template images and the rotating images of all parts through calculation through three or more key point pairs;
the grabbing point coordinate transformation unit specifically comprises:
obtaining camera internal parameters and external parameters matrix through camera calibration, sequentially converting a pixel coordinate system into an image coordinate system, converting an image into a camera coordinate system and converting a camera into a world coordinate system, and combining translation and rotation relations between the world coordinate system and a robot coordinate system to obtain a coordinate conversion matrix capable of converting the pixel coordinate from the pixel coordinate system to the robot coordinate system;
carrying out affine transformation on the designated grabbing points in the part source diagram through the affine transformation matrix to obtain pixel coordinates of the grabbing points in an actual image, and then carrying out space transformation on the pixel coordinates of the grabbing points through the coordinate transformation matrix to obtain coordinates of the grabbing points under a robot coordinate system.
4. A multi-feature fusion visual recognition system as recited in claim 3, further comprising:
the image preprocessing unit is used for preprocessing the part source diagram; the preprocessing comprises image graying, part edge extraction, binarization, morphological operation, part contour extraction and maximum and minimum rectangular frame fitting.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011504074.0A CN112560704B (en) | 2020-12-18 | 2020-12-18 | Visual identification method and system for multi-feature fusion |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011504074.0A CN112560704B (en) | 2020-12-18 | 2020-12-18 | Visual identification method and system for multi-feature fusion |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112560704A CN112560704A (en) | 2021-03-26 |
| CN112560704B true CN112560704B (en) | 2023-07-18 |
Family
ID=75063647
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011504074.0A Active CN112560704B (en) | 2020-12-18 | 2020-12-18 | Visual identification method and system for multi-feature fusion |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112560704B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114734448B (en) * | 2022-05-11 | 2023-11-07 | 上海交通大学 | Plate part grabbing method and device for ship material arranging equipment |
| CN115359112B (en) * | 2022-10-24 | 2023-01-03 | 爱夫迪(沈阳)自动化科技有限公司 | Stacking control method of high-level material warehouse robot |
| CN117409082B (en) * | 2023-12-13 | 2024-02-23 | 南昌工控机器人有限公司 | Visual identification and positioning method and device for modularized assembly of mobile phone shell |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104915957A (en) * | 2015-05-29 | 2015-09-16 | 何再兴 | Matching rectification method for improving three dimensional visual sense identification precision of industrial robot |
| CN110815213A (en) * | 2019-10-21 | 2020-02-21 | 华中科技大学 | Part identification and assembly method and device based on multi-dimensional feature fusion |
Family Cites Families (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP4757142B2 (en) * | 2006-08-10 | 2011-08-24 | キヤノン株式会社 | Imaging environment calibration method and information processing apparatus |
| JP6659098B2 (en) * | 2015-07-29 | 2020-03-04 | キヤノン株式会社 | Image processing method, image processing apparatus, program, and recording medium |
| CN107423770B (en) * | 2017-08-03 | 2021-03-26 | 广东顺德西安交通大学研究院 | Robot vision positioning method for high-speed production line |
| CN107563432A (en) * | 2017-08-29 | 2018-01-09 | 武汉交通职业学院 | A kind of robot multiple-objective recognition methods of view-based access control model shape |
| CN110110760A (en) * | 2019-04-17 | 2019-08-09 | 浙江工业大学 | A kind of workpiece positioning and recognition methods based on machine vision |
| CN110415363A (en) * | 2019-08-05 | 2019-11-05 | 上海神添实业有限公司 | A kind of object recognition positioning method at random based on trinocular vision |
| CN110935644B (en) * | 2019-12-27 | 2020-12-29 | 江南大学 | Bearing needle roller size detection system and method based on machine vision |
| CN111152224B (en) * | 2020-01-10 | 2022-05-10 | 温州大学 | Dual-optimization robot motion trajectory optimization method |
-
2020
- 2020-12-18 CN CN202011504074.0A patent/CN112560704B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104915957A (en) * | 2015-05-29 | 2015-09-16 | 何再兴 | Matching rectification method for improving three dimensional visual sense identification precision of industrial robot |
| CN110815213A (en) * | 2019-10-21 | 2020-02-21 | 华中科技大学 | Part identification and assembly method and device based on multi-dimensional feature fusion |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112560704A (en) | 2021-03-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112560704B (en) | Visual identification method and system for multi-feature fusion | |
| CN112070818B (en) | Robot disordered grabbing method and system based on machine vision and storage medium | |
| CN110580723B (en) | Method for carrying out accurate positioning by utilizing deep learning and computer vision | |
| CN108573221A (en) | A kind of robot target part conspicuousness detection method of view-based access control model | |
| CN111604909A (en) | Visual system of four-axis industrial stacking robot | |
| CN112233181A (en) | 6D pose recognition method and device and computer storage medium | |
| CN111862201A (en) | Deep learning-based spatial non-cooperative target relative pose estimation method | |
| CN115609591B (en) | Visual positioning method and system based on 2D Marker and compound robot | |
| CN111476841A (en) | Point cloud and image-based identification and positioning method and system | |
| CN111199556A (en) | Indoor pedestrian detection and tracking method based on camera | |
| Sommer et al. | Primitect: Fast continuous hough voting for primitive detection | |
| CN113822810A (en) | Method for positioning workpiece in three-dimensional space based on machine vision | |
| CN115816460A (en) | Manipulator grabbing method based on deep learning target detection and image segmentation | |
| CN112734844B (en) | Monocular 6D pose estimation method based on octahedron | |
| CN111161219B (en) | Robust monocular vision SLAM method suitable for shadow environment | |
| CN109410272B (en) | Transformer nut recognition and positioning device and method | |
| CN116188540A (en) | Target identification and pose estimation method based on point cloud information | |
| CN116188763A (en) | Method for measuring carton identification positioning and placement angle based on YOLOv5 | |
| CN113689365B (en) | Target tracking and positioning method based on Azure Kinect | |
| CN112614182B (en) | Deep learning-based method for identifying machining position, storage device and mobile terminal | |
| Tang et al. | A dulcimer keys positioning method based on improved Roberts operator | |
| CN110796019A (en) | Method and device for identifying and tracking spherical object in motion | |
| Chen et al. | Object recognition and localization based on kinect camera in complex environment | |
| Javadi et al. | Change detection in aerial images using a Kendall's TAU distance pattern correlation | |
| JP7191352B2 (en) | Method and computational system for performing object detection |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |