CN112926503A - Automatic captured data set generation method based on rectangle fitting - Google Patents

Automatic captured data set generation method based on rectangle fitting Download PDF

Info

Publication number
CN112926503A
CN112926503A CN202110307109.XA CN202110307109A CN112926503A CN 112926503 A CN112926503 A CN 112926503A CN 202110307109 A CN202110307109 A CN 202110307109A CN 112926503 A CN112926503 A CN 112926503A
Authority
CN
China
Prior art keywords
rectangle
fitting
image
data set
grabbing
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202110307109.XA
Other languages
Chinese (zh)
Other versions
CN112926503B (en
Inventor
李育文
张智辉
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
University of Shanghai for Science and Technology
Original Assignee
University of Shanghai for Science and Technology
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by University of Shanghai for Science and Technology filed Critical University of Shanghai for Science and Technology
Priority to CN202110307109.XA priority Critical patent/CN112926503B/en
Publication of CN112926503A publication Critical patent/CN112926503A/en
Application granted granted Critical
Publication of CN112926503B publication Critical patent/CN112926503B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/10—Terrestrial scenes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/23—Clustering techniques
    • G06F18/232—Non-hierarchical techniques
    • G06F18/2321—Non-hierarchical techniques using statistics or function optimisation, e.g. modelling of probability density functions
    • G06F18/23213—Non-hierarchical techniques using statistics or function optimisation, e.g. modelling of probability density functions with fixed number of clusters, e.g. K-means clustering
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
    • G06V10/267—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion by performing operations on regions, e.g. growing, shrinking or watersheds
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
    • Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
    • Y02P90/30—Computing systems specially adapted for manufacturing

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Artificial Intelligence (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Probability & Statistics with Applications (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Biology (AREA)
  • Evolutionary Computation (AREA)
  • General Engineering & Computer Science (AREA)
  • Image Analysis (AREA)

Abstract

The invention discloses a captured data set automatic generation method based on rectangular fitting, which comprises the steps of clamping the edge of an object by adopting an end effector of a mechanical arm, driving the object to rotate around the center of the object, and acquiring images of the object at various angles by a camera; determining an object region and generating an object mask; detecting contour information of a target object; carrying out Hough transform on the image, detecting straight lines in the object contour, and merging short line segments; sorting the linear lengths, detecting corresponding parallel lines according to the linear lengths, and fitting the object contour by using a plurality of rectangles; generating a plurality of grabbing rectangles suitable for the double-finger paw by the fitting rectangle according to an equidistant sampling mode; the background image and the object image are combined to generate a captured dataset image, while the corresponding label is generated. The invention has simple operation, convenient and quick realization and does not need additional equipment. The problem that manual labeling work of the data sets in the grabbing task wastes time and labor is solved, and a convenient way is provided for manufacturing the training set required by the deep learning model in the grabbing task.

Description

Automatic captured data set generation method based on rectangle fitting
Technical Field
The invention belongs to the field of robots, and particularly relates to a captured data set automatic generation method based on rectangle fitting.
Background
With the development of computer vision and artificial intelligence, robotics has received more and more attention, wherein intelligent robots are more considered as the mainstream development direction in the future. In recent years, various methods for detecting object grabbing postures are proposed in journals and meetings in the robot field, and particularly, a huge target grabbing data set with labeling information is required for training and testing a model for detecting scene object grabbing postures under a deep learning framework. The data set needs to have object image information and a corresponding plurality of capture poses.
In the field of robotics, there are available as grab test data sets Cornell grab lsp Dataset, Jacquard Dataset, VMRD, etc., which are manually labeled by researchers. Each sample in the data set may contain a plurality of objects, each object has a plurality of possible grabbing postures, manual production of the grabbing data set is time-consuming and labor-consuming and is influenced by human subjectivity, the marked grabbing data set has a certain tendency, and the marked grabbing positions only contain a part of available grabbing positions and cannot comprehensively represent the grabbing positions of the objects.
Disclosure of Invention
The invention aims to provide a captured data set automatic generation method based on rectangle fitting, aiming at the problems in the existing captured data set production process. The method is simple to operate, saves time and labor, and can finish the manufacturing work of capturing the database in a short time.
In order to achieve the purpose, the invention adopts the technical scheme that:
a method for automatically generating a captured data set based on rectangle fitting comprises the following steps:
firstly, acquiring an object image:
the acquisition of multi-angle image information of an object is completed in a mode of combining a mechanical arm and a camera;
step two, determining an object region, and generating an object mask:
according to the position relation among all parts in the data collection system, the approximate position of an object in an image is preliminarily determined, and most of background noise is removed; utilizing a background subtraction method to segment an object region from a background to obtain an object mask;
step three, detecting the contour information of the target object:
detecting the edge of the object by using a canny operator, and representing the outline of the object by a large number of short lines;
step four, detecting straight lines by Hough transform:
carrying out Hough transform on the object contour curve, detecting straight lines in the object contour, and integrating line segments meeting merging conditions to reduce the number of the lines;
step five, fitting the contour by using a plurality of rectangles:
sorting the straight lines according to the length of the straight lines, and sequentially detecting corresponding parallel lines according to the sorting of the length of the straight lines; the rectangle used to fit the object contour needs to satisfy four conditions simultaneously:
firstly, the included angle of the two line segments is less than 20 degrees;
② the distance between the parallel lines is larger than threshold th1＝50；
③, the number of pixels between the parallel lines is not zero;
fourthly, the shorter line segment has projection on the longer line segment;
mutually projecting parallel lines which simultaneously meet the conditions of the first to the fourth to generate a plurality of rectangles for fitting the outline of the object;
step six, generating a grabbing rectangle:
according to the finger width w, a plurality of grabbing rectangles suitable for the double-finger paw are generated on the fitting rectangle in an equidistant sampling mode;
step seven, synthesizing the captured data set image, and manufacturing a captured data set:
and combining the color image of the object with a plurality of preset backgrounds by using the object mask generated in the second step to generate a plurality of captured data set images, and simultaneously making label files for all samples in the data set according to the positions of the object in the images and the captured rectangles generated in the sixth step.
Preferably, the data acquisition in the first step is implemented by combining a mechanical arm and a camera, and the camera is mounted right above and vertically downward from the background desktop through a frame. For the convenience of later object segmentation, the background desktop is a single color, and white is selected here. And controlling the gripper to clamp the edge of the object and enabling the object to be positioned under the camera, driving the object to rotate respectively around X, Y, Z three coordinate axes of a gripper coordinate system by plus or minus 30 degrees, and acquiring an object image by the camera after the system is stabilized.
Preferably, in the second step, the relationship between the camera and the robot arm is used to calculate the coordinates (u, v) of the midpoint position (X, Y, Z) of the connecting line between the two fingertips in the robot arm coordinate system in the image through a kinematic forward solution, so as to determine the position of the object in the image:
Figure BDA0002987966230000021
wherein the content of the first and second substances,
Figure BDA0002987966230000022
representing the relation of the robot arm coordinate system with respect to the camera coordinate system, fx,fy,cx,cyIs the camera internal reference. Assuming that the target object is smaller than a cube with a side length of 20cm, the region of the target in the image can be determined by projecting the cube, and the smallest square that can contain the region is clipped, assuming a side length of e. The region only contains the target object and the background image, then the background is subtracted from the image, and the threshold th is setv10, wherein the portion of the difference value less than the threshold is the object region.
Preferably, in the third step, the object mask image obtained in the second step is subjected to edge detection, a Canny operator is used for detecting the edges of the object, and gradient values of all points in the image are calculated. If the point having the gradient value less than 100 is set as the non-boundary point, the point having the gradient value greater than 150 is set as the boundary point. The point in the interval judges the property according to whether the adjacent point is the boundary point, if the adjacent point is adjacent to the boundary point, the point is considered as the boundary point, and if the adjacent point is adjacent to the non-boundary point, the point is considered as the non-boundary point.
Preferably, in the fourth step, all points on the contour are mapped into a hough space through hough transformation, the midpoint of the hough space corresponds to a straight line in an image space, the number of straight lines corresponding to each intersection point in the hough space is counted, and the number of the straight lines is greater than a threshold thcPoint map ofStraight lines in image space, at thc30. And then calculating the included angle between any two line segments and the distance between the two adjacent nearest points of the two line segments to judge whether the two line segments can be merged, and if the included angle of the straight lines where the two line segments are located is less than 10 degrees and the distance between the nearest points is less than 20 pixels, determining that the two line segments are both part of the real line segment. And generating k line segment pairs which can be used for merging through pairwise matching. And then clustering the k line segment pairs, and clustering all the line segments with matching relations into one class. And calculating a minimum bounding rectangle formed by all the points on each type of straight line, wherein the middle point of two short sides in the connecting rectangle is used for representing the merged straight line.
Preferably, in the fifth step, all the lengths of the merged straight lines are sorted, and the rectangle is generated by preferentially using the longer straight line. Whether the rectangle used for fitting the object can be generated by the longer straight line and the residual straight lines is sequentially judged, and the rectangle of the fitting object needs to meet the following conditions:
the first line segment and the second line segment are parallel lines;
secondly, the distance between the parallel lines is larger than the threshold th1＝50；
Thirdly, parallel lines are not empty;
and fourthly, the projection of the shorter line segment on the longer line segment exists.
And establishing a fitting rectangle according to the conditions. Calculating included angle by using included angle residue between vectors
Figure BDA0002987966230000031
The distance between the midpoints of the two line segments is simplified to represent the distance between the two line segments; and respectively connecting two end points of one line segment with two end points of the other line segment to form four vectors, wherein if the vectors generated by the two end points of one line segment at least have one vector which forms an acute angle with the line segment, the two line segments have projections mutually. And connecting corresponding end points of the two line segments to form a quadrangle, and counting the number n of non-zero points in the quadrangle. If the four conditions cannot be simultaneously met, the next straight line is judged until the complete straight line is traversed. For the straight line meeting the conditions, calculating the projection point of the shorter line segment on the longer line segment,if no projection point exists, the side end point of the overlong line segment is perpendicular to the short linearity, and the intersection point is updated to be the end point of the short line segment. Calculating the middle point, the rotation angle, the width and the height of the fitting rectangle, wherein the horizontal and vertical coordinates of the middle point are the average values of the horizontal and vertical coordinates of four end points of the line segment respectively, and the rotation angle thetarThe average of the included angles formed by the two line segments and the horizontal line is shown, the width W is equal to the distance between the middle points of the line segments, and the height H is the shorter length of the line segments.
Preferably, in the sixth step, the finger width w is measured, and the finger projection width w' in the image is calculated according to the hand-eye relationship. And taking w'/2 as a step length, and sampling from one end of the rectangle to the other end of the rectangle along the direction of the middle edge H of the fitted rectangle to generate a plurality of grabbed rectangles. Width w of the grabbing rectanglegW' high
Figure BDA0002987966230000032
The axes of the grabbing rectangle and the fitting rectangle are mutually vertical. The central point of the grabbing rectangle is located on the axis of the fitting rectangle, and the grabbing rectangle is ensured not to exceed the range of the fitting rectangle.
Preferably, in the seventh step, both a monochrome background and a cluttered background are selected as the background. Cutting object region in the image by the object mask obtained in the step two, pasting the object region to the central position of the background image, and rotating the object mask by theta▽，θ▽Values from 0-360 ° are taken at 30 ° intervals, and a total of 2 × 12 — 24 samples are generated for a single object. For color images, the background is directly replaced by the object image, and when the mask value is true, IB(x,y)＝IF(x, y), and the values of other positions are kept unchanged. For depth images, the value of the object region is subtracted from the value of the background, and when the mask value is true, IB(x,y)＝IB(x,y)-IF(x, y), and the values of other positions are kept unchanged. Calculating the grabbing rectangle (c ') of the object in the composite image according to the position and the angle of the object in the background image and the grabbing rectangle obtained in the step six'x,c'y,w'g,h'g,θ'g) Wherein
Figure BDA0002987966230000041
The abscissa representing the center point of the captured rectangle in the composite image,
Figure BDA0002987966230000042
ordinate, w 'representing center point'g＝wgDenotes the width of the finger in the grabbing rectangle, h'g＝hgDenotes the size of the opening of the gripper rectangle, θ'g＝θg+θ▽Representing the angle of the grabbed rectangle to the horizontal axis of the image.
Compared with the prior art, the invention has the following obvious prominent substantive characteristics and obvious advantages:
1. the method is simple and time-saving in the manufacturing operation of the captured data set, only the object needs to be clamped manually in the middle, excessive manual intervention is not needed, and the manufacturing work of the captured data set can be completed in the shortest time;
2. the method is simple, high in efficiency and easy to implement.
Drawings
FIG. 1 is a flow chart of the present invention.
FIG. 2 is a schematic diagram of a data acquisition system of the present invention.
FIG. 3 is a schematic diagram of the calculation of the fitted rectangle of the present invention.
FIG. 4 is a sample diagram of a capture frame according to the present invention.
Fig. 5 is a diagram illustrating the object segmentation result according to the present invention.
Fig. 6 is a schematic diagram of the contour detection of the present invention.
FIG. 7 is a graphical representation of the results of the rectangle fitting of the present invention.
FIG. 8 is a schematic diagram illustrating the capturing of a sample of a data set according to the present invention.
Detailed Description
The invention is further illustrated by the following figures and examples:
the first embodiment is as follows:
referring to fig. 1, a method for automatically generating a captured data set based on rectangle fitting includes the following steps:
firstly, acquiring an object image:
the acquisition of multi-angle image information of an object is completed in a mode of combining a mechanical arm and a camera;
step two, determining an object region, and generating an object mask:
according to the position relation among all parts in the data collection system, the approximate position of an object in an image is preliminarily determined, and most of background noise is removed; utilizing a background subtraction method to segment an object region from a background to obtain an object mask;
step three, detecting the contour information of the target object:
detecting the edge of the object by using a canny operator, and representing the outline of the object by a large number of short lines;
step four, detecting straight lines by Hough transform:
carrying out Hough transform on the object contour curve, detecting straight lines in the object contour, and integrating line segments meeting merging conditions to reduce the number of the lines;
step five, fitting the contour by using a plurality of rectangles:
sorting the straight lines according to the length of the straight lines, and sequentially detecting corresponding parallel lines according to the sorting of the length of the straight lines; the rectangle used to fit the object contour needs to satisfy four conditions simultaneously:
firstly, the included angle of the two line segments is less than 20 degrees;
② the distance between the parallel lines is larger than threshold th1＝50；
③, the number of pixels between the parallel lines is not zero;
fourthly, the shorter line segment has projection on the longer line segment;
mutually projecting parallel lines which simultaneously meet the conditions of the first to the fourth to generate a plurality of rectangles for fitting the outline of the object;
step six, generating a grabbing rectangle:
according to the finger width w, a plurality of grabbing rectangles suitable for the double-finger paw are generated on the fitting rectangle in an equidistant sampling mode;
step seven, synthesizing the captured data set image, and manufacturing a captured data set:
and combining the color image of the object with a plurality of preset backgrounds by using the object mask generated in the second step to generate a plurality of captured data set images, and simultaneously making label files for all samples in the data set according to the positions of the object in the images and the captured rectangles generated in the sixth step.
The method is simple and time-saving in the manufacturing operation of the captured data set, only the object needs to be clamped manually in the middle, excessive manual intervention is not needed, and the manufacturing work of the captured data set can be completed in the shortest time.
Example two:
this embodiment is substantially the same as the first embodiment, and is characterized in that:
in the first step, a fixed Kinect camera is used as data acquisition equipment, the posture of the object is changed by clamping the object through a mechanical arm tail end paw, and image information of the object under various angles is acquired.
In the fifth step, four conditions for forming the rectangle are utilized to detect whether any two parallel lines can be used for synthesizing the fitting rectangle, and the fitting rectangle is established by the two parallel lines in a mutual projection mode.
In the sixth step, the projection w of the finger width is taken as the width of the grabbing rectangle, H of the fitting rectangle which is 6/5 times is taken as the height H of the grabbing rectangle, and sampling is carried out along the H side of the fitting rectangle at the interval of 0.5w to construct the grabbing rectangle.
And 5, in the seventh step, combining the object image and the background image to form a captured data set image, and generating captured parameters according to the position of the object image in the background.
The automatic captured data set generating method based on the rectangular fitting is simple to operate, time-saving and labor-saving, and can complete the manufacturing work of the captured database in a short time.
Example three:
referring to fig. 1 to 8, a method for automatically generating a captured data set based on rectangle fitting includes the following steps:
step one, a data collection system is built according to the structure shown in fig. 2, a camera is installed right above a background desktop and vertically downward through a frame, and a mechanical arm is fixed on the desktop. In order to facilitate later object segmentation, a white desktop background is selected. The camera is connected with the computer through an L-to-USB data line, and the mechanical arm controller is connected with the computer through a twisted pair. And powering on all the devices after all the hardware is successfully connected, and initializing the data acquisition system. And the gripper of the manual control mechanical arm clamps the edge of the object to be collected and drags the edge to the position right below the camera, the object is driven to rotate respectively around X, Y, Z coordinate axes of a gripper coordinate system by plus or minus 30 degrees, and the camera collects an object image after the system is stabilized.
And secondly, establishing a coordinate system at the midpoint position of a connecting line of the two fingertips, and calculating the projection point of the fingertips in the image by using the pose relation between the camera and the mechanical arm so as to determine the approximate position of the object. A rectangle with a side length e of 50 pixels is created parallel to the image edges, and the projected point of the fingertip is located at 3/4 of the rectangle axis, so that the rectangle bounds the area where the object is located. Only the target object and the background image are included in the area, then the background is subtracted from the image, and a threshold th is setv10, where the portion of the difference value less than the threshold is the object region, the effect is shown in fig. 5.
And step three, detecting the edge of the object by utilizing a cvCanny () function provided in OpenCV for the object mask image obtained in the step two. The mask image is input as a function, and the parameters threshold1 is set to 100, threshold2 is set to 150, and alert _ size is set to 3. A two-dimensional matrix of the same size as the input is output, the object contour position is represented by '1', and the other portions are represented by '0', and the effect is shown in fig. 6.
And step four, detecting a point set which is possibly a straight line in the contour by using a probability Hough transform function Hough LinesP () provided in OpenCV. And taking the output in the third step as an input, setting the search step rho to be 2, the search radian interval theta to be pi/180, the accumulated point number threshold to be 30, the line segment maximum interval maxLineGap to be 10 and the line segment minimum length to be 100. The function output is a two-dimensional matrix L consisting of a plurality of line segments, the first dimension represents the number of detected line segments, and the second dimension is the horizontal and vertical coordinates of two end points of each line segment. To simplify the amount of computation in the line segment integration, the line segments are first clustered. Firstly, initializing a set S for storing line segments used by the integrated straight line segments, sequentially extracting one line segment from all detected line segments, and deleting the line segment from L. And judging whether a line segment capable of being merged with the line segment exists in the S, if so, adding the line segment into a corresponding line segment set, otherwise, establishing a new line segment set until all detected line segments are traversed. And calculating a minimum bounding rectangle formed by all the points on each type of straight line, wherein the middle point of two short sides in the connecting rectangle is used for representing the merged straight line.
And step five, calculating the lengths of all the merged straight lines, and sequencing the straight lines by using a quick sequencing algorithm. Preferentially using a longer straight line to generate a rectangle, and sequentially judging whether the longer straight line and the residual straight lines can generate a rectangle for fitting an object, wherein the rectangle for fitting the object needs to meet the following conditions: the included angle of the first line segment and the second line segment is less than 20 degrees; secondly, the distance between the parallel lines is larger than the threshold thl50; thirdly, the number of pixels between parallel lines is not zero; and fourthly, the projection of the shorter line segment on the longer line segment exists. And establishing a fitting rectangle according to the conditions. Let l be the long line of any given two straight lines1Short line is l2，l1Is p at the left end point of1The right end point is p2，l2Is p at the left end point of3The right end point is p4As shown in fig. 3. Connection point p1、p2Form vector V1Point of attachment p3、p4Form vector V2. Calculating the included angle between two vectors
Figure BDA0002987966230000071
By calculating l2Midpoint to l1The distance d of (a) represents the distance between two line segments. Respectively combine p with3、p4Point and p1、p2The point connection forms a vector V13、V14、V23、V24If V is13、V14In which at least one vector and a vector V are present1The included angle is acute and V23、V24In which at least one vector and a vector V are present1The included angle is acute angle, which indicates the angle between two line segmentsHave projections to each other. Connecting point p in sequence1、p2、p4、p3、p1And forming a quadrangle, and counting the number n of non-zero points in the quadrangle. If the four conditions cannot be simultaneously met, the next straight line is judged until the complete straight line is traversed. For a eligible line, calculate point p3、p4In the vector V1Projected point p 'on'3、p'4Abscissa of projection point
Figure BDA0002987966230000072
Ordinate of the curve
Figure BDA0002987966230000073
Where i ∈ {3,4}, α1Represents a vector V1The included angle between the straight line and the horizontal line. If p'3In the vector V1Above, then p'1P 3. Otherwise, a passing point p is established1And vector V2Straight line perpendicular to V2The intersection of the straight lines is point p'1，p'3＝p1. Similarly, if p'4In the vector V1Above, then p'2＝p4. Otherwise, a passing point p is established2And vector V2Straight line perpendicular to V2The intersection of the straight lines is point p'2，p'4＝p2. Calculating the midpoint (x, y) of the fitted rectangle, wherein
Figure BDA0002987966230000074
Figure BDA0002987966230000075
Calculating the rotation angle of the fitted rectangle
Figure BDA0002987966230000076
Calculating the width W of the fitted rectangle to be equal to the point p'1And p'2From midpoint to point p'3And p'4Calculating the fitted rectangular height H as a point p'1And p'2Distance between and point p'3And p'4The smaller the distance between them. The fitting is finally done by a rectangle, as shown in fig. 7.
And step six, measuring the width w of the finger, and calculating the projection width w' of the finger in the image according to the hand-eye relationship. And with w'/2 as a step size, sampling from the P1 end of the rectangle to the P2 end of the rectangle along the direction of the side H in the fitted rectangle to generate a plurality of grabbed rectangles, as shown in FIG. 4. Width w of the grabbing rectanglegW' high
Figure BDA0002987966230000081
When theta isrRotation angle theta of grabbing rectangle with timingg＝θr-90 ° when θrTheta when the value is negativeg＝θr+90 °. Capturing the abscissa c of the center point of the rectanglex＝xP1+k·wgcos(θr) Ordinate cy＝yP1+k·wgsin(θr) Wherein k is 0.5,1,1.5,2 ….
And step seven, selecting a single-color background and a cluttered background as the background, wherein the size of the background is 300 x 300. And D, cutting the object area in the image by using the object mask obtained in the step two, and obtaining a matrix with numerical values only in the object area and 0 in other parts. The image is rotated using the warpAffine () function, the rotation center is set to the center of the image, the boundary fill value is 0, and the rotation angle θ is▽At intervals of 30 from 0-360 deg. And adding the rotated image and the corresponding element of the background image, and generating 2 × 12 to 24 samples in the single object. Calculating the grabbing rectangle (c ') of the object in the composite image according to the position and the angle of the object in the background image and the grabbing rectangle obtained in the step six'x,c'y,w'g,h'g,θ'g) Wherein
Figure BDA0002987966230000082
The abscissa representing the center point of the captured rectangle in the composite image,
Figure BDA0002987966230000083
ordinate, w 'representing center point'g＝wgIndicating the width of the finger in the grabbing rectangleDegree, h'g＝hgDenotes the size of the opening of the gripper rectangle, θ'g＝θg+θ▽Representing the angle of the grab rectangle to the horizontal axis of the image, an example of the obtained grab data set is shown in fig. 8.
In the embodiment, the automatic captured data set generation method based on rectangular fitting utilizes the mechanical arm end effector to clamp the edge of an object, drives the object to rotate around the center of the object, and acquires images of the object at various angles through a camera; determining an object region and generating an object mask; detecting contour information of a target object; carrying out Hough transform on the image, detecting straight lines in the object contour, and merging short line segments; sorting the linear lengths, detecting corresponding parallel lines according to the linear lengths, and fitting the object contour by using a plurality of rectangles; generating a plurality of grabbing rectangles suitable for the double-finger paw by the fitting rectangle in an equidistant sampling mode; the background image and the object image are combined to generate a captured dataset image, while the corresponding label is generated. The method is simple to operate, convenient and quick to implement, and does not need additional equipment. The problem that manual labeling work of the data sets in the grabbing task wastes time and labor is solved, and a convenient way is provided for manufacturing the training set required by the deep learning model in the grabbing task.
The present invention is not limited to the above embodiments, and all changes, substitutions and simplifications made according to the principles of the present invention are equivalent substitutions, or directly or indirectly applied to other related fields, which are included in the protection scope of the present application.

Claims (5)

1. A captured data set automatic generation method based on rectangle fitting is characterized by comprising the following steps:
firstly, acquiring an object image:
the acquisition of multi-angle image information of an object is completed in a mode of combining a mechanical arm and a camera;
step two, determining an object region, and generating an object mask:
according to the position relation among all parts in the data collection system, the approximate position of an object in an image is preliminarily determined, and most of background noise is removed; utilizing a background subtraction method to segment an object region from a background to obtain an object mask;
step three, detecting the contour information of the target object:
detecting the edge of the object by using a canny operator, and representing the outline of the object by a large number of short lines;
step four, detecting straight lines by Hough transform:
carrying out Hough transform on the object contour curve, detecting straight lines in the object contour, and integrating line segments meeting merging conditions to reduce the number of the lines;
step five, fitting the contour by using a plurality of rectangles:
sorting the straight lines according to the length of the straight lines, and sequentially detecting corresponding parallel lines according to the sorting of the length of the straight lines; the rectangle used to fit the object contour needs to satisfy four conditions simultaneously:
firstly, the included angle of the two line segments is less than 20 degrees;
② the distance between the parallel lines is larger than threshold th1＝50；
③, the number of pixels between the parallel lines is not zero;
fourthly, the shorter line segment has projection on the longer line segment;
mutually projecting parallel lines which simultaneously meet the conditions of the first to the fourth to generate a plurality of rectangles for fitting the outline of the object;
step six, generating a grabbing rectangle:
according to the finger width w, a plurality of grabbing rectangles suitable for the double-finger paw are generated on the fitting rectangle in an equidistant sampling mode;
step seven, synthesizing the captured data set image, and manufacturing a captured data set:
and combining the color image of the object with a plurality of preset backgrounds by using the object mask generated in the second step to generate a plurality of captured data set images, and simultaneously making label files for all samples in the data set according to the positions of the object in the images and the captured rectangles generated in the sixth step.
2. The method for automatically generating a grabbed data set based on rectangle fitting as claimed in claim 1, wherein: in the first step, a fixed Kinect camera is used as data acquisition equipment, the posture of the object is changed by clamping the object through a mechanical arm tail end paw, and image information of the object under various angles is acquired.
3. The method for automatically generating a grabbed data set based on rectangle fitting as claimed in claim 1, wherein: in the fifth step, four conditions for forming the rectangle are utilized to detect whether any two parallel lines can be used for synthesizing the fitting rectangle, and the fitting rectangle is established by the two parallel lines in a mutual projection mode.
4. The method for automatically generating a grabbed data set based on rectangle fitting as claimed in claim 1, wherein: in the sixth step, the projection w of the finger width is taken as the width of the grabbing rectangle, H of the fitting rectangle which is 6/5 times is taken as the height H of the grabbing rectangle, and sampling is carried out along the H side of the fitting rectangle at the interval of 0.5w to construct the grabbing rectangle.
5. The method for automatically generating a grabbed data set based on rectangle fitting as claimed in claim 1, wherein: and in the seventh step, combining the object image and the background image to form a captured data set image, and generating captured parameters according to the position of the object image in the background.
CN202110307109.XA 2021-03-23 2021-03-23 Automatic generation method of grabbing data set based on rectangular fitting Active CN112926503B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110307109.XA CN112926503B (en) 2021-03-23 2021-03-23 Automatic generation method of grabbing data set based on rectangular fitting

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110307109.XA CN112926503B (en) 2021-03-23 2021-03-23 Automatic generation method of grabbing data set based on rectangular fitting

Publications (2)

Publication Number Publication Date
CN112926503A true CN112926503A (en) 2021-06-08
CN112926503B CN112926503B (en) 2023-07-18

Family

ID=76175527

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110307109.XA Active CN112926503B (en) 2021-03-23 2021-03-23 Automatic generation method of grabbing data set based on rectangular fitting

Country Status (1)

Country Link
CN (1) CN112926503B (en)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113506314A (en) * 2021-06-25 2021-10-15 北京精密机电控制设备研究所 Automatic grabbing method and device for symmetrical quadrilateral workpiece under complex background
CN114428876A (en) * 2021-12-29 2022-05-03 广州盖盟达工业品有限公司 Image searching method, device, storage medium and equipment for industrial apparatus
WO2023092519A1 (en) * 2021-11-28 2023-06-01 梅卡曼德（北京）机器人科技有限公司 Grabbing control method and apparatus, and electronic device and storage medium
CN116416217A (en) * 2023-03-06 2023-07-11 赛那德科技有限公司 Method, system and equipment for generating unordered stacking parcel image

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105405126A (en) * 2015-10-27 2016-03-16 大连理工大学 Multi-scale air-ground parameter automatic calibration method based on monocular vision system
CN108460799A (en) * 2018-01-26 2018-08-28 中国地质大学（武汉） A kind of Step wise approximation sub-pix image position method and system
CN108656107A (en) * 2018-04-04 2018-10-16 北京航空航天大学 A kind of mechanical arm grasping system and method based on image procossing
CN109948712A (en) * 2019-03-20 2019-06-28 天津工业大学 A kind of nanoparticle size measurement method based on improved Mask R-CNN
CN111507390A (en) * 2020-04-11 2020-08-07 华中科技大学 Storage box body identification and positioning method based on contour features

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105405126A (en) * 2015-10-27 2016-03-16 大连理工大学 Multi-scale air-ground parameter automatic calibration method based on monocular vision system
CN108460799A (en) * 2018-01-26 2018-08-28 中国地质大学（武汉） A kind of Step wise approximation sub-pix image position method and system
CN108656107A (en) * 2018-04-04 2018-10-16 北京航空航天大学 A kind of mechanical arm grasping system and method based on image procossing
CN109948712A (en) * 2019-03-20 2019-06-28 天津工业大学 A kind of nanoparticle size measurement method based on improved Mask R-CNN
CN111507390A (en) * 2020-04-11 2020-08-07 华中科技大学 Storage box body identification and positioning method based on contour features

Non-Patent Citations (6)

* Cited by examiner, † Cited by third party
Title
LSL WONG等: "data association for semantic world modeling from partial views", 《INTERNATIONAL JOURNAL OF ROBOTICS RESEARCH》 *
LSL WONG等: "data association for semantic world modeling from partial views", 《INTERNATIONAL JOURNAL OF ROBOTICS RESEARCH》, 30 June 2015 (2015-06-30), pages 1064 - 1082 *
卢军等: "基于改进Hu矩和矩形拟合的芯片识别定位算法", 《包装工程》 *
卢军等: "基于改进Hu矩和矩形拟合的芯片识别定位算法", 《包装工程》, 16 February 2018 (2018-02-16), pages 151 - 156 *
陆艺等: "基于机器视觉的移动工件抓取和装配的研究", 《计算机测量与控制》 *
陆艺等: "基于机器视觉的移动工件抓取和装配的研究", 《计算机测量与控制》, 31 July 2015 (2015-07-31), pages 1 - 4 *

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113506314A (en) * 2021-06-25 2021-10-15 北京精密机电控制设备研究所 Automatic grabbing method and device for symmetrical quadrilateral workpiece under complex background
CN113506314B (en) * 2021-06-25 2024-04-09 北京精密机电控制设备研究所 Automatic grabbing method and device for symmetrical quadrilateral workpieces under complex background
WO2023092519A1 (en) * 2021-11-28 2023-06-01 梅卡曼德（北京）机器人科技有限公司 Grabbing control method and apparatus, and electronic device and storage medium
CN114428876A (en) * 2021-12-29 2022-05-03 广州盖盟达工业品有限公司 Image searching method, device, storage medium and equipment for industrial apparatus
CN116416217A (en) * 2023-03-06 2023-07-11 赛那德科技有限公司 Method, system and equipment for generating unordered stacking parcel image
CN116416217B (en) * 2023-03-06 2023-11-28 赛那德科技有限公司 Method, system and equipment for generating unordered stacking parcel image

Also Published As

Publication number Publication date
CN112926503B (en) 2023-07-18

Similar Documents

Publication Publication Date Title
CN112926503A (en) Automatic captured data set generation method based on rectangle fitting
CN111340797B (en) Laser radar and binocular camera data fusion detection method and system
WO2018214195A1 (en) Remote sensing imaging bridge detection method based on convolutional neural network
CN111062915A (en) Real-time steel pipe defect detection method based on improved YOLOv3 model
CN111553949B (en) Positioning and grabbing method for irregular workpiece based on single-frame RGB-D image deep learning
CN110910350B (en) Nut loosening detection method for wind power tower cylinder
CN112509063A (en) Mechanical arm grabbing system and method based on edge feature matching
CN112906797A (en) Plane grabbing detection method based on computer vision and deep learning
CN111209864B (en) Power equipment target identification method
CN109708658B (en) Visual odometer method based on convolutional neural network
CN115816460B (en) Mechanical arm grabbing method based on deep learning target detection and image segmentation
CN114037703B (en) Subway valve state detection method based on two-dimensional positioning and three-dimensional attitude calculation
CN114882109A (en) Robot grabbing detection method and system for sheltering and disordered scenes
CN115272204A (en) Bearing surface scratch detection method based on machine vision
CN113327298A (en) Grabbing attitude estimation method based on image instance segmentation and point cloud PCA algorithm
CN111428815A (en) Mechanical arm grabbing detection method based on Anchor angle mechanism
CN114841923A (en) High-precision real-time crack detection method based on unmanned aerial vehicle
CN115830359A (en) Workpiece identification and counting method based on target detection and template matching in complex scene
CN114812398A (en) High-precision real-time crack detection platform based on unmanned aerial vehicle
CN115147380A (en) Small transparent plastic product defect detection method based on YOLOv5
CN115984186A (en) Fine product image anomaly detection method based on multi-resolution knowledge extraction
WO2022148091A1 (en) Target matching method and device, and robot
Jiang et al. Vision-guided active tactile perception for crack detection and reconstruction
CN114331961A (en) Method for defect detection of an object
CN115861780B (en) Robot arm detection grabbing method based on YOLO-GGCNN

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant