CN116188571A - A regular polygonal prism detection method for manipulator - Google Patents

Abstract

The invention discloses a regular polygon prism detection method for a mechanical arm, which comprises the following steps: step one: obtaining a top view image of the operation table; step two: obtaining an object mask, and step three: obtaining object edges, and step four: for each object, denoising the edges obtained in the third step by using the mask obtained in the second step, wherein in the fifth step, whether a circle exists or not is judged, if the circle exists, an upper surface circle is obtained, and the step is jumped to a step nine, otherwise, the step six is carried out; step six: obtaining the information of the straight line where the edge is located from the edge detection result; step seven: selecting an edge straight line of the upper surface of the object; step eight: shortening the straight line into an object edge line segment; step nine: calculating the shape, centroid and angle of the object. The invention can accurately obtain the edge of the upper surface of the regular polygon prism object, and obtain the suction position, the object shape and the object angle required by the placing task, thereby helping the placing task to be successfully completed.

Description

A regular polygonal prism detection method for manipulator

technical field

The invention belongs to the technical field of mechanical arms, and in particular relates to a regular polygonal prism detection method for a mechanical arm to accurately detect the edge, shape, centroid and angle of the upper surface of the regular polygonal prism.

Background technique

A robotic arm usually refers to a complex system with high precision, multiple input and multiple output, highly nonlinear, and strong coupling. Because of its unique operational flexibility, it has been widely used in industrial assembly, safety explosion-proof and other fields. When the manipulation target object is always at a fixed position, fixed, repetitive position control can be used to make the robotic arm perform a pick-and-place step at a fixed position. When the position of the target object to be assembled changes, the robotic arm needs to have the ability to sense, automatically locate the object, and complete the assembly task.

Add perception capabilities to the robotic arm, use sensors on the hardware to obtain external information, and calculate the required information based on the obtained data. For example, patent application 202120538755.2 discloses a distribution network live working robot system, including: an operation table device, a support device and a control cabinet; the operation table device includes an operation table, a mechanical arm assembly and a sensor assembly; on the platform; the mechanical arm assembly includes a first number of mechanical arms and a mechanical arm base; the mechanical arm base is fixed on the operating table, and the mechanical arm is fixed on the mechanical arm base; the supporting device includes an insulating bucket and a supporting frame; On the insulating bucket; the operating table is slidingly set on the supporting frame; a chamber is opened on the insulating bucket, and the control cabinet is installed in the chamber; a processor and a communication module are installed in the control cabinet; the mechanical arm and the sensor assembly are connected to the processor, The processor is connected with the communication module.

In the assembly task, to provide perception capabilities for the robotic arm, the RGB camera can be used as a sensor to obtain the image of the current object to be assembled, and analyze the information required for picking and placing the object, such as the position, angle and shape of the target object, and finally make the robotic arm Perform a placement action. Whether the assembly task can be successfully completed depends on the accuracy of the position, angle,

The process of analyzing the position, angle and shape of the target object. In terms of implementation, target detection or instance segmentation models can be used. However, these two types of models have their own shortcomings. Most of the detection frames output by the target detection model are horizontal or vertical, and there is no tilt angle, which cannot reflect the object angle well, and it is difficult to meet the assembly requirements; the segmentation results of the instance segmentation model are often The actual edge of the object cannot be precisely fitted, and the vertex of the object is not clear enough, resulting in the inability to accurately obtain the center of gravity of the object, which affects the success rate of the robotic arm assembly. In addition, when the object has a certain height, there is an object side surface in the input image, and both types of models will be affected by this. The current technology cannot accurately filter out the upper surface edge of the object to be assembled, making it difficult to obtain the accurate angle and center of gravity of the object, which affects the success rate of the assembly task. Therefore, improvement is badly needed.

Contents of the invention

Based on this, the primary purpose of the present invention is to provide a regular polygonal prism detection method for a robotic arm, which can provide perception capabilities for the robotic arm, overcome the shortcomings of the current target detection model and instance segmentation model, and obtain the placement task The required suction position, object shape and object angle can help the placement task to be completed smoothly.

Another object of the present invention is to provide a regular polygonal prism detection method for a robot arm, which can obtain the edge of the upper surface of the object and eliminate the influence of the side surface. Use the saturation S channel of the HSV color space to do adaptive threshold segmentation to obtain the object mask; then further perform grabcut segmentation in the Lab and YCrCb color spaces to obtain the upper surface mask of the object as much as possible; at the same time, integrate the Lab color space The edge detection of the L component and the V component in the HSV color space can obtain a more complete upper surface edge as much as possible to ensure the accuracy of detection.

To achieve the above object, the technical solution of the present invention is:

A regular polygonal prism detection method for a mechanical arm, characterized in that it comprises the following steps:

Step 1: Obtain the top view image of the console, use the RGB camera to take pictures of the table top and the objects to be assembled on the table from a top view angle, and obtain the picture of the console;

In order to distinguish the operating table and the objects to be assembled, they are marked with different colors; the operating table is designed in white, and the objects to be assembled are in other colors.

Step 2: Obtain the object mask,

Wherein, obtaining the mask of the object refers to obtaining at least a mask of the upper surface of the object.

Specifically include the following steps:

S21. Transfer the image to the HSV color space, and use the saturation S channel therein to perform adaptive threshold segmentation. Since the saturation of the object to be assembled is higher than that of the white table, an object mask will be obtained;

S22. Denoising, using operations such as expansion, erosion, and removal of overly large connected domains to remove part of the noise;

S23, then input the mask after denoising and the countertop picture transferred to Lab color space into grabcut together, so that the module detection process is fully automated, and the image transferred to Lab color space can obtain better upper surface segmentation than RGB color space Effect;

S24. Next, check whether the area of the connected domain in the segmentation mask is too large. If it is too large, it is suspected that there is mask adhesion of multiple objects. Transfer the image to the YCrCb space and do a grabcut again and use it as the upper surface mask of the object. , otherwise directly use the segmentation result in the current Lab color space as the upper surface mask of the object.

Step 3: Detect the edge of the object in the image.

Using Canny edge detection, the edge binary image is obtained. Due to the small change in color on both sides of the side and side edges of some objects, it is difficult to detect this edge. In order to solve this problem, the law that the upper surface and side surface mainly change in brightness is relatively large, respectively in the Lab color space Edge detection is performed on the L component of the HSV color space and the V component in the HSV color space, and then the detection results obtained by the two components are combined.

Step 4: Detect the edge of the object in the image and use the mask obtained in step 2 to denoise. In this step, it is hoped to obtain an edge detection result with as little noise as possible. The removed noise includes two kinds of noise inside and outside the edge of the object. Dilation operation is performed on the mask, and the intersection with the edge detection result is used to remove external edge noise; then the mask is etched, and the difference with the edge detection result is used to remove internal noise.

Further, the mask noise is to expand the mask and intersect with the edge detection result to obtain the edge detection result that removes the external noise of the object; then perform an erosion operation on the mask and calculate the difference with the edge detection result, The edge detection result of removing the internal noise of the object is obtained.

Step five: judge whether there is a circle.

Determine whether the edge image of the current target object contains a circle, if so, go to step 8, otherwise go to step 6. Specifically, use Hough circle detection to detect whether there is a circle in the intersection result obtained by applying the object mask to the Lab color space. If so, obtain the most suitable upper surface circle and enter step nine, otherwise enter step six. Among them, the process of obtaining the upper surface circle is to use Hough circle detection in the images of HSV and Lab color spaces respectively, adaptively find the circle with the largest param2 in the color space, and then use the circle with the outer center as the upper surface circle . Among them, param2 is the input parameter of the Hough circle detection method, which indicates the regularity of the circle. Only shapes larger than param2 can be detected as circles. The outer circle is selected as the upper surface circle, based on the fact that the upper surface is always relatively close to the periphery of the picture in the top view.

Step 6: Obtain the information of the straight line where the edge is located from the edge detection result.

The edge detection result uses the Hough line detection technology to adaptively obtain the line analysis formula of the line segment whose length is greater than th1 in the current edge detection image.

The specific process is as follows:

S41. Obtain the line segment of th=th1;

S42, detecting a line segment whose length is greater than th;

S43, judging whether a straight line is detected; if no straight line is detected, continue to the next step; if yes, make th+=4, and return to S42;

S44, let th-=1;

S45. Further judge that th>th1, if yes, continue to the next step, otherwise end;

S46, detecting a line segment whose length is greater than th;

S47. Determine whether a straight line is detected; if a straight line is detected, record the straight line information and th, and delete the detected straight line in the original image; if not, return to step S44, and set th-=1.

After deleting the detected straight lines, execute the above steps in a loop until all detections are completed.

Through this process, the straight line information of all line segments with a length greater than th1 will be recorded according to the reverse order of the length of the line segment. The information includes the inclination angle, the distance from the straight line to the origin of the image, and the length threshold th at the time of detection.

Step 7: Screen out the straight line on the edge of the upper surface of the object. This step is to eliminate the interference of the edge of the side surface of the object, the edge of the bottom surface and other noises, and select the straight line that best fits the actual edge of the upper surface of the object.

Carry out three operations of line grouping, line denoising and line selection in sequence.

Straight line grouping: group all detected straight lines according to angle, and group all straight lines whose angle difference is within th2, and each group contains at least one straight line;

Further, almost parallel straight lines with a distance greater than th3 are divided into different groups, and th3 needs to be adjusted so that the parallel edges on the upper surface of the object can be divided into two groups, and the parallel upper and lower surface edges can be combined into the same group. The rule for judging that straight lines are almost parallel is that there is no intersection point between two straight lines within the scope of the screen.

Straight line denoising: Mainly remove the side edge line and other noises. According to the short side edge length and the short noise line in the top view, remove the straight line. The judgment rule of the straight line is that there is only one straight line in this group and the line segment is recognized at that time. The length is lower than th4.

Select a straight line: Select the upper surface straight line from the set of remaining upper and lower surface edge straight lines; use the upper surface straight line to compare the lower surface straight line, and compare the difference between the object framed by each straight line and the mask, for each The group selects the straight line with the smallest difference, which is considered to be the edge of the upper surface.

Furthermore, when selecting a straight line, it is necessary to first construct a closed frame for each straight line, and then compare the difference between the closed range and the mask; when constructing a closed frame, match each straight line with the line closest to the center of the mask in other straight line groups to form a closed box; compare all the pixels in the closed box A with all the pixels in the mask B of grabcut, and record the number n of pixels that are in A but not in B for each line; after that, select a minimum n in each group The straight line is regarded as the upper surface edge straight line.

Step 8: The straight line is shortened to the edge segment of the object. Since the straight line information obtained in the fourth step does not include endpoints, it is necessary to obtain the intersection point of the edge lines in this step, that is, the vertex of the upper surface of the object, and obtain a line segment that fits the edge of the object.

To this end, iterate through the pairwise matching of all current straight lines, and calculate the minimum distance from the intersection point of the two straight lines to the mask. If the distance is less than th5, the two straight lines are considered to be adjacent. Among them, the way to calculate the distance from the point to the mask is to draw a circle with gradually increasing radius with the point as the center. When the circle starts to intersect with the mask, the radius of the circle is the distance from the point to the mask. After obtaining the adjacency relationship of all straight lines, calculate the vertices of the adjacent straight lines and connect the vertices in turn to obtain the edge of the upper surface of the object.

Nine: Calculate the shape, centroid, and angle of an object. According to the one-to-one relationship between the shape of a regular polygonal prism and the number of edges, the shape can be obtained from the number of vertices; the centroid can be calculated according to the closed figure surrounded by the upper surface edge; the angle of the object is determined by the rotation angle of the smallest circumscribed rectangle on the upper surface calculated.

Compared with prior art, the beneficial effect of the present invention is:

The invention utilizes the conversion of color space and the fusion of multi-color space information to solve the problem that the segmentation and detection effect in the RGB color space is not good enough. Specifically, for image segmentation, use the S component of HSV to segment the initial mask, and then further perform grabcut segmentation in the Lab and YCrCb color spaces according to the situation, to obtain the upper surface mask of the object as much as possible. For edge detection, the edge detection of the L component in the Lab color space and the V component in the HSV color space can be used to obtain a more complete upper surface edge as much as possible, which can accurately obtain accurate edges, improve the accuracy of detection, and obtain swing The suction position, object shape and object angle required by the placement task can help the placement task to be completed smoothly.

Description of drawings

Figure 1 is a flowchart of the implementation of the present invention.

Fig. 2 is a flow chart of the Hough straight line detection implemented by the present invention.

Figure 3 is a top view of the console.

Fig. 4 is a diagram of obtaining all object masks implemented by the present invention.

Fig. 5 is a schematic diagram of detecting the edges of all objects in an image implemented by the present invention.

Fig. 6 is a schematic diagram of a triangle denoising edge implemented by the present invention.

FIG. 7 is a schematic diagram of a pentagonal denoising edge implemented in the present invention.

Fig. 8 is a schematic diagram of a circular denoising edge implemented by the present invention.

Fig. 9 is a schematic diagram of a square denoising edge implemented by the present invention.

Fig. 10 is a schematic diagram of all the image denoising edges implemented by the present invention.

Fig. 11 is a schematic diagram of the intersection of a triangle mask and an image implemented in the present invention.

FIG. 12 is a schematic diagram of the intersection of a quadrilateral mask and an image implemented in the present invention.

Fig. 13 is a schematic diagram of the intersection of a pentagonal mask and an image implemented in the present invention.

Fig. 14 is a schematic diagram of the intersection of a cylindrical mask and an image implemented in the present invention.

Fig. 15 is a straight line diagram of the detected triangle edge implemented by the present invention.

Fig. 16 is a straight line diagram of detecting pentagonal edges implemented by the present invention.

Fig. 17 is a linear diagram of detecting square edges implemented by the present invention.

Fig. 18 is a schematic diagram of a straight line on the upper surface of a triangle implemented in the present invention.

Fig. 19 is a schematic diagram of a straight line on the upper surface of a pentagon implemented in the present invention.

Fig. 20 is a schematic diagram of a straight line on the upper surface of a square implemented in the present invention.

Fig. 21 is a schematic diagram of a triangle edge line segment implemented in the present invention.

Fig. 22 is a schematic diagram of a pentagonal edge line segment implemented in the present invention.

Fig. 23 is a schematic diagram of a square edge segment implemented in the present invention.

Fig. 24 is a schematic diagram of computing object information implemented by the present invention.

Detailed ways

In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

As shown in FIG. 1 , it is a flowchart of a regular polygonal prism detection method for a mechanical arm implemented in the present invention. Since most of the objects to be assembled are cylinders, the present invention takes a regular polygonal prism as an example for detection.

As shown in the figure, the input of the present invention is a picture of the console from a top view, wherein the console is designed to be white, and the objects to be assembled are cylinders of other colors; the upper surface edge of each object in the output picture and the shape of the object , position and angle information. Among them, the existing technologies used include color space conversion, adaptive threshold segmentation, grabcut segmentation, canny edge detection, Hough line detection and morphological processing, etc. In addition to the combination of these technologies, the present invention is based on the fact that the object to be assembled is a cylinder. Based on the prior knowledge, processing rules are created to filter out the real edges on the surface of the object from the edge detection results containing a lot of noise.

Specifically include the following steps:

Step 1: Obtain the top-view image of the console, use the RGB camera to take pictures of the table top and the objects to be assembled on the table from a top-down perspective, and obtain a picture of the console, as shown in Figure 3;

In order to distinguish the operating table and the objects to be assembled, they are marked with different colors; the operating table is designed in white, and the objects to be assembled are in other colors.

Step 2: Obtain an object mask. The object mask refers to at least obtaining a mask on the upper surface of the object.

Specifically include the following steps:

S21. Transfer the image to the HSV color space, and use the saturation S channel therein to perform adaptive threshold segmentation. Since the saturation of the object to be assembled is higher than that of the white table, an object mask is obtained; first, use the white table and the object to be assembled The color saturation is different, transfer the image to the HSV color space, and use the saturation S channel to do adaptive threshold segmentation. Since the saturation of the object to be assembled is higher than that of the white table, the object mask will be obtained.

S22. Denoising, using operations such as dilation, erosion, and removal of overly large connected domains to remove part of the noise; the denoising technology is an existing technology, and will not be repeated here.

S23, then input the mask after denoising and the table top picture transferred to the Lab color space together into grabcut; when the mask after denoising and the table top picture are input into grabcut together, the mask acts as a manual input foreground for the user The role of the region makes the module detection process fully automated, and the image is transferred to the Lab color space to obtain a better upper surface segmentation effect than the RGB color space.

S24. Next, check whether the area of the connected domain in the segmentation mask is too large. If it is too large, it is suspected that there is mask adhesion of multiple objects. Transfer the image to the YCrCb space and do a grabcut again and use it as the upper surface mask of the object. , otherwise directly use the segmentation result in the current Lab color space as the upper surface mask of the object. Obtaining all object masks is shown in Figure 4.

Step 3: Detect the edge of the object in the image. Using Canny edge detection, the edge binary image is obtained, as shown in Figure 5, which shows the edges of all objects. Even in the top view angle, there are still cases where the sides and side edges of the object are exposed. At this time, the edge connected to the side surface has a small color change on both sides, making it difficult to detect this edge. In order to solve this problem, use the upper surface and The side surface is mainly in the law of large changes in brightness. Edge detection is performed on the L component in the Lab color space and the V component in the HSV color space, because these two components can better reflect the brightness change. Then the detection results obtained by the two components are combined.

Step 4: Use the mask denoising obtained in step 2 to denoise the edge obtained in the third step. In this step, it is hoped to obtain an edge with as little noise as possible. The removed noise includes two kinds of noise inside and outside the edge of the object. Perform an expansion operation on the mask, and intersect with the edge detection result to remove external edge noise; then perform an erosion operation on the mask, and obtain the difference with the edge detection result to remove internal noise, as shown in Figure 6- 11, wherein, Fig. 6 is a schematic diagram of triangle denoising, Fig. 7 is a schematic diagram of pentagonal denoising, Fig. 8 is a schematic diagram of circular denoising, Fig. 9 is a schematic diagram of square denoising, Fig. 10 is a schematic diagram of all graphics Schematic diagram of denoising.

Step 5: Detect whether there is a circle. Since the upper surface of the cylinder is circular and does not contain straight lines, subsequent detection and processing of straight lines cannot be performed. Therefore, in this step, it is necessary to determine whether the target object currently being processed is a cylinder.

The detection process is shown in Figures 11-14. Figure 11 is a schematic diagram of the intersection of a triangular mask and an image. Figure 12 is a schematic diagram of the intersection of a quadrilateral mask and an image. Figure 13 is a schematic diagram of the intersection of a pentagonal mask and an image. Figure 14 is a schematic diagram of the intersection of a cylindrical mask and an image.

When detecting whether there is a circle, the object mask is applied to the intersection result obtained in the Lab color space, and the Hough circle transformation is used to detect whether there is a circle. If so, the most suitable upper surface circle is obtained and enters step 9, otherwise, enters step 6. Among them, the process of obtaining the upper surface circle is to use Hough circle detection in the images of HSV and Lab color spaces respectively, adaptively find the circle with the largest param2 in the color space, and then use the circle with the outer center as the upper surface circle . Among them, param2 is the input parameter of the Hough circle detection function, which indicates the regularity of the circle. Only shapes larger than param2 can be detected as circles. The outer circle is selected as the upper surface circle, based on the fact that the upper surface is always relatively close to the periphery of the picture in the top view.

Step 6: Obtain the information of the straight line where the edge is located. The binary mask in the image form of the edge detection result is not the mathematical expression of each line segment. For subsequent calculations, the straight line analytical formula must be obtained from the binary mask. The present invention uses the Hough straight line detection technology to adaptively obtain the straight line analytical formula where the line segment whose length is greater than th1 in the current edge detection image is located. Among them, the threshold th1 is used to limit the shortest length of the line segment, and the straight line shorter than th1 on the image will not be detected. In actual use, it is necessary to adjust according to the length of the edge of the object to be assembled in the image, so that th1 is slightly smaller than the length of the edge line segment of the object, so as to achieve the purpose of not only detecting all edge line segments, but also filtering out shorter noise line segments . Empirically, it can be set as 0.6 to 0.8 times the length of the edge of the object. For example, when the length of the edge of the image object is at least 100 pixels, then th1 can be set to a value between 60 and 80.

The specific process is shown in Figure 2. Through this process, the line information of all line segments with a length greater than th1 will be recorded according to the reverse order of the length of the line segment. The information includes the inclination angle theta, the distance rho from the line to the origin of the image, and the length when detected. Threshold th, wherein, th is the input parameter of the Hough line detection function, and only the line segment whose length is greater than th will be detected. The specific process is as follows:

S41. Obtain the line segment of th=th1;

S42, detecting a line segment whose length is greater than th;

S43, judging whether a straight line is detected; if no straight line is detected, continue to the next step; if yes, make th+=4, and return to S42;

S44, let th-=1;

S45. Further judge that th>th1, if yes, continue to the next step, otherwise end;

S46, detecting a line segment whose length is greater than th;

S47. Determine whether a straight line is detected; if a straight line is detected, record th and the straight line inclination angle theta output by Hough line detection and the distance rho to the origin of the picture, and delete the detected straight line in the original image; if not, then Return to step S44, set th-=1.

After deleting the detected straight lines, execute the above steps in a loop until all detections are completed.

Figure 15 shows the detection of triangle edge straight lines, detection of pentagon edge straight lines is shown in Figure 16, and detection of square edge straight lines is shown in Figure 17. Step 7: Screen out the straight line on the upper surface edge of the object according to the rules. This step hopes to eliminate the interference of the side surface edge, bottom surface edge and other noises of the object, and select the straight line that best fits the actual edge of the upper surface of the object. Since the straight line obtained by Hough detection in step 5 will be affected by the edge detection result, when the line segment of the edge binary mask has pixel-level curvature, the line segment width is greater than 1, or there is noise, it will be detected near each edge of the object A series of straight lines are drawn, and in order to screen out the straight lines closest to the real edge, three operations of straight line grouping, straight line denoising and straight line selection are performed in sequence. Among them, the straight line grouping is to correspond the straight lines of all the edges to each actual edge respectively, and after classification, each group of straight lines corresponds to an actual edge of the object surface. Line denoising is to remove the noise line segment and the line group corresponding to the side edge line segment on the basis of grouping, so as to ensure that each remaining line group can correspond to an upper surface edge of the object. At this time, in each group of straight lines, there may be straight lines close to the edge of the real upper surface, and there may be straight lines close to the edge of the lower surface. These straight lines also have angle differences, and not all of them can closely fit the actual edge of the surface. The last step, straight line selection, is to select a straight line that best matches the real edge of the object in each group of straight lines, and use it as the result of the upper surface straight line detection.

Straight line grouping: The purpose of grouping is to make each group of straight lines correspond to an edge of the target object, and it is necessary to find the features used for grouping. Since the object is a regular polygonal prism, the upper surface is a regular polygon. It is known that the slope of the straight line on each side of a regular polygon with odd sides is different, and the angle feature can be used to distinguish each side; the opposite sides of a regular polygon with even sides are parallel and have a large distance , the angles between different opposite sides are different, and the angle and distance features can be used to distinguish each side at the same time. Further consider the edge of the side surface of the polygonal prism. When the side surface is exposed, there is an edge parallel to and relatively close to the corresponding edge of the upper surface, that is, the edge of the lower surface, and two edges perpendicular to the operating table in reality. It is the edges that belong only to the side surfaces, and their angles are different from the edges of the upper surface. Therefore, in any case, considering both angle and distance features, straight lines can be mapped to the sides of regular polygonal prisms.

First, set the angle threshold th2 and the distance threshold th3 respectively. Among them, th2 reflects the degree of tolerance of the angle offset. The straight lines with the angle difference within th2 are all classified into the same group. When th2 is larger, the straight lines within the larger angle difference range will be classified into the same group. The specific value It is advisable to be able to distinguish different sides and tolerate a certain angle error, and the desirable value th2=0.3*min (inner angle angle, outer angle angle). Taking a triangle as an example, the interior angle is 60°, and the exterior angle is 120°. Take the minimum of the two angles of 60° and multiply by 0.3, then th2=0.3*60°=18°. The distance threshold th3 is used to distinguish two sides in the opposite side. When the difference between the two straight-line distances is less than th3, they are considered to belong to the same side, otherwise they belong to different opposite sides. The value of Th3 needs to be adjusted according to the size of the target object on the picture, so that the parallel edges on the upper surface of the object can be divided into two groups, and the parallel upper and lower surface edges can be combined into the same group. The possible value th3=0.5*the width of the smallest circumscribed rectangle of the regular polygon on the upper surface.

Next, use th2 and th3 for grouping. According to the recorded straight line information theta, all straight lines whose angle difference is within th2 are divided into one group, and each group contains at least one straight line; on this basis, straight lines whose distance difference is greater than th3 and are almost parallel are divided into different groups. The rule for judging that straight lines are almost parallel is that there is no intersection point between two straight lines within the scope of the screen.

In the actual use process, it is also necessary to judge the rationality of the values of th2 and th3 according to the grouping results. If necessary, the above two steps can be repeated until the grouping effect reaches the expectation, that is, each group of straight lines corresponds to an edge of the object .

Line denoising: The purpose of this step is to remove the two edges perpendicular to the operating table and other noises in the side edges. Taking advantage of the fact that the two edges and the noise are shorter in length from the top view angle, the length threshold th4 can be set to remove them. If a group has only one straight line and corresponds to th<th4, remove this group. Among them, the function of th4 is to separate shorter vertical edges and noises from all straight lines. The range of values is (the length of the vertical edges and the noise line, the length of the upper surface edge). The median value of this range can be taken first, and then according to The actual denoising results are adjusted repeatedly until the purpose of this step can be achieved.

Selecting straight lines: After the previous straight line grouping and straight line denoising steps, the current straight line group only includes straight lines corresponding to the upper surface edge and the lower surface edge parallel to it. The purpose of this step is to select the most suitable straight line among the detected straight lines as the detection result corresponding to the edge of the upper surface. Using the law that the upper surface straight line is compared with the lower surface straight line, the part of the framed object is less, and the possibility that the framed part is not in the range of the mask output in the second step is smaller, and the object framed by each straight line is compared with the mask. For each group, select the straight line with the smallest difference, which is considered to be the edge of the upper surface. Specifically, it is necessary to first construct a closed box for each straight line, and then compare the difference between the closed range and the mask. Among them, the edge of the closed frame is composed of this straight line and the straight line closest to the center of the mask in other straight line groups, so as to ensure that the difference between the enclosed figure and the mask is only determined by this straight line. Comparing all the pixels in the enclosing box A with all the pixels in the grabcut mask B, record the number n of pixels in A but not in B for each line. Afterwards, a straight line with the smallest n is selected from each group, which is considered to be the true upper surface edge straight line.

As shown in Figures 18-20, Figure 18 is a straight line on the upper surface of a triangle, Figure 19 is a straight line on the upper surface of a pentagon, and Figure 20 is a straight line on the upper surface of a square.

Step 8: The straight line is shortened to the edge segment of the object. Since what is obtained in the fourth step is straight line information, not line segment information, and the real edge is a line segment, for this reason, this step first obtains each vertex of the regular polygon, and then connects adjacent vertices to obtain the edge line segment. The vertex is the intersection point of adjacent edge lines, so the adjacent relationship of the lines must be judged first. The specific method is to set the distance threshold th5, and traverse all the current pairwise combinations of straight lines, and calculate the distance from the intersection point of the straight lines to the mask. If the distance is less than th5, the two straight lines are considered adjacent. The rule here is that vertices are closer to the mask than intersections of non-adjacent lines are farther away from the mask. Th5 is designed to distinguish these two distances, and generally can be set to 30 pixels. The way to calculate the distance from the point to the mask is to use the point as the center to draw a circle with increasing radius. When the circle starts to intersect with the mask, the radius of the circle is the distance from the point to the mask. After obtaining the adjacency relationship of all straight lines, calculate the vertices of the adjacent straight lines and connect the vertices in turn to obtain the edge of the upper surface of the object.

As shown in Figures 21-23, Figure 21 is a triangle edge line segment, Figure 22 is a pentagon edge line segment, and Figure 23 is a square edge line segment.

Step 9: Calculate the shape, centroid, and angle of the object. According to the one-to-one relationship between the shape of a regular polygonal prism and the number of vertices, the shape can be obtained from the number of vertices. For example, three vertices indicate that it is a regular triangular prism; the centroid can be calculated according to the closed figure surrounded by the upper surface edge; the object The angle of is calculated from the rotation angle of the smallest circumscribed rectangle on the upper surface. As shown in Figure 24.

Therefore, the present invention can accurately segment the upper surface of the regular polygonal prism, with straight and sharp edges and clear vertices. On this basis, the information required for the manipulator assembly task can be calculated more accurately, including the obtained position, angle and shape. Information, so that the automatic assembly task can be completed smoothly, reduce the possibility of assembly failure due to unsatisfactory segmentation and detection results, so as to obtain the accurate suction position, object shape and object angle required for the placement task, and help the placement task to be completed smoothly.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present invention should be included in the protection of the present invention. within range.

Claims (10)

1. a regular polygonal prism detection method for mechanical arm, is characterized in that comprising the steps: Step 1: Obtain the current state image, specifically, use the RGB camera to shoot down the operating table to obtain the current image of the object to be assembled; Step 2: Obtain the object mask in the image; Step 3: Obtain the edge of the object in the image; Step 4: For each target object, use the mask obtained in Step 2 to denoise the edge obtained in Step 3; Step 5: Determine whether the edge image of the current target object contains a circle, if so, go to step 9, otherwise go to step 6; Step 6: Obtain the information of the straight line where the edge is located from the edge detection result; Specifically, using the Hough line detection technology, adaptively obtain the line analysis formula of the line segment whose length is greater than th1 in the current edge detection image; Step 7: Screen out the straight line on the edge of the upper surface of the object, specifically, eliminate the interference of the edge of the side surface of the object, the edge of the bottom surface and other noises, and select the straight line that best fits the actual edge of the upper surface of the object; Step 8: Obtain the edge segment of the upper surface of the object, specifically, find the intersection point of each straight line on the edge, that is, the vertex of the upper surface of the object, and connect the vertices to obtain a line segment that fits the edge of the object; Step 9: Calculate the shape, centroid, and angle of the object, wherein the shape can be determined by the number of vertices, because the shape of a regular polygonal prism corresponds to the number of edges; the centroid can be calculated from the closed figure surrounded by the upper surface edge; the object The angle of is calculated from the rotation angle of the smallest circumscribed rectangle on the upper surface. 2. the regular polygonal prism detection method that is used for mechanical arm as claimed in claim 1, it is characterized in that in step 1, in order to distinguish operating platform and object to be assembled, they are marked with different colors; Operating platform is designed as White, other colors for the object to be assembled. 3 . The regular polygonal prism detection method for a robotic arm according to claim 1 , wherein in step 2, obtaining the object mask refers to obtaining at least the mask of the upper surface of the object. 4 . 4. The regular polygonal prism detection method for mechanical arm as claimed in claim 3, is characterized in that in step 2, specifically comprises the following steps: S21. Transfer the captured RGB image to the HSV color space, and use the saturation S channel therein for adaptive threshold segmentation. Since the saturation of the object to be assembled is higher than that of the white table, an object mask will be obtained; S22. Denoising, using morphological operations such as expansion, erosion, and removal of overly large connected domains to remove part of the noise; S23, then input the mask after the denoising and the table top picture transferred to the Lab color space into the grabcut together to obtain the result of the grabcut segmenting the foreground; S24, then detect whether the connected domain area in the segmentation mask is too large, if it is too large, transfer the picture to the YCrCb space and do a grabcut again and use it as the upper surface mask of the object, otherwise directly use the current Lab color space (S23 The result of segmentation) is used as the upper surface mask of the object. 5. The regular polygonal prism detection method for the mechanical arm as claimed in claim 1, characterized in that in step 3, the edge of the object in the detection image is specifically implemented as the L component in the Lab color space and the HSV color space respectively Edge detection is performed on the V component, and then the detection results obtained by the two components are combined. 6. The regular polygonal prism detection method for a manipulator as claimed in claim 5, wherein in step 4, the mask is used to remove the noise of the edge detection result, including removing two kinds of noise inside and outside the edge of the object. The specific implementation is to perform an expansion operation on the mask, and obtain the intersection with the edge detection result to remove external edge noise; then perform an erosion operation on the mask, and obtain the difference with the edge detection result to remove internal noise. 7. The regular polygonal prism detection method for manipulator as claimed in claim 5, is characterized in that in step 5, utilizes Hough circle to detect, detects in the intersection result obtained by acting on Lab color space by object mask, whether exists Circle, if so, get the most suitable upper surface circle and go to step 9, otherwise go to step 6; wherein, the process of getting the upper surface circle is to use Hough circle detection in the images of HSV and Lab color space respectively, adaptive Find the circle with the largest param2 in the color space, and then use the circle with the outer center as the upper surface circle; where param2 is the input parameter of the Hough circle detection method, indicating the degree of regularity of the circle. Only shapes larger than param2 can be regarded as Circle detection; select the outer circle as the upper surface circle, based on the fact that the upper surface is always relatively close to the periphery of the picture in the top view. 8. The regular polygonal prism detection method for mechanical arm as claimed in claim 1, characterized in that in step 6, the specific process of adaptively obtaining the longest line segment with a length greater than th1 is as follows: S41. Obtain the line segment of th=th1; S42, detecting a line segment whose length is greater than th; S43, judging whether a straight line is detected; if no straight line is detected, continue to the next step; if yes, make th+=4, and return to S42; S44, let th-=1; S45. Further judge that th>th1, if yes, continue to the next step, otherwise end; S46, detecting a line segment whose length is greater than th; S47, judging whether a straight line is detected; if a straight line is detected, the straight line information and th are recorded, and the detected straight line is deleted in the original image; if not, then return to the S44 step, and th-=1; After deleting the detected straight lines, execute the above steps in a loop until all detections are completed. 9. The regular polygonal prism detection method for a mechanical arm as claimed in claim 1, characterized in that in step seven, in order to screen out the straight lines on the upper surface of the object, it is necessary to perform three straight line grouping, straight line denoising and straight line selection in sequence. operate: Straight line grouping: group all detected straight lines according to angle, and group all straight lines whose angle difference is within th2, and each group contains at least one straight line; Straight line denoising: remove side edge straight lines and other noises, and remove shorter straight lines according to the shorter length of side edges and shorter noise straight lines in the top view; Select a straight line: Select the upper surface straight line from the set of remaining upper and lower surface edge straight lines; use the law that the upper surface straight line is closer to the upper surface mask of the object than the lower surface straight line, and compare the lines framed by each straight line The difference between the object and the mask, the straight line with the smallest difference is selected for each group, which is considered to be the edge of the upper surface; When selecting straight lines, it is necessary to first construct a closed frame for each straight line, and then compare the difference between the closed range and the mask; the method of constructing a closed frame is to match each straight line with the straight line closest to the center of the mask in other straight line groups, and these straight lines Form a closed box; compare all the pixels in the closed box A with all the pixels in the mask B of grabcut, and record the number n of pixels that are in A but not in B for each line; after that, select one n in each group The smallest straight line is considered to be the upper surface edge straight line. 10. The regular polygonal prism detection method for mechanical arm as claimed in claim 1, characterized in that in step 8, traversing the pairwise matching of all current straight lines, calculating the minimum distance from the intersection point of these two straight lines to the mask, If the distance is less than th5, the two straight lines are considered to be adjacent; among them, the way to calculate the distance from the point to the mask is to draw a circle with gradually increasing radius with the point as the center. When the circle starts to intersect with the mask, the circle's The radius is the distance from the point to the mask; after obtaining the adjacency relationship of all straight lines, calculate the vertices of the adjacent straight lines and connect the vertices in turn to obtain the edge of the upper surface of the object.

Images