CN116188571A

CN116188571A - Regular polygon prism detection method for mechanical arm

Info

Publication number: CN116188571A
Application number: CN202310009829.7A
Authority: CN
Inventors: 曹健; 谢晨莹
Original assignee: Peking University
Current assignee: Peking University
Priority date: 2023-01-04
Filing date: 2023-01-04
Publication date: 2023-05-30

Abstract

The invention discloses a regular polygon prism detection method for a mechanical arm, which comprises the following steps: step one: obtaining a top view image of the operation table; step two: obtaining an object mask, and step three: obtaining object edges, and step four: for each object, denoising the edges obtained in the third step by using the mask obtained in the second step, wherein in the fifth step, whether a circle exists or not is judged, if the circle exists, an upper surface circle is obtained, and the step is jumped to a step nine, otherwise, the step six is carried out; step six: obtaining the information of the straight line where the edge is located from the edge detection result; step seven: selecting an edge straight line of the upper surface of the object; step eight: shortening the straight line into an object edge line segment; step nine: calculating the shape, centroid and angle of the object. The invention can accurately obtain the edge of the upper surface of the regular polygon prism object, and obtain the suction position, the object shape and the object angle required by the placing task, thereby helping the placing task to be successfully completed.

Description

Regular polygon prism detection method for mechanical arm

Technical Field

The invention belongs to the technical field of mechanical arms, and particularly relates to a regular polygon prism detection method for a mechanical arm, which is used for accurately detecting the edge, shape, centroid and angle of the upper surface of the regular polygon prism.

Background

Mechanical arms generally refer to high precision, multiple input multiple output, highly nonlinear, strongly coupled complex systems. Because of its unique operational flexibility, it has been widely used in the fields of industrial assembly, safety explosion protection, etc. When the operation target object is always at the fixed position, fixed, repeated position control may be used to cause the robot arm to perform the fixed position pick-and-place step. When the position of the target object to be assembled changes, the mechanical arm needs to have sensing capability, automatically position the object and complete the assembly task.

And (3) increasing the sensing capability for the mechanical arm, acquiring external information on hardware by using a sensor, and calculating the required information according to the acquired data. As in patent application 202120538755.2, a distribution network live working robot system is disclosed, comprising: the control cabinet comprises an operation desk device, a supporting device and a control cabinet; the operation table device comprises an operation table, a mechanical arm assembly and a sensor assembly; the mechanical arm assembly and the sensor assembly are installed on the operation table; the mechanical arm assembly comprises a first number of mechanical arms and a mechanical arm base; the mechanical arm base is fixed on the operation table, and the mechanical arm is fixed on the mechanical arm base; the supporting device comprises an insulating bucket and a supporting frame; the supporting frame is clamped on the insulating hopper in a surrounding manner; the operation table is arranged on the supporting frame in a sliding way; a cavity is formed in the insulating hopper, and the control cabinet is arranged in the cavity; a processor and a communication module are arranged in the control cabinet; the mechanical arm and the sensor component are connected with the processor, and the processor is connected with the communication module.

In the assembly task, the sensing capability is provided for the mechanical arm, an RGB camera can be used as a sensor to acquire an image of an object to be assembled currently, information required by taking and placing the object, such as the position, the angle, the shape and the like of the target object, is analyzed, and finally the mechanical arm performs the placing action. Whether the assembly task can be successfully completed depends on the accuracy of the position, angle,

the process of analyzing the position, angle and shape of the target object may be implemented using a target detection or instance segmentation model. However, the two models have defects, and most of detection frames output by the target detection model are in the horizontal direction or the vertical direction, have no inclination angle, cannot well reflect the angle of an object, and are difficult to meet the assembly requirement; the segmentation result of the example segmentation model often cannot be accurately attached to the actual edge of the object, and the vertex of the object is not clear enough, so that the center of gravity of the object cannot be accurately obtained, and the assembly success rate of the mechanical arm is affected. In addition, when an object exists at a certain height, an object side surface exists in an input image, and both types of models can be affected by the object side surface. The prior art can not accurately filter out the edge of the upper surface of an object to be assembled, so that the accurate angle and the gravity center of the object are difficult to obtain, and the success rate of an assembly task is affected. Thus, improvements are needed.

Disclosure of Invention

Based on the above, the invention aims to provide a regular polygon detection method for a mechanical arm, which can provide sensing capability for the mechanical arm, overcome the defects of a current target detection model and an example segmentation model, obtain the required suction position, object shape and object angle of a placing task and help the placing task to finish smoothly.

Another object of the present invention is to provide a method for detecting regular polygon of a robot arm, which can obtain an edge of an upper surface of an object while eliminating an influence of a side surface. Performing self-adaptive threshold segmentation by using a saturation S channel of the HSV color space to obtain an object mask; further performing grabcut segmentation in Lab and YCrCb color spaces to obtain an upper surface mask of the object as much as possible; meanwhile, the edge detection of the L component in the Lab color space and the V component in the HSV color space is fused, and a more complete upper surface edge is obtained as far as possible, so that the detection accuracy is ensured.

In order to achieve the above purpose, the technical scheme of the invention is as follows:

the regular polygon prism detection method for the mechanical arm is characterized by comprising the following steps of:

step one: obtaining an operation desk overlook image, shooting a table top and an object to be assembled on the table top at an overlook view angle by using an RGB camera, and obtaining an operation desk picture;

in order to distinguish the operation table from the object to be assembled, they are identified in different colours; the operation table is designed to be white, and the object to be assembled is of other colors.

Step two: a mask of the object is obtained,

wherein obtaining an object mask means obtaining at least a mask of an upper surface of the object.

The method specifically comprises the following steps:

s21, transferring the image to an HSV color space, and performing self-adaptive threshold segmentation by using a saturation S channel, wherein an object mask is obtained because the saturation of an object to be assembled is higher than that of a white table top;

s22, denoising, namely removing part of noise by using operations such as expansion, corrosion, connected domain with overlarge area and the like;

s23, inputting the denoised mask and the table top picture transferred to the Lab color space into the grabcut together, so that the module detection process is fully automatic, and the picture transferred to the Lab color space can obtain a better upper surface segmentation effect than the RGB color space;

s24, detecting whether the area of the connected domain in the segmentation mask is too large, if so, suspected that mask adhesion of a plurality of objects occurs, transferring the picture into a YCrCb space to make grabcut again and serve as an object upper surface mask, otherwise, directly using the segmentation result in the current Lab color space as the object upper surface mask.

Step three: object edges in the image are detected.

And (5) obtaining an edge binary image by using Canny edge detection. In order to solve the problem, edge detection is carried out on an L component in a Lab color space and a V component in an HSV color space respectively by utilizing a rule that the upper surface and the side surface mainly change in brightness, and then the detection results obtained by the two components are combined.

Step four: detecting the edge of the object in the image and denoising by using the mask obtained in the step two, wherein the step hopes to obtain an edge detection result with the least noise. The noise removed includes both internal and external noise to the edges of the object. Performing expansion operation on the mask, and solving intersection with an edge detection result to remove external edge noise; and then performing etching operation on the mask, and solving a difference set between the etching operation and the edge detection result to remove internal noise.

Further, mask noise is obtained by performing expansion operation on the mask and obtaining intersection with an edge detection result to obtain an edge detection result for removing external noise of an object; and then carrying out corrosion operation on the mask, and solving a difference set between the mask and the edge detection result to obtain the edge detection result for removing the noise in the object.

And fifthly, judging whether a circle exists.

Judging whether the edge image of the current target object contains a circle, if yes, entering the step eight, otherwise, entering the step six. Specifically, the hough circle detection is used to detect whether a circle exists in the intersection result obtained by the object mask acting on the Lab color space, if so, the most suitable upper surface circle is obtained and the step nine is entered, otherwise, the step six is entered. The flow of obtaining the upper surface circle is to use Hough circle detection in the images of HSV and Lab color spaces respectively, adaptively find the circle with the maximum param2 in the color space, and then use the circle with the outer circle center as the upper surface circle. The param2 is an input parameter of the hough circle detection method, which indicates the rule degree of the circle, and only the shape larger than the param2 can be detected as the circle. The outer circle is chosen as the upper surface circle, based on the fact that the upper surface is always relatively close to the periphery of the picture in a top view.

Step six: and obtaining the information of the straight line where the edge is located from the edge detection result.

And the edge detection result uses a Hough straight line detection technology to adaptively obtain a straight line analysis type where a line segment with the length larger than th1 is located in the current edge detection image.

The specific flow is as follows:

s41, obtaining a line segment with th=th1;

s42, detecting a line segment with the length longer than th;

s43, judging whether a straight line is detected; if no straight line is detected, continuing to the next step; if so, let th+=4, and return to S42;

s44, let th- =1;

s45, further judging that th is more than th1, if yes, continuing the next step, otherwise, ending;

s46, detecting a line segment with the length longer than th;

s47, judging whether a straight line is detected; if the straight line is detected, recording straight line information and th, and deleting the detected straight line in the original image; if not, return to step S44, let th- =1.

And after deleting the detected straight line, circularly executing the steps until all the detection is completed.

Through the process, according to the reverse order of the length of the line segments, the linear information of all the line segments with the length larger than th1 is recorded, wherein the information comprises the inclination angle, the distance from the linear to the origin of the picture and the length threshold th in detection.

Step seven: the method comprises the steps of screening out the straight line of the edge of the upper surface of an object, removing the interference of the edge of the side surface of the object, the edge of the bottom surface and other noise, and selecting the straight line which is most fit with the actual edge of the upper surface of the object.

And sequentially performing three operations of linear grouping, linear denoising and linear selection.

Straight line grouping: grouping all the detected straight lines according to angles, wherein all the straight lines with the angle difference within th2 are grouped into groups, and each group comprises at least one straight line;

further, the straight lines having a distance difference greater than th3 and almost parallel are divided into different groups, and th3 requires adjustment so that the parallel edges of the upper surface of the object can be divided into two groups while the parallel edges of the upper and lower surfaces are combined into the same group. The rule in which the straight lines are judged to be almost parallel is that the two straight lines have no intersection point in the screen range.

Linear denoising: the side edge straight line and other noises are mainly removed, and the straight line is removed according to the fact that the side edge length is short and the noise straight line is short in the overlook view, and the judgment rule of the straight line is that the group has only one straight line and the length of the line segment recognized at the time is lower than th4.

Selecting a straight line: selecting an upper surface line from a collection of remaining upper and lower surface edge lines; and comparing the difference between the object framed by each straight line and the mask by using the upper surface straight line compared with the lower surface straight line, and selecting the straight line with the smallest difference for each group to be regarded as the upper surface edge.

Further, when selecting straight lines, it is necessary to construct a sealing frame for each straight line, and then compare the difference between the sealing range and the mask; when the sealing frame is constructed, matching each straight line with the nearest straight line from the mask center in other straight line groups to form the sealing frame; comparing all pixels in the closed box A with all pixels in the mask B of the grabcut, and recording the number n of pixels in A but not B for each straight line; then, a line with the smallest n is selected from each group and is regarded as the upper surface edge line.

Step eight: the straight line is shortened to an object edge line segment. Since the straight line information obtained in the fourth step does not include an end point, the intersection point of the edge lines, that is, the vertex of the upper surface of the object is obtained in this step, and a line segment that fits the edge of the object is obtained.

For this purpose, the minimum distance from the intersection point of all the current straight lines to the mask is calculated by traversing every two pairs of the straight lines, and if the distance is smaller than th5, the two straight lines are considered to be adjacent. The distance from the point to the mask is calculated by drawing a circle with gradually increased radius by taking the point as the center of a circle, and when the circle starts to intersect with the mask, the radius of the circle is the distance from the point to the mask. And after the adjacent relation of all the straight lines is obtained, calculating the vertexes of the adjacent straight lines and sequentially connecting the vertexes to obtain the edge of the upper surface of the object.

Nine: calculating the shape, centroid and angle of the object. Obtaining the shape according to the one-to-one correspondence between the shape of the regular polygon and the number of the prisms, and the number of the vertexes; the centroid can be calculated according to a closed graph enclosed by the edges of the upper surface; the angle of the object is calculated from the rotation angle of the minimum circumscribed rectangle of the upper surface.

Compared with the prior art, the invention has the beneficial effects that:

the invention solves the problem of poor segmentation and detection effects in RGB color space by utilizing the conversion of the color space and the fusion of multi-color space information. Specifically, for image segmentation, an initial mask is segmented by using the S component of HSV, and then grabcut segmentation is further performed in Lab and YCrCb color spaces according to the situation, so that the upper surface mask of the object is obtained as much as possible. For edge detection, the L component in the Lab color space and the V component in the HSV color space are fused for edge detection, so that a more complete upper surface edge is obtained as far as possible, an accurate edge can be accurately obtained, the detection accuracy is improved, the suction position, the object shape and the object angle required by the placing task are obtained, and the placing task is helped to be completed smoothly.

Drawings

Fig. 1 is a flow chart of an implementation of the present invention.

Fig. 2 is a flowchart of hough straight line detection implemented in the present invention.

Fig. 3 is a top view of the console.

Fig. 4 is a diagram of a full object mask obtained in accordance with an embodiment of the present invention.

Fig. 5 is a schematic diagram of the detection of all object edges of an image embodying the present invention.

Fig. 6 is a schematic diagram of a triangular denoising edge implemented in accordance with the present invention.

Fig. 7 is a schematic diagram of a pentagonal denoising edge embodying the present invention.

Fig. 8 is a schematic diagram of a circular denoising edge embodying the present invention.

Fig. 9 is a schematic diagram of a square denoising edge embodying the present invention.

FIG. 10 is a schematic diagram of an overall pattern denoising edge embodying the present invention.

FIG. 11 is a schematic diagram of the intersection of a triangle mask and an image in accordance with the present invention.

FIG. 12 is a schematic diagram of the intersection of a quadrilateral mask and an image according to the present invention.

FIG. 13 is a schematic diagram of the intersection of a pentagonal mask and an image in accordance with the present invention.

FIG. 14 is a schematic illustration of the intersection of a cylindrical mask and an image in accordance with the present invention.

Fig. 15 is a line drawing of the edges of a test triangle in accordance with the practice of the present invention.

FIG. 16 is a diagram of a line graph of detected pentagonal edges in accordance with the practice of the present invention.

FIG. 17 is a diagram of a line of sight of a square edge for use in the practice of the present invention.

Fig. 18 is a schematic view of a straight line of the upper surface of a triangle in which the present invention is implemented.

Fig. 19 is a schematic view of a straight line of the upper surface of a pentagon in which the present invention is implemented.

FIG. 20 is a schematic view of a straight line of a square upper surface in which the present invention is implemented.

Fig. 21 is a schematic diagram of a triangle edge line segment embodying the present invention.

FIG. 22 is a schematic representation of a pentagonal edge line segment in accordance with the present invention.

FIG. 23 is a schematic view of a square edge line segment in accordance with the present invention.

FIG. 24 is a schematic diagram of calculating object information in accordance with an embodiment of the present invention.

Detailed Description

The present invention will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present invention more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.

Fig. 1 is a flowchart of a method for detecting a regular polygon prism for a mechanical arm according to the present invention. Since the objects to be assembled are mostly columns, the invention takes regular polygon as an example for detection.

The input of the invention is shown in the figure as a picture of the console from a top view, wherein the console is designed to be white and the object to be assembled is a cylinder of other colors; and outputting the upper surface edge of each object in the picture and the shape, position and angle information of the object. The prior art includes color space conversion, adaptive threshold segmentation, grabcut segmentation, canny edge detection, hough straight line detection, morphological processing and the like, and besides the combination of the techniques, the invention creates a processing rule according to the prior knowledge that the object to be assembled is a cylinder, and the processing rule is used for screening real edges of the upper surface of the object from edge detection results containing a large amount of noise.

The method specifically comprises the following steps:

step one: obtaining a top view image of an operation desk, shooting a table top and an object to be assembled on the table top at a top view angle by using an RGB camera, and obtaining an operation desk picture, as shown in figure 3;

Step two: an object mask is obtained, which means that at least the upper surface of the object is obtained.

The method specifically comprises the following steps:

s21, transferring the image to an HSV color space, and performing self-adaptive threshold segmentation by using a saturation S channel, wherein the saturation of an object to be assembled is higher than that of a white table top, so that an object mask is obtained; firstly, the image is transferred to HSV color space by utilizing the difference of color saturation of the white table top and the object to be assembled, and the saturation S channel is used for self-adaptive threshold segmentation, so that the object mask can be obtained because the saturation of the object to be assembled is higher than that of the white table top.

S22, denoising, namely removing part of noise by using operations such as expansion, corrosion, connected domain with overlarge area and the like; the denoising technique is the prior art and will not be described in detail herein.

S23, inputting the denoised mask and the table top picture transferred to the Lab color space into a grabcut; when the denoised mask and the table-board picture are input into the grabcut together, the mask plays a role in manually inputting a foreground area for a user, so that the module detection process is fully automatic, and the picture is transferred to the Lab color space to obtain a better upper surface segmentation effect than the RGB color space.

S24, detecting whether the area of the connected domain in the segmentation mask is too large, if so, suspected that mask adhesion of a plurality of objects occurs, transferring the picture into a YCrCb space to make grabcut again and serve as an object upper surface mask, otherwise, directly using the segmentation result in the current Lab color space as the object upper surface mask. The complete object mask is obtained as shown in fig. 4.

Step three: object edges in the image are detected. Using Canny edge detection, an edge binary image is obtained, as shown in fig. 5, which shows all object edges. Even if there is still a case where the side surfaces and the side edges of the object are exposed in a top view, at this time, the edge connected to the side surface has a small color change on both sides, so that this edge is difficult to detect, and in order to solve this problem, edge detection is performed on the L component in the Lab color space and the V component in the HSV color space, respectively, using a rule that the upper surface and the side surface have a large change in brightness, because these two components are more capable of representing the brightness change. The detection results obtained for the two components are then summed.

Step four: and (3) denoising the edge obtained in the third step by using the mask denoising obtained in the second step, wherein the step is to obtain the edge with the least noise. The noise removed includes both internal and external noise to the edges of the object. Performing expansion operation on the mask, and solving intersection with an edge detection result to remove external edge noise; and performing etching operation on the mask, and obtaining a difference set between the mask and an edge detection result to remove internal noise, as shown in fig. 6-11, wherein fig. 6 is a schematic diagram of triangle denoising, fig. 7 is a schematic diagram of pentagon denoising, fig. 8 is a schematic diagram of circular denoising, fig. 9 is a schematic diagram of square denoising, and fig. 10 is a schematic diagram of all graph denoising.

Step five: it is detected whether a circle is present. Since the upper surface of the cylinder is circular and does not contain a straight line, the subsequent detection and processing of the straight line cannot be performed, and therefore, in this step, it is necessary to determine whether the target object to be processed is a cylinder.

The detection process is shown in fig. 11-14, in which fig. 11 is a schematic diagram of the intersection of a triangle mask and an image, fig. 12 is a schematic diagram of the intersection of a quadrilateral mask and an image, fig. 13 is a schematic diagram of the intersection of a pentagon mask and an image, and fig. 14 is a schematic diagram of the intersection of a cylindrical mask and an image.

And when detecting whether a circle exists, an object mask acts on an intersection result obtained in the Lab color space, detecting whether the circle exists by using Hough circle transformation, if so, obtaining the most suitable upper surface circle and entering a step nine, otherwise, entering a step six. The flow of obtaining the upper surface circle is to use Hough circle detection in the images of HSV and Lab color spaces respectively, adaptively find the circle with the maximum param2 in the color space, and then use the circle with the outer circle center as the upper surface circle. The param2 is an input parameter of the hough circle detection function, which indicates the rule degree of the circle, and only the shape larger than the param2 can be detected as the circle. The outer circle is chosen as the upper surface circle, based on the fact that the upper surface is always relatively close to the periphery of the picture in a top view.

Step six: and obtaining the information of the straight line where the edge is located. The binary mask in the form of an edge detection result image, rather than the mathematical expression of each line segment, is used to obtain a straight line analytical formula from the binary mask for subsequent calculation. The invention uses Hough straight line detection technology to adaptively obtain the straight line analysis type of the line segment with the length larger than th1 in the current edge detection image. Wherein the threshold th1 is used to limit the shortest length of the line segment, a straight line shorter than th1 will not be detected on the image. In the actual use process, the adjustment is needed according to the length of the edge of the object to be assembled in the image, so that th1 is slightly smaller than the length of the edge line segment of the object, and the purposes of detecting all edge line segments and filtering noise line segments with shorter length are achieved. Empirically, it can be set to 0.6 to 0.8 times the length of the edge of the object. For example, when the length of the edge of the image object is 100 pixels at the shortest, th1 may be set to a value from 60 to 80.

The specific flow is shown in fig. 2, through this flow, the straight line information of all the line segments with lengths greater than th1 is recorded according to the reverse order of the length of the line segments, where the information includes the inclination angle theta, the distance rho from the straight line to the origin of the picture, and the length threshold th at the time of detection, where th is the input parameter of the hough straight line detection function, and only the line segments with lengths greater than th can be detected. The specific flow is as follows:

s41, obtaining a line segment with th=th1;

s42, detecting a line segment with the length longer than th;

s44, let th- =1;

s46, detecting a line segment with the length longer than th;

s47, judging whether a straight line is detected; if the straight line is detected, recording the straight line inclination angles theta and the distances rho from the original point of the picture, which are output by th and Hough straight line detection, and deleting the detected straight line in the original image; if not, return to step S44, let th- =1.

The detected triangle edge lines are shown in fig. 15, the detected pentagon edge lines are shown in fig. 16, and the detected square edge lines are shown in fig. 17. Step seven: according to the rule, the straight line of the edge of the upper surface of the object is screened out, the interference of the edge of the side surface of the object, the edge of the bottom surface and other noise is eliminated, and the straight line which is most fit with the actual edge of the upper surface of the object is selected. Because the straight line obtained by Hough detection in the step five is affected by the edge detection result, when the line segment of the edge binary mask has pixel-level bending, the line segment width is larger than 1, or noise exists, a series of straight lines are detected near each edge of the object, and in order to screen out the straight line closest to the real edge from the line segment, three operations of straight line grouping, straight line denoising and straight line selection are sequentially carried out. The straight lines are grouped so as to respectively correspond to all the straight lines of the edges to each actual edge, and each group of straight lines corresponds to the actual edge of the surface of an object after classification. The linear denoising is to remove the noise line segments and the linear groups corresponding to the side edge line segments on the basis of grouping, so that each remaining linear group can correspond to one upper surface edge of an object. At this time, there may be straight lines in each group that are close to the actual upper surface edge, and there may be straight lines that are close to the lower surface edge, and these straight lines may have angular differences, and not every straight line may closely fit the actual upper surface edge. And finally, selecting straight lines, namely, respectively selecting one straight line which can be matched with the real edge of the object in each group of straight lines, and taking the straight line as an upper surface straight line detection result.

Straight line grouping: the grouping is to make each group of straight lines correspond to one edge of the target object, and the characteristics for grouping need to be found. Because the object is a regular polygon, the upper surface is a regular polygon, the slope of the straight line where each side of the regular polygon with an odd side is known to be different, and each side can be distinguished by using angle characteristics; the opposite sides of the regular polygon with even sides are parallel with each other and have larger distance, the angles between different opposite sides are different, and the sides can be distinguished by using the angle and distance characteristics. Further consider the edges of the side surfaces of the polygon, when the side surfaces are exposed, there is a parallel edge that is closer to the corresponding edge of the upper surface, i.e. the edge of the lower surface, and two edges that are in reality perpendicular to the console, i.e. the edges that belong only to the side surfaces, whose angles are different from the edges of the upper surface. In any case, therefore, straight lines can be assigned to each side of the regular polygon, taking into account both the angular and distance characteristics.

First, an angle threshold th2 and a distance threshold th3 are set, respectively. Wherein, th2 reflects the tolerance degree of the angle offset, the straight lines with the angle difference within th2 are all classified into the same group, when th2 is larger, the straight lines within the larger angle difference range are classified into the same group, the specific value is that different edges can be distinguished, a certain angle error can be tolerated, and the value th2=0.3×min (inner angle, outer angle). Taking a triangle as an example, the inner angle is 60 °, the outer angle is 120 °, the minimum value of the two is 60 °, and then the two are multiplied by 0.3, and th2=0.3×60=18° can be set. And a distance threshold th3 for distinguishing two edges of the opposite edges, wherein when the difference of the distances of the two straight lines is smaller than th3, the two straight lines are considered to belong to the same edge, otherwise, the two straight lines respectively belong to different opposite edges. The value of Th3 needs to be adjusted according to the size of the target object on the picture, so that the parallel edges of the upper surface of the object can be divided into two groups, and the parallel edges of the upper surface and the lower surface are combined into the same group. The value th3=0.5 is the width of the smallest circumscribed rectangle of the regular polygon of the upper surface.

Next, grouping is performed using th2 and th3. Dividing all the straight lines with the angle difference within th2 into groups according to the recorded straight line information theta, wherein each group comprises at least one straight line; on this basis, the lines whose distance difference is greater than th3 and which are almost parallel are divided into different groups. The rule in which the straight lines are judged to be almost parallel is that the two straight lines have no intersection point in the screen range.

In the actual use process, the rationality of the values of th2 and th3 is also required to be judged according to the grouping result, and if necessary, the two steps can be repeated until the grouping effect reaches the expected value, namely, each group of straight lines corresponds to one edge of the object.

Linear denoising: the purpose of this step is to remove two edges of the side edges perpendicular to the console and other noise. The length threshold th4 can be set to remove these two edges and the noise by using the rule that the length of the noise is shorter in the top view. If a group has only one straight line and corresponds to th < th4, the group is removed. The effect of th4 is that shorter vertical edges and noise are separated from all straight lines, the value range is (the length of the vertical edges and the noise straight lines and the length of the edge of the upper surface), the median of the range can be firstly taken, and then the adjustment is repeatedly carried out according to the actual denoising result, so that the aim of the step can be achieved.

Selecting a straight line: through the previous straight line grouping and straight line denoising steps, the current straight line group only comprises straight lines corresponding to the upper surface edge and the lower surface edge parallel to the upper surface edge. The purpose of this step is to select a most suitable line among the detected lines as the detection result of the corresponding upper surface edge. And comparing the differences between the objects framed by each straight line and the mask respectively by using the rule that the parts of the objects framed by the upper surface are fewer than the parts of the lower surface, and the probability that the parts framed by the lower surface are not in the mask range output by the second step, and selecting the straight line with the smallest difference for each group to be regarded as the upper surface edge. Specifically, it is necessary to construct a closed frame for each straight line first, and then compare differences between the closed range and the mask. The edge of the sealing frame consists of the straight line and the straight line closest to the center of the mask in other straight line groups, so that the difference between the enclosed sealing pattern and the mask is only determined by the straight line. All pixels in the closed box a are compared with all pixels in the mask B of grabcut, and the number n of pixels in a but not B is recorded for each straight line. Then, a minimum n line is selected from each group, and is considered as a true upper surface edge line.

As shown in fig. 18-20, where fig. 18 is a triangular upper surface line, fig. 19 is a pentagonal upper surface line, and fig. 20 is a square upper surface line.

Step eight: the straight line is shortened to an object edge line segment. Because the straight line information, the non-line segment information and the real edge are line segments are obtained in the fourth step, for this purpose, each vertex of the regular polygon is first obtained in the step, and then adjacent vertices are connected to obtain the edge line segments. The vertex is the intersection of straight lines of adjacent edges, so that the adjacent relationship of the straight lines is judged first. The specific method comprises the steps of setting a distance threshold th5, traversing all the line combinations matched pairwise at present, calculating the distance between the intersection point of the lines and the mask, and considering that the two lines are adjacent if the distance is smaller than th 5. The rule here is that the vertices are closer to the mask than the intersection points of the non-adjacent lines are farther from the mask. Th5 is provided for distinguishing these two distances, and may be generally 30 pixels. The distance from the point to the mask is calculated by drawing a circle with gradually increasing radius by taking the point as the center of a circle, and when the circle starts to intersect with the mask, the radius of the circle is the distance from the point to the mask. And after the adjacent relation of all the straight lines is obtained, calculating the vertexes of the adjacent straight lines and sequentially connecting the vertexes to obtain the edge of the upper surface of the object.

As shown in fig. 21-23, where fig. 21 is a triangular edge line segment, fig. 22 is a pentagonal edge line segment, and fig. 23 is a square edge line segment.

Step nine: calculating the shape, centroid and angle of the object. According to the one-to-one correspondence between the shape of the regular polygon and the number of vertexes, the shape can be obtained by the number of vertexes, for example, three vertexes are described as the regular polygon; the centroid can be calculated according to a closed graph enclosed by the edges of the upper surface; the angle of the object is calculated from the rotation angle of the minimum circumscribed rectangle of the upper surface. As shown in fig. 24.

Therefore, the invention can accurately divide the upper surface of the regular polygon prism object, the edge is flat and sharp, the vertex is clear, the information required by the mechanical arm assembly task can be calculated more accurately on the basis, the information comprises the obtained position, angle and shape information, the automatic assembly task can be completed smoothly, the possibility of assembly failure caused by non-ideal detection results obtained by division is reduced, the accurate suction position, object shape and object angle required by the placement task are obtained, and the smooth completion of the placement task is facilitated.

The foregoing description of the preferred embodiments of the invention is not intended to be limiting, but rather is intended to cover all modifications, equivalents, and alternatives falling within the spirit and principles of the invention.

Claims

1. The regular polygon prism detection method for the mechanical arm is characterized by comprising the following steps of:

step one: obtaining a current state image, specifically, using an RGB camera to take a down-shot operation table to obtain an image of an object to be assembled currently;

step two: obtaining an object mask in the image;

step three: obtaining object edges in the image;

step four: denoising the edges obtained in the third step by using the mask obtained in the second step for each target object;

step five: judging whether the edge image of the current target object contains a circle or not, if yes, entering a step nine, otherwise, entering a step six;

step six: obtaining the information of the straight line where the edge is located from the edge detection result;

specifically, a Hough straight line detection technology is used for adaptively obtaining a straight line analysis type where a line segment with the length larger than th1 is located in a current edge detection image;

step seven: screening out the straight line of the edge of the upper surface of the object, specifically, eliminating the interference of the edge of the side surface of the object, the edge of the bottom surface and other noise, and selecting the straight line which is most fit with the actual edge of the upper surface of the object;

step eight: obtaining edge line segments of the upper surface of the object, specifically, solving the intersection point of all straight lines of the edge, namely the vertex of the upper surface of the object, and connecting all vertexes to obtain line segments attached to the edge of the object;

step nine: calculating the shape, centroid and angle of the object, wherein the shape can be determined by the number of vertexes, because the shape of the regular polygon prism corresponds to the number of edges one by one; the centroid can be calculated by a closed graph surrounded by the edges of the upper surface; the angle of the object is calculated from the rotation angle of the minimum circumscribed rectangle of the upper surface.

2. The method for detecting a regular polygon of a robot arm according to claim 1, wherein in step one, in order to distinguish between the operation panel and the object to be assembled, they are identified in different colors; the operation table is designed to be white, and the object to be assembled is of other colors.

3. The method for detecting regular polygon mirror of claim 1, wherein in the second step, the step of obtaining the object mask is at least obtaining a mask of an upper surface of the object.

4. The method for detecting regular polygon prism of claim 3, wherein in the second step, the method specifically comprises the following steps:

s21, transferring the RGB image obtained by shooting to an HSV color space, and using a saturation S channel in the RGB image to perform self-adaptive threshold segmentation, wherein an object mask is obtained because the saturation of an object to be assembled is higher than that of a white table top;

s22, denoising, namely removing part of noise by using morphological operations such as expansion, corrosion, connected domain with overlarge area and the like;

s23, inputting the denoised mask and the table top picture transferred to the Lab color space into the grabcut together to obtain a result of the grabcut segmentation prospect;

s24, detecting whether the area of the connected domain in the segmentation mask is too large, if so, transferring the picture into the YCrCb space to make grabcut again and using the picture as the upper surface mask of the object, otherwise, directly using the segmentation result in the current Lab color space (obtained in S23) as the upper surface mask of the object.

5. The method for detecting regular polygon of mechanical arm according to claim 1, wherein in the third step, edges of objects in the image are detected by performing edge detection on the L component in Lab color space and the V component in HSV color space, respectively, and then summing the detection results obtained by the two components.

6. The method for detecting a regular polygon prism of a robot arm as claimed in claim 5, wherein in the fourth step, noise of the edge detection result is removed by using a mask, including both internal and external noise of the edge of the object. The method is specifically implemented by performing expansion operation on a mask, and solving an intersection with an edge detection result to remove external edge noise; and then performing etching operation on the mask, and solving a difference set between the etching operation and the edge detection result to remove internal noise.

7. The method for detecting regular polygon prism of mechanical arm as claimed in claim 5, wherein in step five, using hough circle detection to detect whether there is a circle in the intersection result obtained by the object mask acting on Lab color space, if yes, obtaining the most suitable upper surface circle and entering step nine, otherwise entering step six; the method comprises the steps of obtaining an upper surface circle, namely, using Hough circle detection in an image of an HSV color space and an Lab color space respectively, adaptively finding a circle with the maximum param2 in the color space, and using a circle with an outer circle center as the upper surface circle; the param2 is an input parameter of the Hough circle detection method, represents the rule degree of a circle, and can be detected as the circle only in a shape larger than the param 2; the outer circle is chosen as the upper surface circle, based on the fact that the upper surface is always relatively close to the periphery of the picture in a top view.

8. The method for detecting regular polygon prism of mechanical arm as claimed in claim 1, wherein in the sixth step, the specific procedure for adaptively obtaining the longest line segment with a length greater than th1 is as follows:

s41, obtaining a line segment with th=th1;

s42, detecting a line segment with the length longer than th;

s44, let th- =1;

s46, detecting a line segment with the length longer than th;

s47, judging whether a straight line is detected; if the straight line is detected, recording straight line information and th, and deleting the detected straight line in the original image; if not, returning to the step S44, and enabling th- =1;

9. The method for detecting regular polygon prism of claim 1, wherein in step seven, in order to screen out the edge lines of the upper surface of the object, three operations of line grouping, line denoising and line selecting are sequentially performed:

linear denoising: removing side edge lines and other noise, and removing lines with shorter lengths according to shorter side edge lengths and shorter noise lines in a top view;

selecting a straight line: selecting an upper surface line from a collection of remaining upper and lower surface edge lines; comparing the differences between the object framed by each straight line and the mask by utilizing the rule that the upper surface straight line is closer to the mask on the upper surface of the object than the lower surface straight line, and selecting the straight line with the smallest difference for each group to be regarded as the upper surface edge;

when selecting straight lines, a closed frame is required to be constructed for each straight line, and then differences between the closed range and the mask are compared; the method for constructing the closed frame is that each straight line is matched with the straight lines closest to the center of the mask in other straight line groups, and the straight lines form the closed frame; comparing all pixels in the closed box A with all pixels in the mask B of the grabcut, and recording the number n of pixels in A but not B for each straight line; then, a line of n minimum is selected from each group, which is considered as the upper surface edge line.

10. The method for detecting regular polygon prism of mechanical arm according to claim 1, wherein in step eight, traversing every two pairs of the current straight lines, calculating the minimum distance between the intersection point of the two straight lines and the mask, and if the distance is smaller than th5, considering that the two straight lines are adjacent; the method for calculating the distance from the point to the mask is that a circle with gradually increased radius is drawn by taking the point as a circle center, and when the circle starts to intersect with the mask, the radius of the circle is the distance from the point to the mask; and after the adjacent relation of all the straight lines is obtained, calculating the vertexes of the adjacent straight lines and sequentially connecting the vertexes to obtain the edge of the upper surface of the object.