CN115476349A

CN115476349A - Clamp group checking method and device, electronic equipment and storage medium

Info

Publication number: CN115476349A
Application number: CN202110599397.0A
Authority: CN
Inventors: 李辉; 魏海永; 拱忠奇; 张震; 王帅; 魏春生; 丁有爽; 邵天兰
Original assignee: Mech Mind Robotics Technologies Co Ltd
Current assignee: Mech Mind Robotics Technologies Co Ltd
Priority date: 2021-05-31
Filing date: 2021-05-31
Publication date: 2022-12-16

Abstract

The application discloses a clamp group checking method and device, electronic equipment and a storage medium. The clamp group checking method comprises the following steps: acquiring state information of the clamp group; determining the number information of openable clamps and/or the position information of openable clamps in the state based on the state information; judging whether the number of the openable clamps meets a preset condition and/or judging whether the position information of the openable clamps meets the preset condition; and determining whether the clamp group can execute the grabbing in the state according to the judgment result. According to the invention, before the clamp group performs grabbing, whether the used grabbing mode can grab the object correctly is checked in advance, so that the grabbing stability is improved, and the problems of damage, loss and the like of the grabbed object caused by unstable gravity center in the grabbing process are avoided.

Description

Clamp group checking method and device, electronic equipment and storage medium

Technical Field

The present disclosure relates to the field of manipulator control technologies, and more particularly, to a method and an apparatus for checking a fixture set, an electronic device, and a storage medium.

Background

At present, with the wide popularization of intelligent program-controlled robots, more and more articles can be grabbed and transported by means of the intelligent program-controlled robots. For example, commodity circulation packing can snatch through intelligent programming robot to promote by a wide margin and snatch efficiency. In order to improve the grabbing efficiency and flexibly adapt to various object objects, the intelligent programmed robot is usually provided with a clamp group consisting of a plurality of clamps, so that different clamps in the clamp group can be flexibly adjusted according to different object objects. Different objects can be grabbed by different clamps, such as a sucker array, objects of similar glass materials can be sucked, and grabbing failure can be caused once rubber or bulges are met. For objects or structures that such clamps cannot grip, this is called an obstacle.

The conventional intelligent robot can only avoid obstacles and grab aiming at the single and simple condition of grabbing objects and obstacles with fixed models, and under the condition, the size, the shape and the position of the grabbed objects and the obstacles are fixed. In the prior art, a clamp is set at the center of a grabbing object and at a position which is not an obstacle according to the model and the position of the obstacle, so that the grabbing is carried out by avoiding the obstacle. However, the obstacle avoidance and grabbing method has the following defects: firstly, the grabbing method can only be used for objects with fixed models and fixed positions of obstacles, namely when the models of the objects to be grabbed are unknown or the models of the objects to be grabbed are known but the positions of the obstacles are not fixed, the obstacles cannot be accurately avoided for grabbing; secondly, the grabbing method does not determine whether the plurality of clamps can grab the object stably, if the number of the used clamps is not enough or the arrangement of the plurality of clamps is not appropriate, for example, the plurality of clamps are arranged in a straight line, after the object is grabbed in this situation, the center of gravity is unstable, the object may swing during grabbing, and the object may fall or collide with other objects except for the expected object and be damaged.

Disclosure of Invention

In view of the above, the present invention has been made to overcome the above problems or at least partially solve the above problems. Specifically, according to the embodiment, firstly, the possible grabbing modes of the clamp can be obtained, and the best grabbing mode which cannot grab the obstacle is selected to grab the object, so that the object can be accurately grabbed while avoiding the obstacle even if the object has the obstacle which cannot be grabbed; secondly, aiming at the obstacle that the clamp can not be correctly grabbed due to protrusion/recess, the invention provides a method for determining whether a non-planar structure exists in a specific area by grouping and determining the depth value difference of each pixel point in the area and the quantity of different depth values, and the method judges whether the non-planar structure exists in the area by using a numerical value statistical mode instead of a mode of determining the specific position of the non-planar structure, so that the method has high processing efficiency and strong practicability, and is also suitable for grabbing industrial scenes which possibly need to judge whether the non-planar structure exists on the surface of an object; thirdly, before the clamp group performs grabbing, the invention can check in advance whether the used grabbing mode can grab the object correctly, thereby improving the grabbing stability and avoiding the problems of damage, loss and the like of the grabbed object caused by unstable gravity center in the grabbing process; finally, the invention develops the obstacle avoidance grabbing method and the sucker verification method which are specially used for grabbing the industrial scene of the glass by using the sucker array based on the universal obstacle avoidance grabbing method and the universal clamp verification method, and can improve the accuracy and the stability of grabbing the glass by using the sucker array. Therefore, the invention solves the problem of the aspect of article grabbing in the industrial scene by using the clamp.

All the solutions disclosed in the claims and in the description of the present application have one or more of the above-mentioned innovations and, accordingly, are capable of solving one or more of the above-mentioned technical problems. Specifically, the application provides a clamp group checking method and device, electronic equipment and a storage medium.

The clamp group calibration method of the embodiment of the application comprises the following steps:

acquiring state information of the clamp group;

determining the number information of openable clamps and/or the position information of the openable clamps in the state based on the state information;

judging whether the number of the openable clamps meets a preset condition and/or judging whether the position information of the openable clamps meets the preset condition;

and determining whether the clamp group can execute the grabbing in the state according to the judgment result.

In some embodiments, the determining the position information of the openable clamp comprises determining the position information of the openable clamp according to the outline information of the article to be grabbed.

In some embodiments, the conditions that the number of openable clamps needs to satisfy are preset based on the weight of the item to be gripped.

In some embodiments, the conditions to be met are preset based on the center of gravity of the article to be gripped.

In some embodiments, the conditions that need to be met by the position of the openable fixture include: the position connecting lines of the plurality of clamps cannot form a straight line.

In some embodiments, the determining whether the gripping can be performed in the state according to the judgment result includes determining that the gripping can be performed when the number of the grippers satisfies a preset condition and the positions of the grippers satisfy the preset condition.

The utility model provides an embodiment's anchor clamps group verifying attachment includes:

the state information acquisition module is used for acquiring the state information of the clamp array;

the information determining module is used for determining the number information of the openable clamps and/or the position information of the openable clamps based on the state information;

the condition judging module is used for judging whether the number of the openable clamps meets a preset condition and/or judging whether the position information of the openable clamps meets the preset condition;

and the grabbing determining module is used for determining whether the clamp group can carry out grabbing in the state according to the judgment result.

In some embodiments, the information determination module determines position information of the openable clamp according to profile information of the object to be gripped.

In some embodiments, the number of openable clamps that need to be met is preset based on the weight of the item to be grasped.

In some embodiments, the grasping determination module determines that the grasping can be performed when the number of the clamps satisfies a preset condition and the positions of the clamps satisfy the preset condition.

The electronic device of the embodiment of the present application includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor implements the clamp group verification method of any one of the above embodiments when executing the computer program.

The computer-readable storage medium of an embodiment of the present application has stored thereon a computer program that, when executed by a processor, implements the jig set verification method of any of the above-described embodiments.

Additional aspects and advantages of the present application will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the present application.

Drawings

The above and/or additional aspects and advantages of the present application will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings of which:

FIG. 1 is a schematic flow chart of a fixture obstacle avoidance gripping method according to some embodiments of the present disclosure;

FIG. 2 is a schematic flow chart of a method for determining a non-planar structure of a surface of an article according to certain embodiments of the present disclosure;

FIG. 3 is a schematic flow chart of a fixture verification method according to certain embodiments of the present application;

FIG. 4 is a schematic flow chart of a glass gripping method using suction cups according to certain embodiments of the present application;

FIG. 5 is a schematic view of a glass and obstacle point cloud and suction cup arrangement according to certain embodiments of the present application;

FIG. 6 is a schematic flow chart illustrating the determination of an obstacle during glass capture in accordance with certain embodiments of the present application;

FIG. 7 is a schematic illustration of a process for applying glue to a strip to be affixed according to certain embodiments of the present application;

FIG. 8 is a schematic flow chart of a method for verifying a suction cup array according to some embodiments of the present application;

FIG. 9 is a schematic diagram illustrating a fixture obstacle avoidance gripping device according to some embodiments of the present application;

FIG. 10 is a schematic view of a non-planar configuration determining apparatus for determining the configuration of a surface of an article according to certain embodiments of the present application;

FIG. 11 is a schematic diagram of a fixture verification apparatus according to certain embodiments of the present application;

FIG. 12 is a schematic view of a glass gripping apparatus using suction cups according to certain embodiments of the present application;

FIG. 13 is a schematic diagram of a chuck array calibration apparatus in accordance with certain embodiments of the present application;

FIG. 14 is a schematic diagram of an electronic device according to some embodiments of the present application.

Detailed Description

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

Fig. 1 is a schematic flowchart illustrating a robot-based gripper obstacle avoidance gripping method according to an embodiment of the present invention, and as shown in fig. 1, the method includes:

and S100, acquiring point cloud information of the object group to be grabbed.

The group of articles to be gripped may comprise one or more articles to be gripped and may also comprise one or more obstacles. The obstacle may exist on the object to be grabbed, such as a bulge on the object to be grabbed, a rubber strip stuck on the object to be grabbed, and the like; obstacles may also be present outside the object to be gripped, for example the object to be gripped may be placed in a stack of layers, there may be rubber pads between the layers, etc. Accordingly, the point cloud information of the group of items to be grabbed may comprise point cloud information of one or more items to be grabbed and/or obstacles.

As an example, the point cloud information may be obtained by a 3D industrial camera, and the 3D industrial camera is generally equipped with two lenses, which respectively capture the object group to be grabbed from different angles, and after processing, the three-dimensional image of the object can be displayed. And (3) placing the object group to be grabbed below the visual sensor, simultaneously shooting by the two lenses, calculating the X, Y and Z coordinate values of each point of the glass to be glued and the coordinate orientation of each point by using a universal binocular stereo vision algorithm according to the relative attitude parameters of the two obtained images, and further converting the coordinate values into point cloud data of the object group to be grabbed. In specific implementation, the point cloud may also be generated by using elements such as a visible light detector such as a laser detector and an LED, an infrared detector, and a radar detector.

The point cloud data acquired in the above mode are three-dimensional data, and in order to filter data corresponding to dimensions with small influence on grabbing, data processing amount is reduced, data processing speed is accelerated, efficiency is improved, and the acquired three-dimensional to-be-grabbed article group point cloud data orthographic projection can be mapped onto a two-dimensional plane.

As an example, a depth map corresponding to the forward projection may also be generated. A two-dimensional color map corresponding to a three-dimensional object region and a depth map corresponding to the two-dimensional color map can be acquired in a direction perpendicular to the depth direction of the object. The two-dimensional color image corresponds to an image of a plane area vertical to a preset depth direction; each pixel point in the depth map corresponding to the two-dimensional color image corresponds to each pixel point in the two-dimensional color image one by one, and the value of each pixel point is the depth value of the pixel point.

Step S110: and generating a searching state based on the point cloud information of the object group to be grabbed and the searching boundary factor parameters of the clamp.

The search boundary factor parameters of the fixture include the control boundary of the controllable parameters of the fixture, the control granularity and other boundary factor parameters. The search boundary factor parameters and values may be different depending on the actual circumstances, such as the type of gripper used, the type and size of the object to be gripped, etc.

As an example, the controllable parameters may include a moving distance in the X direction, a moving distance in the Y direction, and a rotation angle, the moving distance boundary (i.e., the controllable boundary) in the X direction may be set to-500 mm to 500mm, and the control granularity is 100mm, so that the jig can be moved from-500 mm to 500mm in the X direction in units of 100mm, and thus there are 11 search states in the X direction; the moving distance boundary in the Y direction can be set to be-100 mm to 100mm, the granularity is controlled to be 50mm, and then the clamp can move to 100mm from-100 mm in the Y direction by taking 50mm as a unit, so that 5 searching states are in total in the Y direction; the rotation angle can be set to-50 degrees to +50 degrees, the granularity is controlled to be 10 degrees, and the clamp can rotate to +50 degrees from-50 degrees in a unit of 10 degrees, so that the total number of 11 search states on the rotation angle is obtained. With the above-described setting of the search boundary factor parameter, there are 11 × 5 × 11=605 search states in total. In addition, other boundary factor parameters may be set, for example, if the moving distance of the clamp cannot exceed 100mm of a certain side, the clamp can only move to a position 100mm away from the side when moving to the side.

In order to control the clamp conveniently, the information such as the shape and the size of the clamp can be configured, and the configuration information can be stored by using a json file configuration. The configuration information may be different depending on the actual situation, such as the clamp used or the object to be gripped. For example, when using an array of suction cups for gripping, the configuration information may include the location of the individual suction cups relative to the center of the entire array of suction cups, the number, radius, etc. of the individual suction cups.

The scheme of the invention can be used for various types of clamps, for example, various types of universal clamps are included, and the universal clamp refers to a clamp which has a standardized structure and has a larger application range, for example, a three-jaw chuck and a four-jaw chuck for a lathe, a flat tongs and an indexing head for a milling machine and the like. For another example, the clamping apparatus may be divided into a manual clamping apparatus, a pneumatic clamping apparatus, a hydraulic clamping apparatus, a gas-liquid linkage clamping apparatus, an electromagnetic clamping apparatus, a vacuum clamping apparatus, and the like, according to a clamping power source used by the clamping apparatus. The present invention does not limit the specific type of the clamp as long as the article grasping operation can be achieved.

Step S120: for each search state, obstacle determination is performed, and the search state determined by the obstacle determination is set as an alternative search state.

The obstacle determination is used for determining the obstacle situation faced by the clamp in a certain search state, such as whether the obstacle exists or not, whether the obstacle affects the grabbing or not, and if the obstacle does not exist or does not affect the grabbing, the obstacle determination is passed, otherwise, the obstacle determination is not passed. If a plurality of jigs or a jig array composed of a plurality of jigs is used for grasping, obstacle determination may be performed separately for each jig. One of the important points of the present invention is to traverse all search states to perform obstacle determination, and therefore, the obstacle determination method is not limited, and any obstacle determination method can be used in the present invention. The obstacle condition of the article to be grasped can be determined by only one obstacle determination method, or a plurality of obstacle determination methods can be combined for determination. If a combination of a plurality of judgment modes is adopted, the plurality of judgment modes can be carried out in a certain sequence or in parallel, and when any one of the plurality of judgment modes does not pass, the obstacle judgment of the search state is considered not to pass.

In an optional implementation mode, the invention exemplarily provides four obstacle judgment methods of boundary obstacle avoidance, fixed obstacle avoidance, protrusion/depression obstacle avoidance and custom obstacle avoidance.

Boundary obstacle determination

Some clamps are not able to grip the article at the edge of the article and when using these clamps the boundary of the article itself will constitute an obstacle to gripping and it is therefore necessary to identify the boundary of the article to avoid gripping the article to be gripped at the boundary. In order to identify the boundary, point cloud data of the object group to be captured may be obtained, and specifically, the method of step 100 may be used to obtain the point cloud data, and the portion of the point cloud data at the edge is intercepted to obtain the contour point cloud of the object group to be captured. Therefore, in order to accurately define the outline of the edge of the article, the outline point cloud of the edge of the article can be projected and mapped onto a two-dimensional plane to obtain the outline point of the edge of the article group to be grabbed. Since the contour points of the edges of the group of articles to be grasped are two-dimensional data at this time, the contour points of the edges of the group of articles to be grasped can be made clearer based on the two-dimensional data.

In order to avoid the incompleteness of the acquired point cloud, contour points at four corners of the article to be grabbed can be acquired from the obtained two-dimensional pattern of the article group to be grabbed. And obtaining the minimum external rectangle of the glass to be glued according to the contour points of the four corners of the glass to be glued, and regarding the minimum external rectangle as a contour quadrangle of the object group to be grabbed. Or the contour points at the four corners are sequentially connected clockwise from left to right and from top to bottom to form edge lines of the object to be grabbed, and the quadrangle formed by the edge lines is regarded as the contour quadrangle of the object group to be grabbed.

For each search state, judging the relation between the center of the clamp and the edge contour point of the article in the search state, if the center of the clamp is positioned outside the obtained contour, indicating that the clamp is positioned at the boundary of the article to be grabbed, at the moment, judging that a boundary obstacle exists and judging that the obstacle does not pass; if the center of the clamp is located within the outline, the obstacle is judged to pass.

Fixed obstacle determination

For example, when the object to be grabbed is a steel plate or glass, a plurality of steel plates or glasses may be stacked, each steel plate or glass may be separated by a rubber pad or foam, and after the upper steel plate or glass is grabbed, the rubber pad or foam may be left on the lower steel plate or glass, thereby being an obstacle when grabbing. In such cases, it is necessary to determine obstacles to fixed obstacles whose positions are uncertain and which affect the grip of the gripper to grasp an article to be grasped.

In order to identify a fixed obstacle, three-dimensional point cloud data of the group of items to be grabbed may be acquired and mapped onto a two-dimensional plane, wherein the point cloud data may be acquired and mapped using the method of step 100. The fixed obstacles usually have uniform specifications, such as uniform size or shape, so that specification parameters of the fixed obstacles can be preset, for each search state, whether an article meeting the preset specification parameters exists below the clamp in the search state can be judged based on the two-dimensional point cloud data, if yes, the fixed obstacles exist, and the obstacles are not judged to pass; otherwise, the obstacle determination is passed.

Bump/pit obstacle determination

Depending on the article to be gripped, the article itself may have raised or depressed portions, which may be inherent to the article itself or may be caused by a collision or the like. Some clamps may not be able to properly grip an item in a raised or recessed location, for example, if a suction cup clamp is used to suck on a raised edge, the suction cup may not be in close proximity to the object and may leak air, and thus may not be able to grip the item in that location. Therefore, in some cases, it is necessary to determine whether or not the protrusion/recess is an obstacle.

The applicant found in the course of research and development that the projections/depressions as part of the object to be gripped are difficult to distinguish by means of the contour point cloud data and, unlike fixed obstacles, the projections/depressions do not usually have a fixed specification, and the projections/depressions of the object cannot be identified in the conventional manner.

In order to solve the problem, the invention provides a method for judging a non-planar structure on the surface of an object, which is also one of the important points of the invention, and the method can be used for judging a convex/concave obstacle in some embodiments of the invention, and can also be used under other conditions that whether the surface of the object has the non-planar structure needs to be judged, and is not limited to be used in the scheme of obstacle avoidance and grabbing. Fig. 2 is a flow chart showing a method for determining a non-planar structure of an object surface according to the present invention, wherein the non-planar structure includes a convex structure, a concave structure, and the like. As shown in fig. 2, the method includes:

and 200, acquiring two-dimensional plane depth map information of the area to be determined on the surface of the article.

And after orthographic projection of the three-dimensional point cloud data of the article is mapped onto a two-dimensional plane, generating a depth map corresponding to the orthographic projection as the depth map information of the two-dimensional plane of the area to be judged. The depth map may be acquired in a similar manner as step 100. Instead of acquiring the depth map of the entire article, the depth map of a partial region or a region to be determined of the article may be acquired.

And step 210, obtaining the depth value of each pixel point in the area to be judged according to the two-dimensional plane depth map information.

And each pixel point in the depth map corresponds to each pixel point in the orthographic projection two-dimensional map one to one, and the value of each pixel point is the depth value of the pixel point. In order to determine whether a non-planar structure such as a protrusion or a depression exists in the region to be determined, the depth value of each pixel point in the region to be determined may be obtained in this step.

The obtained depth values are grouped, step 220.

The depth values of the pixels of the planar structure are the same or have small differences, and the depth values of the pixels of the non-planar structure such as the convex or the concave have large differences. In this step, the obtained depth values of all the pixel points are grouped, and if a plurality of same depth values exist, the depth values can be combined into one depth value and then grouped.

In grouping, all the obtained depth values may be grouped in groups of two or more depth values. The grouping mode may be random grouping or any other grouping mode, which is not limited by the present invention. As a preferred embodiment, all depth values may be sorted first, then the highest and lowest values are grouped together, the next highest and next lowest values are grouped together, and so on, and all depth values are grouped together.

Step 230, calculating a difference value of the depth values in each group, and comparing the difference value with a preset difference value threshold; and/or, calculating the number of packets and comparing the number of packets with a preset threshold of the number of packets.

The difference value of the depth values can reflect the distance between the pixel points, and the larger the difference value of the depth values between the two pixel points is, the larger the height difference between the two pixel points is, so that the difference value of the depth values can reflect the degree of unevenness of the region to be judged. In addition, the number of the grouped pixels can reflect the number of the pixels with different heights, and the larger the number of the grouped pixels in the region to be judged is, the more the pixels with fluctuation exist. The number of groups can reflect the degree of concavity and convexity of the region to be determined from another angle. The difference in depth values or the number of groupings can be used alone to determine whether non-planar structures exist for the object. As a preferred embodiment, the difference of the depth values and the number of groups can also be used to jointly determine whether the object has the non-planar structure, and the applicant finds that this way can greatly increase the accuracy of the non-planar structure determination compared with the single use. In another embodiment, the threshold of the difference and/or the threshold of the number of packets may be set according to the needs of the actual situation, for example, when the method is applied to grabbing obstacle avoidance, for some clamps, the smaller protrusions do not affect grabbing, so the threshold may be set to be larger, and thus the smaller protrusions may not be determined as an obstacle that cannot be grabbed.

And step 240, judging the non-planar structure condition of the area according to the comparison result.

The non-planar structure condition may include whether a non-planar structure exists, and when a difference value of depth values and/or the number of packets exceeds a threshold value, it is determined that a non-planar structure exists; it is also possible to include the degree of relief of the non-planar structure, for which a plurality of levels can be set. In other embodiments, multiple levels of thresholds may be provided, and the presence or level of waviness of the non-planar structure may be determined by a combination of different thresholds. As a preferred embodiment, when the non-planar structure is determined by using the difference of the depth values and the number of packets jointly, it may be determined that the non-planar structure exists when the difference of the depth values exceeds the difference threshold and the number of packets exceeds the number of packets threshold.

Custom obstacle determination

In an industrial scene, there are some obstacles that cannot be recognized by the robot vision technology, for example, when the object to be grasped is glass, the glass may be coated with glue, the coating is usually transparent, the characteristics are not obvious, the glass cannot be recognized correctly, and the clamp cannot grasp the glass at the coating position. To identify such obstacles, the size, shape, and location of the obstacle may be customized before grabbing. In this way, for each search state, whether or not there is an obstacle under the jig is determined based on the obstacle information defined in advance. In one embodiment, the edge profile of the obstacle may be generated according to self-defined obstacle information, and then for each search state, the relationship between the center of the jig in the search state and the edge profile of the self-defined obstacle is determined, and if the center of the jig is located outside the obtained profile, it is determined that there is an obstacle and the obstacle determination does not pass; if the center of the clamp is located within the contour, the obstacle is determined to pass. In another embodiment, the retraction distance of the edge profile may be set, that is, the original edge profile of the obstacle is retracted inward by a certain distance, and then the determination of the relative position of the center of the fixture and the edge profile is performed.

In step S120, the scheme may also be extended or improved as follows:

for a certain searching state, if the clamp passes the obstacle judgment, setting to open the clamp in the searching state; if not, in the searching state, the clamp is set to be in a closed state;

if a plurality of jigs or a jig array constituted by a plurality of jigs is used for gripping, obstacle determination may be performed for each of the plurality of jigs when the obstacle determination is performed;

in a certain search state, if at least one jig passes obstacle determination, the search state may be set as an alternative search state;

when the obstacle determination is performed, it may be determined that the obstacle determination is passed when the obstacle is present but the capture is not affected, and it may be determined that the obstacle determination is not passed when the obstacle is present and the capture is affected.

Step S130, selecting the best search state from the candidate search states.

The way of selecting the best search state is different according to the actual situation, such as the object to be grasped and the used clamp, and the invention is not limited to this, and any selection way can be used in the solution of the invention. As a preferred embodiment, the best search condition may be selected according to the distance of the gripper from the center of the object to be gripped in each search condition. Generally, the closer the gripping position of the gripper is to the center of the article, the more stable the gripping is, and therefore, the search state in which the gripper is closest to the center of the article can be set as the optimum search state. In order to calculate the distance from the clamp to the center of the article, the center position P1 of the clamp in a certain search state may be determined, then the circumscribed rectangle of the article is obtained, then the center position P2 of the object is obtained according to the circumscribed rectangle of the article, and then the distance between P1 and P2 is the distance from the clamp to the center of the article.

If a plurality of grippers or a gripper array consisting of a plurality of grippers is used for gripping, it is also possible to select the optimum search state according to the number of grippers judged by the obstacle or to select the optimum search state in consideration of the distance of the grippers from the center of the article and the number of grippers judged by the obstacle in combination. In a preferred embodiment, the number of the search states can be selected according to the number, and if the search state with the largest number passes through is multiple, the search state with the clamp closest to the center position of the article is further selected from the multiple search states passing through as the optimal search state; in other embodiments, a number threshold may be preset, and search states in which the number of opened clamps exceeds the threshold are selected first, and then a search state closest to the center is selected from the search states as an optimal search state.

And step S140, grabbing the object to be grabbed based on the optimal searching state.

The robot sets the position, angle, number, etc. of the gripper according to each configuration parameter of the gripper in the optimum search state, and then performs gripping of the article and places the article at a designated position. The designated positions comprise the ground, an article placing rack and the like, and some articles have position and state requirements when placed, such as the articles need to be placed vertically, and the articles cannot be too high or too low. Therefore, the distance from the center of the clamp of the final planning result to the designated edge of the object can be obtained, and the object can be accurately placed at the designated position.

The inventors have found that when a plurality of grippers or an array of grippers consisting of a plurality of grippers are used for gripping, if the number of grippers used in the optimum search state is insufficient or the arrangement of the plurality of grippers is not proper, for example, the plurality of grippers are arranged in a straight line, the center of gravity of the article may be unstable during gripping, and thus the article may swing, fall off, or collide with another article except for the intended article and be damaged during gripping. Therefore, the invention also provides a method for the gripping verification of the fixture, which can determine whether the gripping mode to be used can stably grip the article before the article is gripped, so that unnecessary loss in gripping is avoided. The method can be used in the obstacle avoidance grabbing method of the invention, and can also be used in other grabbing scenes, the invention does not limit the specific use situation, as long as a certain grabbing scene uses a plurality of clamps or a clamp array consisting of a plurality of clamps, the method can be applied to checking, for convenience of description, the plurality of clamps or the clamp array consisting of a plurality of clamps are collectively called as a clamp group in the invention.

FIG. 3 shows a flow diagram of a method for verifying a gripper group according to an embodiment of the invention. As shown in fig. 3, the method includes:

step S300: and acquiring the state information of the clamp group.

The state of the gripper group refers to a combination of specific parameters of each gripper in the gripper group, and the gripping performed by the gripper group in a certain state refers to an action in which the gripper is configured and performs gripping based on each parameter in the state, for example, assuming that a certain gripper has 3 grippers, respectively No. 1 to No. 3, no. 1 gripper is on the left edge of an article to be gripped and has a rotation angle of 30 degrees, no. 2 gripper is on the right edge of the article and has a rotation angle of 0 degrees, no. 1 and No. 2 grippers are both set to be open, and No. 3 gripper is set to be not open outside the article, such a combination of specific parameters is a state of the gripper group. The state information may include position information of the clamp group in the state, position information of each clamp in the clamp group, angle information, and information about whether the clamps can be opened in the current state, which clamps can be opened, and the like.

Step S310: the number information of the openable clamps and/or the position information of the openable clamps are determined based on the state information of the clamp group.

When the position information of the openable clamp is determined, the position of the clamp relative to the object to be grabbed can be determined according to the outline information of the object to be grabbed and by combining the boundary parameters of the clamp; the position of the fixture relative to other fixtures, or the position of the fixture relative to the fixture array, may also be determined as the position of the fixture. In one embodiment, the plurality of jigs or jig arrays may be divided into a plurality of areas, and the position information of the jigs may be represented by the areas where the jigs are located.

Step S320: judging whether the number of openable clamps meets a preset condition or not; and/or judging whether the position information of the openable clamp meets a preset condition.

Insufficient number of openable clamps, or bad position of openable clamps, may result in unstable gripping. If the number of openable clamps is not adjustable or does not need to be adjusted, or the position of the clamps is not adjustable or does not need to be adjusted, only one item can be judged. Those skilled in the art will appreciate that the determination of both conditions significantly improves the probability of the clamp holding stability compared to determining only one condition. The conditions to be met, for example the conditions to be met by the number of openable clamps, may be preset according to information of the object to be gripped, for example according to the weight of the object to be gripped. For example, when a heavy object is grabbed, the number of the clamps can be set to be 5, so that the preset condition can be met when the number of the openable clamps exceeds 5; and for lighter articles, 3 items may be provided. When the density of the article to be grabbed is uniform, the number of the clamps to be used can be set according to the area size of the article, for example, the article with a larger area is provided with 5 clamps under the preset condition, and the number of the articles with a smaller area is 3, so that the area of the article can be determined according to the profile information of the article. For the condition that the position information of the openable clamps needs to meet, usually at least avoiding that a plurality of clamps are positioned on a straight line or are approximately positioned on a straight line, according to the requirement of the actual situation, more specific position information can be set, for example, the connecting lines of the positions of the openable clamps must form a stable triangle; the position information that the openable clamp needs to satisfy can also be set according to the center of gravity of the object to be grabbed so as to ensure that the center of gravity is stable during grabbing, for example, when the center of the object to be grabbed has a large mass, the preset condition may be that an openable clamp needs to be arranged near the center of the object to be grabbed or a certain number of openable clamps need to be arranged.

Step S330: and determining whether the clamp group can execute grabbing under the state according to the judgment result.

In specific implementation, the grabbing may be performed only when the number of openable clamps satisfies the condition or the position information of the openable clamps satisfies the condition. The grabbing may also be performed when both satisfy the condition. When the condition of the position information includes a plurality of conditions, the capturing may be performed when the plurality of conditions are satisfied.

In one embodiment, the gripper verification may be performed after determining that the gripping of the article is performed in a certain state, and before performing the gripping. If the method is used in combination with the obstacle avoidance grasping scheme, steps S300-S330 may be performed between step S130 and step S140 in the foregoing embodiment, that is, after the optimal search state is selected, the optimal search state is used as the state of the fixture array to perform fixture verification, and it is determined whether the optimal search state can be used for grasping. In another embodiment, it may also be determined in conjunction with a method of fixture verification that the grabbing of the article is performed in a certain state. If the method is used in combination with the obstacle avoidance grabbing scheme, steps S300 to S330 may be executed in step S120 or step S130 of the foregoing embodiment, for example, the alternative search state may be selected in a fixture verification manner, or after the alternative search state is selected, the alternative search state is first used as a state of the fixture array, a search state in which grabbing is not executable is removed from the alternative search state by a fixture verification method, and then an optimal search state is selected according to the number of fixtures and a distance from the center.

The obstacle avoidance grasping method and the fixture grasping and checking method of the invention do not limit the specific fixture and application scenarios. In order to be able to apply the above method in an industrial scenario where the glass is grabbed using the array of suction cups, the inventors have paid hard work to further refine the method to fit the scenario, which is also one of the main points of the present invention.

Fig. 4 shows a flowchart of a glass obstacle avoidance gripping method using a suction cup array according to a preferred embodiment of the present invention, the method comprising:

step S400: and acquiring point cloud information of the glass to be grabbed and the obstacle.

In this embodiment, the glasses may be stacked in layers with a rubber pad between each layer of glass to separate the glasses. A manner similar to step S100 may be used to obtain a rubber pad point cloud and an object point cloud itself, and orthographically project the point clouds to a 2D map and generate a corresponding depth map after orthographic projection. Fig. 5 shows a point cloud image obtained in this way, in which the black part is transparent glass and the white part is a point cloud of non-transparent parts.

Step S410: and generating a search state based on the point cloud information and the search boundary factor parameters of the sucker array.

The search state of the suction cup includes different position states of the suction cup on the object and a rotation state of the suction cup itself. The specified search boundary factor parameters may include X-direction, Y-direction, and rotation, and may also include the distance of the chuck boundary to the specified edge. And generating a plurality of different search states according to different search boundary parameters. As shown in fig. 5, the suction cup array used comprises 10 suction cups, which are numbered 1-10 respectively, and fig. 5 shows the positions of the 10 suction cups in a certain search state, and in other search states, the suction cups may be in other positions or have different rotation angles. As an example, the moving distance boundary (i.e. the controllable boundary) in the X direction may be set to-500 mm to 500mm, and the control granularity is 100mm, then the jig can move from-500 mm to 500mm in the X direction in units of 100mm, so that there are 11 search states in the X direction; the moving distance boundary in the Y direction can be set to be-100 mm to 100mm, the granularity is controlled to be 50mm, the clamp can move to 100mm from-100 mm in the Y direction in a unit of 50mm, and thus 5 search states are totally arranged in the Y direction; the rotation angle can be set to-50 degrees to +50 degrees, the granularity is controlled to be 10 degrees, and the clamp can rotate to +50 degrees from-50 degrees in a unit of 10 degrees, so that the total number of 11 search states on the rotation angle is obtained. With the above setting of the search boundary factor parameter, there are 11 × 5 × 11=605 search states in total. It is also possible to set that the movement distance of the jig cannot exceed 100mm of the upper boundary, and the jig can move only up to 100mm from the upper side in the Y direction.

Step S420: and judging whether an obstacle exists below each sucker or not according to each searching state, and if so, closing the sucker in the searching state.

As shown in fig. 6, determining whether an obstacle exists under the suction cup may include the steps of:

step S421: and judging whether glass boundary obstacles exist below the sucker.

In the current search state, for each chuck, it is checked whether the single chuck center is inside the object outline. Whether the center of a single sucker is located inside the outline of an object or not can be judged by judging the point cloud occupation ratio in the area below the sucker, wherein the point cloud occupation ratio refers to the proportion of a point cloud area to the whole area in the area below the sucker. When the suckers are positioned on the glass boundary, point clouds are increased remarkably, the suckers 4-6 in the picture 5 are positioned on the glass boundary, white point cloud areas below the suckers are larger obviously, therefore, a point cloud area ratio threshold value can be preset, and when the point cloud area ratio in the area below the suckers exceeds the threshold value, the situation that glass boundary obstacles exist below the suckers is judged. The occupation ratio threshold value can be set automatically according to the needs of actual conditions, such as suction force of a suction cup and the condition of an obstacle, and the specific value is not limited by the invention. The inventor finds that the proportion threshold value is selected within 5% -40% to improve the accuracy of the boundary obstacle judgment, and 10% is the best.

Step S422: and judging whether a rubber pad obstacle exists below the sucker.

In this embodiment, the glass layers are stacked one on another and separated from one another by rubber pads, and except for the glass layer on the uppermost layer, a rubber pad for separating the glass layers is present on each glass layer on the lower layer. Therefore, after the upper layer of glass is grabbed away, the lower layer of glass is provided with the remaining rubber pad. Since the size of the rubber blocks is fixed and known, the rubber blocks can be filtered out if they are present according to the size. In one embodiment, the obstacle point cloud area may be preset, and when the point cloud area in the area below the suction cup exceeds the obstacle point cloud area, it is determined that there is a rubber pad obstacle below the suction cup. The obstacle point cloud area can be set arbitrarily according to the size of the used rubber pad.

Step S423: and judging whether the adhesive tape obstacle exists below the sucker.

In an industrial scene, glue may be applied near the border of the glass, the glue strip formed after the glue application is usually transparent, the point cloud feature of the glue strip is not obvious and difficult to identify, but the suction cup cannot grab the glass at the glue strip. In order to correctly identify the rubber strip obstacle, the position of the rubber strip can be set by the user, and the robot judges whether the rubber strip obstacle exists below the sucker array or not according to the rubber strip obstacle information set by the user. Specifically, can set up and generate a round adhesive tape near the marginal profile of glass according to the user, the user can set up the position of adhesive tape, and the distance and the adhesive tape width of contracting in, also can generate two adhesive tapes on same limit. Fig. 7 shows 4 different gluing processes, for process 1, no strip is needed, nor strip information is preset; for the process 2, two inner-layer tracks and two outer-layer tracks are adopted, and the inner-layer track section with a specified edge is generated independently by sticking a strip at a position which is not coated between the inner-layer track and the outer-layer track or sticking a strip on the inner-layer track; for the process 3, the opening section, namely the section without a track on the right side in the figure, is pasted with a strip; for process 4, the opening section outside the inner track, i.e., the section in the figure at a similar position to the opening section of process 3, is taped. For the above processes 2-4, the stripe-pasting obstacle information is set in advance at the position where the stripe is needed. So, after the sticker, the robot can judge whether there is the adhesive tape obstacle in the sucking disc below region according to the adhesive tape information that the user set up.

Step S424: and judging whether a convex obstacle exists below the sucker.

The surface of the glass may have bulges, and if the sucker is sucked at the bulge ridge, the sucker cannot be completely attached to the surface of the glass, so that the sucker leaks air and the sucking fails. Whether protrusion obstacles exist in the area can be judged according to the depth difference value between each pixel point in the point cloud of the area below the sucker and the number of the pixel points. In one embodiment, the depth values of all point clouds in the area under the chuck may be obtained, sorted, and then the highest depth value paired with the lowest depth value, the next highest paired with the next lowest, the third highest paired with the third lowest, \ 8230; \ 8230, all depth values paired in this manner. Calculating the difference between the paired depth values, comparing the depth difference with a preset depth difference threshold, comparing the paired number with a preset logarithmic threshold, and if the height difference exceeds the threshold and the paired number exceeds the threshold, judging that a bulge obstacle exists in the area below the sucker. The depth difference threshold and the logarithm threshold can be set according to the needs of actual conditions, such as suction force of a sucker and the condition of an obstacle, and specific values are not limited by the invention. The inventor finds that the accuracy of judging the convex obstacle can be improved by selecting the logarithmic threshold and the depth difference threshold within the following ranges: the logarithmic threshold may be selected from between 10 pairs and 50 pairs, with 20 pairs being the best; the depth difference threshold may be selected within 0.500mm-0.005mm, preferably 0.015 mm.

As a preferred embodiment, the obstacle determination may be performed strictly in the order of steps S421 to S424, and after determining that there is an obstacle and closing the suction cup in the previous step, the subsequent steps are not performed, for example, if it is determined that there is a glass boundary obstacle below the suction cup in step S421, the suction cup is closed, and steps S422 to S424 are not performed. Since the four steps are relatively independent, the obstacle condition of the glass can be determined only by the obstacle determination method of any one of the steps S421 to S424, or any combination of a plurality of the steps can be determined, for example, the combination of the steps S421 and S422 is used, and the steps S423 and S424 are not used. If a combination of steps is used, the steps may be performed in a sequential order or in parallel. It should be noted that although the obstacle judgment can be implemented in a single step or various combinations, if the obstacle judgment method of the preferred embodiment of the present invention, which executes steps S421 to S424 sequentially, is used, the accuracy of the obstacle judgment can be improved compared to other judgment methods, and because a obstacle with a high probability is judged first and a simpler algorithm is used first, the efficiency of the obstacle judgment can be improved, and thus the efficiency of robot grasping is improved, which is particularly advantageous in an industrial scene.

And step S430, selecting alternative searching states according to the opening condition of the sucker in each searching state.

The selected conditions, such as the number of open, whether a particular suction cup is open, etc., can be set by itself. In one embodiment, if there is at least one suction cup that is open, the search state is selected as the alternative search state.

And step S440, selecting the optimal searching state from the alternative searching states according to the opening number of the suckers and/or the distance between the suckers and the center of the glass.

The optimum search state may be selected according to the distance of each search state from the center of the glass to be grasped, and generally, the closer to the center of the article, the more stable the grasping is, and thus the search state closest to the center of the article may be set as the optimum search state. And for each searching state, determining the center position P1 of the clamp in the searching state, and then obtaining the center position P2 of the object through the circumscribed rectangle of the glass, wherein the distance between P1 and P2 is the distance between the clamp and the center of the glass. The search state with the largest number of passes may be selected as the optimum search state according to the number of suction cups determined by the obstacle. In a preferred embodiment, the best search state can be selected by combining the opening number of the suction cups and the distance between the suction cups and the center of the glass, for example, the best search state can be selected according to the number, and if a plurality of search states with the largest number are selected, the search state closest to the center position of the article is further selected from the selected plurality of search states as the best search state; in another embodiment, a number threshold may be preset, and the search states in which the number of activated suction cups exceeds the threshold are selected first, and then the search state closest to the center is selected from the search states as the optimal search state.

And step S450, grabbing the glass by using the optimal searching state.

The grabbed glass usually needs to be placed on the ground or an article placing frame in an industrial scene, if the glass is placed on the frame, the glass cannot exceed the edge of the frame too much, so that the distance from the center of the sucker array of the final planning result to the appointed edge of the glass can be obtained, and the robot can accurately control the placing position of the glass when placing the glass, so that the glass is accurately placed at the appointed position.

Fig. 8 is a flowchart illustrating a glass grab verification method using a suction cup array according to a preferred embodiment of the present invention. As shown in fig. 8, the method includes:

step S500: all the suction cups in the array are grouped according to the morphology of the suction cup array.

The groups may be based on the distribution of the suction cups over the suction cup array. In the embodiment shown in fig. 5, the suction cup array comprises 12 suction cups, which are respectively numbered 1-12, and the suction cups can be divided into 4 groups of upper left, upper right, lower left and lower right according to their relative positions to the suction cup array, specifically, suction cups 3 and 9 are the first group, suction cups 4 and 8 are the second group, suction cups 1,2 and 10 are the third group, and suction cups 5, 6 and 7 are the fourth group.

Step S510: and acquiring the state information of the sucker array.

The state of the sucker array refers to a combination of specific parameters of each sucker in the sucker array, and the sucker array performs grabbing in a certain state refers to an action that the sucker array performs configuration and grabbing based on each parameter in the state, for example, if a certain sucker array has 3 suckers, which are respectively No. 1-3, no. 1 sucker is at the left edge of an article to be grabbed, and the rotation angle is 30 degrees, no. 2 sucker is at the right edge of the article, and the rotation angle is 0 degree, no. 1,2 suckers are all set to be on, and No. 3 sucker is outside the article, and such a combination of specific parameters is a state of the sucker array. The state information of the sucker array comprises the position of the sucker array, the position and the angle of each sucker in the sucker array, and information about whether the suckers can be opened or not in the current state, which suckers can be opened and the like.

Step S520: and determining the quantity information of the openable suckers and the grouping information of the openable suckers based on the state information of the sucker array.

In this step, for each chuck state, the number of openable chucks and the grouping of openable chucks in that state are determined.

Step S530: and judging whether the number of the openable suckers meets the preset condition and/or whether the grouping information of the openable suckers meets the preset condition.

The grabbing instability can be caused by insufficient quantity of the opened suckers or poor positions of the opened suckers. For the number of open suction cups a single threshold value can be set, e.g. more than 5 suction cups must be exceeded to allow gripping. A threshold value of the number of the suckers can be preset according to the size of the glass to be grabbed, for example, a minimum threshold value of the number of the suckers can be set to be 3, namely, at least 3 suckers can be started no matter how large the area of the glass is; 4 suckers can be arranged on the glass area of 1 square meter to 2 square meters, and 5 suckers can be arranged on the glass area of more than 2 square meters. In this way, the area of the glass and the opening number of the suckers are judged, and whether the grabbing can be executed is judged. Thus, when the number of the openable suckers is judged, the area of the glass is also judged, for example, the area of the glass can be judged to be 2 square meters, the number of the openable suckers is 4, and when the corresponding relation between the area and the number is met, the number condition is considered to be met.

For the grouping information of open suction cups, it can be determined only which groups have open suction cups. For example, in the above embodiment that the suction cups are divided into 4 groups, it can be determined which groups have suction cups that can be opened from the first group to the fourth group according to the opening condition of the suction cups. The distribution conditions of the suckers to be met can be preset, and for example, the distribution conditions of the suckers can be met when at least three groups (in this case, the suckers can be arranged into a stable triangle or quadrangle) are provided with suckers capable of being opened.

Step S540: and determining whether the sucker array can execute grabbing in the state according to the judgment result.

In particular implementations, the grabbing may be performed only when the number of openable suction cups meets a condition or the distribution of openable suction cups meets a condition. The grabbing may be performed when both satisfy the condition. When the preset condition of the sucker grouping comprises a plurality of conditions, the grabbing can be performed when the plurality of conditions are met.

Step S500 may be performed at any time before S510 as long as it is guaranteed that there is already certain grouping information when steps S510-S540 are performed sequentially. In one embodiment, the suction cup verification may be performed after determining that glass gripping is performed in a certain state, and before performing gripping. If the method is used in combination with the obstacle avoidance capture scheme, steps S510 to S540 may be performed between step S430 and step S440 in the foregoing embodiment, that is, after the optimal search state is selected, the optimal search state is used as the state of the chuck array, and chuck verification is performed to determine whether capture can be performed using the optimal search state. In another embodiment, the method of suction cup verification may also be combined with determining that gripping of the item is performed in a certain state. If the method is used in combination with the obstacle avoidance grabbing scheme, steps S510 to S540 may be performed in step S420 or step S430 of the foregoing embodiment, for example, the alternative search state may be selected by a suction cup verification method, or after the alternative search state is selected, the alternative search state is used as the state of the suction cup array, the search state in which grabbing is not executable is removed according to the suction cup verification method, and then the optimal search state is selected according to the number of suction cups and the distance from the center.

In addition, it should be noted that although the present invention describes a general robot gripping method and a gripping method dedicated to glass gripping in a plurality of embodiments respectively, and technical details of the plurality of embodiments are different, those skilled in the art will understand that technical details not described in the specific method but described in the general method may be actually used in the specific method, and vice versa. In other words, although each embodiment of the invention has a specific combination of features, further combinations and cross-combinations of these features between embodiments are possible.

According to the embodiment, firstly, the possible grabbing modes of the clamp can be obtained, the best grabbing mode which cannot grab the obstacle is selected to grab the object, and the object can be grabbed by avoiding the obstacle accurately even if the object has the obstacle which cannot be grabbed; secondly, aiming at an industrial scene that whether the surface of the object has a non-planar structure or not is possibly required to be judged, the invention provides a scheme capable of identifying the non-planar area of the surface of the object; thirdly, the method can verify that the used grabbing mode can accurately grab the object in advance before the clamp group performs grabbing, improves the grabbing stability, and avoids the problems of unstable gravity center and the like possibly occurring in the grabbing process; fourthly, based on the general obstacle avoidance grabbing method and the clamp calibration method, the obstacle avoidance grabbing method and the sucker calibration method which are specially used for grabbing the industrial scene of the glass by using the sucker array are developed, and the accuracy and the stability of grabbing the glass by using the sucker array can be improved. Therefore, the invention solves the problem of the aspect of article grabbing in the industrial scene by using the clamp.

FIG. 9 shows a clamp control device according to yet another embodiment of the present invention, the device comprising:

a point cloud obtaining module 600, configured to obtain point cloud information of the object group to be grabbed, that is, to implement step S100;

a search state generating module 610, configured to generate a search state based on the point cloud information of the object group to be grabbed and the search boundary factor parameter of the fixture, that is, to implement step S110;

a barrier determination module 620, configured to perform a barrier determination for each search state, and set the search state determined by the barrier determination as an alternative search state, that is, to implement step S120;

an optimal search state determination module 630, configured to select an optimal search state from the candidate search states, that is, to implement step S130;

and a grabbing module 640, configured to grab the object to be grabbed based on the optimal search state, that is, to implement step S140.

Fig. 10 shows a robot-based non-planar structure determination apparatus according to still another embodiment of the present invention, the apparatus including:

a depth map obtaining module 700, configured to obtain a two-dimensional planar depth map of an area to be determined on a surface of an article, that is, to implement step S200;

a depth value obtaining module 710, configured to obtain a depth value of each pixel point in the area to be determined according to the two-dimensional plane depth map, that is, to implement step S210;

a grouping module 720, configured to group the obtained depth values, that is, to implement step S220;

a comparing module 730, configured to calculate a difference value of depth values in each packet, and compare the difference value with a preset difference threshold; and/or, calculating the number of packets, and comparing with a preset threshold value of the number of packets, namely, implementing step S230;

and a determining module 740, configured to determine a non-planar structure condition of the area according to the comparison result, that is, to implement step S240.

Fig. 11 is a schematic structural diagram of a jig set verification apparatus according to still another embodiment of the present invention, including:

a status information acquiring module 800, configured to acquire status information of the fixture array, that is, to perform step S300;

an information determining module 810, configured to determine, based on the state information, information on the number of openable clamps and/or information on the positions of openable clamps, that is, to perform step S310;

a condition determining module 820, configured to determine whether the number of openable fixtures meets a preset condition and/or determine whether the position information of the openable fixtures meets the preset condition, that is, to execute step S320;

and a grasping determining module 830, configured to determine whether the fixture group can perform grasping in the state according to the determination result, that is, to perform step S330.

FIG. 12 is a schematic diagram of the structure of a chuck array control device according to another embodiment of the present invention, the device comprising:

a point cloud obtaining module 900, configured to obtain point cloud information of the glass to be captured and the obstacle, that is, to perform step S400;

a search state generating module 910, configured to generate a search state based on the point cloud information and the search boundary factor parameter of the suction cup array, that is, to execute step S410;

an obstacle determining module 920, configured to determine, for each search state, whether an obstacle exists below each suction cup, and if so, close the suction cup in the search state, that is, to execute step S420;

an alternative searching state selecting module 930, configured to select an alternative searching state according to the suction cup opening condition of each searching state, that is, to execute step S430;

an optimal search state selection module 940, configured to select an optimal search state from the candidate search states according to the number of open suction cups and/or the distance between a suction cup and the center of the glass, that is, to perform step S440;

a grasping module 950 for grasping the glass based on the best search state, i.e. for performing step S450.

FIG. 13 is a schematic diagram of a chuck array calibration apparatus according to another embodiment of the present invention, the apparatus comprising:

a grouping module 1000, configured to group all the suction cups in the array according to the shape of the suction cup array, that is, to perform step S500;

a status information obtaining module 1010, configured to obtain status information of the sucker array, that is, to perform step S510;

an information determining module 1020, configured to determine, based on the status information of the sucker array, the number information of the openable suckers and the grouping information where the openable suckers are located, that is, to execute step S520;

a condition determining module 1030, configured to determine whether the number of openable suction cups meets a preset condition and/or whether the grouping information of the openable suction cups meets a preset condition, that is, to execute step S530;

the grabbing determining module 1040 is configured to determine whether the sucker array can perform grabbing in this state according to the determination result, that is, to perform step S540.

In the apparatus embodiments shown in fig. 9 to fig. 13, only the main functions of the modules are described, all the functions of each module correspond to corresponding steps in the method embodiment, and the working principle of each module may also refer to the description of the corresponding steps in the method embodiment, which is not described herein again. In addition, although the correspondence between the functions of the functional modules and the method is defined in the above embodiments, it can be understood by those skilled in the art that the functions of the functional modules are not limited to the correspondence, that is, a specific functional module can also implement other method steps or a part of the method steps. For example, the above embodiment describes that the grabbing determination module 1040 is used to implement the method of step S540, however, the grabbing determination module 1040 may also be used to implement the method or part of the method of steps S500, S510, S520 or S530 according to the needs of the actual situation.

The present application also provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the method of any of the above embodiments. It should be noted that the computer program stored in the computer-readable storage medium according to the embodiments of the present application may be executed by a processor of an electronic device, and in addition, the computer-readable storage medium may be a storage medium built in the electronic device or a storage medium that can be plugged into the electronic device in an pluggable manner, so that the computer-readable storage medium according to the embodiments of the present application has higher flexibility and reliability.

Fig. 14 is a schematic structural diagram of an electronic device according to an embodiment of the present invention, and the specific embodiment of the present invention does not limit the specific implementation of the electronic device.

As shown in fig. 14, the electronic device may include: a processor (processor) 1102, a communication Interface 1104, a memory 1106, and a communication bus 1108.

Wherein:

the processor 1102, communication interface 1104, and memory 1106 communicate with one another via a communication bus 1108.

A communication interface 1104 for communicating with network elements of other devices, such as clients or other servers.

The processor 1102 is configured to execute the program 1110, and may specifically perform relevant steps in the foregoing method embodiments.

In particular, the program 1110 can include program code that includes computer operating instructions.

The processor 1102 may be a central processing unit CPU, or an Application Specific Integrated Circuit (ASIC), or one or more Integrated circuits configured to implement an embodiment of the present invention. The electronic device comprises one or more processors, which can be the same type of processor, such as one or more CPUs; or may be different types of processors such as one or more CPUs and one or more ASICs.

A memory 1106 for storing a program 1110. Memory 1106 may include high-speed RAM memory, and may also include non-volatile memory (non-volatile memory), such as at least one disk memory.

The program 1110 may be specifically configured to cause the processor 1102 to perform the operations in the method embodiments described above.

Broadly, the inventive content of the invention comprises:

a jig control method comprising:

acquiring point cloud information of an article group to be grabbed;

generating a searching state based on point cloud information of the object group to be grabbed and searching boundary factor parameters of the clamp;

for each search state, executing obstacle judgment, and setting the search state judged by the obstacle as an alternative search state;

selecting an optimal search state from the alternative search states;

and grabbing the object to be grabbed based on the optimal searching state.

Optionally, the group of articles to be grabbed comprises one or more articles to be grabbed and/or obstacles.

Optionally, the configuration parameters of the clip are saved using a json file.

Optionally, the obstacle determination includes at least one of: boundary obstacle judgment, fixed obstacle judgment, protrusion/depression obstacle judgment and user-defined obstacle judgment.

Optionally, the determining of the boundary obstacle includes determining whether the boundary obstacle exists according to a relation between the center of the clamp and the edge contour point of the article.

Optionally, the fixed obstacle determination includes determining whether a fixed obstacle is present according to the obstacle size or shape.

Optionally, the determining of the protrusion/depression obstacle includes determining whether there is a protrusion/depression obstacle according to the two-dimensional plane depth map information.

Optionally, the determining the user-defined obstacle includes generating an edge profile of the user-defined obstacle, and determining whether the user-defined obstacle exists based on the edge profile.

Optionally, the method further includes: a fixture check is performed against the best search condition to determine whether the item can be properly grasped using the search condition.

A clamp control device comprising:

the point cloud acquisition module is used for acquiring point cloud information of the object group to be grabbed;

the searching state generating module is used for generating a searching state based on point cloud information of the object group to be grabbed and searching boundary factor parameters of the clamp;

the obstacle judgment module is used for executing obstacle judgment aiming at each search state and setting the search state which passes the obstacle judgment as an alternative search state;

the optimal search state determining module is used for selecting an optimal search state from the alternative search states;

and the grabbing module is used for grabbing the object to be grabbed based on the optimal searching state.

Optionally, the method further includes: the configuration parameters of the fixture are saved using a json file.

Optionally, the obstacle determination module is configured to perform at least one of the following obstacle determinations: boundary obstacle judgment, fixed obstacle judgment, protrusion/depression obstacle judgment and user-defined obstacle judgment.

Optionally, when the obstacle determination module performs the boundary obstacle determination, it determines whether a boundary obstacle exists according to a relationship between the center of the fixture and the edge contour point of the article.

Optionally, the obstacle determination module determines whether there is a fixed obstacle according to the obstacle size or shape when performing the fixed obstacle determination.

Optionally, the obstacle determination module determines whether a protrusion/depression obstacle exists according to the two-dimensional plane depth map information when performing the protrusion/depression obstacle determination.

Optionally, the obstacle determination module determines whether a user-defined obstacle exists according to an edge contour of the user-defined obstacle when performing user-defined obstacle determination.

A non-planar structure determination method based on a robot includes:

acquiring two-dimensional plane depth map information of an area to be determined on the surface of an article;

acquiring the depth value of each pixel point in the area to be judged according to the two-dimensional plane depth map information;

grouping the obtained depth values;

calculating the difference value of the depth values in each group, and comparing the difference value with a preset difference value threshold value; and/or, calculating the number of the packets, and comparing the number of the packets with a preset packet number threshold value;

and judging the non-planar structure condition of the area according to the comparison result.

Optionally, after the depth value of each pixel point is obtained, only one of the depth values is reserved for a plurality of same depth values.

Optionally, the grouping comprises grouping two different depth values.

Optionally, the grouping includes grouping all the depth values as follows: all the obtained depth values are sorted from high to low, the first depth value and the penultimate depth value are divided into a group, the second depth value and the penultimate depth value are divided into a group of \8230, the Nth depth value and the penultimate depth value are divided into a group, and N is a natural number which is larger than or equal to 1.

Optionally, the preset difference threshold and/or the grouping number threshold include a difference threshold and/or a grouping number threshold preset based on the capability of the fixture.

Optionally, the non-planar structure condition includes the presence or absence of a non-planar structure.

Optionally, the non-planar structure condition includes a degree of undulation of the non-planar structure.

A robot-based non-planar structure determination apparatus comprising:

the depth map acquisition module is used for acquiring a two-dimensional plane depth map of an area to be determined on the surface of the article;

the depth value acquisition module is used for acquiring the depth value of each pixel point in the area to be judged according to the two-dimensional plane depth map;

a grouping module for grouping the obtained depth values;

the comparison module is used for calculating the difference value of the depth values in each group and comparing the difference value with a preset difference value threshold value; and/or, calculating the number of the groups, and comparing the number with a preset group number threshold value;

and the judging module is used for judging the non-planar structure condition of the area according to the comparison result.

Optionally, after the depth value obtaining module obtains the depth value of each pixel, only one of the multiple same depth values is reserved.

Optionally, the grouping module groups two different depth values.

Optionally, the grouping module groups all the depth values as follows: the obtained all depth values are sorted from high to low, the first depth value and the last-to-last depth value are divided into a group, the second depth value and the last-to-last depth value are divided into a group of \8230, the \8230theNth depth value and the Nth depth value are divided into a group, and N is a natural number larger than or equal to 1.

Optionally, the preset difference threshold and/or the group number threshold include a difference threshold and/or a group number threshold preset based on the capability of the clamp.

Optionally, the determining module determines that a non-planar structure exists or that a non-planar structure does not exist.

Optionally, the determining module determines the degree of undulation of the non-planar structure.

A method of verifying a set of grippers, comprising:

acquiring state information of the clamp group;

determining the number information of openable clamps and/or the position information of openable clamps in the state based on the state information;

Optionally, the determining the position information of the openable clamp includes determining the position information of the openable clamp according to the profile information of the object to be grabbed.

Optionally, the conditions that the number of openable clamps needs to meet are preset based on the weight of the article to be gripped.

Optionally, the conditions that the position of the openable clamp needs to meet are preset based on the center of gravity of the object to be grabbed.

Optionally, the conditions that the position of the openable clamp needs to satisfy include: the position connecting lines of the plurality of clamps cannot form a straight line.

Optionally, the determining whether the clamp group can perform the grabbing in the state according to the judgment result includes determining that the grabbing can be performed when the number of the clamps meets a preset condition and the positions of the clamps meet the preset condition.

A clamp group verification apparatus comprising:

Optionally, the information determining module determines position information of the openable clamp according to profile information of the object to be grabbed.

Optionally, the conditions that the position of the openable clamp needs to satisfy include: the position connecting lines of the plurality of clamps cannot form a straight line. .

Optionally, the grabbing determining module determines that grabbing can be performed when the number of the clamps meets a preset condition and the positions of the clamps meet the preset condition.

A method of chuck array control comprising:

acquiring point cloud information of glass to be grabbed and obstacles;

generating a searching state based on the point cloud information and the searching boundary factor parameters of the sucker array;

judging whether an obstacle exists below each sucker or not according to each searching state, and closing the sucker in the searching state if the obstacle exists below each sucker;

selecting alternative searching states according to the opening condition of the sucker in each searching state;

selecting an optimal searching state from the alternative searching states according to the opening number of the suckers and/or the distance between the suckers and the center of the glass;

the glass was grabbed using the best search state.

Optionally, the obstacle determination includes at least one of: judging glass boundary obstacles, judging rubber pad obstacles, judging rubber strip obstacles and judging protrusion obstacles.

Optionally, the glass boundary obstacle determination includes determining whether there is a glass boundary obstacle according to a ratio of point clouds in an area below the suction cup.

Optionally, the rubber pad obstacle determination includes: and judging whether rubber pad obstacles exist or not based on the obstacle point cloud area, wherein the obstacle point cloud area is preset according to the rubber pad area.

Optionally, the convex obstacle determination includes: whether protrusion obstacles exist is judged based on a preset depth difference threshold value and a logarithm threshold value.

Optionally, the selecting the alternative search state according to the suction cup opening condition of each search state includes: if there is at least one suction cup that is open, the search state is selected as the alternative search state.

Optionally, the selecting the optimal search state from the alternative search states according to the number of open suckers and/or the distance between the sucker and the center of the glass includes: and selecting the searching state with the largest number of opened suckers, and if the searching state with the largest number of opened suckers is multiple, further selecting the searching state with the sucker closest to the center of the glass.

Optionally, the method further includes: a fixture check is performed for the best search condition to determine whether the glass can be properly grasped using the search condition.

A suction cup array control device comprising:

the point cloud acquisition module is used for acquiring point cloud information of glass and obstacles to be grabbed;

the searching state generating module is used for generating a searching state based on the point cloud information and the searching boundary factor parameters of the sucker array;

the obstacle judging module is used for judging whether an obstacle exists below each sucker or not according to each searching state, and if the obstacle exists, the sucker is closed in the searching state;

the alternative searching state selecting module is used for selecting alternative searching states according to the opening condition of the sucker of each searching state;

the optimal search state selection module is used for selecting an optimal search state from the alternative search states according to the opening number of the suckers and/or the distance between the suckers and the center of the glass;

and the grabbing module is used for grabbing the glass based on the optimal searching state.

Optionally, the obstacle determination module is configured to perform at least one of the following obstacle determinations: judging glass boundary obstacles, judging rubber pad obstacles, judging rubber strip obstacles and judging protrusion obstacles.

Optionally, when the obstacle determining module executes the glass boundary obstacle, whether the glass boundary obstacle exists is determined according to the ratio of point clouds in the area below the suction cup.

Optionally, when the obstacle determination module performs rubber pad obstacle determination, it determines whether a rubber pad obstacle exists based on an obstacle point cloud area, where the obstacle point cloud area is preset according to the rubber pad area.

Optionally, when the obstacle determining module performs the obstacle raised determination, it determines whether there is an obstacle raised based on a preset depth difference threshold and a preset logarithm threshold.

Optionally, when at least one open suction cup exists in a certain search state, the alternative search state selection module selects the search state as the alternative search state.

Optionally, the optimal search state selection module selects a search state with the largest number of open suction cups as the optimal search state, and if there are a plurality of search states with the largest number of open suction cups, further selects a search state with a suction cup closest to the center of the glass.

A method of chuck array verification, comprising:

grouping all the suckers in the array according to the shape of the sucker array;

acquiring state information of the sucker array;

determining the quantity information of the openable suckers and the grouping information of the openable suckers based on the state information of the sucker array;

judging whether the number of openable suckers meets a preset condition and/or whether the grouping information of the openable suckers meets the preset condition;

and determining whether the sucker array can execute grabbing under the state according to the judgment result.

Optionally, the step of grouping all the suckers in the array according to the form of the sucker array includes dividing the sucker array into four areas, namely, an upper left area, a lower left area, an upper right area and a lower right area, and dividing all the suckers into four groups according to the areas where all the suckers are located.

Optionally, the conditions to be met by the number of openable suction cups are preset based on the weight of the article to be gripped.

Optionally, the conditions that the grouping information of the openable suction cups needs to meet are preset based on the gravity center of the object to be grabbed.

Optionally, the condition that the grouping information of the openable suckers needs to satisfy includes: the openable suction cups are distributed in at least three groups.

Optionally, the determining whether the sucker array can execute grabbing in the state according to the judgment result includes: and when the number of the suckers meets the preset condition and the sucker grouping information meets the preset condition, determining that the grabbing can be performed.

A suction cup array verification device comprising:

the grouping module is used for grouping all suckers in the array according to the shape of the sucker array;

the state information acquisition module is used for acquiring the state information of the sucker array;

the information determining module is used for determining the quantity information of the openable suckers and the grouping information of the openable suckers based on the state information of the sucker array;

the condition judging module is used for judging whether the number of openable suckers meets a preset condition and/or whether the grouping information of the openable suckers meets the preset condition;

and the grabbing determining module is used for determining whether the sucker array can grab in the state according to the judgment result.

Optionally, the grouping module is specifically configured to divide the suction cup array into four areas, namely, an upper left area, a lower left area, an upper right area and a lower right area, and divide all the suction cups into four groups according to the areas where all the suction cups are located.

Optionally, the number of openable suction cups is preset to meet the conditions based on the weight of the object to be gripped.

Optionally, the grouping information of the openable suction cups needs to satisfy the conditions including: the openable suction cups are distributed in at least three groups.

Optionally, the capture determining module is specifically configured to: and when the number of the suckers meets the preset condition and the sucker grouping information meets the preset condition, determining that the grabbing can be performed.

In the description of the present specification, reference to the description of "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples" or "some examples" or the like means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present application. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

Any process or method descriptions in flow charts or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps of the process, and the scope of the preferred embodiments of the present application includes other implementations in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art of the present application.

The logic and/or steps represented in the flowcharts or otherwise described herein, such as an ordered listing of executable instructions that can be considered to implement logical functions, can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processing module-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For the purposes of this description, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: an electrical connection (electronic device) having one or more wires, a portable computer diskette (magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium could even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, via for instance optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner if necessary, and then stored in a computer memory.

The Processor may be a Central Processing Unit (CPU), other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic, discrete hardware components, etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

It should be understood that portions of the embodiments of the present application may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the various steps or methods may be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one or combination of the following techniques, which are known in the art, may be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application specific integrated circuit having an appropriate combinational logic gate circuit, a Programmable Gate Array (PGA), a Field Programmable Gate Array (FPGA), or the like.

It will be understood by those skilled in the art that all or part of the steps carried by the method for implementing the above embodiments may be implemented by hardware related to instructions of a program, which may be stored in a computer readable storage medium, and when the program is executed, the program includes one or a combination of the steps of the method embodiments.

In addition, functional units in the embodiments of the present application may be integrated into one processing module, or each unit may exist alone physically, or two or more units are integrated into one module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode. The integrated module, if implemented in the form of a software functional module and sold or used as a separate product, may also be stored in a computer-readable storage medium.

The storage medium mentioned above may be a read-only memory, a magnetic or optical disk, etc.

Although embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present application, and that variations, modifications, substitutions and alterations of the above embodiments may be made by those of ordinary skill in the art within the scope of the present application.

Claims

1. A method for verifying a jig set, comprising:

acquiring state information of the clamp group;

2. The jig set verification method according to claim 1, wherein: the step of determining the position information of the openable clamp comprises the step of determining the position information of the openable clamp according to the outline information of the object to be grabbed.

3. The jig set verification method according to claim 1, wherein: the conditions that the number of openable clamps needs to meet are preset based on the weight of the object to be gripped.

4. The jig set verification method according to claim 1, wherein: the conditions that the position of the openable clamp needs to meet are preset based on the center of gravity of the object to be grabbed.

5. The method of claim 1, wherein the conditions to be satisfied by the positions of the openable clamps include: the position connecting lines of the plurality of clamps cannot form a straight line.

6. The jig set verification method according to claim 1, wherein: and determining whether the clamp group can execute the grabbing in the state according to the judgment result comprises the step of determining that the grabbing can be executed when the number of the clamps meets the preset condition and the positions of the clamps meet the preset condition.

7. A jig set verification device, comprising:

the condition judging module is used for judging whether the number of the openable clamps meets the preset condition and/or judging whether the position information of the openable clamps meets the preset condition;

8. The clamp group verification device of claim 7, wherein: the information determining module determines the position information of the openable clamp according to the outline information of the object to be grabbed.

9. The clamp group verification device of claim 7, wherein: the number of openable clamps is preset based on the weight of the object to be gripped.

10. The clamp group verification device of claim 7, wherein: the conditions that the position of the openable clamp needs to meet are preset based on the center of gravity of the object to be grabbed.

11. The apparatus of claim 7, wherein the position of the openable fixture is subject to conditions including: the position connecting lines of the plurality of clamps cannot form a straight line.

12. The clamp group verification device of claim 7, wherein: and the grabbing determining module determines that grabbing can be performed when the number of the clamps meets a preset condition and the positions of the clamps meet the preset condition.

13. An electronic device, comprising: a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the clamp group verification method of any one of claims 1 to 6 when executing the computer program.

14. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the gripper group verification method according to any one of claims 1 to 6.