WO2022222515A1

WO2022222515A1 - Article surface gluing method and apparatus based on robot vision, device, and medium

Info

Publication number: WO2022222515A1
Application number: PCT/CN2021/138582
Authority: WO
Inventors: 李辉; 魏海永; 丁有爽; 邵天兰
Original assignee: 梅卡曼德（北京）机器人科技有限公司
Priority date: 2021-04-20
Filing date: 2021-12-15
Publication date: 2022-10-27
Also published as: CN112967368A

Abstract

The present application discloses an article surface gluing method based on robot vision, an article surface gluing apparatus based on robot vision, an electronic device, and a storage medium. The article surface gluing method based on robot vision of the present application comprises: acquiring 3D point cloud information of an article; determining 3D image information of the article on the basis of the 3D point cloud information and preset 3D image template information; mapping the 3D image information into 2D image information; generating 2D gluing trajectory points on the basis of the 2D image information; mapping the 2D gluing trajectory points into 3D gluing trajectory points; and performing gluing on the basis of the mapped 3D gluing trajectory points. According to the present invention, complete 3D point cloud information of the article is acquired by a matching method, incomplete 3D point cloud information is replaced with the complete 3D point cloud information, an image processing operation of a 2D image level is performed to obtain the 2D gluing trajectory points, and then the 2D gluing trajectory points are converted into the 3D gluing trajectory points, such that a camera still can obtain the gluing trajectory points on the basis of a contour of the article even acquiring the incomplete point cloud information. The present invention further provides an article 3D image information acquisition method based on robot vision, and in the method, an article matching method is performed on the basis of 3D contour information. Compared with a conventional method, the method can greatly improve the matching efficiency on the premise that the matching accuracy is not lost.

Description

Glue coating method, device, equipment and medium for object surface based on robot vision

claim of priority

This application claims the priority of the Chinese invention patent filed on April 20, 2021, with the application number CN202110426175.9 and the title of "Robot Vision-Based Gluing Method, Device, Electronic Equipment and Storage Media on the Surface of Objects". The above patent All content of .

technical field

The present application relates to the field of B25 intelligent robots, and more particularly, to a method for gluing an object surface based on robot vision, a device for gluing an object surface based on robot vision, an electronic device and a storage medium.

Background technique

At present, with the widespread popularity of intelligent program-controlled robots, it has been possible to implement the operation of applying glue on the surface of objects with the help of intelligent program-controlled robots. In the prior art, after obtaining the complete point cloud information of the item, the outline of the object is extracted, and the trajectory points of the object are obtained according to the outline, or the trajectory points of different types of items are taught in advance, and then after the outline of the item is obtained, according to the item. Place the shape and the incoming material position, and call the track point taught in advance. The above two methods need to obtain the complete point cloud information of the item. However, in industrial scenes, the lighting conditions are complex and changeable, and the glass materials are also different. As a result, when the depth camera is used to collect point clouds, there are often serious point clouds. In the case of missing, especially when the point cloud is incomplete so that the outline point cloud of the item is missing, the correct trajectory point cannot be obtained from the outline point cloud, and the problem of gluing failure occurs.

SUMMARY OF THE INVENTION

In view of the above-mentioned problems, the present invention has been proposed in order to overcome the above-mentioned problems or at least partially solve the above-mentioned problems. Specifically, one of the innovations of the present invention is that, in order to overcome the problem that the obtained point cloud information of the item is incomplete, which leads to the inability to correctly obtain the robot gluing trajectory point, the applicant proposes a method to obtain the complete 3D point of the item through a matching method. Cloud information, replace incomplete 3D point cloud information with complete 3D point cloud information, perform image processing operations at the 2D image level to obtain 2D gluing track points, and then convert them into 3D gluing track points. After the 3D point cloud of the item, the point cloud information is not directly used to obtain the gluing trajectory point, which solves the above technical problems, so that no matter what kind of point cloud information is obtained, the gluing trajectory point can be obtained based on the outline of the item. .

The second innovation of the present invention is that the applicant found that the existing 3D image matching algorithm needs to calculate too many pixel points, so the matching efficiency is not high enough, and the 2D image matching algorithm is difficult to obtain the accurate pose of the item information. Therefore, based on the characteristics of the application scenario of robot gluing, the applicant has developed a method for performing item matching based on 3D contour information. Compared with the traditional method, this method can greatly improve the matching accuracy without losing the matching accuracy. matching efficiency.

All solutions disclosed in the claims and descriptions of the present application have one or more of the above-mentioned innovations, and correspondingly, can solve one or more of the above-mentioned technical problems.

Specifically, the present application provides a method for gluing the surface of an article based on robot vision, a device for gluing the surface of an article based on robot vision, an electronic device and a storage medium.

The robotic vision-based object surface gluing method of the embodiment of the present application includes:

Get the 3D point cloud information of the item;

Determine the 3D image information of the item based on the 3D point cloud information and the preset 3D image template information;

mapping the 3D image information to 2D image information;

generating 2D gluing track points based on the 2D image information;

mapping the 2D gluing track points to 3D gluing track points;

The glue is applied based on the mapped 3D glue track points.

In some embodiments, the 3D point cloud information includes a 3D contour point cloud, and the acquiring 3D point cloud information of the item includes:

Map the 3D point cloud of the item into a 2D image;

obtaining a 2D outline of the item based on the 2D image;

Map 2D contours to 3D contour point clouds.

In some embodiments, the mapping of the 3D point cloud of the item into the 2D image includes: after acquiring the 3D point cloud of the item, first performing point cloud clustering and/or removing outliers, and then processing the processed 3D point cloud. Mapped to a 2D image.

In some embodiments, the mapping of the 3D image information to the 2D image information includes: using an orthogonal projection method to map the matched 3D image information to the 2D image information.

In some embodiments, the generating 2D gluing track points based on the 2D image information includes:

Generate 2D contours based on 2D image information;

Traverse the entire contour at predetermined intervals and generate 2D gluing trajectory points.

In some embodiments, after the contour is indented, the entire contour is traversed at predetermined intervals to generate 2D gluing track points.

In some embodiments, the start and end points of the 2D trajectory points coincide.

In some embodiments, the value range of the predetermined interval includes 50mm-100mm.

In some embodiments, the 3D image information of the item includes 3D image template information matched with the item and/or pose information of the item.

The robot vision-based object surface gluing device according to the embodiment of the present application includes:

3D point cloud acquisition module, used to acquire 3D point cloud information of items;

a 3D image determination module, configured to determine the 3D image information of the item based on the 3D point cloud information and the preset 3D image template information;

a 2D image mapping module for mapping the 3D image information to 2D image information;

a trajectory point generation module for generating 2D gluing trajectory points based on the 2D image information;

a 3D track point mapping module for mapping the 2D gluing track points to 3D gluing track points;

Glue module for gluing based on mapped 3D glue track points.

In some embodiments, the 3D point cloud information includes a 3D contour point cloud, and the 3D point cloud acquisition module is further configured to:

Map the 3D point cloud of the item into a 2D image;

obtaining a 2D outline of the item based on the 2D image;

Map 2D contours to 3D contour point clouds.

In some embodiments, the 2D image mapping module is specifically configured to: map the matched 3D image information into 2D image information by using an orthogonal projection method.

In some embodiments, the trajectory point generation module is specifically used for:

Generate 2D contours based on 2D image information;

In some embodiments, the trajectory point generation module is further configured to: shrink the contour, traverse the entire contour at predetermined intervals, and generate 2D trajectory points.

An electronic device according to an embodiment of the present application includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor implements any of the above-mentioned embodiments when the processor executes the computer program. Glue coating method for object surface based on robot vision.

The computer-readable storage medium of the embodiment of the present application has a computer program stored thereon, and when the computer program is executed by the processor, implements the robot vision-based method for gluing the surface of an article in any of the foregoing embodiments.

Additional aspects and advantages of the present application will be set forth, in part, from the following description, and in part will become apparent from the following description, or may be learned by practice of the present application.

Description of drawings

The above and/or additional aspects and advantages of the present application will become apparent and readily understood from the following description of embodiments taken in conjunction with the accompanying drawings, wherein:

1 is a schematic flowchart of a method for gluing an object surface based on robot vision according to some embodiments of the present application;

2 is a schematic flowchart of a method for obtaining 3D image information of an item based on robot vision according to some embodiments of the present application;

3 is a schematic structural diagram of a robot vision-based object surface gluing device according to some embodiments of the present application;

4 is a schematic structural diagram of a device for obtaining 3D image information of an item based on robot vision according to some embodiments of the present application;

5 is a schematic diagram of the missing point cloud and the complete point cloud of the item according to some embodiments of the present application;

6 is a schematic diagram of the gluing process of certain embodiments of the present application;

7 is a schematic diagram of the outline shape of an article to be glued according to some embodiments of the present application;

FIG. 8 is a schematic structural diagram of an electronic device according to some embodiments of the present application.

Detailed ways

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be more thoroughly understood, and will fully convey the scope of the present disclosure to those skilled in the art.

FIG. 1 shows a method for gluing the surface of an article according to an embodiment of the present invention, including:

Step S100, acquiring 3D point cloud information of the item;

Step S110, based on the 3D point cloud information and the preset 3D image template information, determine the 3D image information of the item;

Step S120, mapping the 3D image information to 2D image information;

Step S130, generating 2D gluing track points based on the 2D image information;

Step S140, mapping the 2D gluing track points to 3D gluing track points;

Step S150, gluing is performed based on the mapped 3D gluing track points.

In step S100, point cloud information can be obtained through a 3D industrial camera. The 3D industrial camera is generally equipped with two lenses, which capture the group of objects to be grasped from different angles, and can display a three-dimensional image of the object after processing. Place the group of objects to be grasped under the vision sensor, and shoot with two lenses at the same time. According to the obtained relative pose parameters of the two images, use a general binocular stereo vision algorithm to calculate the X, X, and X of each point of the object to be filled. The Y, Z coordinate values and the coordinate orientation of each point are then converted into the point cloud data of the item group to be grasped. During specific implementation, components such as laser detectors, visible light detectors such as LEDs, infrared detectors, and radar detectors can also be used to generate point clouds, and the present invention does not limit the specific implementation.

After obtaining the 3D point cloud of the object, the point cloud can be further processed such as point cloud clustering and outlier removal.

The point cloud data acquired in the above manner is three-dimensional data. In order to facilitate data processing and improve efficiency, the acquired three-dimensional point cloud data can be orthographically mapped onto a two-dimensional plane.

As an example, a depth map corresponding to the orthographic projection can also be generated. A two-dimensional color map corresponding to a three-dimensional object region and a depth map corresponding to the two-dimensional color map can be acquired in a depth direction perpendicular to the object. Wherein, the two-dimensional color map corresponds to an image of a plane area perpendicular to the preset depth direction; each pixel in the depth map corresponding to the two-dimensional color map is in one-to-one correspondence with each pixel in the two-dimensional color map, and each pixel The value of the pixel is the depth value of the pixel.

The conversion between 3D pixels and 2D pixels can be performed based on the camera's intrinsic parameters. The camera internal parameters are the internal parameters of the camera, which are only related to the internal properties of the camera (such as focal length, resolution, pixel size, lens distortion, etc.). Using the camera internal parameters, the three-dimensional space points in the camera coordinate system can be transformed into the imaging plane coordinate system, and after correction processes such as lens distortion, they can be further transformed into two-dimensional pixel points in the image pixel coordinate system. Therefore, there is a mapping relationship between the projection point in the image pixel coordinate system and the three-dimensional space point in the camera coordinate system. The process of obtaining this relationship is called camera internal parameter calibration, and based on this mapping relationship, the 3D image can be completed. Convert to and from 2D images.

In step S110, when the point cloud information of the item is acquired, the acquired point cloud may be missing due to the reflection of visible light on the surface of the item or because of the material problem of the item, and the missing point cloud data loses a lot of information. Based on the missing point cloud The information cannot correctly plan the glue path of the robot. Figure 5 shows the situation where the point cloud is missing, the solid line part is the outline formed according to the collected point cloud, and the dashed line part is the actual outline of the item. It can be clearly seen from Figure 5 that the outline of the solid line formed by the point cloud collected by the camera is broken at the left edge of the actual outline of the object and forms a non-existent outline inside the object, which is not the actual outline of the object. contour. If the pre-set gluing process of the robot is to shrink a certain distance along the outline for gluing, planning the gluing route along the solid line in Figure 5 will obviously cause the robot to apply the glue to the inside of the item, instead of near the edge. The contour depicted by the dashed line part opposite to the solid line in Figure 5 is the actual contour of the item. The pre-stored actual contour only has the contour information of the item and no position information. The actual contour corresponding to the item can discard the acquired incomplete contour, and use the pre-stored actual contour at the position of the item according to the position of the item, so as to plan the correct gluing route based on the complete item contour. . Because the existing 2D image matching algorithm is difficult to obtain the accurate pose of the item, and the 3D image matching algorithm has many pixels and complex point attributes, the matching efficiency is low, in order to obtain the item faster in the case of missing point cloud. Complete and correct point cloud information, the applicant developed a method for matching based on the 3D contour point cloud of the item, and obtaining the complete 3D image information of the item including the matched 3D image and the pose information of the item, which is also One of the key points of the present invention

.

FIG. 2 shows a method for acquiring 3D image information according to an embodiment of the present invention, including:

Step S200, obtaining the 3D point cloud of the item;

Step S210, mapping the 3D point cloud of the item into a 2D image, and obtaining a 2D outline of the item based on the 2D image;

Step S220, mapping the 2D contour to a 3D contour point cloud;

Step S230, based on the 3D contour point cloud and the preset 3D image template information, determine the 3D image template information matching the item and the pose information of the item.

For step S200, a method similar to step S100 may be used to obtain the 3D point cloud of the item, which will not be repeated here.

For steps S210 and S220, the 2D image includes a color map and a depth map, each pixel of the 2D depth map is in one-to-one correspondence with each pixel of the color map, and the depth map also includes each pixel. Depth information, the depth information may be the distance between the pixels of the captured picture and the camera used for capturing. The conversion of the 2D image and the 3D image may be performed based on the camera internal parameters. For details, reference may be made to the relevant content in step S100, which will not be repeated here.

For step S230, the image template information corresponding to the item of this model can be input to the robot in advance according to the model of the item to be glued, and the model of the item and the corresponding template can be arbitrarily set according to the needs of the actual situation. Circles, arcs or names with words, such as A, B, C; templates can also use corresponding numbers, such as A template, B template, C template, all types of items should have corresponding templates. Templates can be off-the-shelf, or a 3D image template of the item to be glued can be generated based on the item. In order to generate a 3D image template for a certain type of item, select complete and standard items (ie, standard parts) from multiple items of this type, take pictures of the item at different angles and lighting environments, and obtain multiple sets of point cloud information , and then combine and deduplicate multiple sets of point cloud information to obtain the image template of this type of item.

After the (possibly missing) point cloud of the item is acquired, the image template matching the point cloud of the item and the pose information of the item can be determined based on the feature point matching algorithm and the point set registration method of the iterative closest point algorithm. The feature point matching algorithm extracts the key points in the two images, that is, finds the pixel points with certain features in the image, and then calculates the feature factors of these feature points according to the obtained key point positions. The feature factor is usually a vector, and the distance between two feature factors can reflect the degree of similarity, that is, whether the two feature points are the same. Depending on the eigenfactors, different distance metrics can be chosen. If it is a floating-point descriptor, its Euclidean distance can be used; for binary eigenfactors, its Hamming distance can be used (the Hamming distance between two different binaries refers to the number of different bits in two binary strings) , based on the method of calculating the similarity of feature factors, the feature points can be matched by finding the most similar feature points in the set of feature points. The feature point matching algorithm can efficiently and accurately match the same object in two images from different perspectives. The iterative closest point algorithm enables point cloud data at different coordinates (for example, different point cloud images) to be merged into the same coordinate system, in fact, to find the coordinate system from coordinate system 1 (point cloud image 1) to coordinate system 2 A rigid transformation of (point cloud image 2) that reflects how one point cloud image is rotated and translated to obtain another point cloud image. The algorithm is essentially an optimal registration method based on the least squares method. The corresponding point pairs are selected repeatedly, and the optimal rigid body transformation is calculated until the convergence accuracy requirements of the correct registration are met. In other words, the method minimizes by continuous iteration. The source data and target data correspond to points to achieve precise stitching.

After obtaining the complete point cloud outline information of the item and the pose information of the item, these image information can be used to replace the original incomplete point cloud information, and the subsequent path planning steps can be performed.

In step S120, in order to facilitate performing image morphological operations such as indentation or contouring, the registered complete 3D image information is mapped to 2D image information. The mapping can be in the form of perspective projection or orthographic projection. Although perspective projection is a more commonly used mapping method, when the object is placed obliquely or the viewing angle captured by the camera is inclined, the use of perspective mapping may lead to distortion errors in the mapped image, so the present invention preferably uses the orthogonal projection method for 3D images. Mapping of information to 2D image information. The conversion of the 2D image and the 3D image may be performed based on the camera internal parameters. For details, reference may be made to the relevant content in step S100, which will not be repeated here.

For step S130, in order to obtain the 2D trajectory point, the contour of the 2D image must be obtained first, and after the image contour is obtained, the contour is indented by a certain distance according to the requirements of the process and the model of the item. Figure 6 shows examples of different gluing processes, in which: in process 1, glue is applied evenly on all four edges, so it is necessary to obtain the contour of all 4 edge positions; in process 2, two layers of glue are applied on a specific edge, so On a specific edge, two contours with different indentation distances need to be given to the edge; in process 3, the specific edge is not glued, so it is not necessary to obtain the contour of the specific edge; the glue path of the specific edge in process 4 is compared with other edges For example, there is a certain indentation, so the indentation distance of the outline of that particular edge should be different from other edges. Figure 7 shows examples of different item types. For the first item with a regular rectangle shape, the contours of the four sides can be indented by the same distance, and the preferred indentation distance is 8mm or 10mm. For the second shape, the whole is a rectangle, but one corner is an arc, you can use a different indentation distance from the rectangular section at the position of the arc section. If the rectangular section is indented by 8mm or 10mm, Then the arc segment can be retracted by 15mm or 20mm.

After the indentation is completed, the indented contour points should be sampled at a certain interval, and the entire contour should be traversed in this way to obtain a complete 2D trajectory point. The sampling interval (also called the traversal interval) can be set according to the actual situation. The smaller the sampling interval, the denser the glue is applied, and vice versa. The sampling interval can be a distance, such as 50mm to 100mm, or the number of track points. For example, it can be set to extract 100 points from the outline of the entire item to form gluing track points, or 150 points to form gluing track point. The complete gluing trajectory points should be connected end to end, that is, the robot should start from the starting point and stop at a position close to but not exceeding the starting point when it reaches the end point. However, in industrial scenarios, such trajectory points may cause the robot to move at the end point. The glue density is not enough. In order to solve this problem, the position of the end point can be set to coincide with the starting point or exceed the starting point. In this way, the robot can completely walk through all the positions that need to be glued without any omission.

For step S140, the points of the 2D image are not the points of the real world, only the points of the 3D image can express the real world. For example, when the item has some small protrusions, the 2D points cannot prompt the robot for this information, so that the robot is in the When moving to these positions with small protrusions, the height of the nozzle will not be increased, resulting in an inappropriate distance between the nozzle and the item, and the spraying effect is not good. Therefore, the robot is best to apply glue based on the 3D moving trajectory points. Therefore, after obtaining the 2D gluing track points, it needs to be mapped to the 3D gluing track points. The conversion of the 2D image and the 3D image may be performed based on the camera internal parameters. For details, reference may be made to the relevant content in step S100, which will not be repeated here.

For step S150, in the present invention, a robot can be used for gluing. For this purpose, it is necessary to first plan the movement trajectory of the robot (that is, the gluing trajectory points obtained in the above steps), the moving speed and the gluing rate. Path and moving speed, move on the surface of the object, and execute glue on the surface of the object according to the planned glue discharge rate.

According to the above embodiment, firstly, the present invention obtains the complete point cloud information of the item through template matching, and uses the complete point cloud information to replace the acquired point cloud information to perform contour acquisition, trajectory point calculation, 2D image and 3D image Therefore, even when the obtained point cloud information of the object to be glued is incomplete, the correct glue application trajectory point of the robot can still be calculated from the incomplete point cloud information; secondly, the present invention proposes A method for obtaining 3D image information of an item based on image matching. This method does not use the traditional 2D image and 3D image matching method, but uses the contour of the 3D image for matching. The matching is accurate and fast, and it is especially suitable for gluing objects. industrial scene. It can be seen that the present invention solves the problem that glue cannot be applied correctly due to incomplete point cloud information obtained when using robot gluing in the prior art, as well as the inaccurate and inefficient problems of the existing item matching method .

In addition, those skilled in the art can also make various changes and modifications to the above-mentioned embodiments:

In the process of collecting the point cloud of the glass to be glued, for the glass to be glued on a non-standard plane, the part of the glass to be glued on the conveyor belt is easily disturbed by the conveyor belt, and then interference points appear. The edge of the lifted part will not be disturbed. Therefore, in the process of straight line fitting based on the two-dimensional contour points, for the edge that is in contact with the conveyor belt, the corresponding edge of the edge can be determined according to the point of the edge that is not in contact with the conveyor belt. the straight line. The Z coordinate of the contour point cloud corresponding to the contour point on this edge can be used to determine which contour points on the edge are in contact with the conveyor belt and which contour points are not in contact with the conveyor belt, that is, the contour points corresponding to the contour points. If the Z coordinate of the cloud is zero, it means that the contour point is in contact with the conveyor belt. If the Z coordinate of the contour point cloud corresponding to the contour point is not zero, it means that the contour point is not in contact with the conveyor belt. Line fitting is performed on the contour points in contact with the conveyor belt to obtain a more accurate contour edge of the glass to be glued. It should be noted that, in this embodiment, the coordinate system corresponding to the glass to be glued is established by adhering to the conveyor belt, that is to say, the origin of the above coordinate system is located on the plane where the conveyor belt is located.

It is worth mentioning that the above coordinate system can also be established in other forms. At this time, a certain point cloud screening rule can be set based on the shape of the non-standard flat glass to be glued in advance, and the corresponding point of the above-mentioned better part can be selected, that is, Select the point corresponding to the part of the non-standard flat glass to be glued that is not in contact with the conveyor belt. In this embodiment, the object of the straight line fitting operation is preferably the result after noise removal and smoothing. Of course, it can also be contour points without noise removal and smoothing, which is not limited here.

The robots in various embodiments of the present invention may be industrial robot arms, and these robot arms may be general-purpose or specialized for applying glue to objects. In order to make the robot take fewer redundant trajectories, the initial point of the gluing trajectory point can be set at the position on the gluing path that is closest to the initial pose of the robot, for example, the initial point is set in the middle of the edge close to the robot. That is to say, after the initial pose of the robot is determined, the middle point on the gluing path of the edge closest to the robot's initial pose can be used as the initial point of the gluing trajectory point, and then the robot can be used according to the robot's initial pose. The inherent attribute sets other gluing track points on the gluing path, and then the gluing track point information of the gluing glass can be obtained. It is worth mentioning that the gluing track point information may include, but is not limited to, the coordinates of the gluing track point, the initial track point of the gluing track point, and the direction of the gluing track point (ie, the order of the gluing track points), etc. . After obtaining the gluing track point information of the glued glass, the gluing track point information can be sent to the robot by means of communication. When the robot receives the glue application trajectory point information, it can control its own glue spray nozzle to apply glue to the glass to be glued based on the glue application trajectory point information.

In some embodiments, according to the inherent properties of the robot and the initial pose of the robot, the glue application trajectory point information is generated on the glue application path, including:

Determine the corners and straight lines in the glue path;

According to the glue output rate and movement speed of the robot, set the glue application trajectory points at the corresponding density at the corners and straight lines;

According to the initial pose of the robot, the walking sequence of the gluing track points is determined to obtain the gluing track point information.

Specifically, determining the corners and straight lines in the gluing path can be determined based on the relationship between the coordinate values of each point on the gluing path. The X and Y coordinates of the adjacent points at the corner will be different, while the adjacent points at the straight line may have the same X coordinate or the same Y coordinate. For example: Assuming that the shape of the glass to be glued is a rectangle, in the glued path of the glass to be glued, the X and Y coordinates of the adjacent points at the corners of the four corners will be different, and the adjacent points at the upper line will be different. The Y coordinate of the point will be the same but the X coordinate will be different, the Y coordinate of the adjacent point on the lower line will be the same but the X coordinate will be different and the Y coordinate is smaller than the value of the upper line, and the X coordinate of the adjacent point on the left line It will be the same but the Y coordinate will be different. The X coordinate of the adjacent point on the right line will be the same but the Y coordinate will be different and the X coordinate will be smaller than the value of the left line.

When the robot is applying glue to the glass, it will control the glue head to apply glue based on a certain glue discharge rate. The glue dispensing rate, as an inherent property of the robot, affects the gluing effect in this embodiment. In order to conveniently refer to the glue dispensing rate of the robot and set the gluing trajectory points on the gluing path to avoid the glue stacking situation, the glue dispensing rate of the robot can be determined.

The inherent property of robot motion is also reflected in that if the robot sets the same motion speed parameters at the corner and the straight line, its motion speed at the corner and the straight line will be different, and the specific motion speed at the corner is slower than that at the straight line. In practice, another inherent property of the robot, the glue dispensing rate is unchanged, so for the proper straight line glue dispensing rate and motion speed parameters, the glue stacking situation will occur at the corners. In some embodiments, on the premise of ensuring that the robot moves along the determined gluing path, the spacing between the gluing track points set at the corners of the gluing path may be greater than the spacing between the gluing track points set at the straight line Larger, in order to achieve the balance between the movement speed at the straight line and the movement speed at the corner, and then solve the glue stacking phenomenon that may be caused by the corner. A minimum distance can be set at the straight line to limit the distance between the glue application track points at the straight line, so as to prevent the robot from jamming and stacking glue due to the excessive number of track points at the straight line. You can also set motion speed parameters with different values at the straight line and the corner to achieve a balance between the motion speed at the straight line and the motion speed at the corner, and solve the problem of glue stacking caused by inherent properties.

According to the initial pose of the robot, the walking sequence of the gluing track points is determined, so as to obtain the gluing track point information. It can be understood that, in order to make the robot take fewer redundant trajectories, the initial point of the trajectory point is set as a point close to the initial pose of the robot, for example, it can be the trajectory point corresponding to the middle part of the glass to be glued close to the side of the robot. That is to say, after the initial pose of the robot is determined, the trajectory point corresponding to the middle point on the gluing path of the edge closest to the initial pose of the robot (or the trajectory point closest to the point) As the initial trajectory point of the glue-applied trajectory point, after that, other trajectory points can be moved clockwise or counterclockwise.

In some embodiments, the glue application track point information may specifically include the glue application track point coordinates, the initial track point coordinates, the position sequence of the glue application track points, the motion speed parameters of the glue application track points, and the like.

In some embodiments, the glue application track point information further includes: normal direction information corresponding to the contour points.

Specifically, the normal information can be the angle value of the normal vector corresponding to each contour point cloud relative to a fixed amount, and can also be the deviation of the point cloud in the corresponding position sequence behind the contour point cloud relative to the previous point cloud. angle value.

3 shows a schematic structural diagram of a device for gluing the surface of an article based on robot vision according to another embodiment of the present invention, the device includes:

The 3D point cloud acquisition module 300 is used to acquire the 3D point cloud information of the item, that is, to implement step S100;

A 3D image determination module 310, configured to determine the 3D image information of the item based on the 3D point cloud information and the preset 3D image template information, that is, to implement step S110;

A 2D image mapping module 320, configured to map the 3D image information to 2D image information, that is, to implement step S120;

A trajectory point generation module 330, configured to generate 2D glue application trajectory points based on the 2D image information, that is, to implement step S130;

The 3D track point mapping module 340 is used to map the 2D gluing track points to the 3D gluing track points, that is, for implementing step S140;

The gluing module 350 is used for gluing based on the mapped 3D gluing track points, that is, for implementing step S150.

FIG. 4 shows a schematic structural diagram of a device for acquiring 3D image information of an item based on robot vision according to another embodiment of the present invention, and the device includes:

The 3D point cloud acquisition module 400 is used to acquire the 3D point cloud of the item, that is, to implement step S200;

The 2D contour obtaining module 410 is used to map the 3D point cloud of the item into a 2D image, and obtain the 2D contour of the item based on the 2D image, that is, to implement step S210;

The 3D contour obtaining module 420 is used for mapping the 2D contour into a 3D contour point cloud, that is, for implementing step S220;

The 3D image determination module 430 is configured to determine, based on the 3D contour point cloud and preset 3D image template information, the 3D image template information matching the item and the pose information of the item, that is, for implementing step S230.

In the apparatus embodiments shown in Figures 3 to 4 above, only the main functions of the modules are described, and all the functions of each module correspond to the corresponding steps in the method embodiments, and the working principles of each module can also refer to the corresponding steps in the method embodiments. The description of the steps will not be repeated here. In addition, although the above-mentioned embodiment defines the corresponding relationship between the functions of the functional modules and the methods, those skilled in the art can understand that the functions of the functional modules are not limited to the above-mentioned corresponding relationships, that is, a specific functional module can also implement other method steps. or part of a method step. For example, the above embodiments describe that the 3D point cloud acquisition module 300 is used to implement the method of step S100, however, according to the needs of the actual situation, the 3D point cloud acquisition module 300 can also be used to implement the method of steps S200, S300 or S400 or the method of the method. part.

The present application also provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, implements the method of any one of the foregoing embodiments. It should be noted that the computer program stored in the computer-readable storage medium of the embodiments of the present application may be executed by the processor of the electronic device. In addition, the computer-readable storage medium may be a storage medium built in the electronic device, or a storage medium capable of The storage medium of the electronic device is pluggable and pluggable. Therefore, the computer-readable storage medium of the embodiments of the present application has high flexibility and reliability.

FIG. 8 shows a schematic structural diagram of an electronic device according to an embodiment of the present invention. The specific embodiment of the present invention does not limit the specific implementation of the electronic device.

As shown in FIG. 8 , the electronic device may include: a processor (processor) 902 , a communication interface (Communications Interface) 904 , a memory (memory) 906 , and a communication bus 908 .

in:

The processor 902 , the communication interface 904 , and the memory 906 communicate with each other through the communication bus 908 .

The communication interface 904 is used to communicate with network elements of other devices such as clients or other servers.

The processor 902 is configured to execute the program 910, and specifically may execute the relevant steps in the foregoing method embodiments.

Specifically, the program 910 may include program code including computer operation instructions.

The processor 902 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The one or more processors included in the electronic device may be the same type of processors, such as one or more CPUs; or may be different types of processors, such as one or more CPUs and one or more ASICs.

The memory 906 is used to store the program 910 . Memory 906 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk memory.

The program 910 may specifically be used to cause the processor 902 to perform various operations in the foregoing method embodiments.

In general, the content of the present invention includes:

A method for gluing the surface of objects based on robot vision, comprising:

Get the 3D point cloud information of the item;

mapping the 3D image information to 2D image information;

generating 2D gluing track points based on the 2D image information;

mapping the 2D gluing track points to 3D gluing track points;

The glue is applied based on the mapped 3D glue track points.

Optionally, the 3D point cloud information includes a 3D contour point cloud, and the acquiring the 3D point cloud information of the item includes: mapping the 3D point cloud of the item into a 2D image; acquiring the 2D contour of the item based on the 2D image; 2D contours are mapped to 3D contour point clouds.

Optionally, the mapping of the 3D point cloud of the item into a 2D image includes: after acquiring the 3D point cloud of the item, first performing point cloud clustering and/or processing of removing outliers, and then mapping the processed 3D point cloud into a 2D point cloud. image.

Optionally, the mapping of the 3D image information to the 2D image information includes: using an orthogonal projection method to map the matched 3D image information to the 2D image information.

Optionally, the generating 2D gluing track points based on the 2D image information includes: generating a 2D contour based on the 2D image information; traversing the entire contour at predetermined intervals and generating the 2D gluing track points.

Optionally, after the contour is indented, the entire contour is traversed at predetermined intervals to generate 2D gluing track points.

Optionally, the start point and the end point of the 2D trajectory point coincide.

Optionally, the value range of the predetermined interval includes 50mm-100mm.

Optionally, the 3D image information of the item includes 3D image template information matched with the item and/or pose information of the item.

A device for gluing the surface of objects based on robot vision, comprising:

Glue module for gluing based on mapped 3D glue track points.

Optionally, the 3D point cloud information includes a 3D contour point cloud, and the 3D point cloud acquisition module is further used for:

Map the 3D point cloud of the item into a 2D image;

obtaining a 2D outline of the item based on the 2D image;

Map 2D contours to 3D contour point clouds.

Optionally, the 2D image mapping module is specifically configured to: map the matched 3D image information into 2D image information by using an orthogonal projection method.

Optionally, the trajectory point generating module is specifically configured to: generate a 2D contour based on 2D image information; traverse the entire contour according to a predetermined interval and generate 2D gluing trajectory points.

Optionally, the trajectory point generating module is further configured to: shrink the contour, traverse the entire contour at predetermined intervals, and generate 2D trajectory points.

Optionally, the value range of the predetermined interval includes 50mm-100mm.

A method for acquiring 3D image information of objects based on robot vision, comprising:

Get the 3D point cloud of the item;

mapping the 3D point cloud of the item into a 2D image, and obtaining a 2D outline of the item based on the 2D image;

Map 2D contours to 3D contour point clouds;

Based on the 3D contour point cloud and the preset 3D image template information, the 3D image template information matching the item and the pose information of the item are determined.

Optionally, a feature point-based matching algorithm and/or an iterative closest point algorithm to determine the 3D image template information that matches the item and the pose information of the item

Optionally, obtain the complete point cloud outline of the item in advance as the 3D image template information of the item.

Optionally, the pre-acquiring the complete point cloud profile of the item includes: selecting a standard part from similar items, acquiring the point cloud of the standard part, if the point cloud is incomplete, acquiring the point cloud of the standard part again, and combining with the standard part. If the point cloud combination obtained last time is still incomplete, repeat the steps of point cloud acquisition and combination until a complete point cloud is acquired, and further acquire the contour of the complete point cloud as the complete point cloud contour of the item.

Optionally, use the camera to obtain the 3D point cloud of the item, and map the 3D point cloud of the item into a 2D image based on the camera's internal parameters.

A device for acquiring 3D image information of objects based on robot vision, comprising:

3D point cloud acquisition module, used to acquire 3D point cloud of items;

a 2D contour acquisition module, used to map the 3D point cloud of the item into a 2D image, and obtain the 2D contour of the item based on the 2D image;

3D contour acquisition module for mapping 2D contours to 3D contour point clouds;

The 3D image determination module is configured to determine the 3D image template information matched with the item and the pose information of the item based on the 3D contour point cloud and the preset 3D image template information.

Optionally, the 3D image determination module is also used to: determine the 3D image template information matched with the item and the pose information of the item based on the feature point matching algorithm and/or the iterative closest point algorithm.

Optionally, the 2D contour acquisition module is further configured to: after acquiring the 3D point cloud of the item, first perform point cloud clustering and/or outlier removal processing, and then map the processed 3D point cloud into a 2D image.

In the description of this specification, reference to the terms "one embodiment," "some embodiments," "exemplary embodiment," "example," "specific example," or "some examples" or the like is meant to be used in conjunction with the described embodiments. A particular feature, structure, material, or characteristic described in a manner or example is included in at least one embodiment or example of the present application. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

Any description of a process or method in the flowcharts or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing a specified logical function or step of the process , and the scope of the preferred embodiments of the present application includes alternative implementations in which the functions may be performed out of the order shown or discussed, including performing the functions substantially concurrently or in the reverse order depending upon the functions involved, which should It is understood by those skilled in the art to which the embodiments of the present application belong.

The logic and/or steps represented in flowcharts or otherwise described herein, for example, may be considered an ordered listing of executable instructions for implementing the logical functions, may be embodied in any computer-readable medium, For use by an instruction execution system, apparatus or apparatus (such as a computer-based system, a system including a processing module, or other system that can fetch instructions from and execute instructions from an instruction execution system, apparatus or apparatus), or in conjunction with such instruction execution system, apparatus or equipment. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport the program for use by or in connection with an instruction execution system, apparatus, or apparatus. More specific examples (non-exhaustive list) of computer readable media include the following: electrical connections with one or more wiring (electronic devices), portable computer disk cartridges (magnetic devices), random access memory (RAM), Read Only Memory (ROM), Erasable Editable Read Only Memory (EPROM or Flash Memory), Fiber Optic Devices, and Portable Compact Disc Read Only Memory (CDROM). In addition, the computer readable medium may even be paper or other suitable medium on which the program may be printed, as the paper or other medium may be optically scanned, for example, followed by editing, interpretation, or other suitable medium as necessary process to obtain the program electronically and then store it in computer memory.

The processor may be a Central Processing Unit (CPU), other general-purpose processors, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf programmable processor Field-Programmable Gate Array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

It should be understood that various parts of the embodiments of the present application may be implemented in hardware, software, firmware, or a combination thereof. In the above-described embodiments, various steps or methods may be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented by any one or a combination of the following techniques known in the art: Discrete logic circuits, application specific integrated circuits with suitable combinational logic gates, Programmable Gate Arrays (PGA), Field Programmable Gate Arrays (FPGA), etc.

Those skilled in the art can understand that all or part of the steps carried by the methods of the above embodiments can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium, and the program can be stored in a computer-readable storage medium. When executed, one or a combination of the steps of the method embodiment is included.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing module, or each unit may exist physically alone, or two or more units may be integrated into one module. The above-mentioned integrated modules can be implemented in the form of hardware, and can also be implemented in the form of software function modules. If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they may also be stored in a computer-readable storage medium.

The above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, and the like.

Although the embodiments of the present application have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limitations to the present application. Embodiments are subject to variations, modifications, substitutions and alterations.

Claims

A method for gluing the surface of objects based on robot vision, comprising:

Get the 3D point cloud information of the item;

Determine the 3D image information of the item based on the 3D point cloud information and the preset 3D image template information;

mapping the 3D image information to 2D image information;

generating 2D gluing track points based on the 2D image information;

mapping the 2D gluing track points to 3D gluing track points;

The glue is applied based on the mapped 3D glue track points.
The method for gluing the surface of an item according to claim 1, wherein the 3D point cloud information includes a 3D contour point cloud, and the acquiring the 3D point cloud information of the item includes:

Map the 3D point cloud of the item into a 2D image;

obtaining a 2D outline of the item based on the 2D image;

Map 2D contours to 3D contour point clouds.
The method for gluing the surface of an item according to claim 2, wherein the mapping the 3D point cloud of the item into a 2D image comprises: after acquiring the 3D point cloud of the item, first performing point cloud clustering and/or removing outliers process, and then map the processed 3D point cloud into a 2D image.
The method for gluing the surface of an article according to claim 1, wherein the mapping the 3D image information to the 2D image information comprises: using an orthogonal projection method to map the matched 3D image information to the 2D image information.
The method for gluing the surface of an article according to claim 1, wherein the generating a 2D gluing track point based on the 2D image information comprises:

Generate 2D contours based on 2D image information;

Traverse the entire contour at predetermined intervals and generate 2D gluing trajectory points.
The method for gluing the surface of an article according to claim 5, wherein: after shrinking the contour, the entire contour is traversed at predetermined intervals to generate 2D gluing track points.
The method for gluing the surface of an article according to claim 5, wherein the starting point and the ending point of the 2D trajectory point coincide.
The method for gluing the surface of an article according to claim 5, wherein the value range of the predetermined interval includes 50mm-100mm.
The method for gluing the surface of an article according to any one of claims 1 to 8, wherein the 3D image information of the article includes 3D image template information matched with the article and/or pose information of the article.
A device for gluing an object surface based on robot vision, characterized in that it includes:

3D point cloud acquisition module, used to acquire 3D point cloud information of items;

a 3D image determination module, configured to determine the 3D image information of the item based on the 3D point cloud information and the preset 3D image template information;

a 2D image mapping module for mapping the 3D image information to 2D image information;

a trajectory point generation module for generating 2D gluing trajectory points based on the 2D image information;

a 3D track point mapping module for mapping the 2D gluing track points to 3D gluing track points;

Glue module for gluing based on mapped 3D glue track points.
The robot vision-based object surface gluing device according to claim 10, wherein the 3D point cloud information includes a 3D contour point cloud, and the 3D point cloud acquisition module is further used for:

Map the 3D point cloud of the item into a 2D image;

obtaining a 2D outline of the item based on the 2D image;

Map 2D contours to 3D contour point clouds.
The article surface gluing device according to claim 11, wherein the mapping of the 3D point cloud of the article into a 2D image comprises: after acquiring the 3D point cloud of the article, first performing point cloud clustering and/or removing outliers process, and then map the processed 3D point cloud into a 2D image.
The device for gluing the surface of an article according to claim 10, wherein the 2D image mapping module is specifically used for: using an orthogonal projection method to map the matched 3D image information into 2D image information.
The article surface gluing device according to claim 10, wherein the trajectory point generation module is specifically used for:

Generate 2D contours based on 2D image information;

Traverse the entire contour at predetermined intervals and generate 2D gluing trajectory points.
The article surface gluing device according to claim 14, wherein the trajectory point generating module is further used for: shrinking the contour, then traversing the entire contour at predetermined intervals and generating 2D trajectory points.
The article surface gluing device according to claim 14, wherein the start point and the end point of the 2D track point coincide.
The device for gluing the surface of an article according to claim 14, wherein the predetermined interval ranges from 50 mm to 100 mm.
The device for gluing the surface of an article according to any one of claims 10 to 17, wherein the 3D image information of the article includes 3D image template information matched with the article and/or pose information of the article.
An electronic device, characterized by comprising: a memory, a processor, and a computer program stored on the memory and running on the processor, the processor implementing claims 1 to 1 when the processor executes the computer program The method for gluing the surface of an article based on robot vision according to any one of 9.
A computer-readable storage medium on which a computer program is stored, characterized in that, when the computer program is executed by a processor, the robot vision-based method for gluing the surface of an article according to any one of claims 1 to 9 is realized .