TW202004671A - Method and device for determining spatial position shape of object, storage medium and robot - Google Patents

Abstract

Disclosed by the present application are a method and device for determining the spatial position shape of an object, a storage medium and a robot. The method for determining the spatial position pattern of an object comprises: acquiring a binocular visual image of an object to be tested and a standard mark by means of a binocular vision device; correcting and fitting the binocular visual image of the object to be tested so as to obtain a range image of the object to be tested in a world coordinate system, and determining a point cloud data image of the object to be tested according to the range image; determining point cloud data of an upper surface and the center position of the upper surface as position data of an object to be tested; and determining shape data of the object to be tested according to a fitting surface of the point cloud data of the upper surface of the object to be tested and the distance between the center position of the upper surface of the object to be tested and a boundary position of the point cloud data of the upper surface of the object to be tested.

Description

Method and device for determining space position and shape of object, storage medium and robot

The embodiment of the present invention relates to the technical field of image recognition and image processing, for example, a method and device for determining the spatial position and shape of an object, a storage medium, and a robot.

According to image recognition and image processing methods, it is one of the important factors that affect the development of electronic technology to locate and shape the objects in the image.

The spatial position of the object, that is, the specific position under the spatial coordinate, the spatial form of the object, that is, the shape of the object in the spatial coordinate position. For example, in industrial production, when an industrial robot or robot arm grabs some standard or non-standard parts to achieve installation or assembly functions, if the spatial position and shape of the item are not determined, the mechanical operation method is directly used. It is easy to cause the shedding of standard parts or non-standard parts, affect the industrial production efficiency, and sometimes even damage the assembly line or industrial robot.

In life, such as drones, intelligent robots, etc., if you can not automatically determine the spatial position and shape of the object to be measured, you must need human assistance to achieve the carrying and transportation of items. If you lose human assistance, not only can’t Carry out normal work, and it is difficult to expand the business, which affects the development of electronic technology. Therefore, how to determine the position and shape of space objects has become an urgent technical problem in the field.

The embodiment of the present invention provides a method and device for determining the spatial position and shape of an object, a storage medium, and a robot, which can realize the image of the object to be tested by the binocular vision device, and after processing and analysis, determine the space of the object to be tested Position and shape effects.

In a first aspect, the embodiment of the present invention provides a method for determining the spatial position and shape of an object. The method includes: acquiring a binocular visual image of an object to be tested and a standard mark by a binocular visual device; The device is arranged above the test object; according to the positional relationship between the standard mark and the test object, and the binocular visual image of the standard mark, the binocular visual image of the test object is corrected and fitted to obtain Depth image of the object under test in the world coordinate system, according to the depth image of the object under test in the world coordinate system to determine the point cloud data image of the object under test; use the vertical spatial statistical method to determine the object under test The upper surface point cloud data of the object; determine the center position of the upper surface of the object under test based on the upper surface point cloud data of the object under test, as the position data of the object under test; and according to the upper surface point cloud data of the object under test And the distance between the center position of the upper surface of the object to be measured and the boundary position of the point cloud data of the upper surface of the object to be measured determines the shape data of the object to be tested.

In some embodiments, after determining the position data of the object to be tested and the shape data of the object to be tested, the method further includes: determining a robot based on the position data of the object to be tested and the shape data of the object to be tested The grasping position and grasping posture of the operating arm are used to control the robot operating arm to grab the object to be tested.

In some embodiments, before the binocular vision image of the test object and the standard mark is acquired by the binocular vision device, the method further includes: selecting a fixed structure as the standard in the load space of the test object Mark, or install a mark as the standard mark in the carrying space of the object to be tested, and establish the coordinate system of the standard mark and the position of the binocular vision device through the positional relationship between the binocular vision device and the standard mark The relationship between coordinate systems.

In some embodiments, the binocular visual image of the object to be tested is corrected and fitted according to the positional relationship between the standard mark and the item to be tested and the binocular visual image of the standard mark to obtain the object to be tested The depth image of the object under the world coordinate system includes: using the aforementioned standard mark to determine the position of the binocular vision device; according to the positional relationship between the position of the aforementioned binocular vision device and the aforementioned standard mark, determining the position of the aforementioned object to be tested Correction parameters of the world coordinate system of the binocular visual image; according to the correction parameters of the world coordinate system of the binocular visual image of the object to be tested, convert the binocular visual image of the object to be tested to the world coordinate system, and then proceed Depth image fitting to obtain the depth image of the object under test in the world coordinate system.

In some embodiments, the binocular visual image of the object to be tested is corrected and fitted according to the positional relationship between the standard mark and the item to be tested and the binocular visual image of the standard mark to obtain the object to be tested The depth image of the object in the world coordinate system includes: fitting the depth image of the binocular visual image of the standard mark and the binocular visual image of the object to be tested to obtain their respective primary depth images; The positional relationship between the position of the binocular vision device and the standard mark determines the world coordinate system correction parameters of the primary depth image of the object under test; the world coordinate system correction parameters according to the elementary depth image of the object under test , The primary depth image of the object to be tested is corrected by the world coordinate system to obtain the depth image of the object to be tested under the world coordinate system.

In some embodiments, after determining the point cloud data image of the object under test based on the depth image of the object under test in the world coordinate system, a vertical spatial statistical method is used to determine the upper surface point of the object under test Before the cloud data, the foregoing method further includes: filtering the background point cloud data according to the three-color difference degree of the point cloud data in the point cloud data image of the object to be measured, and obtaining the cloud image data of the former scenic spot; correspondingly, Using the vertical spatial statistical method to determine the upper surface point cloud data of the object to be measured includes: using the vertical spatial statistical method to determine the upper surface point cloud data of the object to be measured from the front spot cloud data image.

In some implementation forms, the background point cloud data is filtered according to the three-color difference of the point cloud data in the point cloud data image of the object to be measured, and the front spot cloud data image is obtained, including: using the following formula Determine the three-color difference value of the point cloud data in the aforementioned point cloud data image: T=|R _point -G _point |-|G _point -B _point |; where R _point represents the red of the RGB colors in the point cloud data G _point represents the value of green in the RGB colors in the point cloud data; B _point represents the value of blue in the RGB colors in the point cloud data; where T is the three-color difference value, and the aforementioned three-color difference value When it is smaller than the background filtering threshold, it is determined that the corresponding point cloud data is background point cloud data, and the foregoing background point cloud data is filtered.

In some implementation forms, the vertical space statistical method is used to determine the point cloud data of the top surface of the object to be measured from the aforementioned cloud data image of the front sight, including: statistical distribution of the vertical data of all cloud data of the front sight ; Determine the number of the front cloud data in each vertical data interval in the statistical distribution; determine the median value of the vertical data interval with the largest number of front cloud data; point cloud that distributes the vertical data within the set range Data, as the point cloud data of the upper surface of the object to be measured; the setting range is the first value obtained by subtracting the preset controllable value from the median value of the vertical data interval, and the median value of the vertical data interval The range of values formed by the value plus the second value obtained from the preset controllable value.

In some implementations, the aforementioned preset controllable value is determined in the following manner: the vertical data of all cloud data images of the front sight are counted to determine the standard deviation; and the set multiple of the aforementioned standard deviation is used as the preset controllable value.

In some embodiments, the center position of the upper surface of the object to be measured is determined according to the point cloud data of the upper surface of the object to be tested, and the position data of the object to be tested includes: according to the upper surface points of all the objects to be tested The average value of the spatial coordinates of the cloud data determines the center position of the upper surface of the test object as the position data of the test object; the fitting surface according to the point cloud data of the upper surface of the test object and the test object The distance between the center position of the upper surface of the upper surface and the boundary position of the upper surface point cloud data of the object under test to determine the morphological data includes: performing plane fitting according to the spatial coordinates of the upper surface point cloud data of the object under test to determine the aforementioned object The upper surface of the test object; determine the normal vector of the upper surface of the test object, and determine the torsion angle Rx of the test object in the world coordinate system along the X axis and the torsion angle Ry along the Y axis according to the normal vector ; And, the point cloud data of the upper surface of the object to be measured is projected on the XOY plane; the minimum value is determined from the distance between the projection position of the upper surface center position and the projection position of the upper surface point cloud data boundary position; The direction of the minimum value determines the torsion angle Rz of the object under test along the Z axis; the Rx, Ry, and Rz are determined as the morphological data of the object under test.

In a second aspect, the embodiment of the present invention further provides a device for determining the spatial position of an object. The device includes: a binocular visual image acquisition module configured to acquire the object to be tested and the standard mark by the binocular visual device Binocular visual image; wherein, the aforementioned binocular visual device is arranged above the object to be tested; the point cloud data image determination module is set according to the positional relationship between the standard mark and the object to be tested, and the standard mark Binocular visual image, correcting and fitting the binocular visual image of the object under test to obtain the depth image of the object under test under the world coordinate system, according to the depth image of the object under test under the world coordinate system Determine the point cloud data image of the object to be tested; the upper surface point cloud data filtering module, set to use the vertical spatial statistical method to determine the upper surface point cloud data of the object to be tested; the position data and shape data determination module, Set to determine the center position of the upper surface of the object to be tested based on the point cloud data of the upper surface of the object to be measured as position data of the object to be tested; and according to the fitting surface of the point cloud data of the upper surface of the object to be tested, And the distance between the center position of the upper surface of the object to be measured and the boundary position of the point cloud data of the upper surface of the object to be measured determines the shape data of the object to be tested.

In a third aspect, the embodiment of the present invention provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, a method for determining the spatial position of an object as described in the embodiment of the present invention is implemented.

According to a fourth aspect, the embodiment of the present invention provides a binocular vision robot, which includes a binocular vision device, a standard mark, a memory, a processor, and a computer program stored on the memory and executable on the processor. The method for determining the spatial position and shape of the object as described in the embodiments of the present invention is implemented when the foregoing computer program is executed.

The technical means provided by the embodiment of the present invention obtain the binocular visual image of the test object and the standard mark by a binocular vision device; wherein the binocular vision device is arranged above the test object; according to the aforementioned standard The positional relationship between the mark and the aforementioned test object, and the binocular visual image of the standard mark, correcting and fitting the binocular visual image of the aforementioned test object to obtain the depth image of the test object under the world coordinate system, Determine the point cloud data image of the object under test based on the depth image of the object under test in the world coordinate system; determine the point cloud data of the upper surface of the object under test using vertical spatial statistical methods; The upper surface point cloud data determines the center position of the upper surface of the object to be measured as the position data of the object to be measured; and the fitting surface according to the point cloud data of the upper surface of the object to be measured and the upper surface of the object to be measured The distance between the center position and the boundary position of the point cloud data on the upper surface of the object to be measured determines the morphological data of the object to be tested, which can be achieved by acquiring and processing the image of the object to be tested by binocular vision device after processing and analysis. The effect of the spatial position and shape of the item to be tested.

10‧‧‧ Binocular vision device

20‧‧‧operator

30‧‧‧Standard mark on the console

50‧‧‧Robot operating arm

FIG. 1 is a flowchart of a method for determining the spatial position and shape of an object provided by Embodiment 1 of the present invention.

FIG. 2 is a flowchart of a method for determining the spatial position and shape of an object provided by Embodiment 2 of the present invention.

[FIG. 3] is a schematic diagram of statistical distribution of point cloud data provided by Embodiment 2 of the present invention.

FIG. 4 is a schematic structural diagram of a device for determining a spatial position and shape of an object according to Embodiment 3 of the present invention.

[Figure 5a] is a schematic diagram of a binocular vision robot provided by Embodiment 5 of the present invention.

[Figure 5b] is a schematic diagram of a binocular vision robot provided by Embodiment 5 of the present invention.

[Figure 5c] is a schematic diagram of a binocular vision robot provided by Embodiment 5 of the present invention.

FIG. 6 is a schematic diagram of a method for determining Rz in object spatial shape data provided by Embodiment 2 of the present invention.

The present invention will be further described in detail below with reference to the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should be further noted that, in order to facilitate description, the drawings only show parts, but not all structures related to the present invention.

Before discussing exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowchart describes the steps as sequential processing, many of the steps can be implemented in parallel, concurrently, or simultaneously. In addition, the order of the steps can be rearranged. The aforementioned processing may be terminated when its operation is completed, but may also have additional steps not included in the drawing. The foregoing processing may correspond to methods, functions, procedures, subroutines, subroutines, and so on.

Example one

FIG. 1 is a flowchart of a method for determining an object’s spatial position and shape according to Embodiment 1 of the present invention. This embodiment is applicable to the positioning and shape determination of objects to be tested. The method may be based on the object space provided by the embodiment of the present invention. The position and shape determination device is executed. The device can be implemented by at least one of software and hardware, and can be integrated into the binocular vision robot.

As shown in FIG. 1, the method for determining the spatial shape of the aforementioned object includes:

S110. Obtain the object to be tested and the binocular vision image of the standard mark by the binocular vision device; wherein the binocular vision device is arranged above the object to be tested.

Among them, the binocular vision device can be used to obtain the spatial position and shape of the object to be tested within a fixed range. For example, in production and installation lines, the binocular vision device can be installed at a fixed position directly above the operating table to make the binocular vision device The center position of the visual vision device corresponds to the center position of the console, so that it can be determined that the image collected by the binocular vision device is directly facing the center of the console. It can also be installed in an unfixed position, such as moving the head of the robot, producing and installing the robotic arm of the assembly line, which can make the setting of the binocular vision device more flexible, but compared to the former, this method is shown in the figure. The image correction process is relatively complicated. If the position of the binocular vision device is fixed, the position of the object to be measured in the world coordinate system can be corrected by the position of the obtained image, and for the movable position of the binocular vision device, it must be collected The binocular vision image contains standard marks to determine the position of the object in the world coordinate system, or the relative position of the robot itself or the robot mechanical arm.

In this embodiment, optionally, before the binocular vision image of the object to be tested and the standard mark is acquired by the binocular vision device, the method further includes: selecting a fixed structure in the object-bearing space of the object to be tested As the standard mark, or install a mark as the standard mark in the load space of the object to be tested, establish the coordinate system of the standard mark and the binocular by the positional relationship between the binocular vision device and the standard mark The relationship between the coordinate system of the visual device. The advantage of this setting is that the image can be corrected and fitted according to fixed or preset standard marks.

Among them, the standard mark is a mark set at a fixed position to calibrate the binocular vision image in the binocular vision image. For example, it can be an arrow that crosses to north and east.

The above-mentioned binocular vision device is arranged above the above-mentioned object to be tested, so the setting is to be able to obtain the image of the upper surface of the object to be tested, because when grabbing some objects by the robot or the robot operating arm, it is often by The grab angle is determined according to the shape of the item to be grabbed. If the robot can grasp the object by lateral grabbing, it can also be determined by acquiring the position and shape of the front surface of the object to be tested.

S120. Correct and fit the binocular visual image of the test object according to the positional relationship between the standard mark and the test object and the binocular visual image of the standard mark to obtain the test object under the world coordinate system According to the depth image of the object under test in the world coordinate system, the point cloud data image of the object under test is determined.

Among them, the standard mark is a mark set at a fixed position to calibrate the binocular vision image in the binocular vision image. For example, it can be an arrow that crosses to north and east. The depth image may be an image with depth information for each pixel. In this embodiment, the corrected depth image may be the depth information from top to bottom, which may be the Z-axis position where the binocular vision device is located. As a starting point, the depth information may be the vertical distance (Z-axis distance) between each pixel constituting the image and the plane where the center of the binocular vision device is located. The point cloud data image can be displayed by each pixel in the form of a point cloud, and the point cloud data image can be transformed from the depth image according to a specific algorithm.

In this embodiment, optionally, the binocular visual image of the test object is corrected and fitted according to the positional relationship between the standard mark and the item to be tested, and the binocular visual image of the standard mark, Obtaining the depth image of the object under test in the world coordinate system includes: determining the position of the binocular vision device using the aforementioned standard mark; determining the aforementioned object according to the positional relationship between the position of the aforementioned binocular vision device and the aforementioned standard mark Correction parameters of the world coordinate system of the binocular visual image of the test object; according to the correction parameters of the world coordinate system of the binocular visual image of the test object, convert the binocular visual image of the test object to the world coordinate system Then, the depth image is fitted to obtain the depth image of the object to be tested in the world coordinate system.

In this embodiment, optionally, the binocular visual image of the test object is corrected and fitted according to the positional relationship between the standard mark and the item to be tested, and the binocular visual image of the standard mark, Obtain the depth image of the test object in the world coordinate system, including: fitting the depth image of the binocular visual image of the standard mark and the binocular visual image of the test object to obtain the respective primary depth map Image; according to the positional relationship between the position of the binocular vision device and the standard mark, determine the world coordinate system correction parameters of the primary depth image of the test object; according to the world coordinate of the primary depth image of the test object According to the correction parameters, the primary depth image of the object to be tested is corrected by the world coordinate system to obtain the depth image of the object to be tested under the world coordinate system.

The above two methods respectively explain the method of first correcting the world coordinate system of the pictures acquired by the two cameras of the binocular vision device, and then performing the fitting, and the method of first fitting and then correcting the world coordinate system. The following describes the process of world coordinate system correction and image fitting: Since the original binocular images were taken independently by the left-eye camera and the right-eye camera, due to the different positions of the camera lens, the two cameras have certain distortion. It is necessary to fit all the pixels in the field of view, and give the compensation amount of the fitting to the camera program based on the measured data.

In addition, it further includes determining the correspondence between the object point in the space coordinate system and its image point on the image plane. Optionally, adjust the internal parameters of the left-eye camera and the right-eye camera to be consistent. The internal parameters include: the camera's internal geometry and optical parameters, and the external parameters include: the conversion between the left-eye and right-eye camera coordinate systems and the world coordinate system.

The fitting here is used to correct the distortion produced by the lens. The distortion caused by the lens can be seen in the original image. For example, a straight line in the scene will become a curve in the original left and right eye images. This effect is especially noticeable at the corners of the left and right eye images. The fitting is to correct this type of distortion.

During image processing, boundary extraction is performed based on the object image. The algorithms that can be used include: Laplacian-Gaussian filtering. The boundary feature is an obvious and main feature for identifying objects, which lays the foundation for subsequent algorithms. In addition, further includes, in image preprocessing and feature extraction, preprocessing: mainly includes image contrast enhancement, random noise removal, low-pass filtering and image enhancement, pseudo color processing, etc.; feature extraction: commonly used matching Features, mainly point-like features, linear features and regional features, are extracted. Among them, in order to fit an image, low-pass filtering is very important to smooth it in advance. So if you want to fit an image, it is a good method to open the low-pass filter to the left and right eye images in advance. Of course, the image can also be corrected without using a low-pass filter, but the corrected image may be confused. If you want to increase the processing speed, you can turn off the low-pass filter.

Edge detection is an optional feature that uses changes in brightness to match features. This function is very useful when the camera in the system has an automatic gain function. If the change of the automatic gain of each camera is inconsistent, then the absolute brightness between the images is inconsistent, and although the absolute brightness is inconsistent, the change in brightness is a constant. Therefore, edge detection is suitable for environments with large changes in lighting. Although edge detection can improve the recognition results of the edge of the item, this is equivalent to introducing another processing step, so we must weigh the relationship between the improvement of the result and the speed to use this function.

During the image processing, according to the principle of binocular stereo imaging, the three-dimensional measurement of binocular stereo vision is based on the principle of parallax.

Baseline distance B = the distance between the projection centers of the two cameras; the focal length of the camera is f. Suppose two cameras are viewing the same feature point of a space object at the same time as P ( x _c , y _c , z _c ) under the space coordinate system, and images of point P are acquired on the "left eye" and "right eye" respectively , Their image coordinates under the image coordinates are P _left = (X _left , Y _left ) and P _right = (X _{right t} , Y _right ). Now that the images of the two cameras are on the same plane, the image coordinates Y of the feature point P are the same, that is, Y _left =Y _right =Y, which is obtained from the triangular geometric relationship:

The parallax is: D _parallax = X _left- X _right . From this, the three-dimensional coordinates of the feature point P under the camera coordinate system can be calculated as:

Therefore, as long as any point on the image surface of the left-eye camera can find a corresponding matching point on the image surface of the right-eye camera, the three-dimensional coordinates of the point can be determined. This method is a complete point-to-point operation. As long as there is a corresponding matching point for all points on the image surface, it can participate in the above operation, so as to obtain its corresponding three-dimensional coordinates.

In addition, when performing image stereo matching, the correspondence between the features is established according to the calculation of the selected features, and the mapping points of the same spatial physical point in different images are mapped. Stereo matching consists of three basic steps: 1) Select the image features corresponding to the actual physical structure from one image in the stereo image pair as shown on the left; 2) Determine on the other image as shown on the right Out the corresponding image features of the same physical structure; 3) determine the relative position between these two features to get the parallax.

Step 2) is the key to matching.

Determining the depth After obtaining the parallax image through stereo matching, the depth image can be determined and the 3D information of the scene can be restored. Stereo matching establishes a correlation library using the method of absolute correlation deviation sum to establish correlation between images. The principle of this method is as follows: for each pixel in the image in the reference image, select a neighborhood according to the given square size, and this neighborhood along the same line with one of the other images The series of neighborhoods is compared to find the best matching end. Calculation using absolute variance correlation:

Where: d _min and d _max are the minimum and maximum disparity; m is the mask size; I _left and I _right are the left and right images.

In the process of image processing, through the calculation of the image after binocular fitting of the object, the correlation calculation depth between the images is established according to the binocular parallax principle formula and the method of the absolute correlation deviation sum, forming a depth map or spatial point Cloud data.

S130. Determine the point cloud data of the upper surface of the object to be measured by using a vertical spatial statistical method.

After obtaining the point cloud data image, the longitudinal (Z-axis) data of each point can be determined, and according to the statistics of the longitudinal data, the number of point clouds in each height range in the current point cloud image can be obtained. Among them, if the background is a plane, such as a console, the number of point clouds can be the largest in the longitudinal data of the background, and among all the point cloud data, the Z-axis data of the background point cloud data is also the largest Or the smallest. With this kind of statistics, background point cloud data can be filtered out. By counting the number of point cloud data in a certain range in the cloud data of the former scenic spot, the point cloud data on the upper surface of the item to be measured can be determined. If the upper surface is level, the point cloud data range of the upper surface is relatively narrow, and if the upper surface is inclined, the point cloud data range of the upper surface is relatively wide.

S140. Determine the center position of the upper surface of the object to be measured according to the point cloud data of the upper surface of the object to be measured as position data of the object to be tested; and according to the fitting surface of the point cloud data of the upper surface of the object to be tested, And the distance between the center position of the upper surface of the object to be measured and the boundary position of the point cloud data of the upper surface of the object to be measured determines the shape data of the object to be tested.

Among them, the center position of the upper surface can be determined by the world coordinate system determined by the standard mark, such as (X, Y, Z). For example, the center position of the upper surface can be determined according to the center position of the geometric shape of the projection of the point cloud data of the upper surface on the XOY plane.

The morphological data is determined according to the fitting surface of the aforementioned upper surface point cloud data and the distance between the center position of the upper surface and the boundary position of the upper surface point cloud data. The morphological data can be expressed by determining three quantities of rotation angles Rx, Ry and Rz of the object to be measured with respect to the X-axis, Y-axis and Z-axis. After the fitting surface of the upper surface is determined, the fitting surface may be a flat surface or a curved surface. After the normal vector on the upper surface is determined, Ry can be determined according to the angle between the normal vector on the upper surface and the XOZ plane, and Rx can be determined based on the included angle with the YOZ plane. Then, Rz is determined by the angle between the vector formed by the center position of the upper surface and the closest point among the boundary points of the upper surface and the XOY plane.

The technical means provided by the embodiments of the present invention obtain the binocular visual image of the test object and the standard mark by a binocular vision device; wherein the binocular vision device is arranged above the test object; according to the standard mark The positional relationship with the aforementioned test object and the binocular visual image of the standard mark are corrected and fitted to the aforementioned binocular visual image of the tested object to obtain the depth image of the tested object under the world coordinate system, according to Determining the point cloud data image of the object under test using the depth image of the object under test in the world coordinate system; determining the point cloud data of the upper surface of the object under test using vertical spatial statistical methods; The surface point cloud data determines the center position of the upper surface of the object to be measured as the position data of the object to be measured; and based on the fitting surface of the point cloud data of the upper surface of the object to be measured and the center of the upper surface of the object to be measured The distance between the position and the boundary position of the point cloud data on the upper surface of the object to be measured determines the morphological data of the object to be tested. After obtaining the image of the object to be tested by the binocular vision device, after processing and analysis, the The effect of measuring the spatial position and form of items.

Based on the above technical means, optionally, after determining the position data and the form data, the method further includes: determining the gripping position and gripping posture of the robot operating arm based on the position data and the form data, To control the robot operating arm to grab the object to be tested.

Among them, the position of the robot operating arm can be corrected to the same world coordinate system as the object to be measured, so that the movement distance, direction and even trajectory of the operation arm can be determined. When the operation arm moves to the position of the object to be measured, the operation can be controlled The gripper of the arm grabs the object to be tested in a form suitable for the object to be tested. The advantage of this setting is that it can be determined that the object to be tested can be grasped smoothly after the position recognition, and the grasping is more compact to avoid accidents such as grasping and falling off.

Example 2

FIG. 2 is a flowchart of a method for determining the spatial position and shape of an object provided by Embodiment 2 of the present invention. Based on the above embodiment, this embodiment is modified as follows: after determining the point cloud data image of the object under test based on the depth image of the object under test in the world coordinate system, the vertical spatial statistical method is used to determine the aforementioned Before the point cloud data on the upper surface of the object to be measured, the foregoing method further includes: filtering the background point cloud data according to the three-color difference degree of the point cloud data in the point cloud data image of the object to be measured, to obtain the front spot cloud Data image; correspondingly, using the vertical spatial statistical method to determine the upper surface point cloud data of the object to be measured includes: using the vertical spatial statistical method to determine the upper surface of the object to be measured from the foregoing cloud data image of the front spot Surface point cloud data.

As shown in FIG. 2, the method for determining the spatial shape of the aforementioned object includes:

S210. Obtain the binocular vision image of the test object and the standard mark by the binocular vision device; wherein the binocular vision device is arranged above the test object.

S220. Correct and fit the binocular visual image of the test object according to the positional relationship between the standard mark and the test object and the binocular visual image of the standard mark to obtain the test object under the world coordinate system According to the depth image of the object under test in the world coordinate system, the point cloud data image of the object under test is determined.

S230. Filter the background point cloud data according to the three-color difference degree of the point cloud data in the point cloud data image of the aforementioned object to be measured, to obtain a cloud image of the former scenic spot.

Among them, the three-color difference degree can be the difference between the values of the three primary colors of red, green, and blue in the pixel color of each data in the point cloud data image. This setting can mainly filter the color or close background point cloud data In addition, you can get a point cloud data image with only the cloud data of the previous attractions.

In this embodiment, optionally, the background point cloud data is filtered according to the three-color difference degree of the point cloud data in the aforementioned point cloud data image, to obtain the front spot cloud data image, which includes: The three-color difference value of the point cloud data in the aforementioned point cloud data image: T=|R _point -G _point |-|G _point -B _point |; where R _point represents the red of the RGB colors in the point cloud data Numerical value; G _point represents the numerical value of green in the RGB color in the point cloud data; B _point represents the numerical value of blue in the RGB color in the point cloud data; where T is the three-color difference value, and the aforementioned three-color difference value is less than When the background filtering threshold is determined, the corresponding point cloud data is determined as background point cloud data, and the filtering operation is performed.

Such setting is advantageous for filtering the point cloud data of the background and other reflective or individual jumping noise points, and improves the accuracy of determining the point cloud data on the upper surface.

S240. Use the vertical spatial statistical method to determine the point cloud data on the upper surface of the object to be measured from the cloud data image of the front sight.

In this embodiment, optionally, the vertical space statistical method is used to determine the point cloud data on the upper surface of the object to be measured, which includes: statistically distributing the vertical data of all the front-point cloud data; determining each vertical data The number of cloud data of the front sight in the interval; determine the median value of the vertical data interval with the largest number of cloud data of the front sight; the point cloud data that distributes the vertical data within the set range as the upper surface point of the aforementioned object to be measured Cloud data; the aforementioned setting range is the first value obtained by subtracting the preset controllable value from the median value of the vertical data interval, and the third value obtained by adding the preset controllable value to the median value of the vertical data interval The range of values formed by the two values.

FIG. 3 is a schematic diagram of statistical distribution of point cloud data provided by Embodiment 2 of the present invention. As shown in Figure 3, the horizontal axis is the vertical data of the point cloud data, which can be understood as the height, in meters, and the vertical coordinate is the number of point cloud data in each data interval, which is expressed as the midpoint of the current vertical data interval The number of cloud data. For example, the data interval is 0.02. In the figure, the maximum number of point clouds in 0.414-0.416 can be determined as point cloud data in a certain range centered on 0.415.

In this embodiment, optionally, the aforementioned preset controllable value is determined in the following manner: the vertical data of all the cloud data images of the front sight are counted to determine the standard deviation; the set multiple of the aforementioned standard deviation is used as the preset controllable Value.

Identify the upper plane of the target object, take the colored points of the target object, and perform statistical distribution according to the vertical direction of the Z direction. Use the point cloud data μ (μ is the statistical mean) and the statistical high frequency peak (Peak) value. As shown in Figure 3, σ (σ is the standard deviation Standard Deviation) is also used to control the selected range. In fact, the point in the range between 1σ-6σ as the upper plane of the target object is a typical data distribution effect, as shown in the figure As shown in 3, the thick line represents the median μ, and the broken line interval represents +/-σ standard deviation. And that these points constitute the main imaging surface or upper surface of the target object. At the same time, this method is used to remove the deviation caused by the point cloud data of reflective points, outliers and shadow points.

S250. Determine the center position of the upper surface of the object to be tested according to the point cloud data of the upper surface of the object to be measured, as the position data of the object to be tested; and according to the fitting surface of the point cloud data of the upper surface of the object to be tested, And the distance between the center position of the upper surface of the object to be measured and the boundary position of the point cloud data of the upper surface of the object to be measured determines the shape data of the object to be tested.

This embodiment provides a method for determining the cloud data of the front sight on the basis of the above embodiments. With this method, the interference caused by the point cloud data of reflective points, outliers, and shadow points can be removed, which improves Determine the accuracy of the point cloud data on the upper surface of the item to be measured.

Based on the above technical means, optionally, the center position of the upper surface of the object to be measured is determined according to the point cloud data of the upper surface of the object to be tested, and the position data of the object to be tested includes: The average value of the spatial coordinates of the surface point cloud data determines the center position of the upper surface of the object under test as position data; the fitting surface according to the point cloud data of the upper surface of the object under test, and the upper surface of the object under test The distance between the center position and the boundary position of the upper surface point cloud data of the object to be measured determines the morphological data of the object to be measured, including: performing plane fitting according to the spatial coordinates of the upper surface point cloud data to determine the object to be measured The upper surface of the test object; determine the normal vector of the upper surface of the test object, determine the torsion angle Rx of the test object in the world coordinate system along the X axis, and the twist angle Ry along the Y axis according to the normal vector; and , Project the point cloud data of the upper surface of the object to be tested on the XOY plane; determine the minimum value between the projection position of the upper surface center position and the projection position of the upper surface point cloud data boundary position; The direction of the value determines the torsion angle Rz of the object under test along the Z axis; the aforementioned Rx, Ry, and Rz are determined as the morphological data of the object under test. The advantage of this setting is that it can improve the accuracy and simplicity of the process of determining the six parameters of the space of the object to be tested, and improve the accuracy of the technical means provided by the embodiments of the present invention.

6 is a schematic diagram of a method for determining Rz in object spatial shape data provided by Embodiment 2 of the present invention. As shown in Figure 6, after determining the point cloud data on the surface of the object to be measured, you can project on the XOY plane, where the Z axis coincides with the O point, which is not shown in the figure, and convert the three-dimensional point into a two-dimensional plane . The original determined center point can be used as the projected center point. After determining the center point, all the peripheral points in the set are extracted to form a convex polygon (Convex hull 2D), and the vertices forming the convex polygon are marked as boundary points (as shown in the figure). , Only partially marked). The center point can form a triangle with two adjacent vertices of the polygon, and any two adjacent vertices form a line segment. As shown in Figure 6, H4 is the triangle formed by the center point and the two boundary points, the height from the center point to the boundary of the part, the figure shows the five height values of H1, H2, H3, H4 and H5, where H4 Is the minimum value and H3 is the maximum value.

Found in any two vertices forming a line segment, the shortest distance (H4) from the center point to the line segment (Segment), that is, in the polygon surrounded by the point cloud on the upper surface, the shortest side distance is vertical foot. After knowing the shortest distance and direction from the center of the upper surface to the boundary polygon, vectorize H4 to obtain the angle formed by the vector H4 and the X axis or the Y axis, which is the angle Rz of the object to be tested rotating around the Z axis, Therefore, the vector H4 can represent the Rz direction of the center point of the upper surface. According to the center point of the upper surface of the target object, take the vector whose center point is the shortest from the boundary polygon (the shortest distance point from the center to the boundary line segment). According to the orientation of the shortest side and the XOZ plane (or YOZ plane), determine the included angle of the Rz of the target object.

Example Three

4 is a schematic structural diagram of a device for determining a spatial position and shape of an object according to Embodiment 3 of the present invention. As shown in FIG. 4, the aforementioned device for determining the spatial position and shape of the object includes: a binocular visual image acquisition module 410, which is configured to acquire a binocular visual image of an object to be tested and a standard mark through the binocular visual device; The aforementioned binocular vision device is arranged above the aforementioned test object; the point cloud data image determination module 420 is set to be based on the positional relationship between the aforementioned standard mark and the aforementioned test object, and the binocular visual image of the standard mark, Correcting and fitting the binocular visual image of the object to be tested to obtain a depth image of the object to be tested in the world coordinate system, and determining the depth of the object to be tested according to the depth image of the object to be tested in the world coordinate system Point cloud data image; upper surface point cloud data filtering module 430, configured to determine the upper surface point cloud data of the object to be measured using vertical spatial statistical methods; position data and shape data determination module 440, configured to be based on the foregoing The point cloud data of the upper surface of the test object determines the center position of the upper surface of the test object as the position data of the test object; and according to the fitting surface of the point cloud data of the upper surface of the test object and the test object The distance between the center position of the upper surface of the article and the boundary position of the point cloud data of the upper surface of the article to be measured determines the shape data of the article to be measured.

The technical means provided by the embodiments of the present invention obtain the binocular visual image of the test object and the standard mark by a binocular vision device; wherein the binocular vision device is arranged above the test object; according to the standard mark The positional relationship with the aforementioned test object and the binocular visual image of the standard mark are corrected and fitted to the aforementioned binocular visual image of the tested object to obtain the depth image of the tested object under the world coordinate system, according to Determining the point cloud data image of the object under test using the depth image of the object under test in the world coordinate system; determining the point cloud data of the upper surface of the object under test using vertical spatial statistical methods; The surface point cloud data determines the center position of the upper surface of the object to be measured as the position data of the object to be measured; and based on the fitting surface of the point cloud data of the upper surface of the object to be measured and the center of the upper surface of the object to be measured The distance between the position and the boundary position of the point cloud data on the upper surface of the object to be measured determines the morphological data of the object to be tested. After obtaining the image of the object to be tested by the binocular vision device, after processing and analysis, the The effect of measuring the spatial position and form of items.

The above-mentioned products can execute the method provided by any embodiment of the present invention, and have the function modules and functions corresponding to the execution method.

Example 4

Embodiments of the present invention further provide a storage medium including computer-executable instructions, which when executed by a computer processor are used to perform a method for determining the spatial position and shape of an object, the method includes: using a binocular vision device Obtain the binocular vision image of the test object and the standard mark; wherein the binocular vision device is arranged above the test object; according to the positional relationship between the standard mark and the test object, and the binocular vision of the standard mark Image, correcting and fitting the binocular vision image of the object under test to obtain the depth image of the object under test in the world coordinate system, and determining the object to be tested according to the depth image of the object under test in the world coordinate system The point cloud data image of the measured object; the vertical surface statistical method is used to determine the point cloud data of the upper surface of the object to be measured; the center position of the upper surface of the object to be measured is determined according to the point cloud data of the upper surface of the object to be measured, as The position data of the aforementioned test object; and the distance between the center position of the upper surface point cloud data of the aforementioned test object and the boundary position of the upper surface point cloud data of the aforementioned test object To determine the shape data of the item to be tested.

Storage medium-any type of memory device or storage device. The term "storage media" is intended to include: installation media such as CD-ROM, floppy disk or tape devices; computer system memory or random access memory such as DRAM, DDR RAM, SRAM, EDO RAM, Lambasse ( Rambus) RAM, etc.; non-volatile memory, such as flash memory, magnetic media (such as hard drives or optical storage); registers or other similar types of memory devices, etc. The storage medium may further include other types of memory or a combination thereof. In addition, the storage medium may be located in a computer system in which the program is executed, or may be located in a different second computer system that is connected to the computer system through a network (such as the Internet). The second computer system can provide program instructions to the computer for execution. The term "storage medium" can include two or more storage media that can reside in different locations (eg, in different computer systems connected by a network). The storage medium may store program instructions executable by one or more processors (for example, embodied as a computer program).

Of course, a storage medium including computer-executable instructions provided by an embodiment of the present invention is not limited to the above-mentioned operation of determining the spatial position and shape of an object, and can also execute objects provided by any embodiment of the present invention Relevant operations in the method of determining spatial position patterns.

Example 5

An embodiment of the present invention provides a binocular vision robot, which includes a binocular vision device, an operating table, a standard mark on the operating table, a robot operating arm, a memory, a processor, and a processor stored on the memory and capable of running on the processor A computer program, when the processor executes the computer program, the method for determining the spatial position and shape of an object according to any one of the embodiments of the present invention is implemented.

FIG. 5a is a schematic diagram of a binocular vision robot provided by Embodiment 5 of the present invention. As shown in FIG. 5a, the binocular vision device 10, the console 20, the standard mark 30 on the console, the robot operating arm 50, the memory, the processor, and the computer program stored on the memory and running on the processor, When the foregoing processor executes the foregoing computer program, the method for determining the spatial position and shape of an object according to any one of the embodiments of the present invention is implemented.

FIG. 5b is a schematic diagram of a binocular vision robot provided by Embodiment 5 of the present invention. As shown in FIG. 5b, relative to the above-mentioned technical means, setting the binocular vision device 10 on the jaws can make the binocular vision image acquisition more flexible, and can be used when there are many items to be tested, or through After the calculation accuracy on one side does not meet the standard or the noise rate is too high, by controlling the movement of the jaws, the six parameters of the space to be measured can be positioned from another angle. The spatial six-parameter results obtained from multiple positions can also be compared and confirmed with each other, thereby improving the accuracy of the results of the determination of the spatial position and shape of the object to be tested by the technical solution provided by the embodiments of the present invention.

5c is a schematic diagram of a binocular vision robot provided by Embodiment 5 of the present invention. As shown in FIG. 5c, relative to the above-mentioned multiple technical means, the binocular vision device is installed on the body of the robot operating arm, so that the situation that the first solution specifically provides a mounting bracket for the binocular vision device can be avoided, and at the same time When the robot operating arm moves to another operation table, the binocular vision device is used to acquire the binocular vision image, and there is no need to install a binocular vision device for each operation station, which achieves the effect of saving system cost.

Among them, the binocular vision device can be installed on the claw of the robot operating arm, or it can be installed on a fixed position of the robot operating arm, as long as the image of the upper surface of the object to be measured and the image of the front of the operating table can be obtained can.

Claims (13)

A method for determining the spatial position and shape of an object, characterized in that: the binocular visual image of the item to be tested and the standard mark is acquired by a binocular visual device; and the binocular visual device is arranged above the item to be tested; According to the positional relationship between the standard mark and the object to be tested, and the binocular visual image of the standard mark, the binocular visual image of the object to be tested is corrected and fitted to obtain the depth of the object to be tested in the world coordinate system Image, based on the depth image of the object under test in the world coordinate system to determine the point cloud data image of the object under test; using the vertical spatial statistical method to determine the point cloud data of the surface of the object under test; according to the aforementioned object The point cloud data of the upper surface of the test object determines the center position of the upper surface of the test object as the position data of the test object; and according to the fitting surface of the point cloud data of the upper surface of the test object and the test object The distance between the center position of the upper surface of the upper surface and the boundary position of the point cloud data of the upper surface of the object to be measured determines the shape data of the object to be measured. The method for determining the spatial position and shape of an object as described in item 1 of the patent application scope, wherein after determining the position data of the object to be tested and the shape data of the object to be tested, the method further includes: The position data and the shape data of the object to be tested determine the grasping position and grasping posture of the robot operating arm to control the robot operating arm to grasp the object to be tested. The method for determining the spatial position and shape of an object as described in item 1 of the patent application scope, wherein, before the binocular visual images of the object to be tested and the standard mark are acquired by the binocular visual device, the method also includes: Select a fixed structure in the test object carrying space as the standard mark, or install a mark in the test object carrying space as the standard mark, established by the positional relationship between the binocular vision device and the standard mark The relationship between the coordinate system of the aforementioned standard mark and the coordinate system of the aforementioned binocular vision device. The method for determining the spatial position form of an object as described in item 1 of the scope of the patent application, wherein the foregoing is based on the positional relationship between the standard mark and the object to be tested, and the binocular visual image of the standard mark Correcting and fitting the binocular vision image to obtain the depth image of the object under test in the world coordinate system, including: using the aforementioned standard mark to determine the position of the binocular vision device; according to the position of the aforementioned binocular vision device and the aforementioned standard The positional relationship between the markers determines the world coordinate system correction parameters of the binocular visual image of the aforementioned test object; according to the world coordinate system correction parameters of the binocular visual image of the aforementioned test object, the The visual image is converted to the world coordinate system, and then the depth image is fitted to obtain the depth image of the aforementioned object under test in the world coordinate system. The method for determining the spatial position form of an object as described in item 1 of the scope of the patent application, wherein the foregoing is based on the positional relationship between the standard mark and the object to be tested, and the binocular visual image of the standard mark Correcting and fitting the binocular visual image to obtain the depth image of the test object in the world coordinate system, including: performing a depth map on the binocular visual image of the aforementioned standard mark and the binocular visual image of the aforementioned test object Image fitting to obtain the respective primary depth images; according to the positional relationship between the position of the binocular vision device and the standard mark, determine the world coordinate system correction parameters of the primary depth image of the test object; The world coordinate system correction parameters of the primary depth image of the object to be measured are corrected by the world coordinate system of the primary depth image of the object to be measured to obtain the depth image of the object to be tested under the world coordinate system. The method for determining the spatial position form of an object as described in item 1 of the patent application scope, wherein after determining the point cloud data image of the object under test based on the depth image of the object under test under the world coordinate system, the vertical Before determining the point cloud data of the upper surface of the object under test by a spatial statistical method, the method further includes: filtering the background point cloud data according to the three-color difference degree of the point cloud data in the point cloud data image of the object under test In addition, obtain the cloud data image of the front sight; correspondingly, use the vertical spatial statistical method to determine the point cloud data of the upper surface of the object to be measured, including: use the vertical space statistical method to determine from the foregoing cloud data image of the front sight Point cloud data on the upper surface of the aforementioned object to be measured. The method for determining the spatial position shape of an object as described in Item 6 of the patent application scope, in which the background point cloud data is filtered according to the three-color difference degree of the point cloud data in the point cloud data image of the object to be measured, Obtaining the cloud data image of the former scenic spot, including: using the following formula to determine the three-color difference value of the point cloud data in the aforementioned point cloud data image: T=|R _point -G _point |-|G _point -B _point |; where , R _point represents the red value in the RGB color in the point cloud data; G _point represents the green value in the RGB color in the point cloud data; B _point represents the blue value in the RGB color in the point cloud data; where, T Is a three-color difference degree value. When the aforementioned three-color difference degree value is less than the background filtering threshold, the corresponding point cloud data is determined to be background point cloud data, and the background point cloud data is filtered. The method for determining the spatial position and shape of an object as described in Item 6 of the patent application scope, wherein the vertical space statistical method is used to determine the point cloud data of the upper surface of the object to be measured from the cloud data image of the front sight, including: Perform statistical distribution on the vertical data of all front sight cloud data; determine the number of front sight cloud data in each vertical data interval in the statistical distribution; determine the median value of the vertical data interval with the largest number of front sight cloud data ; The point cloud data with vertical data distributed within a set range is used as the point cloud data of the upper surface of the object to be measured; the set range is obtained by subtracting the preset controllable value from the median value of the vertical data interval A numerical range formed by a first numerical value and a second numerical value obtained by adding a median value of the foregoing vertical data interval to a preset controllable numerical value. The method for determining the spatial position and shape of the object as described in item 8 of the patent application scope, wherein the aforementioned preset controllable value is determined in the following manner: the vertical data of all the cloud data images of the former scenic spot are counted to determine the standard variance; The set multiple of the standard deviation is used as the preset controllable value. The method for determining the spatial position form of an object as described in item 1 of the patent application scope, wherein the center position of the upper surface of the object to be measured is determined based on the point cloud data of the upper surface of the object to be measured as the position data of the object to be tested , Including: determining the center position of the upper surface of the above-mentioned object as the position data of the above-mentioned object according to the average value of the spatial coordinates of the point cloud data of all the above-mentioned objects under the measurement; according to the upper surface of the above-mentioned object The fitting surface of the point cloud data, and the distance between the center position of the upper surface of the object to be measured and the boundary position of the point cloud data of the upper surface of the object to be measured, determining the morphological data includes: according to the point cloud of the upper surface of the object to be measured The spatial coordinates of the data are plane-fitted to determine the upper surface of the object to be measured; the normal vector of the upper surface of the object to be measured is determined, and the aforementioned object is determined along the X axis in the world coordinate system according to the normal vector The torsion angle Rx and the torsion angle Ry along the Y axis; and, the point cloud data of the upper surface of the object to be tested is projected on the XOY plane; the projection position at the center of the upper surface and the boundary position of the upper surface point cloud data Determine the minimum value among the distances between the projection positions; determine the torsion angle Rz of the object under test along the Z axis according to the direction of the minimum value; determine the Rx, Ry, and Rz as the morphological data of the object under test. A device for determining the spatial position and shape of an object, characterized in that it includes: a binocular visual image acquisition module, which is configured to acquire a binocular visual image of an object to be tested and a standard mark through a binocular visual device; The visual device is arranged above the aforementioned object to be tested; the point cloud data image determination module is set to determine the aforementioned object under test based on the positional relationship between the aforementioned standard mark and the aforementioned object under test and the binocular visual image of the standard mark The binocular vision image is corrected and fitted to obtain the depth image of the object under test in the world coordinate system, and the point cloud data image of the object under test is determined according to the depth image of the object under test in the world coordinate system The upper surface point cloud data screening module is set to determine the upper surface point cloud data of the above-mentioned object to be measured using vertical spatial statistical methods; the position data and shape data determination module is set to be based on the upper surface point of the above-mentioned object to be measured The cloud data determines the center position of the upper surface of the test object as position data of the test object; and based on the fitting surface of the point cloud data of the upper surface of the test object and the center position of the upper surface of the test object and The distance of the boundary position of the point cloud data on the upper surface of the object to be tested determines the shape data of the object to be tested. A computer-readable storage medium characterized by a computer program stored on it, which when executed by a processor implements a method for determining the spatial position and shape of an object as described in any of items 1 to 10 of the patent application. A binocular vision robot, characterized in that it includes a binocular vision device, a standard mark, a robot operating arm, a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor executes the computer program Realize the method of determining the spatial position and shape of the object as described in any one of the items 1 to 10 of the patent application range.

Images