WO2020133407A1 - Structured-light-based locating method and apparatus for industrial robot, and controller and medium - Google Patents

Abstract

The present invention relates to a structured-light-based locating method and apparatus for an industrial robot, and a controller and a medium. The method comprises: acquiring two-dimensional image information of a pre-set area, and segmenting, from the two-dimensional image information of the pre-set area, a pixel area where a target object is located; according to a pre-set mapping relationship between two-dimensional image information and three-dimensional data, converting the pixel area into three-dimensional data corresponding to the target object; and according to the three-dimensional data of the target object, locating a pose of the target object, so as to capture the target object. According to the present invention, based on recognition and locating of a vision-guided industrial robot, a three-dimensional profile of a target object can be accurately and rapidly scanned in a complex environment, and by means of one-to-one mapping and joint analysis of three-dimensional data and data of a two-dimensional image, a pose of the target object is efficiently and accurately located so as to guide the industrial robot to capture the target object.

Description

Industrial robot positioning method and device, controller and medium based on structured light Technical field

The invention relates to the technical field of robot vision, in particular to a method, device, controller and medium for positioning an industrial robot based on structured light.

Background technique

With the rapid development of industrial automation, the application of industrial robots is becoming more and more common. For most industrial robot application scenarios in the prior art, manual teaching or offline programming is usually required to plan the working path of the robot in advance. The highly structured working mode strictly limits the flexibility and intelligence of industrial robots.

Existing industrial robot positioning methods can be divided into two categories according to the type of data: (1) Recognize the target by comparing with the template from the two-dimensional image, and then extract the three-dimensional data using the image area of the recognition target or obtain it through the distance sensor The target local plane calculates the pose of the target. This method depends heavily on the quality of the captured image, but the complex light changes in the industrial production environment, therefore, this method is difficult to adapt to actual production. (2) Compare directly with the CAD model from the 3D data, this method no longer depends on the quality of the acquired 2D image, but for the case of multiple workpieces superimposed together, it is easy to cause registration ambiguity, which affects the comparison with the template Right stability.

In summary, the existing industrial robot positioning method is difficult to use complex and changeable industrial application environment, and cannot meet the needs of flexible production.

Summary of the invention

The technical problem to be solved by the present invention is to provide an industrial robot positioning method and device, controller, and medium based on structured light. The industrial robot recognition and positioning based on visual guidance can achieve accurate and rapid three-dimensional contours of target objects in complex environments. Scanning and analyzing through the one-to-one mapping of three-dimensional data and two-dimensional image data to efficiently and accurately locate the posture of the target object and guide the industrial robot to grab the target object.

In order to solve the above technical problems, according to the first embodiment of the present invention, a method for positioning an industrial robot based on structured light is provided, including the following steps:

Acquiring the two-dimensional image information of the preset area, and segmenting the pixel area where the target object is located from the two-dimensional image information of the preset area;

Converting the pixel area into three-dimensional data corresponding to the target object according to the preset mapping relationship between the two-dimensional image information and the three-dimensional data;

The position and posture of the target object are located according to the three-dimensional data of the target object to grasp the target object.

Further, segmenting the pixel area where the target object is located from the two-dimensional image information of the preset area includes the following steps:

The two-dimensional image information is segmented by AI or image recognition, and the pixel area where the target object is located is segmented.

Further, the method further includes: constructing a mapping relationship between the preset two-dimensional image information and three-dimensional data, specifically including the following steps:

A structured light vision sensor is used to obtain two-dimensional image information and three-dimensional data of the target object, and a one-to-one mapping is formed on the two-dimensional image information and three-dimensional data, wherein the structured light vision sensor includes a light source and at least one camera.

Further, the acquiring the two-dimensional image information of the target object using the structured light vision sensor includes the following steps:

Project an image whose brightness exceeds the preset brightness to obtain the two-dimensional image information after lighting.

Further, the light source is a grating projection device, and the use of a structured light vision sensor to acquire three-dimensional data of a target object includes the following steps:

Calibrate the internal parameters of each camera and the external parameters between each camera and the raster projection device;

The raster projection device projects an image and generates a raster sinusoidal image;

The camera takes the projection image to form a camera image;

Acquiring the main phase value of the grating sine image;

Acquiring the raster sine image coordinates according to the main phase value and the camera image coordinates;

Correcting the camera image coordinates and the raster sine image coordinates;

The three-dimensional coordinates of the target object are obtained according to the corrected camera image coordinates, raster sine image coordinates, and the inner and outer parameters.

Further, the positioning of the target object according to the three-dimensional data of the target object to grab the target object includes the following steps:

Locate the posture of the target object according to the three-dimensional data of the target object and the preset CAD model of the target object;

Converting the posture of the target object to the robot coordinate system to grab the target object.

Further, the method further includes: in the process of grabbing the target object, determining whether the gripping jig interferes with the target object and the material frame where the target object is located, if there is interference, if there is interference, then restart Adjust the position of the fixture and then grab.

Further, determining whether the gripping jig interferes with the target object and the material frame where the target object is located includes the following steps:

Collect the stereoscopic image of the grabbing fixture;

Dividing the stereoscopic image of the gripping jig into a plurality of rectangular parallelepipeds, each of which is used to describe the structural characteristics of each part of the gripping jig;

Determine whether each cuboid has an intersection with each plane of the material frame; if so, there is interference between the cuboid and the material frame;

By analyzing the posture of the target object, it is determined whether each cuboid has an intersection with the target object, and if so, the cuboid interferes with the target object.

According to a second embodiment of the present invention, an industrial robot positioning device based on structured light is provided, including:

An obtaining module, configured to obtain the two-dimensional image information of the preset area, and segment the pixel area where the target object is located from the two-dimensional image information of the preset area;

A conversion module, configured to convert the pixel area into three-dimensional data corresponding to the target object according to a preset mapping relationship between two-dimensional image information and three-dimensional data;

The positioning module is configured to locate the posture of the target object according to the three-dimensional data of the target object to grab the target object.

Further, the acquisition module is also used to:

The two-dimensional image information is segmented by AI or image recognition, and the pixel area where the target object is located is segmented.

Further, the device further includes a construction module for constructing a mapping relationship between the preset two-dimensional image information and three-dimensional data;

The building block includes:

The first acquiring unit is used to acquire the two-dimensional image information of the target object using a structured light vision sensor;

The second acquisition unit is used to acquire three-dimensional data using a structured light vision sensor;

A mapping unit for forming a one-to-one mapping between the two-dimensional image information and the three-dimensional data;

Wherein, the structured light vision sensor includes a light source and at least one camera.

Further, the first obtaining unit is specifically configured to project an image with a brightness exceeding a preset brightness, and obtain two-dimensional image information after lighting.

Further, the light source is a grating projection device, and the second acquiring unit is specifically configured to acquire the three-dimensional data of the target object using a structured light vision sensor:

The second obtaining unit includes:

A stator unit for calibrating the internal parameters of each camera and the external parameters between each camera and the raster projection device;

A first image acquisition subunit, configured to project an image through the raster projection device and generate a raster sine image;

A second image acquisition subunit, configured to capture the projected image by the camera to form a camera image;

A main phase value acquisition subunit, used to obtain the main phase value of the grating sinusoidal image;

A first coordinate obtaining subunit, configured to obtain the raster sine image coordinates according to the phase main value and the camera image coordinates;

A correction subunit for correcting the camera image coordinates and the raster sine image coordinates;

The second coordinate acquisition subunit is configured to acquire the three-dimensional coordinates of the target object according to the corrected camera image coordinates, raster sine image coordinates, and the inner and outer parameters.

Further, the positioning module includes:

A pose acquisition unit for positioning the pose of the target object according to the three-dimensional data of the target object and the preset CAD model of the target object;

The coordinate conversion unit is used to convert the posture of the target object to the manipulator coordinate system to grab the target object.

Further, the device further includes a detection module for determining whether the gripping jig interferes with the target object and the material frame where the target object is located during the process of grabbing the target object, if interference occurs, if it occurs If it interferes, readjust the position of the fixture and then grab.

Further, the detection module includes:

An image collection unit, used to collect the stereoscopic image of the grabbing fixture;

A dividing unit, configured to divide the stereoscopic image of the gripping jig into a plurality of rectangular parallelepipeds, each of which is used to describe the structural characteristics of each part of the gripping jig;

The first judging unit is used to judge whether each cuboid has an intersection with each plane of the material frame, and if so, there is interference between the cuboid and the material frame;

The second determining unit is configured to determine whether each cuboid has an intersection with the target object by analyzing the posture of the target object, and if there is, the cuboid interferes with the target object.

According to a third embodiment of the present invention, a controller is provided, which includes a memory and a processor, the memory stores a computer program, and the program can implement the steps of the method when executed by the processor.

According to a fourth embodiment of the present invention, a computer-readable storage medium is provided for storing a computer program, which when executed by a computer or processor implements the steps of the method.

Compared with the prior art, the invention has obvious advantages and beneficial effects. With the above technical solutions, the present invention provides a method, device, controller, and medium for positioning industrial robots based on structured light, which can achieve considerable technological advancement and practicality, and has wide industrial use value, which has at least the following advantages :

The invention is based on visually guided industrial robot identification and positioning, which can realize accurate and rapid scanning of the three-dimensional contour of the target object in a complex environment, and through the one-to-one mapping and analysis of three-dimensional data and two-dimensional image data, thereby efficiently and accurately locating the The posture of the target object guides the industrial robot to grab the target object, adapts to the complex and changing industrial application environment, and thereby improves the production efficiency of industrial assembly and loading and unloading in a complex environment; in addition, in the process of grabbing the target object, through It is analyzed whether the gripping jig interferes with the target object and the material frame where the target object is located, and makes corresponding adjustments, which satisfies the more flexible and intelligent industrial production using industrial robots.

The above description is only an overview of the technical solutions of the present invention. In order to understand the technical means of the present invention more clearly, it can be implemented in accordance with the content of the specification, and in order to make the above and other objects, features and advantages of the present invention more obvious and understandable In the following, the preferred embodiments are described in detail in conjunction with the drawings, which are described in detail below.

BRIEF DESCRIPTION

FIG. 1 is a schematic diagram of an industrial robot positioning method based on structured light provided by an embodiment of the present invention;

2 is a schematic diagram of two camera space coordinate systems according to an embodiment of the invention;

3 is a schematic diagram of an industrial robot positioning device based on structured light provided by an embodiment of the present invention.

【Symbol Description】

1: Get the module 2: Convert the module

3: Positioning module

detailed description

In order to further illustrate the technical means and effects adopted by the present invention to achieve the intended purpose of the invention, in the following, in conjunction with the drawings and preferred embodiments, a method, device and controller for positioning an industrial robot based on structured light according to the present invention The specific implementation of the medium and its efficacy will be described in detail later.

An embodiment of the present invention provides an industrial robot positioning method based on structured light, as shown in FIG. 1, including the following steps:

Step S1: Obtain the two-dimensional image information of the preset area, and segment the pixel area where the target object is located from the two-dimensional image information of the preset area;

As an example, the step S1 further includes the following steps:

By AI (Artificial Intelligence) segmentation or image recognition of the two-dimensional image information, the pixel area where the target object is located is divided. It should be noted that AI segmentation or image recognition technology is an existing image processing technology. This will not be repeated here. The two-dimensional image information may include one or more target objects, and the pixel region corresponding to each target object may be separated by AI segmentation or image recognition technology. In industrial applications, as an example, the preset area is an area in the material frame, the target object is an object to be grasped in the material frame, the object in the material frame may be referred to as a workpiece, and the target object is the target workpiece.

Step S2: According to the preset mapping relationship between the two-dimensional image information and the three-dimensional data, the pixel area is converted into three-dimensional data AI segmentation or image recognition corresponding to the target object, and each target object can be mapped;

The method further includes step S20, constructing a mapping relationship between the preset two-dimensional image information and three-dimensional data, which specifically includes: acquiring the two-dimensional image information and three-dimensional data of the target object using a structured light vision sensor, and The two-dimensional image information and the three-dimensional data form a one-to-one mapping, wherein the structured light vision sensor includes a light source and at least one camera. As an example, the light source is a raster projection device. The following embodiment uses a raster projection device as an example The description will be made, but it can be understood that other light sources that can be applied to the embodiment of the present invention to obtain two-dimensional image information and three-dimensional data may also be suitable for this. The use of multiple cameras and the combination of the grating projection device can reduce the blind area caused by the limitation of the installation angle of each camera and the grating projection device, and improve the positioning accuracy. In the embodiment of the present invention, the structured light vision sensor includes two cameras as an example Be explained.

Step S20. A structured light vision sensor is used to obtain the two-dimensional image information and three-dimensional data of the target at the same time, and form a one-to-one mapping between the two, so that the two-dimensional image information of the target object can be used when positioning the target object. It can use three-dimensional data to adapt to complex environments and improve the robustness of target object positioning, making industrial robot positioning more flexible and intelligent.

In step S20, the acquiring the two-dimensional image information of the target object using the structured light vision sensor includes the following steps:

Step S200. Project an image with a brightness exceeding the preset brightness to obtain the two-dimensional image information after lighting. Using binocular vision and grating structured light, you can use your own lighting to shoot high-quality two-dimensional images. The preset brightness is set according to projection requirements. As an example, the image exceeding the preset brightness may be a white image, a blue image, or the like.

In step S20, the acquiring the three-dimensional data of the target object using the structured light vision sensor includes the following steps:

Step S201: calibrate the internal parameters of each camera and the external parameters between each camera and the raster projection device;

Step S202, the raster projection device projects an image and generates a raster sine image;

The gray distribution of the raster sine image is:

I(u,v)=a+b*cos(θ(u,v)), θ(u,v)=φ(u,v)+α

Among them, (u, v) represents the coordinates of the projected pixel unit on the projection surface, I (u, v) is the gray value of the (u, v) point, and a and b are the DC fundamental wave component of the sinusoidal grating (light intensity background) Value) and amplitude (modulated light intensity), θ(u,v) is the grating phase corresponding to I(u,v), φ is the main value of the phase to be sought, and α is the phase shift.

Step S203, the camera takes the projected image to form a camera image;

Step S204: Obtain the main phase value of the grating sine image;

The main phase value is the relative phase value. As an example, the standard four-step phase shift method can be used to calculate the main phase value of the raster image, but it can be understood that other methods for calculating the main phase value can also be applied here.

A phase main value image is calculated by using four raster images with the same frequency. The light intensity expression of the four raster images is:

I _i (u,v)=a+b*cos(θ _i (u,v)),θ _i (u,v)=φ(u,v)+π/2*i,i∈{0,1 ,2,3}

The main phase value of the raster image is:

φ(u,v)=atan2(I ₃ -I ₁ ,I ₀ -I ₂ )

Step S205: Acquire the raster sine image coordinates according to the phase main value and the camera image coordinates;

According to the absolute phase value in one direction, a vertical line or a horizontal line in the projected image can be determined; then, according to the absolute phase values in the horizontal and vertical directions, a point in the projected image can be determined.

Through the points in the camera image of vertical and horizontal raster stripes, the point in the camera image is set to (uc, vc), you can determine a corresponding point in the projected image, that is, the coordinate point of the raster sine image, the coordinate point of the raster sine image is set For (up, vp), the coordinates of this point can be calculated with the following formula.

Where Φv and Φh are the absolute phase values in the vertical and horizontal directions at the (uc, vc) point in the raster image, Nv and Nh are the number of raster stripes in the raster image of the vertical and horizontal cameras, and H and W are the cameras, respectively Raster image vertical and horizontal resolution.

Step S206, correcting the camera image coordinates and the raster sine image coordinates;

As an example, (uc, vc) and (up, vp) are corrected using pre-calibrated system distortion parameters. The second-order radial distortion correction formula is as follows:

_{^{u - = u + (uu 0}} ) [k 1 (x 2 + y 2) + k 2 (x 2 + y 2) 2]

_{^{v - = v + (vv 0}} ) [k 1 (x 2 + y 2) + k 2 (x 2 + y 2) 2]

Step S207: Acquire the three-dimensional coordinates of the target object according to the corrected camera image coordinates, raster sine image coordinates, and the inner and outer parameters.

According to the corrected camera image coordinates (uc, vc) and raster sine image coordinates (up, vp), and the internal reference book and external reference book calibrated by the camera in advance, a line between the camera's optical center and the image image point is calculated, Then use the above information to calculate the corresponding three-dimensional coordinates through the principle of triangulation.

Among them, Sc and Sp are the scale factors of the camera and the optical machine, Ac and Ap are internal parameter matrices, [Rc tc] and [Rp tp] are external parameter matrices, and Xw, Yw, Zw, 1 are space vectors.

The following is a detailed description of the corresponding three-dimensional coordinates calculated by the principle of triangulation:

In Figure 2, O1-xyz and O2-xyz are the two-camera space coordinate system; P1, P2 are a pair of points with the same name; S1, S2 are the center position of the camera lens; w is a point in real space. P1, S1 determine a straight line in space, P2, S2 determine another straight line, they intersect at W in space.

Spatial straight line: After the camera takes an image, an image point on the camera CCD and the center of the camera lens can determine a straight line, as shown in Figure 2. The coordinates of the two points of the image point and the center of the lens are in the camera coordinate system. The linear equation of space formed by these two points is:

Among them, X, Y, Z are the three-dimensional coordinates of the target point, which is an unknown number; x, y, f are the image point coordinates, which are known quantities (obtained by analyzing the image); Xs, Ys, Zs are the lens center coordinates, which are Known quantities (obtained during the camera calibration process); a _i , b _i , c _i (i=1, 2, 3) are coordinate system transformation parameters, which are known quantities (obtained during the camera calibration process). An image can list a straight line equation, two images can list two straight line equations, a total of 4 equations, and the unknown number in the formula is only three (three-dimensional point coordinates X, Y, Z), therefore, can be calculated three Unknowns.

Step S3: Position the posture of the target object according to the three-dimensional data of the target object to grab the target object.

As an example, the step S3 includes the following steps:

Step S31: Position the posture of the target object according to the three-dimensional data of the target object and the preset CAD model of the target object;

Each target object has its corresponding CAD model, which is set in advance. By comparing the three-dimensional data of the target object with its corresponding CAD model, the posture of the target object can be located.

Step S32: Convert the position and posture of the target object to the robot coordinate system to grab the target object.

The system corresponding to the method may include an industrial robot, a host computer, and a three-dimensional vision sensor, where the three-dimensional vision sensor is a structured light vision sensor. Compared with existing methods that rely solely on two-dimensional images and three-dimensional data, the method described in the embodiments of the present invention can not only solve the problem that it is difficult to calculate the three-dimensional coordinates of the target object solely on the basis of two-dimensional image analysis, but also avoid the difficulty of relying solely on three-dimensional data. Effectively deal with the segmentation problem when the target objects are closely arranged. The method can achieve accurate and rapid scanning of the three-dimensional contour of the target object in a complex environment by performing steps S1-step S3, and through the mapping relationship of the two-dimensional image of the three-dimensional data, efficiently and accurately locate the posture of the target object, thereby guiding Industrial robots grab workpieces.

In the application of structured light guided industrial robot recognition and positioning, the analysis of the interference between the fixture and the frame and the fixture and the workpiece during the grasping of the workpiece is an important research content and a problem to be solved, because the posture of the workpiece in the frame The diversity of fixtures causes irregularities in fixture design. Based on this, the method further includes step S4. During the process of grasping the target object, it is determined whether the gripping jig interferes with the target object and the material frame where the target object is located. If interference occurs, if interference occurs, Then readjust the position of the fixture and then grab.

In step S4, determining whether the gripping jig interferes with the target object and the material frame where the target object is located includes the following steps:

Step S41: Collect the stereoscopic image of the grabbing fixture;

Step S42: Divide the stereoscopic image of the gripping jig into a plurality of rectangular parallelepipeds, each of which is used to describe the structural characteristics of each part of the gripping jig;

Step S43: Determine whether each cuboid has an intersection with each plane of the material frame. If so, there is interference between the cuboid and the material frame;

Step S44: By analyzing the posture of the target object, it is determined whether each cuboid has an intersection with the target object, and if so, the cuboid interferes with the target object.

Through step S4, the industrial robot can not only avoid the interference between the fixture and the material frame and the workpiece during the process of grasping the workpiece, but also ensure that the workpiece is grasped to the maximum extent.

An embodiment of the present invention also provides an industrial robot positioning device based on structured light. As shown in FIG. 3, it includes an acquisition module 1, a conversion module 2, and a positioning module 3. The acquisition module 1 is used to acquire two of the preset area. Dimensional image information, and segment the pixel area where the target object is located from the two-dimensional image information of the preset area; the conversion module 2 is used to convert all the two-dimensional image information and three-dimensional data according to the preset mapping relationship The pixel area is converted into three-dimensional data corresponding to the target object; the positioning module 3 is used to locate the posture of the target object according to the three-dimensional data of the target object to grab the target object. Compared with existing methods that rely solely on two-dimensional images and three-dimensional data, the device described in the embodiments of the present invention can not only solve the problem that it is difficult to calculate the three-dimensional coordinates of the target object solely on the basis of two-dimensional image analysis, but also avoid the difficulty of relying solely on three-dimensional data. Effectively deal with the segmentation problem when the target objects are closely arranged. The device can realize accurate and rapid scanning of the three-dimensional contour of the target object in a complex environment, and through the mapping relationship of the two-dimensional image of the three-dimensional data, locate the posture of the target object efficiently and accurately, thereby guiding the industrial robot to grab the workpiece.

As an example, the acquisition module 1 is also used to segment the pixel area where the target object is located by AI segmentation or image recognition of the two-dimensional image information. It should be noted that the AI segmentation or image recognition technology is an existing image processing technology, which will not be repeated here. The two-dimensional image information may include one or more target objects, and the pixel area corresponding to each target object may be separated by AI segmentation or image recognition technology. In industrial applications, as an example, the preset area is an area in the material frame, the target object is an object to be grasped in the material frame, the object in the material frame may be called a workpiece, and the target object is the target workpiece.

The device further includes a construction module for constructing a mapping relationship between the preset two-dimensional image information and three-dimensional data.

The construction module includes a first acquisition unit, a second acquisition unit, and a mapping unit, wherein the first acquisition unit is used to acquire the two-dimensional image information of the target object using the structured light vision sensor; the second acquisition unit is used to adopt the structured light vision The sensor acquires three-dimensional data; the mapping unit is used to form a one-to-one mapping between the two-dimensional image information and the three-dimensional data, so that both the two-dimensional image information of the target object and the three-dimensional data can be used when locating the target object, thereby adapting Complex environment, and improve the robustness of target object positioning, making industrial robot positioning more flexible and intelligent. The structured light vision sensor includes a light source and at least one camera. As an example, the light source is a grating projection device. The following embodiment uses a grating projection device as an example for description, but it can be understood that other applications The light source for obtaining two-dimensional image information and three-dimensional data according to the embodiment of the present invention may also be suitable for this. The use of multiple cameras and the combination of the grating projection device can reduce the blind area caused by the limitation of the installation angle of each camera and the grating projection device, and improve the positioning accuracy. In the embodiment of the present invention, the structured light vision sensor includes two cameras as an example Be explained.

As an example, the first acquiring unit is specifically used to project an image with a brightness exceeding a preset brightness, acquire two-dimensional image information after lighting, and use binocular vision in conjunction with grating structured light to use its own lighting to shoot high The quality of the two-dimensional image, wherein the preset brightness is set according to projection requirements. As an example, the image exceeding the preset brightness may be a white image, a blue image, or the like. The second acquisition unit is specifically used to acquire the three-dimensional data of the target object using a structured light vision sensor: the second acquisition unit includes a standard stator unit, a first image acquisition subunit, a second image acquisition subunit, and a phase principal value acquisition subunit , A first coordinate acquisition subunit, a correction subunit, and a second coordinate acquisition subunit, where,

A stator unit for calibrating the internal parameters of each camera and the external parameters between each camera and the raster projection device;

A first image acquisition subunit, configured to project an image through the raster projection device and generate a raster sine image;

The gray distribution of the raster sine image is:

I(u,v)=a+b*cos(θ(u,v)), θ(u,v)=φ(u,v)+α

Among them, (u, v) represents the coordinates of the projected pixel unit on the projection surface, I (u, v) is the gray value of the (u, v) point, and a and b are the DC fundamental wave component of the sinusoidal grating (light intensity background) Value) and amplitude (modulated light intensity), θ(u,v) is the grating phase corresponding to I(u,v), φ is the main value of the phase to be sought, and α is the phase shift.

A second image acquisition subunit, configured to capture the projected image by the camera to form a camera image;

A main phase value acquisition subunit, used to obtain the main phase value of the grating sinusoidal image;

A phase main value image is calculated by using four raster images with the same frequency. The light intensity expression of the four raster images is:

I _i (u,v)=a+b*cos(θ _i (u,v)),θ _i (u,v)=φ(u,v)+π/2*i,i∈{0,1 ,2,3}

The main phase value of the raster image is:

φ(u,v)=atan2(I ₃ -I ₁ ,I ₀ -I ₂ )

A first coordinate obtaining subunit, configured to obtain the coordinate of the raster sine image according to the phase main value and the camera image coordinate;

According to the absolute phase value in one direction, a vertical line or a horizontal line in the projected image can be determined; then, according to the absolute phase values in the horizontal and vertical directions, a point in the projected image can be determined.

Through the points in the camera image of vertical and horizontal raster stripes, the point in the camera image is set to (uc, vc), you can determine a corresponding point in the projected image, that is, the coordinate point of the raster sine image, the coordinate point of the raster sine image is set For (up, vp), the coordinates of this point can be calculated with the following formula.

Where Φv and Φh are the absolute phase values in the vertical and horizontal directions at the (uc, vc) point in the raster image, Nv and Nh are the number of raster stripes in the raster image of the vertical and horizontal cameras, and H and W are the cameras, respectively Raster image vertical and horizontal resolution.

A correction subunit for correcting the camera image coordinates and the raster sine image coordinates;

As an example, (uc, vc) and (up, vp) are corrected using pre-calibrated system distortion parameters. The second-order radial distortion correction formula is as follows:

_{^{u - = u + (uu 0}} ) [k 1 (x 2 + y 2) + k 2 (x 2 + y 2) 2]

_{^{v - = v + (vv 0}} ) [k 1 (x 2 + y 2) + k 2 (x 2 + y 2) 2]

The second coordinate acquisition subunit is configured to acquire the three-dimensional coordinates of the target object according to the corrected camera image coordinates, raster sine image coordinates, and the inner and outer parameters.

According to the corrected camera image coordinates (uc, vc) and raster sine image coordinates (up, vp), and the internal reference book and external reference book calibrated by the camera in advance, a line between the camera's optical center and the image image point is calculated, Then use the above information to calculate the corresponding three-dimensional coordinates through the principle of triangulation.

Among them, Sc and Sp are the scale factors of the camera and the optical machine, Ac and Ap are internal parameter matrices, [Rc tc] and [Rp tp] are external parameter matrices, and Xw, Yw, Zw, 1 are space vectors.

The following is a detailed description of the corresponding three-dimensional coordinates calculated by the principle of triangulation:

In Figure 2, O1-xyz and O2-xyz are the two-camera space coordinate system; P1, P2 are a pair of points with the same name; S1, S2 are the center position of the camera lens; w is a point in real space. P1, S1 determine a straight line in space, P2, S2 determine another straight line, they intersect at W in space.

Spatial straight line: After the camera takes an image, an image point on the camera CCD and the center of the camera lens can determine a straight line, as shown in Figure 2. The coordinates of the two points of the image point and the center of the lens are in the camera coordinate system. The linear equation of space formed by these two points is:

Among them, X, Y, Z are the three-dimensional coordinates of the target point, which is an unknown number; x, y, f are the image point coordinates, which are known quantities (obtained by analyzing the image); Xs, Ys, Zs are the lens center coordinates, which are Known quantities (obtained during the camera calibration process); a _i , b _i , c _i (i=1, 2, 3) are coordinate system transformation parameters, which are known quantities (obtained during the camera calibration process). An image can list a straight line equation, two images can list two straight line equations, a total of 4 equations, and the unknown number in the formula is only three (three-dimensional point coordinates X, Y, Z), therefore, can be calculated three Unknowns.

As an example, the positioning module 3 includes a pose acquisition unit and a coordinate conversion unit, wherein the pose acquisition unit is used to locate the target object according to the three-dimensional data of the target object and a preset CAD model of the target object Pose; the coordinate conversion unit is used to convert the posture of the target object to the manipulator coordinate system to grab the target object. Each target object has its corresponding CAD model, which is set in advance. By comparing the three-dimensional data of the target object with its corresponding CAD model, the posture of the target object can be located.

In the application of structured light guided industrial robot recognition and positioning, the analysis of the interference between the fixture and the frame and the fixture and the workpiece during the grasping of the workpiece is an important research content and a problem to be solved, because the posture of the workpiece in the frame The diversity of fixtures causes irregularities in fixture design. Based on this, the device further includes a detection module for judging whether the gripping jig interferes with the target object and the material frame where the target object is located during the process of grabbing the target object, if interference occurs, if it occurs If it interferes, readjust the position of the fixture and then grab.

The detection module includes an image acquisition unit, a division unit, a first judgment unit and a second judgment unit, among them. An image acquisition unit is used to acquire the stereoscopic image of the gripping jig; a dividing unit is used to divide the stereoscopic image of the gripping jig into a plurality of rectangular parallelepipeds, each of which is used to describe the structural characteristics of each part of the gripping jig The first judgment unit is used to judge whether each of the cuboid has an intersection with each plane of the material frame. If there is, the cuboid interferes with the material frame; the second judgment unit is used to analyze the posture of the target object To determine whether each cuboid has an intersection with the target object, and if so, there is interference between the cuboid and the target object. Through the detection module, the industrial robot can avoid the interference between the fixture and the material frame and the workpiece during the process of grasping the workpiece, and also ensure that the workpiece is grasped to the maximum extent.

The method and device of the present invention are based on visually guided industrial robot recognition and positioning, which can achieve accurate and rapid scanning of the three-dimensional contour of the target object in a complex environment, and through one-to-one mapping and analysis of three-dimensional data and two-dimensional image data, thus efficient and accurate Locate the position of the target object, guide the industrial robot to grab the target object, adapt to the complex and changing industrial application environment, and then improve the production efficiency of industrial assembly and loading and unloading in a complex environment; in addition, grab the target In the process of objects, by analyzing whether the gripping jig interferes with the target object and the material frame where the target object is located, and make corresponding adjustments, it is more flexible and intelligent for industrial production using industrial robots.

An embodiment of the present invention also provides a controller, which includes a memory and a processor, where the memory stores a computer program, and the program, when executed by the processor, can implement the structured light-based industrial robot positioning method step.

An embodiment of the present invention also provides a computer-readable storage medium for storing a computer program, which when executed by a computer or processor implements the steps of the structured light-based industrial robot positioning method.

The above is only the preferred embodiment of the present invention, and does not limit the present invention in any form. Although the present invention has been disclosed in the preferred embodiment as above, it is not intended to limit the present invention. Anyone familiar with the profession Technical personnel, without departing from the scope of the technical solution of the present invention, when the technical contents disclosed above can be used to make some modifications or modifications to equivalent embodiments of equivalent changes, but any content that does not depart from the technical solution of the present invention, according to the present invention Any simple modifications, equivalent changes, and modifications made to the above embodiments by the technical essence of the above still fall within the scope of the technical solution of the present invention.

Claims (18)

1. An industrial robot positioning method based on structured light, characterized by comprising the following steps:

   Acquiring the two-dimensional image information of the preset area, and segmenting the pixel area where the target object is located from the two-dimensional image information of the preset area;

   Converting the pixel area into three-dimensional data corresponding to the target object according to the preset mapping relationship between the two-dimensional image information and the three-dimensional data;

   The position and posture of the target object are located according to the three-dimensional data of the target object to grasp the target object.
2. The method for positioning an industrial robot based on structured light according to claim 1, wherein:

   Segmenting the pixel area where the target object is located from the two-dimensional image information of the preset area includes the following steps:

   The two-dimensional image information is segmented by AI or image recognition, and the pixel area where the target object is located is segmented.
3. The method for positioning an industrial robot based on structured light according to claim 1, wherein:

   The method further includes: constructing a mapping relationship between the preset two-dimensional image information and three-dimensional data, specifically including the following steps:

   A structured light vision sensor is used to obtain two-dimensional image information and three-dimensional data of the target object, and a one-to-one mapping is formed on the two-dimensional image information and three-dimensional data, wherein the structured light vision sensor includes a light source and at least one camera.
4. The method for positioning an industrial robot based on structured light according to claim 3, wherein

   The use of a structured light vision sensor to obtain two-dimensional image information of a target object includes the following steps:

   Project an image whose brightness exceeds the preset brightness to obtain the two-dimensional image information after lighting.
5. The method for positioning an industrial robot based on structured light according to claim 3, wherein

   The light source is a grating projection device. The structured light vision sensor for acquiring three-dimensional data of the target object includes the following steps:

   Calibrate the internal parameters of each camera and the external parameters between each camera and the raster projection device;

   The raster projection device projects an image and generates a raster sinusoidal image;

   The camera takes the projection image to form a camera image;

   Acquiring the main phase value of the grating sine image;

   Acquiring the raster sine image coordinates according to the main phase value and the camera image coordinates;

   Correcting the camera image coordinates and the raster sine image coordinates;

   The three-dimensional coordinates of the target object are obtained according to the corrected camera image coordinates, raster sine image coordinates, and the inner and outer parameters.
6. The method for positioning an industrial robot based on structured light according to claim 1, wherein:

   The positioning the posture of the target object according to the three-dimensional data of the target object to grab the target object includes the following steps:

   Locate the posture of the target object according to the three-dimensional data of the target object and the preset CAD model of the target object;

   Converting the posture of the target object to the robot coordinate system to grab the target object.
7. The method for positioning an industrial robot based on structured light according to claim 1, wherein:

   The method further includes: in the process of grasping the target object, determining whether the gripping jig interferes with the target object and the material frame where the target object is located, if interference occurs, if interference occurs, readjusting the location of the jig Then grab the location.
8. The industrial robot positioning method based on structured light according to claim 7, characterized in that

   Determine whether the gripping jig interferes with the target object and the material frame where the target object is located, including the following steps:

   Collect the stereoscopic image of the grabbing fixture;

   Dividing the stereoscopic image of the gripping jig into a plurality of rectangular parallelepipeds, each of which is used to describe the structural characteristics of each part of the gripping jig;

   Determine whether each cuboid has an intersection with each plane of the material frame; if so, there is interference between the cuboid and the material frame;

   By analyzing the posture of the target object, it is determined whether each cuboid has an intersection with the target object, and if so, the cuboid interferes with the target object.
9. An industrial robot positioning device based on structured light, characterized in that it includes:

   An obtaining module, configured to obtain the two-dimensional image information of the preset area, and segment the pixel area where the target object is located from the two-dimensional image information of the preset area;

   A conversion module, configured to convert the pixel area into three-dimensional data corresponding to the target object according to a preset mapping relationship between two-dimensional image information and three-dimensional data;

   The positioning module is configured to locate the posture of the target object according to the three-dimensional data of the target object to grab the target object.
10. The structured light-based industrial robot positioning device according to claim 9, characterized in that:

    The acquisition module is also used to:

    The two-dimensional image information is segmented by AI or image recognition, and the pixel area where the target object is located is segmented.
11. The industrial robot positioning device based on structured light according to claim 9, characterized in that

    The device further includes a construction module for constructing a mapping relationship between the preset two-dimensional image information and three-dimensional data;

    The building block includes:

    The first acquiring unit is used to acquire the two-dimensional image information of the target object using a structured light vision sensor;

    The second acquisition unit is used to acquire three-dimensional data using a structured light vision sensor;

    A mapping unit for forming a one-to-one mapping between the two-dimensional image information and the three-dimensional data;

    Wherein, the structured light vision sensor includes a light source and at least one camera.
12. The industrial robot positioning device based on structured light according to claim 11, characterized in that

    The first obtaining unit is specifically configured to project an image with a brightness exceeding a preset brightness, and obtain two-dimensional image information after lighting.
13. The industrial robot positioning device based on structured light according to claim 11, characterized in that

    The light source is a grating projection device, and the second acquiring unit is specifically configured to acquire the three-dimensional data of the target object using a structured light vision sensor:

    The second obtaining unit includes:

    A stator unit for calibrating the internal parameters of each camera and the external parameters between each camera and the raster projection device;

    A first image acquisition subunit, configured to project an image through the raster projection device and generate a raster sine image;

    A second image acquisition subunit, configured to capture the projected image by the camera to form a camera image;

    A main phase value acquisition subunit, used to obtain the main phase value of the grating sinusoidal image;

    A first coordinate obtaining subunit, configured to obtain the coordinate of the raster sine image according to the phase main value and the camera image coordinate;

    A correction subunit for correcting the camera image coordinates and the raster sine image coordinates;

    The second coordinate acquisition subunit is configured to acquire the three-dimensional coordinates of the target object according to the corrected camera image coordinates, raster sine image coordinates, and the inner and outer parameters.
14. The structured light-based industrial robot positioning device according to claim 9, characterized in that:

    The positioning module includes:

    A pose acquisition unit for positioning the pose of the target object according to the three-dimensional data of the target object and the preset CAD model of the target object;

    The coordinate conversion unit is used to convert the posture of the target object to the manipulator coordinate system to grab the target object.
15. The industrial robot positioning device based on structured light according to claim 9, characterized in that

    The device further includes a detection module for determining whether the gripping jig interferes with the target object and the material frame where the target object is located during the process of grabbing the target object, if interference occurs, if interference occurs, then Re-adjust the position of the jig and then grab.
16. The industrial robot positioning device based on structured light according to claim 15, characterized in that

    The detection module includes:

    An image collection unit, used to collect the stereoscopic image of the grabbing fixture;

    A dividing unit, configured to divide the stereoscopic image of the gripping jig into a plurality of rectangular parallelepipeds, each of which is used to describe the structural characteristics of each part of the gripping jig;

    The first judging unit is used to judge whether each cuboid has an intersection with each plane of the material frame, and if so, there is interference between the cuboid and the material frame;

    The second determining unit is configured to determine whether each cuboid has an intersection with the target object by analyzing the posture of the target object, and if there is, the cuboid interferes with the target object.
17. A controller, including a memory and a processor, characterized in that: the memory stores a computer program, and when the program is executed by the processor, the program can implement any one of claims 1 to 8. Steps of the method.
18. A computer-readable storage medium for storing a computer program, characterized in that the program, when executed by a computer or processor, implements the steps of the method according to any one of claims 1 to 8.

Images