TWI730575B - Computer device and method for determining depth standard value of marker - Google Patents

Abstract

The present invention provides a method for determining a depth standard value of a marker. The method includes: obtaining a maximum depth value; obtaining a depth reference value of the marker based on a depth image of the marker; obtaining a Z-axis coordinate value of the mark based on a color image of the marker; when the depth reference value of the marker and the Z-axis coordinate value are both smaller than the maximum depth value, and when a difference between the depth reference value of the marker and the Z-axis coordinate value is not greater than 0, the depth reference value is used as the depth standard value of the marker; and when the difference value is greater than 0, the Z-axis coordinate value is used as the depth standard value of the marker. The present invention also provides a computer device. The standard invention can quickly and accurately determine the depth standard value of the marker.

Description

Method and computer device for determining depth standard value of marker

The invention relates to the technical field of robot control, in particular to a method and a computer device for determining the depth standard value of a marker.

When factories use robotic arms to implement automated production, they usually find the gripping position of the target object based on the markers attached to the target object. Therefore, the accuracy of the depth value of the marker directly affects whether the robotic arm can successfully grasp the target object.

In view of the above, it is necessary to propose a method and computer device for determining the depth standard value of the marker, which can quickly and accurately determine the depth value of the marker without changing the existing fixture and environment, so that the robotic arm can be based on The determined depth value of the marker is used to find the gripping position of the target object.

The first aspect of the present invention provides a method for determining the standard value of the depth of a marker, which is applied to a computer device. The method includes: calculating the pressure of the clamp in the Z-axis direction detected by the force sensor of the robot arm. Maximum depth upper limit; controlling the camera of the robotic arm to take photos of the marker, and obtain a depth image and a color image containing the marker; Obtain the depth values of the four corners of the marker from the depth image of the marker; calculate the depth reference value of the marker based on the depth values of the four corners of the marker; based on the color map Obtain the Z-axis coordinate value of the marker; when both the depth reference value of the marker and the Z-axis coordinate value of the marker are less than the maximum depth upper limit, the depth of the marker is calculated The difference between the reference value and the Z-axis coordinate value of the marker; when the difference is less than or equal to 0, the depth reference value is used as the depth standard value of the marker; and when the difference is When the value is greater than 0, the Z-axis coordinate value is used as the depth standard value of the marker.

Preferably, the method further includes: when any one of the depth reference value of the marker and the Z-axis coordinate value of the marker is greater than or equal to the maximum depth upper limit, according to the force of the robot arm The pressure of the clamp in the Z-axis direction detected by the sensor recalculates the upper limit of the maximum depth.

Preferably, the calculation of a maximum depth upper limit value based on the pressure of the clamp in the Z-axis direction detected by the force sensor of the robot arm includes: setting the clamp of the robot arm at one of the distance bearing platforms The position of the preset distance; control the robot arm to drive the clamp to move down at a constant speed, and use the force sensor to detect the pressure of the clamp in the Z-axis direction; when the force sensor detects When the pressure of the clamp in the Z-axis direction determines that the change in the pressure of the clamp in the Z-axis direction is greater than the preset value, read the Z-axis coordinate value of the robot arm; when N numbers are read When the Z-axis coordinate value of the robot arm is used, the maximum depth upper limit value is calculated based on the Z-axis coordinate value of the N robot arms according to a root-mean-square algorithm.

Preferably, the method is based on the depth of the four corners of the marker according to the root-mean-square algorithm. The degree value is calculated to obtain the depth reference value of the marker.

Preferably, the obtaining the coordinate value of the Z axis of the marker based on the color image includes: obtaining the coordinate value of the Z axis of the marker based on the color image by using an OPENCV function.

Preferably, the marker is located on the carrying platform and the four sides of the marker are parallel to the carrying platform.

A second aspect of the present invention provides a computer device, the computer device includes a storage and a processor, the storage is used to store a plurality of modules, the processor is used to execute the plurality of modules, wherein the multiple Each module includes: an execution module, which is used to calculate a maximum depth limit based on the pressure of the fixture in the Z-axis direction detected by the force sensor of the robotic arm; and a shooting module, which is used to control the The camera of the robotic arm takes photos of the marker to obtain a depth image and a color image containing the marker; the execution module is also used to obtain four images of the marker from the depth image of the marker. The depth value of each corner; the execution module is also used to calculate the depth reference value of the marker based on the depth values of the four corners of the marker; the execution module is also used to calculate the depth reference value of the marker based on the The color image obtains the Z-axis coordinate value of the marker; the execution module is also used when the depth reference value of the marker and the Z-axis coordinate value of the marker are both less than the maximum depth. When the limit value is set, the difference between the depth reference value of the marker and the Z-axis coordinate value of the marker is calculated; the execution module is also used to calculate the difference when the difference is less than or equal to 0 The depth reference value is used as the depth standard value of the marker; and the execution module is further configured to use the Z-axis coordinate value as the depth standard value of the marker when the difference is greater than 0 .

Preferably, the execution module is also used for: when the depth reference value of the marker and When any one of the Z-axis coordinate values of the marker is greater than or equal to the upper limit of the maximum depth, recalculate according to the pressure of the clamp in the Z-axis direction detected by the force sensor of the robot arm The upper limit of the maximum depth.

Preferably, the calculation of a maximum depth upper limit value based on the pressure of the clamp in the Z-axis direction detected by the force sensor of the robot arm includes: setting the clamp of the robot arm at one of the distance bearing platforms The position of the preset distance; control the robot arm to drive the clamp to move down at a constant speed, and use the force sensor to detect the pressure of the clamp in the Z-axis direction; when the force sensor detects When the pressure of the clamp in the Z-axis direction determines that the change in the pressure of the clamp in the Z-axis direction is greater than the preset value, read the Z-axis coordinate value of the robot arm; when N numbers are read When the Z-axis coordinate value of the robot arm is used, the maximum depth upper limit value is calculated based on the Z-axis coordinate value of the N robot arms according to a root-mean-square algorithm.

Preferably, the execution module calculates the depth reference value of the marker based on the depth values of the four corners of the marker according to a root-mean-square algorithm.

Preferably, the obtaining the coordinate value of the Z axis of the marker based on the color image includes: obtaining the coordinate value of the Z axis of the marker based on the color image by using an OPENCV function.

Preferably, the marker is located on the carrying platform and the four sides of the marker are parallel to the carrying platform.

The method and computer device for determining the standard value of the depth of the marker described in the embodiment of the present invention obtain an upper limit of the maximum depth by calculating according to the pressure of the clamp in the Z-axis direction detected by the force sensor of the robot arm Value; control the depth camera of the robotic arm to take photos of the marker to obtain a depth image and a color image containing the marker; obtain the four corners of the marker based on the depth image of the marker The depth value of the marker; the depth reference value of the marker is calculated based on the depth values of the four corners of the marker; the Z-axis coordinate value of the marker is obtained based on the color image; when the marker When the depth reference value of and the Z-axis coordinate value of the marker are both less than the maximum depth upper limit, the difference between the depth reference value of the marker and the Z-axis coordinate value of the marker is calculated When the difference is less than or equal to 0, the depth reference value is used as the depth standard value of the marker; and when the difference is greater than 0, the Z-axis coordinate value is used as the marker The standard depth value of the object can quickly and accurately determine the depth value of the marker without changing the existing fixture and environment, so that the robotic arm can find the target object according to the determined depth value of the marker. position.

30: Determine the system

301: Execution module

302: Shooting module

1: computer device

2: Robotic arm

3: External fixture

31: Fixture

4: Bearer platform

51: Storage

52: processor

21: Pedestal

22: Arm

23: Fixture

24: Force sensor

25: Camera

41: Marker

40: target object

D11, D12, D13, D14: the depth of the four corners of the marker

In order to more clearly describe the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the drawings in the following description are merely It is an embodiment of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on the provided drawings without creative work.

Fig. 1 is an application environment diagram of the method for determining the depth standard value of a marker provided by a preferred embodiment of the present invention.

Figure 2 shows the robotic arm and fixture.

Fig. 3 is a flowchart of a method for determining the depth standard value of a marker provided by a preferred embodiment of the present invention.

Fig. 4 is a functional module diagram of a determining system provided by a preferred embodiment of the present invention.

FIG. 5 is a structural diagram of a computer device provided by a preferred embodiment of the present invention.

The following specific embodiments will further illustrate the present invention in conjunction with the above-mentioned drawings.

In order to be able to understand the above objectives, features and advantages of the present invention more clearly, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other if there is no conflict.

In the following description, many specific details are explained in order to fully understand the present invention. The described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the invention The embodiments in and all other embodiments obtained by a person of ordinary skill in the art without creative work shall fall within the protection scope of the present invention.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments, and are not intended to limit the present invention.

Please refer to FIG. 1, in this embodiment, the method for determining the depth standard value of the marker can be applied to an application environment composed of a computer device 1 and a robot arm 2.

In this embodiment, the computer device 1 can establish a wireless communication connection with the robotic arm 2. For example, the computer device 1 may establish a communication connection with the robotic arm 2 through a wireless router (not shown in the figure).

Please also refer to FIG. 2 as shown. In this embodiment, the robotic arm 2 is placed on the carrying platform 4. The bearing platform 4 may be a movable platform. In this embodiment, the carrying platform 4 also carries an external jig 3 at the same time. The external jig 3 may also include a plurality of jigs 31.

In this embodiment, the robotic arm 2 includes a base 21, an arm 22, a clamp 23, a force-torque sensor (Force-Torque Sensor) 24, and a camera 25. The arm 22 is connected to the base 21, and the force sensor 24 is disposed between the arm 22 and the camera 25. The clamp 23 is provided at the front end of the arm 22. The camera 25 may be a depth camera.

Referring to FIG. 2, in this embodiment, the three-dimensional coordinate system where the robotic arm 2 is located can be based on the origin O of the base 21 of the robotic arm 2 and the bottom end of the base 21 is located The horizontal plane of is the XOY plane, and the vertical direction perpendicular to the XOY plane is the Z axis.

In this embodiment, the computer device 1 can control the robotic arm 2 to grab the target object 40 placed on the jig 31 and place the target object 40 on the carrier platform 4.

In this embodiment, the target object 40 can be any product such as a motherboard or other products.

In this embodiment, a marker 41 is provided on the target object 40. The marker 41 may be ArUco Markers.

It should be noted that ArUco marker is a binary square marker, which consists of a wide black border and an internal binary matrix, which determines the ID. The black border is conducive to the rapid detection of the image, the binary code can verify the ID, and allows the application of error detection and correction technology. The size of ArUco Markers determines the size of the internal matrix. For example, a 4cm * 4cm ArUco Markers consists of 16 bits.

In this embodiment, the size of the marker 41 may be 4cm*4cm. Of course, in other embodiments, the size of the marker 41 can also be other sizes. In this embodiment, the robotic arm 2 can find the gripping position of the target object 40 according to the marker 41. Therefore, the accuracy of the depth value of the marker 41 directly affects whether the robotic arm 2 can successfully grasp the target object 40.

In this embodiment, the carrying platform 4 is further provided with a marker 41, and four sides of the marker 41 are parallel to the carrying platform 4. The following will describe how to determine the depth value of the marker 41 in conjunction with FIG. 3. Here, in order to explain the present invention clearly and simply, the depth value of the marker 41 to be determined next is referred to as the depth standard value.

In this embodiment, the method for determining the depth standard value of the marker 41 can be applied to the computer device 1. For the computer device 1 that needs to determine the depth standard value of the marker, the present invention can be directly integrated on the computer device 1. The function provided by the method for determining the depth standard value of the marker 41 may be run on a computer device in the form of a software development kit (SDK).

As shown in FIG. 3, the method for determining the depth standard value of the marker specifically includes the following steps. According to different requirements, the order of the steps in the flowchart can be changed, and some steps can be omitted.

In step S1, the computer device 1 calculates a maximum depth upper limit Zmax based on the pressure of the clamp 23 in the Z-axis direction detected by the force sensor 24 of the robot arm 2. The computer device 1 establishes an XOY plane with the horizontal plane on which the carrying platform 4 carrying the robot arm 2 is located, and establishes the coordinate system O-XYZ with the Z axis as the direction perpendicular to the XOY plane.

Specifically, referring to FIG. 2, in this embodiment, the computer device 1 uses the bottom end of the base 21 of the robot arm 2 as the origin O, and the horizontal plane where the bottom end of the base 21 is located is the XOY plane. And the coordinate system O-XYZ is established with the Z axis as the Z-axis in the vertical upward direction perpendicular to the XOY plane.

In one embodiment, the maximum depth limit Zmax calculated according to the pressure of the clamp 23 in the Z-axis direction detected by the force sensor 24 of the robot arm 2 includes (a1)-(a6):

(a1) The clamp 23 of the robot arm 2 is set at a predetermined distance from the carrying platform 4.

In this embodiment, the preset distance may be 30 cm or 35 cm. The preset distance can be set according to actual application scenarios.

(a2) Control the robot arm 2 to drive the clamp 23 to move down at a uniform speed, and use the force sensor 24 of the robot arm 2 to detect the pressure of the clamp 23 in the Z-axis direction.

Specifically, the robot arm 2 can be controlled to drive the clamp 23 to move vertically downward at a constant speed.

(a3) According to the pressure of the clamp 23 in the Z-axis direction detected by the force sensor 24, determine whether the change in the pressure of the clamp 23 in the Z-axis direction is greater than a preset value (for example, 1 Newton) , 1.5 Newtons, or other values). When the change in the pressure of the clamp 23 in the Z-axis direction is greater than the preset value, (a4) is executed. When the change in the pressure of the clamp 23 in the Z-axis direction is less than or equal to the preset value, continue to perform (a3).

In one embodiment, the change in the pressure of the clamp 23 in the Z-axis direction may refer to the difference between the first pressure value and the second pressure value, where the first pressure value refers to the mechanical When the arm 2 drives the clamp 23 to move down at a uniform speed, the clamp 23 is subjected to pressure in the Z-axis direction. The second pressure value refers to the pressure of the clamp 23 in the Z-axis direction when the robot arm 2 does not drive the clamp 23 to move down at a constant speed. In other words, the second pressure value refers to the pressure value detected by the force sensor 24 before the mechanical arm 2 drives the clamp 23 to move down at a uniform speed.

(a4) When the change in the pressure of the clamp 23 in the Z-axis direction is greater than the preset value, read the Z-axis coordinate value of the robot arm 2 and replace the obtained value of the robot arm 2 The number of Z-axis coordinate values is increased by 1 to count the total number of Z-axis coordinate values of the robot arm 2 that have been obtained.

(a5) Determine whether the Z-axis coordinate values of the N robot arms 2 have been obtained. If N coordinate values of the Z-axis of the robot arm 2 have been obtained, execute (a6). If the Z-axis coordinate values of the N robot arms 2 have not been obtained, execute (a1).

In this embodiment, N may be equal to 30, or other values, such as 40 or 50.

(a6) When the Z-axis coordinate values of the N robot arms 2 have been obtained, calculate the maximum value based on the Z-axis coordinate values of the N robot arms according to the root mean square algorithm (Root Mean Square, abbreviated as RMS) The upper limit of depth Zmax.

The root-mean-square algorithm is also called quadratic mean, which is an operation formula of generalized mean to the power of 2 and can also be called power-of-two mean. Specifically, the calculation formula of the maximum depth upper limit Zmax is:

Wherein, n represents the total number of obtained Z-axis coordinate values of the robotic arm 2 (that is, n=N in this embodiment, for example, n=30), and x _i represents the obtained Z of each of the robotic arms 2 Axis coordinate value.

In step S2, the computer device 1 controls the camera 25 of the robotic arm 2 to take a picture of the marker 41 located on the carrying platform 4 to obtain a depth image and a color image containing the marker 41.

As mentioned above, the four sides of the marker 41 on the carrying platform 4 are parallel to the carrying platform 4. as shown in picture 2.

It should be noted that the markers 41 mentioned in the following steps all refer to the markers 41 on the carrying platform 4, that is, they do not refer to the markers 41 on the target object 40 on the jig 31.

In step S3, the computer device 1 obtains the depth values of the four corners of the marker 41 from the depth image of the marker 41.

For example, referring to FIG. 2, the computer device 1 obtains four of the markers 41 The depth values of the corners are D11, D12, D13, D14.

In step S4, the computer device 1 calculates the depth reference value Zrms of the marker 41 based on the depth values of the four corners of the marker 41.

In this embodiment, the computer device 1 calculates the depth reference value Zrms of the marker 41 based on the depth values of the four corners of the marker 41 according to the root-mean-square algorithm.

Specifically, the formula for calculating the depth reference value Zrms is:

Wherein, n represents the number of corners of the marker 41 (that is, n=4 in this embodiment), and x _i represents the depth value of each corner of the marker 41.

In step S5, the computer device 1 obtains the coordinate value Zcv of the Z axis of the marker 41 based on the color image of the marker 41.

Specifically, the computer device 1 may use the OPENCV function to calculate and obtain the coordinate value Zcv of the Z axis of the marker 41 based on the color image.

In step S6, the computer device 1 determines whether the depth reference value Zrms of the marker 41 and the Z-axis coordinate value Zcv of the marker 41 are both smaller than the maximum depth upper limit Zmax.

When the depth reference value Zrms of the marker 41 and the Z-axis coordinate value Zcv of the marker 41 are both less than the maximum depth upper limit Zmax, step S7 is executed.

When any one of the depth reference value Zrms of the marker 41 and the Z-axis coordinate value Zcv of the marker 41 is greater than or equal to the maximum depth upper limit Zmax, step S61 is executed, and after step S61 is completed Then return to step S1, the computer device 1 recalculates the maximum depth upper limit Zmax based on the pressure of the clamp 23 in the Z axis direction detected by the force sensor 24 of the robot arm 2.

In step S61, the computer device 1 may directly recalibrate the camera 25 or issue a prompt to prompt the user to recalibrate the camera 25.

Specifically, the recalibration of the camera 25 includes calibrating the shooting parameters of the camera 25, such as the focal length.

Step S7, when the depth reference value Zrms of the marker 41 and the Z-axis coordinate value Zcv of the marker 41 are both less than the maximum depth upper limit Zmax, the computer device 1 calculates the depth reference of the marker 41 The difference between the value Zrms and the Z-axis coordinate value Zcv of the marker 41. When the difference is less than or equal to 0, use the depth reference value Zrms as the depth standard value of the marker 41; and when the difference is greater than 0, use the Z-axis coordinate value of the marker 41 Zcv is used as the standard value of the depth of the marker 41.

In other embodiments, the present invention may further include step S8.

In step S8, when the marker 41 is attached to the target object 40 on the jig 31, the computer device 1 can control the robotic arm 2 to grab the marker 41 according to the depth standard value of the marker 41 determined above. 41 corresponds to the target object 40.

According to the above description, the method for determining the standard value of the depth of the marker in the embodiment of the present invention calculates a maximum depth based on the pressure of the clamp in the Z-axis direction detected by the force sensor of the robot arm. Limit, where the XOY plane is established with the horizontal plane of the carrying platform carrying the robot arm, and the coordinate system O-XYZ is established with the Z axis as the direction perpendicular to the XOY plane; and the depth camera of the robot arm is controlled to be positioned at all Taking photos of the marker on the bearing platform to obtain the depth image and color image of the marker; obtaining the depth values of the four corners of the marker based on the depth image of the marker; The depth value of the four corners of the marker is calculated to obtain the depth reference value of the marker; the Z-axis coordinate value of the marker is obtained based on the color image; when the depth reference value of the marker and the marker When the Z-axis coordinate value of the object is less than the upper limit of the maximum depth, the difference between the depth reference value of the marker and the Z-axis coordinate value of the marker is calculated; when the difference is less than or equal to When 0, the depth reference value is used as the depth standard value of the marker; and when the difference is greater than 0, the Z-axis coordinate value is used as the depth standard value of the marker, which can be changed without changing Under the premise of the existing fixture and the environment, the depth value of the marker can be determined quickly and accurately, so that the robotic arm can find the gripping position of the target object according to the determined depth value of the marker.

The above-mentioned Figure 3 details the method for determining the depth standard value of the marker of the present invention, 4 and 5, the functional modules of the software device that implements the method for determining the depth standard value of the marker and the hardware device architecture that implements the method for determining the depth standard value of the marker are respectively introduced below.

It should be understood that the embodiments are only for illustrative purposes, and are not limited by this structure in the scope of the patent application.

Refer to FIG. 4, which is a functional module diagram of the determining system 30 provided by a preferred embodiment of the present invention.

In some embodiments, the determination system 30 runs in the computer device 1. The determination system 30 may include a plurality of functional modules composed of code segments. The computer program code of each program segment in the determination system 30 can be stored in the memory of the computer device 1 and executed by at least one processor of the computer device 1 to determine the depth standard value of the marker ( See the description of Figure 3 for details).

In this embodiment, the determining system 30 can be divided into multiple functional modules according to the functions it performs. The functional modules may include: an execution module 301 and a photographing module 302. The module referred to in the present invention refers to a series of computer program segments that can be executed by at least one processor and can complete fixed functions, and are stored in a memory. In this embodiment, the functions of each module will be described in detail in subsequent embodiments.

The execution module 301 calculates a maximum depth upper limit Zmax based on the pressure of the clamp 23 in the Z-axis direction detected by the force sensor 24 of the robot arm 2. The execution module 301 establishes the XOY plane on the horizontal plane where the carrying platform 4 carrying the robot arm 2 is located, and establishes the coordinate system O-XYZ with the Z axis as the direction perpendicular to the XOY plane.

Specifically, referring to FIG. 2, in this embodiment, the execution module 301 uses the bottom end of the base 21 of the robot arm 2 as the origin O, and the horizontal plane where the bottom end of the base 21 is located is the XOY plane. , And establish a coordinate system O-XYZ with the Z axis as the Z-axis in the vertical upward direction perpendicular to the XOY plane.

In one embodiment, the maximum depth limit Zmax is calculated based on the pressure of the clamp 23 in the Z-axis direction detected by the force sensor 24 of the robot arm 2 including (a1) -(a6):

(a1) The clamp 23 of the robot arm 2 is set at a predetermined distance from the carrying platform 4.

In this embodiment, the preset distance may be 30 cm or 35 cm. The preset distance can be set according to actual application scenarios.

(a2) Control the robot arm 2 to drive the clamp 23 to move down at a uniform speed, and use the force sensor 24 of the robot arm 2 to detect the pressure of the clamp 23 in the Z-axis direction.

Specifically, the robot arm 2 can be controlled to drive the clamp 23 to move vertically downward at a constant speed.

(a3) According to the pressure of the clamp 23 in the Z-axis direction detected by the force sensor 24, determine whether the change in the pressure of the clamp 23 in the Z-axis direction is greater than a preset value (for example, 1 Newton) , 1.5 Newtons, or other values). When the change in the pressure of the clamp 23 in the Z-axis direction is greater than the preset value, (a4) is executed. When the change in the pressure of the clamp 23 in the Z-axis direction is less than or equal to the preset value, continue to perform (a3).

In one embodiment, the change in the pressure of the clamp 23 in the Z-axis direction may refer to the difference between the first pressure value and the second pressure value, where the first pressure value refers to the mechanical When the arm 2 drives the clamp 23 to move down at a uniform speed, the clamp 23 is subjected to pressure in the Z-axis direction. The second pressure value refers to the pressure of the clamp 23 in the Z-axis direction when the robot arm 2 does not drive the clamp 23 to move down at a constant speed. In other words, the second pressure value refers to the pressure value detected by the force sensor 24 before the mechanical arm 2 drives the clamp 23 to move down at a uniform speed.

(a4) When the change in the pressure of the clamp 23 in the Z-axis direction is greater than the preset value, read the Z-axis coordinate value of the robot arm 2 and use the obtained Z-axis coordinate value of the robot arm 2 The number of axis coordinate values is increased by 1 to count the total number of Z axis coordinate values of the robot arm 2 that have been obtained.

(a5) Determine whether the Z-axis coordinate values of the N robot arms 2 have been obtained. If Having obtained N coordinate values of the Z-axis of the robot arm 2, execute (a6). If the Z-axis coordinate values of the N robot arms 2 have not been obtained, execute (a1).

In this embodiment, N may be equal to 30, or other values, such as 40 or 50.

(a6) When the Z-axis coordinate values of the N robot arms 2 have been obtained, calculate the maximum value based on the Z-axis coordinate values of the N robot arms according to the root mean square algorithm (Root Mean Square, abbreviated as RMS) The upper limit of depth Zmax.

The root-mean-square algorithm is also called quadratic mean, which is an operation formula of generalized mean to the power of 2 and can also be called power-of-two mean. Specifically, the calculation formula of the maximum depth upper limit Zmax is:

Wherein, n represents the total number of obtained Z-axis coordinate values of the robot arm 2 (that is, n=N in this embodiment), and x _i represents the obtained Z-axis coordinate value of each robot arm 2.

The photographing module 302 controls the camera 25 of the robotic arm 2 to take photos of the marker 41 located on the carrying platform 4 to obtain a depth image and a color image containing the marker 41.

As mentioned above, the four sides of the marker 41 on the carrying platform 4 are parallel to the carrying platform 4. as shown in picture 2.

It should be noted that the markers 41 mentioned below all refer to the markers 41 on the carrying platform 4, that is, they do not refer to the markers 41 on the target object 40 on the jig 31.

The execution module 301 obtains the depth values of the four corners of the marker 41 from the depth image of the marker 41.

For example, referring to FIG. 2, the execution module 301 obtains the depth values D11, D12, D13, and D14 of the four corners of the marker 41.

The execution module 301 calculates the depth reference value Zrms of the marker 41 based on the depth values of the four corners of the marker 41.

In this embodiment, the execution module 301 is based on the root-mean-square algorithm based on the The depth values of the four corners of the marker 41 are calculated to obtain the depth reference value Zrms of the marker 41.

Specifically, the formula for calculating the depth reference value Zrms is:

Wherein, n represents the number of corners of the marker 41 (that is, n=4 in this embodiment), and x _i represents the depth value of each corner of the marker 41.

The execution module 301 obtains the coordinate value Zcv of the Z axis of the marker 41 based on the color image of the marker 41.

Specifically, the execution module 301 may use the OPENCV function to calculate and obtain the coordinate value Zcv of the Z axis of the marker 41 based on the color image.

The execution module 301 determines whether the depth reference value Zrms of the marker 41 and the Z-axis coordinate value Zcv of the marker 41 are both smaller than the maximum depth upper limit Zmax.

When any one of the depth reference value Zrms of the marker 41 and the Z-axis coordinate value Zcv of the marker 41 is greater than or equal to the maximum depth upper limit Zmax, the execution module 301 recalibrates the camera 25 or sends out Prompt, the user is prompted to recalibrate the camera 25, and then the execution module 301 recalculates the upper limit of the maximum depth based on the pressure of the clamp 23 in the Z-axis direction detected by the force sensor 24 of the robotic arm 2 Zmax.

Specifically, the recalibration of the camera 25 includes calibrating the shooting parameters of the camera 25, such as the focal length.

When the depth reference value Zrms of the marker 41 and the Z-axis coordinate value Zcv of the marker 41 are both less than the maximum depth upper limit Zmax, the execution module 301 calculates the depth reference value Zrms of the marker 41 The difference with the Z-axis coordinate value Zcv of the marker 41. When the difference is less than or equal to 0, use the depth reference value Zrms as the depth standard value of the marker 41; and when the difference is greater than 0, use the Z-axis coordinate value of the marker 41 Zcv is used as the standard value of the depth of the marker 41.

When the marker 41 is affixed to the target object 40 on the jig 31, the execution module 301 can control the robotic arm 2 to grab and hold the target object according to the depth standard value of the marker 41 determined above. The target object 40 corresponding to the marker 41.

According to the above description, the determination system 30 for determining the depth standard value of the marker according to the embodiment of the present invention calculates the pressure of the clamp in the Z-axis direction detected by the force sensor of the robot arm. Obtain a maximum depth upper limit, where the XOY plane is established with the horizontal plane of the bearing platform carrying the robot arm, and the coordinate system O-XYZ is established with the Z axis as the direction perpendicular to the XOY plane; controlling the robot arm The depth camera takes photos of the marker on the bearing platform to obtain the depth image and color image of the marker; and obtains the depth values of the four corners of the marker based on the depth image of the marker ; The depth reference value of the marker is calculated based on the depth values of the four corners of the marker; the Z-axis coordinate value of the marker is obtained based on the color image; when the depth of the marker is referenced When the value and the Z-axis coordinate value of the marker are both less than the upper limit of the maximum depth, the difference between the depth reference value of the marker and the Z-axis coordinate value of the marker is calculated; when the When the difference is less than or equal to 0, the depth reference value is used as the depth standard value of the marker; and when the difference is greater than 0, the Z-axis coordinate value is used as the depth standard value of the marker , Can quickly and accurately determine the depth value of the marker without changing the existing jig and environment, so that the robotic arm can find the gripping position of the target object according to the determined depth value of the marker.

Refer to FIG. 5, which is a schematic structural diagram of a computer device provided by a preferred embodiment of the present invention.

In a preferred embodiment of the present invention, the computer device 1 includes a storage 51 and at least one processor 52 that are electrically connected to each other.

Those skilled in the art should understand that the structure of the computer device 1 shown in FIG. 1 does not constitute a limitation of the embodiment of the present invention. The computer device 1 may also include more or less other hardware or software than shown in the figure. Or different component arrangements. For example, the computer device 1 may also include components such as a display screen.

In some embodiments, the computer device 1 includes a terminal that can automatically perform numerical calculation and/or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to micro Processors, dedicated integrated circuits, programmable gate arrays, digital word processors and embedded devices, etc.

It should be noted that the computer device 1 is only an example, and other existing or future electronic products that can be adapted to the present invention should also be included in the protection scope of the present invention and included here by reference.

In some embodiments, the storage 51 may be used to store program codes and various data of a computer program. For example, the storage 51 can be used to store the determination system 30 installed in the computer device 1 and realize high-speed and automatic access to programs or data during the operation of the computer device 1. The storage 51 may include a read-only memory (Read-Only Memory, ROM), a programmable read-only memory (Programmable Read-Only Memory, PROM), and an erasable programmable read-only memory (Erasable Programmable Read-Only Memory (EPROM), One-time Programmable Read-Only Memory (OTPROM), Electronically-Erasable Programmable Read-Only Memory (Electrically-Erasable Programmable Read-Only Memory, EEPROM), Compact Disc Read-Only Memory (CD-ROM) or other optical disk storage, magnetic disk storage, tape storage, or any other computer-readable storage medium that can be used to carry or store data .

In some embodiments, the at least one processor 52 may be composed of an integrated circuit. For example, it can be composed of a single packaged integrated circuit, or it can be composed of multiple packaged integrated circuits with the same function or different functions, including one or more central processing units (CPU), micro-processing Combinations of processors, digital word processing chips, graphics processors, and various control chips. The at least one processor 52 is the control core (Control Unit) of the computer device 1, which uses various interfaces and lines to connect the various components of the entire computer device 1, by running or executing programs stored in the storage 51 or Modules and call data stored in the storage 51 to perform various functions of the computer device 1 and process data.

Although not shown, the computer device 1 may also include a power source (such as a battery) for supplying power to various components. Preferably, the power source may be logically connected to the at least one processor 52 through a power management device, so as to be realized by the power management device. Manage functions such as charging, discharging, and power management. The power supply may also include one or more DC or AC power supplies, recharging devices, power failure detection circuits, power converters or inverters, power supply status indicators and other arbitrary components. The computer device 1 may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be repeated here.

It should be understood that the embodiments are only for illustrative purposes, and are not limited by this structure in the scope of the patent application.

The above-mentioned integrated unit realized in the form of a software function module can be stored in a computer readable storage medium. The above-mentioned software function module includes several instructions, which are used to make a computer device (can be a vehicle-mounted computer, etc.) or a processor execute part of the method described in each embodiment of the present invention.

In a further embodiment, with reference to FIG. 4, the at least one processor 52 can execute the operating device of the computer device 1 and various installed applications (such as the determination system 30) and so on.

The storage 51 stores computer program codes, and the at least one processor 52 can call the computer program codes stored in the storage 51 to perform related functions. For example, the various modules described in FIG. 5 are computer program codes stored in the storage 51 and executed by the at least one processor 52, so as to realize the functions of the various modules, such as determining markers. The depth value.

In an embodiment of the present invention, the storage 51 stores a plurality of instructions, and the plurality of instructions are executed by the at least one processor 52 to determine the depth value of the marker.

Specifically, as shown in FIG. 3, the method for the at least one processor 52 to execute the above-mentioned multiple instructions to determine the depth value of the marker includes: according to the position of the clamp in the Z-axis direction detected by the force sensor of the robotic arm. The pressure received is calculated to obtain a maximum depth upper limit; the camera of the robotic arm is controlled to take a photo of the marker to obtain a depth image and a color image containing the marker; to obtain the depth image of the marker Depth values of the four corners of the marker; calculating the depth reference value of the marker based on the depth values of the four corners of the marker; Obtain the Z-axis coordinate value of the marker based on the color image; when the depth reference value of the marker and the Z-axis coordinate value of the marker are both less than the maximum depth upper limit, calculate The difference between the depth reference value of the marker and the Z-axis coordinate value of the marker; when the difference is less than or equal to 0, the depth reference value is used as the depth standard value of the marker; And when the difference is greater than 0, the Z-axis coordinate value is used as the depth standard value of the marker.

Preferably, the method further includes: when any one of the depth reference value of the marker and the Z-axis coordinate value of the marker is greater than or equal to the maximum depth upper limit, according to the force of the robot arm The pressure of the clamp in the Z-axis direction detected by the sensor recalculates the upper limit of the maximum depth.

Preferably, said calculating a maximum depth upper limit value according to the pressure of the clamp in the Z-axis direction detected by the force sensor of the robot arm includes: setting the clamp of the robot arm at a distance from the XOY plane Is a preset distance position; controlling the robotic arm to drive the clamp to move down at a constant speed, and at the same time using the force sensor to detect the pressure of the clamp in the Z-axis direction; When the detected pressure of the clamp in the Z-axis direction determines that the change in the pressure of the clamp in the Z-axis direction is greater than the preset value, read the Z-axis coordinate value of the robot arm; when N is read When the Z-axis coordinate values of each of the robot arms are calculated, the maximum depth upper limit value is calculated based on the Z-axis coordinate values of the N robot arms according to a root-mean-square algorithm.

Preferably, the method calculates the depth reference value of the marker based on the depth values of the four corners of the marker according to the root-mean-square algorithm.

Preferably, the obtaining the coordinate value of the Z axis of the marker based on the color image includes: obtaining the coordinate value of the Z axis of the marker based on the color image by using an OPENCV function.

Preferably, the marker is located on the carrying platform and the four sides of the marker are parallel to the carrying platform.

In the several embodiments provided by the present invention, it should be understood that the disclosed computer-readable storage medium, device, and method may be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical units, that is, they may be located in one place, or they may be distributed to multiple networks Unit. Some or all of the modules may be selected according to actual needs to achieve the objectives of the solutions of the embodiments.

In addition, the functional modules in the various embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit. The above-mentioned integrated unit can be realized either in the form of hardware, or in the form of hardware plus software functional modules.

For those skilled in the art, it is obvious that the present invention is not limited to the details of the above exemplary embodiments, and the present invention can be implemented in other specific forms without departing from the spirit or basic characteristics of the present invention. Therefore, no matter from which point of view, the embodiments should be regarded as exemplary and non-restrictive. The scope of the present invention is defined by the scope of the attached patent application rather than the above description, and therefore it is intended to fall within the scope of the application. The meaning of the equivalent elements of the patent scope and all changes within the scope are included in the present invention. Any reference signs in the scope of the patent application should not be regarded as limiting the scope of the patent application involved. In addition, it is obvious that the word "including" does not exclude other elements or the singular number does not exclude the plural number. Multiple units or devices stated in the scope of the device patent application can also be implemented by one unit or device through software or hardware. Words such as first and second are used to denote names, but do not denote any specific order.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than limiting. Although the present invention has been described in detail with reference to preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be Make modifications or equivalent substitutions without departing from the technology of the present invention The spirit and scope of the surgical plan.

Claims (12)

A method for determining the standard value of the depth of a marker, which is applied to a computer device, wherein the method includes: calculating a maximum depth limit based on the pressure of the clamp in the Z-axis direction detected by the force sensor of the robotic arm Value; control the camera of the robotic arm to take a photo of the marker to obtain a depth image and a color image containing the marker, wherein the marker is attached to the target object; from the depth map of the marker Image acquiring the depth values of the four corners of the marker; calculating the depth reference value of the marker based on the depth values of the four corners of the marker; obtaining the Z of the marker based on the color image Axis coordinate value; when the depth reference value of the marker and the Z-axis coordinate value of the marker are both less than the upper limit of the maximum depth, the depth reference value of the marker and the value of the marker are calculated The difference between the Z-axis coordinate values; when the difference is less than or equal to 0, the depth reference value is used as the depth standard value of the marker; when the difference is greater than 0, the Z The axis coordinate value is used as the depth standard value of the marker; and the robotic arm is controlled to grab the target object based on the depth standard value of the marker. The method for determining the depth standard value of a marker according to claim 1, wherein the method further includes: when any one of the depth reference value of the marker and the Z-axis coordinate value of the marker is greater than or equal to When the maximum depth upper limit is used, the maximum depth upper limit is recalculated according to the pressure of the clamp in the Z-axis direction detected by the force sensor of the robot arm. The method for determining the depth standard value of a marker according to claim 2, wherein the maximum depth upper limit value is calculated according to the pressure of the clamp in the Z-axis direction detected by the force sensor of the robot arm include: Set the gripper of the robotic arm at a preset distance from one of the load-bearing platforms; control the robotic arm to drive the gripper to move down at a constant speed, and at the same time, use the force sensor to detect the movement of the gripper in the Z-axis direction. When the pressure of the clamp in the Z-axis direction determined by the force sensor detects that the change in the pressure of the clamp in the Z-axis direction is greater than a preset value, read the The Z-axis coordinate value of the robot arm; when the Z-axis coordinate values of the N robot arms are read, the maximum depth upper limit is calculated based on the Z-axis coordinate values of the N robot arms according to the root-mean-square algorithm . The method for determining the depth standard value of a marker according to claim 1, wherein the method calculates the depth reference value of the marker based on the depth values of the four corners of the marker according to a root-mean-square algorithm. The method for determining the depth standard value of a marker according to claim 1, wherein the obtaining the Z-axis coordinate value of the marker based on the color image includes: obtaining the coordinate value of the marker based on the color image by using an OPENCV function The coordinate value of the Z axis of the marker. The method for determining the depth standard value of a marker according to claim 1, wherein the marker is located on a carrying platform and four sides of the marker are parallel to the carrying platform. A computer device, wherein the computer device includes a memory and a processor, the memory is used to store a plurality of modules, and the processor is used to execute the plurality of modules, wherein the plurality of modules include : Execution module, used to calculate a maximum depth limit based on the pressure of the fixture in the Z-axis direction detected by the force sensor of the robotic arm; shooting module, used to control the camera of the robotic arm Take a picture of the marker to obtain a depth image and a color image containing the marker, wherein the marker is attached to the target object; the execution module is also used to obtain the depth image of the marker Image acquisition of the depth values of the four corners of the marker; the execution module is also used to calculate the depth values of the four corners of the marker to obtain the marker The depth reference value of the marker; the execution module is also used to obtain the Z-axis coordinate value of the marker based on the color image; the execution module is also used to calculate the depth of the marker When the reference value and the Z-axis coordinate value of the marker are both less than the upper limit of the maximum depth, the difference between the depth reference value of the marker and the Z-axis coordinate value of the marker is calculated; The execution module is also used for when the difference is less than or equal to 0, the depth reference value is used as the depth standard value of the marker; the execution module is also used for when the difference is greater than 0 When the Z-axis coordinate value is used as the standard depth value of the marker; and the execution module is also used to control the robotic arm to grab the target object based on the standard depth value of the marker. The computer device according to claim 7, wherein the execution module is further used for: when any one of the depth reference value of the marker and the Z-axis coordinate value of the marker is greater than or equal to the When the maximum depth upper limit is used, the maximum depth upper limit is recalculated according to the pressure of the clamp in the Z-axis direction detected by the force sensor of the robot arm. The computer device according to claim 8, wherein the calculation of a maximum depth upper limit value based on the pressure of the clamp in the Z-axis direction detected by the force sensor of the robot arm includes: The clamp is set at a preset distance from one of the load-bearing platforms; the robot arm is controlled to drive the clamp to move down at a constant speed, and the force sensor is used to detect the pressure of the clamp in the Z-axis direction; when When it is determined according to the pressure of the clamp in the Z-axis direction detected by the force sensor that the change in the pressure of the clamp in the Z-axis direction is greater than a preset value, read the Z-axis of the robot arm Coordinate value; when the Z-axis coordinate values of the N robot arms are read, the maximum depth upper limit is calculated based on the Z-axis coordinate values of the N robot arms according to the root-mean-square algorithm. The computer device according to claim 7, wherein the execution module calculates the depth reference value of the marker based on the depth values of the four corners of the marker according to a root-mean-square algorithm. The computer device according to claim 7, wherein the obtaining the coordinate value of the Z-axis of the marker based on the color image comprises: obtaining the Z-axis of the marker based on the color image by using an OPENCV function The coordinate value. The computer device according to claim 7, wherein the marker is located on a carrying platform and four sides of the marker are parallel to the carrying platform.

Images