CN110433467B - Operation method and device of table tennis ball picking robot based on binocular vision and ant colony algorithm - Google Patents
Operation method and device of table tennis ball picking robot based on binocular vision and ant colony algorithm Download PDFInfo
- Publication number
- CN110433467B CN110433467B CN201910744905.2A CN201910744905A CN110433467B CN 110433467 B CN110433467 B CN 110433467B CN 201910744905 A CN201910744905 A CN 201910744905A CN 110433467 B CN110433467 B CN 110433467B
- Authority
- CN
- China
- Prior art keywords
- table tennis
- coordinate
- coordinate system
- robot
- camera
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 40
- 238000004422 calculation algorithm Methods 0.000 title claims abstract description 22
- 238000001514 detection method Methods 0.000 claims abstract description 9
- 238000006243 chemical reaction Methods 0.000 claims description 8
- 230000009466 transformation Effects 0.000 claims description 8
- 238000003860 storage Methods 0.000 claims description 6
- 238000012549 training Methods 0.000 claims description 6
- 238000013507 mapping Methods 0.000 claims description 3
- 238000004364 calculation method Methods 0.000 abstract description 4
- 238000005457 optimization Methods 0.000 abstract description 2
- 238000010586 diagram Methods 0.000 description 13
- 239000003016 pheromone Substances 0.000 description 11
- 238000004590 computer program Methods 0.000 description 7
- 239000011159 matrix material Substances 0.000 description 7
- 230000006870 function Effects 0.000 description 5
- 238000012545 processing Methods 0.000 description 4
- 241000257303 Hymenoptera Species 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000003384 imaging method Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000001186 cumulative effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000001704 evaporation Methods 0.000 description 1
- 230000008020 evaporation Effects 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 230000001788 irregular Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 239000011664 nicotinic acid Substances 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000004904 shortening Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
- 238000012800 visualization Methods 0.000 description 1
Images
Classifications
-
- A—HUMAN NECESSITIES
- A63—SPORTS; GAMES; AMUSEMENTS
- A63B—APPARATUS FOR PHYSICAL TRAINING, GYMNASTICS, SWIMMING, CLIMBING, OR FENCING; BALL GAMES; TRAINING EQUIPMENT
- A63B47/00—Devices for handling or treating balls, e.g. for holding or carrying balls
- A63B47/02—Devices for handling or treating balls, e.g. for holding or carrying balls for picking-up or collecting
- A63B47/021—Devices for handling or treating balls, e.g. for holding or carrying balls for picking-up or collecting for picking-up automatically, e.g. by apparatus moving over the playing surface
-
- A—HUMAN NECESSITIES
- A63—SPORTS; GAMES; AMUSEMENTS
- A63B—APPARATUS FOR PHYSICAL TRAINING, GYMNASTICS, SWIMMING, CLIMBING, OR FENCING; BALL GAMES; TRAINING EQUIPMENT
- A63B67/00—Sporting games or accessories therefor, not provided for in groups A63B1/00 - A63B65/00
- A63B67/04—Table games physically beneficial for the human body, modelled on outdoor sports, e.g. table tennis
Abstract
The invention discloses a table tennis ball picking robot operation method and equipment based on binocular vision and ant colony algorithm, wherein the method comprises the following steps: acquiring RGB images shot by a binocular camera, detecting table tennis balls from the RGB images and acquiring coordinate positions of the table tennis balls; acquiring depth data of a binocular camera to obtain the distance between each ping-pong ball and the camera; calculating to obtain a coordinate point set of the table tennis under a global coordinate system; carrying out shortest traversal path planning on the coordinate point set by using an ant colony algorithm to obtain a primary ball picking path; and removing the ping-pong balls passing through for many times from the primary ball picking path to obtain a path without a redundant route. The invention uses the binocular camera to simultaneously realize the detection and the positioning of the table tennis, is convenient and efficient, and simultaneously ensures the precision, so that the ball picking process is more intelligent; and the redundant route optimization is carried out on the path drawn by the ant colony calculation rule, so that the walking path of the robot is further reduced, and the time is saved.
Description
Technical Field
The invention relates to the technical field of robots, in particular to a table tennis ball picking robot operation method and device based on binocular vision and an ant colony algorithm.
Background
With the development of artificial intelligence technology and national socioeconomic, the service robot gradually enters the public families. The table tennis ball is popular with people as a national ball. However, a large amount of ball picking work is very annoying, and at present, a special person in the table tennis club is responsible for picking balls, so that the labor cost is high. However, no popular table tennis ball picking robot exists in the market, only research is focused on academic research, actual engineering application is few, and most robots only have scattered methods.
The existing detection and positioning technology is complex and tedious, and a common RGB camera is difficult to complete the task of positioning the table tennis, so most ball-picking robots cannot position the ball, the global traversal path planning of all balls cannot be realized, and the navigation method is very inefficient.
The ant colony algorithm is a bionic algorithm, is mainly used for the TSP problem, and can obtain a near-optimal solution in a short time. However, when the robot is actually applied to a ball picking robot, the robot is limited in width and needs to be further optimized.
Disclosure of Invention
The invention aims to overcome the defects in the prior art, provides the table tennis ball picking robot operation method and equipment based on binocular vision and ant colony algorithm, realizes the detection and positioning of table tennis balls, optimizes redundant routes of the planned routes, further reduces the walking routes of the robot and saves time.
In order to solve the technical problem, the invention provides a table tennis ball picking robot operation method based on binocular vision and ant colony algorithm, which is characterized by comprising the following steps:
establishing a global coordinate system of a table tennis training field;
acquiring RGB images of ground table tennis scattering conditions in a field shot by a binocular camera, detecting table tennis from the RGB images and acquiring coordinate positions of the table tennis;
acquiring depth data of a binocular camera to obtain the distance between each ping-pong ball and the camera;
calculating to obtain a coordinate point set of the table tennis under a global coordinate system according to the coordinate position of each table tennis and the distance from the camera;
carrying out shortest traversal path planning on the coordinate point set by using an ant colony algorithm to obtain a primary ball picking path;
and removing the ping-pong balls passing through for many times from the primary ball picking path to obtain a path without a redundant route.
Further, the specific process of detecting the table tennis balls from the RGB images and obtaining the coordinate positions of the table tennis balls includes:
converting the RGB image into an HSV image;
converting the HSV image into a gray scale image, and performing Gaussian blur;
carrying out contour detection by using a canny operator to obtain the contours of all objects in the image;
and detecting circles in the graph by using Hough circle transformation, namely detecting the table tennis balls in the field, and calculating to obtain coordinate positions of all the table tennis balls in the graph.
Further, the step of calculating and obtaining the coordinate point set of the table tennis balls under the global coordinate system according to the coordinate positions of the table tennis balls and the distance from the camera comprises the following steps:
converting the coordinate position of each table tennis ball and the distance from the camera into a three-dimensional coordinate system of the camera to obtain a coordinate point set of the table tennis balls in a three-dimensional coordinate system of the camera;
then converting the coordinates of the table tennis under the camera three-dimensional coordinate system into the robot coordinate system;
and finally, converting the coordinates of the table tennis under the robot coordinate system into a global coordinate system to obtain a coordinate point set of the table tennis under the global coordinate system.
Further, the process of converting the coordinate position of each table tennis ball and the distance from the camera to the three-dimensional coordinate system of the camera comprises the following steps:
establishing a camera three-dimensional coordinate system XYZ and an image plane coordinate system xy;
the mapping relation between the camera three-dimensional coordinate system midpoint P (X, Y, Z) and the image plane coordinate system midpoint P (X, Y) is as follows:
wherein f is the focal length of the camera, Z is the depth of the table tennis ball obtained from the depth data, X represents the X-axis coordinate of the image plane coordinate system, and X represents the X-axis coordinate of the three-dimensional coordinate system of the camera; y represents a Y-axis coordinate in an image plane coordinate system, and Y represents a Y-axis coordinate in a camera three-dimensional coordinate system; c. CxAs the coordinate of the principal point on the x-axis, cyAs the coordinate of the principal point on the y-axis, fxIs the dimension of the pixel of f on the x-axis, fyIs the pixel dimension of f on the y-axis;
the coordinate position of each table tennis is the coordinate in the image plane coordinate system, the distance from the camera is the Z-axis numerical value in the camera coordinate system, and the three-dimensional coordinate P (X, Y, Z) of the table tennis in the camera coordinate system is calculated according to the formula.
Further, the process of converting the coordinates of the table tennis ball in the robot coordinate system to the global coordinate system is as follows:
the coordinate system of the robot at the beginning is the origin of the global coordinate system, and the coordinate is (x)0，y0，th0) Let (0, 0, 0) and set the robot movement speed V ═ Vx，Vy，Vth)，VxSpeed in the direction of the x-axis, VyRefers to the velocity, V, along the y-axisthThe rotation speed of the robot is indicated; then after dt times, the coordinates of the robot coordinate system with respect to the global coordinate origin are:
(x1，y1，th1)＝(x0+(Vx*cos(th0)-Vy*sin(th0))*dt，y0+(Vx*sin(th0)+Vy*cos(th0))*dt，th0+Vth*dt) (5)
according to the coordinate transformation relation of the robot coordinate system obtained by the formula (5), coordinate transformation can be performed, and the target table tennis coordinate (X ', Y') in the robot coordinate system is transformed into the coordinate (X ", Y") in the global coordinate system as follows:
(X″，Y″)＝(x1+(X′*cos(th1)-Y′*sin(th1))，
y1+(X′*sin(th1)+Y′*cos(th1)))
and completing the coordinate conversion of all table tennis balls to obtain a coordinate point set of the table tennis balls in the global coordinate system.
Further, the step of removing the table tennis balls passing through for multiple times from the primary ball picking path to obtain a path without redundant paths comprises the following processes:
according to the preliminary ball picking path, assuming that the robot moves from the starting point to the 1 st target table tennis,
draw a straight line b1、b2And d1、d2，b1、b2The minimum circumcircle of the robot at the starting point is tangent to the connecting line of the starting point and the target table tennis; straight line d1The direction of the connecting line of the two points is vertical and tangent with the tail part of the minimum circumcircle of the robot at the starting point; straight line d2The direction of the connecting line of the two points is vertical and tangent to the head of the minimum circumcircle of the robot at the target point; the rectangle formed by the four lines represents the area swept by the robot from the departure point to the target point, all table tennis coordinate points in the rectangle formed by the four lines are calculated and obtained, the points are deleted from the coordinate point set,
then taking the target point as a starting point, selecting the next target point from the primary ball picking path, and carrying out the same operation; and repeating the steps until all the points on the ball picking path are processed, and finally obtaining a path without redundant routes.
Accordingly, the present invention also provides a computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by a computing device, cause the computing device to perform the above-described method.
Accordingly, the present invention also provides a computing device comprising,
one or more processors, memory, and one or more programs stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for performing the above-described methods.
Compared with the prior art, the invention has the following beneficial effects: the invention uses the binocular camera to simultaneously realize the detection and the positioning of the table tennis, is convenient and efficient, and simultaneously ensures the precision, so that the ball picking process is more intelligent; and the redundant route optimization is carried out on the path drawn by the ant colony calculation rule, so that the walking path of the robot is further reduced, and the time is saved.
Drawings
FIG. 1 is a schematic flow diagram of the process of the present invention;
FIG. 2 is a schematic flow diagram of table tennis ball detection and positioning;
FIG. 3 is a schematic diagram of a camera aperture imaging model;
FIG. 4 is a base _ link and map coordinate system transformation diagram;
FIG. 5 is a schematic flow chart of the ant colony algorithm for traversal path planning;
fig. 6 is a schematic diagram of redundancy points.
Detailed Description
The invention is further described below with reference to the accompanying drawings. The following examples are only for illustrating the technical solutions of the present invention more clearly, and the protection scope of the present invention is not limited thereby.
The invention relates to a table tennis ball picking robot operation method based on binocular vision and ant colony algorithm, which is shown in figure 1 and comprises the following steps:
step 1, obtaining a map of a table tennis training field and establishing a global coordinate system.
Firstly, a static map of a ping-pong training field is obtained by using a mapping program under a laser radar and ROS (robot operating system) platform, then the map is denoised and modified to obtain an available global static grid map, and the robot navigates according to the map to avoid obstacles. As a global coordinate system map, the coordinate system takes a certain angular point of a rectangular training field as an origin and takes the length and width directions of the training field as horizontal and vertical axes respectively. The table tennis balls detected later will be converted to be displayed under the coordinate system.
And 2, acquiring RGB images of the binocular camera, detecting the table tennis from the RGB images and acquiring a coordinate point of the table tennis in the image.
The binocular camera uses kinect, which has an RGB color camera and a depth camera. The binocular camera is arranged in the center of the front part of the robot, the visual angle looks ahead, and the height from the ground is preferably 15cm to 40 cm. The robot shoots the ping-pong on the ground before the beginning of the ball picking task, receives RGB data and depth data of kinect at the same time in the ROS platform, and ensures that two obtained data timestamps are consistent.
As shown in fig. 2, the RGB images including a plurality of table tennis balls in the field shot at a certain time kinect are taken out, the images are converted into HSV images, filtering is carried out according to HSV value thresholds of white table tennis balls, and white parts in the images are taken out, namely white object images including table tennis balls are taken out. And then converting the HSV image into a gray image, and performing Gaussian blur to make the edge of the image smoother. And then carrying out contour detection by using a canny operator, setting a threshold value to be 100-200, and taking out the contours of all objects in the image. And finally, detecting circles in the graph by using Hough circle transformation, namely detecting the table tennis balls (coordinates of the table tennis balls in the graph), calculating to obtain coordinate point sets of all the table tennis balls in the graph, and numbering the detected table tennis balls from 1 to N (the number of the table tennis balls) according to the detection sequence.
And 3, acquiring depth data of the binocular camera, acquiring depth information (namely the horizontal distance of the table tennis from the camera) corresponding to the two-dimensional coordinates of the table tennis according to the two-dimensional coordinates of the table tennis, and calculating the coordinates of the table tennis in a robot coordinate system according to the depth information.
Obtaining two coordinate systems according to a camera pinhole model, namely an idealized pinhole imaging model, wherein one is a camera three-dimensional coordinate system XYZ as shown in FIG. 3, wherein an X axis is parallel to the ground and an image plane, a Y axis is perpendicular to the ground, and a Z axis is perpendicular to the image plane; one is an image plane coordinate system xy, the image plane is perpendicular to the ground; according to the transversal theorem in geometry, the mapping relation between the midpoint P (X, Y, Z) in the camera three-dimensional coordinate system and the midpoint P (X, Y) in the image plane coordinate system is obtained as follows:
wherein f is the focal length of the camera, Z is the depth of the table tennis ball obtained from the depth data, X represents the X-axis coordinate of the image plane coordinate system, and X represents the X-axis coordinate of the three-dimensional coordinate system of the camera; y represents a Y-axis coordinate in the image plane coordinate system, and Y represents a Y-axis coordinate in the camera three-dimensional coordinate system.
According to camera internal parameters, the steps are improved:
due to practical differences, the principal point of each camera (the origin of the image plane coordinate system) is not necessarily at the very center of the image; there is also a difference between the pixels in the x and y directions, and therefore the above equations (1) and (2) are transformed into:
wherein, cxAs the coordinate of the principal point on the x-axis, cyAs the coordinate of the principal point on the y-axis, fxIs the dimension of the pixel of f on the x-axis, fyIs the pixel dimension of f on the y-axis.
As shown in the flow chart of fig. 2. Firstly, calling an interface function to obtain an internal reference matrix of the camera, and then calculating a three-dimensional coordinate P (X, Y, Z) of the target table tennis in a camera coordinate system according to the formulas (3) and (4) (X, Y is obtained by the calculation, wherein Z is the depth of the point, namely the horizontal distance of the table tennis from the camera). Since the directions of the camera three-dimensional coordinate system and the robot coordinate system (named base _ link, the advancing direction of the robot is the X axis, and the left side is the Y axis) are not consistent, there is a conversion relationship, and finally, the coordinates of the table tennis ball in the robot coordinate system are (X ', Y') (Z, X) (assuming that the table tennis ball is all on the ground and the height coordinate is 0), and the table tennis ball is displayed in the RVIZ (ROS visualization tool) under the ROS platform.
And repeating the processes to obtain a coordinate point set under the robot coordinate system, wherein the coordinate point set consists of all table tennis balls on the ground in the image.
And 4, step 4: and performing coordinate conversion on the table tennis ball under the robot coordinate system according to the conversion relation between the robot coordinate system and the global coordinate system to obtain a coordinate point set of the table tennis ball under the global coordinate system.
The global coordinate system name is map and the robot coordinate system name is base _ link. The base _ link coordinate system at the start is at the origin of the map coordinate system (coordinates are (x)0，y0，th0) The x, y, and th are x-axis coordinates, y-axis coordinates, and angles at which the two coordinate systems are offset, respectively, (0, 0, 0), and the robot moving speed V is (V)x，Vy，Vth)，VxSpeed in the direction of the x-axis, VyRefers to the velocity, V, along the y-axisthRefers to the robot rotation speed. Then after dt times, the base _ link coordinate system has the coordinates relative to the global origin of coordinates:
(x1，y1，th1)＝(x0+(Vx*cos(th0)-Vy*sin(th0))*dt，y0+(Vx*sin(th0)+Vy*cos(th0))*dt，th0+Vth*dt) (5)
the coordinate conversion can be performed according to the coordinate conversion relationship of the base _ link coordinate system obtained by the formula (5). From the origin (x) of the base _ link coordinate system1，y1) And a deflection angle th1The coordinates in the global coordinate system (illustrated by fig. 4) may be calculated. The target table tennis coordinates (X ', Y') in the robot coordinate system base _ link are transformed to coordinates (X ", Y") in the global coordinate system map as:
(X″，Y″)＝(x1+(X′*cos(th1)-Y′*sin(th1))，
y1+(X′*sin(th1)+Y′*cos(th1)))
thus, the whole coordinate conversion work is completed.
And 5: and performing shortest traversal path planning on the obtained table tennis coordinate point set by using an ant colony algorithm to obtain a preliminary ball picking sequence.
Firstly, according to the numbering sequence when detecting the table tennis, calculating the distances between every two table tennis, and combining the distances into a two-dimensional adjacent matrix Graph, wherein the matrix row and column dimensions are the number N of the table tennis detected in the Graph, and the matrix row and column dimensions are used for calculating the shortest path.The elements in the matrix are the distances between the table tennis balls, and for example, the rows i and the columns j represent the distances between the ith ball and the jth ball. With N table tennis balls, the number of elements in the matrix is N2。
As shown in fig. 5, the ant colony algorithm is initialized first with parameters α (pheromone weight), β (visibility weight), ρ (pheromone evaporation rate), Q (pheromone gain coefficient), and initialization pheromone τ0And iteration number N.
Then randomly selecting a ping-pong ball as a starting point for each ant, and adding the starting point to the already passed point set C. The probability P ═ τ of the current point to all points left untouched (set O) is calculated separatelyαWhere τ is the pheromone between two points and γ is the reciprocal of the distance between two points (distance taken from the adjacency matrix graph). The next spot was selected for each ant using a roulette method. Taking a random number q, calculating the probability of a first point, and if the probability is less than q, continuing to calculate the next point; if the probability cumulative sum to a certain point is greater than q, then the point is taken. For example, q is 0.4, p1＝0.2，p2＝0.3，p3＝0.5，(pnRepresenting the probability of selecting the nth point), then since p is1< 0.4, and p1+p2> 0.4, so the second point is taken as the next target. And so on, move all ants to the next target point and add that point to the set that has already passed through (set C).
The above steps are repeated until all points are added to the already passed set. This completes one iteration.
Calculating the path length of each ant according to the selected path, i.e. calculating the distance between ping-pong balls according to the path sequence and summing up, for example, the path sequence is 2-1-3-4 (the serial number is calibrated in the previous step), the path length is the distance d from the ping-pong ball No. 2 to the ping-pong ball No. 11Plus the distance d from ball No. 1 to ball No. 32Plus the distance d from ball No. 3 to ball No. four3I.e. L ═ d1+d2+d3. And updating pheromones of each edge after all ants finish walking: each timeThe pheromone increment between two points on a path is inversely proportional to the length of the path they are on: add Q/Lk(Q is generally 50 to 100), where LkIndicating the length of the kth path and add indicates the increment of the pheromone. The updated pheromone is therefore τ ═ (1- ρ) × τ0+Q/Lk。
Continuing the next iteration using the new pheromone; if (1) the iteration number reaches a set value N (50 times) and is completed, or (2) the pheromone increment add is smaller than a certain threshold (such as 0.01), the approximate optimal solution (with the shortest path length) of all points traversed is output.
Step 6: and removing the ping-pong balls which pass through for many times according to the shape of the robot and the planned path, and further shortening the length of the path.
As shown in fig. 6 (in the global coordinate system map), according to the sequence obtained by the path planning in step 5, it is assumed that the 1 st target table tennis ball moved by the robot is the 2 nd point (the numerical serial number is calibrated according to the table tennis ball sequence identified in step 2, the 0 th point represents the current coordinate point when the robot identifies the table tennis ball, the 2 nd point is the 2 nd (the numerical serial number is calibrated when detecting the table tennis ball), which is the 1 st target point in the path planning), and a straight line b is drawn1、b2And d1、d2，b1、b2Parallel to the line connecting the point 0 and the point 2, and tangent to the minimum circumcircle of the robot at the point 0 (the minimum circle capable of completely enclosing the robot inside is called the minimum circumcircle due to the possible irregular shape of the robot); straight line d1The direction of the connecting line of the two points is vertical and tangent with the tail part of the minimum circumcircle of the robot at the starting point; straight line d2The head of the minimum circumcircle of the robot at the No. 2 point is tangent to the direction vertical to the connecting line of the two points; the rectangle formed by these four lines represents the area swept by the robot from point 0 to point 2 (neglected for the uncovered parts of the four corners of the rectangle for efficiency reasons). All table tennis coordinate points in the rectangle formed by the four lines, i.e. points 4, 7, 15 in fig. 4, are calculated and removed from the coordinate point set, because the robot carries a suction device or a roller ball pick-up device,therefore, the robot can pass through the table tennis balls to pick up the table tennis balls in the process of moving from the point 0 to the point 2, namely the robot can pick up the table tennis balls within the width range of the minimum circumscribed circle radius. Deleting the points reduces redundant traversal points and improves the operation efficiency.
And then taking the point No. 2 as a starting point, selecting a next target point from the path planning in the step 5, and carrying out the same operation on the two points according to the operation processes of the point No. 0 and the point No. 2. And repeating the step 6 by analogy until all the points on the path obtained by calculation in the step 5 are completely processed, and finally obtaining a path without a redundant path, wherein the efficiency of the robot for carrying out ball picking work according to the path is higher than that of the robot running only by using the path in the step 5, and the time is saved.
Accordingly, the present invention also provides a computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by a computing device, cause the computing device to perform the above-described method.
Accordingly, the present invention also provides a computing device comprising,
one or more processors, memory, and one or more programs stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for performing the above-described methods.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The above description is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, several modifications and variations can be made without departing from the technical principle of the present invention, and these modifications and variations should also be regarded as the protection scope of the present invention.
Claims (7)
1. A table tennis ball picking robot operation method based on binocular vision and ant colony algorithm is characterized by comprising the following processes:
establishing a global coordinate system of a table tennis training field;
acquiring RGB images of ground table tennis scattering conditions in a field shot by a binocular camera, detecting table tennis from the RGB images and acquiring coordinate positions of the table tennis;
acquiring depth data of a binocular camera to obtain the distance between each ping-pong ball and the camera;
calculating to obtain a coordinate point set of the table tennis under a global coordinate system according to the coordinate position of each table tennis and the distance from the camera;
carrying out shortest traversal path planning on the coordinate point set by using an ant colony algorithm to obtain a primary ball picking path;
removing the table tennis balls passing through for many times from the primary ball picking path to obtain a path without a redundant route; the method specifically comprises the following steps:
according to the preliminary ball picking path, assuming that the robot moves from the starting point to the 1 st target table tennis,
draw a straight line b1、b2And d1、d2，b1、b2The minimum circumcircle of the robot at the starting point is tangent to the connecting line of the starting point and the target table tennis; straight line d1The direction of the connecting line of the two points is vertical and tangent with the tail part of the minimum circumcircle of the robot at the starting point; straight line d2The direction of the connecting line of the two points is vertical and tangent to the head of the minimum circumcircle of the robot at the target point; the rectangle formed by the four lines represents the area swept by the robot from the departure point to the target point, all table tennis coordinate points in the rectangle formed by the four lines are calculated and obtained, the points are deleted from the coordinate point set,
then taking the target point as a starting point, selecting the next target point from the primary ball picking path, and carrying out the same operation; and repeating the steps until all the points on the ball picking path are processed, and finally obtaining a path without redundant routes.
2. The operation method of the table tennis ball picking robot based on binocular vision and ant colony algorithm as claimed in claim 1, wherein the specific process of detecting the table tennis balls from the RGB images and obtaining the coordinate positions of the table tennis balls comprises:
converting the RGB image into an HSV image;
converting the HSV image into a gray scale image, and performing Gaussian blur;
carrying out contour detection by using a canny operator to obtain the contours of all objects in the image;
and detecting circles in the graph by using Hough circle transformation, namely detecting the table tennis balls in the field, and calculating to obtain coordinate positions of all the table tennis balls in the graph.
3. The operation method of the table tennis ball picking robot based on the binocular vision and the ant colony algorithm as claimed in claim 1, wherein the step of calculating the coordinate point set of the table tennis balls under the global coordinate system according to the coordinate positions of the table tennis balls and the distance of the distance camera comprises the following steps:
converting the coordinate position of each table tennis ball and the distance from the camera into a three-dimensional coordinate system of the camera to obtain a coordinate point set of the table tennis balls in a three-dimensional coordinate system of the camera;
then converting the coordinates of the table tennis under the camera three-dimensional coordinate system into the robot coordinate system;
and finally, converting the coordinates of the table tennis under the robot coordinate system into a global coordinate system to obtain a coordinate point set of the table tennis under the global coordinate system.
4. The operational method of the ping-pong ball picking robot based on binocular vision and ant colony algorithm as claimed in claim 3, wherein the process of converting the coordinate position of each ping-pong ball and the distance of the range camera into the three-dimensional coordinate system of the camera comprises:
establishing a camera three-dimensional coordinate system XYZ and an image plane coordinate system xy;
the mapping relation between the camera three-dimensional coordinate system midpoint P (X, Y, Z) and the image plane coordinate system midpoint P (X, Y) is as follows:
wherein f is the focal length of the camera, Z is the depth of the table tennis ball obtained from the depth data, X represents the X-axis coordinate of the image plane coordinate system, and X represents the X-axis coordinate of the three-dimensional coordinate system of the camera; y represents a Y-axis coordinate in an image plane coordinate system, and Y represents a Y-axis coordinate in a camera three-dimensional coordinate system; c. CxAs the coordinate of the principal point on the x-axis, cyAs the coordinate of the principal point on the y-axis, fxIs the dimension of the pixel of f on the x-axis, fyIs the pixel dimension of f on the y-axis;
the coordinate position of each table tennis is the coordinate in the image plane coordinate system, the distance from the camera is the Z-axis numerical value in the camera coordinate system, and the three-dimensional coordinate P (X, Y, Z) of the table tennis in the camera coordinate system is calculated according to the formula.
5. The operation method of the ping-pong ball picking robot based on binocular vision and ant colony algorithm as claimed in claim 3, wherein the process of converting the coordinates of the ping-pong ball under the robot coordinate system to the global coordinate system comprises:
the coordinate system of the robot at the beginning is the origin of the global coordinate system, and the coordinate is (x)0，y0，th0) Let (0, 0, 0) and set the robot movement speed V ═ Vx，Vy，Vth)，VxSpeed in the direction of the x-axis, VyRefers to the velocity, V, along the y-axisthThe rotation speed of the robot is indicated; then after dt times, the coordinates of the robot coordinate system with respect to the global coordinate origin are:
(x1，y1，th1)＝(x0+(Vx*cos(th0)-Vy*sin(th0))*dt，y0+(Vx*sin(th0)+Vy*costhO*dt，thO+Vth*dt) (5)
according to the coordinate transformation relation of the robot coordinate system obtained by the formula (5), coordinate transformation can be performed, and the target table tennis coordinate (X ', Y') in the robot coordinate system is transformed into the coordinate (X ", Y") in the global coordinate system as follows:
(X″，Y″)＝(x1+(X′*cos(th1)-Y′*sin(th1))，y1+(X′*sin(th1)+Y′*cos(th1)))
and completing the coordinate conversion of all table tennis balls to obtain a coordinate point set of the table tennis balls in the global coordinate system.
6. A computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by a computing device, cause the computing device to perform the method of operation of a ping pong robot based on binocular vision and ant colony algorithm of any one of claims 1-5.
7. A computing device, comprising, in combination,
one or more processors, memory, and one or more programs stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for performing the method of operation of the ping pong robot based on binocular vision and ant colony algorithm of any of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910744905.2A CN110433467B (en) | 2019-08-13 | 2019-08-13 | Operation method and device of table tennis ball picking robot based on binocular vision and ant colony algorithm |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910744905.2A CN110433467B (en) | 2019-08-13 | 2019-08-13 | Operation method and device of table tennis ball picking robot based on binocular vision and ant colony algorithm |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110433467A CN110433467A (en) | 2019-11-12 |
| CN110433467B true CN110433467B (en) | 2021-02-12 |
Family
ID=68435030
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910744905.2A Active CN110433467B (en) | 2019-08-13 | 2019-08-13 | Operation method and device of table tennis ball picking robot based on binocular vision and ant colony algorithm |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110433467B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111530041A (en) * | 2020-04-24 | 2020-08-14 | 贵州民族大学 | Robot ball picking method and system |
| CN111744161A (en) * | 2020-07-29 | 2020-10-09 | 哈尔滨理工大学 | Table tennis falling point detection and edge ball wiping judgment system |
| CN112587884B (en) * | 2020-12-24 | 2022-02-01 | 烟台南山学院 | Full-automatic ball picking robot ball picking method based on monocular vision |
| CN114879669A (en) * | 2022-04-28 | 2022-08-09 | 贵州民族大学 | Robot control method based on multi-target path planning and intelligent picking |
| CN114911270A (en) * | 2022-06-20 | 2022-08-16 | 南京信息工程大学 | Unmanned aerial vehicle coverage path planning method based on parallel self-adaptive ant colony algorithm |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107137890A (en) * | 2016-09-27 | 2017-09-08 | 五邑大学 | A kind of intelligent tennis ball pickup robot based on image recognition |
| CN107767423A (en) * | 2017-10-10 | 2018-03-06 | 大连理工大学 | A kind of mechanical arm target positioning grasping means based on binocular vision |
| CN108579030A (en) * | 2018-04-09 | 2018-09-28 | 宁波乔克兄弟三维科技有限公司 | A kind of robot of view-based access control model picks up ball method |
| CN108786041A (en) * | 2018-07-12 | 2018-11-13 | 贵州理工学院 | Intelligent ball collecting control method and intelligent ball collecting robot |
| CN109333549A (en) * | 2018-10-25 | 2019-02-15 | 湖南大学 | A kind of ping-pong pickup mobile robot and control method based on machine vision |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9216323B2 (en) * | 2011-03-25 | 2015-12-22 | Eric Davis Schwartz | Retrieving device |
-
2019
- 2019-08-13 CN CN201910744905.2A patent/CN110433467B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107137890A (en) * | 2016-09-27 | 2017-09-08 | 五邑大学 | A kind of intelligent tennis ball pickup robot based on image recognition |
| CN107767423A (en) * | 2017-10-10 | 2018-03-06 | 大连理工大学 | A kind of mechanical arm target positioning grasping means based on binocular vision |
| CN108579030A (en) * | 2018-04-09 | 2018-09-28 | 宁波乔克兄弟三维科技有限公司 | A kind of robot of view-based access control model picks up ball method |
| CN108786041A (en) * | 2018-07-12 | 2018-11-13 | 贵州理工学院 | Intelligent ball collecting control method and intelligent ball collecting robot |
| CN109333549A (en) * | 2018-10-25 | 2019-02-15 | 湖南大学 | A kind of ping-pong pickup mobile robot and control method based on machine vision |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110433467A (en) | 2019-11-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110433467B (en) | Operation method and device of table tennis ball picking robot based on binocular vision and ant colony algorithm | |
| CN111325843B (en) | Real-time semantic map construction method based on semantic inverse depth filtering | |
| CN110377015B (en) | Robot positioning method and robot positioning device | |
| WO2020024234A1 (en) | Route navigation method, related device, and computer readable storage medium | |
| WO2020134254A1 (en) | Method employing reinforcement learning to optimize trajectory of spray painting robot | |
| CN110634161B (en) | Rapid high-precision estimation method and device for workpiece pose based on point cloud data | |
| CN109186606B (en) | Robot composition and navigation method based on SLAM and image information | |
| WO2020164092A1 (en) | Image processing method and apparatus, moveable platform, unmanned aerial vehicle and storage medium | |
| CN103678754B (en) | Information processor and information processing method | |
| CN108898676B (en) | Method and system for detecting collision and shielding between virtual and real objects | |
| CN106826833A (en) | Independent navigation robot system based on 3D solid cognition technologies | |
| WO2021237667A1 (en) | Dense height map construction method suitable for legged robot planning | |
| CN110827353B (en) | Robot positioning method based on monocular camera assistance | |
| CN113485350A (en) | Robot movement control method, device, equipment and storage medium | |
| CN110705385B (en) | Method, device, equipment and medium for detecting angle of obstacle | |
| CN106909149B (en) | Method and device for avoiding obstacles by depth camera | |
| CN112102342A (en) | Plane contour recognition method and device, computer equipment and storage medium | |
| CN111401190A (en) | Vehicle detection method, device, computer equipment and storage medium | |
| CN113160401B (en) | Object-oriented visual SLAM lightweight semantic map creation method | |
| CN111198563B (en) | Terrain identification method and system for dynamic motion of foot type robot | |
| CN115542896A (en) | Robot path generation method, system and storage medium | |
| CN115131407A (en) | Robot target tracking method, device and equipment for digital simulation environment | |
| Takegami et al. | Environment map generation in forest using field robot | |
| Wang et al. | Development of a vision system and a strategy simulator for middle size soccer robot | |
| Goll et al. | Construction of the local patency map on the data from Velodyne LiDAR |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| EE01 | Entry into force of recordation of patent licensing contract | ||
| EE01 | Entry into force of recordation of patent licensing contract |
Application publication date: 20191112 Assignee: NANJING NANYOU INSTITUTE OF INFORMATION TECHNOVATION Co.,Ltd. Assignor: NANJING University OF POSTS AND TELECOMMUNICATIONS Contract record no.: X2021980014141 Denomination of invention: Operation method and equipment of table tennis picking robot based on binocular vision and ant colony algorithm Granted publication date: 20210212 License type: Common License Record date: 20211206 |
|
| EC01 | Cancellation of recordation of patent licensing contract | ||
| EC01 | Cancellation of recordation of patent licensing contract |
Assignee: NANJING NANYOU INSTITUTE OF INFORMATION TECHNOVATION Co.,Ltd. Assignor: NANJING University OF POSTS AND TELECOMMUNICATIONS Contract record no.: X2021980014141 Date of cancellation: 20231107 |