CN113814986A - Method and system for controlling SCARA robot based on machine vision - Google Patents
Method and system for controlling SCARA robot based on machine vision Download PDFInfo
- Publication number
- CN113814986A CN113814986A CN202111394916.6A CN202111394916A CN113814986A CN 113814986 A CN113814986 A CN 113814986A CN 202111394916 A CN202111394916 A CN 202111394916A CN 113814986 A CN113814986 A CN 113814986A
- Authority
- CN
- China
- Prior art keywords
- robot
- image
- key frame
- motion
- scara robot
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Programme-controlled manipulators
- B25J9/16—Programme controls
- B25J9/1694—Programme controls characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion
- B25J9/1697—Vision controlled systems
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Programme-controlled manipulators
- B25J9/16—Programme controls
- B25J9/1602—Programme controls characterised by the control system, structure, architecture
- B25J9/1605—Simulation of manipulator lay-out, design, modelling of manipulator
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Programme-controlled manipulators
- B25J9/16—Programme controls
- B25J9/1656—Programme controls characterised by programming, planning systems for manipulators
- B25J9/1664—Programme controls characterised by programming, planning systems for manipulators characterised by motion, path, trajectory planning
Landscapes
- Engineering & Computer Science (AREA)
- Robotics (AREA)
- Mechanical Engineering (AREA)
- Automation & Control Theory (AREA)
- Manipulator (AREA)
Abstract
The invention discloses a method and a system for controlling a SCARA robot based on machine vision, which relate to the technical field of robots, and the method comprises the following steps: the robot controller collects a video data stream of an article to be processed based on a visual sensor; the upper computer extracts key frame images from the video data stream based on a preset time interval, performs visual tracking processing on the key frame images, and generates mechanical arm motion planning information based on a visual tracking processing result; the robot controller generates a motion path and a track instruction based on the mechanical arm motion planning information; and the programmable logic controller analyzes motion control instructions corresponding to the four joint servo drivers based on the motion path and the track instructions and sends the corresponding motion control instructions to the corresponding joint servo drivers. The invention adopts machine vision to extract the video data stream of the object to be processed, and realizes the automatic processing process of the object to be processed.
Description
Technical Field
The invention relates to the technical field of robots, in particular to a method and a system for controlling an SCARA robot based on machine vision.
Background
SCARA (selective Compliance Assembly Robot arm), namely a cylindrical coordinate type industrial Robot with four degrees of freedom and a built-in four-axis servo motor, a harmonic reducer and a brake system. The SCARA robot has four degrees of freedom including two horizontal joints and a link that can move both vertically and rotate. The first degree of freedom and the second degree of freedom are rotary joints formed by a large mechanical arm and a small mechanical arm, so that the horizontal connecting rod can perform rotary motion in a horizontal plane to complete quick and accurate positioning in the plane; the third degree of freedom is a movable joint with a vertical lifting function and can complete the motion vertical to the plane; the fourth degree of freedom is a rotary joint, so that the end effector can conveniently grab the target. Because the SCARA robot can guarantee stronger rigidity and higher precision in the vertical direction, and can freely and quickly rotate in the horizontal plane, the SCARA robot is very suitable for carrying out plane positioning and finishing the tasks of sorting and grabbing workpieces. The actuator parts of four joints of the SCARA robot use alternating current servo motors, can provide large torque, and has good execution effect on the movement speed and precision of a mechanical arm. The first and second joints of the robot are decelerated based on harmonics, while the third and fourth joints are decelerated based on synchronous belts. The first joint and the second joint rotate respectively to realize the movement of the end effector in a horizontal plane, the third joint connects the motor with the ball screw, so that the vertical movement of the mechanical connecting rod can be realized, and the fourth joint connects the motor with the screw through a spline, so that the screw can complete corresponding rotation action.
At present, machine vision is widely applied to an industrial robot control system, and the existing machine vision is realized in a mode of acquiring images and performing vision analysis instead of a mode of acquiring video data streams, so that a robot controller cannot analyze and utilize the video data streams, and a more accurate vision control means cannot be developed in an SCARA robot.
Disclosure of Invention
The invention aims to overcome the defects of the prior art and provides a method and a system for controlling a SCARA robot based on machine vision.
In order to solve the above problems, the present invention proposes a method for controlling a SCARA robot based on machine vision, the method comprising the steps of:
the robot controller collects a video data stream of an article to be processed based on a visual sensor;
the robot controller sends the video data stream to an upper computer;
the upper computer extracts key frame images of the video data stream based on a preset time interval, performs visual tracking processing on the key frame images, and generates mechanical arm motion planning information based on a visual tracking processing result;
the upper computer sends the mechanical arm motion planning information to the robot controller;
the robot controller generates a motion path and track instruction based on the mechanical arm motion planning information and sends the motion path and track instruction to the programmable logic controller;
and the programmable logic controller analyzes motion control instructions corresponding to the four joint servo drivers based on the motion path and the track instructions and sends the corresponding motion control instructions to the corresponding joint servo drivers.
The upper computer extracts key frame images from the video data stream based on a preset time interval, and the visual tracking processing of the key frame images comprises the following steps:
extracting a key frame image based on a preset time interval, then extracting an image block of a target area of the key frame image to serve as a target template, and carrying out relevant matching on the target template and all possible positions in the candidate image, wherein the position with the highest matching degree is the position of the target.
The extracting of the image block of the target area of the key frame image as the target template comprises:
extracting an interested area in the key frame image, and then carrying out contour extraction on an article to be processed in the interested area to obtain a standard contour of a target template, wherein the interested area is an area where image elements of the article to be processed change after work movement on the area where a target is initially located;
and extracting image blocks in the standard contour area based on the standard contour to form a target template.
The extracting of the region of interest in the key frame image and then the contour extraction of the region of interest to obtain the standard contour of the target template comprises:
acquiring a key frame image, and intercepting a rectangular region of interest by using a rectangular interception function; then enhancing the image of the region of interest; and finally, creating an image template under the target template through a template creating function to obtain a standard contour of the image template after binaryzation.
The step of creating the image template under the target template through the template creating function to obtain the standard contour after the binarization of the image template comprises the following steps:
searching the gray gradient of the image of the region of interest, processing the image of the region of interest by using a Sobel operator, and returning the pixel gray value in the X direction and the pixel gray value in the T direction;
searching local maximum values in the gradient direction by using a non-maximum value inhibition algorithm, inhibiting non-maximum value elements and refining edges;
and extracting and connecting edges by adopting a dual-threshold algorithm to obtain a standard outline of the image template.
The robot controller generates a motion path and track instruction based on the mechanical arm motion planning information, and the motion path and track instruction comprises the following steps:
the robot controller carries out track planning on the SCARA robot in a Cartesian space based on the mechanical arm motion planning information to obtain the corresponding relation between the Cartesian space track and joint angles, angular speeds and angular accelerations in each joint space;
performing dynamic modeling on the SCARA robot according to the SCARA robot model parameters and a dynamic analysis method to obtain a dynamic equation of the SCARA robot;
determining the output track state of the robot at each moment through a kinetic equation, outputting the parameter dimension of each joint space to obtain the vector representation of each moment, wherein the parameter dimension comprises joint angle position and joint angular velocity, and simultaneously giving out the vector representation of an expected track state unit on the basis of the vector dimension of the output track state unit;
generating a motion path and trajectory instruction based on the parameter dimensions and a vector representation of the desired trajectory state unit.
The motion control instructions corresponding to the four joint servo drivers are analyzed by the programmable logic controller based on the motion path and the track instructions, and the motion control instructions comprise:
the programmable logic controller analyzes the control parameters of each servo driver and then sends the control parameters to the corresponding servo driver according to the corresponding address relation of the servo driver.
Correspondingly, the invention also provides a control system of the SCARA robot, which comprises the following components:
the visual sensor is used for acquiring a video data stream of an article to be processed;
the robot controller is used for acquiring a video data stream of an article to be processed based on the visual sensor and sending the video data stream to the upper computer; generating a motion path and track instruction based on the mechanical arm motion planning information, and sending the motion path and track instruction to a programmable logic controller;
the upper computer is used for extracting key frame images from the video data stream based on a preset time interval, performing visual tracking processing on the key frame images and generating mechanical arm motion planning information based on a visual tracking processing result; sending the mechanical arm motion planning information to the robot controller;
and the programmable logic controller is used for analyzing motion control instructions corresponding to the four joint servo drivers based on the motion path and the track instructions and sending the corresponding motion control instructions to the corresponding joint servo drivers.
The upper computer extracts the key frame images based on a preset time interval, then uses image blocks of a target area of the key frame images as target templates, and carries out relevant matching on the target templates and all possible positions in the candidate images, wherein the position with the highest matching degree is the position where the target is located.
And the upper computer and the robot controller are communicated through a multi-point interface cable.
According to the SCARA robot control system, the video data stream is acquired by adopting the visual sensor, and the video data stream is uploaded to the upper computer to analyze and process the video image, so that the control system of the SCARA robot can extract the key frame image in combination with the video data stream, and can trigger the mechanical arm movement of the SCARA robot in combination with the key frame image to match the corresponding mechanical arm movement planning information, and the application combination of the video stream acquisition can be well suitable for the control link of the SCARA robot. The programmable logic controller is used for analyzing and distributing motion control instructions to the four joint servo drivers to control the production process of the SCARA robot, so that the reliability of the control of the tail end of the whole SCARA robot is higher.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a schematic structural diagram of a control system of a SCARA robot in an embodiment of the present invention;
FIG. 2 is a state change diagram of extracting image blocks of a target area of a key frame image as a target template in an embodiment of the present invention;
fig. 3 is a flowchart of a method for controlling a SCARA robot based on machine vision in an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Specifically, fig. 1 shows a schematic structural diagram of a control system of a SCARA robot in an embodiment of the present invention, where the control system includes: host computer, robot controller, programmable logic controller, visual sensor, first servo driver, second servo driver, third servo driver, fourth servo driver, first servo motor, second servo motor, third servo motor, fourth servo motor, first decoder, second decoder, third decoder and fourth decoder etc. wherein: the host computer can be a PC or the like, the host computer and the robot controller carry out data communication, the robot controller carries out data communication with the programmable logic controller, the robot controller carries out data communication with the vision sensor, the programmable logic controller carries out data communication with the first servo driver, the second servo driver, the third servo driver and the fourth servo driver, the first servo driver carries out data communication with the first servo motor and the first decoder, the second servo driver carries out data communication with the second servo motor and the second decoder, the third servo driver carries out data communication with the third servo motor and the third decoder, and the fourth servo driver carries out data communication with the fourth servo motor and the fourth decoder.
The vision sensor is arranged at a corresponding position of an application scene of the SCARA robot and can acquire a video data stream of an article to be processed, the vision sensor and the robot controller can communicate by adopting an RS485 communication protocol, the robot controller acquires the video data stream of the article to be processed based on the vision sensor, and the robot controller transmits the video data stream to an upper computer; the upper computer extracts key frame images from the video data stream based on a preset time interval, performs visual tracking processing on the key frame images, and generates mechanical arm motion planning information based on a visual tracking processing result; the upper computer sends the mechanical arm motion planning information to the robot controller; the robot controller generates a motion path and track instruction based on the mechanical arm motion planning information and sends the motion path and track instruction to the programmable logic controller; and the programmable logic controller analyzes motion control instructions corresponding to the four joint servo drivers based on the motion path and the track instructions and sends the corresponding motion control instructions to the corresponding joint servo drivers.
Specifically, after the upper computer acquires a video data stream, extracting a key frame image based on a preset time interval, taking an image block of a target area of the key frame image as a target template, performing relevant matching on the target template and all possible positions in candidate images, wherein the place with the highest matching degree is the position where the target is located, and the upper computer can match the current working state of the SCARA robot based on the visual tracking processing so as to match the next working state, thereby producing mechanical arm motion planning information based on the next working state. The time interval is measured in milliseconds, and the preset time interval is any value between 1ms and 100 ms.
The candidate images are stored in the upper computer, the candidate images are associated with historical working states of the SCARA robot, and a plurality of working images associated with the historical working states form a candidate image set to be stored in a storage space of the upper computer; the candidate images are related to simulation working states of the SCARA robot, working images related to the simulation working states form a candidate image set and are stored in a storage space of an upper computer, key frame images are extracted according to a time relation, the key frame images are subjected to a matching process in a mode of firstly extracting and firstly processing, namely, a first key frame image at a first moment is firstly subjected to related matching with the candidate images, a second key frame image at a second moment is then subjected to related matching with the candidate images, and mechanical arm motion planning information at different moments is generated according to the sequence of a time axis.
Fig. 2 shows a state change diagram of extracting image blocks of a target region of a key frame image as a target template in an embodiment of the present invention, where the left side is an original image, the middle is an image under ROI extraction, and the right side is a final target template, where extracting image blocks of the target region of the key frame image as the target template includes: extracting a region of interest (ROI) in the key frame image, and then extracting the outline of an article to be processed in the ROI to obtain a standard outline of a target template, wherein the ROI can be set based on a motion region of a robot on the article to be processed; and extracting image blocks in the standard contour area based on the standard contour to form a target template.
It should be noted that, here, extracting a region of interest in the key frame image, and then performing contour extraction on the region of interest to obtain a standard contour of the target template includes: acquiring a key frame image, and intercepting a rectangular region of interest by using a rectangular interception function; then enhancing the image of the region of interest; and finally, creating an image template under the target template through a template creating function to obtain a standard contour of the image template after binaryzation.
It should be noted that, here, creating an image template under a target template through a template creation function, and obtaining a standard contour after binarization of the image template includes: (1) and (3) searching the gray gradient of the image of the ROI, and processing the image by using a Sobel operator to return the pixel gray value in the X direction and the pixel gray value in the T direction. (2) Local maxima in the gradient direction are searched using a non-maximum suppression algorithm (NMS), non-maximum elements are suppressed, and edges can be refined. The non-maximum suppression algorithm tracks the left and right pixels in the edge direction, and if the gray value of the current pixel is smaller than the gray values of the left and right pixels, the gray value of the current pixel is suppressed. (3) And extracting and connecting edges by adopting a dual-threshold algorithm to obtain a standard outline of the image template. A small number of non-edge pixels may be included in the result after processing by the non-maxima suppression algorithm, so a cut-off is made by selecting a threshold. The dual-threshold selection and edge connection method assumes that one of two thresholds is a high threshold TH and the other is a low threshold TL, and then: for any edge pixel below TL, discard; for any edge pixel above TH, then hold; any edge pixel value between TL and TH, if edge-connectable one pixel is larger than TH and all pixels of the edge are larger than the minimum threshold TL, then retained, otherwise discarded; and storing the final edge image.
It should be noted that the upper computer and the robot controller can communicate with each other through a multi-point interface (MPI) cable, and the robot controller sends the mechanical arm movement planning information to the robot controller, and the robot controller generates a movement path and a trajectory instruction based on the mechanical arm movement planning information.
Specifically, the generating of the motion path and the trajectory instruction by the robot controller based on the mechanical arm motion planning information includes: and the robot controller performs track planning on the SCARA robot in a Cartesian space to obtain the corresponding relation between the Cartesian space track and joint angles, angular speeds and angular accelerations in each joint space. It should be noted that the motion path and trajectory instructions may enable the robot end effector to reach a set trajectory as much as possible. The trajectory of a robot is generally divided into a basic trajectory plan and an optimal trajectory plan. The basic trajectory planning is carried out in joint space and Cartesian coordinate space, and the kinematics and dynamics constraints of the robot must be considered during planning. For joint space trajectory planning, the planning object is the angle of each joint of the robot, the value generated by the function is the joint value, and the value generated by the Cartesian coordinate space trajectory planning function is the pose of the end hand of the robot hand. The Cartesian coordinate space trajectory is expressed in a common rectangular coordinate space and is very intuitive, but the Cartesian coordinate space trajectory has large calculation amount and can obtain the calculation accuracy of a similar joint space only by needing higher processing speed.
Specifically, the robot controller sends the motion path and the track instruction to the programmable logic controller, and the programmable logic controller analyzes the control parameter of each servo driver and then sends the control parameter to the corresponding servo driver according to the corresponding address relation of the servo driver. After receiving the motion path and track instructions, the programmable logic controller completes the control work of the advancing direction, calculates to obtain the rotating speed and the steering direction to be executed by the four servo motors, and controls different servo drivers to realize the control, so that the SCARA robot completes the corresponding operation process.
The control system of the SCARA robot adopts the visual sensor to collect the video data stream, the video data stream is uploaded to the upper computer to be analyzed and processed by the video image, the control system of the SCARA robot can extract the key frame image by combining the video data stream, the corresponding robot arm motion planning information can be matched by combining the key frame image, the mechanical arm motion of the SCARA robot is triggered, and the application combination of the video stream collection can be better applied to the control link of the SCARA robot. The programmable logic controller is used for analyzing and distributing motion control instructions to the four joint servo drivers to control the production process of the SCARA robot, so that the reliability of the control of the tail end of the whole SCARA robot is higher.
Fig. 3 shows a flow chart of a method for controlling a SCARA robot based on machine vision in an embodiment of the invention, the method comprising the steps of:
s21, acquiring a video data stream of the to-be-processed object based on the visual sensor by the robot controller;
s22, the robot controller sends the video data stream to an upper computer;
it should be noted that the vision sensor is arranged at a corresponding position of an application scene of the SCARA robot, and can acquire a video data stream of an article to be processed, the vision sensor and the robot controller can communicate with each other by adopting an RS485 communication protocol, the robot controller acquires the video data stream of the article to be processed based on the vision sensor, and the robot controller transmits the video data stream to an upper computer.
S23, the upper computer extracts key frame images from the video data stream based on a preset time interval, performs visual tracking processing on the key frame images, and generates mechanical arm motion planning information based on a visual tracking processing result;
after the upper computer obtains the video data stream, extracting key frame images based on a preset time interval, taking image blocks of a target area of the key frame images as a target template, carrying out relevant matching on the target template and all possible positions in candidate images, wherein the place with the highest matching degree is the position where the target is located, and the upper computer can match the current working state of the SCARA robot based on the visual tracking processing so as to match the next working state, thereby producing mechanical arm motion planning information based on the next working state. The time interval is measured in milliseconds, and the preset time interval is any value between 1ms and 100 ms.
It should be noted that, here, extracting the image block of the target area of the key frame image as the target template includes: extracting a region of interest (ROI) in the key frame image, and then carrying out contour extraction on an article to be processed in the ROI to obtain a standard contour of a target template, wherein the ROI is an area where image elements of the article to be processed change after work motion on the area where a target is initially located; and extracting image blocks in the standard contour area based on the standard contour to form a target template. The target area is extracted in an ROI mode, then the extraction processing of the image block of the area where the article to be processed is located is completed through contour extraction, and finally the corresponding target template is obtained.
It should be noted that, here, creating an image template under a target template through a template creation function, and obtaining a standard contour after binarization of the image template includes: (1) and (3) searching the gray gradient of the image of the ROI, and processing the image by using a Sobel operator to return the pixel gray value in the X direction and the pixel gray value in the T direction. (2) Local maxima in the gradient direction are searched using a non-maximum suppression algorithm (NMS), non-maximum elements are suppressed, and edges can be refined. The non-maximum suppression algorithm tracks the left and right pixels in the edge direction, and if the gray value of the current pixel is smaller than the gray values of the left and right pixels, the gray value of the current pixel is suppressed. (3) And extracting and connecting edges by adopting a dual-threshold algorithm to obtain a standard outline of the image template. A small number of non-edge pixels may be included in the result after processing by the non-maxima suppression algorithm, so a cut-off is made by selecting a threshold. The dual-threshold selection and edge connection method assumes that one of two thresholds is a high threshold TH and the other is a low threshold TL, and then: for any edge pixel below TL, discard; for any edge pixel above TH, then hold; any edge pixel value between TL and TH, if edge-connectable one pixel is larger than TH and all pixels of the edge are larger than the minimum threshold TL, then retained, otherwise discarded; and storing the final edge image.
It should be noted that, here, performing correlation matching on the target template and all possible positions in the candidate image includes: and performing a correlation matching process on the candidate image by adopting a BP neural network. The constant training of the neural network with a large number of data samples, which include both training inputs and desired outputs, is a prerequisite for the identification of different targets using the BP neural network. The training input data is often some feature quantity of the image, and the expected output can be set as the corresponding code number of the image.
Specifically, the BP neural network belongs to a multilayer forward type network, and signals thereof are propagated forward and errors are propagated backward. The standard BP neural network adopts a gradient descent method, and the weight of the network is reversely adjusted along the gradient of a performance function. The basic idea of training a feedforward network using the BP learning algorithm is: firstly, calculating an actual output value of input features in a training sample after the input features pass through a forward network through a forward process; then comparing the output value with the expected value to obtain an error signal, and adjusting the connection strength between each neuron from back to front according to the error signal; repeating the calculation of the forward process again to reduce the error, and comparing the new output value with the expected value to obtain a new error signal smaller than the original error signal; and then adjusting the connection strength between the neurons of the neural network layer from back to front according to a small error signal, wherein the forward process and the backward process are continuously carried out for multiple times until the error meets the requirement.
After the visual tracking processing of the key frame image is completed, a candidate image corresponding to the current key frame image can be obtained, the candidate image is associated with the current motion state of the SCARA robot and the next motion state of the SCARA robot, namely, the upper computer can match the current working state of the SCARA robot based on the visual tracking processing so as to match the working state required by the next step, and therefore mechanical arm motion planning information is produced based on the current working state and the next working state.
S24, the upper computer sends the mechanical arm motion planning information to the robot controller;
the upper computer and the robot controller can be communicated through a multi-point interface (MPI) cable, mechanical arm motion planning information is sent to the robot controller, and the robot controller generates motion path and track instructions based on the mechanical arm motion planning information.
S25, the robot controller generates a motion path and track instruction based on the mechanical arm motion planning information, and sends the motion path and track instruction to the programmable logic controller;
specifically, the generating of the motion path and the trajectory instruction by the robot controller based on the mechanical arm motion planning information includes: and the robot controller performs track planning on the SCARA robot in a Cartesian space to obtain the corresponding relation between the Cartesian space track and joint angles, angular speeds and angular accelerations in each joint space. It should be noted that the motion path and trajectory instructions may enable the robot end effector to reach a set trajectory as much as possible. The trajectory of a robot is generally divided into a basic trajectory plan and an optimal trajectory plan. The basic trajectory planning is carried out in joint space and Cartesian coordinate space, and the kinematics and dynamics constraints of the robot must be considered during planning. For joint space trajectory planning, the planning object is the angle of each joint of the robot, the value generated by the function is the joint value, and the value generated by the Cartesian coordinate space trajectory planning function is the pose of the end hand of the robot hand. The Cartesian coordinate space trajectory is expressed in a common rectangular coordinate space and is very intuitive, but the Cartesian coordinate space trajectory has large calculation amount and can obtain the calculation accuracy of a similar joint space only by needing higher processing speed.
Specifically, the robot controller performs track planning on the SCARA robot in a Cartesian space to obtain the corresponding relation between the Cartesian space track and joint angles, angular velocities and angular accelerations in each joint space; performing dynamic modeling on the SCARA robot according to the SCARA robot model parameters and a dynamic analysis method to obtain a dynamic equation of the SCARA robot; determining the output track state of the robot at each moment through a kinetic equation, outputting the parameter dimension of each joint space to obtain the vector representation of each moment, wherein the parameter dimension comprises joint angle position and joint angular velocity, and simultaneously giving out the vector representation of an expected track state unit on the basis of the vector dimension of the output track state unit; generating a motion path and trajectory instruction based on the parameter dimensions and a vector representation of the desired trajectory state unit.
Specifically, the motion trajectory of the tail end of the SCARA robot controlled by the improved quadratic iterative learning is adopted, so that the tracking error of the tail end is reduced, and the expected trajectory is accurately tracked.
And S26, the programmable logic controller analyzes motion control instructions corresponding to the four joint servo drivers based on the motion path and the track instructions, and sends the corresponding motion control instructions to the corresponding joint servo drivers.
Specifically, the robot controller sends the motion path and the track instruction to the programmable logic controller, and the programmable logic controller analyzes the control parameter of each servo driver and then sends the control parameter to the corresponding servo driver according to the corresponding address relation of the servo driver. After receiving the motion path and track instructions, the programmable logic controller completes the control work of the advancing direction, calculates to obtain the rotating speed and the steering direction to be executed by the four servo motors, and controls different servo drivers to realize the control, so that the SCARA robot completes the corresponding operation process.
According to the embodiment of the invention, the video data stream is acquired by adopting the visual sensor, and the video data stream is uploaded to the upper computer for analyzing and processing the video image, so that the control system of the SCARA robot can extract the key frame image by combining with the video data stream, and the mechanical arm motion planning information corresponding to the key frame image matching can be combined to trigger the mechanical arm motion of the SCARA robot, so that the application and combination of the video stream acquisition can be better suitable for the control link of the SCARA robot. The programmable logic controller is used for analyzing and distributing motion control instructions to the four joint servo drivers to control the production process of the SCARA robot, so that the reliability of the control of the tail end of the whole SCARA robot is higher.
The above embodiments of the present invention are described in detail, and the principle and the implementation of the present invention are described herein by using specific embodiments, and the description of the above embodiments is only used to help understanding the method of the present invention and the core idea thereof; meanwhile, for a person skilled in the art, according to the idea of the present invention, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present invention.
Claims (10)
1. A method for controlling a SCARA robot based on machine vision, the method comprising the steps of:
the robot controller collects a video data stream of an article to be processed based on a visual sensor;
the robot controller sends the video data stream to an upper computer;
the upper computer extracts key frame images of the video data stream based on a preset time interval, performs visual tracking processing on the key frame images, and generates mechanical arm motion planning information based on a visual tracking processing result;
the upper computer sends the mechanical arm motion planning information to the robot controller;
the robot controller generates a motion path and track instruction based on the mechanical arm motion planning information and sends the motion path and track instruction to the programmable logic controller;
and the programmable logic controller analyzes motion control instructions corresponding to the four joint servo drivers based on the motion path and the track instructions and sends the corresponding motion control instructions to the corresponding joint servo drivers.
2. The method for controlling a SCARA robot based on machine vision according to claim 1, wherein the upper computer extracts key frame images based on a preset time interval from the video data stream, and the performing the visual tracking process on the key frame images comprises:
extracting a key frame image based on a preset time interval, then extracting an image block of a target area of the key frame image to serve as a target template, and carrying out relevant matching on the target template and all possible positions in the candidate image, wherein the position with the highest matching degree is the position of the target.
3. The method for controlling a SCARA robot based on machine vision according to claim 2, wherein said extracting image blocks of the target area of the key frame image as the target template comprises:
extracting an interested area in the key frame image, and then extracting the outline of the article to be processed in the interested area to obtain a standard outline of the target template, wherein the interested area is set based on the motion area of the robot on the article to be processed;
and extracting image blocks in the standard contour area based on the standard contour to form a target template.
4. The method for controlling a SCARA robot based on machine vision according to claim 3, wherein the extracting the interested region in the key frame image and then performing contour extraction on the interested region, and the obtaining the standard contour of the target template comprises:
acquiring a key frame image, and intercepting a rectangular region of interest by using a rectangular interception function; then enhancing the image of the region of interest; and finally, creating an image template under the target template through a template creating function to obtain a standard contour of the image template after binaryzation.
5. A method for controlling a SCARA robot based on machine vision according to claim 4, wherein the creating an image template under a target template through a template creating function, and the obtaining of the standard contour after the binarization of the image template comprises:
searching the gray gradient of the image of the region of interest, processing the image of the region of interest by using a Sobel operator, and returning the pixel gray value in the X direction and the pixel gray value in the T direction;
searching local maximum values in the gradient direction by using a non-maximum value inhibition algorithm, inhibiting non-maximum value elements and refining edges;
and extracting and connecting edges by adopting a dual-threshold algorithm to obtain a standard outline of the image template.
6. A method for machine vision based control of a SCARA robot according to any of the claims 1-5, wherein the robot controller based on arm motion planning information to generate motion path and trajectory instructions comprises:
the robot controller carries out track planning on the SCARA robot in a Cartesian space based on the mechanical arm motion planning information to obtain the corresponding relation between the Cartesian space track and joint angles, angular speeds and angular accelerations in each joint space;
performing dynamic modeling on the SCARA robot according to the SCARA robot model parameters and a dynamic analysis method to obtain a dynamic equation of the SCARA robot;
determining the output track state of the robot at each moment through a kinetic equation, outputting the parameter dimension of each joint space to obtain the vector representation of each moment, wherein the parameter dimension comprises joint angle position and joint angular velocity, and simultaneously giving out the vector representation of an expected track state unit on the basis of the vector dimension of the output track state unit;
generating a motion path and trajectory instruction based on the parameter dimensions and a vector representation of the desired trajectory state unit.
7. The method for controlling a SCARA robot based on machine vision according to claim 6, wherein the programmable logic controller resolving motion control commands corresponding to four joint servo drivers based on the motion path and trajectory commands comprises:
the programmable logic controller analyzes the control parameters of each servo driver and then sends the control parameters to the corresponding servo driver according to the corresponding address relation of the servo driver.
8. A control system of a SCARA robot, characterized in that the control system comprises:
the visual sensor is used for acquiring a video data stream of an article to be processed;
the robot controller is used for acquiring a video data stream of an article to be processed based on the visual sensor and sending the video data stream to the upper computer; generating a motion path and track instruction based on the mechanical arm motion planning information, and sending the motion path and track instruction to a programmable logic controller;
the upper computer is used for extracting key frame images from the video data stream based on a preset time interval, performing visual tracking processing on the key frame images and generating mechanical arm motion planning information based on a visual tracking processing result; sending the mechanical arm motion planning information to the robot controller;
and the programmable logic controller is used for analyzing motion control instructions corresponding to the four joint servo drivers based on the motion path and the track instructions and sending the corresponding motion control instructions to the corresponding joint servo drivers.
9. The SCARA robot control system according to claim 8, wherein the upper computer extracts the key frame image based on a preset time interval, then uses the image block of the target area of the key frame image as the target template, and performs correlation matching between the target template and all possible positions in the candidate image, and the place with the highest matching degree is the position where the target is located.
10. A control system for a SCARA robot as recited in claim 8, wherein said host computer communicates with said robot controller via a multi-point interface cable.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111394916.6A CN113814986B (en) | 2021-11-23 | 2021-11-23 | Method and system for controlling SCARA robot based on machine vision |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111394916.6A CN113814986B (en) | 2021-11-23 | 2021-11-23 | Method and system for controlling SCARA robot based on machine vision |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113814986A true CN113814986A (en) | 2021-12-21 |
| CN113814986B CN113814986B (en) | 2022-04-26 |
Family
ID=78919761
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111394916.6A Active CN113814986B (en) | 2021-11-23 | 2021-11-23 | Method and system for controlling SCARA robot based on machine vision |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113814986B (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114642588A (en) * | 2022-03-18 | 2022-06-21 | 北京鑫大地机械工程有限公司 | Control method, device and system of rehabilitation robot |
| CN114654473A (en) * | 2022-05-20 | 2022-06-24 | 广东隆崎机器人有限公司 | Remote control method and system for joint driving motor of SCARA robot |
| CN114777676A (en) * | 2022-05-11 | 2022-07-22 | 青岛盛瀚色谱技术有限公司 | Self-adaptive terahertz three-dimensional tomography device and method |
| CN114872055A (en) * | 2022-07-12 | 2022-08-09 | 广东隆崎机器人有限公司 | SCARA robot assembly control method and system |
| CN117226843A (en) * | 2023-09-27 | 2023-12-15 | 盐城工学院 | Robot movement track control method and system based on visual servo |
| CN117541627A (en) * | 2024-01-10 | 2024-02-09 | 山东师范大学 | User motion guidance system based on VR technology |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080065108A1 (en) * | 2006-06-13 | 2008-03-13 | Intuitive Surgical, Inc. | Minimally invasive surgical system |
| US20120307027A1 (en) * | 2010-01-08 | 2012-12-06 | Koninklijke Philips Electronics N.V. | Uncalibrated visual servoing using real-time velocity optimization |
| CN103325106A (en) * | 2013-04-15 | 2013-09-25 | 浙江工业大学 | Moving workpiece sorting method based on LabVIEW |
| CN104441526A (en) * | 2014-09-24 | 2015-03-25 | 上海智觉光电科技有限公司 | Online mold monitoring and protecting system and method based on contour matching |
| CN104898837A (en) * | 2015-05-22 | 2015-09-09 | 燕山大学 | Portable hand virtual rehabilitation experiment box based on gesture interaction and method |
| CN107263468A (en) * | 2017-05-23 | 2017-10-20 | 陕西科技大学 | A kind of SCARA robotic asssembly methods of utilization digital image processing techniques |
| CN108972593A (en) * | 2018-09-07 | 2018-12-11 | 顺德职业技术学院 | Control method and system under a kind of industrial robot system |
| CN111427310A (en) * | 2020-03-31 | 2020-07-17 | 研祥智能科技股份有限公司 | Industrial robot controller operation system |
-
2021
- 2021-11-23 CN CN202111394916.6A patent/CN113814986B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080065108A1 (en) * | 2006-06-13 | 2008-03-13 | Intuitive Surgical, Inc. | Minimally invasive surgical system |
| US20120307027A1 (en) * | 2010-01-08 | 2012-12-06 | Koninklijke Philips Electronics N.V. | Uncalibrated visual servoing using real-time velocity optimization |
| CN103325106A (en) * | 2013-04-15 | 2013-09-25 | 浙江工业大学 | Moving workpiece sorting method based on LabVIEW |
| CN104441526A (en) * | 2014-09-24 | 2015-03-25 | 上海智觉光电科技有限公司 | Online mold monitoring and protecting system and method based on contour matching |
| CN104898837A (en) * | 2015-05-22 | 2015-09-09 | 燕山大学 | Portable hand virtual rehabilitation experiment box based on gesture interaction and method |
| CN107263468A (en) * | 2017-05-23 | 2017-10-20 | 陕西科技大学 | A kind of SCARA robotic asssembly methods of utilization digital image processing techniques |
| CN108972593A (en) * | 2018-09-07 | 2018-12-11 | 顺德职业技术学院 | Control method and system under a kind of industrial robot system |
| CN111427310A (en) * | 2020-03-31 | 2020-07-17 | 研祥智能科技股份有限公司 | Industrial robot controller operation system |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114642588A (en) * | 2022-03-18 | 2022-06-21 | 北京鑫大地机械工程有限公司 | Control method, device and system of rehabilitation robot |
| CN114642588B (en) * | 2022-03-18 | 2024-02-27 | 北京鑫大地机械工程有限公司 | Control method, device and system of rehabilitation robot |
| CN114777676A (en) * | 2022-05-11 | 2022-07-22 | 青岛盛瀚色谱技术有限公司 | Self-adaptive terahertz three-dimensional tomography device and method |
| CN114654473A (en) * | 2022-05-20 | 2022-06-24 | 广东隆崎机器人有限公司 | Remote control method and system for joint driving motor of SCARA robot |
| CN114872055A (en) * | 2022-07-12 | 2022-08-09 | 广东隆崎机器人有限公司 | SCARA robot assembly control method and system |
| CN114872055B (en) * | 2022-07-12 | 2022-11-18 | 广东隆崎机器人有限公司 | SCARA robot assembly control method and system |
| CN117226843A (en) * | 2023-09-27 | 2023-12-15 | 盐城工学院 | Robot movement track control method and system based on visual servo |
| CN117226843B (en) * | 2023-09-27 | 2024-02-27 | 盐城工学院 | Robot movement track control method and system based on visual servo |
| CN117541627A (en) * | 2024-01-10 | 2024-02-09 | 山东师范大学 | User motion guidance system based on VR technology |
| CN117541627B (en) * | 2024-01-10 | 2024-04-16 | 山东师范大学 | User motion guidance system based on VR technology |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113814986B (en) | 2022-04-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113814986B (en) | Method and system for controlling SCARA robot based on machine vision | |
| CN110202583B (en) | Humanoid manipulator control system based on deep learning and control method thereof | |
| CN107901041B (en) | Robot vision servo control method based on image mixing moment | |
| CN111462154B (en) | Target positioning method and device based on depth vision sensor and automatic grabbing robot | |
| Husain et al. | Realtime tracking and grasping of a moving object from range video | |
| CN104325268A (en) | Industrial robot three-dimensional space independent assembly method based on intelligent learning | |
| CN103325106A (en) | Moving workpiece sorting method based on LabVIEW | |
| CN111604942A (en) | Object detection device, control device, and computer program for object detection | |
| Melchiorre et al. | Collison avoidance using point cloud data fusion from multiple depth sensors: a practical approach | |
| CN105500370A (en) | Robot offline teaching programming system and method based on somatosensory technology | |
| CN114851209B (en) | Industrial robot working path planning optimization method and system based on vision | |
| Lai et al. | Integration of visual information and robot offline programming system for improving automatic deburring process | |
| Han et al. | Grasping control method of manipulator based on binocular vision combining target detection and trajectory planning | |
| Liu et al. | A context-aware safety system for human-robot collaboration | |
| CN109079777B (en) | Manipulator hand-eye coordination operation system | |
| CN114770461A (en) | Monocular vision-based mobile robot and automatic grabbing method thereof | |
| Teke et al. | Real-time and robust collaborative robot motion control with Microsoft Kinect® v2 | |
| CN112805127A (en) | Method and apparatus for creating robot control program | |
| CN113681560B (en) | Method for operating articulated object by mechanical arm based on vision fusion | |
| Zhou et al. | Visual servo control system of 2-DOF parallel robot | |
| Hanh et al. | Simultaneously extract 3D seam curve and weld head angle for robot arm using passive vision | |
| Nakhaeinia et al. | Adaptive robotic contour following from low accuracy RGB-D surface profiling and visual servoing | |
| Gietler et al. | Forestry Crane Automation using Learning-based Visual Grasping Point Prediction | |
| CN113510699A (en) | Mechanical arm motion trajectory planning method based on improved ant colony optimization algorithm | |
| Hu et al. | Manipulator arm interactive control in unknown underwater environment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| EE01 | Entry into force of recordation of patent licensing contract |
Application publication date: 20211221 Assignee: Guangdong Biyao Technology Co.,Ltd. Assignor: Guangdong Longqi robot Co.,Ltd. Contract record no.: X2023980042408 Denomination of invention: A Method and System for Controlling SCARA Robots Based on Machine Vision Granted publication date: 20220426 License type: Common License Record date: 20230922 Application publication date: 20211221 Assignee: Foshan Baijing Intelligent Technology Co.,Ltd. Assignor: Guangdong Longqi robot Co.,Ltd. Contract record no.: X2023980042406 Denomination of invention: A Method and System for Controlling SCARA Robots Based on Machine Vision Granted publication date: 20220426 License type: Common License Record date: 20230922 |
|
| EE01 | Entry into force of recordation of patent licensing contract |