CN113742992B - Master-slave control method based on deep learning and application - Google Patents
Master-slave control method based on deep learning and application Download PDFInfo
- Publication number
- CN113742992B CN113742992B CN202110433533.9A CN202110433533A CN113742992B CN 113742992 B CN113742992 B CN 113742992B CN 202110433533 A CN202110433533 A CN 202110433533A CN 113742992 B CN113742992 B CN 113742992B
- Authority
- CN
- China
- Prior art keywords
- master
- slave
- execution unit
- deep learning
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 40
- 238000013135 deep learning Methods 0.000 title claims abstract description 24
- 230000033001 locomotion Effects 0.000 claims abstract description 25
- 238000013507 mapping Methods 0.000 claims abstract description 25
- 230000008569 process Effects 0.000 claims abstract description 11
- 238000013527 convolutional neural network Methods 0.000 claims abstract description 9
- 238000012549 training Methods 0.000 claims description 20
- 230000008859 change Effects 0.000 claims description 10
- 238000012360 testing method Methods 0.000 claims description 9
- 239000012636 effector Substances 0.000 claims description 7
- 238000005070 sampling Methods 0.000 claims description 7
- 238000005452 bending Methods 0.000 claims description 6
- 229910000831 Steel Inorganic materials 0.000 claims description 5
- 238000004364 calculation method Methods 0.000 claims description 5
- 238000001914 filtration Methods 0.000 claims description 5
- 239000010959 steel Substances 0.000 claims description 5
- 230000000694 effects Effects 0.000 claims description 4
- 239000011159 matrix material Substances 0.000 claims description 4
- 238000009499 grossing Methods 0.000 claims description 3
- 230000033228 biological regulation Effects 0.000 claims description 2
- 238000011056 performance test Methods 0.000 claims description 2
- 238000012545 processing Methods 0.000 claims description 2
- 238000010276 construction Methods 0.000 claims 1
- 230000008030 elimination Effects 0.000 claims 1
- 238000003379 elimination reaction Methods 0.000 claims 1
- 238000001356 surgical procedure Methods 0.000 description 7
- 238000002474 experimental method Methods 0.000 description 5
- 206010052428 Wound Diseases 0.000 description 2
- 208000027418 Wounds and injury Diseases 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 2
- 230000008878 coupling Effects 0.000 description 2
- 238000010168 coupling process Methods 0.000 description 2
- 238000005859 coupling reaction Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 210000000056 organ Anatomy 0.000 description 2
- 241000893313 Helochara delta Species 0.000 description 1
- 208000035965 Postoperative Complications Diseases 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 210000004204 blood vessel Anatomy 0.000 description 1
- 238000002790 cross-validation Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000006073 displacement reaction Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000005764 inhibitory process Effects 0.000 description 1
- 230000014759 maintenance of location Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000002324 minimally invasive surgery Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000000737 periodic effect Effects 0.000 description 1
- 239000012466 permeate Substances 0.000 description 1
- 230000002980 postoperative effect Effects 0.000 description 1
- 230000001737 promoting effect Effects 0.000 description 1
- 229910001285 shape-memory alloy Inorganic materials 0.000 description 1
- 238000009864 tensile test Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F30/00—Computer-aided design [CAD]
- G06F30/20—Design optimisation, verification or simulation
- G06F30/27—Design optimisation, verification or simulation using machine learning, e.g. artificial intelligence, neural networks, support vector machines [SVM] or training a model
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/30—Surgical robots
- A61B34/37—Master-slave robots
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
- G06F17/10—Complex mathematical operations
- G06F17/16—Matrix or vector computation, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/02—Total factory control, e.g. smart factories, flexible manufacturing systems [FMS] or integrated manufacturing systems [IMS]
Abstract
According to the problems of the prior art, the invention provides a master-slave control method and application based on deep learning, wherein after a motion coordinate system of an execution unit is established, a master-slave mapping relation is established, the master-slave mapping relation is utilized to describe the motion forward and backward operations of master hands and slave hands corresponding to the execution unit, and a master-slave heterogeneous model is constructed; and carrying out real-time tracking on the target under the master-slave relationship to obtain tracking result data, and finally carrying out deep learning through a convolutional neural network to complete the control process of the execution unit. The invention can facilitate the operator to improve the control precision of the flexible robot according to the assistance of the target tracking algorithm in the master-slave control mode, and facilitate the operation in a complex and narrow operation space, thereby reducing the operation risk and relieving the pressure of doctors.
Description
Technical Field
The present invention relates to the field of deep learning technology, and in particular, to a target tracking algorithm (GOTURN) and an application thereof.
Background
At present, the surgical robot permeates various links of surgical planning, minimally invasive positioning and noninvasive treatment. The operation performed by the operation robot can greatly reduce the pain of postoperative patients caused by overlarge wounds, is convenient for wound rehabilitation and reduces postoperative complications, most developed countries in the world already have more mature medical remote control operation systems, and the more advanced operation robots provide visual feedback and convenient operation modes so that the development of the medical operation robot enters a rapid development period with epoch-making significance, but certain important organ tissues or blood vessels need to be avoided in the actual operation process, so that the flexibility of the manipulator is required to a certain extent. Da Vinci surgical robots are currently available, the main principle of which is to let the doctor control the mechanical arm (slave hand) to enter the human body to complete a complex surgical operation by manipulating the surgical console (master hand). The advantages of this existing solution are as follows:
perfectly combines the advantages of open surgery and minimally invasive surgery, enables doctors to complete complicated large-scale surgery,
the flexible and rapid surgical mechanical arm can rapidly respond according to the operation of a doctor, and helps the doctor easily complete various complicated surgical actions.
The high-precision and stable surgical mechanical arm can ensure the accuracy and safety of surgery, and can eliminate the shake of hands of doctors through corresponding algorithms, thereby preventing misoperation. The high-performance surgical mechanical arm can make the operation more convenient and easier.
However, da Vinci surgical robots have some non-negligible drawbacks, and the manipulator still has the problem of insufficient distal accuracy when it is necessary to avoid organs or to perform surgery in a small space (e.g. the mouth).
Disclosure of Invention
According to the problems of the prior art, the invention provides a master-slave control method and application based on deep learning, and the off-line learning-based GOTURN algorithm is utilized to assist the surgical robot, so that an operator can accurately control the flexible robot to perform surgery in a complex and narrow surgery space under the master-slave control mode according to the assistance of a target tracking algorithm, thereby reducing surgery risk and relieving doctor pressure.
The technical scheme of the invention is as follows:
a master-slave control method based on deep learning comprises the following steps:
establishing a motion coordinate system of the execution unit;
establishing a master-slave mapping relation, describing the kinematic forward and backward operations of master hands and slave hands corresponding to the execution unit through the master-slave mapping relation, and constructing a master-slave heterogeneous model;
real-time tracking of the target is carried out under a master-slave relationship to obtain tracking result data;
and performing deep learning through a convolutional neural network to complete the control process of the execution unit.
The method for establishing the master-slave mapping relation, describing the kinematic forward and backward operations of the master hand and the slave hand corresponding to the execution unit through the master-slave mapping relation, and constructing a master-slave abnormal model comprises the following steps:
establishing a mapping relation from a Cartesian space coordinate system to describe the kinematic forward and inverse operation of the master hand and the slave hand, and solving by using an inverse Jacobian matrix;
speed of the master hand tip: Δx=j (θ) ·Δθ;
velocity from hand end: Δθ=j (θ) -1 ·ΔX；
Wherein,as the angular velocity vector of the joint, J (θ) ∈R 6×6 Is Jacobian matrix->Is the terminal velocity vector.
The method for tracking the target in real time under the master-slave relationship to obtain tracking result data comprises the following steps:
and carrying out real-time tracking of the target by using a GOTURN algorithm.
The "deep learning through convolutional neural network" includes:
the convolution layer trains the network in advance on the ImageNet, trains the network with a learning rate of 1e-5, and other super parameters are taken from default values of the CaffeNet;
each training example is alternately taken from the training set and video cropping is performed using the GOTURN algorithm.
The method for establishing the motion coordinate system of the execution unit and establishing the master-slave mapping relation, describing the positive and negative operation of the kinematics of the master hand and the slave hand corresponding to the execution unit through the master-slave mapping relation, and constructing a master-slave heterogeneous model comprises the following steps: the flexibility testing step:
simultaneously driving one end of the execution unit to perform bending motion to perform flexibility test, wherein the connecting end is defined as a far end;
determining r, L, θ, δ, d is the distance of the line from the central axis, the length of the flexible joint, the rotation angle in the y-direction, the rotation angle in the z-direction, and the distal end in the plane x, respectively b Oy b Position projection in (a);
the position projection can be calculated by:
wherein d x ,d y Respectively representing the position projections on the x and y axes, d x ,d y Can be measured in the operation process, the gesture formula of the flexible joint distal end is:
connecting a clamping jaw at the distal end, wherein coordinates of the tip end of the clamping jaw are as follows:
wherein s represents the connection length between the rigid rod and the processing spring, lg represents the length of the clamping jaw; when s=0, the actuator unit is considered as a bendable joint.
After the "flexibility test step", it includes: the performance testing step:
a driving device for providing rotary power is configured and connected with the execution unit;
ensuring that the execution unit is in a bent state, and simultaneously loading a tensile force on the distal end of the clamp to perform performance test on the execution unit.
The method for tracking the target in real time under the master-slave relationship to obtain tracking result data comprises the following steps: smoothing the tracking effect:
first, the current frame (c' x ,c' y ) The center of the middle bounding box is relative to the previous frame (c x ,c y ) Modeling the center of the middle bounding box:
c' x ＝c x +w·Δx
c' y ＝c y +h·Δy
where w and h are the width and height, respectively, of the bounding box of the previous frame; Δx and y are random variables;
also, the dimensional change was simulated by the following formula:
w'＝w·γ w
h'＝h·γ h
where w 'and h' are the current width and height of the bounding box, w and h are the previous width and height of the bounding box, γ w And gamma h A random variable;
in the training set, gamma with average value of 1 is utilized w And gamma h Modeling the Laplace distribution, and expanding a training set by using a random target extracted from the Laplace distribution;
the proportion parameters of the Laplace distribution are as follows:
for the movement of the bounding box center:for a change in bounding box size: />
After the deep learning through the convolutional neural network, the method comprises the following steps: a step of eliminating continuously accumulated master-slave following errors and eliminating hand shake of doctors by using a PD link;
in "using PD link to eliminate continuously accumulated master-slave following error":
the regulation and control rule is as follows:
wherein the method comprises the steps ofRespectively represent the pose speed and X of the master and slave hand end effectors m ,X s Respectively represent the pose of the master and slave hand end effectors, and k is the same time p And k d Respectively representing a proportional parameter and a differential parameter;
in the process of eliminating hand shake of doctors, two sliding average filtering is adopted for a master hand and a slave hand respectively, specifically:
N i for filtering calculation results, n is the mean digital filter order (i is greater than or equal to n), and i is the ith sampling period.
The application of the master-slave control method based on the deep learning in the direction of the medical flexible robot is disclosed.
The beneficial effects of the invention are as follows:
after a motion coordinate system of an execution unit is established, a master-slave mapping relation is established, the master-slave mapping relation is utilized to describe the positive and negative operation of the kinematics of a master hand and a slave hand corresponding to the execution unit, and a master-slave heterogeneous model is constructed; and carrying out real-time tracking on the target under the master-slave relationship to obtain tracking result data, and finally carrying out deep learning through a convolutional neural network to complete the control process of the execution unit. The invention can facilitate the operator to improve the control precision of the flexible robot according to the assistance of the target tracking algorithm in the master-slave control mode, and facilitate the operation in a complex and narrow operation space, thereby reducing the operation risk and relieving the pressure of doctors.
Meanwhile, the tracking algorithm used by the invention can train in an off-line mode, which ensures that a user can regulate and control the network in a feedforward mode without on-line fine adjustment, and the tracker can run at the speed of 100fps, thereby promoting the network to track the target object in real time.
Drawings
FIG. 1 is a schematic diagram of a coordinate system of an execution unit.
Fig. 2 is a flow chart of the method of the present invention.
Detailed Description
The present application is further described below with reference to the accompanying drawings.
The core idea of the invention is that: because the flexible surgical manipulator can utilize the flexibility of the flexible surgical manipulator to perform surgical operations in a narrow and complex surgical environment; therefore, the master-slave control system utilizes the target tracking algorithm to track the operation area in real time and feed back depth information, so that a doctor can conveniently regulate and control in real time. In actual operation, the user can control the flexible surgical manipulator through the AR image obtained by target tracking, so that the operation is performed in an accurate surgical area (target tracking area), and the operation is smoothly completed.
Specific example I: in this embodiment, the execution unit may be one of flexible mechanisms such as a flexible robot or a flexible manipulator, or a memory alloy-based driving surgical robot or a force feedback device-based pneumatic flexible surgical robot; preferably, the flexible manipulator can be as follows: CN207055546, a minimally invasive manipulator structure related to the disclosed "minimally invasive manipulator structure".
As shown in fig. 1-2, the specific steps of the master-slave control method based on deep learning according to the invention are as follows:
step 1: a motion coordinate system of the flexible manipulator is established. The coordinate system of the flexible joint is shown in figure 1, x b ,y b ,z b And the basic coordinate systems respectively represent the flexible joints and are coincident with the world coordinate system. And x is e ,y e ,z e And x g ,y g ,z g Representing the coordinate systems of the end effector and the jaws, respectively.
Step 2: according to the coordinate system of step 1, four steel wires are distributed outside the processed spring for driving the execution unit together, and the four steel wires are pulled together to driveThe other end (simply referred to as the distal end) of the flexible joint is subjected to a bending motion. Determining r, L, θ, δ, d is the distance of the line from the central axis, the length of the flexible joint, the rotation angle in the y-direction, the rotation angle in the z-direction, and the distal end in the plane x b Oy b Is provided. Thus, the position projection can be calculated by the following formula (1-1):
wherein d x ,d y Respectively representing the position projections on the x and y axes, d x ,d y Can be measured in the operation process, and the gesture formula (1-2) of the flexible joint distal end is:
the jaws are attached to the distal end of the flexible joint and the coordinates of the tips of the jaws are given by equation (1-3).
Where s represents the length of the connection between the rigid rod and the working spring and Lg represents the length of the jaw. The flexible bendable joint ensures that the holder achieves an omnidirectional bending movement, the length of the bendable section depending on the connection distance between the working spring and the rigid rod, the bending curvature of the flexible joint gradually increasing with increasing connection distance, when s=0 the whole flexible joint is regarded as a bendable joint, whereas when s=16 mm the whole flexible joint will be regarded as a rigid body.
Step 3: and the performance of the manipulator is verified through experiments. Four harmonic servomotors (RSF-100-E050-C) are provided for providing rotational power. In a tensile test, the flexible joint is controlled to a bent state while a tensile force is applied to the distal end of the holder; the flexible joint consists of a mechanically processed spring, an elastic framework and a rigid base rod. Thus, when the coupling distance between the machined spring and the rigid rod is zero, the flexible joint will be the weakest rigid state. In this experiment, the flexible joint was bent to 60 ° along the x-axis and y-axis, respectively, and a 3.5N, 5N tensile force was applied at the distal end of the surgical manipulator, each tensile force having 5 trajectories. The test data obtained are shown in the following Table (1-1):
watch (1-1) surgical manipulator tensile experiment
According to the data of Table (1-1), the surgical manipulator can achieve good rigidity retention when the tensile force is 3.5N; when the flexible joint is bent by 60 degrees along the y axis under a 5N load, one track fails; both tests failed when the flexible joint was bent at a coupling angle of 60 ° along the x-axis and y-axis under a 5N load. Since the geometry of the gripper is planar on the up/down sides and curved on the left/right sides, the surgical manipulator maintains a stable load capacity when connected to the up/down sides (bent along the x-axis), and the left/right loads slide due to the shape of the connection plane. During the experiment, the flexible joint did not undergo significant shape deformation on all paths.
Step 4: and establishing a master-slave mapping relation. The robot is of a master-slave abnormal shape, a mapping relation is required to be established from a Cartesian space coordinate system to describe the kinematic forward and inverse operation of master hands and slave hands, wherein the key point of master-slave control based on the Cartesian space coordinate system is to solve the inverse kinematics, and the inverse Jacobian matrix is used for solving.
The mapping of joint space velocity end to Cartesian space velocity is (equations 1-4):
wherein,is the angular velocity vector of the joint, J%θ)∈R 6×6 Is Jacobian matrix->For the tip speed vector, the displacements and speeds of the robot tip and joint angle in a short time are substituted for the instantaneous tip speed and joint speed, respectively, i.e., the equation (1-4) is converted into the equation (1-5):
ΔX=J (θ) ·Δθ (1-5)
Similarly, the inverse kinematics solution equation (1-6) can be derived from equation (1-5):
Δθ＝J(θ) -1 DeltaX (1-6)
The velocity of the end of the master hand can be obtained by the formula (1-5), the velocity of the end of the slave hand can be obtained after the master-slave mapping relation, and the joint velocity can be obtained according to the formula (1-6).
Step 5: and carrying out real-time tracking on the target by using a GOTURN algorithm under a master-slave relationship. According to the master-slave mapping relationship obtained in step 4, it is assumed that in the previous frame, the tracker predicts that the target is located at c= (c x ,c y ) In the target frame which is the center, the width is w, and the height is h. At the current moment, c obtained according to the previous frame is taken as the center to intercept a new target frame with the width of k 1w Height k 1h . The clipping mode can enable the network to identify which object in the graph is being tracked, and how the motion state of the target object changes can be seen by comparing the width and the height of the front frame and the rear frame. Subsequently, at the current frame, c '= (c' x ,c' y ) A piece of area is cropped for the center, where c' is the current location of the target object. The clipping width and height of the current frame are k respectively 2w And k 2h Where w and h are the width and height, k, of the target frame in the previous frame 2 A search range of the target object is defined. For fast moving objects, the size of the search area may be increased at the cost of increasing network complexity; while for occlusion problems, optimization can be done through training.
Step 6: smoothing of tracking effects. First, the current frame (c' x ,c' y ) The center of the middle bounding box is relative to the previous frame (c x ,c y ) Modeling the center of the middle bounding box to obtain formulas (1-7) and (1-8):
c' x ＝c x +w.DELTA.x type (1-7)
c' y ＝c y +h.DELTA.y formula (1-8)
Where w and h are the width and height, respectively, of the bounding box of the previous frame. Δx and y are random variables used to capture the positional change of the bounding box relative to its size. In the training set, the change in position of the target can be modeled with a laplace distribution with an average of 0 for both Δx and Δy, which has a higher probability for motion of smaller magnitude than for motion of larger magnitude.
Also, the dimensional change was simulated by the following formulas (1-9) and (1-10):
w'＝w·γ w (1-9)
h'＝h·γ h (1-10)
Where w 'and h' are the current width and height of the bounding box, w and h are the previous width and height of the bounding box, γ w And gamma h Random variables, which are used to record the size change of the bounding box. In the training set, gamma with average value of 1 is utilized w And gamma h The Laplace distribution modeling can lead to higher probability when the bounding boxes of the front frame and the back frame are similar in size. To teach the network a preference to select small amplitude motions over large amplitude motions, extending the training set with random targets extracted from the laplace distribution described above can be accomplished. Because these training samples are sampled from the laplace distribution, small amplitude motions will be sampled more than large amplitude motions, and thus the network will learn that otherwise small amplitude motions will tend to be more than large amplitude motions. Experiments have shown that the laplace clipping procedure improves the performance of the tracker compared to the standard uniform clipping procedure used in classification tasks. It is also noted that the scale parameters of the Laplace distribution are obtained by cross-validationFor movement of the center of the bounding box; and->For a change in bounding box size. By constraining the random clipping such that it must contain at least half of the target object in each dimension. And also has to limit the size variation, e.g. gamma w ,γ h E (0.6,1.4) to avoid the network stretching or shrinking the bounding box excessively.
Step 7: deep learning is performed through a convolutional neural network. The convolutional layer in the algorithm network needs to be trained on the ImageNet in advance, the network is trained by using a learning rate of 1e-5, and other super parameters are taken from default values of the CaffeNet. To train the network, each training example is alternately taken from a training set of video and images. When a video training example is used, it is necessary to randomly select a video, and a pair of consecutive frames is randomly selected in the video. The video is then cropped according to the method described in step 5. To achieve better results, the current frame also needs to be randomly cropped, with additional examples to augment the dataset. After the training of the video is completed, an image is randomly sampled and the above described process is repeated. Every time a video or image is sampled, new random samples are dynamically generated to create more diversity in the training process.
Step 8: and the PD link is utilized to eliminate the continuously accumulated master-slave following error. In this link, the scaling factor k is used p And differential coefficient k d The system can be quickly enabled to reach a steady state by performing adjustment, so that the flexible manipulator at the tail end of the slave hand can quickly and accurately follow the change of the position and posture coordinates of the tail end of the master hand, and the adjustment rules are as shown in the following formulas (1-11):
wherein the method comprises the steps ofRespectively represent the pose speed and X of the master and slave hand end effectors m ,X s Respectively represent the pose of the master and slave hand end effectors, and k is the same time p And k d Neither should be too large, k p Too large will deteriorate the dynamic performance of the system, k d Too large may reduce the interference rejection capability of the system.
Step 9: the effect of doctor hand shake (shake will be reflected to slave hand through master-slave mapping) is eliminated. The invention filters jitter by respectively adopting twice sliding average filtering (the sliding average algorithm has better inhibition on periodic interference) on a master hand and a slave hand, and sets continuous sampling within a certain time, and then only needs to sample once measured data for each calculation, wherein the specific calculation is as shown in the following formula (1-12):
in the ith sampling period, each discrete point obtained by sampling is calculated by utilizing the above formula to obtain a mean value sampling result before entering the next sampling step; wherein N is i And n is the mean digital filter order (i is greater than or equal to n) for the filtering calculation result.
The foregoing description of the preferred embodiment of the invention is not intended to be limiting, but rather is intended to cover all modifications, equivalents, and alternatives falling within the spirit and principles of the invention.
Claims (7)
1. The master-slave control method based on deep learning is characterized by comprising the following steps:
establishing a motion coordinate system of the execution unit;
establishing a master-slave mapping relation, describing the kinematic forward and backward operations of master hands and slave hands corresponding to the execution unit through the master-slave mapping relation, and constructing a master-slave heterogeneous model;
real-time tracking of the target is carried out under a master-slave relationship to obtain tracking result data;
deep learning is carried out through a convolutional neural network, and the control process of the execution unit is completed;
the real-time tracking of the target is performed under the master-slave relationship to obtain tracking result data, which comprises the following steps:
the step of real-time tracking of the target by using the GOTURN algorithm specifically comprises the following steps:
based on the master-slave mapping relationship, it is assumed that in the previous frame, the tracker predicts that the target is locatedIn the target frame serving as the center, the width is w, and the height is h; at the current moment, c obtained according to the previous frame is taken as the center to intercept a new target frame with the width of +.>Height is +.>The method comprises the steps of carrying out a first treatment on the surface of the At the current frame +.>Clipping a region for the center, wherein +.>Is the current position of the target object; the clipping width and height of the current frame are +.>And->Where w and h are the width and height of the target frame in the previous frame, +.>Defining a search range of the target object; for fast moving objects, the size of the search area is increased at the cost of increasing network complexity; for the shielding problem, the shielding problem is optimized through training;
the deep learning through the convolutional neural network comprises the following steps: a step of eliminating continuously accumulated master-slave following errors and eliminating hand shake of doctors by using a PD link;
in the elimination of continuously accumulated master-slave following errors by the PD link:
the regulation and control rule is as follows:
；
wherein the method comprises the steps ofRepresenting the pose speed of the master and slave hand end effectors respectively, < ->Respectively represent the pose of the master and slave hand end effectors and simultaneously +.>And->Respectively representing a proportion parameter and an integral parameter;
in eliminating the shake of the hands of doctors, the master hand and the slave hand are respectively filtered by adopting twice sliding average values, specifically:
N i =(n i +n i-1 +n i-2 +……+n i-n )/n;
for filtering calculation results, n is the order of the mean digital filter, i is the sampling period in the ith time, and i is more than or equal to n;
the establishment of the motion coordinate system of the execution unit and the establishment of the master-slave mapping relation, the description of the positive and negative operation of the kinematics of the master and slave hands corresponding to the execution unit through the master-slave mapping relation, and the construction of the master-slave heterogeneous model comprise the following steps: the flexibility testing step:
according to the motion coordinate system of the execution unit, four steel wires are distributed on the outer side of the processed spring, the other end of the flexible joint is driven to perform bending motion by pulling the four steel wires together, and the flexible joint and the non-connecting end of the spring are used as the distal end of the flexible joint;
determination ofThe distance between the steel wire and the central shaft, the length of the flexible joint, the rotation angle of the flexible joint along the y direction, the rotation angle of the flexible joint along the z direction and the plane of the distal end of the flexible joint are respectively +>Position projection in (a);
the position projection can be calculated by:
；
wherein,respectively represent the position projection of the distal end of the flexible joint on the x axis and the y axis,/for the flexible joint>Can be measured in the operation process, the gesture formula of the flexible joint distal end is:
connecting a clamping jaw at the distal end, wherein coordinates of the tip end of the clamping jaw are as follows:
；
wherein S represents the connection length between the rigid rod and the processing spring, and Lg represents the length of the clamping jaw; when s=0, the actuator unit is considered as a bendable joint.
2. The master-slave control method based on deep learning according to claim 1, wherein the step of establishing a master-slave mapping relationship, describing the positive and negative operations of the kinematics of the master and slave hands corresponding to the execution unit through the master-slave mapping relationship, and constructing a master-slave heterogeneous model, includes:
establishing a mapping relation from a Cartesian space coordinate system to describe the kinematic forward and inverse operation of the master hand and the slave hand, and solving by using an inverse Jacobian matrix;
speed of the master hand tip: ；
velocity from hand end:；
wherein θ is the rotation angle of the flexible joint in the y direction;is a jacobian matrix.
3. The master-slave control method based on deep learning according to claim 1, wherein the deep learning by a convolutional neural network comprises:
the convolutional layer is pre-trained on ImageNet, training the network with a learning rate of 1 e-5;
each training example is alternately taken from the training set and video cropping is performed using the GOTURN algorithm.
4. A master-slave control method based on deep learning according to claim 3, comprising:
random cropping of the current frame is also required, with additional examples to augment the dataset.
5. The master-slave control method based on deep learning of claim 4, wherein after the flexibility testing step, comprising: the performance testing step:
a driving device for providing rotary power is configured and connected with the execution unit;
ensuring that the execution unit is in a bending state, and simultaneously loading a tensile force on the distal end of the clamp to perform performance test on the execution unit.
6. The master-slave control method based on deep learning according to claim 1, wherein the performing real-time tracking of the target in the master-slave relationship to obtain tracking result data includes: smoothing the tracking effect:
first, the current frameThe center of the middle bounding box is +.>Modeling the center of the middle bounding box:
where w and h are the width and height, respectively, of the bounding box of the previous frame;and->Is a random variable;
also, the dimensional change was simulated by the following formula:
wherein the method comprises the steps ofAnd->Is the current width and height of the bounding box, w and h are the previous width and height of the bounding box,/o>Anda random variable;
in the training set, an average value of 1 is utilizedAnd->Modeling the Laplace distribution, and expanding a training set by using a random target extracted from the Laplace distribution;
the proportion parameters of the Laplace distribution are as follows:
for the movement of the bounding box center:the method comprises the steps of carrying out a first treatment on the surface of the For a change in bounding box size: />。
7. A use of the master-slave control method based on deep learning of claim 1 in a direction of a medical flexible robot.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110433533.9A CN113742992B (en) | 2021-04-19 | 2021-04-19 | Master-slave control method based on deep learning and application |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110433533.9A CN113742992B (en) | 2021-04-19 | 2021-04-19 | Master-slave control method based on deep learning and application |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113742992A CN113742992A (en) | 2021-12-03 |
| CN113742992B true CN113742992B (en) | 2024-03-01 |
Family
ID=78728284
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110433533.9A Active CN113742992B (en) | 2021-04-19 | 2021-04-19 | Master-slave control method based on deep learning and application |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113742992B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113907693B (en) * | 2021-12-10 | 2022-03-01 | 极限人工智能有限公司 | Operation mapping ratio adjusting method and device, electronic equipment and storage medium |
| CN114569252B (en) * | 2022-03-02 | 2024-01-30 | 中南大学 | Master-slave mapping proportion control system and method for surgical robot |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104440864A (en) * | 2014-12-04 | 2015-03-25 | 深圳先进技术研究院 | Master-slaver teleoperation industrial robot system and control method thereof |
| CN109968310A (en) * | 2019-04-12 | 2019-07-05 | 重庆渝博创智能装备研究院有限公司 | A kind of mechanical arm interaction control method and system |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP7415356B2 (en) * | 2019-07-29 | 2024-01-17 | セイコーエプソン株式会社 | Program transfer system and robot system |
-
2021
- 2021-04-19 CN CN202110433533.9A patent/CN113742992B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104440864A (en) * | 2014-12-04 | 2015-03-25 | 深圳先进技术研究院 | Master-slaver teleoperation industrial robot system and control method thereof |
| CN109968310A (en) * | 2019-04-12 | 2019-07-05 | 重庆渝博创智能装备研究院有限公司 | A kind of mechanical arm interaction control method and system |
Non-Patent Citations (1)
| Title |
|---|
| 柔性主从手臂系统自适应混合控制的研究;贾百龙 等;振动与冲击;第30卷(第12期);第51-55页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113742992A (en) | 2021-12-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11653987B2 (en) | Systems and methods for controlling a robotic manipulator or associated tool | |
| US11801100B2 (en) | Estimation of a position and orientation of a frame used in controlling movement of a tool | |
| JP6087368B2 (en) | Application of force feedback at the input device that prompts the operator of the input device to command the joint device to take a suitable posture | |
| CN113742992B (en) | Master-slave control method based on deep learning and application | |
| CN113977602B (en) | Force feedback tail end holder admittance control method | |
| TWI695765B (en) | Robotic arm | |
| Yaşır et al. | Kinematic design of a non-parasitic 2R1T parallel mechanism with remote center of motion to be used in minimally invasive surgery applications | |
| Vandini et al. | Vision-based motion control of a flexible robot for surgical applications | |
| Xia et al. | A constrained optimization approach to virtual fixtures for multi-robot collaborative teleoperation | |
| WO2022166024A1 (en) | Method and device for planning initial poses of surgical arms of laparoscopic surgical robot | |
| Ren et al. | A master-slave control system with workspaces isomerism for teleoperation of a snake robot | |
| CN116392253A (en) | Active positioning method and system applied to surgical robot | |
| Alassi et al. | Development and kinematic analysis of a redundant, modular and backdrivable laparoscopic surgery robot | |
| Chen et al. | Automatic field of view control of laparoscopes with soft RCM constraints | |
| CN114452004B (en) | Control method for tail end position and posture of surgical robot | |
| CN114504385A (en) | Surgical robot tip attitude adjustment method, system, device, and medium | |
| CN113876433A (en) | Robot system and control method | |
| Karadimos et al. | Virtual reality simulation of a robotic laparoscopic surgical system | |
| Wang et al. | Design, control and analysis of a dual-arm continuum flexible robot system | |
| CN114452003B (en) | Surgical robot | |
| Wang et al. | Elimination method of Master-Slave Jitter for laparoscope arm | |
| Kuo et al. | A design for remote center of motion control using a software computing approach | |
| Beira et al. | An external positioning mechanism for robotic surgery | |
| Nawrat et al. | The Robin Heart Vision, telemanipulator for camera holding–preliminary test results | |
| Mitsuishi et al. | A neurosurgical robot for the deep surgical field characterized by an offset-type forceps and natural input capability |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |