WO2023134261A1 - Master-slave tracking delay test method for minimally invasive surgical robot - Google Patents

Master-slave tracking delay test method for minimally invasive surgical robot Download PDF

Info

Publication number
WO2023134261A1
WO2023134261A1 PCT/CN2022/127569 CN2022127569W WO2023134261A1 WO 2023134261 A1 WO2023134261 A1 WO 2023134261A1 CN 2022127569 W CN2022127569 W CN 2022127569W WO 2023134261 A1 WO2023134261 A1 WO 2023134261A1
Authority
WO
WIPO (PCT)
Prior art keywords
hand
delay
slave
rectangular bar
joint
Prior art date
Application number
PCT/CN2022/127569
Other languages
French (fr)
Chinese (zh)
Inventor
王炳强
李晓阳
徐栋
Original Assignee
山东威高手术机器人有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 山东威高手术机器人有限公司 filed Critical 山东威高手术机器人有限公司
Publication of WO2023134261A1 publication Critical patent/WO2023134261A1/en

Links

Images

Classifications

    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
    • A61B34/30—Surgical robots
    • A61B34/35—Surgical robots for telesurgery
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L43/00—Arrangements for monitoring or testing data switching networks
    • H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
    • H04L43/0852—Delays
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L43/00—Arrangements for monitoring or testing data switching networks
    • H04L43/50—Testing arrangements
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/50—Constructional details
    • H04N23/555—Constructional details for picking-up images in sites, inaccessible due to their dimensions or hazardous conditions, e.g. endoscopes or borescopes
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
    • H04N7/183—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast for receiving images from a single remote source
    • H04N7/185—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast for receiving images from a single remote source from a mobile camera, e.g. for remote control
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
    • A61B34/30—Surgical robots
    • A61B2034/301—Surgical robots for introducing or steering flexible instruments inserted into the body, e.g. catheters or endoscopes

Definitions

  • the invention relates to the technical field of endoscopic minimally invasive surgical robots, in particular to a master-slave tracking delay test method for minimally invasive surgical robots.
  • Minimally invasive surgery refers to surgery performed using modern medical instruments such as laparoscopy and thoracoscopic surgery and related equipment. Compared with traditional open surgery, minimally invasive surgery has the advantages of small wound, less pain, less intraoperative blood loss, faster postoperative recovery, and lower risk of infection, so it has been widely recognized and applied in the field of surgery.
  • existing surgical robots also have their limitations, that is, doctors cannot control surgical robots to perform surgery on patients in different places.
  • the authorized announcement number is CN109091237B
  • the Chinese invention patent named split minimally invasive surgical instrument auxiliary system the authorized announcement number is CN210872029U
  • doctor The utility model patent for the operating table discloses a minimally invasive surgical robot system including a doctor's operating table and an instrument operating table.
  • the reference publication number is CN112370168A, and the Chinese invention patent application titled Minimally Invasive Surgical Robot System.
  • the minimally invasive surgical instrument robot is designed to sit in front of the doctor's console under the endoscope, wear 3D glasses to watch the display screen, and use both hands Hold the wrist for manipulation, and use the robotic arm on the patient's console to precisely control the surgical instruments to perform various surgical actions.
  • the control process adopts the master-slave mapping algorithm.
  • the instrument manipulator on the patient console follows the control arm of the doctor's console, that is, the slave hand follows the master hand.
  • remote surgery in different places is an application of minimally invasive surgical robot system.
  • Remote surgery integrates key technologies such as robotics, communication technology, remote control technology, spatial mapping algorithm, and fault tolerance analysis.
  • the doctor operates the robotic arm at the control end according to the operation requirements; the signals of the joint sensors of the robotic arm at the control end are collected and processed in real time and then output. After the data packets are encapsulated by the communication controller at the main end, they are sent to the instrument robotic arm via a dedicated Internet.
  • the slave controller of the instrument mechanical arm verifies and filters the received data packets and sends them to the robot motion controller, where the motion calculation is performed, and finally the desired position of each joint of the instrument mechanical arm is input to the The driver of each motor, and then control the mechanical arm of the instrument to complete the action expected by the doctor.
  • the Internet not only has complex physical circuits, but also complex protocol suites, verification mechanisms, and network security mechanisms.
  • the network delay mainly depends on the transmission distance and the physical link through which the data is transmitted, including the number of routers and routing processing time.
  • the transmission route and routing route of a fixed transmission node are usually fixed, but due to the sharing and competition of the network, the routing processing time and processing tasks change, and the waiting time and processing time of data packets on the route at different times also change randomly Therefore, there will be problems such as out-of-order data packets and delays, which seriously affect the stability and accuracy of surgical operations.
  • the present invention provides a master-slave tracking delay test method for a minimally invasive surgical robot.
  • the invention discloses a minimally invasive surgical robot master-slave tracking delay test method, comprising the following steps:
  • the master-slave tracking process includes the following four delay components: ⁇ m , ⁇ t , ⁇ o , ⁇ a , and at the same time, there are five corresponding states: S 0 , S 1 , S 2 , S 3 , S 4 ;
  • ⁇ m represents the delay when the surgeon touches the main hand and starts to move until the encoder deflection angle is greater than the minimum threshold that the encoder can detect.
  • State S 1 is the command trigger moment, and state S 0 is the moment when the surgeon touches the main hand;
  • ⁇ t is the transmission delay caused by the control signal transmission, and the state S2 represents the moment when the control signal is obtained from the operator controller;
  • ⁇ o represents the delay of data processing, and ⁇ o is the time from state S2 to state S3 after the control system obtains the available signal instruction;
  • ⁇ a represents the time it takes for the servo system to drive the slave to complete the desired task
  • the second step is to build a single-joint delay test tool
  • the delay test model in the joint space coordinate system is as follows:
  • the main hand rectangular bar is fixed on the input joint of the main hand of the test fixture through the flange coupling, rotates at a constant speed around its center and keeps in sync with the main hand encoder; the slave hand rectangular bar is installed on the corresponding test tool from the hand turning joint;
  • the fifth step is to record the orientation of the main-hand rectangular bar and the slave-hand rectangular bar through an external camera, and use image processing technology to calculate the deflection angle of the main-hand rectangular bar and the deflection angle of the slave-hand rectangular bar;
  • the robot sampling-execution delay can be calculated by the following formula (5):
  • ⁇ d and ⁇ d0 represent the current angle difference and the initial angle difference between the main-hand rectangle and the slave-hand rectangle respectively
  • v r represents the rotational speed of the master-hand rectangle.
  • the present invention also discloses a minimally invasive surgery robot master-slave tracking delay test method, including the following steps:
  • the master-slave tracking process includes the following four delay components: ⁇ m , ⁇ t , ⁇ o , ⁇ a , and at the same time, there are five corresponding states: S 0 , S 1 , S 2 , S 3 , S 4 ;
  • ⁇ m represents the delay when the surgeon touches the main hand and starts to move until the encoder deflection angle is greater than the minimum threshold that the encoder can detect.
  • State S 1 is the command trigger moment, and state S 0 is the moment when the surgeon touches the main hand;
  • ⁇ t is the transmission delay caused by the control signal transmission, and the state S2 represents the moment when the control signal is obtained from the operator controller;
  • ⁇ o represents the delay of data processing, and ⁇ o is the time from state S2 to state S3 after the control system obtains the available signal instruction;
  • ⁇ a represents the time it takes for the servo system to drive the slave to complete the desired task
  • the second step is to build a single-joint delay test tool
  • the delay test model in the joint space coordinate system is as follows:
  • the main hand rectangular bar is fixed on the input joint of the main hand of the test fixture through the flange coupling, rotates at a constant speed around its center and keeps in sync with the main hand encoder; the slave hand rectangular bar is installed on the corresponding test tool from the hand turning joint;
  • the fifth step the main hand rectangular bar performs reciprocating periodic motion: first rotate clockwise for a certain period of time, then stand still for a period of time and rotate counterclockwise at the same time, and finally stop for a period of time; the slave hand rectangular bar follows the main hand movement to perform corresponding periodic motion;
  • the whole test process is recorded in real time by an external camera;
  • the range of motion of the slave hand is smaller than that of the master hand, and the backlash d i in the i-th round-trip motion cycle can be obtained according to the following formula (6):
  • Equation 7 Equation 7 below, where n represents the number of cycles of reciprocating motion:
  • the system under test needs to be pre-tensioned to eliminate the effects of mechanical clearance and elastic deformation.
  • the present invention also discloses a minimally invasive surgery robot master-slave tracking delay test method, including the following steps:
  • the master-slave tracking process includes the following four delay components: ⁇ m , ⁇ t , ⁇ o , ⁇ a , and at the same time, there are five corresponding states: S 0 , S 1 , S 2 , S 3 , S 4 ;
  • ⁇ m represents the delay when the surgeon touches the main hand and starts to move until the encoder deflection angle is greater than the minimum threshold that the encoder can detect.
  • State S 1 is the command trigger moment, and state S 0 is the moment when the surgeon touches the main hand;
  • ⁇ t is the transmission delay caused by the control signal transmission, and the state S2 represents the moment when the control signal is obtained from the operator controller;
  • ⁇ o represents the delay of data processing, and ⁇ o is the time from state S2 to state S3 after the control system obtains the available signal instruction;
  • ⁇ a represents the time it takes for the servo system to drive the slave to complete the desired task
  • the second step is to build a single-joint delay test tool
  • the delay test model in the joint space coordinate system is as follows:
  • the main hand rectangular bar is fixed on the input joint of the main hand of the test fixture through the flange coupling, rotates at a constant speed around its center and keeps in sync with the main hand encoder; the slave hand rectangular bar is installed on the corresponding test tool from the hand turning joint;
  • the endoscope is aligned with the rectangular bar on the hand joint corresponding to the measured joint.
  • the real scene is captured by the endoscope, it is digitized in the workstation and transmitted to the LCD display for display.
  • the external camera needs to be placed on the A suitable position so that its field of view can completely cover the main hand movement process and the monitor screen at the same time;
  • the sixth step is to record the orientation of the main-hand rectangular bar and the slave-hand rectangular bar through an external camera, and use image processing technology to calculate the deflection angle of the main-hand rectangular bar and the deflection angle of the slave-hand rectangular bar;
  • the overall system delay ⁇ t can be obtained by formula (8):
  • the deflection angle of the rectangular bar from the hand is calculated by:
  • a certain frame image in the video image captured by the external camera is tracked from the hand rectangle strip, and the edge detection algorithm of the image processing technology is used to detect the edge of the black rectangle strip, and the distance between each point pixel in the line of the rectangle strip edge and a certain line is calculated using the following formula (9)
  • the minimum value of the sum of squares, the straight line angle is the angle value of the rectangular bar;
  • the background superposition method is used for further processing.
  • the background of several frames of images in the image captured by the external camera is superimposed.
  • the two superimposed graphics on the left and right correspond to the slave hand and the master hand respectively.
  • the superimposed image of the slave hand detect the circular edge of the picture and get After the edge function, the coordinates of the center point of the circle can be obtained; then each frame of image is processed separately, the blue point represents the detected coordinates of the center point of the circle, the black rectangle represents the rectangular strip of the captured image, and the blue
  • the colored point is the two points on the edge of the circle whose center has a certain length and the radius intersects the rectangular frame, that is, the black point on the figure, and connects the blue center point and the center points of the two intersection points to form a straight line, which can represent The angle value of the rectangular bar.
  • the beneficial effect of the present invention is to provide a simplified kinematics time interval test analysis and image capture-transmission-display delay test method for a remote surgical robot, and accurately calculate the sampling-communication-execution delay between the master hand and the slave hand And the shooting-transmission-display delay between the endoscope and the display, so as to better control the accuracy of remote surgery.
  • Fig. 1 is a schematic structural view of the control end mechanical arm (main hand) on the doctor's console;
  • Fig. 2 is a structural schematic diagram of the instrument mechanical arm (slave hand) on the patient operating table;
  • Figure 3 is a schematic diagram of the single joint position tracking process
  • Fig. 4 is a schematic diagram of the movement of the main hand and the slave hand structure corresponding to Fig. 3;
  • Fig. 5 is a delay diagram of the visual feedback system of the telesurgery robot
  • Fig. 6 is a schematic diagram of single-joint sampling-execution delay measurement of the robot system
  • Fig. 7 is a schematic diagram of the movement of the master hand and the slave hand structure corresponding to Fig. 6;
  • Fig. 8 is a schematic diagram of single joint backlash measurement
  • Fig. 9 is an ideal tracking result diagram of reciprocating periodic motion
  • Fig. 10 is a schematic diagram of the total delay measurement of the system
  • Figure 11 is a schematic diagram of the total system delay measurement
  • Figure (a) in Figure 12 shows a certain frame of image obtained from the hand rectangle bar tracking video image obtained by an external SLR camera, and figure (b) is the result of detecting the edge of the black rectangle bar through an image processing technology edge detection algorithm;
  • figure (a) is the edge profile formed by splicing several misplaced line segments when the rectangular strip rotates
  • figure (b) is the state where the two long sides of the black rectangular strip are no longer parallel.
  • Figure (a) is the result of overlaying the background of several frames of images captured by an external camera
  • Figure (b) is the processing of each frame of image
  • Figure (c) represents the search for edge points that intersect with the rectangular frame algorithm
  • Figure 15 is the position tracking at a constant speed
  • the graph (a) is the ideal space calibration
  • the graph (b) is the ideal angle change with time
  • the graph (c) shows the ideal angle difference change with time.
  • control-side robotic arm on the doctor's console is shown in Figure 1, and the control-side robotic arm is defined as the main hand.
  • the instrument manipulator on the patient console is shown in Figure 2, and the instrument manipulator is defined as the slave hand.
  • the delay of the minimally invasive surgical robot system is mainly composed of two parts: (1) the sampling-communication-execution delay between the master hand and the slave hand; (2) the shooting between the endoscope and the display -Transfer-Display Delay. Therefore, it is necessary to measure the delay of these two parts separately.
  • the fixture is a single-joint robot, which is consistent with the joint with the largest equivalent inertia of the robot arm (the The main hand joint of the frock is corresponding to the second articulated arm 3 of the robot main hand shown in FIG. Consistent structure), which can reflect the maximum motion delay of the robot.
  • Master-slave tracking refers to a movement process in which the slave hand tracks the master hand to eliminate the mapping deviation when the master-slave mapping deviates.
  • Figure 3 shows the robot joint position tracking process without visual feedback.
  • Figure 4 is a schematic diagram of the corresponding master hand and slave hand structure movement. After collecting the master hand joint motion information, the position signal is sent to the slave hand controller after transmission; After the end of the driving motion task, the slave hand joint sensor returns the current position signal to the master hand controller through transmission. Due to the strict logic and time sequence, master-slave tracking cannot be completely transparent, and delay is inevitable and cannot be ignored.
  • a complete tracking process mainly includes the following four delay components: ⁇ m , ⁇ t , ⁇ o , ⁇ a , and at the same time, there are five corresponding states: S 0 , S 1 , S 2 , S 3 , S 4 .
  • ⁇ m represents the delay until the surgeon touches the main hand (S 0 ) and starts to move until ⁇ q (encoder deflection angle) is greater than Dthr (the minimum threshold that the encoder can detect), and state S 1 is the command trigger moment.
  • State S 0 is the moment when the surgeon touches the main hand.
  • ⁇ t is the transmission delay due to control signal transmission.
  • State S2 represents the moment when a control signal is obtained from the manipulator controller.
  • ⁇ o represents the delay of data processing. ⁇ o is the time from state S2 to state S3 after the control system obtains available signal instructions.
  • ⁇ a represents the time it takes for the servo system to drive the slave hand to complete the desired task. It is related to the driving stiffness, inertia, load, elastic deformation and response speed of the slave hand. Among the four delay sources, ⁇ a is the part with the longest delay. The tracking task is completed when the slave hand reaches the desired position.
  • the delay in the visual feedback system mainly exists in the video capture, data processing and display parts, as shown in Figure 5.
  • the delay of each part is described as follows: ⁇ c represents the time from image acquisition to output video data by the endoscope; ⁇ p represents the time used for image signal processing and remote transmission, including the time consumed by encoding, decoding, network transmission and other processes; ⁇ d Indicates the image display refresh time, and the refresh cycle of the LCD display is generally 5-20ms.
  • the delay test model in the Cartesian space coordinate system and the joint space coordinate system is as follows:
  • Model 1 In the Cartesian space coordinate system, at each moment in a follower unit, the closer the relative pose after the mapping between the end of the slave hand and the end of the main hand is, the higher the transparency of the system position and the smaller the time delay. (This is the terminal comparison method).
  • Model 2 In the joint space coordinate system, at each moment in a following unit, the smaller the relative difference between the slave hand joint angle q and the corresponding master hand joint, the smaller the delay. (This is the joint comparison method)
  • model 2 It is difficult to obtain the master-slave end poses of model 1 at the same time.
  • FIG. 6 is a schematic diagram of a sampling-execution delay measurement method
  • FIG. 7 is a corresponding structural schematic diagram.
  • the input black rectangular bar is fixed on the input joint of the main hand of the test fixture through a flange coupling, rotates at a constant speed around its center and keeps in sync with the main hand encoder.
  • Another identical rectangular bar is mounted on the test fixture corresponding to the slave hand rotation joint. These two rectangular strips must be strictly fixed to their source of motion to avoid position shifting in any direction during the experiment.
  • An external camera is placed in front of the experimental setup with a suitable field of view so that the camera can simultaneously record the orientation of the two rectangular bars. The angular difference between these two matrix bars can be directly translated into system delay.
  • the initial directions of the two rectangular strips need to be measured in advance, and the influence of different initial directions on the experimental results can be eliminated in the subsequent measurement process.
  • Measure the angle value of the black rectangular bar in the acquired image by automatic image processing technology, mainly including the following four steps: (1) image segmentation of the region of interest; (2) histogram equalization; (3) image binarization ; (4) Angle calculation.
  • m pq represents the two-dimensional geometric moment of the p+q image, as shown in the following formula:
  • (x, y) and I(x, y) represent the coordinates of a pixel and the brightness of the pixel, respectively.
  • m 10 represents the two-dimensional geometric moment of the 1+0th image when p and q are 1 and 0 respectively
  • m 01 represents the 0+1th image 2 when p and q are 0 and 1 respectively
  • dimensional geometric moment m 00 represents the 2D geometric moment of the 0+0th image when p and q take 0 and 0 respectively.
  • the angle ⁇ can be obtained as:
  • u 11 represents the central moment of the image when p and q are 1 and 1 respectively
  • u 20 represents the central moment of the image when p and q are 2 and 0 respectively
  • u 02 represents the central moment of the image when p and q are respectively The central moment of the image when taking 0 and 2.
  • ⁇ d and ⁇ d0 represent the current angle difference and the initial angle difference between the master and slave rectangles, respectively, and v r represents the speed of the master-hand rectangle.
  • Minimally invasive surgical robots use more wire transmission mechanisms, and the reverse hysteresis caused by elastic deformation and transmission gap is also one of the important factors that cause robot hysteresis and affect operating accuracy.
  • the influence of backlash can be eliminated by the backlash compensation algorithm in the controller, but accurate backlash measurement is the premise and basis of compensation.
  • the principle diagram of backlash measurement is shown in Figure 8.
  • the input rectangle on the main hand joint under test performs reciprocating periodic motion: first it rotates clockwise for a certain period of time, then it stands still for a period of time and rotates counterclockwise for the same period of time, and finally stops for a period of time. .
  • the output rectangles on the slave hand joints follow the master hand motion to perform corresponding periodic motions.
  • the system under test needs to be pre-tensioned to eliminate the influence of mechanical gap and elastic deformation, and the entire test process is recorded in real time by an external camera. Under the influence of the robot's internal delay and backlash delay, the ideal master-slave tracking trajectory is shown in Figure 9.
  • the backlash d i can be obtained according to the following formula (6):
  • the final measurement result d of the backlash is obtained by averaging multiple cycles, as shown in the following formula 7, where n represents the number of cycles of reciprocating motion:
  • FIG. 10 The schematic diagrams of the experimental measurement of the total delay of the overall system are shown in Figures 10 and 11.
  • the endoscope is aligned with the rectangular strip on the hand joint corresponding to the measured joint.
  • the real scene captured by the endoscope is digitized in the workstation and transmitted to the LCD monitor for display. What is displayed on the graphics workstation monitor is exactly the same as the slave hand motion scene except for some lag.
  • the external camera needs to be placed in a suitable position so that its field of view can completely cover the main hand movement process and the display screen at the same time.
  • the overall system delay ⁇ t can be obtained by formula (8):
  • Figure (a) in Figure 12 shows a certain frame of video image captured by an external SLR camera from the hand rectangular bar tracking video image, and the edge detection algorithm of the image processing technology is used to detect the edge of the black rectangular bar, and the detection result is shown in the middle picture of Figure 12 ( As shown by the green straight line in b), use the following formula (9) to calculate the minimum value of the sum of the squares of the distance between each point pixel and a certain straight line in the straight line on the edge of the rectangular strip, and the angle of the straight line is the angle value of the rectangular strip.
  • the background of several frames of images in the image captured by the external camera is superimposed, as shown in Figure 14(a).
  • the two superimposed graphics on the left and right correspond to the slave hand and the master hand respectively.
  • Taking the superimposed image of the slave hand as an example detect the circular edge of the graph.
  • the blue circle in the figure is the detected edge.
  • the coordinates of the center point of the circle can be obtained.
  • each frame of image is processed separately, as shown in Figure 14(b), the blue point represents the coordinates of the center point of the detected circle, and the black rectangle represents the rectangular strip of the captured image, with the blue point as the center of the circle.
  • Figure 15(c) shows the angular difference between the input and output rectangles (input minus output).
  • the delay is constant, the angular difference of the moving phase remains constant and larger than that of the stationary case.
  • d 1 and d 2 represent the initial angle difference and the maximum angle difference.

Landscapes

  • Engineering & Computer Science (AREA)
  • Signal Processing (AREA)
  • Health & Medical Sciences (AREA)
  • Multimedia (AREA)
  • Surgery (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Medical Informatics (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Molecular Biology (AREA)
  • Animal Behavior & Ethology (AREA)
  • General Health & Medical Sciences (AREA)
  • Public Health (AREA)
  • Veterinary Medicine (AREA)
  • Environmental & Geological Engineering (AREA)
  • Biomedical Technology (AREA)
  • Robotics (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Manipulator (AREA)

Abstract

A master-slave tracking delay test method for a minimally invasive surgical robot for use in an endoscope minimally invasive surgical robot. A delay of the remote operation process of the minimally invasive surgical robot is measured, and a rectangular strip is installed on an input joint of a master manipulator, a rectangular strip is installed on a rotating joint of a slave manipulator, the directions of the two rectangular strips are recorded by means of a camera, an angle difference between the two rectangular strips is calculated and then converted into a system delay.

Description

微创手术机器人主从跟踪延时测试方法Master-slave tracking delay test method for minimally invasive surgery robot 技术领域technical field
本发明涉及内窥镜微创手术机器人技术领域，具体而言，涉及一种用于微创手术机器人主从跟踪延时测试方法。The invention relates to the technical field of endoscopic minimally invasive surgical robots, in particular to a master-slave tracking delay test method for minimally invasive surgical robots.
背景技术Background technique
微创手术是指利用腹腔镜、胸腔镜等现代医疗器械及相关设备进行的手术。与传统的开放手术相比，微创手术具有创口小、疼痛轻、术中出血量少、术后恢复快、感染风险低等优势，因此在外科手术领域得到了广泛的认可与应用。但是现有的手术机器人也有它的局限性，就是医生无法操控手术机器人对病人进行异地手术。Minimally invasive surgery refers to surgery performed using modern medical instruments such as laparoscopy and thoracoscopic surgery and related equipment. Compared with traditional open surgery, minimally invasive surgery has the advantages of small wound, less pain, less intraoperative blood loss, faster postoperative recovery, and lower risk of infection, so it has been widely recognized and applied in the field of surgery. However, existing surgical robots also have their limitations, that is, doctors cannot control surgical robots to perform surgery on patients in different places.
随着现代网络技术的发展，手术机器人远程操控技术的引入可以有效解决这些问题。手术机器人远程操控技术的出现弥补机器人操控距离上的问题，主要表现在保持手术机器人运动精度高，可对病灶区域进行精准定位操作；具有良好的稳定性，消除医生手部抖动，防止出现误操作；手眼协调一致性，降低操作难度，提升了手术安全性；消除了医生长时间站立的疲劳感，等基础上还具有医生可对病人异地手术的优点。With the development of modern network technology, the introduction of remote control technology of surgical robots can effectively solve these problems. The emergence of surgical robot remote control technology makes up for the problem of robot control distance, which is mainly manifested in maintaining high precision of surgical robot movement, and can perform precise positioning operations on lesion areas; it has good stability, eliminates doctor's hand shake, and prevents misuse ;The consistency of hand-eye coordination reduces the difficulty of operation and improves the safety of surgery; it eliminates the fatigue of doctors standing for a long time, and it also has the advantage that doctors can operate on patients in different places.
授权公告号为CN109091237B，名称为微创手术器械辅助系统的中国发明专利以及授权公告号为CN109091238B、名称为分体式微创手术器械辅助系统的中国发明专利，以及授权公告号为CN210872029U、名称为医生操作台的实用新型专利，就公开了包含医生操作台和器械操作台的微创手术机器人系统。The authorized announcement number is CN109091237B, the Chinese invention patent named minimally invasive surgical instrument auxiliary system and the authorized announcement number is CN109091238B, the Chinese invention patent named split minimally invasive surgical instrument auxiliary system, and the authorized announcement number is CN210872029U, named doctor The utility model patent for the operating table discloses a minimally invasive surgical robot system including a doctor's operating table and an instrument operating table.
参考公开号为CN112370168A，名称为微创手术机器人系统的中国发明专利申请，微创手术器机器人被设计为在内窥镜下，医生坐在医生操作台前方，头戴3D眼镜观看显示屏，双手握住手腕进行操控，利用患者操作台上的机械臂精准的控制手术器械执行各种手术动作。控制过程采用主从映射算法，患者操作台上的器械机械臂跟随医生操作台的控制端机械臂，即从手跟随主手。The reference publication number is CN112370168A, and the Chinese invention patent application titled Minimally Invasive Surgical Robot System. The minimally invasive surgical instrument robot is designed to sit in front of the doctor's console under the endoscope, wear 3D glasses to watch the display screen, and use both hands Hold the wrist for manipulation, and use the robotic arm on the patient's console to precisely control the surgical instruments to perform various surgical actions. The control process adopts the master-slave mapping algorithm. The instrument manipulator on the patient console follows the control arm of the doctor's console, that is, the slave hand follows the master hand.
目前，异地远程手术是微创手术机器人系统的一种应用，远程手术融合了机器人技术、通讯技术、远程控制技术、空间映射算法以及容错性分析等关键技术。医生根据手术需求操作控制端机械臂；控制端机械臂的各关节传感器的信号经实时采集并处理后输出，经过主端通讯控制器将数据包进行封装后，经由专用互联网发送至器械机械臂，器械机械臂的从端控制器对接收到的数据包进行校验、滤波后发送到机器人运动控制器，在运动控制器中进行运动解算，最终器械机械臂的各个关节所期望的位置输入到各个电机的驱动器，进而控制器械机械臂完成医生期望的动作。互联网不仅有着复杂的物理线路，同时也存在复杂的协议族、校验机制以及网络安全机制。网络延时主要取决于传输距离和数据传输所经过的物理链路，包括经过路由器个数和路由处理时间。固定传输节点的传输路线和路由线路通常是一定的，但由于网络存在共享和竞争性，路由处理时间和处理任务是变化的，且不同时刻数据包在路由上的等待时间和处理时间也是随机变化的，因此会产生数据包乱序、延迟等问题，严重影响手术操作稳定性、精确性。At present, remote surgery in different places is an application of minimally invasive surgical robot system. Remote surgery integrates key technologies such as robotics, communication technology, remote control technology, spatial mapping algorithm, and fault tolerance analysis. The doctor operates the robotic arm at the control end according to the operation requirements; the signals of the joint sensors of the robotic arm at the control end are collected and processed in real time and then output. After the data packets are encapsulated by the communication controller at the main end, they are sent to the instrument robotic arm via a dedicated Internet. The slave controller of the instrument mechanical arm verifies and filters the received data packets and sends them to the robot motion controller, where the motion calculation is performed, and finally the desired position of each joint of the instrument mechanical arm is input to the The driver of each motor, and then control the mechanical arm of the instrument to complete the action expected by the doctor. The Internet not only has complex physical circuits, but also complex protocol suites, verification mechanisms, and network security mechanisms. The network delay mainly depends on the transmission distance and the physical link through which the data is transmitted, including the number of routers and routing processing time. The transmission route and routing route of a fixed transmission node are usually fixed, but due to the sharing and competition of the network, the routing processing time and processing tasks change, and the waiting time and processing time of data packets on the route at different times also change randomly Therefore, there will be problems such as out-of-order data packets and delays, which seriously affect the stability and accuracy of surgical operations.
在保持高操作精度的前提下，微创手术机器人低透明性和大网络延迟会延长外科医生的响应时间。通过实验可知，当远程手术延迟超过500ms以上时手术风险会明显提高；根据跨大西洋远程“林白手术”的统计，医生能够忍受的延迟为330ms。对于机器人设备的开发者来说，对系统延迟详细的定量描述将有助于发现不足并不断优化，因此手术机器人延迟测试对手术的安全性与可靠性评估具有重要意义，如何进行延迟测试是本领域技术人员亟待解决的技术问题。Under the premise of maintaining high operating precision, the low transparency and large network delay of minimally invasive surgical robots will prolong the surgeon's response time. Experiments show that when the remote surgery delay exceeds 500ms, the risk of surgery will increase significantly; according to the statistics of the transatlantic remote "Lin Bai Surgery", the delay that doctors can tolerate is 330ms. For developers of robotic equipment, a detailed quantitative description of system delay will help to discover deficiencies and optimize continuously. Therefore, the delay test of surgical robots is of great significance to the safety and reliability evaluation of surgery. How to conduct delay test is the key Technical problems to be solved urgently by those skilled in the art.
发明内容Contents of the invention
本发明就是为了解决如何对微创手术机器人远程操作过程的延迟进行测量的技术问题，提供了一种微创手术机器人主从跟踪延时测试方法。In order to solve the technical problem of how to measure the delay in the remote operation process of a minimally invasive surgical robot, the present invention provides a master-slave tracking delay test method for a minimally invasive surgical robot.
本发明公开一种微创手术机器人主从跟踪延时测试方法，包括以下步骤：The invention discloses a minimally invasive surgical robot master-slave tracking delay test method, comprising the following steps:
第一步，主从跟踪过程包括以下四个延时成分：τ m、τ t、τ o、 τ a，与此同时，存在与之对应的五个状态：S 0、S 1、S 2、S 3、S 4； In the first step, the master-slave tracking process includes the following four delay components: τ m , τ t , τ o , τ a , and at the same time, there are five corresponding states: S 0 , S 1 , S 2 , S 3 , S 4 ;
τ m表示外科医生接触主手并开始运动直到编码器偏转角度大于编码器能够检测到的最小阈值的延时，状态S 1为命令触发时刻，状态S 0为外科医生接触主手的时刻； τm represents the delay when the surgeon touches the main hand and starts to move until the encoder deflection angle is greater than the minimum threshold that the encoder can detect. State S 1 is the command trigger moment, and state S 0 is the moment when the surgeon touches the main hand;
τ t是由于控制信号传输造成的传输延迟，状态S 2表示从操作手控制器获得控制信号的瞬间； τt is the transmission delay caused by the control signal transmission, and the state S2 represents the moment when the control signal is obtained from the operator controller;
τ o表示数据处理的延迟，τ o是当控制系统获得到可用的信号指令后，状态S 2持续到状态S 3的时间； τ o represents the delay of data processing, and τ o is the time from state S2 to state S3 after the control system obtains the available signal instruction;
τ a表示伺服系统驱动从手完成期望任务所用的时间； τ a represents the time it takes for the servo system to drive the slave to complete the desired task;
第二步，搭建单关节延时测试工装；The second step is to build a single-joint delay test tool;
第三步，基于所搭建的单关节延时测试工装，在关节空间坐标系下延时测试模型如下：In the third step, based on the built single-joint delay test fixture, the delay test model in the joint space coordinate system is as follows:
Figure PCTCN2022127569-appb-000001
Figure PCTCN2022127569-appb-000001
第四步，主手矩形条通过法兰联轴器固定在测试工装的主手的输入关节上，绕其中心做匀速转动并与主手编码器保持同步；从手矩形条安装在测试工装对应从手转动关节上；In the fourth step, the main hand rectangular bar is fixed on the input joint of the main hand of the test fixture through the flange coupling, rotates at a constant speed around its center and keeps in sync with the main hand encoder; the slave hand rectangular bar is installed on the corresponding test tool from the hand turning joint;
第五步，通过外部相机记录主手矩形条和从手矩形条的方位，采用图像处理技术计算主手矩形条的偏转角度以及从手矩形条的偏转角度；The fifth step is to record the orientation of the main-hand rectangular bar and the slave-hand rectangular bar through an external camera, and use image processing technology to calculate the deflection angle of the main-hand rectangular bar and the deflection angle of the slave-hand rectangular bar;
第六步，机器人采样-执行延时可以通过以下公式(5)计算得到：In the sixth step, the robot sampling-execution delay can be calculated by the following formula (5):
Figure PCTCN2022127569-appb-000002
Figure PCTCN2022127569-appb-000002
公式(5)中，θ d、θ d0分别表示主手矩形条和从手矩形条之间当前角 度差、初始角度差，v r表示主手矩形条的转速。 In formula (5), θ d and θ d0 represent the current angle difference and the initial angle difference between the main-hand rectangle and the slave-hand rectangle respectively, and v r represents the rotational speed of the master-hand rectangle.
本发明还公开一种微创手术机器人主从跟踪延时测试方法，包括以下步骤：The present invention also discloses a minimally invasive surgery robot master-slave tracking delay test method, including the following steps:
第一步，主从跟踪过程包括以下四个延时成分：τ m、τ t、τ o、τ a，与此同时，存在与之对应的五个状态：S 0、S 1、S 2、S 3、S 4； In the first step, the master-slave tracking process includes the following four delay components: τ m , τ t , τ o , τ a , and at the same time, there are five corresponding states: S 0 , S 1 , S 2 , S 3 , S 4 ;
τ m表示外科医生接触主手并开始运动直到编码器偏转角度大于编码器能够检测到的最小阈值的延时，状态S 1为命令触发时刻，状态S 0为外科医生接触主手的时刻； τm represents the delay when the surgeon touches the main hand and starts to move until the encoder deflection angle is greater than the minimum threshold that the encoder can detect. State S 1 is the command trigger moment, and state S 0 is the moment when the surgeon touches the main hand;
τ t是由于控制信号传输造成的传输延迟，状态S 2表示从操作手控制器获得控制信号的瞬间； τt is the transmission delay caused by the control signal transmission, and the state S2 represents the moment when the control signal is obtained from the operator controller;
τ o表示数据处理的延迟，τ o是当控制系统获得到可用的信号指令后，状态S 2持续到状态S 3的时间； τ o represents the delay of data processing, and τ o is the time from state S2 to state S3 after the control system obtains the available signal instruction;
τ a表示伺服系统驱动从手完成期望任务所用的时间； τ a represents the time it takes for the servo system to drive the slave to complete the desired task;
第二步，搭建单关节延时测试工装；The second step is to build a single-joint delay test tool;
第三步，基于所搭建的单关节延时测试工装，在关节空间坐标系下延时测试模型如下：In the third step, based on the built single-joint delay test fixture, the delay test model in the joint space coordinate system is as follows:
Figure PCTCN2022127569-appb-000003
Figure PCTCN2022127569-appb-000003
第四步，主手矩形条通过法兰联轴器固定在测试工装的主手的输入关节上，绕其中心做匀速转动并与主手编码器保持同步；从手矩形条安装在测试工装对应从手转动关节上；In the fourth step, the main hand rectangular bar is fixed on the input joint of the main hand of the test fixture through the flange coupling, rotates at a constant speed around its center and keeps in sync with the main hand encoder; the slave hand rectangular bar is installed on the corresponding test tool from the hand turning joint;
第五步，主手矩形条执行往复周期运动：首先顺时转动一定时间，接着静止一段时间和相同时间逆时针转动，最后停止一段时间；从手矩 形条跟随主手运动执行相应的周期运动；整个测试过程被外部相机实时的记录下来；The fifth step, the main hand rectangular bar performs reciprocating periodic motion: first rotate clockwise for a certain period of time, then stand still for a period of time and rotate counterclockwise at the same time, and finally stop for a period of time; the slave hand rectangular bar follows the main hand movement to perform corresponding periodic motion; The whole test process is recorded in real time by an external camera;
第六步，从手的运动范围要小于主手的运动范围，在第i个往返运动周期中反向间隙d i根据以下公式(6)可得： In the sixth step, the range of motion of the slave hand is smaller than that of the master hand, and the backlash d i in the i-th round-trip motion cycle can be obtained according to the following formula (6):
d i＝b i-a i   (6) d i =b i -a i (6)
；;
反向间隙的最终测量结果d通过多个周期求平均值获得，如下公式7所示，其中n表示往复运动的周期数：The final measurement result d of the backlash is obtained by averaging multiple cycles, as shown in Equation 7 below, where n represents the number of cycles of reciprocating motion:
Figure PCTCN2022127569-appb-000004
Figure PCTCN2022127569-appb-000004
优选地，在主手矩形条开始运动之前，被测系统需要通过预张紧来消除机械间隙和弹性形变的影响。Preferably, before the main-hand rectangular strip starts to move, the system under test needs to be pre-tensioned to eliminate the effects of mechanical clearance and elastic deformation.
本发明还公开一种微创手术机器人主从跟踪延时测试方法，包括以下步骤：The present invention also discloses a minimally invasive surgery robot master-slave tracking delay test method, including the following steps:
第一步，主从跟踪过程包括以下四个延时成分：τ m、τ t、τ o、τ a，与此同时，存在与之对应的五个状态：S 0、S 1、S 2、S 3、S 4； In the first step, the master-slave tracking process includes the following four delay components: τ m , τ t , τ o , τ a , and at the same time, there are five corresponding states: S 0 , S 1 , S 2 , S 3 , S 4 ;
τ m表示外科医生接触主手并开始运动直到编码器偏转角度大于编码器能够检测到的最小阈值的延时，状态S 1为命令触发时刻，状态S 0为外科医生接触主手的时刻； τm represents the delay when the surgeon touches the main hand and starts to move until the encoder deflection angle is greater than the minimum threshold that the encoder can detect. State S 1 is the command trigger moment, and state S 0 is the moment when the surgeon touches the main hand;
τ t是由于控制信号传输造成的传输延迟，状态S 2表示从操作手控制器获得控制信号的瞬间； τt is the transmission delay caused by the control signal transmission, and the state S2 represents the moment when the control signal is obtained from the operator controller;
τ o表示数据处理的延迟，τ o是当控制系统获得到可用的信号指令 后，状态S 2持续到状态S 3的时间； τ o represents the delay of data processing, and τ o is the time from state S2 to state S3 after the control system obtains the available signal instruction;
τ a表示伺服系统驱动从手完成期望任务所用的时间； τ a represents the time it takes for the servo system to drive the slave to complete the desired task;
第二步，搭建单关节延时测试工装；The second step is to build a single-joint delay test tool;
第三步，基于所搭建的单关节延时测试工装，在关节空间坐标系下延时测试模型如下：In the third step, based on the built single-joint delay test fixture, the delay test model in the joint space coordinate system is as follows:
Figure PCTCN2022127569-appb-000005
Figure PCTCN2022127569-appb-000005
第四步，主手矩形条通过法兰联轴器固定在测试工装的主手的输入关节上，绕其中心做匀速转动并与主手编码器保持同步；从手矩形条安装在测试工装对应从手转动关节上；In the fourth step, the main hand rectangular bar is fixed on the input joint of the main hand of the test fixture through the flange coupling, rotates at a constant speed around its center and keeps in sync with the main hand encoder; the slave hand rectangular bar is installed on the corresponding test tool from the hand turning joint;
第五步，内窥镜对准与被测关节相对应的从手关节上的矩形条，现实场景经内窥镜捕获后在工作站中数字化并传输到LCD显示器上进行显示，外部相机需放在一个合适的位置，使它的视野可以同时完全覆盖主手运动过程和显示器屏幕；In the fifth step, the endoscope is aligned with the rectangular bar on the hand joint corresponding to the measured joint. After the real scene is captured by the endoscope, it is digitized in the workstation and transmitted to the LCD display for display. The external camera needs to be placed on the A suitable position so that its field of view can completely cover the main hand movement process and the monitor screen at the same time;
第六步，通过外部相机记录主手矩形条和从手矩形条的方位，采用图像处理技术计算主手矩形条的偏转角度以及从手矩形条的偏转角度；The sixth step is to record the orientation of the main-hand rectangular bar and the slave-hand rectangular bar through an external camera, and use image processing technology to calculate the deflection angle of the main-hand rectangular bar and the deflection angle of the slave-hand rectangular bar;
第七步，系统整体延时τ t可以通过公式(8)获得： In the seventh step, the overall system delay τ t can be obtained by formula (8):
Figure PCTCN2022127569-appb-000006
Figure PCTCN2022127569-appb-000006
优选地，从手矩形条的偏转角度通过以下方法计算：Preferably, the deflection angle of the rectangular bar from the hand is calculated by:
外部相机获取的从手矩形条跟踪视频图像中的某一帧图像，通过图像处理技术边缘检测算法检测黑色矩形条边缘，利用以下公式(9)计算矩形条边缘直线中每点像素与某直线距离平方和的最小值，该直线角度 则为矩形条的角度值；A certain frame image in the video image captured by the external camera is tracked from the hand rectangle strip, and the edge detection algorithm of the image processing technology is used to detect the edge of the black rectangle strip, and the distance between each point pixel in the line of the rectangle strip edge and a certain line is calculated using the following formula (9) The minimum value of the sum of squares, the straight line angle is the angle value of the rectangular bar;
Figure PCTCN2022127569-appb-000007
Figure PCTCN2022127569-appb-000007
优选地，利用背景叠加法进一步处理。首先通过外部相机捕获的图像中若干帧图像的背景进行叠加，左右两个叠加后的图形分别对应从手和主手，以从手叠加后的图像为例，检测该图的圆形边缘，得到边缘函数后，即可得到该圆的中心点坐标；然后分别对每一帧图像进行处理，蓝色点表示检测到的圆环中心点坐标，黑色矩形框表示所捕获图像的矩形条，以蓝色点为圆心一定长为半径的圆与矩形框相交于其边缘上的两点，即图上的黑点，连接蓝色中心点与两个交点的中心点形成一条直线，该直线就可以表示矩形条的角度值。Preferably, the background superposition method is used for further processing. First, the background of several frames of images in the image captured by the external camera is superimposed. The two superimposed graphics on the left and right correspond to the slave hand and the master hand respectively. Taking the superimposed image of the slave hand as an example, detect the circular edge of the picture and get After the edge function, the coordinates of the center point of the circle can be obtained; then each frame of image is processed separately, the blue point represents the detected coordinates of the center point of the circle, the black rectangle represents the rectangular strip of the captured image, and the blue The colored point is the two points on the edge of the circle whose center has a certain length and the radius intersects the rectangular frame, that is, the black point on the figure, and connects the blue center point and the center points of the two intersection points to form a straight line, which can represent The angle value of the rectangular bar.
本发明的有益效果是：提供一种远程手术机器人简化运动学时间间隔测试分析和图像拍摄-传输-显示延时测试方法，准确地计算主手与从手之间的采样-通信-执行延时以及内窥镜与显示器之间的拍摄-传输-显示延时，从而对远程手术精准度更好的把控。The beneficial effect of the present invention is to provide a simplified kinematics time interval test analysis and image capture-transmission-display delay test method for a remote surgical robot, and accurately calculate the sampling-communication-execution delay between the master hand and the slave hand And the shooting-transmission-display delay between the endoscope and the display, so as to better control the accuracy of remote surgery.
本发明进一步的特征和方面，将在以下参考附图的具体实施方式的描述中，得以清楚地记载。Further features and aspects of the present invention will be apparent from the following description of specific embodiments with reference to the accompanying drawings.
附图说明Description of drawings
图1是医生操作台上的控制端机械臂(主手)的结构示意图；Fig. 1 is a schematic structural view of the control end mechanical arm (main hand) on the doctor's console;
图2是患者操作台上的器械机械臂(从手)的结构示意图；Fig. 2 is a structural schematic diagram of the instrument mechanical arm (slave hand) on the patient operating table;
图3是单关节位置跟踪流程原理图；Figure 3 is a schematic diagram of the single joint position tracking process;
图4是图3对应的主手、从手结构运动示意图；Fig. 4 is a schematic diagram of the movement of the main hand and the slave hand structure corresponding to Fig. 3;
图5是远程手术机器人视觉反馈系统延时图；Fig. 5 is a delay diagram of the visual feedback system of the telesurgery robot;
图6是机器人系统单关节采样-执行延时测量的原理图；Fig. 6 is a schematic diagram of single-joint sampling-execution delay measurement of the robot system;
图7是图6对应的主手、从手结构运动示意图；Fig. 7 is a schematic diagram of the movement of the master hand and the slave hand structure corresponding to Fig. 6;
图8是单关节反向间隙测量原理图；Fig. 8 is a schematic diagram of single joint backlash measurement;
图9是往复周期运动的理想跟踪结果图；Fig. 9 is an ideal tracking result diagram of reciprocating periodic motion;
图10是系统总延时测量原理图；Fig. 10 is a schematic diagram of the total delay measurement of the system;
图11是系统总延时测量原理图；Figure 11 is a schematic diagram of the total system delay measurement;
图12中图(a)所示为外部单反相机获取的从手矩形条跟踪视频图像中的某一帧图像，图(b)是通过图像处理技术边缘检测算法检测黑色矩形条边缘的结果；Figure (a) in Figure 12 shows a certain frame of image obtained from the hand rectangle bar tracking video image obtained by an external SLR camera, and figure (b) is the result of detecting the edge of the black rectangle bar through an image processing technology edge detection algorithm;
图13中，图(a)是当矩形条转动时其边缘是由若干条错位的线段拼接而成的边缘轮廓，图(b)是黑色矩形条的两条长边不再平行状态。In Fig. 13, figure (a) is the edge profile formed by splicing several misplaced line segments when the rectangular strip rotates, and figure (b) is the state where the two long sides of the black rectangular strip are no longer parallel.
图14中，图(a)是通过外部相机捕获的图像中若干帧图像的背景进行叠加结果，图(b)是对每一帧图像进行处理，图(c)表示寻找与矩形框相交边缘点的算法；In Figure 14, Figure (a) is the result of overlaying the background of several frames of images captured by an external camera, Figure (b) is the processing of each frame of image, and Figure (c) represents the search for edge points that intersect with the rectangular frame algorithm;
图15是恒定速度下的位置跟踪，图(a)是理想空间校准，图(b)是随时间的理想角度变化，图(c)表示随时间的理想角度差变化。Figure 15 is the position tracking at a constant speed, the graph (a) is the ideal space calibration, the graph (b) is the ideal angle change with time, and the graph (c) shows the ideal angle difference change with time.
图中符号说明：Explanation of symbols in the figure:
1.基座，2.第一关节臂，3.第二关节臂，4.手腕，5.转动连接座，6.第一抱闸，7.第二抱闸，8.第一转动轴线，9.第二转动轴线，10.第三转动轴线，11.锁止按钮；12.旋转关节，13.L型杆，14.连杆一，15.连杆二，16.伸缩关节，17.快换装置，18.夹持关节，19.轴线一，20.轴线二，21.轴线三，22.轴线四；23.输入矩形条，24.输出矩形条。1. Base, 2. First articulated arm, 3. Second articulated arm, 4. Wrist, 5. Rotation connection seat, 6. First holding brake, 7. Second holding brake, 8. First rotation axis, 9. Second rotation axis, 10. Third rotation axis, 11. Lock button; 12. Rotary joint, 13. L-shaped rod, 14. Connecting rod one, 15. Connecting rod two, 16. Telescopic joint, 17. Quick change device, 18. Clamping joint, 19. Axis 1, 20. Axis 2, 21. Axis 3, 22. Axis 4; 23. Input rectangular bar, 24. Output rectangular bar.
具体实施方式Detailed ways
医生操作台上的控制端机械臂如图1所示，定义控制端机械臂为主手。The control-side robotic arm on the doctor's console is shown in Figure 1, and the control-side robotic arm is defined as the main hand.
患者操作台上的器械机械臂如图2所示，定义器械机械臂为从手。The instrument manipulator on the patient console is shown in Figure 2, and the instrument manipulator is defined as the slave hand.
远程手术场景下，微创手术机器人系统的延时主要由两部分构成：(1)主手与从手之间的采样-通信-执行延时；(2)内窥镜与显示器之间的拍摄-传输-显示延时。因此需要分别测定这两部分的延迟情况。In the remote surgery scenario, the delay of the minimally invasive surgical robot system is mainly composed of two parts: (1) the sampling-communication-execution delay between the master hand and the slave hand; (2) the shooting between the endoscope and the display -Transfer-Display Delay. Therefore, it is necessary to measure the delay of these two parts separately.
为了能够使远程操作延迟可测量，且能够反映机器人的远程运行延迟，搭建了专用测试工装；如图3所示，该工装为单关节机器人，与机器人机械臂等效惯量最大的关节一致(该工装的主手关节与图1所示机器人主手的第二关节臂3对应，该工装的从手关节与图2所示机器人从手的连杆一14、连杆二15、伸缩关节16所构成的机构一致)，能够反映机器人最大运动延迟。In order to make the remote operation delay measurable and reflect the remote operation delay of the robot, a special test fixture was built; as shown in Figure 3, the fixture is a single-joint robot, which is consistent with the joint with the largest equivalent inertia of the robot arm (the The main hand joint of the frock is corresponding to the second articulated arm 3 of the robot main hand shown in FIG. Consistent structure), which can reflect the maximum motion delay of the robot.
主从跟踪是指当主从映射产生偏差时，从手跟踪主手消除映射偏差的一种运动过程。图3所示在没有视觉反馈下的机器人关节位置跟踪流程，图4是相应的主手、从手结构运动示意图，通过采集主手关节运动信息，通过传输后向从手控制器发送位置信号；从手关节传感器在驱动运动任务结束后，通过传输向主手控制器返回当前位置信号。由于严格的逻辑和时间顺序，主从跟踪不可能完全透明，延时是不可避免并且不能忽视的。一个完整的跟踪过程主要包含以下四个延时成分：τ m、τ t、τ o、τ a，与此同时，存在与之对应的五个状态：S 0、S 1、S 2、S 3、S 4。 Master-slave tracking refers to a movement process in which the slave hand tracks the master hand to eliminate the mapping deviation when the master-slave mapping deviates. Figure 3 shows the robot joint position tracking process without visual feedback. Figure 4 is a schematic diagram of the corresponding master hand and slave hand structure movement. After collecting the master hand joint motion information, the position signal is sent to the slave hand controller after transmission; After the end of the driving motion task, the slave hand joint sensor returns the current position signal to the master hand controller through transmission. Due to the strict logic and time sequence, master-slave tracking cannot be completely transparent, and delay is inevitable and cannot be ignored. A complete tracking process mainly includes the following four delay components: τ m , τ t , τ o , τ a , and at the same time, there are five corresponding states: S 0 , S 1 , S 2 , S 3 , S 4 .
τ m表示外科医生接触主手(S 0)并开始运动直到τ q(编码器偏转角度)大于Dthr(编码器能够检测到的最小阈值)的延时，状态S 1为命令触发时刻。状态S 0为外科医生接触主手的时刻。 τ m represents the delay until the surgeon touches the main hand (S 0 ) and starts to move until τ q (encoder deflection angle) is greater than Dthr (the minimum threshold that the encoder can detect), and state S 1 is the command trigger moment. State S 0 is the moment when the surgeon touches the main hand.
τ t是由于控制信号传输造成的传输延迟。状态S 2表示从操作手控制器获得控制信号的瞬间。 τ t is the transmission delay due to control signal transmission. State S2 represents the moment when a control signal is obtained from the manipulator controller.
τ o表示数据处理的延迟。τ o是当控制系统获得到可用的信号指令后，状态S 2持续到状态S 3的时间。 τ o represents the delay of data processing. τ o is the time from state S2 to state S3 after the control system obtains available signal instructions.
τ a表示伺服系统驱动从手完成期望任务所用的时间。它与从手的驱 动刚度，惯性，负载，弹性形变以及响应速度有关。在这四个延时源里，τ a是延时最长的部分。当从手到达期望位置时表示跟踪任务完成。 τ a represents the time it takes for the servo system to drive the slave hand to complete the desired task. It is related to the driving stiffness, inertia, load, elastic deformation and response speed of the slave hand. Among the four delay sources, τ a is the part with the longest delay. The tracking task is completed when the slave hand reaches the desired position.
视觉反馈系统中的延时主要存在于视频捕获、数据处理和显示部分，如图5所示。各部分延时描述如下：τ c表示内窥镜采集图像到输出视频数据的时间；τ p表示图像信号处理与远程传输所用的时间，包括编码、解码、网络传输等过程消耗的时间；τ d表示图像显示刷新时间，LCD显示屏的刷新周期一般为5-20ms。 The delay in the visual feedback system mainly exists in the video capture, data processing and display parts, as shown in Figure 5. The delay of each part is described as follows: τ c represents the time from image acquisition to output video data by the endoscope; τ p represents the time used for image signal processing and remote transmission, including the time consumed by encoding, decoding, network transmission and other processes; τ d Indicates the image display refresh time, and the refresh cycle of the LCD display is generally 5-20ms.
在远程手术中，由于许多额外的延时被引入，且商业互联网固有的阻塞丢包特性，在网络带宽有限且用户量高的时段，延时的波动性会进一步加强，会影响视频反馈的流畅性。In remote surgery, due to the introduction of many additional delays and the inherent blocking and packet loss characteristics of the commercial Internet, the fluctuation of delay will be further strengthened during the time period when the network bandwidth is limited and the number of users is high, which will affect the smoothness of video feedback sex.
基于所搭建的单关节延时测试工装，在笛卡尔空间坐标系与关节空间坐标系下延时测试模型如下：Based on the built single-joint delay test fixture, the delay test model in the Cartesian space coordinate system and the joint space coordinate system is as follows:
模型一：在笛卡尔空间坐标系下，在一个跟随单元中的每一时刻，从手末端位姿与主手末端映射后的相对位姿越近，系统位置透明度越高、时延越小。(此为末端比较法)。Model 1: In the Cartesian space coordinate system, at each moment in a follower unit, the closer the relative pose after the mapping between the end of the slave hand and the end of the main hand is, the higher the transparency of the system position and the smaller the time delay. (This is the terminal comparison method).
Figure PCTCN2022127569-appb-000008
Figure PCTCN2022127569-appb-000008
模型二：在关节空间坐标系下，在一个跟随单元中的每一时刻，从手关节角度q与所对应的主手关节相对差值越小时延越小。(此为关节比较法)Model 2: In the joint space coordinate system, at each moment in a following unit, the smaller the relative difference between the slave hand joint angle q and the corresponding master hand joint, the smaller the delay. (This is the joint comparison method)
Figure PCTCN2022127569-appb-000009
Figure PCTCN2022127569-appb-000009
针对模型一同时获取主从末端位姿较困难，此处选择模型二，在关节坐标系下评价其延时大小，基于所搭建的单关节工装即可完成测试，引入的误差也较小，能够相对准确的反映机器人远程操作下的延时状态。It is difficult to obtain the master-slave end poses of model 1 at the same time. Here, choose model 2 and evaluate its delay in the joint coordinate system. The test can be completed based on the single-joint tooling built, and the error introduced is also small. It can Relatively accurately reflect the delay state under the remote operation of the robot.
图6为采样-执行延时测量方法示意图，图7是对应的结构示意图。输入黑色矩形条通过法兰联轴器固定在测试工装的主手的输入关节上，绕其中心做匀速转动并与主手编码器保持同步。另一个相同的矩形条安装在测试工装对应从手转动关节上。这两个矩形条必须严格固定在其运动源上，避免在实验过程中出现任何方向上的位置移动。将外部相机以一个合适的视野放置在实验装置之前，以至于该相机可以同时记录这两个矩形条的方位。这两个矩阵条之间的角度差可以直接转化为系统延时。FIG. 6 is a schematic diagram of a sampling-execution delay measurement method, and FIG. 7 is a corresponding structural schematic diagram. The input black rectangular bar is fixed on the input joint of the main hand of the test fixture through a flange coupling, rotates at a constant speed around its center and keeps in sync with the main hand encoder. Another identical rectangular bar is mounted on the test fixture corresponding to the slave hand rotation joint. These two rectangular strips must be strictly fixed to their source of motion to avoid position shifting in any direction during the experiment. An external camera is placed in front of the experimental setup with a suitable field of view so that the camera can simultaneously record the orientation of the two rectangular bars. The angular difference between these two matrix bars can be directly translated into system delay.
实验之前，两个矩形条的初始方向需要被预先测得，在后续的测量过程中消除初始方向不同对实验结果造成的影响。通过自动图像处理技术来测量所获取图像中黑色矩形条的角度值，主要包括以下四个步骤：(1)对感兴趣区域图像分割；(2)直方图均衡化；(3)图像二值化；(4)角度计算。Before the experiment, the initial directions of the two rectangular strips need to be measured in advance, and the influence of different initial directions on the experimental results can be eliminated in the subsequent measurement process. Measure the angle value of the black rectangular bar in the acquired image by automatic image processing technology, mainly including the following four steps: (1) image segmentation of the region of interest; (2) histogram equalization; (3) image binarization ; (4) Angle calculation.
矩形条的偏转角度计算采用图像矩算法。m pq表示第p+q图像二维几何矩，如下式所示： The calculation of the deflection angle of the rectangular bar adopts the image moment algorithm. m pq represents the two-dimensional geometric moment of the p+q image, as shown in the following formula:
Figure PCTCN2022127569-appb-000010
Figure PCTCN2022127569-appb-000010
公式(1)中，(x，y)和I(x，y)分别表示某像素点坐标和该像素 点亮度。In formula (1), (x, y) and I(x, y) represent the coordinates of a pixel and the brightness of the pixel, respectively.
矩形条的中心为(xc，yc)，计算公式如下：The center of the rectangular bar is (xc, yc), and the calculation formula is as follows:
Figure PCTCN2022127569-appb-000011
Figure PCTCN2022127569-appb-000011
公式(2)中，m 10表示当p和q分别取1和0时的第1+0图像二维几何矩，m 01表示当p和q分别取0和1时的第0+1图像二维几何矩，m 00表示当p和q分别取0和0时的第0+0图像二维几何矩。 In formula (2), m 10 represents the two-dimensional geometric moment of the 1+0th image when p and q are 1 and 0 respectively, and m 01 represents the 0+1th image 2 when p and q are 0 and 1 respectively. dimensional geometric moment, m 00 represents the 2D geometric moment of the 0+0th image when p and q take 0 and 0 respectively.
矩形条的中心矩为u pq，计算公式如下： The central moment of the rectangular bar is u pq , and the calculation formula is as follows:
Figure PCTCN2022127569-appb-000012
Figure PCTCN2022127569-appb-000012
根据图像处理中惯性矩函数，角度θ可得：According to the moment of inertia function in image processing, the angle θ can be obtained as:
Figure PCTCN2022127569-appb-000013
Figure PCTCN2022127569-appb-000013
公式(4)中，u 11表示当p和q分别取1和1时图像的中心矩，u 20表示当p和q分别取2和0时图像的中心矩，u 02表示当p和q分别取0和2时图像的中心矩。 In formula (4), u 11 represents the central moment of the image when p and q are 1 and 1 respectively, u 20 represents the central moment of the image when p and q are 2 and 0 respectively, and u 02 represents the central moment of the image when p and q are respectively The central moment of the image when taking 0 and 2.
最后，机器人采样-执行延时τ t可以通过以下公式(5)计算得到： Finally, the robot sampling-execution delay τ t can be calculated by the following formula (5):
Figure PCTCN2022127569-appb-000014
Figure PCTCN2022127569-appb-000014
公式(5)中，θ d和θ d0分别表示主从矩形条之间当前角度差和初始角度差，v r表示主手矩形条的转速。 In formula (5), θ d and θ d0 represent the current angle difference and the initial angle difference between the master and slave rectangles, respectively, and v r represents the speed of the master-hand rectangle.
微创手术机器人采用了较多丝传动机制，由弹性变形和传动间隙造成的反向迟滞也是造成机器人迟滞性以及影响操作精度的重要因素之一。 可以通过控制器中间隙补偿算法来消除反向间隙的影响，然而精确的间隙测量是补偿的前提和基础。Minimally invasive surgical robots use more wire transmission mechanisms, and the reverse hysteresis caused by elastic deformation and transmission gap is also one of the important factors that cause robot hysteresis and affect operating accuracy. The influence of backlash can be eliminated by the backlash compensation algorithm in the controller, but accurate backlash measurement is the premise and basis of compensation.
反向间隙测量的原理图如图8所示，被测主手关节上的输入矩形条执行往复周期运动：首先顺时转动一定时间，接着静止一段时间和相同时间逆时针转动，最后停止一段时间。从手关节上的输出矩形条跟随主手运动执行相应的周期运动。在输入矩形条开始运动之前，被测系统需要通过预张紧来消除机械间隙和弹性形变的影响，整个测试过程被外部相机实时的记录下来。在机器人内部延时和反向间隙延时的影响下，理想的主从跟踪轨迹如图9所示。The principle diagram of backlash measurement is shown in Figure 8. The input rectangle on the main hand joint under test performs reciprocating periodic motion: first it rotates clockwise for a certain period of time, then it stands still for a period of time and rotates counterclockwise for the same period of time, and finally stops for a period of time. . The output rectangles on the slave hand joints follow the master hand motion to perform corresponding periodic motions. Before the input rectangular bar starts to move, the system under test needs to be pre-tensioned to eliminate the influence of mechanical gap and elastic deformation, and the entire test process is recorded in real time by an external camera. Under the influence of the robot's internal delay and backlash delay, the ideal master-slave tracking trajectory is shown in Figure 9.
由于系统内部延时和反向间隙造成的延时L 2大于仅仅由系统内部延时造成的延时L 1。在反向间隙的影响下输出矩形条的角度变化趋势并不完全重合于输入，从手的运动范围要小于主手的运动范围。在第i个往返运动周期中反向间隙d i根据以下公式(6)可得： The delay L2 due to system internal delay and backlash is greater than the delay L1 due only to system internal delay. Under the influence of the backlash, the angular change trend of the output rectangular bar does not completely coincide with the input, and the range of motion of the slave hand is smaller than that of the master hand. In the i-th reciprocating motion cycle, the backlash d i can be obtained according to the following formula (6):
d i＝b i-a i   (6) d i =b i -a i (6)
为了减小误差，反向间隙的最终测量结果d通过多个周期求平均值获得，如下公式7所示，其中n表示往复运动的周期数：In order to reduce the error, the final measurement result d of the backlash is obtained by averaging multiple cycles, as shown in the following formula 7, where n represents the number of cycles of reciprocating motion:
Figure PCTCN2022127569-appb-000015
Figure PCTCN2022127569-appb-000015
整体系统总延时的实验测量原理图如图10、11，内窥镜对准与被测关节相对应的从手关节上的矩形条。现实场景经内窥镜捕获后在工作站中数字化并传输到LCD显示器上进行显示。图像工作站显示器上显示的 内容与从手运动场景除了存在一定的滞后以外完全相同。外部相机需放在一个合适的位置，使它的视野可以同时完全覆盖主手运动过程和显示器屏幕。系统整体延时τ t可以通过公式(8)获得： The schematic diagrams of the experimental measurement of the total delay of the overall system are shown in Figures 10 and 11. The endoscope is aligned with the rectangular strip on the hand joint corresponding to the measured joint. The real scene captured by the endoscope is digitized in the workstation and transmitted to the LCD monitor for display. What is displayed on the graphics workstation monitor is exactly the same as the slave hand motion scene except for some lag. The external camera needs to be placed in a suitable position so that its field of view can completely cover the main hand movement process and the display screen at the same time. The overall system delay τ t can be obtained by formula (8):
Figure PCTCN2022127569-appb-000016
Figure PCTCN2022127569-appb-000016
图12中图(a)所示为外部单反相机获取的从手矩形条跟踪视频图像中的某一帧图像，通过图像处理技术边缘检测算法检测黑色矩形条边缘，检测结果如图12中图(b)中绿色直线所示，利用以下公式(9)计算矩形条边缘直线中每点像素与某直线距离平方和的最小值，该直线角度则为矩形条的角度值。Figure (a) in Figure 12 shows a certain frame of video image captured by an external SLR camera from the hand rectangular bar tracking video image, and the edge detection algorithm of the image processing technology is used to detect the edge of the black rectangular bar, and the detection result is shown in the middle picture of Figure 12 ( As shown by the green straight line in b), use the following formula (9) to calculate the minimum value of the sum of the squares of the distance between each point pixel and a certain straight line in the straight line on the edge of the rectangular strip, and the angle of the straight line is the angle value of the rectangular strip.
Figure PCTCN2022127569-appb-000017
Figure PCTCN2022127569-appb-000017
由于显示屏是按照从左到右从上到下的顺序逐个像素刷新的，当矩形条转动时会出现如图13中图(a)所示的情况，其边缘是由若干条错位的线段拼接而成的边缘轮廓，这将对边缘检测产生一定的影响，使其测量精度降低。虽然在静止或者低速运转的情况下，该方法依然具有较高的测量精度，但是当速度增加到一定程度，会出现图13中图(b)所示现象，黑色矩形条的两条长边不再平行，如强行以某条边为检测对象，当矩形条匀速转动时其测试的角度值不是均匀变化，转而为周期性变化。按照强行检测某条边的算法显然不合适，因此需要利用背景叠加法进一步处理。Since the display screen is refreshed pixel by pixel in the order from left to right and from top to bottom, when the rectangular bar rotates, the situation shown in Figure 13 (a) will appear, and its edge is spliced by several misplaced line segments The resulting edge profile will have a certain impact on edge detection and reduce its measurement accuracy. Although this method still has a high measurement accuracy in the case of stationary or low-speed operation, when the speed increases to a certain level, the phenomenon shown in (b) in Figure 13 will appear, and the two long sides of the black rectangular bar are not In parallel, if a certain edge is forced to be the detection object, when the rectangular bar rotates at a constant speed, the tested angle value does not change uniformly, but changes periodically. The algorithm of forcibly detecting a certain edge is obviously inappropriate, so it needs to be further processed by the background superposition method.
首先通过外部相机捕获的图像中若干帧图像的背景进行叠加，如图14(a)所示。左右两个叠加后的图形分别对应从手和主手，以从手叠加后的图像为例，检测该图的圆形边缘，如图中蓝色圆圈即为检测后的边缘，得到边缘函数后，即可得到该圆的中心点坐标。然后分别对每一帧图像进行处理，如图14(b)所示，蓝色点表示检测到的圆环中心点坐标，黑色矩形框表示所捕获图像的矩形条，以蓝色点为圆心一定长为半径的圆与矩形框相交于其边缘上的两点，即图上的黑点，连接蓝色中心点与两个交点的中心点形成一条直线，该直线就可以表示矩形条的角度值，此方法可以在一定程度上减小如图13所示由于屏幕刷新造成的影响。图14(c)表示寻找与矩形框相交边缘点的算法，检测以蓝点为中心的圆上的像素值，当圆上的像素点的像素值发生突变时，即找到了矩形条边缘点。通过以上方法获得的延时测量精度与空间校准存在紧密的关系。理想的空间校准如图15(a)所示，输入和输出矩形条的坐标平面需严格平行于外部相机的观察平面，而且输入和输出矩形条末端运动轨迹在外部相机平面上的投影需要是圆形，而非椭圆。通过空间校准，输入矩形条从s 0转到s 1等同于输出矩形条从a 0转到a 1。误差分析实验如图15(b)所示，运动跟踪过程包括三个阶段：停止-运动-停止。输出矩形条的变化趋势除了存在一定的延时以外，完全等同与输入矩形条的变化趋势。图15(c)所示为输入输出矩形条的角度差(输入减输出)。当延时为恒定时，运动阶段的角度差保持恒定，且大于其在静止情况下的角度差。图中d 1，d 2表示初始角度差和最大角度差。 First, the background of several frames of images in the image captured by the external camera is superimposed, as shown in Figure 14(a). The two superimposed graphics on the left and right correspond to the slave hand and the master hand respectively. Taking the superimposed image of the slave hand as an example, detect the circular edge of the graph. The blue circle in the figure is the detected edge. After obtaining the edge function , the coordinates of the center point of the circle can be obtained. Then each frame of image is processed separately, as shown in Figure 14(b), the blue point represents the coordinates of the center point of the detected circle, and the black rectangle represents the rectangular strip of the captured image, with the blue point as the center of the circle. The circle whose length is the radius intersects the two points on the edge of the rectangular frame, that is, the black point on the figure, and connects the blue center point and the center points of the two intersection points to form a straight line, which can represent the angle value of the rectangular bar , this method can reduce the impact caused by screen refresh as shown in Figure 13 to a certain extent. Figure 14(c) shows the algorithm for finding the edge point intersecting with the rectangular frame, detecting the pixel value on the circle centered on the blue point, and when the pixel value of the pixel point on the circle changes abruptly, the edge point of the rectangle bar is found. The delay measurement accuracy obtained by the above method is closely related to the spatial calibration. The ideal spatial calibration is shown in Figure 15(a), the coordinate planes of the input and output rectangles must be strictly parallel to the observation plane of the external camera, and the projections of the motion trajectories of the ends of the input and output rectangles on the external camera plane must be circular shape, not an ellipse. By spatial calibration, an input bar transition from s 0 to s 1 is equivalent to an output bar transition from a 0 to a 1 . Error Analysis Experiment As shown in Fig. 15(b), the motion tracking process includes three stages: stop-motion-stop. The changing trend of the output rectangular bar is exactly the same as the changing trend of the input rectangular bar except for a certain delay. Figure 15(c) shows the angular difference between the input and output rectangles (input minus output). When the delay is constant, the angular difference of the moving phase remains constant and larger than that of the stationary case. In the figure, d 1 and d 2 represent the initial angle difference and the maximum angle difference.

Claims (6)

  1. 一种微创手术机器人主从跟踪延时测试方法，其特征在于，包括以下步骤：A minimally invasive surgical robot master-slave tracking delay test method, characterized in that it comprises the following steps:
    第一步，主从跟踪过程包括以下四个延时成分：τ m、τ t、τ o、τ a，与此同时，存在与之对应的五个状态：S 0、S 1、S 2、S 3、S 4； In the first step, the master-slave tracking process includes the following four delay components: τ m , τ t , τ o , τ a , and at the same time, there are five corresponding states: S 0 , S 1 , S 2 , S 3 , S 4 ;
    τ m表示外科医生接触主手并开始运动直到编码器偏转角度大于编码器能够检测到的最小阈值的延时，状态S 1为命令触发时刻，状态S 0为外科医生接触主手的时刻； τm represents the delay when the surgeon touches the main hand and starts to move until the encoder deflection angle is greater than the minimum threshold that the encoder can detect. State S 1 is the command trigger moment, and state S 0 is the moment when the surgeon touches the main hand;
    τ t是由于控制信号传输造成的传输延迟，状态S 2表示从操作手控制器获得控制信号的瞬间； τt is the transmission delay caused by the control signal transmission, and the state S2 represents the moment when the control signal is obtained from the operator controller;
    τ o表示数据处理的延迟，τ o是当控制系统获得到可用的信号指令后，状态S 2持续到状态S 3的时间； τ o represents the delay of data processing, and τ o is the time from state S2 to state S3 after the control system obtains the available signal instruction;
    τ a表示伺服系统驱动从手完成期望任务所用的时间； τ a represents the time it takes for the servo system to drive the slave to complete the desired task;
    第二步，搭建单关节延时测试工装；The second step is to build a single-joint delay test tool;
    第三步，基于所搭建的单关节延时测试工装，在关节空间坐标系下延时测试模型如下：In the third step, based on the built single-joint delay test fixture, the delay test model in the joint space coordinate system is as follows:
    Figure PCTCN2022127569-appb-100001
    Figure PCTCN2022127569-appb-100001
    第四步，主手矩形条通过法兰联轴器固定在测试工装的主手的输入关节上，绕其中心做匀速转动并与主手编码器保持同步；从手矩形条安装在测试工装对应从手转动关节上；In the fourth step, the main hand rectangular bar is fixed on the input joint of the main hand of the test fixture through the flange coupling, rotates at a constant speed around its center and keeps in sync with the main hand encoder; the slave hand rectangular bar is installed on the corresponding test tool from the hand turning joint;
    第五步，通过外部相机记录主手矩形条和从手矩形条的方位，采用图像处理技术计算主手矩形条的偏转角度以及从手矩形条的偏转角度；The fifth step is to record the orientation of the main-hand rectangular bar and the slave-hand rectangular bar through an external camera, and use image processing technology to calculate the deflection angle of the main-hand rectangular bar and the deflection angle of the slave-hand rectangular bar;
    第六步，机器人采样-执行延时可以通过以下公式(5)计算得到：In the sixth step, the robot sampling-execution delay can be calculated by the following formula (5):
    Figure PCTCN2022127569-appb-100002
    Figure PCTCN2022127569-appb-100002
    公式(5)中，θ d、θ d0分别表示主手矩形条和从手矩形条之间当前角度差、初始角度差，v r表示主手矩形条的转速。 In formula (5), θ d and θ d0 represent the current angle difference and the initial angle difference between the main-hand rectangle and the slave-hand rectangle respectively, and v r represents the rotational speed of the master-hand rectangle.
  2. 一种微创手术机器人主从跟踪延时测试方法，其特征在于，包括以下步骤：A minimally invasive surgical robot master-slave tracking delay test method, characterized in that it comprises the following steps:
    第一步，主从跟踪过程包括以下四个延时成分：τ m、τ t、τ o、τ a，与此同时，存在与之对应的五个状态：S 0、S 1、S 2、S 3、S 4； In the first step, the master-slave tracking process includes the following four delay components: τ m , τ t , τ o , τ a , and at the same time, there are five corresponding states: S 0 , S 1 , S 2 , S 3 , S 4 ;
    τ m表示外科医生接触主手并开始运动直到编码器偏转角度大于编码器能够检测到的最小阈值的延时，状态S 1为命令触发时刻，状态S 0为外科医生接触主手的时刻； τm represents the delay when the surgeon touches the main hand and starts to move until the encoder deflection angle is greater than the minimum threshold that the encoder can detect. State S 1 is the command trigger moment, and state S 0 is the moment when the surgeon touches the main hand;
    τ t是由于控制信号传输造成的传输延迟，状态S 2表示从操作手控制器获得控制信号的瞬间； τt is the transmission delay caused by the control signal transmission, and the state S2 represents the moment when the control signal is obtained from the operator controller;
    τ o表示数据处理的延迟，τ o是当控制系统获得到可用的信号指令后，状态S 2持续到状态S 3的时间； τ o represents the delay of data processing, and τ o is the time from state S2 to state S3 after the control system obtains the available signal instruction;
    τ a表示伺服系统驱动从手完成期望任务所用的时间； τ a represents the time it takes for the servo system to drive the slave to complete the desired task;
    第二步，搭建单关节延时测试工装；The second step is to build a single-joint delay test tool;
    第三步，基于所搭建的单关节延时测试工装，在关节空间坐标系下延时测试模型如下：In the third step, based on the built single-joint delay test fixture, the delay test model in the joint space coordinate system is as follows:
    Figure PCTCN2022127569-appb-100003
    Figure PCTCN2022127569-appb-100003
    第四步，主手矩形条通过法兰联轴器固定在测试工装的主手的输入关节上，绕其中心做匀速转动并与主手编码器保持同步；从手矩形条安装在测试工装对应从手转动关节上；In the fourth step, the main hand rectangular bar is fixed on the input joint of the main hand of the test fixture through the flange coupling, rotates at a constant speed around its center and keeps in sync with the main hand encoder; the slave hand rectangular bar is installed on the corresponding test tool from the hand turning joint;
    第五步，主手矩形条执行往复周期运动：首先顺时转动一定时间，接着静止一段时间和相同时间逆时针转动，最后停止一段时间；从手矩形条跟随主手运动执行相应的周期运动；整个测试过程被外部相机实时的记录下来；The fifth step, the main hand rectangular bar performs reciprocating periodic motion: first rotate clockwise for a certain period of time, then stand still for a period of time and rotate counterclockwise at the same time, and finally stop for a period of time; the slave hand rectangular bar follows the main hand movement to perform corresponding periodic motion; The whole test process is recorded in real time by an external camera;
    第六步，从手的运动范围要小于主手的运动范围，在第i个往返运动周期中反向间隙d i根据以下公式(6)可得： In the sixth step, the range of motion of the slave hand is smaller than that of the master hand, and the backlash d i in the i-th round-trip motion cycle can be obtained according to the following formula (6):
    d i＝b i-a i  (6)； d i =b i -a i (6);
    反向间隙的最终测量结果d通过多个周期求平均值获得，如下公式7所示，其中n表示往复运动的周期数：The final measurement result d of the backlash is obtained by averaging multiple cycles, as shown in Equation 7 below, where n represents the number of cycles of reciprocating motion:
    Figure PCTCN2022127569-appb-100004
    Figure PCTCN2022127569-appb-100004
  3. 根据权利要求2所述的微创手术机器人主从跟踪延时测试方法，其特征在于，在主手矩形条开始运动之前，被测系统需要通过预张紧来消除机械间隙和弹性形变的影响。The master-slave tracking delay test method of a minimally invasive surgical robot according to claim 2, wherein the system under test needs to be pre-tensioned to eliminate the influence of mechanical gap and elastic deformation before the rectangular bar of the master hand starts to move.
  4. 一种微创手术机器人主从跟踪延时测试方法，其特征在于，包括以下步骤：A minimally invasive surgical robot master-slave tracking delay test method, characterized in that it comprises the following steps:
    第一步，主从跟踪过程包括以下四个延时成分：τ m、τ t、τ o、τ a，与此同时，存在与之对应的五个状态：S 0、S 1、S 2、S 3、S 4； In the first step, the master-slave tracking process includes the following four delay components: τ m , τ t , τ o , τ a , and at the same time, there are five corresponding states: S 0 , S 1 , S 2 , S 3 , S 4 ;
    τ m表示外科医生接触主手并开始运动直到编码器偏转角度大于编码器能够检测到的最小阈值的延时，状态S 1为命令触发时刻，状态S 0为外科医生接触主手的时刻； τm represents the delay when the surgeon touches the main hand and starts to move until the encoder deflection angle is greater than the minimum threshold that the encoder can detect. State S 1 is the command trigger moment, and state S 0 is the moment when the surgeon touches the main hand;
    τ t是由于控制信号传输造成的传输延迟，状态S 2表示从操作手控制器获得控制信号的瞬间； τt is the transmission delay caused by the control signal transmission, and the state S2 represents the moment when the control signal is obtained from the operator controller;
    τ o表示数据处理的延迟，τ o是当控制系统获得到可用的信号指令后，状态S 2持续到状态S 3的时间； τ o represents the delay of data processing, and τ o is the time from state S2 to state S3 after the control system obtains the available signal instruction;
    τ a表示伺服系统驱动从手完成期望任务所用的时间； τ a represents the time it takes for the servo system to drive the slave to complete the desired task;
    第二步，搭建单关节延时测试工装；The second step is to build a single-joint delay test tool;
    第三步，基于所搭建的单关节延时测试工装，在关节空间坐标系下延时测试模型如下：In the third step, based on the built single-joint delay test fixture, the delay test model in the joint space coordinate system is as follows:
    Figure PCTCN2022127569-appb-100005
    Figure PCTCN2022127569-appb-100005
    第四步，主手矩形条通过法兰联轴器固定在测试工装的主手的输入关节上，绕其中心做匀速转动并与主手编码器保持同步；从手矩形条安装在测试工装对应从手转动关节上；In the fourth step, the main hand rectangular bar is fixed on the input joint of the main hand of the test fixture through the flange coupling, rotates at a constant speed around its center and keeps in sync with the main hand encoder; the slave hand rectangular bar is installed on the corresponding test tool From the hand turning joint;
    第五步，内窥镜对准与被测关节相对应的从手关节上的矩形条，现实场景经内窥镜捕获后在工作站中数字化并传输到LCD显示器上进行显示，外部相机需放在一个合适的位置，使它的视野可以同时完全覆盖主手运动过程和显示器屏幕；In the fifth step, the endoscope is aligned with the rectangular bar on the hand joint corresponding to the measured joint. After the real scene is captured by the endoscope, it is digitized in the workstation and transmitted to the LCD display for display. The external camera needs to be placed on the A suitable position so that its field of view can completely cover the main hand movement process and the monitor screen at the same time;
    第六步，通过外部相机记录主手矩形条和从手矩形条的方位，采用图像处理技术计算主手矩形条的偏转角度以及从手矩形条的偏转角度；The sixth step is to record the orientation of the main-hand rectangular bar and the slave-hand rectangular bar through an external camera, and use image processing technology to calculate the deflection angle of the main-hand rectangular bar and the deflection angle of the slave-hand rectangular bar;
    第七步，系统整体延时τ t可以通过公式(8)获得： In the seventh step, the overall system delay τ t can be obtained by formula (8):
    Figure PCTCN2022127569-appb-100006
    Figure PCTCN2022127569-appb-100006
  5. 根据权利要求4所述的微创手术机器人主从跟踪延时测试方法，其特征在于，从手矩形条的偏转角度通过以下方法计算：The minimally invasive surgical robot master-slave tracking delay test method according to claim 4, wherein the deflection angle of the rectangular bar from the hand is calculated by the following method:
    外部相机获取的从手矩形条跟踪视频图像中的某一帧图像，通过图像处理技术边缘检测算法检测黑色矩形条边缘，利用以下公式(9)计算矩形条边缘直线中每点像素与某直线距离平方和的最小值，该直线角度则为矩形条的角度值；A certain frame image in the video image captured by the external camera is tracked from the hand rectangle strip, and the edge detection algorithm of the image processing technology is used to detect the edge of the black rectangle strip, and the distance between each point pixel in the line of the rectangle strip edge and a certain line is calculated using the following formula (9) The minimum value of the sum of squares, the straight line angle is the angle value of the rectangular bar;
    Figure PCTCN2022127569-appb-100007
    Figure PCTCN2022127569-appb-100007
  6. 根据权利要求5所述的微创手术机器人主从跟踪延时测试方法，其特征在于，利用背景叠加法进一步处理。首先通过外部相机捕获的图像中若干帧图像的背景进行叠加，左右两个叠加后的图形分别对应从手和主手，以从手叠加后的图像为例，检测该图的圆形边缘，得到边缘函数后，即可得到该圆的中心点坐标；然后分别对每一帧图像进行处理，蓝色点表示检测到的圆环中心点坐标，黑色矩形框表示所捕获图像的矩形条，以蓝色点为圆心一定长为半径的圆与矩形框相交于其边缘上的两点，即图上的黑点，连接蓝色中心点与两个交点的中心点形成一条直线，该直线就可以表示矩形条的角度值。The minimally invasive surgical robot master-slave tracking delay test method according to claim 5, characterized in that the background superposition method is used for further processing. First, the background of several frames of images in the image captured by the external camera is superimposed. The two superimposed graphics on the left and right correspond to the slave hand and the master hand respectively. Taking the superimposed image of the slave hand as an example, detect the circular edge of the picture and get After the edge function, the coordinates of the center point of the circle can be obtained; then each frame of image is processed separately, the blue point represents the detected coordinates of the center point of the circle, the black rectangle represents the rectangular strip of the captured image, and the blue The colored point is the two points on the edge of the circle whose center has a certain length and the radius intersects the rectangular frame, that is, the black point on the figure, and connects the blue center point and the center points of the two intersection points to form a straight line, which can represent The angle value of the rectangular bar.
PCT/CN2022/127569 2022-01-14 2022-10-26 Master-slave tracking delay test method for minimally invasive surgical robot WO2023134261A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202210040020.6 2022-01-14
CN202210040020.6A CN114191099B (en) 2022-01-14 2022-01-14 Master-slave tracking delay test method for minimally invasive surgery robot

Publications (1)

Publication Number Publication Date
WO2023134261A1 true WO2023134261A1 (en) 2023-07-20

Family

ID=80658511

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/127569 WO2023134261A1 (en) 2022-01-14 2022-10-26 Master-slave tracking delay test method for minimally invasive surgical robot

Country Status (2)

Country Link
CN (1) CN114191099B (en)
WO (1) WO2023134261A1 (en)

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114191099B (en) * 2022-01-14 2023-12-01 山东威高手术机器人有限公司 Master-slave tracking delay test method for minimally invasive surgery robot
CN114848155B (en) * 2022-04-29 2023-04-25 电子科技大学 Verification device for time delay measurement of surgical robot
CN115171384B (en) * 2022-07-04 2024-05-14 南京四维智联科技有限公司 Method and device for compensating position delay of key vehicle in vehicle-mounted display process
CN115351819B (en) * 2022-10-20 2023-03-24 合肥合滨智能机器人有限公司 Master-slave operation time delay test system and method for remote ultrasonic robot
CN115355794B (en) * 2022-10-20 2023-01-20 合肥合滨智能机器人有限公司 Master position testing method and master-slave distance accuracy and repeatability testing system
CN117221177B (en) * 2023-11-08 2024-01-09 湖南省华芯医疗器械有限公司 Image transmission delay monitoring method and system

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20070138992A1 (en) * 2005-12-20 2007-06-21 Intuitive Surgical Inc. Medical robotic system with sliding mode control
US20180250086A1 (en) * 2017-03-02 2018-09-06 KindHeart, Inc. Telerobotic surgery system using minimally invasive surgical tool with variable force scaling and feedback and relayed communications between remote surgeon and surgery station
US20190099223A1 (en) * 2017-09-29 2019-04-04 Siemens Healthcare Gmbh Control method for a robotic device with an ultrasound apparatus and robotic device
CN111856942A (en) * 2020-08-03 2020-10-30 天津理工大学 Time lag and error improvement method of remote intervention operation control system
CN112417755A (en) * 2020-11-10 2021-02-26 西安工程大学 Master-slave mode surgical robot track prediction control method
CN112716608A (en) * 2021-01-20 2021-04-30 山东威高手术机器人有限公司 Master-slave tracking control method for minimally invasive surgery robot
CN114191099A (en) * 2022-01-14 2022-03-18 山东威高手术机器人有限公司 Master-slave tracking delay test method for minimally invasive surgical robot

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
DE19961971B4 (en) * 1999-12-22 2009-10-22 Forschungszentrum Karlsruhe Gmbh Device for safely automatically tracking an endoscope and tracking an instrument
CN103536367B (en) * 2013-10-23 2015-07-01 沈阳工业大学 Master-slave minimally invasive surgical robot system
CN104440864B (en) * 2014-12-04 2017-08-11 深圳先进技术研究院 A kind of master-slave mode remote operating industrial robot system and its control method
CN107374727B (en) * 2017-07-28 2019-10-22 重庆金山医疗器械有限公司 A kind of minimally invasive surgical operation robot simplifies the modeling method of kinematics model
CN111839740B (en) * 2020-07-07 2022-06-03 天津大学 Master-slave isomorphic teleoperation force feedback master hand of minimally invasive surgery robot
CN113349939B (en) * 2021-07-12 2023-03-21 哈尔滨思哲睿智能医疗设备股份有限公司 Performance test method and system for passive active hand type master-slave control surgical robot

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20070138992A1 (en) * 2005-12-20 2007-06-21 Intuitive Surgical Inc. Medical robotic system with sliding mode control
US20180250086A1 (en) * 2017-03-02 2018-09-06 KindHeart, Inc. Telerobotic surgery system using minimally invasive surgical tool with variable force scaling and feedback and relayed communications between remote surgeon and surgery station
US20190099223A1 (en) * 2017-09-29 2019-04-04 Siemens Healthcare Gmbh Control method for a robotic device with an ultrasound apparatus and robotic device
CN111856942A (en) * 2020-08-03 2020-10-30 天津理工大学 Time lag and error improvement method of remote intervention operation control system
CN112417755A (en) * 2020-11-10 2021-02-26 西安工程大学 Master-slave mode surgical robot track prediction control method
CN112716608A (en) * 2021-01-20 2021-04-30 山东威高手术机器人有限公司 Master-slave tracking control method for minimally invasive surgery robot
CN114191099A (en) * 2022-01-14 2022-03-18 山东威高手术机器人有限公司 Master-slave tracking delay test method for minimally invasive surgical robot

Also Published As

Publication number Publication date
CN114191099B (en) 2023-12-01
CN114191099A (en) 2022-03-18

Similar Documents

Publication Publication Date Title
WO2023134261A1 (en) Master-slave tracking delay test method for minimally invasive surgical robot
US10730187B2 (en) Tool position and identification indicator displayed in a boundary area of a computer display screen
Richter et al. Augmented reality predictive displays to help mitigate the effects of delayed telesurgery
CN105411681B (en) The hand eye coordination control system and method for split type micro-wound operation robot
US8971597B2 (en) Efficient vision and kinematic data fusion for robotic surgical instruments and other applications
Sznitman et al. Data-driven visual tracking in retinal microsurgery
Özgüner et al. Camera-robot calibration for the da vinci robotic surgery system
WO2011065035A1 (en) Method of creating teaching data for robot, and teaching system for robot
Li et al. Gaze estimation from color image based on the eye model with known head pose
WO2011065034A1 (en) Method for controlling action of robot, and robot system
Placidi et al. Overall design and implementation of the virtual glove
JP2013086206A (en) Operation input device and method of initializing the same
CN112618026A (en) Remote operation data fusion interactive display system and method
Pachtrachai et al. Hand-eye calibration with a remote centre of motion
Gratal et al. Visual servoing on unknown objects
De Pace et al. Leveraging enhanced virtual reality methods and environments for efficient, intuitive, and immersive teleoperation of robots
Song et al. Autonomous and stable tracking of endoscope instrument tools with monocular camera
Peng et al. Endoscope FOV autonomous tracking method for robot-assisted surgery considering pose control, hand–eye coordination, and image definition
Moustris et al. Shared control for motion compensation in robotic beating heart surgery
WO2021097332A1 (en) Scene perception systems and methods
Dockter et al. A fast, low-cost, computer vision approach for tracking surgical tools
Khan et al. On the calibration of active binocular and RGBD vision systems for dual-arm robots
LU504661B1 (en) Robot-assisted automatic introduction method and device for trocar
JP3855038B2 (en) Rigid body motion measuring method, rigid body motion measuring device, rigid body motion measuring program, jaw motion measuring method, jaw motion measuring device and jaw motion measuring program
CN106903665A (en) A kind of master-slave mode telesurgery robot control system based on stereoscopic vision

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22919895

Country of ref document: EP

Kind code of ref document: A1