CN117503364A - Surgical robot, control method and control device thereof - Google Patents
Surgical robot, control method and control device thereof Download PDFInfo
- Publication number
- CN117503364A CN117503364A CN202210899306.XA CN202210899306A CN117503364A CN 117503364 A CN117503364 A CN 117503364A CN 202210899306 A CN202210899306 A CN 202210899306A CN 117503364 A CN117503364 A CN 117503364A
- Authority
- CN
- China
- Prior art keywords
- imaging
- target
- imaging instrument
- instrument
- freedom
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 44
- 238000003384 imaging method Methods 0.000 claims abstract description 480
- 239000013598 vector Substances 0.000 claims abstract description 96
- 230000033001 locomotion Effects 0.000 claims description 195
- 230000003287 optical effect Effects 0.000 claims description 64
- 230000004044 response Effects 0.000 claims description 11
- 238000004590 computer program Methods 0.000 claims description 9
- 238000003780 insertion Methods 0.000 claims description 6
- 230000037431 insertion Effects 0.000 claims description 6
- 238000003860 storage Methods 0.000 claims description 4
- 238000005452 bending Methods 0.000 claims 4
- 230000004886 head movement Effects 0.000 claims 1
- 210000003128 head Anatomy 0.000 description 99
- 239000012636 effector Substances 0.000 description 17
- 210000003857 wrist joint Anatomy 0.000 description 17
- 230000007246 mechanism Effects 0.000 description 16
- 230000008859 change Effects 0.000 description 15
- 238000010586 diagram Methods 0.000 description 12
- 230000000694 effects Effects 0.000 description 11
- 210000001508 eye Anatomy 0.000 description 11
- 238000004891 communication Methods 0.000 description 10
- 238000013507 mapping Methods 0.000 description 9
- 230000005540 biological transmission Effects 0.000 description 7
- 230000001186 cumulative effect Effects 0.000 description 6
- 230000006870 function Effects 0.000 description 6
- 230000008569 process Effects 0.000 description 6
- 238000012545 processing Methods 0.000 description 6
- 238000001356 surgical procedure Methods 0.000 description 5
- 230000009471 action Effects 0.000 description 4
- 238000010276 construction Methods 0.000 description 4
- 238000012544 monitoring process Methods 0.000 description 4
- 230000035945 sensitivity Effects 0.000 description 4
- 238000012937 correction Methods 0.000 description 2
- 238000005520 cutting process Methods 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 210000001061 forehead Anatomy 0.000 description 2
- 239000011521 glass Substances 0.000 description 2
- 230000000977 initiatory effect Effects 0.000 description 2
- 238000005259 measurement Methods 0.000 description 2
- 238000003825 pressing Methods 0.000 description 2
- 210000001747 pupil Anatomy 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 241001465754 Metazoa Species 0.000 description 1
- 208000002847 Surgical Wound Diseases 0.000 description 1
- 206010052428 Wound Diseases 0.000 description 1
- 208000027418 Wounds and injury Diseases 0.000 description 1
- 238000012084 abdominal surgery Methods 0.000 description 1
- 230000004913 activation Effects 0.000 description 1
- 210000000436 anus Anatomy 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 210000005252 bulbus oculi Anatomy 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000004140 cleaning Methods 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 239000002131 composite material Substances 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000005684 electric field Effects 0.000 description 1
- 230000003028 elevating effect Effects 0.000 description 1
- 210000004247 hand Anatomy 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000007257 malfunction Effects 0.000 description 1
- 239000002184 metal Substances 0.000 description 1
- 238000002324 minimally invasive surgery Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003032 molecular docking Methods 0.000 description 1
- 238000000465 moulding Methods 0.000 description 1
- 238000011022 operating instruction Methods 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 208000014733 refractive error Diseases 0.000 description 1
- 230000002040 relaxant effect Effects 0.000 description 1
- 238000005096 rolling process Methods 0.000 description 1
- 239000000523 sample Substances 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/30—Surgical robots
- A61B34/37—Master-slave robots
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/36—Image-producing devices or illumination devices not otherwise provided for
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/30—Surgical robots
- A61B2034/301—Surgical robots for introducing or steering flexible instruments inserted into the body, e.g. catheters or endoscopes
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/30—Surgical robots
- A61B2034/302—Surgical robots specifically adapted for manipulations within body cavities, e.g. within abdominal or thoracic cavities
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/30—Surgical robots
- A61B2034/305—Details of wrist mechanisms at distal ends of robotic arms
Abstract
The application relates to a surgical robot, a control method thereof and a control device thereof. The control method comprises the following steps: acquiring a first target rotation vector input by an input device; determining a target position and/or posture of a target imaging center of the imaging instrument in a reference coordinate system based on the first target rotation vector; determining a target joint variable for a joint in the manipulator and the imaging instrument based on the target position and/or pose; the manipulator and the articulation in the imaging instrument are controlled in accordance with the target articulation variable to bring the imaging center of the imaging instrument to the target imaging center. By the above embodiment, it can be ensured that the image acquired by the imaging instrument is a desired image.
Description
Technical Field
The present application relates to the field of medical devices, and in particular, to a surgical robot, and a control method and a control device thereof.
Background
Minimally invasive surgery refers to a surgical mode for performing surgery in a human cavity by using modern medical instruments such as laparoscopes, thoracoscopes and related devices. Compared with the traditional operation mode, the minimally invasive operation has the advantages of small wound, light pain, quick recovery and the like.
With the progress of technology, minimally invasive surgical robotic system technology is becoming mature and widely used. The surgical robotic system includes a master console and a slave manipulator including a plurality of medical instruments having end instruments including imaging instruments having image end instruments and surgical instruments having manipulator end instruments. The main console includes a display and an operation unit. The doctor operates the operating portion to manipulate the imaging instrument or the surgical instrument in motion under the field of view provided by the imaging instrument displayed by the display.
In adjusting the field of view, the physician typically desires to rotate the imaging instrument about its distal point to obtain the desired image, however, in practice, because of the structural and kinematic characteristics of the minimally invasive surgical robotic system, the imaging instrument is often not directly controlled to rotate about its distal point of rotation, but rather is required to rotate about other centers of rotation, such as a particular telecentric stop at the time of abdominal surgery. The imaging instrument's actual acquired images are not desirable to the physician due to the changing center of rotation as the imaging instrument moves.
Disclosure of Invention
Accordingly, it is necessary to provide a surgical robot capable of obtaining a desired image, a control method therefor, and a control device therefor.
In one aspect, the present application provides a surgical robot comprising: an imaging instrument for insertion into a body opening to acquire an image; a manipulator for manipulating the imaging instrument for rotation about a telecentric stop positioned at the body opening and for feeding in an axial direction of the imaging instrument; input means for inputting a first target rotation vector of the imaging instrument about its distal point in a target pose degree of freedom rotation; a control device coupled with the manipulator and the input device, configured to: acquiring the first target rotation vector input by the input device; determining a target position and/or posture of a target imaging center of the imaging instrument in a reference coordinate system based on the first target rotation vector; determining a target joint variable for a joint in the manipulator and the imaging instrument based on the target position and/or pose; and controlling articulation in the manipulator and the imaging instrument according to the target articulation variable so as to enable the imaging center of the imaging instrument to reach the target imaging center.
Wherein the determining a target joint variable for a joint in the manipulator and the imaging instrument based on the target position and/or pose comprises: acquiring configuration parameters of an imaging optical axis of an imaging instrument; constructing a kinematic model associated with the manipulator, the imaging instrument and the imaging optical axis in combination with configuration parameters of the imaging optical axis; the target joint variables of the joints in the manipulator and the imaging instrument are determined based on the target position and/or pose, the kinematic model.
Wherein the configuration parameters include a length of the imaging optical axis and/or an angle of the imaging optical axis relative to an imaging plane of the imaging instrument.
Wherein the control device is configured to: acquiring an imaging distance range of the imaging instrument; generating a configuration interface comprising one or more selectable target imaging distances based on the imaging distance range, the target imaging distance being between a minimum imaging distance and a maximum imaging distance of the imaging distance range; in response to selection of the target imaging distance through the configuration interface, the selected target imaging distance is configured to be the length of the imaging optical axis.
Wherein the control device is configured to: generating a configuration interface comprising one or more selectable target angles, the target angles being between 0 ° and 90 °; in response to selection of the target angle through the configuration interface, the selected target angle is configured as an angle of the imaging optical axis relative to the imaging plane.
Wherein the determining, based on the first target rotation vector, a target position and/or pose of a target imaging center of the imaging instrument in a reference coordinate system includes: determining a first position and/or pose of the target imaging center relative to a distal point of the imaging instrument; and determining the target position and/or posture of the first position and/or posture in the reference coordinate system based on the current position and/or posture of the remote point in the reference coordinate system.
Wherein movement of the user's head in the target pose degrees of freedom is associated with the first target rotation vector acquired, the surgical robot further comprising: the image host comprises an observation assembly and an attitude adjustment assembly, wherein the attitude adjustment assembly is used for adjusting the attitude of the observation assembly, and the observation assembly is used for observing the image acquired by the imaging instrument; the control device is coupled with the attitude adjustment assembly and is further configured to: and controlling the gesture adjusting component to move according to the first target rotation vector, so that the observing component moves in the target gesture degree of freedom along with the movement of the head of the user.
Wherein the viewing assembly is configured to move in synchronization with movement of the imaging instrument.
Wherein the attitude adjustment assembly includes: a base; a first pivot pivotally connected to the base and configured to be rotatable relative to the base in a first attitude degree of freedom; and a second pivot member pivotally connected to the first pivot member and configured to pivot in a second attitude degree of freedom with respect to the first pivot member; the observation assembly is fixedly connected with the second pivoting piece relatively.
Wherein the base comprises a first curved slide rail, the first pivot comprises a second curved slide rail and a third curved slide rail, the second pivot comprises a fourth curved slide rail, the first curved slide rail and the second curved slide rail are slidably engaged to be rotatable in a first attitude degree of freedom, and the third curved slide rail and the fourth curved slide rail are slidably engaged to be rotatable in a second attitude degree of freedom; or, the base includes a first curved slide rail, the first pivot member includes a second curved slide rail, the first curved slide rail and the second curved slide rail are slidably engaged to be rotatable in a first attitude degree of freedom, the attitude adjustment joint further includes a revolute joint, and the second pivot member and the first pivot member are connected by the revolute joint to be rotatable in a second attitude degree of freedom.
Wherein the input means comprises one or more base stations having the capability to receive and/or transmit wireless signals, the input means further comprising a wearable device wearable on the head of a user, the wearable device comprising one or more beacons having the capability to transmit and/or receive wireless signals, the first target rotation vector associated with the movement of the head of the user being determinable based on sensing the distance between the one or more beacons and the one or more base stations.
Wherein the imaging instrument comprises a rigid imaging instrument.
Wherein the imaging instrument comprises a flexible imaging instrument.
In another aspect, the present application provides a control method of a surgical robot, the surgical robot including:
an imaging instrument for insertion into a body opening to acquire an image; a manipulator for manipulating the imaging instrument for rotation about a telecentric stop positioned at the body opening and for feeding in an axial direction of the imaging instrument; input means for inputting a first target rotation vector of the imaging instrument about a distal point in a target pose degree of freedom rotation; the control method comprises the following steps: acquiring the first target rotation vector input by the input device; determining a target position and/or posture of a target imaging center of the imaging instrument in a reference coordinate system based on the first target rotation vector; determining a target joint variable for a joint in the manipulator and the imaging instrument based on the target position and/or pose; and controlling articulation in the manipulator and the imaging instrument according to the target articulation variable so as to enable the imaging center of the imaging instrument to reach the target imaging center.
In another aspect, the present application provides a computer readable storage medium storing a computer program configured to be loaded by a processor and to execute steps of implementing a control method according to any one of the embodiments described above.
In another aspect, the present application provides a control device of a surgical robot, including: a memory for storing a computer program; and a processor for loading and executing the computer program; wherein the computer program is configured to be loaded by the processor and to execute steps of implementing the control method according to any of the embodiments described above.
The surgical robot, the control method and the control device thereof have the following beneficial effects:
by determining the target joint variables of the manipulator and the joints in the imaging instrument based on the target imaging center aligned by rotating the first target rotation vector of the imaging instrument about its distal point, and further controlling the joint movements in the manipulator and the imaging instrument according to the target joint variables, the imaging center of the imaging instrument can be brought to the target imaging center, and the same imaging effect as when the imaging instrument is manipulated to rotate the first target rotation vector about its distal point can be achieved, i.e. it can be ensured that the image actually acquired by the imaging instrument coincides with the image desired to be acquired.
Drawings
FIG. 1 is a simplified diagram of the device relationships of a surgical system, according to one embodiment;
FIG. 2 is a schematic diagram illustrating a surgical system according to one embodiment;
FIG. 3 is a schematic diagram illustrating the construction of an imaging instrument according to one embodiment;
FIG. 4 is a schematic diagram of a doctor's primary console according to one embodiment;
FIG. 5 is a flow chart illustrating a method of controlling a surgical robot according to one embodiment;
FIG. 6 is a schematic structural view of another imaging instrument shown in accordance with an embodiment;
FIG. 7 is a schematic illustration of a structural principle of the imaging instrument of FIG. 4;
FIG. 8 is a schematic illustration of a structural principle of the imaging instrument of FIG. 6;
FIG. 9 is a schematic view of the motion state of an embodiment of the imaging instrument of FIG. 7;
FIG. 10 is a schematic view of a motion state of yet another embodiment of the imaging instrument of FIG. 7;
FIG. 11 is a schematic view of a motion state of yet another embodiment of the imaging instrument of FIG. 7;
FIG. 12 is a flow chart illustrating another method of controlling a surgical robot, according to one embodiment;
FIG. 13 is a flow chart illustrating yet another control method of a surgical robot, according to one embodiment;
FIG. 14 is a schematic diagram of the structural principle of an input device according to an embodiment;
FIG. 15 is a screen diagram of an image host shown according to one embodiment;
FIG. 16 is a schematic diagram of an image host, according to one embodiment;
FIG. 17 is a schematic diagram illustrating an exemplary configuration of a posture adjustment assembly of the image mainframe shown in FIG. 16;
FIG. 18 is an enlarged schematic view of a second pivot member of the attitude adjustment assembly shown in FIG. 17;
FIG. 19 is a schematic diagram illustrating the construction of a further embodiment of a posture adjustment assembly in an image host, according to one embodiment;
FIG. 20 is a schematic diagram of a portion of the structure of an input device according to one embodiment;
fig. 21 is a schematic structural view of a control device of a surgical robot system according to an embodiment of the present application.
Detailed Description
In order to facilitate an understanding of the present application, a more complete description of the present application will now be provided with reference to the relevant figures. Preferred embodiments of the present application are shown in the accompanying drawings. This application may, however, be embodied in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete.
It will be understood that when an element is referred to as being "disposed on" another element, it can be directly on the other element or intervening elements may also be present. When an element is referred to as being "connected" to another element, it can be directly connected to the other element or intervening elements may also be present. When an element is referred to as being "coupled" to another element, it can be directly coupled to the other element or intervening elements may also be present. The terms "vertical," "horizontal," "left," "right," and the like are used herein for illustrative purposes only and are not meant to be the only embodiment. The terms "distal" and "proximal" are used herein as directional terms that are conventional in the art of interventional medical devices, wherein "distal" refers to the end of the procedure that is distal to the operator and "proximal" refers to the end of the procedure that is proximal to the operator. The terms "first/second" and the like as used herein may refer to one component as well as a class of more than two components having a common characteristic.
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the present application is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. The term "and/or" as used herein includes any and all combinations of one or more of the associated listed items. The terms "each," "plurality," and "plurality" as used in this application include one or more than two.
Fig. 1 is a simplified diagram of the device relationships of a surgical system, according to one embodiment. As shown in fig. 1, the surgical system 100 includes a surgical robot and a surgical bed 105, the surgical robot includes a bedside robot arm system 101, a doctor main console 103, and an image cart imaging system 108, and it is understood that the constitution of the surgical robot is not limited thereto.
The bedside robotic arm system 101 comprises a drive arm having a plurality of joints, the distal end of which is provided with a piercing device for insertion into a body opening of a patient 106 lying on the table top of the operating table 105, the piercing device being capable of providing a passageway between the surgical robot and a living being (including a human/animal) through which a medical instrument is inserted into the body of the living being, the medical instrument comprising an imaging instrument providing a field of view and a surgical instrument providing a surgical operation such as cutting, stapling, cutting, burning, cleaning. The body opening includes, among other things, surgical incisions and/or natural orifice of a living being, such as a patient's body, which illustratively includes the mouth, nose, anus.
The doctor main console 103 communicates with the bedside mechanical arm system 101 in real time through the data transmission path 120, and the operation actions of the doctor on the doctor main console 103 operate the medical devices in the bedside mechanical arm system 101 through the master-slave mapping relationship based on the kinematic model, and meanwhile, the doctor main console 103 can monitor the state of the bedside mechanical arm system 101, for example, monitor the motion information of each joint in the bedside mechanical arm system 101. When the operation table 105 performs the movement of the corresponding degree of freedom, the patient 106 fixed on the table top of the operation table 105 maintains a relatively stationary state with respect to the table top, the body position change of the patient 106 is achieved by the operation table 105 performing the movement of the corresponding degree of freedom, the movement information of each movement joint of the operation table 105 is recorded and stored in real time, and the movement information of the operation table 105 is transmitted to the bedside robot arm system 101 through the data transmission path 150. Data transmission between the doctor's home console 103 and the operating table 105 is via a data transmission path 130. The image of the operation position of patient 106 is gathered by installing the imaging apparatus of bedside arm system 101, and imaging apparatus connects image car imaging system 108, and the image of imaging apparatus collection is transmitted in the image car imaging system 108 through data transmission path 110, and then, image car imaging system 108 feeds back the image of imaging apparatus collection to doctor's main control board 103 through data transmission path 160 in real time, provides the operation field of vision for the doctor to the smooth implementation of operation of being convenient for. In actual implementation, the data transmission paths 110, 120, 130, 150, 160 may be wired or wireless.
Fig. 2 is a schematic structural view of a surgical system according to an embodiment. As shown in fig. 2, the structure of the bedside robotic arm system 101 and the operating table 105 of the surgical robot is mainly illustrated. The bedside robot system 101 includes a motion chassis 201, a robot arm 250, and a driving arm, where the motion chassis 201 can integrally move the bedside robot system 101 in any direction on a horizontal floor, the robot arm 250 is used to integrally position one or more driving arms, and the driving arm includes an adjusting arm 260 and a manipulator 270, where the manipulator 270 may also be referred to as a manipulator assembly. The robotic arm 250, the adjustment arm 260, and the manipulator 270 typically each include more than one joint, including more than one of a rotational joint and a translational joint.
The motion chassis 201 can adopt a wheeled moving structure, so that the relative position relationship between the bedside mechanical arm system 101 and the operating table 105 is more flexible, the constraint condition of regional appointed positions does not exist, on-site medical staff can push the on-site medical staff to complete the positioning operation and the locking operation after positioning according to the actual operation use requirement, and the on-site medical staff can be fully close to the operating table 105 and simultaneously facilitate the preoperative positioning action of each manipulator 270 above the outside of the patient.
The robotic arm 250 includes a stationary support 203 fixedly coupled to the motion chassis 201 for supporting all of the motion joints, a lifting column 204 for performing the overall lifting linear motion J1 of the robotic arm 250, a large arm 205 and a small arm 206 for performing the rotational motions J2 and J3, respectively, and a directional platform 207 for controlling one or more adjustment arms 260 to perform the overall rotational motion J4, the motions of these joints enabling a rapid reach to the intended preoperative swing area, facilitating a reduced docking time between the preoperative bedside robotic arm system 101 and the patient 106.
The one or more adjustment arms 260 are coupled to the orienting platform 207 by the rotary joint J5 either alone or in parallel, and in some examples, the bedside robotic arm system 101 has multiple adjustment arms 260, and, given that the configuration between the multiple adjustment arms 260 is substantially identical and the descriptions of the various joints are substantially identical, the structural presentation and description of the various joint relationships is presented below using only one adjustment arm 260 and one manipulator 270 as examples in fig. 2. In some examples, the adjustment arm 260 includes a small rotating platform 208, a telescoping arm 209 that performs a linear translational motion J6 in a horizontal direction parallel to the ground, a moving vertical arm 211 that performs an up-down elevating motion J7 in a vertical direction perpendicular to the ground, a swivel joint 213 that performs a rotational motion J8, and a swivel joint 213 that performs a rotational motion J9, with respect to a fixed vertical arm 210 that is fixedly connected to the telescoping arm 209.
The manipulator 270 comprises a yaw joint 214 which is in rotational movement J10 with the whirlwind joint 213, a parallelogram linkage base 215, a first link 216 and a second link 217 which perform rotational movement J11, and a holding arm 218 for causing the medical instrument 219 to perform a linear movement J12 in the direction of the guide rail. A puncturing device (trocon) 229 is mounted at the distal end of manipulator 270. The telecentric stop 220 of the puncturing device 229, which is located at the same position as the body opening of the patient 106, is defined by the intersection of the axis of the whirlwind joint 213 and the axis of the deflection joint 214, and the intersection of these two axes with the lateral center plane of the parallelogram linkage base 215 likewise converges at the telecentric stop 220 of the puncturing device 229, furthermore, the first link 216 and the second link 217 form a parallelogram motion mechanism as two adjacent sides with two virtually adjacent sides parallel to them, the folding and unfolding movements of the parallelogram motion being performed by one motor and about the axis of the rotational motion J11, the telecentric stop of the parallelogram likewise converging at one point with the telecentric stop 220 of the puncturing device 229, and this intersection lying on the central axis of the medical instrument 219, the medical instrument tip 221 being inserted into the body of the patient 106, and performing the surgical actions of the doctor at the main console based on a master-slave mapping relationship. As shown in fig. 2, the rotational movement J10 of the yaw joint 214 relative to the whirlwind joint 213 is controlled such that the puncturing device 229 and the medical instrument 219 move in a yaw degree of freedom about the telecentric stop 220; the rotational movement J11 of the second link 217 relative to the first link 216 is controlled to move the puncturing device 229 and the medical instrument 219 in a pitch degree of freedom about the telecentric stop 220. Illustratively, a pitch joint controlling the rotation of the second link 217 relative to the first link 216 may be provided at the parallelogram linkage base 215. The puncturing device 229 and the medical instrument 219 move around the telecentric stop 220 primarily to avoid stress tears in the body opening caused by changes in the position of the puncturing device 229 and the medical instrument 219.
In a medical instrument suitable for use with the bedside robotic arm of fig. 2, for example, the rigid endoscope (i.e., imaging instrument) 219 of fig. 3, generally includes four degrees of freedom, a translational degree of freedom, and three rotational degrees of freedom that rotate about the telecentric stop 220, wherein the manipulator 270 provides the imaging instrument 219 with a pitch degree of freedom, a yaw degree of freedom, and a translational degree of freedom along J12, and the imaging instrument 219 itself provides a roll degree of freedom. Telecentric stop 220 is typically the intersection of the central axis of imaging instrument 219 and the body opening, and generally telecentric stop 220 is primarily defined by the physical structure of manipulator 270, and the telecentric stop 220 needs to be maintained stationary during surgery to reduce the effects on the human body during surgery, such as without enlarging the incision.
The manipulator assembly includes a manipulator 270 and a medical instrument 219. Illustratively, controlling the first articulation of the joints in the manipulator assembly while controlling the pitch degrees of freedom of a medical instrument, such as imaging instrument 219, about telecentric fixed point 220, e.g., controlling the pitch articulation shown in fig. 2, controls the rotation of imaging instrument 219 about telecentric fixed point 220. Illustratively, controlling the imaging instrument 219 to rotate about the telecentric fixed point 220 may involve controlling a second articulation of the joints in the manipulator assembly, e.g., controlling the yaw articulation 214 to control the imaging instrument 219 to rotate about the telecentric fixed point 220 about the yaw degree of freedom.
In some embodiments, the surgical robot further comprises an input device that may be configured to input a target motion vector of the medical instrument 219, wherein the target motion vector may be configured as a target motion vector of a distal end of the medical instrument 219, i.e. the end effector, with respect to a certain reference point, i.e. a certain reference coordinate system. The input device may be integrated with the physician's main console 103 or may be provided independently of the physician's main console 103. In some embodiments, the input device includes one or more of a linkage-type operation portion, a magnetically permeable operation portion, a voice recognition device, an eye tracking device, a head tracking device, and the like. The doctor work console 103 shown in fig. 4 includes a link type operation unit 1031.
In some embodiments, the surgical robot further comprises a control device comprising one or more processors. The one or more processors are coupled to the bedside robotic arm system 101, the doctor's main console 103, and the imaging system 108. The one or more processors may be integrated in one of the bedside robotic arm system 101, the doctor's main console 103, and the imaging system 108, may be distributed in two or more of the bedside robotic arm system 101, the doctor's main console 103, and the imaging system 108, or may be deployed in the cloud.
In some embodiments, please provide a control method of a surgical robot, the method is configured to be executed by a control device of the surgical robot, referring to fig. 5, the method includes:
step S11, acquiring a first target rotation vector of the desired imaging apparatus input by the input device, which rotates around its distal point in the target attitude degree of freedom.
Wherein the distal end of the imaging instrument generally comprises an image end effector of the molding instrument, the distal end point of the imaging instrument illustratively comprises a point in the image end effector that illustratively comprises a center point of the imaging plane.
The target attitude degrees of freedom may include a single attitude degree of freedom or may include two or more composite attitude degrees of freedom.
Step S12, determining a second target rotation vector of the imaging apparatus rotating around the telecentric fixed point in the target pose degree of freedom based on the first target rotation information.
Among them, in the surgical robot, since it is difficult to expect the distal end of the imaging instrument to rotate in the target attitude degree of freedom about the distal end point on the distal end, it is expected that the distal end of the imaging instrument rotates in the target attitude degree of freedom about other rotation centers. In combination with the safety and reliability requirements of surgical robots in performing surgery, for example, to meet the need to avoid stress tears in the body opening caused by changes in the position of the puncturing device and medical instrument, the desired field of view range adjustment may be achieved by actually translating rotation of the distal end of the desired imaging instrument about its distal point into rotation about a telecentric stop.
Step S13, controlling the manipulator to control the imaging instrument to rotate around the telecentric fixed point according to the second target rotation vector.
The imaging instrument may be steered to rotate about the telecentric stop in a number of ways.
In some embodiments, a target joint and a target joint amount thereof in the manipulator associated with the movement of the target attitude degrees of freedom may be determined based on the second target rotation vector, and then the target joint may be controlled to move the target joint amount. For example, in the bedside robotic arm system 101 shown in fig. 2, when the second target rotation vector is associated with a yaw degree of freedom, the yaw joint 214 may be determined as the target joint and the target joint amount thereof may be determined; when the second target rotation vector is associated with the pitch degree of freedom, a pitch joint, that is, a joint that controls rotation of the second link 217 with respect to the first link 216 may be determined as a target joint and a target joint amount thereof may be determined; when the second target rotation vector is associated with a yaw degree of freedom and a pitch degree of freedom, yaw joint 214 may be determined as one target joint and its target joint amount, while pitch joint may be determined as another target joint and its target joint amount. These target joints in the control manipulator 270 are then moved by their corresponding target joint amounts to steer the imaging instrument 219 about the telecentric stop 220.
In some embodiments, imaging instrument 219 may also be steered to rotate about telecentric stop 220 based on a kinematic control manipulator 270. For example, a current pose of a distal point of the imaging instrument 219 may be obtained, and a target pose of the distal point of the imaging instrument 219 may be determined based on the current pose and a second target rotation vector, and then a target joint amount of a joint in the manipulator assembly may be determined based on the target pose and using inverse kinematics, and finally joint motion in the manipulator assembly may be controlled based on the target joint amount.
In response to the imaging instrument rotating the second target rotation vector about the telecentric fixed point relative to the imaging distance deviation caused by rotating the first target rotation vector about the distal point, the control manipulator steers the imaging instrument along its axis to compensate for the imaging distance deviation, step S14.
Wherein the feeding and rotation of the imaging instrument are synchronized to ensure stable effect in the whole process of adjusting the imaging instrument.
Wherein, step S14 includes: acquiring an imaging distance deviation of the imaging instrument rotating the second target rotation vector around the telecentric fixed point relative to rotating the first target rotation vector around the distal point; based on the imaging distance deviation, the control manipulator manipulates the imaging instrument to feed along an axis of the imaging instrument.
In some embodiments, an imaging instrument includes a linkage and an image end effector coupled to a distal end of the linkage. In the imaging device, when the wrist joint is not arranged between the connecting rod and the image end effector, the imaging device is a rigid imaging device, and the image end effector does not have the performance of relative distal movement of the connecting rod; with a wrist joint between the linkage and the image end effector, the image end effector is a flexible imaging instrument that has the ability to move distally relative to the linkage. In the imaging instrument 219 (219 ') shown in fig. 3 and 6, the wrist joint is not provided between the link 2191 (2191 ') and the image end effector 2192 (2192 '), and the imaging instrument is a rigid imaging instrument.
In some embodiments, the imaging instrument may also be of an additional type. For example, the imaging optical axis of the imaging instrument may be classified according to whether or not the imaging optical axis is parallel (including overlapping) with the extending direction of the link. Wherein, if the imaging optical axis is parallel to the extending direction of the connecting rod, the imaging instrument is a zero-degree endoscope; if the imaging instrument is not parallel to the extending direction of the connecting rod, namely an included angle is formed, the imaging instrument is a non-zero endoscope. In the imaging apparatus 219 (219 ') shown in fig. 7 and 8, for example, a straight line between the telecentric fixed point 220 (220 ') and the distal end point 2193 (2193 ') may be used as the extending direction 2196 (2196 ') of the link 2191 (2191 '), and the optical axis perpendicular to the imaging plane, that is, the mirror plane 2194 (2194 '), that is, the imaging optical axis 2195 (2195 '). The imaging optical axis includes innumerable bundles of optical axes that are parallel or non-parallel to each other. For example, the imaging optical axis described herein may refer to an optical axis passing through the center of the imaging plane and/or the center of the image end effector and perpendicular to the imaging plane, which may also be referred to as a central imaging optical axis. In fig. 7, the imaging optical axis 2995 is parallel to the link extension direction 2196 and the imaging instrument 219 is a zero degree endoscope. In fig. 8, imaging optical axis 2995 'is parallel to link extension direction 2196', imaging instrument 219 'is a non-zero degree endoscope, wherein imaging optical axis 2995' is 30 ° from link extension direction 2196', and imaging instrument 219' is a 30 degree endoscope.
Wherein fig. 7 is a schematic illustration of fig. 3 and fig. 8 is a schematic illustration of fig. 6, and thus, the imaging instrument 219 shown in fig. 3 or fig. 7 is a rigid zero degree endoscope and the imaging instrument 219' shown in fig. 6 or fig. 8 is a rigid 30 degree endoscope.
The above embodiment will be described by taking a rigid zero endoscope (also simply referred to as a rigid zero endoscope) shown in fig. 7 as an example. Where the actual imaging center point at which the imaging instrument 219 is currently aligned is a, the user desires to rotate the imaging instrument 219 about the distal point 2193 to reach the first target rotation vector α to see the target imaging center B without changing the target imaging distance of the imaging instrument 219. In reality, however, the imaging instrument 219 is only controlled to rotate about the telecentric fixed point 220 because the distal end of the imaging instrument 219 is not rotatable about the distal point 2193, but only about the telecentric fixed point 220.
Referring to fig. 9, if the imaging instrument 219 is rotated about the telecentric stop 220 to reach the first target rotation vector α, the actual imaging center point C of alignment tends to deviate far from the target imaging center point B of desired alignment due to the magnification of the rigid link 2191 upon rotation. Therefore, the imaging instrument 219 cannot be directly controlled to rotate the first target rotation vector α around the telecentric motionless point 220, and further, the second target rotation vector β can be determined based on the first target rotation vector in consideration of the user's desire.
Further, referring to fig. 10, after determining the second target rotation vector β, if the imaging instrument 219 is rotated about the telecentric stop 220 to reach the second target rotation vector β, it is easy to cause the alignment actual imaging center D to have a distance difference, i.e., an imaging distance deviation, from the desired alignment target imaging center B. This imaging distance deviation will affect the imaging effect, at least the user expects a deviation of the center of field seen by the imaging instrument from the center of field actually seen, e.g. a deviation in depth of field in the optimal imaging plane.
Thus, by controlling manipulator 270 to manipulate imaging instrument 219 to advance along its axis J12 to compensate for the imaging distance deviation described above while controlling manipulator 270 to manipulate imaging instrument 219 to rotate about telecentric stop 220, it is possible to achieve consistency of the actual viewing with the desired viewing imaging effect, i.e., both the actual imaging center point and the target imaging center point being B, as shown in fig. 11. For example, it may ensure a desired sharpness of the boundary of the corresponding circle and the region outside the boundary assuming that the imaging distance is a radius centered on the distal point of the imaging instrument.
In some embodiments, determining a second target rotation vector for rotation of the imaging instrument about the telecentric fixed point in the target pose degrees of freedom based on the first target rotation vector comprises:
step 121, obtaining a first distance and a second distance. Wherein the first distance comprises a distance of the imaging instrument between the telecentric stationary point and the distal point and the second distance comprises a target imaging distance of the imaging instrument. In some embodiments, the target imaging distance comprises an imaging distance from the center of the image end effector that is perpendicular to its imaging plane, i.e., the mirror plane, which is illustratively the optimal imaging distance or other suitable imaging distance for the imaging instrument. The first target rotation vector entered by the user may be understood as an exemplary target rotation vector for which the center rotation is desired. In some embodiments, the center of the image end effector may be a feature point associated with a distal point of the imaging instrument, which may coincide with the distal point or be slightly offset from the distal point. For example, the distal point of the imaging instrument may be the center of the image end effector. For another example, the center of the imaging plane in the image end effector may be taken as the center of the image end effector. Step 122, determining a second target rotation vector of the imaging instrument rotating around the telecentric fixed point in the target pose degree of freedom in combination with the first target rotation vector, the first distance, and the second distance.
In one embodiment, a trigonometric function may be utilized and the second target rotation vector may be determined based on the first target rotation vector, the first distance, and the second distance. With continued reference to fig. 10, for example, the determination may be made using an arctangent trigonometric function, and the second target rotation vector may be determined by the following formula:
where α represents a first target rotation vector, β represents a second target rotation vector, L represents a first distance, and d represents a second distance.
For example, based on the same principle, the second target rotation vector may also be determined by trigonometric function formulas such as anticontrol, arcsine, arccosine, etc., which are not illustrated here.
In some embodiments, the step S14 described above, the controlling the manipulator to manipulate the imaging apparatus to feed along the axis thereof, includes:
s141, acquiring an initial position and/or an initial posture of the imaging instrument.
The initial position and/or initial posture of the imaging instrument comprises the position and/or posture of the input device and the manipulator assembly at the moment of establishing a master-slave mapping relation, and the zero state of the imaging instrument is expressed. The initial position and/or initial pose of the imaging instrument may refer to an initial position and/or initial pose of a feature region on the imaging instrument, which feature region in this embodiment comprises some preset region located distally of the rotation point. Wherein the feature region comprises a region of one or more points.
S142, the current position and/or the current posture of the imaging instrument are acquired.
Wherein the current position and/or current pose of the imaging instrument may be determined based on current joint variables of a manipulator assembly, e.g., a joint in a manipulator, and using positive kinematics.
S143, determining a target direction along which the imaging apparatus is fed along the axis thereof based on a positional relationship and/or a posture relationship between the target position and/or the target posture of the imaging apparatus compared to the current position and/or the current posture thereof and the initial position and/or the initial posture.
Wherein the target position and/or target pose of the imaging instrument is determined based on the second target rotation vector. More specifically, the target position and/or target pose of the imaging instrument may be determined based on the second target rotation vector and a current joint variable of a joint in a manipulator assembly, e.g., a manipulator, and using positive kinematics.
Wherein the target direction is determined to be the direction in which the imaging instrument withdraws the body opening along its axis when the target position and/or target pose of the imaging instrument is relatively far from the initial position and/or initial pose as compared to its current position and/or current pose; alternatively, the target direction is determined to be the direction along which the imaging instrument is inserted into the body opening along its axis when the target position and/or target pose is relatively close to the initial position and/or initial pose as compared to its current position and/or current pose.
S144, controlling the imaging instrument to feed along the axis of the imaging instrument in the target direction so as to compensate imaging distance deviation.
Wherein the feed direction of the imaging instrument is substantially consistent with the target direction, and the feed amount of the imaging instrument is substantially consistent with the imaging distance deviation.
In some embodiments, the imaging distance deviation may be determined based on the following principles:
a first length of the imaging instrument between the telecentric stationary point and the current imaging center is obtained, and a second length of the imaging instrument between the telecentric stationary point and the actual imaging center is obtained assuming a second rotation vector around the telecentric stationary point, based on which an imaging distance deviation can be determined.
Illustratively, based on the above principle, the imaging distance deviation may be determined as follows:
step S21, a first distance and a second distance are acquired.
Wherein the first distance comprises a distance of the imaging instrument between the telecentric stationary point and the distal point and the second distance comprises a target imaging distance of the imaging instrument.
In one embodiment, a method for acquiring a first distance includes:
acquiring joint variables of joints in the manipulator assembly; combining the joint variable and a first kinematic model of the manipulator assembly, determining a first position of a distal point in a reference frame by utilizing positive kinematics, and determining a second position of a telecentric fixed point in the reference frame; a first distance is determined based on the first location and the second location.
In one embodiment, the method for obtaining the second distance includes:
acquiring an imaging distance range of an imaging instrument; generating a configuration interface comprising one or more selectable target imaging distances based on the imaging distance range; in response to selection of the target imaging distance by the configuration interface, the selected target imaging distance is taken as the second distance. Through flexible configuration of the second distance, different imaging effects can be achieved to meet different surgical demands.
With continued reference to fig. 10, the first length described above may be determined based on a sum of the first distance and the second distance. Wherein:
l1=l+d type (2)
Wherein L1 represents a first length, L represents a first distance, and d represents a second distance.
Step S22, combining the first target rotation vector, the first distance and the second distance, and determining the imaging distance deviation.
The second length may be determined based on the first target rotation vector, the first distance, and the second distance, for example, by using a trigonometric function, and the calculation formula may be exemplarily expressed as follows:
where L2 represents the second length and α represents the first target rotation vector.
Δl=l1-L2 (4)
Where Δl represents the imaging distance deviation, and L1 represents the first length. For example, in fig. 10, Δl represents the distance between DBs.
Substituting the formula (2) and the formula (3) into the formula (4) can obtain:
that is, the imaging distance deviation can be determined according to the equation (5).
The embodiment of controlling the imaging apparatus by determining the second target rotation vector and/or the imaging distance deviation based on the trigonometric function can be better suitable for the use scene of the zero-degree endoscope, and is particularly suitable for the use scene of the rigid zero-degree endoscope. The bedside robotic arm system shown in fig. 2 may employ either a rigid zero degree endoscope or a rigid non-zero degree endoscope.
In some embodiments, rigid zero degree endoscopes and rigid non-zero degree endoscopes are typically implemented using structural designs, e.g., in rigid zero degree endoscopes, the mirror surface may be disposed perpendicular to the direction of extension of the links. For another example, in a rigid non-zero endoscope, the mirror surface may be provided obliquely to the extending direction of the link.
In some embodiments, the flexible endoscope is generally adapted for and manipulated by a manipulator of a single-hole surgical robot to provide more degrees of freedom of movement. The telecentric dead point of the single-hole surgical robot can be controlled by a software algorithm or defined by a physical structure of a manipulator with a parallelogram mechanism, which is the same as the bedside robotic arm system shown in fig. 2. Flexible endoscopes include flexible zero degree endoscopes and flexible non-zero degree endoscopes. The flexible zero-degree endoscope may be defined as a flexible endoscope in which the imaging optical axis is parallel to (including coincides with) the extending direction of the link in an initial (i.e., zero position) state of the link and the wrist joint, in which state the imaging plane is set perpendicular to the extending direction of the link. The flexible non-zero endoscope may be defined as a flexible endoscope in which the imaging optical axis and the extending direction of the link have an angle in an initial state of the link and the wrist joint, and in this state, the imaging surface may be set to be inclined to the extending direction of the link.
Wherein, the configuration of the flexible zero-degree endoscope can be changed by controlling the movement of the wrist joint so as to realize the use requirement of the non-zero-degree endoscope; and the configuration of the flexible non-zero endoscope can be changed by controlling the wrist joint to move so as to realize the use requirement of the zero endoscope.
In some embodiments, maintaining the wrist joint and linkage in a straight line for a flexible zero degree endoscope may be applicable to the various embodiments of S11-S14 described above. For example, when the technical means described in each of the embodiments of S11 to S14 is to be applied, if the wrist joint and the link are not in a straight line in the flexible zero-degree endoscope, the wrist joint is controlled to be returned to a zero state relative to the link, that is, to be in a straight line.
For example, the initial joint variation of the wrist joint may be recorded when the wrist joint and the link are in a straight line. When the wrist joint motion is controlled to enable the wrist joint and the connecting rod to be in a straight line, the corresponding wrist joint can be controlled to reset directly based on the current joint variable and the initial joint variable of the wrist joint.
In some embodiments, where the rotation of the imaging instrument about its distal end point by the first target rotation vector is translated into rotation about other centers of rotation, such as a telecentric fixed point, other means may be employed to achieve the desired imaging effect or to maintain the imaging effect unchanged, where the distance between the distal end of the imaging instrument and the target imaging center remains consistent with the target imaging distance as the center of rotation of the imaging instrument is changed as previously described. The control method generally includes: the target position and/or posture of the target imaging center of the imaging apparatus in a reference coordinate system such as a basic coordinate system of a bedside mechanical arm system is obtained, then a target joint amount of a joint in the manipulator assembly is determined based on inverse kinematics, and then the joint movement in the manipulator assembly is controlled according to the target joint amount, so that the imaging center of the imaging apparatus reaches the target position and/or posture from the current position and/or posture. The method is applicable to imaging instruments with or without wrist joints and is a versatile method. For example, referring to fig. 12, the control method includes:
Step S11' acquires a first target rotation vector input by the input device about which the imaging instrument is expected to rotate in a target pose degree of freedom about its distal point.
Wherein the first target rotation vector comprises a target rotation vector of the distal end of the desired imaging instrument input by the input device about its distal point rotated in a target pose degree of freedom.
Step S12' determines a target position and/or pose of a target imaging center of the imaging instrument in a reference coordinate system based on the first target rotation vector.
Step S13' determines a target joint variable for a joint in the manipulator assembly based on the target position and/or pose.
Wherein the target joint variables of the joints in the manipulator and imaging apparatus are determined.
Step S14' controls articulation in the manipulator assembly in accordance with the target articulation variable to bring the imaging center of the imaging instrument to the target imaging center.
Wherein the joints in the manipulator and imaging apparatus are controlled to move according to the target joint variable.
By the above steps S11 'to S14', the same imaging effect can be achieved without determining the second target rotation vector and the imaging distance deviation required in the above steps S11 to S14.
In some embodiments, determining a target position and/or pose of a target imaging center of an imaging instrument in a reference coordinate system based on a first target rotation vector comprises:
Step 131' determines a first position and/or pose of the target imaging center relative to a distal point of the imaging instrument.
For example, assume that the current position and/or pose of the far-end point in the reference coordinate system is P 0 (P x0 ,P y0 ) And assuming that the imaging instrument rotates a first target rotation vector α about the distal point in the two-dimensional xy-plane, a first position and/or pose of the target imaging center relative to the distal point of the imaging center is P 1 (P x0 +dcosα,P y0 +dsinα)。
Step 132' determines a target position and/or pose of the first position and/or pose in the reference coordinate system based on the current position and/or pose of the distal point in the reference coordinate system.
Wherein the current position and/or pose of the distal point of the imaging instrument in the reference coordinate system may be determined based on the first kinematic model associated with the manipulator assembly and the acquired current joint variables of the joints in the manipulator assembly.
In some embodiments, referring to fig. 13, determining a target joint variable for a joint in a manipulator assembly based on a target position and/or pose includes:
step S141' acquires configuration parameters of an imaging optical axis of the imaging instrument.
In some embodiments, the configuration parameters of the imaging optical axis include the length of the imaging optical axis and the angle of the imaging optical axis relative to an imaging plane, such as a mirror. The configuration parameters of the imaging optical axis may be determined based on the acquired attribute information of the imaging instrument. Wherein the attribute information of the imaging instrument includes an imaging distance range of the imaging instrument, the imaging distance range of the imaging instrument including at least one of a minimum imaging distance, a maximum imaging distance, and an optimal imaging distance between the minimum imaging distance and the maximum imaging distance. The attribute information of the imaging instrument also includes a type of imaging instrument, including a zero degree endoscope or a non-zero degree endoscope.
In some embodiments, the length of the imaging optical axis may be determined based on the acquired imaging distance range of the imaging instrument, wherein the length of the imaging optical axis may be understood as the target imaging distance. For example, when the imaging distance range includes a minimum imaging distance, any imaging distance equal to or greater than the minimum imaging distance may be configured as the length of the imaging optical axis. For example, when the imaging distance range includes an optimal imaging distance, the optimal imaging distance may be configured as a length of the imaging optical axis. For example, when the imaging distance range includes a minimum imaging distance and a maximum imaging distance, any imaging distance between the minimum imaging distance and the maximum imaging distance may be configured as the length of the imaging optical axis. In some embodiments, the imaging range of distances of the imaging instrument is related to its focal length, the imaging range of distances being relatively unique for imaging instruments having fixed focus image end effectors, and the imaging range of distances including different imaging ranges of distances corresponding to different focal lengths for imaging instruments having zoomed image end effectors.
In some embodiments, the angle of the imaging optical axis relative to the imaging plane may be determined based on the type of imaging instrument acquired. For example, when the imaging instrument is acquired as a zero degree endoscope, it may be determined that the imaging optical axis is at an angle of 90 degrees with respect to the imaging plane. For another example, when an imaging instrument of +30 degrees endoscope is acquired, it may be determined that the imaging optical axis is at +60 degrees with respect to the imaging plane. For another example, when an imaging instrument is acquired as a-30 degree endoscope, it may be determined that the imaging optical axis is at an angle of-60 degrees relative to the imaging plane. In some embodiments, the first angle of the imaging optical axis relative to the extending direction of the connecting rod and the second angle of the imaging optical axis relative to the imaging surface are generally complementary angles. For example, the second included angle may be determined based on the first included angle when the first included angle is stored in the memory chip. For another example, the second angle may be stored directly in the memory chip.
The attribute information of the imaging instrument can be stored in a memory chip of the imaging instrument, and when the imaging instrument is arranged on the manipulator, the attribute information is read by a reading interface arranged in the manipulator and transmitted to the control device for processing.
Step S142' constructs a second kinematic model associated with the manipulator assembly and the imaging optical axis in combination with the configuration parameters of the imaging optical axis.
In some embodiments, the second kinematic model may only correlate the manipulator assembly and the imaging optical axis. In some embodiments, the second kinematic model may be associated with at least a portion of the drive arm including the manipulator assembly and the imaging optical axis. The aforementioned first kinematic model does not take into account the imaging optical axis, and the second kinematic model herein takes into account the imaging optical axis, in particular the length of the imaging optical axis and its angle relative to the imaging plane, and thus differs from the first kinematic model. More specifically, the construction of the first kinematic model is associated with only the physical arm structure, while the construction of the second kinematic model is associated with not only the physical arm structure but also the virtual arm structure, i.e., the imaging optical axis. In the second kinematic model, when the imaging device is extended and the imaging device is formed into a physical shape, the distance between the imaging device and the target imaging center, i.e., the target imaging distance, can be always maintained at the length of the imaging optical axis when the imaging device reaches the target imaging center, as in the case of the distal point 2193 shown in fig. 7 or the distal point 2193' shown in fig. 8.
Step S143' determines a target joint variable for a joint in the manipulator assembly based on the target position and/or pose, the second kinematic model, and using inverse kinematics.
Wherein, for example, when the second kinematic model only correlates the manipulator assembly and the imaging optical axis, a target joint variable for a joint in the manipulator and the imaging instrument may be determined. For another example, where the second kinematic model is associated with the entire drive arm including the manipulator assembly and the imaging optical axis, a target joint variable for a joint in the drive arm may be determined.
In some embodiments, a second, different kinematic model may be constructed in order to achieve a user's desire to see a different image effect. In some embodiments, configuration parameters of the imaging optical axis may be configured. For example, the angle of the imaging optical axis relative to the imaging plane may be configured, and/or the length of the imaging optical axis may be configured.
For example, a configuration interface including one or more selectable target imaging distances may be generated based on the imaging distance range; in response to selection of the target imaging distance by the configuration interface, the selected target imaging distance is taken as the length of the imaging optical axis. For example, where the imaging distance range includes a minimum imaging distance and a maximum imaging distance, a plurality of target imaging distances may be generated between the minimum imaging distance and the maximum imaging distance for configuration.
For example, a configuration interface may be generated that includes one or more selectable target angles; in response to selection of the target angle by the configuration interface, the selected target angle is taken as the angle of the imaging optical axis relative to the imaging plane. For example, a plurality of target angles between 0-90 may be generated for configuration.
Of course, the length of the imaging optical axis may be implemented in other manners, and/or the configuration of the angle of the imaging optical axis with respect to the imaging plane may be configured by voice recognition or the like, for example.
In some embodiments, the method of the above embodiments is particularly useful for imaging instrument rotation in pitch and/or yaw degrees of freedom, provided that the center of rotation of the imaging instrument is changed, since rotation of the imaging instrument in the autorotation degree of freedom (i.e., roll degree of freedom) does not change the target imaging distance, while rotation in the pitch degree of freedom and/or yaw degree of freedom changes the target imaging distance.
The first target rotation vector described above may be input in a variety of ways. For example, the input of the first target rotation vector may be realized by an operation unit provided in the main console. The input of the first target rotation vector may be realized by a touch screen provided on the main console, for example.
Generally, the imaging instrument and the surgical instrument are controlled by manipulating the operation portion. When a doctor is operating the surgical instruments with both hands, if the imaging instruments need to be controlled to adjust the field of view, the doctor needs to pause at least the control of one surgical instrument to switch the control of the imaging instruments, however, in operation, there is generally a high switching frequency between the imaging instruments and the surgical instruments, and such switching operation is complex and inefficient. Moreover, even if the surgical instrument can be operated by one hand and the imaging instrument can be operated by the other hand at the same time, the coordination problem exists, the user experience is poor, and the requirements of doctors are difficult to meet. Accordingly, the present application provides an additional form of input device that enables control of an imaging instrument without requiring manipulation by a physician's hand. In other embodiments, the input of the input device may also control the surgical instrument according to the need, only the master-slave mapping relationship between the input device and the surgical instrument is required to be established, and it is noted that, as the control of the surgical instrument does not have the problem related to the imaging distance, for the surgical instrument, only the distal end point of the surgical instrument is required to be controlled to move according to the target movement information input by the input device, where the target movement information may include not only the movement information in the attitude degree of freedom, but also the movement information in the position degree of freedom.
In some embodiments, the input device includes a voice recognition component, and the doctor can generate a specific instruction by making a specific sound and processing the specific instruction by the voice recognition component, wherein the specific instruction corresponds to the first target rotation vector. For example, the physician may issue instructions of "left", "right", "up", "down", "up-left", "up-right", "down-left", "down-right", "forward", "backward", etc., to generate incremental motion vectors for the corresponding degrees of freedom. For example, the physician may issue instructions of "5 degrees left", "5 degrees right", "1 cm forward", "1 cm backward", etc., resulting in incremental motion vectors for the corresponding degrees of freedom. Further, the rotation vector in the incremental motion vector may be the first target rotation vector.
In some embodiments, the input device includes a sensing assembly that senses movement of the user's head, including any movement in space, including rotation and translation. The motion of the user's head sensed by the sensing assembly is an incremental motion vector corresponding to the degree of freedom, and a rotation vector in the incremental motion vector may be used as the aforementioned first target rotation vector.
In some embodiments, the sensing assembly includes one or more sets of sensors, each set of sensors including one or more sensors. For example, the sensing assembly includes a first set of sensors for sensing movement of the user's head in a first attitude degree of freedom, such as a yaw degree of freedom, which corresponds to a side-to-side rotation of the user's head. For example, the sensing assembly includes a second set of sensors for sensing movement of the user's head in a second degree of attitude freedom, e.g., a pitch degree of freedom, which corresponds to up and down rotation of the user's head. For example, the sensing assembly includes a third set of sensors for sensing movement of the user's head in a first positional degree of freedom, e.g., depth, that moves back and forth.
The sensors of the first, second and third sets of sensors may be of the same type, or may be of completely or partially different types, and may be selected from force sensors, deformation sensors, distance sensors, etc. For example, the force sensor may be further selected from a pressure sensor, a torsion sensor, and the like. For example, the distance sensor may be further selected from optical sensors such as infrared sensors and the like. The movement of the user's head may be monitored by sensing the force applied by the user's head and/or the distance moved.
In one embodiment, the sensors in the first, second and third sets of sensors may each employ a force sensor.
For the first and/or second attitude degrees of freedom described above, the corresponding set of sensors (first and/or second set of sensors) includes at least two force sensors, such as pressure sensors, disposed at both ends in the respective attitude directions, and the direction of movement of the user's head can be determined by the difference in force, such as pressure difference, sensed by the force sensors at both ends in the corresponding set of sensors, for example, in the yaw degree of freedom corresponding to the left-right rotation of the user's head, if the left-end force sensor is subjected to a pressure greater than the right-end force sensor, the direction of movement of the user's head is determined to be to the left. Further, an accumulated time at which a force difference exists may be recorded, and a rotation vector in the first and/or second attitude degrees of freedom may be determined based on the movement direction and the accumulated time.
The control device can compare the force difference with a force difference threshold value, and when the force difference exceeds the force difference threshold value, the intention of the user for adjusting the imaging instrument is indicated, and then the moving direction of the head of the user is determined according to the condition of the force difference; and when the force difference is smaller than the force difference threshold value, the user does not have the intention of adjusting the imaging instrument, and the moving direction of the head of the user is not required to be determined. By judging the force difference and the force difference threshold value, the sensitivity of the response of the imaging instrument and the like can be reduced, and the problem of preventing false touch is further solved. Further, the control means records the cumulative time for the force difference to exceed the force difference threshold and determines the rotational vector in the first and/or second attitude degrees of freedom based on the direction of movement, the speed of movement and the cumulative time.
In one example, the movement speed may be configured as a unit speed without regard to other factors.
In one example, the velocity of movement of the imaging instrument may also be determined based on the magnitude of the force differential. For example, the movement speed of the imaging instrument may be determined from the ratio of the force difference to the force difference threshold when the force difference exceeds the force difference threshold, and in combination with the unit speed, for example, the movement speed may be obtained by multiplying the ratio by the unit speed, whereby the determined movement speed is relatively linear. For another example, the movement speed of the imaging instrument may be determined based on the degree of difference between the force difference and the force difference threshold, in combination with the unit speed, e.g., when the degree of difference is within a first degree, the movement speed is determined using the sum of the unit speed and the first increasing speed, and when the degree of difference is within a second degree, the movement speed is determined using the sum of the unit speed and the second increasing speed, whereby the determined movement speed has a stepwise nature. Preferably, the movement speed has a maximum value to ensure safety, and the movement speed is configured to be the maximum value when the movement speed is adjusted linearly or stepwise or in other manners to reach the maximum value.
For the first degree of freedom of the position described above, the corresponding set of sensors (third set of sensors) comprises at least one force sensor arranged at one end of the degree of freedom of the position. For example, the force sensor may be one, and the force sensor is usually disposed on the front side of the user's head, i.e. on the face side, and in normal operation, the user's head is usually attached to or at least acts on the force sensor, and in the case that the force sensor senses a force (e.g. the force is not zero), if the force is between the first force threshold and the second force threshold, which is smaller than the first force threshold, this means that the user is attached to the force sensor normally, without the intention of adjusting the imaging apparatus; the force sensor senses that the force is greater than a first force threshold or less than a second force threshold, wherein if the force is greater than the first force threshold, the direction of movement of the user's head is determined to be in the forward direction, and if the force is less than the second force threshold, the direction of movement of the user's head is determined to be in the reverse direction, e.g., the force is between 0 and the second force threshold when the direction of movement associated with the user's head is in the reverse direction. By way of example, a first threshold of 6N may be configured and a second threshold of 4N may be configured to provide a comfortable pressure feel for user operation, where forces sensed by the force sensor are between 4N and 6N (including 4N and 6N) without the desire and intent to adjust the imaging instrument, i.e., without the need to determine the direction of movement of the user's head; when the force sensed by the force sensor is greater than 6N, determining that the head of the user moves towards the front direction; when the force sensed by the force sensor is smaller than 4N, the head of the user is determined to move towards the reverse direction. Wherein the movement of the user's head in the frontal direction is associated with insertion of the imaging instrument in the depth direction along its axis, and the movement of the user's head in the frontal direction is associated with withdrawal of the imaging instrument in the depth direction along its axis. Further, the control device records a cumulative time when the force is greater than the first force threshold or less than the second force threshold, and may determine a motion vector in the first positional degree of freedom, i.e., in the axial direction of the imaging instrument, based on the moving direction, the moving speed, and the cumulative time.
In one example, the movement speed may be configured to achieve a configured unit speed in the feed direction of the imaging instrument, i.e., the axial direction, without regard to the others. In yet another example, the movement speed may be dynamically determined based on the unit speed, and the force versus the first threshold and/or the second threshold. For example, when the force is greater than a first threshold, determining the movement speed of the imaging instrument according to the ratio of the force to the first threshold, and combining the unit speed, e.g., obtaining the movement speed by using the product of the ratio and the unit speed, wherein the determined movement speed is relatively linear; for another example, when the force is less than the second threshold, the movement speed of the imaging instrument is determined based on the ratio of the force to the second threshold, and the unit speed is combined, e.g., the product of the ratio and the unit speed is used to obtain the movement speed, and the movement speed thus determined is also relatively linear. Preferably, the movement speed also has a maximum value to ensure safety, and the movement speed is configured to be the maximum value when the movement speed reaches the maximum value by one or more ways of adjusting the movement speed.
When the sensors of the first, second and/or third set of sensors comprise deformation sensors, the deformation sensors may be illustratively configured with reference to pressure sensors, the deformation of the deformation sensors being substantially due to the force exerted by the user's head. For example, a deformation sensor is disposed at each of two ends of the corresponding degree of freedom of the posture, and a deformation sensor is disposed at the corresponding degree of freedom of the position. Further, for example, in the attitude degree of freedom, the moving direction of the user's head may be determined according to the deformation difference sensed by the deformation sensors at both ends and the deformation difference threshold, and the accumulated time for which the deformation difference exceeds the deformation difference threshold may be recorded, so that the moving direction, the moving speed and the accumulated time may be used to determine the motion vector in the corresponding degree of freedom. The movement speed may also be configured as a unit speed or may be dynamically adjusted in advance linearly or stepwise or the like based on the unit speed and related factors. And meanwhile, a maximum value can be set for the movement speed, and when the dynamically adjusted movement speed is greater than the maximum value, the movement speed is determined to be the maximum value.
For another example, in terms of positional degrees of freedom, processing is simpler, for example, when the deformation amount is between a first deformation amount threshold and a second deformation amount threshold smaller than the first deformation amount threshold, it is indicated that the user is normally attached to the deformation sensor without the intention to adjust the imaging apparatus, and when the deformation amount is greater than the first deformation amount threshold or smaller than the second deformation amount threshold, it is indicated that the user has the intention to adjust the imaging apparatus, wherein if the deformation amount is greater than the first deformation amount threshold, it is determined that the movement direction of the user's head is toward the front direction, and if the deformation amount is smaller than the second deformation amount threshold, it is determined that the movement direction of the user's head is toward the back direction. The amount of deformation induced is typically non-zero when the user is operating normally, e.g. between 0 and a second deformation threshold when the direction of movement associated with the user's head is in the opposite direction. The amount of movement can also be determined in relation to the unit speed and the accumulated time, and the movement speed should also have a maximum value, and a detailed description thereof will not be repeated.
When the sensors of the first, second and/or third set of sensors comprise distance sensors, the distance sensors may be illustratively configured with reference to the pressure sensors, e.g., with distance sensors disposed at both ends of the respective degrees of freedom of posture, and a distance sensor disposed at the respective degrees of freedom of position. Further, for example, in the degree of freedom of the posture, the moving direction of the user's head may be determined based on the distance difference sensed by the distance sensors at both ends and the distance difference threshold, and the cumulative time for which the distance difference exceeds the distance difference threshold may be recorded, and the moving direction, the moving speed, and the cumulative time may be used to determine the moving vector in the corresponding degree of freedom. The movement speed may also be configured as a unit speed or may be dynamically adjusted in advance linearly or stepwise or the like based on the unit speed and related factors. And meanwhile, a maximum value can be set for the movement speed, and when the dynamically adjusted movement speed is greater than the maximum value, the movement speed is determined to be the maximum value.
For another example, in the positional degrees of freedom, the processing is simpler, for example, when the distance amount is between a first distance amount threshold and a second distance amount threshold that is smaller than the first distance amount threshold, it is indicated that the user is at a normal distance from the distance sensor, and when the distance amount is greater than the first distance amount threshold or smaller than the second distance amount threshold, it is indicated that the user has an intention to adjust the imaging apparatus, wherein if the distance amount is greater than the first distance amount threshold, it is determined that the movement direction of the user's head is toward the front direction, and if the distance amount is smaller than the second distance amount threshold, it is determined that the movement direction of the user's head is toward the back direction. The sensed distance is typically non-zero when the user is operating normally, e.g. between 0 and a second distance threshold when the direction of movement associated with the user's head is facing in the opposite direction. The amount of movement can also be determined in relation to the unit speed and the accumulated time, and the movement speed should also have a maximum value, and a detailed description thereof will not be repeated.
In the above-described embodiment, the motion vector may be used as target movement information for which the imaging instrument is desired to be fed in the axial direction thereof, and the control device may control the manipulator to manipulate the imaging instrument to be fed in the axial direction thereof based on the target movement information after acquiring the target movement information.
In some embodiments, referring to fig. 16 and 20, the input device includes one or more beacons 61 and one or more detectors that detect the position of the one or more beacons 61 in space. The beacon 61 may optionally comprise an active or passive beacon, the beacon 61 may optionally comprise a coil, a metal sheet, a magnet, or the like. The detector may optionally include a detector that emits a magnetic field, an electric field, infrared light, or the like. In some embodiments, the beacon 61 may be configured in a wearable device, e.g., comprising a cap 63, a mask 64, and the wearable device may further comprise accessories, e.g., glasses, earrings, hair clips, stickers, etc., that facilitate the wearing of the user's head. By detecting a change in the position of one or more beacons 61 in the wearable device 63 and/or 65 at different moments in time, for example detecting a change in the position of one, two, three, four beacons 61, the movement of the user's head can be determined. The wearable device provided with a plurality of beacons 61 allows posture monitoring of almost the entire head of the user, not being limited to head parts such as face parts or forehead parts, and thus the sensitivity and accuracy of detection are higher, and false touches caused by other unintentional actions such as relaxing the head of the user are correspondingly less.
In some embodiments, when the beacons include more than three beacons, at least two lines may be formed between different beacons 61, i.e., not necessarily disposed on the same line. In some embodiments, when the beacon 61 includes more than four beacons, at least two planes may be formed between different beacons 61, i.e., not necessarily disposed on the same plane. This facilitates the realization of the attitude positioning.
Optionally, the motion information associated with the movement of the user's head may be configured to control the motion of any portion of any medical instrument, including imaging instruments and surgical instruments, for example, the motion information may correspond to control information configured to control the motion of the imaging instrument about a telecentric stop, or may be configured to control the motion of the imaging instrument about a distal point, although other may be used, and are not listed herein.
In some embodiments, the detector comprises a magnetic field generator for generating a magnetic field in a space. Accordingly, the beacon includes a magnetic sensor. The change of the magnetic sensor at different moments in the magnetic field is detected by sensing the change of the magnetic field intensity of the magnetic sensor in the magnetic field, so that the movement of the head of the user can be determined. In some embodiments, when the beacon, such as the magnetic sensor, includes a plurality of beacons, the identity identifier may be configured for the beacon worn by the same user, so that the movement of the head of the specific user can be accurately determined based on the obtained position change of the beacon with the same identity identifier sensed in the detector, such as the magnetic field generator.
In some embodiments, the probe includes a base station that can receive and/or transmit wireless communication signals, such as a signal transceiver; the beacons include beacons that transmit and/or receive signals for wireless communications, including for example signal transceivers. The signal transceiver may include a signal transceiver supporting one or more of bluetooth, 2G, 3G, 4G, 5G, infrared, wiFi, zigbee, etc. The more base stations, the more accurate the positioning. The position change of the beacon in space can be determined by acquiring the distances between the beacon and different base stations at different moments, and the movement of the user head can be determined.
In some embodiments, as shown in fig. 14, the input device includes an eye tracking device, which is mainly used for automatically controlling the pitching degree of freedom and the yawing degree of freedom of the medical apparatus, such as the imaging apparatus, and the rolling degree of freedom and the axial depth motion of the imaging apparatus, and may be controlled by a user's hand or foot, such as a foot control pedal, or a hand control button, etc. The eyeball tracking device comprises one or more camera devices 303, wherein the camera devices 303 are associated with a viewing assembly in an image host, for example, are arranged at the periphery of a display unit 302 of the viewing assembly and are used for tracking pupils of a user so as to acquire an area of a surgical field of view watched by eyes 301 of the user, and a control device responds to the acquired gazing area and controls an imaging instrument to move towards the direction associated with the area so as to expand the field of view.
By way of example, fig. 15 shows a screen diagram of an image host, wherein a surgical field 401 may be configured as a rectangular field, and a series of peripheral fields may be provided around the surgical field, such as above 402, below 403, left 404 and right 405 sides of the surgical field. When the user's gaze point falls above 402 or below 403 the edge of the surgical field, the user may consider that the user wishes to expand, i.e. adjust, the surgical field of view above or below, and accordingly, the surgical robot may drive the imaging instrument to adjust the pitch degree of freedom, or possibly adjust both the pitch degree of freedom and the axial depth. When the user's gaze point falls to the left 404 or right 405 of the edge of the surgical field, the user may consider that he wants the surgical field to be expanded to the left or right, and accordingly, the surgical robot may drive the endoscope to adjust the yaw degree of freedom, or may adjust both the yaw degree of freedom and the axial depth.
In some embodiments, the imaging apparatus may be adjusted to move toward the corresponding region only after detecting that the duration of the user's gaze continues to look at the specific region for a threshold time, e.g., when the threshold time is 2 seconds, the adjustment may be performed in the corresponding direction after the gaze continues for 2 seconds, so as to prevent false triggering of the adjustment of the imaging apparatus. Of course, the length and sensitivity of the period of time may also be adapted and set accordingly, e.g., to facilitate user configuration, a configuration interface may be generated for user configuration that includes one or more configuration parameters of the length and/or sensitivity of the period of time.
Through the various methods described above, it is possible to obtain, for example, a first target rotation vector in which the user desires the imaging instrument to rotate about the distal point, and/or target movement information fed in the axial direction thereof.
In some embodiments, as shown in fig. 16, the surgical robot further includes an image host 500. The image host includes a viewing component 501 and a pose adjustment component 502. The pose adjustment assembly 502 is used to adjust the pose of the viewing assembly 501, and the viewing assembly 501 is used to view images acquired by the imaging instrument. The control device is coupled to the viewing assembly 501 and the attitude adjustment assembly 502 and is configured to:
the gesture adjustment assembly is controlled to move according to the first target rotation vector, so that the observation assembly moves in the target gesture degree of freedom along with the movement of the head of the user. Therefore, when the head gesture of the user changes, the observation component for observing the image can be correspondingly and automatically adjusted; particularly, when the movement of the imaging device is related to the first target rotation vector for movement, the movement of the imaging device is always synchronous with the movement of the observing assembly, the movement of the observing assembly is always matched with the related movement of the head of the user, and the three movements comprise movements with basically the same speed, so that when the head of the user moves, the eyes of the user can always see a desired image, and the eyes of the user and the center of the image always keep a relatively constant state.
In some embodiments, with continued reference to fig. 16, the attitude adjustment assembly 502 includes a base 503, a first pivot 504, and a second pivot 505. The base 503 may be disposed on any fixed or movable object or structure, for example, the base 503 may be disposed on a wall or a ceiling, and when the base 503 is disposed on a fixed object or structure, the image host 500 has a relatively high rigidity, and is not prone to vibrating during movement, so that the stability of the image for viewing by the user can be maintained; for another example, the base 503 may be disposed at a distal end of a robot arm having one or more degrees of freedom, and the motion performance of the image host may be extended by means of the degrees of freedom of the robot arm. Referring to fig. 4, the image mainframe 500 may be integrated in the doctor main console 103, the image mainframe 500 may be accommodated in an operation space 510 of the doctor main console 103, a base 503 of the posture adjustment assembly 502 is relatively fixedly disposed in the operation space 510, and the viewing assembly 501 may move freely in the operation space 510 by means of the movement properties of the first pivot member 504 and the second pivot member 505 of the posture adjustment assembly 502.
Further, the first pivot member 504 is pivotally coupled to the base 503 and configured to be rotatable with respect to the base 503 in a first attitude degree of freedom. The second pivot member 505 is pivotally connected to the first pivot member 504 and is configured to be rotatable with respect to the first pivot member 504 in a second attitude degree of freedom. The viewing assembly 501 is fixedly coupled relative to the second pivot member 505. Further, the viewing assembly 501 may be positionally adjustable in at least one of the first and second degrees of freedom.
The image host 500 further includes a first driving mechanism and a second driving mechanism coupled with the control device. The first drive mechanism is configured to drive rotation of the first pivot member 504 in a first attitude degree of freedom relative to the base 503, and the second drive mechanism is configured to drive rotation of the second pivot member 505 in a second attitude degree of freedom relative to the first pivot member 504.
In some embodiments, referring to fig. 17, the base 503 includes a first curved track 5031, the first pivot 504 includes a second curved track 5041 and a third curved track 5042, the second pivot 505 includes a fourth curved track 5051, the first curved track 5031 and the second curved track 5041 cooperate to be rotatable in a first pose degree of freedom, and the third curved track 5042 and the fourth curved track 5051 cooperate to be rotatable in a second pose degree of freedom. The second curved sliding rail 5041 and the third curved sliding rail 5042 are generally disposed on opposite sides of the first pivot member 504, one side of which is configured to slidably engage the base 503, and the other side of which is configured to slidably engage the second pivot member 505.
Referring to fig. 18, the observation assembly 501 includes a display unit, the display unit includes a 2D or 3D display unit, and the display unit is coupled to the control device and is relatively fixedly disposed on the second pivot member 505. The viewing assembly 501 may also include a viewing window 5011, also referred to as a binocular viewing window, the viewing window 5011 being fixedly disposed relative to the display unit for viewing images displayed by the display unit. In some embodiments, a cushion 5012 for the user to rest against may be disposed above the viewing window 5011 to improve the comfort of the user's forehead when the user's eyes view the image displayed by the display unit proximate to the viewing window 5011. One or more of the above-described various sensors may be provided in the back cushion 5012 or may be provided at a position outside the back cushion 5012.
In some embodiments, as shown in fig. 19, the base 503 'in the attitude adjustment assembly 502' includes a first curved track 5031', the first pivot member 504' includes a second curved track, the first curved track 5031 'and the second curved track cooperate to be rotatable in a first attitude degree of freedom, and the attitude adjustment assembly 502' further includes a rotational joint 506', the second pivot member 505' being coupled to the first pivot member 504 'by the rotational joint 506' to be rotatable in a second attitude degree of freedom.
The curved track illustrated in fig. 17 or 19 described above illustratively includes a curved track having a circular arc shape. For a structure in which rotation is achieved by a curved slide, for example, in fig. 17, for the base 503 and the first pivoting member 504, the first driving mechanism may employ, for example, a planar four-bar mechanism, a crank block mechanism, or the like, and the first pivoting member 504 is driven to rotate on the curved slide 5031 with respect to the base 503. For the structure in which the rotation is achieved by the rotary joint 506', for example, in fig. 19, the implementation of the second driving mechanism is simpler for the second pivoting member 505' and the first pivoting member 504', for example, the second driving mechanism may employ, for example, a gear engagement mechanism, a pulley mechanism, or the like, to rotate the second pivoting member 505' relative to the first pivoting member by driving the rotary joint 506' to rotate.
In some embodiments, with continued reference to fig. 18, the second pivot member 505 includes a base plate 5051 and side walls 5052 extending away from the base plate 503 from both sides of the base plate 5051. The viewing assembly 501 is disposed on the base plate 5051. When viewing an image, the user's head may be accommodated in a semi-enclosed space formed by the side walls 5052 and the base plate 5051 and may allow for flexible up-down, left-right rotation, and front-back movement. Sensors and/or detectors that monitor movement of the user's head may be provided on the posture adjustment assembly 502 and/or the viewing assembly 501. For example, a set of sensors 507 that monitor the rotation of the user's head in an up-down direction may be provided on the viewing assembly 501, for example, on the base plate 5051 on both the upper and lower sides of the viewing window 5011; a set of sensors that monitor the rotation of the user's head in the left-right direction may be provided on the second pivot member 505, for example on the oppositely disposed side walls 5052 of the second pivot member 505; a set of sensors that monitor the movement of the user's head in the depth direction may be provided on the viewing assembly 501, for example integrally provided inside the cushion 5012. In some embodiments, the input device employs a detector that monitors movement of the user's head, the detector may also be positioned at the same or different locations as the sets of sensors, e.g., 507, 508, described above.
In some embodiments, the first pose degree of freedom of the first pivot 504 (504 ') includes one of a yaw degree of freedom and a pitch degree of freedom associated with the imaging modality, and the second pose degree of freedom of the second pivot 505 (505') includes the other of the yaw degree of freedom and the pitch degree of freedom associated with the imaging modality. For example, the first pose degree of freedom includes a yaw degree of freedom associated with the imaging instrument and the second pose degree of freedom includes a pitch degree of freedom associated with the imaging instrument.
Further, a yaw degree of freedom of the imaging instrument is associated with one of a left-right rotational motion and an up-down rotational motion of the user's head, and a pitch degree of freedom of the imaging instrument is associated with the other of the left-right rotational motion and the up-down rotational motion of the user's head. For example, the yaw degree of freedom of the imaging instrument is associated with the left and right rotational movements of the user's head, and the pitch degree of freedom of the imaging instrument is associated with the up and down rotational movements of the user's head, such association being intuitive in operation and capable of achieving a better user experience.
In some embodiments, the axis of rotation of each pivot member is substantially the same as the axis of rotation of the associated user's head, e.g., with continued reference to FIG. 16, the first pivot member has a first axis of rotation J20 that is substantially the same as the axis of rotation of the user's head (i.e., neck) in yaw degrees of freedom, and the second pivot member has a second axis of rotation J30 that is substantially the same as the axis of rotation of the user's head (i.e., neck) in pitch degrees of freedom. The design can ensure that the motion trail of the observation component is basically the same as the natural motion trail of the head of the user, and can ensure the comfort of the user during use.
In some embodiments, the range of motion of the doctor's head in the left-right rotational direction is typically around ±75°, and thus the range of motion of the pivoting member providing the attitude degrees of freedom corresponding to the left-right rotational motion may be configured to be between-75 ° and +75°, i.e., may be rotated by 75 ° to the left and right, respectively, for the center position corresponding to the attitude degrees of freedom. Preferably, the range of motion for which the doctor's head is comfortable is typically around ±60°, and thus the range of motion of the pivoting member providing the degree of freedom of posture corresponding to this left-right rotational motion can be configured to be between-60 ° and +60°, for example between-45 ° and +45°. For example, the pivoting member is the first pivoting member shown in fig. 17 or fig. 19.
In some embodiments, the range of motion of the doctor's head in the up-and-down rotational direction is typically around ±45°, and thus the range of motion of the pivoting member providing the attitude degree of freedom corresponding to the up-and-down rotational motion may be configured to be between-45 ° and +45°, i.e., may be rotated up and down by 45 ° respectively for the center position corresponding to the attitude degree of freedom. Preferably, the range of motion for which the doctor's head is comfortable is typically around ±30°, and thus the range of motion of the pivoting member providing the degree of freedom of posture corresponding to the left-right rotational motion can be configured to be between-30 ° and +30°, for example between-25 ° and +25°. For example, the pivoting member is a second pivoting member shown in fig. 17 or 19.
In some embodiments, a more specific procedure for a physician to control a medical device, such as an imaging device, includes:
step 401, adjusting the pose of the image host.
Including user adjustments to the height, depth, tilt, etc. of the image mainframe to obtain a more comfortable surgical pose for the user to observe. Different users may have different operation habits, and the pose of the observation component can be automatically adjusted according to the operation habits corresponding to the acquired user identity. At the discretion of the user, with or without step 401 being performed at the discretion of the user.
Step 402, the initial position and/or pose of the image host is acquired and recorded.
The initial position and/or posture is the position and/or posture of the image host machine when the master-slave mapping relation is established between the input device and the manipulator assembly, and the initial position and/or posture can be simply referred to as an initial position and/or posture, and can also be referred to as a position and posture zero position. The initial position and/or attitude includes the position and/or attitude of the first pivoting member relative to the base as previously described, and the position and/or attitude of the second pivoting member relative to the first pivoting member. Illustratively, the pose zero position is recorded once when the user first enters the procedure.
Step 403, monitoring whether the operation starting condition is satisfied in real time.
The operation starting condition includes, but is not limited to, one or more of detecting that the head of the user is close to the image host, acquiring a master-slave activation instruction, acquiring an instruction that the input device and the imaging apparatus establish a mapping relation, and the like. Illustratively, the proximity of the user's head to the image host may be confirmed by force, deformation, distance, etc. by the sensors as mentioned above. The obtaining of the mapping relation between the input device and the imaging apparatus can be obtained by triggering pressing a specific key, outputting a specific voice, stepping on a specific pedal, and the like, wherein the mapping relation between the input device and the imaging apparatus is established at the same time.
Step 404, determining movement of the user's head when the operation start condition is satisfied.
Wherein movement of the user's head, including a change in the user's head pose, such as position and/or pose, comprises sensing movement of the user's head using an input device as in any of the embodiments described above. For example, a change in the position and/or posture of the user's head may be determined by determining a change in the position and/or posture of the beacon-configured wearable device.
Step 405 determines if the movement of the user's head is an intentional movement.
Among them, the movement of the user's head includes an intended movement or an unintended movement, and the unintended movement is generally considered to be a malfunction, and needs to be excluded when controlling the medical instrument. For example, an initial position and/or pose of the user's head may be established in response to the imaging host and/or imaging instrument, and after the procedure begins, the user's head current position and/or pose may be compared to the initial position and/or pose of the user's head to determine if the movement of the user's head is intended, e.g., if the amount of change between the current position and/or pose and the initial position and/or pose exceeds a set threshold.
In step 406, upon determining that the movement of the user's head is intended movement, adjustments are made to the image host and imaging modality in response to the movement of the user's head.
Thus, the matched movement of the head of the user, the image host and the imaging instrument is realized. Particularly when the imaging apparatus is adjusted, if the imaging apparatus is rotated around different rotation centers in association with the movement of the user's head, in order to ensure consistency of imaging effects, the solution of the foregoing embodiment may be adopted to compensate for the deviation of the imaging distance, which is not repeated here.
Accordingly, in step 406, the image displayed in the viewing assembly will also change, where the change in image will also be fed back to the user, thereby supporting the user in deciding whether further adjustment is needed, if so, the user's head will move further in the desired direction, otherwise, the user's head will not need to move any more.
In some embodiments, the process further includes step 407, where the control device monitors in real time whether the surgical interrupt condition is met after the surgical procedure is initiated. Wherein the surgical interrupt condition includes, but is not limited to, for example, one or more of detecting that the user's head is off the image host, that the user's hand is off the operator, that an instruction to disconnect the input device from the imaging instrument is obtained, and the like.
The process also includes a step 408 of disconnecting the mapping relationship between the input device and the imaging instrument, the image master, and locking the position and/or posture of the imaging instrument and the image master when the surgical interrupt condition is satisfied.
In some embodiments, the process further includes step 409 of monitoring whether the image host clutch command is acquired in real time. The image host clutch command may be input by means such as outputting a specific voice, pressing a specific key, stepping on a specific pedal, etc. The process further includes step 410, when the image host clutch command is acquired, the image host is controlled to return to the initial position and/or posture, i.e. return to the zero position, according to the initial position and/or posture of the image host recorded in advance. In the process of controlling the image host to return to the zero position, the imaging instrument is in a locking state, and the visual field does not change correspondingly. The steps 409 and 410 can meet the requirement of large-scale adjustment of the imaging apparatus, and facilitate the operation of the user. In other embodiments, the control device may interrupt the association between the movement of the user's head and the movement of the imaging instrument, and may also lock the position and/or pose of the imaging host if the adjustment of the imaging instrument reaches a limit.
In some embodiments, if the user desires to restart the procedure, steps 404-406 described above may be repeated.
In some embodiments, the viewing assembly may be movable to follow the movement of the user's head, and may be implemented without additional drive mechanisms. For example, a head-mounted display device may be provided that can naturally move following the movement of the user's head as long as it is worn on the user's head.
Further, in order to achieve the control of a medical instrument, such as an imaging instrument, as described in any of the embodiments above, input means for inputting control instructions of the medical instrument, such as instructions for the imaging instrument to rotate a target rotation vector about a distal point, such as the aforementioned instructions for the imaging instrument, may be integrated in the head-mounted display device. In some embodiments, the input device includes one or more sensors for sensing movement of the user's head. For example, these sensors include inertial sensors, such as in particular one or more of accelerometers, angular rate gyroscopes, and Inertial Measurement Units (IMUs), etc., that monitor movement of the user's head by monitoring pose and motion of the head mounted display device. The sensors can be uniformly arranged at all positions of the head-mounted display device, mutual correction among a plurality of sensors is facilitated, and measurement accuracy is improved.
In some embodiments, the head-mounted display device includes a display module and an adjustment mechanism, wherein the display module includes two display units for left and right eyes to watch, and the adjustment mechanism is used for adjusting a horizontal distance between the two display units so as to adapt to pupil distances of different users.
In some embodiments, the head mounted display device may further include one or more sensors for sensing whether the head of the user is wearing the head mounted display device, which may be, for example, a proximity sensor, a deformation sensor, a pressure sensor, or the like. In some embodiments, the head mounted display device may further comprise a communication unit coupled to the surgical system by wired and/or wireless means, for example coupled to the control means.
In some embodiments, the display unit includes a display screen and a lens group, where the distance between the lens group and the display screen is adjustable, so as to adapt to the diopters of both eyes of different users, so that the user with refractive errors of both eyes can still normally use the head-mounted display without wearing glasses or other devices. The display screen may be configured to adjust the image according to the state of the lens group such that the image can form a normal image in the human eye after correction by the lens group.
In some embodiments, the display unit may further include one or more cameras, which may be a visible light camera or an infrared light camera, mainly used for an eye tracking function of the user to determine an area where the user looks at, or simply determine whether the user looks at a screen, so as to at least determine whether the user wears the head-mounted display device. The surgical operation is allowed to begin after the user is determined to wear the head mounted display device and when other surgical initiation conditions are met, and is discontinued, e.g., the head mounted display device is disconnected from the imaging modality, when the surgical initiation conditions are not met.
In some embodiments, the head mounted display device is adapted to control a pitch degree of freedom, a yaw degree of freedom, and a feed degree of freedom of the imaging apparatus, and may also be adapted to control a roll degree of freedom of the imaging apparatus. Wherein rotation of the user's head about the longitudinal axis of the neck may correspond to a yaw degree of freedom of the imaging device, rotation of the user's head about the transverse axis of the neck may correspond to a pitch degree of freedom of the imaging device, and rotation of the user's head away from the longitudinal axis of the neck may correspond to a roll degree of freedom of the imaging device.
In an embodiment, the present application further provides a computer readable storage medium storing a computer program configured to be loaded and executed by a processor to implement a control method according to any one of the embodiments above.
In an embodiment, the present application further provides a control device of the surgical robot. As shown in fig. 21, the control device may include: a processor (processor) 501, a communication interface (Communications Interface) 502, a memory (memory) 503, and a communication bus 504.
The processor 501, the communication interface 502, and the memory 503 perform communication with each other via the communication bus 504.
A communication interface 502 for communicating with other devices such as various types of sensors or motors or solenoid valves or other network elements of clients or servers, etc.
The processor 501 is configured to execute the program 505, and may specifically perform relevant steps in the above-described method embodiments.
In particular, program 505 may comprise program code comprising computer operating instructions.
The processor 505 may be a central processing unit CPU, or a specific integrated circuit ASIC (ApplicationSpecific Integrated Circuit), or one or more integrated circuits configured to implement embodiments of the present application, or a graphics processor GPU (Graphics Processing Unit). The one or more processors included in the control device may be the same type of processor, such as one or more CPUs, or one or more GPUs; but may also be different types of processors such as one or more CPUs and one or more GPUs.
A memory 503 for storing a program 505. The memory 503 may comprise high-speed RAM memory or may further comprise non-volatile memory (non-volatile memory), such as at least one disk memory.
The program 505 is particularly useful for causing the processor 501 to execute the control method described in any one of the embodiments described above.
The technical features of the above-described embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above-described embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.
The above examples merely represent a few embodiments of the present application, which are described in more detail and are not to be construed as limiting the scope of the invention. It should be noted that it would be apparent to those skilled in the art that various modifications and improvements could be made without departing from the spirit of the present application, which would be within the scope of the present application. Accordingly, the scope of protection of the present application is to be determined by the claims appended hereto.
Claims (14)
1. A surgical robot, comprising:
An imaging instrument for insertion into a body opening to acquire an image;
a manipulator for manipulating the imaging instrument for rotation about a telecentric stop positioned at the body opening and for feeding in an axial direction of the imaging instrument;
input means for inputting a first target rotation vector of the imaging instrument about its distal point in a target pose degree of freedom rotation;
a control device coupled with the manipulator and the input device, configured to:
acquiring the first target rotation vector input by the input device;
determining a target position and/or posture of a target imaging center of the imaging instrument in a reference coordinate system based on the first target rotation vector;
determining a target joint variable for a joint in the manipulator and the imaging instrument based on the target position and/or pose;
and controlling articulation in the manipulator and the imaging instrument according to the target articulation variable so as to enable the imaging center of the imaging instrument to reach the target imaging center.
2. The surgical robot of claim 1, wherein the determining a target joint variable for a joint in the manipulator and the imaging instrument based on the target position and/or pose comprises:
Acquiring configuration parameters of an imaging optical axis of an imaging instrument;
constructing a kinematic model associated with the manipulator, the imaging instrument and the imaging optical axis in combination with configuration parameters of the imaging optical axis;
the target joint variables of the joints in the manipulator and the imaging instrument are determined based on the target position and/or pose, the kinematic model.
3. A surgical robot as claimed in claim 2, wherein the configuration parameters include a length of the imaging optical axis and/or an angle of the imaging optical axis relative to an imaging plane of the imaging instrument.
4. A surgical robot as claimed in claim 3, wherein the control means is configured to:
acquiring an imaging distance range of the imaging instrument;
generating a configuration interface comprising one or more selectable target imaging distances based on the imaging distance range, the target imaging distance being between a minimum imaging distance and a maximum imaging distance of the imaging distance range;
in response to selection of the target imaging distance through the configuration interface, the selected target imaging distance is configured to be the length of the imaging optical axis.
5. A surgical robot as claimed in claim 3, wherein the control means is configured to:
generating a configuration interface comprising one or more selectable target angles, the target angles being between 0 ° and 90 °;
in response to selection of the target angle through the configuration interface, the selected target angle is configured as an angle of the imaging optical axis relative to the imaging plane.
6. The surgical robot of claim 1, wherein the determining a target position and/or pose of a target imaging center of the imaging instrument in a reference coordinate system based on the first target rotation vector comprises:
determining a first position and/or pose of the target imaging center relative to a distal point of the imaging instrument;
and determining the target position and/or posture of the first position and/or posture in the reference coordinate system based on the current position and/or posture of the remote point in the reference coordinate system.
7. The surgical robot of claim 1, wherein movement of a user's head in the target pose degrees of freedom is associated with the acquired first target rotation vector, the surgical robot further comprising:
The image host comprises an observation assembly and an attitude adjustment assembly, wherein the attitude adjustment assembly is used for adjusting the attitude of the observation assembly, and the observation assembly is used for observing the image acquired by the imaging instrument;
the control device is coupled with the attitude adjustment assembly and is further configured to:
and controlling the gesture adjusting component to move according to the first target rotation vector, so that the observing component moves in the target gesture degree of freedom along with the movement of the head of the user.
8. The surgical robot of claim 7, wherein the viewing assembly is configured to move in synchronization with movement of the imaging instrument.
9. The surgical robot of claim 7, wherein the pose adjustment assembly comprises:
a base;
a first pivot pivotally connected to the base and configured to be rotatable relative to the base in a first attitude degree of freedom;
and a second pivot member pivotally connected to the first pivot member and configured to pivot in a second attitude degree of freedom with respect to the first pivot member;
the observation assembly is fixedly connected with the second pivoting piece relatively.
10. The surgical robot of claim 9, wherein the base comprises a first curved slide, the first pivot comprises a second curved slide and a third curved slide, the second pivot comprises a fourth curved slide, the first curved slide and the second curved slide are slidably engaged to be rotatable in a first pose degree of freedom, the third curved slide and the fourth curved slide are slidably engaged to be rotatable in a second pose degree of freedom; or,
The base comprises a first bending slide rail, the first pivoting piece comprises a second bending slide rail, the first bending slide rail and the second bending slide rail are in sliding fit to rotate in a first attitude degree of freedom, the attitude adjusting joint further comprises a rotary joint, and the second pivoting piece and the first pivoting piece are connected through the rotary joint to rotate in a second attitude degree of freedom.
11. The surgical robot of claim 1, wherein the input device comprises one or more base stations capable of receiving and/or transmitting wireless signals, the input device further comprising a wearable device wearable on a user's head, the wearable device comprising one or more beacons capable of transmitting and/or receiving wireless signals, the first target rotation vector associated with the user's head movement being determinable based on sensing a distance between the one or more beacons and the one or more base stations.
12. A method of controlling a surgical robot, the surgical robot comprising:
an imaging instrument for insertion into a body opening to acquire an image;
a manipulator for manipulating the imaging instrument for rotation about a telecentric stop positioned at the body opening and for feeding in an axial direction of the imaging instrument;
Input means for inputting a first target rotation vector of the imaging instrument about a distal point in a target pose degree of freedom rotation;
the control method comprises the following steps:
acquiring the first target rotation vector input by the input device;
determining a target position and/or posture of a target imaging center of the imaging instrument in a reference coordinate system based on the first target rotation vector;
determining a target joint variable for a joint in the manipulator and the imaging instrument based on the target position and/or pose;
and controlling articulation in the manipulator and the imaging instrument according to the target articulation variable so as to enable the imaging center of the imaging instrument to reach the target imaging center.
13. A computer readable storage medium, characterized in that the computer readable storage medium stores a computer program configured to be loaded by a processor and to execute steps implementing the control method according to claim 12.
14. A control device for a surgical robot, the control device comprising:
a memory for storing a computer program;
and a processor for loading and executing the computer program;
Wherein the computer program is configured to be loaded by the processor and to perform the steps of implementing the control method of claim 12.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210899306.XA CN117503364A (en) | 2022-07-28 | 2022-07-28 | Surgical robot, control method and control device thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210899306.XA CN117503364A (en) | 2022-07-28 | 2022-07-28 | Surgical robot, control method and control device thereof |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117503364A true CN117503364A (en) | 2024-02-06 |
Family
ID=89740574
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210899306.XA Pending CN117503364A (en) | 2022-07-28 | 2022-07-28 | Surgical robot, control method and control device thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117503364A (en) |
-
2022
- 2022-07-28 CN CN202210899306.XA patent/CN117503364A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11792386B2 (en) | Medical devices, systems, and methods using eye gaze tracking for stereo viewer | |
| CN108472097B (en) | User console system for robotic surgery | |
| KR102414405B1 (en) | Computer-assisted teleoperated surgical systems and methods | |
| JP6576002B2 (en) | Synthetic representation of surgical robot | |
| US11333899B2 (en) | Systems and methods for three-dimensional visualization during robotic surgery | |
| WO2018159338A1 (en) | Medical support arm system and control device | |
| CN106456148B (en) | Medical devices, systems, and methods using eye gaze tracking | |
| KR102482803B1 (en) | Secondary mechanism control in computer-assisted remote control system | |
| CN109171977A (en) | The skilful type surgery systems of oversoul | |
| JP2017104455A (en) | Remote handling equipment and remote surgery system | |
| EP3592276B1 (en) | Systems for controlling tool with articulatable distal portion | |
| JP7445377B2 (en) | Virtual console for controlling surgical robots | |
| JP2020532385A (en) | Robotic surgery system that realigns rolls, pitches, and yaws with algorithms including trim and reversal algorithms | |
| JP2020044354A (en) | Remote operation device and remote operation system | |
| CN114795495A (en) | Master-slave operation minimally invasive surgery robot system | |
| CN116196111A (en) | Ophthalmic surgery robot system and control method thereof | |
| Zinchenko et al. | Virtual reality control of a robotic camera holder for minimally invasive surgery | |
| CN117503364A (en) | Surgical robot, control method and control device thereof | |
| CN117503365A (en) | Surgical robot, control method and control device thereof | |
| CN218739128U (en) | Main console and image host thereof | |
| JP2022545684A (en) | Movable display unit on track | |
| JP2018075395A (en) | Remote handling equipment and remote surgery system | |
| CN116869669B (en) | Surgical robot system | |
| CN116869668B (en) | Surgical robot system | |
| US20240033023A1 (en) | Adjustable user console for a surgical robotic system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |