CN114631886A - Mechanical arm positioning method, readable storage medium and surgical robot system - Google Patents
Mechanical arm positioning method, readable storage medium and surgical robot system Download PDFInfo
- Publication number
- CN114631886A CN114631886A CN202011492045.7A CN202011492045A CN114631886A CN 114631886 A CN114631886 A CN 114631886A CN 202011492045 A CN202011492045 A CN 202011492045A CN 114631886 A CN114631886 A CN 114631886A
- Authority
- CN
- China
- Prior art keywords
- arm
- positioning
- adjustment
- surgical
- coordinate system
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/70—Manipulators specially adapted for use in surgery
- A61B34/74—Manipulators with manual electric input means
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/10—Computer-aided planning, simulation or modelling of surgical operations
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/25—User interfaces for surgical systems
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/30—Surgical robots
- A61B34/37—Master-slave robots
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/10—Computer-aided planning, simulation or modelling of surgical operations
- A61B2034/107—Visualisation of planned trajectories or target regions
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
- A61B2034/2046—Tracking techniques
Abstract
The invention provides a mechanical arm positioning method, a readable storage medium and a surgical robot system, wherein the mechanical arm positioning method comprises the following steps: acquiring a surgical object, a pre-punched position on the surgical object and a virtual three-dimensional model of a mechanical arm of a surgical console at a patient end; obtaining the swing configuration of the adjusting arm of the mechanical arm according to the virtual three-dimensional model; performing virtual-real fusion coordinate registration on a patient-end operation table coordinate system and a real body surface coordinate system of the operation object according to the current positioning configuration of the adjusting arm; obtaining a positioning path plan of the adjusting arm according to the real-time position of the AR equipment and the positioning configuration of the adjusting arm; and prompting the positioning path planning through AR equipment. By the aid of the arrangement, an operator can be conveniently guided to carry out the positioning operation of the adjusting arm, and the accuracy and the execution efficiency of preoperative positioning are improved.
Description
Technical Field
The invention relates to the field of robot-assisted surgery systems and methods, in particular to a mechanical arm positioning method, a readable storage medium and a surgical robot system.
Background
The appearance of surgical robots is in line with the development trend of precision surgery. The surgical robot becomes a powerful tool for helping doctors to complete the operation, and the design concept of the surgical robot is to adopt a minimally invasive mode and accurately implement complex surgical operations. Under the condition that the traditional operation faces various limitations, a surgical robot is developed to replace the traditional operation, the surgical robot breaks through the limitation of human eyes, and the internal organs are more clearly displayed to an operator by adopting a three-dimensional imaging technology. In the original area that the hand can not stretch into, the robot hand can accomplish 360 degrees rotations, move, swing, centre gripping to avoid the shake. The surgical robot has little harm to the patient, little bleeding of the patient and quick recovery, greatly shortens the hospitalization time of the patient after the operation, can obviously improve the survival rate and the recovery rate after the operation, is popular among doctors and patients, and is widely applied to various clinical operations as a high-end medical instrument at present.
The existing surgical robot always carries out preoperative positioning on the mechanical arm by means of hand pulling or control end adjustment and the like, whether the positioning is at an expected optimal position cannot be determined, and the preoperative positioning operation is complex and the efficiency is low.
Disclosure of Invention
The invention aims to provide a mechanical arm positioning method, a readable storage medium and a surgical robot system, so as to solve the problem that the preoperative positioning of the existing surgical robot is inaccurate.
In order to solve the above technical problem, according to a first aspect of the present invention, there is provided a robot arm positioning method, including:
acquiring a surgical object, a pre-punched position on the surgical object and a virtual three-dimensional model of a mechanical arm of a surgical console at a patient end;
obtaining the swing configuration of the adjusting arm of the mechanical arm according to the virtual three-dimensional model;
performing virtual-real fusion coordinate registration on a patient-end operation table coordinate system and a real body surface coordinate system of the operation object according to the current positioning configuration of the adjusting arm;
obtaining a positioning path plan of the adjusting arm according to the real-time position of the AR equipment and the positioning configuration of the adjusting arm; and
and prompting the positioning path planning through AR equipment.
Optionally, the method for positioning the mechanical arm further includes:
acquiring the actual position of the adjusting arm after the positioning is finished according to the positioning path planning;
acquiring a planned position of the adjusting arm calculated according to the positioning path plan;
calculating the deviation between the planned position and the actual position, and if the deviation exceeds a preset standard, executing the following steps again: performing virtual-real fusion coordinate registration on a coordinate system of a patient-end operation console and a real body surface coordinate system of the operation object according to the current swing configuration of the adjusting arm; obtaining a positioning path plan of the adjusting arm according to the real-time position of the AR equipment; prompting the planning of the positioning path through AR equipment; and if the deviation is within the preset standard, determining that the positioning of the adjusting arm is finished.
Optionally, in the method for positioning the mechanical arm, the surgical object, the pre-punching position, and the virtual three-dimensional model of the mechanical arm are established based on parallax information fed back by a binocular vision device.
Optionally, in the mechanical arm positioning method, according to the current positioning configuration of the adjustment arm, the method for performing virtual-real fusion coordinate registration on the coordinate system of the patient-end surgical console and the real body surface coordinate system of the surgical object includes:
establishing coordinate mapping relations with a coordinate system of the patient-side operation console and a coordinate system of the AR equipment in a world coordinate system through a binocular vision device coordinate system;
and establishing a coordinate mapping relation between the patient-side operation console coordinate system and the real body surface coordinate system of the operation object according to the mapping relation between the real body surface coordinate system of the operation object and the binocular vision device coordinate system.
Optionally, in the method for positioning the mechanical arm, the coordinate relationship between the coordinate system of the binocular vision device and the coordinate system of the AR device is fixed.
Optionally, in the method for positioning the mechanical arm, the method for prompting the positioning path planning through the AR device includes:
the AR equipment displays the planned positions of the adjusting arms and displays the swing paths of the adjusting arms for selection to reach the planned positions;
prompting the selected joint of the adjusting arm to be adjusted; and
the adjustment amount of the joint to be adjusted is presented.
Optionally, in the method for positioning the mechanical arm, the method for prompting the positioning path planning through the AR device further includes:
acquiring an adjustment difference value fed back by a joint to be adjusted in the adjustment process;
if the adjustment difference exceeds the range of the preset target value, executing the following steps again: and prompting the adjustment amount of the joint to be adjusted until the adjustment difference value is within the range of a preset target value.
Optionally, in the method for positioning the mechanical arm, the method for prompting the positioning path planning through the AR device further includes:
and after the positioning of all the adjusting arms is detected to be completed, prompting that the positioning adjustment is completed, and displaying and/or recording the operation time.
Optionally, in the method for positioning a mechanical arm, in the step of displaying, by the AR device, planned positions of the plurality of adjustment arms and displaying positioning paths of the plurality of adjustment arms for selection, the closest adjustment arm is recommended as an adjustment arm for initial adjustment according to a relative position between a real-time position of the AR device and the plurality of adjustment arms.
Optionally, in the mechanical arm positioning method, before the joints of the selected adjustment arm that need to be adjusted are prompted, the joints with the highest collision probability are recommended as the joints to be adjusted at the beginning according to the positions of the joints and the adjacent joints of the selected adjustment arm.
Optionally, the current positioning configuration of the adjusting arm is obtained according to historical surgical data under the same surgical formula and the current virtual three-dimensional model.
In order to solve the above technical problem, according to a second aspect of the present invention, there is also provided a readable storage medium having a program stored thereon, the program, when executed, implementing the robot arm positioning method as described above.
In order to solve the above technical problem, according to a third aspect of the present invention, there is provided a surgical robot system, including a patient-side surgical console, an AR device, and a control device, wherein the patient-side surgical console includes a robot arm including an adjustment arm, and the control device controls the AR device to prompt a plan of a positioning path of the adjustment arm by using the robot arm positioning method according to any one of claims 1 to 10.
Optionally, the surgical robot system further includes a binocular vision device, the binocular vision device is configured to obtain a surgical object, the pre-punching position, and parallax information of the mechanical arm, and the control device is configured to establish a virtual three-dimensional model of the surgical object, the punching position, and the mechanical arm according to the parallax information.
Optionally, in the surgical robot, the binocular vision device and the AR device are integrally or separately arranged.
Optionally, in the surgical robot, the mechanical arm further includes a tool arm, one end of the tool arm is used to connect with a surgical instrument, the other end of the tool arm is used to connect with the adjustment arm, the adjustment arm includes a plurality of joints, and the control device controls the AR device to prompt the selected joint to be adjusted in the adjustment arm according to a swing path plan.
Optionally, in the surgical robot, the plurality of joints of the adjustment arm include at least three rotational joints and one mobile joint, and the control device controls the AR device to prompt the selected at least one of the rotational joint and the mobile joint of the adjustment arm to be adjusted according to the positioning path plan.
In summary, in the robot arm positioning method, the readable storage medium, and the surgical robot system provided by the present invention, the robot arm positioning method includes: acquiring an operation object, a pre-punching position on the operation object and a virtual three-dimensional model of a mechanical arm of a patient-end operation console; obtaining the positioning configuration of the adjusting arm of the mechanical arm according to the virtual three-dimensional model; performing virtual-real fusion coordinate registration on a coordinate system of a patient-end operation console and a real body surface coordinate system of the operation object according to the current swing configuration of the adjusting arm; obtaining a positioning path plan of the adjusting arm according to the real-time position of the AR equipment and the positioning configuration of the adjusting arm; and prompting the positioning path planning through AR equipment.
According to the configuration, based on virtual-real fusion coordinate registration, the obtained positioning path plan of the adjusting arm is prompted through the AR equipment according to the real-time position of the AR equipment, so that an operator can be conveniently guided to perform positioning operation of the adjusting arm in an auxiliary mode, and the accuracy and the execution efficiency of preoperative positioning are improved.
Drawings
It will be appreciated by those skilled in the art that the drawings are provided for a better understanding of the invention and do not constitute any limitation to the scope of the invention. Wherein:
FIG. 1 is a schematic view of an application scenario of a surgical robot in accordance with an embodiment of the present invention;
FIG. 2 is a schematic view of a patient-side surgical console of an embodiment of the present invention;
FIG. 3 is a schematic view of an adjusting arm joint of a robotic arm according to one embodiment of the present invention;
fig. 4 is a schematic view of the acquisition principle of a binocular vision apparatus according to an embodiment of the present invention;
fig. 5 is a schematic view of a binocular vision apparatus according to an embodiment of the present invention;
FIG. 6 is a flow chart of a robotic arm positioning method in accordance with an embodiment of the present invention;
FIG. 7 is a diagram illustrating an application scenario of path planning according to an embodiment of the present invention;
FIG. 8 is a schematic diagram illustrating the principle of virtual-real fused coordinate registration according to an embodiment of the present invention, wherein the binocular vision device is separated from the AR device;
fig. 9 is a schematic view of a binocular vision apparatus and AR equipment of an embodiment of the present invention separately provided;
FIG. 10 is a schematic diagram of the principle of virtual-real fused coordinate registration according to an embodiment of the present invention, in which a binocular vision device is integrated with an AR apparatus;
fig. 11 is a schematic view of a binocular vision apparatus according to an embodiment of the present invention integrally provided with an AR device;
fig. 12 is a schematic diagram of a coordinate mapping relationship between a binocular vision device and an AR apparatus according to an embodiment of the present invention;
FIG. 13 is a flowchart of prompting a relocation path plan by an AR device according to an embodiment of the present invention;
FIG. 14 is a diagram illustrating a path planning method according to an embodiment of the invention;
fig. 15 is a schematic view of a scenario of AR device prompting relocation path planning according to an embodiment of the present invention;
FIG. 16 is a schematic diagram of a real-time path planning method according to an embodiment of the invention;
fig. 17 is a schematic diagram of an optimal path selection method according to an embodiment of the present invention.
In the drawings:
1-a first revolute joint 1; 2-a second revolute joint; 3-a mobile joint; 4-a third revolute joint; 10-doctor end console; 20-a patient-end surgical console; 21-a mechanical arm; 21 a-an adjusting arm; 21 b-a tool arm; 30-an AR device; 40-the subject of surgery; 50-auxiliary equipment; 71-binocular vision device; 711-left camera; 712-right camera.
Detailed Description
To further clarify the objects, advantages and features of the present invention, a more particular description of the invention will be rendered by reference to specific embodiments thereof which are illustrated in the appended drawings. It is to be noted that the drawings are in greatly simplified form and are not to scale, but are merely intended to facilitate and clarify the explanation of the embodiments of the present invention. Further, the structures illustrated in the drawings are often part of actual structures. In particular, the drawings may have different emphasis points and may sometimes be scaled differently.
As used in this disclosure, the singular forms "a," "an," and "the" include plural referents, the term "or" is generally employed in a sense including "and/or," the terms "a," "an," and "the" are generally employed in a sense including "at least one," the terms "at least two" are generally employed in a sense including "two or more," and further, the terms "first," "second," and "third" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or to imply that the number of indicated technical features is essential. Thus, features defined as "first", "second", "third" may explicitly or implicitly include one or at least two of such features, the term "proximal" generally being the end near the operator, the term "distal" generally being the end near the patient, i.e. near the lesion, the terms "end" and "proximal" and "distal" generally referring to the corresponding two parts, which include not only the end points, the terms "mounted", "connected" and "connected" being to be understood in a broad sense, e.g. as being fixedly connected, as well as detachably connected, or as an integral part; either directly or indirectly through intervening media, either internally or in any other relationship. Furthermore, as used in the present invention, the disposition of an element with another element generally only means that there is a connection, coupling, fit or driving relationship between the two elements, and the connection, coupling, fit or driving relationship between the two elements may be direct or indirect through intermediate elements, and cannot be understood as indicating or implying any spatial positional relationship between the two elements, i.e., an element may be in any orientation inside, outside, above, below or to one side of another element, unless the content clearly indicates otherwise. The specific meanings of the above terms in the present invention can be understood by those skilled in the art according to specific situations.
The invention mainly aims to provide an AR (Augmented Reality) equipment auxiliary mechanical arm positioning method, a readable storage medium and a surgical robot, so as to solve the problem that the existing surgical robot is inaccurate in preoperative positioning.
The following description refers to the accompanying drawings.
Referring to fig. 1 to 17, fig. 1 is a schematic view illustrating an application scenario of a surgical robot according to an embodiment of the present invention; FIG. 2 is a schematic view of a patient-side surgical console of an embodiment of the present invention; FIG. 3 is a schematic view of an adjusting arm joint of a robotic arm according to one embodiment of the present invention; fig. 4 is a schematic view of the acquisition principle of a binocular vision apparatus according to an embodiment of the present invention; fig. 5 is a schematic view of a binocular vision apparatus according to an embodiment of the present invention; FIG. 6 is a flow chart of a robotic arm positioning method in accordance with an embodiment of the present invention; FIG. 7 is a diagram illustrating an application scenario of path planning according to an embodiment of the present invention; FIG. 8 is a schematic diagram of a principle of virtual-real fused coordinate registration in accordance with an embodiment of the present invention, wherein the binocular vision device is arranged separately from the AR apparatus; fig. 9 is a schematic view of a binocular vision apparatus and AR equipment of an embodiment of the present invention separately provided; FIG. 10 is a schematic diagram of the principle of virtual-real fused coordinate registration according to an embodiment of the present invention, in which a binocular vision device is integrated with an AR apparatus; fig. 11 is a schematic view of a binocular vision apparatus according to an embodiment of the present invention integrally provided with an AR device; fig. 12 is a schematic diagram of a coordinate mapping relationship between a binocular vision device and an AR apparatus according to an embodiment of the present invention; FIG. 13 is a flowchart of prompting a relocation path plan by an AR device according to an embodiment of the present invention; FIG. 14 is a diagram illustrating a path planning method according to an embodiment of the invention; fig. 15 is a schematic view of a scenario of AR device prompting relocation path planning according to an embodiment of the present invention; FIG. 16 is a schematic diagram of a real-time path planning method according to an embodiment of the invention; fig. 17 is a schematic diagram of an optimal path selection method according to an embodiment of the present invention.
Fig. 1 shows an application scenario of a surgical robot for performing a surgical operation in an exemplary embodiment. The surgical robot of the present invention is not particularly limited to the application environment. The surgical system comprises a doctor-end console 10, a patient-end surgical console 20, an AR device 30 (such as AR glasses), and a control device (not shown); the patient-side operation console 20 includes at least one mechanical arm, on which a surgical instrument or an endoscope is mounted, and an operator (e.g., a doctor) operates from the doctor-side operation console 10 to control the patient-side operation console 20 to drive the mechanical arm, so as to operate the surgical instrument to perform an operation. Preferably, the surgical robot further comprises a patient bed and other auxiliary equipment 50 (e.g., a display trolley, a sterile table, a ventilator or detection device, etc.). The control device, such as a processor, is communicatively connected to the physician-side console 10 and the patient-side surgical console 20, respectively, for controlling and interacting with various components of the overall system, and may be integrated or divided into multiple parts, and may be located at the same location or distributed at different locations.
Fig. 2 exemplarily shows a patient-end surgical console 20, which includes 4 robot arms 21, each including an adjustment arm 21a and a tool arm 21b, wherein the tool arm 21b is disposed at a distal end of the adjustment arm 21a, that is, one end (distal end) of the tool arm 21b is used for connecting with a surgical instrument (or an endoscope), and the other end (proximal end) is used for connecting with the adjustment arm 21 a; the adjusting arm 21a can adopt different positioning modes for different operation styles and patient signs, so that the tool arm 21b can accurately point to the focus of an operation object, an operator can complete the operation in an optimal operation space, and the interference probability generated in the operation process of the tool arm 21b is reduced. The adjustment arm 21a includes a plurality of joints, and in the example shown in fig. 2, the adjustment arm 21a includes four joints, three rotational joints (three rotational joints are a first rotational joint 1, a second rotational joint 2, and a third rotational joint 4), and one moving joint 3, as shown in fig. 3. Thus, there are four articulation configurations for each adjustment arm 21 a:
1. the first rotating joint 1 rotates by an angle theta 1 in the clockwise/anticlockwise direction from a zero point position (which is a default initial position) and is kept in a limiting state; alternatively, the rotation of the first rotary joint 1 may adjust the left-right rotation of the tool arm 21 b;
2. the second rotary joint 2 rotates by an angle theta 2 along the zero point position in the clockwise/anticlockwise direction and is kept in the limit position; alternatively, the rotation of the second revolute joint 2 can adjust the up-and-down rotation of the tool arm 21 b;
3. the movable joint 3 linearly translates from the zero position by the distance of s and is kept in the limit; alternatively, translation of the prismatic joint 3 may adjust the fore-aft translation of the tool arm 21 b;
4. the third rotating joint 4 rotates by an angle theta 3 along the clockwise/anticlockwise direction from the zero position and is kept in the limit position; alternatively, the rotation of the third revolute joint 4 can adjust the left-right rotation of the tool arm 21 b.
For easy understanding, please refer to a scene schematic diagram of the AR device prompting the planning of the positioning path shown in fig. 15, where fig. 15 shows the adjustment sequence and the adjustment amount of the joint of the adjustment arm 21a in an exemplary embodiment, where: (i) indicates that the joint adjusted in the first step is the first rotating joint 1 and the adjustment amount is θ 1, (ii) indicates that the joint adjusted in the second step is the third rotating joint 4 and the adjustment amount is θ 3, and (iii) indicates that the joint adjusted in the third step is the moving joint 3 and the adjustment amount is s; in this example, the second revolute joint 2 remains in the initial position and is not adjusted. It can be understood that the adjustment of the three rotational joints and the one moving joint on the adjustment arm 21a will bring about the change of the pose of the tool arm 21b, thereby realizing the adjustment of the pose of the tool arm 21b and enabling the tool arm 21b to move to the most suitable pose.
It should be noted that the patient-side surgical console 20 and the robotic arms 21 thereof shown in fig. 2 and 3 are only an example, and are not limited to the patient-side surgical console 20, and those skilled in the art may configure the number, structure, number of joints, joint forms, and the like of the robotic arms 21 differently according to the actual situation, and the invention is not limited thereto. Based on the above surgical robot, the present embodiment further provides a surgical robot system, which includes the surgical robot and the AR device 30, where the control device is in communication connection with the AR device 30, and is configured to control the AR device 30 to prompt the positioning path planning of the adjustment arm. The control device controls the AR device 30 to prompt the selected joint to be adjusted in the adjustment arm 21a according to the positioning path plan. For the example shown in fig. 2 and 3, the control means controls the AR device 30 to indicate at least one of the rotational joint and the translational joint to be adjusted of the selected adjustment arm 21a according to the swing path plan.
Further, the surgical robot further includes a binocular vision device 71, as shown in fig. 4, the binocular vision device 71 is configured to obtain a surgical object 40, a pre-punching position on the surgical object, and parallax information of the mechanical arm 21, and the control device is configured to establish a virtual three-dimensional model of the surgical object 40, the pre-punching position, and the mechanical arm 21 according to the parallax information.
As shown in fig. 4, the binocular vision device 71 acquires image information through a binocular camera, acquires two pieces of image information of the object to be measured from different angles based on the parallax principle to acquire two-dimensional information thereof, further establishes a correspondence relationship of the feature points, and calculates a position deviation, thereby realizing three-dimensional reconstruction of the surgical object 40, the pre-punching position, and the mechanical arm 21. Of course, in other embodiments, the virtual three-dimensional model may be obtained by scanning through tomography such as preoperative CT and MRI. Further, based on the virtual three-dimensional model, the configuration of the adjustment arm 21a can be determined by the reverse geometry of the surgical object and the pre-perforation position.
Please refer to fig. 5, which is a schematic diagram of a binocular vision device 71, the principle of binocular vision is as follows: in an exemplary embodiment, the binocular camera includes a left camera 711 and a right camera 712, a distance (similar to a pupil distance of binoculars of a human eye) between a center of the left camera 711 and a center of the right camera 712 is B, a reference plane c parallel to a line connecting the binoculars is provided at a distance f in front of the binocular camera, an intersection point of the line connecting the left camera 711 and the point P to be measured and the reference plane c is a1, an intersection point of a line connecting the right camera 712 and the point P to be measured and the reference plane c is a2, a left optical axis B1 of the left camera 711 extends perpendicular to the reference plane c through the left camera 711, a right optical axis B2 of the right camera 712 extends perpendicular to the reference plane c through the right camera 712, and a distance x between the a1 and the left optical axis B1 is xlThe distance between the point A2 and the right optical axis B2 is xrIf a rectangular coordinate system is established with a1 as the origin of coordinates, and the left optical axis B1 as the z-axis, the x-axis and the y-axis are both on the reference plane c, the x-axis is parallel to the line connecting the centers of the two eyes, and the y-axis is perpendicular to the line connecting the centers of the two eyes, then the coordinates (x, y, z) of the point P can be derived from the following formula:
based on the surgical robot system, please refer to fig. 6, this embodiment provides a method for positioning a mechanical arm, where the surgical robot system controls the AR device 30 to prompt planning of a positioning path of an adjustment arm by using the method for positioning the mechanical arm. The mechanical arm positioning method comprises the following steps:
step S1: acquiring a surgical object 40, a pre-drilling position on the surgical object and a virtual three-dimensional model of a mechanical arm 21 of a patient-end surgical console 20; preferably, the virtual three-dimensional model of the surgical object 40, the pre-punching position and the mechanical arm 21 is established based on parallax information fed back by the binocular vision device 71.
Step S2: obtaining the positioning configuration of the adjusting arm 21a of the mechanical arm 21 according to the virtual three-dimensional model; the skilled person can obtain the swing configuration of the mechanical arm 21, and further, the swing configuration of the adjusting arm 21a of the mechanical arm 21 according to the prior art. For example, the arrangement configuration of the adjustment arm 21a of the robot arm 21 can be obtained by combining the current virtual three-dimensional model with the historical surgical data in the same surgical formula. This step may be performed by the control means. Specifically, referring to fig. 7, the current position of the mechanical arm 21 and the modeling position in the virtual three-dimensional model are combined and calculated through the positioning of the binocular vision device 71, so as to obtain the positioning configuration.
Step S3: performing virtual-real fusion coordinate registration on a coordinate system of a patient-end operation table and a real body surface coordinate system of the operation object 40 according to the current swing configuration of the adjusting arm 21 a;
step S4: obtaining a positioning path plan of the adjusting arm 21a according to the real-time position of the AR device 30 and the positioning configuration of the adjusting arm 21 a; specifically, the positioning path planning can be adjusted in real time due to the different real-time positions of the AR devices 30 and the different positioning configurations of the adjusting arms 21 a. The skilled person can obtain the plan of the swing path of the adjusting arm 21a according to the real-time position of the AR device 30 and the swing configuration of the adjusting arm 21a, and the description is not repeated here.
Step S5: the placement path planning is prompted by the AR device 30.
With the configuration, based on the virtual-real fusion coordinate registration, the obtained positioning path plan of the adjusting arm 21a is prompted through the AR device 30 according to the real-time position of the AR device 30, so that the positioning operation of the adjusting arm 21a by an operator can be conveniently guided in an auxiliary manner, and the accuracy and the execution efficiency of preoperative positioning are improved.
As shown in fig. 8 and 9, in an alternative embodiment, the binocular vision device 71 is provided separately from the AR apparatus 30, and the step S3 specifically includes:
establishing coordinate mapping relations with the patient-end surgical console coordinate system (X2/Y2/Z2) and the AR equipment coordinate system (X3/Y3/Z3) in a world coordinate system (X0/Y0/Z0) through a binocular vision device coordinate system (X7/Y7/Z7); establishing a coordinate mapping relation between the patient-end surgical console coordinate system (X2/Y2/Z2) and the real body surface coordinate system (X4/Y4/Z4) of the surgical object 40 according to the mapping relation (if calibrated before the step) between the real body surface coordinate system (X4/Y4/Z4) of the surgical object 40 and the binocular vision device coordinate system (X7/Y7/Z7).
Further, referring to fig. 10 and 11, in some embodiments, the binocular vision device 71 is integrated with the AR apparatus 30, and the relative coordinate relationship between the coordinate system of the binocular vision device (X7/Y7/Z7) and the coordinate system of the AR apparatus (X3/Y3/Z3) is fixed. At this time, in step S3, a coordinate mapping relationship between the binocular vision apparatus coordinate system (X7/Y7/Z7) and the AR apparatus coordinate system (X3/Y3/Z3) may be established according to the design profiles of the binocular vision apparatus 71 and the AR apparatus 30. As shown in FIG. 12, the relative coordinate relationship of the AR device 30 to the binocular vision apparatus 71 is fixed, i.e., the binocular vision apparatus coordinate system (X7/Y7/Z7)
The mapping relationship can be established by the machine position and the AR equipment coordinate system (X3/Y3/Z3), and the binocular vision device coordinate system (X7/Y7/Z7) and the world coordinate system (Z0/X0/Y0) can be established by the rotation matrix R and the translation vector t. The basic principle is as follows:
where M1 is a transformation matrix. The establishment of the transformation matrix and the mapping relationship between coordinate systems can be understood by those skilled in the art according to the prior art, and will not be described herein.
Optionally, the method for positioning the mechanical arm further includes:
step S6: acquiring the actual position of the adjusting arm 21a after the positioning is finished according to the positioning path planning; this step S6 is preferably performed after step S5.
Step S7: acquiring a planned position of the mechanical arm 21 calculated according to the positioning path plan; this step S7 is preferably executed in step S4, step S5, or after step S5.
Step S8: calculating the deviation between the planned position and the actual position, and if the deviation exceeds a preset standard, executing the following steps again: step S3 to step S5; and if the deviation is within the preset standard, determining that the positioning of the mechanical arm 21 is completed.
Step S6 to step S8 are swing position detection steps, an operator may adjust the swing position of the adjustment arm 21a according to the plan of the swing position path prompted by the AR device 30 in step S5, detect whether the deviation between the planned position and the actual position exceeds a preset standard after the swing position is completed, if so, indicate that the swing position is not completed, return to step S3 to be repeatedly executed, and if the deviation does not exceed the preset standard, determine that the swing position of the adjustment arm 21a is completed. Preferably, the adjustment complete or incomplete information may be prompted, for example, by the AR device 30 to inform the operator. It will be appreciated that the preset criteria can be set by those skilled in the art to meet the actual needs.
Preferably, referring to fig. 13, step S5 includes:
step S51: the AR device 30 displays a plurality of planned positions of the adjustment arm 21a, and displays a plurality of positioning paths of the adjustment arm 21a for selecting to reach the planned positions; after the AR device 30 is turned on, the AR device 30 displays the planned positions of the plurality of adjustment arms 21a as virtual figures, and fuses the virtual figures to the real object image of the adjustment arms 21a to form a positioning path for the operator to select.
Step S52: prompting the selected joint of the adjusting arm 21a to be adjusted; after the operator selects the adjustment arm 21a to be adjusted in the pendulum position, the AR device 30 presents the joint to be adjusted by the selected adjustment arm 21 a.
Step S53: the adjustment amount of the joint to be adjusted is presented. The AR device 30 also prompts the amount of adjustment of the joint that needs to be adjusted, as displayed in the form of data, for reference by the operator. The adjustment amount of the joint here may include the position and posture of the joint adjustment, and may be different according to the movement form of the joint, for example, the rotation direction and angle of the joint rotation indicator, the direction and distance of the joint movement indicator displacement, and the like. Referring to fig. 14, in an exemplary embodiment, the spatial movement position of the joint of the adjustment arm 21a may be converted by the actual adjustment arm 21a and inverse kinematics modeling data, for example, for the first rotary joint 1, the second rotary joint 2 and the third rotary joint 4, the planned position is mapped to the same plane by adjusting the positions through angles, and the adjustment angles θ and φ based on the current position are calculated according to the mechanical structure data. For the mobile joint 3, the movement path can be planned by the displacement values S of the actual position and the modeled position.
Fig. 15 shows a schematic of a scene in which the AR device 30 displays a plurality of planned positions of the adjustment arm 21a, which roughly represents what the operator can observe on the AR device 30.
Further, after the step S53, the step S5 further includes:
step S54: acquiring an adjustment difference fed back by a joint to be adjusted in the adjustment process;
step S55: if the adjustment difference value exceeds a preset standard value, executing the following steps again: and prompting the adjustment amount of the joint to be adjusted until the adjustment difference value is within the range of a preset standard value.
Steps S54 and S55 are joint adjustment detection steps, in the process of adjusting the position of the adjustment arm 21a, the operator may generate an adjustment difference value during the adjustment process of each joint of the adjustment arm 21a, if the adjustment difference value exceeds the range of the preset target value, the position is not completed, the operation returns to step S53 to be repeatedly executed, and the adjustment amount of the joint is continuously presented to the operator for the operator to continue to adjust. And if the adjustment difference value does not exceed the range of the preset target value, determining that the adjustment of the joint is finished. It will be appreciated that the target values can be set by those skilled in the art to meet the actual needs.
Further, after the step S53, the step S5 further includes:
step S56: and after all the positioning of all the adjusting arms 21a are detected to be completed, prompting that the positioning adjustment is completed, and displaying and/or recording the operation time.
Alternatively, referring to fig. 16 and 17, in an exemplary embodiment, the binocular vision device 71 is mounted on the AR apparatus 30, and the AR apparatus 30 may be wearable AR glasses, so that the binocular vision device 71 may match the feature points of the mechanical arm 21 in real time according to the change of the visual angle of the operator, and further achieve real-time calibration through the transformation of the coordinate position of the AR apparatus 30 relative to the mechanical arm 21. In fig. 16, the position of the AR device 30 is shown to change from (X3/Y3/Z3) to (X31/Y31/Z31), where the coordinate system (X31/Y31/Z31) and the coordinate system (X3/Y3/Z3) are transformed according to the translation vector t.
Preferably, in step S51, the adjustment arm 21a closest to the real-time position of the AR device 30 is recommended as the adjustment arm to start the adjustment, based on the relative positions between the plurality of adjustment arms 21a and the real-time position. As shown in fig. 17, in the process of actually adjusting the swing position of the adjusting arm 21a, the adjusting arm 21a closest to the AR device 30 may be recommended as the adjusting arm for starting adjustment according to the real-time position of the AR device 30 and the relative position between each adjusting arm 21 a. The adjustment arm for initial adjustment recommended here means the adjustment arm 21a that first recommends the operator to make an adjustment, which can be prompted at the AR device 30.
Further, before presenting the joint to be adjusted of the selected adjustment arm 21a in step S52, the joint having the highest collision probability is recommended as the joint to be adjusted at the start, based on the positions of each joint and the adjacent joint of the selected adjustment arm 21 a. After the operator selects the adjustment arm 21a to be adjusted, the control device may calculate the collision probability of each joint of the adjustment arm 21a to be adjusted based on the positional relationship between each joint of the adjustment arm 21a and the joint of the adjacent other adjustment arm 21a, thereby determining the order of adjustment of the joints, preferentially adjusting the joints having the high collision probability, and so on to obtain the optimal path of the positioning of all the adjustment arms 21a of the entire patient-side surgical console 20.
The present embodiment also provides a readable storage medium, on which a program is stored, the program, when executed, implements the mechanical arm collision prompting method, and the readable storage medium may be integrally disposed on the surgical robot, such as in the control device, or may be attached separately.
In summary, in the robot positioning method, the readable storage medium and the surgical robot system provided by the present invention, the robot positioning method includes: acquiring a surgical object, a pre-punched position on the surgical object and a virtual three-dimensional model of a mechanical arm of a surgical console at a patient end; obtaining the positioning configuration of the adjusting arm of the mechanical arm according to the virtual three-dimensional model; performing virtual-real fusion coordinate registration on a coordinate system of a patient-end operation console and a real body surface coordinate system of the operation object according to the current swing configuration of the adjusting arm; obtaining a positioning path plan of the adjusting arm according to the real-time position of the AR equipment and the positioning configuration of the adjusting arm; and prompting the positioning path planning through AR equipment. According to the configuration, based on virtual-real fusion coordinate registration, the obtained positioning path plan of the adjusting arm is prompted through the AR equipment according to the real-time position of the AR equipment, so that an operator can be conveniently guided to perform positioning operation of the adjusting arm in an auxiliary mode, and the accuracy and the execution efficiency of preoperative positioning are improved.
It should be noted that, in the present specification, the embodiments are described in a progressive manner, each embodiment focuses on differences from other embodiments, similar parts between the embodiments may be referred to each other, and different parts between the embodiments may also be used in combination with each other, which is not limited by the present invention. The above description is only for the purpose of describing the preferred embodiments of the present invention, and is not intended to limit the scope of the present invention, and any variations and modifications made by those skilled in the art based on the above disclosure are within the scope of the present invention.
Claims (17)
1. A mechanical arm positioning method is characterized by comprising the following steps:
acquiring a surgical object, a pre-punched position on the surgical object and a virtual three-dimensional model of a mechanical arm of a surgical console at a patient end;
obtaining the positioning configuration of the adjusting arm of the mechanical arm according to the virtual three-dimensional model;
performing virtual-real fusion coordinate registration on a coordinate system of a patient-end operation console and a real body surface coordinate system of the operation object according to the current swing configuration of the adjusting arm;
obtaining a positioning path plan of the adjusting arm according to the real-time position of the AR equipment and the positioning configuration of the adjusting arm; and
and prompting the positioning path planning through AR equipment.
2. The robotic arm positioning method of claim 1, further comprising:
acquiring the actual position of the adjusting arm after the positioning is finished according to the positioning path planning;
acquiring a planned position of the adjusting arm calculated according to the positioning path plan;
calculating the deviation between the planned position and the actual position, and if the deviation exceeds a preset standard, executing the following steps again: performing virtual-real fusion coordinate registration on a patient-end operation table coordinate system and a real body surface coordinate system of the operation object according to the current positioning configuration of the adjusting arm; obtaining a positioning path plan of the adjusting arm according to the real-time position of the AR equipment; prompting the planning of the positioning path through AR equipment; and if the deviation is within the preset standard, determining that the positioning of the adjusting arm is finished.
3. The method of claim 1, wherein the surgical object, the pre-perforated location, and the virtual three-dimensional model of the robotic arm are created based on parallax information fed back by a binocular vision device.
4. The robotic arm positioning method of claim 3, wherein the virtual-real fused coordinate registration of the patient-end surgical console coordinate system and the real body surface coordinate system of the surgical object according to the current positioning configuration of the adjustment arm comprises:
establishing coordinate mapping relations with a coordinate system of the patient-side operation console and a coordinate system of the AR equipment in a world coordinate system through a binocular vision device coordinate system;
and establishing a coordinate mapping relation between the patient-side surgical console coordinate system and the real body surface coordinate system of the surgical object according to the mapping relation between the real body surface coordinate system of the surgical object and the binocular vision device coordinate system.
5. The robotic arm positioning method of claim 4, wherein the binocular vision device coordinate system is fixed in relation to the AR equipment coordinate system.
6. The robotic arm positioning method of claim 1, wherein the method of prompting the positioning path plan through the AR device comprises:
the AR equipment displays the planning positions of the plurality of adjusting arms and displays the swing paths of the plurality of adjusting arms for selecting to reach the planning positions;
prompting the selected joint of the adjusting arm to be adjusted; and
the adjustment amount of the joint to be adjusted is presented.
7. The robotic arm placement method of claim 6, wherein the method of prompting the placement path plan by the AR device further comprises:
acquiring an adjustment difference fed back by a joint to be adjusted in the adjustment process;
if the adjustment difference exceeds the range of the preset target value, executing the following steps again: and prompting the adjustment amount of the joint to be adjusted until the adjustment difference value is within the range of a preset target value.
8. The robotic arm positioning method of claim 6, wherein the method of prompting the positioning path plan via the AR device further comprises:
and after the positioning of all the adjusting arms is detected to be completed, prompting that the positioning adjustment is completed, and displaying and/or recording the operation time.
9. The method of claim 6, wherein in the step of displaying the planned positions of the plurality of adjustment arms and displaying the positioning paths of the plurality of adjustment arms for selection by the AR device, the closest adjustment arm is recommended as the adjustment arm for initial adjustment according to the relative positions between the real-time position of the AR device and the plurality of adjustment arms.
10. The method of positioning a robot arm according to claim 6, wherein a joint having a maximum collision probability is recommended as a joint to be initially adjusted according to a position of each joint and an adjacent joint of the selected adjustment arm before presenting a joint to be adjusted of the selected adjustment arm.
11. The method as claimed in claim 1, wherein the current positioning configuration of the adjustment arm is obtained according to the current virtual three-dimensional model and the historical surgical data under the same surgical formula.
12. A readable storage medium on which a program is stored, wherein the program, when executed, implements a robot arm positioning method according to any one of claims 1 to 11.
13. A surgical robot system, comprising a patient-side surgical console, an AR device and a control device, wherein the patient-side surgical console comprises a mechanical arm, the mechanical arm comprises an adjusting arm, and the control device controls the AR device to prompt the planning of the positioning path of the adjusting arm by using the mechanical arm positioning method according to any one of claims 1 to 11.
14. The surgical robotic system of claim 13, further comprising a binocular vision device for acquiring parallax information of a surgical object, the pre-perforation location, and the robotic arm, the control device for creating a virtual three-dimensional model of the surgical object, the perforation location, and the robotic arm based on the parallax information.
15. The surgical robotic system as claimed in claim 14, wherein the binocular vision device is provided integrally or separately with the AR apparatus.
16. The surgical robotic system of claim 13, wherein the robotic arm further comprises a tool arm having one end for coupling to a surgical instrument and another end for coupling to the adjustment arm, the adjustment arm comprising a plurality of joints, the control device controlling the AR device to indicate which of the selected adjustment arms that need to be adjusted according to a swing path plan.
17. The surgical robotic system of claim 16, wherein the plurality of joints of the adjustment arm include at least three rotational joints and a translational joint, and the control means controls the AR device to indicate at least one of the rotational joints and the translational joint of the selected adjustment arm to be adjusted based on the positioning path plan.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011492045.7A CN114631886A (en) | 2020-12-16 | 2020-12-16 | Mechanical arm positioning method, readable storage medium and surgical robot system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011492045.7A CN114631886A (en) | 2020-12-16 | 2020-12-16 | Mechanical arm positioning method, readable storage medium and surgical robot system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114631886A true CN114631886A (en) | 2022-06-17 |
Family
ID=81944983
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011492045.7A Pending CN114631886A (en) | 2020-12-16 | 2020-12-16 | Mechanical arm positioning method, readable storage medium and surgical robot system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114631886A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115359896A (en) * | 2022-10-20 | 2022-11-18 | 山东曲阜康尔健医疗科技有限公司 | Operation and monitoring analysis system based on data analysis and remote control |
| CN116531111A (en) * | 2023-07-04 | 2023-08-04 | 梅奥心磁(杭州)医疗科技有限公司 | Double-stage self-adaptive power operation mechanical arm device and control method |
-
2020
- 2020-12-16 CN CN202011492045.7A patent/CN114631886A/en active Pending
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115359896A (en) * | 2022-10-20 | 2022-11-18 | 山东曲阜康尔健医疗科技有限公司 | Operation and monitoring analysis system based on data analysis and remote control |
| CN115359896B (en) * | 2022-10-20 | 2023-01-24 | 山东曲阜康尔健医疗科技有限公司 | Operation and monitoring analysis system based on data analysis and remote control |
| CN116531111A (en) * | 2023-07-04 | 2023-08-04 | 梅奥心磁(杭州)医疗科技有限公司 | Double-stage self-adaptive power operation mechanical arm device and control method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP7086150B2 (en) | Systems and methods for rendering on-screen identification of instruments in remote-controlled medical systems | |
| JP7080945B2 (en) | Systems and methods for on-screen identification of instruments in remote-controlled medical systems | |
| EP3636194B1 (en) | Surgical robot system, and method for displaying position of surgical instrument | |
| JP6058111B2 (en) | Synthetic representation of surgical robot | |
| US11877816B2 (en) | Systems and methods for master/tool registration and control for intuitive motion | |
| JP5372225B2 (en) | Tool position and identification indicator displayed in the border area of the computer display screen | |
| JP2020062521A (en) | Graphical user interface for robot surgery system | |
| CN114631886A (en) | Mechanical arm positioning method, readable storage medium and surgical robot system | |
| EP3973540A1 (en) | Systems and methods for generating workspace volumes and identifying reachable workspaces of surgical instruments | |
| CN113081273B (en) | Punching auxiliary system and surgical robot system | |
| WO2023018684A1 (en) | Systems and methods for depth-based measurement in a three-dimensional view | |
| WO2023018685A1 (en) | Systems and methods for a differentiated interaction environment | |
| CN117813631A (en) | System and method for depth-based measurement in three-dimensional views | |
| Jung et al. | Immersive Human-Machine Interface Design for Teleoperated Surgical System |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |