CN117752376A - endoscope adjusting device and method and surgical robot system - Google Patents
endoscope adjusting device and method and surgical robot system Download PDFInfo
- Publication number
- CN117752376A CN117752376A CN202410108945.9A CN202410108945A CN117752376A CN 117752376 A CN117752376 A CN 117752376A CN 202410108945 A CN202410108945 A CN 202410108945A CN 117752376 A CN117752376 A CN 117752376A
- Authority
- CN
- China
- Prior art keywords
- target
- endoscope
- mechanical arm
- joint
- current
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims description 36
- 230000007246 mechanism Effects 0.000 description 8
- 239000003086 colorant Substances 0.000 description 7
- 238000004891 communication Methods 0.000 description 6
- 230000008569 process Effects 0.000 description 6
- 238000012790 confirmation Methods 0.000 description 5
- 238000005286 illumination Methods 0.000 description 5
- 238000001356 surgical procedure Methods 0.000 description 5
- 238000013519 translation Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 4
- 230000000007 visual effect Effects 0.000 description 4
- 238000013461 design Methods 0.000 description 3
- 230000009977 dual effect Effects 0.000 description 3
- 230000004075 alteration Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- 230000002708 enhancing effect Effects 0.000 description 1
- 238000003709 image segmentation Methods 0.000 description 1
- 238000002324 minimally invasive surgery Methods 0.000 description 1
- 238000011179 visual inspection Methods 0.000 description 1
Abstract
The invention provides an endoscope adjusting device and a surgical robot system, wherein the endoscope adjusting device comprises: the first pose acquisition module is used for acquiring the pose of a target tissue relative to the mechanical arm base based on the image acquired by the current endoscope, and the pose planning module is used for acquiring the target pose of the target mechanical arm; the tail end of the target mechanical arm is provided with a target endoscope, and when the target mechanical arm is in a target pose, a target tissue is positioned in the field of view of the target endoscope; the second pose acquisition module is used for acquiring the current position of the joint of the target mechanical arm, and the motion planning module is used for planning the target motion direction of at least one joint of the target mechanical arm based on the current position of the joint of the target mechanical arm and the target pose of the target mechanical arm. The endoscope adjusting device is applied to a surgical robot system, and when the surgical robot system is used for performing a minimally invasive interventional operation, switching of two endoscopes can be rapidly realized through operation of the endoscope adjusting device.
Description
Technical Field
The invention belongs to the technical field of medical instruments, and particularly relates to an endoscope adjusting device, an endoscope adjusting method and a surgical robot system.
Background
in the field of view in minimally invasive surgery, certain target tissues are difficult to identify or are at least partially obscured by other tissues. For this case, a dual endoscope is often used for surgery, typically one of the dual endoscopes is a white light endoscope and the other is a fluorescent endoscope, the use of which helps to identify the target tissue.
however, when the endoscope is switched during application of the dual endoscope, a change in the field of view occurs. In the related art, in order to cope with this, it is necessary for an operator to manually adjust the position of the mechanical arm on which the endoscope is mounted to adjust the field of view of the endoscope. However, the operator mainly relies on visual inspection to adjust repeatedly in the manual adjustment process, the accuracy is poor, the efficiency is low, and the possibility that the mechanical arm collides with other mechanical arms and the endoscope collides with other instruments in the manual adjustment process is easy to occur for the operator, which may cause some surgical risks.
Disclosure of Invention
the invention aims to provide an endoscope adjusting device, an endoscope adjusting method and a surgical robot system, which aim to quickly and accurately realize the view adjustment during the switching of endoscopes in the surgical process of applying double endoscopes, reduce the possibility of collision among devices and improve the surgical safety.
To achieve the above object, the present invention provides an endoscope adjusting apparatus comprising:
The first pose acquisition module is used for acquiring the position of a target tissue under a mechanical arm base coordinate system based on an image acquired by a current endoscope; the target tissue is positioned in the field of view of the current endoscope, and the current endoscope is hung at the tail end of the current mechanical arm;
The pose planning module is used for planning the target pose of the target mechanical arm on which the target endoscope is mounted based on the position of the target tissue under the mechanical arm base coordinate system; when the target mechanical arm is in the target pose, the target tissue is positioned in the field of view of the target endoscope, and the target mechanical arm is different from the current mechanical arm;
the second pose acquisition module is used for acquiring the current position of the joint of the target mechanical arm; the method comprises the steps of,
And the motion planning module is used for planning the target motion direction of at least one joint of the target mechanical arm based on the current position of the joint of the target mechanical arm and the target pose of the target mechanical arm.
optionally, the first pose acquisition module is configured to:
Acquiring the position of the target tissue under the coordinate system of an image acquisition element of the current endoscope based on the image acquired by the current endoscope;
Acquiring the position of an image acquisition element of the current endoscope under the coordinate system of the mechanical arm base;
and acquiring the position of the target tissue under the coordinate system of the mechanical arm base based on the position of the target tissue under the coordinate system of the image acquisition element of the current endoscope and the position of the image acquisition element of the current endoscope under the coordinate system of the mechanical arm base.
Optionally, the first pose acquisition module is further configured to:
and performing positive kinematic calibration on the current mechanical arm.
Optionally, the motion planning module is configured to:
acquiring a target position of an N joint according to a target pose of the target mechanical arm;
Calculating the difference value between the target position and the current position of the N joint;
Judging whether the N joint is at the corresponding target position according to the difference value between the target position of the N joint and the current position; if not, acquiring the target movement direction of the N joint according to the difference value between the target position and the current position of the N joint.
optionally, the endoscope adjusting device further comprises a display control module, wherein the display control module is used for:
And controlling a prompting device to display the movement direction of the target.
optionally, the endoscope adjusting device further comprises a motion control module for: and controlling the corresponding joints of the target mechanical arm to move according to the target movement direction.
Optionally, the endoscope adjusting device further comprises a prompt information generating module and a display control module;
The prompt information generation module is used for: generating first prompt information when the N joint is positioned at a corresponding target position;
the display control module is used for: controlling a prompt device to display the first prompt information; and/or the number of the groups of groups,
the prompt information generation module is used for: generating second prompt information when the target mechanical arm reaches the target pose;
The display control module is used for: and controlling a display device to display the second prompt information.
Optionally, the target motion direction is a rotation direction or a translation direction of the joint.
In order to achieve the above object, the present invention further provides an endoscope adjusting method, including:
Acquiring the position of a target tissue under a mechanical arm base coordinate system based on an image acquired by a current endoscope; the target tissue is positioned in the field of view of the current endoscope, and the current endoscope is hung at the tail end of the current mechanical arm;
planning a target pose of a target mechanical arm on which a target endoscope is mounted based on the position of the target tissue under the mechanical arm base coordinate system; when the target mechanical arm is in the target pose, the target tissue is positioned in the field of view of the target endoscope, and the target mechanical arm is different from the current mechanical arm;
acquiring the current position of the joint of the target mechanical arm, and planning the target motion direction of at least one joint of the target mechanical arm based on the current position of the joint of the target mechanical arm and the target pose of the target mechanical arm.
To achieve the above object, the present invention also provides a surgical robot system comprising:
the surgical operation device comprises a first mechanical arm and a second mechanical arm;
An endoscope assembly for capturing an image of a target tissue and comprising a first endoscope mounted on a distal end of the first mechanical arm and a second endoscope mounted on a distal end of the second mechanical arm; one of the first endoscope and the second endoscope is a current endoscope, the other is a target endoscope, a mechanical arm for mounting the current endoscope is a current mechanical arm, and a mechanical arm for mounting the target endoscope is a target mechanical arm; the method comprises the steps of,
The endoscope adjustment device of any one of the preceding claims, which is communicatively connected to the surgical operation device and the endoscope assembly.
compared with the prior art, the endoscope adjusting device, the endoscope adjusting method and the surgical robot system have the following advantages:
The endoscope adjusting device includes: the system comprises a first pose acquisition module, a pose planning module, a second pose acquisition module and a motion planning module, wherein the first pose acquisition module acquires the position of a target tissue under the coordinate system of the mechanical arm base based on an image acquired by a current endoscope; the target tissue is positioned in the field of view of the current endoscope, and the current endoscope is hung at the tail end of the current mechanical arm; the pose planning module is used for acquiring a target pose of a target mechanical arm on which a target endoscope is mounted based on the position of the target tissue under the mechanical arm base coordinate system; when the target mechanical arm is in the target pose, the target tissue is positioned in the field of view of the target endoscope, and the target mechanical arm is different from the current mechanical arm; the second pose acquisition module is used for acquiring the current position of the joint of the target mechanical arm; the motion planning module is used for planning a target motion direction of at least one joint of the target mechanical arm based on the current position of the target endoscope and the target pose of the target mechanical arm. The endoscope adjusting device is applied to a surgical robot system, so that when the surgical robot system is used for performing a minimally invasive interventional operation, switching of the two endoscopes can be rapidly realized through operation of the endoscope adjusting device, and device collision during switching of the endoscopes is reduced or even avoided.
Drawings
The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:
Fig. 1 is a schematic view of an application scenario of a surgical robot system according to an embodiment of the present invention;
FIG. 2 is a schematic view of a surgical robotic system in accordance with an embodiment of the present invention, not shown with an endoscope;
FIG. 3 is a schematic view of a partial application scenario of a surgical robotic system according to an embodiment of the present invention, showing two endoscopes;
FIG. 4 is a schematic view of a surgical handling device of a surgical robotic system provided in accordance with an embodiment of the present invention;
FIG. 5 is a schematic view of an endoscope of a surgical robotic system provided in accordance with an embodiment of the present invention;
FIG. 6 is a schematic view of an application scenario of a surgical robotic system according to an embodiment of the present invention, showing a schematic view when switching endoscopes, in which a current second endoscope orientation is depicted in solid lines and a pose-adjusted second endoscope orientation is depicted in dashed lines;
FIG. 7 is a partial flow chart of an endoscope adjustment method performed by an endoscope adjustment device when the surgical robotic system of the present invention is in use, according to one embodiment;
FIG. 8 is a schematic diagram of the mathematical modeling principle of a binocular vision apparatus;
FIG. 9 is a schematic diagram of the positioning principle of a binocular vision apparatus;
FIG. 10 is a schematic view of a robotic arm of a surgical robotic system provided in accordance with an embodiment of the present invention;
FIG. 11 is a schematic view of a robotic arm of a surgical robotic system according to an embodiment of the present invention, the movement pattern of the various joints of the robotic arm being schematically represented by arrows in FIG. 11;
FIG. 12 is a schematic view of an adjustment arm of a robotic arm of a surgical robotic system provided in accordance with an embodiment of the present invention;
FIG. 13 is a schematic illustration of the positioning of a tool arm of a robotic arm of a surgical robotic system according to one embodiment of the present invention;
FIG. 14 is a schematic view of a movement direction of a first joint of a robotic arm of a surgical robotic system according to an embodiment of the present invention, wherein "+" indicates a positive direction and "-" indicates a negative direction;
FIG. 15 is a partial flow chart of step S30 of an endoscopic method performed by an endoscopic adjustment apparatus when the surgical robotic system provided in accordance with an embodiment of the present invention is in use;
FIG. 16 is a partial flow chart of a surgical robotic system according to one embodiment of the present invention when switching the field of view of an endoscope during a surgical procedure, wherein the endoscope adjusting device automatically controls the movement of the target manipulator and prompts the movement direction information, the first prompt information and the second prompt information of the target by the voice mechanism;
FIG. 17 is a partial flow chart of a surgical robotic system for switching the field of view of an endoscope during a surgical procedure, according to one embodiment of the present invention, wherein the target robotic arm is manually actuated by an operator;
fig. 18 is a partial flow chart of the surgical robotic system of the present invention in performing a switch of view of an endoscope during a surgical procedure, wherein the movement of the target robotic arm is controlled by an operator through manipulation of a master hand.
FIG. 19 is a schematic view of a partial structure of a surgical robot system according to an embodiment of the present invention, in which a prompting device is a first light prompting mechanism;
FIG. 20 is an enlarged schematic view of the surgical robotic system of FIG. 19 at A;
FIG. 21 is a partial flow chart illustrating a surgical robotic system according to an embodiment of the present invention when performing a view switch of an endoscope during a surgical procedure, wherein an endoscope adjusting device automatically controls a target mechanical arm to move, and a target movement direction, a first prompt message, and a second prompt message are prompted by a light change of a first light prompt mechanism;
FIG. 22 is a schematic view of a partial structure of a surgical robotic system according to an embodiment of the present invention, wherein the prompting device is a second light prompting mechanism;
FIG. 23 is an enlarged schematic view of the surgical robotic system shown in FIG. 22 at B
Fig. 24 is a schematic view showing a partial structure of a surgical robot system according to an embodiment of the present invention, in which a prompting device is a display screen.
Detailed Description
Other advantages and effects of the present invention will become apparent to those skilled in the art from the following disclosure, which describes the embodiments of the present invention with reference to specific examples. The invention may be practiced or carried out in other embodiments that depart from the specific details, and the details of the present description may be modified or varied from the spirit and scope of the present invention. It should be noted that, the illustrations provided in the present embodiment merely illustrate the basic concept of the present invention by way of illustration, and only the components related to the present invention are shown in the drawings and are not drawn according to the number, shape and size of the components in actual implementation, and the form, number and proportion of the components in actual implementation may be arbitrarily changed, and the layout of the components may be more complex.
in addition, each embodiment of the following description has one or more features, respectively, which does not mean that the inventor must implement all features of any embodiment at the same time, or that only some or all of the features of different embodiments can be implemented separately. In other words, those skilled in the art can implement some or all of the features of any one embodiment or a combination of some or all of the features of multiple embodiments selectively, depending on the design specifications or implementation requirements, thereby increasing the flexibility of the implementation of the invention where implemented as possible.
The invention will be further described in detail with reference to the accompanying drawings, in order to make the objects, advantages and features of the invention more apparent. It should be noted that the drawings are in a very simplified form and are all to a non-precise scale, merely for convenience and clarity in aiding in the description of embodiments of the invention. The same or similar reference numbers in the drawings refer to the same or similar parts.
Fig. 1 to 3 are schematic views illustrating an application scenario of a surgical robot system according to an embodiment of the present invention. As shown in fig. 1 to 3, the surgical robot system includes a control end and an execution end, the control end includes a doctor console and a doctor end control device 10 disposed on the doctor console, and an operation master hand (in the drawing) is disposed on the doctor end control device 10. The execution end includes devices such as a patient end control device (not shown), a surgical operation device 20, an image display device 30, an endoscope assembly (not shown) and the like. Wherein the endoscope assembly is used for entering the body of the target object 1 and acquiring an image of a target tissue 2 located in the body of the target object 1. The endoscope assembly comprises a first endoscope 41 and a second endoscope 42. The image display device 30 is communicatively connected to the endoscope assembly and is configured to receive and display the image of the target tissue 2 acquired by the endoscope assembly.
As shown in fig. 2 to 4, the surgical operation device includes an operation platform 21 and a plurality of robot arms 22. The plurality of robot arms 22 are respectively referred to as a first robot arm 22a, a second robot arm 22b, and a third robot arm (not labeled in the drawing). The first mechanical arm 22a is used for mounting the first endoscope 41, the second mechanical arm 22b is used for mounting the second endoscope 42, the number of the third mechanical arms 22c is at least one, and each third mechanical arm is used for mounting a surgical instrument (not shown in the figure) which is used for entering the interior of the target object 1 to perform a surgical operation on the target tissue 2. Here, the target object 1 is, for example, a patient or, in a simulated operation, a phantom, and the guiding means is, for example, a stamp card.
Generally, as shown in fig. 5, the endoscope includes a light source interface 40a, a mirror lever 40b, and an image capturing element 40c. Wherein the light source interface 40a is disposed at the proximal end of the mirror rod 40b and is configured to interface with a light source (not shown). The image pickup element 40c is disposed at the distal end of the mirror lever 40b, and picks up an image of the target tissue 2 based on the illumination light generated by the light source.
In operation, the light source interface 40a of the first endoscope 41 is connected to a first light source, the light source interface 40a of the second endoscope 42 is connected to a second light source, and the second illumination light generated by the second light source is different from the first illumination light generated by the first light source. In one exemplary embodiment, one of the first light source and the second light source is a white light source, the other is a fluorescent light source, and then one of the first illumination light and the second illumination light is white light, and the other is fluorescent light. Thus, the endoscope connected to the light source generating white light is a white light endoscope, and the endoscope connected to the light source generating fluorescence is a fluorescence endoscope. When the surgical robot system is used to perform a surgical operation, the combined use of the first endoscope 41 and the second endoscope 42 can effectively achieve the effect of identifying or enhancing the visual saliency of the target tissue.
During the operation, the operator alternately observes the target tissue 2 using the first endoscope 41 and the second endoscope 42. That is, the operator switches the endoscope according to the actual need during the operation. It will be appreciated that if the operator is currently viewing the target tissue 2 with the first endoscope 41, then at the present time the target tissue 2 must be within the field of view of the first endoscope 41, but it may not be within the field of view of the second endoscope 42. For the case where the target tissue 2 is not within the visual field of the second endoscope 42 at the present time (as shown by the second endoscope 42 drawn with a solid line in fig. 6), if the operator needs to switch to observe the target tissue 2 with the second endoscope 42, the second robot arm 22b needs to be controlled to move to adjust the pose of the second endoscope 42 so that the second endoscope 42 moves to a state where its visual field covers the target tissue 2 (as shown by the second endoscope 42 drawn with a broken line in fig. 6) to satisfy the requirement that the target tissue 2 appears within the visual field of the second endoscope 42.
Further, the surgical robotic system further includes an endoscope adjustment device communicatively coupled to the surgical manipulation apparatus and the endoscope assembly and configured to receive the images acquired by the endoscope assembly and to perform an endoscope adjustment method aimed at quickly and accurately adjusting the pose of the second robotic arm 22b to the field of view of the second endoscope 42 covering the target tissue by the execution of the endoscope adjustment method.
The embodiment of the invention does not limit the specific setting mode of the endoscope adjusting device, as long as the method can very execute the pose adjusting method of the mechanical arm. Optionally, the endoscope adjusting device is disposed on a doctor-side control device, or the endoscope adjusting device is disposed on a patient-side control device, or a part of the endoscope adjusting device is disposed on the doctor-side control device, and another part of the endoscope adjusting device is disposed on the patient-side control device, or at least a part of the endoscope adjusting device is independent from the doctor-side control device and the patient-side control device.
The robot arm pose adjustment method will be described below by taking a currently used endoscope as the first endoscope 41 and taking a target endoscope to be switched as the second endoscope 42 as an example. It can be understood that, when the second endoscope 42 is the target endoscope to be switched, the second mechanical arm 22b is the target mechanical arm that needs to be adjusted in pose, and the first mechanical arm 22a is the current mechanical arm on which the current endoscope is mounted. In addition, the following description may be modified by those skilled in the art to adapt to the case where the currently used endoscope is the second endoscope 42, the second manipulator 22b is the current manipulator, the target endoscope to be switched is the first endoscope 41, and the target manipulator required for pose adjustment is the first manipulator 22 a.
Fig. 7 shows an overall flowchart of the endoscope adjustment method. Referring to fig. 7, the endoscope adjusting method includes:
Step S10, acquiring the position of the target tissue 2 relative to the manipulator base based on the image acquired by the current endoscope, that is, the first endoscope 41. It will be appreciated that the image of the target tissue 2 is displayed in the image acquired by the first endoscope 41.
Step S20, acquiring a target pose of the target mechanical arm, that is, the second mechanical arm 22b, based on the pose of the target tissue 2 relative to the mechanical arm base. The target pose is a corresponding pose of the second robotic arm 22b when the target tissue 2 is within the field of view of the second endoscope 42.
Step S30, acquiring a current position of a joint of the second mechanical arm 22b, and planning a target movement direction of at least one joint of the second mechanical arm 22b based on the current position of the joint of the second mechanical arm 22b and the target pose of the second mechanical arm 22 b.
Thereafter, the second mechanical arm 22b may be controlled to move according to the target movement direction of each joint in any suitable manner, and the second mechanical arm 22b is caused to move to the target pose, so as to drive the second endoscope 42 to move to the view of the second endoscope to cover the target tissue 2. In practice, the movement of the second robot arm 22b may be automatically controlled by the endoscope adjusting device according to the target movement direction of each joint (as shown in fig. 16 and 21), or the movement of the second robot arm 22b may be manually controlled by an operator according to the target movement direction of each joint (as shown in fig. 17), or the movement of the second robot arm 22b may be controlled by an operator according to the target movement direction of each joint through manipulation of the manipulation master hand (as shown in fig. 18).
The current time is considered to be any time during the execution of the robot pose adjustment method.
Based on the above, the endoscope adjusting device at least comprises a first pose acquisition module, a pose planning module, a second pose acquisition module and a motion planning module. The first pose acquisition module is in communication connection with the endoscope assembly, the pose planning module is in communication connection with the first pose acquisition module, and the motion planning module is in communication connection with the pose planning module and the second pose acquisition module. The first pose acquisition module is used for executing the step S10, the pose planning module is used for executing the step S20, and the second pose acquisition module and the motion planning module are used for executing the step S30 together. In addition, "pose" includes position and pose.
Next, step S10, step S20, and step S30 will be further described.
The specific process of step S10 described above includes step S11, step S12, and step S13.
Wherein the step S11 comprises acquiring a position of the target tissue 2 in a coordinate system of an image acquisition element 40c of the first endoscope 41 based on the image acquired by the first endoscope 41. Those skilled in the art will appreciate that the image capturing element 40c of the endoscope is a binocular vision device including two cameras. Therefore, the specific operation in step S11 is to first segment the image acquired by the first endoscope 41 by using any suitable image segmentation method, so as to extract the image of the target tissue 2. The position of the target tissue 2 in the coordinate system of the image acquisition element 40c of the first endoscope 41 is then obtained based on the positioning principle of the binocular vision apparatus.
Two cameras of the binocular vision device respectively acquire two digital images of the measured object from different angles at the same time, and recover three-dimensional geometric information of the measured object based on a parallax principle to acquire the position of the measured object. Fig. 8 schematically shows a mathematical modeling diagram of a binocular vision apparatus, and fig. 9 schematically shows a positioning principle diagram of the binocular vision apparatus. The two cameras of the binocular vision apparatus are referred to herein as the left camera and the right camera, respectively. As shown in fig. 8 and 9, the point P ((x, y, z) is a characteristic point on the object to be measured, the coordinate system Flis the coordinate system of the left camera, Olis the optical center of the left camera and the coordinate system FRIs the coordinate system of the right camera, ORIs the optical center of the right camera. If the left camera view point P is utilized, the image of the point P in the image plane 01 of the left camera is the point PLbut we cannot go through PlKnowing the three-dimensional position of P, in fact, at OlPlThe image points on the connecting line, which are arbitrary to the left camera, are all PlThus from PlIs only aware of the position of point P in line OlPlAnd (3) upper part. Similarly, from the right camera, only the point P is known to be located at the execution OrPrand (3) upper part. Thus, when the left camera and the right camera capture the same feature point P ((x, y, z) of the object under test at the same time, the straight line OlPlAnd straight line OrPrThe intersection point of (a) is the position of point P in space.
Optical center O of left cameraLand the right camera ORThe distance between the optical centers of the left camera and the right camera is the base line b, and the focal lengths of the left camera and the right camera are f. The two cameras shoot characteristic points P (x, y, z) of the measured object at the same moment, and the following relational expression is obtained according to the principle of similar triangles:
and then obtain:
Thus, the coordinates of the feature point P on the object under test in the coordinate system of the binocular vision apparatus can be obtained. And similarly, obtaining the coordinates of other characteristic points on the measured object under the coordinate system of the binocular vision device, thereby obtaining the position of the measured object under the binocular vision device. In the embodiment of the present invention, the position of the target tissue 2 in the coordinate system of the image capturing element 40c of the first endoscope 41 is obtained.
step S12 includes acquiring a position of the image capturing element 40c of the first endoscope 41 in the robot arm base coordinate system. As is well known, the first endoscope 41 is mounted on the distal end of the first mechanical arm 22a, and the image capturing element 40c is disposed on the distal end of the first endoscope 41. The position of the image capturing element 40c of the first endoscope 41 in the arm base coordinate system can be obtained based on the design parameters of the first arm 22a, the current joint angles of the respective joints of the first arm 22a, and the design parameters of the first endoscope 41. Preferably, the first pose acquisition module performs the step S12, and performs a positive kinematic calibration (dnavit-Hartenberg calibration, DH calibration) on the first mechanical arm 22a to improve the accuracy of the acquired position of the image acquisition element 40c of the first endoscope 41 under the mechanical arm base coordinate system.
step S13 includes acquiring a position of the target tissue 2 in the arm base coordinate system based on a position of the target tissue 2 in the coordinate system of the image capturing element 40c of the first endoscope 41 and a position of the image capturing element 40c of the first endoscope 41 in the arm base coordinate system.
Step S20 may be implemented by any suitable method in the prior art, as long as the target pose of the second robot arm 22b can be obtained when the target tissue 2 is located within the field of view of the second endoscope 42. It should be appreciated that when planning the target pose of the second robot arm 22b, it should be considered that the target pose of the second robot arm 22b should not interfere with other robots or devices, and that the second robot arm 22b should not collide with other robots or devices when moving from the current pose to the target pose.
The mechanical arm 22 may include a plurality of joints, and as illustrated in fig. 10 to 13, the mechanical arm 22 includes seven joints connected in sequence. In some non-limiting embodiments, the mechanical arm includes an adjusting arm 221 and a tool arm 222 that are connected to each other, the adjusting arm 221 is connected to the operation platform 21 and includes four joints, namely, a first joint 221a, a second joint 221b, a third joint 221c, and a fourth joint 221d that are sequentially connected, and the tool arm 222 includes three joints, namely, a fifth joint 222a, a sixth joint 222b, and a seventh joint 222c that are sequentially connected. The first joint 221a, the third joint 221c, the fourth joint 221d, the fifth joint 222a, and the sixth joint 222b are rotation joints, and the second joint 221b and the seventh joint 222c are translation joints. For the second mechanical arm 22b, the pose adjustment includes position adjustment of seven joints at most, namely: rotating the first joint 221a in either a positive or negative direction from the current position until the corresponding target position is reached; translating the second joint 221b from the current position in either the positive or negative direction until the corresponding target position is reached; rotating the third joint 221c in either a positive or negative direction from the current position until the corresponding target position is reached; rotating the fourth joint 221d in either a positive or negative direction from the current position until the corresponding target position is reached; rotating the fifth joint 222a in either a positive or negative direction from the current position until the corresponding target position is reached; rotating the sixth joint 222b in either a positive or negative direction from the current position until the corresponding target position is reached; the seventh joint 222c is translated in either a positive or negative direction from the current position until the corresponding target position is reached. Here, for the rotary joint, one of the positive direction and the negative direction is clockwise, the other is counterclockwise, for example, the positive direction is counterclockwise, and the negative direction is clockwise (as shown in fig. 14), for the translational joint, the positive direction is, for example, a direction approaching the target object 1, and the negative direction is a direction separating from the target object 1. It will be appreciated that each joint should move within the allowable range during movement from the current position to the corresponding target position to ensure safety.
as shown in fig. 15, the step S30 includes at least step S31, step S32, step S33, and step S34.
Wherein, the step S31 includes obtaining the current position of a designated joint, such as an nth joint, of the second mechanical arm 22b, where N is a positive integer greater than or equal to 1. Specifically, the second pose acquisition module receives position data monitored by a position sensor installed at the joint, and further acquires the current position of the corresponding joint according to the position data. The position sensor may be an angle encoder or other type of angle sensor for a rotary joint and a distance sensor for a translational joint. It is to be appreciated that the second pose acquisition module is communicatively coupled to the position sensor.
The step S32 includes acquiring a target position of each joint of the second mechanical arm 22b according to the target pose of the second mechanical arm 22 b. The target pose of the second robot arm 22b is determined by the target positions of the respective joints of the second robot arm 22 b. Conversely, knowing the target pose of the second robotic arm 22b, the target positions of the various joints of the second robotic arm 22b can be deduced back from the configuration of the second robotic arm 22 b. In practice, the step S32 may obtain the target position of each joint by calculating the second robot arm 22b using a robot inverse kinematics algorithm.
the step S33 includes calculating a difference between the target position and the current position of the nth joint.
step S34 includes determining whether the nth joint is located at a position corresponding to the target position according to a difference between the target position and the current position of the nth joint, if not, step S35 is further performed, and step S35 includes obtaining a target movement direction of the nth joint according to a difference between the target position and the current position of the nth joint.
In step S34, if the difference between the target position and the current position of the nth joint is within the preset range, it is determined that the nth joint is at the target position. And if the difference value between the target position and the current position of the N joint is not in the preset range, judging that the N joint is not in the target pose.
if the difference between the target position of the nth joint and the current position is a positive value, the target movement direction of the nth joint obtained in the step S35 is a positive direction, otherwise, if the difference between the target position of the nth joint and the current position is a negative value, the target movement direction of the nth joint obtained in the step S35 is a negative direction. For example, when the nth joint is a rotary joint, the target position of the nth joint is θgoalThe current position is theta0If thetagoal-θ0And when the target motion direction of the N joint is positive, the target motion direction of the N joint is along the positive direction, and when the target motion direction of the N joint is negative, the target motion direction of the N joint is along the negative direction. When the nth joint is a translation joint and the nth joint translates in the Y direction (as shown in fig. 11, for example), the target position of the nth joint is Ngoal(0，ygoal0), the current position is N0(0，y00), the difference between the two can be directly expressed as ygoal-y0If ygoal-y0If positive, the target movement direction of the nth joint in translation is positive, if ygoal-y0And if the motion direction is negative, the target motion direction of the nth joint during translation is the reverse direction.
it will be appreciated that steps S32 to S35 are performed by the motion planning module, and the motion planning module performs step S32 once, and performs at least steps S33 and S34 for each joint.
In some embodiments, the endoscope adjustment device further comprises a motion control module in communication with the motion planning module and the robot. The endoscope adjustment method further includes a step S50 performed by the motion control module, the step S50 including controlling each joint of the second mechanical arm 22b to move in a corresponding target motion direction. That is, as shown in fig. 16 and 21, after the target movement direction of the nth joint is acquired, the nth joint is automatically controlled by the movement control module to move in accordance with the target movement direction thereof.
in other embodiments, as shown in fig. 17, after the target movement direction of the nth joint of the second robot arm 22b is acquired, the nth joint is manually controlled by the operator to move in accordance with the target movement direction thereof, or, as shown in fig. 18, the nth joint is controlled by the operator to move in accordance with the target movement direction thereof by manipulating the manipulation master hand.
it will be appreciated that, for the joint performing the step S35, the steps S33 and S34 are also repeatedly performed during the nth joint movement of the second mechanical arm 22b until the corresponding joint reaches the corresponding target position.
Further, the surgical robot system further comprises a display device, and correspondingly, the endoscope adjusting device further comprises a display control module, and the display control module is in communication connection with the display device. As shown in fig. 15, the endoscope adjusting method further includes a step S60 performed by the display control module, the step S60 including controlling the display device to display the target movement direction of the nth joint. Thus, when the motion control module automatically controls the nth joint to move according to the target motion direction, an operator can confirm whether the motion of the nth joint is correct through the display of the display device. And when the operator controls the nth joint movement manually or by manipulating the operation master hand, the operator can intuitively see the specific direction of the target movement direction through the display of the display device.
Further, the endoscope adjusting device further comprises a prompt information generating module, and the prompt information generating module is in communication connection with the motion planning module and the display device. The mechanical arm pose adjustment method further comprises a step S70 executed by the prompt information generation module, wherein the step S70 comprises the step of generating first prompt information when the Nth joint is located at the target position. Subsequently, the display control module executes step S80, where step S80 includes controlling the display device to display the first prompt message.
In addition, the endoscope adjusting method further includes a step S01 executed by the prompt message generating module and a step S02 executed by the display control module. The step S01 includes generating a confirmation message. The step S02 includes controlling the display device to display the confirmation information, so as to prompt the operator to confirm the current nth joint. It will be appreciated that in adjusting the pose of the second robot arm 22b, the adjustment operation is typically performed on all the joints sequentially in the connection order, starting from the first joint 221 a. The endoscope adjustment method further includes a step S36 executed by the motion planning module after the position adjustment of each joint is finished, where the step S36 includes determining whether all joints of the second mechanical arm 22b are at respective corresponding target positions (i.e., determining whether the second mechanical arm 22b reaches the target pose), if not, updating N, and executing the step S01, the step S02, the step S42, etc. for the N-th joint after the update, if yes, determining that the second mechanical arm 2b reaches the target pose, and executing the step S90 and the step S100, where the step S90 includes the prompt information generating module generating second prompt information, and the step S100 includes the display control module controlling the display device to display the second prompt information.
accordingly, as shown in fig. 16 to 18 and 21, a specific procedure when the operator switches the currently used first endoscope 41 to the second endoscope 42 in the process of performing the surgical operation by the surgical robot is as follows:
First, the steps S10, S20, and S32 are performed, and then the steps S01 and S02 are performed. After confirming that the joint to be currently adjusted is the nth joint, the operator executes the step S31, the step S33 and the step S34 for the nth joint. If the determination result of the step S34 is yes, the steps S70, S80 and S36 are executed. If the determination result of the step S36 is yes, the step S90 and the step S100 are executed again, and if the determination result of the step S36 is no, the N is updated, and the step S01 and the subsequent steps are executed based on the updated N. If the determination result of the above step S34 is "no", the step S35 is executed for the nth joint, and then the steps S60 and S50 are executed. And repeatedly executing the step S31 and the following steps in the process of executing the step S50 on the nth joint until the nth joint reaches the target position, that is, the judgment result of the step S34 is yes, and then executing the steps after the step S34.
As previously described, step S50 may be performed by the motion control module of the endoscope adjusting device, or manually controlled by an operator (as shown in fig. 17) or by manipulating the operating master hand to control the nth joint (as shown in fig. 18) to move in its target motion direction. The other steps are all performed by the endoscope adjusting device.
in an optional embodiment, the display device is a voice device, which may prompt the operator to confirm the current nth joint through voice broadcasting, or may display the first prompt information and the second prompt information through voice broadcasting. That is, step S02, step S60, step S80, and step S100 are performed by the voice device.
In an alternative embodiment, as shown in fig. 19 and 20, the display device is a first light prompting mechanism, and the first light prompting mechanism includes at least two indicator light groups 50. The first mechanical arm 22a is provided with one indicator lamp set 50, the second mechanical arm 22b is provided with one indicator lamp set 50, when the pose of the first mechanical arm 22a is adjusted, the indicator lamp set 50 arranged on the first mechanical arm 22a works, and when the pose of the second mechanical arm 22b is adjusted, the indicator lamp set 50 arranged on the second mechanical arm 22b works. Each of the indicator light groups 50 may include a first indicator light 51 and a second indicator light 52, where each of the first indicator light 51 and the second indicator light 52 has a state of flashing, always-on, always-off, etc. in different colors, as shown in fig. 21, a person skilled in the art may set the states of the first indicator light 51 and the second indicator light 52 so that they can display corresponding information, for example, by controlling the first indicator light 51 and the second indicator light 52 to flash in corresponding colors to prompt the operator to confirm the nth joint to be currently adjusted, for example, controlling the first indicator light 51 and the second indicator light 52 to flash in a first color to prompt the operator to confirm the joint to be currently adjusted as the first joint 221a and a second color different from the first color to prompt the operator to confirm the joint to be currently adjusted as the second joint 221b light … …. The target movement direction is displayed by controlling one of the first indicator lamp 51 and the second indicator lamp 52 to be normally on and the other to be normally off, for example, controlling the first indicator lamp 51 to be normally on and the second indicator lamp 52 to be normally off to display the target movement direction as a positive direction, and controlling the first indicator lamp 51 to be normally off and the second indicator lamp 52 to be normally on to display the target movement direction as a negative direction. The first indicator lamp 51 and the second indicator lamp 52 are controlled to be turned off normally to display the first prompt information and the second prompt information.
Alternatively, referring to fig. 22 and 23, the display device is a second light prompting mechanism, and the second light prompting mechanism includes at least two third indicator lamps 60. The first mechanical arm 22a is provided with a third indicator lamp 60, the second mechanical arm 22b is provided with a third indicator lamp 60, when the pose of the first mechanical arm 22a is adjusted, the third indicator lamp 60 arranged on the first mechanical arm 22a works, and when the pose of the second mechanical arm 22b is adjusted, the third indicator lamp 60 arranged on the second mechanical arm 22b works. The third indicator lamp 60 has various states, such as flashing in different colors, being normally on, off in different colors, etc. Those skilled in the art can also set the state of the third indicator lamp 60 to enable the third indicator lamp 60 to display different information, for example, control the third indicator lamp 60 to flash with different colors to prompt an operator to confirm the nth joint to be currently adjusted, control the third indicator lamp 60 to always light with different colors to display the target display direction, and control the third indicator lamp 60 to display the first prompt information and the second prompt information in an always-off manner.
In another alternative embodiment, as shown in fig. 24, the display device includes a display screen 70, and the number of the display screens 70 is at least two. The first mechanical arm 22a is provided with one display screen 70, the second mechanical arm 22b is provided with one display screen 70, when the pose of the first mechanical arm 22a is adjusted, the display screen 70 arranged on the first mechanical arm 22a works, and when the pose of the second mechanical arm 22b is adjusted, the display screen 70 arranged on the second mechanical arm 22b works. The display screen 70 is provided with an indication icon comprising a first arrow 71 pointing in the positive direction, a second arrow 72 pointing in the negative direction, and a confirmation symbol 73, for example a "v". The first and second arrows 71, 72 have normally on and normally off states, and the confirmation symbol 73 may blink in different colors, and may also be normally on or normally off. When the device works, the operator is prompted to confirm the current N joint to be confirmed by controlling the confirmation symbol to flash with the corresponding color; displaying the target movement direction by controlling one of the first arrow 71 and the second arrow 72 to be normally bright, for example, displaying the target movement direction to be a positive direction by controlling the first arrow 71 to be normally bright, and displaying the target movement direction to be a negative direction by controlling the second arrow 72 to be normally bright; the first hint information and the second hint information are displayed by controlling the confirm symbol 73 to be normally lit.
further, an embodiment of the present invention also provides a computer-readable storage medium having a program stored thereon, which when executed, performs the above-described endoscope adjustment method.
Although the present invention is disclosed above, it is not limited thereto. Various modifications and alterations of this invention may be made by those skilled in the art without departing from the spirit and scope of this invention. Thus, it is intended that the present invention also include such modifications and alterations insofar as they come within the scope of the appended claims or the equivalents thereof.
Claims (10)
1. an endoscope adjustment device, comprising:
The first pose acquisition module is used for acquiring the position of a target tissue under a mechanical arm base coordinate system based on an image acquired by a current endoscope; the target tissue is positioned in the field of view of the current endoscope, and the current endoscope is hung at the tail end of the current mechanical arm;
The pose planning module is used for planning the target pose of the target mechanical arm on which the target endoscope is mounted based on the position of the target tissue under the mechanical arm base coordinate system; when the target mechanical arm is in the target pose, the target tissue is positioned in the field of view of the target endoscope, and the target mechanical arm is different from the current mechanical arm;
the second pose acquisition module is used for acquiring the current position of the joint of the target mechanical arm; the method comprises the steps of,
And the motion planning module is used for planning the target motion direction of at least one joint of the target mechanical arm based on the current position of the joint of the target mechanical arm and the target pose of the target mechanical arm.
2. the endoscope adjustment device of claim 1, wherein the first pose acquisition module is configured to:
Acquiring the position of the target tissue under the coordinate system of an image acquisition element of the current endoscope based on the image acquired by the current endoscope;
Acquiring the position of an image acquisition element of the current endoscope under the coordinate system of the mechanical arm base;
and acquiring the position of the target tissue under the coordinate system of the mechanical arm base based on the position of the target tissue under the coordinate system of the image acquisition element of the current endoscope and the position of the image acquisition element of the current endoscope under the coordinate system of the mechanical arm base.
3. the endoscope adjustment device of claim 2, wherein the first pose acquisition module is further configured to:
and performing positive kinematic calibration on the current mechanical arm.
4. The endoscope adjustment device of claim 1, wherein the motion planning module is configured to:
acquiring a target position of an N joint according to a target pose of the target mechanical arm;
Calculating the difference value between the target position and the current position of the N joint;
Judging whether the N joint is at the corresponding target position according to the difference value between the target position of the N joint and the current position; if not, acquiring the target movement direction of the N joint according to the difference value between the target position and the current position of the N joint.
5. The endoscope adjustment device of any one of claims 1-4, further comprising a display control module to:
And controlling a prompting device to display the movement direction of the target.
6. The endoscope adjustment device of any one of claims 1-4, further comprising a motion control module for: and controlling the corresponding joints of the target mechanical arm to move according to the target movement direction.
7. The endoscope adjustment device of claim 4, further comprising a hint information generation module and a display control module;
The prompt information generation module is used for: generating first prompt information when the N joint is positioned at a corresponding target position;
the display control module is used for: controlling a prompt device to display the first prompt information; and/or the number of the groups of groups,
the prompt information generation module is used for: generating second prompt information when the target mechanical arm reaches the target pose;
The display control module is used for: and controlling a display device to display the second prompt information.
8. an endoscope adjustment device according to claim 3 and wherein said target direction of movement is a rotational or translational direction of said joint.
9. An endoscope adjustment method, comprising:
Acquiring the position of a target tissue under a mechanical arm base coordinate system based on an image acquired by a current endoscope; the target tissue is positioned in the field of view of the current endoscope, and the current endoscope is hung at the tail end of the current mechanical arm;
planning a target pose of a target mechanical arm on which a target endoscope is mounted based on the position of the target tissue under the mechanical arm base coordinate system; when the target mechanical arm is in the target pose, the target tissue is positioned in the field of view of the target endoscope, and the target mechanical arm is different from the current mechanical arm;
acquiring the current position of the joint of the target mechanical arm, and planning the target motion direction of at least one joint of the target mechanical arm based on the current position of the joint of the target mechanical arm and the target pose of the target mechanical arm.
10. A surgical robotic system, comprising:
the surgical operation device comprises a first mechanical arm and a second mechanical arm;
An endoscope assembly for capturing an image of a target tissue and comprising a first endoscope mounted on a distal end of the first mechanical arm and a second endoscope mounted on a distal end of the second mechanical arm; one of the first endoscope and the second endoscope is a current endoscope, and the other is a target endoscope; the method comprises the steps of,
The endoscope adjustment device of any one of claims 1-8, communicatively coupled to the surgical manipulation device and the endoscope assembly.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410108945.9A CN117752376A (en) | 2024-01-25 | 2024-01-25 | endoscope adjusting device and method and surgical robot system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410108945.9A CN117752376A (en) | 2024-01-25 | 2024-01-25 | endoscope adjusting device and method and surgical robot system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117752376A true CN117752376A (en) | 2024-03-26 |
Family
ID=90314779
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410108945.9A Pending CN117752376A (en) | 2024-01-25 | 2024-01-25 | endoscope adjusting device and method and surgical robot system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117752376A (en) |
-
2024
- 2024-01-25 CN CN202410108945.9A patent/CN117752376A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107049492B (en) | Surgical robot system and method for displaying position of surgical instrument | |
| KR102501099B1 (en) | Systems and methods for rendering on-screen identification of instruments in teleoperated medical systems | |
| CN110786819B (en) | Surgical imaging system | |
| JP5908172B2 (en) | Surgery support system and control method for surgery support system | |
| CN113616338B (en) | Computer readable storage medium, alignment method and system, and surgical robot system | |
| US9271797B2 (en) | Robotic surgery | |
| CN109965981B (en) | Starting method of surgical robot, readable memory and surgical robot | |
| CN108433809B (en) | Device for setting and retrieving a reference point during a surgical procedure | |
| US10932657B2 (en) | Endoscope with wide angle lens and adjustable view | |
| JP2019537461A (en) | Optical system for surgical probe, system and method incorporating the same, and method of performing surgery | |
| KR20230003408A (en) | Systems and methods for onscreen identification of instruments in a teleoperational medical system | |
| JP2021531910A (en) | Robot-operated surgical instrument location tracking system and method | |
| CN107847285B (en) | Input mechanism and medical system | |
| US11832790B2 (en) | Method of alerting a user to off-screen events during surgery | |
| CN113456221B (en) | Positioning guiding method and system of movable equipment and surgical robot system | |
| EP4181811A2 (en) | Medical arm control system, medical arm control method, medical arm simulator, medical arm learning model, and associated programs | |
| US11638000B2 (en) | Medical observation apparatus | |
| US20230142404A1 (en) | Medical imaging apparatus, learning model generation method, and learning model generation program | |
| WO2017082047A1 (en) | Endoscope system | |
| US20230126611A1 (en) | Information processing apparatus, information processing system, and information processing method | |
| CN117752376A (en) | endoscope adjusting device and method and surgical robot system | |
| EP3840629B1 (en) | Image correction of a surgical endoscope video stream | |
| US20210267435A1 (en) | A system, method and computer program for verifying features of a scene | |
| CN114305695B (en) | Mobile guidance method and system, readable storage medium, and surgical robot system | |
| KR20120052574A (en) | Surgical robitc system and method of driving endoscope of the same |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |