TW202222270A - Surgery assistant system and related surgery assistant method - Google Patents

Abstract

A surgery assistant system includes an algorithm module, a monitoring module, a signal conversion unit and a surgery robotic unit. The algorithm module generates an augmented reality scene and an augmented reality object according to an image of a physical target, and the algorithm module further displays a three-dimensional-path indicator in the augmented reality scene. The monitoring module includes a display device and a control device, wherein the display device displays the augmented reality scene including the augmented reality object and the three-dimensional-path indicator. The control device is operated to control the moving path of the three-dimensional-path indicator in in the augmented reality scene to generate a moving trajectory. The signal conversion unit converts the moving trajectory into a motion signal. The surgery robotic unit receives and executes the motion signal to perform surgery actions on the physical target along the moving trajectory.

Description

Surgical assistance system and related surgical assistance methods

The present invention relates to a monitoring system and method thereof, and in particular to a surgical assistance system and method thereof.

In general surgery, medical images are usually captured by taking X-rays to formulate surgical planning. During the operation, the medical images can be displayed on a monitor of a nearby doctor. However, the surgical operation with high complexity requires real-time monitoring of the patient's on-site status, which is obviously insufficient. This is because the medical image and the actual patient to be operated on are in different fields of view, and the doctor must repeatedly compare the positions of the surgical instruments in the medical image and the actual operating field of view to move the knife correctly during the operation, resulting in The doctors were unable to focus on the same field of vision, which caused great inconvenience. For example, pedicle screw placement is a difficult implantation procedure that improves the condition by implanting pedicle screws into the spine. Any slight angular error can have a big impact on the surgery.

In conclusion, there is a need for a novel surgical assistance system and a related surgical assistance method to improve the stability of surgical operations to avoid the above-mentioned problem of human error.

The purpose of the present invention is to provide a surgical assistance system suitable for surgery, so as to solve the above-mentioned problem of human error through Augmented Reality (AR) technology and a surgical robot unit.

In order to achieve the above objective, an embodiment of the present invention provides a surgical assistance system, which includes an arithmetic module, a monitoring module, and a signal conversion unit. The calculation module is used to generate the augmented reality scene and the augmented reality object according to the physical target object, the augmented reality object system simulates the target object, and the augmented reality object system is displayed in the augmented reality scene. The group also displays 3D path indicators in the augmented reality scene. The monitoring module includes a display device and a control device, wherein the display device is used to display an augmented reality scene including augmented reality objects and three-dimensional path indicators. The control device is operated to control the movement path of the three-dimensional path pointer in the augmented reality scene to generate the motion trajectory. The signal converting unit is used for converting the motion track into a first motion signal. The surgical robot unit receives and executes the first action signal, so as to perform a surgical action on the physical target along the motion track.

In an embodiment of the present invention, the calculation module generates a user interface, the display device simultaneously displays the user interface and the augmented reality scene, the user interface is controlled by the control device, and is used to adjust the spatial parameters of the three-dimensional path index, to determine the movement trajectory.

In an embodiment of the present invention, the spatial parameters include a plurality of adjustment fields of X-axis coordinate, Y-axis coordinate, Z-axis coordinate, rotation angle around X-axis, rotation angle around Y-axis, and rotation angle around Z-axis of the three-dimensional path index , which are used to control the three-dimensional path index to move forward and backward along the X axis, left and right along the Y axis, up and down along the Z axis, rotate around the X axis, rotate forward and backward around the Y axis, and rotate left and right around the Z axis.

In an embodiment of the present invention, the control device is further operated to send an interrupt signal to the surgical robot unit to disable the surgical robot unit from executing the first motion signal.

In an embodiment of the present invention, after the motion trajectory is generated, in the case that the surgical robot unit has not yet executed the first motion signal or the surgical robot unit has been interrupted to execute the first motion signal, the calculation module additionally generates a candidate 3D path index, the candidate 3D path indicator The path indicator and the 3D path indicator are simultaneously displayed in the augmented reality scene; the control device is further operated to control the movement path of the candidate 3D path indicator in the augmented reality scene to generate a candidate motion trajectory; the signal conversion unit receives the candidate motion trajectory and converting the motion trajectory into a second motion signal; and the surgical robot unit executes the second motion signal to perform a surgical motion along the candidate motion trajectory.

In an embodiment of the present invention, the surgical robot unit approaches or moves away from the physical target according to the first motion signal, resects at least a part of the physical target, or implants an implant into the physical target.

In an embodiment of the present invention, the augmented reality scene is simulated but does not include: a real scene where the target is located.

In an embodiment of the present invention, the augmented reality scene includes a real scene where the target object is located.

In an embodiment of the present invention, the control device is further operated to change the viewing angle of the augmented reality scene. When the viewing angle of the augmented reality scene is changed, the viewing angle of the augmented reality object and the viewing angle of the three-dimensional path index are calculated through a model. The group operation changes accordingly.

In one embodiment of the present invention, the display device is a head-mounted display, and the control device is a portable controller coupled to the head-mounted display.

In an embodiment of the present invention, the display device is a liquid crystal display, and the control device is an input device coupled to the liquid crystal display.

In addition to the above surgical assistance system, another embodiment of the present invention provides a surgical assistance method, including: an algorithm module generates an augmented reality scene and an augmented reality object according to a physical target, and further generates an augmented reality scene and an augmented reality object in the augmented reality scene The 3D path index is displayed, the augmented reality object system simulates the target, and the augmented reality object system is displayed in the augmented reality scene; the augmented reality object including the augmented reality object and the 3D path index is displayed on the display device operating the control device to control the moving path of the three-dimensional path indicator in the augmented reality scene to generate a motion trajectory; the signal conversion unit converts the motion trajectory into a first motion signal, and the surgical robot unit receives and executes the first motion The signal is used to perform surgical action on the physical target along the motion trajectory.

In an embodiment of the present invention, the calculation module generates a user interface, the display device simultaneously displays the user interface and the augmented reality scene, the user interface is controlled by the control device, and is used to adjust the spatial parameters of the three-dimensional path index, to determine the movement trajectory.

In an embodiment of the present invention, the spatial parameters include a plurality of adjustment fields of X-axis coordinate, Y-axis coordinate, Z-axis coordinate, rotation angle around X-axis, rotation angle around Y-axis, and rotation angle around Z-axis of the three-dimensional path index , which are used to control the three-dimensional path index to move forward and backward along the X axis, left and right along the Y axis, up and down along the Z axis, rotate around the X axis, rotate forward and backward around the Y axis, and rotate left and right around the Z axis.

In an embodiment of the present invention, the following step is further included: the control device sends an interrupt signal to the surgical robot unit to disable the surgical robot unit from executing the first action signal.

In an embodiment of the present invention, after the motion trajectory is generated, in the case that the surgical robot unit has not yet executed the first motion signal or the surgical robot unit is interrupted to execute the first motion signal, the surgical assistance method further executes the following steps: an arithmetic module In addition, a candidate 3D path indicator is generated, and the candidate 3D path indicator and the 3D path indicator are simultaneously displayed in the augmented reality scene; the control device is operated to control the movement path of the candidate 3D path indicator in the augmented reality scene, so as to generate a candidate motion trajectory; The signal conversion unit receives the candidate motion trajectory and converts the motion trajectory into a second motion signal; and the surgical robot unit receives and executes the second motion signal to perform a surgical action along the candidate motion trajectory.

To sum up, the present invention uses augmented reality technology to solve the bottleneck encountered by surgeons in performing surgical operations. The surgeon can rotate the field of view of the augmented reality scene to grasp all the perspectives of the target object, so there is no need to move himself. The relative positional relationship between the target object and the surrounding environment can be grasped by the position of the robot arm; in addition, the robot arm can execute the pre-input robot arm movement trajectory or the robot arm movement trajectory adjusted by the surgeon on the spot, which can avoid the mistakes of manual operation. The present invention also allows the surgeon to interrupt and modify the motion track of the robotic arm, so that it can cope with the unexpected situation of the actual operation.

The disclosure is specifically described with the following examples, which are only for illustration, because for those skilled in the art, various changes and modifications can be made without departing from the spirit and scope of the present disclosure. The scope of protection of the disclosed contents shall be determined by the scope of the appended patent application. Throughout the specification and claims, unless the content clearly dictates otherwise, the meanings of "a" and "the" include that such recitations include "one or at least one" of the element or ingredient. Furthermore, as used in this disclosure, singular articles also include recitations of plural elements or components unless it is obvious from the specific context that the plural is excluded. Also, as used in this description and throughout the claims below, the meaning of "in" may include "in" and "on" unless the content clearly dictates otherwise. The terms used throughout the specification and the scope of the patent application, unless otherwise specified, generally have the ordinary meaning of each term used in the field, in the content disclosed herein and in the specific content. Certain terms used to describe the present disclosure are discussed below or elsewhere in this specification to provide practitioners with additional guidance in describing the present disclosure. Examples anywhere throughout the specification, including the use of examples of any terms discussed herein, are by way of illustration only, and of course do not limit the scope and meaning of the disclosure or any exemplified terms. Likewise, the present disclosure is not limited to the various embodiments set forth in this specification.

The terms "substantially", "around", "about" or "approximately" as used herein shall generally mean within 20% of a given value or range, The best line is within 10%. Furthermore, quantities provided herein may be approximate, and thus are meant to be expressed in terms of the words "about," "about," or "approximately," unless specifically stated otherwise. When a quantity, concentration or other value or parameter has a specified range, a preferred range or a table listing upper and lower ideal values, it shall be deemed to specifically disclose all ranges formed by any pair of upper and lower limits or ideal values, regardless of whether Whether those ranges are disclosed separately. For example, if a certain length of the disclosed range is X cm to Y cm, it should be deemed that the disclosed length is H cm and H can be any real number between X and Y.

Furthermore, if the term "electrically (sexually) coupled" or "electrically (sexually) connected" is used herein, it includes any means of direct and indirect electrical connection. For example, if it is described in the text that a first device is electrically coupled to a second device, it means that the first device can be directly connected to the second device, or indirectly connected to the second device through other devices or connecting means device. In addition, if the transmission and provision of electrical signals are described, those skilled in the art should be able to understand that the transmission of electrical signals may be accompanied by attenuation or other non-ideal changes. In fact, it should be regarded as the same signal. For example, if the electrical signal S is transmitted (or provided) from the terminal A of the electronic circuit to the terminal B of the electronic circuit, it may pass through the source and drain terminals of a transistor switch and/or possible stray capacitance. A voltage drop is generated, but if the purpose of this design is not to use the attenuation or other non-ideal changes generated during transmission (or supply) to achieve some specific technical effects, the electrical signal S is at the terminal A and the terminal of the electronic circuit. B should be regarded as substantially the same signal.

It is understood that the terms "comprising", "including", "having", "containing", "involving", etc. as used herein are open-ended open-ended, which means including but not limited to. In addition, any embodiment of the present invention or the scope of the claims is not required to achieve all of the objects or advantages or features disclosed herein. In addition, the abstract part and the title are only used to assist the search of patent documents, and are not used to limit the scope of the patent application of the present invention.

The system architecture of the present invention can roughly include three applications: augmented reality visualization, surgical robot navigation, and real-time monitoring. The augmented reality visualization application uses augmented reality technology to combine virtual images with real images, or in virtual images and real images. After matching the real environment image, only the virtual image is output; the real-time monitoring application allows users to adjust the movement trajectory of the robotic arm while viewing the 3D image of the target through augmented reality visualization. The surgeon can adjust the trajectory through translation and rotation through the operation interface of the monitoring system, and the adjusted trajectory will also be sent to the robotic arm navigation system for correction. The surgical robot navigation application converts the user's control signal into the movement trajectory of the robot arm. The surgeon can adjust the trajectory through translation and rotation through the operation interface of the monitoring system, and the adjusted trajectory will also be sent to the robot arm navigation system. make corrections. In the robotic arm navigation application, the system first calibrates the robotic arm and the augmented reality environment to the same coordinate system to ensure that the robotic arm coordinates can correctly execute the user's input instructions. The user's input command may be a trajectory planned in advance, or a trajectory planned by the on-site operation monitoring module. The surgeon can adjust the trajectory on the operation interface of the monitoring module (as shown in Figure 5). The task of the entire operation includes confirming the position and angle of the target, as well as the position and angle of the implantation point on the target. Through the present invention, doctors can monitor and adjust these key parameters in real time. After receiving the new trajectory and position adjustment, the surgical robot unit calculates the new execution trajectory, and feeds it back to the augmented reality vision system for confirmation by the doctor. After confirmation, it is handed over to the robotic arm for execution. The above design includes the conversion between the coordinate system of the robot arm and the coordinate system of the virtual augmented reality environment, and the interface for adjusting the trajectory. Therefore, the path planning on the visualization side of the augmented reality can be converted into the trajectory of the robotic arm, and the user can directly plan the surgical action of the robotic arm to achieve the purpose of real-time monitoring.

Please refer to FIG. 1A . FIG. 1A is a schematic diagram of a surgical assistance system 100 according to an embodiment of the present invention. The surgical assistance system 100 includes a monitoring module 20 , an arithmetic module 30 , a surgical robot navigation module 40 and a photographing unit 60 . The photographing unit 60 can be independent of the monitoring module 20 as shown in FIG. 1A , and can also be integrated with the monitoring module 20 or the surgical robot navigation module 40 . For example, the camera unit 60 may be a camera externally connected to a computer device, or a camera lens integrated in smart glasses. The signal transmission mode between the monitoring module 20, the calculation module 30, the surgical robot navigation module 40 and the camera unit 60 can be wired or wireless, that is, these components can be in a coupled relationship, or can be transmitted through each other. Communicate with wireless transmission technologies such as WiFi or Bluetooth. The surgical robot navigation module 40 may include a signal conversion unit 42 and a surgical robot unit (eg, the manipulator 44 shown in FIG. 1A ). The signal conversion unit 42 is used to convert the received control signals to control the manipulator 44 to perform related actions. The surgical robot unit generally refers to a component that is controlled by electricity and can generate torque and displacement. It can be, for example, a robotic arm 44 or a surgical robot, or a minimally invasive device that can be placed in a patient to perform surgery. The material of the surgical robot unit is not Limited to metals. For ease of understanding, most of the examples in the following paragraphs use the robotic arm 44 to represent the surgical robot unit, but the present invention does not exclude the use of other types of devices to perform surgery.

The calculation module 30 simulates the motion of the robotic arm 44 and controls the robotic arm 44 through a kinematic calculation tool, so as to plan the implantation point of the surgical action target as a series of trajectory information of the robotic arm 44 . Finally, the calculation module 30 sends the control information of the robotic arm 44 to the driving motor of the robotic arm 44 to complete the control of the robotic arm 44 . In FIG. 1A , the calculation module 30 generates an augmented reality scene corresponding to the target object 50 (for example, symbols 350 and 450 shown in FIG. 3 and FIG. 4 ), and the augmented reality scene is displayed in the augmented reality scene. environment objects (eg symbol 55 shown in Figures 3 and 4). In addition to augmented reality objects, three-dimensional path indicators corresponding to augmented reality objects (such as the symbol 56 shown in Figure 5) also appear in augmented reality scenes. The monitoring module 20 includes a display device 24 and a control device 22. The display device 24 is used to display the augmented reality scene, and the control device 22 is controlled by a user to transmit control signals, wherein the user 10 can be, for example, a surgeon Or related medical personnel, and the target 50 can be a patient's spine, or a spinal model (if the surgical assistance system 100 is used for simulation or academic research), the present invention does not limit the type of the target 50 .

In the case where the monitoring module 20 is a computer device, the display device 24 and the control device 22 can be integrated into a machine, or can be externally connected to the monitoring module 20 . For example, the display device 24 can be a liquid crystal display, and the control device 22 can be a computer device including an input device coupled to the liquid crystal display, such as a keyboard, a mouse, a button handle, and other devices with an input interface. In another example, the display device 24 may be a head-mounted display (eg, smart glasses with a display screen, such as Epson BT-300 AR glasses), and the control device 22 may be a portable device coupled to the head-mounted display A controller (such as a button disposed on the smart glasses, or a button handle connected to the smart glasses); the display device 24 can pass through the transparent microdisplay on the lens of the smart glasses, so it will not block the user 10 Sight. In the above two embodiments, the user 10 can simultaneously see the augmented reality object and the current scene on the same screen through the display device 24, and the signal transmission method between the display device 24 and the control device 22 can be wired or wireless.

Specifically, the control device 22 is controlled by the user 10 (as indicated by the input operation A1 ) to generate the control signal C1 to the calculation module 30 , and the calculation module 30 executes the relevant operation according to the control signal C1 , and converts the calculation result to the image The mode is displayed on the display device 24 (image data V1 is transmitted as shown in FIG. 1A ). On the other hand, the arithmetic module 30 sends a control signal C2 to the signal conversion unit 42 according to the control signal C1 sent from the control device 22 , and the signal conversion unit 42 converts the control signal C2 into a first action signal S1 to control the mechanical arm 44 , so that the robot arm 44 operates the target 50 according to the requirements of the user 10 (as shown in the operation action A2 ).

Regarding the motion planning part of the robotic arm 44 , the calculation module 30 can plan the drilling trajectory of the steel nail for the affected part in space (it can be a drilling trajectory that conforms to the Karl Fischer coordinates), and calculate the motion of each joint of the robotic arm 44 . parameters (such as torque, displacement, speed, acceleration, etc.), and generate commands to the signal conversion unit 42 according to these parameters. The trajectory planning and kinematic calculation of the robotic arm 44 can be applied to orthopedic surgery, such as pedicle screw implantation surgery. For example, the robotic arm 44 for performing surgery can be implemented with a six-axis robotic arm system that can move arbitrarily in six degrees of freedom. The so-called six degrees of freedom refers to the freedom of movement of an object in three-dimensional space, especially that the object can translate on three mutually perpendicular coordinate axes, front, back, up and down, and left and right, and can also move in three axes (X, Y, Z axes). Displacement in the direction and rotation around three axes, where rotation around the X axis is called roll, rotation around the Y axis is called pitch, and rotation around the Z axis is called yaw.

The present invention can be applied in spinal fusion surgery, where an implant, such as a steel screw, is implanted in a patient, and the trajectory includes a starting position and a straight trajectory. Since the orientation of the bone screws must be kept consistent during the drilling process, otherwise there is a risk of bone fracture. Therefore, the key to planning the trajectory is to appropriately determine the penetration point of the steel nail on the affected part, including the position, angle and length of the penetration path. Before the actual action of the robotic arm, the drilling point will be defined according to the bone model. The reference factors include avoiding the nerve marrow and the effect of the implant after drilling. The path planned by the present invention will be a straight line starting from the drilling hole and ending at the position where the implant is finally placed. During the movement, it is necessary to keep the straight line and not rotate.

Specifically, the establishment of the trajectory of the robotic arm 44 can output the information of the robotic arm 44 through a computer-aided design program so as to be applied to computing software (eg, ROS MoveIt). Table 1 shows the parameters of each axis of the robot arm 44, which is the mathematical model of the robot arm 44. For example, the conversion of Joint1 relative to the base of the robot arm 44 is 0.183 meters in the z direction, and the angle remains unchanged. This is an important basis for inverse kinematics to convert the Cartesian coordinate system to the joint coordinate system. In Table 1, Base represents the base of the robotic arm 44, and Joint1 to Joint6 represent six different joints of the robotic arm 44 (six-axis robotic arm system). Table 1

The robotic arm 44 is calibrated with the camera lens to obtain the relative relationship between the two coordinate systems. The camera detects the mark of the virtual coordinate system set in the actual environment, and then the trajectory planning of the simulation system can be obtained to ensure that the coordinates planned by the virtual system end are consistent with the actual execution machinery. The arms 44 execute the coordinates to coincide with each other. The relative relationship between the camera and the base of the robotic arm 44 is obtained by using the hand-eye correction method. By identifying the position of the tag (TAG) of the bone model through the camera, the relative position of the robotic arm 44 and the bone model can be obtained. In this way, the preoperative trajectory planned on the bone model can be converted into the execution trajectory of the robotic arm 44 .

（1.1）     其中

係為工具座標系（Tool coordinate system, TCS）與機械臂座標系（Robot Base Coordinate System, RBCS）的轉換矩陣，

係為標記座標系與機械臂座標系的轉換矩陣，

標記座標系與相機座標系的轉換矩陣，

係為相機座標系與機械臂座標系的轉換矩陣。 The present invention can use a digital camera (eg Realsense D435, not shown) as the camera unit 60 to detect the surrounding environment, and then provide the information to the robotic arm 44 for execution. The coordinates of the digital camera (D435) are not equal to the coordinate system of the manipulator, however, there is a fixed relationship transformation between these two coordinate systems, which can be estimated by hand-eye correction. Fix a mark (such as AprilTag) on the end axis of the robot arm 44, and capture a variety of postures, and simultaneously record the color image of the digital camera and the end axis posture information of the coordinate system of the robot arm 44 (referred to as the robot arm coordinate system), such as Section 1B The hand-eye correction and image navigation shown in Fig. The camera captures the image and then uses the image to identify the position and attitude of the mark, and the relationship between the mark and the camera coordinate system can be obtained. The relative relationship between the coordinate system of the robot arm and the end axis is known by the robot arm 44 system, and the mark and the end of the robot arm 44 are separated from each other. After the attachment is fixed, it is a fixed distance. With these three pieces of information, the following equation (1.1) can be obtained, and after n iterations, the most consistent relative relationship between the robotic arm 44 and the camera coordinate system can be obtained.

(1.1) in

is the transformation matrix between the Tool coordinate system (TCS) and the Robot Base Coordinate System (RBCS),

is the transformation matrix between the marker coordinate system and the manipulator coordinate system,

The transformation matrix between the marker coordinate system and the camera coordinate system,

is the transformation matrix of the camera coordinate system and the manipulator coordinate system.

Nowadays, there have been many progresses in introducing augmented reality technology into the medical environment. In the operating room, the pre-shot X-ray tomography (Computed Tomography, CT) can be imaged on the affected area in real time using the positioning system. The most difficult part is the real-time imaging. The virtual image overlays the user's visual overlay and is precisely aligned with objects in the real world. In the augmented reality visualization design, the present invention will combine the augmented reality technology to design a visual interface to intuitively display medical information to the user. In the system, a simulation environment will be established first, which contains the medical image model and the trajectory of the robotic arm, and the medical image model will be registered with the real model to obtain the relative relationship between the real and virtual models; the trajectory of the robotic arm is divided into There are two types, one is the preoperative trajectory, which is planned according to the image model, and the other is the current posture and expected trajectory of the robotic arm. This part will communicate with the robotic arm control system in real time. In addition, it also includes a graphical interface for users to adjust the trajectory. The present invention can use Epson BT-300 AR glasses to develop an augmented reality visualization interface and monitoring system.

FIG. 2 is a schematic diagram of a real image captured by the photographing unit 60 . The image 200 is displayed on the screen of the monitoring module 20 and displays the target 50 and the physical scene 250 in which it is located. The monitoring module 20 in FIG. 1 can be replaced by the computer device 20A or the smart glasses 20B in FIG. 2 . For example, the computer device 20A may be a desktop computer or a notebook computer, and the smart glasses 20B may be smart glasses with a display screen. In addition, the object 50 is an entity, that is, it is not a computer-generated image; the physical scene 250 is a real scene (that is, the real scene where the object 50 is located). In FIG. 2 , the shape of the target 50 is drawn as the spinal column of the human body, so that the target 50 can be a patient's spinal column or a spinal model. That is, the subsequent steps of the present invention may be actual clinical operations, or a doctor may perform a simulation test before the operation. For example, the spine model can be completed by 3D printing technology.

Please refer to FIG. 1A and FIG. 3. FIG. 3 illustrates that the calculation module 30 generates a picture 300 with an augmented reality scene and an augmented reality object according to an embodiment of the present invention. The object 50 is used to display the augmented reality scene 350 and the augmented reality object 55 , wherein the augmented reality scene 350 is an augmented reality scene, and the augmented reality object 55 is generated by simulating the target object 50 . The object 50 displayed on the screen 300 is real, and the augmented reality object 55 is virtual. The augmented reality object 55 may be a computer-generated imagery (CGI) that matches the object 50 in appearance and spatial characteristics. The initial position of the augmented reality object 55 displayed on the screen 300 may overlap with the target object 50 (as shown in FIG. 3 ), or may not overlap the target object 50 at the beginning, and the user 10 can then manually correct the augmented reality The position of the object 55 may be automatically corrected by the calculation module 30 . It should be noted that although the augmented reality scene 350 is mainly composed of real scenes, because the augmented reality object 55 appears in the augmented reality scene 350 , the user 10 will simultaneously view the physical scene and the virtual scene on the screen 300 . scene, so it still meets the definition of augmented reality.

Please refer to FIG. 1A and FIG. 4. FIG. 4 illustrates that the calculation module 30 according to another embodiment of the present invention generates a picture 400 with an augmented reality scene and an augmented reality object. The augmented reality scene 450 is composed of The augmented reality scene simulated by the calculation module 30 according to the data is not a real scene, but the augmented reality scene 450 is consistent with the augmented reality scene 350 in appearance and spatial characteristics. Since the entire augmented reality scene 450 is virtual, only the augmented reality object 55 will appear in the augmented reality scene 450 , and the physical target object 50 will not appear. That is, although this embodiment uses augmented reality for description, it can actually include various reality technologies, such as: virtual reality (Virtual Reality, VR), augmented reality, and alternative reality (Substitutional Reality, SR) or Mixed Reality (MR), not limited to this.

Please continue to refer to Figures 1A, 3, and 4. The user 10 can operate the control device 22 to operate the augmented reality scenes 350 and 450 to rotate the viewing angle (for example, the screen 800 in Figure 8 and the screen 900 in Figure 9), The rotation angle can be 360 degrees, so the user 10 can grasp the information of each viewing angle of the target object 50 by viewing the augmented reality objects 55 in the screens 300 and 400 without changing his position, including the straight line path Features that are not visible from the current viewing angle that are blocked on the top (such as switching to other viewing angles to see details, or the internal structure of the target 50). For example, the augmented reality object 55 may be the patient itself or some part of the body (which may be the spine or other parts), and the augmented reality scene 350, 450 may be the patient's surroundings (eg, the patient is lying on it). hospital bed or the entire indoor environment). In another example, the augmented reality scene 450 may be a patient's internal environment, the augmented reality object 55 may be a patient's internal organs, the surgical robot navigation module 40 may be a minimally invasive surgical device, and the camera unit 60 may be Is a miniature camera or other camera that captures digital images.

The calculation module 30 can display the three-dimensional path indicator 56 in the augmented reality scene 450 (or the augmented reality scene 350 ), as shown in FIG. 5 . FIG. 5 is a schematic diagram of displaying a three-dimensional path indicator in an augmented reality scene. The three-dimensional path indicator 56 is displayed in the augmented reality scene 450 and can be moved in the augmented reality scene 450 under the control of the user 10 . The three-dimensional path indicator 56 is a virtual augmented reality pattern, and its shape may be the arrow shape shown in FIG. 5 , but the invention is not limited thereto, and the arrow shape may be replaced by other directional patterns. Preferably, the three-dimensional path index 56 can represent the moving direction of the robotic arm 44 or the direction of performing the operation in shape. The user 10 can operate the control device 22 to control the movement of the three-dimensional path index corresponding to the augmented reality object in the augmented reality scene, so as to generate the first motion signal S1, and the processor 42 of the surgical robot navigation module 40 then moves according to the first motion signal S1. An action signal S1 controls the robotic arm 44 , so that the robotic arm 44 moves or operates on the target 50 .

Please refer to FIG. 1A and FIG. 6A. FIG. 6A is a schematic diagram of the present invention for calibrating an augmented reality scene and an augmented reality object. The screen 600 shows that the target 50 is placed on the calibration chart 630 , and the calibration chart 630 has an irregular geometric pattern, which helps the calculation module 30 to create an augmented reality scene more accurately. Specifically, before the calculation module 30 visualizes all the information, the mapping relationship between the virtual and real world coordinate systems is obtained, and a camera is used to identify a plurality of feature points in the geometric pattern of the calibration chart 630 , and Comparing with the augmented reality object 55 in the virtual coordinates, the mapping relationship between the above two world coordinates can be obtained after comparing and calculating a plurality of feature points. In this way, the augmented reality object 55 can be smoothly projected into the augmented reality scenes 350 and 450 shown in FIGS. 3 and 4 .

In detail, the relative relationship between the virtual and real world coordinate systems must be found before all the information can be visualized. The imaging unit 60 is used to identify an augmented reality marker (AR marker) in the environment, and compare it with the object in the virtual coordinates, and through multi-point comparison calculation, the corresponding relationship between the two world coordinates can be obtained. After obtaining the transformed coordinates, the present invention can project the virtual object into the picture of the photographing unit 60 . In this embodiment, the augmented reality sign is illustrated by taking the calibration chart 630 set in the environment as an example. The calibration chart 630 is composed of a variety of geometric lines, and has multiple feature points to make the image recognition effect better. Using the image recognition to analyze the deformation of the logo in the picture and the original logo, the camera coordinate system of the photographing unit 60 (Xc, Yc, The transformation relationship between Zc) and the mark coordinate system (Xm, Ym, Zm) of the calibration chart 630 . The image of the user's field of view is composed of the image captured by the camera unit 60 superimposed on the virtual object. With the relative conversion relationship between the calibration chart 630 and the camera unit 60, the position of the virtual object on the screen field of view can be calculated accordingly ( x, y).

The core of augmented reality technology is to project virtual objects into the real environment for virtual and real integration. Broadly speaking, as long as the information is covered in the real environment, it can be regarded as an augmented reality. However, in the framework of the present invention, the present invention It is not only necessary to superimpose the virtual object, but also to superimpose it accurately at the designated position in the real environment, so as to further carry out trajectory planning.

The calibration chart 630 (also known as augmented reality marker (AR marker)) can be self-made and used as a medium in virtual and real environments. Common tools for implementing augmented reality technology include Vuforia, ARToolkit, and OpenCV. Vuforia performs image recognition based on texture features, and is widely used due to its simple and fast application characteristics. However, it does not support the BT-300 AR glasses used in this embodiment. ARToolkit uses a specific format of logo for image recognition. The black regular quadrilateral thick border is its characteristic. The recognition principle is to detect the deformation of the quadrilateral line to complete, but currently it only supports the MacOS system. OpenCV is famous for its rich and diverse image recognition tools. Using this tool to identify and project augmented reality signs can be done with the most basic graphic recognition. Although OpenCV can be customized according to different cases, it is more complicated. Since this embodiment takes EPSON's AR glasses BT-300 as an example, EPSON Moverio AR SDK can be used to perform image registration of augmented reality signs. Like Vuforia, EPSON Moverio AR SDK's requirements for augmented reality signs are based on texture features, so the more geometric line feature points, the better the recognition effect. Therefore, the present invention designs a calibration chart 630 composed of a plurality of geometric lines as an augmented reality mark. Since the calibration chart 630 has multiple feature points, the recognition effect is better compared with the existing AprilTag.

In order to realize the superimposition of the object projection of the virtual system in the actual environment, the present invention realizes the projection design based on the Moverio SDK tool, which includes two steps, one is the identification of the calibration chart 630 and the real-time projection. In the first part of image recognition, the feature points of the picture of the calibration chart 630 are captured and recorded in the computing system, so that the camera unit 60 on the smart glasses BT-300 can identify the calibration chart 630, and further calculate its position on the world coordinate system through its position in the picture. The second part is real-time projection, the purpose is to integrate the virtual image with the real image, so as to correct the graphics card 630 to connect the real and virtual environments. Since the origin of the virtual coordinate system is located on the calibration chart 630, after knowing the position of the entity mark in the world coordinate system, the positions of other virtual objects in the virtual coordinate system in the actual coordinate system can be inferred, and the projection can be calculated in reverse at the same time. The position of the object on the glasses screen, and the ability to successfully project the object into the field of view. During the projection process, the photographing unit 60 will continuously track and identify the calibration chart 630 , and process the position information through a tracking loader to obtain the coordinate positions of each projected object in the real environment, and finally the virtual objects can be superimposed on the in the user's field of vision. With this, even if the glasses change with the user's head movement, the virtual objects can be projected instantly and correctly.

。第6B圖顯示了相關座標系，其X、Y、Z軸（如第6B圖所示之座標軸）可在顯示器上分別用不同顏色表示。機械臂座標系RBCS位於機械臂44第一軸之基座上。第六軸末端效應器（end effector）末端的工具中心點（Tool center Point, TCP）是工具座標系（Tool coordinate system, TCS）的原點。世界座標系（World Coordinate System, WCS）的原點定義在工作空間上，該工作空間的長度和寬度分別為m和n。虛擬座標系VCS是代表AR可視化設計虛擬環境中的座標系，其中包含定義的校正圖卡630。虛擬座標系VCS的原點是基於校正圖卡630，因此實際上標記的位置可以獲取虛擬座標系VCS的相應原點。 For the coordinate transformation and correction of the augmented reality manipulation of the robotic arm 44, please refer to Figure 6B. The purpose of Coordinate Registration is to obtain the robotic arm coordinates of the Virtual Coordinate System (VCS) VCS and the robotic arm 44. The transformation matrix between Robot Base Coordinate System (RBCS)

. Figure 6B shows the relevant coordinate system, whose X, Y, Z axes (as shown in Figure 6B) can be represented by different colors on the display, respectively. The robotic arm coordinate system RBCS is located on the base of the first axis of the robotic arm 44 . The Tool center Point (TCP) at the end of the sixth axis end effector is the origin of the Tool coordinate system (TCS). The origin of the World Coordinate System (WCS) is defined on the workspace, and the length and width of the workspace are m and n, respectively. The virtual coordinate system VCS is a coordinate system in the virtual environment representing the AR visual design, and includes a defined calibration chart 630 therein. The origin of the virtual coordinate system VCS is based on the calibration chart 630, so the actual marked position can obtain the corresponding origin of the virtual coordinate system VCS.

）。虛擬座標系VCS的原點是透過校正圖卡630定義，將校正圖卡630置放於機械臂44手術檯面之工作區間內，並定義該工作區間為世界座標系WCS，即可獲得世界座標系WCS與虛擬座標系VCS之轉換矩陣

。因此，為獲得機械臂座標系RBCS與虛擬座標系VCS的轉換，本發明需要找到世界座標系WCS與機械臂座標系RBCS之間的轉換矩陣

，並依式（2.1）獲得。

(2.1） In practical applications, in order to let the trajectory of the virtual coordinate system VCS of the AR visualization system be executed by the manipulator 44, it is necessary to find the transformation matrix of the manipulator coordinate system RBCS and the virtual coordinate system VCS (

). The origin of the virtual coordinate system VCS is defined by the calibration chart 630. Place the calibration chart 630 in the working area of the operating table of the robotic arm 44, and define the working area as the world coordinate system WCS, then the world coordinate system can be obtained. Conversion matrix between WCS and virtual coordinate system VCS

. Therefore, in order to obtain the conversion between the robot arm coordinate system RBCS and the virtual coordinate system VCS, the present invention needs to find the transformation matrix between the world coordinate system WCS and the robot arm coordinate system RBCS

, and obtained according to formula (2.1).

(2.1)

。由於在先前的投影工作中已經找到了虛擬系統到世界座標系的轉換，所以接下來要找到的就是機械臂座標系RBCS與世界座標系WCS的轉換。如第6C圖所示，首先本發明定義一矩形工作區間在世界座標系中，就是機械臂44作動的範圍，而本發明在機械臂44工作平台上的四邊形作為這個區間，其長寬分別為m和n（m、n為正整數），並定義左上角A點為世界座標系原點。利用機械臂44校點的方式，本發明可以獲取機械臂座標系上A、B、C及D點四點之點位置集。而此四邊形為校正圖卡630的四點，因此虛擬座標系上的A、B、C、D四點之點位置集，就是校正圖卡630四個角落於虛擬座標系上之座標點。利用這兩組點位置集，可以計算機械臂座標系與虛擬座標系之三維空間轉換關係。 In order to allow the trajectory adjusted on the UI interface to be executed by the manipulator 44, the present invention needs to find the transformation matrix between the manipulator coordinate system RBCS and the virtual coordinate system VCS

. Since the conversion of the virtual system to the world coordinate system has been found in the previous projection work, the next thing to find is the conversion of the robot arm coordinate system RBCS and the world coordinate system WCS. As shown in Fig. 6C, first of all, the present invention defines a rectangular working area in the world coordinate system, which is the operating range of the robotic arm 44, and the quadrilateral of the present invention on the working platform of the robotic arm 44 is used as this area, and its length and width are respectively m and n (m, n are positive integers), and define point A in the upper left corner as the origin of the world coordinate system. By means of the point calibration method of the manipulator 44, the present invention can obtain the point position set of the four points A, B, C and D on the coordinate system of the manipulator. The quadrilateral is the four points of the calibration chart 630 , so the point position set of the four points A, B, C, and D on the virtual coordinate system is the coordinate points of the four corners of the calibration chart 630 on the virtual coordinate system. Using these two sets of point position sets, the three-dimensional space transformation relationship between the coordinate system of the robot arm and the virtual coordinate system can be calculated.

（2.2）   The method of the present invention is to find a transformation matrix so that the origin of the virtual coordinate system VCS can be coincident with the robot arm coordinate system RBCS and have the same orientation, so that the two coordinate systems have six degrees of freedom transformation. Since the Z-axis direction of the virtual coordinate system is positioned in the direction of the paper surface, and the calibration chart 630 is placed on the platform of the robot arm 44, the Z-axis directions of the virtual coordinate system VCS and the robot arm coordinate system RBCS are consistent. Therefore, the next steps are to find the relative displacement and rotate around the Z axis so that the X axis is aligned with the Y axis. The transformation relationship between the robot arm coordinate system RBCS and the virtual coordinate system VCS can be divided into position coincidence and direction coincidence, as shown in the following formula (2.2).

(2.2)

）: 先找到兩座標系原點在位置上的位移，座標系之原點即為A點，機械臂座標系RBCS之A點座標為式（2.3），虛擬座標系VCS的A點座標

為式（2.4），因此可得位置重合矩陣

（式（2.5）），其中

係為機械臂座標系的A點座標。

,

,

  （2.3）

  （2.4）

（2.5） position coincidence (

): First find the displacement of the origin of the two coordinate systems, the origin of the coordinate system is point A, the coordinate of point A of the robotic arm coordinate system RBCS is formula (2.3), and the coordinate of point A of the virtual coordinate system VCS

is formula (2.4), so the position coincidence matrix can be obtained

(Equation (2.5)), where

is the coordinate of point A in the coordinate system of the manipulator.

,

,

(2.3)

(2.4)

(2.5)

）: 由於機械臂座標系RBCS與虛擬座標系VCS之Z軸方向相同，為了要讓兩座標系的方向一致，以Z軸為旋轉軸，旋轉

度，要讓剩餘的兩軸重合，同時B、C、 D三點也將重合，得方向重合矩陣（

），式（2.6）。

（2.6） direction coincidence (

): Since the direction of the Z axis of the robot arm coordinate system RBCS and the virtual coordinate system VCS is the same, in order to make the directions of the two coordinate systems consistent, take the Z axis as the rotation axis, rotate

degree, to make the remaining two axes coincide, and the three points B, C, and D will also coincide, and the direction coincidence matrix (

), formula (2.6).

(2.6)

度，使得兩矩形重合。其中，虛擬座標系VCS之B點經旋轉後會與機械臂座標系RBCS之B’點重合，得式（2.7）。

（2.7） 而虛擬座標系之C點經旋轉後會與機械臂座標系RBCS之C’點重合，得式（2.8）。透過式（2.7） 和（2.8）可以求得

。

（2.8） In Figure 6D, the coordinates of the robotic arm coordinate system RBCS are shifted so that the origin of the robotic arm coordinate system RBCS is aligned with the origin of the virtual coordinate system VCS. In this figure, since the direction of the Z axis is the same, only x and y are considered. coordinate. When the origins of the two coordinate systems have been aligned, then as shown in Figure 6E, the virtual coordinate system VCS rotates around the Z axis

degrees so that the two rectangles overlap. Among them, the B point of the virtual coordinate system VCS will coincide with the B' point of the robotic arm coordinate system RBCS after rotation, and the formula (2.7) is obtained.

(2.7) The point C of the virtual coordinate system will coincide with the point C' of the coordinate system RBCS of the robot arm after being rotated, and Equation (2.8) is obtained. Through equations (2.7) and (2.8), it can be obtained

.

(2.8)

Please refer to FIG. 1A and FIG. 7. FIG. 7 is a schematic diagram of the monitoring module 20 of the present invention simultaneously displaying a three-dimensional path index and an adjustment interface corresponding to the three-dimensional path index. As shown in FIG. 7 , in addition to the augmented reality scene 650 and the augmented reality object 55 , the screen 700 further displays a parameter adjustment area 710 that includes the corresponding 3D path index 56 . The parameter adjustment area 710 includes a plurality of spatial parameter scrolls 701 - 706 and an execution key 707 . The user 10 can easily pull the spatial parameter scrolls 701 - 706 in the parameter adjustment area 710 to change the spatial parameters of the 3D path indicator 56 . When the user 10 adjusts any one of the spatial parameter scrolls 701 to 706 , the 3D path indicator 56 will rotate correspondingly, so that the user 10 can confirm in real time whether the current position and angle of the 3D path indicator 56 meet the requirements. After the user 10 has adjusted the parameters (that is, it has been confirmed that the current position and angle of the three-dimensional path index 56 are correct), the user 10 can further press the execution key 707 to send the control signal C1 to control the robot arm 44 to execute A related action, such as implanting a spike in the spine according to the orientation of the three-dimensional path index 56 . The above-mentioned spatial parameters may include six degrees of freedom parameters of the three-dimensional path index 56 . In detail, the six-degree-of-freedom parameters of the three-dimensional path index 56 are the spatial parameter scrolls 701 to 706 : the X-axis coordinates, the Y-axis coordinates, the Z-axis coordinates, the roll angle, the pitch angle, and the yaw angle of the three-dimensional path index 56 . They are respectively used to control the three-dimensional path index 56 to perform the following actions: move back and forth along the X axis, move left and right along the Y axis, move up and down along the Z axis, rotate around the X axis, rotate back and forth around the Y axis, and rotate left and right around the Z axis. Approaching or moving away from the target 50 , resecting at least a portion of the target 50 , or implanting an implant into the target 50 according to the three-dimensional path index 56 . For example, the spatial parameter reels 701 - 703 may respectively correspond to the three-axis (X, Y, Z axis) displacements of the control robot 44 , and the spatial parameter reels 704 - 706 may respectively correspond to the control of the robot 44 to rotate three times. The angle by which the axis is rotated. The three-dimensional path indicator 56 can be continuously displayed in the augmented reality scene 650 as a predetermined trajectory, so that the user 10 can visually judge whether the actual action of the robotic arm 44 conforms to the predetermined trajectory, or the calculation module 30 can automatically judge. In addition, the user 10 can send an interruption signal D1 to the surgical robot navigation module 40 through the control device 22 to disable the surgical robot navigation module 40 from executing the first action signal S1. For example, if the user 10 has sent a command to the surgical robot navigation module 40, pressing the execution key 707 again can interrupt the execution of the command by the surgical robot navigation module 40, or the parameter adjustment area 710 can add another key to disable the execution of the command. The purpose of the surgical robot navigation module 40.

Figures 8 to 10 are schematic diagrams of different perspectives of the augmented reality scene of the present invention, which are used to illustrate the application of the present invention to spinal fusion surgery, wherein the spine 755 is an augmented reality object, using A spine part of a patient or a spine model is presented, and the augmented reality scene 750 is an augmented reality scene. In Figure 8, the screen 800 presents a back view of the spine 755. At this time, the surgeon plans to implant the nail at the expected implantation point of the spine 755, but the current perspective cannot see the specific position, resulting in a three-dimensional path indicator 756. It cannot be positioned smoothly, so it is necessary to rotate the angle of the augmented reality scene 750 .

In Figure 9, the augmented reality scene 750 has been rotated so that the frame 900 presents a frontal view of the spine 755, and the surgeon can already see the projected implantation point 759 of the spine 755, which helps determine the 3D path metrics 756's motion trajectory and surgical actions. For example, when the monitoring module 20 is a computer device, the operation of rotating the augmented reality scene 750 can be realized through the scroll wheel on the mouse; when the monitoring module 20 is a smart glasses, the operation of rotating the augmented reality scene 750 can be realized by A scroll wheel is arranged on the smart glasses, or a scroll wheel is arranged on a button handle connected to the smart glasses, so that the operation of rotating the augmented reality scene 750 can be realized through the scroll wheel. Although the frame 900 has presented decisive reference information, the surgeon may still need to refer to other perspectives to confirm the exact location of the 3D path indicator 756 in space, and further rotate the augmented reality scene 750 to the frame in Fig. 10 A side view of spine 755 as presented by 1000. Although the above only describes the perspective as in the frames 800 , 900 , 1000 , the present invention can actually rotate the augmented reality scene 750 to various angles, even including the bottom of the spine 755 .

The present invention combines the 3D image and data with the actual surgical field of view in the augmented reality smart glasses through the augmented reality technology, so as to solve the dilemma of hand-eye incongruity that the surgeon had to repeatedly confirm in the two fields of view in the past. In addition, in spinal fusion surgery, it is necessary to make holes in hard bones and implant bone nails, and the surface of the bones is smooth. Artificial implantation is laborious and prone to slippage. Implantation through surgical robotic units such as robotic arms or robots can accurately perform predetermined surgical actions and improve the stability of surgery. In addition, the surgical assistance system 100 of the present invention can achieve the effect of real-time monitoring. Once the action of the robotic arm 44 does not proceed as expected, or the user 10 changes the original plan, the control device 22 can be used to stop/pause (pause)/resume the robotic arm 44 to achieve the effect of real-time monitoring.

Please refer to FIG. 11, which is a schematic diagram of the monitoring module 20 of the present invention simultaneously displaying two three-dimensional path indicators. As shown in FIG. 11, after the motion trajectory is generated (that is, the user has sent the three-dimensional path index 756 through the execution key 707), the surgical robot navigation module 40 has not yet executed the first motion signal S1 or the surgical robot unit has been interrupted When the first motion signal S1 is executed, the calculation module 30 further generates a candidate 3D path indicator 757 and displays the 3D path indicator 757 on the screen 1000 . At this time, in the screen 1100 , the candidate 3D path indicator 757 and the 3D path indicator 756 are simultaneously displayed in the augmented reality scene 750 , and the user 10 operates the control device 22 to control the candidate 3D path indicator 757 in the augmented reality scene 750 . Move the path to generate a candidate motion trajectory. The signal conversion unit 42 receives the candidate motion trajectory and converts the predetermined candidate motion trajectory into a second motion signal S2 (refer to FIG. 1A ). The surgical robot unit executes the second motion signal S2 to move the target along the candidate motion trajectory. 50 (eg patient spine area) to perform surgical action. For example, there are three virtual objects in the above-mentioned visual image, which are the patient's spine 755 , the predicted trajectory of the robotic arm (3D path indicator 756 ), and the trajectory adjusted by the doctor in real time (3D path indicator 757 ). Medical bone models are converted from CT images of real lumbar vertebrae to plan and simulate surgical movements. In the virtual environment coordinate system, the upper left corner of the calibration chart 630 is used as the origin of the coordinate system. Each track and bone model have relative points in this coordinate system.

Please refer to FIG. 12. FIG. 12 is a flowchart of a surgical assistance method according to an embodiment of the present invention. Note that these steps do not have to be performed exactly in the order of execution shown in FIG. 12 if substantially the same result can be obtained. The method shown in FIG. 12 can be used by the surgical assistance system 100 shown in FIG. 1A and can be briefly summarized as follows: Step 1202: start; Step 1204: Display the augmented reality object and the 3D path indicator in the augmented reality scene; Step 1206: Control the movement path of the 3D path indicator in the augmented reality scene to generate a motion trajectory; Step 1208: The signal conversion unit converts the motion trajectory into motion signals; Step 1210: The surgical robot unit receives and executes the action signal, so as to execute the surgical action on the physical target along the motion track; Step 1212: End.

Since those skilled in the art should be able to easily understand the details of each step in Fig. 12 after reading the above paragraphs, further descriptions are omitted here for brevity.

To sum up, the present invention uses augmented reality technology to solve the bottleneck encountered by surgeons in performing surgical operations. The surgeon can rotate the field of view of the augmented reality scene to grasp all the perspectives of the target object, so there is no need to move himself. The relative positional relationship between the target object and the surrounding environment can be grasped by the position of the robot arm; in addition, the robot arm can execute the pre-input robot arm movement trajectory or the robot arm movement trajectory adjusted by the surgeon on the spot, which can avoid the mistakes of manual operation. The present invention also allows the surgeon to interrupt the current task of the robotic arm and adjust the motion trajectory of the robotic arm in real time, so that it can respond to various emergencies in actual operations.

、

、

、

、

、

、

:轉換矩陣 A、B、C、D、A’、B’、C’、D’:座標點 T ₁:位置重合矩陣 T ₂:方向重合矩陣 10: User 20: Monitoring module 20A: Computer device 20B: Smart glasses 22: Control device 24: Display device 30: Algorithm module 40: Surgical robot navigation module 42: Signal conversion unit 44: Robot arm 50: Target Object 55: Augmented reality Object 56, 756: 3D path index 60: Photography unit 100: Surgical assistance system 250: Physical scene 200, 300, 400, 500, 600, 700, 800, 900, 1000, 1100: Screen 350 , 450, 650, 750: Augmented reality scene 630: Correction chart 701～706: Spatial parameter scroll 707: Execute button 710: Parameter adjustment area 755: Spine 757: Candidate 3D path index 759: Estimated implantation point 1202～ 1212: Step A1: Input operation A2: Surgical action C1, C2: Control signal D1: Interruption signal S1: First action signal S2: Second action signal V1: Image data VCS: Virtual coordinate system WCS: World coordinate system RBCS: Mechanical Arm coordinate system

,

,

,

,

,

,

: Conversion matrix A, B, C, D, A', B', C', D': Coordinate point T ₁ : Position coincidence matrix T ₂ : Direction coincidence matrix

FIG. 1A is a schematic diagram of a surgical assistance system according to an embodiment of the present invention. FIG. 1B is a schematic diagram of hand-eye correction and image navigation according to the present invention. FIG. 2 is a schematic diagram of a real picture captured by the photographing unit. FIG. 3 illustrates the generation of a picture with an augmented reality scene and an augmented reality object by an algorithm module according to an embodiment of the present invention. FIG. 4 illustrates the generation of a picture with an augmented reality scene and an augmented reality object by an algorithm module according to another embodiment of the present invention. FIG. 5 is a schematic diagram of displaying a three-dimensional path indicator in an augmented reality scene. FIG. 6A is a schematic diagram of correcting an augmented reality scene and an augmented reality object according to the present invention. FIG. 6B is a schematic diagram of coordinate transformation and calibration of the augmented reality manipulation of the robotic arm according to the present invention. Fig. 6C is a schematic diagram of defining a rectangular working area in the world coordinate system according to the present invention. Fig. 6D and Fig. 6E are schematic diagrams of aligning the coordinate system of the manipulator with the coordinate system of the manipulator. FIG. 7 is a schematic diagram of the monitoring module of the present invention simultaneously displaying three-dimensional path indicators and an adjustment interface corresponding to the three-dimensional path indicators. Figures 8 to 10 are schematic diagrams of different perspectives of the augmented reality scene of the present invention. FIG. 11 is a schematic diagram of the monitoring module of the present invention simultaneously displaying two three-dimensional path indicators. FIG. 12 is a flowchart of a surgical assistance method according to an embodiment of the present invention.

20: Monitoring module

20A: Computer equipment

20B: Smart Glasses

55: Augmented Reality Objects

56: 3D Path Indicator

400: Screen

450: Augmented Reality Scenarios

Claims (20)

A surgical assistance system comprising: an arithmetic module for generating an augmented reality object in an augmented reality scene according to an image of a physical object, the augmented reality object system simulates the physical object, and is located in the augmented reality displaying the augmented reality object and a three-dimensional path indicator in the scene; A monitoring module includes a display device and a control device, wherein the display device is used to display the augmented reality scene including the augmented reality object and the three-dimensional path indicator; the control device is operated to control the three-dimensional The path indicator is a moving path in the augmented reality scene, so as to generate a motion trajectory for the surgical robot; a signal converting unit for converting the motion trajectory into a first motion signal, and A surgical robot unit receives and executes the first action signal, so as to perform a surgical action on the physical target along the movement track. The surgical assistance system according to claim 1, wherein the computing module generates a user interface, the display device simultaneously displays the user interface and the augmented reality scene, and the user interface is controlled by the control device, And it is used to adjust the spatial parameters of the three-dimensional path index to determine the movement trajectory. The surgical assistance system of claim 2, wherein the spatial parameters include X-axis coordinates, Y-axis coordinates, Z-axis coordinates, rotation angle around X-axis, rotation angle around Y-axis, and rotation around Z-axis of the three-dimensional path index Multiple adjustment fields of the angle, respectively used to control the 3D path index to move forward and backward along the X axis, left and right along the Y axis, up and down along the Z axis, rotate around the X axis, rotate forward and backward around the Y axis, and rotate left and right around the Z axis . The surgical assistance system of claim 2, wherein the user interface is operated to send an interrupt signal to the surgical robotic unit to disable the surgical robotic unit from executing the first motion signal. The surgical assistance system as claimed in claim 2, wherein after the motion trajectory is generated, in the case that the surgical robot unit has not yet executed the first action signal or the surgical robot unit has been interrupted to execute the first action signal, the calculation module In addition, a candidate 3D path indicator is generated, and the candidate 3D path indicator and the 3D path indicator are simultaneously displayed in the augmented reality scene; the control device is further operated to control the candidate 3D path indicator in the augmented reality scene. moving a path to generate a candidate motion trajectory; the signal conversion unit receives the candidate motion trajectory and converts the motion trajectory into a second motion signal; and the surgical robot unit executes the second motion signal to follow the candidate motion The motion trajectory performs the surgical action. The surgical assistance system of claim 5, wherein the candidate motion trajectory is adjusted in real time through the user interface, and the adjusted candidate motion trajectory is confirmed by the user interface to generate the second motion signal. The surgical assistance system of claim 1, wherein the surgical robot unit approaches or moves away from the physical target according to the first motion signal, resects at least a part of the physical target, or implants an implant into the physical target thing. The surgical assistance method of claim 1, wherein the augmented reality scene is simulated but does not include: a real scene where the physical object is located. The surgical assistance system of claim 1, wherein the augmented reality scene includes a real scene where the physical object is located. The surgical assistance system of claim 1, wherein the control device is further operated to change the viewing angle of the augmented reality scene, and when the viewing angle of the augmented reality scene is changed, the viewing angle of the augmented reality object is the same as the viewing angle of the augmented reality object. The viewing angle of the three-dimensional path index is correspondingly changed by the calculation module. The surgical assistance system of claim 1, wherein the display device is a head-mounted display, and the control device is a portable controller coupled to the head-mounted display. The surgical assistance system of claim 1, wherein the display device is a liquid crystal display, and the control device is an input device coupled to the liquid crystal display. A surgical aid method comprising: An arithmetic module generates an augmented reality scene and an augmented reality object according to a physical object, and further displays a three-dimensional path indicator in the augmented reality scene, and the augmented reality object system simulates the entity a target object, and the augmented reality object system is displayed in the augmented reality scene; Displaying the augmented reality scene including the augmented reality object and the 3D path indicator on a display device; operating a control device to control the movement path of the 3D path indicator in the augmented reality scene to generate a movement trajectory; a signal converting unit converts the motion trajectory into a first motion signal, and A surgical robot unit receives and executes the first action signal to perform a surgical action on the physical target along the movement track. The surgical assistance method according to claim 13, wherein the calculation module generates a user interface, the display device simultaneously displays the user interface and the augmented reality scene, and the user interface is controlled by the control device, And it is used to adjust the spatial parameters of the three-dimensional path index to determine the movement trajectory. The surgical assistance method of claim 14, wherein the spatial parameters include X-axis coordinates, Y-axis coordinates, Z-axis coordinates, rotation angle around X axis, rotation angle around Y axis, and rotation around Z axis of the three-dimensional path index Multiple adjustment fields of the angle, respectively used to control the 3D path index to move forward and backward along the X axis, left and right along the Y axis, up and down along the Z axis, rotate around the X axis, rotate forward and backward around the Y axis, and rotate left and right around the Z axis . The surgical assistance method as described in claim 13, further comprising: Operating the user interface sends an interruption signal to the surgical robot unit to disable the surgical robot unit from executing the first motion signal. The surgical assistance method as claimed in claim 13, wherein after the motion trajectory is generated, the surgical assistance method is provided in the case that the surgical robot unit has not yet executed the first motion signal or the surgical robot unit has been interrupted to execute the first motion signal Also perform the following steps: The calculation module further generates a candidate 3D path indicator, and the candidate 3D path indicator and the 3D path indicator are simultaneously displayed in the augmented reality scene; operating the control device to control the moving path of the candidate three-dimensional path indicator in the augmented reality scene to generate a candidate motion trajectory; the signal conversion unit receives the candidate motion trajectory and converts the motion trajectory into a second motion signal; and The surgical robot unit receives and executes the second action signal to perform a surgical action along the candidate motion trajectory. The surgical assistance method according to claim 17, wherein the candidate motion trajectory is adjusted in real time through the user interface, and the adjusted candidate motion trajectory is confirmed by the user interface to generate the second motion signal. The surgical assistance method as claimed in claim 13, wherein the surgical robot unit performs: approaching, moving away, removing or implanting the physical target according to the first motion signal. The surgical assistance method as described in claim 13, further comprising: Operate the control device to change the viewing angle of the augmented reality scene. When the viewing angle of the augmented reality scene changes, the viewing angle of the augmented reality object and the viewing angle of the three-dimensional path index are calculated by the arithmetic module to correspond to change.

Images