KR20120098342A - Master robot, surgical robot system using thereof, control method thereof, and recording medium thereof - Google Patents

Abstract

PURPOSE: A master robot, surgery robot system, and control method thereof are provided to improve surgery effects and surgery accuracy and to reduce surgery time by increasing the concentration of an operator. CONSTITUTION: A master robot(100) includes an image input unit(130), a screen display unit(140), an arm control unit(150), a control signal creation unit(160), a virtual control board realization unit(170), and a control unit(180). A slave robot(200) includes a robot arm(210) and a laparoscope(220). The slave robot includes a bio information measurement unit for measuring bio-information for patients. A surgery robot system executes predetermined operations using the robot arm in a surgery. The virtual control board is overlapped with a real image on a display member. The virtual control board is operated by controlling the robot arm displayed on the display member. [Reference numerals] (130) Image input unit; (140) Screen display unit; (150) Arm control unit; (160) Control signal creation unit; (170) Virtual control board realization unit; (180) Control unit; (210) Robot arm; (220) Laparoscope

Description

Master robot, surgical robot system including the same, and control method thereof, and recording medium recording the same

The present invention relates to a master robot, a surgical robot system including the same, and a control method thereof, and a recording medium recording the same, in detail, in a surgical robot system including a master robot and a slave robot, a main operator alone without the assistance of an assistant operator. Master robot, which can perform various operations of the surgical robot system while watching the monitor screen while the operator gripping the operation lever of the master robot, so that all the surgical operations can be performed as a surgical robot system including the same And a control method thereof and a recording medium recording the same.

Surgical robot refers to a robot having a function that can replace a surgical operation performed by a surgeon. Such a surgical robot has the advantage of being capable of accurate and precise operation and remote surgery compared to humans.

Surgical robots currently being developed worldwide include bone surgery robots, laparoscopic surgery robots, and stereotactic surgery robots. The laparoscopic surgical robot is a robot that performs minimally invasive surgery using a laparoscope and a small surgical tool.

Laparoscopic surgery is an advanced surgical technique that involves surgery after inserting a laparoscope, which is an endoscope for looking into the belly with a hole about 1 cm in the navel area, and is expected to be developed in the future. A recent laparoscope is equipped with a computer chip to provide a sharper, more magnified image than is seen with the naked eye, and the laparoscopic surgical instruments specially designed by looking at the screen through the monitor have developed enough to perform any operation.

Furthermore, laparoscopic surgery has the same range of surgery as laparotomy, but has fewer complications than laparotomy, and can start treatment much earlier than the procedure, and has the ability to maintain the stamina or immune function of the patient. have. As a result, laparoscopic surgery is increasingly recognized as a standard surgery for treating colorectal cancer in the US and Europe.

On the other hand, the surgical robot system is generally composed of a master robot and a slave robot. When the operator manipulates a steering lever (for example, a handle) provided in the master robot, a surgical tool coupled to the robot arm of the slave robot or held by the robot arm is operated to perform surgery.

Operation of the endoscope in the above-described laparoscopic surgery is a process that plays a major role in the operation, and in the conventional laparoscopic surgery, such as not using a surgical robot system, a separate assistant to perform the operation only for the endoscope. In addition, the surgical robot system which automates the operation of the endoscope or the surgical robot system including both the endoscope and the surgical tool operation does not need such an assistant operator, and the operator can directly operate the endoscope during the operation. It is a type of surgery in which the main operator performs all the surgical tasks alone.

The present invention allows the operator to operate the master robot in various operations of the surgical robot system so that the main surgeon can perform a single operation alone without the help of an assistant operator in the surgical robot system having a master robot and a slave robot. An object of the present invention is to provide a master robot, a surgical robot system including the same, a control method thereof, and a recording medium recording the same, which can be performed while watching a monitor screen while holding a lever.

The present invention provides a master robot for controlling a slave robot including a surgical endoscope and a robot arm, comprising: a display member configured to display an image photographed through the surgical endoscope; A virtual operation panel display unit displaying a virtual operation panel on the display member; A motion matching unit determining whether the robot arm displayed on the display member performs an operation of manipulating the virtual operation panel; And an operation implementation signal generator configured to generate an operation implementation signal for performing an actual operation corresponding to the operation operation of the virtual operation panel.

In the present invention, the virtual operation panel display unit may display the virtual operation panel by overlapping the image captured by the surgical endoscope displayed on the display member.

In the present invention, the virtual operation panel display unit may display the virtual operation panel on the display member when a predetermined virtual operation panel activation signal is received.

In the present invention, the virtual operation panel display unit controls the operation of the surgical endoscope or the robot arm, selection of the robot arm to be manipulated by using the master robot among a plurality of the robot arm, output control of the electric cauterizer, At least one of setting the operation scale relationship between the master robot and the slave robot, pausing and resuming the slave robot, manipulating anesthesia, turning on / off the CO2 injector and adjusting gas pressure A function operation unit for performing the operation may be displayed on the virtual operation panel.

In the present invention, the virtual operation panel display unit may display an information display unit for displaying the biometric information of the subject and / or the medical image information of the subject on the virtual operation panel.

Here, at least some of the biometric information and the medical image information may be fixedly displayed at a predetermined position on the display member.

In the present invention, when the robot arm performs an operation of operating the virtual operation panel displayed on the display member, the reaction force providing unit for providing a predetermined reaction force corresponding to the operation to the operation lever of the master robot It may include.

In the present invention, an operation of controlling the ratio of the operation distance of the robot arm of the slave robot to the operation distance of the operation lever of the master robot in the state in which the virtual operation panel is enlarged / reduced and the virtual operation panel is enlarged. It may further include a scaling controller.

In the present invention, the display member may include a stereoscopic display device.

According to another aspect of the present invention, a surgical endoscope for imaging the surgical site; And a robot arm for driving with multiple degrees of freedom; and a display member for displaying an image photographed through the surgical endoscope; And displaying a virtual operation panel on the display member and operating a signal corresponding to the operation operation by operating the virtual operation panel of the robot arm displayed on the display member. It provides a surgical robot system including a master robot comprising; a virtual operation panel implementation unit for generating.

In the present invention, the virtual operation panel implementation unit, the virtual operation panel display unit for displaying the virtual operation panel on the display member; A motion matching unit determining whether the robot arm displayed on the display member performs an operation of manipulating the virtual operation panel; And an operation implementation signal generator configured to generate an operation implementation signal for performing an actual operation corresponding to the operation operation of the virtual operation panel.

In the present invention, when the robot arm performs a predetermined predetermined operation, a virtual operation panel activation signal for displaying a virtual operation panel on the display member may be generated.

In the present invention, the robot arm may further include a reaction force providing unit for providing a predetermined reaction force to the operation lever of the master robot when performing the operation of operating the virtual operation panel displayed on the display member.

In the present invention, an operation of controlling the ratio of the operation distance of the robot arm of the slave robot to the operation distance of the operation lever of the master robot in the state in which the virtual operation panel is enlarged / reduced and the virtual operation panel is enlarged. It may further include a scaling controller.

In the present invention, the virtual operation panel display unit controls the operation of the surgical endoscope or the robot arm, selection of the robot arm to be manipulated by using the master robot among a plurality of the robot arm, output control of the electric cauterizer, At least one of setting the operation scale relationship between the master robot and the slave robot, pausing and resuming the slave robot, manipulating anesthesia, turning on / off the CO2 injector and adjusting gas pressure A function operation unit for performing the operation may be displayed on the virtual operation panel.

In the present invention, the virtual operation panel may include an information display unit for displaying the biometric information of the subject and / or the medical image information of the subject.

Here, the medical image information is reconstructed three-dimensional based on X-ray image, computed tomography (CT) image, magnetic resonance imaging (MRI) image and the computed tomography (CT) image data. It may include one or more of the image data or the digitized model.

The slave robot further includes a biometric information measuring unit including at least one of a body temperature measuring module, a pulse measuring module, a respiratory measuring module, a blood pressure measuring module, an electrocardiogram measuring module, and the like, measured by the biometric information measuring unit. Information may be displayed on the information display unit.

Here, at least some of the biometric information and the medical image information may be fixedly displayed at a predetermined position on the display member.

In the present invention, the display member may include a stereoscopic display device.

According to another aspect of the present invention, there is provided a control method of a surgical robot system including a master robot and a slave robot, the method comprising: displaying an image captured by a surgical endoscope of the slave robot on a display member of the master robot; Generating a predetermined virtual operation panel activation signal; Displaying an image captured by the surgical endoscope on the display member of the master robot and the virtual operation panel; Performing operation of manipulating the virtual operation panel by a robot arm of the slave robot displayed on the display member; And actually performing an operation manipulated by the robot arm on the virtual operation panel.

In the present invention, the step of generating the predetermined virtual operation panel activation signal includes: predetermined operation in which the robot arm is pre-set by operation of the master robot while the operator holds the operation lever of the master robot. By performing the operation of, the predetermined virtual operation panel activation signal may be generated.

Here, the predetermined predetermined operation may include pressing or holding the virtual operation panel activation signal generation button formed on the operation lever of the master robot, rotating the robot arm in the air, and rotating the robot arm at a constant speed. The robot arm may include one or more of an operation of moving straight in a predetermined direction, an operation of drawing the predetermined mark in the air by the robot arm, and an operation of pressing or holding the predetermined icon displayed on the display member.

In the present invention, the virtual operation panel may include a functional manipulation unit for controlling the operation of the surgical endoscope or the robot arm.

Here, when the operation operated by the robot arm of the slave on the virtual operation panel is an operation of operating the function operation unit, the step of actually performing the operation operated by the robot arm of the slave on the virtual operation panel, The control operation for the robot arm or the surgical endoscope of the slave robot may be performed.

In the present invention, the virtual operation panel may include an information display unit for displaying the biometric information of the subject and / or the medical image information of the subject.

Here, when the operation operated by the robot arm of the slave on the virtual operation panel is an operation of operating the information display unit, the step of actually performing the operation operated by the robot arm of the slave on the virtual operation panel, The control operation for the information display unit shown in the display member may be performed.

Here, when the operation operated by the robot arm of the slave on the virtual operation panel is an operation for moving at least a portion of the information display unit to a specific position on the display member, at least some of the biometric information and the medical image information The display member may be fixedly displayed at a predetermined position on the display member.

In the present invention, the step of overlapping and displaying the image photographed through the surgical endoscope on the display member of the master robot, the virtual operation panel, the display from the surgical site is displayed on the display member the virtual operation panel It may be displayed in a state of injury to the user side of the member to some extent.

In the present invention, when the robot arm of the slave robot displayed on the display member to operate the virtual operation panel, a predetermined reaction force is provided to the operation lever of the master robot; Can be.

According to another aspect of the present invention, a program of instructions that can be executed by a digital processing apparatus for performing the control method of the surgical robot system according to any one of the above is tangibly embodied and can be read by the digital processing apparatus. Provide a recording medium recording the program.

According to the present invention, it is possible to easily operate the entire surgical robot system and information systems such as various biometric information, medical image information within a single system, it is possible to obtain the effect that can be operated alone. Furthermore, the single surgery as in the technique of the present technology can increase the attention of the operator to improve the accuracy of the operation, shorten the operation time, and increase the effectiveness of the surgery, as well as the efficiency of the operation and the medical cost, which are advantages of the single surgery. It can incidentally provide the effect of.

In addition, by applying a haptic interface to the surgical robot system, it is possible to improve the accuracy and intuitiveness of the operator's motion input, and to apply augmented reality interface to perform a more intuitive operation more intuitive.

1 is a plan view showing the overall structure of a surgical robot system according to an embodiment of the present invention.
FIG. 2 is a perspective view illustrating a master robot of the surgical robot system of FIG. 1.
3 is a block diagram schematically showing the configuration of a master robot and a slave robot according to an embodiment of the present invention.
FIG. 4 is a block diagram schematically illustrating a configuration of a virtual operation panel implementing unit provided in the master robot of FIG. 3.
5 is a diagram illustrating a screen display form in which an image captured by a laparoscope is output through a display member.
FIG. 6 is a diagram illustrating a screen display form in which a virtual operation panel is superimposed on the display member of FIG. 5.
7 is a flow chart schematically showing a control method of a surgical robot system according to an embodiment of the present invention.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art may easily implement the present invention. The present invention may be embodied in many different forms and is not limited to the embodiments described herein.

Herein, the present invention is a technical idea that can be used universally for surgeries using surgical endoscopes (for example, laparoscopic, thoracoscopic, arthroscopic, non-rigid, etc.), but in describing the embodiments of the present invention, For convenience, a laparoscope is used as an example.

1 is a plan view showing the overall structure of the surgical robot system according to an embodiment of the present invention, Figure 2 is a perspective view showing a master robot of the surgical robot system of FIG.

1 and 2, the surgical robot system 1 includes a slave robot 200 performing surgery on a patient lying on an operating table, and a master robot 100 for allowing an operator to remotely control the slave robot 200. It includes. The master robot 100 and the slave robot 200 are not necessarily separated into separate devices that are physically independent, but may be integrated into one and be integrally formed.

The master robot 100 includes an operation lever 110 and a display member 120, and the slave robot 200 includes a robot arm 210 and a laparoscope 220.

In detail, the master robot 100 is provided with an operation lever 110 so that the operator can operate by holding each hand in both hands. The manipulation lever 110 may be implemented with two or more handles as illustrated in FIGS. 1 and 2, and an operation signal according to a handle manipulation of an operator may be controlled by a slave robot 200 through a wired or wireless communication network. Is transmitted to the robot arm 210 is controlled. That is, a surgical operation such as a position movement, rotation, and cutting operation of the robot arm 210 may be performed by the operator's handle manipulation.

For example, the operator may manipulate the slave robot arm 210, the laparoscope 220, or the like using a handle lever of a handle. Such an operation lever may have various mechanical configurations according to its operation method, and includes a master handle for manipulating operations such as a slave robot arm 210 and a laparoscope 220, and a master robot for manipulating the functions of the entire system. It may be provided in various forms for operating the robot arm 210 and / or other surgical equipment of the slave robot 200, such as various input tools such as a joystick, a keypad, a trackball, and a touch screen added to the 100. . Here, the manipulation lever 110 is not limited to the shape of the handle, and may be applied without any limitation as long as it can control the operation of the robot arm 210 through a network such as a wired or wireless communication network.

The image captured by the laparoscope 220 is displayed on the display member 120 of the master robot 100 as an image image. In addition, a predetermined virtual operation panel may be displayed on the display member 120 together with the image photographed through the laparoscope 220 or may be displayed independently. The arrangement, configuration, and the like of such a virtual operation panel will be described in detail later with reference to related drawings.

In this case, the display member 120 may be configured of one or more monitors, and information necessary for surgery may be individually displayed on each monitor. 1 and 2 illustrate the case in which the display member 120 includes three monitors, the quantity of monitors may be variously determined according to the type or type of information requiring display.

Meanwhile, the slave robot 200 may include one or more robot arms 210. In general, a robot arm refers to a device having a function similar to that of a human arm and / or wrist and capable of attaching a predetermined tool to a wrist region. In the present specification, the robot arm 210 may be defined as a concept encompassing all the components such as upper and lower chests, wrists, elbows, and surgical instruments coupled to the wrist region. As such, the robot arm 210 of the slave robot 200 may be implemented to be driven with multiple degrees of freedom. The robot arm 210 includes, for example, a surgical instrument inserted into a surgical site of a patient, a rocking drive unit for rotating the surgical instrument in a yaw direction according to a surgical position, and a pitch direction perpendicular to the rotational drive of the rocking drive unit. It comprises a pitch drive unit for rotating the surgical instruments, a transfer drive for moving the surgical instruments in the longitudinal direction, a rotation drive for rotating the surgical instruments, a surgical instrument drive unit installed on the end of the surgical instruments to cut or cut the surgical lesion Can be. However, the configuration of the robot arm 210 is not limited thereto, and it should be understood that this example does not limit the scope of the present invention. Here, the detailed description of the actual control process, such as the robot arm 210 rotates, moves in the corresponding direction by the operator operating the manipulation lever 110 will be omitted.

One or more slave robots 200 may be used to operate the patient, and the laparoscope 220 may be implemented as an independent slave robot 200 so that the surgical site is displayed as an image image through the display member 120. It may be. In addition, as described above, embodiments of the present invention may be used universally in operations in which various surgical endoscopes (eg, thoracoscopic, arthroscopy, parenteral, etc.) other than laparoscopic are used.

3 is a block diagram schematically showing the configuration of a master robot and a slave robot according to an embodiment of the present invention.

Referring to FIG. 3, the master robot 100 includes an image input unit 130, a screen display unit 140, an arm operation unit 150, an operation signal generation unit 160, a virtual operation panel implementation unit 170, and a controller 180. ). The slave robot 200 includes a robot arm 210 and a laparoscope 220. Although not shown in FIG. 3, the slave robot 200 may further include a biometric information measuring unit for measuring and providing biometric information about the patient.

Here, when the surgical robot system 1 according to an embodiment of the present invention performs a predetermined operation using the robot arm 210 during surgery, a virtual operation panel is displayed on the display member 120 by being superimposed with an actual image. When the virtual operation panel is operated by manipulating the robot arm 210 displayed on the display member 120, an actual operation corresponding thereto is performed.

In detail, in an endoscope-based surgical robot system having a master-slave structure, the operator generally holds the two operation levers of the master robot with both hands and controls the movement of the robot arm of the slave robot. However, during such a robot surgery process, input of various system operation commands other than the movement of the robot arm of the slave robot is required. For example, endoscope camera position shift, posture change, field of view zoom in / out, setting the motion scale relationship between master and slave robots, and for systems with three or more slave robot arms, It is necessary to input system operation commands such as selecting and connecting / disconnecting two robot arms to be controlled, suspending the slave robot, and controlling the operation of the electric cauterizer.

In order to input such a system operation command, when an input of a system operation command is conventionally required during an operation, the operator pauses the surgical robot system and moves both hands or one of them to the operation panel outside the master robot. The command was inputted by pressing the button switch. In this case, there is a problem that the attention of the operator is psychologically distracted because it is necessary to temporarily suspend the operation work and physically remove the working hand from the master robot and shift the attention between different controls.

In order to solve this problem, by using a device that performs a function by receiving a voice command from the operator to recognize the command content, or by using a sensor tool attached to the operator's head to recognize the change in posture, that is, the gesture A device has been developed that performs this function. In this case, there is no need to stop the system or take the hands off the operating lever of the master robot, but even in this case, there is still a problem that the attention of the operator is distracted psychologically. In addition, in the case of such a voice input or a head gesture input method, there is a possibility that accurate recognition cannot be performed during command recognition (voice recognition through a computer signal processing operation or pattern recognition of an operation signal). In particular, in the case of the voice input method, the operation is performed by reconfirming the synthesized voice in the system after inputting the voice to prevent an error, and by putting a non-compliance for several seconds between consecutive commands. Difficulties existed in performing endoscopy manipulation.

Furthermore, in the current surgical robot system, only a relatively small number of operation commands are required. However, according to the development trend of the intelligent system, various types of safer and more accurate surgical procedures such as various real-time biometric information and medical image data of the patient are available. Techniques for incorporating the information into the surgical robotic system environment are being developed. Therefore, a plurality of additional input manipulation commands for viewing and using such various information are required. Currently, in some commercialized systems, functions for selecting the patient's biometric information to be shown separately on the operator's display screen are implemented. However, a function for controlling the operator is not implemented. You need a surgical assistant. Therefore, the operator can operate the operation of the surgical robot system as well as the various operation operations of various integrated information systems without the need for additional assistants or assistants during the operation, minimizing distraction, and removing the hand from the master robot. In the absence of this, there is a need for an improved interface for single surgery that allows for quick and intuitive operation.

To this end, in the surgical robot system 1 according to an embodiment of the present invention, when a predetermined operation is performed using the robot arm 210 during surgery, a virtual operation panel is displayed on the display member 120 by being superimposed with an actual image. If the virtual operation panel is operated by manipulating the robot arm 210 displayed on the display member 120, the actual operation corresponding thereto is performed.

Referring to FIG. 3 again, each component of the master robot 100 will be described in detail.

The image input unit 130 receives an image captured by a camera provided in the laparoscope 220 of the slave robot 200 through a wired or wireless communication network.

The screen display unit 140 outputs an image image corresponding to the image received through the image input unit 130 as visual information. In addition, the screen display unit 140 may further output information corresponding to the biometric information of the operator from the slave robot 200. In addition, the screen display unit 140 may further output relevant image data (eg, X-ray image, CT image, MRI image, etc.) of the patient for the surgical site. Here, the screen display unit 140 may be implemented in the form of a display member (see 120 of FIG. 2) or the like, and an image processing process for outputting a received image as an image image through the screen display unit 140 may include a controller ( 180, the virtual operation panel implementer 170 or an image processor (not shown).

Here, the screen display unit 140 (that is, the display member 120 of FIG. 2) may be provided as a stereoscopic display device. In detail, the stereoscopic display device applies stereoscopic technology to add depth information to a two-dimensional image, and uses the depth information to display an image display device that allows the observer to feel three-dimensional liveness and reality. Refers to. Surgical robot system 1 according to an embodiment of the present invention is characterized by providing a more practical virtual environment to the user by having a three-dimensional display device as a screen display, which will be described in more detail later.

The arm manipulation unit 150 is a means for allowing the operator to manipulate the position and function of the robot arm 210 of the slave robot 200. As illustrated in FIG. 2, the arm operating unit 150 may be formed in the shape of a handle member (see 110 of FIG. 2), but the shape is not limited thereto and may be modified in various shapes to achieve the same purpose. Can be. Further, for example, some may be formed in the shape of a handle, others may be formed in a different shape, such as a clutch button, finger insertion tube or insertion to enable the operator's fingers can be inserted and fixed to facilitate the operation of the surgical tool More rings may be formed.

The manipulation signal generating unit 160 generates a manipulation signal corresponding to the manipulation of the arm manipulation unit 150 when the operator manipulates the arm manipulation unit 150 for the movement of the robot arm 210 and / or the laparoscope 220 or the manipulation operation. To the slave robot 200. As described above, the manipulation signal may be transmitted and received through a wired or wireless communication network.

The virtual operation panel implementing unit 170 displays the virtual operation panel on the display member (see 120 of FIG. 2), and the robot arm 210 displayed on the display member (see 120 of FIG. 2) controls the virtual operation panel. When the manipulation operation is performed, an operation implementation signal for performing an actual operation corresponding to the manipulation operation is generated. Specific functions, various detailed configurations, and the like of the virtual operation panel implementing unit 170 will be described in detail with reference to the accompanying drawings.

The controller 180 controls the operation of each component so that the above-described function can be performed. The controller 180 may perform a function of converting an image input through the image input unit 130 into an image image to be displayed through the screen display unit 140. In addition, the controller 180 transmits an image input from the image input unit 130 to the virtual operation panel implementer 170, and executes an operation implementation signal generated by the virtual operation panel implementer 170. It may be.

FIG. 4 is a block diagram schematically illustrating a configuration of a virtual operation panel implementing unit provided in the master robot of FIG. 3.

Referring to FIG. 4, the virtual operation panel implementing unit 170 of the master robot 100 of the surgical robot system 1 according to an embodiment of the present invention may include a virtual operation panel display unit 171 and a motion matching unit 173. , An operation implementation signal generator 175, a reaction force output unit 177, and an operation-scaling control unit 179. One or more components included in the virtual operation panel implementation 170 may be implemented in the form of a software program by a combination of program codes.

In detail, the virtual operation panel display unit 171 serves to display the virtual operation panel on the display member 120. Such a virtual operation panel (see 300 in FIG. 6) includes a function operation unit (see 310 in FIG. 6) and an information display unit (see 320 in FIG. 6).

Here, the virtual operation panel 300 (refer to 300 in FIG. 6) may be an operation panel having a predetermined shape existing in the virtual three-dimensional space. As described above, the display member 120 may be provided as a stereoscopic display device. In addition, the virtual operation panel 300 (refer to 300 of FIG. 6) may be a structure such as a three-dimensional flat panel or a cuboid that is displayed on the stereoscopic display device. The virtual operation panel (see 300 of FIG. 6) is a virtual structure existing only on the display member 120, not an object in the real world, and is operated on the display member 120 in the form of a small structure of several millimeters in all directions. The area may be displayed to float in the air above the vertical. The area where the virtual operation panel (see 300 in FIG. 6) is normally located is connected to the main body of the robot arm 210 (for example, the upper and lower parts of the robot arm, the lower arm, the wrist, the elbow, etc.) or the main body of the robot arm 210. Surgical instruments are often overlooked as they move and deform, and are not related to the surgical operation, so the user manipulates the virtual operation panel (see 300 in FIG. 6) with the actual robot arm 210, but the surgical site is thus Since it is spaced a lot from the safety can be guaranteed. The configuration of such a virtual operation panel (see 300 of FIG. 6) will be described in detail with reference to FIGS. 5 and 6.

Here, the virtual operation panel display unit 171 performs an operation of pressing or holding a predetermined button provided on the master robot 100, or the robot arm 210 being displayed on the display member 120 is displayed. When a virtual operation panel activation signal is generated by performing a predetermined operation, the virtual operation panel may be controlled to be activated on the display member 120.

For example, the operation lever 110 of the master robot 100 is provided with a separate virtual operation panel activation signal generation button (not shown), and the operation lever 110 in a state in which the user grips the operation lever 110. By pressing or holding the virtual operation panel activating signal generation button (not shown) of FIG. 1, the virtual operation panel may be displayed on the display member 120 in a desired size.

Alternatively, the surgical instrument provided in the robotic arm 210 rotates in the air, or moves straight in a certain direction over a certain speed, or a predetermined specific motion (for example, a surgical instrument has a predetermined shape such as a "Z" shape in the air). Operation such as drawing a mark of the user), such as a virtual operation panel activation operation, which is possible irrespective of the surgical operation, and when such an operation is performed, a virtual operation panel activation signal is generated and displayed on the display member 120. You can also control the virtual operation panel to be displayed in the desired size.

Alternatively, the virtual operation panel minimized on the display member 120 is displayed in the form of an icon, and the robot arm 210 displayed on the display member 120 presses or grabs the minimized virtual operation panel in the form of the icon. The virtual operation panel activation signal may be generated to control the virtual operation panel to be displayed on the display member 120.

That is, in this specification, activation of the virtual operation panel may mean that a virtual operation panel that was not displayed on the display member 120 is displayed, and the virtual operation that was displayed in a minimized state on the display member 120. It may be said that the plate is enlarged and displayed to a desired size.

Here, the virtual operation panel display unit 171 may display the virtual operation panel by overlapping the image displayed on the display member 120, that is, the image photographed through the laparoscope 220. As such, displaying the virtual operation panel by overlapping the image photographed through the laparoscope 220 may be performed using augmented reality. Augmented reality is a technology that superimposes a virtual object on the real world that the user sees with his eyes, and shows the virtual world with additional information in real time as a single image to show mixed images (Mixed Reality, MR) Also called. Detailed description of such augmented reality will be omitted herein.

Next, the motion matching unit 173 determines whether the robot arm 210 displayed on the display member 120 performs an operation of operating the virtual operation panel. In detail, the movement of the robot arm 210 is an actual movement, while the virtual operation panel actuated by the movement of the robot arm is not actually present but is simply an image being displayed on the display member 120. However, when the motion of operating the virtual operation panel is taken through the movement of the robot arm, it is a feature of the present invention that an operation corresponding to the motion is actually performed. Therefore, in order to perform such an operation, the motion matching unit 173 determines whether the robot arm 210 displayed on the display member 120 performs an operation of operating the virtual operation panel. That is, the actual movement of the robot arm 210 is performed in the air, but if the movement of the robot arm 210 displayed on the display member 120 is a motion of pressing or holding a predetermined button on the virtual operation panel, this is the case. The action is actually performed. In other words, the motion matching unit 173 determines that the motion of the robot arm 210 may be, for example, pressing a predetermined button on the virtual operation panel, turning a dial on the virtual operation panel, or turning on / off a switch on the virtual operation panel. It is to determine whether the motion is to turn off or to pick up a specific part on the virtual operation panel to move to another area.

Next, the operation implementation signal generator 175 generates an operation implementation signal for performing an actual operation corresponding to the operation operation of the performed virtual operation panel. In detail, as a result of analyzing the operation of the robot arm 210 displayed on the display member 120 by the motion matching unit 173, the operation of the robot arm 210 displayed on the display member 120 is virtually operated. When the operation is performed, the operation implementation signal generator 175 actually generates an operation implementation signal for performing the operation.

For example, when the motion matching unit 173 determines that the operation of the robot arm 210 displayed on the display member 120 is to press or hold the laparoscope movement button on the virtual operation panel, the operation implementation signal generator 175 generates a laparoscopic movement signal and transmits it to the controller 180. In addition, the controller 180 controls the laparoscope 220 of the slave robot 200 to actually move the laparoscope 220 by a predetermined distance.

Alternatively, when the motion matching unit 173 determines that the operation of the robot arm 210 displayed on the display member 120 is an operation of pressing or grabbing an MRI image magnification button on the virtual operation panel, the operation implementation signal generator 175. ) Generates an MRI image magnification signal and transmits it to the controller 180. The controller 180 actually displays the enlarged MRI image on the display member 120.

On the other hand, the virtual operation panel implementation unit 170 of the surgical robot system 1 according to an embodiment of the present invention performs an operation for the robot arm 210 to operate the virtual operation panel displayed on the display member 120. The reaction force output unit 177 may further include a reaction force output unit 177 that provides a predetermined reaction force corresponding to the operation to the operation lever 110 of the master robot 100.

In detail, the operator actually moves the robot arm of the slave robot through the master robot to operate a switch on the virtual operation panel, or to apply an operation to the virtual operation panel itself (for example, in contact with the virtual operation panel or to operate the virtual operation panel). Reaction part when the switch is actually pressed, the contact feeling when contacting the actual operation panel, or the actual operation panel, is performed. When moving the haptic feeling, such as a feeling of gravity moving the object having a predetermined weight is output to the operation lever 110 in synchronization with the image, it is possible to implement a realistic and intuitive interaction. By the reaction force output unit 177 of the present invention, when the operator operates the virtual switch in the virtual operation panel provides an intuitive reaction force, through which an effect of intuitively confirming that the corresponding function is correctly selected can be obtained. .

On the other hand, the virtual operation panel implementation unit 170 of the surgical robot system 1 according to an embodiment of the present invention is the robot of the slave robot 200 in preparation for the operation distance of the operation lever 110 of the master robot 100 The apparatus may further include an operation-scaling controller 179 for controlling a ratio of an operation distance of the arm 210.

As described above, the virtual operation panel 300 (refer to 300 in FIG. 6) may be an operation panel having a predetermined shape existing in the virtual three-dimensional space. Here, the virtual operation panel (refer to 300 of FIG. 6) may be a structure such as a three-dimensional flat panel or a cuboid that is displayed on the stereoscopic display device. This virtual operation panel (see 300 in FIG. 6) may be displayed as a small structure measuring several millimeters in all directions so that it floats in the air vertically above the surgical site on the display member 120. When the virtual operation panel activation signal is generated while the virtual operation panel (refer to 300 in FIG. 6) of such a small size is being displayed, the virtual operation panel is activated and enlarged to an appropriate size on the display member 120. Is displayed.

Here, the operation-scaling control unit 179 is normally displayed in a state where the virtual operation panel (see 300 of FIG. 6) is minimized. When the virtual operation panel activation signal is generated, only the portion of the virtual operation panel (see 300 of FIG. 6) is displayed. At the same time, it serves to control the scaling of the operation of the master robot and the slave robot in the enlarged state. This is a concept separate from the master-slave motion scaling concept during general surgery, and is a function provided for controlling the virtual operation panel when the virtual operation panel is activated and enlarged. By the motion-scaling control unit 179 of the present invention, it is possible to obtain an effect of eliminating the possibility that the movement of the robot arm of the slave robot for the virtual operation panel operation affects the affected part.

Meanwhile, the area where the virtual operation panel is displayed should be a space that is safely spaced apart from the actual surgical site. In general, in all surgical operations, the space immediately above the surgical site in the center of the endoscope screen becomes empty to secure the surgical field of vision, thereby defining a cube-shaped space in this area within the total spatial coordinate system where real and virtual spaces overlap. The virtual operation panel can be arranged in the space. Furthermore, in order to construct a safer surgical space, the operator may directly designate a location on the space where the virtual operation panel is to be located before or during the operation as a safe space. In addition, while the control system continuously observes the movement of the robot arm during surgery, a space that is safe even when the virtual operation panel is located, that is, a space formed by a path through which the robot arm's body repeatedly passes (an obstacle or a human body requiring attention) The space that can be assumed not to be on the path) can be automatically recognized and adaptively automatically adjusted so that the virtual operation panel is always positioned at the safest position.

On the other hand, when the surgical robot system 1 according to an embodiment of the present invention operates the virtual operation panel by actually moving the robot arm 210 displayed on the display member 120 as described above, the actual operation corresponding thereto. Although described as being performed, the spirit of the present invention is not limited thereto. That is, when the virtual operation panel is displayed on the display member 120, a virtual mouse, a virtual pointer, a virtual finger, etc. are displayed on the display member 120 at the same time, and in this state, the user manipulates the handle of the master robot. It will be possible to operate the virtual operation panel while moving the displayed virtual mouse / pointer / finger instead of actually moving the robot arm 210.

FIG. 5 is an example of a screen display form in which an image captured by a laparoscope is output through a display member, and FIG. 6 is a diagram illustrating a screen display form in which a virtual operation panel is displayed on the display member of FIG.

First, as shown in Figure 5, the operator performs the operation while checking the image of the surgical site taken through the laparoscopic 220 through the display member 120. In this case, the first robot arm 211, the second robot arm 212, and the like are displayed on the screen output through the display member 120. This state of FIG. 5 is a state before the virtual operation panel is activated.

In this state, when the operator operates the operation lever 110 of the master robot 100 to move the robot arm 210 of the slave robot 200 to perform a predetermined operation to generate a virtual operation panel activation signal, FIG. 6 A virtual operating panel 300 as shown in is shown on the display member 120. The virtual operation panel 300 includes a function operation unit 310 and an information display unit 320.

In detail, the function manipulation unit 310 operates the slave robot 200 such as manipulation of the laparoscope 220 and selection of a robot arm to be manipulated using the master robot 100 among a plurality of robot arms of the slave robot 200. And control functions. The function manipulation unit 310 may include a robot arm selection button 311, a laparoscope movement button 312, an electrocauter control button 313, and the like. By using the function operation unit 310 as described above, the position movement of the laparoscope 220 is changed, the posture change, the view enlargement / reduction, the setting of the operation scale relationship between the master robot 100 and the slave robot 200, and the slave robot 200. In the case of a system having three or more robot arms 210, selecting and connecting / disconnecting the two robot arms 210 to be manipulated by the two operation levers 110 of the master robot 100, and the slave robot 200 It is possible to perform system operation commands such as pausing, controlling the operation of the electrocauterizer, controlling the operation of the anesthesia, turning the CO2 injector on and off and adjusting the gas pressure.

Furthermore, the functional manipulation unit 310 can control not only the control of the surgical robot system itself but also other environments inside the operating room, so that the operator can control all the instruments and the environment within the operating room by the operator while holding the operation handle of the master robot. You can also control it. For example, the function manipulation unit 310 may be used to control the illumination in the operating room, to control the brightness of the laparoscope, or to control the height and posture of the surgical bed.

For example, the operator operates the manipulation lever 110 of the master robot 100 to move the robot arm 210 of the slave robot 200 so that the robot arm 210 displayed on the display member 120 is virtual. When the laparoscopic movement button 312 of the function operation unit 310 of the operation panel 300 is pressed or grasped, the laparoscopic 220 moves in the direction of the button. Here, although the laparoscopic movement button 312 is shown as being provided as a four-way button, the spirit of the present invention is not limited thereto, and may be provided in various forms that can be designated in various forms such as a joystick and a track ball. will be.

Alternatively, the operator manipulates the operation lever 110 of the master robot 100 to move the robot arm 210 of the slave robot 200, so that the robot arm 210 displayed on the display member 120 is virtually operated. When pressing or grabbing the electrocauter control button 313 of the function manipulation unit 310 of 300, the operation and control of the cauterizer is performed. That is, various operations such as ON / OFF of the electric cauterizer and output control of the electric cauterizer may be performed through the manipulation of the robot arm 210.

Meanwhile, the information display unit 320 may display one or more real-time biometric information of the patient or medical image information (eg, X-ray image, CT image, MRI image, etc.) of the patient for the surgical site. The information display unit 320 may include a biometric information display unit 321, a medical image information display unit 322, and a selection button unit 323. Here, each of the real-time biometric information displayed on the biometric information display unit 321 or the related image of the patient displayed on the medical image information display unit 322 through the manipulation of the selection button unit 323 (ON / OFF) ), Zoom, position movement, etc. can be controlled.

The biometric information display unit 321 may output one or more indicators indicating a patient's condition, for example, real-time biometric information such as an electrocardiogram, a body temperature, a pulse, a breath, and a blood pressure of the patient. It may be.

In the case of the biometric information display unit 321, various real-time numerical values or waveform information such as multi-channel electrocardiogram and real-time blood pressure at various sites are selected and viewed in whole or in part, and if necessary, some of these are separated and displayed on the screen. It can also be displayed continuously on the part. For example, during cardiac surgery, certain signal waveforms collected from various parts of the heart that require constant attention can be removed by attaching only one channel from the operation panel to another specific area of the entire endoscope screen. Intuitive operation, such as placing, allows the operator to reconfigure the information depiction form within the display member 120.

In order to provide such biometric information to the master robot 100, the slave robot 200 may include one or more of a body temperature measurement module, a pulse measurement module, a respiration measurement module, a blood pressure measurement module, an electrocardiogram measurement module, and the like. (Not shown). The biometric information measured by each module may be transmitted from the slave robot 200 to the master robot 100 in the form of an analog signal or a digital signal, and the master robot 100 displays the received biometric information on the display member 120. It can be displayed through the virtual operation panel 300.

The medical image information display unit 322 may display relevant image data of the patient on the surgical site such as an X-ray image, a CT image, and an MRI image. In addition, the medical image information display unit 322 may display even a reconstructed three-dimensional image data or a digitized model based on computed tomography (CT) image data. The medical image data can also be used to change the screen scroll, zoom, and change the 3D visual rotation in a manner similar to that of a multi-touch gesture method or a 3D mouse or a 3D haptic interface according to the characteristics of 2D and 3D data. It can be easily performed to read.

Meanwhile, by utilizing the characteristics of the present invention that can be interacted with, the information display unit 320 does not merely display biometric information or a medical image, but selects only information necessary for a specific surgical operation process and moves it to a desired position on the screen to fix it. You could do it. For example, in the process of physiologically monitoring the momentary change of the patient's condition, the ECG of a specific channel and the blood pressure (waveform and) of a specific part may be separated and operated while being fixed on the screen. Or, during a work process that requires anatomical structure information, a specific scene from a medical image may be selected and fixed on one side of the screen, and then the surgery may be performed. In this way, the selection of the cooking ability for the biometric information and the medical image can be performed, and thus, the operator's ease of operation can be further improved.

Hereinafter, a control method of a surgical robot system according to an exemplary embodiment of the present invention will be described with reference to related drawings. 7 is a flow chart schematically showing a control method of a surgical robot system according to an embodiment of the present invention.

Referring to FIG. 7, in the method for controlling a surgical robot system according to an exemplary embodiment of the present invention, an image captured by the laparoscope 220 of the slave robot 200 is displayed on the display member 120 of the master robot 100. The displayed step (step S110), the step of generating a virtual operation panel activation signal (step S120), the image and the virtual operation panel 300 taken through the laparoscope 220 on the display member 120 of the master robot 100 ) Overlapping and displayed (step S130), the operation of the robot arm 210 displayed on the display member 120 to operate the operation member on the virtual operation panel 300 through the operation of the master robot 100. Performing (step S140) and the operation manipulated by the robot arm 210 of the slave robot 200 on the virtual operation panel 300 is actually performed (step S150).

First, an image photographed through the laparoscope 220 of the slave robot 200 is displayed on the display member 120 of the master robot 100 (S110). As such, a screen display form in which the image captured by the laparoscope 220 is output through the display member 120 is illustrated in FIG. 5.

In operation S120, it is determined whether a virtual operation panel activation signal is generated. In detail, the virtual operation panel implementing unit 170 of the master robot 100 performs an operation of pressing or holding a predetermined button provided on the master robot 100 or the robot arm being displayed on the display member 120. It is determined whether or not 210 has performed a predetermined predetermined operation. Here, the predetermined predetermined operation may be, for example, an operation that can be performed irrespective of the surgical operation, such as the robot arm 210 rotating in the air or moving straight in a direction more than a predetermined speed, or the display member. The virtual operation panel minimized on the 120 may be displayed in the form of a predetermined icon, and the robot arm 210 displayed on the display member 120 may press or hold the minimized virtual operation panel. When the robot arm 210 performs a predetermined operation as described above, the virtual operation panel implementing unit 170 generates a virtual operation panel activation signal.

Next, when the virtual operation panel activation signal is generated, the image photographed through the laparoscope 220 and the virtual operation panel 300 is superimposed on the display member 120 of the master robot 100 to display (step S130). do. In detail, when the virtual operation panel display unit 171 of the virtual operation panel implementation unit 170 of the master robot 100 receives the virtual operation panel activation signal, the image displayed on the display member 120, that is, the laparoscope 220 The virtual operation panel 300 may be superimposed on the image photographed through). As such, displaying the virtual operation panel 300 by overlapping the image photographed through the laparoscope 220 may be performed using augmented reality. 6 illustrates a screen display form in which the virtual operation panel 300 is overlapped and displayed on the display member 120.

The virtual operation panel 300 includes a function operation unit 310 and an information display unit 320. In detail, the function manipulation unit 310 operates the slave robot 200 such as manipulation of the laparoscope 220 and selection of a robot arm to be manipulated using the master robot 100 among a plurality of robot arms of the slave robot 200. And control functions. The function manipulation unit 310 may include a robot arm selection button 311, a laparoscope movement button 312, an electrocauter control button 313, and the like. Meanwhile, the information display unit 320 may display one or more real-time biometric information of the patient or medical image information (eg, X-ray image, CT image, MRI image, etc.) of the patient about the surgical site. The information display unit 320 may include a biometric information display unit 321, a medical image information display unit 322, and a selection button unit 323.

Next, the robot arm 210 displayed on the display member 120 performs an operation of manipulating the manipulation member on the virtual manipulation panel 300 through manipulation of the master robot 100 (operation S140). This will be described in more detail as follows.

The operator manipulates the operation lever 110 of the master robot 100, and the robot arm 210 displayed on the display member 120 takes an action to operate a predetermined operation member on the virtual operation panel 300. . The predetermined operation member may be each button of the function operation unit 310 or the information display unit 320. Alternatively, operations such as rotating the image itself shown on the information display unit 320 may be possible.

In this case, the actual movement of the robot arm 210 is performed in the air, but the movement of the robot arm 210 displayed on the display member 120 may be a motion of pressing or holding a predetermined button on the virtual operation panel 300. have. Accordingly, the motion matching unit 173 determines whether the robot arm 210 displayed on the display member 120 performs an operation of manipulating the virtual operation panel 300. That is, while the movement of the robot arm 210 is an actual movement, the virtual operation panel operated through the movement of the robot arm is not an image, but an image that is simply displayed on the display member 120. However, in the present invention, when the motion of operating the virtual operation panel is taken through the movement of the robot arm, an operation corresponding to the motion is actually performed. To this end, the motion matching unit 173 determines whether the robot arm 210 displayed on the display member 120 performs an operation of operating the virtual operation panel.

Next, an operation manipulated by the robot arm 210 of the slave robot 200 on the virtual operation panel 300 is actually performed (step S150).

In detail, the operation implementation signal generator 175 generates an operation implementation signal for performing an actual operation corresponding to the operation operation of the virtual operation panel 300. That is, as a result of analyzing the operation of the robot arm 210 displayed on the display member 120 by the motion matching unit 173, the operation of the robot arm 210 displayed on the display member 120 is virtual. When the operation is performed, the operation implementation signal generator 175 actually generates an operation implementation signal for performing the operation.

For example, when the motion matching unit 173 determines that the operation of the robot arm 210 displayed on the display member 120 is to press or hold the laparoscope movement button on the virtual operation panel, the operation implementation signal generator 175 generates a laparoscopic movement signal and transmits it to the controller 180. In addition, the controller 180 controls the laparoscope 220 of the slave robot 200 to actually move the laparoscope 220 by a predetermined distance. Alternatively, when the operation of the robot arm 210 displayed on the display member 120 is an operation of pressing or holding the viewing position moving button or the zoom button of the laparoscope on the virtual operation panel, the operation implementing signal generator 175. Generates a laparoscopic field of view movement or zoom signal and transmits it to the controller 180. In addition, the controller 180 controls the laparoscope 220 of the slave robot 200 so that the laparoscope 220 actually moves forward or backward or enlarges / reduces the image itself.

Alternatively, when the motion matching unit 173 determines that the operation of the robot arm 210 displayed on the display member 120 is an operation of pressing or grabbing an MRI image magnification button on the virtual operation panel, the operation implementation signal generator 175. ) Generates an MRI image magnification signal and transmits it to the controller 180. The controller 180 actually displays the enlarged MRI image on the display member 120.

That is, in the conventional surgical robot system, setting the operation scale relationship between the master robot and the slave robot, in the case of a system having three or more slave robot arms, selecting and connecting / disconnecting the robot arm to be manipulated at present, and pausing the slave robot. In order to implement various functions, a separate operation panel is required.To operate the operation panel, hold the handle of the master robot, press the button provided on the operation panel of the master robot, and then hold the handle of the master robot again. I had to proceed with the surgery.

On the contrary, the surgical robot system according to an embodiment of the present invention generates a predetermined virtual operation panel activation signal by manipulating the handle while the handle of the master robot is held in order to move the laparoscopic image. After the operation panel is displayed on the screen display unit, the slave robot arm presses or grabs the movement button of the virtual operation panel displayed on the screen display unit by operating the handle of the master robot. will be.

On the other hand, the control method of the surgical robot system according to an embodiment of the present invention, when the robot arm 210 of the slave robot 200 operates the virtual operation panel 300, the master robot 100 has a predetermined reaction force The method may further include transmitting to the user through S160.

In detail, the operator actually moves the robot arm of the slave robot through the master robot to operate a switch on the virtual operation panel, or to apply an operation to the virtual operation panel itself (for example, in contact with the virtual operation panel or to operate the virtual operation panel). Reaction part when the switch is actually pressed, the contact feeling when contacting the actual operation panel, or the actual operation panel, is performed. When moving the haptic feeling, such as a feeling of gravity moving the object having a predetermined weight is output to the operation lever 110 in synchronization with the image, it is possible to implement a realistic and intuitive interaction. By the reaction force output unit 177 of the present invention, when the operator operates the virtual switch in the virtual operation panel provides an intuitive reaction force, through which an effect of intuitively confirming that the corresponding function is correctly selected can be obtained. .

Finally, when the robot arm 210 performs a predetermined predetermined operation, it is determined whether the virtual operation panel deactivation signal is generated (step S170), and when the virtual operation panel deactivation signal is generated, the virtual The operation panel 300 is deactivated, and only the image photographed through the laparoscope 220 of the slave robot 200 is displayed on the display member 120 of the master robot 100.

According to the present invention, it is possible to easily operate the entire surgical robot system and information systems such as various biometric information, medical image information within a single system, it is possible to obtain the effect that can be operated alone. Furthermore, the single surgery as in the technique of the present technology can increase the attention of the operator to improve the accuracy of the operation, shorten the operation time, and increase the effectiveness of the surgery, as well as the efficiency of the operation and the medical cost, which are advantages of the single surgery. It can incidentally provide the effect of. In addition, by applying a haptic interface to the surgical robot system, it is possible to improve the accuracy and intuitiveness of the operator's motion input, and to apply augmented reality interface to perform a more intuitive operation more intuitive.

In the present specification, the present invention has been described with reference to limited embodiments, but various embodiments are possible within the scope of the present invention. In addition, although not described, equivalent means will also be referred to as incorporated in the present invention. Therefore, the true scope of the present invention will be defined by the claims below.

1: Surgical Robot System 100: Master Robot
110: operation lever 120: display member
200: slave robot 210: robot arm
220: Laparoscopic

Claims (31)

In the master robot for controlling a slave robot including a surgical endoscope and a robot arm,
A display member configured to display an image photographed through the surgical endoscope;
A virtual operation panel display unit displaying a virtual operation panel on the display member;
A motion matching unit determining whether the robot arm displayed on the display member performs an operation of manipulating the virtual operation panel; And
And an operation implementation signal generator configured to generate an operation implementation signal for performing an actual operation corresponding to the operation operation of the virtual operation panel. The method of claim 1,
And the virtual operation panel display unit superimposes the virtual operation panel on an image captured by the surgical endoscope displayed on the display member. The method of claim 1,
And the virtual operation panel display unit displays the virtual operation panel on the display member when a predetermined virtual operation panel activation signal is received. The method of claim 1,
The virtual operation panel display unit controls the operation of the surgical endoscope or the robot arm, selection of a robot arm to be manipulated using the master robot among a plurality of the robot arms, output control of an electric cauterizer, the master robot and the Function to perform at least one operation of setting the operation scale relationship of the slave robot, pausing and resuming the slave robot, manipulating anesthesia, turning on / off the CO2 injector and adjusting gas pressure. And displaying an operation unit on the virtual operation panel. The method of claim 1,
And the virtual operation panel display unit displays an information display unit for displaying the biometric information of the subject and / or the medical image information of the subject on the virtual operation panel. The method of claim 5, wherein
And at least some of the biometric information and the medical image information are fixedly displayed at a predetermined position on the display member. The method of claim 1,
When the robot arm performs an operation to operate the virtual operation panel displayed on the display member, further comprising a reaction force providing unit for providing a predetermined reaction force corresponding to the operation to the operation lever of the master robot; Master robot. The method of claim 1,
And an operation-scaling control unit for controlling a ratio of an operation distance of the robot arm of the slave robot to an operation distance of the operation lever of the master robot in a state where the virtual operation panel is enlarged / reduced. Master robot comprising a. The method of claim 1,
The display member is a master robot, characterized in that it comprises a three-dimensional display device. Surgical endoscopes for imaging the surgical site; And
Slave robot comprising; and a robot arm for driving with multiple degrees of freedom,
A display member configured to display an image photographed through the surgical endoscope; And
Displaying a virtual operation panel on the display member and generating an operation realization signal for performing an actual operation corresponding to the operation operation by operating the virtual operation panel of the robot arm displayed on the display member; Surgical robot system comprising a master robot including; virtual operation panel implementation. 11. The method of claim 10,
The virtual operation panel implementation unit,
A virtual operation panel display unit displaying the virtual operation panel on the display member;
A motion matching unit determining whether the robot arm displayed on the display member performs an operation of manipulating the virtual operation panel; And
And an operation implementation signal generator for generating an operation implementation signal for performing an actual operation corresponding to the operation operation of the virtual operation panel. 11. The method of claim 10,
And a virtual operation panel activation signal for displaying a virtual operation panel on the display member when the robot arm performs a predetermined predetermined operation. 11. The method of claim 10,
And a reaction force providing unit configured to provide a predetermined reaction force to an operation lever of the master robot when the robot arm performs an operation of operating the virtual operation panel displayed on the display member. 11. The method of claim 10,
And an operation-scaling control unit for controlling a ratio of an operation distance of the robot arm of the slave robot to an operation distance of the operation lever of the master robot in a state where the virtual operation panel is enlarged / reduced. Surgical robot system comprising a. 11. The method of claim 10,
The virtual operation panel display unit controls the operation of the surgical endoscope or the robot arm, selection of a robot arm to be manipulated using the master robot among a plurality of the robot arms, output control of an electric cauterizer, the master robot and the Function to perform at least one operation of setting the operation scale relationship of the slave robot, pausing and resuming the slave robot, manipulating anesthesia, turning on / off the CO2 injector and adjusting gas pressure. And a control panel on the virtual operation panel. 11. The method of claim 10,
The virtual operation panel includes an information display unit for displaying the biometric information of the subject and / or the medical image information of the subject. 17. The method of claim 16,
The medical image information is reconstructed three-dimensional image data based on X-ray image, computed tomography (CT) image, magnetic resonance imaging (MRI) image and the computed tomography (CT) image data. Or one or more of a quantified model. 17. The method of claim 16,
The slave robot further includes a biometric information measuring unit including at least one of a body temperature measuring module, a pulse measuring module, a respiratory measuring module, a blood pressure measuring module, an electrocardiogram measuring module, and the like,
And the information measured by the biometric information measuring unit is displayed on the information display unit. 17. The method of claim 16,
And at least a portion of the biometric information and the medical image information is fixedly displayed at a predetermined position on the display member. 11. The method of claim 10,
And the display member comprises a stereoscopic display device. In the control method of the surgical robot system comprising a master robot and a slave robot,
Displaying an image captured by a surgical endoscope of the slave robot on a display member of the master robot;
Generating a predetermined virtual operation panel activation signal;
Displaying an image captured by the surgical endoscope on the display member of the master robot and the virtual operation panel;
Performing operation of manipulating the virtual operation panel by a robot arm of the slave robot displayed on the display member; And
And actually performing the operation manipulated by the robot arm on the virtual operation panel. 22. The method of claim 21,
Generating the predetermined virtual operation panel activation signal,
In the state where the operator holds the operation lever of the master robot, the robot arm performs a predetermined operation by the operation of the master robot, whereby the predetermined virtual operation panel activation signal is generated. The control method of the surgical robot system. The method of claim 22,
The predetermined operation includes pressing or holding the virtual operation panel activation signal generation button formed on the operation lever of the master robot, rotating the robot arm in the air, and constantly moving the robot arm at a constant speed or more. And a motion of moving the robot arm in a straight direction, drawing the predetermined mark in the air, and pressing or holding the predetermined icon displayed on the display member. How to control the robotic system. 22. The method of claim 21,
The virtual operation plate is a control method of the surgical robot system, characterized in that it comprises a functional operation for controlling the operation of the surgical endoscope or the robot arm. 25. The method of claim 24,
When the operation operated by the robot arm of the slave on the virtual operation panel is an operation of operating the function operation unit,
The step of actually operating the operation by the robot arm of the slave on the virtual operation panel, the control method of the surgical robot system, characterized in that the control operation for the robot arm or the surgical endoscope of the slave robot is performed. 22. The method of claim 21,
The virtual operation panel is a control method of the surgical robot system, characterized in that it comprises an information display for displaying the biometric information of the subject and / or the medical image information of the subject. The method of claim 26,
When the operation operated by the robot arm of the slave on the virtual operation panel is an operation of operating the information display unit,
The step of actually performing the operation operated by the robot arm of the slave on the virtual operation panel, the control method of the surgical robot system, characterized in that the control operation for the information display unit shown in the display member is performed. The method of claim 26,
When the operation operated by the robot arm of the slave on the virtual operation panel is an operation of moving at least a part of the information display unit to a specific position on the display member,
And at least some of the biometric information and the medical image information are fixedly displayed at a predetermined position on the display member. 22. The method of claim 21,
The displaying of the virtual operation panel and the image photographed through the surgical endoscope are displayed on the display member of the master robot by overlapping with each other.
And the virtual operation panel is displayed in a state of being injured to the user side of the display member to a certain degree from the surgical site displayed on the display member. 22. The method of claim 21,
And when a robot arm of the slave robot displayed on the display member operates the virtual operation panel, a predetermined reaction force is provided to the operation lever of the master robot. How to control the robotic system. A program of instructions, which can be executed by a digital processing apparatus, is tangibly implemented to perform the control method of the surgical robot system according to any one of claims 21 to 30, and a program that can be read by the digital processing apparatus. Recorded media.

Images