KR101798590B1 - Master robot, surgical robot system using thereof, control method thereof, and recording medium thereof - Google Patents

Abstract

In the surgical robot system including the master robot and the slave robot, various operations of the surgical robot system are performed by the operator to grasp the operation lever of the master robot so that the main operator can perform the single operation without performing the assistant The present invention provides a master robot for controlling a slave robot including a surgical endoscope and a robot arm, the master robot comprising: a display member for displaying an image photographed through the surgical endoscope; A virtual operation panel display unit for displaying a virtual operation panel on the display member; A motion matching unit for determining whether the robot arm displayed on the display member performs an operation of operating the virtual operation board; And an operation implementation signal generation unit for generating an operation implementation signal for performing an actual operation corresponding to the operation operation of the virtual operation board performed.

Description

TECHNICAL FIELD The present invention relates to a master robot, a surgical robot system including the robot, a control method thereof, and a recording medium on which the master robot is recorded,

The present invention relates to a master robot, a surgical robot system including the master robot, a control method thereof, and a recording medium on which the master robot is mounted. More particularly, the present invention relates to a master robot, A master robot capable of performing various operations of the surgical robot system while watching the monitor screen while the operator is holding the operation lever of the master robot, and a surgical robot system including the same A control method thereof, and a recording medium recording the same.

The surgical robot refers to a robot having a function of substituting surgical operation performed by a surgeon. Such a surgical robot has an advantage that it can perform accurate and precise operation compared to a human and can perform a remote operation.

Surgical robots currently being developed around the world include bone surgery robots, laparoscopic surgery robots, and stereotactic robots. Here, a laparoscopic surgical robot is a robot that performs minimally invasive surgery using laparoscopic and small surgical instruments.

Laparoscopic surgery is an advanced surgical technique that inserts a laparoscope, which is an endoscope for inserting a 1 cm hole in the umbilicus and looks into the abdomen. A recent laparoscope is equipped with a computer chip to provide a sharper, more magnified image than is seen with the naked eye, and the laparoscopic surgical instruments specially designed by looking at the screen through the monitor have developed enough to perform any operation.

In addition, laparoscopic surgery has the advantage of being less complicated than open surgery, initiating treatment within a very short time after surgery, and maintaining the physical strength and immune function of the surgical patients, have. Therefore, in the United States and Europe, laparoscopic surgery is becoming increasingly recognized as a standard surgery for colorectal cancer treatment and the like.

On the other hand, the surgical robot system generally comprises a master robot and a slave robot. When the operator manipulates a steering lever (for example, a steering wheel) provided on the master robot, the surgical tool held by the robot arm is coupled to the robot arm of the slave robot.

In the laparoscopic surgery described above, the manipulation of the endoscope plays a major role in the surgical operation. In the conventional laparoscopic surgery without using the surgical robot system, a separate auxiliary operation is performed only for manipulation of the endoscope. In addition, a surgical robot system that automates the operation of an endoscope or a surgical robot system including both an endoscope and a surgical tool manipulator does not require such an assistant operator, and the operator can directly manipulate the endoscope during the operation. It is called "single operation" in the sense that the main operator performs all the surgical operations alone.

The present invention relates to a surgical robot system including a master robot and a slave robot, in which various operations of a surgical robot system are operated by a master robot The present invention provides a master robot, a surgical robot system including the same, and a control method thereof, and a recording medium on which the master robot can be operated while viewing the monitor screen while holding the lever.

A master robot for controlling a slave robot including a surgical endoscope and a robot arm, the master robot comprising: a display member for displaying an image photographed through the surgical endoscope; A virtual operation panel display unit for displaying a virtual operation panel on the display member; A motion matching unit for determining whether the robot arm displayed on the display member performs an operation of operating the virtual operation board; And an operation implementation signal generation unit for generating an operation implementation signal for performing an actual operation corresponding to the operation operation of the virtual operation board performed.

In the present invention, the virtual operation panel display unit may superimpose the virtual operation panel on an image photographed through the surgical endoscope displayed on the display member.

In the present invention, the virtual operation panel display unit may display the virtual operation panel on the display member when a preset virtual operation panel activation signal is received.

In the present invention, the virtual operation panel display unit may be configured to control operation of the surgical endoscope or the robot arm, selection of a robot arm to be operated using the master robot among the plurality of robot arms, Wherein at least one of setting of an operation scale relationship between the master robot and the slave robot, pause and resumption of the slave robot, operation control of an anesthesia machine, on / off of a CO2 insulator, and gas pressure control On the virtual operation panel can be displayed.

In the present invention, the virtual operation panel display unit may display, on the virtual operation panel, an information display unit for displaying the biometric information of the subject and / or the medical image information of the subject.

Here, at least a part of the biometric information and the medical image information may be fixed and displayed at a predetermined position on the display member.

In the present invention, when the robot arm performs the operation of manipulating the virtual manipulation plate displayed on the display member, the reaction force providing section for providing the manipulation lever of the master robot with a predetermined reaction force corresponding to the motion .

In the present invention, an operation of controlling the ratio of the operation distance of the robot arm of the slave robot in relation to the enlargement / reduction of the virtual operation plate and the operation distance of the operation lever of the master robot in a state in which the virtual operation plate is enlarged - a scaling control unit.

In the present invention, the display member may include a stereoscopic display device.

According to another aspect of the present invention, there is provided a surgical endoscope comprising: a surgical endoscope for imaging a surgical site; And a robot arm for driving the robot arm with multiple degrees of freedom, a display member for displaying an image photographed through the surgical endoscope; And an operation implementing signal for performing an actual operation corresponding to the operation operation by displaying a virtual operation board on the display member and operating the virtual operation board of the robot arm displayed on the display member And a virtual manipulation plate implementing unit for generating a manipulation robot manipulating the manipulation robot.

In the present invention, the virtual operation panel implementing unit may include: a virtual operation panel display unit displaying the virtual operation panel on the display member; A motion matching unit for determining whether the robot arm displayed on the display member performs an operation of operating the virtual operation board; And an operation implementation signal generator for generating an operation implementation signal for performing an actual operation corresponding to the operation operation of the virtual operation board performed.

In the present invention, when the robot arm performs predetermined predetermined operations, a virtual operation panel activation signal for displaying a virtual operation panel on the display member may be generated.

In the present invention, the robot arm may further include a reaction force providing unit for providing a predetermined reaction force to the operation lever of the master robot when the robot arm performs an operation of operating the virtual operation board displayed on the display member.

In the present invention, an operation of controlling the ratio of the operation distance of the robot arm of the slave robot in relation to the enlargement / reduction of the virtual operation plate and the operation distance of the operation lever of the master robot in a state in which the virtual operation plate is enlarged - a scaling control unit.

In the present invention, the virtual operation panel display unit may be configured to control operation of the surgical endoscope or the robot arm, selection of a robot arm to be operated using the master robot among the plurality of robot arms, Wherein at least one of setting of an operation scale relationship between the master robot and the slave robot, pause and resumption of the slave robot, operation control of an anesthesia machine, on / off of a CO2 insulator, and gas pressure control On the virtual operation panel can be displayed.

In the present invention, the virtual operation board may include an information display unit for displaying biometric information of the subject and / or medical image information of the subject.

Here, the medical image information may be reconstructed based on X-ray images, computed tomography (CT) images, magnetic resonance imaging (MRI) images, and computed tomography (CT) Image data or a numerical model.

Here, the slave robot may further include a bio-information measuring unit including at least one of a body temperature measuring module, a pulse measuring module, a respiration measuring module, a blood pressure measuring module, an electrocardiogram measuring module, etc., Information can be displayed on the information display section.

Here, at least a part of the biometric information and the medical image information may be fixed and displayed at a predetermined position on the display member.

In the present invention, the display member may include a stereoscopic display device.

According to still another aspect of the present invention, there is provided a control method of a surgical robot system including a master robot and a slave robot, comprising: displaying an image photographed through a surgical endoscope of the slave robot on a display member of the master robot; Generating a virtual manipulation plate activation signal; Displaying an image photographed through the surgical endoscope on a display member of the master robot and the virtual operation panel superimposed on the display member; Performing a manipulation of the robot arm of the slave robot displayed on the display member to manipulate the virtual manipulation plate; And a step of actually performing an operation operated by the robot arm on the virtual operation board.

In the present invention, the step of generating the predetermined virtual manipulation plate activating signal may include a step of, when the operator holds the manipulation lever of the master robot, The predetermined virtual operation plate activation signal can be generated.

Here, the predetermined operation may include an operation of pressing or holding a virtual operation panel activation signal generation button formed on an operation lever of the master robot, an operation of rotating the robot arm in the air, An operation of linearly moving the robot arm in a predetermined direction, an operation in which the robot arm draws a predetermined mark in the air, and an operation in which the robot arm presses and holds a predetermined icon displayed on the display member.

In the present invention, the virtual operation plate may include a function operation unit for controlling the operation of the surgical endoscope or the robot arm.

Here, when the operation operated by the robot arm of the slave on the virtual operation board is the operation of operating the functional operation unit, the operation performed by the robot arm of the slave on the virtual operation board is actually performed, A control operation for the robot arm or the surgical endoscope of the slave robot may be performed.

In the present invention, the virtual operation board may include an information display unit for displaying biometric information of the subject and / or medical image information of the subject.

Here, when the operation operated by the robot arm of the slave on the virtual operation board is an operation of operating the information display unit, the operation performed by the robot arm of the slave on the virtual operation board is actually performed, A control operation for the information display unit shown in the display member can be performed.

Here, when the operation operated by the robot arm of the slave on the virtual operation board is an operation of moving at least a part of the information display part to a specific position on the display member, at least part of the biometric information and the medical image information And can be fixedly displayed at a predetermined position on the display member.

In the present invention, the step of superimposing and displaying the image photographed through the surgical endoscope on the display member of the master robot and the virtual operation plate may be performed by the virtual operation plate from the surgical site displayed on the display member, And can be displayed with a certain degree of floating on the user side of the member.

In the present invention, when the robot arm of the slave robot displayed on the display member performs the operation of manipulating the virtual manipulation plate, a predetermined reaction force is provided to the manipulation lever of the master robot .

According to still another aspect of the present invention, there is provided a surgical robot system comprising: a display unit that displays a program of instructions executable by a digital processing apparatus to perform a control method of a surgical robot system described in any one of the above aspects, A recording medium on which a program is recorded is provided.

According to the present invention, it is possible to easily operate an information system such as a whole surgical robot system and various kinds of biometric information, medical image information, etc. in a single system, thereby achieving an effect of enabling a single operation. In addition, the single operation as in the technique of the present invention can increase the attention of the operator to improve the accuracy of the operation, shorten the operation time, increase the efficiency of the operation, Can be provided incidentally.

In addition, by applying the haptic interface to the surgical robot system, it is possible to enhance the accuracy and intuitiveness of the operation input of the operator, and to perform more various operation functions more intuitively by applying the augmented reality interface.

1 is a plan view showing the entire structure of a surgical robot system according to an embodiment of the present invention.
2 is a perspective view showing a master robot of the surgical robot system of FIG.
3 is a block diagram schematically showing the configuration of a master robot and a slave robot according to an embodiment of the present invention.
FIG. 4 is a block diagram schematically showing a configuration of a virtual operation plate implementing unit provided in the master robot of FIG. 3. FIG.
5 is a view showing a screen display mode in which an image photographed by a laparoscope is output through a display member.
FIG. 6 is a view showing a screen display mode in which a virtual operation panel is superimposed on the display member of FIG. 5 to be displayed.
7 is a flowchart schematically showing a control method of a surgical robot system according to an embodiment of the present invention.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings, which will be readily apparent to those skilled in the art to which the present invention pertains. The present invention may be embodied in many different forms and is not limited to the embodiments described herein.

Herein, the present invention is a technical idea that can be used universally for surgeries using surgical endoscopes (for example, laparoscopic, thoracoscopic, arthroscopic, non-rigid, etc.), but in describing the embodiments of the present invention, For convenience, a laparoscope is used as an example.

FIG. 1 is a plan view showing an overall structure of a surgical robot system according to an embodiment of the present invention, and FIG. 2 is a perspective view showing a master robot of the surgical robot system of FIG.

1 and 2, the surgical robot system 1 includes a slave robot 200 for performing surgery on a patient lying on the operating table, a master robot 100 for allowing the operator to remotely operate the slave robot 200, . The master robot 100 and the slave robot 200 are not necessarily separated from each other by a separate device that is physically independent and may be integrated into one unit.

The master robot 100 includes an operation lever 110 and a display member 120 and the slave robot 200 includes a robot arm 210 and a laparoscope 220.

In detail, the master robot 100 is provided with an operation lever 110 so that the operator can grasp and operate the operator's hands, respectively. The manipulation lever 110 may be implemented by two or more handles as illustrated in FIGS. 1 and 2. The manipulation lever 110 may be manipulated by a slave robot 200 through a wired or wireless communication network, And the robot arm 210 is controlled. That is, the operation of the robot arm 210, such as the movement, rotation, cutting, etc., of the robot arm 210 can be performed by operating the handle of the operator.

For example, the operator can manipulate the slave robot arm 210, the laparoscope 220, and the like using a handle-shaped operation lever. Such an operation lever may have various mechanical configurations according to its operation mode, and may include a master handle for operating operations of the slave robot arm 210 and the laparoscope 220 and the like, and a master robot Such as various input devices such as a joystick, a keypad, a trackball, and a touch screen, which are added to the slave robot 100, for operating the robot arm 210 and / or other surgical equipment of the slave robot 200 . Here, the operation lever 110 is not limited to the shape of the handle, and can be applied without any limitation as long as it can control the operation of the robot arm 210 through a network such as a wired or wireless communication network.

An image photographed through the laparoscope 220 is displayed as an image image on the display member 120 of the master robot 100. In addition, a predetermined virtual operation plate may be displayed on the display member 120 together with the image taken through the laparoscope 220, or may be independently displayed. The arrangement, configuration, and the like of the virtual operation plate will be described in detail with reference to related drawings hereinafter.

Here, the display member 120 may be constituted by one or more monitors, and information necessary for operation may be displayed on each monitor individually. Although FIGS. 1 and 2 illustrate the case where the display member 120 includes three monitors, the number of monitors can be variously determined according to the type and kind of information required to be displayed.

Meanwhile, the slave robot 200 may include one or more robot arms 210. Generally, a robot arm has a function similar to a human arm and / or wrist, and means a device capable of attaching a predetermined tool to a wrist part. In the present specification, the robot arm 210 may be defined as a concept including both upper and lower components such as upper arm, lower arm, wrist, and elbow, and a surgical instrument coupled to the wrist. The robot arm 210 of the slave robot 200 may be configured to be driven with multiple degrees of freedom. For example, the robot arm 210 may include a surgical instrument inserted into a surgical site of a patient, a swinging drive unit that rotates the surgical instrument in a yaw direction according to a surgical position, a pitch direction , A pitch driving part for rotating the surgical instrument, a feed driving part for moving the surgical instrument in the longitudinal direction, a rotation driving part for rotating the surgical instrument, and a surgical instrument driving part installed at the end of the surgical instrument for cutting or cutting the surgical lesion. . However, the configuration of the robot arm 210 is not limited thereto, and it should be understood that these examples do not limit the scope of the present invention. Here, a detailed description of the actual control process, such as rotation and movement of the robot arm 210 in the corresponding direction by operating the operation lever 110 by the operator, will be omitted.

The slave robot 200 may be used more than once to perform a surgery on a patient and the laparoscope 220 for displaying a surgical site through a display member 120 as an image image may be implemented as an independent slave robot 200 It is possible. Further, as described above, the embodiments of the present invention can be widely used in surgeries in which a variety of surgical endoscopes other than laparoscopes (e.g., thoracoscopic, arthroscopic, non-circumferential, etc.) are used.

3 is a block diagram schematically showing the configuration of a master robot and a slave robot according to an embodiment of the present invention.

3, the master robot 100 includes an image input unit 130, a screen display unit 140, a dark operation unit 150, an operation signal generation unit 160, a virtual operation plate implementation unit 170, and a control unit 180 ). The slave robot 200 includes a robot arm 210 and a laparoscope 220. Although not shown in FIG. 3, the slave robot 200 may further include a bio-information measuring unit for measuring and providing bio-information about the patient.

Here, in the surgical robot system 1 according to the embodiment of the present invention, when a predetermined operation is performed using the robot arm 210 during the operation, the virtual operation panel is displayed on the display member 120 in superposition with the actual image, When the virtual operation plate is operated by operating the robot arm 210 displayed on the display member 120, the corresponding actual operation is performed.

In detail, in an endoscope-based surgical robot system of a master-slave structure, a practitioner generally grasps the two operation levers of the master robot with both hands and controls the movement of the robot arm of the slave robot. However, during such a robot operation procedure, it is necessary to input various system operation commands other than the motion of the robot arm of the slave robot. For example, if the position of the endoscope camera, the posture change, the magnification / reduction of the field of view, the setting of the scale relationship between the master robot and the slave robot, and the system having three or more slave robot arms, Selection of the two robot arms to be controlled, connection / disconnection of the two robot arms to be controlled, pause of the slave robot, and operation control of the electric squeezer are required.

In order to input such a system operation command, conventionally, when inputting a system operation command is required during operation, the operator temporarily stops the surgical robot system, and either one of the hands manipulating the master robot, or one of them, The user presses a button switch to perform the operation, and the corresponding command is input. In this case, there is a problem that the operator 's attention is dispersed psychologically since it is necessary to temporarily suspend the surgical operation and physically remove the hand from the master robot and transfer the attention to the other manipulator.

In order to solve such a problem, a voice command is input from an operator, and a device performing a function by recognizing the contents of the command is used, or a change in the posture, that is, a gesture is recognized using a sensor tool attached to the head of the operator A device has been developed to perform this function. In such a case, it is not necessary to stop the system or to release the operation lever of the master robot, but there is still a problem that the operator's attention is dispersed psychologically. In the case of such a voice input or a head gesture input method, there is a possibility that accurate recognition can not be performed during command recognition (speech recognition through computer signal processing calculation or pattern recognition of operation signal). In particular, in one case of voice input method, in order to prevent the error, after the input of voice, the system is reconfirmed through the synthetic voice transmission in the system, and the operation is performed in such a manner that a few seconds between consecutive commands are refused, There was a difficulty in performing endoscopic manipulation.

In addition, the current surgical robot system requires only a relatively small number of manipulation command inputs. However, according to the development trend of the intelligent system, various kinds of real-time biometric information and various medical image data Are integrated into a surgical robot system environment. Therefore, a large number of input operation commands for reading and using such various information are required. In some commercialized systems, functions for selecting the biometric information of the patient to be displayed separately on the operator's display screen have been implemented. However, since the function for allowing the operator to control the biometric information is not implemented, Surgical assistant is needed. Accordingly, it is an object of the present invention to provide a surgical robot that minimizes distraction of attention by not only requiring operation of a surgical robot system, but also various kinds of operation of various integrated information systems, There is a need for an improved interface for single surgery that allows for quick and intuitive manipulation.

To this end, in the surgical robot system 1 according to the embodiment of the present invention, when a predetermined operation is performed using the robot arm 210 during the operation, the virtual operation panel is displayed on the display member 120 in superposition with the actual image , When the virtual operation plate is operated by operating the robot arm 210 displayed on the display member 120, the corresponding actual operation is performed.

Referring again to FIG. 3, each component of the master robot 100 will be described in detail.

The image input unit 130 receives an image photographed through a camera provided in the laparoscope 220 of the slave robot 200 through a wired or wireless communication network.

The screen display unit 140 outputs visual images corresponding to the images received through the image input unit 130 as visual information. In addition, when biometric information of the subject is input from the slave robot 200, the screen display unit 140 can output more information corresponding to the biometric information. Further, the screen display unit 140 may further output the patient-related image data (for example, an X-ray image, a CT image, an MRI image, and the like) for the surgical site. Here, the screen display unit 140 may be implemented in the form of a display member (see 120 in FIG. 2), and an image processing process for causing the received image to be output as an image image through the screen display unit 140 may be performed by a control unit 180), a virtual operation panel implementing section 170 or an image processing section (not shown).

Here, the screen display unit 140 (i.e., the display member 120 of FIG. 2) may be provided as a stereoscopic display device. In detail, the stereoscopic display device applies depth information to a two-dimensional image by applying a stereoscopic technique, and uses an image display device (not shown) to allow the observer to feel three- Quot; The surgical robot system 1 according to an embodiment of the present invention includes a stereoscopic display device as a screen display unit to provide a more realistic virtual environment to the user, which will be described in detail later.

The arm control unit 150 is a means for allowing the operator to operate the position and function of the robot arm 210 of the slave robot 200. 2, the shape of the arm control unit 150 is not limited thereto, but may be modified into various shapes for achieving the same purpose. . Further, for example, a part of the finger may be formed in a handle shape and the other part may be formed in a different shape such as a clutch button, and a finger insertion tube or insertion More rings may be formed.

The manipulation signal generating unit 160 generates a manipulation signal corresponding to the manipulation of the arm manipulation unit 150 by the operator in order to move the robot arm 210 and / or the laparoscope 220, And transmits it to the slave robot 200. The operation signal can be transmitted and received through a wired or wireless communication network as described above.

The virtual operating plate implementing section 170 displays a virtual operating panel on the display member (see 120 in FIG. 2), and the robot arm 210 displayed on the display member (see 120 in FIG. 2) And generates an operation implementation signal for performing an actual operation corresponding to the operation operation. The specific functions, various detailed configurations, and the like of the virtual operation plate implementing unit 170 will be described in detail below with reference to related drawings.

The control unit 180 controls the operation of each component so that the above-described functions can be performed. The control unit 180 may convert the image input through the image input unit 130 into an image to be displayed through the screen display unit 140. The control unit 180 transmits the image input from the image input unit 130 to the virtual operation panel implementing unit 170 and executes the operation implementing signal generated by the virtual operation panel implementing unit 170 It is possible.

FIG. 4 is a block diagram schematically showing a configuration of a virtual operation plate implementing unit provided in the master robot of FIG. 3. FIG.

4, the virtual operation plate implementation unit 170 of the master robot 100 of the surgical robot system 1 according to the embodiment of the present invention includes a virtual operation panel display unit 171, a motion matching unit 173, An operation implementation signal generation unit 175, a reaction force output unit 177, and an operation-scaling control unit 179. One or more components included in the virtual operation board implementing section 170 may be implemented in the form of a software program by a combination of program codes.

In detail, the virtual operation panel display section 171 serves to display a virtual operation panel on the display member 120. [ Such a virtual operation panel (see 300 in Fig. 6) includes a function control section (see 310 in Fig. 6) and an information display section (see 320 in Fig. 6).

Here, the virtual operation plate (see 300 in Fig. 6) may be an operation plate of a predetermined shape existing on the virtual three-dimensional space. As described above, the display member 120 may be provided as a stereoscopic display device. The virtual operation plate (see 300 in FIG. 6) may be a three-dimensional plate-like or cubic structure displayed on the stereoscopic display device. 6) is a virtual structure existing only on the display member 120, not on an object existing in the real world, but on the display member 120 in the form of a small structure of four millimeters in size. The site can be displayed to float upright above the air. The area where the virtual operating plate (see 300 in FIG. 6) is normally located is the area of the robot arm 210 (or the upper arm of the robot arm, the lower arm, the wrist, the elbow, etc.) The operator often manipulates the virtual manipulation plate (see 300 in FIG. 6) with the actual robot arm 210, but the surgical site is not affected by the movement of the surgical instrument. So that safety can be guaranteed. The configuration of such a virtual operation plate (see 300 in FIG. 6) will be described in detail in FIGS. 5 and 6. FIG.

Here, the virtual operation panel display unit 171 performs an operation of pressing or holding a predetermined button provided on the master robot 100, or when the robot arm 210 displayed on the display member 120 is operated The virtual operation panel can be controlled to be activated on the display member 120 when the virtual operation panel activation signal is generated by performing the predetermined operation.

For example, the master robot 100 may be provided with a separate virtual operation panel activation signal generation button (not shown) in the operation lever 110. When the user holds the operation lever 110, the operation lever 110 (Not shown) of the virtual operation panel on the display member 120 by pressing or holding the virtual operation panel activation signal generating button (not shown) of the virtual operation panel.

Or, if the surgical instrument provided in the robot arm 210 rotates in the air or moves linearly in a certain direction at a predetermined speed or in a predetermined specific operation (for example, when the surgical instrument is in a predetermined "Z" A virtual operation panel activation signal is generated when such operation is performed, and a virtual operation panel activation signal is generated and displayed on the display member 120 So that the virtual operation panel is displayed at a desired size.

Or when the minimized virtual operation panel on the display member 120 is displayed in the form of an icon and the robot arm 210 displayed on the display member 120 presses or holds the minimized virtual operation panel in the icon form The virtual operation panel activation signal may be generated by the virtual operation panel display unit 120 so that the virtual operation panel is displayed on the display member 120. [

That is, in the present specification, the activation of the virtual operation board may mean that the virtual operation board that has not been displayed on the display member 120 is displayed, and the virtual operation that has been displayed in the minimized state on the display member 120 It may be said that the plate is enlarged and displayed to a desired size.

Here, the virtual operation panel display unit 171 can display a virtual operation panel superimposed on an image displayed on the display member 120, that is, an image photographed through the laparoscope 220. As described above, the virtual manipulation plate can be superimposed and displayed on the image photographed through the laparoscope 220 by using augmented reality. Augmented reality is a technique of superimposing virtual objects on the real world that the user sees. The augmented reality is a mixed reality (MR) system because it displays the virtual world combined with the additional information in real- . A detailed description of such augmented reality will be omitted herein.

Next, the motion matching unit 173 performs a role of determining whether the robot arm 210 displayed on the display member 120 performs the operation of operating the virtual operation board. Specifically, while the movement of the robot arm 210 is an actual movement, the virtual operation plate operated through the movement of the robot arm is not actually present, but is an image that is simply being displayed on the display member 120. However, it is a feature of the present invention that when a motion for operating the virtual operation board is taken through the motion of the robot arm, an operation corresponding to the motion is actually performed. Therefore, in order to perform such operation, the motion matching unit 173 determines whether or not the robot arm 210 displayed on the display member 120 performs the operation of operating the virtual operation board. That is, if the movement of the robot arm 210 displayed on the display member 120 is a motion to press or hold a predetermined button on the virtual operation plate, the actual movement of the robot arm 210 is performed on the air, Is actually performed. In other words, the motion matching unit 173 determines whether or not the motion of the robot arm 210 is in a state in which the motion of the robot arm 210 is detected by, for example, pressing a predetermined button on the virtual operation plate, rotating the dial on the virtual operation plate, Off, or to move a specific portion on the virtual operation board to another area.

Next, the operation implementation signal generation unit 175 generates an operation implementation signal for performing an actual operation corresponding to the operation operation of the performed virtual operation board. More specifically, the motion matching unit 173 analyzes the operation of the robot arm 210 displayed on the display member 120, so that the operation of the robot arm 210 displayed on the display member 120 is controlled by the virtual operation plate The operation implementation signal generation unit 175 generates an operation implementation signal for actually performing the operation.

For example, when the motion matching unit 173 determines that the operation of the robot arm 210 displayed on the display member 120 is an operation of pressing or holding the laparoscopic movement button on the virtual operation plate, (175) generates a laparoscopic movement signal and transmits it to the controller (180). The control unit 180 controls the laparoscope 220 of the slave robot 200 to actually move the laparoscope 220 by a predetermined distance.

Or when the operation of the robot arm 210 displayed on the display member 120 is an operation of pressing or holding the MRI image enlargement button on the virtual operation plate as a result of the determination by the motion matching unit 173, Generates an MRI image enlargement signal, and transmits it to the controller 180. And the control unit 180 actually displays the magnified MRI image on the display member 120. [

Meanwhile, the virtual operation plate implementing unit 170 of the surgical robot system 1 according to an embodiment of the present invention performs an operation of manipulating the virtual operation board displayed on the display member 120 by the robot arm 210 A reaction force output unit 177 for providing the reaction lever 110 of the master robot 100 with a predetermined reaction force corresponding to the operation.

In detail, the operator can actually move the robot arm of the slave robot through the master robot to operate the switch of the virtual operation board, or to perform an operation on the virtual operation board itself (for example, contact with the virtual operation board, The reaction force output section 177 outputs a reaction force when an actual switch is pressed or a contact feeling when the contact is made with the actual operation plate, A haptic sense such as a gravity sense that moves an object having a predetermined weight when it is moved is synchronized with the image and is outputted to the operation lever 110, realizing intuitive and intuitive interaction. The reactive force output unit 177 of the present invention provides an intuitive reaction force when the operator manipulates the virtual switch in the virtual operation plate, thereby obtaining an effect of intuitively confirming that the function is correctly selected .

The virtual operation plate implementation unit 170 of the surgical robot system 1 according to an embodiment of the present invention may be configured such that the virtual operation plate implementation unit 170 of the slave robot 200, And an operation-scaling control unit 179 for controlling the ratio of the working distance of the arm 210. [

As described above, the virtual operation plate (see 300 in Fig. 6) may be an operation plate of a predetermined shape existing on the virtual three-dimensional space. Here, the virtual operation plate (see 300 in FIG. 6) may be a three-dimensional plate-like or cubic structure displayed on the stereoscopic display device. This virtual manipulation plate (see 300 in FIG. 6) can be displayed to float in the vertical upper air space of the surgical site on the display member 120 as a small structure of the size of four millimeters in size. Then, when a virtual operation panel activation signal is generated in a state in which a very small virtual operation panel (see 300 in FIG. 6) is being displayed, the virtual operation panel is activated and enlarged to an appropriate size on the display member 120 Is displayed.

Here, the operation-scaling control unit 179 displays only the virtual operation panel (see 300 in FIG. 6) when the virtual operation panel is normally displayed while minimizing the virtual operation panel And controls the operation scaling of the master robot and the slave robot in a state in which the virtual operation panel (see 300 in FIG. 6) is enlarged in this way. This is a function separate from the concept of master-slave motion scaling during general operation, and is a function provided for controlling the virtual operation plate when the virtual operation plate is activated and enlarged. The operation-scaling controller 179 of the present invention can eliminate the possibility that the movement of the robot arm of the slave robot for the virtual manipulation plate operation affects the affected part during surgery.

On the other hand, the area where the virtual manipulation plate is displayed should be a space that is securely separated from the actual operation site. In general, in all surgical operations, the space at the upper end of the endoscope at the center of the endoscopic screen is empty for securing the visual field of the surgical operation. Therefore, a cubic space is defined in the entire space coordinate system , The virtual operation board can be arranged in the space. Further, in order to construct a safer surgical space, it is possible to allow the operator to directly designate a safe space as a position on the space where the virtual operation plate is to be positioned before or during surgery. In addition, while the control system constantly observes the movement of the robot arm during surgery, the space that the robot arm repeatedly passes through (the obstacle or the body part requiring attention) Is automatically determined to be not in the path), so that the position can be automatically adjusted adaptively so that the virtual operation panel can always be positioned at the safest position.

Meanwhile, in the surgical robot system 1 according to the embodiment of the present invention, when the virtual operation plate is operated by actually moving the robot arm 210 displayed on the display member 120 as described above, Has been described as being performed, the spirit of the present invention is not limited thereto. That is, when a virtual operation panel is displayed on the display member 120, a virtual mouse or a virtual pointer or a virtual finger or the like is displayed on the display member 120 at the same time, and in this state, It is also possible that the robot arm 210 does not actually move but the virtual manipulation plate is operated while the displayed virtual mouse / pointer / finger moves.

FIG. 5 is an example of a screen display mode in which an image photographed by a laparoscope is output through a display member, and FIG. 6 is a diagram illustrating a screen display mode in which a virtual operation panel is superimposed on a display member of FIG.

First, as shown in FIG. 5, the operator performs an operation while confirming the image of the surgical site photographed through the laparoscope 220 through the display member 120. At this time, the first robot arm 211, the second robot arm 212, and the like are displayed on the screen output through the display member 120. 5 is a state before the virtual operation plate is activated.

In this state, when the operator operates the operation lever 110 of the master robot 100 to move the robot arm 210 of the slave robot 200 to perform a predetermined operation to generate a virtual operation plate activation signal, A virtual operating panel 300 as shown in FIG. The virtual operation board 300 includes a function operation unit 310 and an information display unit 320. [

More specifically, the function control unit 310 controls operation of the slave robot 200 such as operation of the laparoscope 220, selection of a robot arm to be operated by using the master robot 100 among the plurality of robot arms of the slave robot 200, And control functions. The function control unit 310 may include a robot arm selection button 311, a laparoscopic movement button 312, an electric dimmer control button 313, and the like. The positional shift, the posture change, the field of view enlargement / reduction of the laparoscope 220 using the function control unit 310, the setting of the operation scale relationship between the master robot 100 and the slave robot 200, In the case of a system having three or more robot arms 210, the two operation levers 110 of the master robot 100 can select and connect / disconnect two robot arms 210 to be operated at present, The operation control of the electric decanter, the operation control of the anesthesia machine, the ON / OFF of the CO2 insufflator, and the input of system operation commands such as the gas pressure control.

In addition, the function operation unit 310 can control not only the operation of the surgical robot system itself but also other environments inside the operation room, so that the operator can operate all the apparatuses and environments in the operating room alone while holding the operation handle of the master robot. Control. For example, it is also possible to control the illuminance in the operating room, control the brightness of the laparoscope, or control the height and posture of the surgical bed by using the function control unit 310.

For example, when the operator operates the operation lever 110 of the master robot 100 to move the robot arm 210 of the slave robot 200, the robot arm 210 displayed on the display member 120 is moved When the laparoscopic movement button 312 of the function control unit 310 of the operation plate 300 is pressed or held, the laparoscope 220 moves in the button direction. Although the laparoscopic movement button 312 is illustrated as being provided as a four-way button in the drawing, the concept of the present invention is not limited thereto and may be provided in various forms such as a joystick, a trackball, will be.

Or the operator operates the operation lever 110 of the master robot 100 to move the robot arm 210 of the slave robot 200 so that the robot arm 210 displayed on the display member 120 is moved to the virtual operation plate The operation and manipulation of the electric groomer are performed when the electric groomer control button 313 of the function control unit 310 of the electric power source 300 is pressed or held. That is, various operations such as ON / OFF of the electric groomer and control of the output of the electric groomer may be performed through the operation of the robot arm 210.

On the other hand, the information display unit 320 may display one or more real-time biometric information about the patient or medical image information (e.g., X-ray image, CT image, MRI image, etc.) of the patient on the operation site. The information display unit 320 may include a biometric information display unit 321, a medical image information display unit 322, and a selection button unit 323. The ON / OFF of each of the related images of the patient displayed on the medical image information display unit 322 is displayed on the biometric information display unit 321 through the operation of the selection button unit 323, ), Enlargement / reduction, position movement, and the like.

One or more pieces of real-time biometric information such as an electrocardiogram, a body temperature, a pulse, respiration, a blood pressure, etc. of the patient can be output to the biometric information display unit 321, It is possible.

In the case of such a bio-information display unit 321, various real-time numerical values or waveform information such as a multi-channel electrocardiogram and real-time blood pressure at various sites can be selected as a whole or a part thereof, Portion of the image. For example, a specific signal waveform requiring continuous attention among the blood pressure signals collected from various parts of the heart during heart surgery is taken off the display panel of only one channel on the operation panel, and is taken to another specific part of the entire endoscopic screen Thereby allowing the operator to reconstruct the information urban form within the display member 120. [0064]

In order to provide the biometric information to the master robot 100, the slave robot 200 may include a body-part measuring unit including at least one of a body temperature measuring module, a pulse measuring module, a respiration measuring module, a blood pressure measuring module, (Not shown). The biometric information measured by each module may be transmitted from the slave robot 200 to the master robot 100 in the form of an analog signal or a digital signal and the master robot 100 transmits the received biometric information to the display member 120, Can be displayed through the virtual operating plate 300 of the display unit 300. [

The medical image information display unit 322 may display relevant image data of the patient on the surgical site such as an X-ray image, a CT image, and an MRI image. Further, the medical image information display unit 322 may display reconstructed three-dimensional image data or a numerical model based on the CT image data. Such medical image data can also be scrolled, zoomed in / out, and changed in three-dimensional visual rotation in a manner similar to various multi-touch gesture methods or three-dimensional mouse and three-dimensional haptic interfaces depending on the characteristics of two- and three- It can be easily performed and read.

On the other hand, by utilizing the characteristic of the present invention capable of interacting, the information display unit 320 does not merely display biometric information or a medical image, but selects only information necessary for a specific operation work process, You can do it. For example, physiologically, in a process of carefully observing a momentary change in a patient's condition, an electrocardiogram of a specific channel and blood pressure (waveform and) values of a specific region can be separated and fixedly viewed on the screen. Or, during a work process that requires anatomical structure information, you can select a specific scene from a medical image and fix it on one side of the screen, and proceed with the operation while referring to it. In this manner, biometric information and medical images can be selected for cooking, thereby making it possible to obtain an effect of further improving ease of operation of the operator.

Hereinafter, a method of controlling a surgical robot system according to an embodiment of the present invention will be described with reference to related drawings. 7 is a flowchart schematically showing a control method of a surgical robot system according to an embodiment of the present invention.

7, an image captured through the laparoscope 220 of the slave robot 200 is displayed on the display member 120 of the master robot 100. In the control method of the surgical robot system according to the embodiment of the present invention, The image captured through the laparoscope 220 on the display member 120 of the master robot 100 and the image captured through the virtual operation panel 300 (step S120) The robot arm 210 displayed on the display member 120 through the operation of the master robot 100 operates to operate the operation member on the virtual operation board 300 (Operation S140) and the operation performed by the robot arm 210 of the slave robot 200 on the virtual operation board 300 is actually performed (operation S150).

First, an image photographed through the laparoscope 220 of the slave robot 200 is displayed on the display member 120 of the master robot 100 (step S110). FIG. 5 shows a screen display form in which an image photographed by the laparoscope 220 is output through the display member 120. FIG.

Next, it is determined whether or not a virtual operation panel activation signal is generated (step S120). In detail, the virtual operating plate implementing section 170 of the master robot 100 performs an operation of pressing or holding a predetermined button provided on the master robot 100, or performing a pressing operation of the robot arm 100 displayed on the display member 120 (210) has performed a predetermined set operation. Here, the predetermined motion may be a motion that is possible regardless of a surgical operation, for example, the robot arm 210 rotates in the air or linearly moves in a constant direction over a predetermined speed, The minimized virtual operation panel on the display member 120 may be displayed in a predetermined icon form and the robot arm 210 displayed on the display member 120 may be an operation of pressing or holding the minimized virtual operation panel. When the robot arm 210 performs the preset predetermined operation, the virtual operation panel implementing unit 170 generates the virtual operation panel activation signal.

Next, when the virtual operation plate activation signal is generated, the image photographed through the laparoscope 220 on the display member 120 of the master robot 100 is overlapped with the virtual operation plate 300 and displayed (step S130) do. In detail, when the virtual operation panel display unit 171 of the virtual operation panel implementing unit 170 of the master robot 100 receives the virtual operation panel activation signal, the image displayed on the display member 120, that is, the laparoscope 220 The virtual operation panel 300 can be superimposed and displayed on an image photographed through the virtual operation panel 300. The virtual manipulation plate 300 can be superimposed and displayed on an image photographed through the laparoscope 220 by using augmented reality. A screen display mode in which the virtual operation board 300 is superimposed and displayed on the display member 120 is shown in FIG.

The virtual operation board 300 includes a function operation unit 310 and an information display unit 320. [ More specifically, the function control unit 310 controls operation of the slave robot 200 such as operation of the laparoscope 220, selection of a robot arm to be operated by using the master robot 100 among the plurality of robot arms of the slave robot 200, And control functions. The function control unit 310 may include a robot arm selection button 311, a laparoscopic movement button 312, an electric dimmer control button 313, and the like. On the other hand, the information display unit 320 may display one or more real-time biometric information about the patient or medical image information (e.g., X-ray image, CT image, MRI image) of the patient on the operation site. The information display unit 320 may include a biometric information display unit 321, a medical image information display unit 322, and a selection button unit 323.

Next, the robot arm 210 displayed on the display member 120 operates the operation member on the virtual operation board 300 through the operation of the master robot 100 (step S140). This will be described in more detail as follows.

The operator operates the operation lever 110 of the master robot 100 to take an action that the robot arm 210 displayed on the display member 120 operates a predetermined operation member on the virtual operation board 300 . Here, the predetermined operating member may be one of the buttons of the function operating unit 310 or the information displaying unit 320. Or to rotate the image itself displayed on the information display unit 320, for example.

At this time, although the actual movement of the robot arm 210 is performed on the air, the movement of the robot arm 210 displayed on the display member 120 may be a motion to press or hold a predetermined button on the virtual operation board 300 have. The motion matching unit 173 determines whether or not the robot arm 210 displayed on the display member 120 performs an operation of operating the virtual operation board 300. [ That is, while the movement of the robot arm 210 is an actual movement, the virtual operation plate operated through the movement of the robot arm is not actually present but is an image that is simply displayed on the display member 120. However, in the present invention, when a motion for operating the virtual operation board is taken through the motion of the robot arm, an operation corresponding to the motion is actually performed. To this end, the motion matching unit 173 determines whether or not the robot arm 210 displayed on the display member 120 performs the operation of operating the virtual operation board.

Next, the operation performed by the robot arm 210 of the slave robot 200 on the virtual operation board 300 is actually performed (step S150).

In detail, the operation implementation signal generation unit 175 generates an operation implementation signal for performing an actual operation corresponding to the operation operation of the virtual operation board 300 performed. That is, when the motion matching unit 173 analyzes the operation of the robot arm 210 displayed on the display member 120, it is determined that the operation of the robot arm 210 displayed on the display member 120 is & The operation implementation signal generation unit 175 generates an operation implementation signal for actually performing the operation.

For example, when the motion matching unit 173 determines that the operation of the robot arm 210 displayed on the display member 120 is an operation of pressing or holding the laparoscopic movement button on the virtual operation plate, (175) generates a laparoscopic movement signal and transmits it to the controller (180). The control unit 180 controls the laparoscope 220 of the slave robot 200 to actually move the laparoscope 220 by a predetermined distance. Alternatively, when the operation of the robot arm 210 displayed on the display member 120 is an operation of pressing or holding the position change button or the zoom button of the laparoscopic view position on the virtual operation plate, Generates a laparoscopic field-of-view position movement or an enlargement / reduction signal, and transmits it to the controller 180. The control unit 180 controls the laparoscope 220 of the slave robot 200 so that the laparoscope 220 actually advances or retreats or enlarges / reduces the image itself.

Or when the operation of the robot arm 210 displayed on the display member 120 is an operation of pressing or holding the MRI image enlargement button on the virtual operation plate as a result of the determination by the motion matching unit 173, Generates an MRI image enlargement signal, and transmits it to the controller 180. And the control unit 180 actually displays the magnified MRI image on the display member 120. [

That is, in the case of a system having three or more slave robot arms, it is possible to set the operation scale relationship between the master robot and the slave robot in the conventional surgical robot system, such as selection and connection / disconnection of the robot arm to be controlled, In order to implement various functions, a separate operation panel is required. In order to operate such an operation panel, a hand holding the handle of the master robot is held and a button provided on the operation panel of the master robot is pressed, The operation had to proceed.

In contrast, in the surgical robot system according to an embodiment of the present invention, in order to move the laparoscopic image, a predetermined virtual manipulation plate activation signal is generated by operating the handle while holding the handle of the master robot, When the slave robot arm operates the handle of the master robot to display the operation plate on the screen display unit and press or hold the movement button of the virtual operation plate displayed on the screen display unit, the laparoscopic movement operation is actually performed will be.

In the control method of the surgical robot system according to the embodiment of the present invention, when the robot arm 210 of the slave robot 200 operates the virtual operation plate 300, a predetermined reaction force is applied to the master robot 100, (S160). ≪ / RTI >

In detail, the operator can actually move the robot arm of the slave robot through the master robot to operate the switch of the virtual operation board, or to perform an operation on the virtual operation board itself (for example, contact with the virtual operation board, The reaction force output section 177 outputs a reaction force when an actual switch is pressed or a contact feeling when the contact is made with the actual operation plate, A haptic sense such as a gravity sense that moves an object having a predetermined weight when it is moved is synchronized with the image and is outputted to the operation lever 110, realizing intuitive and intuitive interaction. The reactive force output unit 177 of the present invention provides an intuitive reaction force when the operator manipulates the virtual switch in the virtual operation plate, thereby obtaining an effect of intuitively confirming that the function is correctly selected .

Finally, when the robot arm 210 has performed predetermined predetermined operations, it is determined whether or not a virtual operation panel inactivation signal has been generated (operation S170). If the virtual operation panel inactivation signal is generated, The control panel 300 is deactivated and only the images photographed through the laparoscope 220 of the slave robot 200 are displayed on the display member 120 of the master robot 100.

According to the present invention, it is possible to easily operate an information system such as a whole surgical robot system and various kinds of biometric information, medical image information, etc. in a single system, thereby achieving an effect of enabling a single operation. In addition, the single operation as in the technique of the present invention can increase the attention of the operator to improve the accuracy of the operation, shorten the operation time, increase the efficiency of the operation, Can be provided incidentally. In addition, by applying the haptic interface to the surgical robot system, it is possible to enhance the accuracy and intuitiveness of the operation input of the operator, and to perform more various operation functions more intuitively by applying the augmented reality interface.

Although the present invention has been described with reference to the limited embodiments, various embodiments are possible within the scope of the present invention. It will also be understood that, although not described, equivalent means are also incorporated into the present invention. Therefore, the true scope of protection of the present invention should be defined by the following claims.

1: Surgical robot system 100: Master robot
110: Operation lever 120: Display member
200: Slave robot 210: Robot arm
220: Laparoscopic

Claims (31)

1. A master robot for controlling a slave robot including a surgical endoscope and a robot arm,
A display member for displaying an image taken through the surgical endoscope;
A virtual operation panel display unit for displaying a virtual operation panel on the display member;
A motion matching unit for determining whether the robot arm displayed on the display member performs an operation of operating the virtual operation board; And
And an operation implementation signal generator for generating an operation implementation signal for performing an actual operation corresponding to the operation operation of the virtual operation board performed,
Wherein the virtual manipulation plate includes an information display unit for displaying at least one of biometric information of the subject and medical image information of the subject,
Wherein the slave robot includes a bio-information measuring unit including at least one of a body temperature measuring module, a pulse measuring module, a respiration measuring module, a blood pressure measuring module, and an electrocardiogram measuring module,
And the information measured by the bio-information measuring unit is displayed on the information display unit. The method according to claim 1,
Wherein the virtual manipulation plate display unit superimposes the virtual manipulation plate on an image photographed through the surgical endoscope displayed on the display member. The method according to claim 1,
Wherein the virtual manipulation plate display unit displays the virtual manipulation plate on the display member when a preset virtual manipulation plate activation signal is received. The method according to claim 1,
Wherein the virtual operation panel display unit is configured to control operation of the surgical endoscope or the robot arm, selection of a robot arm to be operated using the master robot among a plurality of robot arms, output control of the electric arc generator, A function for performing at least one of setting of the operation scale relationship of the slave robot, pause and resumption of the slave robot, operation control of the anesthesia machine, ON / OFF of the CO2 insulator, and gas pressure control And displays an operation unit on the virtual operation board. The method according to claim 1,
Wherein the virtual operation panel display unit displays on the virtual operation panel an information display unit for displaying biometric information of the subject and / or medical image information of the subject. 6. The method of claim 5,
Wherein at least a part of the biometric information and the medical image information is fixed and displayed at a predetermined position on the display member. The method according to claim 1,
And a reaction force providing unit for providing the reaction lever corresponding to the operation to the operation lever of the master robot when the robot arm performs the operation of operating the virtual operation board displayed on the display member Master robot. The method according to claim 1,
And an operation-scaling control unit for controlling the ratio of the operation distance of the robot arm of the slave robot to the distance of operation of the operation lever of the master robot in the state in which the virtual operation plate is enlarged The master robot comprising: The method according to claim 1,
Wherein the display member comprises a stereoscopic display device. A surgical endoscope for imaging a surgical site; And
A slave robot including a robot arm driving with multiple degrees of freedom,
A display member for displaying an image taken through the surgical endoscope; And
Generating an operation implementation signal for performing an actual operation corresponding to the operation operation by displaying a virtual operation board on the display member and operating the virtual operation board of the robot arm displayed on the display member And a virtual operating plate implementing section,
Wherein the virtual manipulation plate includes an information display unit for displaying at least one of biometric information of the subject and medical image information of the subject,
Wherein the slave robot further comprises a bio-information measuring unit including at least one of a body temperature measuring module, a pulse measuring module, a respiration measuring module, a blood pressure measuring module, and an electrocardiogram measuring module,
And the information measured by the bio-information measuring unit is displayed on the information display unit. delete delete delete delete delete delete delete delete delete delete 1. A control method for a surgical robot system including a master robot and a slave robot,
Displaying an image photographed through a surgical endoscope of the slave robot on a display member of the master robot;
Generating a virtual manipulation plate activation signal;
Displaying an image photographed through the surgical endoscope on a display member of the master robot and the virtual operation panel superimposed on the display member;
Determining whether the robot arm of the slave robot displayed on the display member performs an operation of operating the virtual operation board; And
Generating an operation implementation signal for implementing an actual operation corresponding to an operation operation of the virtual operation board,
Wherein the virtual manipulation plate includes an information display unit for displaying at least one of biometric information of the subject and medical image information of the subject,
Wherein the slave robot includes a bio-information measuring unit including at least one of a body temperature measuring module, a pulse measuring module, a respiration measuring module, a blood pressure measuring module, and an electrocardiogram measuring module,
Wherein the step of superimposing and displaying the virtual manipulation plate includes displaying information measured by the biometric information measurement unit on the information display unit. 22. The method of claim 21,
The step of generating the virtual manipulation plate activation signal includes:
The robot arm performs a predetermined operation that is set by the operation of the master robot in a state in which the operator holds the operation lever of the master robot, Wherein the control unit controls the operation of the robot. 23. The method of claim 22,
The predetermined operation may include an operation of pressing or holding a virtual operation panel activation signal generation button formed on an operation lever of the master robot, an operation of rotating the robot arm in the air, Wherein the robot arm includes at least one of an operation for drawing a predetermined mark on the hollow space and an operation for pressing and holding the predetermined icon displayed on the display member by the robot arm. Control method of robot system. 22. The method of claim 21,
Wherein the virtual operation plate includes a function operation unit for controlling the operation of the surgical endoscope or the robot arm. 25. The method of claim 24,
When the operation operated by the robot arm of the slave on the virtual operation board is an operation of operating the function operation portion,
Wherein the step of generating the operation implementation signal generates a control operation implementation signal for the robot arm or the surgical endoscope of the slave robot. 22. The method of claim 21,
Wherein the virtual manipulation plate includes an information display unit for displaying biometric information of the subject and / or medical image information of the subject. 27. The method of claim 26,
When the operation operated by the robot arm of the slave on the virtual operation board is an operation of operating the information display section,
Wherein the step of generating the operation implementation signal generates a control operation implementation signal for the information display section shown in the display member. 27. The method of claim 26,
When the operation operated by the robot arm of the slave on the virtual operation board is an operation of moving at least a part of the information display part to a specific position on the display member,
Wherein at least a part of the biometric information and the medical image information is fixed and displayed at a predetermined position on the display member. 22. The method of claim 21,
Wherein the step of displaying an image photographed through the surgical endoscope on the display member of the master robot and the virtual operation panel are overlapped and displayed,
Wherein the virtual manipulation plate is displayed in a floating state from a surgical site displayed on the display member to a user side of the display member to a certain extent. 22. The method of claim 21,
Further comprising the step of providing a predetermined reaction force to the operation lever of the master robot when the robot arm of the slave robot displayed on the display member performs an operation of operating the virtual operation plate Control method of robot system. 29. A program for executing a control method of a surgical robot system according to any one of claims 21 to 30, wherein a program of instructions executable by the digital processing apparatus is tangibly embodied and can be read by a digital processing apparatus Recording medium.

Images