KR20100106834A - Surgical robot system using augmented reality and control method thereof - Google Patents

Abstract

PURPOSE: An operating robot system using augmented reality and a method for controlling the same are provided to facilitate the progress of an operation by displaying actual surgical tools and virtual surgical tools. CONSTITUTION: A master robot controls a slave robot(2) including a robot arm(3). A display unit(320) displays endoscopic images from a surgical endoscope. One or more arm manipulating unit(330) controls the robot arm. An augmented reality implementing unit(350) generates information related to virtual surgical tools. A manipulating signal generating unit(340) generates a manipulating signal for controlling the robot arm and transmitting the manipulating signal to the slave robot.

Description

Surgical robot system using augmented reality and control method

The present invention relates to surgery, and more particularly, to a surgical robot system using augmented reality and a control method thereof.

Surgical robot refers to a robot having a function that can replace a surgical operation performed by a surgeon. Such a surgical robot has the advantage of being capable of accurate and precise operation and remote surgery compared to humans.

Surgical robots currently being developed worldwide include bone surgery robots, laparoscopic surgery robots, and stereotactic surgery robots. The laparoscopic surgical robot is a robot that performs minimally invasive surgery using a laparoscope and a small surgical tool.

Laparoscopic surgery is an advanced surgical technique that involves surgery after inserting a laparoscope, an endoscope that looks into the belly and drills a hole about 1 cm in the navel area.

Recent laparoscopy is equipped with a computer chip to obtain a clearer and more magnified image than the naked eye, and has been developed so that any operation can be performed using specially designed laparoscopic surgical instruments while viewing the screen through the monitor.

Moreover, laparoscopic surgery has the same range of surgery as laparotomy, but has fewer complications than laparotomy, and can start treatment much faster after the procedure, and it has the advantage of maintaining the stamina or immune function of the patient. have. As a result, laparoscopic surgery is increasingly recognized as a standard surgery for treating colorectal cancer in the US and Europe.

The surgical robot system generally consists of a master robot and a slave robot. When the operator manipulates a manipulator (for example, a handle) provided in the master robot, a surgical tool coupled to the robot arm of the slave robot or held by the robot arm is operated to perform surgery.

The master robot and the slave robot are combined through a communication network to perform network communication. At this time, if the network communication speed is not fast enough, until the operation signal transmitted from the master robot is received by the slave robot and / or until the laparoscope image transmitted from the laparoscopic camera mounted on the slave robot is received by the master robot It takes a lot of time.

In general, it is known that the operation using the master robot and the slave robot is possible only when the network communication speed is less than 150ms. If the communication speed is further delayed, the operator's hand does not match the movement of the slave robot through the screen, which makes the operator very uncomfortable.

In addition, when the network communication speed between the master robot and the slave robot is slow, the operator performs the operation by consciously or predicting the movement of the slave robot shown on the screen in advance. This causes unnatural movements and, in severe cases, causes normal surgery to fail.

The present invention is to provide a surgical robot system and a control method using augmented reality to enable a smooth operation of the operator by displaying a real surgical tool and a virtual surgical tool using augmented reality (augmented reality).

In addition, the present invention is to provide a surgical robot system using augmented reality that can output a variety of information on the patient during surgery and its control method.

In addition, the present invention is to provide a surgical robot system using augmented reality and its control method that can be varied in the surgical screen display method to enable the operation of the operator smoothly according to the network communication speed between the master robot and the slave robot.

In addition, the present invention is to provide a surgical robot system using augmented reality and the control method that can be notified to the operator immediately by emergency processing the image input through the endoscope and the like.

In addition, the present invention using augmented reality to ensure that the long-term contact, such as the movement of the virtual surgical tool by the operation of the master robot can be detected in real time to the operator, the intuitive relationship between the virtual surgical tool and the long term position can be intuitively recognized. It is to provide a surgical robot system and a control method thereof.

In addition, the present invention is a surgical robot system using augmented reality to enable the operation progress using a variety of information can be presented in real time the relevant image data (for example, CT image, MRI image, etc.) of the patient to the surgical site And a control method thereof.

In addition, the present invention is to provide a surgical robot system using augmented reality that can maximize the real-time education by allowing the surgical robot system to be compatible and shared between the learner (learner) and trainer (trainer) and its control method. .

In addition, the present invention is to provide a surgical robot system using augmented reality and its control method to predict the progress and results of the actual surgical process in advance by using a three-dimensional modeled virtual organs.

Technical problems other than the present invention will be easily understood through the following description.

According to one aspect of the invention, there is provided a surgical robot system, a slave robot and a master robot using augmented reality.

According to an embodiment of the present invention, an interface mounted on a master robot that controls a slave robot including one or more robot arms, which corresponds to an image signal provided from a surgical endoscope. Virtual surgical tool information according to a user operation using a screen display unit for displaying an endoscope image, one or more arm manipulation units provided to control at least one robot arm, and a virtual surgical tool to be displayed through the screen display unit Provided is a master interface of a surgical robot including an augmented reality implementation to generate the.

The surgical endoscope may be one or more of laparoscopic, thoracoscopic, arthroscopic, parenteral, bladder, rectal, duodenum, mediastinal, cardiac.

The master interface of the surgical robot may further include an operation signal generation unit configured to generate an operation signal according to a user operation for controlling the robot arm and transmit the operation signal to the slave robot.

The master interface of the surgical robot may include a driving mode selection unit for designating a driving mode of the master robot and at least one of an endoscope image and a virtual surgical tool through a screen display corresponding to the driving mode selected through the driving mode selection unit. The control unit may further include a control unit.

The controller may control a mode indicator corresponding to the selected driving mode to be displayed through the screen display unit. The mode indicator may be predefined as one or more of a text message, a border color, an icon, a background color, and the like.

The slave robot may further include a biometric information measuring unit. The biometric information measured by the biometric information measuring unit may be displayed through the screen display unit.

The augmented reality implementation unit, a characteristic value calculation unit for calculating a characteristic value using at least one of the endoscope image and the position coordinate information of the actual surgical tool coupled to one or more robot arms, and virtual surgical tool information according to the user operation using the arm operation unit It may include a virtual surgical tool generating unit for generating a.

The characteristic value calculated by the characteristic value calculator may include one or more of a field of view (FOV), an enlargement ratio, a viewpoint, a viewing depth, a type of an actual surgical tool, a direction, a depth, and a bending angle of the surgical endoscope.

The augmented reality implementation unit transmits a test signal to the slave robot and receives a response signal corresponding to the test signal from the slave robot, and a master robot and a slave robot using a transmission time of the test signal and a reception time of the response signal. The apparatus may further include a delay time calculator configured to calculate a delay value for at least one of a network communication speed and a delay time in network communication.

The master interface may further include a control unit for controlling one or more of the endoscope image and the virtual surgical tool to be displayed through the screen display unit. Here, when the delay value is less than or equal to a preset delay threshold, the controller may control only the endoscope image to be displayed through the screen display unit.

The augmented reality implementation may further include an interval calculator configured to calculate an interval value between each surgical tool by using the position coordinates of the actual surgical tool and the virtual surgical tool displayed through the screen display unit.

The virtual surgical tool generating unit may process the virtual surgical tool not to be displayed through the screen display unit when the interval value calculated by the interval calculating unit is equal to or less than a preset interval threshold.

The virtual surgical tool generation unit may perform processing for one or more of semi-transparency control, color change, and outline thickness change for the virtual surgical tool in proportion to the interval value calculated by the interval calculator.

The augmented reality implementation unit may further include an image analyzer extracting feature information through image processing of the endoscope image displayed through the screen display unit. Here, the feature information may be at least one of color values of pixels of the endoscope image, position coordinates of the actual surgical tool, and manipulation shapes.

The image analyzer may output a warning request when an area or a quantity of pixels having a color value included in a preset color value range in the endoscope image exceeds a threshold. According to the warning request, one or more of displaying a warning message through the screen display unit, outputting a warning sound through the speaker unit, and stopping the display of the virtual surgical tool may be performed.

The master interface uses the position coordinate information of the virtual surgical tool included in the virtual surgical tool information generated by the virtual surgical tool generator and the position coordinate information of the actual surgical tool included in the characteristic value calculated by the characteristic value calculator. The apparatus may further include a network verification unit verifying a network communication state between the master robot and the slave robot.

The master interface may further include a network verification unit verifying a network communication state between the master robot and the slave robot by using positional coordinate information of the real surgical tool and the virtual surgical tool included in the feature information extracted by the image analyzer. Can be.

The network verification unit may further use at least one of a movement trajectory and a manipulation form of each surgical tool to verify the network communication state.

The network verification unit may verify the network communication state as a determination of whether the position coordinate information of the virtual surgical tool coincides with the position coordinate information of the previously stored actual surgical tool within an error range.

The network verification unit may output a warning request when the position coordinate information of the actual surgical tool and the position coordinate information of the virtual surgical tool do not match within an error range. According to the warning request, one or more of displaying a warning message through the screen display unit, outputting a warning sound through the speaker unit, and stopping the display of the virtual surgical tool may be performed.

The augmented reality implementation unit, an image analysis unit for extracting the feature information including the area coordinate information of the organ displayed through the surgical site or the endoscope image through the image processing for the endoscope image displayed through the screen display unit; The virtual surgical tool information and the area coordinate information are used to determine whether the virtual surgical tool is located at the rear side due to the overlapping of the area coordinate information, and when the overlap occurs, concealing the overlapped area of the shape of the virtual surgical tool. It may further include an overlap processing unit for processing.

The augmented reality implementation unit, the image analysis unit for extracting feature information including the coordinate information of the area of the organ displayed through the surgical site or the endoscope image through the image processing for the endoscope image displayed through the screen display unit and The virtual surgical tool may further include a contact recognition unit that determines whether the virtual surgical tool is in contact with the region coordinate information by using the virtual surgical tool information and the region coordinate information, and performs a contact warning process when the contact is generated.

The contact warning processing may be one or more of a force feedback processing, an operation restriction of the arm operation unit, a warning message display through the screen display unit, and an alarm sound output through the speaker unit.

The master interface includes a storage unit for storing a reference image of at least one of an X-ray image, a computed tomography (CT) image, and a magnetic resonance imaging (MRI) image, and an endoscope image displayed through a screen display. The image processing unit may further include an image analyzing unit configured to recognize an organ displayed on the surgical site or the endoscope image through image processing. The reference image may be displayed through a display screen independent of the display screen on which the endoscope image is displayed, corresponding to the name of the organ recognized by the image analyzer.

The master interface may further include a storage unit which stores a reference image which is one or more of an X-ray image, a computed tomography (CT) image, and a magnetic resonance imaging (MRI) image. The reference image may be displayed together on the display screen on which the endoscope image is displayed, or may be displayed through a display screen independent of the display screen, corresponding to the position coordinate information of the actual surgical tool calculated by the characteristic value calculator.

The reference image may be displayed as a 3D image using MPR (Multi Planner Reformat).

According to another embodiment of the present invention, a surgical robot system comprising two or more master robots coupled to each other via a communication network, and one or more robot arms controlled according to an operation signal received from any master robot. A surgical robot system is provided that includes a slave robot.

Each of the master robots may include a screen display unit displaying an endoscope image corresponding to an image signal provided from a surgical endoscope, at least one arm operation unit provided to control at least one robot arm, and a virtual surgical tool through the screen display unit. It may include an augmented reality implementation unit for generating virtual surgical tool information according to a user operation using the arm operation unit to be displayed.

The operation of the arm operation unit of the first master robot, which is one of the two or more master robots, functions to generate virtual surgical tool information, and the operation of the arm operation unit of the second master robot, which is the other of the two or more master robots, controls the control of the robot arm. Can function.

The virtual surgical tool corresponding to the virtual surgical tool information according to the manipulation of the arm manipulation unit of the first master robot may be displayed through the screen display of the second master robot.

According to another aspect of the present invention, there is provided a recording medium in which a control method of a surgical robot system, an operating method of a surgical robot system, and a program for implementing each method are recorded.

According to an embodiment of the present invention, as a control method of a surgical robot system performed in a master robot for controlling a slave robot including one or more robot arms, displaying an endoscope image corresponding to an image signal input from a surgical endoscope. A method of controlling a surgical robot system is provided, the method comprising: generating virtual surgical tool information according to manipulation of an arm manipulation unit, and causing a virtual surgical tool corresponding to the virtual surgical tool information to be displayed together with an endoscope image. .

The surgical endoscope may be one or more of laparoscopic, thoracoscopic, arthroscopic, parenteral, bladder, rectal, duodenum, mediastinal, cardiac.

The generating of the virtual surgical tool information may include receiving operation information according to an operation of the arm manipulation unit and generating an operation signal for controlling the virtual surgical tool information and the robot arm according to the operation information. . The operation signal may be transmitted to the slave robot for the control of the robot arm.

In the control method of the surgical robot system, receiving a driving mode selection command for designating a driving mode of the master robot, and controlling one or more of the endoscope image and the virtual surgical tool to be displayed through the screen display according to the driving mode selection command. It may further comprise the step. The method may further include controlling a mode indicator corresponding to the driving mode designated by the driving mode selection command to be displayed on the screen display unit.

The mode indicator may be predefined as one or more of a text message, a border color, an icon, a background color, and the like.

The control method of the surgical robot system may further include receiving biometric information measured from a slave robot, and displaying the biometric information on a display area independent of the display area on which the endoscope image is displayed.

The control method of the surgical robot system may further include calculating a characteristic value using at least one of an endoscope image and position coordinate information of an actual surgical tool coupled to the robotic arm. The characteristic value may include one or more of the field of view (FOV), magnification, viewpoint, depth of view, actual type of surgical instrument, direction, depth, angle of bending of the surgical endoscope.

The control method of the surgical robot system includes transmitting the test signal to the slave robot, receiving a response signal according to the test signal from the slave robot, and transmitting the test signal and the receiving time of the response signal. The method may further include calculating a delay value for at least one of a network communication speed between slave robots and a delay time in network communication.

The step of causing the virtual surgical tool to be displayed together with the endoscope image may include determining whether the delay value is less than or equal to a preset delay threshold, and processing the virtual surgical tool to be displayed together with the endoscope image if the delay threshold value is exceeded. And processing to display only the endoscope image when the delay threshold value is less than or equal to.

The control method of the surgical robot system includes calculating the position coordinates of the displayed endoscope image and the displayed virtual surgical tool including the actual surgical tool, and calculating the interval value between each surgical tool by using the position coordinate of each surgical tool. It may further comprise a step.

The step of displaying the virtual surgical tool with the endoscope image may include determining whether the interval value is less than or equal to a preset interval threshold value, and processing the virtual surgical tool to be displayed together with the endoscope image only when it is less than or equal to. Can be.

Also, the step of causing the virtual surgical tool to be displayed together with the endoscope image may include determining whether the interval value exceeds a preset interval threshold, and if exceeding, adjusting the translucency, changing the color, and changing the outline thickness. It may include the step of processing such that the above-described virtual surgical tool is displayed with the endoscope image.

The control method of the surgical robot system may further include determining whether the position coordinates of each surgical tool are within a preset error range, and verifying a communication state between the master robot and the slave robot based on the result of the determination. Can be.

In the determining step, it may be determined whether the current position coordinates for the virtual surgical instrument and the previous position coordinates for the actual surgical instrument coincide within the error range.

Further, in the determining step, it may be further determined whether one or more of the movement trajectory and the manipulation form of each surgical tool coincide within an error range.

The control method of the surgical robot system includes extracting feature information including color values for each pixel from the displayed endoscope image, and the area or quantity of pixels having a color value included in a preset color value range in the endoscope image is thresholded. Determining whether or not exceeding, and outputting the warning information if exceeded.

In response to the alert request, one or more of the alert message display, alert sound output, and disabling display for the virtual surgical instrument may be performed.

The step of displaying the virtual surgical tool together with the endoscope image may include extracting region coordinate information of the organ displayed through the surgical site or the endoscope image through image processing of the endoscope image, and the virtual surgical tool. Determining whether the virtual surgical tool is located at the rear side by overlapping with the region coordinate information by using the information and the area coordinate information; and when the overlap is generated, concealing the overlapped area of the shape of the virtual surgical tool; It may include the step.

The control method of the surgical robot system extracts the region coordinate information of the organ displayed through the surgical site or the endoscope image through image processing on the endoscope image, and uses the virtual surgical tool information and the region coordinate information. The virtual surgical tool may further include determining whether a contact has occurred with the region coordinate information, and performing contact warning processing when a contact occurs.

The contact warning processing may be one or more of force feedback processing, an operation restriction of the arm control unit, a warning message display and a warning sound output.

The control method of the surgical robot system includes recognizing an organ displayed through a surgical site or an endoscope image through image processing of an endoscope image, and a position corresponding to a name of an organ recognized in a pre-stored reference image. The method may include extracting and displaying a reference image of the. Here, the reference image may be at least one of an X-ray image, a computed tomography (CT) image, and a magnetic resonance imaging (MRI) image.

The control method of the surgical robot system may include extracting a reference image corresponding to the position coordinates of the actual surgical instrument from a pre-stored reference image, and extracting and displaying the extracted reference image. The reference image may be at least one of an X-ray image, a computed tomography (CT) image, and a magnetic resonance imaging (MRI) image.

The reference image may be displayed together on a display screen on which the endoscope image is displayed or may be displayed on a display screen independent of the display screen.

The reference image may be displayed as a 3D image using MPR (Multi Planner Reformat).

According to another embodiment of the present invention, a method of operating a surgical robot system comprising a slave robot including at least one robot arm and a master robot for controlling the slave robot, wherein the first master robot corresponds to the operation of the arm manipulation unit to perform virtual surgery. Generating virtual surgical tool information for displaying the tool and an operation signal for controlling the robot arm, and the first master robot transmits the operation signal to the slave robot and provides one or more of the operation signal or the virtual surgical tool information. And a step of transmitting to the master robot, wherein the second master robot displays a virtual surgical tool corresponding to at least one of an operation signal or virtual surgical tool information through a screen display unit. This is provided.

Each of the first and second master robots may display an endoscope image received from a slave robot through a screen display unit, and the virtual surgical tool may be displayed together with the endoscope image.

In the operating method of the surgical robot system, the first master robot determines whether the operation authority recovery command has been received from the second master robot, and if the operation authority recovery command is received, the first master robot is virtual operation of the arm operation unit The method may further include controlling to function only for generation of the surgical tool information.

According to another embodiment of the present invention, as a surgical simulation method performed in a master robot controlling a slave robot including a robot arm, recognizing organ selection information and using previously stored organ modeling information And displaying a three-dimensional organ image corresponding to the organ selection information, wherein the organ modeling information has characteristic information including one or more of the shape, color, and feel of each point of the inside and the outside of the corresponding organ. A surgical simulation method is provided.

In order to recognize the organ selection information, using the image signal input from the surgical endoscope to interpret the information on one or more of the color and appearance of the organ included in the surgical site, and information that is interpreted in the pre-stored organ modeling information The step of recognizing the organ matching to may be performed.

The organ selection information may be optionally entered by the operator as one or more organs.

The method may further include receiving a surgical operation command for the 3D organ image according to the manipulation of the cancer manipulation unit, and outputting the tactile information according to the surgical operation command using the organ modeling information.

The tactile information may be control information for controlling one or more of manipulation sensitivity and manipulation resistance according to the operation of the arm manipulation unit or control information for force feedback processing.

The method may further include receiving a surgical operation command for the 3D organ image according to the manipulation of the arm manipulation unit, and displaying a cut plane image according to the surgical operation command using the organ modeling information.

The surgical operation command described above may be one or more of cutting, stitching, pulling, pressing, organ deformation, organ damage by electrosurgery, bleeding in blood vessels, and the like.

The method may further include recognizing an organ according to organ selection information, and extracting and displaying a reference image at a position corresponding to a name of the recognized organ from a previously stored reference image. The reference image may be at least one of an X-ray image, a computed tomography (CT) image, a magnetic resonance imaging (MRI) image, and the like.

Other aspects, features, and advantages other than those described above will become apparent from the following drawings, claims, and detailed description of the invention.

According to an embodiment of the present invention, by using the augmented reality (augmented reality) by displaying the actual surgical tool and the virtual surgical tool is effective to enable the operation of the operator smoothly.

In addition, there is an effect that is provided to the operator to output a variety of information about the patient during surgery.

In addition, according to the network communication speed between the master robot and the slave robot, there is an effect that the operation screen display method is diversified to enable the operator to proceed smoothly.

In addition, there is an effect that the operator can be notified immediately of the emergency by automatically processing the image input through the endoscope.

In addition, long-term contact due to the movement of the virtual surgical tool by the operation of the master robot can be sensed by the operator in real time, and there is an effect to intuitively recognize the long-term positional relationship with the virtual surgical tool.

In addition, the relevant image data (for example, CT image, MRI image, etc.) of the patient for the surgical site can be presented in real time has the effect of enabling the operation progress using a variety of information.

In addition, the surgical robot system can be compatible and shared between the learner (learner) and the trainer (trainer) has the effect of maximizing the real-time education effect.

In addition, the present invention also has the effect of using the three-dimensional modeled virtual organs to predict in advance the progress and results of the actual surgical process.

As the invention allows for various changes and numerous embodiments, particular embodiments will be illustrated in the drawings and described in detail in the written description. However, this is not intended to limit the present invention to specific embodiments, it should be understood to include all transformations, equivalents, and substitutes included in the spirit and scope of the present invention. In the following description of the present invention, if it is determined that the detailed description of the related known technology may obscure the gist of the present invention, the detailed description thereof will be omitted.

The terms first, second, etc. may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another.

The terminology used herein is for the purpose of describing particular example embodiments only and is not intended to be limiting of the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, the terms "comprise" or "have" are intended to indicate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, and one or more other features. It is to be understood that the present invention does not exclude the possibility of the presence or the addition of numbers, steps, operations, components, components, or a combination thereof.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, and in the following description with reference to the accompanying drawings, the same or corresponding components are given the same reference numerals and redundant description thereof will be omitted. Shall be.

In addition, in describing various embodiments of the present invention, each embodiment should not be interpreted or implemented independently, and the technical spirit described in each embodiment may be interpreted or implemented in combination with other embodiments separately described. It should be understood that there is.

In addition, the present invention is a technical concept that can be used universally in the surgery that is used for surgical endoscopes (for example, laparoscopic, thoracoscopic, arthroscopy, parenteral, etc.), in the description of the embodiments of the present invention For convenience, a case where laparoscope is used will be described as an example.

1 is a plan view showing the overall structure of a surgical robot according to an embodiment of the present invention, Figure 2 is a conceptual diagram showing a master interface of a surgical robot according to an embodiment of the present invention.

1 and 2, the laparoscopic surgical robot system includes a slave robot 2 performing surgery on a patient lying on an operating table and a master robot 1 on which an operator remotely controls the slave robot 2. It is composed. The master robot 1 and the slave robot 2 are not necessarily separated into separate devices that are physically independent, but may be integrated into one and integrally formed, in which case the master interface 4 may be, for example, of an integrated robot. May correspond to an interface portion.

The master interface 4 of the master robot 1 comprises a monitor 6 and a master manipulator, and the slave robot 2 comprises a robot arm 3 and a laparoscope 5. The master interface 4 may further include a mode switch control button. The mode switching control button may be implemented in the form of a clutch button 14 or a pedal (not shown), and the mode switching control button is not limited thereto. For example, the mode switching control button may be displayed through the monitor unit 6. It may be implemented as a function menu or a mode selection menu.

The master interface 4 is provided with a master controller so that the operator can be gripped and manipulated by both hands. As illustrated in FIGS. 1 and 2, the master controller may be implemented with two handles 10. An operation signal according to the manipulation of the operator's handle 10 is transmitted to the slave robot 2 so that the robot arm 3 may be moved. Controlled. By operating the handle 10 of the operator, the position movement, rotation, and cutting operation of the robot arm 3 may be performed.

For example, the handle 10 may be composed of a main handle and a sub handle. The slave robot arm 3, the laparoscope 5, etc. may be operated with only one handle, or a plurality of surgical equipment may be operated in real time by adding a sub handle. The main handle and the sub handle may have various mechanical configurations depending on the operation method thereof. For example, the robot arm 3 of the slave robot 2 and / or the like may be a joystick type, a keypad, a trackball, a touch screen, or the like. Various input means for operating the surgical equipment may be used.

The master controller is not limited to the shape of the handle 10 and may be applied without any limitation as long as it can control the operation of the robot arm 3 through a network.

The monitor 6 of the master interface 4 displays an image input by the laparoscope 5 as an image image. In addition, the monitor 6 may be displayed together with the virtual surgical tool controlled by the operator's handle 10, or may be displayed on a separate screen. In addition, the information displayed on the monitor unit 6 may vary depending on the selected driving mode. Whether to display the virtual surgical tool, a control method, display information for each driving mode, etc. will be described in detail with reference to the accompanying drawings.

The monitor unit 6 may be composed of one or more monitors, and may display information necessary for surgery on each monitor separately. 1 and 2 illustrate a case in which the monitor unit 6 includes three monitors, the quantity of the monitors may be variously determined according to the type or type of information requiring display.

The monitor unit 6 may further output a plurality of biometric information about the patient. In this case, one or more indicators indicating a patient's condition, for example, body temperature, pulse rate, respiration, and blood pressure, may be output through one or more monitors of the monitor unit 6. It may be output. In order to provide such biometric information to the master robot 1, the slave robot 2 measures biometric information including at least one of a body temperature measuring module, a pulse measuring module, a respiratory measuring module, a blood pressure measuring module, an electrocardiogram measuring module, and the like. It may include a unit. The biometric information measured by each module may be transmitted from the slave robot 2 to the master robot 1 in the form of an analog signal or a digital signal, and the master robot 1 monitors the received biometric information. Can be displayed via

The slave robot 2 and the master robot 1 may be coupled to each other through a wired communication network or a wireless communication network so that an operation signal and a laparoscope image input through the laparoscope 5 may be transmitted to the counterpart. If two operation signals by the two handles 10 provided in the master interface 4 and / or operation signals for adjusting the position of the laparoscope 5 need to be transmitted at the same time and / or at a similar point in time. Each operation signal may be independently transmitted to the slave robot 2. Herein, when each operation signal is 'independently' transmitted, it means that the operation signals do not interfere with each other and one operation signal does not affect the other signal. As described above, in order to transmit the plurality of operation signals independently of each other, in the generation step of each operation signal, header information for each operation signal is added and transmitted, or each operation signal is transmitted in the generation order thereof, or Various methods may be used such as prioritizing each operation signal in advance and transmitting the operation signal accordingly. In this case, the transmission path through which each operation signal is transmitted may be provided independently so that interference between each operation signal may be fundamentally prevented.

The robot arm 3 of the slave robot 2 can be implemented to be driven with multiple degrees of freedom. The robot arm 3 includes, for example, a surgical instrument inserted into a surgical site of a patient, a rocking drive unit for rotating the surgical instrument in the yaw direction according to the surgical position, and a pitch direction perpendicular to the rotational drive of the rocking drive unit. It comprises a pitch drive unit for rotating the surgical instruments, a transfer drive for moving the surgical instruments in the longitudinal direction, a rotation drive for rotating the surgical instruments, a surgical instrument drive unit installed on the end of the surgical instruments to cut or cut the surgical lesion Can be. However, the configuration of the robot arm 3 is not limited thereto, and it should be understood that this example does not limit the scope of the present invention. In addition, the actual control process, such as the operator rotates the robot arm 3 in the corresponding direction by operating the handle 10 is somewhat distanced from the subject matter of the present invention, so a detailed description thereof will be omitted.

One or more slave robots 2 may be used to operate the patient, and the laparoscope 5 for allowing the surgical site to be displayed as an image image through the monitor unit 6 may be implemented as an independent slave robot 2. It may be. In addition, as described above, embodiments of the present invention may be used universally in operations in which various surgical endoscopes (eg, thoracoscopic, arthroscopy, parenteral, etc.) other than laparoscopic are used.

3 is a block diagram schematically showing the configuration of a master robot and a slave robot according to an embodiment of the present invention, Figure 4a is a view illustrating a driving mode of a surgical robot system according to an embodiment of the present invention, 4B is a diagram illustrating a mode indicator indicating a running mode in accordance with one embodiment of the present invention.

Referring to FIG. 3 in which the configurations of the master robot 1 and the slave robot 2 are schematically shown, the master robot 1 includes an image input unit 310, a screen display unit 320, an arm operation unit 330, and an operation signal. The generation unit 340, the augmented reality implementation unit 350 and the control unit 360. The slave robot 2 comprises a robot arm 3 and a laparoscope 5. Although not shown in FIG. 3, the slave robot 2 may further include a biometric information measuring unit for measuring and providing biometric information about the patient.

The image input unit 310 receives an image input through a camera provided in the laparoscope 5 of the slave robot 2 through a wired or wireless communication network.

The screen display unit 320 outputs an image image corresponding to the image received through the image input unit 310 as visual information. In addition, the screen display unit 320 may further output the virtual surgical tool according to the operation of the arm operation unit 330 as visual information, and further outputs corresponding information when the biometric information is input from the slave robot 2. Can be. The screen display unit 320 may be implemented in the form of a monitor unit 6, and the image processing process for outputting the received image as an image image through the screen display unit 320, the control unit 360, augmented reality implementation It may be performed by the unit 350 or an image processor (not shown).

The arm manipulation unit 330 is a means for allowing the operator to manipulate the position and function of the robot arm 3 of the slave robot 2. The arm manipulation unit 330 may be formed in the shape of the handle 10 as illustrated in FIG. 2, but the shape is not limited thereto and may be modified in various shapes for achieving the same purpose. Further, for example, some may be formed in the shape of a handle, others may be formed in other shapes, such as a clutch button, finger insertion tube or insertion to allow the operator's fingers can be inserted and fixed to facilitate the operation of the surgical tool More rings may be formed.

As described above, the arm operation unit 330 may be provided with a clutch button 14, the clutch button 14 may be used as a mode switching control button. In addition, the mode switching control button may be implemented in a mechanical form such as a pedal (not shown) or may be implemented as a function menu or a mode selection menu displayed through the monitor unit 6. In addition, if the laparoscope 5 is not fixedly present at a specific position to receive an image, and the position and / or the image input angle can be moved or changed by the operator's adjustment, the clutch button 14 or the like is used as the laparoscope ( 5) may be set to function for adjustment of the position and / or the image input angle.

The operation signal generator 340 generates a corresponding operation signal when the operator manipulates the arm operation unit 330 for the movement of the robot arm 3 and / or the laparoscope 5 or the operation for surgery. Transfer to the robot (2). As described above, the manipulation signal may be transmitted and received through a wired or wireless communication network.

When the master robot 1 is driven in the comparison mode that is the second mode, the augmented reality implementation unit 350 is linked to the operation of the arm operation unit 330 as well as the image of the surgical site input through the laparoscope 5 in real time. The virtual surgical tool is processed to be output together through the screen display unit 320. Specific functions, various detailed configurations, and the like, of the augmented reality implementation unit 350 will be described in detail with reference to the accompanying drawings.

The controller 360 controls the operation of each component so that the above-described function can be performed. The controller 360 may perform a function of converting an image input through the image input unit 310 into an image image to be displayed through the screen display unit 320. In addition, the controller 360 controls the augmented reality implementation unit 350 to output the virtual surgical tool through the screen display unit 320 to correspond to the operation information according to the operation of the arm operation unit 330. In addition, the controller 360 may control to grant or retrieve the operation right between the learner and the educator when the fourth mode is performed.

As illustrated in FIG. 4A, the master robot 1 and / or the slave robot 2 may operate in a drive mode selected by an operator or the like among various drive modes.

For example, the driving mode may include a first mode that is an actual mode, a second mode that is a comparison mode, a third mode that is a virtual mode, a fourth mode that is an education mode, a fifth mode that is a simulation mode, and the like.

When the master robot 1 and / or the slave robot 2 operates in the first mode, which is the actual mode, the image displayed through the monitor unit 6 of the master robot 1 may display an operation site, an actual surgical tool, or the like. It may include. That is, the virtual surgical tool may not be displayed, which may be the same as or similar to the display screen during remote surgery using a conventional surgical robot system. Of course, even when operating in the first mode, the corresponding information may be displayed when the biometric information of the patient is measured and received by the slave robot 2, and the display method may vary. .

When the master robot 1 and / or the slave robot 2 operates in the second mode, which is the comparison mode, the image displayed through the monitor unit 6 of the master robot 1 may be displayed on the surgical site, the actual surgical tool, and the virtual. Surgical instruments and the like.

For reference, the actual surgical tool is a surgical tool that is applied by the laparoscopic (5) and included in the image transmitted to the master robot (1) is a surgical tool that applies a direct surgical action to the body of the patient. In contrast, the virtual surgical tool is displayed only on the screen as controlled by operation information (that is, information about movement and rotation of the surgical tool) recognized by the master robot 1 as the operator manipulates the arm manipulation unit 330. It is a virtual surgical tool. The actual surgical tool and the virtual surgical tool will be determined by the operation information of the position and operation shape.

 The operation signal generation unit 340 generates an operation signal using the operation information according to the operation of the operator's arm operation unit 340, and transmits the generated operation signal to the slave robot 2. To be manipulated accordingly. In addition, the position and operation shape of the actual surgical tool operated by the operation signal can be confirmed by the operator by the image input by the laparoscope (5). In other words, if the network communication speed between the master robot 1 and the slave robot 2 is fast enough, the actual surgical tool and the virtual surgical tool will move at almost the same speed. In contrast, if the network communication speed is somewhat slow, the virtual surgical tool will move first, and then the actual surgical tool will move in the same manner as the virtual surgical tool operation with a slight time difference. However, if the network communication speed is slow (for example, the delay time exceeds 150ms), the actual surgical tool will move with a certain amount of time after the virtual surgical tool moves.

In the case where the master robot 1 and / or the slave robot 2 operate in the third mode, which is the virtual mode, the learner (ie, the student) or the educator (ie, the teacher) of the arm operation unit 330. The master robot 1 does not transmit the signal to the slave robot 2 so that the image displayed through the monitor unit 6 of the master robot 1 includes at least one of a surgical site and a virtual surgical tool. Can be. The educator can select the third mode and perform pre-testing on the actual surgical tool. Entering the third mode may be made by selection of the clutch button 14 or the like, and the actual surgical tool moves when the handle 10 is operated while the corresponding button is pressed (or the third mode is selected). Instead of this, only the virtual surgical instruments will be able to move. In this state, when the button is pressed (or the first mode or the second mode is selected), the virtual surgical tool moves the actual surgical tool to correspond to the operation information moved or the handle 10 is pressed to the point where the button is pressed. May be returned (or the position and operation form of the virtual surgical tool is returned).

In the case where the master robot 1 and / or the slave robot 2 operate in the fourth mode in the education mode, the learner (ie, the student) or the educator (ie, the teacher) of the arm operation unit 330 is operated. The signal may be transmitted to the master robot 1 operated by an educator or learner. To this end, two or more master robots 1 may be connected to one slave robot 2, or a separate master robot 1 may be connected to the master robot 1. In this case, when the arm operation unit 330 of the master robot 1 for educators is operated, a corresponding operation signal may be transmitted to the slave robot 2, and each monitor unit of the master robot 1 for educators and learners is provided. In (6), the image input through the laparoscope (5) can be displayed to confirm the operation progress. On the contrary, when the arm operation unit 330 of the learner master robot 1 is operated, the corresponding operation signal may be provided only to the master master robot 1 for education, and may not be transmitted to the slave robot 2. In this manner, the educator's operation can be operated in the first mode, but the learner's operation can be made in the third mode. Operation in the fourth mode, which is the education mode, will be described in detail with reference to the accompanying drawings.

When operating in the fifth mode, which is the simulation mode, the master robot 1 functions as a surgical simulator using characteristics of organs of the three-dimensional modeled three-dimensional shape (for example, shape, texture, and tactile feel during ablation). do. That is, the fifth mode may be understood as the evolution of the virtual mode, which is the third mode, to operate as a surgical simulator by combining the characteristics of organs with a three-dimensional shape obtained using a stereoscopic endoscope or the like.

If the liver is output through the screen display unit 320, the shape of the liver may be grasped three-dimensionally by using a stereoscopic endoscope, and the characteristic information of the liver modeled mathematically (this is a storage unit (not shown)) Can be stored in advance in the virtual mode during surgery. For example, before actually cutting the liver, surgery simulation may be performed in advance on how and in what direction the liver should be excised while matching the characteristics of the liver to the shape of the liver. In addition, based on mathematical modeling information and characteristic information, it is possible to feel in advance the touch of which part is hard or which part is soft. In this case, the surface shape information of the organ obtained three-dimensionally is matched with the three-dimensional shape of the organ surface reconstructed with reference to CT (Computer Tomography) or MRI (Magnetic Resonance Imaging) image, and the like. Matching mathematical modeling information with the three-dimensional shape of the organs reconstructed from can enable more realistic surgical simulation.

The driving modes of the first to fifth modes have been described so far, but it may be possible to add the driving mode according to various purposes.

In addition, when the master robot 1 is driven in each mode, the operator may be confused about what the current driving mode is. The mode indicator may be further displayed through the screen display unit 320 to identify the driving mode more clearly.

4B illustrates a display form in which a mode indicator is further displayed on a screen on which a surgical site and an actual surgical tool 460 are displayed. The mode indicator is for clearly recognizing which driving mode is currently being driven. The mode indicator may vary with the message 450 and the border color 480. In addition, the mode indicator may be implemented with an icon, a background color, or the like, and only one mode indicator may be displayed or two or more mode indicators may be displayed together.

5 is a flowchart illustrating a driving mode selection process of a first mode and a second mode according to an embodiment of the present invention, and FIG. 6 is a screen display output through a monitor unit in a second mode according to an embodiment of the present invention. An illustration of the.

In FIG. 5, it is assumed that one of the first mode and the second mode is selected. However, if the driving mode is applied to the first to fifth modes as illustrated in FIG. 4A, in operation 520 described below. The mode selection input of may be for any one of the first mode to FIG. 5 mode.

Referring to FIG. 5, in operation 510, the surgical robot system is driven. After the driving of the surgical robot system starts, the image input through the laparoscope 5 will be output through the monitor unit 6 of the master robot 1.

In operation 520, the master robot 1 receives a selection of a driving mode from an operator. Selection of the driving mode may be made by, for example, pressing a mechanically implemented clutch button 14 or a pedal (not shown) or using a function menu or mode selection menu displayed through the monitor unit 6. There will be.

If the first mode is selected in step 520, the master robot 1 operates in the driving mode, which is the actual mode, and displays the image input from the laparoscope 5 on the monitor unit 6.

However, if the second mode is selected in step 520, the master robot 1 operates in the drive mode of the comparison mode, and not only the image input from the laparoscope 5 but also the operation information according to the operation of the arm operating unit 330. The virtual surgical tool controlled by the display is displayed on the monitor unit 6 together.

6 illustrates a screen display form output through the monitor unit 6 in the second mode.

As illustrated in FIG. 6, in the comparison mode, an image provided by the laparoscope 5 is provided on the screen (that is, an image in which the surgical site and the actual surgical tool 460 are displayed) and the arm manipulation unit 330. The virtual surgical tool 610 controlled by the operation information is displayed together.

The difference in the display position between the actual surgical tool 460 and the virtual surgical tool 610 is caused by the network communication speed between the master robot 1 and the slave robot 2, and after a predetermined time elapses, the actual surgery The tool 460 will be moved to the location where the current virtual surgical tool 610 is displayed.

In FIG. 6, the virtual surgical tool 610 is illustrated in the shape of an arrow for convenience of distinguishing it from the actual surgical tool 460, but the display shape of the virtual surgical tool 610 is the same as the display shape of the actual surgical tool or mutually. For convenience of identification, it may be expressed in various ways, such as a semi-transparent form and a dotted line having only an outline. Whether the virtual surgical tool 610 is displayed or not will be described later with reference to the accompanying drawings.

In addition, a method for displaying the image and the input provided by the laparoscopic 5 and the virtual surgical tool 610 together, for example, a method such that the virtual surgical tool 610 is displayed overlapped on the laparoscopic image, laparoscopic image And a method such that the virtual surgical tool 610 is reconstructed and displayed as one image.

7 is a view showing a detailed configuration of the augmented reality implementation unit 350 according to an embodiment of the present invention, Figure 8 is a drive of the master robot 1 in the second mode according to an embodiment of the present invention A flowchart illustrating the method.

Referring to FIG. 7, the augmented reality implementation unit 350 includes a characteristic value calculator 710, a virtual surgical tool generator 720, a test signal processor 730, and a delay time calculator 740. Some components (for example, the test signal processor 730 and the delay time calculator 740, etc.) of the components of the augmented reality implementation unit 350 may be omitted, and some components (for example, slaves). Components for processing the biometric information received from the robot 2 to be output through the screen display unit 320). One or more components included in the augmented reality implementation unit 350 may be implemented in the form of a software program by a combination of program codes.

The characteristic value calculator 710 uses the characteristic value by using the image inputted by the laparoscope 5 of the slave robot 2 and / or the coordinate information on the position of the actual surgical tool coupled to the robot arm 3. Calculate The actual position of the surgical tool can be recognized by referring to the position value of the robot arm 3 of the slave robot 2, the information about the position may be provided from the slave robot 2 to the master robot (1). .

The characteristic value calculator 710 may use, for example, a field of view (FOV), an enlargement ratio, a perspective (for example, a viewing direction), a viewing depth, etc. of the laparoscope 5 using an image by the laparoscope 5. In addition, the type of the actual surgical tool 460, the direction, the depth, the degree of bending and the like can be calculated. When the characteristic value is calculated using the image of the laparoscope 5, an image recognition technique for recognizing the outline of the subject included in the image, shape recognition, tilt angle, or the like may be used. In addition, the type of the actual surgical tool 460 may be input in advance in the process of coupling the corresponding surgical tool to the robot arm (3).

The virtual surgical tool generation unit 720 generates a virtual surgical tool 610 to be output through the screen display unit 320 with reference to the operation information according to the operation of the operator's robot arm (3). The position at which the virtual surgical tool 610 is initially displayed may be based on the display position at which the actual surgical tool 460 is displayed through the screen display 320, for example, and is manipulated according to the operation of the arm manipulation unit 330. The movement displacement of the surgical tool 610 may be set in advance with reference to the measured value at which the actual surgical tool 460 is moved in correspondence with the manipulation signal, for example.

The virtual surgical tool generating unit 720 may generate only the virtual surgical tool information (for example, a characteristic value for the display of the virtual surgical tool) for the virtual surgical tool 610 is output through the screen display unit 320. have. The virtual surgical tool generating unit 720 is used to express the characteristic value or the virtual surgical tool 610 calculated by the characteristic value calculating unit 710 in determining the shape or position of the virtual surgical tool 610 according to the operation information. You can also refer to the characteristic value just before. This is to enable the virtual surgical tool 710 or the actual surgical tool 460 to quickly generate the corresponding information when only the translation operation is performed in the same state as the previous shape (for example, the inclined angle, etc.). to be.

The test signal processor 730 transmits a test signal to the slave robot 2 so as to determine what network communication speed is between the master robot 1 and the slave robot 2, and responds from the slave robot 2. Receive the signal. The test signal transmitted by the test signal processing unit 730 is a typical signal that is included in the form of a time stamp in a control signal transmitted and received between the master robot 1 and the slave robot 2, for example. It may be a signal additionally used to measure the network communication speed. In addition, the network communication speed may be previously determined to be measured only at some of the time points at which the test signal is transmitted and received.

The delay time calculator 740 calculates a delay time in network communication using the transmission time of the test signal and the reception time of the response signal. If the network communication speed is the same in a section in which the master robot 1 transmits an arbitrary signal to the slave robot 2 and in a section in which the master robot 1 receives an arbitrary signal from the slave robot 2, the delay time is YES. For example, it may be 1/2 of the difference between the transmission time of the test signal and the reception time of the response signal. This is because the slave robot will immediately perform the corresponding processing when an operation signal is received from the master robot 1. Of course, the delay time may further include a processing delay time for performing a process such as controlling the robot arm 3 according to the manipulation signal in the slave robot 2. As another example, if the delay time in the network communication is important when the difference between the operator's operation time and the observation time is important, the delay time may indicate the operation time of the operator through the transmission time and the reception time of the response signal (for example, the display unit). The time difference may be calculated as the difference value. In addition, the method of calculating the delay time may vary.

If the delay time is less than or equal to a predetermined threshold (eg, 150 ms), the difference in the display position between the actual surgical tool 460 and the virtual surgical tool 610 will not be large. In this case, the virtual surgical tool generation unit 720 may prevent the virtual surgical tool 610 from being displayed through the screen display unit 320. This is because the actual surgical tool 460 and the virtual surgical tool 610 are displayed in duplicate in a coincident or very close position so that there is no need to cause operator confusion.

However, if the delay time exceeds a predetermined threshold (eg, 150 ms), the difference in the display position between the actual surgical tool 460 and the virtual surgical tool 610 may be large. In this case, the virtual surgical tool generation unit 720 may allow the virtual surgical tool 610 to be displayed through the screen display unit 320. This is to remove the operator's confusion caused by the operation situation of the operator's cancer operation unit 330 and the operation state of the actual surgical tool 460 in real time, the operator with reference to the virtual surgical tool 610 This is because the actual surgical tool 460 is subsequently operated in the operation form of the virtual surgical tool 610 even if the surgery.

8 is a flowchart illustrating a method of driving the master robot 1 in the second mode. In describing each step of the flowchart, it will be described that the master robot 1 performs each step for convenience of explanation and understanding.

Referring to FIG. 8, in operation 810, the master robot 1 generates a test signal and transmits a test signal to the slave robot 2 through a wired or wireless communication network.

In operation 820, the master robot 1 receives a response signal to the test signal from the slave robot 2.

In step 830, the master robot 1 calculates a delay time on the network communication speed using the transmission time of the test signal and the reception time of the response signal.

In operation 840, the master robot 1 determines whether the calculated delay time is equal to or less than a preset threshold. Here, the threshold value is a delay time in network communication speed required for the smooth operation of the operator using the surgical robot system, and may be determined and applied in an experimental and / or statistical manner.

If the calculated delay time is less than or equal to a predetermined threshold value, the process proceeds to step 850 where the master robot 1 inputs the image input through the laparoscope 5 to the screen display 320 (that is, the surgical site and the actual surgical tool 460). Image), which is displayed on the screen, to be displayed. At this time, the virtual surgical tool 610 may not be displayed. Of course, even in this case, the virtual surgical tool 610 and the actual surgical tool 460 may be displayed together.

However, if the calculated delay time exceeds the preset threshold, the process proceeds to step 860 where the master robot 1 inputs the image (that is, the surgical site and the actual surgery) input through the laparoscope 5 to the screen display 320. And the virtual surgical tool 610 is displayed together with the image including the tool 460. Of course, even in this case, the virtual surgical tool 610 may not be displayed.

9 is a view showing a detailed configuration of the augmented reality implementation unit 350 according to another embodiment of the present invention, Figures 10 and 11 are each a master robot 1 in the second mode according to another embodiment of the present invention; ) Is a flow chart showing the driving method.

Referring to FIG. 9, the augmented reality implementation unit 350 may include a feature value calculator 710, a virtual surgical tool generator 720, an interval calculator 910, and an image analyzer 920. Some components of the components of the augmented reality implementation unit 350 may be omitted, and may process some components (for example, outputting biometric information received from the slave robot 2 through the screen display unit 320). Components, etc.) may be further added. One or more components included in the augmented reality implementation unit 350 may be implemented in the form of a software program by a combination of program codes.

The characteristic value calculator 710 uses the characteristic value by using the image inputted by the laparoscope 5 of the slave robot 2 and / or the coordinate information on the position of the actual surgical tool coupled to the robot arm 3. Calculate The characteristic value may be, for example, the field of view (FOV), magnification, perspective (for example, viewing direction), viewing depth, etc. of the laparoscope 5, and the type, direction, depth, and bending of the actual surgical tool 460. Degree or the like.

The virtual surgical tool generation unit 720 generates a virtual surgical tool 610 to be output through the screen display unit 320 with reference to the operation information according to the operation of the operator's robot arm (3).

The interval calculator 910 uses the position coordinates of the actual surgical tool 460 calculated through the characteristic value calculator 710 and the position coordinates of the virtual surgical tool 610 linked to the operation of the arm operator 330. Calculate the interval of. For example, if the position coordinates of the virtual surgical tool 610 and the actual surgical tool 460 are determined, respectively, it may be calculated as the length of a line connecting two points. Here, the position coordinate may be, for example, a coordinate value of a point in three-dimensional space defined by the xyz axis, and the point is previously a point of a specific position on the virtual surgical tool 610 and the actual surgical tool 460. Can be specified. In addition, the interval between the surgical instruments may be further used, such as the length of the path or trajectory generated by the operation method. For example, in the case of drawing a circle, when the time difference exists for the time of drawing the circle, the length of the line segment between each surgical tool becomes very small, but the difference in path or trajectory may be generated as the circumference of the circle generated by the manipulation method. Because.

The position coordinates of the actual surgical tool 460 used for the interval calculation may be used as an absolute coordinate value or a relative coordinate value calculated based on a specific point, or the actual surgical tool 460 displayed through the screen display 320. ) Can be used by coordinates. Similarly, the position coordinates of the virtual surgical tool 610 are also used as the absolute coordinates of the virtual position moved by the operation of the arm operation unit 330 based on the initial position of the virtual surgical tool 610 or calculated based on a specific point. The relative coordinate value may be used, or the position of the virtual surgical tool 610 displayed through the screen display unit 320 may be coordinated. Here, the feature information analyzed by the image analyzer 920 to be described below may be used to interpret the position of each surgical tool displayed through the screen display unit 320.

If the distance between the virtual surgical tool 610 and the actual surgical tool 460 is narrow or 0, it may be understood that the network communication speed is good, but if the interval is wide it may be understood that the network communication speed is insufficient.

The virtual surgical tool generating unit 720 determines one or more of whether to display the virtual surgical tool 610, the display color or the display form of the virtual surgical tool 610, etc. by using the interval information calculated by the interval calculating unit 910. Can be. For example, if the distance between the virtual surgical tool 610 and the actual surgical tool 460 is less than a predetermined threshold, the virtual surgical tool 610 is not to be output through the screen display 320 Can be. In addition, when the interval between the virtual surgical tool 610 and the actual surgical tool 460 exceeds a predetermined threshold (threshold), in order to adjust the translucency in proportion to each other or to distort the color or virtual surgical tool (610 The operator may be able to clearly recognize the network communication speed through a process such as changing the thickness of the outline. Here, the threshold value may be designated as a distance value, for example, 5 mm.

The image analyzer 920 may use one of preset feature information (for example, color value for each pixel, position coordinates of the actual surgical tool 460, and manipulation shape) by using the image input and provided by the laparoscope 5. Above). For example, the image analyzer 920 analyzes the color values of each pixel of the image so as to enable immediate response to an emergency situation (for example, excessive bleeding) that may occur during surgery. It may be determined whether a pixel having a color value indicating a is greater than or equal to a reference value, or whether a region or area formed by pixels having a color value indicating blood is a predetermined size or more. In addition, the image analyzer 920 may generate a position coordinate of each surgical tool by capturing a display screen of the screen display unit 320 on which the image input by the laparoscope 5 and the virtual surgical tool 610 are displayed. .

10 is a flowchart illustrating a method of driving the master robot 1 in the second mode according to another embodiment of the present invention.

Referring to FIG. 10, in operation 1010, the master robot 1 receives a laparoscope image (that is, an image provided through the laparoscope 5) from the slave robot 2.

In operation 1020, the master robot 1 calculates coordinate information of the actual surgical tool 460 and the virtual surgical tool 610. In this case, the coordinate information may be calculated using, for example, feature values and manipulation information calculated by the feature value calculator 710, or feature information extracted by the image analyzer 920 may be used.

In step 1030, the master robot 1 calculates the distance between each other using the coordinate information of each surgical tool calculated in step 1020.

In step 1040, the master robot 1 determines whether the calculated interval is less than or equal to the threshold.

If the calculated interval is less than or equal to the threshold value, the process proceeds to step 1050 where the master robot 1 outputs the laparoscope image through the screen display unit 320, but the virtual surgical tool 610 is not displayed.

However, if the calculated interval exceeds the threshold, the process proceeds to step 1060 where the master robot 1 causes the laparoscopic image and the virtual surgical tool 610 to be displayed together through the screen display. In this case, a process such as adjusting the translucency, distorting the color, or changing the thickness of the outline of the virtual surgical tool 610 may be performed in proportion to the distance between the two.

11 is a flow chart showing a method of driving the master robot 1 in the second mode according to another embodiment of the present invention.

Referring to FIG. 11, in operation 1110, the master robot 1 receives a laparoscope image. The received laparoscopic image will be output through the screen display 320.

In steps 1120 and 1130, the master robot 1 analyzes the received laparoscopic image, and calculates and analyzes color values of pixels of the corresponding image. As described above, the color value calculation for each pixel may be performed by the image analyzer 920 or by the feature value calculator 710 to which an image recognition technique is applied. In addition, one or more of, for example, a color value frequency, an area or an area formed by pixels having a color value to be analyzed may be calculated by analyzing the color value for each pixel.

In operation 1140, the master robot 1 determines whether an emergency situation is performed based on the information analyzed in operation 1130. The type of emergency (eg, excessive bleeding, etc.) and when the analyzed information will be recognized as an emergency may be predefined.

If it is determined that there is an emergency, the process proceeds to step 1150 and the master robot 1 outputs warning information. The warning information may be, for example, a warning message output through the screen display unit 320 or a warning sound output through a speaker unit (not shown). Although not shown in Figure 3, it is obvious that the speaker unit for outputting warning information or announcements may be further included in the master robot (1). In addition, when the virtual surgical tool 610 is displayed together through the screen display 320 at the time determined as an emergency, the virtual surgical tool 610 is displayed so that the operator can accurately determine the surgical site. May be controlled so as not to.

However, if it is determined that it is not an emergency, the process proceeds to step 1110 again.

12 is a block diagram schematically showing the configuration of a master robot and a slave robot according to another embodiment of the present invention, Figure 13 is a method for verifying the normal operation of the surgical robot system according to another embodiment of the present invention Is a flow chart showing.

Referring to FIG. 12, in which the configuration of the master robot 1 and the slave robot 2 is schematically illustrated, the master robot 1 generates an image input unit 310, a screen display unit 320, an arm operation unit 330, and an operation signal. The unit 340, the augmented reality implementation unit 350, a controller 360, and a network verification unit 1210 are included. The slave robot 2 comprises a robot arm 3 and a laparoscope 5.

The image input unit 310 receives an image input through a camera provided in the laparoscope 5 of the slave robot 2 through a wired or wireless communication network.

The screen display unit 320 outputs an image image received through the image input unit 310 and / or an image image corresponding to the virtual surgical tool 610 according to the operation of the arm operation unit 330 as visual information.

The arm manipulation unit 330 is a means for allowing the operator to manipulate the position and function of the robot arm 3 of the slave robot 2.

The operation signal generator 340 generates a corresponding operation signal when the operator manipulates the arm operation unit 330 for the movement of the robot arm 3 and / or the laparoscope 5 or the operation for surgery. Transfer to the robot (2).

The network verification unit 1210 uses the characteristic value calculated by the characteristic value calculating unit 710 and the virtual surgical tool information generated by the virtual surgical tool generating unit 720 to display the master robot 1 and the slave robot 2. Verify network communication between To this end, for example, the position information, direction, depth, of the virtual surgical tool 610 according to the virtual surgical tool information and one or more of the position information, direction, depth, degree of bending of the actual surgical tool 460 among the characteristic values One or more of the degree of bending may be used, and the characteristic value and the virtual surgical tool information may be stored in a storage unit (not shown).

According to the exemplary embodiment of the present invention, when the operation information is generated by the operation of the operator's arm operation unit 330, the virtual surgical tool 610 is controlled to correspond thereto, and the operation signal corresponding to the operation information is slave robot 2. It is transmitted to and used for the operation of the actual surgical tool 460. In addition, the movement of the position of the actual surgical tool 460 that is manipulated and controlled by the manipulation signal may be confirmed through a laparoscope image. In this case, since the manipulation of the virtual surgical tool 610 is made in the master robot 1, the operation of the virtual surgical tool 460 will generally be made in advance in consideration of the network communication speed.

Accordingly, the network verification unit 1210 is operated so that the actual surgical tool 460 is delayed in time but can be identified with the same movement trajectory or operation type of the virtual surgical tool 610 or within a preset error range. It can be determined whether the network communication is normal by determining. To this end, the virtual surgical tool information stored in the storage unit, such as the current position of the actual surgical tool 460 may be used. In addition, the error range may be set to, for example, a distance value between mutual coordinate information or a time value until it is recognized as a coincidence, and the like, which may be specified arbitrarily, experimentally, and / or statistically.

In addition, the network verification unit 1210 may verify network communication using the feature information interpreted by the image analyzer 920.

The controller 360 controls the operation of each component so that the above-described function can be performed. In addition, as described in an exemplary embodiment, the controller 360 may further perform various additional functions.

FIG. 13 illustrates a method of verifying whether the surgical robot system is normally driven by verifying network communication.

Referring to FIG. 13, in steps 1310 and 1320, the master robot 1 receives a manipulation of the arm manipulation unit 330 from an operator and interprets manipulation information according to manipulation of the arm manipulation unit 330. The operation information is, for example, information according to the operation of the arm operation unit 330 for moving the position of the actual surgical tool 460, incision of the surgical site, and the like.

In operation 1330, the master robot 1 generates virtual surgical tool information using the analyzed operation information, and outputs the virtual surgical tool 610 according to the generated virtual surgical tool information to the screen display unit 320. In this case, the generated virtual surgical tool information may be stored in a storage unit (not shown).

In operation 1340, the master robot 1 calculates characteristic values of the actual surgical tool 460. The calculation of the feature value may be performed by the feature value calculator 710 or the image analyzer 920, for example.

In operation 1350, the master robot 1 determines whether a coordinate value coincidence point of each surgical tool exists. If the coordinate information of each surgical tool is matched or matched within the error range, it may be determined that a coordinate value match point exists. Here, the error range may be preset with, for example, a distance value on three-dimensional coordinates. As described above, since the result of the operation of the operator's cancer operation unit 330 will be reflected to the virtual surgical tool 610 first than the actual surgical tool 460, step 1350 is a characteristic value for the actual surgical tool 460 It may be performed by determining whether or not the information matches the virtual surgical tool information stored in the storage unit.

If the coordinate value coincidence point does not exist, the process proceeds to step 1360 and the master robot 1 outputs warning information. The warning information may be, for example, a warning message output through the screen display unit 320 or a warning sound output through a speaker unit (not shown).

However, if there is a coordinate value coincidence point, it is determined that the network communication is normal and the process proceeds to step 1310 again.

Steps 1310 to 1360 described above may be performed in real time during the operation of the operator, or may be performed periodically or at a predetermined time point.

14 is a view showing a detailed configuration of the augmented reality implementation unit 350 according to another embodiment of the present invention, Figures 15 and 16 are respectively for outputting a virtual surgical tool according to another embodiment of the present invention It is a flowchart showing the driving method of the master robot 1.

Referring to FIG. 14, the augmented reality implementation unit 350 may include a feature value calculator 710, a virtual surgical tool generator 720, an image analyzer 920, an overlap processor 1410, and a contact recognizer 1420. Include. Some components of the components of the augmented reality implementation unit 350 may be omitted, and may process some components (for example, outputting biometric information received from the slave robot 2 through the screen display unit 320). Components, etc.) may be further added. One or more components included in the augmented reality implementation unit 350 may be implemented in the form of a software program by a combination of program codes.

The characteristic value calculator 710 uses the characteristic value by using the image inputted by the laparoscope 5 of the slave robot 2 and / or the coordinate information on the position of the actual surgical tool coupled to the robot arm 3. Calculate The characteristic value may be, for example, the field of view (FOV), magnification, perspective (for example, viewing direction), viewing depth, etc. of the laparoscope 5, and the type, direction, depth, and bending of the actual surgical tool 460. Degree or the like.

The virtual surgical tool generation unit 720 generates virtual surgical tool information for outputting the virtual surgical tool 610 through the screen display 320 with reference to the operation information according to the operation of the operator's robot arm 3.

The image analyzing unit 920 may use the image input and provided by the laparoscope 5 to preset feature information (for example, the shape of an organ in a surgical site, a position coordinate of an actual surgical tool 460, an operation shape, etc.). Extract one or more). For example, the image analyzer 920 may analyze what organs are displayed by using an image recognition technique such as extracting an outline of organs displayed in a laparoscopic image and analyzing color values of pixels representing organs. To this end, the storage unit (not shown) may store in advance information about the shape, color, and coordinate information of the region where each organ and / or the surgical site is located in a three-dimensional space. Alternatively, the image analyzer 920 may analyze coordinate information (absolute coordinates or relative coordinates) of an area occupied by the corresponding organ through image analysis.

The overlap processing unit 1410 overlaps each other using the virtual surgical tool information generated by the virtual surgical tool generating unit 720 and the region coordinate information of the organs and / or surgical sites recognized by the image analyzer 920. It is judged whether or not it occurs and processes accordingly. If some or all of the virtual surgical instruments are located below or laterally to the side of the organ, it may be determined that overlapping (i.e., obscuration) occurs as much as the corresponding portions, and the facts of the virtual surgical instrument 610 display may be determined. In order to increase, the area of the virtual surgical tool 610 corresponding to the overlapped portion is processed to be concealed (that is, not displayed through the screen display unit 320). As a method of concealing the overlapped portion, for example, a method of allowing the region corresponding to the overlapped portion of the shape of the virtual surgical tool 610 to be transparently processed.

Alternatively, when the overlapping processor 1410 determines that there is an overlap between the organ and the virtual surgical tool 610, the area coordinate information of the organ is provided to the virtual surgical tool generator 720 or the virtual surgical tool generator 720. May request that the information be read from the storage to prevent the virtual surgical tool generation unit 720 from generating virtual surgical tool information for the overlapped portion.

The contact recognition unit 1420 may determine whether contact is generated by using the virtual surgical tool information generated by the virtual surgical tool generator 720 and the region coordinate information of the organ recognized by the image analyzer 920. Determine and deal with them accordingly. If the surface coordinate information of the organ coordinate information of the organ and the coordinate information of part or all of the virtual surgical tool coincide with each other, it may be determined that there is a contact there. When it is determined that there is a contact by the contact recognition unit 1420, the master robot 1 processes, for example, the arm operation unit 330 so that it is no longer operated, or forces feedback through the arm operation unit 330. Can be generated or warning information (e.g., warning messages and / or warning sounds) can be output. Components for processing force feedback or outputting warning information may be included as components of the master robot 1.

15 illustrates a driving method of the master robot 1 for outputting a virtual surgical tool according to another embodiment of the present invention.

Referring to FIG. 15, in operation 1510, the master robot 1 receives a manipulation of the arm manipulation unit 330 from an operator.

Subsequently, in steps 1520 and 1530, the master robot 1 analyzes the operator's manipulation information according to the manipulation of the arm manipulation unit 330 and generates virtual surgical tool information. The virtual surgical tool information may include, for example, coordinate information about an outline or an area of the virtual surgical tool 610 for outputting the virtual surgical tool 610 through the screen display unit 320.

In addition, in steps 1540 and 1550, the master robot 1 receives a laparoscope image from the slave robot 2 and interprets the received image. For example, the analysis of the received image may be performed by the image analyzer 920, and the image analyzer 920 may recognize which organs are included in the laparoscopic image.

In operation 1560, the master robot 1 reads region coordinate information about the organ recognized through the laparoscope image from the storage.

In operation 1570, the master robot 1 determines whether overlapping portions exist between each other using coordinate information of the virtual surgical tool 610 and area coordinate information of the organ.

If there is an overlapping portion, in step 1580, the master robot 1 processes the virtual surgical tool 610 in which the overlapping portion is concealed to be output through the screen display unit 320.

However, if there is no overlapping part, the master robot 1 processes the virtual surgical tool 610 in which all parts are normally displayed through the screen display unit 320 in step 1590.

16 illustrates an embodiment for notifying the operator when the virtual surgical tool 610 is in contact with an organ of the patient. Steps 1510 to 1560 of FIG. 16 have already been described with reference to FIG. 15, and thus description thereof will be omitted.

Referring to FIG. 16, in step 1610, the master robot 1 determines whether some or all of the virtual surgical tool 610 is in contact with an organ. The contact between the organ and the virtual surgical tool 610 may be determined using, for example, coordinate information about each region.

If the organ is in contact with the virtual surgical tool 610, the process proceeds to step 1620, the master robot 1 performs a force feedback process to inform the operator. As described above, for example, the arm manipulation unit 330 may be processed to no longer be operated, or may be processed to output warning information (eg, a warning message and / or a warning sound).

However, if the organ is not in contact with the virtual surgical tool 610, and waits in step 1610.

Through the above-described process, the operator can predict in advance whether the actual surgical tool 460 will be in contact with the organ in advance, thereby enabling a safer and more sophisticated surgical procedure.

17 is a flowchart illustrating a method of providing a reference image according to another embodiment of the present invention.

Typically, the patient will take various reference images, such as X-rays, CT, and / or MRI, before surgery. If such a reference image can be presented to the operator in conjunction with the laparoscopic image during the operation or on any monitor of the monitor 6, the operation of the operator will be smoother. For example, the reference image may be previously stored in a storage unit included in the master robot 1, or may be stored in a database accessible by the master robot 1 through a communication network.

Referring to FIG. 17, in operation 1710, the master robot 1 receives a laparoscope image from the laparoscope 5 of the slave robot 2.

In operation 1720, the master robot 1 extracts predetermined feature information by using the laparoscope image. Here, the feature information may be, for example, one or more of the shape of the organ in the surgical site, the position coordinates of the actual surgical tool 460, the manipulation shape, and the like. Extraction of the feature information may be performed by, for example, the image analyzer 920.

In operation 1730, the master robot 1 recognizes what organs are included and displayed in the laparoscope image using the feature information extracted in operation 1720 and information previously stored in the storage unit.

Subsequently, in operation 1740, the master robot 1 reads a reference image including an image corresponding to the organ recognized in operation 1730 from a database accessible through a storage unit or a communication network, and then any part of the reference image is monitored. Determine if it should be indicated through (6). The reference image to be output through the monitor unit 6 is an image of the organ, for example, may be an X-ray, CT, and / or MRI image. In addition, which part of the reference image (for example, which part of the whole body image of the patient) is output for reference may be determined by, for example, the name of the recognized organ or coordinate information of the actual surgical tool 460. have. To this end, the coordinate information or the name of each part of the reference image or the number of reference frames of the sequential frames may be previously specified. Any one reference image may be output through the monitor unit 6, or two or more reference images (eg, X-ray image and CT image) having different properties may be displayed together.

In operation 1750, the master robot 1 outputs the laparoscopic image and the reference image through the monitor 6. At this time, by processing the reference image to be displayed in a direction similar to the input angle (for example, the camera angle) of the laparoscopic image, the intuitiveness of the operator may be maximized. For example, when the reference image is a planar image captured in a specific direction, the 3D image using a real-time multi-planner format (MPR) may be output according to the camera angle calculated by the characteristic value calculator 710. For reference, MPR is a technique of selectively drawing only an arbitrary portion of a cross-sectional image, which is necessary for one or several slices, to form a partially three-dimensional image. It is a technique that developed the technique of drawing.

The case where the master robot 1 functions in the first mode which is the actual mode, the second mode which is the comparison mode, and / or the third mode which is the virtual mode has been described. Hereinafter, the case where the master robot 1 functions in the fourth mode which is the education mode or the fifth mode which is the simulation mode will be described. However, various embodiments related to the display of the virtual surgical tool 610 described above with reference to the related drawings are not technical ideas that are limitedly applied in a specific driving mode, and the driving mode in which the virtual surgical tool 610 needs to be displayed. If it is not described, it can be applied without limitation.

18 is a plan view showing the overall structure of a surgical robot according to another embodiment of the present invention.

Referring to FIG. 18, a laparoscopic surgical robot system includes two or more master robots 1 and slave robots 2. Of the two or more master robots 1, the first master robot 1a may be a student master robot used by a learner (eg, a training student), and the second master robot 1b may be an educator (eg, Teacher master robot used by a training teacher). Since the configurations of the master robot 1 and the slave robot 2 are as described above, a brief description thereof will be provided.

As described above with reference to FIG. 1, the master interface 4 of the master robot 1 includes a monitor unit 6 and a master controller, and the slave robot 2 includes the robot arm 3 and the laparoscope 5. It may include. The master interface 4 may further include a mode switch control button for selecting one of the plurality of driving modes. The master manipulator may be embodied, for example, in a form (for example, a handle) in which an operator can grip and manipulate each hand. The monitor 6 may further output a plurality of biometric information or reference images as well as a laparoscope image.

The two master robots 1 illustrated in FIG. 18 are coupled to each other through a communication network, and may be coupled to the slave robot 2 through a communication network, respectively. The master robot 1 coupled to each other through a communication network may be provided in various quantities as necessary. In addition, although the use of the first master robot 1a and the second master robot 1b, the training teacher and the training student may be predetermined in advance, the roles may be interchanged with each other according to a request or a need.

For example, the first master robot 1a for the learner is coupled via a communication network only with the second master robot 1b for the training teacher, and the second master robot 1b is the first master robot 1a and the slave robot. It may be combined with (2) through a communication network. That is, when the training student manipulates the master controller provided in the first master robot 1a, only the virtual surgical tool 610 may be operated to be output through the screen display unit 320. At this time, the operation signal is provided from the first master robot (1a) to the second master robot (1b), the operation state of the virtual surgical tool 610 according to the monitor portion 6b of the second master robot (1b) Through the output, the lab teacher can check whether the lab student is operating in the normal course.

As another example, the first master robot 1a and the second master robot 1b may be coupled through a communication network, and each may be coupled with the slave robot 2 through a communication network. In this case, when the training student manipulates the master controller provided in the first master robot 1a, the actual surgical tool 460 is manipulated, and the corresponding operation signal is also provided to the second master robot 1b to provide the training teacher. The student can check whether the student is undergoing surgery in the normal course.

In this case, the training teacher may manipulate his master robot to control in which mode the master robot of the training student functions. To this end, any master robot may be set in advance such that a driving mode is determined by a control signal received from another master robot to enable manipulation of the actual surgical tool 460 and / or the virtual surgical tool 610.

19 is a view showing a method of operating a surgical robot system in the training mode according to another embodiment of the present invention.

In FIG. 19, the operation of the arm manipulation unit 330 in the first master robot 1a is functioned only for the operation of the virtual surgical tool 610, and an operation signal is transmitted from the first master robot 1a to the second master robot 1b. The operation method of the surgical robot system when provided with is illustrated. This is to be used for the purpose of checking the operation situation of the first master robot 1a by one of the training students or one of the training teachers by using the second master robot 1b by the other one of the training teachers or training students. Can be.

Referring to FIG. 19, in operation 1905, communication connection establishment is performed between the first master robot 1a and the second master robot 1b. The communication connection setting may be, for example, for transmitting and receiving one or more of an operation signal, an authority command, and the like. The communication connection setting may be made at the request of at least one of the first master robot 1a and the second master robot 1b, or may be made immediately when each master robot is powered on.

In operation 1910, the first master robot 1a receives a user manipulation according to the manipulation of the arm manipulation unit 330. Here, the user may be, for example, either a training student or a training teacher.

In operation 1920 and operation 1930, the first master robot 1a generates an operation signal according to the user operation of operation 1910 and generates virtual surgical tool information corresponding to the generated operation signal. As described above, the virtual surgical tool information may be generated using the operation information according to the operation of the arm manipulation unit 330.

In operation 1940, the first master robot 1a determines whether there is an overlap or contact portion with the organs based on the generated virtual surgical instrument information. Since the method for determining the existence of a long-term overlapping or contact portion with the virtual surgical tool has been described above with reference to FIGS. 15 and / or 16, a description thereof will be omitted.

If there is an overlap or contact portion, the flow proceeds to step 1950 to generate processing information according to the overlap or contact. The processing information may be transparent processing on the overlapping portion, force feedback performance according to contact, and the like as described above with reference to FIGS. 15 and / or 16.

In operation 1960, the first master robot 1a transmits virtual surgical tool information and / or processing information to the second master robot 1b. The first master robot 1a may transmit an operation signal to the second master robot 1b, and generate virtual surgical tool information using the operation signal received by the second master robot 1b, and then overlap or touch it. You can also judge.

In steps 1970 and 1980, the first master robot 1a and the second master robot 1b output the virtual surgical tool 610 to the screen display unit 320 using the virtual surgical tool information. At this time, the matter corresponding to the processing information may be processed together.

Up to now, the case in which the first master robot 1a controls only the virtual surgical tool 610 and a manipulation signal according to the same is provided to the second master robot 1b has been described with reference to FIG. 19. However, the first master robot 1a may control the actual surgical tool 460 according to the driving mode selection, and an operation signal and the like may be provided to the second master robot 1b.

20 is a view showing a method of operating a surgical robot system in the training mode according to another embodiment of the present invention.

In the operation method of the surgical robot system with reference to FIG. 20, it is assumed that the second master robot 1b has control authority over the first master robot 1a.

Referring to FIG. 20, in operation 2010, communication connection is established between the first master robot 1a and the second master robot 1b. The communication connection setting may be, for example, for transmitting and receiving one or more of an operation signal, an authority command, and the like. The communication connection setting may be made at the request of at least one of the first master robot 1a and the second master robot 1b, or may be made immediately when each master robot is powered on.

In operation 2020, the second master robot 1b transmits an operation authority grant command to the first master robot 1a. By the operation authority grant command, the first master robot 1a has the authority to actually control the robot arm 3 provided in the slave robot 2. The surgical authority grant command may be generated by the second master robot 1b to be configured, for example, in a signal form and information form predefined between the master robots.

In operation 2030, the first master robot 1a receives a user manipulation according to the manipulation of the arm manipulation unit 330. Here, the user may be, for example, a training student.

In operation 2040, the first master robot 1a generates an operation signal according to the user operation in operation 1910 and transmits the operation signal to the slave robot 2 through the communication network. The first master robot 1a may generate the virtual surgical tool information corresponding to the generated operation signal or the operation information according to the operation of the arm manipulation unit 330 to display the virtual surgical tool 610 through the monitor unit 6. Make sure

In addition, the first master robot 1a may transmit operation signals and / or virtual surgical tool information to the second master robot 1b in order to confirm the actual operating condition of the surgical tool 460. In operation 2050, the second master robot 1b receives an operation signal and / or virtual surgical tool information.

In steps 2060 and 2070, the first master robot 1a and the second master robot 1b respectively receive the laparoscope image received from the slave robot 2 and the operation of the arm manipulation unit 330 of the first master robot 1a. The virtual surgical tool 610 is output through the screen display unit 320.

If the second master robot 1b is received from the slave robot 2 without outputting the virtual surgical tool 610 according to the operation of the arm operation unit 330 of the first master robot 1a to the screen display unit 320. In the case of confirming the actual operating condition of the surgical tool 460 through the laparoscopic image, step 2050 may be omitted, and in step 2070, only the received laparoscopic image will be output.

In operation 2080, the second master robot 1b determines whether a request for retrieving the surgery authority granted to the first master robot 1a is input from the user. Here, the user may be, for example, a training student, and may recover the operation right when the normal surgery is not performed by the user of the first master robot 1a.

If the request for retrieval authority has not been entered, the process proceeds to step 2050 again to allow the user to observe the actual operation of the surgical tool 460 by the first master robot 1a.

However, if the operation right recovery request is input, in step 2090, the second master robot 1b transmits the operation right termination command to the first master robot 1a through the communication network.

By transmitting the operation authority end command, the first master robot 1a may switch to an education mode in which the operation state of the actual surgical tool 460 by the second master robot 1b may be observed (step 2095). .

Up to now, the second master robot 1b has a control authority over the first master robot 1a with reference to FIG. 20. However, in the opposite case, the first master robot 1a may transmit a request for terminating the surgery right to the second master robot 1b. This is to transfer the authority to operate the actual surgical tool 460 by the user of the second master robot (1b), for example, the operation of the operation site is not easy or the operation of the operation site is very easy. As one case, it may be used when necessary for education.

In addition, various methods for transferring surgical or control rights among a plurality of master robots or allowing one master robot to proactively grant / recover can be considered and applied without limitation.

So far, various embodiments of the present invention have been described with reference to the accompanying drawings. However, the present invention is not limited by the above-described embodiments, and more various embodiments may further be presented.

In an embodiment, when a plurality of master robots are connected through a communication network and function in a fourth mode, which is an education mode, an evaluation function of a learner's master robot 1 control ability or surgical ability may be performed.

The evaluation function of the education mode is a virtual surgery tool 610 by training the arm operation unit 330 of the second master robot 1b while the training teacher is performing the operation using the first master robot 1a. It is executed in the process of controlling. The second master robot 1b receives the laparoscope image from the slave robot 2, interprets the characteristic value or the characteristic information about the actual surgical tool 460, and also performs the virtual surgery according to the operation of the arm operation unit 330 of the training student. Analyze the control process of the tool 610. Subsequently, the second master robot 1b is trained by analyzing the similarity between the movement trajectory and the operation form of the actual surgical tool 460 included in the laparoscopic image and the movement trajectory and the operation form of the virtual surgical tool 610 by the student. Evaluate scores for students.

In another embodiment, in the fifth mode, which is a simulation mode in which the virtual mode is further improved, the master robot 1 may operate as a surgical simulator by combining the characteristics of an organ with a three-dimensional shape obtained by using a stereoscopic endoscope.

For example, when liver is included in a laparoscopic image or a virtual screen output through the screen display unit 320, the master robot 1 extracts the liver liver information stored in the storage unit and is output to the screen display unit 320. The matching may also allow the surgical simulation to be performed during the surgery or in a virtual mode separately from the surgery. Which organs are included in the laparoscopic image can be interpreted by, for example, recognizing the color, shape, etc. of the corresponding organs using conventional image processing and recognition techniques, and comparing the recognized information with characteristics information of previously stored organs. Can be. Of course, which organs are included and / or which organs to perform a surgical simulation may be chosen by the operator.

Using this, the operator can perform a pre-surgical simulation of how and in what direction the liver should be excised using the shape of the liver matched with the characteristic information before actually cutting or cutting the liver. In the surgical simulation process, the master robot 1 has a hard part in which a surgical operation (for example, one or more of ablation, cutting, stitching, pulling, pressing, etc.) is performed based on characteristic information (for example, mathematical modeling information). Paper or soft paper may be transmitted to the operator.

As a method of transmitting the touch, for example, the force feedback processing or the sensitivity of the operation of the arm operation unit 330 or the resistance during operation (for example, the resistance force to block when pushing the arm operation unit 330 forward) ), Etc.).

In addition, the section of the organ virtually cut or cut by the operator's operation may be output through the screen display unit 320 to allow the operator to predict the result of actual cutting or cutting.

In addition, the master robot 1 functions as a surgical simulator in order to reconstruct the organ surface surface information reconstructed from reference image such as CT, MRI, and the like by using the stereoscopic endoscope through the screen display unit 320. By matching the three-dimensional shape and matching the three-dimensional shape and characteristic information (for example, mathematical modeling information) inside the organ reconstructed from the reference image, it may be possible for the operator to perform more realistic surgical simulation. The characteristic information may be characteristic information specific to the patient, or may be characteristic information generated for general use.

The control method of the surgical robot system using the augmented reality described above may be implemented as a software program. Codes and code segments constituting a program can be easily inferred by a computer programmer in the art. The program is also stored in a computer readable media, and read and executed by a computer to implement the method. The information storage medium includes a magnetic recording medium, an optical recording medium and a carrier wave medium.

It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit or scope of the invention as defined in the appended claims. It will be understood that the invention may be varied and varied without departing from the scope of the invention.

1 is a plan view showing the overall structure of a surgical robot according to an embodiment of the present invention.

2 is a conceptual diagram showing a master interface of a surgical robot according to an embodiment of the present invention.

Figure 3 is a block diagram schematically showing the configuration of a master robot and a slave robot according to an embodiment of the present invention.

Figure 4a is a diagram illustrating a driving mode of the surgical robot system according to an embodiment of the present invention.

4B illustrates a mode indicator indicative of a running drive mode in accordance with one embodiment of the present invention.

5 is a flowchart illustrating a driving mode selection process of a first mode and a second mode according to an embodiment of the present invention.

6 is an exemplary view of a screen display output through the monitor unit in the second mode according to an embodiment of the present invention.

7 is a view showing a detailed configuration of the augmented reality implementation unit 350 according to an embodiment of the present invention.

8 is a flowchart illustrating a method of driving a master robot in a second mode according to an embodiment of the present invention.

9 is a view showing a detailed configuration of the augmented reality implementation unit 350 according to another embodiment of the present invention.

10 and 11 are flowcharts illustrating a method of driving a master robot in a second mode according to another embodiment of the present invention, respectively.

12 is a block diagram schematically showing the configuration of a master robot and a slave robot according to another embodiment of the present invention.

13 is a flow chart illustrating a method for verifying normal operation of a surgical robot system according to another embodiment of the present invention.

14 is a view showing a detailed configuration of the augmented reality implementation unit according to another embodiment of the present invention.

15 and 16 are flowcharts showing a method of driving a master robot for outputting a virtual surgical tool according to another embodiment of the present invention, respectively.

17 is a flowchart illustrating a method of providing a reference image according to another embodiment of the present invention.

18 is a plan view showing the overall structure of a surgical robot according to another embodiment of the present invention.

19 illustrates a method of operating a surgical robot system in an educational mode according to another embodiment of the present invention.

20 illustrates a method of operating a surgical robot system in a training mode according to another embodiment of the present invention.

<Explanation of symbols for main parts of the drawings>

1, 1a, 1b: master robot 2: slave robot

3: robot arm 4: master interface

5: Laparoscope 6: Monitor

10: handle 14: clutch button

310: image input unit 320: screen display unit

330: arm operation unit 340: operation signal generation unit

350: augmented reality implementation unit 360: control unit

710: characteristic value calculator 720: virtual surgical tool generation unit

730: test signal processing unit 740: delay time calculation unit

910: interval calculation unit 920: image analysis unit

1210: network verification unit 1410: overlapping processing unit

1420: contact recognition unit

Claims (70)

An interface mounted on a master robot that controls a slave robot including a robot arm, A screen display unit displaying an endoscope image corresponding to an image signal provided from a surgical endoscope; One or more arm manipulation units provided to control the robot arms, respectively; And Master interface of the surgical robot including an augmented reality implementation unit for generating virtual surgical tool information according to a user operation using the cancer operation unit to display the virtual surgical tool through the screen display. The method of claim 1, The surgical endoscope is a master interface of the surgical robot, characterized in that at least one of the laparoscope, thoracoscopic, arthroscopy, parenteral, bladder, rectal, duodenum, mediastinoscope, cardiac. The method of claim 1, And a manipulation signal generator for generating a manipulation signal according to the user manipulation for controlling the robot arm and transmitting the manipulation signal to the slave robot. The method of claim 1, A drive mode selection unit for designating a drive mode of the master robot; And And a control unit for controlling one or more of the endoscope image and the virtual surgical tool to be displayed through the screen display unit in correspondence with the driving mode selected by the driving mode selecting unit. The method of claim 4, wherein The control unit is a master interface of the surgical robot, characterized in that for controlling the mode indicator corresponding to the selected driving mode is displayed on the screen display. The method of claim 5, The mode indicator is a master interface of a surgical robot, characterized in that predetermined in at least one of a text message, a border color, an icon, a background color. The method of claim 1, The slave robot further includes a biometric information measuring unit, The biometric information measured by the biometric information measuring unit is displayed on the screen display unit, the master interface of the surgical robot. The method of claim 1, The augmented reality implementation unit, A characteristic value calculator configured to calculate characteristic values using at least one of the endoscope image and position coordinate information of an actual surgical tool coupled to at least one robot arm; And Master interface of the surgical robot, characterized in that it comprises a virtual surgical tool generating unit for generating virtual surgical tool information according to the user operation using the arm operation unit. The method of claim 8, The characteristic value calculated by the characteristic value calculator includes one or more of an angle of view (FOV), an enlargement ratio, a viewpoint, a viewing depth, a type of the actual surgical tool, a direction, a depth, and a bending angle of the surgical endoscope. Master interface of the surgical robot, characterized in that. The method of claim 8, The augmented reality implementation unit, A test signal processor which transmits a test signal to the slave robot and receives a response signal according to the test signal from the slave robot; And And a delay time calculator configured to calculate a delay value for at least one of a network communication speed between the master robot and the slave robot and a delay time in network communication using the transmission time of the test signal and the reception time of the response signal. Master interface of the surgical robot, characterized in that. The method of claim 10, Further comprising a control unit for controlling one or more of the endoscope image and the virtual surgical tool to be displayed through the screen display, And the control unit controls to display only the endoscope image through the screen display unit when the delay value is less than or equal to a preset delay threshold. The method of claim 8, The augmented reality implementation unit, The master interface of the surgical robot further comprises an interval calculating unit for calculating the interval value between each surgical tool using the position coordinates of the actual surgical tool and the virtual surgical tool displayed through the screen display. The method of claim 12, And the virtual surgical tool generating unit processes the virtual surgical tool not to be displayed through the screen display unit when the interval value calculated by the interval calculating unit is equal to or less than a preset interval threshold value. The method of claim 12, The virtual surgical tool generating unit for the surgical operation, characterized in that for processing at least one of semi-transparency adjustment, color change and outline thickness change for the virtual surgical tool in proportion to the interval value calculated by the interval calculating unit The master interface of the robot. The method of claim 8, The augmented reality implementation unit, The master interface of the surgical robot further comprises an image analysis unit for extracting feature information through the image processing for the endoscope image displayed through the screen display. The method of claim 15, The characteristic information is a master interface of the surgical robot, characterized in that at least one of the color value of each pixel of the endoscope image, the position coordinates and the operation shape of the actual surgical tool. The method of claim 16, The image analyzer outputs a warning request when an area or quantity of pixels having a color value included in a preset color value range in the endoscope image exceeds a threshold value. According to the warning request, the master interface of the surgical robot, characterized in that one or more of the warning message display through the screen display unit, the warning sound output through the speaker unit and the display stop for the virtual surgical tool is performed. The method of claim 8, By using the position coordinate information of the actual surgical tool included in the characteristic value calculated by the characteristic value calculator and the position coordinate information of the virtual surgical tool included in the virtual surgical tool information generated by the virtual surgical tool generating unit. The master interface of the surgical robot further comprises a network verification unit for verifying the network communication state between the master robot and the slave robot. The method of claim 15, And further comprising a network verification unit for verifying a network communication state between the master robot and the slave robot by using the position coordinate information of the real surgical tool and the virtual surgical tool included in the feature information extracted by the image analyzer. Master interface of the robot. The method according to any one of claims 18 or 19, The network verification unit master interface of the surgical robot, characterized in that further using at least one of the movement trajectory and operation form of each surgical tool for verification of the network communication state. The method according to any one of claims 18 or 19, The network verification unit is a master interface of the surgical robot, characterized in that for verifying whether the position coordinate information of the virtual surgical tool is matched with the position coordinate information of the actual surgical tool previously stored within the error range . The method of claim 18 or 19, The network verification unit outputs a warning request if the position coordinate information of the actual surgical tool and the position coordinate information of the virtual surgical tool do not match within an error range, According to the warning request, one or more of the warning message display through the screen display unit, the warning sound output through the speaker unit and the display stop for the virtual surgical tool is performed. The method of claim 8, The augmented reality implementation unit, An image analyzer extracting feature information including region coordinate information of a surgical site or an organ displayed through the endoscope image through image processing of the endoscope image displayed through the screen display unit; And Using the virtual surgical tool information and the area coordinate information, it is determined whether the virtual surgical tool is located behind the area by generating overlap with the area coordinate information. When the overlap occurs, the overlap of the shape of the virtual surgical tool occurs. The master interface of the surgical robot further comprises an overlapping processing unit for concealing the closed area. The method of claim 8, The augmented reality implementation unit, An image analyzer extracting feature information including region coordinate information of a surgical site or an organ displayed through the endoscope image through image processing of the endoscope image displayed through the screen display unit; And The virtual surgical tool further includes a contact recognition unit configured to determine whether contact with the region coordinate information is generated using the virtual surgical tool information and the region coordinate information, and to perform a contact warning process when a contact is generated. Master interface for surgical robots. The method of claim 24, The touch alert processing is at least one of a force feedback process, an operation restriction of the arm operation unit, a warning message display through the screen display unit and a warning sound output through the speaker unit. The method of claim 1, A storage unit for storing a reference image which is one or more of an X-ray image, a computed tomography (CT) image, and a magnetic resonance imaging (MRI) image; And Further comprising an image analysis unit for recognizing the surgical site or the organ displayed through the endoscope image through the image processing for the endoscope image displayed through the screen display, And the reference image is displayed on a display screen independent of a display screen on which the endoscope image is displayed, corresponding to a name of an organ recognized by the image analyzer. The method of claim 8, Further comprising a storage unit for storing a reference image of at least one of an X-ray image, a computed tomography (CT) image and a magnetic resonance imaging (MRI) image, The reference image is displayed together on a display screen on which the endoscope image is displayed or on a display screen independent of the display screen, corresponding to the position coordinate information of the actual surgical tool calculated by the characteristic value calculator. Master interface for surgical robots. The method of claim 26 or 27, wherein The reference image is a master interface of a surgical robot, characterized in that displayed as a three-dimensional image using the Multi Planner Reformat (MPR). As a surgical robotic system, Two or more master robots coupled to one another via a communication network; And Surgical robot system including a slave (slave) robot including one or more robot arms controlled in accordance with the operation signal received from any master robot. 30. The method of claim 29, Each of the master robot, A screen display unit displaying an endoscope image corresponding to an image signal provided from a surgical endoscope; One or more arm manipulation units provided for respectively controlling one or more robot arms; And Surgical robot system comprising augmented reality implementation for generating a virtual surgical tool information according to a user operation using the cancer operation unit to display the virtual surgical tool through the screen display. 30. The method of claim 29, The operation of the arm operation unit of the first master robot, which is one of the at least two master robots, functions to generate the virtual surgical tool information, and the operation of the arm operation unit of the second master robot, which is the other of the at least two master robots, is the robot. Surgical robot system, characterized in that it functions for the control of cancer. The method of claim 31, wherein And a virtual surgical tool corresponding to the virtual surgical tool information according to the operation of the arm manipulation unit of the first master robot is displayed on the screen display of the second master robot. A control method of a surgical robot system performed in a master robot for controlling a slave robot including a robot arm, Displaying an endoscope image corresponding to an image signal input from a surgical endoscope; Generating virtual surgical tool information according to manipulation of the arm manipulation unit; And And controlling the virtual surgical tool corresponding to the virtual surgical tool information to be displayed together with the endoscope image. 34. The method of claim 33, The surgical endoscope is a control method of a surgical robot system, characterized in that at least one of the laparoscope, thoracoscopic, arthroscopy, parenteral, bladder, rectal, duodenum, mediastinoscope, cardiac. 34. The method of claim 33, Generating the virtual surgical tool information, Receiving manipulation information according to manipulation of the arm manipulation unit; And Generating an operation signal for controlling the virtual surgical tool information and the robot arm according to the operation information, And the manipulation signal is transmitted to the slave robot for controlling the robot arm. 34. The method of claim 33, Receiving a driving mode selection command for designating a driving mode of the master robot; And And controlling one or more of the endoscope image and the virtual surgical tool to be displayed through a screen display according to the driving mode selection command. The method of claim 36, And controlling a mode indicator corresponding to the drive mode designated by the drive mode selection command to be displayed on the screen display unit. The method of claim 37, The mode indicator is a control method of a surgical robot system, characterized in that predetermined in at least one of a text message, a border color, an icon, a background color. 34. The method of claim 33, Receiving biometric information measured from the slave robot; And And displaying the biometric information in a display area independent of the display area in which the endoscope image is displayed. 34. The method of claim 33, Computing a characteristic value using at least one of the endoscope image and the position coordinate information of the actual surgical tool coupled to the robot arm, The characteristic value may include at least one of a field of view (FOV), an enlargement ratio, a viewpoint, a viewing depth, a type, a direction, a depth, and a bending angle of the surgical endoscope. Control method. 34. The method of claim 33, Transmitting a test signal to the slave robot; Receiving a response signal according to the test signal from the slave robot; And And calculating a delay value for at least one of a network communication speed between the master robot and the slave robot and a delay time in network communication using the transmission time of the test signal and the reception time of the response signal. How to control the system. The method of claim 41, wherein The step of displaying the virtual surgical tool with the endoscope image, Determining whether the delay value is equal to or less than a preset delay threshold value; If the delay threshold is exceeded, processing the virtual surgical instrument to be displayed together with the endoscope image; And And controlling the endoscope image to display only the endoscope image when the delay threshold is less than or equal to the delay threshold value. 34. The method of claim 33, Calculating a position coordinate of the displayed endoscope image and the displayed virtual surgical tool including an actual surgical tool; And Comprising a step of calculating the interval value between each surgical tool using the position coordinates of each surgical tool. The method of claim 43, The step of displaying the virtual surgical tool with the endoscope image, Determining whether the interval value is equal to or less than a preset interval threshold value; And And controlling the virtual surgical tool to be displayed together with the endoscope image only in the following cases. The method of claim 43, The step of displaying the virtual surgical tool with the endoscope image, Determining whether the interval value exceeds a preset interval threshold; And If exceeding, the method of controlling the surgical robot system characterized in that it comprises the step of displaying the virtual surgical tool is processed with one or more of the semi-transparency control, color change and outline thickness change is displayed with the endoscope image. The method of claim 43, Determining whether or not the position coordinates of the respective surgical instruments coincide within a preset error range; And And verifying a communication state between the master robot and the slave robot based on a result of the determination. 47. The method of claim 46 wherein In the determining step, it is determined whether the current position coordinates for the virtual surgical tool and the previous position coordinates for the actual surgical tool is within the error range is determined. 47. The method of claim 46 wherein In the determining step, the control method of the surgical robot system, characterized in that it is further determined whether one or more of the movement trajectory and operation form of each surgical tool is within the error range. 34. The method of claim 33, Extracting feature information including color values of pixels from the displayed endoscope image; Determining whether an area or a quantity of pixels having a color value included in a preset color value range in the endoscope image exceeds a threshold value; And If exceeded, the control method of the surgical robot system further comprising the step of outputting warning information. The method of claim 49, And controlling at least one of displaying a warning message, outputting a warning sound, and stopping the display of the virtual surgical tool according to the warning request. 34. The method of claim 33, The step of displaying the virtual surgical tool with the endoscope image, Extracting region coordinate information of an organ displayed through a surgical site or the endoscope image through image processing of the endoscope image; Determining whether the virtual surgical tool is located behind the virtual surgical tool by using the virtual surgical tool information and the region coordinate information and overlapping the region coordinate information; And If the overlap is generated, the control method of the surgical robot system comprising the step of concealing the area in which the overlap occurs in the shape of the virtual surgical tool. 34. The method of claim 33, Extracting region coordinate information of an organ displayed through a surgical site or the endoscope image through image processing of the endoscope image; Determining whether the virtual surgical tool is in contact with the area coordinate information by using the virtual surgical tool information and the area coordinate information; And And performing contact warning processing when a contact is generated. The method of claim 52, wherein And the contact warning processing is at least one of a force feedback processing, an operation restriction of the arm manipulation unit, a warning message display, and a warning sound output. 34. The method of claim 33, Recognizing an organ displayed on the surgical site or the endoscope image through image processing of the endoscope image; And Extracting and displaying a reference image of a position corresponding to the recognized name of the organ from a pre-stored reference image, The reference image is a control method of a surgical robot system, characterized in that at least one of the X-ray (X-ray) image, computed tomography (CT) image and magnetic resonance imaging (MRI) image. The method of claim 40, Extracting a reference image corresponding to the position coordinates of the actual surgical instrument from a pre-stored reference image; And Extracting and displaying the extracted reference image, The reference image is a control method of a surgical robot system, characterized in that at least one of the X-ray (X-ray) image, computed tomography (CT) image and magnetic resonance imaging (MRI) image. The method of claim 54 or 55, wherein And the reference image is displayed together on a display screen on which the endoscope image is displayed or is displayed through a display screen independent of the display screen. The method of claim 54 or 55, wherein The reference image is a control method of a surgical robot system, characterized in that displayed as a three-dimensional image using a multi-plane reformat (MPR). A program of instructions that can be executed by a digital processing apparatus is tangibly embodied in order to perform the control method of the surgical robot system according to any one of claims 33 to 55, and a program that can be read by the digital processing apparatus. Recorded media. A method of operation of a surgical robot system comprising a slave robot including a robot arm and a master robot controlling the slave robot, Generating, by a first master robot, virtual surgical tool information for displaying a virtual surgical tool and an operation signal for controlling the robot arm according to an operation of the arm manipulation unit; And And transmitting, by the first master robot, the manipulation signal to the slave robot, and transmitting one or more of the manipulation signal or the virtual surgical tool information to a second master robot. And the second master robot displays a virtual surgical tool corresponding to at least one of the manipulation signal and the virtual surgical tool information through a screen display. The method of claim 59, Each of the first master robot and the second master robot displays an endoscope image received from the slave robot through a screen display unit, and the virtual surgical tool is displayed together with the endoscope image. Way. The method of claim 59, Determining, by the first master robot, whether an operation right recovery command has been received from the second master robot; And And when the surgical authority recovery command is received, the first master robot controlling the operation of the arm manipulation unit so that the operation only functions for generation of the virtual surgical tool information. A surgical simulation method performed in a master robot for controlling a slave robot including a robot arm, Recognizing organ selection information; And And displaying a 3D organ image corresponding to the organ selection information by using previously stored organ modeling information. And the organ modeling information has characteristic information including one or more of the shape, color, and feel of each point of the inside and outside of the corresponding organ. The method of claim 62, In order to recognize the organ selection information, Interpreting information on at least one of color and appearance of an organ included in a surgical site by using an image signal input from a surgical endoscope; And And a step of recognizing organs matching the interpreted information among pre-stored organ modeling information. The method of claim 62, And the organ selection information is input by the operator as one or more organs. The method of claim 62, Receiving a surgical operation command on the 3D organ image according to an operation of an arm manipulation unit; And And outputting the tactile information according to the surgical operation command by using the long-term modeling information. 66. The method of claim 65, And the tactile information is control information for controlling at least one of manipulation sensitivity and manipulation resistance according to manipulation of the arm manipulation unit, or control information for force feedback processing. The method of claim 62, Receiving a surgical operation command on the 3D organ image according to an operation of an arm manipulation unit; And And displaying a manipulation result image according to the surgery manipulation command using the organ modeling information. The method of any one of claims 65 and 67, The surgical operation command is a surgical simulation method, characterized in that at least one of cutting, stitching, pulling, pressing, long-term deformation due to contact, organ damage by electrosurgery, bleeding in the blood vessels. The method of claim 62, Recognizing an organ according to the organ selection information; And The method may further include extracting and displaying a reference image of a position corresponding to the recognized name of the organ from a pre-stored reference image. And the reference image is at least one of an X-ray image, a computed tomography (CT) image, and a magnetic resonance imaging (MRI) image. 70. A recording medium on which a program of instructions that can be executed by a digital processing apparatus is tangibly implemented to perform the surgical simulation method according to any one of claims 62 to 69, and which records a program that can be read by the digital processing apparatus. .

Images