KR101447931B1 - Surgical robot system using augmented reality and control method thereof - Google Patents

Abstract

A surgical robot system using an augmented reality and a control method thereof are disclosed. An interface mounted on a master robot for controlling a slave robot including a robot arm, comprising: a screen display unit for displaying an endoscopic image corresponding to a video signal provided from a surgical endoscope; The master interface of the surgical robot including the manipulation unit and the augmented reality implementation unit for generating the virtual surgery tool information according to the user operation using the arm operation unit to display the virtual surgery tool through the screen display unit, By displaying the tool together with the virtual surgical tool, the operator can perform a smooth operation.

Description

Technical Field [0001] The present invention relates to a surgical robot system using augmented reality,

The present invention relates to surgery, and more particularly, to a surgical robot system using an augmented reality and a control method thereof.

The surgical robot refers to a robot having a function of substituting surgical operation performed by a surgeon. Such a surgical robot has an advantage that it can perform accurate and precise operation compared to a human and can perform a remote operation.

Surgical robots currently being developed around the world include bone surgery robots, laparoscopic surgery robots, and stereotactic robots. Here, a laparoscopic surgical robot is a robot that performs minimally invasive surgery using laparoscopic and small surgical instruments.

Laparoscopic surgery is an advanced surgical technique that inserts a laparoscope, which is an endoscope that penetrates a 1 cm hole in the umbilicus and looks into the abdomen.

A recent laparoscope is equipped with a computer chip that provides a sharper, larger image than what the naked eye can see, and the laparoscopic surgical instruments specially designed for viewing through the monitor have been developed to allow any surgery.

In addition, laparoscopic surgery is similar to laparotomy, but it has fewer complications than laparotomy, can start treatment within a short time after surgery, and has the advantage of maintaining the patient's physical strength and immune function have. Therefore, in the United States and Europe, laparoscopic surgery is becoming increasingly recognized as a standard surgery for colorectal cancer treatment and the like.

Surgical robot systems generally consist of a master robot and a slave robot. When the operator manipulates a manipulator (e.g., a handle) provided in the master robot, the surgical tool held by the robot arm is operated by being coupled to the robot arm of the slave robot.

The master robot and the slave robot are connected through a communication network to perform network communication. At this time, if the network communication speed is not sufficiently fast, the operation is not performed until the operation signal transmitted from the master robot is received by the slave robot or / and the laparoscopic image transmitted from the laparoscopic camera mounted on the slave robot is received by the master robot It takes a lot of time.

It is generally known that the network communication speed between the master and slave robots must be within 150 ms. If the communication speed is further delayed, the movement of the operator's hand and the movement of the slave robot seen through the screen do not coincide with each other, so that the operator feels uncomfortable.

In addition, when the network communication speed between the master robot and the slave robot is slow, the operator considers the motion of the slave robot displayed on the screen or anticipates and predicts the operation of the slave robot. This causes unnatural movements, and in severe cases, normal operation can not be performed.

The present invention relates to a surgical robot system using an augmented reality and a control method thereof, in which an actual surgical tool and a virtual surgical tool are displayed together using an augmented reality, thereby enabling smooth operation of the operator.

In addition, the present invention provides a surgical robot system using an augmented reality capable of outputting various information about a patient during surgery and a control method thereof.

It is another object of the present invention to provide a surgical robot system and a control method thereof using an augmented reality that can diversify a surgical screen display method so that an operator can smoothly proceed according to a network communication speed between a master robot and a slave robot.

The present invention also provides a surgical robot system using an augmented reality capable of automatically notifying an operator of an emergency situation by automatically processing an image input through an endoscope or the like, and a control method thereof.

In addition, the present invention can be applied to augmented reality in which a long-term contact or the like due to movement of a virtual surgical tool by manipulation of a master robot can be sensed in real time by a surgeon and intuitive recognition of a long- A surgical robot system and a control method thereof.

In addition, the present invention provides a surgical robot system using an augmented reality which enables the progression of surgery using a variety of information because the related image data (e.g., CT image, MRI image, etc.) And a control method thereof.

In addition, the present invention provides a surgical robot system using an augmented reality capable of maximizing a real-time educational effect by allowing a surgical robot system to be compatible and shared between a learner and a trainer, and a control method thereof .

The present invention also provides a surgical robot system and a control method thereof using an augmented reality in which a progress of a surgical procedure can be predicted in advance using a three-dimensional modeled virtual organ.

The technical problems other than the present invention can be easily understood from the following description.

According to one aspect of the present invention, a surgical robot system using augmented reality, a slave robot, and a master robot are provided.

According to an embodiment of the present invention, there is provided an interface mounted on a master robot for controlling a slave robot including at least one robot arm, the interface including: an endoscope corresponding to a video signal provided from a surgical endoscope; And a virtual operation tool information display unit for displaying virtual surgery tool information according to a user operation using a dark operation unit to display a virtual surgery tool through the screen display unit, A master interface of a surgical robot including an augmented reality implementing section is provided.

Surgical endoscopy may be one or more of laparoscopic, thoracoscopic, arthroscopic, non-operative, cystoscopic, rectal, duodenal, mediastinal, and cardiac.

The master interface of the surgical robot may further include an operation signal generation unit for generating an operation signal according to a user operation for controlling the robot arm and transmitting the operation signal to the slave robot.

The master interface of the surgical robot includes a drive mode selection unit for designating a drive mode of the master robot and a display unit for displaying at least one of the endoscopic image and the virtual surgery tool through the screen display unit corresponding to the drive mode selected through the drive mode selection unit And a control unit for controlling the display unit.

The controller may control the display of the mode indicator corresponding to the selected drive mode through the screen display unit. The mode indicator may be predefined by one or more of a text message, a border color, an icon, a background color, and the like.

The slave robot may further include a bio-information measurement unit. The biometric information measured by the biometric information measuring unit can be displayed on the screen display unit.

The augmented reality implementation unit includes a feature value calculation unit that calculates a feature value using at least one of an endoscopic image and positional coordinate information of an actual surgical tool coupled to the at least one robot arm, And a virtual surgery tool generation unit for generating a virtual surgery tool.

The characteristic value calculated by the characteristic value calculation unit may include at least one of an angle of view (FOV) of a surgical endoscope, an enlargement ratio, a viewpoint, a viewing depth, a kind of an actual surgical tool, a direction, a depth and a bending angle.

The augmented reality implementing unit includes a test signal processing unit that transmits a test signal to the slave robot and receives a response signal in response to the test signal from the slave robot and a test signal processing unit that tests the master robot and the slave robot And a delay time calculation unit for calculating a delay value for at least one of a network communication speed between the base station and the base station, and a network communication delay time.

The master interface may further include a control unit for controlling one or more of the endoscopic image and the virtual surgical tool to be displayed through the screen display unit. Here, when the delay value is equal to or less than a preset delay threshold value, the controller may control only the endoscopic image to be displayed through the screen display unit.

The augmented reality implementing unit may further include an interval calculating unit for calculating an interval value of each operation section using the coordinates of the actual operation tool and the virtual operation tool displayed through the screen display unit.

The virtual surgery tool generation unit may process the virtual surgery tool so that the virtual surgery tool is not displayed on the screen display unit when the interval value calculated by the interval operation unit is equal to or less than a preset interval threshold value.

The virtual surgery tool creation unit may perform processing for at least one of adjusting translucency, color change, and outline thickness change for the virtual surgical tool in proportion to the interval value calculated by the interval calculation unit.

The augmented reality implementing unit may further include an image analyzing unit for extracting the feature information through image processing on the endoscopic image displayed through the screen display unit. Here, the feature information may be at least one of a color value per pixel of the endoscopic image, a position coordinate of the actual surgical tool, and an operation shape.

The image analyzing unit may output a warning request when the area or the number of pixels having a color value included in the color value range set in advance in the endoscopic image exceeds a threshold value. According to the warning request, at least one of a warning message display through a screen display unit, a warning sound output through a speaker unit, and a display stop for a virtual surgical tool can be performed.

The master interface uses the position coordinate information of the actual surgical tool contained in the characteristic value calculated by the characteristic value calculation unit and the position coordinate information of the virtual surgery tool included in the virtual surgical tool information generated by the virtual surgery tool generation unit And a network verifying unit for verifying a network communication state between the master robot and the slave robot.

The master interface may further include a network verifying unit for verifying a network communication state between the master robot and the slave robot using each position coordinate information of the actual surgical tool and the virtual surgical tool included in the feature information extracted by the video analyzing unit have.

The network verification unit may further use at least one of the movement trajectory and the operation mode of each surgical tool to verify the network communication state.

The network verification unit can verify the network communication status by determining whether the position coordinate information of the virtual surgery tool matches the position coordinate information of the actual surgical tool stored in advance within the error range.

The network verification unit can output a warning request when the position coordinate information of the actual surgical tool and the position coordinate information of the virtual surgery tool do not match within the error range. According to the warning request, at least one of a warning message display through a screen display unit, a warning sound output through a speaker unit, and a display stop for a virtual surgical tool can be performed.

The augmented reality implementing unit includes an image analyzing unit that extracts feature information including area coordinate information displayed on a surgical site or an endoscopic image through image processing of an endoscopic image displayed through a screen display unit, The virtual surgical tool information and the area coordinate information are used to determine whether or not the virtual surgical tool overlaps with the area coordinate information and is located at the back side. When overlapping occurs, the area of the virtual surgery tool where the overlap occurs is concealed And a superimposing unit for processing the superimposed image data.

The augmented reality implementing unit includes an image analyzing unit that extracts feature information including area coordinate information displayed on a surgical site or an endoscopic image through image processing of an endoscopic image displayed through a screen display unit, The contact detection unit may further include a contact recognition unit for determining whether or not the virtual surgery tool has contacted with the area coordinate information using the virtual surgery tool information and the area coordinate information and performing the contact warning processing when the contact is generated.

The contact alert process may be at least one of a force feedback process, an operation restriction of a dark operation unit, a warning message display through a screen display unit, and a warning sound output through a speaker unit.

The master interface includes a storage unit for storing a reference image that is at least one of an X-ray image, a computed tomography (CT) image, and a magnetic resonance imaging (MRI) image, And an image analyzing unit for recognizing an organ displayed on a surgical site or an endoscopic image through image processing on the image. The reference image can be displayed on the display screen independent of the display screen on which the endoscopic image is displayed in accordance with the name of the organ recognized by the image analyzing unit.

The master interface may further include a storage unit for storing a reference image that is at least one of an X-ray image, a computed tomography (CT) image, and a magnetic resonance imaging (MRI) image. The reference image may be displayed together with the display screen on which the endoscopic image is displayed or on the display screen independent of the display screen in correspondence with the position coordinate information of the actual surgical tool calculated by the characteristic value calculation unit.

The reference image can be displayed as a three-dimensional image using MPR (Multi Planar Reformat).

According to another embodiment of the present invention there is provided a surgical robot system comprising two or more master robots coupled to each other via a communication network and one or more robotic arms controlled in accordance with an operation signal received from any of the master robots A surgical robot system including a slave robot is provided.

Each of the master robots includes a screen display unit for displaying an endoscopic image corresponding to a video signal provided from a surgical endoscope, at least one arm control unit provided for controlling each of the at least one robot arm, and a virtual operation tool And an augmented reality implementing unit for generating virtual surgery tool information according to a user operation using the arm operation unit to display the virtual surgery tool information.

The manipulation of the arm manipulation part of the first master robot, which is one of the two or more master robots, functions to generate virtual surgical tool information, and the manipulation of the arm manipulation part of the second master robot, which is another one of the two or more master robots, Can function.

A virtual surgical tool corresponding to the virtual surgical tool information according to the operation of the dark operation unit of the first master robot can be displayed through the screen display unit of the second master robot.

According to another aspect of the present invention, there is provided a control method of a surgical robot system, a method of operating a surgical robot system, and a recording medium on which a program for implementing the respective methods is recorded.

According to an embodiment of the present invention, there is provided a method of controlling a surgical robot system performed by a master robot that controls a slave robot including at least one robot arm, the method comprising: displaying an endoscopic image corresponding to a video signal input from a surgical endoscope; A step of generating virtual surgery tool information according to the operation of the arm operation unit and a step of displaying the virtual surgery tool corresponding to the virtual surgery tool information together with the endoscope image, .

Surgical endoscopy may be one or more of laparoscopic, thoracoscopic, arthroscopic, non-operative, cystoscopic, rectal, duodenal, mediastinal, and cardiac.

The step of generating the virtual surgical tool information may include the steps of receiving the manipulation information according to the manipulation of the arm manipulation unit and generating manipulation signals for controlling the virtual manipulation tool information and the robot arm according to the manipulation information . The operation signal can be transmitted to the slave robot for control of the robot arm.

A control method of a surgical robot system includes the steps of receiving a drive mode selection command for designating a drive mode of a master robot and controlling the display mode to display at least one of an endoscopic image and a virtual surgical tool through a screen display unit according to a drive mode selection command The method comprising the steps of: The method may further include controlling a mode indicator corresponding to the drive mode designated by the drive mode selection command to be displayed on the screen display unit.

The mode indicator may be predefined by one or more of a text message, a border color, an icon, a background color, and the like.

The control method of the surgical robot system may further include receiving biometric information measured by the slave robot and displaying the biometric information in a display area independent of the display area where the endoscopic image is displayed.

The control method of the surgical robot system may further include calculating the characteristic value using at least one of the endoscopic image and the position coordinate information of the actual surgical tool coupled to the robot arm. The characteristic value may include at least one of an angle of view (FOV) of a surgical endoscope, a magnification, a viewpoint, a viewing depth, a kind of an actual surgical tool, a direction, a depth and a bending angle.

A method of controlling a surgical robot system includes transmitting a test signal to a slave robot, receiving a response signal according to a test signal from the slave robot, and transmitting the test signal to the master robot Calculating a delay value for at least one of a network communication speed between the slave robots and a network communication delay time.

The step of causing the virtual surgical instrument to be displayed with the endoscopic image may include determining whether the delay value is less than or equal to a preset delay threshold value, And processing so that only the endoscopic image is displayed if it is equal to or less than the delay threshold value.

The control method of the surgical robot system includes calculating a position coordinate of an endoscopic image and a virtual surgical tool to be displayed including an actual surgical tool and calculating an interval value of each operation section using the position coordinates of each surgical tool Step < / RTI >

The step of causing the virtual surgical tool to be displayed with the endoscopic image includes determining whether the interval value is less than or equal to a preset interval threshold value and processing the virtual surgical tool to be displayed with the endoscopic image only when .

In addition, the step of causing the virtual surgical tool to be displayed with the endoscopic image further comprises the steps of: determining whether the interval value exceeds a preset interval threshold; and if so, selecting one of semitransparency adjustment, color change and outline thickness change And processing the processed virtual surgery tool to be displayed together with the endoscopic image.

The control method of the surgical robot system further includes a step of determining whether or not the position coordinates of each surgical tool coincide with each other within a predetermined error range and verifying the communication state between the master robot and the slave robot based on the determination result .

In the judging step, it can be judged whether or not the current position coordinates of the virtual surgical tool and the previous position coordinates of the actual surgical tool coincide within the error range.

Further, in the judging step, it may be further judged whether or not at least one of the movement trajectory and the operation form of each surgical tool coincide within the error range.

A method of controlling a surgical robot system includes the steps of extracting feature information including a color value for each pixel in an endoscopic image to be displayed, determining an area or a quantity of pixels having a color value included in a preset color value range in the endoscopic image, , And outputting warning information if it is exceeded.

Depending on the alert request, one or more of a warning message display, a warning sound output and a display stop for the virtual surgical instrument may be performed.

The step of displaying the virtual surgical tool together with the endoscopic image may include extracting area coordinate information of an organ displayed on a surgical site or an endoscopic image through image processing on the endoscopic image, A step of determining whether the virtual surgery tool is overlapped with the area coordinate information and located behind the virtual surgery tool using the information and the area coordinate information, .

The control method of the surgical robot system includes the steps of extracting area coordinate information of an organ displayed through a surgical site or an endoscopic image through image processing on an endoscopic image, using virtual surgery tool information and area coordinate information Determining whether or not the virtual surgery tool has contacted with the area coordinate information, and performing the contact warning processing when the contact is generated.

The contact alert processing may be one or more of force feedback processing, operation restriction of the dark control section, warning message display, and warning sound output.

A control method of a surgical robot system includes the steps of recognizing an organ displayed through a surgical site or an endoscopic image through image processing on an endoscopic image, And extracting and displaying a reference image of the reference image. Here, the reference image may be at least one of an X-ray image, a computed tomography (CT) image, and a magnetic resonance imaging (MRI) image.

The control method of the surgical robot system may include extracting a reference image corresponding to the coordinates of the actual surgical tool from a reference image stored in advance, and extracting and displaying the extracted reference image. The reference image may be at least one of an X-ray image, a computed tomography (CT) image, and a magnetic resonance imaging (MRI) image.

The reference image may be displayed together with the display screen on which the endoscopic image is displayed or on the display screen independent of the display screen.

The reference image may be displayed as a three-dimensional image using MPR (Multi Planar Reformat).

According to another embodiment of the present invention, there is provided a method of operating a surgical robot system including a master robot for controlling a slave robot and a slave robot including at least one robot arm, A step of generating a manipulation signal for controlling the robot arm and virtual surgery tool information for displaying the tool, a step of transmitting the manipulation signal to the slave robot, 2 master robot, wherein the second master robot displays a virtual surgical tool corresponding to at least one of an operation signal or virtual surgical tool information through a screen display unit, / RTI >

Each of the first master robot and the second master robot displays an endoscopic image received from the slave robot through a screen display unit and the virtual surgical tool is displayed together with the endoscopic image.

The operation method of the surgical robot system according to the present invention is characterized in that the first master robot judges whether or not a surgical operation right recovery command has been received from the second master robot, and when the operation right recovery instruction has been received, And controlling to function only for generation of the surgical tool information.

According to another embodiment of the present invention, there is provided a surgical simulation method performed by a master robot for controlling a slave robot including a robot arm, the method comprising: recognizing long-term organ selection information; And displaying the three-dimensional long-term image corresponding to the long-term selection information, wherein the long-term modeling information is characterized by having characteristic information including at least one of shape, color, and touch of each internal and external point of the corresponding organ A surgical simulation method is provided.

Analyzing information on at least one of color and appearance of organs contained in a surgical site using a video signal input from a surgical endoscope to recognize long term selection information; A step of recognizing the organ matched to the user can be performed.

The long-term selection information may be selected by the operator as one or more organs.

The method may further include receiving a surgical operation command for a three-dimensional long-term image according to the operation of the dark operation unit, and outputting tactile information according to a surgical operation command using the long-term modeling information.

The tactile information may be control information for controlling at least one of the operation sensitivity and the operation resistance according to the operation of the dark operation unit, or the control information for the force feedback process.

Receiving a surgical operation command for a three-dimensional long-term image according to the operation of the arm operation unit, and displaying a sectional image according to a surgical operation command using the long-term modeling information.

The above-described surgical operation command may be at least one of cutting, stitching, pulling, pushing, long-term deformation, organs damage by electrosurgery, bleeding in a blood vessel, and the like.

The method may further include recognizing an organ according to the organ selection information and extracting and displaying a reference image at a position corresponding to a name of the organ recognized in the reference image stored in advance. Here, the reference image may be at least one of an X-ray image, a computed tomography (CT) image, and a magnetic resonance imaging (MRI) image.

Other aspects, features, and advantages will become apparent from the following drawings, claims, and detailed description of the invention.

According to the embodiment of the present invention, an actual surgical tool and a virtual surgical tool are displayed together using an augmented reality, thereby enabling smooth operation of the operator.

In addition, various information about the patient is outputted at the time of operation, and the operation is provided to the operator.

In addition, there is an effect that the operation screen display method is diversified so that the operator can smoothly perform the operation according to the network communication speed between the master robot and the slave robot.

In addition, there is also an effect that an image inputted through an endoscope or the like is automatically processed and an emergency situation is immediately notified to the operator.

In addition, long-term contact or the like due to the movement of the virtual surgical tool by the operation of the master robot can be detected in real time by the operator, and the long-term positional relationship with the virtual surgical tool can be intuitively recognized.

In addition, related image data (e.g., a CT image, an MRI image, etc.) of the patient with respect to the surgical site can be presented in real time, so that the operation can be performed using various information.

In addition, the surgical robot system can be compatible and shared between the learner and the trainer, thereby maximizing the real-time educational effect.

The present invention also has the effect of predicting the progress and results of an actual surgical procedure using a three-dimensional modeled virtual organ.

1 is a plan view showing the entire structure of a surgical robot according to an embodiment of the present invention;
2 is a conceptual diagram illustrating a master interface of a surgical robot according to an embodiment of the present invention.
3 is a block diagram schematically showing the configuration of a master robot and a slave robot according to an embodiment of the present invention.
FIG. 4A illustrates a driving mode of a surgical robot system according to an embodiment of the present invention; FIG.
FIG. 4B illustrates a mode indicator representing a running mode of operation in accordance with an embodiment of the present invention. FIG.
5 is a flowchart illustrating a driving mode selection process of a first mode and a second mode according to an embodiment of the present invention.
6 is an exemplary view of a screen display output through a monitor unit in a second mode according to an embodiment of the present invention;
7 is a detailed configuration of an augmented reality implementing unit 350 according to an embodiment of the present invention.
8 is a flowchart illustrating a method of driving a master robot in a second mode according to an embodiment of the present invention.
9 is a detailed configuration of an augmented reality implementing unit 350 according to another embodiment of the present invention.
10 and 11 are flowcharts showing a method of driving a master robot in a second mode according to another embodiment of the present invention, respectively.
12 is a block diagram schematically showing the configuration of a master robot and a slave robot according to another embodiment of the present invention.
13 is a flowchart illustrating a method for verifying normal operation of a surgical robot system according to another embodiment of the present invention.
14 is a detailed configuration of an augmented reality implementing unit according to another embodiment of the present invention.
15 and 16 are flowcharts showing a driving method of a master robot for outputting a virtual surgical tool according to another embodiment of the present invention, respectively.
17 is a flowchart showing a reference image providing method according to another embodiment of the present invention.
18 is a plan view showing the entire structure of a surgical robot according to another embodiment of the present invention.
19 is a diagram illustrating a method of operating a surgical robot system in a training mode according to another embodiment of the present invention.
20 illustrates a method of operating a surgical robot system in a training mode according to another embodiment of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS The present invention is capable of various modifications and various embodiments, and specific embodiments are illustrated in the drawings and described in detail in the detailed description. It is to be understood, however, that the invention is not to be limited to the specific embodiments, but includes all modifications, equivalents, and alternatives falling within the spirit and scope of the invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, the present invention will be described in detail with reference to the accompanying drawings.

The terms first, second, etc. may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another.

The terminology used in this application is used only to describe a specific embodiment and is not intended to limit the invention. The singular expressions include plural expressions unless the context clearly dictates otherwise. In the present application, the terms "comprises" or "having" and the like are used to specify that there is a feature, a number, a step, an operation, an element, a component or a combination thereof described in the specification, But do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or combinations thereof.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. Referring to the accompanying drawings, the same or corresponding components are denoted by the same reference numerals, .

Furthermore, in describing various embodiments of the present invention, it is not necessary for each embodiment to be interpreted or practiced independently, and the technical ideas described in each embodiment may be interpreted or practiced in combination with other embodiments, It should be understood that there is.

In addition, while the present invention is a technical idea that can be used universally for surgeries using surgical endoscopes (e.g., laparoscopic, thoracoscopic, arthroscopic, non-rigid, etc.) For convenience, a laparoscope is used as an example.

FIG. 1 is a plan view showing the entire structure of a surgical robot according to an embodiment of the present invention, and FIG. 2 is a conceptual view illustrating a master interface of a surgical robot according to an embodiment of the present invention.

1 and 2, the robot system for laparoscopic surgery includes a slave robot 2 for performing surgery on a patient lying on the operating table and a master robot 1 for remotely controlling the slave robot 2 do. The master robot 1 and the slave robot 2 are not necessarily separated from each other by a separate device that is physically independent and may be integrated into one unit. In this case, the master interface 4 is, for example, It can correspond to the interface part.

The master interface 4 of the master robot 1 includes a monitor unit 6 and a master manipulator and the slave robot 2 includes a robot arm 3 and a laparoscope 5. [ The master interface 4 may further include a mode switching control button. The mode changeover control button may be implemented in the form of a clutch button 14 or a pedal (not shown), and an embodiment of the mode changeover control button is not limited thereto. For example, A function menu, a mode selection menu, or the like.

The master interface 4 is provided with a master controller so that the operator can be grasped by both hands and operated. The master manipulator can be implemented with two handles 10 as illustrated in Figures 1 and 2, and an operation signal according to operation of the operating handle 10 is transmitted to the slave robot 2, Respectively. The robot arm 3 can be moved, rotated, cut, or the like by operating the handle 10 of the operator.

For example, the handle 10 may comprise a main handle and a sub handle. The slave robot arm 3 or the laparoscope 5 can be operated with only one handle or a plurality of surgical instruments can be operated simultaneously in real time by adding a sub handle. The main handle and the sub handle may have various mechanical configurations according to their operation modes. For example, a robot arm 3 of a slave robot 2 such as a joystick type, a keypad, a trackball, a touch screen, Various input means for operating the equipment can be used.

The master manipulator is not limited to the shape of the handle 10, and can be applied without any limitation as long as it can control the operation of the robot arm 3 via the network.

In the monitor section 6 of the master interface 4, an image input by the laparoscope 5 is displayed as an image image. In addition, the virtual surgical tool controlled by the operation of the handle 10 of the operator may be displayed on the monitor unit 6 or may be displayed on an independent screen. The information displayed on the monitor unit 6 may vary depending on the selected drive mode. The display state of the virtual surgical tool, the control method, the display information for each drive mode, and the like will be described in detail with reference to related drawings.

The monitor unit 6 may be constituted by one or more monitors, and information necessary for operation may be individually displayed on each monitor. 1 and 2 illustrate the case where the monitor unit 6 includes three monitors, but the number of monitors can be variously determined according to the type and kind of information required to be displayed.

The monitor unit 6 may further output a plurality of pieces of biometric information for the patient. In this case, one or more biometric information such as an index indicating the state of the patient, for example, body temperature, pulse, respiration, blood pressure, etc., may be outputted through one or more monitors of the monitor unit 6, May be output. In order to provide such biometric information to the master robot 1, the slave robot 2 includes a biometric information measurement unit including at least one of a body temperature measurement module, a pulse measurement module, a breath measurement module, a blood pressure measurement module, . ≪ / RTI > The biometric information measured by each module may be transmitted from the slave robot 2 to the master robot 1 in the form of an analog signal or a digital signal and the master robot 1 transmits the received biometric information to the monitor unit 6. [ Lt; / RTI >

The slave robot 2 and the master robot 1 are coupled to each other through a wired communication network or a wireless communication network so that operation signals and laparoscopic images inputted through the laparoscope 5 can be transmitted to the other party. If two operation signals by the two handles 10 provided in the master interface 4 and / or operation signals for adjusting the position of the laparoscope 5 need to be transmitted at the same time and / , The respective operation signals can be transmitted to the slave robot 2 independently of each other. Here, the fact that each operation signal is transmitted 'independently' means that the operation signals do not interfere with each other, and that any one operation signal does not affect the other. In order to allow a plurality of operation signals to be independently transmitted, header information for each operation signal is added and transmitted in the generation of each operation signal, or each operation signal is transmitted in accordance with the generation order, or A variety of schemes may be used, such that priorities are given in advance regarding the transmission order of each operation signal and are transmitted in accordance with the priorities. In this case, the transmission path through which each operation signal is transmitted may be provided independently so that interference between each operation signal may be fundamentally prevented.

The robot arm 3 of the slave robot 2 can be implemented to be driven with multiple degrees of freedom. The robot arm 3 includes, for example, a surgical instrument inserted into a surgical site of a patient, a swinging drive unit for rotating the surgical instrument in a yaw direction according to an operation position, a pitch direction , A pitch driving part for rotating the surgical instrument, a feed driving part for moving the surgical instrument in the longitudinal direction, a rotation driving part for rotating the surgical instrument, and a surgical instrument driving part installed at the end of the surgical instrument for cutting or cutting the surgical lesion. . However, it should be understood that the configuration of the robot arm 3 is not limited thereto, and that these examples do not limit the scope of the present invention. The actual control process of rotating or moving the robot arm 3 in the corresponding direction by operating the handle 10 by the surgeon is somewhat distant from the gist of the present invention, and a detailed description thereof will be omitted.

The slave robot 2 can be used more than one to perform surgery on the patient and the laparoscope 5 for displaying the surgical site through the monitor unit 6 as an image image is implemented as an independent slave robot 2 It is possible. Further, as described above, the embodiments of the present invention can be widely used in surgeries in which a variety of surgical endoscopes other than laparoscopes (e.g., thoracoscopic, arthroscopic, non-circumferential, etc.) are used.

FIG. 3 is a block diagram schematically showing the configuration of a master robot and a slave robot according to an embodiment of the present invention. FIG. 4A is a view illustrating a driving mode of a surgical robot system according to an embodiment of the present invention, 4B is a diagram illustrating a mode indicator indicating a running mode of operation according to an embodiment of the present invention.

3, the master robot 1 includes an image input unit 310, a screen display unit 320, a dark operation unit 330, an operation signal generation unit 330, and a display unit 320. The slave robot 2 includes a master robot 1 and a slave robot 2, Unit 340, an augmented reality implementing unit 350, and a control unit 360. [ The slave robot 2 includes a robot arm 3 and a laparoscope 5. [ Although not shown in FIG. 3, the slave robot 2 may further include a bio-information measuring unit for measuring and providing bio-information about the patient.

The image input unit 310 receives an image input through a camera provided in the laparoscope 5 of the slave robot 2 through a wired or wireless communication network.

The screen display unit 320 outputs visual images corresponding to the images received through the image input unit 310 as visual information. Further, the screen display unit 320 can further output the virtual surgical tool according to the operation of the arm control unit 330 as visual information, and when biometric information is input from the slave robot 2, . The screen display unit 320 may be implemented in the form of a monitor unit 6 and the like. The image processing process for causing the received image to be outputted as an image image through the screen display unit 320 may include a controller 360, (350) or an image processing unit (not shown).

The arm control unit 330 is a means for enabling the operator to operate the position and function of the robot arm 3 of the slave robot 2. [ 2, a shape of the handle 10 is not limited to the shape of the handle 10, but may be modified into various shapes to achieve the same purpose. In addition, for example, a part may be formed in a handle shape and the other part may be formed in another shape such as a clutch button or the like. In order to facilitate manipulation of the surgical tool, a finger insertion tube or insertion More rings may be formed.

As described above, the clutch operation button 330 may be provided with the clutch button 14, and the clutch button 14 may be used as the mode change control button. In addition, the mode changeover control button may be implemented in a mechanical form such as a pedal (not shown) or a function menu displayed through the monitor 6 or a mode selection menu. In addition, if the laparoscope 5 is not fixed at a specific position to receive an image and its position and / or image input angle can be shifted or changed by adjusting the operator, the clutch button 14, 5 and / or for adjusting the image input angle.

The manipulation signal generator 340 generates a manipulation signal corresponding to the manipulation of the arm manipulation unit 330 by the operator to move the robot arm 3 and / or the laparoscope 5, To the robot (2). The operation signal can be transmitted and received through a wired or wireless communication network as described above.

When the master robot 1 is driven in a comparison mode or the like, which is the second mode, the augmented reality implementing unit 350 realizes an image of a surgical site input through the laparoscope 5, So that the virtual surgery tool can be output together with the screen display unit 320. [ The detailed functions, various detailed configurations, and the like of the augmented reality implementing unit 350 will be described in detail with reference to related drawings hereinafter.

The control unit 360 controls the operation of each component so that the above-described functions can be performed. The control unit 360 may convert the image input through the image input unit 310 into an image to be displayed through the screen display unit 320. [ The control unit 360 controls the augmented reality implementing unit 350 so that the virtual surgery tool is output through the screen display unit 320 so that the operation information corresponding to the operation of the arm operation unit 330 is input. In addition, the control unit 360 may control the learner and the educator to grant or withdraw the right of operation when performing the fourth mode, which is an education mode.

As illustrated in FIG. 4A, the master robot 1 and / or the slave robot 2 can operate in a drive mode selected by the operator or the like in various drive modes.

For example, the driving mode may include a first mode that is an actual mode, a second mode that is a comparison mode, a third mode that is a virtual mode, a fourth mode that is an education mode, and a fifth mode that is a simulation mode.

When the master robot 1 and / or the slave robot 2 operate in the first mode, which is an actual mode, the image displayed through the monitor unit 6 of the master robot 1 is displayed on a surgical site, . That is, the virtual surgical tool may not be displayed, which may be the same or similar to the display screen during the remote operation using the conventional surgical robot system. Of course, even when operating in the first mode, information corresponding to the patient's biometric information is measured and received in the slave robot 2, and the display method may be varied as described above .

When the master robot 1 and / or the slave robot 2 operate in the second mode, which is the comparison mode, the image displayed through the monitor unit 6 of the master robot 1 is displayed on the operation part, Surgical tools, and the like.

For reference, an actual surgical tool is a surgical tool included in an image input by the laparoscope 5 and transmitted to the master robot 1, and is a surgical tool for directly applying a surgical operation to a patient's body. On the other hand, the virtual surgical tool is controlled by the operation information recognized by the master robot 1 (that is, information related to movement, rotation, and the like of the surgical tool) as the operator operates the arm control unit 330, It is a virtual surgical tool. Actual surgical tools and virtual surgical instruments will have their positions and manipulated shapes determined by the manipulation information.

 The operation signal generating unit 340 generates an operation signal by using the operation information according to the operation of the operation member 340 of the operator and transmits the generated operation signal to the slave robot 2. As a result, As shown in FIG. In addition, the position and manipulation shape of the actual surgical tool manipulated by the manipulation signal can be confirmed by the operator through the image input by the laparoscope 5. That is, if the network communication speed between the master robot 1 and the slave robot 2 is sufficiently fast, the actual surgical tool and the virtual surgical tool will move at almost the same speed. On the contrary, if the network communication speed is somewhat slow, the virtual surgical tool will move first, and then the actual surgical tool will move in the same manner as the virtual operation tool. However, if the network communication speed is slow (for example, the delay time is more than 150 ms), the virtual surgical tool will move and the actual surgical tool will move with a certain time difference.

(I.e., a training student) or an educator (i.e., a training teacher) with respect to the arm control unit 330 when the master robot 1 and / or the slave robot 2 operate in the third mode, The master robot 1 does not transmit the signal to the slave robot 2 so that the image displayed through the monitor unit 6 of the master robot 1 includes at least one of a surgical site and a virtual surgical tool . Educators can select a third mode and perform pre-test operations on actual surgical tools. The third mode can be entered by selecting the clutch button 14 or the like. When the button 10 is pressed (or the third mode is selected), the actual operation tool moves It will be possible to move only the virtual surgical tool without it. When the depression of the corresponding button is terminated (or the first mode or the second mode is selected) in this state, the virtual surgical tool is moved so that the actual surgical tool is moved so as to match the operated manipulation information, (Or the position and manipulation of the virtual surgical instrument may be restored).

(That is, a training student) or an educator (that is, a training teacher) with respect to the arm control unit 330 when the master robot 1 and / or the slave robot 2 operate in the fourth mode, The signal can be transmitted to the master robot 1 operated by the educator or the learner. To this end, two or more master robots 1 may be connected to one slave robot 2, or a separate master robot 1 may be connected to the master robot 1. In this case, a corresponding operation signal can be transmitted to the slave robot 2 when the dark operation unit 330 of the master robot 1 for the educator is operated, The image input through the laparoscope 5 for the confirmation of the progress of the operation can be displayed on the display unit 6. In contrast, when the arm control unit 330 of the learner master robot 1 is operated, a corresponding operation signal may be provided only to the master robot 1 for educator and not to the slave robot 2. Thus, the operation of the educator can be made to function in the first mode, while the operation of the learner can be made to function in the third mode. The operation in the fourth mode which is the education mode will be described in detail with reference to the related drawings hereinafter.

When operating in the fifth mode, which is a simulation mode, the master robot 1 functions as a surgical simulator that uses three-dimensional modeled three-dimensional organs characteristics (e.g., shape, texture, do. In other words, the fifth mode can be understood as a virtual mode, which is a third mode, evolved, and operates as a surgical simulator by combining characteristics of an organ in a three-dimensional shape obtained by using a stereoscopic endoscope or the like.

If the liver is output through the screen display unit 320, the liver shape can be grasped three-dimensionally using the stereoscopic endoscope, and mathematically modeled liver characteristic information (which is stored in a storage unit (not shown) Which can be simulated in a virtual mode during surgery. For example, before performing a liver resection, it is possible to perform a surgical simulation in advance, in which the liver is resected in a certain direction while matching the liver characteristic information to the liver shape. In addition, based on mathematical modeling information and characteristic information, it is possible to feel the tactile sensation in advance at which part is hard or which portion is soft. In this case, the surface shape information of the organ obtained in three dimensions is matched with the three-dimensional shape of the reconstructed organ surface with reference to CT (Computer Tomography) or / and MRI (Magnetic Resonance Imaging) And the mathematical modeling information can be matched with the three-dimensional shape reconstructed from the reconstructed image.

Although the driving modes of the first to fifth modes have been described so far, it is also possible to add driving modes according to various purposes.

Further, when the master robot 1 is driven in each mode, the operator can be confused as to what the current drive mode is. It is possible to display a mode indicator further through the screen display unit 320 so that a clearer drive mode can be identified.

FIG. 4B illustrates a display mode in which a mode indicator is further displayed on the screen where the surgical site and the actual surgical tool 460 are displayed. The mode indicator may be a message 450, a border color (480), or the like in order to clearly recognize in which drive mode the drive is currently being operated. In addition, the mode indicator may be implemented as an icon, a background color, or the like, and only one mode indicator may be displayed or two or more mode indicators may be displayed together.

FIG. 5 is a flowchart illustrating a driving mode selection process of a first mode and a second mode according to an exemplary embodiment of the present invention. FIG. Fig.

5, it is assumed that either the first mode or the second mode is selected. However, if the driving mode is applied in the first mode to the fifth mode as illustrated in FIG. 4A, The mode selection input of the first mode may be for any one of the first mode to the fifth mode.

Referring to FIG. 5, the surgical robot system is started in step 510. An image input through the laparoscope 5 after the start of the operation of the surgical robot system will be outputted through the monitor unit 6 of the master robot 1. [

In step 520, the master robot 1 receives the selection of the drive mode from the operator. The selection of the drive mode can be made, for example, by depressing a clutch button 14 or a pedal (not shown) implemented mechanically or by using a function menu or a mode selection menu displayed through the monitor 6 There will be.

If the first mode is selected in step 520, the master robot 1 operates in the driving mode, which is the actual mode, and displays the image input from the laparoscope 5 on the monitor unit 6.

However, if the second mode is selected in step 520, the master robot 1 operates in the driving mode, which is the comparison mode, and receives not only the image input from the laparoscope 5 but also the operation information according to the operation of the dark operation unit 330 And displays the virtual surgery tool on the monitor unit 6 together.

FIG. 6 illustrates a screen display form output through the monitor unit 6 in the second mode.

6, in the comparison mode, on the screen, the image input by the laparoscope 5 and provided (i.e., the image in which the surgical site and the actual surgical tool 460 are displayed) and the operation A virtual surgical tool 610 controlled by information is displayed together.

The difference in the display position between the actual surgical tool 460 and the virtual surgical tool 610 is caused by the network communication speed between the master robot 1 and the slave robot 2, The tool 460 will be moved to the location where the current virtual surgical tool 610 is displayed and displayed.

6, the virtual surgical tool 610 is illustrated in the form of an arrow for convenience in distinguishing it from the actual surgical tool 460. However, the display shape of the virtual surgical tool 610 may be the same as that of the actual surgical tool, A semi-transparent form for distinguishing convenience, a dotted line form including only an outline, and the like. The display state and the display form of the virtual surgical tool 610 will be described later with reference to the related drawings.

In addition, a method of causing the virtual surgical tool 610 to be displayed together with the image inputted and provided by the laparoscope 5 includes a method of displaying the virtual surgical tool 610 in an overlapped manner on the laparoscopic image, And a method for causing the virtual surgical tool 610 to be reconstructed and displayed as a single image.

FIG. 7 is a diagram illustrating a detailed configuration of an augmented reality implementing unit 350 according to an embodiment of the present invention. FIG. 8 is a flowchart illustrating a method of driving a master robot 1 in a second mode according to an embodiment of the present invention Fig.

7, the augmented reality implementing unit 350 includes a characteristic value calculating unit 710, a virtual surgery tool generating unit 720, a test signal processing unit 730, and a delay time calculating unit 740. [ Some components (for example, the test signal processing unit 730, the delay time calculating unit 740, and the like) of the components of the augmented reality implementing unit 350 may be omitted and some components (e.g., A component for processing the biometric information received from the robot 2 so that the biometric information can be outputted through the screen display unit 320) may be further added. One or more components included in the augmented reality implementing unit 350 may be implemented in the form of a software program by a combination of program codes.

The characteristic value computing unit 710 computes a characteristic value by using the coordinates of the image provided by the laparoscope 5 of the slave robot 2 and / or the position of the actual surgical tool coupled to the robot arm 3, . The position of the actual surgical tool can be recognized by referring to the position value of the robot arm 3 of the slave robot 2 and information on the position can be provided from the slave robot 2 to the master robot 1 .

The characteristic value calculator 710 calculates the characteristic value of the laparoscope 5 based on the image of the laparoscope 5 using the image of the laparoscope 5, the field of view of the laparoscope 5, the enlargement ratio, the viewpoint (for example, , The type of the actual surgical tool 460, the direction, the depth, and the degree of tearing. In the case of calculating the characteristic value using the image by the laparoscope 5, an image recognition technique for recognizing the contour extraction, shape recognition, tilted angle, etc. of the subject included in the image may be used. In addition, the type of the actual surgical tool 460 and the like may be input in advance in a process of coupling the surgical tool to the robot arm 3 or the like.

The virtual surgery tool generation unit 720 generates a virtual surgery tool 610 to be output through the screen display unit 320 by referring to operation information on operation of the robot arm 3 of the operator. The position at which the virtual surgical tool 610 is initially displayed may be based on the display position where the actual surgical tool 460 is displayed through the screen display unit 320, The moving displacement of the surgical tool 610 may be set in advance by referring to an actual value at which the actual surgical tool 460 is moved corresponding to, for example, an operation signal.

The virtual surgery tool generation unit 720 may generate only virtual surgery tool information (e.g., a property value for displaying the virtual surgery tool) for causing the virtual surgery tool 610 to be output through the screen display unit 320 have. The virtual surgery tool generation unit 720 generates a virtual surgery tool 610 by using the virtual surgery tool 610 to display the characteristic value calculated by the characteristic value calculation unit 710 or the virtual surgery tool 610 in determining the shape or position of the virtual surgery tool 610 according to the manipulation information. The value of the immediately preceding characteristic, and the like. This is to allow the virtual surgical tool 710 or the actual surgical tool 460 to quickly generate the information when only the translating operation is performed in the same state as the previous shape (for example, the inclined angle, etc.) to be.

The test signal processing unit 730 transmits a test signal to the slave robot 2 so that the network communication speed between the master robot 1 and the slave robot 2 can be determined, Signal. The test signal transmitted by the test signal processing unit 730 may be a conventional signal used as a time stamp in a control signal transmitted and received between the master robot 1 and the slave robot 2, It may be a signal that is additionally used to measure the communication speed. In addition, it may be specified in advance that a measurement of the network communication speed is performed only at a time point of each time point at which the test signal is transmitted and received.

The delay time calculating unit 740 calculates the delay time on the network communication by using the transmission time of the test signal and the reception time of the response signal. If the network communication speeds of the master robot 1 and the slave robot 2 are the same and the delay time is the same as the delay time of the slave robot 2 For example, 1/2 of the difference between the transmission time of the test signal and the reception time of the response signal. This is because the slave robot will perform the corresponding processing immediately when an operation signal is received from the master robot 1. [ Of course, the delay time may further include a processing delay time for performing processing such as control of the robot arm 3 in response to an operation signal from the slave robot 2. As another example, if the difference between the operation time of the operator and the observation time is emphasized, the delay time may be determined based on the transmission time and the reception time of the response signal (for example, Of the time difference). In addition, the method of calculating the delay time may be various.

If the delay time is less than a predetermined threshold value (for example, 150 ms), the difference in display position between the actual surgical tool 460 and the virtual surgical tool 610 may not be large. In this case, the virtual surgery tool creation unit 720 may prevent the virtual surgery tool 610 from being displayed through the screen display unit 320. This is because it is unnecessary to cause the surgeon's confusion because the actual surgical tool 460 and the virtual surgical tool 610 are displayed at the same position or in close proximity to each other.

However, if the delay time exceeds a predetermined threshold value (for example, 150 ms), a difference in display position between the actual surgical tool 460 and the virtual surgical tool 610 may be large. In this case, the virtual surgery tool generator 720 may cause the virtual surgery tool 610 to be displayed on the screen display unit 320. This is to remove the confusion of the operator caused by the fact that the operation state of the operator's arm operation unit 330 and the operation state of the actual operation tool 460 do not coincide with each other in real time and the operator refers to the virtual surgery tool 610 Because the actual surgical tool 460 will be subsequently manipulated in accordance with the manipulated form of the virtual surgical tool 610 even if it is operated.

FIG. 8 illustrates a flowchart showing a method of driving the master robot 1 in the second mode. In describing each step of the flowchart, it will be described that the master robot 1 performs each step for convenience of explanation and understanding.

Referring to FIG. 8, in step 810, the master robot 1 generates a test signal to measure the network communication speed and transmits the test signal to the slave robot 2 via the wired or wireless communication network.

In step 820, the master robot 1 receives a response signal to the test signal from the slave robot 2.

In step 830, the master robot 1 calculates the delay time on the network communication speed using the transmission time of the test signal and the reception time of the response signal.

Then, in step 840, the master robot 1 determines whether the calculated delay time is equal to or less than a preset threshold value. Here, the threshold value is a delay time on the network communication speed required for smooth operation of the operator using the surgical robot system, and can be determined and applied by an experimental and / or statistical method.

If the calculated delay time is equal to or less than the predetermined threshold value, the process proceeds to step 850 and the master robot 1 transmits the image input through the laparoscope 5 to the screen display unit 320 ) Is displayed on the screen. At this time, the virtual surgical tool 610 may not be displayed. Of course, in this case, the virtual surgical tool 610 and the actual surgical tool 460 may be displayed together.

However, if the calculated delay time exceeds the preset threshold, the master robot 1 proceeds to step 860 so that the image input through the laparoscope 5 to the screen display unit 320 (i.e., The virtual surgical tool 610 may be displayed together with the image including the tool 460. Of course, in this case, the virtual surgical tool 610 may not be displayed.

FIG. 9 is a diagram illustrating a detailed configuration of an augmented reality implementing unit 350 according to another embodiment of the present invention. FIGS. 10 and 11 are diagrams showing a configuration of a master robot 1 ) According to the second embodiment of the present invention.

9, the augmented reality implementing unit 350 includes a characteristic value calculating unit 710, a virtual surgery tool generating unit 720, an interval calculating unit 910, and an image analyzing unit 920. Some of the components of the augmented reality implementing unit 350 may be omitted and some components (e.g., biometric information received from the slave robot 2) may be processed to be output through the screen display unit 320 And the like) may be further added. One or more components included in the augmented reality implementing unit 350 may be implemented in the form of a software program by a combination of program codes.

The characteristic value computing unit 710 computes a characteristic value by using the coordinates of the image provided by the laparoscope 5 of the slave robot 2 and / or the position of the actual surgical tool coupled to the robot arm 3, . The characteristic values include, for example, the field of view (FOV) of the laparoscope 5, the magnification, the viewpoint (e.g., viewing direction), the viewing depth, and the type, direction, depth, And the like.

The virtual surgery tool generation unit 720 generates a virtual surgery tool 610 to be output through the screen display unit 320 by referring to operation information on operation of the robot arm 3 of the operator.

The interval calculator 910 calculates the distance between the operation coordinates of the actual operation tool 460 and the virtual operation tool 610 by using the position coordinates of the actual operation tool 460 calculated through the characteristic value calculator 710 and the position coordinates of the virtual operation tool 610, . For example, when the position coordinates of the virtual surgical tool 610 and the actual surgical tool 460 are determined, they can be calculated as the length of a line connecting two points. Here, the position coordinate may be a coordinate value of a point on a three-dimensional space defined by, for example, an xyz axis, and the point may correspond to a point of a specific position on the virtual surgical tool 610 and the actual surgical tool 460 Can be specified. In addition, the interval of each operation section can be further utilized by the path or the length of the trajectory generated by the operation method. For example, when drawing circles, when the time difference exists for the time of drawing a circle, the length of the line segment of each operation section becomes very small, but the difference in path or trajectory can be generated by the circumference of the circle generated by the manipulation method Because.

The positional coordinates of the actual surgical tool 460 used for the gap calculation may be used as absolute coordinate values or relative coordinate values calculated on the basis of a specific point, ) May be used as coordinates. Likewise, the positional coordinates of the virtual surgical tool 610 may be obtained by using the virtual position shifted by the operation of the dark operation unit 330 on the basis of the initial position of the virtual surgical tool 610, Alternatively, the position of the virtual surgical tool 610 displayed through the screen display unit 320 may be used as coordinates. Here, the feature information interpreted by the image analyzing unit 920, which will be described below, may be used to analyze the position of each surgical tool displayed on the screen display unit 320.

If the interval between the virtual surgical tool 610 and the actual surgical tool 460 is narrow or zero, it can be understood that the network communication speed is good. However, if the interval is wide, it can be understood that the network communication speed is insufficient.

The virtual surgery tool generation unit 720 determines at least one of the display state of the virtual surgery tool 610, the color or display form of the virtual surgery tool 610, and the like using the interval information calculated by the interval operation unit 910 . For example, if the interval between the virtual surgical tool 610 and the actual surgical tool 460 is equal to or less than a predetermined threshold value, the virtual surgical tool 610 can be prevented from being output through the screen display unit 320 have. When the interval between the virtual surgical tool 610 and the actual surgical tool 460 exceeds a preset threshold value, the translucency is adjusted or the color is distorted in proportion to the interval between the virtual surgical tool 610 and the virtual surgical tool 460, ) May be changed to allow the operator to clearly recognize the network communication speed. Here, the threshold value may be designated as a distance value such as 5 mm, for example.

The image analyzing unit 920 analyzes the image data inputted by the laparoscope 5 and outputs predetermined feature information (for example, color value per pixel, position coordinates of the actual surgical tool 460, Or more). For example, the image analyzing unit 920 analyzes the color value of each pixel of the corresponding image so that an immediate response to an emergency (for example, excessive bleeding) Or whether or not the area or area formed by the pixels having the color value indicating the blood is greater than or equal to a predetermined size. The image analyzing unit 920 may capture the image input by the laparoscope 5 and the display screen of the screen display unit 320 on which the virtual surgical tool 610 is displayed to generate the position coordinates of each surgical tool .

10 is a flowchart showing a method of driving the master robot 1 in the second mode according to another embodiment of the present invention.

10, in step 1010, the master robot 1 receives a laparoscopic image (that is, an image input through the laparoscope 5) from the slave robot 2.

In step 1020, the master robot 1 calculates coordinate information of the actual surgical tool 460 and the virtual surgical tool 610. Here, the coordinate information may be calculated using, for example, the feature value and operation information calculated by the feature value calculating unit 710, or the feature information extracted by the image analyzing unit 920 may be used.

In step 1030, the master robot 1 calculates the distance between the two using the coordinate information of each surgical tool calculated in step 1020.

In step 1040, the master robot 1 determines whether the calculated interval is below a threshold value.

If the calculated interval is less than the threshold value, the master robot 1 proceeds to step 1050 to output the laparoscopic image through the screen display unit 320, but does not display the virtual surgical tool 610.

However, if the calculated interval exceeds the threshold value, the master robot 1 proceeds to step 1060 so that the laparoscopic image and the virtual surgical tool 610 are displayed together on the screen display unit. At this time, processing such as adjusting the translucency, distorting the color, or changing the outline thickness of the virtual surgical tool 610 may be performed in proportion to the interval between them.

11 is a flowchart showing a method of driving the master robot 1 in the second mode according to another embodiment of the present invention.

Referring to FIG. 11, in step 1110, the master robot 1 receives a laparoscopic image. The received laparoscopic image will be output through the screen display unit 320.

In steps 1120 and 1130, the master robot 1 analyzes the received laparoscopic image, and calculates and analyzes the color value of each pixel of the image. The calculation of the pixel-by-pixel color value may be performed by the image analyzing unit 920 or may be performed by the characteristic value calculating unit 710 to which the image recognizing technique is applied. Also, by analyzing the color value per pixel, at least one of, for example, a color value frequency, a region or an area formed by pixels having a color value to be analyzed, etc. can be calculated.

In step 1140, the master robot 1 determines whether it is an emergency situation based on the information analyzed in step 1130. The types of emergencies (eg, excessive bleeding) or whether the analyzed information will be perceived as an emergency can be defined in advance.

If it is determined as an emergency, the master robot 1 proceeds to step 1150 and outputs warning information. The warning information may be, for example, a warning message output through the screen display unit 320, a warning sound output through a speaker unit (not shown), or the like. Although not shown in FIG. 3, it is needless to say that a speaker unit for outputting warning information, announcement messages, and the like can be further included in the master robot 1. If the virtual surgical tool 610 is displayed together with the screen display unit 320 at the time of the emergency, the virtual surgical tool 610 is displayed so that the operator can be accurately determined .

However, if it is determined that it is not an emergency situation, the flow returns to step 1110.

FIG. 12 is a block diagram schematically showing a configuration of a master robot and a slave robot according to another embodiment of the present invention, FIG. 13 is a flowchart illustrating a method of verifying normal operation of a surgical robot system according to another embodiment of the present invention Fig.

12, the master robot 1 includes an image input unit 310, a screen display unit 320, a dark operation unit 330, an operation signal generation unit 330, and a display unit 320. The slave robot 2 includes a master robot 1 and a slave robot 2, An augmented reality implementing unit 350, a control unit 360, and a network verifying unit 1210, as shown in FIG. The slave robot 2 includes a robot arm 3 and a laparoscope 5. [

The image input unit 310 receives an image input through a camera provided in the laparoscope 5 of the slave robot 2 through a wired or wireless communication network.

The screen display unit 320 outputs visual images corresponding to the images received through the image input unit 310 and / or the virtual operation tool 610 according to the operation of the dark operation unit 330 as vision information.

The arm control unit 330 is a means for enabling the operator to operate the position and function of the robot arm 3 of the slave robot 2. [

The manipulation signal generator 340 generates a manipulation signal corresponding to the manipulation of the arm manipulation unit 330 by the operator to move the robot arm 3 and / or the laparoscope 5, To the robot (2).

The network verifying unit 1210 verifies whether the master robot 1 and the slave robot 2 are using the characteristic value calculated by the characteristic value calculating unit 710 and the virtual surgical tool information generated by the virtual surgical tool generating unit 720, Lt; / RTI > For this purpose, for example, at least one of the position information, the direction, the depth, the degree of bending, etc. of the actual surgical tool 460 and the position information, direction, depth and direction of the virtual surgical tool 610, And the degree of distortion, etc., and the characteristic value and the virtual surgical tool information may be stored in a storage unit (not shown).

According to the embodiment of the present invention, when the manipulation information is generated by the operation of the arm manipulation part 330 of the operator, the virtual surgical tool 610 is controlled to correspond to the manipulation information, and an operation signal corresponding to the manipulation information is transmitted to the slave robot 2 And is used for manipulation of the actual surgical tool 460. [ In addition, the positional movement of the actual surgical tool 460, which is manipulated and controlled by the manipulation signal, can be confirmed through the laparoscopic image. In this case, since the manipulation of the virtual surgical tool 610 is performed in the master robot 1, the manipulation of the virtual surgical tool 610 will be performed ahead of the operation of the actual surgical tool 460 in consideration of the network communication speed and the like.

Accordingly, the network verifying unit 1210 determines whether the actual surgical tool 460 is late or is manipulated so as to be identifiable within the predetermined error range in the same manner as the movement trajectory or the operation form of the virtual surgical tool 610 It is possible to determine whether or not the network communication is normal. For this purpose, the virtual surgery tool information stored in the storage unit may be used as a characteristic value of the actual operation tool 460. Further, the error range may be set, for example, as a distance value between mutual coordinate information, a time value until it is recognized as a coincidence, and the like, which may be arbitrarily, experimentally, and / or statistically specified.

In addition, the network verification unit 1210 may perform verification of network communication using the feature information analyzed by the image analysis unit 920.

The control unit 360 controls the operation of each component so that the above-described functions can be performed. In addition, as illustrated in other embodiments, the control unit 360 may perform additional various functions.

13 illustrates a method for verifying whether the surgical robot system is normally operated by verifying network communication.

Referring to FIG. 13, in steps 1310 and 1320, the master robot 1 receives the manipulation of the manipulation part 330 from the operator and interprets the manipulation information according to the manipulation of the manipulation part 330. The operation information is information related to the operation of the arm operation unit 330 for, for example, moving the actual surgical tool 460, cutting the surgical site, and the like.

In step 1330, the master robot 1 generates virtual surgery tool information using the analyzed operation information, and outputs the virtual surgery tool 610 according to the generated virtual surgery tool information to the screen display unit 320. At this time, the generated virtual surgical tool information may be stored in a storage unit (not shown).

In step 1340, the master robot 1 calculates a characteristic value for the actual surgical tool 460. The characteristic value calculation may be performed, for example, by the characteristic value calculation unit 710 or the image analysis unit 920.

In step 1350, the master robot 1 determines whether there is a coordinate value coincidence point of each surgical tool. It can be judged that a coordinate value coincident point exists when the coordinate information of each surgical tool coincides with each other within the coincidence or error range. Here, the error range can be preset, for example, as a distance value on three-dimensional coordinates or the like. As described above, since the result of operation of the operator's manipulation part 330 of the operator will be reflected first on the virtual surgical tool 610 rather than on the actual surgical tool 460, step 1350 determines the characteristic value for the actual surgical tool 460 May be performed by determining whether or not the virtual surgery tool information stored in the storage unit matches the virtual surgery tool information stored in the storage unit.

If there is no coordinate coincidence point, the master robot 1 proceeds to step 1360 and outputs the warning information. The warning information may be, for example, a warning message output through the screen display unit 320, a warning sound output through a speaker unit (not shown), or the like.

However, if there is a coordinate value coincidence point, it is determined that the network communication is normal, and the process proceeds to step 1310 again.

The above-described steps 1310 to 1360 may be performed in real time during the operation of the operator, or may be performed periodically or at a preset time.

FIG. 14 is a diagram illustrating a detailed configuration of an augmented reality implementing unit 350 according to another embodiment of the present invention, and FIGS. 15 and 16 are diagrams for explaining a method for outputting a virtual surgery tool according to another embodiment of the present invention Fig. 2 is a flowchart showing a driving method of the master robot 1. Fig.

14, the augmented reality implementing unit 350 includes a characteristic value calculating unit 710, a virtual surgery tool generating unit 720, an image analyzing unit 920, an overlapping processing unit 1410, and a contact recognizing unit 1420 . Some of the components of the augmented reality implementing unit 350 may be omitted and some components (e.g., biometric information received from the slave robot 2) may be processed to be output through the screen display unit 320 And the like) may be further added. One or more components included in the augmented reality implementing unit 350 may be implemented in the form of a software program by a combination of program codes.

The characteristic value computing unit 710 computes a characteristic value by using the coordinates of the image provided by the laparoscope 5 of the slave robot 2 and / or the position of the actual surgical tool coupled to the robot arm 3, . The characteristic values include, for example, the field of view (FOV) of the laparoscope 5, the magnification, the viewpoint (e.g., viewing direction), the viewing depth, and the type, direction, depth, And the like.

The virtual surgical tool generating unit 720 generates virtual surgical tool information for outputting the virtual surgical tool 610 through the screen display unit 320 referring to the operation information of the operation of the robot arm 3 of the operator.

The image analyzing unit 920 analyzes the feature information (for example, the shape of the organ in the surgical site, the position coordinates of the actual surgical tool 460, the manipulated shape, etc.) using the image input by the laparoscope 5, Or more). For example, the image analyzing unit 920 can analyze what the displayed organs are by using an image recognition technique such as extraction of an outline of an organ displayed in a laparoscopic image, color value analysis of each pixel representing an organ, and the like. For this purpose, information on the shape of each organ, hue, coordinate information of the region where each organ or / and surgical site is located in the three-dimensional space, and the like may be stored in advance in a storage unit (not shown). Alternatively, the image analyzing unit 920 may analyze coordinate information (absolute coordinates or relative coordinates) of an area occupied by the organ through image analysis.

The superposition processor 1410 superimposes the virtual surgery tool information generated by the virtual surgery tool generator 720 and the region coordinate information of the organs and / or surgery sites recognized by the image analyzer 920 It is judged whether or not it occurs and processed in a corresponding manner. If some or all of the virtual surgical instruments are located at the lower or lateral side of the organ, it can be determined that overlapping (i.e., masked) with respect to each other occurs, and the reality of the virtual surgical tool 610 display The area of the virtual surgical tool 610 corresponding to the overlapping portion is processed so as to be hidden (i.e., not displayed through the screen display unit 320). As a method of concealing the overlapping portion, for example, a method of making a region corresponding to the overlapped portion of the shape of the virtual surgical tool 610 transparent is used.

Or, if the superposition processor 1410 determines that there is an overlap between the organ and the virtual surgical tool 610, it may provide the long-term area coordinate information to the virtual surgery tool generator 720 or the virtual surgery tool generator 720, The virtual surgery tool creation unit 720 may request that the corresponding information be read from the storage unit so that the virtual surgery tool creation unit 720 does not generate the virtual surgery tool information for the overlapping part.

The contact recognition unit 1420 determines whether or not contact is generated between the virtual surgery tool information generated by the virtual surgery tool creation unit 720 and the region coordinate information of the organ recognized by the image analysis unit 920 And judges them to be correspondingly processed. If the surface coordinate information and the coordinate information of some or all of the virtual surgical tool among the long-term area coordinate information are matched, it can be determined that there is a contact in the corresponding part. The master robot 1 performs processing such that the arm control unit 330 is no longer operated or force feedback is performed through the arm control unit 330. [ (For example, a warning message and / or a warning sound) may be output. The components for processing the force feedback or outputting the warning information may be included as a component of the master robot 1.

FIG. 15 illustrates a driving method of the master robot 1 for outputting a virtual surgical tool according to another embodiment of the present invention.

Referring to FIG. 15, in step 1510, the master robot 1 receives an operation of the arm control unit 330 from the operator.

Then, in steps 1520 and 1530, the master robot 1 analyzes the operation information of the operator according to the operation of the arm control unit 330 and generates virtual surgical tool information. The virtual surgical tool information may include coordinate information for the outline or area of the virtual surgical tool 610 for outputting the virtual surgical tool 610 through the screen display unit 320, for example.

Further, in steps 1540 and 1550, the master robot 1 receives the laparoscopic image from the slave robot 2 and analyzes the received image. The analysis of the received image can be performed, for example, by the image analyzing unit 920, and the image analyzing unit 920 can recognize the organs included in the laparoscopic image.

In step 1560, the master robot 1 reads the area coordinate information for the organ recognized through the laparoscopic image from the storage unit.

In step 1570, the master robot 1 determines whether or not there is an overlapping part between the robot and the robot using the coordinate information of the virtual surgical tool 610 and the area coordinate information of the organ.

If there is an overlapping portion, the master robot 1 processes the virtual surgical tool 610 in which the overlapped portion is concealed so as to be output through the screen display portion 320 in Step 1580. [

However, if there is no overlapping part, the master robot 1 processes the virtual surgical tool 610 in which all the parts are normally displayed in step 1590 through the screen display part 320.

FIG. 16 shows an embodiment for notifying the operator of the virtual surgical tool 610 when the virtual surgical tool 610 is in contact with the organ of the patient. Steps 1510 to 1560 of FIG. 16 have already been described with reference to FIG. 15, and a description thereof will be omitted.

Referring to FIG. 16, in step 1610, the master robot 1 determines whether part or all of the virtual surgical tool 610 has contacted organs. The contact between the organ and the virtual surgical tool 610 can be determined using, for example, coordinate information for each region.

If the organ has been in contact with the virtual surgical tool 610, the process proceeds to step 1620, where the master robot 1 performs force feedback processing to inform the operator. As described above, for example, the dark operation section 330 may be processed so as to be no longer operated or the warning information (for example, a warning message and / or a warning sound) may be output.

However, if the organ is not in contact with the virtual surgical tool 610, the process waits at step 1610.

Through the above-described process, the operator can predict whether or not the actual surgical tool 460 will be in contact with the organ in advance, thereby enabling safer and more precise surgery progression.

17 is a flowchart illustrating a method of providing a reference image according to another embodiment of the present invention.

Generally, the patient will take a variety of reference images such as X-ray, CT, and / or MRI prior to surgery. If the reference image can be presented to the surgeon along with the laparoscopic image at the time of surgery or on any monitor of the monitor unit 6, the operation of the surgeon may be more smooth. For example, the reference image may be stored in a storage unit included in the master robot 1 or may be stored in a database that the master robot 1 can access via a communication network.

17, in step 1710, the master robot 1 receives a laparoscopic image from the laparoscope 5 of the slave robot 2.

In step 1720, the master robot 1 extracts preset feature information using the laparoscopic image. Here, the feature information may be at least one of, for example, the shape of the organ in the surgical site, the position coordinates of the actual surgical tool 460, the manipulated shape, and the like. The extraction of the feature information may be performed by the image analyzing unit 920, for example.

In step 1730, the master robot 1 recognizes the organ displayed in the laparoscopic image by using the feature information extracted in step 1720 and information stored in the storage unit in advance.

Then, in step 1740, the master robot 1 reads out a reference image including an image corresponding to the organ recognized in step 1730 from a database connectable via a storage unit or a communication network, (6). ≪ / RTI > The reference image to be output through the monitor unit 6 may be an image of a state of a relevant organ, for example, an X-ray, a CT, and / or an MRI image. Also, which part of the reference image (for example, which part of the whole body image for the patient) is to be output for reference can be determined by, for example, the name of the recognized organ or the coordinate information of the actual surgical tool 460 have. For this purpose, the coordinate information or the name of each part of the reference image or the frame of the reference image of the sequential frame may be specified in advance. Any one reference image may be outputted through the monitor unit 6, or two or more reference images (for example, an X-ray image and a CT image) having different properties may be displayed together.

In step 1750, the master robot 1 outputs the laparoscopic image and the reference image through the monitor unit 6, respectively. At this time, by processing the reference image so as to be displayed in a direction similar to the input angle (e.g., camera angle) of the laparoscopic image, the intuitiveness of the operator can be maximized. For example, when the reference image is a planar image taken in a specific direction, a three-dimensional image using a real-time MPR (Multi Planar Reformat) may be outputted according to the camera angle calculated by the characteristic value calculator 710. For reference, the MPR is a technique for selectively forming only a necessary part of a single or multiple slices in a cross-sectional image and constructing a partial three-dimensional image. The MPR is a technique for constructing a region of interest (ROI) Is a technique that develops the technique of drawing.

The case where the master robot 1 functions in the first mode which is the actual mode, the second mode which is the comparison mode and / or the third mode which is the virtual mode has been described. Hereinafter, the case where the master robot 1 functions in the fourth mode which is the education mode or the fifth mode which is the simulation mode will be mainly described. It should be noted that the various embodiments relating to the display of the virtual surgical tool 610 described above with reference to the related drawings are not technical ideas that are limitedly applied in a specific driving mode, May be applied without limitation without any explanation.

18 is a plan view showing the entire structure of a surgical robot according to another embodiment of the present invention.

Referring to FIG. 18, the robot system for laparoscopic surgery includes two or more master robots 1 and slave robots 2. The first master robot 1a among the two or more master robots 1 may be a student master robot used by a learner (for example, a student in training) and the second master robot 1b may be a student master robot A teacher master robot used by a teacher). Since the configurations of the master robot 1 and the slave robot 2 are as described above, they will be briefly described.

1, the master interface 4 of the master robot 1 includes a monitor unit 6 and a master controller. The slave robot 2 includes a robot arm 3 and a laparoscope 5, . ≪ / RTI > The master interface 4 may further include a mode switching control button for selecting any one of the plurality of driving modes. The master manipulator may be embodied, for example, in a form (e.g., a handle) that can be manipulated by the operator to be gripped by his or her hands, respectively. Not only the laparoscopic image but also a plurality of biometric information and reference images can be output to the monitor unit 6. [

The two master robots 1 illustrated in FIG. 18 are coupled to each other through a communication network, and can be coupled to the slave robot 2 through a communication network. The master robot 1, which is coupled to each other through a communication network, may be provided in various quantities as needed. In addition, the use of the first master robot 1a and the second master robot 1b, the training teacher and the training student may be determined in advance, but their roles may be exchanged with each other as required or required.

For example, the first master robot 1a for the learner is connected to the second master robot 1b for the teacher only through a communication network, and the second master robot 1b is connected to the first master robot 1a and the slave robot 1a. Or may be coupled to the base station 2 via a communication network. That is, when the trainee student manipulates the master manipulator provided in the first master robot 1a, only the virtual surgical tool 610 is manipulated and output through the screen display unit 320. At this time, an operation signal is supplied from the first master robot 1a to the second master robot 1b, and the operation state of the virtual surgical tool 610 is transmitted to the monitor unit 6b of the second master robot 1b The trainee can confirm that the trainee is proceeding with the normal course.

As another example, the first master robot 1a and the second master robot 1b may be coupled through a communication network and each may be coupled to the slave robot 2 via a communication network. In this case, when the trainee manipulates the master manipulator provided in the first master robot 1a, the actual surgical tool 460 is manipulated, and the corresponding manipulation signal is also provided to the second master robot 1b, Can confirm that the trainee student is performing the operation in the normal course.

In this case, the practical teacher can control his / her master robot to control the mode of the master robot of the practical student. For this purpose, any master robot may be preset to enable the operation mode of the actual surgical tool 460 and / or the virtual surgical tool 610 to be determined by a control signal received from another master robot.

FIG. 19 is a diagram illustrating a method of operating a surgical robot system in a training mode according to another embodiment of the present invention.

19 shows that the operation of the arm operating section 330 in the first master robot 1a functions only for the operation of the virtual surgical tool 610 and the operation signal from the first master robot 1a is transmitted to the second master robot 1b, The operation method of the surgical robot system is exemplified. This is because it is possible to use the second master robot 1b to confirm the operation status of the first master robot 1a by one of the practicing student or the practicing teacher, .

Referring to Fig. 19, in step 1905, communication connection establishment is performed between the first master robot 1a and the second master robot 1b. The communication connection setting may be, for example, to transmit and receive at least one of an operation signal, an authority command, and the like to each other. The communication connection setting may be made at the request of one or more of the first master robot 1a and the second master robot 1b or may be performed immediately when each master robot is powered on.

In step 1910, the first master robot 1a receives a user operation according to the operation of the arm control unit 330. [ Here, the user may be, for example, either a student who is practicing or a teacher who is practicing.

In steps 1920 and 1930, the first master robot 1a generates an operation signal in accordance with the user operation in step 1910, and generates virtual surgical tool information corresponding to the generated operation signal. As described above, the virtual surgery tool information may be generated using the operation information according to the operation of the arm operation unit 330. [

In step 1940, the first master robot 1a determines whether there is an overlapping part or a contact part with the organ based on the generated virtual surgical tool information. A method of determining whether there is a long-term overlap or a contact portion with the virtual surgical tool has been described with reference to FIG. 15 and / or FIG. 16, and a description thereof will be omitted.

If overlap or contact portions are present, proceed to step 1950 to generate processing information according to the overlap or contact. The processing information may be transparency processing for the overlapping portion, performing force feedback according to the contact, and the like, as exemplified in Figs. 15 and / or 16 above.

In step 1960, the first master robot 1a transmits the virtual surgical tool information and / or the processing information to the second master robot 1b. The first master robot 1a may transmit an operation signal to the second master robot 1b or generate virtual surgical tool information using the operation signal received by the second master robot 1b, .

In steps 1970 and 1980, the first master robot 1a and the second master robot 1b output the virtual surgical tool 610 to the screen display unit 320 using the virtual surgical tool information. At this time, items corresponding to the processing information may be processed together.

19, a case has been described in which the first master robot 1a controls only the virtual surgical tool 610, and an operation signal or the like corresponding thereto is provided to the second master robot 1b. However, the first master robot 1a may control the actual operation tool 460 according to the drive mode selection, and an operation signal or the like may be provided to the second master robot 1b.

20 is a diagram illustrating a method of operating a surgical robot system in a training mode according to another embodiment of the present invention.

Referring to FIG. 20, a description will be given of a method of operating the surgical robot system, assuming that the second master robot 1b has a control right for the first master robot 1a.

Referring to Fig. 20, in step 2010, communication connection establishment is performed between the first master robot 1a and the second master robot 1b. The communication connection setting may be, for example, to transmit and receive at least one of an operation signal, an authority command, and the like to each other. The communication connection setting may be made at the request of one or more of the first master robot 1a and the second master robot 1b or may be performed immediately when each master robot is turned on.

In step 2020, the second master robot 1b transmits the operation authorization command to the first master robot 1a. The first master robot 1a has the authority to actually control the robot arm 3 provided in the slave robot 2 by the operation grant command. The operation authorization command may be generated by the second master robot 1b so as to be constituted of, for example, predefined signal forms and information forms between the master robots.

In step 2030, the first master robot 1a receives a user operation according to the operation of the arm control unit 330. [ Here, the user may be, for example, a practicing student.

In step 2040, the first master robot 1a generates an operation signal in accordance with the user operation in step 1910 and transmits it to the slave robot 2 through the communication network. The first master robot 1a generates virtual operation tool information corresponding to the generated operation signal or the operation information according to the operation of the arm operation unit 330 and displays the virtual operation tool 610 through the monitor unit 6 .

In addition, the first master robot 1a may transmit operation signals and / or virtual surgical tool information to the second master robot 1b so that the actual operation tool 460 may be checked. In step 2050, the second master robot 1b receives the manipulation signal and / or virtual surgical tool information.

In steps 2060 and 2070, the first master robot 1a and the second master robot 1b respectively receive the laparoscopic image received from the slave robot 2 and the laparoscopic image received from the slave robot 2 in accordance with the operation of the arm control part 330 of the first master robot 1a And outputs the virtual surgical tool 610 through the screen display unit 320. [

If the second master robot 1b does not output the virtual surgical tool 610 according to the operation of the arm control unit 330 of the first master robot 1a to the screen display unit 320 but receives the virtual surgical tool 610 from the slave robot 2 If the operation situation of the actual surgical tool 460 is confirmed through the laparoscopic image, step 2050 may be omitted, and only the received laparoscopic image will be output in step 2070.

In step 2080, the second master robot 1b determines whether a request for retrieving the operation right granted to the first master robot 1a from the user is input. Here, the user may be, for example, a trainee student, and can recover the operation authority in the case where normal operation can not be performed by the user of the first master robot 1a.

If the operation right withdrawal request is not input, the process returns to step 2050 to allow the user to observe the operation state of the actual operation tool 460 by the first master robot 1a.

However, if it is determined that the request for surgical removal is input, in step 2090, the second master robot 1b transmits a surgical operation termination command via the communication network to the first master robot 1a.

The first master robot 1a can switch to the training mode in which the operation state of the actual surgical tool 460 by the second master robot 1b can be observed by transmitting the operation authority end command (step 2095) .

20, the case where the second master robot 1b has the control authority for the first master robot 1a has been mainly described. However, in the opposite case, the first master robot 1a may transmit a surgical operation termination request to the second master robot 1b. This is for transferring authority to allow the user of the second master robot 1b to actually operate the surgical tool 460. For example, it is not easy to perform the operation at the operation site, or the operation of the operation site is very easy In one case, it could be used in case of education.

In addition, various methods for transferring operation authority or control authority among a plurality of master robots to each other, or for allowing a single master robot to grant / revoke authority can be considered and applied without limitation.

Various embodiments of the present invention have been described above with reference to the related drawings. However, the present invention is not limited by the above-described embodiments, and a more various embodiments can be additionally suggested.

In one embodiment, when a plurality of master robots are connected through a communication network and function in the fourth mode which is an education mode, the learner's ability to control the master robot 1 or the operative ability may be performed.

The training mode evaluating function is a function that the practitioner manipulates the arm control unit 330 of the second master robot 1b to perform the virtual surgery tool 610 while the training teacher is performing the operation using the first master robot 1a, As shown in FIG. The second master robot 1b receives the laparoscopic image from the slave robot 2 and analyzes the characteristic value or characteristic information about the actual surgical tool 460. The second master robot 1b also analyzes the characteristic value or characteristic information about the actual surgical tool 460, And interprets the control process of the tool 610. Then, the second master robot 1b analyzes the movement trajectory and the operation form of the actual surgical tool 460 included in the laparoscopic image and the similarity between the movement trajectory and the operation form of the virtual surgical tool 610 by the practicing student, It is possible to calculate the evaluation score for the student.

In another embodiment, the master robot 1 may operate as a surgical simulator by combining characteristics of an organ in a three-dimensional shape obtained by using a stereoscopic endoscope in a fifth mode, which is a simulation mode in which a virtual mode is further improved.

For example, when the laparoscopic image or the virtual screen output through the screen display unit 320 includes the liver, the master robot 1 extracts the liver characteristic information stored in the storage unit, So that the surgical simulation can be performed in a virtual mode during surgery or separately from surgery. Whether organs such as laparoscopic images are included can be determined by, for example, recognizing the color or shape of a relevant organ using normal image processing and recognition technology, and comparing the recognized information with previously stored characteristic information . Of course, the operator may choose which organs were involved and / or which surgical simulations to perform on which organs.

Using this, the operator can perform pre-surgery simulation of how the liver should be resected in any direction using the characteristic information and the matching liver shape before actually cutting or cutting the liver. In the course of the surgical simulation, the master robot 1 performs a surgical operation (for example, one or more of cutting, stitching, pulling, pushing, etc.) based on characteristic information (e.g., mathematical modeling information) The tactile sensation on the ground or soft tissue may be transmitted to the operator.

As a method of transmitting the tactile sensation, for example, there is a method of performing force feedback processing, or the operation sensitivity of the arm operation unit 330 or the resistance force in operation (for example, resistance to stop the arm operation unit 330 when pushing the arm operation unit 330 forward ), And the like.

In addition, a section of the organ virtually resected or cut by the operation of the operator can be output through the screen display unit 320, so that the operator can be made to predict the result of the actual resection or disconnection.

The master robot 1 functions as a surgical simulator. The master robot 1 displays the organs obtained by three-dimensionally using the stereoscopic endoscope through the screen display unit 320 and the organ images reconstructed from reference images such as CT and MRI By matching the three-dimensional shape and matching the three-dimensional shape of the organs reconstructed from the reference image with the characteristic information (for example, mathematical modeling information), a more realistic surgical simulation can be made available to the operator. The characteristic information may be characteristic information specific to the patient or characteristic information generated for universal use.

The control method of the surgical robot system using the augmented reality may be implemented by a software program or the like. The code and code segments that make up the program can be easily deduced by a computer programmer in the field. In addition, the program is stored in a computer readable medium, readable and executed by a computer, and implements the method. The information storage medium includes a magnetic recording medium, an optical recording medium, and a carrier wave medium.

It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit or scope of the invention as defined in the appended claims. It will be understood that the invention may be varied and varied without departing from the scope of the invention.

1, 1a, 1b: master robot 2: slave robot
3: Robot arm 4: Master interface
5: Laparoscope 6: Monitor section
10: Handle 14: Clutch button
310: image input unit 320:
330: Cancer control unit 340: Operation signal generation unit
350: augmented reality realization part 360: control part
710: characteristic value calculating unit 720: virtual surgery tool generating unit
730: test signal processing unit 740: delay time calculating unit
910: interval calculating unit 920:
1210: Network verification unit 1410: Overlay processing unit
1420:

Claims (16)

As a surgical robot system,
Two or more master robots coupled to each other through a communication network; And
And a slave robot including at least one robot arm controlled in accordance with an operation signal received from an arbitrary master robot,
Wherein each of the master robots comprises:
A screen display unit for displaying an endoscopic image corresponding to a video signal provided from a surgical endoscope;
At least one arm operating section provided for respectively controlling at least one robot arm; And
And an augmented reality implementation unit for generating virtual surgery tool information according to a user operation using the dark operation unit to display a virtual surgery tool through the screen display unit.
delete The method according to claim 1,
Wherein the operation of the arm manipulation part of the first master robot, which is one of the two or more master robots, functions to generate the virtual surgical tool information, and the operation of the arm manipulation part of the second master robot, which is another one of the two or more master robots, Wherein the surgical robot system functions to control the cancer.
The method of claim 3,
Wherein a virtual surgical tool corresponding to virtual surgical tool information according to an operation of a dark operation unit of the first master robot is displayed through a screen display unit of the second master robot.
delete delete delete 1. A surgical simulation method performed by a master robot for controlling a slave robot including a robot arm,
Recognizing the organ selection information; And
Displaying the three-dimensional organ image corresponding to the organ selection information using previously stored long-term modeling information,
Wherein the long term modeling information has characteristic information including at least one of a shape, a hue, and a touch of each of internal and external points of a corresponding organ.
9. The method of claim 8,
To recognize the organ selection information,
Analyzing information on at least one of a color and an appearance of an organ included in a surgical site using a video signal input from a surgical endoscope; And
And recognizing an organ matched with the interpreted information among the previously stored long term modeling information.
9. The method of claim 8,
Wherein the organ selection information is selected by one or more organs as an operator.
9. The method of claim 8,
Receiving a surgical operation command for the three-dimensional organ image in accordance with the operation of the dark operation unit; And
Further comprising the step of outputting tactile information according to the operation command using the long term modeling information.
12. The method of claim 11,
Wherein the tactile information is control information for controlling at least one of an operation sensitivity and an operation resistance according to the operation of the arm operation unit, or control information for a force feedback process.
9. The method of claim 8,
Receiving a surgical operation command for the three-dimensional organ image in accordance with the operation of the dark operation unit; And
Further comprising the step of displaying an operation result image according to the surgical operation command using the long term modeling information.
The method according to any one of claims 11 to 13,
Wherein the surgical operation command is at least one of cutting, stitching, pulling, pushing, long-term deformation due to contact, organ damage due to electrosurgery, and bleeding in a blood vessel.
9. The method of claim 8,
Recognizing an organ according to the organ selection information; And
Extracting and displaying a reference image at a position corresponding to a name of the recognized organ in a previously stored reference image,
Wherein the reference image is at least one of an X-ray image, a computed tomography (CT) image, and a magnetic resonance imaging (MRI) image.
A recording medium on which a program of instructions executable by a digital processing apparatus to perform the surgical simulation method according to claim 8 is tangibly embodied and which can be read by a digital processing apparatus.

Images