KR101795720B1 - Control method of surgical robot system, recording medium thereof, and surgical robot system - Google Patents

Abstract

A control method of a surgical robot system for improving safety during surgery using a surgical robot by judging a current surgical situation from laparoscopic images and performing a predetermined operation to cope with the situation, and a recording medium and a surgical robot system recording the same The method comprising: generating image information including at least one of an RGB value or an HSV value representing pixel information of the laparoscopic image from a laparoscopic image; Extracting a region of interest in the laparoscopic image from the generated image information; Determining a current surgical situation from a change in the extracted region of interest over time; And a step of performing a predetermined operation corresponding to the determined surgical condition.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a control method of a surgical robot system for judging and responding to a surgical situation, a recording medium on which the surgical robot system is controlled,

BACKGROUND OF THE INVENTION 1. Field of the Invention [0001] The present invention relates to a control method for a surgical robot system, a recording medium on which the surgical robot system is controlled, and a surgical robot system. More particularly, And more particularly, to a control method of a surgical robot system for improving safety during surgery using a surgical instrument, a recording medium on which the surgical robot system is recorded, and a surgical robot system.

The surgical robot refers to a robot having a function of substituting surgical operation performed by a surgeon. Such a surgical robot has an advantage that it can perform accurate and precise operation compared to a human and can perform a remote operation.

Surgical robots currently being developed around the world include bone surgery robots, laparoscopic surgery robots, and stereotactic robots. Here, a laparoscopic surgical robot is a robot that performs minimally invasive surgery using laparoscopic and small surgical instruments.

Laparoscopic surgery is an advanced surgical technique that inserts a laparoscope, which is an endoscope for inserting a 1 cm hole in the umbilicus and looks into the abdomen. A recent laparoscope is equipped with a computer chip to provide a sharper, more magnified image than is seen with the naked eye, and the laparoscopic surgical instruments specially designed by looking at the screen through the monitor have developed enough to perform any operation. In addition, laparoscopic surgery has the advantage of being less complicated than open surgery, initiating treatment within a very short time after surgery, and maintaining the physical strength and immune function of the surgical patients, have.

On the other hand, the surgical robot system generally comprises a master robot and a slave robot. When the operator manipulates a steering lever (for example, a steering wheel) provided on the master robot, the surgical tool held by the robot arm is coupled to the robot arm of the slave robot.

The above-described background technology is technical information that the inventor holds for the derivation of the present invention or acquired in the process of deriving the present invention, and can not necessarily be a known technology disclosed to the general public prior to the filing of the present invention.

The present invention relates to a control method of a surgical robot system for judging a current surgical situation from laparoscopic images and performing a predetermined operation to cope with the situation, thereby improving safety during surgery using a surgical robot, System.

The method includes generating image information including at least one of an RGB value or an HSV value representing pixel information of the laparoscopic image from a laparoscopic image; Extracting a region of interest in the laparoscopic image from the generated image information; Determining a current surgical situation from a change in the extracted region of interest over time; And a step of performing a predetermined operation corresponding to the determined surgical condition.

In the present invention, the region of interest may be a surgical tool region in which the surgical tool is located within the laparoscopic image.

In the present invention, the step of extracting the region of interest in the laparoscopic image from the generated image information may include applying a K-means clustering method to the generated image information, Performing grouping into a plurality of regions having similar brightness; And binarizing the group after leaving only the group of surgical tools among the groups obtained through the K-means clustering method.

Here, after the binarization step, a Kalman filter may be applied to the binarized image information to remove temporal errors of the binarized image information.

In the present invention, the step of determining the current operation status from the change of the extracted ROI according to the passage of time may include the step of determining whether or not the extracted surgical tool area invades a preset predetermined restriction area .

Here, the predetermined predetermined restriction region may include at least one of blood vessel, tissue, and organ of the subject.

Here, it is determined whether the surgical tool region is within a predetermined distance from the preset predetermined restriction region, whether the surgical tool region is in contact with the preset predetermined restriction region, It can be determined whether or not it is drawn into a predetermined restriction area.

In the present invention, in the step of performing a predetermined operation corresponding to the determined surgical condition, an alarm message through time or auditory information may be diverted.

In the present invention, the step of performing a predetermined operation corresponding to the determined surgical condition may include a step of performing a force feedback operation for applying a predetermined reaction force to deform or increase the reaction force detected by the surgical tool to a certain degree, (force feedback) function can be performed.

In the present invention, the region of interest may be a bleeding region in which bleeding occurs in the laparoscopic image.

Here, extracting the region of interest in the laparoscopic image from the generated image information may include: performing histogram flattening on the image information; And performing mutually inclusive RGB thresholding on the image information on which the histogram smoothing is performed.

The mutually inclusive RGB thresholding performs binarization by assigning a predetermined threshold value to each space of R, G, and B, and then performs binarization on the binarized R, G, and B spaces It is possible to calculate an intersection and output only an area having the same value.

Herein, after the mutually inclusive RGB thresholding is performed, a border of the bleeding region is detected by applying a Canny edge filter to the image information subjected to the mutually inclusive RGB thresholding. And an entropy filter is applied to image information to which the canyon edge filter is applied to smoothen the image information in operation S220f.

Here, after the step of applying the entropy filter, a threshold value is applied to the image information based on the edge detected part of the image information to which the entropy filter is applied, And a Binary edge detection step.

In the present invention, the step of determining the current operation state from the change of the extracted interest area with the passage of time may include determining whether the bleeding area is newly extracted or the area of the extracted bleeding area is increased or decreased Or whether the bleeding area has disappeared can be determined.

In the present invention, it is preferable that the current operation state is determined from the change of the extracted region of interest over time, the first region extracted from the first image at a certain point of time is divided into a plurality of Region, and can be performed by template matching that finds the smallest difference in pixel values between two objects.

In the present invention, the step of determining the current operation state from the change of the extracted ROI with the passage of time may include the steps of removing the noise of the values of the sensor itself used in the sub Kalman filter, And a Federated Kalman filter applying a main Kalman filter to the position of the sensor.

In the present invention, in the step of performing a predetermined operation corresponding to the determined surgical condition, an alarm message through time or auditory information may be diverted.

In the present invention, the region of interest may be a smoke generation region where smoke occurs in the laparoscopic image.

In the present invention, extracting the region of interest in the laparoscopic image from the generated image information may include converting RGB image information into HSV image information; Determining whether or not a smoke is generated through H histogram analysis of the HSV image information; Determining whether or not the smoke is generated through S histogram analysis of the HSV image information; And a step of finally determining whether or not a smoke occurrence situation exists by combining the H histogram analysis and the S histogram analysis result.

Here, the step of determining whether or not the smoke occurrence state is determined through the H histogram analysis of the HSV image information may be determined as a smoke occurrence state if the frequency for H of the specific section in the H histogram distribution is higher than a predetermined threshold value have.

Here, the step of determining whether or not the smoke occurrence status is determined through the S histogram analysis of the HSV image information is determined as a smoke occurrence status when the frequency for S of the specific interval in the S histogram distribution is lower than a predetermined threshold value .

The final step of determining whether or not the smoke is generated by combining the H histogram analysis and the S histogram analysis may include determining whether a smoke occurrence occurs through H histogram analysis of the HSV image information, It is possible to judge that the case where the judgment result of the stage in which the smoke occurrence status is judged through the S histogram analysis of the smoke occurrence status is the final smoke occurrence situation.

In the present invention, the step of determining the current operation state from the change of the extracted region of interest over time may include determining whether or not the smoke generation is started, whether the smoke generation is continued, Can be determined.

In the present invention, in the step of performing a predetermined operation corresponding to the determined surgical condition, an alarm message through time or auditory information may be diverted.

In the present invention, a predetermined operation corresponding to the determined surgical condition is performed, and a control signal for gas discharge may be generated.

In the present invention, the surgical tool may include a surgical needle or a surgical stapler.

Here, in the step of determining the current operation state from the change of the extracted interest area with time, it may be determined whether the surgical needle or the surgical stapler has been moved from a predetermined fixed position.

Here, in the step of performing a predetermined operation corresponding to the determined surgical condition, the movement path of the surgical needle or the surgical stapler may be displayed.

According to another aspect of the present invention, there is provided a method of controlling a surgical robot system, which comprises the steps of: (a) providing a program that is tangibly embodied in a program of instructions executable by a digital processing apparatus to perform a control method of the surgical robot system described in And provides a recorded medium.

Another aspect of the present invention relates to a surgical endoscope for imaging a surgical site; And a robot arm for driving the robot arm with multiple degrees of freedom, a display member for displaying an image photographed through the surgical endoscope; And generating image information including at least one of an RGB value or an HSV value representing pixel information of the laparoscopic image from the laparoscopic image picked up through the surgical endoscope, and extracting, from the generated image information, And a surgical condition determiner for extracting a region of interest, determining a current surgical situation from a change with time of the extracted region of interest, and controlling a predetermined operation corresponding to the determined surgical situation to be performed And a master robot to which the robot is connected.

According to the present invention, during a surgical operation using a surgical robot, a current operation status is determined from a laparoscopic image in real time, and a predetermined operation is automatically performed to cope with the situation, thereby improving safety during surgery using a surgical robot .

1 is a plan view showing the entire structure of a surgical robot system according to an embodiment of the present invention.
2 is a perspective view showing a master robot of the surgical robot system of FIG.
3 is a block diagram schematically showing the configuration of a master robot and a slave robot according to an embodiment of the present invention.
4 is a flowchart illustrating a method of controlling a surgical robot system according to an embodiment of the present invention.
5 is a flowchart showing a first embodiment of a control method of the surgical robot system of FIG.
Figs. 6A to 6D and 7 are diagrams showing respective steps of the first embodiment of the control method of the surgical robot system of Fig. 5; Fig.
Fig. 8 is a flowchart showing a second embodiment of the control method of the surgical robot system of Fig. 4;
Figs. 9A to 9K and Figs. 10A to 10D are views showing respective steps of the second embodiment of the control method of the surgical robot system of Fig.
11A to 11C are views showing a bleeding region extracted by a control method of a surgical robot system according to a comparative example of the present invention.
12A to 12D are views showing a bleeding region extracted by a control method of a surgical robot system according to an embodiment of the present invention.
13 is a flowchart showing a third embodiment of the control method of the surgical robot system of Fig.
14A and 14B are laparoscopic images when smoke occurs and H histograms therefor.
15A and 15B are laparoscopic images when smoke is not generated and H histograms therefor.
Figs. 16A and 16B are laparoscopic images and smoke histograms when smoke occurs.
17A and 17B are laparoscopic images when no smoke is generated and S histograms therefor.
18A, 18B and 18C are laparoscopic images when no smoke is actually generated, H histograms and S histograms therefor.
19A, 19B, and 19C are laparoscopic images when smoke actually occurs, H histograms and S histograms thereof.
20A, 20B and 20C are laparoscopic images when no smoke is actually generated, and H histograms and S histograms therefor.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings, which will be readily apparent to those skilled in the art to which the present invention pertains. The present invention may be embodied in many different forms and is not limited to the embodiments described herein.

Herein, the present invention is a technical idea that can be used universally for surgeries using surgical endoscopes (for example, laparoscopic, thoracoscopic, arthroscopic, non-rigid, etc.), but in describing the embodiments of the present invention, For convenience, a laparoscope is used as an example.

FIG. 1 is a plan view showing an overall structure of a surgical robot system according to an embodiment of the present invention, and FIG. 2 is a perspective view showing a master robot of the surgical robot system of FIG.

1 and 2, the surgical robot system 1 includes a slave robot 200 for performing surgery on a patient lying on the operating table, a master robot 100 for allowing the operator to remotely operate the slave robot 200, . The master robot 100 and the slave robot 200 are not necessarily separated from each other by a separate device that is physically independent and may be integrated into one unit.

The master robot 100 includes an operation lever 110 and a display member 120 and the slave robot 200 includes a robot arm 210 and a laparoscope 220.

In detail, the master robot 100 is provided with an operation lever 110 so that the operator can grasp and operate the operator's hands, respectively. The operation lever 110 may be implemented by two or more handles as illustrated in FIGS. 1 and 2. An operation signal according to a manipulation of a handle of a surgical operator may be transmitted to a slave robot 200 via a wired or wireless communication network And the robot arm 210 is controlled. That is, the operation of the robot arm 210, such as the movement, rotation, cutting, etc., of the robot arm 210 can be performed by operating the handle of the operator.

For example, the operator can manipulate the slave robot arm 210, the laparoscope 220, and the like using a handle-shaped operation lever. Such an operation lever may have various mechanical configurations according to its operation mode, and may include a master handle for operating operations of the slave robot arm 210 and the laparoscope 220 and the like, and a master robot Such as various input devices such as a joystick, a keypad, a trackball, and a touch screen, which are added to the slave robot 100, for operating the robot arm 210 and / or other surgical equipment of the slave robot 200 . Here, the operation lever 110 is not limited to the shape of the handle, and can be applied without any limitation as long as it can control the operation of the robot arm 210 through a network such as a wired or wireless communication network.

An image photographed through the laparoscope 220 is displayed as an image image on the display member 120 of the master robot 100. The display member 120 may also have a touch screen function. Further, the display member 120 may be provided as a stereoscopic display device so that an observer can feel a three-dimensional feeling of life and realism.

Here, the display member 120 may be constituted by one or more monitors, and information necessary for operation may be displayed on each monitor individually. Although FIGS. 1 and 2 illustrate the case where the display member 120 includes three monitors, the number of monitors can be variously determined according to the type and kind of information required to be displayed.

Meanwhile, the slave robot 200 may include one or more robot arms 210. Generally, a robot arm has a function similar to a human arm and / or wrist, and means a device capable of attaching a predetermined tool to a wrist part. In the present specification, the robot arm 210 may be defined as a concept including both upper and lower components such as upper arm, lower arm, wrist, and elbow, and a surgical instrument coupled to the wrist. The robot arm 210 of the slave robot 200 may be configured to be driven with multiple degrees of freedom. For example, the robot arm 210 may include a surgical instrument inserted into a surgical site of a patient, a swinging drive unit that rotates the surgical instrument in a yaw direction according to a surgical position, a pitch direction , A pitch driving part for rotating the surgical instrument, a feed driving part for moving the surgical instrument in the longitudinal direction, a rotation driving part for rotating the surgical instrument, and a surgical instrument driving part installed at the end of the surgical instrument for cutting or cutting the surgical lesion. . However, the configuration of the robot arm 210 is not limited thereto, and it should be understood that these examples do not limit the scope of the present invention. Here, a detailed description of the actual control process, such as rotation and movement of the robot arm 210 in the corresponding direction by operating the operation lever 110 by the operator, will be omitted.

The slave robot 200 may be used more than once to perform a surgery on a patient and the laparoscope 220 for displaying a surgical site through a display member 120 as an image image may be implemented as an independent slave robot 200 It is possible. Also, as described above, embodiments of the present invention can be used universally for surgery in which a variety of surgical endoscopes other than laparoscopes (e.g., thoracoscopic, arthroscopic, non-circumferential, etc.) are used.

3 is a block diagram schematically showing the configuration of a master robot and a slave robot according to an embodiment of the present invention.

3, the master robot 100 includes an image input unit 130, a screen display unit 140, a user operation unit 150, an operation signal generation unit 160, a surgical condition determination unit 170, and a control unit 180. [ . The slave robot 200 includes a robot arm 210 and a laparoscope 220.

The image input unit 130 receives an image photographed through a camera provided in the laparoscope 220 of the slave robot 200 through a wired or wireless communication network.

The screen display unit 140 outputs visual images corresponding to the images received through the image input unit 130 as visual information. In addition, when biometric information of the subject is input from the slave robot 200, the screen display unit 140 can output more information corresponding to the biometric information. The screen display unit 140 may display the patient's related image data (for example, an X-ray image, a computed tomography (CT) image, a magnetic resonance imaging (MRI) A reconstructed three-dimensional image data or a numerical model based on computed tomography (CT) image data). Here, the screen display unit 140 may be implemented in the form of a display member (see 120 in FIG. 2), and an image processing process for causing the received image to be output as an image image through the screen display unit 140 may be performed by a control unit 180, a surgical condition determination unit 170, or an image processing unit (not shown).

Here, the screen display unit 140 (i.e., the display member 120 of FIG. 2) may be provided as a stereoscopic display device. In detail, the stereoscopic display device applies depth information to a two-dimensional image by applying a stereoscopic technique, and uses an image display device (not shown) to allow the observer to feel three- Quot; Herein, the surgical robot system 1 according to an embodiment of the present invention includes a stereoscopic display device as a screen display unit, thereby providing a more realistic virtual environment to a user.

The user manipulation unit 150 is a means for allowing an operator to manipulate the position and function of the robot arm 210 of the slave robot 200. 2, the user operation unit 150 may be formed in the form of a handle-like operating member (see 110 in FIG. 2), but the shape is not limited thereto and may be modified into various shapes for achieving the same purpose . Further, for example, a part of the finger may be formed in a handle shape and the other part may be formed in a different shape such as a clutch button, and a finger insertion tube or insertion More rings may be formed.

The user operation unit 150 may provide a predetermined reaction force to an operation input by the user under the control of the operation situation determination unit 170 and the control unit 180 connected thereto. For example, when the user grasps the user manipulation part 150 and the user manipulation part 150 is pushed in a certain direction, the user manipulation part 150 provides a reaction force of a certain magnitude in a direction opposite to the direction in which the user pushes the manipulation part 150 . As a result, the user must exert a greater force than usual in order to operate the user's operating unit 150, and this can draw attention to the possibility of malfunction of the user.

Or the user operation unit 150 may emit a predetermined alarm message for the operation input by the user under the control of the operation situation determination unit 170 and the control unit 180 connected thereto. For example, when the user manipulates the user's operation unit 150 while the user grasps the user's operation unit 150 and the robot arm 210 enters the predetermined restricted area, the user's operation unit 150 may generate a predetermined alarm the user can divert an alarm message to the user and thereby alert the user of the possibility of malfunction.

The operation signal generating unit 160 generates an operation signal corresponding to the operation of the user operation unit 150 by the operator in order to move the robot arm 210 and / or the laparoscope 220, And transmits it to the slave robot 200. The operation signal can be transmitted and received through a wired or wireless communication network as described above.

The surgical condition determination unit 170 generates image information including at least one of an RGB value or an HSV value representing pixel information of the laparoscopic image from the laparoscopic image, extracts a region of interest in the laparoscopic image from the generated image information, Performs a function of judging a current operation state from a change with time of the extracted region of interest, and performing a predetermined operation corresponding to the determined operation state. The control method such as the specific function of the operation condition determination unit 170 will be described in detail with reference to the related drawings hereinafter.

The control unit 180 controls the operation of each component so that the above-described functions can be performed. The control unit 180 may convert the image input through the image input unit 130 into an image to be displayed through the screen display unit 140. The control unit 180 may also transmit the image input from the image input unit 130 to the operation condition determination unit 170 and execute the operation implementation signal generated by the operation condition determination unit 170 .

Hereinafter, a method of controlling a surgical robot system according to an embodiment of the present invention will be described in detail. 4 is a flowchart illustrating a method of controlling a surgical robot system according to an embodiment of the present invention. Referring to FIG. 4, in a method of controlling a surgical robot system according to an exemplary embodiment of the present invention, image information including at least one of an RGB value or an HSV value representing pixel information of the laparoscopic image is generated from a laparoscopic image A step of extracting a region of interest in the laparoscopic image from the generated image information in operation S20, a step of determining a current operation state from a change in the extracted region of interest over time, Step S30) and a predetermined operation corresponding to the determined surgical condition is performed (operation S40).

Herein, the control method of the surgical robot system according to an embodiment of the present invention is a method of controlling a surgical robot system in which a region of interest is a surgical tool region on a laparoscopic screen, a region of interest is a bleeding region on a laparoscopic screen, smoke generation region. In the following, each case will be described separately.

First, a first embodiment of the present invention relating to detection and control of surgical instruments on the laparoscopic screen will be described. 5 is a flowchart showing a first embodiment of a control method of the surgical robot system of FIG. 6A to 6D and 7 are diagrams showing respective steps of the first embodiment of the control method of the surgical robot system of FIG.

5, 6A to 6D and 7, a first embodiment of a control method of a surgical robot system according to the present invention is characterized in that image information is generated from a laparoscopic image (step S110) (Step S120) of extracting a region of interest in the laparoscopic image from the laparoscopic image, determining whether the surgical tool region invades a predetermined predetermined region (operation S130) And a predetermined operation is performed on the operation of the user operation unit when the restricted area is invaded (step S140).

The step of extracting the region of interest in the laparoscopic image from the generated image information in operation S 120 may include applying a K-means clustering method to the image information in operation S 121, A step S122 of leaving only the group of surgical tools among the groups obtained through the K-means clustering method, binarization thereof, and a step of extracting a center point of the extracted surgical tool region ).

This will be described in more detail as follows.

First, image information including at least one of an RGB value and an HSV value representing pixel information of the laparoscopic image is generated using the laparoscopic image (step S110). That is, it is possible to generate image information including image information represented by RGB values using the provided laparoscopic image, generate image information including image information represented by HSV values using the provided laparoscopic image, Image information including both HSV values can be generated.

Here, R represents red, G represents green, and B represents blue. The numerical range of each component is freely determined. In general, the normalized value is 0 And may be set to have a value between 0 and 255,

The HSV value is a value obtained by subtracting a predetermined value from the RGB value when the RGB value has a value between 0 and 1, and H, H, S, Can be calculated by the following equation. In this case, as in the case of RGB values, the numerical range can be freely determined, but in general, H can be expressed by a degree value of 0 to 360 degrees, and S is a value in which 1 is white with 100% , And V can be expressed as having a value of 1, which is a black color, 0 to white. That is, H (color) is the dominant color recognized by the observer, S (saturation) is the degree of pureness or pure color of the color diluted by white light, and V (brightness) .

Next, the region of interest in the laparoscopic image is extracted from the generated image information (step S120). This will be described in more detail as follows.

First, a K-means clustering method is applied to the image information (step S121). Here, the K-means clustering method is a method of distinguishing regions corresponding to surgical tools by analyzing image information obtained from laparoscopic images.

In detail, in order to recognize a surgical tool in a laparoscopic surgical image, brightness information of a pixel is most significant. Therefore, if the brightness information is divided into a plurality of regions by any method, desired results (i.e., extraction of a surgical tool region) Can be obtained. To do this, image information is converted to gray scale, and the brightness information of each pixel is automatically segmented using an unsupervised clustering method such as a K-means clustering method, Because of the similarity of information, the tool part can easily be automatically classified into one cluster. Unsupervised clustering is a generic term for computation algorithms that are designed to automatically classify clusters of data without an administrator intervention. The K-means clustering method refers to a technique of grouping data near a reference point based on the distance between two images using intensity in an image.

That is, a laparoscopic image as shown in FIG. 6A is converted into gray scale, and a K-means clustering method is performed on the brightness information of each pixel. For example, (Clustering), it is classified into three clusters as shown in Figs. 6B, 6C and 6D. The area to which the surgical tool belongs is a bright color area (non-black area) B in FIG. 6B. By clustering (clustering) the laparoscopic image with the brightness information of each pixel in this way, the surgical tool area can be classified into one area.

Next, after leaving only the group of surgical tools among the groups obtained through the K-means clustering method, binarization of the image is performed in order to determine the boundary of the region corresponding to the surgical tool (S122) do.

That is, among the plurality of regions (for example, the three regions shown in Figs. 6B to 6D) classified through the above-described step S121, the region corresponding to the surgical tool (for example, ). Then, binarization is performed on an area corresponding to the surgical tool. Here, binarization means that the image is monochromatic by a certain threshold value.

Next, the geometric center point of the extracted surgical tool area is extracted (step S123), and assumed to be representative coordinates of the extracted surgical tool area. The geometric center point of the surgical tool area thus extracted is shown in Fig. 7 (see Fig. 7C).

Meanwhile, although not shown in FIG. 5, the step of extracting a region of interest in the laparoscopic image from the generated image information (step S120) may further include removing a temporary error with respect to the position detection result of the surgical tool .

In detail, as the area of the surgical tool is temporally overlapped by other parts and the area of the image is reduced, or the distribution of the color information is temporarily changed due to the smoke generated in the electric decanter, Temporary errors may occur in extracting regions. As a method for eliminating temporal errors due to a logically negligible cause caused by noise in the laparoscopic image and irregular transient errors, there is a method in which the output result of the system based on the characteristic at the previous time such as the Kalman filter , And the stability of the dynamic detection result of the position of the surgical tool can be enhanced by using a method of randomly excluding the abnormal output. Here, the Kalman filter predicts the present output on the assumption that statistical general characteristics will be shared based on the data of the past state of the system, and corrects the characteristic model of the system with the current output value measured again And the output of the system is estimated. Such a Kalman filter is applied to various fields because it plays a solid role as a filter for filtering a temporary abnormal signal due to noise or disturbance.

Next, it is determined whether or not the surgical tool area invades predetermined predetermined restriction areas (step S130). Here, the predetermined restriction region may be a region that can be damaged by surgical instruments during surgery, such as blood vessel, tissue, or organ of the subject. In this case, it is possible to use a template such as a blood vessel / tissue / organ shape on the laparoscopic screen to find an area having the same pattern, and to set it as a predetermined restriction area. Or a predetermined predetermined restriction area may be an area arbitrarily selected by the user in such a manner that the user touches the touch screen on the display member.

Whether or not the surgical instrument region invades predetermined predetermined restriction regions can be determined by determining the position of the surgical instrument region according to the passage of time and determining whether the surgical instrument region is within a predetermined distance from a predetermined predetermined restriction region, Whether the surgical tool area is in contact with a predetermined predetermined restriction area, whether or not the surgical tool area is drawn into a predetermined predetermined restriction area, and the like.

Next, when the surgical tool area violates predetermined predetermined restriction areas, a predetermined operation is performed on the operation of the user's operation unit (step S140).

Here, the predetermined operation may be an operation of emitting a predetermined alarm message to the user. That is, when the surgical tool area has invaded a predetermined restricted area or is close to a restricted object, an alarm message can be transmitted to the user through visual or auditory information or the like.

Or a force feedback function for applying a predetermined reaction force to deform or increase the reaction force sensed by the surgical tool to the operation of the user's operation unit when the surgical tool area invades a predetermined predetermined restriction area May be performed.

In detail, in a surgical-robot system of a master-slave structure, there is a risk that the unintentional movement of the slave robot in operation risks damaging tissues and organs. In order to prevent the unintentional movement of the operator, a virtual fixture concept and force feedback concept for telepresence have been proposed.

Here, the basic concept of force-feedback control is to feedback the reaction force value detected by the sensor or the like attached to the robot arm of the slave robot or its vicinity and the current position value of the attached robot arm to the master robot side , And on the master robot side, taking into account the kinematic structure of the robot itself, the reaction force sensed at the most distal end of the user interaction is controlled to be as close as possible to the feedback value from the slave robot. In other words, this means a function of returning the result of the operation back to the information of the force to the side of operating the mechanism, or a system using the function. Such a force feedback function can be used to perform a manual operation The same feeling can be reproduced.

Here, the surgical robot system according to an embodiment of the present invention may perform a force feedback function to apply a predetermined reaction force to the operation of the user's operation unit when invading a predetermined restriction area set as a surgical tool area have. Here, the predetermined reaction force refers not to the actual reaction force sensed by the robot arm, but to a reaction force that deforms or augments the reaction force sensed by the robot arm to a certain degree. The reaction force that is deformed or augmented to a certain degree is given to the user, And to improve the safety during surgery by drawing attention of the user.

Although not shown in the drawings, in the first embodiment of the control method of the surgical robot system of the present invention, the surgical tool further includes not only the surgical instrument shown in Fig. 6A or the like, but also a surgical needle or a surgical stapler It is possible. That is, the surgical robot system of the present invention may include various surgical instruments that can be inserted into a user's body during surgery. Since the surgical needle, the surgical stapler, or the like is formed of a metal material such as the surgical instrument shown in FIG. 6A or the like, it can be detected from the laparoscopic image by a method similar to the method of detecting the surgical instrument described above. When the surgical needle or the surgical stapler thus detected is inadvertently disappeared to a place other than the surgical site, the movement trajectory of the surgical needle or the surgical stapler is displayed to the user, It is possible to easily find a surgical needle or a surgical stapler.

Next, a second embodiment of the present invention relating to detection and control of a bleeding area on the laparoscopic screen will be described. Fig. 8 is a flowchart showing a second embodiment of the control method of the surgical robot system of Fig. 4; Figs. 9A to 9K and Figs. 10A to 10D are diagrams showing respective steps of the second embodiment of the control method of the surgical robot system of Fig. 8. Fig.

8, 9A to 9K and 10A to 10D, the second embodiment of the control method of the surgical robot system of the present invention is characterized in that image information is generated from the laparoscopic image (step S210) A step S230 of extracting a region of interest in the laparoscopic image from the image information obtained in operation S230, a step S230 of determining a current operation state from a change in the extracted region of interest with time, And a predetermined operation corresponding to the situation is performed (step S240).

The step of extracting the region of interest in the laparoscopic image from the generated image information in operation S220 may include resizing the generated image information and cutting the boundary region in operation S220b, (Step S220c), mutually inclusive RGB thresholding is performed on the image information on which the histogram has been smoothed (S220d), and the image information subjected to the mutually inclusive RGB thresholding is subjected to a canny edge filter The entropy filter is applied to the image information to which the canyon edge filter is applied in step S220f, the background is removed from the image information to which the entropy filter is applied in step S220g, In step S220h, a bleeding area is extracted from the removed image information, A step of filling the inside of the extracted bleeding area (step S220i), a boundary of the bleeding area is extracted from the image information filled in the bleeding area (step S220j), the boundary of the extracted bleeding area (Step S220k), and the center point of the extracted bleeding area is extracted (step S220l). This will be described in more detail as follows.

First, the original image information as shown in FIG. 9A is resized as shown in FIG. 9B, and the boundary region is cut (step S220b).

Next, histogram flattening is performed on the image information (step S220c). In detail, Histogram equalization of the RGB histogram is performed as a preprocessing step for analyzing the color information of the image information. Here, the RGB histogram planarization is a process of converting the intensity histogram of the image into R, G, B space ) To a wide range from 0 to 255, respectively, so that the contrast is markedly increased. Thus, the image information on which the RGB histogram has been smoothed is shown in FIG. 9C.

Next, mutually inclusive RGB thresholding is performed on the image information on which the histogram smoothing is performed (step S220d). In detail, Mutually inclusive RGB thresholding is a method of performing binarization by assigning a predetermined threshold value to each space of R, G, and B, and then performing an intersection of R, G, and B spaces subjected to binarization And outputting only the region having the same value. FIG. 10A shows a screen in which the image of FIG. 9C in which the RGB histogram is flattened is binarized by giving a predetermined threshold value in the R space, FIG. 10B shows a screen in which the RGB histogram is flattened, FIG. 10C shows a screen in which the image of FIG. 9C in which the RGB histogram is flattened is subjected to a predetermined threshold on the G space And the binarization is performed. FIG. 10D (and FIG. 9D) shows a screen in which the intersection areas of FIGS. 10A, 10B, and 10C are extracted.

Next, as shown in FIG. 9E, a Canny edge filter is applied to the image information subjected to the mutually inclusive RGB thresholding (step S220e), and the border of the bleeding area is roughly detected. Here, the Canny edge filter is an example of an optimal edge detection technique, and refers to a filter that detects an edge by finding a portion having a large gradient strength.

Next, as shown in FIG. 9F, an entropy filter is applied to the image information to which the CannyEdge filter is applied (step S220f), so that the border of the bleeding area is smoothly connected. Here, an entropy filter is an example of a technique of smoothing an image. In this case, the value of the pixel is divided into a certain peripheral region (for example, a 9 pixel x 9 pixel region surrounding the pixel) Quot; is replaced with an entropy value of " a "

Next, as shown in FIG. 9G, the background is removed from the image information to which the entropy filter is applied (step S220g). Here, a binary edge detection method may be applied to remove the background. Binary edge detection refers to a method of removing a small island (i.e., noise) by applying a predetermined threshold to an image based on a portion where an edge is detected in a monochrome image. That is, by applying the result of applying the canyon edge filter and the result of applying the entropy filter and applying the binary edge detection method, the boundary portion can be accurately detected without being disconnected.

By applying Canny edge filter, Entropy filter and Binary edge detection method, it is possible to obtain the effect of detecting the boundary of the bleeding area more stably by using Canny edge filter, Entropy filter and Binary edge detection.

Next, as shown in FIG. 9H, a bleeding area is extracted from the image information from which the background is removed (step S220h), and then the extracted bleeding area is filled in the image information as shown in FIG. 9I ( S220i step). Then, as shown in FIG. 9J, the border of the bleeding area is extracted from the image information filled in the bleeding area (step S220j).

The boundary B of the extracted bleeding area may be overlapped with the original image information and displayed (step S220k), as shown in FIG. 9K. Then, a geometric center point of the extracted bleeding region is extracted (step S220l), and assumed to be representative coordinates of the extracted bleeding region.

11A to 11C are views showing a bleeding region extracted by a control method of a surgical robot system according to a comparative example of the present invention. 11A to 11C, in the case of the bleeding area extracted by the control method of the surgical robot system according to the comparative example of the present invention, it is larger than the actual bleeding area (C1 in FIG. 11A), smaller than the actual bleeding area (C2 in Fig. 11B), or it includes surgical tools and organs in addition to the actual bleeding area (C3 in Fig. 11C).

12A to 12D are views showing a bleeding region extracted by a control method of a surgical robot system according to an embodiment of the present invention. 12A to 12D, it can be seen that the bleeding area extracted by the control method of the surgical robot system according to the embodiment of the present invention is completely the same as the actual bleeding area (D1, D2 in FIG. 12B, and D3 in FIG. 12C), the bleeding area is not extracted when there is no bleeding area in the laparoscopic image (FIG. 12D).

Next, the current operation state is determined from the change with time of the extracted interest region (step S230).

In detail, when a bleeding occurs during the operation using the surgical robot system, it is possible to determine the place where the bleeding occurs, whether the bleeding temporarily occurs or not, whether the bleeding amount increases or decreases, and the like It can have a great influence on the progress of surgery. Therefore, it can be very important to understand the pattern of bleeding. Therefore, in the present invention, the aspect of the bleeding from the change of the bleeding area according to the passage of time is grasped and transmitted to the user.

There are two main ways to determine the current surgical situation from the changes of the extracted region of interest over time.

The first method is template matching.

The basic template matching is based on the shape perception. The first template extracted from the first image at a certain point of time is sequentially compared with the plurality of regions of the second image at another point of time, This means finding the smallest difference between the values. That is, the first area is compared with the entire second image. However, this method has a disadvantage in that the processing speed is slow and the amount of calculation is large because the first region must be compared over the entire second image.

In order to solve such a disadvantage, a Sum of Squared Difference (SSD), which is one type of template matching, is a kind of block matching technique. It is an error of the difference between the first image and the second image of the first image And finds an area having the smallest error. SSD uses block matching based on Fast Fourier Transform and starts with the assumption that the area with the smallest error is the most similar area. The advantage of SSD over basic template matching is that it computes not only the intensity of the pixel but also the phase component so that the intensity and the external environment are less It is sensitive. However, this method has a problem that there is a high rate of change for illumination and image light.

To solve these drawbacks, the proposed method is NCC (Normalized Cross Correlation). This means that the geometric similarity between the first image and the second image of the first image is measured independently of the linear difference between the brightness of the first image and the brightness of the second image of the first image. When the first region of the first image and the second image are compared with each other using the NCC, the position of the largest NCC value is output, and the image of the first region of the first image in the second image You can find the location. The advantage of Fast Fourier Transform-based Normalized Cross Correlation (NCC) is that it is a normalization process before the effects of ambient environment such as illumination light are compared. It is less influenced by the surrounding environment, is less computationally fast, The bigger it is, the faster it is.

On the other hand, the second method is a method of applying a federated Kalman filter. As described above, the Kalman filter is a method of statistically predicting the next position using the stored position. Here, the federated Kalman filter means that the Sub Kalman filter and the Main Kalman filter exist to remove the noise of the values of the sensor itself used in the Sub Kalman filter, , And applying a main Kalman filter to the positions of various sensors output, it is possible to predict a more accurate position.

Further, it is also possible to use the above two methods in combination. That is, when movement of a center value of a specific region is detected through a template matching method, an error occurring between a center value and an actual center value detected by applying a federated Kalman filter is corrected .

Using these methods, the current surgical situation is determined from the change over time of the extracted region of interest. That is, it can be grasped by analyzing the temporal / spatial change of the bleeding area extracted from the image, such as whether the bleeding starts, the amount of continuous bleeding increases, the amount of bleeding increases, and how much is maintained or the bleeding is stopped .

Finally, a predetermined operation corresponding to the determined surgical condition is performed (step S240). In particular, the predetermined operation may be an operation of emitting a predetermined alarm message to the user. That is, when it is determined that a dangerous situation such as the start of bleeding or the amount of bleeding has occurred, an alarm message can be transmitted to the user through visual or auditory information or the like. In addition, medical imaging information of the lower or proximal portion of the bleeding (suspicion) region may be provided in addition to the laparoscopic image to identify the origin of the bleeding.

According to the present invention, it is possible to quickly identify the occurrence of bleeding and the position of occurrence of bleeding, so that it is possible to promptly cope with the occurrence of an emergency situation at the time of surgery, thereby improving the safety during surgery.

Next, a third embodiment of the present invention relating to detection and control of a smoke generation area on the laparoscopic screen will be described. 13 is a flowchart showing a third embodiment of the control method of the surgical robot system of Fig. FIGS. 14A and 14B are laparoscopic images and smoke histograms when smoke occurs, and FIGS. 15A and 15B are laparoscopic images and smoke histograms when no smoke is generated. 16A and 16B are laparoscopic images and smoke histograms when smoke occurs, and FIGS. 17A and 17B are laparoscopic images and smoke S histograms when no smoke is generated.

Referring to FIG. 13 and the like, a third embodiment of a control method of a surgical robot system of the present invention includes generating image information from a laparoscopic image (step S310), extracting, from the generated image information, A control signal for alarm and / or gas discharge corresponding to the determined surgical condition is determined (step S320), whether or not smoke is generated, maintained, increased, (Step S340).

The step of extracting the region of interest in the laparoscopic image from the generated image information in operation S320 may include converting the RGB image information into HSV image information in operation S321, (Step S322), determining whether or not a smoke is generated through S histogram analysis (S323), and analyzing H histogram analysis and S histogram analysis to determine whether or not a smoke is generated (S324) . This will be described in more detail as follows.

First, the RGB image information is converted into HSV image information (step S321). In detail, in order to detect smoke on the laparoscopic screen, RGB image data must be converted into HSV image data. As described above, R represents red (red), G represents green (green), and B represents blue (blue). The HSV value is a value obtained by subtracting the next RGB value from the RGB value when the RGB value has a value between 0 and 1. The HV value is represented by Hue, S is saturation, and V is brightness and Brightness. Can be calculated by Equation (1).

Next, it is determined whether or not the smoke is generated through the H histogram analysis (step S322). In other words, in the H histogram distribution, if the frequency for H of a specific section is higher than a predetermined threshold, or if the (weighted) average of the H values of all or a portion of the image data in the image data exceeds a predetermined threshold value It is judged that there is no smoke occurrence in the case of smoke occurrence.

14A and 14B are laparoscopic images and smoke histograms when smoke occurs, and FIGS. 15A and 15B are laparoscopic images and smoke histograms when no smoke is generated. 14A and 14B, it can be seen that a large number of histogram values are distributed in the H1 region on the right side of the H histogram when smoke is generated. On the other hand, referring to FIGS. 15A and 15B, it can be seen that the histogram values are hardly distributed in the H2 area on the right side of the H histogram when no smoke is generated. From this, when H histogram is extracted from the HSV image information of the laparoscopic image, it can be judged that smoke is generated when a plurality of histogram values are distributed in the right region of the H histogram, and histogram values in the right region of the H histogram are almost If it is not distributed, it can be judged that no smoke has occurred.

Next, it is determined whether or not the smoke is generated through the S histogram analysis (step S323). In other words, in the S histogram distribution, if the frequency for S of a specific section is lower than a predetermined threshold, or if the (weighted) average of the S values of all or some of the regions of interest of the image data exceeds a predetermined threshold ), It is judged that there is a smoke generation situation, otherwise, it is determined that there is no smoke generation.

16A and 16B are laparoscopic images and smoke histograms when smoke is generated, and Figs. 17A and 17B are laparoscopic images when no smoke is generated and S histograms thereof. 16A and 16B, it can be seen that the histogram values are hardly distributed in the S1 area on the right side of the S histogram when smoke is generated. On the other hand, referring to FIGS. 17A and 17B, it can be seen that a large number of histogram values are distributed in the S2 area on the right side of the S histogram when no smoke is generated. From this, when the S histogram is extracted from the HSV image information of the laparoscopic image, it can be judged that no smoke occurs when a plurality of histogram values are distributed in the right region of the S histogram, and the histogram value It can be determined that smoke has occurred.

Next, the H histogram analysis and the S histogram analysis result are combined to determine whether or not the smoke occurrence state is finally determined (step S324). That is, in order to judge the smoke occurrence situation stably, it is the final judgment as the smoke occurrence situation when both of the above items are satisfied.

Specifically, Figs. 18A, 18B and 18C are laparoscopic images when no smoke is actually generated, H histograms and S histograms therefor. 19A, 19B, and 19C are laparoscopic images when smoke actually occurs, H histograms and S histograms thereof. 20A, 20B and 20C are laparoscopic images when no smoke is actually generated, and H histograms and S histograms therefor.

18A, 18B and 18C, laparoscopic images when no smoke is actually generated are converted into HSV image information, and H histograms and S histograms are extracted. First, in the H histogram shown in FIG. 18B, it can be seen that the histogram values are hardly distributed in the H3 region on the right side of the H histogram. From this, it can be judged that the current situation does not cause the postponement . On the other hand, in the S histogram shown in FIG. 18C, it can be seen that a large number of histogram values are distributed in the S3 region on the right side of the S histogram. From this, it can be judged that the current situation does not cause a delay. And, since the analysis result of the H histogram and the analysis result of the S histogram coincide with each other, it can be concluded that the present situation is the situation in which the smoke does not occur.

19A, 19B and 19C, the laparoscopic image at the time of actual smoke generation is converted into HSV image information, and H histogram and S histogram are extracted. First, in the H histogram shown in FIG. 19B, it can be seen that a large number of histogram values are distributed in the H4 region on the right side of the H histogram, and it can be judged that the current situation is in a state of delay. On the other hand, in the S histogram shown in FIG. 19C, it can be seen that the histogram values are not distributed in the S4 region on the right side of the S histogram. From this, it can be judged that the current situation is a state where the smoke is occurring. In addition, since the analysis result of the H histogram and the analysis result of the S histogram coincide with each other, it can be concluded that the current situation is the situation where the smoke is occurring.

20A, 20B and 20C, laparoscopic images when no smoke is actually generated are converted into HSV image information, and H histograms and S histograms are extracted. First, in the H histogram shown in FIG. 20B, it can be seen that a plurality of histogram values are distributed in the H5 area on the right side of the H histogram. From this, it can be judged that the current situation is in a state of delay. On the other hand, in the S histogram shown in FIG. 20C, it can be seen that a large number of histogram values are distributed in the S5 area on the right side of the S histogram, and it can be judged that the present situation is not causing the smoke . If the analysis result of the H histogram and the analysis result of the S histogram do not match in this way, it can be finally judged that the present situation is not a postponement. In other words, it can be judged that only when the analysis result of the H histogram and the analysis result of the S histogram are all in the state of the occurrence of the smoke, the smoke is finally generated.

According to the present invention as described above, it is possible to more stably judge the smoke generation situation.

Next, it is determined whether or not smoke is generated, maintained, increased, decreased, and eliminated (step S330). That is, based on the temporal / spatial change of the information obtained from the histogram distribution of the image information, it is possible to grasp the situation information such as whether the smoke is generated, the smoke is continuing, or the smoke is decreasing.

Finally, a control signal for alarm and / or gas discharge corresponding to the determined surgical condition is generated (operation S340). That is, when it is determined that a dangerous situation such as the start of smoke generation or an increase in smoke has occurred, an alarm message can be transmitted to the user through visual or auditory information or the like. In addition, a control signal for gas discharge may be transmitted to the control unit so that the gas discharge apparatus is operated.

Although the present invention has been described with reference to the limited embodiments, various embodiments are possible within the scope of the present invention. It will also be understood that, although not described, equivalent means are also incorporated into the present invention. Therefore, the true scope of protection of the present invention should be defined by the following claims.

1: Surgical robot system 100: Master robot
110: Operation lever 120: Display member
200: Slave robot 210: Robot arm
220: Laparoscopic

Claims (31)

delete Generating image information including at least one of an RGB value or an HSV value representing pixel information of the laparoscopic image from a laparoscopic image captured through a camera provided in the laparoscope;
Extracting a region of interest in the laparoscopic image from the generated image information;
Determining a current surgical situation from a change in the extracted region of interest over time; And
And performing a predetermined operation corresponding to the determined surgical condition,
Wherein the region of interest is a surgical tool region in which the surgical tool is located within the laparoscopic image. 3. The method of claim 2,
Extracting a region of interest in the laparoscopic image from the generated image information,
Applying a K-means clustering method to the generated image information, and grouping the image information into a plurality of regions having similar brightness; And
And a step of binarizing the group after leaving only the group of surgical tools among the groups obtained through the K-means clustering method. The method of claim 3,
After the binarization step,
Further comprising the step of applying a Kalman filter to the binarized image information to remove temporal errors in the binarized image information. 3. The method of claim 2,
Wherein the current operation state is determined from the change of the extracted ROI according to the flow of time,
Wherein the step of determining whether or not the extracted surgical tool region invades a predetermined predetermined restriction region is determined. 6. The method of claim 5,
Wherein the predetermined restriction region includes at least one of blood vessels, tissues, and organs of the subject's surgeon. 6. The method of claim 5,
Whether the surgical tool region is within a predetermined distance from the preset predetermined restriction region or whether the surgical tool region is in contact with the preset predetermined restriction region or whether the surgical tool region is in contact with the predetermined predetermined region Wherein the control unit determines whether or not the robot enters the restricted area. 3. The method of claim 2,
Wherein the predetermined operation corresponding to the determined surgical condition is performed,
And an alarm message is emitted through the visual or auditory information. 3. The method of claim 2,
Wherein the predetermined operation corresponding to the determined surgical condition is performed,
Wherein a force feedback function for applying a predetermined reaction force to deform or increase the reaction force detected by the surgical tool to a certain degree is performed for the operation of the user's operation unit. Generating image information including at least one of an RGB value or an HSV value representing pixel information of the laparoscopic image from a laparoscopic image captured through a camera provided in the laparoscope;
Extracting a region of interest in the laparoscopic image from the generated image information;
Determining a current surgical situation from a change in the extracted region of interest over time; And
And performing a predetermined operation corresponding to the determined surgical condition,
Wherein the region of interest is a bleeding region in which bleeding occurs in the laparoscopic image. 11. The method of claim 10,
Extracting a region of interest in the laparoscopic image from the generated image information,
Performing histogram smoothing on the image information; And
And performing mutually inclusive RGB thresholding on the image information on which the histogram smoothing is performed. delete delete delete 11. The method of claim 10,
Wherein the current operation state is determined from the change of the extracted ROI according to the flow of time,
Wherein it is determined whether or not the bleeding area is newly extracted, whether the area of the extracted bleeding area is increased or decreased, or whether the bleeding area is extinguished. 11. The method of claim 10,
Wherein the current operation state is determined from the change of the extracted ROI according to the flow of time,
A first region extracted from a first image at a certain point of time is sequentially compared with a plurality of regions of a second image at another point of view to perform template matching to find a position having a smallest difference in pixel values between two objects Wherein the control unit controls the operation of the surgical robot. 11. The method of claim 10,
Wherein the current operation state is determined from the change of the extracted ROI according to the flow of time,
This is performed by applying a federated Kalman filter which removes the noise of the values of the sensor itself used in the sub Kalman filter and applies a main Kalman filter to the positions of the outputted sensors Wherein the control unit controls the operation of the surgical robot system. 11. The method of claim 10,
Wherein the predetermined operation corresponding to the determined surgical condition is performed,
And an alarm message is emitted through the visual or auditory information. Generating image information including at least one of an RGB value or an HSV value representing pixel information of the laparoscopic image from a laparoscopic image captured through a camera provided in the laparoscope;
Extracting a region of interest in the laparoscopic image from the generated image information;
Determining a current surgical situation from a change in the extracted region of interest over time; And
And performing a predetermined operation corresponding to the determined surgical condition,
Wherein the region of interest is a smoke generation region where smoke has occurred in the laparoscopic image. 20. The method of claim 19,
Extracting a region of interest in the laparoscopic image from the generated image information,
Converting RGB image information into HSV image information;
Determining whether or not a smoke is generated through H histogram analysis of the HSV image information;
Determining whether or not the smoke is generated through S histogram analysis of the HSV image information; And
And finally determining whether or not the smoke occurrence state is obtained by combining the H histogram analysis and the S histogram analysis result. delete delete delete 20. The method of claim 19,
Wherein the current operation state is determined from the change of the extracted ROI according to the flow of time,
Wherein it is determined whether or not the generation of the smoke is started, whether or not the generation of the smoke is continued, or whether the smoke is extinguished. 20. The method of claim 19,
Wherein the predetermined operation corresponding to the determined surgical condition is performed,
And an alarm message is emitted through the visual or auditory information. 20. The method of claim 19,
Wherein the predetermined operation corresponding to the determined surgical condition is performed,
And a control signal for gas discharge is generated. delete delete delete A program of instructions executable by a digital processing apparatus to perform a method of controlling a surgical robotic system according to any one of claims 2 to 11, 15 to 20, 24 to 26, A recording medium on which a program that is tangibly embodied and which can be read by a digital processing apparatus is recorded. A surgical endoscope for imaging a surgical site; And
A slave robot including a robot arm driving with multiple degrees of freedom,
A display member for displaying an image taken through the surgical endoscope; And
Generating image information including at least one of an RGB value or an HSV value representing pixel information of the laparoscopic image from the laparoscopic image taken through the surgical endoscope, extracting, from the generated image information, interest in the laparoscopic image And a surgical condition determining unit for determining a current operation state from the change of the extracted interest area over time and controlling a predetermined operation corresponding to the determined operation state to be performed, Including a master robot,
Wherein the region of interest is at least one of a surgical tool region in which the surgical tool is located in the laparoscopic image, a bleeding region in which bleeding occurs in the laparoscopic image, and a smoke generation region in which smoke occurs in the laparoscopic image, A surgical robot system.

Images