WO2024144133A1

WO2024144133A1 - Surgical robot system and control method thereof

Info

Publication number: WO2024144133A1
Application number: PCT/KR2023/021377
Authority: WO
Inventors: 김형주; 최종도; 왕준상; 고경완
Original assignee: (주)미래컴퍼니
Priority date: 2022-12-26
Filing date: 2023-12-22
Publication date: 2024-07-04

Abstract

Disclosed are a surgical robot system and a control method thereof, the surgical robot system comprising: a plurality of robot arms; an instrument which is selectively installed on at least one of the plurality of robot arms and has, at the tip, an end effector that operates by being inserted into a surgical site; a camera which is installed on at least one of the plurality of robot arms and inserted into the surgical site, and generates image information by imaging the surgical site; and a control unit which controls the operations of the plurality of robot arms and the end effector, and recognizes the state of the end effector within the image information and thereby adjusts the strength of a driving force transmitted to the end effector.

Description

Surgical robot system and its control method

Embodiments disclosed herein relate to a surgical robot system and a control method thereof. More specifically, a robot system and control method that is inserted into the surgical site of the human body and performs surgery using surgical tools while confirming the surgical site through an image. It's about.

This study was conducted as a result of the research results of the “Pan-Ministerial Medical Device Research and Development Project (R&D) (Ministry of Science and ICT, Ministry of Health and Welfare, Ministry of Trade, Industry and Energy)” of the Multi-Ministry and (Foundation) Pan-Ministry Life Cycle Medical Device Research and Development Project Group (KMDF_PR_20200901_0090 -01).

Minimally invasive surgery is a general term for surgery that minimizes the size of the affected area.

Specifically, minimally invasive surgery, unlike open surgery, which is performed by opening a large incision in a part of the human body (e.g., the abdomen), involves forming at least one incision (or invasive hole) measuring 0.5 cm to 1.5 cm in the human body. , This is a surgical method that is performed while watching images after inserting an endoscope camera and various surgical tools through this incision.

Unlike open surgery, this minimally invasive surgery has the advantages of less pain after surgery, early recovery of bowel movement and early food intake, short hospitalization period, quick return to normal condition, and excellent cosmetic effect due to the narrow incision range. has Due to these advantages, minimally invasive surgery is being used for cholecystectomy, prostate cancer surgery, and hernia repair, and its field is expanding.

Surgical robots generally used in minimally invasive surgery include a master robot and a slave robot, such as the surgical robot disclosed in Korean Patent Publication No. 10-2014-0102465. The master robot generates control signals according to the doctor's operations and transmits them to the slave robot, and the slave robots receive control signals from the master robot and perform the operations necessary for surgery on the patient.

Here, the slave robot is equipped with at least one robot arm, and the end of each robot arm is equipped with an instrument that performs surgery and a camera that provides images of the surgical site, and an end effector for holding a surgical tool is installed at the end of the instrument. is installed.

In minimally invasive surgery using such a surgical robot, the end effector of the instrument and the camera enter the inside of the patient's body, and the necessary procedure is performed while confirming the surgical site through the camera's image.

The end effector performs the procedure by holding a surgical tool such as a needle at the surgical site of the human body. When holding the needle at the surgical site, the needle is held with relatively strong force to firmly fix the needle.

Here, there is a risk that the surgical robot may grip human tissue along with the needle during the process of the end effector gripping the needle, or grip human tissue without grasping the needle.

Here, in the case of a conventional surgical robot, the grip force of the end effector is controlled to a constant set value for the purpose of gripping only the needle, and in particular, when gripping a relatively thin needle, the grip force is relatively small. It is controlled by strong power. Accordingly, if a conventional end effector grips human tissue other than a needle, there is a problem of causing serious damage to human tissue.

Accordingly, new technologies are required to overcome the limitations of the prior art as described above.

Meanwhile, the above-described background technology is technical information that the inventor possessed for deriving the present invention or acquired in the process of deriving the present invention, and cannot necessarily be said to be known technology disclosed to the general public before filing the application for the present invention. .

Embodiments disclosed herein present a surgical robot system and control method that is inserted into the surgical site of the human body and performs surgery by selectively mounting surgical tools on the end effector while confirming the surgical site through an image through a camera. There is a purpose.

In particular, the purpose of the embodiments disclosed in this specification is to present a surgical robot system and control method that can determine whether a surgical tool such as a needle is mounted on the end effector through image information provided by a camera.

In addition, in the embodiments disclosed in this specification, when the end effector is composed of a gripper, it is possible to check whether the surgical tool is gripped through the image information of the camera, and the grip force of the gripper can be adjusted to different strengths depending on whether the surgical tool is gripped. The purpose is to present a surgical robot system and control method that can be controlled.

In addition, embodiments disclosed herein can adjust the grip force of the end effector according to the grip direction, which may lower the grip force of the end effector when the grip direction is adjusted while rotating while the end effector is holding a surgical tool. The purpose is to present a surgical robot system and control method.

In addition, the purpose of the embodiments disclosed herein is to present a surgical robot system and control method that can detect the gripping state of the end effector through a change in current acting on the drive motor during the end effector gripping process. there is.

As a technical means for achieving the above-described technical problem, one embodiment of a surgical robot system includes a plurality of robot arms; An instrument that is selectively installed on at least one of the plurality of robot arms and has an end effector at the tip that operates by being inserted into the surgical site; a camera installed on at least one of the plurality of robot arms, inserted into the surgical site, and generating image information by photographing the surgical site; It may include a control unit that controls the operations of the plurality of robot arms and the end effector and recognizes the state of the end effector within the image information to adjust the strength of the driving force transmitted to the end effector.

In addition, as a technical means for achieving the above-described technical problem, an embodiment of a control method of a surgical robot system includes a plurality of robot arms, selectively installed on at least one of the plurality of robot arms, and a surgical site at the tip. An instrument equipped with an end effector that is inserted and operated, a camera installed on at least one of the plurality of robot arms and inserted into the surgical site, and generating image information by photographing the surgical site, and the plurality of robot arms and the end. A surgical robot system comprising a control unit that controls the operation of an effector, comprising: recognizing the state of the end effector within the image information, performed by the control unit; The method may also include adjusting the intensity of the driving force transmitted to the end effector according to the state of the end effector.

According to one of the above-mentioned problem solving means, a surgical robot system and control method are provided that are inserted into the surgical site of the human body and perform surgery by selectively holding surgical tools with an end effector while confirming the surgical site through an image through a camera. can be presented.

In particular, according to one of the above-mentioned problem solving means, the control unit determines whether a surgical tool is mounted on the end effector through image information from a camera and adjusts the grip force of the end effector to different strengths depending on whether the surgical tool is determined. Since it is controlled by , damage to human tissue caused by the end effector can be prevented, and a surgical robot system and control method that can hold surgical tools such as needles more safely and firmly can be proposed.

In addition, according to one of the means for solving the above-described problem, the control unit adjusts the grip force according to the gripping direction of the end effector, so that the surgical tool can be held more stably even when the end effector rotates while holding the surgical tool. A surgical robot system and control method can be presented.

The effects that can be obtained from the disclosed embodiments are not limited to the effects mentioned above, and other effects not mentioned are clear to those skilled in the art to which the disclosed embodiments belong from the description below. It will be understandable.

Figure 1 is a configuration diagram showing the overall configuration of a surgical robot system according to an embodiment.

Figure 2 is a configuration diagram showing the configuration of instruments of a surgical robot system according to an embodiment.

Figure 3 is an example diagram showing image information of the surgical site captured by a camera.

Figure 4 is a flowchart showing a control method of a surgical robot system according to an embodiment.

FIG. 5 is a flowchart illustrating an embodiment of step S410 in the control method shown in FIG. 4.

FIG. 6 is a flowchart showing another embodiment of step S410 in the control method shown in FIG. 4.

FIG. 7 is a flowchart showing another embodiment of step S410 in the control method shown in FIG. 4.

FIG. 8 is a flowchart showing an embodiment of step S420 in the control method shown in FIG. 4.

Below, various embodiments will be described in detail with reference to the attached drawings. The embodiments described below may be modified and implemented in various different forms. In order to more clearly explain the characteristics of the embodiments, detailed descriptions of matters widely known to those skilled in the art to which the following embodiments belong have been omitted. In addition, in the drawings, parts that are not related to the description of the embodiments are omitted, and similar parts are given similar reference numerals throughout the specification.

Throughout the specification, when a configuration is said to be “connected” to another configuration, this includes not only cases where it is “directly connected,” but also cases where it is “connected with another configuration in between.” In addition, when a configuration “includes” a configuration, this means that other configurations may be further included rather than excluding other configurations, unless specifically stated to the contrary.

Hereinafter, embodiments will be described in detail with reference to the attached drawings.

The surgical robot system according to one embodiment is a system designed to have a robot arm operate under the doctor's control to perform surgery on behalf of the doctor. Instruments and a camera mounted on the tip of the robot arm are inserted into the body to allow the doctor to view the surgical site. It is a device that properly controls the robot arm and instruments while confirming the video.

Referring to FIG. 1, the surgical robot system 1 according to one embodiment includes a slave robot 10 that performs surgery on a patient P lying on the operating table 2, and an operator (10) of the slave robot 10. It may include a master console 20 that allows remote control by O, and a vision cart 30 that allows the assistant A to check the progress of the surgery through the display unit 35.

The slave robot 10 may include one or more robot arms 11. In general, the robot arm 11 has functions similar to those of a human arm and/or wrist and refers to a device that can attach a predetermined tool to the area corresponding to the wrist. In this specification, the robot arm 11 can be defined as a concept encompassing all components such as the upper arm, lower arm, wrist, and elbow. In this way, the robot arm 11 of the slave robot 10 can be implemented to be driven with multiple degrees of freedom.

In addition, an instrument 12 that is inserted into the surgical site of the patient (P) to perform surgery and a camera 13 that photographs the surgical site and provides image information may be installed at the tip of the robot arm 11. The tip of the instrument 12 may be provided with an end effector 123 on which a surgical tool such as a needle is selectively mounted. Additionally, the instrument 12 may be configured in various ways to include different types of end effectors 123 depending on the purpose. For example, the needle holder, which is one of the instruments 12, may be configured to include a gripper as the end effector 123 that grips the needle with strong force. Here, the gripper is an end effector with a structure for holding surgical tools such as needles. Any type of end effector can be used as long as it includes a structure that can selectively hold or place the surgical tool according to the operation of the operator (O). do.

Meanwhile, the robot arm 11 operates with multiple degrees of freedom, enabling the instrument 12 to move or rotate within a certain range, and at the same time provide driving force to the end effector 123 of the instrument 12. .

However, the configuration of the robot arm 11 is not limited to this, and it should be understood that this example does not limit the scope of the present invention. Here, a detailed description of the actual control process, such as the robot arm 11 rotating and moving in the corresponding direction when the operator O manipulates the operation lever, will be omitted.

Meanwhile, one or more slave robots 10 can be used to perform surgery on a patient P, and the camera 13 for displaying the surgical site as an image through the display unit 35 is an independent slave robot 10. It can also be implemented as:

Additionally, embodiments of the present invention can be universally used in surgeries using various surgical endoscopes other than laparoscopes (eg, thoracoscope, arthroscope, rhinoscope, etc.).

Additionally, the master console 20, the slave robot 10, and the vision cart 30 do not necessarily have to be physically separated into separate devices, and at least some of them may be integrated to form an integrated system. However, hereinafter, the case where the master console 20, the slave robot 10, and the vision cart 30 are physically separated will be described as an example.

The master console 20 may include an operating lever (not shown) and a display member (not shown). In addition, the master console 20 may be further equipped with an external display device 25 capable of displaying the status of the operator O.

In detail, the master console 20 may be provided with an operating lever (not shown) so that the operator O can operate it. The manipulation lever can be implemented as one or more handles, and the manipulation signal according to the handle manipulation by the operator (O) is transmitted to the slave robot 10 through a wired or wireless communication network to control the robot arm 11. It can be. That is, surgical operations such as moving the position of the robot arm 11, rotating the instrument 12, and cutting operations can be performed by manipulating the handle of the operator O.

For example, the operator O can operate the robot arm 11 or the instrument 12 of the slave robot 10 using a handle-shaped operating lever. This operating lever can have various mechanical configurations depending on the operation method, and includes a master handle that operates the operation of the robot arm 11 or instrument 12 of the slave robot 10, and the function of the entire system. Various input tools such as joysticks, keypads, trackballs, and touch screens added to the master console 20 for operation are provided in various forms to operate the robot arm 11 of the slave robot 10 or other surgical equipment. It can be. Here, the operating lever is not limited to the shape of the handle, and can be applied without any restrictions as long as it has a shape that can control the operation of the robot arm 11 through a network such as a wired or wireless communication network.

An image captured through the camera 13 may be displayed as an image image on the display member of the master console 20. Additionally, a predetermined virtual operation panel may be displayed on the display member together with the image captured through the camera 13 or may be displayed independently. Furthermore, the image captured by the camera 13 may be provided on the display member of the master console 20 as a three-dimensional image with different left-eye and right-eye images.

The display member may be provided in various forms through which the operator (O) can check the image. For example, a display device may be installed to correspond to both eyes of the operator O. As another example, it may consist of one or more monitors, and information necessary for surgery may be individually displayed on each monitor. The quantity of display members may be determined in various ways depending on the type or type of information required to be displayed.

The vision cart 30 is installed to be spaced apart from the slave robot 10 or the master console 20, and the progress of the surgery can be checked from the outside through the display unit 35.

The image displayed on the display unit 35 may be the same as the image displayed on the display member of the operator O. The assistant (A) can assist the operator (O) in the surgical work while checking the image on the display unit 35. For example, the assistant A can replace the instruments 12 or surgical tools in the instrument cart 3 depending on the progress of the surgery.

Meanwhile, the control unit 40 is connected to the slave robot 10, the master console 20, and the vision cart 30, and can transmit and receive respective signals, and the slave robot 10, the master console 20, and the vision cart It can be installed in any one of (30) or independently.

The control unit 40 is a component that controls the overall operation of the surgical robot system 1 including the master console 20 and the slave robot 10. For example, the control unit 40 may be configured to include one or more processors such as a CPU. there is. The control unit 40 controls other components included in the surgical robot system and can perform calculations required for the slave robot 10 to perform operations required for surgery.

This control unit 40 may be configured to include, for example, a main control unit provided in the master console 20 and an auxiliary control unit provided in the slave robot 10.

The control unit 40 processes information input from the master console 20 to generate commands to be transmitted to the slave robot 10, and receives signals transmitted from the slave robot 10 to determine the operating state of the slave robot 10. or the image information transmitted from the camera 13 provided in the slave robot 10 can be output to the display unit 35 of the vision cart 30 and the display member of the master console 2. To this end, the control unit 40 may further include additional components such as memory or GPU. More specific operations of the control unit 40 will be described later.

Meanwhile, Figure 2 is a configuration diagram showing the configuration of instruments of a surgical robot system according to an embodiment. Referring to FIG. 2, the instrument 12 is formed to extend in the longitudinal direction, and its tip is inserted into the surgical site of the human body. An end effector 123, which will be described later, is installed at the tip, thereby providing a needle-like effect to the end effector 123. It is a component that performs surgery by selectively mounting surgical tools.

The instrument 12 can be installed on at least one of the plurality of robot arms 11 provided in the slave robot 10 and inserted into the surgical site, and includes a drive motor (not shown) that operates under the control of the control unit 40. The end effector 123 can be operated by transmitting the driving force of ) to the tip.

Specifically, the instrument 12 may be configured to include a housing 121, a shaft 122, an end effector 123, and a power transmission member 124, as shown in FIG. 2.

The housing 121 may be detachably coupled to the robot arm 11 of the slave robot 10. Specifically, the tip of the robot arm 11 is provided with a connector (not shown) that is coupled to the housing 121, and the housing 121 can be fitted and coupled to the connector of the robot arm 11. At this time, the connector and housing 121 may be provided with one or more driving wheels that rotate together while engaging with each other in corresponding shapes. Accordingly, the driving force of the wire wound or unwound by the driving force of the driving motor (not shown) provided on one side of the slave robot 10, for example, the robot arm 11, is transmitted to the driving wheel of the connector, which in turn is transmitted to the driving wheel of the connector. Power is provided to the end effector 123 of the instrument 12 by rotating the drive wheel provided in the housing 121.

Meanwhile, the shaft 122 is a component formed to a predetermined length and inserted into the surgical site of the human body. It extends in the longitudinal direction from the housing 121, and the tip in the longitudinal direction is moved to the surgical site by the operation of the robot arm 11. It can be inserted into the site. At this time, the tip of the shaft 122 refers to the distal end of the robot arm 11 among both ends of the shaft 122.

An end effector 123 is installed at the front end of the shaft 122, and a housing 121 is installed at the opposite end of the end effector 123, that is, at the rear end, to be coupled to the robot arm 11 together with the housing 121. You can.

And the shaft 122 can transmit the driving force of the above-described drive motor to the end effector 123 installed at the tip. For this purpose, the shaft 122 is formed in the shape of a hollow tube, and a power transmission member 124 may be formed inside.

The power transmission member 124 is a component that transmits the rotational force of the driving wheel of the rear end of the shaft 122, that is, the housing 121, to the end effector 123 provided at the front end of the shaft 122, for example. , may include a wire (not shown) that is unwound or wound by the rotation of the driving wheel provided in the housing 121.

As described above, the end effector 123 is installed at the tip of the shaft 122 constituting the instrument 12, is inserted into the surgical site together with the shaft 122, and operates by driving force. At this time, the end effector 123 may be configured to include a gripping means for gripping a surgical tool, that is, the above-described gripper, depending on the embodiment.

Here, the surgical tool can be any tool that performs procedures such as suturing and incision on the surgical site, such as a needle or scalpel, and the end effector 123 can selectively mount the surgical tool in various ways. It can be composed of a structure such as

This end effector 123 may be composed of a gripper including a pair of jaws 123a, as shown in FIG. 2, and operates by using the driving force transmitted to the power transmission member 124. As the pair of jaws 123a opens or retracts, the surgical tool can be gripped and mounted.

Meanwhile, referring again to FIG. 1, the camera 13 is installed on one of the robot arms 11 of the slave robot 10 and can be inserted into the surgical site of the human body to provide image information of the surgical site. For example, the camera 13 may be configured as an endoscope assembly in which an optical device such as a CCD camera is provided at the tip of a tube extending in the longitudinal direction. The camera 13 is moved by the robot arm 11 that operates according to commands transmitted to the master console 20 and can generate image information by photographing the body with its tip inserted into the human body. And the generated image information may be transmitted back to the vision cart 30 and/or the master console 20.

Meanwhile, the control unit 40 can control the robot arm 11 and the end effector 123 of the instrument 12. The control unit 40 provides driving force to the end effector 123, for example, a pair of jaws 123a of the gripper, so that the pair of jaws 123a come closer to each other and engage or spread apart, or the pair of jaws 123a moves apart. (123a) can be controlled to rotate together in a certain direction. For example, the control unit 40 can control the gripper by controlling the rotation of the drive motor that provides driving force to each of the pair of jaws 123a so that the wire is unwound or wound.

In particular, the control unit 40 can detect the state of the end effector 123 and control the grip force of the end effector 123 at different strengths. Here, the 'state' of the end effector 123 is, for example, whether the end effector 123 is holding a surgical tool such as a needle or is not holding a surgical tool, and a pair of groups ( This may include whether 123a) is open or engaged, whether it is moving in the same direction, and which direction the gripper is facing.

According to one embodiment, the control unit 40 may detect the state of the end effector 123 based on image information received from the camera 13. The control unit 40 may analyze image information and determine whether the end effector 123 is holding a surgical tool in the image.

For this purpose, the control unit 40 can use a machine learning model that has been learned in advance. For example, the machine learning model uses an image of the end effector 123 in a state holding a surgical tool and an image of the end effector 123 in a state not holding a surgical tool, so that it can be determined whether the end effector 123 is in a state holding a surgical tool. It can be trained in advance using a plurality of learning images prepared by securing a sufficient number of images of the end effector 123 in the state. At this time, each learning image recognizes the end effector 123 from the actual surgical image information obtained in advance, crops the image to a certain size based on the end effector 123, and then provides information about whether the surgical tool is held. It can be created by recording in association.

Accordingly, the machine learning model sufficiently learns whether each learning image shows whether the end effector 123 is holding a surgical tool, thereby making it possible to classify whether the end effector 123 is holding a surgical tool.

Accordingly, the control unit 40 can use the machine learning model trained in this way to detect whether the end effector 123 is holding a surgical tool in at least some frames among the image information of the actual surgery in progress. At this time, the control unit 40 preferentially recognizes the end effector 123 in the image information, generates detection data by cropping the image in a certain range based on the end effector 123, and applies the detection data to a machine learning model. By inputting and obtaining a result, it is possible to detect whether the end effector 123 is holding a surgical tool.

Meanwhile, in another embodiment, the control unit 40 recognizes the end effector 123 and the surgical tool from the image information, and controls the end effector 123 to use the surgical tool based on the position of the recognized end effector 123 or the surgical tool. It is possible to determine whether it is in a gripped state.

To this end, the control unit 40 can perform object detection or object recognition within the image signal. And in order to perform such object detection or recognition, the control unit 40 can use a machine learning model that has been trained in advance.

Here, the machine learning model may be a learning model that has previously learned a learning data set containing a sufficient number of learning images of various end effectors 123 and various surgical tools.

Accordingly, the control unit 40 may detect the end effector 123 or a surgical tool while performing object detection or object recognition by inputting at least some frames of image information into a machine learning model, It is possible to determine whether the surgical tool is installed by determining the distance between the detected end effector 123 and the surgical tool.

That is, as shown in FIG. 3, when the surgical tool is a needle N, the control unit 40 detects the needle N and the end effector 123 from the image information provided by the camera 13, respectively, The distance between the detected end effector 123 and the needle (N) is determined, and if the determined distance is less than or equal to a set value, it can be determined that the end effector 123 is equipped with the needle (N).

Depending on the embodiment, if the surgical tool is not recognized within the image information or the surgical tool is not recognized within a certain range from the end effector 123, the control unit 40 determines whether the end effector 123 is not holding the surgical tool. It can be identified as being in a state.

Meanwhile, depending on the embodiment, the machine learning model determines whether the end effector 123 is equipped with a surgical tool by learning all images of the end effector 123 holding two or more different types of surgical tools. It can also be learned to identify the type of surgical tool.

The control unit 40 can control the grip force of the end effector 123 at different strengths by controlling the size of the current applied to the drive motor 14 depending on whether the surgical tool is mounted on the end effector 123. there is.

That is, in order to appropriately adjust the grip force of the end effector 123 according to the situation, the control unit 40 can detect whether a surgical tool is held through image information and appropriately adjust the strength of the grip force of the end effector 123. .

Specifically, when the end effector 123 is not gripping a surgical tool, the control unit 40 sets the current value applied to the drive motor 14 lower than the default value to increase the relative grip force of the end effector 123. A control signal can be applied to adjust it low.

Accordingly, when the control unit 40 determines that the end effector 123 is not equipped with a surgical tool, the control unit 40 sets the grip force of the end effector 123 lower than the default value so that the end effector 123 grips human tissue. Even so, tissue damage can be prevented.

In addition, the control unit 40 can adjust the grip force of the end effector 123 to be relatively high by setting the current value applied to the drive motor 14 to a default value when the end effector 123 is holding a surgical tool. A control signal can be applied to enable

Accordingly, when it is determined that the end effector 123 is equipped with a surgical tool, the control unit 40 sets the grip force of the end effector 123 to a relatively high default value to place the surgical tool on the end effector 123. It can be mounted firmly.

Meanwhile, the control unit 40 refers to the control value provided by the drive motor that provides driving force to the end effector 123 or the current value measured from the drive motor in addition to the image information, so that the end effector 123 holds the surgical tool. You can also determine whether it is in a certain state or not. For example, when a command is received to cause the pair of jaws 123a to engage and close according to the control value provided by the drive motor, the control unit 40 analyzes the image information at the corresponding time and operates the end effector 123. ) can determine whether the surgical tool is being held. Alternatively, when the current value measured from the drive motor increases significantly, at least temporarily, the control unit 40 recognizes that the end effector 123 is grasping something, analyzes the image information at that time, and detects the end effector 123. It is possible to determine whether the surgical tool is being held.

Meanwhile, when a control signal for a situation such as performing a suturing operation while the end effector 123 is holding a surgical tool is applied through the master console 20, the current value applied to the drive motor 14 is changed to By setting it higher than the gripping state of the tool, the grip force of the end effector 123 can be adjusted to be stronger than the gripping state.

Meanwhile, the pair of jaws 123a constituting the end effector 123 can be rotated and adjusted through control of the master console 20 while holding the surgical tool, and the gripping direction can be adjusted through the control unit 40. The grip force of the end effector 123 can be adjusted to different strengths while adjusting the amount of current applied to the drive motor 14 differently depending on the gripping direction of the end effector 123 controlled through the master console 20.

Here, when the pair of jaws 123a are rotated while holding a surgical tool, one of the pair must move in the direction of opening and the other of the pair must move in the direction of retraction, so the grip force is reduced during the rotation. may deteriorate.

When the gripping direction of the pair of jaws 123a rotates in a direction in which the grip force may be reduced as described above, the control unit 40 adjusts the control value applied to the drive motor 14 to control the pair of jaws 123a. ) can be adjusted to be stronger than the grip state. Specifically, the control unit 40 can adjust the grip force between the pair of jaws 123a to be stronger than the default value.

At this time, the control unit 40 may change the grip force by detecting the rotation direction of the pair of jaws 123a as described above only when it recognizes that the end effector 123 is holding the surgical tool.

For example, when the control unit 40 determines that the end effector 123 is holding a surgical tool, the control unit 40 controls the grip force of the end effector 123 to the default value, and then controls the grip force of the end effector 123 to a default value, and then controls the grip force of the end effector 123 to a default value. When turning in one direction, the grip strength can be instantly increased.

At this time, the control unit 40 may refer to image information to detect the rotation direction of the pair of jaws 123a, or may refer to a control value transmitted to the drive motor or a current value measured from the drive motor. For example, if the driving command for a pair of groups 123a is a command to move in the same direction according to the same value, the control unit 40 adjusts the control value of at least one of the pair of groups 123a differently to The grip force between the pair of jaws 123a can be increased. Alternatively, if the current value measured by the drive motor temporarily decreases by a small amount while the surgical tool is held between the pair of jaws 123a, the control unit 40 recognizes that the grip force between the two has decreased and increases the grip force. You can do it.

Meanwhile, depending on the embodiment, the control unit 40 detects the gripping direction of the end effector 123 based on image information applied from the camera 13, and adjusts the gripping force of the end effector 123 according to the detected gripping direction. You can also adjust it.

Meanwhile, as described above, the control unit 40 can determine the grip state information of the state in which the end effector 123 is gripping the surgical tool based on the change in current measured from the drive motor as well as the image information. To this end, measuring means may be provided to measure the current of the driving motor and apply it to the control unit 40.

The control method of the surgical robot system 1 including the above components will be described with reference to FIGS. 4 to 8.

Figure 4 is a flowchart showing a control method of a surgical robot system according to an embodiment, Figures 5 to 7 are flowcharts showing various embodiments of step S410 in the control method of Figure 4, and Figure 8 is a flowchart of Figure 4 This is a flowchart showing an example of step S420 in the control method.

The method of controlling a surgical robot system according to an embodiment includes recognizing the state of the end effector 123 (S410) and adjusting the intensity of the driving force transmitted to the end effector 123, as shown in FIG. 4. It may be performed including (S420).

The step (S410) of recognizing the state of the end effector 123 is a step in which the above-described control unit 40 recognizes the state of the end effector 123 through the image information of the camera 13, and the end effector 123 This is the step of recognizing whether a surgical tool such as a needle is being held, or whether the surgical tool is not being held.

In more detail, referring to FIG. 5, in the step (S410) of recognizing the state of the end effector 123, the control unit 40 performs machine learning model training using a learning image for the end effector 123 ( After S510), the state of the end effector 123 can be detected from the image information by inputting the image information into the learned machine learning model (S520).

In the step of learning the machine learning model (S510), the control unit 40 generates images of two or more different states for the end effector 123, for example, a state in which a surgical tool is being held and a state in which the surgical tool is not being held. A machine learning model can be learned using a plurality of training images prepared by securing a sufficient number of.

In the step of detecting the state of the end effector 123 (S520), the control unit 40 inputs image information into the machine learning model that has completed learning and determines whether the end effector 123 is holding a surgical tool based on the image information. can be detected.

Meanwhile, referring to FIG. 6, when the end effector 123 is composed of a gripper consisting of a pair of jaws 123a, the control unit 40 performs a step (S610) of learning a machine learning model while the gripper performs surgery. Machine learning model training can be performed using multiple learning images for the state in which the tool is held and the state in which the tool is not held. At this time, the control unit 40 recognizes the gripper from the actual surgical image information obtained in advance, crops the image to a certain size based on the gripper, and then creates a learning image by correlating the information about whether the surgical tool is gripped. You can learn.

In addition, while performing the step (S620) of detecting the state of the end effector 123, the control unit 40 inputs image information into a machine learning model that has completed learning to determine whether the gripper is holding a surgical tool based on the image information. It can be detected whether or not At this time, the control unit 40 preferentially recognizes the end effector 123 in the image information, generates detection data by cropping the image in a certain range based on the end effector 123, and applies the detection data to the machine learning model. By inputting, it is possible to detect whether the end effector 123 is holding a surgical tool.

As another embodiment, referring to FIG. 7, the control unit 40 performs machine learning model training using a plurality of learning images of the gripper and surgical tool while performing the step (S710) of training the machine learning model. You can. At this time, the control unit 40 may learn in advance a learning data set containing a sufficient number of learning images of the end effector 123 and various surgical tools.

In addition, the control unit 40 performs the step of detecting the state of the end effector 123 and inputs image information into the machine learning model that has completed learning, so that the gripper can recognize each surgical tool from the image information (S720) . At this time, the control unit 40 may detect the gripper or surgical tool while performing object detection or object recognition by inputting at least some frames of image information into a machine learning model.

Additionally, the control unit 40 may determine whether the gripper is holding the surgical tool based on the recognized distance between the gripper and the surgical tool (S730). At this time, the control unit 40 can determine the distance between the gripper and the surgical tool based on the positions of the gripper and the surgical tool detected in the image information, and if the determined distance is less than the set value, the gripper is equipped with the surgical tool. It can be judged that

Meanwhile, in the above step S410, the state of the end effector 123 that the control unit 40 wants to recognize is a state in which the surgical tool is held and a state in which the surgical tool is not held. However, as needed, other examples include the state of the end effector 123. A machine learning model can be trained to recognize whether a pair of jaws 123a are engaged or apart, and the rotation state of each pair of jaws 123a.

Referring again to FIG. 4, in the step S420 of adjusting the intensity of the driving force transmitted to the end effector 123, the intensity of the driving force can be adjusted according to the state of the end effector 123 recognized through step S410.

At this time, the control unit 4 can control the size of the current applied to the drive motor and adjust the grip force of the end effector 123 differently depending on whether the surgical tool is mounted on the end effector 123 determined through step S410. there is. That is, the control unit 40 sets the current value to the default value when the surgical tool is mounted and adjusts the grip force of the end effector 123 to be relatively strong, and when the surgical tool is not mounted, the control unit 40 sets the current value to the default value. By setting it lower than the default value, the grip force of the end effector 123 can be adjusted to be relatively weak.

Meanwhile, referring to Figure 8, when the end effector 123 is composed of a gripper consisting of a pair of jaws 123a, the control unit 40 performs a step of adjusting the strength of the driving force while allowing the gripper to grip the surgical tool. In one state, the rotation direction of each pair of pairs 123a can be detected (S810).

That is, the control unit S810 can detect whether the pair of jaws 123a rotate in a direction that may reduce the grip force by rotating in the same direction. At this time, the control unit 40 can recognize the pair of jaws 123 in the image information and detect each rotation direction. As another example, the control unit 40 can rotate the gripper based on the control value transmitted to the drive motor that rotates the gripper. Direction can be sensed.

And, the control unit 40 can adjust the grip force of the gripper based on the detected rotation direction of the gripper (S820). At this time, when the gripping direction of the pair of jaws 123a rotates in a direction that may reduce the grip force, the control unit 40 adjusts the control value applied to the drive motor 14 to be higher than the default value to control the pair of jaws 123a. The grip force of (123a) can be strongly adjusted.

As discussed above, in the surgical robot system 1 and its control method according to an embodiment, the control unit 40 determines whether a surgical tool is mounted on the end effector 123 through image information from the camera, and Since the grip force of the end effector 123 is controlled at different strengths depending on whether the surgical tool is determined to be installed, damage to human tissue caused by the end effector 123 can be prevented, and surgical tools such as needles can be used more safely. A surgical robot system and control method that can hold firmly can be proposed.

The term '~unit' used in the above embodiments refers to software or hardware components such as FPGA (field programmable gate array) or ASIC, and the '~unit' performs certain roles. However, '~part' is not limited to software or hardware. The '~ part' may be configured to reside in an addressable storage medium and may be configured to reproduce on one or more processors. Therefore, as an example, '~ part' refers to components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, and procedures. , subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables.

The functions provided within the components and 'parts' may be combined into a smaller number of components and 'parts' or may be separated from additional components and 'parts'.

In addition, the components and 'parts' may be implemented to regenerate one or more CPUs within the device or secure multimedia card.

The control method according to the embodiment described with reference to FIGS. 4 to 8 may also be implemented in the form of a computer-readable medium that stores instructions and data executable by a computer. At this time, instructions and data can be stored in the form of program code, and when executed by a processor, they can generate a certain program module and perform a certain operation. Additionally, computer-readable media can be any available media that can be accessed by a computer and includes both volatile and non-volatile media, removable and non-removable media. Additionally, computer-readable media may be computer recording media, which are volatile and non-volatile implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, or other data. It can include both volatile, removable, and non-removable media. For example, computer recording media may include magnetic storage media such as HDDs and SSDs, optical recording media such as CDs, DVDs, and Blu-ray discs, or accessible via a network. It may be memory included in the server.

Additionally, the control method according to the embodiment described with reference to FIGS. 4 to 8 may be implemented as a computer program (or computer program product) including instructions executable by a computer. A computer program includes programmable machine instructions processed by a processor and may be implemented in a high-level programming language, object-oriented programming language, assembly language, or machine language. . Additionally, the computer program may be recorded on a tangible computer-readable recording medium (eg, memory, hard disk, magnetic/optical medium, or solid-state drive (SSD)).

Therefore, the control method according to the embodiment described with reference to FIGS. 4 to 8 can be implemented by executing the above-described computer program by a computing device. The computing device may include at least some of a processor, memory, a storage device, a high-speed interface connected to the memory and a high-speed expansion port, and a low-speed interface connected to a low-speed bus and a storage device. Each of these components is connected to one another using various buses and may be mounted on a common motherboard or in some other suitable manner.

Here, the processor can process instructions within the computing device, such as to display graphical information to provide a graphic user interface (GUI) on an external input or output device, such as a display connected to a high-speed interface. These may include instructions stored in memory or a storage device. In other embodiments, multiple processors and/or multiple buses may be utilized along with multiple memories and memory types as appropriate. Additionally, the processor may be implemented as a chipset consisting of chips including multiple independent analog and/or digital processors.

Memory also stores information within a computing device. In one example, memory may be comprised of volatile memory units or sets thereof. As another example, memory may consist of non-volatile memory units or sets thereof. The memory may also be another type of computer-readable medium, such as a magnetic or optical disk.

And the storage device can provide a large amount of storage space to the computing device. A storage device may be a computer-readable medium or a configuration that includes such media, and may include, for example, devices or other components within a storage area network (SAN), such as a floppy disk device, a hard disk device, an optical disk device, Or it may be a tape device, flash memory, or other similar semiconductor memory device or device array.

The above-described embodiments are for illustrative purposes, and those skilled in the art will recognize that the above-described embodiments can be easily modified into other specific forms without changing the technical idea or essential features of the above-described embodiments. You will understand. Therefore, the above-described embodiments should be understood in all respects as illustrative and not restrictive. For example, each component described as single may be implemented in a distributed manner, and similarly, components described as distributed may also be implemented in a combined form.

The scope sought to be protected through this specification is indicated by the patent claims described later rather than the detailed description above, and should be interpreted to include the meaning and scope of the claims and all changes or modified forms derived from the equivalent concept. .

Claims

Multiple robotic arms;

An instrument that is selectively installed on at least one of the plurality of robot arms and has an end effector at the tip that operates by being inserted into the surgical site;

a camera installed on at least one of the plurality of robot arms, inserted into the surgical site, and generating image information by photographing the surgical site; and

A surgical robot system comprising a control unit that controls the operations of the plurality of robot arms and the end effector and recognizes the state of the end effector within the image information to adjust the strength of the driving force transmitted to the end effector.
According to paragraph 1,

The end effector is,

It is composed of a gripper that selectively grips the surgical tool,

The control unit,

Recognize at least one of the gripper and the surgical tool from the image information, determine whether the gripper is holding the surgical tool, and determine whether the gripper is holding the surgical tool. A surgical robot system that adjusts the gripping force of the gripper.
According to paragraph 2,

The control unit,

When the surgical tool is recognized within the image information, it is determined that the gripper is holding the surgical tool, or

A surgical robot system that determines that the gripper is holding the surgical tool when the surgical tool is recognized within the image information and the distance between the gripper and the surgical tool is within a certain range.
According to paragraph 1,

The control unit,

A surgical robot system that detects the state of the end effector from the image information using a machine learning model that has previously learned a plurality of learning images representing two or more different states of the end effector.
According to paragraph 2,

The control unit,

Using a machine learning model that has previously learned a plurality of learning images for the gripper and the surgical tool, at least one of the gripper and the surgical tool is recognized from the image information, and based on the recognition result, the gripper performs the surgery. A surgical robot system that determines whether a tool is being gripped.
According to paragraph 2,

The gripper is,

It includes a pair of jaws that selectively grasp the surgical tool by opening or closing by a driving force controlled by the control unit,

The control unit,

When the gripper is in a state of holding the surgical tool, a surgical robot system that detects the rotational direction of each of the pair of jaws and adjusts the grip force of the gripper based on the detected rotational direction of the pair of jaws. .
According to clause 6,

The control unit,

Recognizing the pair of jaws from the image information to detect the rotational direction of the pair of jaws, or detecting the rotational direction of the pair of jaws based on the control value of the drive motor that generates the driving force transmitted to the gripper , surgical robotic system.
A plurality of robot arms, an instrument selectively installed on at least one of the plurality of robot arms and having an end effector at the tip that operates by being inserted into the surgical site, installed on at least one of the plurality of robot arms and inserted into the surgical site. In the surgical robot system, which includes a camera that photographs the surgical site and generates image information, and a control unit that controls the operation of the plurality of robot arms and the end effector,

Performed by the control unit,

Recognizing the state of the end effector within the image information; and

A control method for a surgical robot system comprising the step of adjusting the intensity of the driving force transmitted to the end effector according to the state of the end effector.
According to clause 8,

The step of recognizing the state of the end effector is,

Learning a machine learning model using a plurality of learning images representing two or more different states of the end effector;

A control method of a surgical robot system comprising the step of inputting the image information into the machine learning model on which learning has been completed and detecting the state of the end effector from the image information.
According to clause 9,

The end effector is,

It is composed of a gripper that selectively grips the surgical tool,

The learning step is,

Comprising the step of training the machine learning model using a plurality of learning images for a state in which the gripper grips the surgical tool and a state in which the gripper does not grip the surgical tool,

The step of detecting the state of the end effector is,

A control method of a surgical robot system comprising inputting the image information into the machine learning model on which learning has been completed and detecting from the image information whether the gripper is in a state of holding the surgical tool.
According to clause 9,

The end effector is,

It is composed of a gripper that selectively grips the surgical tool,

The learning step is,

Comprising the step of training the machine learning model using a plurality of learning images of the gripper and the surgical tool,

The step of detecting the state of the end effector is,

Inputting the image information into the machine learning model for which learning has been completed, and recognizing the gripper and the surgical tool from the image information; and

A control method of a surgical robot system comprising detecting whether the gripper is in a state of gripping the surgical tool based on the recognized distance between the gripper and the surgical tool.
According to any one of claims 10 and 11,

The gripper is,

It includes a pair of jaws that selectively grasp the surgical tool by opening or closing by a driving force controlled by the control unit,

The step of adjusting the intensity of the driving force transmitted to the end effector is:

The grip force of the gripper is adjusted depending on whether the gripper is in a state of holding the surgical tool, and when the gripper is in a state of holding the surgical tool, the rotation direction of each of the pair of jaws is detected, and the detected A method of controlling a surgical robot system, comprising the step of adjusting the grip force of the gripper based on the rotation direction of the pair of jaws.
According to clause 12,

The step of adjusting the grip force of the gripper is,

Recognizing the pair of jaws from the image information to detect the rotational direction of the pair of jaws, or detecting the rotational direction of the pair of jaws based on the control value of the drive motor that generates the driving force transmitted to the gripper A method of controlling a surgical robot system, comprising the steps:
A computer-readable recording medium on which a program for performing the method of claim 8 is recorded.