KR20230028816A - Laparoscopic camera holder Robot having adapter and remote center of motion structure - Google Patents

Abstract

The present invention relates to a laparoscope holder robot having a laparoscope adapter and a remote center of motion (RCM) structure to provide four degrees of freedom and, more specifically, to a laparoscope holder robot operated to rotate a laparoscope, which includes a camera at one end and a lens at the other end, around an RCM point. According to the present invention, the laparoscope holder robot comprises: a main body; an RCM structure in which the rear end part is coupled to a first rotation joint of the main body and the front end part is coupled to the laparoscope side on a second rotation joint; a first rotation drive unit provided in the main body to rotate the RCM structure with respect to the first rotation joint to allow the laparoscope to rotate around the RCM point; a second rotation drive unit installed on one side of the main body to allow the laparoscope to rotate on the basis of an imaginary line connecting the RCM point and the first rotation joint; a linear movement device connected to the second rotation joint to move the laparoscope in the longitudinal direction; and a laparoscope adapter device configured to attach and detach the linear movement device and the laparoscope.

Description

Laparoscopic camera holder robot having adapter and remote center of motion structure}

The present invention relates to a laparoscopic holder robot having a laparoscopic mounting adapter and an RCM structure.

Medically, surgery refers to repairing a disease by cutting, cutting, or manipulating skin, mucous membranes, or other tissues using medical machines. In particular, open surgery, which incises and opens the skin at the surgical site and treats, molds, or removes internal organs, etc., has recently been using robots due to problems such as bleeding, side effects, patient pain, and scars. Surgery is emerging as an alternative.

A surgical robot refers to a robot having a function that can substitute for a surgical operation performed by a surgeon. Compared to humans, these surgical robots have the advantage of being able to perform more accurate and precise movements and enabling remote surgery. Currently, surgical robots being developed worldwide include bone surgery robots, laparoscopic surgery robots, and stereotactic surgery robots.

A surgical robot device is generally composed of a master console and a slave robot. When an operator manipulates a control lever (for example, a handle) provided in the master console, an instrument coupled to the robot arm of the slave robot or held by the robot arm is manipulated to perform surgery.

The surgical robot is provided with a robot arm for manipulation for surgery, and an instrument is mounted on a front end of the robot arm. In this way, when an instrument is mounted on the front end of the robot arm to perform surgery, the instrument moves along with the movement of the robot arm, which means that the patient's skin is partially punctured and the instrument is inserted thereto to perform surgery. There is a risk of causing unnecessary damage to the skin. In addition, when the surgical area is wide, there is a concern that the advantage of robot surgery may be halved, such as incision of the skin as much as the path the instrument moves or perforation of the skin for each surgical area.

Therefore, the instrument mounted on the front end of the robot arm sets a virtual rotation center point at a predetermined position at the distal end and controls the robot arm so that the instrument rotates around this point. This virtual center point is referred to as a 'remote center' or ' It is called RCM (remote center of motion).

US Patent Publication US2019/0069965 A Korean Registered Patent No. 10-2206647 Korean registered patent 10-1550451 Japanese registered patent 6778769

Therefore, the present invention has been made to solve the above conventional problems, and according to an embodiment of the present invention, tilting up and down (①) based on the RCM point on the laparoscope through the RCM structure, the RCM point and the first rotation joint The purpose is to provide a laparoscopic camera holder robot with four degrees of freedom that rotates the laparoscope based on the connecting virtual axis (②), moves the laparoscope in the longitudinal direction through a linear movement device, and rotates the laparoscope based on the longitudinal axis (④). there is.

According to an embodiment of the present invention, an object of the present invention is to provide a laparoscopic camera holder robot having a laparoscopic mounting adapter so that it can be easily attached and detached from the laparoscopic mirror and the laparoscopic holder robot (linear movement device of the laparoscopic holder).

In addition, the adapter device according to an embodiment of the present invention solves the problem of different laparoscope diameters, can be mounted on a robot like a module, is coupled with the laparoscope axial rotation device, and can detach only the laparoscope itself. When the user places the laparoscope at the standard position after removing the laparoscope and performing the required operation, the position of the laparoscope at the procedure stop point is calculated from the standard position, and then the driving is controlled to move the laparoscope to the laparoscope position at the procedure stop point. Its purpose is to provide a laparoscopic holder robot having a laparoscopic mounting adapter and an RCM structure that can be automatically moved.

On the other hand, the technical problems to be achieved in the present invention are not limited to the above-mentioned technical problems, and other technical problems that are not mentioned will become clear to those skilled in the art from the description below. You will be able to understand.

An object of the present invention is to provide a laparoscopic holder robot operated to rotate a laparoscope equipped with a camera at one end and an image sensor at the other end about a remote center of motion (RCM) point, comprising: a body; an RCM structure in which a rear end is engaged on a first rotation joint of the body and a front end is engaged with the laparoscopic side on a second rotation joint; a first rotational driving unit provided in the body to rotationally drive the RCM structure based on the first rotational joint so that the laparoscope rotates around the RCM point; a second rotation drive unit installed on one side of the body and driving the laparoscope to rotate based on an imaginary line connecting the RCM point and the first rotation joint; a linear movement device connected to the second rotation joint and moving the laparoscope in a longitudinal direction; It can be achieved as a laparoscopic holder robot having a laparoscope mounting adapter and an RCM structure, characterized in that it comprises; and a laparoscopic adapter device configured to attach and detach the linear movement device and the laparoscope.

And, the RCM structure has a structure in which a first link part and a second link part are combined, and the first rotation joint is a 1-1 rotation joint, and a third spaced apart from the 1-1 rotation joint downward by a specific distance. It includes a 1-2 rotation joint, the second rotation joint includes a 2-1 rotation joint and the 2-2 rotation joint, and the first link part has one end connected to the 1-1 rotation joint. The 1-1 link connected, one end of which is connected to the other end of the 1-1 link by a first hinge, and the other end of the 1-2 link connected to the linear movement device through the 2-1 rotation joint The second link part has a link, one end of which is connected to the 2-1 link connected to the 1-2 rotation joint, and one end connected to the other end of the 2-1 link by a second hinge, and the other end The end has a 2-2 link connected to the linear movement device through a 2-2 rotation joint, and the first link part and the second link part intersect the 1-2 link and the 2-1 link. It may be characterized in that it is hinged at the point of being.

The RCM point may be a point where a virtual line connecting the 1-1 rotation point and the 1-2 rotation point intersects the laparoscope.

In addition, the laparoscopic adapter device includes a fastening means provided on one side of the upper portion and configured to be detachable from the linear moving device, and a detachable unit configured to detach the laparoscope, and the detachable unit can be replaced according to the size of the diameter of the laparoscope. It may be characterized in that it is configured to.

And the laparoscopic adapter device may be detachably installed between the linear moving device and the detachable unit to further include a laparoscopic axial rotation device for rotating the laparoscope based on a longitudinal axis.

In addition, the detachable unit may include a mounting portion having a cylindrical inner surface into which the laparoscope is inserted and having an incision formed in a longitudinal direction, and a cam clamping member for fixing the laparoscope by tightening the mounting portion by manipulation. there is.

And it may be characterized in that it comprises a controller for controlling the driving of the first rotary drive unit, the second rotation drive unit, the linear movement device, and the laparoscopic axial rotation device.

In addition, it may be characterized in that it comprises a position recognition unit for recognizing the movement position of the laparoscope in the longitudinal direction and the position of the laparoscope's rotational angle based on the longitudinal direction axis from the reference position based on the angle reference and the length reference.

And, during the procedure, when the user places the laparoscope at the reference position after the necessary operation by removing the laparoscope after the procedure is stopped, the controller calculates the position of the laparoscope at the procedure stop point from the reference position, It may be characterized in that the laparoscope is moved to the laparoscope position at the procedure interruption point by controlling driving of the linear moving device and the laparoscope axial rotation device.

According to the laparoscopic holder robot having the laparoscope mounting adapter and the RCM structure according to an embodiment of the present invention, tilting up and down (①) based on the RCM point on the laparoscope through the RCM structure, based on the virtual axis connecting the RCM point and the first rotation joint It has an effect that can be configured to have four degrees of freedom by rotating the laparoscope (②), moving the laparoscope in the longitudinal direction through a linear movement device, and rotating the laparoscope (④) based on the longitudinal axis.

According to the laparoscopic holder robot having the laparoscope mounting adapter and the RCM structure according to an embodiment of the present invention, it has the advantage of being easy to attach and detach from the laparoscope and the laparoscopic holder robot (a linear moving device for the laparoscopic holder).

According to the laparoscope mounting adapter and the laparoscope holder robot having an RCM structure according to an embodiment of the present invention, it solves the problem of having a different diameter of the laparoscope, can be mounted on the robot like a module, is combined with the laparoscope axial rotation device, and only the laparoscope itself Detachable, during the procedure, stop the procedure, remove the laparoscope, and after the necessary operation, if the user places the laparoscope at the reference position, after calculating the position of the laparoscope at the point of interruption of the procedure from the reference position, control the laparoscope drive It has the effect of automatically moving to the laparoscopic position at the point where the procedure is stopped.

On the other hand, the effects obtainable in the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below. You will be able to.

The following drawings attached to this specification illustrate preferred embodiments of the present invention, and serve to further understand the technical idea of the present invention together with the detailed description of the present invention, so the present invention is limited only to those described in the drawings. and should not be interpreted.
1 is a schematic diagram showing a driving mechanism having an RCM structure according to an embodiment of the present invention;
2 is a side view of a laparoscopic holder robot having a laparoscopic mounting adapter and an RCM structure according to an embodiment of the present invention;
3 is a side cross-sectional view of a laparoscopic adapter device attached to a laparoscope according to an embodiment of the present invention;
Figure 4a is a side cross-sectional view of a detachable unit according to an embodiment of the present invention;
Figure 4b is a front view of a detachable unit according to an embodiment of the present invention;
5 is a block diagram showing a control flow of a controller according to an embodiment of the present invention;
6 is a block diagram of a laparoscopic camera holder robot control system according to an embodiment of the present invention;
7 is a block diagram of an external control device and a controller that are communicatively connected by an external interface according to an embodiment of the present invention;
8 is a block diagram of a four-mode laparoscopic camera holder robot control system according to an embodiment of the present invention;
9 is a flowchart of a method for controlling a laparoscopic camera holder robot according to an embodiment of the present invention;
10 is an example of a screen when a grid command is input in a voice command mode according to an embodiment of the present invention and a screen when a voice command is number 4;
11 is a block diagram of a laparoscopic robot artificial intelligence surgery guide system based on image information according to an embodiment of the present invention;
12 is a block diagram of a data collection unit according to an embodiment of the present invention;
13 is a block diagram of a learning DB according to an embodiment of the present invention;
14 shows a block diagram of a guide monitoring unit according to an embodiment of the present invention.

The above objects, other objects, features and advantages of the present invention will be easily understood through the following preferred embodiments in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments described herein and may be embodied in other forms. Rather, the embodiments introduced herein are provided so that the disclosed content will be thorough and complete and the spirit of the present invention will be sufficiently conveyed to those skilled in the art.

In this specification, when an element is referred to as being on another element, it means that it may be directly formed on the other element or a third element may be interposed therebetween. Also, in the drawings, the thickness of components is exaggerated for effective description of technical content.

Embodiments described in this specification will be described with reference to cross-sectional views and/or plan views, which are ideal exemplary views of the present invention. In the drawings, the thicknesses of films and regions are exaggerated for effective explanation of technical content. Accordingly, the shape of the illustrated drawings may be modified due to manufacturing techniques and/or tolerances. Therefore, embodiments of the present invention are not limited to the specific shape shown, but also include changes in the shape generated according to the manufacturing process. For example, a region shown at right angles may be rounded or have a predetermined curvature. Accordingly, the regions illustrated in the drawings have attributes, and the shapes of the regions illustrated in the drawings are intended to illustrate a specific shape of a region of a device and are not intended to limit the scope of the invention. Although terms such as first and second are used to describe various elements in various embodiments of the present specification, these elements should not be limited by these terms. These terms are only used to distinguish one component from another. Embodiments described and illustrated herein also include complementary embodiments thereof.

Terminology used herein is for describing the embodiments and is not intended to limit the present invention. In this specification, singular forms also include plural forms unless specifically stated otherwise in a phrase. The terms 'comprises' and/or 'comprising' used in the specification do not exclude the presence or addition of one or more other elements.

In describing the specific embodiments below, several specific contents are prepared to more specifically describe the invention and aid understanding. However, readers who have knowledge in this field to the extent that they can understand the present invention can recognize that it can be used without these various specific details. In some cases, it is mentioned in advance that parts that are commonly known in describing the invention and are not greatly related to the invention are not described in order to prevent confusion for no particular reason in explaining the present invention.

Hereinafter, a laparoscopic holder robot system having a laparoscopic mounting adapter and an RCM structure according to an embodiment of the present invention will be described.

First, the configuration and function of the laparoscopic holder robot having the laparoscopic mounting adapter and the RCM structure according to an embodiment of the present invention are described focusing on the driving mechanism, and second, the control system and control method for the laparoscopic holder robot are described. Then, thirdly, the method and system for monitoring the laparoscopic surgery process based on image information will be described.

1 is a schematic diagram showing a driving mechanism having an RCM structure according to an embodiment of the present invention. And Figure 2 shows a side view of a laparoscopic holder robot having a laparoscopic mounting adapter and an RCM structure according to an embodiment of the present invention.

The laparoscopic holder robot 100 having a laparoscope mounting adapter and an RCM structure according to an embodiment of the present invention basically includes a laparoscope 1 equipped with a camera 2 at one end and a lens 3 at the other end. It is operated to rotate about a remote center of motion (RCM) point (4).

The laparoscopic holder robot 100 having a laparoscopic mounting adapter and an RCM structure according to an embodiment of the present invention generally includes a body 5, an RCM structure 10, a first rotation drive unit 20, and a second rotation drive unit 30. , Linear movement device 40, the laparoscopic adapter device 50 having a laparoscopic axial rotation device 60, and the like.

The laparoscopic holder robot 100 having the laparoscopic mounting adapter and the RCM structure according to an embodiment of the present invention has four degrees of freedom and enables the laparoscope 1 to be detached from the holder robot 100 by the laparoscopic adapter device 50. It consists of

As will be described later, the laparoscope 1 is tilted up and down based on the RCM point 4 on the laparoscope by the first rotation drive unit 20, and the RCM point 4 and the RCM point 4 by the second rotation drive unit 30. It is rotationally driven based on the imaginary axis connecting the first rotation joint 11, and the laparoscope 1 can be moved in the longitudinal direction through the linear moving device 40, and the laparoscope 1 can be moved in the longitudinal direction by the laparoscope axial rotation device 60. has four degrees of freedom to rotate about its longitudinal axis.

The RCM structure 10 has a rear end coupled on the first rotary joint 11 of the body, and a front end coupled to the laparoscope 1 side on the second rotary joint 16. The first rotation driving unit 20 is provided in the body 5 and drives the RCM structure 10 to rotate with respect to the first rotation joint 11, so that the laparoscope 1 rotates around the RCM point 4 .

More specifically, as shown in Figure 1, it can be seen that the RCM structure 10 has a structure in which the first link unit 12 and the second link unit 14 are coupled. In addition, the first rotation joint 11 is a 1-1 rotation joint 11-1, a 1-2 rotation joint 11-1 spaced apart from the 1-1 rotation joint 11-1 downward by a specific interval. 2), and the second rotation joint 16 is configured to include a 2-1 rotation joint 16-1 and a 2-2 rotation joint 16-2.

The first link unit 12 includes a 1-1 link 12-1 having one end connected to the 1-1 rotation joint 11-1, and a 1-1 link 12-1 having one end connected to the 1-1 rotation joint 11-1. ) is connected to the other end of the first hinge 13, and the other end connects the 1-2 link 12-1 connected to the linear movement device 40 through the 2-1 rotation joint 16-1. consists of including

In addition, the second link unit 14 includes a 2-1 link 14-1 having one end connected to the 1-2 rotation joint 11-2, and a 2-1 link 14-1 having one end connected to the 1-2 rotation joint 11-2. The 2-2 link 140-2 connected to the other end of 1) by the second hinge 15 and the other end connected to the linear movement device 40 through the 2-2 rotation joint 16-2 have

The first link unit 12 and the second link unit 14 are hinged at a point where the 1-2 link 12-2 and the 2-1 link 14-1 intersect.

In addition, the second rotation drive unit 30 is installed on one side of the body 5 and drives the laparoscope 1 to rotate based on a virtual line connecting the RCM point 4 and the first rotation joint 11.

And, as shown in FIGS. 1 and 2, the RCM point 4 is a virtual line connecting the 1-1 rotation point 11-1 and the 1-2 rotation point 11-2 and the laparoscope 1 It can be seen that it is located at the point where it intersects. Therefore, the laparoscope 1 can be rotated with respect to the RCM point 4 by driving the first rotary driving unit through the RCM structure 10 .

The linear movement device 40 is connected to the second rotation joint 16 and is configured to move the laparoscope 1 in the longitudinal direction. The specific configuration, means, and form of the linear movement device 40 are not limited as long as it can move the laparoscope 1 along the longitudinal direction of the laparoscope 1.

In addition, the laparoscopic adapter device is configured to attach and detach the linear movement device and the laparoscope.

In addition, Figure 3 shows a side cross-sectional view of the laparoscopic adapter device attached to the laparoscope according to an embodiment of the present invention. And Figure 4a is a cross-sectional side view of the detachable unit according to an embodiment of the present invention, Figure 4b shows a front view of the detachable unit according to an embodiment of the present invention. And Figure 5 shows a block diagram showing the control flow of the controller according to an embodiment of the present invention.

The laparoscopic adapter device 50 according to an embodiment of the present invention includes a fastening means provided on one side of the upper portion and configured to be detachable from the linear movement device 40, and a detachable unit 70 configured to attach and detach the laparoscope 1. do. This detachable unit 70 is configured to be replaceable according to the size of the diameter of the laparoscope.

In addition, in the laparoscopic adapter device 50, the laparoscopic axial rotation device 60 may be detachably installed between the linear movement device 40 and the detachable unit 70. Through the laparoscope axial rotation device 60, the laparoscope can be rotated based on the longitudinal axis.

That is, according to an embodiment of the present invention, it is possible to solve the problem of having a different diameter of the laparoscope 1 and mount it on the robot 100 like a module. Since the diameter of the laparoscope 1 varies, it can be divided into 2-3 pieces and mounted, and the external size of the adapter 50 is kept constant so that it can be combined with the robot 100. Since there are cases where axial rotation is not required, it is configured to be equipped with a modular laparoscopic axial rotation device. Since the laparoscope 1 having an inclined angle can be mounted, it is configured to align the central axis, and the motor drive 62 and the electric motor 61 are installed together as one module.

As shown in FIGS. 4A and 4B, the detachable unit 70 according to an embodiment of the present invention has a cylindrical inner surface, into which the laparoscope 1 is inserted, and a mounting portion 71 having an incision in the longitudinal direction and , It may be configured to include a cam clamping member 72 for fixing the laparoscope 1 by tightening the mounting portion 71 by manipulation.

And the controller 200 controls the driving of the first rotary drive unit 20, the second rotation drive unit 30, the linear movement unit 40, and the axial rotation unit 60 of the laparoscope to position the end of the laparoscope 10. As will be described later by adjusting, the imaging position of the laparoscopic camera 2 is adjusted.

In addition, during surgery, it is frequently removed and inserted to clean the laparoscopic lens. It should be easy to mount for this. The adapter device 50 is coupled to the laparoscope axial rotation device 60 and is configured to detach only the laparoscope 1 itself.

The position recognition unit according to an embodiment of the present invention recognizes a longitudinal movement position of the laparoscope and a rotation angle position based on a longitudinal axis of the laparoscope from a reference position based on an angle criterion and a length criterion.

Therefore, during the procedure, when the user places the laparoscope 1 at the reference position after the necessary operation by stopping the procedure and taking out the laparoscope 1, the controller 200 moves the position of the laparoscope at the point where the procedure is stopped from the reference position. After the calculation, the driving of the linear moving device 40 and the laparoscope axial rotation device 60 are controlled to move the laparoscope 1 to the laparoscope position at the procedure stop point.

For example, when the laparoscope 1 is removed and reinstalled after necessary work, the longitudinal position of the laparoscope 1 should not be changed. To this end, a reference line is made so that the light source mechanical part and the adapter device 50 already mounted on the laparoscope (1) match, and the angle part is matched, and the length of the laparoscope (1) is matched with the step part in the laparoscope to match the length. That is, after the laparoscope holder robot 100 is coupled to the laparoscope 1 and mounted in the robot system, the RCM point 4 and the state position are determined through a calibration process. After that, when it is detached, this calibration process is not needed.

Hereinafter, the configuration, function, and control method of the laparoscopic camera holder robot control system according to an embodiment of the present invention will be described.

First, FIG. 6 shows a block diagram of a laparoscopic camera holder robot control system according to an embodiment of the present invention. And Figure 7 shows a block diagram of an external control device and a controller that are communicatively connected by an external interface according to an embodiment of the present invention. 8 is a block diagram of a four-mode laparoscopic camera holder robot control system according to an embodiment of the present invention.

The laparoscopic camera holder robot control system according to an embodiment of the present invention is a system for controlling the driving of the aforementioned laparoscopic camera holder robot 100 .

As mentioned above, the controller 20 basically drives the holder robot 10 based on the control command signal, that is, the first rotation drive unit 20, the second rotation drive unit 30, and the linear movement device 40. , Controls the driving of the laparoscopic axial rotation device 60.

In addition, the controller 200 is divided into a real-time control area and a non-real-time control area, and is configured to mutually exchange data through internal communication. The real-time controller 202 of the controller 200 receives a control signal from the attitude measurement unit and the first control input means in the real-time domain.

The top priority control input means is a means for inputting a top priority control command signal to the controller 200 by a user in a real-time control area, and may be composed of a foot pedal 110 in an embodiment of the present invention. The user inputs a control signal to the controller 200 through manipulation of the foot pedal 110 .

The posture measurement unit is configured to measure the posture data of the holder robot 100 in the real-time control area and transmits the data to the controller 200. In an embodiment of the present invention, the posture measurement unit may be configured with the IMU sensor 120.

The display unit 150 is configured to display image data captured by the laparoscopic camera 2 in real time.

The external control device 210 is communicatively connected to the external interface 201 provided in the controller 200, and adjusts the position of the end of the laparoscope 1 by means of an external adjustment input to adjust the position of the image displayed on the display unit 150. configured to adjust.

The external interface 201 operates in the non-real-time control area and exchanges data with the real-time control area of the controller through internal communication.

The external adjustment input is a position command based on the displayed image, and the controller 200 controls the driving of the holder robot 100 so that the position of the image is changed based on the external adjustment input.

That is, the external control device 210 is a device for the operator (user) to move the camera holder robot 100 during surgery, and the laparoscope 1 indirectly attached to the camera holder robot 100 by controlling the position shown in the laparoscopic image. ) to control the end position to control the displayed image. The connection with the external control device 210 is connected to the camera holder robot controller 200 through the "external interface 201" module in the camera holder robot controller 200, and the external interface 201 module uses various communication methods. Support (ADS, TCP/IP, Serial, etc.). The external interface 201 module operates in a non-real-time area and exchanges data with the real-time control area of the camera holder robot controller 200 through internal communication.

In addition, the image processing device 130 is configured to receive image data captured by the camera 2 and exchange data with the controller 200 through communication in a non-real-time control area.

In addition, the image processing device 130 learns sample surgical tool images, includes a surgical tool learning DB classified by type of surgical tool, recognizes the surgical tool in the image data, identifies the location and type, and connects the controller 200 and non-real-time It is configured to exchange data in the control domain.

Further, the voice command processing device 140 receives the voice data from the microphone 6 that receives voice data from the user, recognizes voice control commands, and exchanges data with the controller through communication in the non-real-time control area.

Voice control input ( is a position command based on the display image, and the controller 200 controls the driving of the holder robot 100 so that the position of the image is changed based on the voice control input.

In addition, the voice command processing device 140 learns the characteristics of each person and includes a voice command DB classified for each voice control command, and recognizes the voice control command from the voice data so that the controller 200 and the data in the non-real-time control area It is configured to exchange

In addition, the laparoscopic camera holder robot control system according to an embodiment of the present invention, when there is an inclination angle at the rear end of the laparoscope 1, a kinematic map between the inclination angle, the screen coordinate system of the image data, and the laparoscope end coordinate system. ), it is configured to include a laparoscopic inclination angle correction unit that calculates the movement of the laparoscopic end coordinate system according to the movement of the screen coordinate system based on the screen coordinate system.

That is, in the case of the laparoscope 1 having an inclination angle, even if the laparoscope 1 moves in a straight line, the screen moves obliquely and is seen. In order to solve this problem, it is necessary to complement the laparoscopic axial rotation motion and the linear motion based on the RCM.

In an embodiment of the present invention, a device (voice command, selection button, etc.) capable of inputting an installed laparoscopic inclination angle is provided, and based on a kinematic map between the screen coordinate system and the laparoscopic end coordinate system, the movement of the laparoscopic end coordinate system according to the movement of the laparoscopic end coordinate system is calculated. Thus, the robot motion for generating the laparoscopic distal coordinate system motion is generated.

And Figure 9 shows a flow chart of a laparoscopic camera holder robot control method according to an embodiment of the present invention. 10 shows an example of a screen when a grid command is input in the voice command mode according to an embodiment of the present invention and a screen when a voice command is number 4.

Conventionally, the laparoscopic camera holder robot control system according to an embodiment of the present invention can be operated in a manual operation mode, an external operation mode, a basic operation mode, and an automatic operation mode, and the basic operation mode is a foot pedal and a voice command. As mentioned above, the control by the foot pedal is applied with the highest priority for other modes as a real-time control area.

First, after the laparoscope 1 is mounted on the holder robot 100 through the adapter device 50 (S1), the user adjusts the position of the holder robot 100 through the manual operation mode (S2, S3), Image data is displayed on the display unit 150 by the laparoscopic camera 2 . For example, the user can control the position of the robot 100 after pressing the “manual mode” button on the robot 100 or the controller 200. At this time, the manual operation mode button is lit and the button is pressed again. When pressed, the light turns off and the manual operation mode is released.

When the manual operation mode is released, the basic operation mode is executed, the posture data measured through the IMU sensor 120 in the real-time control area is input, and the user inputs a control signal through the foot pedal 110 and the controller 200 In the real-time control area, the driving of the holder robot 100 is controlled based on the foot pedal 110 input signal (S5).

And when voice data is input through the microphone 6 (S6), the voice command processing device 140 recognizes a specific voice control command from the voice data, and the controller 200 operates the holder robot 100 based on the voice control command. The driving of is controlled (S8). In this voice command mode, if there is a control input by the foot pedal 110 (S7), the control input by the foot pedal takes precedence over the voice command.

That is, the basic operation mode is a mode that basically operates when the power of the camera holder robot 100 is turned on, and the foot pedal 110 device directly connected to the real-time control area and the voice command connected to the non-real-time control area can be performed. . When both external input devices are connected, the foot pedal command connected to the real-time area takes precedence.

As mentioned above, the voice command processing device 140 receives voice data from the microphone 6 that receives voice data from the user, recognizes voice control commands, and transmits data through communication with the controller in a non-real-time control area. will exchange

The voice control input is a position command based on the display image, and the controller 200 controls the driving of the holder robot 100 so that the position of the image is changed based on the voice control input.

In addition, the voice command processing device 140 learns the characteristics of each person and includes a voice command DB classified for each voice control command, and recognizes the voice control command from the voice data so that the controller 200 and the data in the non-real-time control area It is configured to exchange

In addition, in the voice command mode, as shown in FIG. 10, the voice control command includes a grid voice command, and when the grid voice command is performed, the image data is divided into a plurality of pieces, an index is displayed in each divided area, and the user selects a specific index. In the case of voice input, the controller 200 controls the driving of the holder robot 100 so that the specific index partition area becomes the entire screen.

That is, in the voice command mode, the user issues a movement command through the wirelessly connected microphone 6, provides a learning algorithm that can learn the characteristics of each person, and easily transfers the learned result to the voice command processing device 140. be able to upload.

For example, the voice command is based on the displayed image, and may be moved by up/down, right/left, near/far, right rotation/left rotation commands of the image. In addition, it provides a voice command system definition (up, down, left, right, near, far, etc.) of fine-tuning movements according to the degree of freedom of laparoscope (1) movement and a scale calibration algorithm optimized for laparoscopic surgery. In addition, screen movement, i.e. robot movement speed, can be optimized to meet user requirements.

In addition, when the user designates an external operation mode (S9), the external control device 210 is communicatively connected to the external interface 201 provided in the controller 200, and the controller 200 operates based on an external control input by the user. is to control the driving of the laparoscopic robot 100 (S11).

This external operation mode operates in the non-real-time control area, exchanges data with the real-time control area of the controller 200 through internal communication, the external control input is a position command based on the display image, and the controller 200 controls the driving of the holder robot 100 so that the position of the image is changed based on the external adjustment input.

For example, when the user selects “external manipulation mode” in the external control device 210, control input (position, speed) of the external control device 210 is received and the robot 100 is controlled through the camera holder robot controller 200. ) to move. At this time, the external control device 210 must transmit data to the camera holder robot controller 200 according to a predetermined protocol, and related APIs are provided.

In addition, when the user designates the automatic operation mode (S11), the image processing device 130 having a surgical tool learning DB classified by type of surgical tool by learning the sample surgical tool image recognizes the surgical tool in the image data The location and type are identified and data is exchanged with the controller 200 in the non-real-time control area, and the controller 200 controls the drive 100 of the holder robot so that the recognized surgical tool is maintained within a specific area within the image data. (S14).

In addition, this automatic operation mode is executed only when the user designates the automatic operation mode, image data is input (S12), and the surgical tool is recognized by the image processing device 130 (S13).

In the laparoscopic camera holder robot control system according to an embodiment of the present invention, except for the manual mode, the driving priority of the holder robot 100 is foot pedal 110 input, voice control command, external adjustment input, and automatic operation mode in order. am.

Hereinafter, an image information-based laparoscopic robot artificial intelligence surgery guide system and a guide method will be described. These artificial intelligence surgical guides are basically operated in an automatic operation mode in the aforementioned control method.

11 is a block diagram of a laparoscopic robot artificial intelligence surgery guide system based on image information according to an embodiment of the present invention. 12 is a block diagram of a data collection unit according to an embodiment of the present invention, FIG. 13 is a block diagram of a learning DB according to an embodiment of the present invention, and FIG. 14 is a block diagram of a guide monitoring unit according to an embodiment of the present invention. it did

The image information-based laparoscopic robot artificial intelligence surgery guide system according to an embodiment of the present invention can guide surgery by monitoring the surgical process based on image data captured by the laparoscopic camera 1 of the laparoscopic camera holder robot 100. It is a system that has

As shown in FIG. 11, the image information-based laparoscopic robot artificial intelligence surgery guide system according to an embodiment of the present invention, in the aforementioned control system, data collection unit, data learning unit, learning DB, guide monitoring unit, notification It can be seen that it is configured to further include means and the like.

As shown in FIG. 12 , the data collection unit is configured to collect surgical image data 311 , sample surgical tool image 312 , sample removal target tool image 313 , and audio data 314 .

And the data learning unit learns the collected surgical image data 311, sample surgical tool image 312, sample removal target tool image 313, and audio data 314, and each of the learned data is of the learning DB 330. It is stored in the surgical image learning DB 331, the surgical tool learning DB 332, the removal target tool learning DB 333, and the voice command DB 334.

The image processing device 130 receives the current surgical image data captured by the camera and exchanges data with the controller 200 that controls the driving of the holder robot 100 through communication in a non-real-time control area.

The data learning unit is configured to learn the collected surgical image data 311, classify the surgical image data by surgery type and operator, and store the surgical learning data in the surgical image learning DB. In the surgical learning data, the surgical sequence characteristics are learned by surgery type and by operator.

In addition, the data learning unit 320 learns the sample surgical instrument images 312, classifies them by surgical instrument type, and stores them in the surgical instrument learning DB 332. The image processing device 130 is configured to exchange data with the controller 200 in a non-real-time control area by recognizing a surgical tool in the current surgical image data to determine the location and type. In addition, the surgical learning data can learn the characteristics of the position and direction of the surgical tool according to the surgical sequence.

In addition, the data learning unit 320 learns the sample removal target tool images 313, classifies them according to removal target tool types, and stores them in the removal target tool learning DB 333. The image processing device 130 recognizes the tool to be removed in the current surgical image data, identifies the location and type, and exchanges data with the controller 200 in the non-real-time control area.

In addition, the data learning unit 320 learns the characteristics of each person from the voice data, classifies them according to voice control commands, and stores them in the voice command DB 334. The voice command processing device 140 recognizes a voice control command from voice data and exchanges data with the controller 200 in a non-real-time control area.

In addition, the guide monitoring unit 350 is basically configured to generate surgical guide data by comparing the surgical learning data stored in the learning DB 330 with the current surgical image data captured by the camera, and the notification means 340 It is configured to guide the surgical guide data.

Specifically, as shown in FIG. 14, the guide monitoring unit 350 includes a search engine 351 that detects surgical learning data to be compared and analyzed based on current surgical image data, and the current surgical image data and surgery A comparison and analysis unit 352 that compares and analyzes the learning data in real time, and an event judgment that determines whether a sequence is missing or a change of the current surgical image data compared to the surgical learning data exceeds a threshold according to the comparison and analysis of the comparison and analysis unit 352. It can be seen that it is configured to include the unit 353.

In addition, the notification means 340 is configured to transmit a notification signal when an event occurs.

And when an event occurs, the controller 200 controls the driving of the holder robot 200 to capture an image of the location where the event occurred.

As mentioned above, the surgical learning data can be learned the characteristics of the position and direction of the surgical tool according to the surgical sequence. Therefore, the comparison and determination unit 352 compares and analyzes the characteristics of the position and direction of the surgical instruments according to the surgical sequence of the surgical learning data and the surgical instruments in the current surgical image data, and the event determination unit 353 compares and analyzes the surgical instruments according to the sequence It is possible to determine an event about whether there is a change in the characteristics of location and direction that exceeds a threshold value.

In addition, as mentioned above, the image processing device 130 includes the removal target tool learning DB 333 classified by removal target tool type by learning the sample removal target tool image 313, and the image processing device 130 removes the current surgical image data. Data can be exchanged with the controller 200 in the non-real-time control area by recognizing the target tool and figuring out the location and type.

Also, when the tool to be removed is recognized, the controller 200 controls driving of the holder robot 200 to capture an image of the position of the tool to be removed right before the surgery is completed.

In addition, when the tool to be removed is recognized, the controller 200 includes a removal decision unit 356 that determines whether the tool to be removed is removed, and when it is determined that the tool to be removed is not removed right before the surgery is completed, the tool to be removed is removed. The driving of the holder robot 100 is controlled to take an image.

In addition, the voice command processing device 140 is configured to receive voice data from the microphone 6 that receives voice data from the user, recognize voice control commands, and exchange data with the controller 200 through communication in a non-real-time control area. do.

The voice control input is a position command based on the display image, and the controller 200 controls the driving of the holder robot to change the position of the image based on the voice control input. As mentioned above, in the voice command DB, characteristics of each person are learned and classified according to voice control commands, and voice control commands are recognized from voice data to exchange data with a controller in a non-real-time control area.

In addition, the robot posture storage unit 357 is configured to command to store the posture of the robot during surgery, at a corresponding time point, or during a specific time range. That is, when a storage command is input by a voice command or the foot pedal 110, the robot posture at that time is stored. Therefore, the controller can control the driving of the holder robot to be converted to the stored robot posture according to the user's request.

That is, the driving is controlled so that the momentary robot posture to be memorized is stored through a voice command or the foot pedal 110, and the memorized instantaneous image is later displayed according to the user's request. Also, when a command to move to the previous location is given, the current location is automatically saved and then moved according to the command to return to the previous location.

In addition, a situation in which movement to a target point cannot be achieved due to a changed environment (movement of organs) during surgery and the current position of a surgical tool may be identified and an object that becomes an obstacle during movement may be displayed on the screen.

That is, the obstacle recognition unit according to an embodiment of the present invention may be configured to recognize a surgical tool DB and a tool that does not match the removal target tool DB as an obstacle and display it in the surgical image data when a tool that does not match the current surgical image data exists. .

In addition, the device and method described above are not limited to the configuration and method of the above-described embodiments, but all or part of each embodiment is selectively combined so that various modifications can be made. may be configured.

1:laparoscopic
2: Camera
3: Lens
4: RCM Point
5: body
6: microphone
10: RCM structure
11: 1st rotation joint
11-1: 1-1 rotation joint
11-2: 1st-2nd rotation joint
12: first link part
12-1: Link 1-1
12-2: Link 1-2
13: first hinge
14: second link part
14-1: Link 2-1
14-2: Link 2-2
15: second hinge
16: 2nd rotation joint
16-1: 2-1 rotation joint
16-2: 2-2 rotation joint
20: 1st rotation driving unit
30: 2nd rotation driving unit
40: linear movement device
50: laparoscopic adapter device
60: laparoscopic axial rotation device
61: motor
62: motor driver
63: reducer
70: detachable unit
71: mounting part
72: bearing
73: clamping member
100: Laparoscopic holder robot with laparoscopic mounting adapter and RCM structure
110: foot pedal
120: IMU sensor
130: image processing device
140: voice command processing device
150: display unit
200: controller
201: external interface
202: real-time controller
210: external control device
300: Image information-based laparoscopic robot artificial intelligence surgery guide system
310: data collection unit
311: surgical image data
312: sample surgical tool image
313: sample removal target tool image
314: voice data
320: data learning unit
330: Learning DB
331: surgical image learning DB
332: Surgical tool learning DB
333: Removal target tool learning DB
334: Voice command DB
340: notification means
350: guide monitoring unit
351: search engine
352: comparison analysis unit
353: event judgment unit
354: surgical tool recognition unit
355: removal target tool recognition unit
356: Removal determination unit
357: robot posture storage unit

Claims (9)

In the laparoscopic holder robot operated to rotate a laparoscope equipped with a camera at one end and a lens at the other end about a remote center of motion (RCM) point,
body;
an RCM structure in which a rear end is engaged on a first rotation joint of the body and a front end is engaged with the laparoscopic side on a second rotation joint;
a first rotational driving unit provided in the body to rotationally drive the RCM structure based on the first rotational joint so that the laparoscope rotates around the RCM point;
a second rotation drive unit installed on one side of the body and driving the laparoscope to rotate based on an imaginary line connecting the RCM point and the first rotation joint;
a linear movement device connected to the second rotation joint and moving the laparoscope in a longitudinal direction; and
A laparoscopic holder robot having a laparoscopic mounting adapter and an RCM structure, characterized in that it includes; a laparoscopic adapter device configured to attach and detach the linear movement device and the laparoscope.
According to claim 1,
The RCM structure has a structure in which a first link part and a second link part are combined, and the first rotation joint is a 1-1 rotation joint, and a first position spaced apart from the 1-1 rotation joint downward by a specific distance. -Including a 2-rotation joint, wherein the 2-rotation joint includes a 2-1 rotation joint and the 2-2 rotation joint,
The first link unit has one end connected to the 1-1 link connected to the 1-1 rotation joint, one end connected to the other end of the 1-1 link by a first hinge, and the other end connected to the second link. -Has a 1-2 link connected to the linear movement device through a 1-rotation joint,
The second link part has one end connected to the 2-1 link connected to the 1-2 rotation joint, one end connected to the other end of the 2-1 link by a second hinge, and the other end connected to the second link. -Has a 2-2 link connected to the linear movement device through a 2-rotation joint,
Laparoscopic holder robot having a laparoscopic mounting adapter and an RCM structure, characterized in that the first link part and the second link part are hinged at the point where the 1-2 link and the 2-1 link intersect.
According to claim 2,
The RCM point is a laparoscopic holder robot having a laparoscope mounting adapter and an RCM structure, characterized in that the virtual line connecting the 1-1 rotation point and the 1-2 rotation point and the laparoscope intersect.
According to claim 3,
The laparoscopic adapter device,
It includes a fastening means provided on one side of the upper portion and configured to be detachable from the linear movement device, and a detachable unit configured to detach the laparoscope, wherein the detachable unit is configured to be replaceable according to the diameter size of the laparoscope. Characterized in that Laparoscopic holder robot with laparoscope mounting adapter and RCM structure.
According to claim 4,
The laparoscopic adapter device further comprises a laparoscopic axial rotation device detachably installed between the linear moving device and the detachable unit to rotate the laparoscope based on a longitudinal axis, characterized in that the laparoscopic mounting adapter and RCM structure Laparoscopic holder robot having a.
According to claim 5,
The detachable unit,
A laparoscope mounting adapter and an RCM structure comprising a mounting portion having a cylindrical inner surface into which the laparoscope is inserted and a cutout formed in the longitudinal direction, and a cam clamping member for fixing the laparoscope by tightening the mounting portion by manipulation. It has a laparoscopic holder robot.
According to claim 5,
Laparoscopic holder robot having a laparoscopic mounting adapter and an RCM structure, characterized in that it comprises a controller for controlling the driving of the first rotation driving unit, the second rotation driving unit, the linear movement unit, and the laparoscope axial rotation unit.
According to claim 7,
A laparoscope holder robot having a laparoscope mounting adapter and an RCM structure, characterized in that it comprises a position recognition unit for recognizing a longitudinal movement position of the laparoscope and a rotation angle position based on a longitudinal axis of the laparoscope from a reference position based on an angle reference and a length reference.
According to claim 8,
During the procedure, when the user places the laparoscope at the reference position after the procedure is stopped and the laparoscope is taken out and the necessary operation is performed,
The controller calculates the position of the laparoscope at the procedure stopping point from the reference position, and then controls the driving of the linear moving device and the laparoscope axial rotation device to move the laparoscope to the laparoscope position at the procedure stopping point. A laparoscopic holder robot having a laparoscopic mounting adapter and an RCM structure characterized in that it moves.

Images