WO2023022257A1

WO2023022257A1 - System and method for controlling laparoscope camera holder robot

Info

Publication number: WO2023022257A1
Application number: PCT/KR2021/011020
Authority: WO
Inventors: 황희선; 노경석; 김정준; 김종찬; 박지현
Original assignee: 한국로봇융합연구원
Priority date: 2021-08-19
Filing date: 2021-08-19
Publication date: 2023-02-23
Also published as: KR102522211B1; KR20230028817A

Abstract

The present invention relates to a system and a method for controlling a laparoscope camera holder robot and, more specifically, to a system for controlling a laparoscope camera holder robot for driving a laparoscope having a camera provided on one end thereof and having an image sensor provided on the other end thereof, the system comprising: a controller for controlling a driver of the holder robot on the basis of a control command signal, and exchanging data with each other through internal communication on the basis of distinction between a real-time control area and a non-real-time control area; a preferential control input means used by a user to input a preferential control command signal to the controller in the real-time control area; a posture measuring unit for measuring posture data of the holder robot in the real-time control area and transmitting same to the controller; a display unit for displaying image data captured by the laparoscope camera; and an external adjustment device connected for communication with an external interface provided in the controller so as to adjust the end position of the laparoscope by an external adjustment input, thereby adjusting the position of images displayed by the display unit.

Description

Laparoscopic camera holder robot control system and control method

The present invention relates to a laparoscopic camera holder robot control system and control method.

Medically, surgery refers to repairing a disease by cutting, cutting, or manipulating skin, mucous membranes, or other tissues using medical machines. In particular, open surgery, which incises and opens the skin at the surgical site and treats, molds, or removes internal organs, etc., has recently been using robots due to problems such as bleeding, side effects, patient pain, and scars. Surgery is emerging as an alternative.

A surgical robot refers to a robot having a function that can substitute for a surgical operation performed by a surgeon. Compared to humans, these surgical robots have the advantage of being able to perform more accurate and precise movements and enabling remote surgery. Surgical robots currently being developed worldwide include bone surgery robots, laparoscopic surgery robots, and stereotactic surgery robots.

A surgical robot device is generally composed of a master console and a slave robot. When an operator manipulates a control lever (for example, a handle) provided in the master console, an instrument coupled to the robot arm of the slave robot or held by the robot arm is manipulated to perform surgery.

The surgical robot is provided with a robot arm for manipulation for surgery, and an instrument is mounted on a front end of the robot arm. In this way, when an instrument is mounted on the front end of the robot arm to perform surgery, the instrument moves along with the movement of the robot arm, which means that the patient's skin is partially punctured and the instrument is inserted thereto to perform surgery. There is a risk of causing unnecessary damage to the skin. In addition, when the surgical area is wide, there is a concern that the advantage of robot surgery may be halved, such as incision of the skin as much as the path the instrument moves or perforation of the skin for each surgical area.

Therefore, the instrument mounted on the front end of the robot arm sets a virtual rotation center point at a predetermined position at the distal end and controls the robot arm so that the instrument rotates around this point. This virtual center point is referred to as a 'remote center' or ' It is called RCM (remote center of motion).

Therefore, the present invention has been made to solve the above conventional problems, and according to an embodiment of the present invention, manual operation mode, external operation mode, basic operation mode (foot pedal operation mode, voice command mode), automatic operation The purpose is to provide a laparoscopic camera holder robot control system and control method that can be controlled in a mode.

And according to the embodiment of the present invention, in the external manipulation mode, the operator indirectly controls the position shown in the laparoscopic image through an external control device for moving the camera holder robot during surgery, thereby indirectly adjusting the position of the tip of the laparoscope attached to the camera holder robot. It is possible to control the displayed image by controlling, the user control device is connected through the external interface of the controller, and the external interface module supports various communication methods (ADS, TCP/IP, Serial, etc.), and in the non-real-time area Its purpose is to provide a laparoscopic camera holder robot control system and control method that operate and can exchange data through real-time control area and internal communication of the camera holder robot controller.

In addition, according to an embodiment of the present invention, in the automatic operation mode, which is executed after an external image input signal is confirmed and the position of a surgical tool in the image is confirmed, the surgical tool in the image data is recognized and the position and type are identified to communicate with the controller in non-real-time. An object of the present invention is to provide a laparoscopic camera holder robot control system and control method in which a controller can automatically control a holder robot so that a recognized surgical tool is maintained within a specific region in image data by exchanging data in a control region.

According to an embodiment of the present invention, a user can receive a motion command through a wirelessly connected microphone, provide a learning algorithm capable of learning characteristics of each person, and easily upload the learned result to the voice command processing device. These voice commands are based on the displayed image and can be moved by up/down, right/left, near/far, right/left rotation commands of the image, and voice commands for fine-tuning movements according to the degree of freedom of laparoscope movement. System definition (up, down, left, right, near, far, etc.) and scale calibration algorithm optimized for laparoscopic surgery can be provided, and a grid can be displayed on the screen and the screen can be moved through assigned numbers, Its purpose is to provide a laparoscopic camera holder robot control system and control method capable of optimizing the screen movement, that is, the robot movement speed, to meet the requirements.

In addition, in the case of a laparoscope with an inclination angle, even if the laparoscope moves in a straight line, the screen may move obliquely and be seen. Its purpose is to provide a robot motion generation algorithm for generating laparoscopic distal coordinate system motion by calculating laparoscopic distal coordinate system motion according to.

On the other hand, the technical problems to be achieved in the present invention are not limited to the above-mentioned technical problems, and other technical problems that are not mentioned will become clear to those skilled in the art from the description below. You will be able to understand.

A first object of the present invention is a control system for a laparoscopic camera holder robot that drives a laparoscope equipped with a camera at one end and an image sensor at the other end, and controls the driver of the holder robot based on a control command signal. and a controller that is divided into a real-time control area and a non-real-time control area and exchanges data with each other through internal communication; a first priority control input means for inputting a first priority control command signal to the controller by a user in a real-time control area; a posture measuring unit that measures posture data of the holder robot in a real-time control area and transmits the measured posture data to the controller; a display unit for displaying image data captured by the laparoscopic camera; an external adjustment device that is connected to an external interface provided in the controller and adjusts the position of the image displayed on the display unit by adjusting the position of the end of the laparoscope by an external adjustment input; And an image processing device receiving the image data and exchanging data through communication in a non-real-time control area with the controller; it can be achieved as a laparoscopic camera holder robot control system comprising a.

The external interface operates in a non-real-time control area and exchanges data with the real-time control area of the controller through internal communication, the external adjustment input is a position command based on a display image, and the controller controls the external adjustment It may be characterized in that the driving of the holder robot is controlled so that the position of the image is changed based on an input.

In addition, the top priority control input means may be composed of a foot pedal, and the posture measuring unit may include an IMU sensor.

And a microphone for receiving voice data from a user, and a voice command processing device for receiving the voice data, recognizing a voice control command, and exchanging data with the controller through communication in a non-real-time control area. The control input may be a position command based on a display image, and the controller may control driving of the holder robot so that the position of the image is changed based on the voice control input.

In addition, the image processing device includes a surgical tool learning DB classified by surgical tool type by learning sample surgical tool images, and recognizes the surgical tool in the image data to determine the location and type in the controller and non-real-time control area. It can be characterized by exchanging data.

The voice command processing device includes a voice command DB that learns characteristics of each person and is classified for each voice control command, recognizes a voice control command from the voice data, and exchanges data with the controller in a non-real-time control area. that can be characterized.

In addition, when there is an inclination angle at the rear end of the laparoscope, based on the inclination angle and a kinematic map between the screen coordinate system of the image data and the laparoscope end coordinate system, calculating the laparoscope end coordinate system movement according to the screen coordinate system movement It may be characterized in that it further comprises; laparoscopic inclination angle correction unit.

A second object of the present invention is a control method for a laparoscopic camera holder robot using the control system according to the first object mentioned above. Adjusting the position, and displaying the image data on the display unit by the laparoscopic camera; The basic operation mode is executed, the posture data measured through the IMU sensor is input in the real-time control area, the user inputs the control signal through the foot pedal, and the controller controls the holder robot based on the foot pedal input signal in the real-time control area. controlling the driving of; When voice data is input through the microphone, the voice command processing device recognizes a specific voice control command from the voice data, and the controller controls driving of the holder robot based on the voice control command; and when a user designates an external manipulation mode, an external control device being communicatively connected to an external interface provided in the controller so that the controller controls driving of the laparoscopic robot based on an external adjustment input by the user. It can be achieved as a laparoscopic camera holder robot control method characterized in that.

The external operation mode operates in a non-real-time control area, exchanges data with the real-time control area of the controller through internal communication, the external control input is a position command based on a display image, and the controller It may be characterized in that the driving of the holder robot is controlled so that the position of the image is changed based on an external adjustment input.

In addition, when the user designates the automatic operation mode, the image processing device equipped with a surgical tool learning DB classified by type of surgical tool by learning sample surgical tool images recognizes the surgical tool in the image data and identifies the location and type of the surgical tool. data is exchanged with the controller in a non-real-time control area, and the controller controls driving of the holder robot so that the recognized surgical tool is maintained within a specific area within the image data.

The automatic operation mode may be executed when a user designates the automatic operation mode, image data is input, and a surgical tool is recognized by the image processing device.

In addition, the driving priority of the holder robot excluding the manual mode may be characterized in that the order of foot pedal input, voice control command, external control input, and automatic operation mode.

In the voice command mode, the voice control command includes a grid voice command, and when the grid voice command divides the image data into a plurality of pieces, displays an index in each divided area, and the user voices a specific index. The controller may control driving of the holder robot so that the specific index partition area becomes the entire screen.

According to the laparoscopic camera holder robot control system and control method according to an embodiment of the present invention, effects that can be controlled by manual operation mode, external operation mode, basic operation mode (foot pedal operation mode, voice command mode), and automatic operation mode have

And according to the laparoscopic camera holder robot control system and control method according to an embodiment of the present invention, in the external manipulation mode, the operator manipulates the position shown by the laparoscopic image through an external adjustment device for moving the camera holder robot during surgery. It is possible to control the displayed image by indirectly controlling the position of the end of the laparoscope attached to the camera holder robot, and the user control device is connected through the external interface of the controller, and the external interface module supports various communication methods (ADS, TCP /IP, Serial, etc.), operates in the non-real-time area and has the effect of enabling data exchange through the real-time control area of the camera holder robot controller and internal communication.

In addition, according to the laparoscopic camera holder robot control system and control method according to an embodiment of the present invention, the external image input signal is confirmed and the surgical tool in the image data is recognized in the automatic operation mode executed after the location of the surgical tool is confirmed in the image. By identifying the position and type and exchanging data with the controller in the non-real-time control area, the controller has the effect of automatically controlling the holder robot so that the recognized surgical tool is maintained within a specific area within the image data.

And according to the laparoscopic camera holder robot control system and control method according to an embodiment of the present invention, a user can receive a motion command through a wirelessly connected microphone, and a learning algorithm capable of learning characteristics for each person is provided and learned. The resulting output can be easily uploaded to the voice command processing device, and these voice commands are based on the displayed image and can be moved with commands for up/down, right/left, near/far, right/left rotation of the image, and laparoscope It can provide a voice command system definition (up, down, left, right, near, far, etc.) of fine-tuning movements according to the degree of freedom of movement and a scale calibration algorithm optimized for laparoscopic surgery, showing a grid on the screen and assigning numbers Since the screen can be moved through, it has the effect of optimizing the screen movement, that is, the robot movement speed, to meet the user's requirements.

In addition, in the case of a laparoscope with an inclination angle, even if the laparoscope moves in a straight line, the screen may move obliquely and be seen. It has an effect of providing a robot motion generation algorithm for generating laparoscopic distal coordinate system motion by calculating the laparoscopic distal coordinate system motion according to.

On the other hand, the effects obtainable in the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below. You will be able to.

The following drawings attached to this specification illustrate preferred embodiments of the present invention, and together with the detailed description of the invention serve to further understand the technical idea of the present invention, the present invention is limited only to those described in the drawings. and should not be interpreted.

1 is a configuration diagram of a laparoscopic holder robot system having a laparoscopic mounting adapter and an RCM structure according to an embodiment of the present invention;

2 is a schematic diagram showing a driving mechanism having an RCM structure according to an embodiment of the present invention;

3 is a side view of a laparoscopic holder robot having a laparoscopic mounting adapter and an RCM structure according to an embodiment of the present invention;

Figure 4 is a side cross-sectional view of the laparoscopic adapter device attached to the laparoscope according to an embodiment of the present invention;

5 is a side cross-sectional view of a detachable unit according to an embodiment of the present invention;

6 is a front view of a detachable unit according to an embodiment of the present invention;

7 is a block diagram showing a control flow of a controller according to an embodiment of the present invention;

8 is a diagram of a laparoscopic camera holder robot control system according to an embodiment of the present invention;

9 is a block diagram of an external control device and a controller that are communicatively connected by an external interface according to an embodiment of the present invention;

10 is a block diagram of a four-mode laparoscopic camera holder robot control system according to an embodiment of the present invention;

11 is a flowchart of a method for controlling a laparoscopic camera holder robot according to an embodiment of the present invention;

12 is an example of a screen when a grid command is input in a voice command mode according to an embodiment of the present invention and a screen when a voice command is number 4;

13 is a block diagram of a laparoscopic robot artificial intelligence surgery guide system based on image information according to an embodiment of the present invention;

14 is a block diagram of a data collection unit according to an embodiment of the present invention;

15 is a block diagram of a learning DB according to an embodiment of the present invention;

16 shows a block diagram of a guide monitoring unit according to an embodiment of the present invention.

Hereinafter, a laparoscopic holder robot system having a laparoscopic mounting adapter and an RCM structure according to an embodiment of the present invention will be described. First, FIG. 1 shows a configuration diagram of a laparoscopic holder robot system having a laparoscopic mounting adapter and an RCM structure according to an embodiment of the present invention.

First, the configuration and function of the laparoscopic holder robot having the laparoscopic mounting adapter and the RCM structure according to an embodiment of the present invention are described focusing on the driving mechanism, and second, the control system and control method for the laparoscopic holder robot are described. Then, thirdly, the method and system for monitoring the laparoscopic surgery process based on image information will be described.

2 is a schematic diagram showing a driving mechanism having an RCM structure according to an embodiment of the present invention. And Figure 3 shows a side view of the laparoscopic holder robot having a laparoscopic mounting adapter and RCM structure according to an embodiment of the present invention.

Laparoscope holder robot 100 having a laparoscope mounting adapter and an RCM structure according to an embodiment of the present invention basically has a camera 2 at one end and a laparoscope 1 equipped with an image sensor 3 at the other end. rotates around a remote center of motion (RCM) point (4).

The laparoscopic holder robot 100 having a laparoscopic mounting adapter and an RCM structure according to an embodiment of the present invention generally includes a body 5, an RCM structure 10, a first rotation drive unit 20, and a second rotation drive unit 30. , Linear movement device 40, the laparoscopic adapter device 50 having a laparoscopic axial rotation device 60, and the like.

The laparoscopic holder robot 100 having the laparoscopic mounting adapter and the RCM structure according to an embodiment of the present invention has four degrees of freedom and enables the laparoscope 1 to be detached from the holder robot 100 by the laparoscopic adapter device 50. It consists of

As will be described later, the laparoscope 1 is tilted up and down based on the RCM point 4 on the laparoscope by the first rotation drive unit 20, and the RCM point 4 and the RCM point 4 by the second rotation drive unit 30. It is rotationally driven based on the imaginary axis connecting the first rotation joint 11, and the laparoscope 1 can be moved in the longitudinal direction through the linear moving device 40, and the laparoscope 1 can be moved in the longitudinal direction by the laparoscope axial rotation device 60. has four degrees of freedom to rotate about its longitudinal axis.

The RCM structure 10 has a rear end coupled on the first rotary joint 11 of the body, and a front end coupled to the laparoscope 1 side on the second rotary joint 16. The first rotation driving unit 20 is provided in the body 5 and drives the RCM structure 10 to rotate with respect to the first rotation joint 11, so that the laparoscope 1 rotates around the RCM point 4 .

More specifically, as shown in FIG. 2 , it can be seen that the RCM structure 10 has a structure in which the first link unit 12 and the second link unit 14 are combined. In addition, the first rotation joint 11 is a 1-1 rotation joint 11-1, a 1-2 rotation joint 11-1 spaced apart from the 1-1 rotation joint 11-1 downward by a specific interval. 2), and the second rotation joint 16 is configured to include a 2-1 rotation joint 16-1 and a 2-2 rotation joint 16-2.

The first link unit 12 includes a 1-1 link 12-1 having one end connected to the 1-1 rotation joint 11-1, and a 1-1 link 12-1 having one end connected to the 1-1 rotation joint 11-1. ) is connected to the other end of the first hinge 13, and the other end connects the 1-2 link 12-1 connected to the linear movement device 40 through the 2-1 rotation joint 16-1. consists of including

In addition, the second link unit 14 includes a 2-1 link 14-1 having one end connected to the 1-2 rotation joint 11-2, and a 2-1 link 14-1 having one end connected to the 1-2 rotation joint 11-2. The 2-2 link 140-2 connected to the other end of 1) by the second hinge 15 and the other end connected to the linear movement device 40 through the 2-2 rotation joint 16-2 have

The first link unit 12 and the second link unit 14 are hinged at a point where the 1-2 link 12-2 and the 2-1 link 14-1 intersect.

In addition, the second rotation drive unit 30 is installed on one side of the body 5 and drives the laparoscope 1 to rotate based on a virtual line connecting the RCM point 4 and the first rotation joint 11.

And, as shown in FIGS. 2 and 3, the RCM point 4 is a virtual line connecting the 1-1 rotation point 11-1 and the 1-2 rotation point 11-2 and the laparoscope 1 It can be seen that it is located at the point where it intersects. Therefore, the laparoscope 1 can be rotated with respect to the RCM point 4 by driving the first rotary driving unit through the RCM structure 10 .

The linear movement device 40 is connected to the second rotation joint 16 and is configured to move the laparoscope 1 in the longitudinal direction. The specific configuration, means, and form of the linear movement device 40 are not limited as long as it can move the laparoscope 1 along the longitudinal direction of the laparoscope 1.

In addition, the laparoscopic adapter device is configured to attach and detach the linear movement device and the laparoscope.

In addition, Figure 4 shows a side cross-sectional view of the laparoscopic adapter device attached to the laparoscope according to an embodiment of the present invention. 5 is a cross-sectional side view of a detachable unit according to an embodiment of the present invention, and FIG. 6 is a front view of the detachable unit according to an embodiment of the present invention. And Figure 7 shows a block diagram showing the control flow of the controller according to an embodiment of the present invention.

The laparoscopic adapter device 50 according to an embodiment of the present invention includes a fastening means provided on one side of the upper portion and configured to be detachable from the linear movement device 40, and a detachable unit 70 configured to attach and detach the laparoscope 1. do. This detachable unit 70 is configured to be replaceable according to the size of the diameter of the laparoscope.

In addition, in the laparoscopic adapter device 50, the laparoscopic axial rotation device 60 may be detachably installed between the linear movement device 40 and the detachable unit 70. Through the laparoscope axial rotation device 60, the laparoscope can be rotated based on the longitudinal axis.

That is, according to an embodiment of the present invention, it is possible to solve the problem of having a different diameter of the laparoscope 1 and mount it on the robot 100 like a module. Since the diameter of the laparoscope 1 varies, it can be divided into 2-3 pieces and mounted, and the external size of the adapter 50 is kept constant so that it can be combined with the robot 100. Since there are cases where axial rotation is not required, it is configured to be equipped with a modular laparoscopic axial rotation device. Since the laparoscope 1 having an inclined angle can be mounted, it is configured to align the central axis, and the motor drive 62 and the electric motor 61 are installed together as one module.

As shown in FIGS. 5A and 5B, the attachment/detachment unit 70 according to an embodiment of the present invention has a cylindrical inner surface, into which the laparoscope 1 is inserted, and a mounting portion 71 having an incision in the longitudinal direction and , It may be configured to include a cam clamping member 72 for fixing the laparoscope 1 by tightening the mounting portion 71 by manipulation.

And the controller 200 controls the driving of the first rotary drive unit 20, the second rotation drive unit 30, the linear movement unit 40, and the axial rotation unit 60 of the laparoscope to position the end of the laparoscope 10. As will be described later by adjusting, the imaging position of the laparoscopic camera 2 is adjusted.

In addition, during surgery, it is frequently removed and inserted to clean the laparoscopic lens. It should be easy to mount for this. The adapter device 50 is coupled to the laparoscope axial rotation device 60 and is configured to detach only the laparoscope 1 itself.

The position recognition unit according to an embodiment of the present invention recognizes a longitudinal movement position of the laparoscope and a rotation angle position based on a longitudinal axis of the laparoscope from a reference position based on an angle criterion and a length criterion.

Therefore, during the procedure, when the user places the laparoscope 1 at the reference position after the necessary operation by stopping the procedure and taking out the laparoscope 1, the controller 200 moves the position of the laparoscope at the point where the procedure is stopped from the reference position. After the calculation, the driving of the linear moving device 40 and the laparoscope axial rotation device 60 are controlled to move the laparoscope 1 to the laparoscope position at the procedure stop point.

For example, when the laparoscope 1 is removed and reinstalled after necessary work, the longitudinal position of the laparoscope 1 should not be changed. To this end, a reference line is made so that the light source mechanical part and the adapter device 50 already mounted on the laparoscope (1) match, and the angle part is matched, and the length of the laparoscope (1) is matched with the step part in the laparoscope to match the length. That is, after the laparoscope holder robot 100 is coupled to the laparoscope 1 and mounted in the robot system, the RCM point 4 and the state position are determined through a calibration process. After that, when it is detached, this calibration process is not needed.

Hereinafter, the configuration, function, and control method of the laparoscopic camera holder robot control system according to an embodiment of the present invention will be described.

First, FIG. 8 shows a block diagram of a laparoscopic camera holder robot control system according to an embodiment of the present invention. 9 is a block diagram of an external control device and a controller that are communicatively connected by an external interface according to an embodiment of the present invention. 10 also shows a block diagram of a four-mode laparoscopic camera holder robot control system according to an embodiment of the present invention.

The laparoscopic camera holder robot control system according to an embodiment of the present invention is a system for controlling the driving of the aforementioned laparoscopic camera holder robot 100 .

As mentioned above, the controller 20 basically drives the holder robot 10 based on the control command signal, that is, the first rotation drive unit 20, the second rotation drive unit 30, and the linear movement device 40. , Controls the driving of the laparoscopic axial rotation device 60.

In addition, the controller 200 is divided into a real-time control area and a non-real-time control area, and is configured to mutually exchange data through internal communication. The real-time controller 202 of the controller 200 receives a control signal from the attitude measurement unit and the first control input means in the real-time domain.

The top priority control input means is a means for inputting a top priority control command signal to the controller 200 by a user in a real-time control area, and may be composed of a foot pedal 110 in an embodiment of the present invention. The user inputs a control signal to the controller 200 through manipulation of the foot pedal 110 .

The posture measurement unit is configured to measure the posture data of the holder robot 100 in the real-time control area and transmits the data to the controller 200. In an embodiment of the present invention, the posture measurement unit may be configured with the IMU sensor 120.

The display unit 150 is configured to display image data captured by the laparoscopic camera 2 in real time.

The external control device 210 is communicatively connected to the external interface 201 provided in the controller 200, and adjusts the position of the end of the laparoscope 1 by means of an external adjustment input to adjust the position of the image displayed on the display unit 150. configured to adjust.

The external interface 201 operates in the non-real-time control area and exchanges data with the real-time control area of the controller through internal communication.

The external adjustment input is a position command based on the displayed image, and the controller 200 controls the driving of the holder robot 100 so that the position of the image is changed based on the external adjustment input.

That is, the external control device 210 is a device for the operator (user) to move the camera holder robot 100 during surgery, and the laparoscope 1 indirectly attached to the camera holder robot 100 by controlling the position shown in the laparoscopic image. ) to control the end position to control the displayed image. The connection with the external control device 210 is connected to the camera holder robot controller 200 through the "external interface 201" module in the camera holder robot controller 200, and the external interface 201 module uses various communication methods. Support (ADS, TCP/IP, Serial, etc.). The external interface 201 module operates in a non-real-time area and exchanges data with the real-time control area of the camera holder robot controller 200 through internal communication.

In addition, the image processing device 130 is configured to receive image data captured by the camera 2 and exchange data with the controller 200 through communication in a non-real-time control area.

In addition, the image processing device 130 learns sample surgical tool images, includes a surgical tool learning DB classified by type of surgical tool, recognizes the surgical tool in the image data, identifies the location and type, and connects the controller 200 and non-real-time It is configured to exchange data in the control domain.

Further, the voice command processing device 140 receives the voice data from the microphone 6 that receives voice data from the user, recognizes voice control commands, and exchanges data with the controller through communication in the non-real-time control area.

Voice control input ( is a position command based on the display image, and the controller 200 controls the driving of the holder robot 100 so that the position of the image is changed based on the voice control input.

In addition, the voice command processing device 140 learns the characteristics of each person and includes a voice command DB classified for each voice control command, and recognizes the voice control command from the voice data so that the controller 200 and the data in the non-real-time control area It is configured to exchange

In addition, the laparoscopic camera holder robot control system according to an embodiment of the present invention, when there is an inclination angle at the rear end of the laparoscope 1, a kinematic map between the inclination angle, the screen coordinate system of the image data, and the laparoscope end coordinate system. ), it is configured to include a laparoscopic inclination angle correction unit that calculates the movement of the laparoscopic end coordinate system according to the movement of the screen coordinate system based on the screen coordinate system.

That is, in the case of the laparoscope 1 having an inclination angle, even if the laparoscope 1 moves in a straight line, the screen moves obliquely and is seen. In order to solve this problem, it is necessary to complement the laparoscopic axial rotation motion and the linear motion based on the RCM.

In an embodiment of the present invention, a device (voice command, selection button, etc.) capable of inputting an installed laparoscopic inclination angle is provided, and based on a kinematic map between the screen coordinate system and the laparoscopic end coordinate system, the motion of the laparoscopic end coordinate system is calculated according to the movement of the laparoscopic end coordinate system. Thus, the robot motion for generating the laparoscopic distal coordinate system motion is generated.

And Figure 11 shows a flow chart of a laparoscopic camera holder robot control method according to an embodiment of the present invention. 12 shows an example of a screen when a grid command is input in the voice command mode according to an embodiment of the present invention and a screen when a voice command is number 4.

Conventionally, the laparoscopic camera holder robot control system according to an embodiment of the present invention can be operated in a manual operation mode, an external operation mode, a basic operation mode, and an automatic operation mode, and the basic operation mode is a foot pedal and a voice command. As mentioned above, the control by the foot pedal is applied with the highest priority for other modes as a real-time control area.

First, after the laparoscope 1 is mounted on the holder robot 100 through the adapter device 50 (S1), the user adjusts the position of the holder robot 100 through the manual operation mode (S2, S3), Image data is displayed on the display unit 150 by the laparoscopic camera 2 . For example, the user can control the position of the robot 100 after pressing the “manual mode” button on the robot 100 or the controller 200. At this time, the manual operation mode button is lit and the button is pressed again. When pressed, the light turns off and the manual operation mode is released.

When the manual operation mode is released, the basic operation mode is executed, the posture data measured through the IMU sensor 120 in the real-time control area is input, and the user inputs a control signal through the foot pedal 110 and the controller 200 In the real-time control area, the driving of the holder robot 100 is controlled based on the foot pedal 110 input signal (S5).

And when voice data is input through the microphone 6 (S6), the voice command processing device 140 recognizes a specific voice control command from the voice data, and the controller 200 operates the holder robot 100 based on the voice control command. The driving of is controlled (S8). In this voice command mode, if there is a control input by the foot pedal 110 (S7), the control input by the foot pedal takes precedence over the voice command.

That is, the basic operation mode is a mode that basically operates when the power of the camera holder robot 100 is turned on, and the foot pedal 110 device directly connected to the real-time control area and the voice command connected to the non-real-time control area can be performed. . When both external input devices are connected, the foot pedal command connected to the real-time area takes precedence.

As mentioned above, the voice command processing device 140 receives voice data from the microphone 6 that receives voice data from the user, recognizes voice control commands, and transmits data through communication with the controller in a non-real-time control area. will exchange

The voice control input is a position command based on the display image, and the controller 200 controls the driving of the holder robot 100 so that the position of the image is changed based on the voice control input.

In addition, in the voice command mode, as shown in FIG. 12, the voice control command includes a grid voice command, and at the time of the grid voice command, the image data is divided into a plurality of pieces, an index is displayed in each divided area, and the user selects a specific index. In the case of voice input, the controller 200 controls the driving of the holder robot 100 so that the specific index partition area becomes the entire screen.

That is, in the voice command mode, the user issues a movement command through the wirelessly connected microphone 6, provides a learning algorithm that can learn the characteristics of each person, and easily transfers the learned result to the voice command processing device 140. be able to upload.

For example, the voice command is based on the displayed image, and may be moved by up/down, right/left, near/far, right rotation/left rotation commands of the image. In addition, it provides a voice command system definition (up, down, left, right, near, far, etc.) of fine-tuning movements according to the degree of freedom of laparoscope (1) movement and a scale calibration algorithm optimized for laparoscopic surgery. In addition, screen movement, i.e. robot movement speed, can be optimized to meet user requirements.

In addition, when the user designates an external operation mode (S9), the external control device 210 is communicatively connected to the external interface 201 provided in the controller 200, and the controller 200 operates based on an external control input by the user. is to control the driving of the laparoscopic robot 100 (S11).

This external operation mode operates in the non-real-time control area, exchanges data with the real-time control area of the controller 200 through internal communication, the external control input is a position command based on the display image, and the controller 200 controls the driving of the holder robot 100 so that the position of the image is changed based on the external adjustment input.

For example, when the user selects “external manipulation mode” in the external control device 210, control input (position, speed) of the external control device 210 is received and the robot 100 is controlled through the camera holder robot controller 200. ) to move. At this time, the external control device 210 must transmit data to the camera holder robot controller 200 according to a predetermined protocol, and related APIs are provided.

In addition, when the user designates the automatic operation mode (S11), the image processing device 130 having a surgical tool learning DB classified by type of surgical tool by learning the sample surgical tool image recognizes the surgical tool in the image data The location and type are identified and data is exchanged with the controller 200 in the non-real-time control area, and the controller 200 controls the drive 100 of the holder robot so that the recognized surgical tool is maintained within a specific area within the image data. (S14).

In addition, this automatic operation mode is executed only when the user designates the automatic operation mode, image data is input (S12), and the surgical tool is recognized by the image processing device 130 (S13).

In the laparoscopic camera holder robot control system according to an embodiment of the present invention, except for the manual mode, the driving priority of the holder robot 100 is foot pedal 110 input, voice control command, external adjustment input, and automatic operation mode in order. am.

Hereinafter, an image information-based laparoscopic robot artificial intelligence surgery guide system and a guide method will be described. These artificial intelligence surgical guides are basically operated in an automatic operation mode in the aforementioned control method.

13 is a block diagram of a laparoscopic robot artificial intelligence surgery guide system based on image information according to an embodiment of the present invention. 14 is a block diagram of a data collection unit according to an embodiment of the present invention, FIG. 15 is a block diagram of a learning DB according to an embodiment of the present invention, and FIG. 16 is a block diagram of a guide monitoring unit according to an embodiment of the present invention. it did

The image information-based laparoscopic robot artificial intelligence surgery guide system according to an embodiment of the present invention can guide surgery by monitoring the surgical process based on image data captured by the laparoscopic camera 1 of the laparoscopic camera holder robot 100. It is a system that has

As shown in FIG. 13, the image information-based laparoscopic robot artificial intelligence surgery guide system according to an embodiment of the present invention, in the aforementioned control system, data collection unit, data learning unit, learning DB, guide monitoring unit, notification It can be seen that it is configured to further include means and the like.

As shown in FIG. 14 , the data collection unit is configured to collect surgical image data 311 , sample surgical tool image 312 , sample removal target tool image 313 , and audio data 314 .

And the data learning unit learns the collected surgical image data 311, sample surgical tool image 312, sample removal target tool image 313, and audio data 314, and each of the learned data is of the learning DB 330. It is stored in the surgical image learning DB 331, the surgical tool learning DB 332, the removal target tool learning DB 333, and the voice command DB 334.

The image processing device 130 receives the current surgical image data captured by the camera and exchanges data with the controller 200 that controls the driving of the holder robot 100 through communication in a non-real-time control area.

The data learning unit is configured to learn the collected surgical image data 311, classify the surgical image data by surgery type and operator, and store the surgical learning data in the surgical image learning DB. In the surgical learning data, the surgical sequence characteristics are learned by surgery type and by operator.

In addition, the data learning unit 320 learns the sample surgical instrument images 312, classifies them by surgical instrument type, and stores them in the surgical instrument learning DB 332. The image processing device 130 is configured to exchange data with the controller 200 in a non-real-time control area by recognizing a surgical tool in the current surgical image data to determine the location and type. In addition, the surgical learning data can learn the characteristics of the position and direction of the surgical tool according to the surgical sequence.

In addition, the data learning unit 320 learns the sample removal target tool images 313, classifies them according to removal target tool types, and stores them in the removal target tool learning DB 333. The image processing device 130 recognizes the tool to be removed in the current surgical image data, identifies the location and type, and exchanges data with the controller 200 in the non-real-time control area.

In addition, the data learning unit 320 learns the characteristics of each person from the voice data, classifies them according to voice control commands, and stores them in the voice command DB 334. The voice command processing device 140 recognizes a voice control command from voice data and exchanges data with the controller 200 in a non-real-time control area.

In addition, the guide monitoring unit 350 is basically configured to generate surgical guide data by comparing the surgical learning data stored in the learning DB 330 with the current surgical image data captured by the camera. It is configured to guide the surgical guide data.

Specifically, as shown in FIG. 16, the guide monitoring unit 350 includes a search engine 351 that detects surgical learning data to be compared and analyzed based on the current surgical image data, and the current surgical image data and surgery A comparison and analysis unit 352 that compares and analyzes the learning data in real time, and an event judgment that determines whether a sequence is missing or a change of the current surgical image data compared to the surgical learning data exceeds a threshold according to the comparison and analysis of the comparison and analysis unit 352. It can be seen that it is configured to include the unit 353.

In addition, the notification means 340 is configured to transmit a notification signal when an event occurs.

And when an event occurs, the controller 200 controls the driving of the holder robot 200 to capture an image of the location where the event occurred.

As mentioned above, the surgical learning data can be learned the characteristics of the position and direction of the surgical tool according to the surgical sequence. Therefore, the comparison and determination unit 352 compares and analyzes the characteristics of the position and direction of the surgical instruments according to the surgical sequence of the surgical learning data and the surgical instruments in the current surgical image data, and the event determination unit 353 compares and analyzes the surgical instruments according to the sequence It is possible to determine an event about whether there is a change in the characteristics of location and direction that exceeds a threshold value.

In addition, as mentioned above, the image processing device 130 includes the removal target tool learning DB 333 classified by removal target tool type by learning the sample removal target tool image 313, and the image processing device 130 removes the current surgical image data. Data can be exchanged with the controller 200 in the non-real-time control area by recognizing the target tool and figuring out the location and type.

Also, when the tool to be removed is recognized, the controller 200 controls driving of the holder robot 200 to capture an image of the position of the tool to be removed right before the surgery is completed.

In addition, when the tool to be removed is recognized, the controller 200 includes a removal decision unit 356 that determines whether the tool to be removed is removed, and when it is determined that the tool to be removed is not removed right before the surgery is completed, the tool to be removed is removed. The driving of the holder robot 100 is controlled to take an image.

In addition, the voice command processing device 140 is configured to receive voice data from the microphone 6 that receives voice data from the user, recognize voice control commands, and exchange data with the controller 200 through communication in a non-real-time control area. do.

The voice control input is a position command based on the display image, and the controller 200 controls the driving of the holder robot to change the position of the image based on the voice control input. As mentioned above, in the voice command DB, characteristics of each person are learned and classified according to voice control commands, and voice control commands are recognized from voice data to exchange data with a controller in a non-real-time control area.

In addition, the robot posture storage unit 357 is configured to command to store the posture of the robot during surgery, at a corresponding time point, or during a specific time range. That is, when a storage command is input by a voice command or the foot pedal 110, the robot posture at that time is stored. Therefore, the controller can control the driving of the holder robot to be converted to the stored robot posture according to the user's request.

That is, the driving is controlled so that the momentary robot posture to be memorized is stored through a voice command or the foot pedal 110, and the memorized instantaneous image is later displayed according to the user's request. Also, when a command to move to the previous location is given, the current location is automatically saved and then moved according to the command to return to the previous location.

In addition, a situation in which movement to a target point cannot be achieved due to a changed environment (movement of organs) during surgery and the current position of a surgical tool may be identified and an object that becomes an obstacle during movement may be displayed on the screen.

That is, the obstacle recognition unit according to an embodiment of the present invention may be configured to recognize a surgical tool DB and a tool that does not match the removal target tool DB as an obstacle and display it in the surgical image data when a tool that does not match the current surgical image data exists. .

Claims

In order to control the laparoscopic camera holder robot driving the laparoscope equipped with a camera at one end and an image sensor at the other end, a controller for controlling the driver of the holder robot based on a control command signal, and a user in a real-time control area A first-priority control input means for inputting a top-priority control command signal to the controller, a display unit for displaying image data captured by the laparoscopic camera, and receiving the image data to transmit data through communication in a non-real-time control area with the controller As a control system having an image processing device that exchanges,

An external adjustment device that is connected to an external interface provided in the controller and adjusts the position of the image displayed on the display unit by adjusting the position of the tip of the laparoscope by an external adjustment input;

The controller is divided into a real-time control area and a non-real-time control area and mutually exchanges data through internal communication.

The external adjustment input is a position command based on the display image, and the controller controls driving of the holder robot so that the position of the image is changed based on the external adjustment input. Laparoscopic camera holder robot control system .
According to claim 1,

A posture measurement unit for measuring posture data of the holder robot in a real-time control area and transmitting the data to the controller;

The external interface operates in a non-real-time control area and exchanges data with the real-time control area of the controller through internal communication;

The controller calculates the movement of the laparoscopic end coordinate system according to the movement of the screen coordinate system based on a kinematic map between the screen coordinate system and the laparoscopic end coordinate system of the image data. Control system for the laparoscopic camera holder robot.
According to claim 2,

Laparoscopic camera holder robot control system, characterized in that the first priority control input means is composed of a foot pedal, and the posture measurement unit includes an IMU sensor.
According to claim 2,

A microphone for receiving voice data from a user, and a voice command processing device for receiving the voice data, recognizing a voice control command, and exchanging data with the controller through communication in a non-real-time control area;

Laparoscopic camera holder robot control system, characterized in that the voice control input is a position command based on the display image, and the controller controls the driving of the holder robot so that the position of the image is changed based on the voice control input .
According to claim 4,

The image processing device includes a surgical tool learning DB classified by type of surgical tool by learning sample surgical tool images, and recognizes the surgical tool in the image data to determine the location and type of data in the controller and non-real-time control area. Laparoscopic camera holder robot control system, characterized in that for exchanging.
According to claim 5,

The voice command processing device includes a voice command DB that learns characteristics of each person and is classified for each voice control command, recognizes a voice control command from the voice data, and exchanges data with the controller in a non-real-time control area. Characterized by a laparoscopic camera holder robot control system.
According to claim 6,

If there is an inclination angle at the rear end of the laparoscope,

Further comprising a laparoscopic inclination angle correction unit for calculating the movement of the laparoscopic end coordinate system according to the movement of the screen coordinate system based on the inclination angle and a kinematic map between the screen coordinate system and the laparoscopic end coordinate system of the image data. Laparoscopic camera holder robot control system.
In the control method of the laparoscopic camera holder robot using the control system according to any one of claims 1 to 7,

After mounting the laparoscope on the holder robot, the user adjusts the position of the holder robot through a manual operation mode, and the image data is displayed on the display unit by the laparoscopic camera;

The basic operation mode is executed, the posture data measured through the IMU sensor is input in the real-time control area, the user inputs the control signal through the foot pedal, and the controller controls the holder robot based on the foot pedal input signal in the real-time control area. Controlling the driving of;

When voice data is input through the microphone, the voice command processing device recognizes a specific voice control command from the voice data, and the controller controls driving of the holder robot based on the voice control command; and

When the user designates an external manipulation mode, an external control device is communicatively connected to an external interface provided in the controller, and the controller controls driving of the laparoscopic robot based on an external adjustment input by the user. Laparoscopic camera holder robot control method characterized by.
According to claim 8,

The external operation mode operates in a non-real-time control area, exchanges data with the real-time control area of the controller through internal communication, the external control input is a position command based on a display image, and the controller Laparoscopic camera holder robot control method characterized in that for controlling the driving of the holder robot so that the position of the image is changed based on the adjustment input.
According to claim 9,

If the user specifies the automatic operation mode,

An image processing device equipped with a surgical tool learning DB classified by surgical tool type by learning sample surgical tool images recognizes the surgical tool in the image data, identifies the location and type, and converts the data in the controller and non-real-time control area. and the controller controls driving of the holder robot so that the recognized surgical tool is maintained within a specific region within the image data.
According to claim 10,

The automatic operation mode is executed when a user designates the automatic operation mode, image data is input, and a surgical tool is recognized by the image processing device.
According to claim 10,

Laparoscopic camera holder robot control method, characterized in that the driving priority of the holder robot excluding the manual mode is in the order of foot pedal input, voice control command, external adjustment input, and automatic operation mode.
According to claim 11,

In the voice command mode,

The voice control command includes a grid voice command, and upon the grid voice command, the image data is divided into a plurality of pieces and an index is displayed in each divided area, and when a user voices a specific index, the entire divided area at the specific index Laparoscopic camera holder robot control method, characterized in that the controller controls the driving of the holder robot so that the screen.