WO2023022258A1 - Image information-based laparoscope robot artificial intelligence surgery guide system - Google Patents

Abstract

The present invention relates to an image information-based laparoscope robot artificial intelligence surgery guide system and, more specifically, to a system for monitoring a surgical process on the basis of image data captured by a laparoscope camera of a laparoscope camera holder robot, thereby guiding the surgery. The image information-based laparoscope robot artificial intelligence surgery guide system comprises: a data collecting unit for collecting surgery image data; a surgery image learning DB for learning the collected surgery image data, classifying same with regard to each surgery type and each surgeon, and storing surgery learning data; an image processing device for receiving current surgery image data captured by the camera and exchanging data with a controller for controlling the driving of the holder robot through communication in a non-real-time control area; a guide monitoring unit for generating surgery guide data in comparison with the surgery learning data and the current surgery image data captured by the camera; and a notification means for guiding the surgery guide data.

Description

Image information-based laparoscopic robot artificial intelligence surgery guide system

The present invention relates to an image information-based laparoscopic robot artificial intelligence surgical guide system and guide method.

Medically, surgery refers to repairing a disease by cutting, cutting, or manipulating skin, mucous membranes, or other tissues using medical machines. In particular, open surgery, which incises and opens the skin at the surgical site and treats, molds, or removes internal organs, etc., has recently been using robots due to problems such as bleeding, side effects, patient pain, and scars. Surgery is emerging as an alternative.

A surgical robot refers to a robot having a function that can substitute for a surgical operation performed by a surgeon. Compared to humans, these surgical robots have the advantage of being able to perform more accurate and precise movements and enabling remote surgery. Surgical robots currently being developed worldwide include bone surgery robots, laparoscopic surgery robots, and stereotactic surgery robots.

A surgical robot device is generally composed of a master console and a slave robot. When an operator manipulates a control lever (for example, a handle) provided in the master console, an instrument coupled to the robot arm of the slave robot or held by the robot arm is manipulated to perform surgery.

The surgical robot is provided with a robot arm for manipulation for surgery, and an instrument is mounted on a front end of the robot arm. In this way, when an instrument is mounted on the front end of the robot arm to perform surgery, the instrument moves along with the movement of the robot arm, which means that the patient's skin is partially punctured and the instrument is inserted thereto to perform surgery. There is a risk of causing unnecessary damage to the skin. In addition, when the surgical area is wide, there is a concern that the advantage of robot surgery may be halved, such as incision of the skin as much as the path the instrument moves or perforation of the skin for each surgical area.

Therefore, the instrument mounted on the front end of the robot arm sets a virtual rotation center point at a predetermined position at the distal end and controls the robot arm so that the instrument rotates around this point. This virtual center point is referred to as a 'remote center' or ' It is called RCM (remote center of motion).

Therefore, the present invention has been devised to solve the above conventional problems, and according to an embodiment of the present invention, while mapping the surgical image data to be a guide and the current captured image data, there is a missing in the surgical sequence or a change of more than a threshold value. Its purpose is to provide an image information-based laparoscopic robot artificial intelligence surgery guide system that can be transmitted as a notification signal or controlled to capture an image of the location where the corresponding event exists immediately before surgery is completed.

According to an embodiment of the present invention, it is possible to memorize the location of a tool (gauze, mass, etc.) inserted into the affected area during surgery and automatically photograph the location right before the surgery is completed to determine whether the tool has been removed or not. Its purpose is to provide an image information-based laparoscopic robot artificial intelligence surgery guide system that can be controlled to photograph the location immediately before surgery is completed if it is not removed by judgment.

According to an embodiment of the present invention, it is possible to store a momentary robot posture to be memorized through a voice command or a foot pedal, and to establish a movement plan to move to show the memorized momentary image according to a user's request later. The purpose is to provide an image information-based laparoscopic robot artificial intelligence surgical guide system.

In addition, according to an embodiment of the present invention, a situation in which movement to a target point is impossible due to a changed environment (organ movement) during surgery and the current surgical tool position, etc., an object that becomes an obstacle during movement is displayed on the screen, and kinematic characteristics of the laparoscopic robot To provide an image information-based laparoscopic robot artificial intelligence surgery guide system that can display the optimal screen possible considering the There is a purpose.

On the other hand, the technical problems to be achieved in the present invention are not limited to the above-mentioned technical problems, and other technical problems that are not mentioned will become clear to those skilled in the art from the description below. You will be able to understand.

An object of the present invention is to provide a system for guiding surgery by monitoring a surgical procedure based on image data captured by a laparoscopic camera of a laparoscopic camera holder robot, comprising: a data collection unit for collecting surgical image data; a surgical image learning DB that stores the surgical learning data by learning the collected surgical image data and classifying them by surgery type and operator; an image processing device that receives current surgical image data captured by the camera and exchanges data through communication in a non-real-time control area with a controller that controls driving of the holder robot; a guide monitoring unit generating surgical guide data by comparing the surgical learning data with the current surgical image data captured by the camera; And notification means for guiding the surgical guide data; it can be achieved as an image information-based laparoscopic robot artificial intelligence surgery guide system comprising a.

In addition, the surgical learning data may be characterized in that surgical sequence characteristics are learned for each surgical type and each operator.

In addition, the guide monitoring unit, a comparison analysis unit for comparing and analyzing the current surgical image data and the surgical learning data in real time; and an event determination unit for determining an event as to whether a sequence is missing or whether a change in the current surgical image data compared to the surgical learning data exceeds a threshold value according to the comparative analysis by the comparison and analysis unit, wherein the notification means comprises: It may be characterized in that a notification signal is transmitted when the event occurs.

Further, when the event occurs, the controller may control driving of the holder robot to capture an image of a location where the event occurs.

In addition, it includes a surgical tool learning DB classified by surgical tool type by learning sample surgical tool images, and the image processing device recognizes the surgical tool in the current surgical image data and identifies the location and type to perform non-real-time control with the controller. data is exchanged in the area, and the surgical learning data learns the position and direction characteristics of surgical instruments according to the surgical sequence, and the comparison and determination unit position and direction characteristics of the surgical instruments according to the surgical sequence of the surgical learning data, The surgical tools in the current surgical image data may be compared and analyzed, and the event determination unit may determine an event as to whether there is a change of more than a threshold value in the position and direction characteristics of the surgical tool according to the sequence.

And includes a tool to be removed learning DB classified by type of tool to be removed by learning sample tool images to be removed, and the image processing device recognizes the tool to be removed in the current surgical image data to identify the location and type of the tool to be removed, and the controller and exchanges data in a non-real-time control area, and the controller, when the tool to be removed is recognized, controls driving of the holder robot to capture an image of the position of the tool to be removed right before surgery is completed. .

The controller may further include a removal decision unit for determining whether the tool to be removed is removed when the tool to be removed is recognized, and the controller controls the tool to be removed when it is determined that the tool to be removed is not removed immediately before completion of the operation. It may be characterized in that driving of the holder robot is controlled to capture an image.

And a microphone for receiving voice data from a user, and a voice command processing device for receiving the voice data, recognizing a voice control command, and exchanging data with the controller through communication in a non-real-time control area. The control input may be a position command based on a display image, and the controller may control driving of the holder robot so that the position of the image is changed based on the voice control input.

In addition, the voice command processing device includes a voice command DB that learns characteristics of each person and is classified for each voice control command, recognizes a voice control command from the voice data, and exchanges data with the controller in a non-real-time control area. that can be characterized.

And a robot posture storage unit for commanding to store the posture of the robot at a specific point in time or during a specific time range during surgery, and the controller drives the holder robot to switch to the stored robot posture according to a user's request. It can be characterized as controlling.

In addition, when a tool that does not match the surgical tool DB and the removal target tool DB exists in the current surgical image data, it may be characterized by further comprising an obstacle recognizing unit that recognizes it as an obstacle and displays it in the surgical image data.

According to the image information-based laparoscopic robot artificial intelligence surgery guide system according to an embodiment of the present invention, while mapping the surgical image data that serves as a guide and the current captured image data, if there is an omission in the surgical sequence or a change greater than the threshold value, it is sent as a notification signal. Alternatively, it has an effect of being able to control to capture an image of a location where a corresponding event exists right before surgery is completed.

According to the image information-based laparoscopic robot artificial intelligence surgery guide system according to an embodiment of the present invention, the surgery is automatically completed to determine whether the tool is removed by memorizing the position of a tool (gauze, mass, etc.) inserted into the affected area during surgery. If it is not removed by determining whether or not the technology or control to shoot the corresponding position just before the surgery is completed, it has the effect of being able to control the photographing of the corresponding position immediately before the surgery is completed.

According to the image information-based laparoscopic robot artificial intelligence surgery guide system according to an embodiment of the present invention, the momentary robot posture to be memorized through a voice command or foot pedal is stored, and the instantaneous image that is remembered according to the user's request later It has the effect of establishing a movement plan to move to show

In addition, according to the image information-based laparoscopic robot artificial intelligence surgery guide system according to an embodiment of the present invention, it is possible to grasp the situation in which movement to the target point cannot be performed due to the changed environment (organ movement) and the current surgical tool position during surgery, and obstacles when moving to the screen. This object can be displayed, the optimal screen can be displayed considering the kinematic characteristics of the laparoscopic robot, and the current position is automatically saved when a move command is given to the previous position, and then the move can be performed according to the return command to the previous position. .

On the other hand, the effects obtainable in the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below. You will be able to.

The following drawings attached to this specification illustrate preferred embodiments of the present invention, and together with the detailed description of the invention serve to further understand the technical idea of the present invention, the present invention is limited only to those described in the drawings. and should not be interpreted.

1 is a configuration diagram of a laparoscopic holder robot system having a laparoscopic mounting adapter and an RCM structure according to an embodiment of the present invention;

2 is a schematic diagram showing a driving mechanism having an RCM structure according to an embodiment of the present invention;

3 is a side view of a laparoscopic holder robot having a laparoscopic mounting adapter and an RCM structure according to an embodiment of the present invention;

Figure 4 is a side cross-sectional view of the laparoscopic adapter device attached to the laparoscope according to an embodiment of the present invention;

5 is a side cross-sectional view of a detachable unit according to an embodiment of the present invention;

6 is a front view of a detachable unit according to an embodiment of the present invention;

7 is a block diagram showing a control flow of a controller according to an embodiment of the present invention;

8 is a diagram of a laparoscopic camera holder robot control system according to an embodiment of the present invention;

9 is a block diagram of an external control device and a controller that are communicatively connected by an external interface according to an embodiment of the present invention;

10 is a block diagram of a four-mode laparoscopic camera holder robot control system according to an embodiment of the present invention;

11 is a flowchart of a method for controlling a laparoscopic camera holder robot according to an embodiment of the present invention;

12 is an example of a screen when a grid command is input in a voice command mode according to an embodiment of the present invention and a screen when a voice command is number 4;

13 is a block diagram of a laparoscopic robot artificial intelligence surgery guide system based on image information according to an embodiment of the present invention;

14 is a block diagram of a data collection unit according to an embodiment of the present invention;

15 is a block diagram of a learning DB according to an embodiment of the present invention;

16 shows a block diagram of a guide monitoring unit according to an embodiment of the present invention.

Hereinafter, a laparoscopic holder robot system having a laparoscopic mounting adapter and an RCM structure according to an embodiment of the present invention will be described. First, FIG. 1 shows a configuration diagram of a laparoscopic holder robot system having a laparoscopic mounting adapter and an RCM structure according to an embodiment of the present invention.

First, the configuration and function of the laparoscopic holder robot having the laparoscopic mounting adapter and the RCM structure according to an embodiment of the present invention are described focusing on the driving mechanism, and second, the control system and control method for the laparoscopic holder robot are described. Then, thirdly, the method and system for monitoring the laparoscopic surgery process based on image information will be described.

2 is a schematic diagram showing a driving mechanism having an RCM structure according to an embodiment of the present invention. And Figure 3 shows a side view of the laparoscopic holder robot having a laparoscopic mounting adapter and RCM structure according to an embodiment of the present invention.

Laparoscope holder robot 100 having a laparoscope mounting adapter and an RCM structure according to an embodiment of the present invention basically has a camera 2 at one end and a laparoscope 1 equipped with an image sensor 3 at the other end. rotates around a remote center of motion (RCM) point (4).

The laparoscopic holder robot 100 having a laparoscopic mounting adapter and an RCM structure according to an embodiment of the present invention generally includes a body 5, an RCM structure 10, a first rotation drive unit 20, and a second rotation drive unit 30. , Linear movement device 40, the laparoscopic adapter device 50 having a laparoscopic axial rotation device 60, and the like.

The laparoscopic holder robot 100 having the laparoscopic mounting adapter and the RCM structure according to an embodiment of the present invention has four degrees of freedom and enables the laparoscope 1 to be detached from the holder robot 100 by the laparoscopic adapter device 50. It consists of

As will be described later, the laparoscope 1 is tilted up and down based on the RCM point 4 on the laparoscope by the first rotation drive unit 20, and the RCM point 4 and the RCM point 4 by the second rotation drive unit 30. It is rotationally driven based on the imaginary axis connecting the first rotation joint 11, and the laparoscope 1 can be moved in the longitudinal direction through the linear moving device 40, and the laparoscope 1 can be moved in the longitudinal direction by the laparoscope axial rotation device 60. has four degrees of freedom to rotate about its longitudinal axis.

The RCM structure 10 has a rear end coupled on the first rotary joint 11 of the body, and a front end coupled to the laparoscope 1 side on the second rotary joint 16. The first rotation driving unit 20 is provided in the body 5 and drives the RCM structure 10 to rotate with respect to the first rotation joint 11, so that the laparoscope 1 rotates around the RCM point 4 .

More specifically, as shown in FIG. 2 , it can be seen that the RCM structure 10 has a structure in which the first link unit 12 and the second link unit 14 are combined. In addition, the first rotation joint 11 is a 1-1 rotation joint 11-1, a 1-2 rotation joint 11-1 spaced apart from the 1-1 rotation joint 11-1 downward by a specific interval. 2), and the second rotation joint 16 is configured to include a 2-1 rotation joint 16-1 and a 2-2 rotation joint 16-2.

The first link unit 12 includes a 1-1 link 12-1 having one end connected to the 1-1 rotation joint 11-1, and a 1-1 link 12-1 having one end connected to the 1-1 rotation joint 11-1. ) is connected to the other end of the first hinge 13, and the other end connects the 1-2 link 12-1 connected to the linear movement device 40 through the 2-1 rotation joint 16-1. consists of including

In addition, the second link unit 14 includes a 2-1 link 14-1 having one end connected to the 1-2 rotation joint 11-2, and a 2-1 link 14-1 having one end connected to the 1-2 rotation joint 11-2. The 2-2 link 140-2 connected to the other end of 1) by the second hinge 15 and the other end connected to the linear movement device 40 through the 2-2 rotation joint 16-2 have

The first link unit 12 and the second link unit 14 are hinged at a point where the 1-2 link 12-2 and the 2-1 link 14-1 intersect.

In addition, the second rotation drive unit 30 is installed on one side of the body 5 and drives the laparoscope 1 to rotate based on a virtual line connecting the RCM point 4 and the first rotation joint 11.

And, as shown in FIGS. 2 and 3, the RCM point 4 is a virtual line connecting the 1-1 rotation point 11-1 and the 1-2 rotation point 11-2 and the laparoscope 1 It can be seen that it is located at the point where it intersects. Therefore, the laparoscope 1 can be rotated with respect to the RCM point 4 by driving the first rotary driving unit through the RCM structure 10 .

The linear movement device 40 is connected to the second rotation joint 16 and is configured to move the laparoscope 1 in the longitudinal direction. The specific configuration, means, and form of the linear movement device 40 are not limited as long as it can move the laparoscope 1 along the longitudinal direction of the laparoscope 1.

In addition, the laparoscopic adapter device is configured to attach and detach the linear movement device and the laparoscope.

In addition, Figure 4 shows a side cross-sectional view of the laparoscopic adapter device attached to the laparoscope according to an embodiment of the present invention. 5 is a cross-sectional side view of a detachable unit according to an embodiment of the present invention, and FIG. 6 is a front view of the detachable unit according to an embodiment of the present invention. And Figure 7 shows a block diagram showing the control flow of the controller according to an embodiment of the present invention.

The laparoscopic adapter device 50 according to an embodiment of the present invention includes a fastening means provided on one side of the upper portion and configured to be detachable from the linear movement device 40, and a detachable unit 70 configured to attach and detach the laparoscope 1. do. This detachable unit 70 is configured to be replaceable according to the size of the diameter of the laparoscope.

In addition, in the laparoscopic adapter device 50, the laparoscopic axial rotation device 60 may be detachably installed between the linear movement device 40 and the detachable unit 70. Through the laparoscope axial rotation device 60, the laparoscope can be rotated based on the longitudinal axis.

That is, according to an embodiment of the present invention, it is possible to solve the problem of having a different diameter of the laparoscope 1 and mount it on the robot 100 like a module. Since the diameter of the laparoscope 1 varies, it can be divided into 2-3 pieces and mounted, and the external size of the adapter 50 is kept constant so that it can be combined with the robot 100. Since there are cases where axial rotation is not required, it is configured to be equipped with a modular laparoscopic axial rotation device. Since the laparoscope 1 having an inclined angle can be mounted, it is configured to align the central axis, and the motor drive 62 and the electric motor 61 are installed together as one module.

As shown in FIGS. 5A and 5B, the attachment/detachment unit 70 according to an embodiment of the present invention has a cylindrical inner surface, into which the laparoscope 1 is inserted, and a mounting portion 71 having an incision in the longitudinal direction and , It may be configured to include a cam clamping member 72 for fixing the laparoscope 1 by tightening the mounting portion 71 by manipulation.

And the controller 200 controls the driving of the first rotary drive unit 20, the second rotation drive unit 30, the linear movement unit 40, and the axial rotation unit 60 of the laparoscope to position the end of the laparoscope 10. As will be described later by adjusting, the imaging position of the laparoscopic camera 2 is adjusted.

In addition, during surgery, it is frequently removed and inserted to clean the laparoscopic lens. It should be easy to mount for this. The adapter device 50 is coupled to the laparoscope axial rotation device 60 and is configured to detach only the laparoscope 1 itself.

The position recognition unit according to an embodiment of the present invention recognizes a longitudinal movement position of the laparoscope and a rotation angle position based on a longitudinal axis of the laparoscope from a reference position based on an angle criterion and a length criterion.

Therefore, during the procedure, when the user places the laparoscope 1 at the reference position after the necessary operation by stopping the procedure and taking out the laparoscope 1, the controller 200 moves the position of the laparoscope at the point where the procedure is stopped from the reference position. After the calculation, the driving of the linear moving device 40 and the laparoscope axial rotation device 60 are controlled to move the laparoscope 1 to the laparoscope position at the procedure stop point.

For example, when the laparoscope 1 is removed and reinstalled after necessary work, the longitudinal position of the laparoscope 1 should not be changed. To this end, a reference line is made so that the light source mechanical part and the adapter device 50 already mounted on the laparoscope (1) match, and the angle part is matched, and the length of the laparoscope (1) is matched with the step part in the laparoscope to match the length. That is, after the laparoscope holder robot 100 is coupled to the laparoscope 1 and mounted in the robot system, the RCM point 4 and the state position are determined through a calibration process. After that, when it is detached, this calibration process is not needed.

Hereinafter, the configuration, function, and control method of the laparoscopic camera holder robot control system according to an embodiment of the present invention will be described.

First, FIG. 8 shows a block diagram of a laparoscopic camera holder robot control system according to an embodiment of the present invention. 9 is a block diagram of an external control device and a controller that are communicatively connected by an external interface according to an embodiment of the present invention. 10 also shows a block diagram of a four-mode laparoscopic camera holder robot control system according to an embodiment of the present invention.

The laparoscopic camera holder robot control system according to an embodiment of the present invention is a system for controlling the driving of the aforementioned laparoscopic camera holder robot 100 .

As mentioned above, the controller 20 basically drives the holder robot 10 based on the control command signal, that is, the first rotation drive unit 20, the second rotation drive unit 30, and the linear movement device 40. , Controls the driving of the laparoscopic axial rotation device 60.

In addition, the controller 200 is divided into a real-time control area and a non-real-time control area, and is configured to mutually exchange data through internal communication. The real-time controller 202 of the controller 200 receives a control signal from the attitude measurement unit and the first control input means in the real-time domain.

The top priority control input means is a means for inputting a top priority control command signal to the controller 200 by a user in a real-time control area, and may be composed of a foot pedal 110 in an embodiment of the present invention. The user inputs a control signal to the controller 200 through manipulation of the foot pedal 110 .

The posture measurement unit is configured to measure the posture data of the holder robot 100 in the real-time control area and transmits the data to the controller 200. In an embodiment of the present invention, the posture measurement unit may be configured with the IMU sensor 120.

The display unit 150 is configured to display image data captured by the laparoscopic camera 2 in real time.

The external control device 210 is communicatively connected to the external interface 201 provided in the controller 200, and adjusts the position of the end of the laparoscope 1 by means of an external adjustment input to adjust the position of the image displayed on the display unit 150. configured to adjust.

The external interface 201 operates in the non-real-time control area and exchanges data with the real-time control area of the controller through internal communication.

The external adjustment input is a position command based on the displayed image, and the controller 200 controls the driving of the holder robot 100 so that the position of the image is changed based on the external adjustment input.

That is, the external control device 210 is a device for the operator (user) to move the camera holder robot 100 during surgery, and the laparoscope 1 indirectly attached to the camera holder robot 100 by controlling the position shown in the laparoscopic image. ) to control the end position to control the displayed image. The connection with the external control device 210 is connected to the camera holder robot controller 200 through the "external interface 201" module in the camera holder robot controller 200, and the external interface 201 module uses various communication methods. Support (ADS, TCP/IP, Serial, etc.). The external interface 201 module operates in a non-real-time area and exchanges data with the real-time control area of the camera holder robot controller 200 through internal communication.

In addition, the image processing device 130 is configured to receive image data captured by the camera 2 and exchange data with the controller 200 through communication in a non-real-time control area.

In addition, the image processing device 130 learns sample surgical tool images, includes a surgical tool learning DB classified by type of surgical tool, recognizes the surgical tool in the image data, identifies the location and type, and connects the controller 200 and non-real-time It is configured to exchange data in the control domain.

Further, the voice command processing device 140 receives the voice data from the microphone 6 that receives voice data from the user, recognizes voice control commands, and exchanges data with the controller through communication in the non-real-time control area.

Voice control input ( is a position command based on the display image, and the controller 200 controls the driving of the holder robot 100 so that the position of the image is changed based on the voice control input.

In addition, the voice command processing device 140 learns the characteristics of each person and includes a voice command DB classified for each voice control command, and recognizes the voice control command from the voice data so that the controller 200 and the data in the non-real-time control area It is configured to exchange

In addition, the laparoscopic camera holder robot control system according to an embodiment of the present invention, when there is an inclination angle at the rear end of the laparoscope 1, a kinematic map between the inclination angle, the screen coordinate system of the image data, and the laparoscope end coordinate system. ), it is configured to include a laparoscopic inclination angle correction unit that calculates the movement of the laparoscopic end coordinate system according to the movement of the screen coordinate system based on the screen coordinate system.

That is, in the case of the laparoscope 1 having an inclination angle, even if the laparoscope 1 moves in a straight line, the screen moves obliquely and is seen. In order to solve this problem, it is necessary to complement the laparoscopic axial rotation motion and the linear motion based on the RCM.

In an embodiment of the present invention, a device (voice command, selection button, etc.) capable of inputting an installed laparoscopic inclination angle is provided, and based on a kinematic map between the screen coordinate system and the laparoscopic end coordinate system, the movement of the laparoscopic end coordinate system according to the movement of the laparoscopic end coordinate system is calculated. Thus, the robot motion for generating the laparoscopic distal coordinate system motion is generated.

And Figure 11 shows a flow chart of a laparoscopic camera holder robot control method according to an embodiment of the present invention. 12 shows an example of a screen when a grid command is input in the voice command mode according to an embodiment of the present invention and a screen when a voice command is number 4.

Conventionally, the laparoscopic camera holder robot control system according to an embodiment of the present invention can be operated in a manual operation mode, an external operation mode, a basic operation mode, and an automatic operation mode, and the basic operation mode is a foot pedal and a voice command. As mentioned above, the control by the foot pedal is applied with the highest priority for other modes as a real-time control area.

First, after the laparoscope 1 is mounted on the holder robot 100 through the adapter device 50 (S1), the user adjusts the position of the holder robot 100 through the manual operation mode (S2, S3), Image data is displayed on the display unit 150 by the laparoscopic camera 2 . For example, the user can control the position of the robot 100 after pressing the “manual mode” button on the robot 100 or the controller 200. At this time, the manual operation mode button is lit and the button is pressed again. When pressed, the light turns off and the manual operation mode is released.

When the manual operation mode is released, the basic operation mode is executed, the posture data measured through the IMU sensor 120 in the real-time control area is input, and the user inputs a control signal through the foot pedal 110 and the controller 200 In the real-time control area, the driving of the holder robot 100 is controlled based on the foot pedal 110 input signal (S5).

And when voice data is input through the microphone 6 (S6), the voice command processing device 140 recognizes a specific voice control command from the voice data, and the controller 200 operates the holder robot 100 based on the voice control command. The driving of is controlled (S8). In this voice command mode, if there is a control input by the foot pedal 110 (S7), the control input by the foot pedal takes precedence over the voice command.

That is, the basic operation mode is a mode that basically operates when the power of the camera holder robot 100 is turned on, and the foot pedal 110 device directly connected to the real-time control area and the voice command connected to the non-real-time control area can be performed. . When both external input devices are connected, the foot pedal command connected to the real-time area takes precedence.

As mentioned above, the voice command processing device 140 receives voice data from the microphone 6 that receives voice data from the user, recognizes voice control commands, and transmits data through communication with the controller in a non-real-time control area. will exchange

The voice control input is a position command based on the display image, and the controller 200 controls the driving of the holder robot 100 so that the position of the image is changed based on the voice control input.

In addition, the voice command processing device 140 learns the characteristics of each person and includes a voice command DB classified for each voice control command, and recognizes the voice control command from the voice data so that the controller 200 and the data in the non-real-time control area It is configured to exchange

In addition, in the voice command mode, as shown in FIG. 12, the voice control command includes a grid voice command, and at the time of the grid voice command, the image data is divided into a plurality of pieces, an index is displayed in each divided area, and the user selects a specific index. In the case of voice input, the controller 200 controls the driving of the holder robot 100 so that the specific index partition area becomes the entire screen.

That is, in the voice command mode, the user issues a movement command through the wirelessly connected microphone 6, provides a learning algorithm that can learn the characteristics of each person, and easily transfers the learned result to the voice command processing device 140. be able to upload.

For example, the voice command is based on the displayed image, and may be moved by up/down, right/left, near/far, right rotation/left rotation commands of the image. In addition, it provides a voice command system definition (up, down, left, right, near, far, etc.) of fine-tuning movements according to the degree of freedom of laparoscope (1) movement and a scale calibration algorithm optimized for laparoscopic surgery. In addition, screen movement, i.e. robot movement speed, can be optimized to meet user requirements.

In addition, when the user designates an external operation mode (S9), the external control device 210 is communicatively connected to the external interface 201 provided in the controller 200, and the controller 200 operates based on an external control input by the user. is to control the driving of the laparoscopic robot 100 (S11).

This external operation mode operates in the non-real-time control area, exchanges data with the real-time control area of the controller 200 through internal communication, the external control input is a position command based on the display image, and the controller 200 controls the driving of the holder robot 100 so that the position of the image is changed based on the external adjustment input.

For example, when the user selects “external manipulation mode” in the external control device 210, control input (position, speed) of the external control device 210 is received and the robot 100 is controlled through the camera holder robot controller 200. ) to move. At this time, the external control device 210 must transmit data to the camera holder robot controller 200 according to a predetermined protocol, and related APIs are provided.

In addition, when the user designates the automatic operation mode (S11), the image processing device 130 having a surgical tool learning DB classified by type of surgical tool by learning the sample surgical tool image recognizes the surgical tool in the image data The location and type are identified and data is exchanged with the controller 200 in the non-real-time control area, and the controller 200 controls the drive 100 of the holder robot so that the recognized surgical tool is maintained within a specific area within the image data. (S14).

In addition, this automatic operation mode is executed only when the user designates the automatic operation mode, image data is input (S12), and the surgical tool is recognized by the image processing device 130 (S13).

In the laparoscopic camera holder robot control system according to an embodiment of the present invention, except for the manual mode, the driving priority of the holder robot 100 is foot pedal 110 input, voice control command, external adjustment input, and automatic operation mode in order. am.

Hereinafter, an image information-based laparoscopic robot artificial intelligence surgery guide system and a guide method will be described. These artificial intelligence surgical guides are basically operated in an automatic operation mode in the aforementioned control method.

13 is a block diagram of a laparoscopic robot artificial intelligence surgery guide system based on image information according to an embodiment of the present invention. 14 is a block diagram of a data collection unit according to an embodiment of the present invention, FIG. 15 is a block diagram of a learning DB according to an embodiment of the present invention, and FIG. 16 is a block diagram of a guide monitoring unit according to an embodiment of the present invention. it did

The image information-based laparoscopic robot artificial intelligence surgery guide system according to an embodiment of the present invention can guide surgery by monitoring the surgical process based on image data captured by the laparoscopic camera 1 of the laparoscopic camera holder robot 100. It is a system that has

As shown in FIG. 13, the image information-based laparoscopic robot artificial intelligence surgery guide system according to an embodiment of the present invention, in the aforementioned control system, data collection unit, data learning unit, learning DB, guide monitoring unit, notification It can be seen that it is configured to further include means and the like.

As shown in FIG. 14 , the data collection unit is configured to collect surgical image data 311 , sample surgical tool image 312 , sample removal target tool image 313 , and audio data 314 .

And the data learning unit learns the collected surgical image data 311, sample surgical tool image 312, sample removal target tool image 313, and audio data 314, and each of the learned data is of the learning DB 330. It is stored in the surgical image learning DB 331, the surgical tool learning DB 332, the removal target tool learning DB 333, and the voice command DB 334.

The image processing device 130 receives the current surgical image data captured by the camera and exchanges data with the controller 200 that controls the driving of the holder robot 100 through communication in a non-real-time control area.

The data learning unit is configured to learn the collected surgical image data 311, classify the surgical image data by surgery type and operator, and store the surgical learning data in the surgical image learning DB. In the surgical learning data, the surgical sequence characteristics are learned by surgery type and by operator.

In addition, the data learning unit 320 learns the sample surgical instrument images 312, classifies them by surgical instrument type, and stores them in the surgical instrument learning DB 332. The image processing device 130 is configured to exchange data with the controller 200 in a non-real-time control area by recognizing a surgical tool in the current surgical image data to determine the location and type. In addition, the surgical learning data can learn the characteristics of the position and direction of the surgical tool according to the surgical sequence.

In addition, the data learning unit 320 learns the sample removal target tool images 313, classifies them according to removal target tool types, and stores them in the removal target tool learning DB 333. The image processing device 130 recognizes the tool to be removed in the current surgical image data, identifies the location and type, and exchanges data with the controller 200 in the non-real-time control area.

In addition, the data learning unit 320 learns the characteristics of each person from the voice data, classifies them according to voice control commands, and stores them in the voice command DB 334. The voice command processing device 140 recognizes a voice control command from voice data and exchanges data with the controller 200 in a non-real-time control area.

In addition, the guide monitoring unit 350 is basically configured to generate surgical guide data by comparing the surgical learning data stored in the learning DB 330 with the current surgical image data captured by the camera, and the notification means 340 It is configured to guide the surgical guide data.

Specifically, as shown in FIG. 16, the guide monitoring unit 350 includes a search engine 351 that detects surgical learning data to be compared and analyzed based on the current surgical image data, and the current surgical image data and surgery A comparison and analysis unit 352 that compares and analyzes the learning data in real time, and an event judgment that determines whether a sequence is missing or a change of the current surgical image data compared to the surgical learning data exceeds a threshold according to the comparison and analysis of the comparison and analysis unit 352. It can be seen that it is configured to include the unit 353.

In addition, the notification means 340 is configured to transmit a notification signal when an event occurs.

And when an event occurs, the controller 200 controls the driving of the holder robot 200 to capture an image of the location where the event occurred.

As mentioned above, the surgical learning data can be learned the characteristics of the position and direction of the surgical tool according to the surgical sequence. Therefore, the comparison and determination unit 352 compares and analyzes the characteristics of the position and direction of the surgical instruments according to the surgical sequence of the surgical learning data and the surgical instruments in the current surgical image data, and the event determination unit 353 compares and analyzes the surgical instruments according to the sequence It is possible to determine an event about whether there is a change in the characteristics of location and direction that exceeds a threshold value.

In addition, as mentioned above, the image processing device 130 includes the removal target tool learning DB 333 classified by removal target tool type by learning the sample removal target tool image 313, and the image processing device 130 removes the current surgical image data. Data can be exchanged with the controller 200 in the non-real-time control area by recognizing the target tool and figuring out the location and type.

Also, when the tool to be removed is recognized, the controller 200 controls driving of the holder robot 200 to capture an image of the position of the tool to be removed right before the surgery is completed.

In addition, when the tool to be removed is recognized, the controller 200 includes a removal decision unit 356 that determines whether the tool to be removed is removed, and when it is determined that the tool to be removed is not removed right before the surgery is completed, the tool to be removed is removed. The driving of the holder robot 100 is controlled to take an image.

In addition, the voice command processing device 140 is configured to receive voice data from the microphone 6 that receives voice data from the user, recognize voice control commands, and exchange data with the controller 200 through communication in a non-real-time control area. do.

The voice control input is a position command based on the display image, and the controller 200 controls the driving of the holder robot to change the position of the image based on the voice control input. As mentioned above, in the voice command DB, characteristics of each person are learned and classified according to voice control commands, and voice control commands are recognized from voice data to exchange data with a controller in a non-real-time control area.

In addition, the robot posture storage unit 357 is configured to command to store the posture of the robot during surgery, at a corresponding time point, or during a specific time range. That is, when a storage command is input by a voice command or the foot pedal 110, the robot posture at that time is stored. Therefore, the controller can control the driving of the holder robot to be converted to the stored robot posture according to the user's request.

That is, the driving is controlled so that the momentary robot posture to be memorized is stored through a voice command or the foot pedal 110, and the memorized instantaneous image is later displayed according to the user's request. Also, when a command to move to the previous location is given, the current location is automatically saved and then moved according to the command to return to the previous location.

In addition, a situation in which movement to a target point cannot be achieved due to a changed environment (movement of organs) during surgery and the current position of a surgical tool may be identified and an object that becomes an obstacle during movement may be displayed on the screen.

That is, the obstacle recognition unit according to an embodiment of the present invention may be configured to recognize a surgical tool DB and a tool that does not match the removal target tool DB as an obstacle and display it in the surgical image data when a tool that does not match the current surgical image data exists. .

Claims (11)

1. In the system for guiding surgery by monitoring the surgical process based on image data captured by the laparoscopic camera of the laparoscopic camera holder robot,

   Data collection unit for collecting surgical image data;

   a surgical image learning DB that stores the surgical learning data by learning the collected surgical image data and classifying them by surgery type and operator;

   an image processing device that receives current surgical image data captured by the camera and exchanges data through communication in a non-real-time control area with a controller that controls driving of the holder robot;

   a guide monitoring unit that generates customized surgical guide data for each type of surgery and for each operator based on the surgical learning data; and

   The controller is an image information-based laparoscopic robot artificial intelligence surgery guide system, characterized in that for controlling the driving of the holder robot based on the surgical guide data.
2. According to claim 1,

   In the surgical learning data, surgical sequence characteristics are learned by surgery type and by operator,

   An image information-based laparoscopic robot artificial intelligence surgery guide system further comprising a notification means for guiding the surgical guide data.
3. According to claim 2,

   The guide monitoring unit,

   a comparison analysis unit that compares and analyzes the current surgical image data and the surgical learning data in real time; and

   An event judgment unit for determining an event whether a sequence is missing or a change of the current surgical image data compared to the surgical learning data exceeds a threshold value according to the comparative analysis of the comparative analysis unit; and

   The notification means is an image information-based laparoscopic robot artificial intelligence surgery guide system, characterized in that for transmitting a notification signal when the event occurs.
4. According to claim 3,

   When the event occurs, the controller controls the driving of the holder robot to capture an image of the location where the event occurs. Image information-based laparoscopic robot artificial intelligence surgery guide system.
5. According to claim 4,

   It includes a surgical tool learning DB classified by type of surgical tool by learning sample surgical tool images, and the image processing device recognizes the surgical tool in the current surgical image data and identifies the location and type of the surgical tool to determine the location and type of the surgical tool and the controller and the non-real-time control area. exchange data in

   In the surgical learning data, the characteristics of the position and direction of the surgical tool according to the surgical sequence are learned,

   The comparison and determination unit compares and analyzes the position and direction characteristics of surgical instruments according to the surgical sequence of the surgical learning data and the surgical instruments in the current surgical image data, and the event determination unit compares and analyzes the characteristics of the position and direction of the surgical instruments according to the sequence as a threshold value. An image information-based laparoscopic robot artificial intelligence surgery guide system characterized in that it determines the event for the existence of abnormal changes.
6. According to claim 5,

   A tool to be removed learning DB classified by type of tool to be removed by learning sample tool to be removed images is included, and the image processing device recognizes the tool to be removed in the current surgical image data to identify the location and type of the tool to be removed, and the controller and Exchange data in the non-real-time control area,

   The controller, when the tool to be removed is recognized, controls the driving of the holder robot to image the position of the tool to be removed immediately before surgery is completed.
7. According to claim 6,

   The controller further includes a removal decision unit for determining whether the tool to be removed is removed when the tool to be removed is recognized, and the controller captures an image of the tool to be removed when it is determined that the tool to be removed is not removed immediately before completion of the operation. Image information-based laparoscopic robot artificial intelligence surgery guide system, characterized in that for controlling the driving of the holder robot so as to.
8. According to claim 7,

   A microphone for receiving voice data from a user, and a voice command processing device for receiving the voice data, recognizing a voice control command, and exchanging data with the controller through communication in a non-real-time control area;

   The voice control input is a position command based on the display image, and the controller controls the driving of the holder robot so that the position of the image is changed based on the voice control input. Intelligent surgical guide system.
9. According to claim 8,

   The voice command processing device includes a voice command DB that learns characteristics of each person and is classified for each voice control command, recognizes a voice control command from the voice data, and exchanges data with the controller in a non-real-time control area. An image information-based laparoscopic robot artificial intelligence surgical guide system.
10. According to claim 7,

    It further includes a robot posture storage unit for commanding to store the posture of the robot at a specific point in time or during a specific time range during surgery, and the controller controls driving of the holder robot to be converted to the stored robot posture according to a user's request. Image information-based laparoscopic robot artificial intelligence surgery guide system, characterized in that.
11. According to claim 10,

    An image information-based laparoscopic robot further comprising an obstacle recognizing unit that recognizes a tool that does not match the surgical tool DB and the removal target tool DB as an obstacle when it exists in the current surgical image data and displays it in the surgical image data. Artificial intelligence surgical guide system.

Images