KR20240049859A - Hysteresis compensation apparatus of flexible tube - Google Patents

Abstract

The present invention relates to a hysteresis compensation control device and method for a flexible tube, and more particularly, to a hysteresis compensation control device for a flexible tube for compensating and controlling the hysteresis of a surgical tool disposed in a channel of an overtube. For this purpose, an input unit receives image information and wire tension information of the flexible surgical tool required for mode switching, determines the current state of the surgical tool based on the image information and tension information of the flexible surgical tool, and determines the current state of the surgical tool. A mode switching determination unit that generates a control mode change signal for control mode switching, a flexible tube based on image information in flexible tube control based on tension information according to the control mode change signal of the mode switching determination unit. A mode switching unit that switches the control mode to control or switches the control mode from flexible tube control based on image information to flexible tube control based on tension information, based on the tension error value calculated by the learning model. A hysteresis compensation control device for a flexible tube is disclosed, which includes a compensation control unit that compensates and controls a flexible tube having hysteresis characteristics.

Description

Hysteresis compensation apparatus of flexible tube}

The present invention relates to a hysteresis compensation control device and method for a flexible tube, and more particularly, to a hysteresis compensation control device for a flexible tube for compensating and controlling the hysteresis of a surgical tool disposed in a channel of an overtube.

As shown in Figure 1, the flexible overtube 10 includes a plurality of flexible surgical tools 21 and 22. A plurality of flexible surgical tools 21 and 22 are inserted and arranged into each channel 11 and 12 provided in the body of the overtube. As an example, the position and direction of the first surgical tool 21 can be controlled by being pulled by first and second traction wires (not shown).

The flexible surgical tool has a hysteresis section as shown in FIG. 2. In other words, even if the same traction force is given, a hysteresis section occurs depending on the shape of the flexible overtube or surgical tool. Control of the overtube or surgical tool in this hysteresis occurrence section has the disadvantage of not being precise.

US 2019/0290109 KR 10-2340169

Accordingly, the present invention was created to solve the problems described above, and its purpose is to provide an invention that can precisely control a flexible surgical tool even within a hysteresis section.

However, the objects of the present invention are not limited to the objects mentioned above, and other objects not mentioned will be clearly understood by those skilled in the art from the description below.

The purpose of the present invention described above is to determine the current state of the surgical tool based on the input unit that receives image information and tension information of the flexible surgical tool required for mode switching, and the image information of the surgical tool, and to determine the current state of the surgical tool according to the current state of the surgical tool. A mode switching decision unit that generates a control mode change signal, a mode that switches the control mode from flexible overtube control based on image information to flexible overtube control based on tension information according to the control mode change signal from the mode switching decision unit. This can be achieved by providing a hysteresis compensation control device for a flexible overtube, which includes a switching unit and a compensation control unit that compensates and controls a flexible surgical tool with hysteresis characteristics based on the tension error value calculated by the learning model. .

In addition, the image information that determines the current state of the surgical tool is image information taken of the tip area of the surgical tool that is inserted into the channel of the body of the flexible overtube and pulled by the traction wire.

In addition, the input unit is inserted into the channel of the body of the overtube and includes a surgical tool image capture unit for photographing the surgical tool, a tension measurement unit for measuring the traction force of the traction wire of the surgical tool, and a surgical tool image capture unit photographed by the surgical tool image capture unit. The current state of the surgical tool is determined by transmitting image information of the tip area to the mode switching determination unit.

Additionally, the mode switching determination unit determines whether control is possible in the flexible overtube control mode based on image information based on image information of the tip area of the surgical tool.

In addition, the image information learning unit learns according to a preset learning model based on the image information of the surgical tool and calculates the image error value through learning, and learns according to a preset learning model based on the tension information of the surgical tool, It further includes a tension information learning unit that calculates the tension error value through learning.

Meanwhile, the purpose of the present invention is to generate a first control mode change signal by determining the current state of the surgical tool based on the image information of the flexible surgical tool or to perform predefined tension control modeling based on the tension information of the flexible surgical tool. a mode switching determination unit that generates a second control mode change signal by determining whether the mode is applicable; an image mode switching unit that switches the control mode from flexible overtube control based on tension information to flexible overtube control based on image information; A tension mode switching unit that switches the control mode from flexible overtube control based on image information to flexible overtube control based on tension information, and hysteresis characteristics using compensation error values generated based on tension information or image information. This can be achieved by providing a hysteresis compensation control device for a flexible overtube, which includes a compensation control unit for compensation control of a flexible surgical tool having.

Additionally, the first control mode change signal is generated when the tip area of the surgical tool included in the image information for determining the current state of the surgical tool is not visible or is obscured by another surgical tool.

Additionally, the tension mode switching unit switches the control mode according to the first control mode change signal.

Additionally, the second control mode change signal is generated when there is a change in the load of the surgical tool or when one of the plurality of tension measurement values of the surgical tool is zero and does not correspond to tension control modeling.

Additionally, the image mode switching unit switches the control mode according to the second control mode change signal.

Meanwhile, the purpose of the present invention is to compensate and control a flexible surgical tool having hysteresis characteristics based on image information of the flexible surgical tool, and determine whether the surgical tool is visible based on image information that determines the current state of the surgical tool. , if it is determined that the surgical tool is not visible, switching the control mode from compensation control based on image information to compensation control based on tension information; Compensating control of the flexible surgical tool with hysteresis characteristics based on the tension information of the flexible surgical tool. This can be achieved by providing a hysteresis compensation control method of a flexible overtube, comprising the steps:

Additionally, after the control mode is changed from compensation control based on image information to compensation control based on tension information,

A step of determining whether the flexible surgical tool corresponds to predefined tension control modeling based on tension information. If it is determined that it does not correspond to predefined tension control modeling, control is changed from compensation control based on tension information to compensation control based on image information. It includes switching modes and compensatingly controlling a flexible surgical tool having hysteresis characteristics based on image information of the flexible surgical tool.

According to the present invention as described above, there is an effect of precisely controlling a flexible surgical tool even within a hysteresis section.

The following drawings attached to this specification illustrate a preferred embodiment of the present invention, and serve to further understand the technical idea of the present invention along with the detailed description of the invention. Therefore, the present invention is limited to the matters described in such drawings. It should not be interpreted in a limited way.
1 is a diagram showing a flexible overtube according to an embodiment of the present invention,
Figure 2 is a diagram showing the hysteresis characteristics or curve of a flexible surgical tool according to an embodiment of the present invention;
Figure 3 is a diagram showing the approximate configuration of a hysteresis compensation control device for a flexible overtube according to a first embodiment of the present invention;
Figure 4 is a diagram sequentially explaining the control mode switching from image information-based hysteresis compensation control to tension information-based hysteresis compensation control according to the first embodiment of the present invention;
Figure 5 shows the control mode changed from tension information-based hysteresis compensation control to image information-based hysteresis compensation control after switching the control mode from image information-based hysteresis compensation control to tension information-based hysteresis compensation control according to the first embodiment of the present invention. This is a diagram sequentially explaining the change and switching,
Figure 6 is a diagram showing the approximate configuration of a hysteresis compensation control device for a flexible overtube according to a second embodiment of the present invention;
Figure 7 shows a control mode change from image information-based hysteresis compensation control to tension information-based hysteresis compensation control after switching the control mode from tension information-based hysteresis compensation control to image information-based hysteresis compensation control according to the second embodiment of the present invention. This is a diagram sequentially explaining the change and switching,
Figure 8 is a diagram showing the approximate configuration according to learning of the hysteresis compensation model according to the third embodiment of the present invention;
Figure 9 is a diagram showing the approximate configuration of hysteresis compensation control according to the third embodiment of the present invention.

Hereinafter, a preferred embodiment of the present invention will be described with reference to the drawings. In addition, the embodiment described below does not unduly limit the content of the present invention described in the claims, and it cannot be said that all of the configurations described in this embodiment are essential as a solution to the present invention. In addition, descriptions of matters that are obvious to those skilled in the art and skilled in the art may be omitted, and descriptions of such omitted components (methods) and functions may be sufficiently referenced without departing from the technical spirit of the present invention.

A hysteresis compensation control device for a flexible tube according to an embodiment of the present invention is a device that compensates and controls hysteresis occurring in a flexible tube. At this time, the flexible tube is a tube that is driven or pulled by a wire, and examples include a flexible overtube or a flexible surgical tool or flexible robotic surgical tool inserted into a channel of the flexible overtube. Hereinafter, the flexible overtube 10 or the flexible surgical tool 21 and 22 will be described as an example of a flexible tube.

In the flexible overtube 10, a flexible surgical tool is inserted into the flexible tube body and the channel of the flexible tube. The flexible overtube 10 is inserted into the human body and is a device that performs surgery on the human body by controlling the position and orientation of the flexible tube and surgical tools through a traction wire.

As shown in FIG. 1, the flexible overtube 10 (or tube) includes an overtube body 10a having a flexible body, first and second tubes formed with a predetermined diameter inside the overtube body 10a. It includes 3 channels (11, 12, 13), surgical tools and a camera unit inserted into each channel. As shown in Figure 1, the first surgical tool 21 is inserted into the first channel 11, the second surgical tool 22 is inserted into the second channel 12, and the third channel 13 is inserted. The camera unit 23 may be inserted. However, the number of channels and surgical tools may be changed as needed. A camera unit 23 and a surgical tool inserted into each channel protrude from the front end of the overtube body 10a. Accordingly, the camera unit 23 can capture images of the front, and each surgical tool can control surgery to pinch or excise tissue according to the manipulation of the traction wire (or drive wire). An operating unit (or control unit, not shown) that can manipulate a traction wire (not shown) is disposed at the rear end of the overtube body 10a. The traction wire is connected from the operating unit to the overtube body or the front end of the surgical tool. Therefore, the position and direction of the traction wire can be controlled by the user's operation of the control panel. At this time, as an example, in order to control the position and direction of the flexible overtube body 10a, the first traction wire 10b and the second traction wire 10c are adjusted by the operating unit, so that the position and direction can be changed according to the traction force. . And, just as the position and direction control of the flexible overtube body 10a, the position and direction control of each surgical tool can be explained using the same principle.

Meanwhile, as shown in Figure 2, the flexible overtube body 10a or the flexible surgical tools 21 and 22 have a linear slope section according to the traction force of the traction wire to control the position and direction of the surgical tool. You can. However, when controlling a surgical tool using a traction wire, even though the same traction force is provided as shown in FIG. 2, there are sections where the position and direction cannot be consistently controlled to correspond to the traction force. This section can be called a hysteresis area, and in this hysteresis area, the position and direction control of the surgical tool may vary even if the traction force is the same. The reason why this hysteresis occurs is, for example, when a flexible surgical tool or flexible overtube is inserted into the body, its shape or twist may change, which may cause friction between the wire and the wire sheath, or wire elongation due to the use of a traction wire ( elongation may be the cause. Additionally, slack in the wire may cause a so-called dead zone in which there is no change in output even though a control command is input to tow the wire. Additionally, it may be caused by un-uniform tube characteristics due to sheaths or wires within the flexible overtube.

Therefore, in the present invention, the surgical tool can be precisely controlled by compensating and controlling the flexible surgical tool having such hysteresis characteristics.

(Configuration and function of hysteresis compensation control device of flexible overtube: first embodiment)

As shown in Figure 3, the hysteresis compensation control device for a flexible overtube according to the first embodiment of the present invention includes an input unit 110, a mode switching determination unit 120, a mode switching unit 130, and a learning unit 140. ), and includes a compensation control unit 150.

The input unit 110 receives image information and tension information of the flexible surgical tool required for mode switching for hysteresis control. For this purpose, the input unit 110 includes a surgical tool image capture unit 111, a tension measurement unit 112, a tension change calculation unit 113, and a tension input value comparison determination unit 114.

The surgical tool image capturing unit 111 is inserted into the third channel 13, and the end effector of the first surgical tool 21 or the tip 21a of the surgical tool is inserted into the third channel 13 and is the subject of hysteresis compensation control. ) Create a captured image of the area. At this time, if the control object is the second surgical tool 22, the second surgical tool 22 may be photographed, and, if necessary, both the first and second surgical tools 21 and 22 may be photographed. Meanwhile, the surgical tool image capture unit 111 may be fixed or inserted and fixed into the channel in a structure that allows movement control as needed. At this time, it is desirable that the shooting conditions of the camera actually inserted into the body for surgery exactly match the shooting position, shooting angle, or shooting area of the surgical tool already used for learning in the learning unit 140.

The tension measuring unit 112 measures the traction force of the traction wire of the surgical tool. To measure the traction force, for example, a tension measuring sensor may be placed to measure the tension of the traction wire, or the load of the motor that drives the traction wire placed on the operating unit may be measured.

As shown in Figure 5, after the control mode is switched from the surgical tool hysteresis compensation control based on image information to the surgical tool hysteresis compensation control based on tension information, the mode is additionally switched back to the surgical tool hysteresis compensation control based on image information. It includes a tension change calculation unit 113 and a tension input value comparison determination unit 114.

The tension change calculation unit 113 operates when there is no load on the first surgical tool 21, which is the object of hysteresis compensation control, and then a load is suddenly applied, or when a load is applied to the first surgical tool 21 and then the load disappears. Calculate the amount of change in the wire tension value. In order to control one surgical tool with a traction wire, each of the two traction wires must be manipulated as shown in FIG. 1 described above, and accordingly, wire tension or traction force measuring sensors for the first and second traction wires are required.

However, for convenience of explanation, each traction wire is described as one traction wire. In reality, 2 traction wires are needed to move 1 degree of freedom, and in the case of 2 degrees of freedom, a total of 4 traction wires are needed. Additionally, one drive motor is required per traction wire, or it can be driven with one drive motor per 2 traction wires.

The tension change calculation unit 113 calculates the change in tension value measured by each measurement sensor. By comparing the amount of change in tension value, you can know when it changes from no load to load or from load to no load. In these two cases, hysteresis compensation control based on tension information is difficult, so the control mode must be switched. That is, in this case, it is necessary to change from tension information-based control to image information-based control.

The tension input value comparison determination unit 114 finds a case where one of the tension or traction measurement values (or tension values) of the above-described first and second wires is '0'. If one of the tension values of the first and second wires is '0', hysteresis compensation control based on tension information is difficult and the control mode must be switched. That is, in this case, it is necessary to change from tension information-based control to image information-based control. Hereinafter, the description will be made assuming that the hysteresis compensation control object is the first surgical tool 21. However, hysteresis compensation control of the second surgical tool 22 can also be explained using the same principle.

The mode switching determination unit 120 according to an embodiment of the present invention determines the current state of the first surgical tool 21 based on the image information of the flexible first surgical tool 21, thereby generating a first control mode change signal ( A signal generated to enable mode switching from image information-based hysteresis compensation control to tension information-based hysteresis compensation control) is generated.

The mode switching determination unit 120 determines whether it corresponds to predefined tension control modeling based on the tension information of the flexible first surgical tool 21, thereby transmitting a second control mode change signal (from tension information-based hysteresis compensation control to image information-based A signal generated to enable mode switching through hysteresis compensation control) is generated.

Image information for determining the current state of the first surgical tool 21 is image information obtained by photographing the tip 21a area of the first surgical tool 21.

The mode switching determination unit 120 includes an image analysis and determination unit 121 and a tension modeling analysis and determination unit 122.

The image analysis and determination unit 121 receives a captured image of the tip area of the first surgical tool 21 from the surgical tool image capturing unit 111. The current state of the first surgical tool 21 can be determined through the received captured image. That is, by analyzing the captured image, if the tip area of the first surgical tool 21 included in the captured image is not visible or the first surgical tool 21 is obscured by the second surgical tool 22, image information is provided. Since based compensation control cannot be performed, the image analysis and determination unit 121 generates a first control mode change signal. According to the generation of the first control mode change signal, the control mode is switched from image information-based hysteresis compensation control to tension information-based hysteresis compensation control.

The tension modeling analysis and determination unit 122 determines whether it corresponds to tension control modeling according to the input value of the tension change amount calculation unit 113 or the tension input value comparison determination unit 114, and performs tension control modeling according to the determination. If it is determined that it is not applicable, hysteresis compensation control based on tension information can no longer be performed, so a second control mode change signal is generated to switch the control mode. That is, the control mode is switched from tension information-based hysteresis compensation control to image information-based hysteresis compensation control. At this time, as an example of a case that does not correspond to tension control modeling, there is a change in the load of the first surgical tool 21 (when there is no load and then a load is applied, when a load is applied and then the load disappears) or the first This is a case where one of the tension measurement values of the first and second traction wires for pulling the surgical tool 21 is '0'. The above-mentioned example is a case where the prediction model is not assumed during deep learning learning by the tension information learning unit 142, which will be described later, and in this case, hysteresis compensation control cannot be performed based on tension information. Therefore, switching of control modes is necessary.

The mode switching unit 130 according to an embodiment of the present invention includes an image mode switching unit 131 and a tension mode switching unit 132.

The image mode switching unit 131 performs the first surgery based on tension information in the compensation control of the first surgical tool 21 based on image information according to the first control mode change signal from the image analysis and determination unit 121. Tool 21 switches the control mode to compensation control.

The tension mode switching unit 132 controls the compensation control of the first surgical tool 21 based on image information according to the second control mode change signal from the tension modeling analysis and determination unit 122 to the first compensation control based on tension information. Switch the control mode to compensation control for the surgical tool (21).

The learning unit 140 according to an embodiment of the present invention performs deep learning or machine learning based on image information or tension information, and calculates an image error value or a tension error value through learning, respectively. For this purpose, the learning unit 140 includes an image information learning unit 141 and a tension information learning unit 142.

The image information learning unit 141 performs machine learning or deep learning based on the captured image of the tip 21a area of the first surgical tool 21. Therefore, it is desirable that the shooting conditions of the camera 23 for acquiring images necessary for learning and the shooting conditions of the camera 23 when actually inserted into the body match each other. The image information learning unit 141 calculates an image compensation control value or an image compensation error value corresponding to the image of the first surgical tool 21 currently captured and inserted into the body through learning according to a predefined learning model. The image compensation error value is transmitted to the compensation control unit 150. Of course, the image compensation error value generated at this time may be information related to the driving of the traction wire drive motor for actually driving the first surgical tool.

The tension information learning unit 142 performs machine learning or deep learning based on the tension information of the traction wire of the first surgical tool 21. The tension information learning unit 142 calculates a tension compensation control value or a tension compensation error value corresponding to the tension measurement value of the first surgical tool 21 inserted into the body by learning according to a predefined learning model. The tension compensation error value is transmitted to the compensation control unit 150.

The compensation control unit 150 compensates and controls the first surgical tool 21 having a hysteresis curve based on the image compensation error value or the tension compensation error value. For this purpose, the compensation control unit 150 includes an image information-based hysteresis compensation control unit 151 and a tension information-based hysteresis compensation control unit 152.

The image information-based hysteresis compensation control unit 151 controls the first surgical tool 21 with hysteresis compensation using the image compensation error value. In other words, hysteresis compensation is controlled using the image compensation error value calculated through learning based on the image information of the currently captured first surgical tool.

The tension information-based hysteresis compensation control unit 152 controls the first surgical tool 21 with hysteresis compensation using the tension compensation error value. That is, hysteresis compensation is controlled using the tension compensation error value calculated through learning based on the currently measured tension information of the first surgical tool.

(Hysteresis compensation control method for flexible overtube)

The hysteresis compensation control method of a flexible overtube according to an embodiment of the present invention includes a first method in which the control mode is changed from the image information-based compensation control shown in FIG. 4 to the tension information-based compensation control and the image information shown in FIG. 5. A second method in which the control mode is changed from based compensation control to tension information-based compensation control, and then again from tension information-based compensation control to image information-based compensation control will be described below.

As shown in FIG. 4, tension information-based compensation control in image information-based compensation control can be achieved by performing the following steps.

The compensation control unit 150 first compensates and controls the first surgical tool 21 having a hysteresis curve based on the image information of the first surgical tool 21 (S11). In more detail, the image information-based hysteresis compensation control unit 151 controls the tension of the traction wire of the first surgical tool 21 based on the image information of the first surgical tool 21.

At this time, the image analysis and determination unit 121 determines whether the first surgical tool 21 is visible based on image information that determines the current state of the first surgical tool 21. When the first surgical tool 21 is obscured or disappears from the field of view of the camera unit 23, the current state of the first surgical tool 21 is determined because image information-based compensation control can no longer be performed (S12).

As a result of the determination, if it is determined that the surgical tool is not visible, the mode switching unit 130 switches the control mode from compensation control based on image information to compensation control based on tension information (S13).

According to the switching of the control mode, the tension information-based hysteresis compensation control unit 152 compensates and controls the first surgical tool 21 having a hysteresis curve based on the tension information of the first surgical tool 21.

Meanwhile, as shown in FIG. 5, after performing tension information-based compensation control in image information-based compensation control, image information-based compensation control can be performed again by performing the following steps.

First, as shown in FIGS. 4 and 5, steps S11 to S14 are the same. The tension information-based hysteresis compensation control unit 152 compensates and controls the first surgical tool 21 based on the tension information of the first surgical tool 21.

The tension modeling analysis and determination unit 122 corresponds to predefined tension control modeling based on the tension information of the first surgical tool 21 after the control mode is changed from compensation control based on image information to compensation control based on tension information. Determine whether it is done (S15). This is because compensation control based on tension information can no longer be performed in cases that do not correspond to tension control modeling.

As a result of the determination, if it is determined that it does not correspond to the predefined tension control modeling, the image mode switching unit 131 switches the control mode from compensation control based on tension information to compensation control based on image information (S16).

According to the switching of the control mode, the image information-based hysteresis compensation control unit 151 compensates and controls the first surgical tool 21 having a hysteresis curve based on the image information of the first surgical tool 21.

(Configuration and function of hysteresis compensation control device of flexible overtube: Second embodiment)

The hysteresis compensation control device for a flexible overtube according to a second embodiment of the present invention will be described in detail with reference to the attached drawings 6 and 7. However, the description of the input unit 110, mode switching determination unit 120, mode switching unit 130, learning unit 140, and compensation control unit 250 may be referred to as needed, and the same description is omitted. I decided to do it.

As shown in FIG. 6, the input unit 210 receives image information and wire tension information of a flexible surgical tool required for mode switching. The input unit 210 includes a surgical tool image capturing unit 211, a tension measuring unit 212, a tension change calculation unit 213, and a tension input value comparison determination unit 214. I will replace it with an explanation. However, the tension input value comparison determination unit 214 receives the tension measurement values of the first and second wires and finds a case where one of the tension input values is '0'. For this purpose, it is desirable to adjust the traction force so that the initial tension measurement value according to the traction force of the wire is measured as '0' or more even when there is no load.

The mode switching determination unit 220 includes an image analysis determination unit 221 and a tension modeling analysis and determination unit 222. The mode switching determination unit 220 determines the current state of the surgical tool based on the image information and tension information of the flexible surgical tool, and generates a control mode change signal for control mode switching according to the current state of the surgical tool. Control mode change signals are generated separately according to each switching condition. The mode switching determination unit 220 generates a first and second switching condition control mode change signal, a third forced switching condition control mode change signal, and a signal that induces a change in the state of the surgical tool, respectively.

The tension modeling analysis and determination unit 222 determines that flexible tube control based on tension information cannot be performed while performing control based on tension information. In the first switching condition, the flexible tube control based on tension information is performed. Generates a first switching condition control mode change signal for switching the control mode to flexible tube control based on image information. The example of the first switching condition will replace the previous description in which control based on tension information cannot be performed.

In addition, while performing flexible tube control based on image information by the first switching condition control mode change signal, the tension modeling analysis and determination unit 222 analyzes the image based on the signal from the image analysis and determination unit 221. If it is determined that the flexible tube control based on the information cannot be performed, and the second switching condition is determined to be released, the control based on the tension information in the flexible tube control based on the image information is determined. A second switching condition control mode change signal is generated for switching the control mode to flexible tube control.

The image analysis and determination unit 221 determines that flexible tube control based on image information cannot be performed while flexible tube control based on image information is being performed by the first switching condition control mode change signal, and And, if the third forced switching condition is determined to determine that the first switching condition is not released, the third control mode is forcibly switched from flexible tube control based on image information to flexible tube control based on tension information. Forced switching condition: Generates a control mode change signal. As an example of the third forced switching condition, a partial area of at least one surgical tool among a plurality of surgical tools is not visible on the read image, or two or more surgical tools overlap each other and an overlap area occurs on the read image, resulting in image reading. There may be difficult cases, or cases where two or more surgical tools are in contact with each other, making image interpretation difficult. In addition, as another example, if blood gets on the camera lens or surgical tools, if the camera's view or surgical tools are obscured by smoke from organ tissue or organ resection in the human body, or if the organ or the environment surrounding the organ causes surgery There are cases where it is difficult to identify the surgical tool because the overall image outline of the tool changes, when one surgical tool blocks the camera's view, and when the LED light of the recording camera is reflected on the surgical tool, making it difficult to identify the image of the surgical tool. .

The mode switching determination unit 220 determines that flexible tube control based on image information cannot be performed while performing flexible tube control based on image information by the first switching condition control mode change signal, and If the fourth switching condition, which determines that the 1 switching condition is not released, is met, the wire or surgical tool is controlled to generate a signal that induces a change in the state of the surgical tool to enable flexible tube control based on tension information or image information. The generated signal is transmitted to the control unit 260, which will be described later.

The control unit 260 includes a wire control unit 261 and a surgical tool control unit 262. The wire control unit 261 receives a signal that induces a change in the state of the surgical tool from the mode switching determination unit 220 and controls the wire to enable flexible tube control based on tension information. As an example, if the wire is not broken, the tension measurement value can be measured by repeatedly controlling the tow line to be pulled or released.

In addition, the surgical tool control unit 262 receives a signal that induces a change in the state of the surgical tool from the mode switching determination unit 220 and controls the surgical tool image capture unit 211 or the surgical tool to provide image information that can be image readable. Enables flexible tube-based control.

When the tension measurement value is normally input or the image can be read under the control of the control unit 260, a tension information available signal and an image information available signal are generated and transmitted to the mode switching determination unit 220.

When the mode switching determination unit 220 receives a tension information enable signal from the control unit 260, the mode switching determination unit 220 provides a tension state change control mode change signal that changes the control mode from flexible tube control based on image information to flexible tube control based on tension information. creates . When the mode switching determination unit 220 receives an image information available signal from the control unit 260, it continues to be controlled in the flexible tube control mode based on the image information. Therefore, no special control mode change is required.

If the state change does not occur even under the control of the controller 260 and control is not possible based on tension information and image information, feedback is sent to the user.

The mode switching unit 230 receives the above-described first and second switching condition control mode change signals, the third forced switching condition control mode change signal, and the tension state change control mode change signal from the mode switching determination unit 220. Depending on each control mode change signal from the mode switching determination unit 220, the control mode is switched from flexible tube control based on tension information to flexible tube control based on image information, or from flexible tube control based on image information. The control mode is controlled to switch from flexible tube control to flexible tube control based on tension information.

The description of the learning unit 240 and the compensation control unit 250 will replace the previous description.

As shown in Figure 7, flexible tube control based on tension information is first performed (S21). If it is determined that the flexible tube control based on tension information does not correspond to the predefined tension modeling (first switching condition) (S22), the control mode is changed to flexible tube control based on image information ( S23,S24).

Meanwhile, while controlling from flexible tube control based on tension information to flexible tube control based on image information, the first switching condition is released (second switching condition), or the first switching condition is not released and the surgical tool If image reading is also not possible and flexible tube control based on image information can no longer be maintained, control is performed respectively as follows.

First, when the first switching condition is released (second switching condition) and the measured tension measurement value is greater than a set threshold (in this case, the threshold is '0' as an example) (second' switching condition), image information The control mode is changed back from flexible tube control based on tension information to flexible tube control based on tension information (S31, S32, S33, S34).

Meanwhile, if the first switching condition is not released and the image of the surgical tool cannot be read (S51), control is performed in three ways as follows.

First, in the first method, it is determined whether the control mode change forced condition is met (third forced switching condition), and if the third forced switching condition is met, the flexible tube control based on image information is unconditionally changed from flexible tube control based on tension information. Control by changing the control mode back to sex tube control (S41.S54a)

In the second method, the wire is controlled so that the first switching condition is released by repeatedly controlling the wire so that the wire is pulled or released (S52a). If the wire tension measurement value is normally input according to the wire control (fourth switching condition), the control mode is changed back from flexible tube control based on image information to flexible tube control based on tension information (S53a, S54a) ). If the tension measurement value is not input properly and the fourth switching condition is not satisfied, the system switches to the third method described later or finally sends feedback to the user.

In the third method, the position of the surgical tool or the surgical tool image capture unit 211 is controlled so that the image of the surgical tool is read (S52b). If it is determined that image reading is possible according to the position control (fourth switching condition), the flexible tube control based on the image information is maintained as is (S53b, S54b). If image reading is not possible and the fourth switching condition is not satisfied, the method switches to the second method described above or finally provides feedback to the user.

(Configuration and function of hysteresis compensation control device of flexible overtube: Third embodiment)

In general, hysteresis varies depending on the degree of friction between the wire and the sheath, and the shape of the sheath also varies depending on the shape of the overtube. Accordingly, the friction between the outer ear and the sheath may vary, which changes the characteristics of hysteresis, causing the position and posture (bending angle) of the flexible surgical tool to vary depending on the shape of the overtube despite the driving force of the same wire. Therefore, the hysteresis compensation control device for a flexible overtube according to the third embodiment of the present invention can derive a compensation value by exploring a hysteresis characteristic model appropriate for the change in the shape of the overtube and the change in the position and posture of the flexible surgical tool. It is a device that allows this. The hysteresis compensation control device for a flexible overtube according to a third embodiment of the present invention will be described in detail with reference to the attached FIGS. 8 and 9.

In the present invention, the hysteresis compensation model is first learned and corrected through learning the hysteresis model, the learned and corrected hysteresis compensation model is searched according to model search conditions to determine the hysteresis compensation model, and then the compensation value is derived based on the determined compensation model to provide a flexible surgical tool. (21,22) is controlled for compensation.

First, learning the hysteresis model will be described with reference to FIG. 8. As shown in FIG. 8, the flexible surgical tool position and posture calculation unit 310 calculates the position and posture values of the flexible surgical tools 21 and 22 that are inserted into the channel of the overtube and driven according to the traction of the wire. The position and posture values of flexible surgical tools can be calculated by the following two methods.

In the first method, the flexible surgical tool position and posture detection unit 311 detects and calculates the actual position and posture values of the flexible surgical tool through a sensor. For this purpose, various sensors can be deployed. As an example of a sensor, an FBG sensor (fiber optic grid element), a magnetic sensor, or a camera can be used. The actual position and posture of the flexible surgical tool may be the bending angle of the flexible surgical tool or a measurement value based on the forward, backward, and rotation of the flexible surgical tool. The tension information may be a tension value or a wire pulling amount measured by an encoder.

In the second method, the flexible surgical tool image acquisition unit 312 acquires an image of the flexible surgical tool. The acquired image of each flexible surgical tool is saved as a data set. The flexible surgical tool position and posture estimation unit 313 matches the input image data set of the flexible surgical tool with the position and posture values of the flexible surgical tool and stores it as a data set. Therefore, the flexible surgical tool position and posture estimation unit 313 can estimate the actual position and posture values of the matched flexible surgical tool based on the image data set transmitted from the flexible surgical tool image acquisition unit 312. The image information of the flexible surgical tool is a set of image data viewed from the camera 111 inserted into the channel of the overtube.

In order to set hysteresis compensation model conditions, the model condition setting unit 321 controls the shape of the overtube to a preset shape or controls the motion of the flexible surgical tool to a preset motion. As an example, the overtube can be transformed into various shapes, and the motion of the flexible surgical tool can also vary depending on the surgical operation. Therefore, various possible environmental variables (shape of the overtube and motion of the flexible surgical tool) are set, and a hysteresis compensation model is created according to the environmental variable settings.

The tension information and image information acquisition unit 322 acquires each tension information and image information according to the hysteresis compensation model conditions variously set in the model condition setting unit 321. Additionally, the tension information and image information acquisition unit 322 stores the obtained tension information and image information as an input data set (input information) to be input to the hysteresis compensation model generator. Tension information can be measured through the tension measuring unit 112 described above, and image information can be acquired through the flexible surgical tool image acquisition unit 312 or the surgical tool image capturing unit 111 described above.

The flexible surgical tool position and posture derivation unit 323 derives the position and posture values of the flexible surgical tool at the time of acquiring the input information from the tension information and image information acquisition unit 322. As described above, the position and posture values of the flexible surgical tool can be derived from sensors or image data sets. Additionally, the flexible surgical tool position and posture derivation unit 323 stores the derived position and posture values of the flexible surgical tool as an output data set (output information).

The hysteresis compensation model generation unit 324 acquires wire tension information and image information, which are input data sets obtained according to the shape of the overtube and the motion control of the flexible surgical tool set in the model condition setting unit 321, and input information. The position and posture values of the flexible surgical tool, which is the output data set, are respectively input. The hysteresis compensation model generation unit 324 receives the input data set and the output data set and generates data according to various environmental variables set in the model condition setting unit 321 (depending on the shape change of the overtube and the motion change of the flexible surgical tool). Create each hysteresis compensation model.

The hysteresis model learning unit 325 learns and calibrates the hysteresis compensation model by including tension information, image information, and the hysteresis compensation model generated by the hysteresis compensation model generation unit 324.

As described above, after learning and correcting the hysteresis compensation model through learning the hysteresis model, the learned and corrected hysteresis compensation model is searched according to the model search conditions to determine the hysteresis compensation model, and the compensation value is derived based on the determined compensation model to provide a flexible Surgical tools (21, 22) are compensated and controlled. Hereinafter, this will be explained with reference to the attached FIG. 9.

The model search condition input unit 330 acquires tension information and image information resulting from a specific motion of the flexible surgical tool after the overtube reaches the target location where the lesion is located. As described above, tension information can be measured through the tension measuring unit 112, and image information can be acquired through the flexible surgical tool image acquisition unit 312 or the surgical tool image capture unit 111. The obtained tension information and image information are transmitted to the hysteresis compensation model search unit 340.

The hysteresis compensation model search unit 340 searches for a hysteresis compensation model calibrated by the learning of the hysteresis model learner 325 based on the tension information and image information input from the model search condition input unit 330.

The hysteresis compensation model determination unit 350 determines a learning-corrected hysteresis compensation model searched by the hysteresis compensation model search unit 340, and calculates a compensation value, which is a tension error value, based on the determined hysteresis compensation model.

The compensation control unit 360 compensates and controls the flexible tube having hysteresis characteristics based on the tension error value of the hysteresis compensation model determined in the hysteresis compensation model determination unit 350.

The hysteresis change detection unit 370 detects movement of the overtube or movement of the patient. Movement of the overtube or the patient corresponds to a hysteresis change condition. If hysteresis changes occur, it may be necessary to re-search the determined hysteresis compensation model. Therefore, when the hysteresis change detection unit 370 detects a hysteresis change condition, it transmits a detection signal to the model search condition input unit 330. The model search condition input unit 330 that receives the hysteresis change detection signal re-acquires tension information and image information to re-search the hysteresis compensation model.

Detection of movement of the overtube may, for example, occur when an image changes without a control input controlling the overtube or a flexible surgical tool, or when a master manipulation signal controlling the overtube is input.

Meanwhile, when the patient moves, the shape of the overtube inserted inside the human body may change. Therefore, the patient's movement is detected and hysteresis changes are predicted. Detection of a patient's movement can be done, for example, by checking the patient's breathing or by changing the image of a video taken of the patient.

In explaining the present invention, matters that are obvious to those skilled in the art and skilled in the art may be omitted, and descriptions of such omitted components (methods) and functions may be sufficiently referenced without departing from the technical spirit of the present invention. You will be able to. In addition, the components of the present invention described above are only described for the convenience of explaining the present invention, and components not described herein may be added without departing from the technical spirit of the present invention.

The description of the configuration and function of each part described above is explained separately from each other for convenience of explanation, and as needed, one configuration and function may be implemented by integrating with other components, or may be implemented in further detail.

Although the present invention has been described above with reference to one embodiment, the present invention is not limited to this, and various modifications and applications are possible. That is, those skilled in the art will easily understand that many modifications are possible without departing from the gist of the present invention. In addition, it should be noted that if a specific description of the known functions and their configurations related to the present invention or the combination relationship between each component of the present invention is judged to unnecessarily obscure the gist of the present invention, the detailed description has been omitted. something to do.

10: Flexible overtube
10a: Overtube body
10b: first traction wire
10c: second traction wire
11: 1st channel
12: 2nd channel
13: Third channel
21: first surgical tool
21a: End effector or surgical tool tip
22: Second surgical tool
22a: End effector or surgical tool tip
23: Camera unit
110: input unit
111: Surgical tool image recording unit
112: Tension measuring unit
113: Tension change calculation unit
114: Tension input value comparison determination unit
120: mode switching determination unit
121: Image analysis and judgment unit
122: Tension modeling analysis and judgment unit
130: mode switching unit
131: Image mode switching unit
132: Tension mode switching unit
140: Learning Department
141: Image information learning unit
142: Tension information learning department
150: compensation control unit
151: Image information-based hysteresis compensation control unit
152: Tension information-based hysteresis compensation control unit
210: input unit
211: Surgical tool image recording unit
212: Tension measuring unit
213: Tension change calculation unit
214: Tension input value comparison determination unit
220: Mode switching determination unit
221: Image analysis and judgment unit
222: Tension modeling analysis and judgment unit
230: mode switching unit
231: Image mode switching unit
232: Tension mode switching unit
240: Learning Department
241: Image information learning department
242: Tension information learning department
250: compensation control unit
251: Image information-based hysteresis compensation control unit
252: Tension information-based hysteresis compensation control unit
260: control unit
261: wire control unit
262: Surgical tool control unit
310: Flexible surgical tool position and posture calculation unit
311: Flexible surgical tool position and posture detection unit
312: Flexible surgical tool image acquisition unit
313: Flexible surgical tool position and posture estimation unit
320: Learning Department
321: Model condition setting unit
322: Tension information and image information acquisition unit
323: Flexible surgical tool location and posture derivation unit
324: Hysteresis compensation model generation unit
325: Hysteresis model learning unit
330: Model search condition input unit
340: Hysteresis compensation model exploration unit
350: Hysteresis compensation model decision unit
360: Compensation control unit
370: Hysteresis change detection unit
371: Overtube motion detection unit
372: Patient movement detection unit

Claims (11)

An input unit that receives image information and wire tension information of flexible surgical tools required for mode switching,
A mode switching determination unit that determines the current state of the surgical tool based on the image information and tension information of the flexible surgical tool and generates a control mode change signal for control mode switching according to the current state of the surgical tool;
Depending on the control mode change signal from the mode switching determination unit, the control mode is switched from flexible tube control based on tension information to flexible tube control based on image information, or from flexible tube control based on image information. A mode switching unit that switches control modes with flexible tube control based on tension information,
A hysteresis compensation control device for a flexible tube, comprising a compensation control unit that compensates and controls a flexible tube having hysteresis characteristics based on the tension error value calculated by the learning model.
According to claim 1,
The input unit,
A surgical tool image capture unit that is inserted into the channel of the body of the tube and takes pictures of the surgical tool inserted in the channel,
A tension measuring unit that measures the traction force of the traction wire of the surgical tool,
A tension change calculation unit that calculates a tension change amount by which the wire tension value of the wire changes based on the traction force measured by the tension measurement unit,
A hysteresis compensation control device for a flexible tube, comprising a tension input value comparison and determination unit that compares and determines the input wire tension value based on the traction force measured by the tension measurement unit.
According to claim 1,
The mode switching determination unit,
In the first switching condition in which it is determined that flexible tube control based on the tension information cannot be performed, the control mode is switched from flexible tube control based on tension information to flexible tube control based on image information. Generate a first switching condition control mode change signal, or
Determining that the first switching condition is released by determining that flexible tube control based on the image information cannot be performed while flexible tube control based on image information is being performed by the first switching condition control mode change signal In the second switching condition, a control mode change signal is generated that switches the control mode from flexible tube control based on image information to flexible tube control based on tension information, or
While performing flexible tube control based on image information by the first switching condition control mode change signal, it is determined that flexible tube control based on the image information cannot be performed and the first switching condition is not released. In the third forced switching condition, a control mode change signal is generated that forcibly switches the control mode from flexible tube control based on image information to flexible tube control based on tension information, or ,
While performing flexible tube control based on image information by the first switching condition control mode change signal, it is determined that flexible tube control based on the image information cannot be performed and the first switching condition is not released. In the fourth switching condition to be determined, the wire or surgical tool is controlled to generate a signal that induces a change in the state of the surgical tool to enable control of the flexible tube based on the tension information or image information. Hysteresis compensation control device.
According to claim 3,
By receiving a signal that induces a change in the state of the surgical tool and controlling the wire, the flexible tube can be controlled based on tension information, or by controlling any one of the surgical tool image capture unit and the surgical tool to control the wire based on image information. A hysteresis compensation control device for a flexible tube, further comprising a control unit for controlling the flexible tube.
According to claim 1,
An image information learning unit that learns according to a preset learning model based on the image information of the surgical tool and calculates an image error value through learning;
A hysteresis compensation control device for a flexible tube, further comprising a tension information learning unit that learns according to a preset learning model based on the tension information of the surgical tool and calculates a tension error value through learning.
A flexible surgical tool position and posture calculation unit that calculates the position and posture values of a flexible surgical tool that is inserted into the channel of the overtube and driven according to the traction of the wire;
The wire tension information and image information, which are input information obtained according to the shape of the overtube and the motion control of the flexible surgical tool, and the position and posture values of the flexible surgical tool, which are output information when acquiring the input information, are respectively input. A hysteresis compensation model generator that generates each hysteresis compensation model according to the shape change of the overtube and the motion change of the flexible surgical tool,
A hysteresis compensation control device for a flexible tube, comprising a hysteresis model learning unit that learns and corrects the hysteresis compensation model by learning the tension information, the image information, and the hysteresis compensation model generated by the hysteresis compensation model generation unit.
According to claim 6,
The flexible surgical tool position and posture calculation unit,
Hysteresis of a flexible tube, characterized in that the position and posture values of the flexible surgical tool are calculated using an attached sensor, or the position and posture values of the flexible surgical tool are matched with image information consisting of a data set. Compensation control device.
According to claim 6,
In order to set hysteresis compensation model conditions, a model condition setting unit that controls the shape of the overtube to a preset shape or the motion of the flexible surgical tool to a preset motion,
A tension information and image information acquisition unit that acquires tension information and image information according to each of the hysteresis compensation model conditions, and stores the obtained tension information and image information as an input data set of the input information,
Deriving the position and posture values of the flexible surgical tool when acquiring the input information, and storing the derived position and posture values of the flexible surgical tool as an output data set of the output information. Deriving the position and posture of the flexible surgical tool. A hysteresis compensation control device for a flexible tube, further comprising:
According to claim 6,
A model search condition input unit for acquiring tension information and image information due to a specific motion of the flexible surgical tool after the overtube reaches the target position,
A hysteresis compensation model search unit that searches for the learned and corrected hysteresis compensation model based on the tension information and image information input from the model search condition input unit;
A hysteresis compensation model determination unit that determines a learned and corrected hysteresis compensation model searched by the hysteresis compensation model search unit and calculates a compensation value that is a tension error value based on the determined hysteresis compensation model;
A hysteresis compensation control device for a flexible tube, further comprising a compensation control unit that compensates and controls the flexible tube having hysteresis characteristics based on the tension error value.
According to clause 9,
A hysteresis compensation control device for a flexible tube, further comprising a hysteresis change detection unit that detects a hysteresis change condition that causes a hysteresis change when movement of the overtube or movement of the patient is sensed.
According to claim 10,
The hysteresis change detection unit,
A flexible tube characterized by detecting the hysteresis change condition and transmitting a detection signal to the model search condition input unit so that tension information and image information are re-acquired from the model search condition input unit and the hysteresis compensation model is re-searched. Hysteresis compensation control device.