WO2023021538A1

WO2023021538A1 - Manipulator system, control device, and shape estimation method

Info

Publication number: WO2023021538A1
Application number: PCT/JP2021/029852
Authority: WO
Inventors: 晋平宮原
Original assignee: オリンパスメディカルシステムズ株式会社
Priority date: 2021-08-16
Filing date: 2021-08-16
Publication date: 2023-02-23

Abstract

A manipulator system (10) comprises: a manipulator (100); a first sensor (150) which acquires the motion of a leading end of the manipulator; an operating unit (200) for performing a bending operation on the manipulator; a second sensor (250) which acquires an operation input amount of the bending operation performed through the operating unit; and a control device (300) which estimates the shape of the manipulator. The control device estimates, from the shape of the manipulator in a first state, a start timing (ts) at which the manipulator is deemed to have started bending, on the basis of a first input value that is a value inputted from the first sensor. In addition, on the basis of a second input value that is a value inputted from the second sensor at a time subsequent to the estimated start timing, the control device estimates the shape of the manipulator in a second state being a state that has changed from the first state as a result of the bending operation by the operating unit.

Description

Manipulator system, controller and shape estimation method

The present invention relates to a manipulator system, a control device, a shape estimation method, and the like.

Conventionally, manipulators and systems containing manipulators have been used in the medical and industrial fields. When a manipulator is used by being inserted into, for example, the body, the operator who operates the manipulator cannot directly see the tip of the insertion portion of the manipulator. In order for the operator to easily operate the manipulator, there is a need to grasp the curved shape of the tip of the manipulator. Japanese Patent Application Laid-Open No. 2002-200001 discloses a technique of sensing the amount of wire pulling of an endoscope, which is an example of a manipulator, and predicting the shape of a bending portion of the endoscope.

JP 2013-172905 A

However, since the wires used in the manipulator have slack, estimating the shape of the manipulator only from the amount of wire pulling results in a large error. Moreover, it is difficult to accurately predict the shape of the manipulator without considering the timing of starting to pull the wire and the shape of the manipulator at that timing. Patent document 1 does not consider such circumstances.

One aspect of the present disclosure is a manipulator, a first sensor that acquires movement of a tip of the manipulator, an operation unit that performs a bending operation on the manipulator, and a second sensor that acquires an operation input amount of the bending operation of the operation unit. and a control device for estimating a shape of the manipulator, wherein the control device receives an input from the first sensor, from the shape of the manipulator in a first state, a start timing at which the manipulator begins to bend. based on a first input value that is a value, and based on a second input value that is an input value from the second sensor after the estimated start timing, a bending operation of the operation unit causes the state to shift from the first state It concerns a manipulator system that estimates the shape of said manipulator in a second state, which is a changed state.

Another aspect of the present disclosure is a control device for estimating a shape of a manipulator, which acquires a first input value, which is an input value from a first sensor that acquires movement of a tip of the manipulator, and bends the manipulator. Acquiring a second input value that is an input value from a second sensor that acquires an operation input amount based on a bending operation of the operating portion to be operated, and based on the first input value, from the shape of the manipulator in the first state. estimating a start timing at which it is assumed that the manipulator has begun to bend, and based on the second input value after the estimated start timing, a second state that is a state changed from the first state by a bending operation of the operation unit; It concerns a controller that estimates the shape of the manipulator in a state.

Yet another aspect of the present disclosure is a shape estimation method for estimating a shape of a manipulator, comprising: acquiring a first input value that is an input value from a first sensor that acquires movement of a tip of the manipulator; acquiring a second input value that is an input value from a second sensor that acquires an operation input amount based on a bending operation of an operation unit that bends the manipulator; and obtaining a first state based on the first input value. estimating a start timing at which it is assumed that the manipulator begins to bend from the shape of the manipulator; estimating the shape of the manipulator in a second state, which is a state changed from .

FIG. 2 is a block diagram for explaining a configuration example of a manipulator system; FIG. FIG. 4 is a diagram schematically explaining an example of a manipulator and an operation unit; The figure explaining the example of a wire. FIG. 5 is a diagram for explaining start timing; The figure explaining the slack of a wire. Another figure explaining slack of a wire. Another figure explaining slack of a wire. The figure explaining a 1st state and a 2nd state. FIG. 5 is a diagram for explaining the relationship between the angle change of the tip of the manipulator, the first input value, and the second input value for each timing; 4A and 4B are diagrams for explaining a processing example of the embodiment; FIG. FIG. 5 is a diagram for explaining a processing example of tip angle estimation processing; FIG. 4 is a diagram for explaining a first input value and a first input value excluding a disturbance value; The figure explaining the model of an articulated manipulator. The figure explaining the relationship between a 2nd input value and a tip angle.

The present embodiment will be described below. In addition, this embodiment described below does not unduly limit the content described in the claims. Moreover, not all the configurations described in the present embodiment are essential constituent elements of the present disclosure.

A configuration example of the manipulator system 10 of the present embodiment will be described using FIGS. 1, 2 and 3. FIG. FIG. 1 is a block diagram illustrating a configuration example of a manipulator system 10 of this embodiment. Manipulator system 10 includes manipulator 100 , operation unit 200 , and control device 300 . Manipulator 100 includes a first sensor 150 . Operation unit 200 includes a second sensor 250 . The configuration of the manipulator system 10 is not limited to that shown in FIG. 1, and various modifications such as adding other components are possible. For example, the manipulator 100 can be configured to include the operation unit 200 .

FIG. 2 is a diagram schematically illustrating the manipulator 100 and the operation unit 200 of this embodiment. Manipulator 100 includes a curved portion 102 and a flexible portion 104 . The bending portion 102 is positioned on the distal end side of the outer sheath 110 and includes a plurality of bending pieces 120 and a distal end portion 130 connected to the distal ends of the bending pieces 120 . The bending portion 102 bends when the wire 160 is pulled based on the operation of the operation portion 200, which will be described later. Also, the flexible portion 104 is positioned on the proximal end side of the outer sheath 110, does not include the bending piece 120, and bends passively by an external force. Although illustration is omitted, the flexible portion 104 may further include a flexible annular member as long as it is passively bent by an external force. The bending piece 120 is a short cylindrical member made of metal, and the number thereof is not particularly limited. The plurality of bending pieces 120 and the distal end portion 130 are each connected by a rotatable connecting portion 140 . That is, the manipulator 100 of this embodiment has an articulated structure. The manipulator 100 is not limited to that shown in FIG. 2, and various modifications such as addition of other components are possible. For example, although illustration is omitted, the distal end portion 130 may include a treatment tool, a lighting device, or an imaging device. Also, all or part of these may be included. For example, the lighting device and imaging device included in the distal end portion 130 can be controlled from the outside by passing an optical fiber connected to the lighting device and a cable connecting to the imaging device through a cavity (not shown) in the operation unit 200 and the bending piece 120. can be realized. By doing so, the manipulator 100 of the present embodiment can be used as a medical manipulator such as a medical endoscope, a catheter, and a surgical support robot arm, or as an industrial manipulator such as an industrial endoscope and an industrial robot arm. Can be used as a manipulator. The specific structures of the bending piece 120, the distal end portion 130, the connecting portion 140, etc. are known, and detailed description thereof will be omitted. Further, in the following description, bending operation of the bending portion 102 may be referred to as bending operation of the manipulator 100 . Further, in the following description, bending of the bending portion 102 may be referred to as bending of the manipulator 100 .

Also, for convenience of explanation, the A axis, the UD axis and the LR axis are shown as appropriate from FIG. 2 onward as the three mutually orthogonal axes. The direction along the A-axis is called the A-axis direction, which is the direction along the longitudinal direction of the manipulator 100 . Further, the direction in which the distal end side of the manipulator 100 is inserted into a body cavity, for example, is direction A1, and the direction in which the manipulator 100 is pulled out is direction A2. Also, the direction along the UD axis is called the UD axis direction, and the direction along the LR axis is called the LR axis direction. Here, the LR axis, UD axis, and A axis can also be called the X axis, Y axis, and Z axis, respectively. Note that the term "perpendicular" includes not only intersecting at 90° but also intersecting at an angle slightly inclined from 90°.

The first sensor 150 acquires the movement of the tip of the manipulator 100. The first sensor is, for example, a two-axis or three-axis acceleration sensor, and outputs detected acceleration data to the control device 300, which will be described later, through wireless communication or wired communication. Note that the first sensor 150 may integrate the detected acceleration data with an integrator (not shown) and output it to the control device 300 as velocity data. Note that the two axes here are the aforementioned X axis and Y axis, and the three axes are the X axis, Y axis, and Z axis. The same applies to the rest. Also, the first sensor may be a two-axis or three-axis gyro sensor, and may output detected angular acceleration data or angular velocity data to the control device 300, which will be described later, via wireless or wired communication. A gyro sensor may also be called an angular velocity sensor. Also, the first sensor may be a motion sensor including a biaxial or triaxial acceleration sensor and a biaxial or triaxial gyro sensor. Note that the motion sensor may refer to either one of an acceleration sensor and a gyro sensor. By analyzing the data detected by the first sensor 150, it is possible to obtain information for estimating the start timing ts at which the tip of the manipulator 100 starts moving, the details of which will be described later.

The operation unit 200 bends the manipulator 100 . For example, the pair of wires 160 can be moved in opposite directions by a wire driving mechanism (not shown) based on an operation input of the operation section 200, and the bending section 102 can be bent in a desired direction. In FIG. 2, the bending portion 102 is shown bent upward in the UD direction by the two wires 160, but it may be bent in the RL direction. For example, as shown in FIG. 3, the manipulator 100 may be configured to include four wires 160 consisting of an upper bend wire 160u, a lower bend wire 160d, a left bend wire 160l, and a right bend wire 160r. . In the following embodiments, when the operator operates the angle knob 200A that is the operation unit 200, the left bending wire 160l and the right bending wire 160r move in mutually opposite directions, and the predetermined bending piece 120 rotates. It is assumed that the bending portion 102 can be bent in a desired direction in the RL direction. Similarly, when the operator operates the angle knob 200B that is the operation unit 200, the upper bending wire 160u and the lower bending wire 160d move in opposite directions, and the specific bending piece 120 rotates, thereby bending the bending portion 102. can be bent in any desired direction in the UD direction. Note that the wire drive mechanism (not shown) can be realized by a known method, so detailed description thereof will be omitted.

The second sensor 250 acquires the operation input amount of the bending operation of the operation section 200 . For example, when the operation unit 200 is the angle knobs 200A and 200B described above, by attaching an angle sensor or a rotary encoder to the rotation shaft of the operation unit 200, the amount of rotation of the angle knobs 200A and 200B can be grasped as the operation input amount of the bending operation. can do. A method of estimating the amount of pulling of the wire 160 based on the amount of rotation of the angle knobs 200A and 200B and estimating the curved shape of the manipulator 100 is known. Note that the second sensor 250 may be a sensor that directly detects the amount of movement of the wire 160 .

The control device 300 acquires a first input value input from the first sensor and a second input value input from the second sensor. The first input value is the aforementioned acceleration data or angular acceleration data, or position change data or angle data obtained by integrating these data. The second input value is data indicating the operation input amount of the operation unit 200 described above and data indicating the movement amount of the wire 160 .

The control device 300 is configured with the following hardware. The hardware may include circuitry for processing digital signals and/or circuitry for processing analog signals. For example, the hardware may consist of one or more circuit devices or one or more circuit elements mounted on a circuit board. The one or more circuit devices are, for example, ICs (Integrated Circuits), FPGAs (field-programmable gate arrays), or the like. The one or more circuit elements are, for example, resistors, capacitors, and the like.

Also, the control device 300 may be realized by the following processor. Controller 300 includes a memory that stores information and a processor that operates on the information stored in the memory. The information is, for example, programs and various data. A processor includes hardware. Various processors such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and a DSP (Digital Signal Processor) can be used as the processor. The memory may be a semiconductor memory such as SRAM (Static Random Access Memory) or DRAM (Dynamic Random Access Memory), a register, or a magnetic storage device such as HDD (Hard Disk Drive). Alternatively, it may be an optical storage device such as an optical disc device. For example, the memory stores computer-readable instructions, and when the instructions are executed by the processor, some or all of the functions of the units of the control device 300 are realized as processes. The instruction here may be an instruction set that constitutes a program, or an instruction that instructs a hardware circuit of a processor to perform an operation. Furthermore, all or part of each unit of the control device 300 can be realized by cloud computing, and each process described later with reference to FIG. 10 and the like can be performed on the cloud computing.

Also, the control device 300 estimates the shape of the manipulator 100 . A method of estimating the shape of the manipulator 100 using the amount of pulling of the wire 160 and the amount of operation input of the operation unit 200 is disclosed in Patent Document 1 and the like. Therefore, it is considered that the control device 300 can estimate the shape of the manipulator 100 by using these methods based on the second input value from the second sensor 250 described above.

However, the shape of the manipulator 100 cannot be accurately estimated only by applying the method disclosed in Patent Document 1 and the like. This is because the method disclosed in Patent Document 1 or the like assumes that uniform and constant tension always acts on the wire 160 . In practice, wire 160 in flexible section 104 is slack. As a result, as will be described later, the bending portion 102 starts bending at the start timing ts, which is the timing after the timing at which the operation of the operation portion 200 is started. Unlike the conventional method, the start timing cannot be grasped from the second input value from the second sensor 250 .

A specific explanation will be given using FIG. FIG. 4 is a diagram for explaining the relationship between the time for operating the operation unit 200 and the first input value. In the schematic diagram of the manipulator 100 shown in FIG. 4, the bending pieces 120 and the like, which are unnecessary for explanation, are omitted as appropriate. In addition, the reference numerals of the outer sheath 110, the distal end portion 130, the wire 160, and the operating portion 200 are attached to only one portion as a representative, and the others are omitted. Moreover, in FIG. 4, the specific direction in which the manipulator 100 bends is not particularly limited. It is also assumed that the operator starts operating the operation unit 200 at the timing when the vertical axes intersect, and that the manipulator 100 as a whole is not curved at the time of starting the operation of the operation unit 200 .

Since the wire 160 is slack during the period indicated by B1 in FIG. No force from wire 160 acts on tip 130 . Therefore, the first sensor 150 attached to the distal end portion 130 never acquires the first input value. The period indicated by B2 is a period in which the slackness of the wire 160 corresponding to the operation of the operation section 200 is eliminated. Part 130 begins to act. Then, when the operator continues to operate the operation section 200, the bending section 102 starts bending, for example, as indicated by B3 at the start timing ts.

By establishing a technique for estimating the start timing ts in this manner, the bending angle of the bending portion 102 can be accurately estimated. For example, the bending angle of the bending portion 102 can be accurately estimated by adding a certain offset value to the amount of operation input to the above-described conventional method after considering the start timing. It seems so.

However, the start timing ts is not a constant value, but a value that changes according to the shape of the flexible portion 104 . This is because the slackness of the wire 160 depends on the shape of the flexible portion 104 . 5, 6, and 7 are diagrams illustrating slackness of the wire 160. FIG. Specifically, FIG. 5 shows the state in which the flexible portion 104 is not curved, FIG. 6 shows the state in which the flexible portion 104 is curved by 90°, and FIG. 7 is the state in which the flexible portion 104 is curved by 180°. is in a state of 5 to 7 show only the upward bending wire 160u and the downward bending wire 160d as examples of the wires 160, and show the case where the flexible portion 104 bends upward in the UD axis direction. is the same, and the same is true for the RL axis direction. Further, illustration of the bending portion 102 is omitted in FIG. The same applies to FIGS. 6 and 7. FIG.

In FIG. 5, the slackness of the upper bending wire 160u and the lower bending wire 160d is the same. The same here includes substantially the same. However, in FIGS. 6 and 7, when the flexible portion 104 bends, the upper bending wire 160u stretches while the lower bending wire 160d loosens. 6 and 7, the greater the bending of the flexible portion 104, the smaller the slack of the downward bending wire 160d. Thus, the slackness of wire 160 depends on the shape of flexible portion 104 of manipulator 100 . Therefore, for example, when the manipulator 100 is a medical manipulator such as an endoscope, since the manipulator 100 is inserted through an object having a complicated shape such as an intestine, the shape of the flexible portion 104 changes intricately over time. , and the start timing ts also changes accordingly. Therefore, it is difficult to easily estimate the shape of the manipulator 100 using the conventional method.

Therefore, in this embodiment, the control device 300 estimates the start timing ts based on the first input value from the shape of the manipulator 100 in the first state. For example, assume that the shape of the manipulator 100 before the start timing ts was in the first state. However, although FIG. 8 schematically illustrates a state where the manipulator 100 is not curved, the manipulator 100 may be curved. Then, the start timing ts is estimated by applying a method described later to the first input value input from the first sensor 150 .

Furthermore, after the estimated start timing ts, the control device 300 estimates the shape of the manipulator 100 in the second state, which is the state changed from the first state by the bending operation of the operation unit 200, based on the second input value. . For example, since the manipulator 100 begins to bend after the start timing ts, by applying a predetermined method to the second input value input from the second sensor 250, the shape of the manipulator 100 can be determined at a predetermined timing after the start timing ts. can be estimated. For example, the control device 300 can estimate that the tip of the manipulator 100 is bent by 90° at the timing indicated by C1, and that the tip of the manipulator 100 is bent by 180° at the timing indicated by C2. The predetermined method is the conventional method described above, the method described later with reference to FIG. 13, or the like.

As described above, the manipulator system 10 of the present embodiment includes the manipulator 100, the first sensor 150 that acquires the movement of the tip of the manipulator 100, the operation unit 200 that performs bending operation of the manipulator 100, and the operation unit 200 that performs bending operation. and a control device 300 for estimating the shape of the manipulator 100 . Based on the shape of manipulator 100 in the first state, controller 300 estimates the start timing ts at which manipulator 100 begins to bend, based on the first input value from first sensor 150 . Further, based on the second input value, which is the input value from the second sensor 250 after the estimated start timing ts, the control device 300 controls the second state, which is the state changed from the first state by the bending operation of the operation unit 200 . Estimate the shape of the state manipulator 100 .

Thus, the manipulator system 10 of this embodiment not only includes the second sensor 250 in the operation section 200, but also includes the first sensor 150 in the manipulator 100. By doing so, the control device 300 can use not only the second input value but also the first input value in estimating the shape of the manipulator 100 . Thus, the start timing ts can be estimated using the first state before the manipulator 100 starts bending and the first input value. This makes it possible to update the estimation result of the start timing ts every time the shape of the manipulator 100 changes. As a result, the start timing ts can be accurately estimated, so the shape of the manipulator 100 can be accurately estimated by using the second input value from the second sensor. This allows the operator to operate the manipulator 100 easily and safely.

Also, the method of the present embodiment may be implemented as the control device 300. That is, the control device 300 of the present embodiment is a control device 300 that estimates the shape of the manipulator 100, and acquires the first input value that is the input value from the first sensor 150 that acquires the movement of the tip of the manipulator 100. do. The control device 300 also acquires a second input value, which is an input value from the second sensor 250 that acquires the amount of operation input based on the bending operation of the operation unit 200 that bends the manipulator 100 . Based on the first input value and the shape of the manipulator 100 in the first state, the control device 300 estimates the start timing ts at which the manipulator 100 begins to bend, and the second input after the estimated start timing ts. Based on the values, the shape of the manipulator 100 in the second state, which is the state changed from the first state by the bending operation of the operation unit 200, is estimated. By doing so, an effect similar to that described above can be obtained.

Also, the method of the present embodiment may be implemented as a shape estimation method. That is, the shape estimation method of the present embodiment is a shape estimation method for estimating the shape of the manipulator 100, and acquires the first input value, which is the input value from the first sensor 150 that acquires the movement of the tip of the manipulator 100. and obtaining a second input value that is an input value from the second sensor 250 that obtains an operation input amount based on the bending operation of the operation unit 200 that bends the manipulator 100 . Furthermore, the shape estimating method includes, based on the first input value, estimating the start timing ts at which the manipulator 100 is considered to start bending from the shape of the manipulator 100 in the first state, and It includes estimating the shape of the manipulator 100 in the second state, which is a state changed from the first state by the bending operation of the operation unit 200, based on the second input value. By doing so, an effect similar to that described above can be obtained.

Next, a specific method for estimating the start timing ts will be described. FIG. 9 is a diagram for explaining the relationship between the angle change of the tip portion 130 of the manipulator 100, the behavior of the first input value, and the behavior of the second input value. For convenience of explanation, each graph in FIG. 9 shows either the component in the LR axis direction or the component in the UD axis direction, but the same applies to the other. The same applies to FIG. 13 to be described later.

Although the first sensor 150 is assumed to be an acceleration sensor in the following description, it may be the gyro sensor or motion sensor described above. That is, the first sensor 150 is an acceleration sensor provided at the tip of the manipulator 100, an angular velocity sensor, or both.

Also, in the following description, it is assumed that the operation unit 200 is rotated like the angle knobs 200A and 200B in FIG. 2, and the second sensor 250 is an angle sensor. That is, the operation unit 200 is rotated within a predetermined angle range, and the second sensor 250 is an angle sensor that measures the angle. That is, the change in the second input value shown in FIG. 9 is the change in the angle of the operation unit 200 .

Also, the second sensor 250 may be a sensor that can directly acquire the amount of traction of the wire 160, and in this case there is no need to use the aforementioned angle sensor. In other words, the manipulator 100 includes the wire 160 and the second sensor 250 is a sensor capable of acquiring the amount of pulling of the wire 160 .

Assume that the bending portion 102 is bent by an angle φ1. Note that the curved portion 102 at this time may not be curved, that is, the angle φ1 may be 0°. Then, when the operator bends the operation unit 200, the bending portion 102 starts bending at a first timing t1, and bends by an angle φ2 at a second timing t2. That is, the first timing t1 in FIG. 9 is timing corresponding to the start timing ts in FIG. As described above, since the wire 160 connecting the operation unit 200 and the distal end portion 130 has slack or the like, the first timing t1 at which the bending portion 102 begins to bend is the timing at which the operation input of the operation unit 200 is started. It does not match. That is, the operation input of the operation unit 200 is started at the third timing t3 which is the timing before the first timing t1, and the second sensor 250 acquires the second input value at the third timing t3. It will be.

In other words, since the shape of the manipulator 100 does not change at the third timing t3 or at timings before the third timing t3, it corresponds to the first state in FIG. That is, the first state is the state of the shape of the manipulator 100 at the timing before the third timing t3, which is the timing at which the operation input to the operation unit 200 is performed, and the state of the shape of the manipulator 100 at the third timing t3. at least one of them.

The first input value starts rising after the third timing t3. The timing at which the first input value exceeds the threshold value L is the first timing t1, and the bending portion 102 starts bending. Then, the timing when the first input value is divided by the threshold value L is the second timing t2, and the bending portion 102 stops bending at the aforementioned angle φ2. The threshold value L here is a value that depends on the shape of the manipulator 100 at the third timing t3, and the details will be described later. Also, since the first input value acquired by the first sensor 150 includes a disturbance value, which will be described later, it is necessary to estimate the first timing t1 in consideration of the disturbance value, the method of which will be described later.

Also, as in FIG. 8, the shape of the manipulator 100 can be estimated at a timing after the first timing t1. That is, at a predetermined timing after the first timing t1, the shape of the manipulator 100 can be estimated based on the second input value. In the case of FIG. 9, the predetermined timing is the timing from the first timing t1 to the second timing t2.

As described above, in the manipulator system 10 of the present embodiment, the control device 300 estimates the threshold value L of the operation input amount at which the manipulator 100 is assumed to start bending from the shape of the manipulator 100 in the first state. , the first timing t1, which is the timing at which the operation input amount exceeds the threshold L, is determined as the start timing ts, and based on the second input value at a predetermined timing after the first timing t1, the second state The shape of the manipulator 100 is estimated. By estimating the threshold L that depends on the shape of the manipulator 100 in this manner, the first timing t1, which is the timing at which the first input value exceeds the threshold L, can be made to correspond to the aforementioned start timing ts. Thereby, the shape of the manipulator 100 can be accurately estimated.

Also, the method of the present embodiment may be implemented as the control device 300. That is, the control device 300 of the present embodiment estimates the threshold value L of the amount of operation input at which the manipulator 100 is considered to start bending from the shape of the manipulator 100 in the first state, and calculates the amount of operation input based on the first input value. exceeds the threshold value L as the start timing ts, and the shape of the manipulator 100 in the second state is estimated based on the second input value at a predetermined timing after the first timing t1. . Thereby, an effect similar to that described above can be obtained.

Also, the method of the present embodiment may be implemented as a shape estimation method. In other words, the shape estimation method of the present embodiment includes estimating the threshold value L of the amount of operation input at which the manipulator 100 is assumed to start bending from the shape of the manipulator 100 in the first state. Further, the shape estimating method includes, based on the first input value, determining the first timing t1, which is the timing at which the operation input amount exceeds the threshold value L, as the start timing ts, estimating the shape of the manipulator 100 in the second state based on the second input value at the timing; Thereby, an effect similar to that described above can be obtained.

Next, a processing example for realizing the method of this embodiment will be described using the flowcharts of FIGS. The control device 300 first acquires a first input value (step S100). The process of step S100 is a loop process until YES in step S120, which will be described later, and is always performed at a constant cycle. In other words, the control device 300 continues to acquire the first input value as time-series data.

After that, the control device 300 acquires a disturbance value (step S110). The disturbance value is, among the first input values, a value that does not depend on the bending operation from the operation unit 200, and specifically is noise or the like. That is, in a situation where noise or the like occurs, the control device 300 obtains the first input value as a value to which the disturbance value is added. Therefore, when the control device 300 continues to acquire the first input value until the second timing t2, the actual waveform of the first input value becomes, for example, D1 in FIG. The situation where noise or the like occurs is, for example, when the manipulator 100 is a medical endoscope, it is inserted into the patient's lumen.

Specifically, the control device 300 acquires a disturbance value estimated by applying a predetermined time-series model to the time-series data of the first input value. The predetermined time series model is, for example, an autoregressive model, but may be other time series models such as an autoregressive moving average model or an autoregressive integrated moving average model. The autoregressive model can also be called an AR (autoregressive) model, the autoregressive moving average model can also be called an ARMA (autoregressive moving average) model, and the autoregressive integrated moving average model can be called an ARIMA (autoregressive integrated moving average) model. ) can also be called a model. Note that the theory and the like of these predetermined time-series models are publicly known, and therefore the description thereof is omitted.

Then, the control device 300 performs a process of determining whether or not a bending operation has been detected, and if a bending operation has not been detected (NO in step S120), the process returns to step S100. On the other hand, if a bending operation has been detected (YES in step S120), a tip angle calculation process (step S200), which will be described later, is performed, and then the process returns to step S100.

That is, step S110 described above is repeatedly performed until the bending operation is detected. The disturbance value is also determined. That is, the disturbance value is estimated based on the waveform portion indicated by E in the waveform indicated by D1 in FIG. 12 and the above-described predetermined time-series model. Then, as will be described later with reference to FIG. 11, the estimated disturbance value is used to estimate the first timing t1.

Next, a processing example of the tip angle calculation processing (step S200) will be described using the flowchart of FIG. The control device 300 acquires the second input value at the third timing (step S210), and acquires the threshold value L of the amount of operation input (step S220). Specifically, control device 300 acquires threshold value L estimated according to a predetermined algorithm based on the second input value at the third timing. The predetermined algorithm, for example, assumes that the manipulator 100 is a multi-joint manipulator connected to i joints as shown in FIG. This is an algorithm that estimates the velocity and angular velocity of the point F connected to the terminal joint by distributing. Although each joint in the multi-joint manipulator in FIG. 13 is illustrated as rotating only in the direction along the XY plane, it is possible to make a model in which the rotating direction is extended to three dimensions, and each It is also possible to create a model in which a mechanism that rotates about the axis of the link that connects the joints is added.

In the multi-joint manipulator model of FIG. 13, a homogeneous transformation matrix from the origin O to the link coordinate system at the joint i-1 can be expressed by the following equation (1).

Also, in the articulated manipulator model of FIG. 13, when only the joint i is moving, the velocity and angular velocity of the point F located ahead of the joint i are given by the following equation (2) based on the above equation (1). and the Jacobian matrix at joint i can be derived.

Also, when all the joints (joint 1, joint 2, . , the velocity and acceleration of the point F can be expressed by the following equation (3).

The model of the articulated manipulator in FIG. 13 can correspond to the illustrated manipulator 100 in FIG. For example, the operating portion 200 in FIG. 2 corresponds to the origin O in FIG. 13, the connecting portion 140 in FIG. 2 corresponds to the joint in FIG. 13, and the tip portion 130 in FIG. 2 corresponds to the point F in FIG. The displacement of the bending piece 120 and the change in the angle between the bending pieces 120 due to the pulling of the wire 160 by the operation input of the operation unit 200 in FIG. corresponds to changes in Therefore, if the second input value, which is the value obtained by operating the operation unit 200 in FIG. 2, corresponds to the velocity and angular velocity at the origin O in FIG. The velocity and angular velocity of F are estimated. Based on the velocity and angular velocity at point F, the velocity and angular velocity of tip portion 130 of manipulator 100 can be estimated. Therefore, by setting the acceleration obtained by partially differentiating the estimated speed of the tip end portion 130 as the threshold value L, it can be compared with the first input value, which is the acceleration data. Accordingly, after the third timing t3, the first timing t1 at which the first input value input to the control device 300 exceeds the threshold value L is considered to match the start timing ts with high accuracy. It can be regarded as the start timing ts.

As described above, the estimated threshold value L is a value that depends on the shape of the manipulator 100 at the third timing t3. For example, when the manipulator 100 is not bent at all, and the manipulator 200 is quickly operated, that is, when the wire 160 is quickly pulled, the distal end portion 130 undergoes a large acceleration. Further, for example, even if the wire 160 is pulled at the same speed while the manipulator 100 is curved, the acceleration generated in the tip portion 130 is smaller than when the manipulator 100 is not curved at all. These events mean that the Jacobian matrix in the multi-joint manipulator model of FIG. 13 depends on the angle of each joint. Therefore, when the control device 300 calculates the threshold L based on the above model, estimation of the threshold L can be realized by storing Jacobian matrix data according to the angle in a storage unit (not shown). can be done.

A large estimated threshold value L means that the tip portion 130 of the manipulator 100 is difficult to move. I have something to say. In other words, the operability of the manipulator 100 can be predicted from the shape of the manipulator 100 at the third timing t3, and the second input value at the third timing t3 is used to obtain the first timing t1. A threshold L can be estimated. In other words, the operability is the responsiveness of the first input value to the bending operation of the operating section 200, and the responsiveness is shown in the above equation (3). That is, in Equation (3), the velocity and angular velocity of each joint change according to the bending operation, and as a result, the angle and/or angular velocity of the tip portion 130 corresponding to the first input value responds. Therefore, even if the velocity or angular velocity input to the operation unit 200 is the same, if the Jacobian matrix is different, the angle or angular velocity of the tip portion 130 corresponding to the first input value will be different. Similarly, even if the velocity and angular velocity input to the operation unit 200 are the same, the bending operation amount differs according to the operability, so different threshold values L are estimated according to the operability. That is, the operability and the threshold L are estimated based on the Jacobian matrix of Equation (3).

As described above, in the manipulator system 10 of the present embodiment, the control device 300 converts the second input value at the third timing t3, which is the timing at which the operation input to the operation unit 200 is performed, to the Jacobian matrix in a predetermined algorithm. A threshold value L is obtained based on the threshold value L, and the first timing t1 is obtained by comparing the threshold value L and the first input value. By doing so, the threshold value L can be determined such that the first timing t1 and the start timing ts match with high precision.

Also, the method of the present embodiment may be implemented as the control device 300. That is, the control device 300 of the present embodiment obtains the threshold value L based on the second input value at the third timing t3, which is the timing at which the operation input to the operation unit 200 is performed, and the Jacobian matrix in a predetermined algorithm, A first timing t1 is obtained by comparing the threshold value L and the first input value. Thereby, an effect similar to that described above can be obtained.

Also, the method of the present embodiment may be implemented as a shape estimation method. That is, the shape estimation method of the present embodiment obtains the threshold value L based on the second input value at the third timing t3, which is the timing at which the operation input to the operation unit 200 is performed, and the Jacobian matrix in a predetermined algorithm, and determining the first timing t1 by comparing the threshold value L and the first input value. Thereby, an effect similar to that described above can be obtained.

After that, the control device 300 acquires the first input value (step S230). The process of step S230 is the same as step S100 of FIG. 10, but step S100 is a process of acquiring the first input value before the third timing t3, whereas step S230 is a process of acquiring the first input value after the third timing t3. It differs in that it is a process of acquiring the first input value.

After that, the control device 300 determines whether or not the value obtained by subtracting the disturbance value obtained in step S110 from the first input value obtained in step S230 is greater than the threshold value L obtained in step S220. For example, when the disturbance value is subtracted from the first input value based on the waveform shown in D1 of FIG. 12, the waveform shown in D2, for example, is obtained.

When the value obtained by subtracting the disturbance value from the first input value is smaller than the threshold value L (NO in step S240), the control device 300 performs the process of step S230 again. That is, as long as the value obtained by subtracting the disturbance value from the first input value is smaller than the threshold value L, step S230 is looped. As described above, the first input value obtained in step S230 increases over time, and the timing at which the value obtained by subtracting the disturbance value from the first input value exceeds the threshold value L is the first timing t1, that is, the estimated start It is the timing ts.

As described above, in the manipulator system 10 of the present embodiment, the control device 300 uses the first input value in a predetermined period before the third timing t3, which is the timing at which the operation input to the operation unit 200 is performed, Among the first input values, a disturbance value that does not depend on the bending operation from the operation unit 200 is estimated. Then, the control device 300 estimates the start timing ts based on the disturbance value and the first input value.

Also, the method of the present embodiment may be implemented as the control device 300. That is, the control device 300 of the present embodiment uses the first input value in a predetermined period before the third timing t3, which is the timing at which the operation input to the operation unit 200 is performed, to A disturbance value that does not depend on the bending operation from the operation unit 200 is estimated. Then, the control device 300 estimates the start timing ts based on the disturbance value and the first input value. Thereby, an effect similar to that described above can be obtained.

The method of this embodiment may be implemented as a shape estimation method. That is, the shape estimation method of the present embodiment uses the first input value in a predetermined period before the third timing t3, which is the timing at which the operation input to the operation unit 200 is performed, It includes estimating a disturbance value that does not depend on the bending operation from the operation unit 200 . The shape estimation method also includes estimating the start timing ts based on the disturbance value and the first input value. Thereby, an effect similar to that described above can be obtained.

Then, if the value obtained by subtracting the disturbance value from the first input value is greater than the threshold value L (NO in step S240), the control device 300 integrates the second input value within a predetermined range (step S250). Specifically, the timing at which the first input value becomes greater than the threshold value L is the first timing t1 as described above, which is the timing at which the bending portion 102 is estimated to start bending. Therefore, as shown in FIG. 14, the control device 300 integrates the second input value in the section from the first timing t1 to the second timing, which is the timing at which the bending portion 102 finishes bending, so that the second timing t2 is estimated. In the example of FIG. 9, the higher the estimation accuracy, the closer the estimated angle is to the angle φ2. For example, when the second input value is input from the second sensor 250 so that the second input value has the dimension of the angular velocity, the angle information of the operation input of the operation unit 200 is obtained by integration. Based on this, the pulling amount of the wire 160 can be grasped. Thereby, the shape of the manipulator 100 can be estimated using the conventional technique described above.

As described above, the manipulator 100 is configured by a plurality of bending pieces 120 connected by the connecting portion 140, but the control device 300 estimates the bending angle assuming that the angles formed by the adjacent bending pieces 120 are equal. are doing.

As described above, in the manipulator system 10 of the present embodiment, the control device 300 integrates the second input value in the range from the first timing t1 to the second timing t2, which is the timing after the first timing t1. , the shape of the manipulator 100 at the second timing t2 is estimated. By doing so, the first timing t1, which is estimated to be the timing at which bending of the bending portion 102 is started, can be set as one end of the integration interval. can be estimated to

Also, the method of the present embodiment may be implemented as the control device 300. That is, the control device 300 of the present embodiment integrates the second input value in the range from the first timing t1 to the second timing t2, which is the timing after the first timing t1, based on the second timing Estimate the shape of the manipulator 100 at t2. Thereby, an effect similar to that described above can be obtained.

Also, the method of the present embodiment may be implemented as a shape estimation method. That is, the shape estimation method of the present embodiment calculates the second input value at the second timing based on the value obtained by integrating the second input value in the range from the first timing t1 to the second timing t2, which is the timing after the first timing t1. including estimating the shape of the manipulator 100 at t2. Thereby, an effect similar to that described above can be obtained.

The method of this embodiment is not limited to the above, and various modifications are possible. For example, the first input value is acceleration data input from the first sensor 150, but the control device 300 may acquire velocity data from the first sensor 150 as the first input value. By doing so, for example, the speed of the point F estimated in the above-described multi-joint manipulator model equation (3) can be used as the threshold value L and compared with the first input value. Even in this way, the first timing t1 can be obtained in the same manner, so that the same effect as when the first input value is acceleration data can be obtained.

Although the present embodiment has been described in detail as above, those skilled in the art will easily understand that many modifications that do not substantially deviate from the novel matters and effects of the present embodiment are possible. . Accordingly, all such modifications are intended to be included within the scope of this disclosure. For example, a term described at least once in the specification or drawings together with a different broader or synonymous term can be replaced with the different term anywhere in the specification or drawings. All combinations of this embodiment and modifications are also included in the scope of the present disclosure. Also, the configurations and operations of the manipulator system, control device, and shape estimation method are not limited to those described in the present embodiment, and various modifications are possible.

DESCRIPTION OF SYMBOLS 10... Manipulator system, 100... Manipulator, 110... Outer sheath, 120... Bending piece, 130... Tip, 140... Connection part, 150... First sensor, 160... Wire, 160r... Right bending wire, 160l... Left bending wire, 160u...upward bending wire, 160d...downward bending wire, 200...operation unit, 200A, 200B...angle knob, 250...second sensor, 300...control device, L...threshold value, t1...first timing, t2...second timing , t3... third timing, ts... start timing

Claims

a manipulator;
a first sensor that acquires movement of the tip of the manipulator;
an operation unit that bends the manipulator;
a second sensor that acquires an operation input amount of the bending operation of the operation unit;
a control device for estimating the shape of the manipulator;
including
The control device is
from the shape of the manipulator in the first state, estimating a start timing at which the manipulator begins to bend based on a first input value that is an input value from the first sensor;
A shape of the manipulator in a second state, which is a state changed from the first state by a bending operation of the operation unit, based on a second input value that is an input value from the second sensor after the estimated start timing. A manipulator system characterized by estimating .
In claim 1,
The control device is
estimating a threshold value of the operation input amount at which the manipulator is considered to start bending from the shape of the manipulator in the first state;
determining, based on the first input value, a first timing, which is a timing at which the operation input amount exceeds the threshold value, as the start timing;
A manipulator system that estimates a shape of the manipulator in the second state based on the second input value at a predetermined timing after the first timing.
In claim 2,
The control device is
estimating the shape of the manipulator at the second timing based on a value obtained by integrating the second input value in a range from the first timing to a second timing that is a timing after the first timing; Characterized manipulator system.
In claim 2,
The control device is
The threshold is obtained based on the second input value at a third timing, which is the timing at which an operation input to the operation unit is performed, and a Jacobian matrix in a predetermined algorithm, and the threshold and the first input value are compared. A manipulator system, wherein the first timing is obtained by:
In claim 1,
The first state is at least a state of the shape of the manipulator at a timing before a third timing at which an operation input to the operation unit is performed, and a state of the shape of the manipulator at the third timing. A manipulator system characterized by being one.
In claim 1,
The control device is
Using the first input value in a predetermined period before a third timing at which an operation input to the operation unit is performed, the first input value does not depend on the bending operation from the operation unit. Estimate the disturbance value,
A manipulator system, wherein the start timing is estimated based on the disturbance value and the first input value.
In claim 1,
A manipulator system, wherein the first sensor is an acceleration sensor, an angular velocity sensor, or both, provided at the tip of the manipulator.
In claim 1,
The operation unit is rotated within a predetermined angle range,
The manipulator system, wherein the second sensor is an angle sensor that measures the angle.
In claim 1,
the manipulator comprises a wire;
The manipulator system, wherein the second sensor is a sensor capable of acquiring a pulling amount of the wire.
A control device for estimating the shape of a manipulator,
Acquiring a first input value that is an input value from a first sensor that acquires the movement of the tip of the manipulator;
acquiring a second input value that is an input value from a second sensor that acquires an operation input amount based on a bending operation of an operation unit that bends the manipulator;
estimating a start timing at which the manipulator begins to bend from the shape of the manipulator in the first state, based on the first input value;
Control characterized by estimating the shape of the manipulator in a second state, which is a state changed from the first state by a bending operation of the operation unit, based on the second input value after the estimated start timing. Device.
In claim 10,
estimating a threshold value of the operation input amount at which the manipulator is considered to start bending from the shape of the manipulator in the first state;
determining, based on the first input value, a first timing, which is a timing at which the operation input amount exceeds the threshold value, as the start timing;
A control device that estimates a shape of the manipulator in the second state based on the second input value at a predetermined timing after the first timing.
In claim 11,
estimating the shape of the manipulator at the second timing based on a value obtained by integrating the second input value in a range from the first timing to a second timing that is a timing after the first timing; A controller characterized by:
In claim 11,
The threshold is obtained based on the second input value at a third timing, which is the timing at which an operation input to the operation unit is performed, and a Jacobian matrix in a predetermined algorithm, and the threshold and the first input value are compared. and obtaining the first timing.
In claim 10,
Using the first input value in a predetermined period before a third timing at which an operation input to the operation unit is performed, the first input value does not depend on the bending operation from the operation unit. Estimate the disturbance value,
A control device, wherein the start timing is estimated based on the disturbance value and the first input value.
A shape estimation method for estimating the shape of a manipulator, comprising:
obtaining a first input value that is an input value from a first sensor that obtains movement of the tip of the manipulator;
Acquiring a second input value that is an input value from a second sensor that acquires an operation input amount based on a bending operation of an operation unit that bends the manipulator;
estimating a start timing at which the manipulator begins to bend from the shape of the manipulator in the first state, based on the first input value;
estimating the shape of the manipulator in a second state, which is a state changed from the first state by a bending operation of the operation unit, based on the second input value after the estimated start timing. A shape estimation method characterized by:
In claim 15,
estimating a threshold value of the amount of operation input for determining that the manipulator has started to bend from the shape of the manipulator in the first state;
Determining a first timing at which the operation input amount exceeds the threshold based on the first input value;
estimating a shape of the manipulator in the second state based on the second input value at a predetermined timing after the first timing.
In claim 16,
estimating the shape of the manipulator at the second timing based on a value obtained by integrating the second input value in a range from the first timing to a second timing that is a timing after the first timing; A shape estimation method, comprising:
In claim 16,
determining the threshold value based on the second input value at a third timing, which is the timing at which an operation input to the operation unit is performed, and a Jacobian matrix in a predetermined algorithm; A shape estimation method, comprising: obtaining the first timing by comparison.
In claim 15,
Using the first input value in a predetermined period before a third timing at which an operation input to the operation unit is performed, the first input value does not depend on the bending operation from the operation unit. estimating a disturbance value;
A shape estimation method, comprising estimating the start timing based on the disturbance value and the first input value.