WO2017138208A1 - Système d'endoscope - Google Patents

Système d'endoscope Download PDF

Info

Publication number
WO2017138208A1
WO2017138208A1 PCT/JP2016/083788 JP2016083788W WO2017138208A1 WO 2017138208 A1 WO2017138208 A1 WO 2017138208A1 JP 2016083788 W JP2016083788 W JP 2016083788W WO 2017138208 A1 WO2017138208 A1 WO 2017138208A1
Authority
WO
WIPO (PCT)
Prior art keywords
endoscope
imaging unit
arm
optical axis
control device
Prior art date
Application number
PCT/JP2016/083788
Other languages
English (en)
Japanese (ja)
Inventor
西澤　幸司
Original Assignee
オリンパス株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by オリンパス株式会社 filed Critical オリンパス株式会社
Priority to CN201680065126.4A priority Critical patent/CN108348134B/zh
Priority to JP2017516540A priority patent/JP6150968B1/ja
Priority to EP16889909.4A priority patent/EP3415071A4/fr
Publication of WO2017138208A1 publication Critical patent/WO2017138208A1/fr
Priority to US15/976,142 priority patent/US10638915B2/en

Links

Images

Classifications

    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/00002—Operational features of endoscopes
    • A61B1/00004—Operational features of endoscopes characterised by electronic signal processing
    • A61B1/00006—Operational features of endoscopes characterised by electronic signal processing of control signals
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/00147—Holding or positioning arrangements
    • A61B1/00149—Holding or positioning arrangements using articulated arms
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/04—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor combined with photographic or television appliances
    • A61B1/05—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor combined with photographic or television appliances characterised by the image sensor, e.g. camera, being in the distal end portion
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
    • A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/50—Constructional details
    • H04N23/555—Constructional details for picking-up images in sites, inaccessible due to their dimensions or hazardous conditions, e.g. endoscopes or borescopes
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/66—Remote control of cameras or camera parts, e.g. by remote control devices
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
    • A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
    • A61B2034/2046—Tracking techniques
    • A61B2034/2055—Optical tracking systems
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
    • A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
    • A61B2034/2072—Reference field transducer attached to an instrument or patient
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10068—Endoscopic image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30204—Marker
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30244—Camera pose

Definitions

  • the present invention relates to an endoscope system.
  • This application claims priority based on Japanese Patent Application No. 2016-023670 filed in Japan on February 10, 2016, the contents of which are incorporated herein by reference.
  • Patent Literature 1 discloses an ultrasonic observation system including an ultrasonic probe for grasping a three-dimensional state of an observation object and a holder for the ultrasonic probe.
  • Patent Document 2 discloses a surgical microscope capable of observing other image information during observation using a surgical microscope.
  • Patent Document 3 discloses an endoscope system including a treatment scope for obtaining a narrow-angle image of a treatment site and an observation endoscope for obtaining a wide-angle image of the treatment site. ing.
  • Japanese Unexamined Patent Publication No. 2004-136066 Japanese Unexamined Patent Publication No. 2001-161638 Japanese Unexamined Patent Publication No. 2000-32442
  • the present invention has been made in view of the above-described circumstances, and is an endoscope that can easily associate images captured by two endoscopes with less burden on a surgeon or an operator of an endoscope.
  • the purpose is to provide a system.
  • An endoscope system is attached to a first endoscope having a first imaging unit, a second endoscope having a second imaging unit, and the first endoscope.
  • An arm operable to move the first endoscope; first position and orientation detection means for detecting a position and posture of the first endoscope in a predetermined coordinate system; and the arm in the predetermined coordinate system.
  • a second position and orientation detection means for detecting the position and orientation of the second endoscope; and a control device for controlling the operation of the arm, the control device as a control procedure for operating the arm,
  • Position and figure of the first endoscope in a predetermined coordinate system Is obtained from the first position and orientation detection means, and the arm is operated so that the first imaging unit moves in a direction in which the optical axis of the first imaging unit coincides with the optical axis of the second imaging unit.
  • the control device repeatedly executes the first calculation step, the motion amount calculation step, and the instruction step in this order.
  • the position and direction of the optical axis of the second imaging unit in the first calculation step executed immediately before the instruction step that has already been executed and immediately after the instruction step If the position and direction of the optical axis of the second imaging unit in the first calculation step have a deviation amount equal to or greater than a predetermined threshold, the optical axis of the first imaging unit moves equal to the deviation amount.
  • the movement amount may be calculated as described above.
  • the control device repeatedly executes the first calculation step, the motion amount calculation step, and the instruction step in this order. If the motion amount calculated in the motion amount calculation step executed after the instruction step that has already been executed does not exceed a predetermined threshold, the execution of the instruction step is skipped and the first calculation step is performed. When the operation amount exceeds the predetermined threshold, the instruction step may be executed.
  • the first endoscope in the endoscope system according to the first aspect, includes a joint portion operable to change a visual field direction of the first imaging unit.
  • a shaft portion connected to the joint portion, and an adapter for connecting the shaft portion to the arm, and the control device includes an optical axis of the first imaging unit in the operation amount calculating step.
  • An operation instruction for operating the arm and the joint portion may be output based on the above.
  • the endoscope system according to the above aspect is attached to the second endoscope and moves the second endoscope.
  • a second arm operable is further provided.
  • the control device uses the first coordinate system in the predetermined coordinate system as a control procedure for operating the second arm.
  • the position and orientation of the endoscope are acquired from the first position and orientation detection means so that the first imaging unit moves in a direction in which the optical axis of the first imaging unit coincides with the optical axis of the second imaging unit.
  • a second operation amount calculating step for calculating an operation amount of the second arm for operating the second arm, and an operation instruction for operating the second arm based on the operation amount of the second arm are output.
  • An instruction step to perform the control It may control the operation of the second arm in the order.
  • the control device includes a restriction step for restricting a movable range of the arm in a control procedure, and the control device The arm is operated within the range of motion limited by the limiting step so that the angle formed by the vector of the optical axis of the first imaging unit and the vector of the optical axis of the second imaging unit is minimized. Also good.
  • the control device has a minimum distance between the optical axis of the first imaging unit and the optical axis of the second imaging unit.
  • the arm may be operated within the range of motion limited by the limiting step.
  • an endoscope system in which the images captured by the two endoscopes can be easily associated with each other and the burden on the operator is small.
  • FIG. 1 is a schematic overall view of an endoscope system according to a first embodiment of the present invention. It is a block diagram of the principal part of the endoscope system. It is a flowchart which shows the control procedure by the control apparatus of the same endoscope system. It is a figure for demonstrating the effect
  • FIG. 1 is a schematic overall view of an endoscope system according to the present embodiment.
  • FIG. 2 is a block diagram of a main part of the endoscope system.
  • FIG. 3 is a flowchart showing a control procedure by the control device of the endoscope system.
  • 4 to 7 are diagrams for explaining the operation of the endoscope system.
  • An endoscope system 1 is a medical system used for surgery or the like. As shown in FIG. 1, the endoscope system 1 includes a first endoscope 10, an arm 20, a first image processing device 25, a first sub monitor 26, a second endoscope 30, and a second endoscope. An image processing device 36, a second sub-monitor 37, a position / orientation detection device 41, a main monitor 45, and a control device 50 are provided.
  • the first endoscope 10 acquires an image (narrow angle image) in which a narrow region including a treatment site is an imaging target (narrow angle image), and supplies the first sub monitor and the main monitor 45. It is an endoscope for displaying.
  • the first endoscope 10 is connected to the first image processing device 25 via a signal line SL1. Furthermore, the first image processing device 25 and the control device 50 are connected via a signal line SL1 ′.
  • the first endoscope 10 has an insertion part 11 and a drive part 15.
  • the insertion portion 11 is a long portion that is inserted from outside the body into the body of a patient to be treated.
  • the insertion unit 11 includes an imaging unit 12 (first imaging unit), a joint unit 13, and a shaft unit 14.
  • the imaging unit 12 has, for example, an image sensor and an objective optical system (both not shown) in order to acquire an image with the treatment site as an imaging target site.
  • the image sensor of the imaging unit 12 can acquire, for example, a bright field image of the imaging target part.
  • the objective optical system of the imaging unit 12 has a predetermined optical axis L1.
  • the center of the visual field of the image captured by the imaging unit 12 is the position of the optical axis L1 of the imaging unit 12.
  • the image captured by the imaging unit 12 is transmitted to the first image processing device 25 described later, and further transmitted to the control device 50.
  • the angle of view of the imaging unit 12 of the first endoscope 10 may be the same as the angle of view of the imaging unit 32 of the second endoscope 30 or different from the angle of view of the imaging unit 32 of the second endoscope 30. May be.
  • the narrow-angle image captured by the imaging unit 12 of the first endoscope 10 is an image in which a part of the image captured by the imaging unit 32 of the second endoscope 30 is an imaging field of view.
  • the angle of view of the imaging unit 12 of the first endoscope 10 is not limited to mean that the angle of view of the imaging unit 32 of the second endoscope 30 is narrower.
  • the joint portion 13 has a plurality of joint elements connected to each other via a rotating shaft.
  • the joint portion 13 can be deformed so as to move the imaging unit 12 with respect to the shaft portion 14.
  • the joint unit 13 is operable to adjust the position of the imaging unit 12 and the direction of the imaging field of view.
  • An operation wire 18 is connected to the joint portion 13 for transmitting a force for operating the joint portion 13.
  • the shaft portion 14 has a hard cylindrical shape in order to guide the imaging unit 12 to the vicinity of the treatment site in the body. Inside the shaft portion 14, there are wired a signal line (not shown) from the imaging unit 12, an operation wire 18 for connecting the joint portion 13 to a driving portion 15 described later in order to operate the joint portion 13, and the like. Yes.
  • the drive unit 15 is disposed at the end of the insertion unit 11 opposite to the imaging unit 12 (referred to as the base end of the first endoscope 10 in this embodiment).
  • the drive unit 15 is fixed to the shaft unit 14.
  • a signal line (not shown) extending from the imaging unit 12 and an operation wire 18 extending from the joint unit 13 are wired.
  • the drive unit 15 includes an actuator 16 connected to the operation wire 18 for moving the operation wire 18, and an encoder 17 for detecting an operation amount of the actuator 16.
  • the actuator 16 provided in the drive unit 15 operates according to control by the control device 50.
  • the drive unit 15 also has a function as an adapter for attaching the first endoscope 10 to the arm 20.
  • the first endoscope 10 of the present embodiment does not need to have a structure for incising or suturing the treatment site directly.
  • the insertion unit 11 may have a treatment tool (not shown) for incising or suturing the treatment site at the distal end.
  • the insertion part 11 may have a channel for inserting a known endoscope treatment tool.
  • the first endoscope 10 may be configured to rotate the imaging field of the imaging unit 12 with the center of the field of view as the rotation center (for example, with the optical axis L1 of the imaging unit 12 as the rotation center).
  • the rotation of the imaging field of view may be a configuration in which the imaging unit 12 is mechanically rotated, or a configuration in which an image acquired by the image sensor of the imaging unit 12 is rotated by image processing.
  • the arm 20 holds the first endoscope 10 in a desired position and posture, and moves the first endoscope 10 in a desired direction.
  • the arm 20 is connected to the control device 50 via the signal line SL2.
  • the arm 20 includes a link portion 21, a joint portion 22, an actuator 23, and an encoder 24.
  • the link part 21 includes a plurality of proximal side link parts 21 a having a detachable structure to which the drive part 15 of the first endoscope 10 can be attached and the joint elements constituting the joint part 22. And a link portion 21b.
  • the joint portion 22 connects two adjacent link portions 21 so as to be bendable, for example. Moreover, some of each joint part 22 which comprises the several joint part 22 may connect two adjacent link parts 21 so that each centerline may make coaxial, so that rotation is possible.
  • the upper limit of the degree of freedom of the arm 20 is not particularly limited. The arrangement, the number, and the like of the joint portions 22 may be such that the arm 20 can be provided with the minimum degree of freedom necessary for controlling the position and posture of the first endoscope 10.
  • the actuator 23 is disposed in the joint portion 22.
  • the actuator 23 operates the joint portion 22 according to the operation control by the control device 50.
  • the encoder 24 is disposed at the plurality of joint portions 22.
  • encoders 24 are arranged at all joint portions 22 included in the arm 20.
  • the encoder 24 is electrically connected to a control device 50 described later.
  • the encoder 24 outputs the movement amount of each joint portion 22 to the control device 50.
  • the encoder 24 may output a signal indicating the absolute angle of each joint portion 22 to the control device 50.
  • the first image processing device 25 is connected to the imaging unit 12 of the first endoscope 10.
  • the first image processing device 25 receives image information captured by the image capturing unit 12 of the first endoscope 10 from the image capturing unit 12 and outputs the image information to the first sub-monitor 26 as a video signal. Is output to the image control unit 52.
  • the first sub-monitor 26 displays a video based on the video signal output from the first image processing device 25.
  • the first sub-monitor 26 can display a narrow-angle image including the treatment site.
  • the second endoscope 30 acquires an image (wide-angle image) that captures a wide area including the treatment site and displays the acquired image on the second sub-monitor and the main monitor 45. It is an endoscope for.
  • the second endoscope 30 is connected to the second image processing device 36 via a signal line SL3. Further, the second image processing device 36 and the control device 50 are connected via a signal line SL3 ′.
  • the second endoscope 30 has an insertion part 31 and an operation part 34.
  • the insertion portion 31 of the second endoscope 30 has an elongated rod shape as a whole.
  • the insertion part 31 of the second endoscope 30 in the present embodiment has, for example, a hard property. That is, in the present embodiment, the second endoscope 30 is a rigid endoscope.
  • the insertion part 31 of the second endoscope 30 has an imaging part 32 (second imaging part) and a shaft part 33.
  • the imaging unit 32 of the second endoscope 30 has, for example, an image sensor and an objective optical system (both not shown) in order to acquire an image of the treatment site.
  • the image sensor of the imaging unit 32 of the second endoscope 30 can acquire, for example, a bright field image of the imaging target part.
  • the objective optical system of the imaging unit 32 has a predetermined optical axis L2.
  • the center of the visual field of the image captured by the imaging unit 32 is the position of the optical axis L ⁇ b> 2 of the imaging unit 32.
  • the image captured by the imaging unit 32 of the second endoscope 30 is transmitted to the control device 50 described later.
  • the angle of view of the imaging unit 32 of the second endoscope 30 may be the same as the angle of view of the imaging unit 12 of the first endoscope 10 or different from the angle of view of the imaging unit 12 of the first endoscope 10. May be.
  • the wide-angle image captured by the imaging unit 32 of the second endoscope 30 is an image that captures a wider range than the imaging field of view of the imaging unit 12 of the first endoscope 10.
  • the angle of view of the imaging unit 32 of the endoscope 30 is not limited to that which means that the angle of view of the imaging unit 12 of the first endoscope 10 is wider.
  • the shaft portion 33 of the second endoscope 30 has a hard cylindrical shape in order to guide the imaging unit 32 of the second endoscope 30 to the vicinity of the treatment site in the body.
  • a signal line (not shown) for connecting the imaging unit 32 of the second endoscope 30 to a control device 50 described later is wired inside the shaft portion 33 of the second endoscope 30.
  • the operation unit 34 is an end of the insertion unit 31 of the second endoscope 30 opposite to the imaging unit 32 side (second endoscope in the present embodiment). (Referred to as the base end of the mirror 30).
  • the operation part 34 has a grip part 35.
  • the grip part 35 is a part that can be gripped by an operator who operates the second endoscope 30.
  • the operator can move the entire insertion portion 31 of the second endoscope 30 by gripping and moving the grip portion 35.
  • the imaging field of view of the imaging unit 32 of the second endoscope 30 can be moved by the operator holding and moving the holding unit 35.
  • a viewpoint changing switch 39 and a field matching switch 40 which will be described later, may be disposed on the grip portion 35.
  • the configuration of the second endoscope 30 is not limited to the above configuration.
  • the second endoscope 30 may be a rigid endoscope or a flexible endoscope having the imaging unit 32 for capturing the wide-angle image.
  • the direction of the visual field of the imaging unit 32 of the second endoscope 30 is fixed to the insertion unit 31 of the second endoscope 30 or can be grasped by the control device 50, It is not limited to the configuration.
  • the second image processing device 36 is connected to the imaging unit 32 of the second endoscope 30.
  • the second image processing device 36 receives the image information captured by the image capturing unit 32 of the second endoscope 30 from the image capturing unit 32 and outputs the image information to the second sub-monitor 37 as well as the image information. Is output to the image control unit 52.
  • the second sub monitor 37 displays a video based on the video signal output from the second image processing device 36.
  • the second sub-monitor 37 can display a wide-angle image including the treatment site.
  • the arm input unit 38 is connected to the drive control unit 53 of the control device 50.
  • the arm input unit 38 outputs a predetermined operation signal for operating the arm 20 to the drive control unit 53 by being operated by a scopist, for example.
  • the viewpoint change switch 39 is electrically connected to the image control unit 52 of the control device 50.
  • the viewpoint change switch 39 is a change-over switch that selects one desired image from the narrow-angle image obtained by the first endoscope 10 and the wide-angle image obtained by the second endoscope 30.
  • the viewpoint change switch 39 is a push button switch, and switches the image displayed on the main monitor 45 to a wide-angle image or a narrow-angle image each time the viewpoint change switch 39 is pressed.
  • the viewpoint change switch 39 is a position that is easy for a person (for example, a surgeon) who performs treatment to a treatment site, a person (for example, a scoopist) who operates the first endoscope 10 or the second endoscope 30, etc. Can be arranged.
  • the viewpoint change switch 39 can be attached to a medical instrument operated by a surgeon who performs treatment on a treatment site. Further, the viewpoint change switch 39 may be disposed in the vicinity of the grip portion 35 of the second endoscope 30 (see FIG. 1). By changing the viewpoint change switch 39 of the present embodiment at a position that can be easily operated by an operator who operates the second endoscope 30, the viewpoint can be changed during the treatment using the second endoscope 30. Becomes easy.
  • the visual field alignment switch 40 is electrically connected to the drive control unit 53 of the control device 50.
  • the field matching switch 40 operates the first endoscope 10 so that the field center of the narrow-angle image obtained by the first endoscope 10 matches the field center of the wide-angle image obtained by the second endoscope 30. This is a switch for performing an input to the drive control unit 53 of the control device 50.
  • the field alignment switch 40 is a position that is easy for a person (for example, a surgeon) who performs treatment to a treatment site, a person (for example, a scoopist) who operates the first endoscope 10 or the second endoscope 30, etc. Can be arranged.
  • the field matching switch 40 is disposed in the vicinity of the grip portion 35 of the second endoscope 30.
  • the field-of-view switch 40 is a push button switch, and each time the field-of-view switch 40 is pressed, the controller 50 controls the first endoscope 10 so that the center of the narrow-angle image matches the center of the wide-angle image. The movement is started.
  • the visual field alignment switch 40 according to the present embodiment is arranged at a position that can be easily operated by an operator who operates the second endoscope 30, so that the field of view of the narrow-angle image during the treatment using the second endoscope 30. The center and the visual field center of the wide-angle image can be easily matched.
  • the position / orientation detection device 41 detects the positions and orientations of the first endoscope 10 and the second endoscope 30 in a space such as an operating room in which the endoscope system 1 according to the present embodiment is installed.
  • the position / orientation detection device 41 includes a first marker 42, a second marker 43, and a camera 44.
  • the position / orientation detection device 41 is connected to a control device 50 described later.
  • the first marker 42 is disposed on the drive unit 15 of the first endoscope 10.
  • the first marker 42 is a marker different from the second marker 43 so that the first endoscope 10 and the second endoscope 30 can be distinguished.
  • the configuration of the first marker 42 is not particularly limited as long as it can be imaged by the camera 44 and can be recognized by the control device 50.
  • the first marker 42 may have a shape unique to the first marker 42, have a pattern with a predetermined shape, or emit predetermined light to the camera 44. Absent.
  • the second marker 43 is disposed on the operation unit 34 of the second endoscope 30.
  • the second marker 43 is a marker different from the first marker 42 so that the first endoscope 10 and the second endoscope 30 can be distinguished.
  • the configuration of the second marker 43 is not particularly limited as long as it can be imaged by the camera 44 and can be recognized by the control device 50.
  • the second marker 43 may have a shape unique to the second marker 43, have a pattern with a predetermined shape, or emit predetermined light to the camera 44. Absent.
  • first marker 42 and the second marker 43 are indistinguishable from each other, they are associated with each other when the endoscope system 1 is used, and the positions and orientations of the respective markers are always used when the endoscope system 1 is used. If it comes to track.
  • the camera 44 captures an image including the first marker 42 and the second marker 43 and transmits the image to the control device 50.
  • the camera 44 acquires a plurality of images having different fields of view.
  • the camera 44 includes two imaging devices having different fields of view.
  • the camera 44 is fixed to a part of a space such as an operating room where the endoscope system 1 is arranged.
  • the camera 44 is fixed to a part of the operating room and an image including the first marker 42 and the second marker 43 is captured, so that the first coordinate system in the predetermined coordinate system (reference coordinate system) with the position of the camera 44 as a reference.
  • the positions and postures of the one marker 42 and the second marker 43 can be detected.
  • the main monitor 45 is connected to the control device 50 through a signal line SL4.
  • the main monitor 45 can display an image captured by the imaging unit 12 of the first endoscope 10 and an image captured by the imaging unit 32 of the second endoscope 30 via the image control unit 52.
  • the main monitor 45 selects one image selected by the switching input by the viewpoint change switch 39 from the image captured by the first endoscope 10 and the image captured by the second endoscope 30. indicate.
  • the display state on the main monitor 45 is controlled by the image control unit 52 of the control device 50.
  • the specific configuration of the main monitor 45 is not particularly limited. For example, as the main monitor 45, a known display system that displays an image based on an analog or digital video signal may be appropriately selected.
  • the control device 50 controls the arm 20 and the second endoscope 30. Furthermore, the control device 50 causes the main monitor 45 to display an image from the first endoscope 10, an image from the second endoscope 30, and the like.
  • the control device 50 illustrated in FIGS. 1 and 2 includes a position calculation unit 51, an image control unit 52, and a drive control unit 53.
  • the position calculation unit 51 receives an image signal from the camera 44 and calculates the positions of the first marker 42 and the second marker 43.
  • the position calculation unit 51 stores the positions and orientations of the first marker 42 and the second marker 43 in a predetermined coordinate system (reference coordinate system) based on the position of the camera 44 as coordinate information.
  • the image control unit 52 is connected to the first image processing device 25, the second image processing device 36, the drive control unit 53, the viewpoint change switch 39, and the main monitor 45. Furthermore, the image control unit 52 uses a predetermined information for operating the drive control unit 53 using the image information output from the first image processing device 25 and the image information output from the second image processing device 36. Information is generated and output to the drive control unit 53. Further, the image control unit 52 acquires an image captured by the first endoscope 10 and an image captured by the second endoscope 30, and outputs an image corresponding to the switching input by the viewpoint change switch 39 to the main monitor 45. Is output.
  • the main monitor 45 may be composed of two screens, a main screen and a sub screen, and the image control unit 52 displays an image designated by the switching input by the viewpoint change switch 39 on the main screen, and by the switching input. An image that has not been selected may be displayed on the sub-screen.
  • the drive control unit 53 is connected to the position calculation unit 51. Further, the drive control unit 53 is connected to the actuator 16 and encoder 17 of the first endoscope 10 and the actuator 23 and encoder 24 of the arm 20. Further, the drive control unit 53 is connected to the arm input unit 38. The drive control unit 53 controls the operation of the joint unit 13 of the first endoscope 10 and the arm 20 based on an operation signal generated by an operation by the operator on the arm input unit 38.
  • the position calculation unit 51 recognizes the first marker 42 from the image captured by the camera 44 of the position and orientation detection device 41.
  • the image transmitted from the camera 44 to the position calculation unit 51 includes a plurality of images having different fields of view.
  • the position calculation unit 51 determines the position of the first marker 42 in the reference coordinate system and the position of the first marker 42 from the plurality of images. Recognize posture.
  • the first marker 42 is disposed in the drive unit 15 of the first endoscope 10, and the position of the first endoscope 10 with respect to the position of the first marker 42 is stored in the control device 50 in advance.
  • the position calculation unit 51 can recognize the position and posture of the first endoscope 10 based on the position and posture of the first marker 42.
  • the position calculation unit 51 uses a predetermined reference point in the driving unit 15 of the first endoscope 10 as the position of the first endoscope 10. That is, the position calculation unit 51 recognizes the coordinates of the reference point of the first endoscope 10 in the reference coordinate system as the position of the first endoscope 10. Further, the position calculation unit 51 uses the direction of the first marker 42 in the reference coordinate system as the posture of the first endoscope 10.
  • the drive control unit 53 refers to the encoder 17 provided in the joint portion 13 of the first endoscope 10, so that the distal end portion of the joint portion 13 is added to the position and posture of the first endoscope 10.
  • the position and orientation of the imaging unit 12 arranged in the are recognized.
  • the drive control unit 53 recognizes some coordinates of the imaging unit 12 in the reference coordinate system as the position of the imaging unit 12. Further, the drive control unit 53 recognizes the direction of the optical axis in the reference coordinate system as the posture of the imaging unit 12.
  • the position / orientation detection device 41, the position calculation unit 51, and the drive control unit 53 constitute a first position / orientation detection unit that detects the position and orientation of the first endoscope 10 in the reference coordinate system.
  • the control apparatus 50 can detect the position and attitude
  • the position calculation unit 51 recognizes the second marker 43 from the image captured by the camera 44 of the position and orientation detection device 41.
  • the position calculation unit 51 recognizes the position and orientation of the second marker 43 in the same manner as the recognition of the position and orientation of the first marker 42.
  • the drive control unit 53 recognizes some coordinates of the imaging unit 32 of the second endoscope 30 in the reference coordinate system as the position of the imaging unit 32 of the second endoscope 30. Further, the drive control unit 53 recognizes the direction of the optical axis in the reference coordinate system as the posture of the imaging unit 32 of the second endoscope 30.
  • the position / orientation detection device 41, the position calculation unit 51, and the drive control unit 53 constitute second position / orientation detection means for detecting the position and orientation of the second endoscope 30 in the reference coordinate system.
  • the control device 50 can detect the position and posture of the optical axis L2 of the second endoscope 30.
  • the drive control unit 53 receives an input to the arm input unit 38 disposed on the grip unit 35 of the second endoscope 30 and operates the first endoscope 10 and the arm 20 based on this input. Further, the drive control unit 53 receives an input to the visual field alignment switch 40 disposed on the grip unit 35 of the second endoscope 30 and operates the first endoscope 10 and the arm 20 based on this input. As shown in FIG. 3, the drive control unit 53 that has received an input to the field matching switch 40 operates the first endoscope 10 and the arm 20 as shown in each step from step S1 to step S4.
  • the drive control unit 53 recognizes the position and orientation of the first endoscope 10 in the reference coordinate system and the position and orientation of the optical axis L1 of the first endoscope 10 in the reference coordinate system (first calculation step). , Step S1). Further, the drive control unit 53 recognizes the position and orientation of the optical axis L2 of the second endoscope 30 in the reference coordinate system (second calculation step, step S2). In addition, step S2 may be performed before said step S1, and may be performed after said step S1.
  • step S1 and step S2 the drive control unit 53 operates the first endoscope 10 and the arm 20 so as to approach the position and posture of the optical axis L2 of the second endoscope 30 recognized in step S2. Is calculated (operation amount calculation step, step S3).
  • step S ⁇ b> 3 the drive control unit 53 moves the first endoscope 10 at a position shifted by a predetermined distance toward the front side in the optical axis direction with respect to the coordinates indicating the position of the optical axis L ⁇ b> 2 of the second endoscope 30. Is set to the movement target position of the optical axis L1.
  • step S3 the drive control unit 53 sets the movement target position of the optical axis L1 of the first endoscope 10 in a region between the distal end of the imaging unit 32 of the second endoscope 30 and the imaging target site.
  • the movement target position set by the drive control unit 53 in step S3 is the coordinates of one point on the optical axis L2 of the second endoscope 30 in the reference coordinate system.
  • the accuracy of the movement target position set by the drive control unit 53 in step S3 may not be so strict. This is because if the optical axis L1 of the first endoscope 10 and the optical axis L2 of the second endoscope 30 are parallel, the center of the field of view on the image is practical even if the distance between the optical axes is somewhat apart. This is because the difference is within the range of no problem.
  • the movement target position set by the drive control unit 53 in step S3 may be set as an area within a certain range in order to allow a certain amount of error. For example, as shown in FIG.
  • the movement target position X1 set by the drive control unit 53 in step S3 is centered on the optical axis along a plane orthogonal to the optical axis L2 of the second endoscope 30 in the reference coordinate system.
  • the coordinates in a circle C1 having a predetermined diameter may be used.
  • the movement target position set by the drive control unit 53 in step S3 may be coordinates in a sphere centered on the coordinates of the position of the optical axis L2 of the second endoscope 30 in the reference coordinate system.
  • the movement target position set by the drive control unit 53 in step S3 may be coordinates in a cylinder with the optical axis L2 of the second endoscope 30 as the center line in the reference coordinate system.
  • the second endoscope 30 since an error can be allowed when determining that the position of the optical axis L1 of the first endoscope 10 has reached the optical axis L2 of the second endoscope 30, for example, the second endoscope 30 is operated. Therefore, it is possible to prevent the movement of the optical axis L1 of the first endoscope 10 from being completed due to the movement of the optical axis L2 of the second endoscope 30 due to the hand shake of the operator who performs the operation. Further, in this case, when both the image from the first endoscope 10 and the image from the second endoscope 30 are displayed on the main monitor 45, due to camera shake when operating the second endoscope 30. It is also possible to reduce image disturbance due to the imaging unit 12 of the first endoscope 10 continuing to move following the optical axis L2 of the second endoscope 30.
  • step S3 the drive control unit 53 sets a movement path from the current position of the optical axis L1 of the first endoscope 10 to the movement target position.
  • the method for setting the movement path from the current position of the optical axis L1 of the first endoscope 10 to the movement target position is not particularly limited as long as there is no obstacle in the movement path.
  • the drive control unit 53 moves the first endoscope 10 so that the optical axis moves along the movement path. Twenty movement amounts are calculated.
  • step S3 the drive control unit 53 operates the target direction of the imaging unit 12 so that the direction of the optical axis L1 of the first endoscope 10 matches the direction of the optical axis L2 of the second endoscope 30.
  • the drive control unit 53 determines that the direction of the optical axis L1 of the first endoscope 10 when the optical axis L1 of the first endoscope 10 is positioned at the movement target position is the second inner position.
  • the joint drive amount of the joint unit 13 for rotating the imaging unit 12 around the coordinates indicating the position of the optical axis L1 of the first endoscope 10 so as to be parallel to the direction of the optical axis L2 of the endoscope 30 is set. calculate.
  • step S3 the operation target direction of the imaging unit 12 set by the drive control unit 53 in step S3 may be set as an area within a certain range in order to allow a certain amount of error.
  • step S ⁇ b> 3 the drive control unit 53 calculates the operation amount of the arm 20 and the operation amount of the joint unit 13. Step S3 is complete
  • step S4 an operation instruction for operating the arm 20 and the joint unit 13 is output to the arm 20 and the joint unit 13 based on the operation amount of the arm 20 and the operation amount of the joint unit 13 calculated in step S3.
  • This is an instruction step.
  • the drive control unit 53 operates the arm 20 and the joint unit 13 by outputting an operation instruction to the arm 20 and the joint unit 13.
  • the drive control unit 53 first outputs an operation instruction to the arm 20, so that the position of the optical axis L1 of the first endoscope 10 is on the optical axis L2 of the second endoscope 30. The first endoscope 10 is moved until the movement target position is reached.
  • the drive control unit 53 outputs an operation instruction to the joint unit 13, so that the optical axis L ⁇ b> 1 of the first endoscope 10 is coaxial with the optical axis L ⁇ b> 2 of the second endoscope 30.
  • the imaging unit 12 of the first endoscope 10 is rotated on the optical axis L2 of the endoscope 30.
  • the drive control unit 53 uses inverse kinematics to calculate the motion amount of each degree of freedom of the joint unit 13 and the arm 20 in order to move the arm 20 and the joint unit 13 in the motion direction calculated in step S3. To determine the control amount of each actuator. Based on this control amount, the drive control unit 53 outputs a drive signal.
  • step S4 ends.
  • the drive control unit 53 corresponds to the input to the visual field alignment switch 40 and the optical axis L1 of the first endoscope 10 and the second endoscope.
  • the first endoscope 10 and the arm 20 are operated so that the 30 optical axes L2 are coaxial.
  • the operation of the endoscope system 1 will be described together with the operation during use of the endoscope system 1.
  • the first endoscope 10 and the second endoscope 30 are inserted into the body. That is, the insertion part 11 of the first endoscope 10 and the insertion part 31 of the second endoscope 30 are inserted into the body through, for example, a trocar, and as shown in FIG. 4, the imaging part 32 of the second endoscope 30. Is guided to the vicinity of the treatment site in the body, and the imaging unit 12 of the first endoscope 10 is guided to a position closer to the treatment endoscope near the second endoscope 30.
  • the first endoscope 10 and the second endoscope 30 may have optical functions such as zoom and macro, but an explanation in consideration of these optical functions will be described in this specification. Omitted.
  • the operator For example, a surgeon or a scoopist
  • the operator uses the second endoscope 30 to observe a wide area including the treatment site from a bird's-eye view.
  • the operator mainly operates the second endoscope 30.
  • the operator grasps the situation near the treatment site by moving the operation unit 34 of the second endoscope 30 and observing a desired site.
  • the operator After grasping the status of the treatment site, the operator maintains the second endoscope 30 so that the treatment site is positioned at the center of the visual field of the image, and further performs input to the visual field matching switch 40. Then, the drive control part 53 controls the arm 20 and the 1st endoscope 10 according to the control procedure from said step S1 to step S4.
  • the imaging unit 12 of the first endoscope 10 enters the field of view of the second endoscope 30. Furthermore, as shown in FIG. 6, when the optical axis L1 of the imaging unit 12 of the first endoscope 10 reaches the circle C1 centered on the point X1 on the optical axis L2 of the second endoscope 30, The movement of the imaging unit 12 stops, and then, as shown in FIG. 7, the first endoscope so that the optical axis L1 of the first endoscope 10 is coaxial with the optical axis L2 of the second endoscope 30. The orientation of the imaging unit 12 of the mirror 10 changes.
  • the second endoscope In the state where the imaging unit 12 of the first endoscope 10 is arranged so that the optical axis L1 of the first endoscope 10 is coaxial with the optical axis L2 of the second endoscope 30, the second endoscope The imaging unit 12 of the first endoscope 10 is located at the center of the visual field of the image captured by the 30 imaging units 32. The operator can grasp that the imaging unit 12 of the first endoscope 10 is arranged in the vicinity of the treatment site from the wide-angle image captured by the imaging unit 32 of the second endoscope 30. In a state where the imaging unit 12 of the first endoscope 10 is in the vicinity of the treatment site, the operator inputs to the viewpoint change switch 39.
  • the image control unit 52 of the control device 50 captures the image displayed on the main monitor 45 from the image captured by the imaging unit 32 of the second endoscope 30 by the imaging unit 12 of the first endoscope 10. Switch to the selected image. Since the optical axis L1 of the first endoscope 10 and the optical axis L2 of the second endoscope 30 are coaxial, the field center in the image captured by the imaging unit 12 of the first endoscope 10 is the first This coincides with the center of the visual field in the image captured by the imaging unit 32 of the two endoscopes 30.
  • the display on the main monitor 45 includes a part including the center of the field of view of the wide-angle image.
  • the image looks like an enlarged (zoomed-in) image.
  • the image captured by the imaging unit 12 of the first endoscope 10 is a narrow-angle image in which the treatment site is captured in the field of view. The operator can perform treatment on the treatment site while viewing the image captured by the imaging unit 12 of the first endoscope 10.
  • the operator can use the viewpoint change switch 39 to display an image to be displayed on the main monitor 45 from the narrow-angle image captured by the imaging unit 12 of the first endoscope 10 as necessary. Switching to a wide-angle image captured by the imaging unit 32 of the mirror 30 is possible. This can be performed, for example, when the state around the treatment site is confirmed during the treatment using the narrow-angle image.
  • the imaging unit 32 of the second endoscope 30 uses a wide area including the imaging field of view of the first endoscope 10 as the imaging field. For this reason, the image displayed on the main monitor 45 is switched from the image captured by the image capturing unit 12 of the first endoscope 10 to the image captured by the image capturing unit 32 of the second endoscope 30.
  • the display is an image in which the field of view of the narrow-angle image is widened (zoomed out).
  • the operator holds and moves the operation unit 34 of the second endoscope 30 in a state where the image captured by the imaging unit 32 of the second endoscope 30 is displayed on the main monitor 45.
  • the periphery of the treatment site can be observed.
  • the first endoscope 10 is stopped in a state where a narrow-angle image in which the treatment site is captured at the center of the visual field can be acquired.
  • a narrow-angle image including the treatment site is displayed on the main monitor 45 again.
  • the operator first refers to the position of the imaging unit 12 of the first endoscope 10 based on the image captured by the imaging unit 32 of the second endoscope 30, and the second endoscope 30.
  • the center of the visual field of the imaging unit 32 can be easily returned to the treatment site.
  • the operator inputs to the viewpoint change switch 39 and displays an image displayed on the main monitor 45.
  • the image captured by the image capturing unit 32 of the second endoscope 30 is switched to the image captured by the image capturing unit 12 of the first endoscope 10.
  • the first endoscope 10 is stopped in a state where a narrow-angle image in which the treatment site is captured at the center of the visual field can be acquired. That is, the first endoscope 10 maintains the positional relationship before switching from the narrow-angle image to the wide-angle image. For this reason, after the image displayed on the main monitor 45 is switched from the wide-angle image to the narrow-angle image, the operator can easily restart the treatment.
  • the positional relationship between the treatment site and the first endoscope 10 changes, and the first endoscope
  • the position and posture of the treatment site in the image captured by the ten imaging units 12 may change.
  • the operator inputs to the visual field alignment switch 40 in a state where the treatment site is captured at the center of the visual field of the imaging unit 32 of the second endoscope 30, and the imaging unit of the first endoscope 10 is input to the treatment site. 12 can be moved.
  • the drive control unit 53 controls the arm 20 and the first endoscope 10 according to the control procedure from step S1 to step S4 described above, so that the optical axis L2 of the imaging unit 32 of the second endoscope 30 and The optical axis L1 of the imaging unit 12 of the first endoscope 10 is coaxial.
  • the viewpoint change switch 39 By inputting to the viewpoint change switch 39 in this state, a narrow-angle image in which the treatment site is captured at the center of the visual field is displayed on the main monitor 45, and the new site can be treated.
  • the operator needs to perform another treatment using the new site as the treatment site.
  • the operator performs input to the field matching switch 40 in a state where the new part as a treatment part is captured at the center of the visual field of the imaging unit 32 of the second endoscope 30.
  • the imaging unit 12 of the first endoscope 10 can be moved to a new site.
  • the drive control unit 53 controls the arm 20 and the first endoscope 10 according to the control procedure from step S1 to step S4 described above, so that the optical axis L2 of the imaging unit 32 of the second endoscope 30 and The optical axis L1 of the imaging unit 12 of the first endoscope 10 is coaxial.
  • a narrow-angle image capturing a new part as a treatment part with the center of the visual field being displayed is displayed on the main monitor 45, so that the new part can be treated.
  • a surgeon who performs treatment using the endoscope system 1 according to the present embodiment can perform treatment while viewing a wide-angle image and a narrow-angle image including a treatment site by using the main monitor 45. it can.
  • the scopist who operates the first endoscope 10 and the second endoscope 30 of the endoscope system 1 according to the present embodiment uses the first sub-monitor 26 and the second sub-monitor 37 to perform the treatment by the surgeon. Can adjust the position and posture of each endoscope independently or in cooperation with a surgeon's procedure.
  • the optical axes of the two endoscopes are coaxial with each other, and a narrow-angle image and a wide-angle image are respectively displayed. Since the image is captured, when the image control unit 52 switches the image displayed on the main monitor 45 based on the input to the viewpoint change switch 39, an operation such as zooming out the narrow-angle image or zooming in on the wide-angle image A feeling can be obtained.
  • the burden on the operator is small.
  • a close visual field In a case where treatment is performed using a single endoscope in a conventional laparoscopic operation, during a procedure such as a peeling operation or blood vessel exposure in a close visual field (narrow angle visual field) In some cases, it is necessary to check the state of surrounding tissue or to check the running of the blood vessel currently being treated. In this case, the endoscope that has been brought close to is moved away, and observation is performed at a position where the whole can be seen, and the operation of returning the endoscope to the original position is performed after confirming the surrounding tissue and blood vessel running. .
  • the first close visual field was the operative field (visual field) created by instructing the scopist to be easy to treat. However, by observing the whole, the field of view created first has been destroyed.
  • the surgeon In order to reproduce the condition (field of view) that is easy to treat after the overall observation, the surgeon must repeat the instructions to the scopist, and it takes a long time to be able to return to the treatment. Accumulate. For this reason, in order to avoid this stress and fatigue, the surgeon wants to avoid losing his field of vision with a small amount of confirmation, and may give up even if he wants to confirm the whole.
  • the surrounding tissue can be appropriately confirmed using the wide-angle image, so there is a possibility that safer surgery can be performed.
  • FIG. 8 is a schematic diagram showing a part of the endoscope system according to the present embodiment.
  • FIG. 9 is a flowchart showing a control procedure by the control device of the endoscope system.
  • the endoscope system 2 includes distance measuring means for measuring the distance to the imaging target region, the first endoscope 10, the second endoscope 30, and The controller 66 is provided.
  • the imaging unit 60 having a configuration different from that of the imaging unit 12 disclosed in the first embodiment is replaced with the imaging unit 12 of the first embodiment.
  • the second endoscope 30 of the present embodiment includes an imaging unit 63 having a configuration different from that of the imaging unit 32 disclosed in the first embodiment, instead of the imaging unit 32 of the first embodiment. Yes.
  • the endoscope system 2 considers the distance to the imaging target based on the images captured by the imaging units 60 and 63 of the present embodiment, and the first endoscope 10 and the arm 20.
  • a control device 66 including a step of operating (see FIG. 1) in the operation procedure is provided instead of the control device 50 of the first embodiment.
  • the imaging unit 60 of the first endoscope 10 is different in configuration from the imaging unit 12 of the first endoscope 10 disclosed in the first embodiment in that it can acquire three-dimensional information of the imaging target part. ing.
  • the first endoscope 10 includes a left imaging unit 61 and a right imaging unit 62.
  • the left imaging unit 61 and the right imaging unit 62 capture a pair of images having parallax with respect to the same imaging target part.
  • the left imaging unit 61 and the right imaging unit 62 each independently have an image sensor and an objective optical system.
  • the image captured by the left imaging unit 61 and the image captured by the right imaging unit 62 are transmitted to the image control unit 67 of the control device 66 through a signal line (not shown) as in the first embodiment.
  • the imaging unit 63 of the second endoscope 30 includes a left imaging unit 64 and a right imaging unit 65, for example, like the imaging unit 60 of the first endoscope 10.
  • the image control unit 67 of the control device 66 measures the distance from the first endoscope 10 to the imaging target region based on the images acquired by the left imaging unit 61 and the right imaging unit 62 of the first endoscope 10, respectively.
  • the image control unit 67 of the control device 66 measures the distance to the imaging target region using the parallax of the images acquired by the left imaging unit 61 and the right imaging unit 62 of the first endoscope 10.
  • the image control unit 67 of the control device 66 is configured so that the distance from the second endoscope 30 to the imaging target region is based on the images acquired by the left imaging unit 64 and the right imaging unit 65 of the second endoscope 30, respectively. It has a function to measure. As an example, the image control unit 67 of the control device 66 measures the distance to the imaging target region using the parallax of the images acquired by the left imaging unit 64 and the right imaging unit 65 of the second endoscope 30.
  • the configuration for measuring the distance between the first endoscope 10 and its imaging target site or the distance between the second endoscope 30 and its imaging target site is not limited to the above configuration.
  • the first endoscope 10 and the second endoscope 30 may appropriately have a configuration such as a known laser range finder or infrared range finder.
  • control device 66 of the present embodiment for moving the first endoscope 10 and the arm 20 will be described focusing on portions different from the control procedure in the first embodiment.
  • the control device 66 includes the steps from Step S1 to Step S4 disclosed in the first embodiment in the control procedure.
  • the endoscope system 2 according to the present embodiment has a positional relationship in which the optical axes of the first endoscope 10 and the second endoscope 30 coincide with each other with the treatment site being centered on the visual field.
  • a narrow-angle image and a wide-angle image can be taken and either one or both of the narrow-angle image and the wide-angle image can be displayed on the main monitor 45.
  • the first endoscope 10 and the second endoscope 30 both follow the movement of the second endoscope 30 in a state where an image in which the treatment site is captured at the center of the visual field is captured.
  • the control procedure (see FIG. 9) when the control device 66 operates the first endoscope 10 and the arm 20 is different from that in the first embodiment.
  • the control device 66 and the imaging target region of the first endoscope 10 The distance from the endoscope 10 is measured. For example, the control device 66 determines a predetermined image in a set of two images captured by the imaging unit 60 of the first endoscope 10 from the coordinates of the position of the optical axis of the imaging unit 60 of the first endoscope 10 in the reference coordinate system. The distance to the visual field center of one of the images is calculated (first ranging step, step S11, see FIG. 9). Furthermore, the control device 66 measures the distance between the imaging target portion of the second endoscope 30 and the second endoscope 30.
  • control device 66 determines a predetermined value in a set of two images captured by the imaging unit 63 of the second endoscope 30 from the coordinates of the position of the optical axis of the imaging unit 63 of the second endoscope 30 in the reference coordinate system.
  • the distance to the center of the visual field of one of the images is calculated (second distance measuring step, step S12, see FIG. 9).
  • Step S12 may be executed before step S11 described above, or may be executed after step S11 described above.
  • the first endoscope 10 and the second endoscope 30 both grasp the treatment site at the center of the visual field, and the respective distances calculated in the above steps S11 and S12 are respectively , The distance between the first endoscope 10 and the treatment site, and the distance between the second endoscope 30 and the treatment site.
  • the operator moves the second endoscope 30 and starts treatment using a new part as a treatment part, the operator treats the center of the visual field of the image captured by the imaging unit 63 of the second endoscope 30 as a treatment.
  • the second endoscope 30 is operated so as to match with a new part to be a part. Thereafter, the operator performs input to the visual field alignment switch 40.
  • step S13 the control device 66 detects the position and orientation of the optical axis L2 of the second endoscope 30 in the same manner as in the first embodiment.
  • step S ⁇ b> 13 a new treatment site is located at the center of the field of view of the image captured by the imaging unit 63 of the second endoscope 30. For this reason, in step S13, the distance between the second endoscope 30 and the new treatment site is calculated.
  • step S13 the drive control unit 68 of the control device 66 operates the first endoscope 10 and the arm 20 so as to approach the position and posture of the optical axis L2 of the second endoscope 30 recognized in step S13.
  • the operation amount for this is calculated (see step S14, FIG. 9).
  • step S ⁇ b> 14 the drive control unit 68 sets the first endoscope 10 at a position shifted by a predetermined distance toward the front side in the optical axis direction with respect to the coordinates indicating the position of the optical axis L ⁇ b> 2 of the second endoscope 30. Is set to the movement target position of the optical axis L1.
  • the control device 66 controls the second endoscope 30. For example, a message that prompts the user to move the first endoscope 10 or the second endoscope 30 is displayed on the main monitor 45 so as to move away from the imaging target region. In this case, the control device 66 returns to the above step S12 to recalculate the distance between the second endoscope 30 and its imaging target site, and to set the coordinates indicating the position of the optical axis L2 of the second endoscope 30.
  • step S14 the control device 66 sets the movement target position of the optical axis L1 of the first endoscope 10 in the region between the tip of the imaging unit 63 of the second endoscope 30 and the imaging target site.
  • the drive control unit 68 of the control device 66 sets a movement path from the current position of the optical axis L1 of the first endoscope 10 to the movement target position.
  • the method for setting the movement path from the current position of the optical axis L1 of the first endoscope 10 to the movement target position takes into account that there are no obstacles in the movement path.
  • the travel route set in step S14 may be set by the same setting method as the travel route set in step S3 in the first embodiment. After the movement path of the optical axis L1 of the first endoscope 10 is set, the drive control unit 68 moves the first endoscope 10 so that the optical axis moves along the movement path. Twenty movement amounts are calculated.
  • step S14 the drive control unit 68 sets the first endoscope 10 so that the direction of the optical axis L1 of the first endoscope 10 coincides with the direction of the optical axis L2 of the second endoscope 30.
  • the operation target direction of the imaging unit 60 is set.
  • the drive control unit 68 determines that the direction of the optical axis L1 of the first endoscope 10 is the second inner point when the optical axis L1 of the first endoscope 10 is positioned at the movement target position.
  • Step S ⁇ b> 14 the drive control unit 68 calculates the movement amount of the arm 20 and the movement amount of the joint portion 13. Step S14 is completed now and it progresses to Step S15.
  • step S15 an operation instruction for operating the arm 20 and the joint portion 13 is output to the arm 20 and the joint portion 13 based on the movement amount of the arm 20 and the motion amount of the joint portion 13 calculated in step S14.
  • This is an instruction step (see FIG. 9).
  • the drive control unit 68 operates the arm 20 and the joint unit 13 by outputting an operation instruction to the arm 20 and the joint unit 13.
  • the drive control unit 68 first outputs an operation instruction to the arm 20, so that the position of the optical axis L1 of the first endoscope 10 is on the optical axis L2 of the second endoscope 30. The first endoscope 10 is moved until the movement target position is reached.
  • the drive control unit 68 outputs an operation instruction to the joint unit 13, so that the optical axis L ⁇ b> 1 of the first endoscope 10 is coaxial with the optical axis L ⁇ b> 2 of the second endoscope 30.
  • the imaging unit 60 of the first endoscope 10 is rotated on the optical axis L2 of the endoscope 30.
  • the drive control unit 68 corresponds to the input to the visual field alignment switch 40 and the optical axis L1 of the first endoscope 10 and the second endoscope.
  • the first endoscope 10 and the arm 20 are operated until the 30 optical axes L2 are coaxial.
  • the distance between the first endoscope 10 and the treatment site before the operation of the first endoscope 10 and the arm 20, and the first inner after the operation of the first endoscope 10 and the arm 20 are performed.
  • the distance between the endoscope 10 and the new treatment site is equal to each other. For this reason, when a new treatment is performed using the image captured by the imaging unit 60 of the first endoscope 10, it is easy to grasp the status of the treatment site.
  • the endoscope system 2 according to the present modification is configured such that a treatment path is used to move the imaging unit 60 of the first endoscope 10 from a certain treatment site (first site) to another new treatment site (second site).
  • the imaging unit 60 of the first endoscope 10 can be moved while keeping the distance to the tissue including the region constant.
  • the drive control unit 68 is based on the image captured by the imaging unit 60 of the first endoscope 10 in step S14 in the second embodiment.
  • the optical axis L1 of the first endoscope 10 is always such that the distance between the line connecting the first part and the second part, for example, by the shortest path, and the position of the optical axis L1 of the first endoscope 10 always coincide. Set the travel route.
  • the imaging unit 60 of the first endoscope 10 corresponds to the obstacle having the irregularities. It can move while avoiding obstacles with unevenness.
  • the drive control unit 68 removes an obstacle having irregularities on the line connecting the first site and the second site of the tissue from the imaging unit 60 of the first endoscope 10.
  • the first endoscope 10 may be moved while recognizing based on the image and avoiding an obstacle having unevenness.
  • the drive control unit 68 calculates the distance between the first endoscope 10 and the tissue by calculating the distance from the tissue while appropriately adjusting the direction of the imaging unit 60 in the process of moving the first endoscope 10.
  • the first endoscope 10 is moved while keeping the distance constant.
  • FIG. 10 is a schematic overall view showing the endoscope system according to the present embodiment.
  • FIG. 11 is a flowchart showing a control procedure by the control device 80 of the endoscope system.
  • the endoscope system 3 according to the present embodiment shown in FIG. 10 is provided with an arm (second arm 71) for moving the second endoscope 30, and the endoscope according to the first embodiment described above.
  • the configuration is different from the mirror system 1. That is, the endoscope system 3 according to the present embodiment includes the first endoscope 10 and the second endoscope 30 disclosed in the first embodiment, and the first for moving the first endoscope 10.
  • the arm 70, the 2nd arm 71 for moving the 2nd endoscope 30, and the control apparatus 80 are provided.
  • the configurations of the first endoscope 10 and the second endoscope 30 are the same as those in the first embodiment.
  • the first endoscope 10 and the second endoscope 30 in the present embodiment may be the same type machine having the same configuration.
  • the first arm 70 has the same configuration as the arm 20 disclosed in the first embodiment.
  • the second arm 71 has the same configuration as the arm 20 disclosed in the first embodiment. That is, the second arm 71 has a link part 72, a joint part 73, an actuator 74, and an encoder 75.
  • the operation part 34 of the second endoscope 30 can be attached to the link part 72 of the second arm 71.
  • the second arm 71 is connected to the control device 80.
  • the control device 80 included in the endoscope system 3 controls operations of the first endoscope 10, the second endoscope 30, the first arm 70, and the second arm 71.
  • the control device 80 can output an operation instruction for operating the first endoscope 10 and the first arm 70 to the first endoscope 10 and the arm 20 in the same manner as in the first embodiment.
  • the drive control unit 83 of the present embodiment outputs an operation instruction for operating the second arm 71 to the second arm 71.
  • the drive control unit 83 is provided in the second endoscope 30.
  • the above-described joint portion may be controlled.
  • the control device 80 of the endoscope system 3 includes a control procedure for operating the second arm 71. That is, the control device 80 is configured so that the first endoscope 10 is capturing a narrow-angle image including the treatment site, and the optical axis L1 of the imaging unit 12 of the first endoscope 10 and the second endoscope. The second endoscope 30 is moved by operating the second arm 71 so that the optical axis L2 of the 30 imaging units 32 is coaxial.
  • the control procedure for moving the second endoscope 30 includes the following steps from step S21 to step S24.
  • the control device 80 recognizes the position and direction of the optical axis L1 of the first endoscope 10 based on an image including the first marker 42 captured by the camera 44 of the position and orientation detection device 41 (first calculation).
  • Step, step S21 see FIG. 11).
  • the control device 80 recognizes the position and posture of the second endoscope 30 based on an image including the second marker 43 imaged by the camera 44 of the position and posture detection device 41 (second calculation step, step S22).
  • Step S22 may be executed before step S21 or may be executed after step S21.
  • the drive control unit 83 of the control device 80 causes the optical axis L1 of the imaging unit 12 of the first endoscope 10 and the optical axis L2 of the imaging unit 32 of the second endoscope 30 to be coaxial. Then, the operation amount of the second endoscope 30 is calculated (operation amount calculation step, step S23, see FIG. 11).
  • the drive control unit 83 of the control device 80 causes the position and direction of the optical axis L2 of the second endoscope 30 to pass through the position of the optical axis L1 of the first endoscope 10 in the reference coordinate system.
  • the amount of movement of the second endoscope 30 is calculated so as to be parallel to the direction of the optical axis.
  • the operation amount of the second arm 71 to which the second endoscope 30 is attached is calculated. Step S23 is ended now and it progresses to Step S24.
  • Step S24 is an instruction step for outputting, to the second arm 71, an operation instruction for operating the second arm 71 based on the operation amount calculated in step S23 (see FIG. 11).
  • the motion amount calculated in step S23 is converted into the motion amount of each joint portion 22 of the second arm 71, and the actuator 23 provided in each joint portion 22 of the second arm 71 respectively. It is output as an operation instruction. Accordingly, the second arm 71 operates in accordance with the operation instruction output in step S24, and the second arm 71 moves the second endoscope 30.
  • the control device 80 causes the optical axis of the imaging unit 32 of the second endoscope 30 to move.
  • the second endoscope 30 can be operated following the operation of the first endoscope 10 so that L2 is coaxial with the optical axis L1 of the imaging unit 12 of the first endoscope 10.
  • the control device 80 of the present embodiment mainly sets one of the first endoscope 10 and the second endoscope 30, and mainly sets the first endoscope 10 and the second endoscope 30.
  • the person who was not done is set as a subordinate.
  • the endoscope set mainly between the first endoscope 10 and the second endoscope 30 is an endoscope operated by an operator, and the first endoscope 10 and the second endoscope 30 are connected to each other.
  • the endoscope set as a slave is automatically controlled by the control device 80 so as to move following the movement of the optical axis L1 of the imaging unit 12 of the endoscope.
  • the master-slave relationship between the first endoscope 10 and the second endoscope 30 is set based on the input state to the viewpoint change switch 39.
  • the control device 80 when an input for making the image displayed on the main monitor 45 an image from the first endoscope 10 is made to the viewpoint change switch 39, 10 is set as the primary endoscope, and the second endoscope 30 is set as the secondary endoscope.
  • the control device 80 when the input to change the image displayed on the main monitor 45 to the image from the second endoscope 30 is made to the viewpoint change switch 39, the control device 80 performs the second endoscope.
  • the mirror 30 is set as the main endoscope
  • the first endoscope 10 is set as the slave endoscope. In this way, the control device 80 sets the master-slave relationship between the first endoscope 10 and the second endoscope 30 based on the input state to the viewpoint change switch 39.
  • control procedure by the control device 80 in the endoscope system 3 according to the present embodiment will be described in more detail together with the overall operation of the endoscope system 3.
  • the optical axis L1 of the first endoscope 10 is coaxial with the optical axis L2 of the second endoscope 30.
  • the imaging unit 12 of the first endoscope 10 can be moved. That is, the second endoscope 30 that captures a wide-angle image for observing the treatment site from a bird's-eye view is set as the main endoscope, and the second endoscope 30 that is the main endoscope is imaged.
  • the first endoscope 10, which is a slave endoscope is automatically controlled by the control device 80 so as to be coaxial with the optical axis L2 of the unit 32.
  • the imaging unit 12 of the first endoscope 10 that is the main endoscope moves, and the second endoscope 30 that is the slave endoscope captures the image of the first endoscope 10. It moves following the movement of the part 12.
  • the second endoscope 30 is the second endoscope 30.
  • the position and orientation are controlled by the control device 80 so that the optical axis L2 of the imaging unit 32 is coaxial with the optical axis L1 of the imaging unit 12 of the first endoscope 10. Therefore, when the operator inputs to the viewpoint change switch 39 and the image displayed on the main monitor 45 is switched from the narrow-angle image to the wide-angle image by the image control unit 82, the wide-angle image displayed on the main monitor 45 is The image is obtained by capturing the center of the field of view of the narrow-angle image as the center of the field of view.
  • the image displayed on the main monitor 45 is switched by the image control unit 82 from an image captured by the imaging unit 12 of the first endoscope 10 to an image captured by the imaging unit 32 of the second endoscope 30.
  • the display on the main monitor 45 becomes an image in which the field of view of the narrow-angle image is widened (zoomed out).
  • the control device 80 sets the second endoscope 30 as the main endoscope in a state where the image captured by the imaging unit 32 of the second endoscope 30 is displayed on the main monitor 45.
  • the first endoscope 10 is set as a slave endoscope. Furthermore, in a state where the image captured by the imaging unit 32 of the second endoscope 30 is displayed on the main monitor 45, the control device 80 does not change the state of the second endoscope 30 until there is an input to the field matching switch 40. The linkage between the movement and the movement of the first endoscope 10 is released.
  • the operator stops the imaging unit 12 of the first endoscope 10 and stops the second
  • the state around the treatment site can be observed by moving the endoscope 30.
  • the operator inputs to the viewpoint change switch 39 and switches the image displayed on the main monitor 45.
  • the main endoscope is switched from the second endoscope 30 to the first endoscope 10
  • the slave endoscope is switched from the first endoscope 10 to the second endoscope 30.
  • the second endoscope 30, which is a slave endoscope is automatically controlled by the control device 80.
  • the optical axis L2 of the imaging unit 32 of the second endoscope 30 is the optical axis of the imaging unit 12 of the first endoscope 10 as in steps S21 to S24 in the present embodiment.
  • the second arm 71 is controlled by the control device 80 so as to be coaxial with L1.
  • Treatment can be resumed while viewing the included narrow-angle image. That is, the operation of returning the second endoscope 30 after operating the second endoscope 30 to observe the state around the treatment site is performed after the image displayed on the main monitor 45 is switched. It is automatically controlled by the control device 80 at the ground.
  • the operator does not need to manually move the second endoscope 30 to return the second endoscope 30 to the original position, and the optical axis L2 of the second endoscope 30 is Since it automatically returns to a state where it is coaxial with the optical axis L1 of the endoscope 10, the center of the field of view of the narrow-angle image and the wide-angle image match when a wide-angle image is captured again.
  • the main endoscope is the second endoscope 30, and the first endoscope 10 which is the slave endoscope is automatically controlled by the control device 80.
  • the optical axis L1 of the imaging unit 12 of the first endoscope 10 is the same as that of the imaging unit 32 of the second endoscope 30 in the same manner as each step from step S1 to step S4 in the first embodiment.
  • the first arm 70 and the joint portion 13 are controlled by the control device 80 so as to be coaxial with the optical axis L2.
  • the control device 80 can be moved so as to follow the movement of the imaging unit 12 of the one endoscope 10.
  • the deviation between the visual field center of the imaging unit 12 of the first endoscope 10 and the visual field center of the imaging unit 32 of the second endoscope 30 is always eliminated. State.
  • the position of the second endoscope 30 is controlled by the control device 80. Is restored to its original position and posture by automatic control.
  • the control device 80 performs an operation for moving the second endoscope 30 so that the second endoscope 30 is in a state where the original treatment site is captured at the center of the visual field. The operation burden can be reduced.
  • the endoscope system 3 of this modification is different from the above embodiment in part of the control procedure for moving the second endoscope 30 following the movement of the imaging unit 12 of the first endoscope 10. Yes. That is, in the present modification, when the operator inputs to the visual field alignment switch 40, the optical axis L2 of the imaging unit 32 of the second endoscope 30 is the same as that of the imaging unit 12 of the first endoscope 10.
  • the control device 80 operates the second arm 71 so as to be coaxial with the optical axis L1.
  • step S21 and step S22 described above are triggered by the detection of the control device 80 that the operator has made an input to the visual field matching switch 40.
  • the second endoscope 30 when the imaging unit 12 of the first endoscope 10 is moved, the second endoscope 30 is not used until the operator inputs to the field matching switch 40. Do not move.
  • the second endoscope 30 in the state where only the narrow-angle image from the first endoscope 10 is displayed on the main monitor 45 when the treatment site is imaged using the imaging unit 12 of the first endoscope 10, the second There may be a case where it is not possible to grasp what posture the endoscope 30 is in the body, and it is not preferable to always move the second endoscope 30 automatically in the body.
  • the drive control unit 83 of the control device 80 controls the second arm 71 so as to prohibit the operation of the second arm 71 that moves the second endoscope 30 when there is no input to the visual field alignment switch 40.
  • the second arm 71 can be operated only when the second endoscope 30 needs to be moved by the second arm 71.
  • the drive control unit 83 of the control device 80 operates the second arm 71 so that the optical axis L2 of the imaging unit 32 is coaxial with the optical axis L1 of the imaging unit 12 of the first endoscope 10.
  • steps S21 and S22 the optical axis L1 of the imaging unit 12 of the first endoscope 10 and the optical axis L2 of the imaging unit 32 of the second endoscope 30 are separated by a predetermined distance or more. Is triggered by the detection of the drive control unit 83 via the position calculation unit 81.
  • the distance between the optical axis L1 of the imaging unit 12 of the first endoscope 10 and the optical axis L2 of the imaging unit 32 of the second endoscope 30 is the position and orientation of the first endoscope 10 in the reference coordinate system, and
  • the position calculation unit 81 or the drive control unit 83 calculates based on the position and orientation of the two endoscopes 30.
  • the control device 80 is configured such that a deviation of a predetermined threshold value or more occurs between the optical axis L1 of the imaging unit 12 of the first endoscope 10 and the optical axis L2 of the imaging unit 32 of the second endoscope 30. Only the second endoscope 30 is moved.
  • the control device 80 calculates the total sum of deviations accumulated from the most recent movement of the second endoscope 30 to the present as the movement amount to move the second endoscope 30. Furthermore, the control device 80 does not move the second endoscope 30 when the movement amount calculated based on the total sum of the deviations does not exceed a predetermined threshold, and based on the total sum of the deviations. When the calculated movement amount exceeds a predetermined threshold, the second endoscope 30 is moved so as to reduce the deviation. For this reason, the second endoscope 30 can be used as a fixed point camera for the vicinity and the periphery of the treatment site without moving during a fine operation when imaging the treatment site using the first endoscope 10.
  • the second endoscope 30 Since the second endoscope 30 is farther from the imaging target site than the first endoscope 10, the second endoscope 30 moves in accordance with the movement of the imaging unit 12 of the first endoscope 10. There is a lot to make. For this reason, when the second endoscope 30 is always moved following the fine movement of the imaging unit 12 of the first endoscope 10, the insertion unit 31 of the second endoscope 30 is moved through the trocar. It is considered that the body wall or the like is often moved, and the burden on the patient is increased. In the present modification, the visual field centers of the wide-angle image and the narrow-angle image do not always exactly match each other, but are within a shift that does not cause a problem in treatment and observation. For this reason, in this modification, the unnecessary operation
  • the first endoscope 10 and the second endoscope 30 are each provided with distance measuring means as in the second embodiment.
  • the configuration of the distance measuring means includes, for example, the left imaging unit 61 and the right imaging unit 62 as in the second embodiment, and captures a pair of images having parallax and uses them for distance measurement. It may be a thing. Similar to the second embodiment, the configuration of the distance measuring means is not particularly limited also in this modification.
  • the control device 80 uses the second endoscope. Instead of the distance between 30 and the treatment site, the distance between the imaging unit 12 of the first endoscope 10 and the second endoscope 30 may be calculated. In this case, based on the information on the distance between the first endoscope 10 and the treatment site and the information on the distance between the imaging unit 12 of the first endoscope 10 and the second endoscope 30, the treatment site and the second The distance from the endoscope 30 can be calculated.
  • the control device 80 calculates the distance between the second endoscope 30 and its imaging target part, and the control device calculates the distance between the second endoscope 30 and its imaging target part. 80 remains constant. As a result, when the second endoscope 30 is moved by the second arm 71 following the movement of the imaging unit 12 of the first endoscope 10, the second endoscope 30 is moved before and after the movement of the second endoscope 30. The distance between the endoscope 30 and its imaging target part does not change.
  • FIG. 12 is a schematic overall view showing the endoscope system according to the present embodiment.
  • 13 and 14 are schematic diagrams for explaining the operation of the endoscope system.
  • FIG. 15 is a flowchart illustrating a control procedure by the control device of the endoscope system.
  • the endoscope system 4 according to this embodiment shown in FIG. 12 is coaxial with the optical axis L2 of the imaging unit 32 of the second endoscope 30 with respect to the optical axis L1 of the imaging unit 12 of the first endoscope 10.
  • a control device 85 capable of controlling in consideration of the case where the second endoscope 30 cannot be moved is provided instead of the control device 80 disclosed in the third embodiment.
  • the configuration of the endoscope system 4 according to the present embodiment is the same as that of the third embodiment except that a different control device 85 is provided instead of the control device 80 disclosed in the third embodiment. It's okay.
  • the imaging unit 12 of the first endoscope 10 is movable by the first arm 70 and the joint unit 13, whereas the imaging of the second endoscope 30 is performed.
  • the movable range of the part 32 is limited to the movable range of the second arm 71.
  • the insertion portion 31 of the second endoscope 30 swings around the trocar 100.
  • the control device 85 swings the movable portion of the second arm 71 for moving the second endoscope 30, with the insertion portion 31 of the second endoscope 30 swinging around the trocar 100.
  • a range (indicated by reference numeral D1 in FIG. 12), a range in which the insertion portion 31 of the second endoscope 30 moves back and forth in the trocar 100 (indicated by reference numeral D2 in FIG. 12), and a rotation axis about the central axis of the trocar 100 Is limited to a range in which the insertion portion 31 of the second endoscope 30 is rotated (indicated by reference numeral D3 in FIG. 12) (refer to step S31, FIG. 15).
  • step S ⁇ b> 31 the control device 85 calculates the position and direction of the optical axis L ⁇ b> 2 of the second endoscope 30 based on the position and posture of the second endoscope 30 positioned in the above-described restriction range.
  • the control device 85 corresponds to the arrangement of the organ and other medical instruments in the body so that the insertion portion 31 of the second endoscope 30 does not collide in advance so that the insertion portion 31 of the second endoscope 30 does not collide.
  • the movable range may be limited.
  • control device 85 calculates an operation amount for moving the optical axis L2 of the second endoscope 30 in a direction coaxial with the optical axis L1 of the first endoscope 10 (operation amount calculating step, step S32, see FIG. 15).
  • step S32 the second angle is set so that the angle formed by the vector of the optical axis L1 of the imaging unit 12 of the first endoscope 10 and the vector of the optical axis L2 of the imaging unit 32 of the second endoscope 30 is minimized.
  • the control device 85 sets the movement target position and movement target direction of the optical axis L2 of the endoscope 30.
  • control device 85 sets a movement path (operation amount) for operating the second arm 71 within the range of motion limited by the limiting step.
  • step S32 the control device 85 outputs an operation instruction to the second arm 71 based on the operation amount calculated in step S32 (instruction step, step S33, see FIG. 15).
  • the second arm 71 moves the optical axis L2 of the second endoscope 30 in a direction coaxial with the optical axis L1 of the first endoscope 10 in accordance with the operation instruction output from the control device 85.
  • the optical axis L1 of the first endoscope 10 can be used even when the optical axis L2 of the second endoscope 30 is not coaxial with the optical axis L1 of the first endoscope 10. And almost parallel.
  • the center of the field of view of the image captured by the imaging unit 12 of the first endoscope 10 may be positioned approximately at the center of the imaging field of the imaging unit 32 of the second endoscope 30. it can.
  • control device 85 is configured so that the distance between the optical axis L1 of the imaging unit 12 of the first endoscope 10 and the optical axis L2 of the imaging unit 32 of the second endoscope 30 is minimized.
  • the second arm 71 may be operated within the range of motion limited by the limiting step. In this case, the optical axis L1 of the imaging unit 12 of the first endoscope 10 and the optical axis L2 of the imaging unit 32 of the second endoscope 30 are brought close to each other so that they can be regarded as substantially coaxial. it can.
  • FIG. 16 is a flowchart showing a control procedure by the control device 86 of the endoscope system according to the present embodiment.
  • FIG. 17 and FIG. 18 are schematic diagrams for explaining the operation during use of the endoscope system.
  • An endoscope system 5 (see FIG. 17) according to this embodiment includes a first endoscope 10, an arm 20, and a second endoscope 30 (see FIG. 1) similar to those in the first embodiment,
  • the control apparatus 86 (refer FIG. 17) for controlling operation
  • the control device 86 includes a control procedure different from the control procedure included in the control device 50 of the first embodiment. Below, the structure of the control apparatus 86 is demonstrated by showing the control procedure of the principal part in the endoscope system 5 which concerns on this embodiment.
  • the control device 86 according to the present embodiment performs the following when the input to the viewpoint change switch 39 (see FIG. 1) is performed so that the image displayed on the main monitor 45 is the image from the second endoscope 30.
  • the imaging unit 12 of the first endoscope 10 is moved by the steps from step S41 to step S44.
  • control device 86 stores the current position and posture of the first endoscope 10 (see FIG. 17) based on the input to the viewpoint change switch 39 (position storage step, step S41, (See FIG. 16).
  • step S41 the control device 86 moves the insertion portion 11 of the first endoscope 10 by a predetermined distance in the direction in which the insertion portion 11 of the first endoscope 10 is removed from the body (retraction step, step S42). , FIG. 16).
  • step S42 the control device 86 is interlocked with the movement of the insertion portion 11 of the first endoscope 10 being removed from the body so that the joint portion 13 and the imaging portion 12 do not contact the tissue in the vicinity of the treatment site.
  • the joint portion 13 is driven so as to be linear in order from the proximal end side of the joint portion 13 (see FIG. 18).
  • step S42 the insertion portion 11 of the first endoscope 10 may be moved in a direction in which the insertion portion 11 of the first endoscope 10 is removed from the body while the shape of the joint portion 13 is maintained.
  • the movement of the insertion portion 11 of the first endoscope 10 in step S42 is performed by the control device 86 controlling the operation of each joint portion 22 of the arm 20.
  • the imaging unit 12 of the first endoscope 10 moves to a position away from the optical axis L2 of the imaging unit 32 of the second endoscope 30.
  • the moving destination of the imaging unit 12 of the first endoscope 10 in step S42 is, for example, outside the imaging field of the imaging unit 32 of the second endoscope 30 or within the imaging field of the imaging unit 32 of the second endoscope 30. Even so, the position does not interfere with the observation using the second endoscope 30. This ends step S42.
  • the imaging unit 12 of the first endoscope 10 is not located at the center of the imaging field of the imaging unit 32 of the second endoscope 30.
  • the imaging unit 32 can preferably observe the treatment site and the state in the vicinity thereof.
  • the imaging unit of the first endoscope 10 moved in step S42 described above. 12 is moved to the position stored in step S41.
  • the operator uses the viewpoint change switch as an input for switching the image displayed on the main monitor 45 to an image captured by the imaging unit 12 of the first endoscope 10. 39 is input.
  • the control device 86 reads out the position and posture of the first endoscope 10 stored in step S41 based on the input to the viewpoint change switch 39 (see reading step, step S43, FIG. 16). ).
  • step S43 the control device 86 controls the first endoscope 10 and the arm 20 so that the position and posture of the first endoscope 10 become the position and posture read in step S41.
  • step S44 an operation instruction
  • the imaging unit 12 of the first endoscope 10 returns to the position stored before the movement in step S41 (see FIG. 17), and the treatment site is centered on the visual field in the posture before the movement. It is possible to capture a narrow-angle image captured in
  • the imaging unit 12 of the first endoscope 10 is retracted from the center of the visual field of the wide-angle image captured using the second endoscope 30.
  • the imaging unit 12 of the first endoscope 10 can be automatically returned from the retracted state after observation using the wide-angle image, the operation burden on the operator can be reduced.
  • FIG. 19 is a schematic diagram for explaining an operation during use of the endoscope system according to the present embodiment.
  • FIG. 20 is a schematic diagram for explaining an operation when the endoscope system is used.
  • An endoscope system 6 (see FIG. 19) according to this embodiment includes a first endoscope 10, a second endoscope 30, a first arm 70, and a second arm 71 (FIG. 10) similar to those in the third embodiment. And a control device 87 (see FIG. 19) for controlling operations of the first endoscope 10, the first arm 70, and the second arm 71.
  • the control device 87 differs from the fifth embodiment in that the control device 87 includes a control procedure different from that of the control device 86 of the fifth embodiment.
  • the control device 87 converts a narrow-angle image from the first endoscope 10 to a wide-angle image from the second endoscope 30.
  • the imaging unit 12 of the first endoscope 10 is retracted (see FIG. 19), and then the wide-angle image by the second endoscope 30 is changed to the narrow-angle image by the first endoscope 10.
  • the imaging unit 12 of the first endoscope 10 is returned to the original position.
  • control device 87 of the present embodiment returns the imaging unit 12 of the first endoscope 10 to the original position and displays a narrow-angle image on the main monitor 45, and then is disclosed in the third embodiment.
  • the optical axis L1 of the imaging unit 12 of the first endoscope 10 and the optical axis L2 of the imaging unit 32 of the second endoscope 30 are coaxial. (See FIG. 20), the second arm 71 is moved.
  • the endoscope system 6 when the treatment site and its surroundings are observed from a bird's-eye view using a wide-angle image of the imaging target site using the second endoscope 30, The imaging unit 12 of the first endoscope 10 can be retracted, and the imaging unit 12 of the first endoscope 10 does not get in the way when the treatment site is observed using the second endoscope 30. Further, the endoscope system 6 according to the present embodiment returns the image displayed on the main monitor 45 to the narrow-angle image after the operator moves the second endoscope 30 and observes the periphery of the treatment site.
  • the imaging unit 12 of the first endoscope 10 moves from the retracted position to a position where the treatment site is captured at the center of the visual field and returns to the original position. Further, the imaging unit 32 of the second endoscope 30 of the endoscope system 6 is moved by the second arm 71 in the background so as to be coaxial with the optical axis L1 of the imaging unit 12 of the first endoscope 10. Is done.
  • the control device 87 causes the first endoscope 10 and the second endoscope 10 so that the imaging unit 12 of the first endoscope 10 and the imaging unit 32 of the second endoscope 30 both capture the treatment site at the center of the visual field.
  • the endoscope 30 can be automatically moved. Thereby, according to the endoscope system 6 which concerns on this embodiment, an operator's operation burden can be eased.
  • FIG. 21 and 22 are schematic views illustrating an example of an image displayed on the main monitor of the endoscope system according to the present embodiment.
  • the endoscope system 7 according to the present embodiment includes a control device 89 having a configuration different from that of the control device 50 disclosed in the first embodiment, instead of the control device 50 of the first embodiment ( (See FIG. 1).
  • the configuration of the endoscope system 7 according to the present embodiment is the same as that of the first embodiment except for the configuration of the control device 89.
  • the control device 89 of the endoscope system 7 according to the present embodiment can display the position of the optical axis L2 of the second endoscope 30 on the main monitor 45.
  • the control device 89 of the present embodiment applies a predetermined target mark M1 to the center of the field of view of the image captured by the imaging unit 32 of the second endoscope 30, as shown in FIG. Are superimposed and output to the main monitor 45.
  • both the imaging unit 12 of the first endoscope 10 and the imaging unit 32 of the second endoscope 30 are in a state where the treatment site is captured at the center of the visual field.
  • the operator first uses the image captured by the imaging unit 32 of the second endoscope 30 as in the first embodiment. To observe.
  • the image and the target mark M1 captured by the imaging unit 32 of the second endoscope 30 are displayed on the main monitor 45 as shown in FIG. It is possible to easily grasp where the optical axis L2 of the imaging unit 32 of the two endoscopes 30 is located in the imaging target part. Therefore, the operator moves the imaging unit 12 of the first endoscope 10 by moving the second endoscope 30 so that the target mark M1 overlaps the treatment site as shown in FIG. Can be specified accurately.
  • the imaging unit 12 of the first endoscope 10 overlaps the target mark. Move to position. Since the position of the target mark M2 reflects the position determined by the operator as the treatment target, the imaging unit 12 of the first endoscope 10 captures the position determined by the operator as the center of the visual field. Get an image.
  • an image captured by the imaging unit 12 of the first endoscope 10 is displayed on the main monitor 45.
  • the operator can perform a treatment on the treatment site using the image captured by the imaging unit 12 of the first endoscope 10.
  • the control mark 89 is superimposed on the image captured by the imaging unit 32 of the second endoscope 30 and displayed on the main monitor 45, so that the optical axis L2 of the imaging unit 32 of the second endoscope 30 is displayed. It is easy to move the second endoscope 30 so as to match the treatment site. As a result, the operator can easily create a state where both the imaging unit 32 of the second endoscope 30 and the imaging unit 12 of the first endoscope 10 accurately capture the treatment site at the center of the visual field.
  • FIG. 23 to FIG. 25 are schematic diagrams showing examples of images displayed on the main monitor of the endoscope system of the present modification.
  • the target mark is not the optical axis L2 of the second endoscope 30, but an arbitrary point designated by the operator on the image captured by the imaging unit 32 of the second endoscope 30. .
  • both the first mark M1 indicating the center of the visual field of the second endoscope 30 and the second mark M2 which is the target mark in the present modification are displayed on the main monitor 45. Is done.
  • the control device 89 moves the first endoscope 10 so that the optical axis L2 of the imaging unit 32 of the second endoscope 30 and the optical axis L1 of the first endoscope 10 are parallel to each other.
  • the imaging unit of the first endoscope 10 is arranged such that the optical axis L1 of the first endoscope 10 overlaps the target mark (second mark M2) on the image by the imaging unit 32 of the second endoscope 30. 12 is moved (see FIG. 24).
  • the control device 89 causes the main monitor 45 to display an image captured by the imaging unit 12 of the first endoscope 10, so that a narrow angle including a position corresponding to the target mark (second mark M ⁇ b> 2) at the center of the visual field. An image is displayed on the main monitor 45.
  • the target mark (second mark) is moved without moving the second endoscope 30.
  • the first endoscope 10 can be operated so that the imaging unit 12 of the first endoscope 10 can capture an image in which the treatment site is centered on the visual field.
  • the second endoscope in a state where the imaging unit 12 of the first endoscope 10 captures the treatment site at the center of the visual field.
  • the second endoscope 30 can be moved so that the optical axis L2 of the 30 imaging units 32 is coaxial with the optical axis L1 of the imaging unit 12 of the first endoscope 10 (see FIG. 25).
  • the first marker may include information serving as a basis for specifying the position of the optical axis of the imaging unit of the first endoscope. This information may be obtained by encoding an identifier unique to the model of the first endoscope, for example.
  • the control device may be configured to recognize the position of the optical axis based on the structure of the first endoscope based on the identifier included in the first marker.
  • the second marker may include information serving as a basis for specifying the position of the optical axis of the imaging unit of the second endoscope.
  • the control device may be configured to recognize the position of the optical axis based on the structure of the second endoscope based on the identifier included in the second marker.

Landscapes

  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Engineering & Computer Science (AREA)
  • Surgery (AREA)
  • Public Health (AREA)
  • Medical Informatics (AREA)
  • Veterinary Medicine (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • General Health & Medical Sciences (AREA)
  • Animal Behavior & Ethology (AREA)
  • Physics & Mathematics (AREA)
  • Molecular Biology (AREA)
  • Biomedical Technology (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Radiology & Medical Imaging (AREA)
  • Pathology (AREA)
  • Optics & Photonics (AREA)
  • Biophysics (AREA)
  • Signal Processing (AREA)
  • Multimedia (AREA)
  • Robotics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Endoscopes (AREA)
  • Instruments For Viewing The Inside Of Hollow Bodies (AREA)

Abstract

Le système d'endoscope de l'invention comporte : un premier endoscope comportant une première unité d'imagerie ; un second endoscope comportant une seconde unité d'imagerie ; un bras, fixé au premier endoscope et qui peut être actionné pour déplacer le premier endoscope ; un premier moyen de détection de position/orientation, qui détecte la position et l'orientation du premier endoscope dans un système de coordonnées prédéfini ; un second moyen de détection de position/orientation, qui détecte la position et l'orientation du second endoscope dans le système de coordonnées prédéfini ; et un dispositif de commande, qui commande le déplacement du bras. Le dispositif de commande commande le déplacement du bras selon une procédure de commande d'actionnement du bras comprenant : une première étape de calcul consistant à acquérir la position et l'orientation du second endoscope dans le système de coordonnées prédéfini provenant du second moyen de détection de position/orientation, et à calculer la position et la direction de l'axe optique de la seconde unité d'imagerie ; une étape de calcul de valeur de déplacement consistant à acquérir la position et l'orientation du premier endoscope dans le système de coordonnées prédéfini provenant du premier moyen de détection de position/orientation, et à calculer la valeur de déplacement du bras pour actionner le bras de sorte que la première unité d'imagerie se déplace dans une direction telle que l'axe optique de la première unité d'imagerie coïncide avec l'axe optique de la seconde unité d'imagerie ; et une étape de commande consistant à produire une commande d'actionnement pour actionner le bras sur la base de la valeur de déplacement.
PCT/JP2016/083788 2016-02-10 2016-11-15 Système d'endoscope WO2017138208A1 (fr)

Priority Applications (4)

Application Number Priority Date Filing Date Title
CN201680065126.4A CN108348134B (zh) 2016-02-10 2016-11-15 内窥镜系统
JP2017516540A JP6150968B1 (ja) 2016-02-10 2016-11-15 内視鏡システム
EP16889909.4A EP3415071A4 (fr) 2016-02-10 2016-11-15 Système d'endoscope
US15/976,142 US10638915B2 (en) 2016-02-10 2018-05-10 System for moving first insertable instrument and second insertable instrument, controller, and computer-readable storage device

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2016023670 2016-02-10
JP2016-023670 2016-02-10

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US15/976,142 Continuation US10638915B2 (en) 2016-02-10 2018-05-10 System for moving first insertable instrument and second insertable instrument, controller, and computer-readable storage device

Publications (1)

Publication Number Publication Date
WO2017138208A1 true WO2017138208A1 (fr) 2017-08-17

Family

ID=59563753

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2016/083788 WO2017138208A1 (fr) 2016-02-10 2016-11-15 Système d'endoscope

Country Status (4)

Country Link
US (1) US10638915B2 (fr)
EP (1) EP3415071A4 (fr)
CN (1) CN108348134B (fr)
WO (1) WO2017138208A1 (fr)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2021024804A1 (fr) * 2019-08-05 2021-02-11 ソニー株式会社 Système d'imagerie chirurgicale, dispositif de traitement de signal et procédé de traitement de signal

Families Citing this family (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9204939B2 (en) * 2011-08-21 2015-12-08 M.S.T. Medical Surgery Technologies Ltd. Device and method for assisting laparoscopic surgery—rule based approach
CN110786819B (zh) 2016-03-30 2022-07-26 索尼公司 手术成像系统
CN112336295B (zh) * 2019-08-08 2024-07-05 上海安翰医疗技术有限公司 磁性胶囊内窥镜的控制方法、装置、存储介质、电子装置
US20230096406A1 (en) * 2021-09-29 2023-03-30 Cilag Gmbh International Surgical devices, systems, and methods using multi-source imaging
US11937798B2 (en) 2021-09-29 2024-03-26 Cilag Gmbh International Surgical systems with port devices for instrument control
CN113952033B (zh) * 2021-12-21 2022-04-19 广东欧谱曼迪科技有限公司 一种双源内窥手术导航系统及方法
KR20240128994A (ko) * 2021-12-31 2024-08-27 아우리스 헬스, 인코포레이티드 기계적 연동을 사용한 포지셔닝 시스템 정합
CN115842848B (zh) * 2023-03-01 2023-04-28 成都远峰科技发展有限公司 一种基于工业物联网的动态监控系统及其控制方法

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2000032442A (ja) 1998-07-13 2000-01-28 Olympus Optical Co Ltd 内視鏡システム
JP2001161638A (ja) 1999-12-13 2001-06-19 Olympus Optical Co Ltd 手術用顕微鏡
JP2003127076A (ja) * 1993-12-28 2003-05-08 Olympus Optical Co Ltd 医療器具
JP2004136066A (ja) 2002-03-25 2004-05-13 Olympus Corp 超音波観察システム
JP2009125392A (ja) * 2007-11-26 2009-06-11 Olympus Medical Systems Corp 内視鏡システムおよび内視鏡装置
WO2013141155A1 (fr) * 2012-03-17 2013-09-26 学校法人早稲田大学 Système d'exécution d'image pour une région tronquée dans l'image, dispositif de traitement d'image et programme associé

Family Cites Families (16)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH08164148A (ja) 1994-12-13 1996-06-25 Olympus Optical Co Ltd 内視鏡下手術装置
JP3506809B2 (ja) 1995-06-08 2004-03-15 オリンパス株式会社 体腔内観察装置
US5836869A (en) 1994-12-13 1998-11-17 Olympus Optical Co., Ltd. Image tracking endoscope system
JP3532660B2 (ja) 1995-06-09 2004-05-31 オリンパス株式会社 体腔内観察装置
JP4656700B2 (ja) 2000-07-11 2011-03-23 オリンパス株式会社 内視鏡外科手術システム
JP4179846B2 (ja) 2002-10-25 2008-11-12 オリンパス株式会社 内視鏡手術システム
JP2006167867A (ja) 2004-12-16 2006-06-29 Fuji Heavy Ind Ltd 遠隔操作装置
US8469945B2 (en) * 2006-01-25 2013-06-25 Intuitive Surgical Operations, Inc. Center robotic arm with five-bar spherical linkage for endoscopic camera
JP4861540B2 (ja) 2010-05-10 2012-01-25 オリンパスメディカルシステムズ株式会社 医療装置
CN103188987B (zh) * 2010-11-02 2015-08-05 伊顿株式会社 手术机器人系统及其腹腔镜操作方法以及体感式手术用图像处理装置及其方法
US10231791B2 (en) * 2012-06-21 2019-03-19 Globus Medical, Inc. Infrared signal based position recognition system for use with a robot-assisted surgery
KR102218244B1 (ko) * 2012-12-10 2021-02-22 인튜어티브 서지컬 오퍼레이션즈 인코포레이티드 영상 포착 장치 및 조작 가능 장치 가동 아암들의 제어된 이동 중의 충돌 회피
DE102012025100A1 (de) * 2012-12-20 2014-06-26 avateramedical GmBH Entkoppeltes Mehrkamerasystem für die minimal-invasive Chirurgie
JP6265630B2 (ja) 2013-06-13 2018-01-24 オリンパス株式会社 内視鏡装置及び内視鏡装置の作動方法
JP6120715B2 (ja) 2013-07-26 2017-04-26 オリンパス株式会社 医療システム
WO2016189878A1 (fr) * 2015-05-27 2016-12-01 京セラ株式会社 Dispositif logique arithmétique, dispositif à caméra, véhicule et procédé d'étalonnage

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2003127076A (ja) * 1993-12-28 2003-05-08 Olympus Optical Co Ltd 医療器具
JP2000032442A (ja) 1998-07-13 2000-01-28 Olympus Optical Co Ltd 内視鏡システム
JP2001161638A (ja) 1999-12-13 2001-06-19 Olympus Optical Co Ltd 手術用顕微鏡
JP2004136066A (ja) 2002-03-25 2004-05-13 Olympus Corp 超音波観察システム
JP2009125392A (ja) * 2007-11-26 2009-06-11 Olympus Medical Systems Corp 内視鏡システムおよび内視鏡装置
WO2013141155A1 (fr) * 2012-03-17 2013-09-26 学校法人早稲田大学 Système d'exécution d'image pour une région tronquée dans l'image, dispositif de traitement d'image et programme associé

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP3415071A4

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2021024804A1 (fr) * 2019-08-05 2021-02-11 ソニー株式会社 Système d'imagerie chirurgicale, dispositif de traitement de signal et procédé de traitement de signal
JP7484922B2 (ja) 2019-08-05 2024-05-16 ソニーグループ株式会社 手術撮像システム、信号処理装置、及び、信号処理方法

Also Published As

Publication number Publication date
CN108348134B (zh) 2020-05-19
US20180256008A1 (en) 2018-09-13
EP3415071A1 (fr) 2018-12-19
CN108348134A (zh) 2018-07-31
US10638915B2 (en) 2020-05-05
EP3415071A4 (fr) 2019-11-20

Similar Documents

Publication Publication Date Title
WO2017138208A1 (fr) Système d'endoscope
JP7229319B2 (ja) 手術支援装置、その制御方法、プログラム並びに手術支援システム
JP6982605B2 (ja) 外科用器具可視性のロボット制御
US9630323B2 (en) Operation support system and control method of operation support system
JP4744595B2 (ja) 内視鏡外科手術器具
KR102105142B1 (ko) 입력 장치의 오퍼레이터가 볼 수 있는 디스플레이 영역으로 기구가 진입할 때 기구의 제어를 입력 장치로 전환하는 방법
KR102363661B1 (ko) 원격동작 의료 시스템 내의 기구의 화면외 표시를 위한 시스템 및 방법
KR102585602B1 (ko) 눈 시선 추적을 사용하는 의료 디바이스, 시스템, 및 방법
CN108433809B (zh) 用于在手术过程期间设置并检索参考点的设备
JP6234643B2 (ja) 医療システム及びその作動方法
JP5384108B2 (ja) 遠隔操作システム
JPWO2020054566A1 (ja) 医療用観察システム、医療用観察装置及び医療用観察方法
JP4027876B2 (ja) 体腔内観察システム
WO2017082047A1 (fr) Système d'endoscope
JP6150968B1 (ja) 内視鏡システム
JP4426770B2 (ja) 内視鏡保持装置
WO2019106711A1 (fr) Système médical et procédé pour faire fonctionner le système médical
CN117651533A (zh) 多分叉通道中的管状器件导航方法、设备及存储介质

Legal Events

Date Code Title Description
ENP Entry into the national phase

Ref document number: 2017516540

Country of ref document: JP

Kind code of ref document: A

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16889909

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

WWE Wipo information: entry into national phase

Ref document number: 2016889909

Country of ref document: EP

ENP Entry into the national phase

Ref document number: 2016889909

Country of ref document: EP

Effective date: 20180910