JP2012065698A - Operation support system, and operation support method using the same - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable an operator himself to perform operation support work without using an assistant in an operation support system for supporting operation performed using a treatment tool while looking a screen, and an operation support method using the same.SOLUTION: This operation support system includes an imaging part 20 for taking the image of a part to be observed and an image display part 4 for displaying the image of the part to be observed on the screen and supports operation performed using the treatment tool 6 while looking the screen on which the image is displayed. This operation support system is equipped with an image processing part 33 for extracting the position of the representative point P of the treatment tool 6 in the image in a case that the treatment tool 6 is contained in the image, a menu forming part 39a for forming a menu N, an item selection part 39b for selecting a support item I on the basis of the data of a position where the menu N is displayed and the data of the position of the representative point P and an operation support means for executing the operation support work related to the support item I on the basis of the selected support item I.

Description

  The present invention relates to an operation support system that supports an operation of an observed portion performed using a treatment tool while viewing a screen, and an operation support method using the same.

  2. Description of the Related Art Conventionally, an operation is performed in which a treatment tool is used to perform a treatment on a portion to be observed that is projected on a screen while viewing the screen. For example, an operation using an endoscope system as shown in Patent Document 1 is an example of such an operation. Such an endoscope system inserts an endoscope into a region to be examined, irradiates illumination light from the light source device to the observation portion, obtains an optical image of the observation portion with the endoscope, and The image signal of the image is subjected to desired signal processing, and an image of the optical image is displayed on the screen of the monitor. Then, the surgeon uses the treatment tool such as forceps inserted into the endoscope to treat the observed portion while viewing the screen.

  In the above-mentioned surgery, in order to secure the optimal surgical field for the surgeon performing the surgery, the range and position of the observed part displayed on the screen, the brightness of the screen, etc. are adjusted as appropriate according to the situation. There is a need to. However, since the surgeon usually holds the treatment tool for performing surgery in both hands, the operator cannot quickly perform the screen adjustment work as described above. Alternatively, it is not preferable that the surgeon directly operates the adjustment device or the like for screen adjustment from the viewpoint of ensuring the cleanliness of the surgical field, medical device, operating room, and the like.

  Therefore, in general, when the surgeon gives an instruction to an assistant called a scopist or the like, the assistant operates the adjustment device according to the instruction and performs the screen adjustment as described above. Has been adopted.

JP 2002-263053 A

  However, the method using the assistant as described above has a problem that the operator's instruction is not accurately transmitted and the screen adjustment desired by the operator is not always performed. Therefore, there is a demand for a system that enables the surgeon to perform screen adjustment without using an assistant so that the screen adjustment desired by the surgeon is appropriately performed.

  Furthermore, the above problems are not limited to screen adjustment. For example, operations for supporting surgery in general, such as switching of the wavelength of illumination light and synthesis processing of a visible image and a fluorescent image (operation support) The same can happen for (work).

  The present invention has been made in view of the above problems, and in a system and method for assisting surgery for performing treatment on a portion to be observed displayed on a screen, an operation support operation desired by an operator is appropriately performed. An object of the present invention is to provide an operation support system and an operation support method using the operation support system that enable an operator himself / herself to control an operation support means for performing an operation support operation without using an assistant. .

In order to solve the above-described problem, a surgery support system according to the present invention includes:
An imaging unit that captures an image of an observed part and an image display unit that displays an image of the observed part on a screen, and is an observation part that is performed using a treatment instrument while viewing the screen on which the image is displayed. In a surgery support system that supports surgery,
An extraction unit for extracting the position of the representative point of the treatment tool in the image when the treatment tool is included in the image;
A menu forming unit for forming a menu of one or more support items related to a surgery support operation for supporting a surgery performed by the treatment tool;
An image combining unit that combines the image and the menu so that the menu can be displayed on an image display area where the image is displayed in the screen so as to be superimposed on a predetermined part of the image;
An item selection unit for selecting any one of the support items based on the information on the position of the predetermined portion where the menu is displayed and the information on the position of the representative point;
A surgical operation support means for performing a surgical operation support operation related to the support item based on the selected support item is provided.

  In this specification, the “representative point” means a point representing the position (coordinates) of the treatment tool in the image. Although the number of representative points may be plural, it is preferable that the number of points is one in order to simplify the processing.

  “Representative point position” means the coordinates of a representative point in an image.

  “Surgery support work” means work performed to support surgery, and includes, for example, adjustment of a screen on which an image is displayed (change in image magnification, change in image brightness), and the like.

  “The information of the position of the predetermined portion where the menu is displayed” means, for example, the coordinates in the image of the menu indicating the position where the menu is displayed in the image.

  Furthermore, in the surgery support system according to the present invention, the treatment tool has a marker at a position that can be recognized in the image of the treatment tool, and the extraction unit uses the marker as the position of the representative point of the treatment tool. It is preferable to extract the position.

  Furthermore, the marker is preferably a light emitting diode.

  In addition, the treatment tool includes a switch for lighting the light emitting diode on the handle portion of the treatment tool, and the extraction unit is located at the position of the representative point from the lighting portion of the light emitting diode only when the light emitting diode is in the lighting state. Is preferably extracted.

  Furthermore, the treatment tool includes a switch for lighting the light emitting diode on the handle portion of the treatment tool, and the item selection unit selects a support item on the condition that the light emitting diode is in a lighting state. It is preferable.

  Furthermore, the treatment instrument includes a switch for blinking the light emitting diode on the handle portion of the treatment instrument, and the item selection unit selects a support item on the condition that the light emitting diode is blinking. It is preferable.

  Furthermore, it is preferable that the menu forming unit forms a menu in conjunction with the lighting state of the light emitting diode.

  Furthermore, the imaging unit receives reflected light from the observed portion generated by irradiating illumination light as the image, and picks up a visible image, and the light emitting diode generates blue light. Preferably there is.

  In this specification, “blue light” means light included in a wavelength range of about 450 nm.

  Alternatively, the imaging unit receives the fluorescence from the observed portion generated by irradiating the excitation light as the image, and captures the fluorescence image. The light emitting diode emits light included in the fluorescence wavelength band. It is preferable that it is generated.

  In the present specification, “light included in the fluorescence wavelength band” means light included in a wavelength range of about 900 nm.

  Furthermore, the treatment instrument can be provided with a switch for selecting whether or not to display a menu on the handle portion of the treatment instrument.

  Furthermore, support items include image magnification change, image brightness change, image enhancement, still image creation, image composition, imaging location movement, illumination light wavelength switching, presence / absence of excitation light illumination , Switching of image display, and cleaning of the lens.

  Further, the treatment tool is preferably a forceps, and the surgery support system according to the present invention can be used for an endoscope system.

And, the surgery support method according to the present invention comprises:
In a surgery support method for capturing an image of a portion to be observed, displaying the image of the portion to be observed on a screen, and supporting the operation of the portion to be observed performed using a treatment tool while viewing the screen on which the image is displayed ,
When a treatment tool is included in the image, the position of the representative point of the treatment tool in the image is extracted,
Forming a menu of one or more support items related to the operation support operation for supporting the operation performed by the treatment tool;
The image and the menu are combined so that the menu can be displayed on the image display area where the image is displayed on the screen so as to be superimposed on a predetermined portion of the image.
Based on the information on the position of the predetermined portion where the menu is displayed and the information on the position of the representative point, select one of the support items,
Based on the selected support item, an operation support operation for the support item is performed.

  An operation support system and an operation support method using the same according to the present invention include an imaging unit that captures an image of an observed part, and an image display unit that displays an image of the observed part on a screen, and the image is displayed. In a surgery support system for assisting surgery of an observed portion performed using a treatment tool while viewing the screen, when the treatment tool is included in the image, the representative point position of the treatment tool in the image is extracted. An extraction unit, a menu forming unit that forms a menu of one or more support items related to a surgical support operation for supporting a surgery performed by the treatment tool, and an image on an image display area in which an image is displayed on the screen. An image composition unit for compositing the image and the menu, information on the position of the predetermined part where the menu is displayed, and the position of the representative point. An item selection unit that selects any one of the support items based on the information, and a surgery support unit that performs a surgery support operation on the support item based on the selected support item. . As a result, the surgeon can use the treatment tool to select a support item, control the surgical support means, and execute the surgical support work. As a result, in the system and method for supporting the operation for performing the treatment on the observed portion displayed on the screen, the operator himself / herself is not involved with the assistant so that the operation support operation desired by the operator is appropriately performed. It becomes possible to control the operation support means for performing the operation support operation.

It is the schematic which shows one Embodiment of the endoscope system which comprises the surgery assistance system of this invention. It is the schematic which shows the structure of the hard insertion part which comprises the endoscope system of FIG. It is the schematic which shows the internal structure of the imaging part which comprises the endoscope system of FIG. It is a block diagram which shows the internal structure of the image processing apparatus and light source device of the endoscope system of FIG. It is the schematic which shows one Embodiment of the treatment tool which comprises the surgery assistance system of this invention. It is the schematic which shows a mode that the marker was provided in the treatment tool of FIG. 5A. It is a figure which shows an example of the state by which the hard insertion part and the treatment tool were inserted in the body cavity. It is the schematic which shows a mode that the normal image on which the menu was superimposed was displayed on the image display part in the surgery assistance system of this invention. It is the schematic which shows other embodiment of the treatment tool which comprises the surgery assistance system of this invention. It is the schematic which shows other embodiment of the treatment tool which comprises the surgery assistance system of this invention. It is the schematic explaining how a marker looks when a normal image and a fluorescence image switch. It is the schematic which shows a mode that the fluorescence image on which the menu was superimposed was displayed on the image display part in the surgery assistance system of this invention. It is the schematic which shows other embodiment of the display method of the menu in the surgery assistance system of this invention.

  Hereinafter, although an embodiment of the present invention is described using a drawing, the present invention is not limited to this. In addition, for easy visual recognition, the scale of each component in the drawings is appropriately changed from the actual one.

“Embodiment of Surgery Support System and Surgery Support Method Using It”
The surgical operation support system according to the present embodiment includes a treatment tool such as forceps and an endoscope system. Here, the endoscope system according to the present embodiment includes an imaging unit that captures an image of the observed part, a monitor (image display part) that displays the image of the observed part on a screen, and a treatment tool in the image. An extraction unit that extracts the position of the representative point of the treatment tool in the image, and a menu formation unit that forms a menu of one or more support items related to a surgical support operation that supports the surgery performed by the treatment tool. The menu is displayed on the image display area where the image is displayed in the screen so that the menu can be displayed by superimposing the menu on a predetermined portion of the image. Based on the information on the position of the predetermined portion and the information on the position of the representative point, an item selection unit that selects any one of the support items, and the support item based on the selected support item Surgery And an operation support means for executing the 援作 industry.

  FIG. 1 is an external view showing a schematic configuration of an endoscope (rigid endoscope) system 1 constituting the surgery support system of the present embodiment. As shown in FIG. 1, the rigid endoscope system 1 according to the present embodiment guides the normal light and excitation light emitted from the light source device 2 and the light source device 2 that emits white normal light and excitation light. A rigid mirror imaging apparatus that irradiates the observation unit and captures a normal image based on reflected light reflected from the observed portion by irradiation of normal light and a fluorescent image based on fluorescence emitted from the observed portion by irradiation of excitation light 10, an image processing device 3 that performs a predetermined process on the image signal captured by the rigid endoscope imaging device 10, and a normal image and a fluorescence image of the observed portion based on the display control signal generated in the image processing device 3. And a monitor 4 for display.

  As shown in FIG. 1, the rigid endoscope imaging apparatus 10 includes a hard insertion unit 30 that is inserted into the abdominal cavity, and an imaging unit 20 that captures a normal image and a fluorescence image of the observed portion guided by the hard insertion unit 30. And.

  In addition, as shown in FIG. 2, the rigid endoscope imaging device 10 has a hard insertion unit 30 and an imaging unit 20 that are detachably connected. The hard insertion portion 30 includes a connection member 30a, an insertion member 30b, a cable connection port 30c, an irradiation lens 30d, and an imaging lens 30e.

  The connection member 30a is provided on one end side 30X of the hard insertion portion 30 (insertion member 30b). For example, the connection member 30a is fitted into an opening 20a formed on the imaging portion 20 side, whereby the imaging portion 20 and the hard insertion portion 30 are connected. Are detachably connected.

  The insertion member 30b is inserted into the abdominal cavity when photographing inside the abdominal cavity, and is formed of a hard material and has, for example, a cylindrical shape with a diameter of approximately 10 mm. The insertion member 30b accommodates a lens group for forming an image of the observed portion. The normal image and the fluorescent image of the observed portion incident from the other end 30Y are the imaging lens 30e and the above-described lens. The light is emitted to the imaging unit 20 side on the one end side 30X through the lens group.

  A cable connection port 30c is provided on the side surface of the insertion member 30b, and the optical cable LC is mechanically connected to the cable connection port 30c. Thereby, the light source device 2 and the insertion member 30b are optically connected via the optical cable LC.

  The irradiation lens 30d is provided on the other end side 30Y of the hard insertion portion 30, and irradiates the observed portion with normal light and excitation light guided by the optical cable LC. In addition, a light guide for guiding normal light and excitation light from the cable connection port 30c to the irradiation lens 30d is accommodated in the insertion member 30b (not shown), and the irradiation lens 30d is guided by the light guide. The light to be observed is irradiated with normal light and excitation light.

  FIG. 3 is a diagram illustrating a schematic configuration of the imaging unit 20. The imaging unit 20 includes a first imaging system that captures a fluorescent image of the observed part imaged by the lens group in the hard insertion part 30 and generates a fluorescent image signal of the observed part, and the hard insertion part 30 And a second imaging system that generates a normal image signal by capturing a normal image of the observed portion formed by the lens group. These imaging systems are divided into two optical axes orthogonal to each other by a dichroic prism 21 having a spectral characteristic that reflects a normal image and transmits a fluorescent image.

  The first imaging system includes an excitation light cut filter 22 that cuts the excitation light emitted from the hard insertion portion 30 and transmitted through the dichroic prism 21, and the dichroic prism 21 and the excitation light cut filter 22 emitted from the hard insertion portion 30. The first imaging optical system 23 that forms an image of the fluorescent image L4 that has passed through the first imaging optical system 23, and the high-sensitivity imaging device 24 that images the fluorescent image L4 imaged by the first imaging optical system 23.

  The second imaging system is a normal image formed by the second imaging optical system 25 and a second imaging optical system 25 that forms a normal image L3 emitted from the hard insertion portion 30 and reflected by the dichroic prism 21. An image sensor 26 that captures the image L3 is provided.

  The high-sensitivity imaging element 24 detects light in the wavelength band of the fluorescent image L4 with high sensitivity, converts it into a fluorescent image signal, and outputs it. The high sensitivity image sensor 24 is a monochrome image sensor.

  The image sensor 26 detects light in the wavelength band of the normal image, converts it into a normal image signal, and outputs it. On the image pickup surface of the image pickup element 26, color filters of three primary colors red (R), green (G) and blue (B), or cyan (C), magenta (M) and yellow (Y) are arranged in a Bayer array or a honeycomb. It is provided in an array.

  In addition, the imaging unit 20 includes an imaging control unit 27. The imaging control unit 27 performs CDS / AGC (correlated double sampling / automatic gain control) processing and A / D on the fluorescence image signal output from the high-sensitivity imaging device 24 and the normal image signal output from the imaging device 26. A conversion process is performed and output to the image processing apparatus 3 via the cable 5 (see FIG. 1).

  As shown in FIG. 4, the image processing apparatus 3 includes a normal image input controller 31, a fluorescence image input controller 32, an image processing unit 33, a memory 34, a video output unit 35, an operation unit 36, a TG (timing generator) 37, and a CPU 38. The menu forming unit 39a and the item selecting unit 39b are provided.

  The normal image input controller 31 and the fluorescence image input controller 32 include a line buffer having a predetermined capacity, and temporarily output the normal image signal and the fluorescence image signal for each frame output from the imaging control unit 27 of the imaging unit 20. To remember. Then, the normal image signal stored in the normal image input controller 31 and the fluorescent image signal stored in the fluorescent image input controller 32 are stored in the memory 34 via the bus.

  The image processing unit 33 receives a normal image signal and a fluorescence image signal for each frame read from the memory 34, performs predetermined image processing on these image signals, and outputs them to the bus. In the above image processing, in addition to image brightness adjustment, normal image and fluorescence image synthesis processing, etc., when a treatment tool is included in the normal image and / or fluorescence image, the treatment tool in the image is recognized. The process to perform is also included. That is, the image processing unit 33 has a function as an extraction unit in the present invention. Specifically, the image processing unit 33 recognizes the treatment tool in the image, extracts the position of the representative point (coordinate in the image) of the recognized treatment tool, and outputs the extracted position information to the bus. To do. The image recognition process is not particularly limited and can be performed by a general method. Information on the position of the representative point is stored in the memory 34 via the bus. This representative point is used as a pointer when selecting a support item in the menu displayed on the screen. The representative point is preferably set at the distal end of the treatment tool from the viewpoint of operability. Further, as will be described later, the image processing unit 33 also performs a composition process for superimposing the menu formed by the menu forming unit 39a on the image captured by the imaging unit 20. That is, the image processing unit 33 has a function as an image composition unit in the present invention. In addition, the image processing unit 33 also generates information on where the menu is superimposed and displayed in the image, and this information is stored in the memory 34 via the bus.

  The menu formation part 39a forms the menu containing the support item regarding the operation | work (operation support operation) performed in order to support an operation. The menu formed by the menu forming unit 39a is transferred to the image processing unit 33 via the bus. The menu is superimposed on the image captured by the image capturing unit 20 by the image processing unit 33 and displayed on the monitor 4. The support items included in the menu can be set in advance, and are stored in the memory 34, for example. Surgery support operations include changing the magnification of the image, changing the brightness of the image, enhancing the image, creating a still image, compositing the image, moving the imaging location, switching the wavelength of the illumination light, whether or not the excitation light is illuminated Switching, image display switching, lens cleaning, and the like. Specifically, “changing the magnification of an image” means enlarging or reducing the image displayed on the monitor. “Changing the brightness of an image” means adjusting the brightness of the monitor. “Image enhancement” means to increase the sharpness of an image or remove noise by image processing, and to display a specific area in the image by color. “Creating a still image” refers to creating a still image while a moving image is being captured. “Image synthesis” is, for example, the synthesis of a normal image and a fluorescent image as described above. “Moving the imaging location” means changing the observation direction of the endoscope so that the region of the observed portion displayed on the monitor is changed. “Switching the wavelength of illumination light” refers to switching, for example, white light and excitation light as illumination light. “Presence / absence of excitation light illumination” is, for example, switching between the case of illuminating the excitation light together with white light and the case of not illuminating the excitation light. “Switching the display format of an image” refers to, for example, a display of whether to switch the format of an image such as a normal image or a fluorescent image as an image to be displayed on a monitor, or to display images of these multiple formats individually or simultaneously. It is to switch the method. “Lens cleaning” is to remove deposits attached to the lens by a cleaning mechanism provided in the endoscope.

  The item selection unit 39b receives information on the position of the predetermined portion where the menu is displayed and information on the position of the representative point from the memory 34, and selects one of the support items based on this information. . Specifically, for example, when the representative point of the treatment tool is included in the area in the image where the support item of the menu is displayed, the support item is selected.

The treatment tool constituting the surgery support system of the present embodiment is a forceps 6 as shown in FIG. 5A, for example. The forceps 6 shown in FIG. 5A includes an operating part 6a for holding an observed part or performing a predetermined treatment, a handle part 6b having a thin cylindrical shape provided with the operating part 6a, and an operator operating it. And an operation unit 6c for performing the operation of the unit 6a. And, as shown in FIG. 5B, a representative point extraction marker M is provided on the operating portion 6 a at the distal end portion of the forceps 6. The color of the marker M is not particularly limited, but it is preferable to select a hue range with a small abundance in the living body in consideration of the color of blood in order to facilitate identification in the abdominal cavity on a normal image. When expressed in the L ^* a ^* b ^* (CIE) color system, hues in the range of (34, 80, −106) to (40, −20, −14) are preferable. Further, the color of the marker M is particularly preferably blue. The marker M is formed by including a fluorescent material that emits fluorescence of 800 nm or more and 850 nm or less when irradiated with near infrared excitation light of 750 nm or more and 800 nm or less in order to facilitate identification in the abdominal cavity on the fluorescence image. You can also. For example, as such a fluorescent material, it is desirable to use a fluorescent agent such as ICG, which will be described later, or a material that emits fluorescence in the same wavelength band as the wavelength band of autofluorescence. In addition, forceps without a marker can also be used. In such a case, the extraction unit that extracts the representative point of the treatment tool recognizes the image of the entire operation unit 6a of the forceps 6 and extracts the representative point from the image.

  The video output unit 35 receives the normal image signal and the fluorescence image signal output from the image processing unit 33 or these images superimposed with the menu via the bus, and performs a predetermined process to generate a display control signal. The display control signal is output to the monitor 4.

  The operation unit 36 accepts necessary inputs by the operator in advance such as various operation instructions and control parameters. The TG 37 outputs a driving pulse signal for driving the high-sensitivity imaging device 24, the imaging device 26 of the imaging unit 20, and the LD driver 45 of the light source device 2 described later.

  The CPU 36 controls the entire apparatus. And in this invention, CPU36 receives the assistance item selected by the item selection part via the bus | bath, and issues an instruction | indication to the surgery assistance means which performs an applicable operation | work so that the surgery assistance work regarding the said assistance item may be performed. The surgery support means refers to a component that actually performs the surgery support work. For example, the surgery support means is the imaging unit 20 when the support item relates to the magnification of the image, and the support item is a change in the brightness of the image, a switching of the wavelength of the illumination light, and the presence or absence of illumination of the excitation light. In the case of switching, the light source device 2 is used. For example, in the case where the endoscope is a flexible mirror and the support item is related to the movement of the imaging location, it is a drive mechanism for the insertion portion of the flexible mirror, and the support item Is the video output unit 35 when the display is related to switching of the image display format.

  As shown in FIG. 4, the light source device 2 condenses the normal light source 40 that emits normal light (white light) L <b> 1 having a broadband wavelength of about 400 to 700 nm and the normal light L <b> 1 emitted from the normal light source 40. The condensing lens 42 and the normal light L1 collected by the condensing lens 42 are transmitted, the excitation light L2 described later is reflected, and the normal light L1 and the excitation light L2 are incident on the incident end of the optical cable LC. And a dichroic mirror 43. For example, a xenon lamp is used as the normal light source 40. A diaphragm 41 is provided between the normal light source 40 and the condenser lens 42, and the amount of the diaphragm is controlled based on a control signal from ALC (Automatic light control).

  Further, the light source device 2 emits near-infrared light of 750 to 800 nm that excites the fluorescent dye ICG to generate fluorescence, as an excitation light L2, and an LD driver 45 that drives the LD light source 44. , A condenser lens 46 that condenses the excitation light L2 emitted from the LD light source 44, and a mirror 47 that reflects the excitation light L2 collected by the condenser lens 46 toward the dichroic mirror 43.

  In the present embodiment, the light in the wavelength band as described above is used as the excitation light L2. However, the excitation light L2 is not limited to the light in the wavelength band, and is not limited to the fluorescent material that forms the marker M and the subject. It is determined appropriately depending on the type of fluorescent dye to be introduced or the type of biological tissue to be autofluorescent.

  The LD driver 45 controls the intensity of the excitation light output from the LD light source 44.

Next, the operation of the surgery support system of this embodiment will be described.
First, after the hard insertion portion 30 and the cable 5 to which the optical cable LC is connected are attached to the imaging unit 20, the light source device 2, the imaging unit 20, and the image processing device 3 are powered on and driven.

  Next, as shown in FIG. 6, the operator inserts the hard insertion portion 30 and the forceps 6 into the abdominal cavity, the observation portion is sandwiched by the forceps 6, and the distal end of the hard insertion portion 30 is the observation portion. It is installed in the vicinity.

  Then, the normal light L1 emitted from the normal light source 40 of the light source device 2 is incident on the hard insertion portion 30 via the condenser lens 42, the dichroic mirror 43, and the optical cable LC, and is irradiated from the irradiation window 30d of the hard insertion portion 30. Irradiate the observation part. On the other hand, the excitation light L2 emitted from the LD light source 44 of the light source device 2 is incident on the hard insertion section 30 via the condenser lens 46, the mirror 47, the dichroic mirror 43, and the optical cable LC, and the irradiation window of the hard insertion section 30 The portion to be observed is irradiated with normal light from 30d.

  Then, a normal image based on the reflected light reflected from the observed portion by the irradiation of the normal light L1 is captured, and a fluorescent image based on the fluorescence emitted from the observed portion by the irradiation of the excitation light L2 is captured. It should be noted that ICG is administered to the observed part in advance, and fluorescence emitted from the ICG is imaged.

  More specifically, when the normal image is captured, the normal image L3 based on the reflected light reflected from the observed portion by the irradiation of the normal light L1 is incident from the tip 30Y of the insertion member 30b, and the inside of the insertion member 30b. Are guided by the lens group and emitted toward the imaging unit 20.

  The normal image L3 incident on the imaging unit 20 is reflected by the dichroic prism 21 in a direction perpendicular to the imaging element 26, and is imaged on the imaging surface of the imaging element 26 by the second imaging optical system 25. 26 sequentially captures images at predetermined intervals. In the present embodiment, it is assumed that a normal image is captured at a frame rate of 30 fps.

  The normal image signal sequentially output from the image sensor 26 is subjected to CDS / AGC (correlated double sampling / automatic gain control) processing and A / D conversion processing in the imaging control unit 27, and then the image is transmitted through the cable 5. The data is sequentially output to the processing device 3.

  On the other hand, when the fluorescent image is picked up, a fluorescent image L4 based on the fluorescence emitted from the observed portion by the irradiation of the excitation light is incident from the tip 30Y of the insertion member 30b and guided by the lens group in the insertion member 30b. And emitted toward the imaging unit 20.

  The fluorescent image L4 incident on the imaging unit 20 passes through the dichroic prism 21 and the excitation light cut filter 22, and then is imaged on the imaging surface of the high-sensitivity imaging device 24 by the first imaging optical system 23, and has high sensitivity. Images are taken at a predetermined interval by the image sensor 24. In the present embodiment, it is assumed that the fluorescence image is captured at a frame rate of 5 to 10 fps.

  The fluorescent image signals sequentially output from the high-sensitivity image sensor 24 are subjected to CDS / AGC (correlated double sampling / automatic gain control) processing and A / D conversion processing in the imaging control unit 27, and then passed through the cable 5. Are sequentially output to the image processing apparatus 3.

  The normal image signal input to the image processing device 3 is temporarily stored in the normal image input controller 31 and then stored in the memory 34. The normal image signal for each frame read out from the memory 34 is subjected to gradation correction processing and sharpness correction processing in the image processing unit 33, and further, the representative point P of the forceps 6 is extracted from the image. Processing and menu N superimposing processing are performed. After these image processes are performed, normal image signals are sequentially output to the video output unit 35.

  Then, the video output unit 35 performs a predetermined process on the input normal image signal to generate a display control signal, and sequentially outputs the display control signal for each frame to the monitor 4. The monitor 4 displays a normal image based on the input display control signal. FIG. 7 is a schematic view showing a state where a normal image on which a menu is superimposed is displayed through the above-described steps.

  On the other hand, the fluorescence image signal input to the image processing device 3 is temporarily stored in the fluorescence image input controller 32 and then stored in the memory 34. Then, the fluorescence image signal for each frame read from the memory 34 is subjected to predetermined image processing in the image processing unit 33, and then the representative of the forceps 6 in the image as in the case of the normal image. A point P extraction process and a menu N superimposition process are performed. After these image processes are performed, the fluorescence image signal is sequentially output to the video output unit 35.

  The video output unit 35 performs a predetermined process on the input fluorescent image signal to generate a display control signal, and sequentially outputs the display control signal for each frame to the monitor 4. The monitor 4 displays a fluorescent image based on the input display control signal.

  In the present embodiment, the normal image and the fluorescence image are switched and displayed in accordance with an instruction from the operator. However, the present invention is not limited to this and may be displayed simultaneously. In the case where the normal image and the fluorescent image are displayed simultaneously, it is sufficient that the menu is superimposed only on the normal image, for example.

  The surgeon operates the forceps 6 while looking at the monitor 4 on which an image as shown in FIG. 7 and a menu superimposed on the image are displayed. When the surgeon desires screen adjustment as the operation support operation, the operator selects a support item related to the screen adjustment from the menu, and matches the representative point P of the forceps 6 to the support item on the screen of the monitor 4. For example, if you want to enlarge the screen, the representative point P is set to the support item “Zoom +” in the menu, and if you want to increase the screen brightness, the representative item “Brightness +” in the menu is representative. Point P will be matched.

  The item selection unit 39b receives information from the memory 34 that the representative point P has been matched as described above, information on the position of the predetermined portion where the menu N is displayed, and information on the position of the representative point P. To select one of the support items. The selected support item is output to the CPU via the bus, and the CPU 38 gives an instruction to the operation support means for performing the corresponding operation so as to execute the operation support operation related to the support item. For example, if the support item is “zoom +” as described above, the CPU 38 instructs the imaging unit 20 to enlarge the image.

  As described above, the surgeon can use the treatment tool to select a support item, control the operation support means, and execute the operation support operation. As a result, in the system and method for supporting the operation for performing the treatment on the observed portion displayed on the screen, the operator himself / herself is not involved with the assistant so that the operation support operation desired by the operator is appropriately performed. It becomes possible to control the operation support means for performing the operation support operation.

(Design change of operation support system and operation support method using it)
In the above embodiment, the marker is formed of a paint or a fluorescent material, but the present invention is not limited to this.

  For example, as shown in FIG. In this case, the representative point can be easily extracted by detecting the light emission of the light emitting diode D. Furthermore, the convenience of the operator's menu selection can be improved by utilizing the lighting state of the light emitting diode D. For example, as shown in FIG. 8b, the treatment tool includes a switch SW for turning on the light-emitting diode D in the handle portion of the treatment tool, and the image processing unit is only in the case where the light-emitting diode D is in a lighting state. When the representative point position is extracted from the lighted portion of the light emitting diode, the representative point P is not extracted unless the switch SW is pressed. It can be lowered by the switch SW. Thus, it is possible to prevent a surgical support operation not desired by the operator from being performed by touching an unintended support item while the representative point is moving. On the other hand, the treatment instrument is provided with a switch SW for turning on the light emitting diode D in the handle portion of the treatment instrument as shown in FIG. 8b, and the item selection unit is provided that the light emitting diode D is in a lighting state. When the support item is set to be selected, since the support item is not selected unless the switch SW is pressed, the surgeon can make a final decision on the selection of the support item by the switch SW. Therefore, the same effect as described above can be obtained.

  Furthermore, the treatment instrument is provided with a switch SW for causing the light emitting diode D to blink on the handle portion of the treatment instrument as shown in FIG. 8b, and the item selection unit is provided that the light emitting diode D is in a blinking state. Even when the support item is set to be selected, since the support item is not selected unless the switch SW is pressed, the surgeon can make a final decision on the selection of the support item with the switch SW. Therefore, the same effect as described above can be obtained.

  Further, the menu forming unit may form a menu in conjunction with the lighting state of the light emitting diode. Thereby, only when the menu is necessary, the menu is displayed at the same time as the light emitting diode D is turned on, so that a wide operating field when the menu is unnecessary can be secured.

  In addition, the imaging unit receives reflected light from the observed portion generated by irradiating illumination light as an image and captures a visible image (for example, an image based on white light in the above embodiment). In this case, it is preferable that the light-emitting diodes generate blue light from the viewpoint of selecting a hue range with a small abundance in the living body. The blue light is preferably light having a wavelength included in 400 to 500 nm, and particularly preferably light having a wavelength included in 430 to 470 nm. Thereby, the light emission of the light emitting diode can be clearly identified for blood vessels and organs in the abdominal cavity. On the other hand, in the case where the imaging unit receives fluorescence from the observed portion generated by irradiating excitation light as an image and captures a fluorescent image, the light-emitting diode is invisible and is a living body of light. From the viewpoint of high transmittance and a sensitivity range of the image sensor, it is preferable that light included in the fluorescence wavelength band is generated. The light contained in the fluorescence wavelength band is preferably light contained in a wavelength range of 800 to 1000 nm, and particularly preferably light contained in a wavelength range of 850 to 950 nm. Thereby, the representative point P of the forceps 6 can be extracted also in the fluorescence image.

  In addition, as shown in FIG. 9, the representative point extraction marker M is provided on the operating portion 6a of the forceps 6 and the light emitting diode D is provided, and the final selection of the support item is performed using the lighting or blinking state of the light emitting diode D. It may be used as a judgment signal.

  In the above embodiment, the case where the menu is superimposed on the normal image has been described in detail. However, as described above, the present invention may superimpose the menu on the fluorescent image. FIG. 10A is a schematic diagram showing a normal image in a state where the marker M provided on the forceps 6 can be recognized. FIG. 10B shows a case where the image display format is changed from the normal image state to the fluorescent image. It is the schematic which shows a fluorescence image. As can be seen from FIG. 10b, in the present invention, if the marker M is formed of a fluorescent material, a representative point can be extracted based on the marker MG that emits fluorescence even in a fluorescent image. Therefore, as shown in FIG. 11, by displaying a menu in the fluorescent image, the surgeon himself uses the treatment tool to select a support item and execute a surgical support operation as in the case of the normal image. be able to. As a result, in the system and method for supporting the operation for performing the treatment on the observed portion displayed on the screen, the operator himself / herself is not involved with the assistant so that the operation support operation desired by the operator is appropriately performed. Surgery support work can be performed. The same applies to the case where the light emitting diode generates light included in the fluorescence wavelength band.

  In the present invention, the size and shape of the menu N are not limited to those shown in FIG. 7 described above. For example, the menu in the form of a matrix in which support items I are arranged at predetermined positions as shown in FIG. N may also be used, or a menu N may be used in which the entire image display area in which an image is displayed in the screen as shown in FIG.

  Moreover, although said embodiment demonstrated the case where the surgery assistance system of this invention was applied to the endoscope system, this invention is not limited to this. For example, the present invention can be applied to an operation using a telesurgical robot because it is common in that the operation of the observed portion is performed using the treatment tool while viewing the screen.

DESCRIPTION OF SYMBOLS 1 Rigid endoscope system 2 Light source apparatus 3 Image processing apparatus 4 Monitor 5 Cable 6 Forceps 6a Forceps operation part 6b Forceps handle part 6c Forceps operation part 10 Rigid mirror imaging apparatus 20 Imaging part 27 Imaging control unit 30 Rigid insertion part 30a Connection Member 30b Insertion member 31 Normal image input controller 32 Fluorescent image input controller 33 Image processing unit 34 Memory 35 Video output unit 36 Operation unit 39a Menu formation unit 39b Item selection unit 40 Normal light source 44 LD light source D Light emitting diode I Support item L1 Normal light L2 Excitation light L3 Normal image L4 Fluorescence image LC Optical cable M Marker MG Marker N Menu P Representative point SW switch

Claims (14)

An imaging unit configured to capture an image of the observed part; and an image display unit configured to display the image of the observed part on a screen, and the subject to be performed using a treatment tool while viewing the screen on which the image is displayed. In the surgery support system that supports the surgery of the observation part,
An extraction unit that extracts a position of a representative point of the treatment tool in the image when the treatment tool is included in the image;
A menu forming unit for forming a menu of one or more support items related to a surgery support operation for supporting a surgery performed by the treatment tool;
An image combining unit that combines the image and the menu so that the menu can be displayed on an image display area where the image is displayed in the screen so as to be superimposed on a predetermined portion of the image;
An item selection unit that selects any one of the support items based on the information on the position of the predetermined portion on which the menu is displayed and the information on the position of the representative point;
An operation support system comprising operation support means for executing the operation support operation related to the support item based on the selected support item. The treatment tool has a marker at a position recognizable in the image of the treatment tool,
The surgery support system according to claim 1, wherein the extraction unit extracts the position of the marker as the position of the representative point of the treatment instrument.   The surgery support system according to claim 2, wherein the marker is a light emitting diode. The treatment tool includes a switch for lighting the light emitting diode on a handle portion of the treatment tool,
The surgical operation support system according to claim 3, wherein the extraction unit extracts the position of the representative point from a lighting portion of the light emitting diode only when the light emitting diode is in a lighting state. . The treatment tool includes a switch for lighting the light emitting diode on a handle portion of the treatment tool,
The surgical operation support system according to claim 3, wherein the item selection unit selects the support item on condition that the light emitting diode is in a lighting state. The treatment instrument includes a switch for blinking the light emitting diode on a handle portion of the treatment instrument,
The surgery support system according to claim 3 or 4, wherein the item selection unit selects the support item on condition that the light emitting diode is in a blinking state.   The operation support system according to any one of claims 3 to 6, wherein the menu forming unit forms the menu in conjunction with a lighting state of the light emitting diode. The imaging unit receives the reflected light from the observed portion generated by irradiating illumination light as the image, and captures a visible image;
The surgery support system according to any one of claims 3 to 7, wherein the light-emitting diodes generate blue light. The imaging unit receives fluorescence from the observed portion generated by irradiating excitation light as the image, and captures a fluorescence image;
The surgery support system according to any one of claims 3 to 7, wherein the light emitting diode generates light included in the fluorescence wavelength band.   The operation support system according to any one of claims 1 to 9, wherein the treatment tool includes a switch for selecting whether or not the menu is displayed on a handle portion of the treatment tool.   The support items include image magnification change, image brightness change, imaging location movement, illumination light wavelength switching, excitation light illumination presence / absence switching, image display format switching, lens washing and The operation support system according to any one of claims 1 to 10, wherein the operation support system relates to at least one operation support operation among image synthesis.   The surgery support system according to claim 1, wherein the treatment tool is a forceps.   The surgery support system according to any one of claims 1 to 12, wherein the surgery support system is used in an endoscope system. An operation that captures an image of the observed portion, displays the image of the observed portion on a screen, and supports the operation of the observed portion performed using a treatment tool while viewing the screen on which the image is displayed. In the support method,
When the treatment tool is included in the image, extract the position of the representative point of the treatment tool in the image,
Forming a menu of one or more support items related to surgery support work for supporting the surgery performed by the treatment tool;
The image and the menu are combined so that the menu can be displayed on an image display area where the image is displayed in the screen so as to be superimposed on a predetermined portion in the image.
Based on the information on the position of the predetermined portion where the menu is displayed and the information on the position of the representative point, select one of the support items,
The operation support method according to claim 1, wherein the operation support operation for the support item is executed based on the selected support item.

Images