JP2014000120A - Endoscope and endoscope system including the same - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve positioning accuracy in super-resolving processing when acquiring multiple low-resolution images with deviated imaging positions required for super-resolving processing with the use of blurring.SOLUTION: An endoscope includes: an insertion unit 11 to be inserted to the inside of an observation object; an illumination window 16 arranged in the insertion unit and illuminating a subject S inside the observation object; an imaging unit 14 arranged in the insertion unit and imaging a subject; and a marking unit 17 arranged in the insertion unit and allowing a marker material M to adhere onto a subject. Especially, the marking unit injects a marker material to a subject by tension of an operation fluid. The marker material contains a fluorescent material and emits light by light to be emitted from the illumination window.

Description

  The present invention relates to an endoscope that images the inside of an observation target that cannot be directly observed from the outside and an endoscope system including the endoscope, and particularly suitable for performing super-resolution processing on an image obtained by imaging. The present invention relates to an endoscope and an endoscope system including the endoscope.

  In the endoscope, an image sensor is arranged at the distal end of an insertion part that is inserted into an observation target such as a human body, and an image obtained by imaging with the image sensor is displayed on a monitor. So-called electronic endoscopes are widely used. Even in this type of endoscope, it is desirable to form the insertion portion as thin as possible. However, since the size of the image sensor is restricted, an image sensor with high resolution cannot be employed. There has been a problem that it is not possible to sufficiently satisfy the desire to observe internal tissues such as internal organs of the human body with fine images.

  In response to such a demand, the resolution of an image captured by an endoscope is reduced by using a so-called inter-frame super-resolution technique that generates a high-resolution image from a plurality of low-resolution images obtained by imaging. It is performed (patent document 2). In addition, since super-resolution processing requires a plurality of images whose imaging positions are shifted, a plurality of pixels whose imaging positions are shifted using a pixel shifting mechanism that relatively vibrates the imaging element and the objective lens with an actuator. A technique for acquiring the image is also known (Patent Document 1).

JP-A-6-178214 Special table 2010-512173 gazette

  In the inter-frame super-resolution technique, it is necessary to grasp the amount of misalignment between low-resolution images in order to align a plurality of low-resolution images. When the pixel shifting mechanism is used, for example, by controlling the applied voltage in consideration of the hysteresis characteristic of the expansion / contraction amount of the stacked piezoelectric element used in the pixel shifting mechanism, the positional shift amount between images can be managed. Based on this, it is possible to obtain an advantage that alignment with sub-pixel accuracy can be performed. However, in the configuration using the pixel shifting mechanism, it is necessary to provide the pixel shifting mechanism in the insertion portion together with the image sensor, and there is a problem that the insertion portion becomes thick.

  On the other hand, in the case of an endoscope, it is inevitable that the insertion part shakes due to camera shake, that is, the hand holding the endoscope shakes, but the imaging position is shifted by this camera shake, so if this camera shake is used, pixel shifting will occur. Without providing a mechanism, it is possible to acquire a plurality of images whose imaging positions are shifted.

  However, in the case of acquiring a plurality of images whose imaging positions are shifted by using camera shake in this way, unlike the case of using the pixel shifting mechanism, the amount of positional shift between images is not known. Therefore, it is necessary to acquire the amount of misalignment from the image itself. For this purpose, a so-called feature point matching method is used in which feature points are extracted from the image and the amount of misalignment is obtained from the correspondence between the feature points. That's fine. In general inter-frame super-resolution processing, a predetermined standard image and other reference images are defined for a plurality of captured frames, and the positional deviation amount of the reference image with respect to the standard image is set, for example, SURF (Speeded Up Robust Features). And a feature point matching method such as SIFT (Scale-invariant feature transform), and the image is reconstructed by convolving a plurality of images after alignment. However, when the subject is an internal tissue such as an organ, the subject has almost no characteristic texture, and there are few places where the color and shape change greatly. There is a problem in that the effect of increasing the resolution by the inter-frame super-resolution processing is small because it cannot be performed with high accuracy.

  The present invention has been devised to solve such problems of the prior art, and the main object of the present invention is to use a plurality of low-level images that are shifted in imaging positions necessary for super-resolution processing using camera shake. An object of the present invention is to provide an endoscope and an endoscope system including the endoscope that are configured to improve the accuracy of alignment in super-resolution processing when a resolution image is acquired.

  The endoscope according to the present invention includes an insertion unit that is inserted into an observation target, an illumination unit that is provided in the insertion unit and illuminates a subject inside the observation target, and the subject that is provided in the insertion unit. And a marking unit that is provided in the insertion unit and attaches a marker material to the subject.

  In addition, the endoscope system of the present invention is an image that performs super-resolution processing for generating a high-resolution image from the endoscope configured as described above and a plurality of low-resolution images acquired by imaging by the imaging unit. A processing device, and when the image processing device performs alignment of the plurality of low resolution images in the super-resolution processing, the feature point is located at a position corresponding to the marker material in the low resolution image. It is set as the structure to extract.

  According to the present invention, the marker material is imaged together with the subject by the imaging unit by attaching the marker material to the subject. Thereby, when performing alignment of a plurality of low resolution images acquired by imaging at the imaging unit in the super-resolution processing, feature points are reliably extracted at positions corresponding to the marker material in the low resolution image. , Positioning accuracy can be increased.

Overall configuration diagram showing an endoscope system according to the present embodiment Sectional drawing which shows the front-end | tip part 12 of the insertion part 11 of the endoscope 1 Schematic diagram showing the schematic configuration of the pressure generator 22 Sectional view of marking part 17 Sectional drawing which expands and shows the principal part of the marking part 17 Explanatory drawing which shows the condition (pressure profile) of the pressure p of the working fluid generated with the pressure generation part 22 The side view which shows the condition of the marking which attaches the marker material M to the to-be-photographed object S Functional block diagram showing the super-resolution processing unit 32 Explanatory drawing explaining the outline | summary of the reconstruction process performed in the reconstruction process part 43 of the super-resolution process part 32 The perspective view which shows the condition of the marking which attaches the marker material M to the to-be-photographed object S Explanatory drawing which shows the low-resolution image (captured image) obtained by the imaging by the endoscope 1

  A first invention made to solve the above-described problems is an insertion unit that is inserted into an observation target, an illumination unit that is provided in the insertion unit and illuminates a subject inside the observation target, and the insertion An imaging unit that is provided in a part and images the subject, and a marking part that is provided in the insertion unit and attaches a marker material to the subject.

  According to this, by attaching the marker material to the subject, the marker material is imaged together with the subject by the imaging unit. Thereby, when performing alignment of a plurality of low resolution images acquired by imaging at the imaging unit in the super-resolution processing, feature points are reliably extracted at positions corresponding to the marker material in the low resolution image. , Positioning accuracy can be increased.

  In the second aspect of the present invention, the marking portion emits a marker material toward the subject with the pressure of the working fluid.

  According to this, since the marker material can be attached to the subject without bringing the marking portion into contact with the subject, the operation of attaching the marker material to the subject is facilitated, and the subject is damaged by contact with the marking portion. You can avoid that.

  According to a third aspect of the invention, the marker material includes a fluorescent material and emits light by light emitted from the illumination unit.

  According to this, the brightness of the marker material image shown in the captured image is higher than that of the periphery, and the feature points are extracted at the position of the region where the brightness is high. For this reason, it becomes possible to extract a feature point more reliably at a position corresponding to the marker material, and it is possible to further improve the accuracy of alignment.

  According to a fourth aspect of the present invention, the marker material is a liquid, and the marking portion includes a meniscus forming portion that forms a liquid mass of the marker material in a meniscus shape.

  According to this, a liquid mass of the marker material having a uniform size can be formed. In this case, if a plurality of meniscus forming portions are arranged at equal intervals, the liquid mass of the marker material can be continuously ejected at a constant timing.

  Moreover, 5th invention is set as the structure by which the said marking part was provided in the said insertion part so that advancement / retraction was possible.

  According to this, by moving the marking portion forward, the marking portion can be brought close to the subject without moving the insertion portion itself, so that the marker material can be accurately attached to a required position on the subject. Further, by retracting the marking portion, the insertion portion can be smoothly inserted into and removed from the observation target without damaging the observation target with the marking portion or damaging the marking portion.

  The sixth invention includes an endoscope configured as described above, an image processing apparatus that performs super-resolution processing for generating a high-resolution image from a plurality of low-resolution images acquired by imaging by the imaging unit, and And the image processing device extracts a feature point at a position corresponding to the marker material in the low resolution image when aligning the plurality of low resolution images in the super-resolution processing. And

  According to this, since the feature point is reliably extracted at the position corresponding to the marker material in the low-resolution image, the alignment accuracy can be improved.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is an overall configuration diagram showing an endoscope system according to the present embodiment. The endoscope system includes an endoscope 1 that is inserted into a human body (observation target) and images a subject S (an outer surface of an internal organ or an inner wall of a digestive tract) such as an internal organ of the body; Generated by a controller (control device) 2 that controls the endoscope 1, an image processing unit (image processing device) 3 that performs super-resolution processing on an image captured by the endoscope 1, and the image processing unit 3 And a monitor (display device) 4 for displaying the processed image.

  The endoscope 1 includes an insertion portion 11 that is inserted into the body. In the present embodiment, a so-called rigid endoscope in which the insertion portion 11 is mainly configured in a hard rod shape will be described, but the present invention may be applied to a flexible endoscope using a flexible material for the insertion portion. . In endoscopic surgery using a laparoscope, which is a typical example of a rigid endoscope, for example, a minute hole is made in a part of the abdomen, and the insertion part 11 is inserted from there to observe the inside of the body cavity. Perform surgery. An imaging unit 14 that images a subject (such as an organ) S inside the body through an imaging window 13 is provided at the distal end 12 of the insertion unit 11. In addition, an illumination window (illumination unit) 16 that emits light guided by the optical fiber 15 to illuminate the subject S is provided at the distal end portion 12 of the insertion unit 11.

  Furthermore, a marking portion 17 for attaching the marker material M to the subject S is provided at the distal end portion 12 of the insertion portion 11 of the endoscope 1. In particular, in the present embodiment, the marker material M forms a liquid, and the marking unit 17 forms a liquid mass of the marker material M in a required size and is directed toward the subject S with the pressure of the working fluid (air). The marker material M is ejected. The marker material M includes a fluorescent material, and the marker material M emits light by light emitted from the illumination window 16 in order to illuminate the subject S.

  The controller 2 includes a light source unit 21 that sends illumination light to the illumination window 16 provided in the insertion unit 11 of the endoscope 1 via the optical fiber 15, and a marking unit provided in the insertion unit 11 of the endoscope 1. 17, a pressure generating unit 22 that generates a pressure of a working fluid for injecting the marker material M, a marker material storing unit 23 that stores the marker material M, an operation switch 24 that instructs various operations, and an imaging unit 14. Are operated, and an image capturing control unit 25 that receives image data from the image capturing unit 14 and a main control unit 26 that collectively controls each unit are provided.

  The main control unit 26 operates the imaging control unit 25 and the light source unit 21 in synchronization. In this embodiment, the imaging sensor 52 (see FIG. 2) provided in the imaging unit 14 performs a global shutter operation at, for example, 30 fps (Frames Per Second) (the shutter cycle is 33.3 ms), but the charge accumulation period (shutter opening time) ) Is set to about 1 ms. The main control unit 26 outputs a light emission instruction to the light source unit 21 in synchronization with this timing, and as a result, the light source unit 21 emits light only during a period corresponding to the charge accumulation period. This realizes high-luminance light emission of a light source (for example, white LED) mounted on the light source unit 21, acquires a low-resolution image with less blur, and eliminates a factor that suppresses the super-resolution effect.

  The endoscope 1 is provided with a trigger switch 18 for instructing the injection of the marker material M at the marking unit 17, and a trigger signal output from the trigger switch 18 is input to the main control unit 26 of the controller 2 to trigger In response to the operation of the switch 18, the pressure generating unit 22 operates and the marker material M is ejected from the marking unit 17. Further, the operation switch 24 provided in the controller 2 gives instructions such as lighting of the light source unit 21, imaging by the imaging unit 14, execution of image processing by the image processing unit 3, and switching between moving images and still images. .

  The marking unit 17 provided in the endoscope 1, the pressure generation unit 22 and the marker material storage unit 23 provided in the controller 2 are connected via a connection pipe 27. The connection pipe 27 includes a working fluid supply path 28 that supplies the working fluid from the pressure generating unit 22 to the marking unit 17, and a marker material supply path 29 that supplies the marker material M from the marker material storage unit 23 to the marking unit 17. Is provided.

  The connection pipe 27 is pulled out from a pull-out portion 19 provided on the base side of the insertion portion 11 and connected to the controller 2. The insertion hole of the connection pipe 27 provided in the lead-out portion 19 may be used also as a forceps hole for inserting forceps, or may be dedicated to the connection pipe 27 separately from the forceps hole. Good.

  Further, in the present embodiment, the marking portion 17 is provided in the insertion portion 11 so as to be movable back and forth in the longitudinal direction of the insertion portion 11, and the portion of the connection pipe 27 drawn out from the drawer portion 19 of the insertion portion 11 is manually formed. The marking part 17 can be advanced and retracted by picking and pulling.

  The image processing unit 3 is configured by a PC or the like, and includes a storage unit 31 that stores a captured image acquired by the imaging unit 14 provided in the insertion unit 11 of the endoscope 1, and an ultra-high resolution image that is generated from the captured image. A resolution processing unit 32, a display control unit 33 that controls the monitor 4, and a main control unit 34 that collectively controls each unit are provided. The image processing unit 3 is connected to an input unit 35 constituted by a keyboard or the like. Here, drive control (frame rate, exposure time, white balance, etc.) of the imaging unit 14 and a super-resolution processing unit. For example, parameters related to super-resolution processing at 32 are set. The main control unit 25 of the controller 2 and the main control unit 34 of the image processing unit 24 can bidirectionally transmit information via a signal line (serial communication) (not shown). The bit of the control parameter related to the specific drive control of the imaging unit 14 can be referred to through the control unit, and the value of the control parameter already set in the imaging unit 14 (imaging sensor 52, see FIG. 2) can be referred to. Can be rewritten and set in the imaging unit 14 again.

  FIG. 2 is a cross-sectional view showing the distal end portion 12 of the insertion portion 11 of the endoscope 1. At the distal end portion 12 of the insertion portion 11, an imaging portion 14, an illumination optical fiber 15, and a marking portion 17 are accommodated in a cylindrical casing 51. The illumination light guided by the optical fiber 15 is emitted through an illumination window 16 made of a cover glass. The imaging unit 14 includes an imaging sensor 52 composed of CMOS (Complementary Metal Oxide Semiconductor) and an optical system 53 including a plurality of lenses, and the reflected light from the subject S illuminated by the illumination light is reflected. The light enters the optical system 53 through the imaging window 13 made of a cover glass. The marking portion 17 is inserted into a guide hole 55 extending in the longitudinal direction of the insertion portion 11 so as to freely advance and retract.

  FIG. 3 is a schematic diagram showing a schematic configuration of the pressure generator 22. The pressure generating unit 22 includes a diaphragm pump, and includes a diaphragm 61 to which the working fluid supply path 28 is connected, a voice coil motor 62 that drives the diaphragm 61, and a check valve 63 that is connected to the diaphragm 61. ing. In the pressure generator 22, a suction process in which the working fluid is sucked into the diaphragm 61 through the check valve 63 by a reciprocating motion of the output shaft of the voice coil motor 62, and a discharge process in which the working fluid in the diaphragm 61 is discharged. And the working fluid is sent to the marking portion 17 of the endoscope 1 through the working fluid supply path 28 in the discharge process.

  In the present embodiment, air is used as the working fluid, and the inlet side of the check valve 63 may be opened to the atmosphere so that outside air is directly taken into the diaphragm 61, but clean air is stored. A tank may be provided, and the air in the tank may be taken into the diaphragm 61.

  Further, when the secondary pressure (back pressure) of the pressure generating unit 22, that is, the pressure in the body cavity into which the insertion unit 11 of the endoscope 1 is inserted is higher than the atmospheric pressure, the marker is considered in consideration of the pressure difference. It is necessary to control the pressure of the pressure generating unit 22 so that the material M is appropriately injected. Therefore, an opening for obtaining back pressure may be provided in the vicinity of the marking portion 17 in the insertion portion 11 and this opening may be connected to the check valve 63 of the pressure generating portion 22 through the back pressure introduction path. Thereby, the pressure control of the pressure generation part 22 becomes easy. In this case, as shown in FIG. 1, the connection pipe 27 connecting the marking unit 17 and the controller 2 is provided with a back pressure introduction path in addition to the working fluid supply path 28 and the marker material supply path 29.

  The pressure generating unit 22 is preferably composed of a positive displacement pump capable of precisely controlling the discharge pressure and discharge amount of the working fluid. In addition to the diaphragm pump, a rotary pump or a syringe pump is also possible. is there. In addition, an actuator in which piezoelectric elements are stacked on a circular metal diaphragm may be employed as the diaphragm pump, or a stacked piezoelectric element may be used instead of the voice coil motor 62.

  FIG. 4 is a cross-sectional view of the marking portion 17 and shows the transition state of the liquid mass of the marker material M. FIG. 5 is an enlarged cross-sectional view showing a main part of the marking portion 17.

  As shown in FIG. 4, the marking portion 17 has a tubular shape, and a plurality of meniscus forming portions 71 that form a liquid mass of the marker material M in a meniscus shape are arranged at equal intervals. The meniscus forming portion 71 is formed with an annular convex portion 72 that protrudes inward, and has an inner diameter smaller than that of other portions.

  Here, when the marker material M is water-based, as shown in FIG. 5, an oil repellent (hydrophilic) process is performed on the surface portion 73 that forms the liquid mass of the marker material M in the meniscus forming portion 71, and the others The surface portion 74 and the tip portion for releasing the liquid mass of the marker material M into the body cavity are subjected to a water repellent (lipophilic) treatment. On the other hand, when the marker material M is oily, the oil repellent (hydrophilic) treatment and the water repellent (lipophilic) treatment are reversed. Thereby, the liquid lump of a required magnitude | size can be formed stably.

  In the marking unit 17 configured as described above, when the marker material M is supplied, a liquid mass of the marker material M is formed in a meniscus shape in the meniscus forming unit 71 as shown in FIG. When the pressure of the working fluid is increased, the liquid mass of the marker material M is pushed out as shown in FIG. 4B, and the liquid mass of the marker material M forms meniscus as shown in FIG. 4C. Fly away from the unit 71. Here, the liquid mass of the marker material M separated from the meniscus forming portion 71 at the tip flies within the body cavity and adheres to the subject S, and the liquid mass of the marker material M separated from the other meniscus forming portions 71 is adjacent to the meniscus. It reaches the forming portion 71 and returns to the state shown in FIG. 4A, and the liquid mass of the marker material M is formed in a meniscus shape at each meniscus forming portion 71.

  In this way, the liquid mass of the marker material M separated from the meniscus forming portion 71 is made to reach the adjacent meniscus forming portion 71 so that the states of FIGS. 4A to 4C are stably repeated. Is necessary to appropriately set the relationship between the arrangement interval L1 of the meniscus forming portion 71, the length L2 of the annular convex portion 72, and the inner diameter r of the annular convex portion 72, and the working fluid generated by the pressure generating portion 22 The pressure needs to be set appropriately.

  FIG. 6 is an explanatory diagram showing the state (pressure profile) of the pressure p of the working fluid generated by the pressure generator 22. The pressure generator 22 may basically generate pressure in a pulsed manner, but in the example shown in FIG. 6, the pressure p is changed according to the state of the liquid mass of the marker material M shown in FIG. I am doing so.

  Specifically, first, the pressure p is raised from a pressure p0 equal to the back pressure. Thereby, the liquid mass of the marker material M flies away from the meniscus forming portion 71 (see FIGS. 4B and 4C). At this time, the pressure p is increased to a predetermined pressure p2 in the vicinity of the maximum pressure p1, and when the pressure p reaches the pressure p2, the pressure p is gradually increased to the maximum pressure p1. Thereby, the liquid mass which flies in the vicinity of the adjacent meniscus formation part 71 can be decelerated. Next, after reducing the pressure p to a predetermined pressure p3 lower than the back pressure, the pressure p is returned to the pressure p0. Thereby, the liquid mass which reached the adjacent meniscus formation part 71 can be reliably capture | acquired by the meniscus formation part 71 (refer FIG. 4 (A)).

  By providing the meniscus forming portion 71 in the marking portion 17 in this way, a liquid mass of the marker material M having a uniform size can be formed. In particular, by arranging a plurality of meniscus forming portions 71 at equal intervals, the liquid mass of the marker material M can be continuously ejected at a constant timing. Thereby, the marking for attaching the marker material M to the subject S can be performed promptly or when the marking for the next imaging is performed.

  FIG. 7 is a side view illustrating a marking state in which the marker material M is attached to the subject S. FIG. As described above, the marking portion 17 provided at the distal end portion 12 of the insertion portion 11 ejects the marker material M toward the subject S with the pressure of the working fluid. For this reason, since the marker material M can be attached to the subject S without bringing the marking portion 17 into contact with the subject S, the operation of attaching the marker material M to the subject S is facilitated. It is possible to avoid damage due to contact with 17.

  Further, as described above, the marking unit 17 is provided so as to be able to move forward and backward in the insertion unit 11. By moving the marking unit 17 forward, the marking unit 17 is brought closer to the subject S without moving the insertion unit 11 itself. Therefore, the marker material M can be attached to a required position on the subject S with high accuracy. Further, by retracting the marking portion 17, the insertion portion 11 can be smoothly inserted into and removed from the human body without damaging the human body with the marking portion 17 or damaging the marking portion 17.

  FIG. 8 is a functional block diagram showing the super-resolution processing unit 32. FIG. 9 is an explanatory diagram for explaining an overview of the reconstruction processing performed by the reconstruction processing unit 43 of the super-resolution processing unit 32.

  As shown in FIG. 8, the super-resolution processing unit 32 includes a low-resolution image acquisition unit 41, an alignment unit 42, a reconstruction processing unit 43, and a high-resolution image generation unit 44. The super-resolution processing unit 32 is realized by executing a program by a CPU.

  When the operation switch 24 (see FIG. 1) instructs the still image shooting, the main control unit 26 of the controller 2 performs super-resolution on the super-resolution processing unit 32 via the main control unit 34 of the image processing unit 3. Give instructions to perform image processing. The low resolution image acquisition unit 41 acquires a plurality (about 16 to 32) of low resolution images (captured images) necessary for the super-resolution processing from the storage unit 31. The plurality of low-resolution images are images (frame images) generated by performing imaging a plurality of times with the imaging unit 14 provided in the endoscope 1.

  The alignment unit 42 selects one of a plurality of low resolution images as a reference image, and aligns the low resolution images other than the reference image with respect to the reference image. A transformation matrix for coordinate transformation of a low resolution image is obtained. In this alignment, feature point matching is performed. Specifically, feature points are extracted for a reference image and a low-resolution image other than the reference image, and based on the correspondence between the feature points between the images, the reference image is extracted. A projection transformation matrix is obtained for each low-resolution image other than.

  In this feature point matching, it is desirable to extract four or more feature points. Therefore, the marker material M may be attached to the subject S so that four or more feature points are extracted. In point matching, for example, it is often possible to extract feature points using a known algorithm known as a gradient method or SIFT, and it is not an absolute requirement to extract four or more feature points.

  As shown in FIG. 9, the reconstruction processing unit 43 first enlarges the reference image at a super-resolution enlargement ratio, and generates a mapping image in which pixels of the reference image are discretely arranged. Then, using the conversion matrix acquired by the alignment unit 42, the low-resolution image other than the reference image is converted into the coordinate system of the reference image, and the pixels of the low-resolution image other than the reference image are converted based on the converted coordinates. Is plotted on the mapping image. Thereby, a reconstructed image is obtained.

  Here, in general, plotting all low-resolution image pixels on the mapping image does not plot low-resolution image pixels on all mapping image pixels. There are pixels where the pixels of the low resolution image are not plotted.

  In FIG. 9, for convenience of illustration, the low-resolution image has a size of 4 pixels × 4 pixels and the super-resolution enlargement ratio is four times in the vertical and horizontal directions. The magnification of the image is not limited to this.

  The high resolution image generation unit 44 illustrated in FIG. 8 generates a high resolution image from the reconstructed image acquired by the reconstruction processing unit 43. Here, first, processing is performed to fill in pixels in which the pixels of the low-resolution image are not plotted in the reconstructed image acquired by the reconstruction processing unit 43 by interpolation, and further, the amount of image blur (PSF: Point Spread Function) , And a blur restoration process is also performed by inverse transformation to output as a high resolution image.

  Now, as shown in FIG. 7, the imaging unit 14 provided at the distal end portion 12 of the insertion unit 11 of the endoscope 1 captures the subject S within the imaging range with a predetermined angle of view, and within the imaging range. By attaching the marker material M on the subject S, the marker material M is imaged together with the subject S, and a captured image in which the marker material M is reflected on the subject S is obtained.

  Here, when the subject 1 is shaken when the subject S is imaged, that is, when the insertion portion 11 is shaken by shaking the hand holding the endoscope 1, the distal end portion 12 of the insertion portion 11 moves. However, since the marker material M is attached to the subject S, the positional relationship between the subject S and the marker material M does not change.

  FIG. 10 is a perspective view illustrating a marking state in which the marker material M is attached to the subject S. FIG. In the marking for attaching the marker material M to the subject S, in order to detect the rotational component of the image in the alignment in the super-resolution processing, a plurality of markings are performed within the imaging range, and more preferably the observation site is the center. Marking may be performed at a plurality of locations (for example, four locations) around the periphery. As a result, it is possible to accurately obtain the projective transformation parameter and increase the accuracy of alignment.

  In the example shown in FIG. 10, marking is performed on four points. First, as shown in FIG. 10A, the marking point P1 is marked. Next, the insertion portion 11 of the endoscope 1 is rotated by 90 ° about the central axis, and marking is performed at the marking point P2 as shown in FIG. Hereinafter, similarly to this, marking points P3 and P4 are marked while rotating the insertion portion 11 by 90 °. In the example shown in FIG. 10, four markings are performed, but the present invention is not limited to this.

  FIG. 11 is an explanatory diagram showing a low-resolution image (captured image) obtained by imaging with the endoscope 1. In the low-resolution image obtained by imaging with the endoscope 1, an image of the marker material M attached to the subject S appears together with the image of the subject S. Although the imaging range of each low-resolution image is shifted due to camera shake, there is no difference in the positional relationship between the image of the subject S and the image of the marker material M. Therefore, if the position of the marker material M image is used as a reference, the low-resolution image can be accurately aligned.

  As described above, the alignment unit 42 of the super-resolution processing unit 32 performs alignment of other low-resolution images with respect to the reference image by feature point matching. In this alignment, feature points are extracted from the low-resolution image. However, since the image of the marker material M is significantly different from the image of the surrounding subject S, the feature point is reliably extracted at the position of the image of the marker material M. For this reason, the alignment process can be performed with high accuracy.

  In particular, in the present embodiment, since the marker material M includes a fluorescent material, the marker material M emits light by light that illuminates the subject S. As a result, the brightness of the image of the marker material M shown in the captured image is higher than that of the periphery, and the feature points are extracted at the position of the region where the brightness is high. For this reason, a feature point can be extracted more reliably at a position corresponding to the marker material M, and the alignment accuracy can be further increased.

  In this embodiment, the marker material is a liquid, but the present invention is not limited to this, and any of a liquid and a solid can be used. However, since the marker material is left in the human body, a material that does not adversely affect the human body (for example, powdered sugar, physiological saline, edible oil gel, etc.) is used for both liquid and solid.

  Here, even when the marker material is solid, as in the case of liquid, the marking portion may be configured to inject the marker material with the pressure of the working fluid. In this case, a plurality of marker materials are provided in the guide hole. If the pressure is applied to the working fluid in a pulsed manner, the marker material can be continuously ejected at a constant timing.

  In the present embodiment, the working fluid is a gas, but the present invention is not limited to this, and any of a gas and a liquid can be used.

  Further, in the present embodiment, a single type of marker material is ejected. However, the present invention is not limited to this, and is configured to eject a plurality of types of marker materials having different colors. May be. In this case, when marking a plurality of locations, the feature points can be easily extracted by changing the color of the marker material for each marking location.

  In the present embodiment, the marking portion is formed by ejecting a liquid mass of the marker material in a meniscus shape. However, the present invention is not limited to this, and is used, for example, in an inkjet printer head. It is also possible to configure with a MEMS device.

  Further, in the present embodiment, the marker material is projected from the marking portion and attached to the subject, but the present invention is not limited to this, and the marker portion is brought into contact with the subject and the marker material is attached to the subject. Alternatively, a marker material bulged by surface tension in a state where the marking portion is close to the subject may be attached to the subject.

  In the present embodiment, the marker material includes a fluorescent material, but a configuration not including the fluorescent material is also possible. In this case, if the marker material is colored in a color that is significantly different from the color of the body tissue such as the organ that is the subject, for example, a color that is complementary to the color of the body tissue (for example, cyan that is a complementary color of red), Extraction of points becomes easy.

  In the present embodiment, the case where a high-resolution still image is obtained from a plurality of low-resolution images by super-resolution processing has been described. For example, a reference image is taken in the time axis direction with respect to a plurality of captured frame images. If the super-resolution processing is executed by selecting in order, a moving image is obtained as a result. That is, the present invention can be applied to moving images.

  Further, in the present embodiment, the description has been made on the premise that super-resolution processing is performed using a plurality of low-resolution images that have been displaced due to camera shake, but the present invention is not limited to camera shake, for example, the endoscope itself can be vibrated, The relative positional relationship between the imaging optical system and the image sensor may be changed by a simple mechanism such as a bimorph so that a low-resolution image is obtained by forcibly giving a pixel shift. In any case, if alignment is performed using the marker according to the present invention, a high-resolution image can be obtained without using a high-precision alignment mechanism such as a conventional pixel shift.

  An endoscope according to the present invention and an endoscope system provided with the endoscope are capable of performing super-resolution processing when acquiring a plurality of low-resolution images whose imaging positions necessary for super-resolution processing are shifted using camera shake. In particular, the present invention relates to an endoscope for imaging an inside of an observation object that cannot be directly observed from the outside, and an endoscope system including the endoscope. The present invention is useful as an endoscope suitable for performing super-resolution processing and an endoscope system equipped with the endoscope.

DESCRIPTION OF SYMBOLS 1 Endoscope 2 Controller 3 Image processing unit (image processing apparatus)
4 Monitor 17 Marking part 11 Insertion part 12 Tip part 13 Imaging window 14 Imaging part 16 Illumination window (illumination part)
32 Super-resolution processing unit 42 Positioning unit 71 Meniscus forming unit M Marker material S Subject

Claims (6)

An insertion part to be inserted inside the observation object;
An illumination unit that is provided in the insertion unit and illuminates a subject inside the observation target;
An imaging unit provided in the insertion unit for imaging the subject;
An endoscope, comprising: a marking portion provided in the insertion portion to attach a marker material to the subject.   The endoscope according to claim 1, wherein the marking unit ejects a marker material toward the subject with a pressure of a working fluid.   The endoscope according to claim 1, wherein the marker material includes a fluorescent material and emits light by light emitted from the illumination unit. The marker material is a liquid,
The endoscope according to any one of claims 1 to 3, wherein the marking portion includes a meniscus forming portion that forms a liquid mass of the marker material in a meniscus shape.   The endoscope according to any one of claims 1 to 4, wherein the marking portion is provided in the insertion portion so as to freely advance and retract. An endoscope according to any one of claims 1 to 5, and an image processing apparatus that performs super-resolution processing for generating a high-resolution image from a plurality of low-resolution images acquired by imaging by the imaging unit, Prepared,
The image processing device extracts a feature point at a position corresponding to the marker material in the low-resolution image when aligning a plurality of the low-resolution images in the super-resolution processing. Endoscope system.