JP2014108193A - Image processing device and endoscope - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing device capable of obtaining both an image highlighting a concave area on a surface of a subject and a stereoscopic image.SOLUTION: In a polarization imaging mode, an image processing device obtains: a first polarization image formed by a signal of light after penetrating a polarizer having a polarization transmission axis in a direction intersecting with a first direction when a subject is exposed to first illumination light polarized in the first direction; and a second polarization image formed by a signal of light after penetrating the polarizer having the polarization transmission axis in a direction intersecting with a second direction intersecting with the first direction when the subject is exposed to second illumination light polarized in the second direction. Then, the device detects a concave area on a surface of the subject on the basis of at least one of the polarization images, and obtains an image highlighting the concave area by combining the polarization image with a normally-taken image. In a stereoscopic imaging mode, the device obtains a stereoscopic image by combining multiple parallax images formed by a signal of light after penetrating the polarizer or each transparent area.

Description

  The present disclosure relates to an image processing device and an endoscope used in the image processing device.

  In the field of endoscopes that illuminate the wall surface of living organ organs covered with mucous membranes and perform imaging, the texture of the surface of the subject is changed at the same time as the color of the subject surface changes. It is necessary to confirm. The surface texture is semi-transparent fine irregularities having an average size of 0.5 to 1.0 mm and a depth of about 0.1 to 0.2 mm, for example, like a stomach subdivision in the stomach. Since it is very difficult to capture this with a shadow of luminance by an endoscope, a blue pigment liquid such as an indigo carmine solution is distributed on the mucous membrane, and the state where the liquid accumulates in the groove is observed with luminance.

  However, this treatment has problems such as bleeding because the liquid is sprayed on the mucous membrane and the color of the mucous membrane changing. A polarization endoscope using polarization illumination and polarization imaging is described in Patent Document 1 for the problem of observing surface irregularities in this way.

JP 2009-246770 A JP 2009-210780 A

Viktor Gruev, Rob Perkins, and Timothy York, "CCD polarization imaging sensor with aluminum nanowire optical filters", 30 August 2010 / Vol. 18, No. 18 / OPTICS EXPRESS PP. 19087-19094

  In the prior art using polarized light as disclosed in Patent Document 1, illumination light of a specific polarization component is irradiated onto an object, and in the return light from the object, a polarization component parallel to illumination and a polarization component orthogonal to each other The surface shape change is calculated based on these images.

  Embodiments of the image processing apparatus of the present disclosure enhance the concave region on the surface of the subject by detecting a concave region on the surface of the subject in the polarization imaging mode and obtaining a non-polarized image in the non-polarization imaging mode. Both an unillustrated image and a non-polarized image can be obtained.

  In the polarization imaging mode, the image processing apparatus according to the present disclosure includes the first illumination light that is polarized in the first direction and the first illumination light that is polarized in the second direction that intersects the first direction. The illumination unit sequentially irradiates the subject with the illumination light of 2 and irradiates the subject with the unpolarized illumination light in the non-polarization imaging mode, wherein the wavelength range of the first illumination light is the second An illumination unit that sequentially emits the first and second illumination lights so as to have a portion that does not overlap in the wavelength range of the illumination light, an aperture region provided with a polarizer and a transparent region, and a polarizer in the aperture region or In the imaging device having a photodetecting device array that receives light transmitted through the transparent region and outputs a signal, each having a plurality of microlenses covering a plurality of photodetecting devices, and in the polarization imaging mode The first illumination When the subject is irradiated with the first polarized image and / or the subject composed of a signal of light transmitted through a polarizer having the polarization transmission axis in a direction crossing the first direction. Obtaining a second polarization image composed of a signal of light transmitted through a polarizer having the polarization transmission axis in a direction intersecting the second direction when irradiated with the second illumination light; In the non-polarized imaging mode, a non-polarized image composed of a signal of light transmitted through a polarizer or a transparent region is obtained when the subject is illuminated with the non-polarized illumination light. A pixel selective re-integration unit that obtains a plurality of parallax images composed of light signals transmitted through a child or a transparent region, and the subject based on at least one of the first and second polarization images A concave area detection unit that detects a concave area on a surface, an image synthesis unit that forms an image showing the concave area on the surface of the subject in an emphasized manner, and a parallax image synthesis unit that alternately reproduces the parallax image temporally With.

  Another image processing apparatus of the present disclosure includes a first white illumination light that is polarized in a first direction and a polarization that is polarized in a second direction that intersects the first direction in the polarization imaging mode. A plurality of polarizers having different directions of polarization transmission axes from the illumination unit that sequentially irradiates the subject with the second white illumination light and irradiating the subject with the non-polarized white illumination light in the non-polarized imaging mode And a color mosaic filter in which color filters having different light transmission characteristics are arranged, and after passing through each polarizer or each transparent region of the opening region, each color filter An image sensor having a photo-detecting element array that receives transmitted light and outputs a signal, the image sensor having a plurality of microlenses each covering a plurality of photo-detecting elements, and the polarization imaging mode. The first white light is configured by a signal of light transmitted through a polarizer having the polarization transmission axis in a direction intersecting the first direction when the subject is irradiated with the first white illumination light. And a signal of light transmitted through a polarizer having the polarization transmission axis in a direction crossing the second direction when the subject is illuminated with the second white illumination light. A second polarization image is formed based on signals from a part of the light detection elements selected from the plurality of light detection elements covered by each microlens, and in the non-polarization imaging mode, the non-polarization image When the subject is illuminated with white illumination light, a non-polarized image composed of light signals transmitted through each polarizer is obtained, and in stereoscopic imaging mode, it is composed of light signals transmitted through each transparent area. Multiple A pixel selection / reintegration unit that obtains a parallax image, a concave region detection unit that detects a concave region on the surface of the subject based on at least one of the first and second polarization images, and the concave region on the surface of the subject An image composition unit that forms an image showing the emphasis on the image, and a parallax image composition unit that reproduces the parallax image alternately in time.

  Still another image processing apparatus of the present disclosure includes a first illumination light that is polarized in a first direction and a polarization that is polarized in a second direction that intersects the first direction in the polarization imaging mode. In the non-polarization imaging mode, the illumination unit irradiates the subject with non-polarized illumination light, and the wavelength range of the first illumination light is An illumination unit that sequentially emits the first and second illumination lights so as to have a portion that does not overlap with the wavelength range of the second illumination light, and a plurality of polarizers and a plurality of transparent regions having different directions of polarization transmission axes A compound eye lens unit provided with a compound eye lens unit having a plurality of lenses each provided with a polarizer or a transparent region, and receiving light transmitted through each polarizer or each transparent region of the compound eye lens unit Photodetector array that outputs signals In the polarization imaging mode, the light transmitted through the polarizer having the polarization transmission axis in the direction intersecting the first direction when the subject is irradiated with the first illumination light. A first polarized image constituted by the signal of the first and the polarizer having the polarization transmission axis in a direction intersecting the second direction when the subject is illuminated with the second illumination light. A second polarization image constituted by light signals, and in the non-polarization imaging mode, constituted by light signals transmitted through the respective polarizers when the subject is irradiated with the non-polarization illumination light. A non-polarized image to be obtained, and in a stereoscopic imaging mode, a pixel selective re-integration unit that obtains a plurality of parallax images composed of light signals transmitted through the transparent regions, and the first and second polarized images Less A concave region detection unit that detects a concave region on the surface of the subject based on the other, an image composition unit that forms an image that highlights the concave region on the surface of the subject, and temporally converts the parallax image And a parallax image synthesizing unit that reproduces alternately.

  Another image processing apparatus of the present disclosure includes a first white illumination light that is polarized in a first direction and a polarization that is polarized in a second direction that intersects the first direction in the polarization imaging mode. A plurality of polarizers having different directions of polarization transmission axes from the illumination unit that sequentially irradiates the subject with the second white illumination light and irradiating the subject with the non-polarized white illumination light in the non-polarized imaging mode And a compound eye lens portion provided with a plurality of transparent regions, each having a plurality of lenses each provided with a polarizer or a transparent region, and a color mosaic in which color filters having different light transmission characteristics are arranged An image sensor light having a filter, and having a light detection element array that outputs a signal after receiving light transmitted through each color filter after passing through each polarizer or each transparent region of the compound eye lens unit; The polarization imaging mode is configured by a signal of light transmitted through a polarizer having the polarization transmission axis in a direction crossing the first direction when the subject is irradiated with the first illumination light. And a signal of light transmitted through a polarizer having the polarization transmission axis in a direction crossing the second direction when the subject is illuminated with the second illumination light. A non-polarized image composed of a signal of light transmitted through each polarizer when the subject is illuminated with the non-polarized illumination light in the non-polarized imaging mode. And in a stereoscopic imaging mode, based on at least one of the first and second polarization images, a pixel selection re-integration unit that obtains a plurality of parallax images composed of light signals transmitted through the transparent regions A concave area detecting unit for detecting a concave area on the surface of the subject, an image composition unit for forming an image showing the concave area on the surface of the subject in an emphasized manner, and parallax for alternately reproducing the parallax image temporally An image composition unit.

Another image processing apparatus of the present disclosure includes a first white illumination light that is polarized in a first direction and a polarization that is polarized in a second direction that intersects the first direction in the polarization imaging mode. A plurality of polarizers having different directions of polarization transmission axes from the illumination unit that sequentially irradiates the subject with the second white illumination light and irradiating the subject with the non-polarized white illumination light in the non-polarized imaging mode And a color mosaic filter in which color filters having different light transmission characteristics are arranged, and after passing through each polarizer of the opening region, further receiving light transmitted through each color filter and receiving a signal An image sensor having a photo-detecting element array that outputs a signal, a pixel selective reintegrating unit that obtains a plurality of time-division parallax images composed of light signals transmitted through each polarizer,
A parallax image synthesizing unit that alternately reproduces the parallax images in terms of time.

  According to the embodiment of the present disclosure, in the polarization imaging mode, the first illumination light that is polarized in the first direction and the first illumination light that is polarized in the second direction that intersects the first direction. The object is sequentially irradiated with the illumination light of 2 and the object is irradiated with the non-polarized illumination light in the non-polarization imaging mode, so that unevenness information and inclination information on the object surface are obtained separately from the normal object image. be able to. For this reason, it is possible to synthesize an image similar to an image in which a blue pigment liquid such as an indigo carmine solution is distributed on the mucous membrane (an image in which the concave region is emphasized). In addition, since a parallax image can be obtained, a stereoscopic image can be displayed.

(A) and (b) are both endoscopic images of gastric mucosa. The figure which shows the observation state of the translucent uneven part by brightness (A) And (b) is a figure which shows the principle of polarization observation Graph showing the emission angle and transmittance when light exits the medium (A) And (b) is a figure which shows the groove | channel on the acrylic board in the case of crossed nicols imaging of an acrylic lenticular board, and the directionality of polarized light illumination The block diagram which shows the structure of 1st Embodiment of this indication. (A) to (c) is a diagram showing a color wheel in the first embodiment of the present disclosure. The figure which shows the characteristic of the illumination filter in 1st Embodiment of this indication. FIGS. 4A to 4C are diagrams illustrating a planar structure and a transmission axis direction of a wire grid forming a monochrome broadband polarization imaging device according to the first embodiment of the present disclosure. FIGS. The figure which shows the cross-section of the monochrome broadband polarization | polarized-light imaging element in 1st Embodiment of this indication. FIG. 10 is a diagram illustrating an operation in a normal imaging mode according to the first embodiment of the present disclosure. Timing chart of normal imaging mode in the first embodiment of the present disclosure The figure which shows operation | movement of the polarization mosaic process part 202 of the polarization imaging mode in 1st Embodiment of this indication. Timing chart of polarization imaging mode in the first embodiment of the present disclosure The figure which shows operation | movement of the polarization mosaic process part 202 of the polarization imaging mode in 1st Embodiment of this indication. Timing chart of polarization imaging mode in the first embodiment of the present disclosure The block diagram explaining the process of the recessed area detection part 204 and the image synthetic | combination part 206 in 1st Embodiment of this indication Example of differential processing mask used in concave region detection unit 204 Example of blue emphasis processing used in the concave area detection unit 204 (A) to (C) are experimental results using rat gastric mucosa. The block diagram which shows the structure of 2nd Embodiment of this indication. The figure which shows the endoscope front-end | tip part and polarization | polarized-light rotation illumination in 2nd Embodiment of this indication. (A) And (B) is a figure showing another composition of polarization rotation illumination in a 2nd embodiment of this indication. The figure which shows the cross-section of the color polarization image pick-up element in 2nd Embodiment of this indication. FIGS. 9A to 9C are diagrams illustrating a planar arrangement of a color mosaic and a polarization mosaic according to the second embodiment of the present disclosure. FIGS. The figure which shows operation | movement of the polarization mosaic process part 202 of the normal imaging mode in 2nd Embodiment of this indication. Timing chart of normal imaging mode in the second embodiment of the present disclosure The figure which shows operation | movement of the polarization mosaic process part 202 of the polarization imaging mode in 2nd Embodiment of this indication. Timing chart of polarization imaging mode in the second embodiment of the present disclosure FIGS. 9A to 9C are diagrams illustrating a planar arrangement of a color mosaic and a polarization mosaic in Modification 1 of the second embodiment of the present disclosure. FIGS. The figure which shows operation | movement of the polarization mosaic process part 202 of the normal imaging mode in the modification 1 of 2nd Embodiment of this indication. The figure which shows operation | movement of the polarization mosaic process part 202 of the polarization imaging mode in the modification 1 of 2nd Embodiment of this indication. FIGS. 9A to 9C are diagrams illustrating planar arrangements of a color mosaic and a polarization mosaic in Modification 2 of the second embodiment of the present disclosure. FIGS. The figure which shows operation | movement of the polarization mosaic process part 202 of the normal imaging mode in the modification 2 of 2nd Embodiment of this indication. The figure which shows operation | movement of the polarization mosaic process part 202 of the polarization imaging mode in the modification 2 of 2nd Embodiment of this indication. A block diagram showing composition of a 3rd embodiment of this indication. The figure which shows the front-end | tip part of the endoscope in 3rd Embodiment of this indication. The figure which shows the structure of the micro lens array type | mold color polarization imaging part in 3rd Embodiment of this indication. The figure which shows the structure of the color polarizing filter area | region of the opening part in 3rd Embodiment of this indication. 6A and 6B are diagrams for describing processing of the pixel selective re-integration unit 210 according to the third embodiment of the present disclosure. 6A and 6B are diagrams illustrating images acquired in the normal imaging mode and the polarization imaging mode according to the third embodiment of the present disclosure. The figure explaining the structure of 4th Embodiment of this indication Configuration diagram of microlens array type color polarization imaging device according to the fourth embodiment of the present disclosure Cross-sectional configuration diagram of a microlens array type color polarization imaging unit in the fourth embodiment of the present disclosure Polarization filter region of the opening in the fourth embodiment of the present disclosure 6A and 6B are diagrams for describing processing of the pixel selective re-integration unit 210 according to the fourth embodiment of the present disclosure. The figure which shows the structure of 5th Embodiment of this indication. (A) to (D) is a diagram showing a distal end portion of an endoscope according to a fifth embodiment of the present disclosure. FIG. 10 is a diagram for describing processing of a pixel selection recorrection unit according to the fifth embodiment of the present disclosure. The figure of the relation between the right-and-left image which has parallax, and subject distance in a 5th embodiment of this indication. The figure explaining the stereoscopic observation in the stereoscopic mode of the fifth embodiment of the present disclosure The figure explaining the structure of 6th Embodiment of this indication FIGS. 9A to 9D are diagrams illustrating a distal end portion of an endoscope according to a sixth embodiment of the present disclosure; FIGS. The figure which shows the relationship between the polarizing filter of the opening part in the 6th Embodiment of this indication, and the color mosaic filter on an imaging surface. 6A and 6B are diagrams for describing processing of a pixel selection reintegration unit according to the sixth embodiment of the present disclosure. (A)-(C) are figures of the modification in a 6th embodiment of this indication. Configuration diagram of the seventh embodiment of the present disclosure FIGS. 9A to 9C are views of a distal end portion of an endoscope according to a seventh embodiment of the present disclosure. FIGS. The figure which shows the process of the image separation part in 7th Embodiment of this indication. Configuration diagram of the eighth embodiment of the present disclosure FIGS. 9A to 9C are views of a distal end portion of an endoscope according to an eighth embodiment of the present disclosure. FIGS. The figure which shows the process of the image separation part in 8th Embodiment of this indication. Configuration diagram of ninth embodiment of the present disclosure FIGS. 9A to 9C are views of a distal end portion of an endoscope according to a ninth embodiment of the present disclosure; FIGS. The figure which shows the process of the pixel selection re-integration part in 9th Embodiment of this indication. (A) And (B) is a figure showing a modification of a ninth embodiment of this indication. Configuration diagram of tenth embodiment of the present disclosure (A) to (C) are views of the distal end portion of the endoscope according to the tenth embodiment of the present disclosure. The figure which shows the process of the pixel selection re-integration part in 10th Embodiment of this indication. Configuration diagram of eleventh embodiment of the present disclosure The figure of the front-end | tip part of the endoscope of 11th Embodiment of this indication Time division stereoscopic vision principle diagram of eleventh embodiment of the present disclosure

  FIG. 1 shows an image obtained by observing the surface mucosa of a human stomach with an endoscope. FIG. 1A shows a normal color image, and only a gentle undulation is felt on the surface. In other words, it is difficult for an endoscope for probing the digestive organ or the like to detect irregularities formed on a transparent or translucent surface of an organ by ordinary color image processing. Here, normal color image processing is processing for obtaining a color luminance image obtained by irradiating non-polarized white light. The color image thus obtained is referred to as a “color luminance image”, and photographing for obtaining a color luminance image may be referred to as color luminance photographing.

  On the other hand, FIG. 1 (b) shows a color image after the indigo carmine solution has been distributed, and the texture of fine irregularities on the surface (size is 0.5 to 1.0 mm, depth is 0.1 to 0). .About 2 mm) can be clearly confirmed.

  FIG. 2 shows a simplified cross section formed on the surface of an organ of the stomach or intestine. The unevenness present on the surface of the stomach or intestine is generally considered to be formed by a repetitive array of upward convex convex shapes. A concave region located between two adjacent convex portions is typically a small “groove” extending in a certain direction. The plurality of grooves are locally aligned in substantially the same direction, but can generally exhibit a complex curvilinear shape or other pattern. The actual unevenness on the surface of the subject may include a dot-like recess or protrusion, but in this specification, such an uneven recess may be simply referred to as a “groove” or “groove region”. FIG. 2 schematically shows a cross section across several grooves existing in a narrow region on the surface of the subject. For simplicity, in the following description, it may be assumed that the concave and convex portions as shown in FIG. 2 extend in a direction perpendicular to the drawing sheet.

  The observation by the endoscope is coaxial illumination in which a light source is arranged in the vicinity of the photographing optical axis. Therefore, the illumination light is irradiated on the subject shown in FIG. Shooting starts from. There are two types of reflected light that can be observed in normal color luminance photography using such coaxial illumination. One is specular reflection light that reflects light on the surface, and the other is internal diffusion light that permeates into the medium, reflects off the lower layer, returns, and is re-emitted from the surface. Since the specular reflection light is generated only when the irradiated light and the photographing optical axis are close to the specular reflection condition, the specular reflection light is generated only very locally in the photographing scene of the endoscope. The color of the specular reflected light is the color of illumination, that is, white, and the luminance is very strong. The subject image by the specular reflection light is generally strong and bright at the convex and concave portions on the surface of the subject, and weak and dark at the concave portion due to the above-described regular reflection conditions. On the other hand, the internal diffused light is observed over the entire area of the photographing scene. The color of the internal diffused light is the color of the medium itself, and the whole medium shines without the luminance being so strong. The subject image by the internal diffused light tends to be dark at the thick convex portion of the medium and bright at the thin concave portion. That is, the correspondence between the brightness level and the unevenness of the subject surface is opposite between the specular reflection light and the internal diffused light.

  In normal shooting, the above two types of reflected light are superimposed to form one luminance image (shooting scene). For this reason, in an area where the difference in luminance between the two types of reflected light almost antagonizes in the shooting scene, there is almost no difference in luminance between the uneven portions. Therefore, in a normal luminance image, there is almost no difference in luminance in the surface unevenness of the subject. Even if there is a brightness difference between light and dark, based on that information, for example, if processing is performed by detecting a pixel having a lower brightness than the pixel as a recess, the position of the unevenness by the specular reflection part and the internal diffuse reflection part There is a detrimental effect on the relationship, and luminance image processing is very difficult.

  FIGS. 3A and 3B are diagrams for explaining observation using polarized light. In the example of FIG. 3 (a), when polarized illumination in a direction parallel to the surface irregularities is irradiated, FIG. 3 (b) shows a polarized image in a crossed Nicol state when irradiated with polarized illumination in a direction perpendicular to the surface irregularities. Is getting.

  In the setting of FIG. 3A, an illumination 300 and a P-polarization filter 302 are arranged with respect to a schematic model 301 showing an uneven surface of the organ surface. As a result, the model 301 is irradiated with light (P-polarized light or P-wave) polarized in parallel with the direction in which the unevenness of the model 301 extends (direction perpendicular to the drawing sheet). On the other hand, the observation-side polarizing filter 303 (S) is arranged so as to form an orthogonal Nicol state, and the imaging device 304 acquires an image.

  In the setting of FIG. 3B, the polarizing filter 309 (S) for S-polarized illumination is arranged, and the observation-side polarizing filter 310 (P) is arranged so as to form a crossed Nicols state. In the example of FIG. 3, the polarized light when the polarization transmission axis of the polarizing filter is perpendicular to the paper surface of the drawing is P-polarized light, and the polarized light when the polarization filter is parallel to the paper surface is S-polarized light (S wave).

  There are two types of reflected light that can be observed in photographing using the linearly polarized illumination 305, 311. One is a specular reflection component 306, 312 in which light is regularly reflected by a convex portion, and the other is penetrating into the medium. In the lower layer, the non-polarized scattered light 307 is polarized light 308 and 313 of internally diffused light that is emitted again from the surface from the inclined surface inclined with respect to the imaging system. The internally diffused light becomes significantly polarized when the emission angle, which is the inclination between the boundary surface normal and the line of sight, is large.

  Next, a qualitative explanation will be given regarding the direction of polarization of the reflected light.

  First, since the specular reflection components 306 and 312 are regular reflection in the coaxial illumination state, the same polarization state as the polarization state of the irradiated polarized light is maintained. Therefore, the specular reflection component 306 is polarized light perpendicular to the paper surface, and the specular reflection component 312 is polarized light parallel to the paper surface.

  On the other hand, the polarization direction of the polarized light 308 and 313 of the internally diffused light is determined by Fresnel theory with respect to the polarized light emitted from the medium having a refractive index larger than 1 into the air. FIG. 4 is a graph showing the state of polarization when light is emitted from a medium having a refractive index greater than 1 determined based on Fresnel theory into the air. The transmittance is always T‖> T⊥, that is, P-polarized light> S-polarized light with respect to the emission angle on the horizontal axis. Accordingly, both the internally diffused light polarizations 308 and 313 are polarized light having strong P-polarized light with respect to the inclination of the surface of the model 301.

  Next, when observing these reflected lights with the imaging device 304, in FIG. 3A, the reflected light 306 is P-polarized light (perpendicular to the paper surface), and the reflected light 308 is S-polarized light (within the paper surface). Since the light is observed by the polarizing filter 303, the reflected light 306 becomes dark in the blocked state, and the reflected light 308 becomes bright in the transmitted state. That is, when the 301 scallop shape is observed from directly above, it is clearly observed as a striped pattern in which the vicinity of the central axis is dark and the vicinity of the slope is bright as in an image 314. In FIG. 3B, since the reflected light 312 and 313 are both in S (in the paper plane), the P-polarization filter 310 of the imaging device 304 is observed, so that the reflected lights 312 and 313 are both blocked and darkened. That is, when the kamaboko shape of the model 301 is observed from directly above, it becomes dark as a whole as in the image 315 and the unevenness becomes unclear.

  FIG. 5 is a diagram showing a result of executing this polarization imaging using a lenticular plate in which irregularities (many grooves) are actually formed in stripes on the acrylic plate surface. The subject is a transparent sheet imitating a blood vessel placed on a complete diffusion plate and a milky white 2 mm thick acrylic lenticular plate placed thereon. This subject was observed from directly above. The grooves of the lenticular plate were arranged parallel to the direction of 0 ° with respect to the horizontal direction in the drawing sheet. FIG. 5A and FIG. 5B show images taken in a state where the polarized illumination is P and S, respectively. That is, in FIG. 5A, the polarization direction of the polarized illumination and the groove direction of the subject surface are parallel, and in FIG. 5B, the polarization direction of the polarized illumination and the groove direction of the subject surface are orthogonal. . Both the polarization imaging in FIGS. 5A and 5B were performed in a crossed Nicols state. When the orthogonal Nicol image captured at this time is observed, bright and dark stripes are clearly observed in the P direction in FIG. On the other hand, the brightness becomes unclear in FIG.

  As described above, when polarized light in the crossed Nicols state is imaged in the polarization imaging mode, if the polarization direction of the polarized illumination is nearly parallel to the concave portion (groove) of the subject, the brightness becomes brighter than the surroundings, and the concave portion can be detected clearly. . Therefore, when the direction of the unevenness of the subject is unknown, the concave portion on the surface can be detected by image processing such as differential filter processing if at least two types of orthogonal Nicol images in which polarized illuminations are orthogonal can be obtained. An image similar to an image in which a blue pigment liquid such as an indigo carmine solution is distributed on the mucous membrane can be reproduced by coloring the detected recesses in blue (blue) by color digital image processing.

  According to the experiments of the present inventor, in the coaxial illumination state like an endoscope, the brightness of the specular reflection is extremely high as about 10 to 100 when the complete diffusion plate is 1. For this reason, when this high luminance is reduced so that the image pickup device is not saturated, and a crossed Nicols image is picked up, a polarizing filter having an extinction ratio performance of 100: 1 or more can be used.

  Hereinafter, embodiments of the present disclosure will be described.

(First embodiment)
FIG. 6 is a diagram schematically illustrating the overall configuration of the image processing apparatus according to the first embodiment of the present disclosure. The image processing apparatus includes an endoscope 101, a control device 102, and a display unit 114.

  The endoscope 101 has a distal end portion 106 having a monochrome broadband polarization imaging device 115 and an insertion portion 103 having a light guide 105 and a video signal line 108 inside. The insertion portion 103 of the endoscope 101 has a structure that is long to the left and right and can be bent flexibly as shown. The light guide 105 can transmit light even in a bent state.

  The control device 102 includes a light source device 104 and an image processor 110. Within the light source device 104 is provided a light source 118 which can be, for example, a xenon light source, a halogen light source, an LED light source, or a laser light source. When the non-polarized light emitted from the light source 118 passes through the color wheel 116 (ab) having a rotating RGB filter, it sequentially becomes R red, G green, and B blue light. These lights are guided to the distal end portion 106 via the light guide 105 and become polarized or non-polarized light when passing through the illumination filter 200. Then, the illumination lens 107 irradiates the visceral mucosa surface 111 which is a subject as polarized or non-polarized illumination light 117. The reflected light 113 from the subject passes through the objective lens 109 and forms an image on the monochrome broadband polarization image sensor 115.

  The synchronizer 112 sends an imaging start signal to the monochrome broadband polarization image sensor 115 in synchronization with the rotation of the color wheel 106a to acquire an image by reflected light. The imaged video signal is sent to the image processor 110 via the video signal line 108.

  The above processing is continuously executed by the RGB frame sequential method to execute color imaging and polarization imaging. Hereinafter, a mode for performing normal color imaging may be referred to as a “non-polarized imaging mode” or a “normal imaging mode”, and a mode for performing polarized imaging may be referred to as a “polarized imaging mode”.

  When a signal indicating whether the operation mode of the endoscope is the normal imaging mode or the polarization imaging mode is input from the outside, the illumination control unit 120 illuminates the color wheel corresponding to each of the signals. It is inserted into the optical path 121 of light. By doing so, the spectral characteristic of the illumination used for the surface sequential irradiation is changed.

  In the normal imaging mode, the color image processed by the polarization mosaic processing unit 202 is assembled into a full-color moving image by the image combining unit 206 and displayed on the display unit 114 as, for example, a moving image. Further, in the polarization imaging mode, an image processed by the polarization mosaic processing unit 202 is detected on the display unit in which the concave region on the surface is detected by the concave region detection unit 204 and is highlighted in blue by the image composition unit 206. Is displayed.

  FIGS. 7A to 7C show examples of color wheels that can be used for illumination. The normal imaging color wheel 116a shown in FIG. 7A has three fan-shaped regions provided around the rotation axis. The three fan-shaped regions are respectively a red filter that transmits the substantially same red wavelength region indicated by R1R2 simultaneously, a green filter that transmits the substantially same green wavelength region indicated by G1G2, and a blue wavelength of substantially the same color indicated by B1B2. It consists of a blue filter that transmits through the area simultaneously. Here, R1 and R2 of “R1R2” indicate the wavelength range of red (R), for example, the short wavelength side half of 600 nm to 700 nm and the long wavelength side half wavelength range, respectively. In the color filter 116a of FIG. 7A, the fan-shaped region described as “R1R2” is a region that transmits light in both the wavelength region of R1 and the wavelength region of R2, and is described as “R”. May be. The same applies to the other symbols “G1G2” and “B1B2”. In this specification, a symbol such as “R1” may be used to indicate a specific wavelength range, or may be used to indicate a filter that selectively transmits light in such a wavelength range.

  The polarization imaging color wheel 116b may have various configurations depending on the setting of the wavelength range for performing polarization imaging. FIG. 7B shows an example of a color wheel 116b that sequentially transmits light of two types of wavelength bands in which green and blue are mixed. The color wheel 116b can select a wavelength range suitable for detecting the surface uneven texture. On the other hand, FIG. 7C shows another example of the color wheel 116b that sequentially transmits light of six different wavelength ranges. The color wheel 116b having such a configuration is suitable for acquiring a full-color orthogonal Nicol image. Note that one of the color wheel 116a of FIG. 7A and the color wheel 116b of FIG. 7B or 7C is designated by an external signal and selectively used. More specifically, the color wheel 116a is used in the non-polarization imaging mode or the normal imaging mode, and the color wheel 116b is used in the polarization imaging mode.

  FIG. 8 is a diagram illustrating the transmission characteristics of the illumination filter 200. It has comb-shaped transmission characteristics in which P-polarized light and S-polarized light are transmitted alternately in the visible light wavelength ranges of B, G, and R. For example, in the example shown in FIG. 8, only the P-polarized light is transmitted in the wavelength region B1 (range of 400 to 450 nm), and only the S-polarized light is transmitted in the wavelength region B2 (range of 450 to 500 nm). For this reason, when the wavelength of light passing through the light guide from the light source is included in the wavelength band B1, the light is converted into P-polarized polarized illumination light by the illumination filter 200. Similarly, when the wavelength of light passing through the light guide from the light source is included in the wavelength band B2, the light is converted into S-polarized polarized illumination light by the illumination filter 200. In the normal imaging mode, when the wavelength of light passing from the light source through the light guide spreads over the entire wavelength band B1B2, unpolarized illumination light is obtained because P-polarized light and S-polarized light are mixed. .

  The filter having the characteristics as shown in FIG. 8 can be realized as a multilayer polarizer. For example, Patent Document 2 describes an example thereof.

  FIG. 9 is a diagram schematically illustrating an example of the structure of the pattern polarizer (polarization mosaic or polarization mosaic array) on the imaging surface of the monochrome broadband polarization imaging device 115. As shown in FIG. 9A, pixels are regularly arranged in rows and columns (XY directions) on the imaging surface.

  Since the imaging element 115 is used in a surface sequential method in which the color of illumination light sequentially changes to R, G, and B, no color mosaic filter is arranged on the imaging surface. That is, the image sensor 115 itself is a monochrome image sensor, and a polarizer is arranged in each pixel. Since light contained in the visible light wavelength range sequentially enters each pixel, the polarization selection characteristics of the polarizer used in the present embodiment are realized in the visible light band. Specifically, in the band of 400 nm to 800 nm, the extinction ratio indicating the polarization acquisition performance of the polarizer of the present embodiment is 100: 1 or more. For this reason, this embodiment does not use a polarizer that exhibits polarization characteristics only at a specific wavelength that occupies a part of the visible light band, but instead uses a metal wire grid polarizer that can exhibit high polarization acquisition performance in a wide wavelength range. Used.

  FIG. 9B shows one unit 801 of a polarizing filter corresponding to four pixels (2 × 2 blocks) arranged in 2 rows and 2 columns. In this unit 801, polarizing filters (four polarizers) rotated by 90 ° in a plane are arranged. In FIG. 9B, a plurality of straight lines shown on each polarizing filter indicate the direction of the transmission axis of polarized light.

  FIG. 9C shows a wire arrangement in the case of realizing with a metal wire grid corresponding to FIG. 9B. In the wire grid, the direction perpendicular to the direction of the metal wire (TE axis) is the polarization transmission axis. Therefore, when the wire is represented by a straight line in the schematic diagram, the direction of the polarization axis in FIG. ) Is 90 degrees different from the direction of the straight line of the metal wire. In order to avoid this confusion, when expressing the direction of the transmission axis of the polarization filter used in the embodiment of the present disclosure, the straight line (parallel to the polarization transmission axis) in FIG. Plan views that directly depict the wire direction shall not be used.

  As will be described later, the arrangement plane of the metal wire grid can take various positions within the range from the uppermost surface of the imaging device to the lower layer. In order to avoid interference with other pixels in a plane, the wire grid is set in a region that is on the inner side by a margin Δ from the outer edge of the pixel unit region. For example, if the length D of one side of a square pixel region is 3 to 4 μm, Δ can be set to 0.2 μm = 200 nm or more. The duty ratio between the width L and the interval S of each wire constituting the wire grid is a trade-off between the transmittance and the extinction ratio. In the embodiment of the present disclosure, the width L and the interval S are assumed to be equal. As will be described later, when L = S = 0.1 μm = 100 nm and Δ = 0.2 μm = 200 nm, the number of wire grids is 0 ° and 90 ° with respect to the vertical axis or horizontal axis in the imaging plane. In the case of forming an angle of °, the number is 17.

  Conventionally, Non-Patent Document 1 discloses an example in which a prototype of a polarization imaging device using an aluminum wire grid polarizer was actually manufactured and performance of extinction ratio was measured. According to Non-Patent Document 1, the extinction ratio of a fine wire grid polarizer arranged with a pitch P = 140 nmm and a height H = 70 nm in a region of pixel size = 7.4 × 7.4 μm is 450, 580, 700 nm. It was about 30: 1, 45: 1, and 60: 1 at each wavelength. From this example, even if the wire grid is miniaturized and mounted on the image sensor, it is predicted that it is difficult to realize 100: 1 as the extinction ratio performance. Therefore, in the present embodiment, the wire grid has a two-layer structure to achieve a high extinction ratio.

  A cross-sectional configuration example of the image sensor 115 will be described with reference to FIG.

  Incident light reaches the imaging surface from the objective lens 109 disposed above the imaging element 115. In the image sensor 115, the structure in which light reaches will be described in order. First, the microlens 220 is disposed on the uppermost surface. The role of the microlens 220 is to condense light onto an efficient PD (photodiode) 232. The microlens 220 is effective particularly in a case where wide-angle shooting is frequently used like an endoscope because the optical path of obliquely incident light is turned to an angle close to perpendicular to the imaging surface. In addition, the presence of the microlens 220 has the effect of preventing the TM transmittance and the extinction ratio from being lowered because light can be incident on the wire grid layers 222 and 224 from almost right above. Below the microlens 220 is a planarization layer 226. A first wire grid layer 222 is disposed in the lower layer. The first wire grid layer 222 transmits only polarized light in a specific direction rotated by 90 ° in the imaging plane, and reflects or absorbs other light.

  The first wire grid layer 222 in the present embodiment has a hollow structure between metal wires, and each metal wire is in contact with air. For this reason, it has the effect which prevents the performance fall of an extinction ratio.

  A second wire grid layer 224 is disposed below the first wire grid layer 222. The second wire grid layer 224 is basically the same arrangement direction, size, and material as the first wire grid layer 222 and has the same hollow structure.

  By using the first and second wire grid layers 222 and 224 in an overlapping manner, the extinction ratio of the entire two layers can be reduced to about 100: 1 even in a fine wire grid in which each extinction ratio is reduced to about 10: 1. Can be improved. A planarizing layer 228 and wiring 230 are provided below the second wire grid 224. Since the wiring 230 is not disposed in the light transmitting portion, the light reaches the PD (photodiode) 232 in the lower layer without being shielded from light. On the imaging surface, a number of PDs 232 are arranged in rows and columns to form a photosensitive cell array.

  In the imaging device, it is important to reduce the height of the microlens 220 to the PD 232 by shortening the distance. Also in the polarization imaging device of the present embodiment, when the distance from the microlens 220 to the PD 232 is long, crosstalk between pixels occurs and the polarization characteristics, particularly the extinction ratio, are reduced. In order to reduce the height, in this embodiment, the distance from the wire grid to the PD is set to about 2 to 3 μm. Also, in the wire grid polarizer, TE waves whose polarization direction is orthogonal to the transmitted TM waves are reflected, which becomes stray light and may cause performance deterioration. In order to avoid this, it is effective to provide the wire grids 222 and 224 with an absorption action as a multilayer structure instead of a single layer.

  Hereinafter, the imaging operation of the image processing apparatus in the present embodiment will be described.

  First, the operation in the normal imaging mode will be described with reference to FIGS. 11A and 11B.

  FIG. 11A and FIG. 11B are a diagram and a timing chart showing the illumination and imaging operations in the normal imaging mode, respectively. The left side of FIG. 11A shows a spectrum of frame sequential illumination. B-color illumination is composed of mixed light of B1 (P-polarized light) and B2 (S-polarized light), which are strictly different colors and different polarized lights. The same applies to G and R illumination lights. These illumination lights can be regarded as non-polarized light of B, G, and R, respectively, at the time of irradiation. For this reason, it is substantially the same as the known frame sequential illumination.

  When illumination light is applied to the subject, the return light reflected by the subject is observed by the monochrome broadband polarization image sensor 105. In FIG. 11A, only the basic unit 801 of the polarization mosaic included in the polarization imaging device 105 is shown. Of the four polarizers included in the basic unit 801, two polarizers (P polarization filters) positioned at the upper left and lower right transmit P-polarized light polarized in the horizontal direction within the imaging surface. The two polarizers (S polarization filters) located in the upper right and lower left transmit S-polarized light polarized in the direction perpendicular to the imaging surface.

  Since the polarization operation of the monochrome broadband polarization imaging device 115 functions over a range of 400 nm to 800 nm corresponding to the entire visible light region, it is 1 regardless of whether the subject is irradiated with B, G, or R color illumination light. A single image sensor can be used.

  The obtained captured image receives return light from a subject that has received non-polarized illumination through a P polarizing filter or an S polarizing filter. For this reason, a non-polarized image can be acquired by averaging the obtained pixel values in a 2 × 2 four-pixel region. The averaged pixel value is virtually located at the center of 2 × 2 4 pixels. For this reason, on the rightmost side of FIG. 11A, NP (non-polarized light) is described in the pixel region indicated by the dotted line. By shifting 2 × 2 4 pixels in units of 1 pixel, resolution degradation does not substantially occur.

  In this way, non-polarized imaging is realized for the non-polarized plane sequential illumination of B, G, and R, respectively. When the three primary color images are sequentially stored in the color image buffer memory and the three primary color images are obtained, these images are combined to generate a full-color moving image. The above processing is referred to as “polarization mosaic pixel averaging processing” in this specification. Polarization mosaic pixel averaging processing is executed by the polarization mosaic processing unit 202 in FIG. 6, and full-color moving image generation is executed by the image composition unit 206.

  FIG. 11B shows the above operation in a timing chart. FIG. 11B illustrates a color component image processed in the light emission operation, the imaging operation, and the polarization mosaic processing unit 202 from the upper part of FIG. 11B. Each operation at this timing is realized by the synchronization circuit 112 controlling the illumination control unit 120, the monochrome broadband polarization imaging device 115, and the polarization mosaic processing unit 202.

  Next, the operation in the polarization imaging mode will be described with reference to FIGS.

  FIG. 12 and FIG. 13 are a diagram and a timing chart showing an outline of operation of illumination and imaging in the polarization imaging mode, respectively. In this example, the polarization imaging color wheel shown in FIG. 7A is used.

  The left side of FIG. 12 shows the spectrum of frame sequential illumination. The illumination of B color and G color is determined in consideration of characteristics of mucous membranes such as a digestive organ that is a subject. Since the B and G bands are shorter in wavelength than the R band, they are more likely to be surface scattered, and are suitable for observing light scattering on the surface texture. In the B and G bands, since strong absorption exists in the reflection characteristics of the biological mucous membrane, the contrast becomes high, and it is suitable for observing the surface uneven texture.

  In the present embodiment, B1G1 that is P-polarized light and B2G2 that is S-polarized light are alternately irradiated on the subject by rotation of the color wheel for polarization imaging. The illumination light is applied to the subject, and the return light reflected from the subject is observed by the monochrome broadband polarization image sensor 105. Different polarization components are measured in the basic unit 801 of the polarization mosaic. A total of four types of pixel information of orthogonal Nicols (P⊥) and parallel Nicols (P‖) in P-polarized illumination and orthogonal Nicols (S⊥) and parallel Nicols (S‖) in S-polarized illumination are obtained. However, it is necessary to interpolate the values of the pixels displayed with ★ which are spatially insufficient as pixel information based on the values of the pixels located in the surrounding area. This interpolation process can be realized by a simple averaging process from four surrounding pixels.

  FIG. 13 is a timing chart showing the above operation. From the upper part of FIG. 13, a light emission operation, an imaging operation, and a color component image processed in the mosaic processing unit are shown. The synchronization circuit 112 controls the operation at this timing. At the timing of alternately irradiating P-polarized light and S-polarized light, orthogonal Nicol (P⊥) (S さ れ る) images and parallel Nicol (P‖) (S⊥) images corresponding to each are output as monochrome images. Here, the “monochrome image” is a luminance image indicating polarization information in the B and G wavelength bands. By displaying the orthogonal Nicols (P⊥) (S⊥) alternately and continuously, an image in which the invisible surface irregularities as shown in FIG. 5 are clarified can be obtained.

  FIG. 14 and FIG. 15 are a diagram and a timing chart, respectively, showing an outline of illumination and imaging operations in the polarization imaging mode when the polarization imaging color wheel of FIG. 7C is used. In this case, BGR frame sequential color illumination is performed. Such imaging is particularly effective when the mucosal surface is visually observed while removing specular reflection and the like. It is also suitable for applications in which the polarization characteristics inside the biological mucous membrane are to be observed as a narrow wavelength band.

  As for the illumination light, the subject is sequentially irradiated with B-polarized B1G1R1 and S-polarized B2G2R2 by the rotation of the color wheel for polarization imaging in FIG. 7C. The return light reflected by the subject is observed by the monochrome broadband polarization imaging device 105, and different components are imaged by the basic unit 801 of the polarization mosaic. The obtained captured images are RGB full-color orthogonal Nicols (P⊥) and parallel Nicols (P‖) in P-polarized illumination, and orthogonal Nicols (S⊥) and parallel Nicols (S‖) for each wavelength band of RGB in S-polarized illumination. A total of 12 types of image information. In this case as well, since there is a shortage of pixel information, it is necessary to interpolate the pixels displayed with ★ from the surroundings. By adopting such a configuration, it is easy to observe the mucosal state by removing specular reflection and the like from the mucosal surface during real-time observation with an endoscope. In this example, in order to enable high-speed reproduction as a moving image, the order of the filter arrangement on the circumference on the wheel is B1-G2-R1-B2-G1-R2, and RGB colors and polarization P And the type of S are alternating.

  FIG. 15 is a timing chart showing the above operation. The upper part of FIG. 15 shows the light emission operation of the illumination, the imaging operation, and the color component image processed in the mosaic processing unit. The synchronization circuit 112 controls the operation at this timing. A crossed Nicol (P⊥) (S⊥) image and a parallel Nicol (P‖) (S⊥) image corresponding to the P-polarized light and S-polarized light are output at the timing of alternate irradiation. However, in order to obtain an RGB full-color image of crossed Nicols (P⊥) or (S⊥), it is necessary to have a time of Tps, which is a period in which B, G, R frame sequential light is irradiated with polarized illumination fixed. Become. Therefore, this time cannot be displayed as a moving image. This is because both polarized light and color illumination are realized in a frame sequential manner, and it takes time to finish irradiating all types of illumination. In observation with an endoscope, real-time performance is required to reproduce the motion. Therefore, in the embodiment, for the purpose of displaying moving images of orthogonal Nicol images, an image in which (P （) and (S⊥) are mixed as much as possible is processed and displayed in a pipeline manner. For example, the RGB full-color orthogonal Nicol image 1510 in FIG. 15 includes an image 1501 under B1 (P) illumination, an image 1502 under G2 (S) illumination, and an image 1503 under R1 (P) illumination. An image 1511 processed and displayed at the next timing in a crossed Nicol image in which polarization is mixed includes an image 1502 under G2 (S) illumination, an image 1503 under R1 (P) illumination, and under B2 (S) illumination. This is an orthogonal Nicol image in which P-polarized light and S-polarized light are mixed as image 1504 in FIG.

  When such processing is performed, the illumination of the orthogonal Nicol image includes P⊥ and S⊥ in a well-balanced manner, which is advantageous for detection of a concave region in the subsequent stage. In addition, by changing the color plane sequential change of BGR faster than the change of polarization, recognition as a color image is not lost when a moving image is observed by a human.

  FIG. 16 is a diagram for explaining processing in the recessed area detection unit 204 and the image composition unit 206. FIG. 17 is a diagram illustrating a mask pattern of peripheral pixel positions used for the difference between the central pixel value and the peripheral pixel average value in the concave area detection unit 204. The concave area detection unit 204 receives the orthogonal Nicol image generated by the above processing.

  Hereinafter, an example of processing when a full-color orthogonal Nicol image described with reference to FIGS. 14 and 15 is acquired for each frame will be described. Here, an orthogonal Nicol image composed of full-color R, G, and B components is extracted from the G component and processed in the order of smoothing processing, differentiation processing, and blue enhancement processing shown in FIG.

(1) Smoothing process The input image is subjected to removal of noise components in a higher frequency range than the frequency of the texture to be emphasized before executing the subsequent differentiation process. Specifically, smoothing filter processing is executed to remove noise components. In this embodiment, a general Gaussian filter is used. By making the mask size of the filter the same as the mask size of the differential mask filter described later, emphasis of fine granular noise can be avoided.

(2) Differentiation Processing The following differentiation mask processing is performed on the G component image that has been subjected to the smoothing filter processing in order to detect a pixel region brighter than the surroundings. The reason why the pixel area brighter than the surroundings is detected is that, as described with reference to FIGS. 3 to 5, when the polarization direction of the polarized illumination is close to parallel to the surface groove of the subject, the brightness becomes brighter than the surroundings. . Actually, the direction of the surface unevenness of the subject is unknown. However, in the embodiment of the present disclosure, the polarization direction of the polarized illumination is alternately changed between two orthogonal directions, and two types of orthogonal Nicols images (P⊥) and (S⊥) can be obtained alternately. ,No problem. In the differentiation process, a mask (in this example, 3 × 3, 5 × 5, and 7 × 7 pixels) for specifying the center pixel and the peripheral pixels as shown in FIG. 17 is set in the smoothed image. Then, a difference value Δ between the average value of the pixel values Vkl of the peripheral surrounding N = 8 pixels, N = 16 pixels, or N = 24 pixels and the central pixel value Cij is calculated. The difference value Δ thus obtained is expressed by the following equation (1).

Next, when the central pixel value is brighter than the surroundings, ΔC is a value obtained by amplifying the difference value Δ by k times as shown in the following formula 2. If it is dark, ΔC is 0.
(Equation 2)
If (Δ> 0) ΔC = k * Δ
else ΔC = 0

(3) Enhancement of blue component Blue is emphasized by subtracting the ΔC value from the R and G components. Here, when the R and G components become 0 or less, the shortage is subtracted from other color components to maintain continuity. For this reason, although the hue changes depending on the magnitude of Δ, the connection can be made smoothly. Of the R and G components, the smaller value is C1, and the larger value is C2, and the three types are classified as follows.

  FIG. 18 shows the following three cases.

First, 1) When ΔC is small, a process of subtracting from the R and G signals is executed. Next, when ΔC becomes a value exceeding C1 in 2), the smallest signal becomes zero and the remainder is subtracted from the intermediate signal. Next, when the subtraction from the R and G signals becomes zero in 3), the remaining signals are subtracted from the B signal. With the above processing, the color signal of the pixel area where the center pixel is brighter than the surrounding pixels is blue-enhanced according to the degree, and a color image similar to the indigo carmine distribution is generated.
(Equation 3)
1) When ΔC ≦ C1 C1 = C1−ΔC
C2 = C2-ΔC

2) When C1 <ΔC ≦ (C1 + C2) / 2 C1 = 0 C2 = (C1 + C2) − (2ΔC)

3) When (C1 + C2) / 2 <(ΔC) C1 = 0 C2 = 0
B = B-((2ΔC) -C1-C2)

(4) Processing of Image Composition Unit As shown in FIG. 15, the image composition unit 206 accumulates R, G, and B images acquired by frame sequential illumination. Then, the three images R, G, and B are synthesized for each frame to synthesize a full-color image that can be displayed in real time. Furthermore, a full-color image in which the blue component enhancement is applied to the concave portion of the surface texture is displayed as a moving image without delay every frame time.

  FIG. 19 is a diagram illustrating an example of an image obtained by the image processing apparatus of the present embodiment. The subject is a mucous membrane obtained by dissecting a rat, opening a stomach, and extending and fixing it on a cork board. FIGS. 19A and 19B show a parallel Nicol image and an orthogonal Nicol image of the subject, respectively. FIG. 19C shows an image obtained by executing the recess detection process shown in the present embodiment. In FIG. 19C, a monochrome image is displayed, but the actually obtained image is a full-color image, and the surface uneven texture existing on the surface mucosa of the stomach is detected and colored pseudo blue. Indicates the state.

(Second Embodiment)
FIG. 20 is a diagram schematically illustrating the overall configuration of the image processing apparatus according to the second embodiment of the present disclosure. In this embodiment, white (white) light is irradiated and color imaging is performed by the single plate color imaging device 119. In the present embodiment, polarization rotation illumination is realized when irradiating a subject with white light. For this purpose, in this embodiment, only the illumination control unit is arranged in the light source device 104, and the illumination light is generated by the LED arranged at the distal end portion of the endoscope or the organic EL surface light source.

  In the present embodiment, for example, as shown in FIG. 21, a large number of illumination light irradiation apertures having a polarization plane of 0 ° (P) and 90 ° (S) are alternately arranged at the distal end of the endoscope (here, in 16 divisions). ) Is arranged. In this example, eight identical types of LEDs that are not adjacent to each other are selected and turned on to realize alternately polarized illumination of P-polarized light and S-polarized light.

FIG. 22A shows another example of polarization rotation illumination. In this example, the illumination pixel unit that is sequentially turned on is sufficiently small and the quantity is increased, so that the fluctuation of the light source position that is turned on can be suppressed within one pixel on the imaging side. FIG. 22B is a diagram showing the overall configuration of this surface illumination, and a data driver for sequentially controlling lighting is prepared for each of the X-axis and Y-axis of the surface illumination. Pixels addressed by the axis are lit all at once. For example, here, if pixels with an even number on both the X axis and the Y axis (X _2m and Y _2m ) are turned on all at once, it becomes illumination light with a polarization plane of 0 °. Then, illumination light having a polarization transmission plane of 0 ° (P) and 90 ° (S) can be obtained by combining the even and odd numbers of the X-axis and Y-axis data drivers.

  An advantage of using such surface illumination is that only the polarization state of the illumination can be changed while the overall illuminance and light distribution state are constant. By using a surface light source for illumination, the uniformity of illumination light is improved. As a result, a very strong specular brightness on the surface mucosa of the organ can be reduced, and imaging can be performed satisfactorily.

  FIG. 23 is a diagram illustrating an example of a cross-sectional structure of the color polarization image sensor 119 used in the present embodiment. A difference from the monochrome broadband polarization image pickup device 115 shown in FIG. 10 is that a color filter 240 is disposed between a wire grid layer 224 and a PD (photodiode) 232. The color filter 240 may be formed of an organic material, or may be formed of a photonic crystal or a metal. When viewed from the light traveling direction from the light incident side to the PD 232, there are a total of six possible arrangement orders of the micro lens 220, the first wire grid layer 222, the second wire grid layer 224, and the color filter 240, The advantages are different. Since the distance DEPTH between the wire grid 224 and the PD 232 becomes longer as the color filter 240 enters, it is typically 4 to 6 μm.

  In the configuration of FIG. 23 (from the top layer, the microlens 220, the first wire grid layer 222, the second wire grid layer 224, and the color filter 240), since the microlens 220 is located on the top layer, light is emitted perpendicularly to the wire grid. Easy to enter.

  FIG. 24 is a diagram showing a planar structure of the color polarization image sensor 119 of FIG. FIG. 24A shows the same planar structure as the color single-plate image sensor. In the configuration example of FIG. 24A, the 4 × 4 pixel region is enlarged and observed from directly above, and the color mosaic structure of FIG. 24B and the polarization mosaic structure of FIG. Overlapping condition.

  FIG. 24B illustrates an example of the color mosaic filter, but the color mosaic filter that can be used in the embodiment of the present disclosure is not limited to this example. Other mosaic structures other than the Bayer mosaic may be used. In this example, one color filter included in the color mosaic covers an area of four pixels (four photodiodes) arranged in two rows and two columns. The 2 × 2 pixel region corresponds to the four types of polarization mosaic regions in FIG. That is, when the sub-pixel is used as a reference, the resolution of the image sensor is 1/2 × 1/2 of the original pixel and the number of pixels is reduced. However, the polarization processing is generated by performing the polarization processing within one pixel. Artifacts to be reduced.

Next, the operation in the normal imaging mode in this embodiment will be described with reference to FIG. The subject is irradiated with white (white) P-polarized light and white (white) S-polarized light alternately, and a polarized color mosaic image is acquired each time. A polarized pixel pattern 2503 is acquired by the polarization mosaic 2502 when the P-polarized light is irradiated, and a polarized pixel pattern 2504 is acquired when the S-polarized light is irradiated. Here, P‖ and P⊥ indicate pixels in the parallel Nicols state and the orthogonal Nicols state at the time of irradiation of P-polarized light, respectively. Similarly, S‖ and S⊥ denote pixels in the parallel Nicols state and the orthogonal Nicols state at the time of S-polarized light irradiation, respectively. The polarization mosaic processing unit 202 performs addition averaging of the images of the polarization pixel pattern 2503 and the polarization pixel pattern 2504 for each pixel. In the polarization pixel pattern 2503 and the polarization pixel pattern 2504, when the averaging process is performed on each color pixel, the pixel value in the parallel Nicol state and the pixel value in the orthogonal Nicol state are evenly mixed as follows.
(Equation 4)
(NP) = (P‖ + P⊥ + S‖ + S⊥) / 4

  As a result V of the addition average, a non-polarized (NP) color mosaic image 2505 is obtained which is reduced to 1/2 × 1/2 from the original resolution. The process of generating a full color image from the non-polarized color mosaic image 2505 can be performed by normal color mosaic interpolation.

  FIG. 26 is a timing chart showing the above operation. 26 shows the light emission operation of the illumination, the imaging operation, and the color component image processed in the polarization mosaic processing unit 202 from the upper part of FIG. The synchronization circuit 112 controls the operation at this timing. Polarized pixel patterns 2503 and 2504 corresponding to the P-polarized light and S-polarized light are alternately imaged at the timing of alternate irradiation. In the polarization mosaic processing unit 202, an addition average processing of the polarization pixel patterns 2503 and 2504 shown in FIG. 25 is executed to obtain a non-polarization color mosaic image 2505. Next, an RGB full-color image is obtained by color mosaic interpolation processing. Therefore, it is possible to obtain a single RGB full-color image by irradiation with P-polarized illumination and S-polarized illumination. In practice, as shown in FIG. 26, an image is generated as a moving image without delay for each frame time by continuously processing the P-polarized illumination and the S-polarized illumination that are temporally adjacent.

FIG. 27 is a diagram for explaining the operation in the polarization imaging mode in the present embodiment. In the polarization imaging mode, P-polarized light and S-polarized light are alternately irradiated on the subject, and a polarized color mosaic image is acquired each time. Polarized pixel patterns 2503 and 2504 obtained here are the same as the polarized pixel patterns 2503 and 2504 in FIG. The polarization mosaic processing unit 202 uses both the pixel patterns 2503 and 2504 to select and accumulate P‖ and S‖ and P⊥ and S⊥ for each corresponding pixel. In this way, a parallel Nicol image 2701 with PS polarization mixing and an orthogonal Nicol image 2702 with PS polarization mixing are generated separately. Then, the four pixels in the same color pixel are added and averaged as follows.
(Equation 5)
(PS‖) = (P‖ + S‖ + P‖ + S‖) / 4
(PS⊥) = (P⊥ + S⊥ + P⊥ + S⊥) / 4

  Polarized images obtained as a result of this processing are color mosaic images 2703 and 2704 that have been reduced to ½ × ½ from the original resolution. The same color pixel in the color mosaic images 2703 and 2704 constitutes a parallel Nicol image PS‖ and an orthogonal Nicol image PS⊥ in a state where P-polarized illumination and S-polarized illumination are equally included. Color orthogonal interpolation is performed on the orthogonal Nicol image PS 、 to generate a full-color orthogonal Nicol image. For the orthogonal Nicol image, the concave region detection unit 204 and the image composition unit 206 perform the same processing as that described in the first embodiment.

  FIG. 28 is a timing chart showing the above operation. FIG. 28 shows a color image processed by the light emission operation, the imaging operation, and the polarization mosaic processing unit 202, the color mosaic interpolation unit 208, the concave region detection unit 204, and the image composition unit 206 from the top. The operation up to the illumination and imaging is the same as the timing chart in the normal imaging mode in FIG. In the operation of the polarization mosaic processing unit 202, a parallel Nicol image 2701 with PS mixed polarization and a crossed Nicol image 2702 with PS polarized light are generated by using captured images of P polarization illumination and S polarization illumination for each frame. That is, two types of polarized pixel patterns 2701 and 2702 are generated simultaneously for each frame. In addition, a color mosaic image 2703 and a color mosaic image 2704 obtained by averaging these are also generated at the same time. The resolutions of the color mosaic images 2703 and 2704 are reduced to ½ × ½. As described with reference to FIG. 16 relating to the first embodiment, the orthogonal Nicol image 2704 is a full-color image obtained by emphasizing the blue component in the concave portion of the surface texture in the concave region detection unit 204 and the image synthesis unit 206 every frame. A moving image is displayed on the display unit 114 every time.

(Modification 1 of 2nd Embodiment)
FIG. 29 is a diagram illustrating a first modification of the second embodiment of the present disclosure. FIG. 29A is a diagram showing a planar structure of the color polarization image sensor 119 in the second embodiment shown in FIG. FIG. 29A shows the same planar structure as that of the color single plate image sensor. FIG. 29B shows an arrangement example of 4 × 4 color filters in the color mosaic, and FIG. 29C shows an arrangement example of eight polarizers in the polarization mosaic. These color mosaic and polarization mosaic are stacked so as to cover 4 × 4 pixels (PD: photodiode).

  In the present embodiment, two color filters of a color mosaic correspond to one rectangular polarizer. Other configurations are the same as those of the second embodiment.

  In FIG. 29C, the pixel where the 0 ° polarizer of angle display is located is a P-polarized transmission pixel, and the pixel where the 90 ° polarizer is located is an S-polarized transmission pixel. The 0 ° polarizer and the 90 ° polarizer do not form a checkered array. That is, the polarization mosaic is configured such that the same polarized light is incident on two pixels adjacent in the vertical or horizontal direction within the imaging surface. This is because a 0 ° polarizer and a 90 ° polarizer are always assigned to G pixels that occupy two of the RGB pixels. According to such a configuration, a 0 ° polarizer is assigned to two RG pixels and two BG pixels, and a 90 ° polarizer is assigned to two GB pixels and two GR pixels.

FIG. 30 is a diagram illustrating the operation in the normal imaging mode in the present embodiment. In the normal imaging mode, white (white) P-polarized light and S-polarized light are alternately irradiated, and an image is acquired each time, and a polarized color mosaic image is acquired. Since the polarization mosaic has an arrangement such as 3001, a polarization pixel pattern indicated by 3002 is acquired when P-polarized light is irradiated, and a polarization pixel pattern indicated by 3003 is acquired when S-polarized light is irradiated. Here, P‖ and P⊥ indicate pixels in a crossed Nicols state when irradiated with P-polarized light, and similarly, S‖ and S⊥ indicate a crossed Nicols state when irradiated with S-polarized light. Indicates a pixel. In the polarization mosaic processing unit 202, the polarization pixel patterns 3002 and 3003 are averaged for each pixel. In the averaging process, it is considered that pixels in the parallel Nicols state and the orthogonal Nicols state are evenly mixed as follows.
(Equation 6)
(NP) = (P⊥ + S‖) / 2
(NP) = (P‖ + S⊥) / 2

  As a result V of the addition averaging, a non-polarized color mosaic image 3004 is obtained. At this time, there is no reduction in resolution as in the second embodiment. The process of generating a full color image from the non-polarized color mosaic image 3004 can be performed by normal color mosaic interpolation.

  FIG. 31 is a diagram for explaining the operation in the polarization imaging mode in the present embodiment. In the polarized illumination, P-polarized light and S-polarized light are alternately irradiated, and an image is acquired each time, and polarized image patterns 3102 and 3103 are acquired. The polarization mosaic processing unit 202 collects P‖ and S‖ and P⊥ and S⊥ for each corresponding pixel by using both of the polarization pixel patterns 3102 and 3103 and performs the embedding process. In this way, a parallel Nicol image 3104 with PS polarization mixing and an orthogonal Nicol image 3105 with PS polarization mixing are generated separately. Polarized images obtained as a result of this processing are color mosaic images 3106 and 3107, which are a parallel Nicol image PS‖ and an orthogonal Nicol image PS⊥ in a state where P-polarized illumination and S-polarized illumination are equally included, respectively. For the orthogonal Nicol image PS 画像, color mosaic interpolation is performed to generate a full-color orthogonal Nicol image. The orthogonal Nicol image is subjected to processing similar to the processing described in the first embodiment in the concave area detection unit 204 and the image synthesis unit 206. The timing chart related to this embodiment is the same as the timing chart related to the second embodiment.

(Modification 2 of the second embodiment)
FIG. 32 is a diagram illustrating a second modification of the second embodiment of the present disclosure. FIG. 32A is a diagram showing a planar structure of the color polarization image sensor 119 of FIG. FIG. 32B shows an arrangement example of 4 × 4 color filters in the color mosaic, and FIG. 32C shows an arrangement example of four polarizers in the polarization mosaic. These color mosaic and polarization mosaic are stacked so as to cover 4 × 4 pixels (PD: photodiode).

  In the present embodiment, one unit of four pixels of the color Bayer mosaic corresponds to one unit of the polarization mosaic. Other configurations are the same as those of the second embodiment. In FIG. 32C, the pixel where the 0 ° polarizer in the angle display is located is a P-polarized transmission pixel, and the pixel where the 90 ° polarizer is located is an S-polarized transmission pixel. The 0 ° polarizer and the 90 ° polarizer of the polarization mosaic form a checkered arrangement. Each array includes the same color Bayer mosaic.

  FIG. 33 is a diagram illustrating the operation in the normal imaging mode in the present embodiment. In the normal imaging mode, white (white) P-polarized light and S-polarized light are alternately irradiated, and an image is acquired each time, and a polarized color mosaic image is acquired. Since the polarization mosaic has the arrangement 3301, the polarization pixel pattern 3302 is acquired when the P-polarized light is irradiated, and the polarization pixel pattern 3303 is acquired when the S-polarized light is irradiated. Here, P‖, P⊥, S‖ and S⊥ have the meanings already mentioned. The polarization mosaic processing unit 202 executes addition averaging for each pixel with respect to the image having the polarization pixel patterns 3302 and 3303. In the averaging process, parallel Nicols and orthogonal Nicols pixels are evenly mixed (Equation 6).

  As a result V of the addition averaging, a non-polarized color mosaic image 3304 is obtained. In this case, the feature is that there is no reduction in resolution as in the second embodiment. The process of generating a full color image from this non-polarized color mosaic image can be performed by normal color mosaic interpolation.

  FIG. 34 is a diagram for explaining the operation in the polarization imaging mode in the present embodiment. Polarized illumination is alternately irradiated with P-polarized light and S-polarized light, and an image is acquired each time, and polarized pixel patterns 3402 and 3403 are acquired.

  The polarization mosaic processing unit 202 collects and embeds P‖ and S‖ and P ご と and S⊥ for each corresponding pixel using both of the polarized pixel patterns 3402 and 3403, thereby performing parallel Nicols of PS polarization mixing. An image 3404 and a PS polarized mixed orthogonal Nicol image 3405 are separately generated. The polarization images obtained as a result of this processing are color mosaic images such as 3406 and 3407, and the parallel Nicol image PS‖ and the orthogonal Nicol image PS⊥ with the P polarization illumination and S polarization illumination included equally, respectively. It is. For the orthogonal Nicol image PS 画像, color mosaic interpolation is performed to generate a full-color orthogonal Nicol image. The orthogonal Nicol image is subjected to processing similar to the processing described in the first embodiment in the concave area detection unit 204 and the image synthesis unit 206. The timing chart relating to this embodiment is the same as the timing chart of the second embodiment.

(Third embodiment)
FIG. 35 is a diagram showing the configuration of the third exemplary embodiment of the present invention. Also in this embodiment, white (white) light is irradiated onto a subject and color imaging is performed with a single-plate color imaging device. The difference from the second embodiment is that a polarizing plate and a color filter are arranged in the lens opening, and a microlens array type color polarization imaging unit 3501 in which a microlens array is arranged on the imaging surface. And a pixel selection / re-integration unit 210 for performing image processing unique to the microlens array type element.

  FIG. 36 is an enlarged view of the distal end portion of the endoscope in the present embodiment from the front. A large number (in this case, 16 divisions) of illumination light irradiation ports having different polarization planes of 0 ° (P) and 90 ° (S) are arranged at the distal end of the endoscope. With this illumination, eight LEDs of the same type that are not adjacent to each other are selected and turned on, and P and S alternately polarized illumination is realized by imaging. On the objective lens 3502 serving as the opening, there are arranged 2 × 2 total four types of color and polarized composite filter regions as shown in the figure. These are the R and B non-polarized color filter regions and two orthogonal color polarizing filter regions of G 0 ° (P) and G 90 ° (S).

  FIG. 37 is a diagram illustrating a configuration example of the microlens array type color polarization imaging unit 3501. In FIG. 37, for convenience of illustration, among the four regions on the objective lens 3502, two regions of the G filter in FIG. 36, that is, the G filter similar to the region 3701 in which the G filter and the 0 ° (P) polarizing filter are arranged. And only two regions 3702 in which a 90 ° (S) polarizing filter is arranged.

  As shown in FIG. 37, the light diverging from one point 3700 on the subject passes through two regions 3701 and 3702 on the objective lens 3502, respectively, and passes through the arrayed optical element 3703, thereby taking an imaging surface 3704 of the monochrome imaging element. To reach. At this time, since the image of the region on the objective lens reaches two different pixels 3705, the image formed on the imaging surface 3704 is the image of the subject as a whole, but in detail, the images of two different regions From this, the image is established. By performing digital image processing for selecting and accumulating pixels from here, it is possible to separate and generate an image that has passed through two regions, and at the same time, it is possible to obtain a color image using a monochrome imaging device.

  FIG. 38 is a diagram showing a cross-sectional structure of the color polarizing filter regions 3701 and 3701 in the opening. In this example, a metal wire grid layer 3801 is used as a polarizing filter. For the wire grid layer 3801, for example, a metal wire having a pitch of about 100 nm can be formed on the transparent substrate 3802. A color filter 3803 is disposed in the lower layer. An objective lens 3502 is arranged at the next stage of the color filter 3803. Here, the arrangement order of the color filter, the wire grid layer, the objective lens, and the presence or absence of a gap with the lens are arbitrary. Further, as the polarizing plate, existing techniques such as a polymer polarizing plate other than the wire grid, a polarizing plate using a photonic crystal, and a polarizing plate using structural clothing refraction can be used.

  FIG. 39 is a diagram for explaining processing in the pixel selection / re-integration unit 210 that generates a color polarization image from an imaging result using the microlens array type color polarization imaging device. By selecting the upper left, upper right, lower left, and lower right pixels for each 2 × 2 pixel unit from the image on the image sensor 3704 over the entire image and integrating them again, the resolution is reduced to ½ × ½. However, it is possible to separate the P (0 °) polarization image 3901 of G, the non-polarization image 3902 of R, the non-polarization image 3903 of B, and the S (90 °) polarization image of G, respectively. From this, an RGB non-polarized color image and a P / S polarized image in the G wavelength region can be obtained.

FIG. 40 shows images acquired in the normal imaging mode and the polarization imaging mode in the endoscope according to the present embodiment. In both the normal imaging mode and the polarization imaging mode, the white (white) polarized illumination is alternately irradiated with P-polarized light and S-polarized light, and an image is acquired each time, and the process of FIG. A seeded color polarization image is acquired separately. The four types of color polarization images obtained separately are displayed as 4001. This display method does not represent individual pixels as in the prior art, but represents the entire four images. A polarized image 4002 is acquired when P-polarized light is irradiated, and a polarized image 4003 is acquired when S-polarized light is irradiated. Here, P‖, P⊥, S‖, and S⊥ have the above-mentioned meanings, but P or S is an image captured in a non-polarized light without a special polarizing filter under P-polarized illumination or S-polarized illumination. It has the meaning. In the normal imaging mode, the images 4002 and 4003 are averaged for each pixel. In the averaging process, pixels in the parallel Nicols state and the orthogonal Nicols state are considered to be evenly mixed as follows, and this result is approximately an unpolarized image.
(Equation 7)
(NP) = (P⊥ + S‖) / 2
(NP) = (P‖ + S⊥) / 2
(NP) = P + S

  As a result V of the addition average, a non-polarized color mosaic image 4004 is obtained. The process of generating a full color image from this non-polarized color mosaic image can be performed by normal color mosaic interpolation.

  Even in the polarization imaging mode, the same process is performed until the image 4002 and the image 4003 are alternately acquired. However, by collecting parallel Nicols and orthogonal Nicols images using only the G image, PS‖4005 and PS⊥4006 are collected in the G wavelength region. These two types of polarization images can be generated, and the output image is a monochrome image as shown in FIG. 13 in the first embodiment.

  The advantage of the microlens array type polarization imaging device as in the present embodiment is that a polarizing plate can be arranged in the lens opening, so that the size of each polarization mosaic element can be made larger than that arranged on the imaging device. is there. For example, in the polarization mosaic type image sensor used in Embodiments 1 and 2, the length of the metal wire forming the polarization mosaic unit is equal to the pixel size of the image sensor and is 2 to 3 μm. With such a fine size, even if the pitch of each metal wire of the wire grid is fine, the wire grid length and the number of repetitions are limited, so that the extinction ratio performance as a polarizing plate is reduced to about 10: 1. . In the present embodiment, a relatively large wire grid polarizer having a lens opening size of about 0.5 mm = 500 μm can be used, and a high extinction ratio of about 100: 1 can be realized. This is extremely advantageous in clarifying the observation of the surface fine pattern in the mirror.

(Fourth embodiment)
FIG. 41 is a diagram showing the configuration of the fourth exemplary embodiment of the present invention. Similarly to the second embodiment, the present embodiment performs white-white light irradiation and performs color imaging with a single-plate color imaging device. This embodiment is different from the second embodiment in that a microlens array type color polarization imaging unit 4101 is used. The microlens array type color polarization imaging unit 4101 in the present embodiment is different from the microlens array type color polarization imaging unit used in the third embodiment in the following configuration.

  FIG. 42A is a diagram illustrating a configuration example of the microlens array type color polarization imaging unit 4101. In the lens opening, only a polarizing filter 4103 having a broadband type 0 ° (P) transmission axis and 90 ° (S) transmission axis is arranged, and colorization is performed by a single-plate color imaging device 4104 having a Bayer mosaic 4105. To do. By doing so, the operation of polarization and color is separated. As a result, RGB full-color parallel / orthogonal Nicol images can be obtained. As will be described later, the light transmitted through the four regions (UL), (UR), (DL), and (DR) of the polarizing filter 4103 by the action of the microlens 3703 causes (UL1) (UR1) ( Each of the images is formed in four areas DL1) and DR1.

  FIG. 42B schematically illustrates an example of a cross-sectional configuration of the microlens array type color polarization imaging unit 4101. FIG. 42B shows only two areas of the four areas on the objective lens 3502, the area 4202 where the 90 ° polarization filter is arranged in FIG. 41 and the area 4202 where the 0 ° polarization filter is arranged. The light diverging from one point 3700 on the subject passes through the two areas 4201 and 4202 on the objective lens 3502 and reaches the color imaging element surface 4203 where the color mosaic is arranged via the arrayed optical element 3703. At this time, an image formed by light rays passing through the two areas 4201 and 4202 on the objective lens reaches different pixels 4204. For this reason, the image formed on the imaging surface 4203 is an image of the subject as a whole, but in detail, is formed from images of different polarization regions of 0 ° and 90 °. The images of the areas 4201 and 4202 correspond to two color mosaic pixels on the color image sensor 4203.

  FIG. 43 is a diagram showing a cross-sectional structure of the polarizing filter regions 4201 and 4202 in the opening according to the present embodiment. In this example, a metal wire grid layer 3801 is used as a polarizing filter. For the wire grid layer 3801, for example, a metal wire having a pitch of about 100 nm can be formed on the transparent substrate 3802. According to such a wire grid layer 3801, a polarization operation can be realized in a wide band in the visible light range.

  In the next stage, an objective lens 3502 is arranged. Here, the arrangement order of the wire grid layer 3801 and the objective lens 3502 and the presence or absence of a gap between the wire grid layer 3801 and the objective lens 3502 are arbitrary. The polarizing plate may be a polymer polarizing plate other than the wire grid polarizing plate as long as the polarizing operation is realized in a wide range of the visible light band.

  FIG. 44 is a diagram for explaining the processing of the pixel selective re-integration unit 210. By selecting the upper left, upper right, lower left, and lower right pixels from the image on the image sensor 4203 for each 2 × 2 pixel unit over the entire image and integrating them again, the resolution is reduced to 1/2 × 1/2. However, the P (0 °) polarized color image 4401, the S (90 °) polarized color image 4402, the S (90 °) polarized color image 4403, and the P (0 °) polarized color image 4404 can be separated, respectively. . Thereafter, the processing of the color mosaic interpolation unit 208 is performed.

  In this embodiment, since a 2 × 2 color Bayer mosaic unit called RGBG is included in one type of polarizing filter, the same information as that of the second modification of the second embodiment can be obtained. Further, as a merit of the micro lens type polarization imaging device, since a polarizing plate can be disposed in the lens opening, a large size wire grid polarizing device can be used, and a high extinction ratio of about 100: 1 can be realized. Have advantages.

  The above-mentioned images 4401 and 4404 and images 4402 and 4403 are almost the same image, and thus have redundancy in information. However, since these images are formed from light rays that have passed through different positions of the lens, parallax exists when the distance to the subject is short. In the following embodiments, stereoscopic viewing is enabled using parallax.

(Fifth embodiment)
FIG. 45 is a diagram illustrating an overall configuration of an image processing device according to the fifth embodiment of the present disclosure. In the present embodiment, it has a function of observing the uneven texture on the surface of the subject, and further enables stereoscopic viewing using parallax. This embodiment is different from the first embodiment in that the monochrome broadband polarization imaging device 115 is changed to a microlens array type monochrome polarization imaging unit 4501, a pixel selection / integration unit 210, a parallax image in the image processor 110. It is to have the composition unit 250, to change the image display unit 114 to a stereoscopic image display unit 4501, and to use stereoscopic glasses 4502 at the time of image observation.

  The microlens array type monochrome polarization imaging unit 4501 basically has the same configuration as the microlens array type color polarization imaging unit 3501 shown in FIG. 37, but the opening is as shown in FIG. Only a polarizing plate is provided, and no color filter is provided.

  For inspection endoscopes, stereoscopic vision for recognizing the distance between the forceps and the mucosa is effective in collecting mucous membranes using so-called fine forceps (so-called biopsy) or performing surgery to remove the mucosa. . In this embodiment, a stereo vision input in which the distance between the centers of gravity of the left and right regions in the lens is set as the base line length by providing left and right transparent (through) regions in a part of the aperture-type polarization imaging unit used in the previous embodiment. Form a system. That is, a parallax image is generated from light rays that have passed through the left and right regions using a microlens array type optical system. The parallax image is displayed on the stereoscopic image display unit 4501. When parallax images are displayed alternately, if the 3D glasses 4502 in which shutters provided on the left and right are alternately opened and closed are mounted, the displayed image can be viewed in 3D. Note that depending on the configuration of the stereoscopic image display 4501, a stereoscopic image can be visually recognized without the stereoscopic glasses 4502.

  FIG. 46 is a diagram for explaining an opening at the distal end portion of the endoscope, an objective lens, a filter arrangement on the lens, and the like. FIG. 46A is a perspective view of the distal end portion of the endoscope, and shows a state where the forceps 4604 protrudes from the forceps opening during the operation. FIG. 46B is an enlarged view of the distal end portion of the endoscope as viewed from the front. As shown in FIG. 46B, the distal end portion of the endoscope has an illumination lens 107, an opening 4602, and a forceps port 4603. An objective lens is provided in the opening 4602. As shown in FIG. 46 (A), the forceps 4604 protrudes from the forceps opening 4603 during the operation, but is usually stored inside the endoscope. FIG. 46C is an enlarged view of the opening 4602. As shown in FIG. 46C, four regions arranged in two rows and two columns are provided on the illumination lens 107.

  FIG. 46D is an enlarged view of the opening 4602. Of the four regions of 2 rows and 2 columns arranged on the opening 4602, the upper left (UL) and upper right (UR) of the first row are transparent (through), and the lower left of the second row ( The two regions (DL) and lower right (DR) are a P (0 °) polarized light transmission region and an S (90 °) polarized light transmission region, respectively. In these polarization transmission regions, two polarization filters whose polarization transmission axis directions intersect are arranged. Such a polarizing filter may be composed of, for example, a wire grid as shown in FIG.

  According to the present embodiment, the image by the light beam that has passed through the region located in the left column (left region) of the opening 4602 and the image that has passed through the region located in the right column (right region) are separated. Can be acquired. A parallax is generated between the two types of images by a distance (DIS) between the center of gravity of the left region of the opening 4602 and the center of gravity of the right region.

  FIG. 47 is a diagram schematically illustrating image processing of the pixel selection / re-integration unit 210. According to the present embodiment, the light beam reflected at one point of the subject passes through each of the regions UL, UR, DL, and DR of the opening 4602 and then passes to different light detection cells depending on the microphone lens in the vicinity of the imaging surface 3704. A microlens array is configured to be incident. That is, each microlens is arranged so as to cover a unit structure composed of four photodetecting elements on the imaging surface 3704. The light rays that have passed through the four regions shown in FIG. 46D are incident on the four light detection cells included in each unit structure of the imaging surface 3704 shown in the upper part of FIG. For this reason, four types of images generated by light rays that have passed through each of the regions UL, UR, DL, and DR in the opening 4602 of the microlens array type monochrome polarization imaging unit 4501 are separately generated as shown in FIG. Can be done. As a result, left and right parallax images consisting of an image obtained from a light beam that has passed through the transparent area UL and an image that has passed through the transparent area UR are obtained, and two areas DL in which polarization filters are arranged, From the DR, a P-polarized image and an S-polarized image are obtained by the above-described processing.

  If a large parallax occurs between the polarization images obtained from the regions DL and DR, it becomes difficult to detect the groove on the subject surface by the above-described processing. However, such parallax is not a problem in the mode of acquiring a polarization image. Hereinafter, the reason will be described.

  The upper side of FIG. 48 shows the distance (subject distance) from the distal end portion of the endoscope to the subject when performing an operation using the endoscope. The lower side of FIG. 48 shows the distance from the distal end portion of the endoscope to the subject when the inspection is performed using the endoscope. When excising the mucosal surface using forceps during surgery, the distance from the endoscope tip to the subject mucosa is approximately 5 to 20 mm. In general, in stereoscopic vision, a subject distance of about 15 to 30 times the baseline length can be stereoscopically viewed. Since the diameter of the objective lens of the endoscope is about 1 mm, the center-of-gravity distance DIS between the transparent regions is about 0.5 mm, and an optimal parallax for stereoscopic viewing of the subject distance is obtained. On the left and right images, for example, a pixel shift of the scene can occur by a distance corresponding to the difference between the distance X1 and the distance X2. On the other hand, at the time of examination, the subject distance is about 50 mm to 100 mm because it is necessary to observe a wide area on the organ surface. In this case, since the distance from the distal end portion of the endoscope to the subject is long, no parallax occurs, and the pixel shift can be regarded as substantially zero in the left and right images.

  Therefore, when the subject distance is long, in the polarization imaging mode, the uneven texture on the subject surface is detected, an image that artificially colors the groove is confirmed, and then the subject distance is shortened to reduce the subject distance. After shifting to, the mucosal surface of the subject may be excised while performing stereoscopic viewing. During the operation, the parallax becomes large and it is difficult to realize the polarization imaging process. However, since the indigo carmine solution can be actually distributed at a close distance, there is no problem in surface texture observation.

  The left and right images having parallax shown in FIG. 48 are reproduced on the stereoscopic image display unit 4501 as moving images in the present embodiment. The image is reproduced by using stereoscopic glasses 4502 at the time of image observation.

  FIG. 49 is a diagram illustrating a state in which the user observes the stereoscopic image by alternately displaying the left image LL and the right image RR of the parallax image on the stereoscopic image display unit 4501. In the stereoscopic image display unit 4501, only the LL image is shown during the period T1, and only the RR image is shown during the next period T2. This image is emitted as linearly or circularly polarized light 4903 by passing through a polarizing filter plate or the like. An observer observes this by wearing stereoscopic glasses 4502 composed of a liquid crystal shutter or the like. The liquid crystal shutter of the stereoscopic glasses repeatedly opens (transmits) and closes (blocks) the polarized light 4903 in synchronization with the stereoscopic image display unit 4501. That is, during the period T1, the L liquid crystal shutter is open and the R liquid crystal shutter is closed, and during the period T2, the L liquid crystal shutter is closed and the R liquid crystal shutter is open. By setting the periods T1 and T2 sufficiently short and repeating these periods, stereoscopic vision is established in the human visual system.

  In the present embodiment, two types of polarized images of 0 ° (P) and 90 ° (S) are obtained simultaneously for one scene. Since this corresponds to the polarization image obtained by the apparatus of the first embodiment, the normal imaging mode and the polarization imaging mode can be realized by combining with the field sequential illumination described in the first embodiment. In the form, a stereoscopic imaging mode is added.

(Sixth embodiment)
FIG. 50 is a diagram illustrating an overall configuration of an image processing device according to the sixth embodiment of the present disclosure. As in the second embodiment, the image processing apparatus irradiates white light and performs color imaging with a single-plate color imaging device. Even in the present embodiment, stereoscopic viewing using parallax is also possible while maintaining the function of observing the textured surface texture.

  This embodiment is different from the fourth embodiment in that transparent (through) regions are provided in the left and right regions of the filter of the aperture used in the microlens array type color polarization imaging unit 4101, and in the image processor 110. A parallax image synthesizing unit 250, the image display unit 114 is changed to a three-dimensional image display unit 4501, and three-dimensional glasses 4502 are used for image observation. The configuration of the microlens array type color polarization imaging unit 4101 is the same as the configuration shown in FIG. 42B except that a pair of transparent regions is provided in front of the objective lens 3502. Also in this embodiment, each microlens included in the microlens array 3703 shown in FIG. 42B covers 4 × 4 light detection cells on the imaging surface 4203.

  FIG. 51 is a diagram illustrating a configuration example of the endoscope distal end portion in the present embodiment. FIG. 51A is a perspective view of the distal end portion of the endoscope, and shows a forceps 4604 protruding from the forceps opening 4603 during surgery. FIG. 51B is an enlarged view of the front end portion of the endoscope as viewed from the front, and shows the illumination lens 107, the opening 4602, and the forceps port 4603. FIG. 51C is an enlarged view of the opening 4602. FIG. 51D is an enlarged view of four regions provided in two rows and two columns on the opening 4602. The configuration of the opening 4602 is the same as the configuration of the opening 4602 in the fifth embodiment.

  FIG. 52 illustrates an example of a correspondence relationship between a pair of polarizing filters provided in the regions DL and DR of the opening 4602 and the color mosaic filter on the imaging surface 4203. Due to the action of the microlens array 3703, light rays that have passed through the regions UL, UR, DL, and DR of the opening 4602 are respectively imaged on the four unit regions UL1, UR1, DL1, and DR1 of the color mosaic filter. The four unit areas UL1, UR1, DL1, and DR1 each include color filters of different colors. In the present embodiment, each unit region forms a Bayer array.

  FIG. 53 is a diagram for explaining image processing in the pixel selective re-integration unit. For each of four pixel units arranged in 2 rows and 2 columns from the image formed on the imaging surface 4023, the upper left, upper right, lower left, and lower right pixels are selected over the entire image and accumulated again. By doing so, the resolution is reduced to 1/2 × 1/2, but the left color image LL5301, right color image RR5302, S (90 °) polarized color image 5303, and P (0 °) polarized color image 5304, respectively. Can be separated.

  In the present embodiment, since 2 × 2 Bayer color mosaic units called RGBG are included corresponding to each polarization filter, information equivalent to the information obtained by the apparatus of the fourth embodiment can be obtained, and parallax can be obtained. Images can also be acquired.

(Modification of the sixth embodiment)
A modification of the sixth embodiment will be described with reference to FIG. The arrangement of the polarization transmission region and the transparent region in the opening 4602 of the microlens array type color polarization imaging device 4101 is not limited to the example shown in FIG. In the example of FIG. 54A, polarization filters of 0 ° and 90 ° are arranged at diagonal positions of four regions of 2 rows and 2 columns, and the other two regions are transparent regions. In this arrangement, the center-of-gravity distance DIS of the transparent region can be increased to about 1.4 times the center-of-gravity distance DIS in the above-described arrangement example. For this reason, the parallax of the left and right images can be increased.

  In the example of FIG. 51B, six regions arranged in two rows and three columns are provided in the opening 4602. Polarization filters of 0 ° and 90 ° are provided in a region with little parallax near the center of the opening 4602, and transparent regions are provided in left and right positions within the opening 4602. By this arrangement, the center-of-gravity distance DIS can be further increased. By adopting this arrangement, even when the subject distance is short, the pixel shift between the left and right images is reduced during polarization imaging, and stereoscopic viewing is possible even with a longer subject distance.

  In the example of FIG. 51C, 16 regions arranged in 4 rows and 4 columns are provided in the opening 4602. Polarization filters of 0 ° and 90 ° are arranged in a region with little parallax near the center of the opening 4602, and transparent regions are provided at left and right and up and down positions in the opening 4602. Therefore, the pixel shift of the image during polarization imaging is reduced, and stereoscopic viewing is possible not only on the left and right but also on the top and bottom. In the examples of FIGS. 54B and 54C, the arrangement of the microlens array 3703 is set so as to match the arrangement pattern of the openings 4602. That is, on the imaging surface, 6 photodetecting cells in 2 rows and 3 columns and 16 photodetecting cells in 4 rows and 4 columns constitute each basic unit.

  Conventionally, endoscope optical systems have set the depth of focus deep in order to make focusing unnecessary. In such an optical system, the amount of blur is small, and light rays that have passed through each of the left and right regions of the objective lens do not cause a large pixel shift on the imaging surface. Therefore, monocular stereoscopic vision is difficult with a conventional endoscope. In the embodiment of the present disclosure, monocular stereoscopic vision can be validated using an optical system with a shallow depth of focus. When using a microlens array-type image sensor, if an image with a deep focal depth is required, if the signals from the light rays that have passed through only the vicinity of the lens center are integrated by pixel selection, the same effect as the effect of narrowing the lens can be obtained. And an image with a deep focal depth can be generated.

(Seventh embodiment)
FIG. 55 is a diagram illustrating an overall configuration of an image processing device according to the seventh embodiment of the present disclosure. As in the first and fifth embodiments, the image processing apparatus operates in a frame sequential manner. In the present embodiment, a stereoscopic view using parallax is also possible while having a function of observing the textured surface texture. The feature of this embodiment is that a compound eye lens type color image pickup unit is mounted instead of the micro lens array type color polarization image pickup unit in the fifth embodiment. In the compound eye type color imaging unit, a compound eye lens unit 5601 including a plurality of rod lenses is arranged at the distal end of the endoscope as in the compound eye camera, and a plurality of images of the subject are formed on the imaging surface of the color imaging element 5503. To do.

  FIG. 56 is an enlarged view showing the distal end portion of the endoscope. 56A shows a perspective view of the distal end portion of the endoscope, and FIG. 56B shows a front view of the distal end portion. FIG. 56C is an enlarged view of the compound eye lens portion 5601. In the present embodiment, four rod lenses UL, UR, DL, DR are arranged in the compound eye lens portion 5601. The rod lenses UL and UR are transparent regions, but the rod lens DL is provided with a polarizing filter having a 0 ° (P) transmission axis, and the rod lens DR is a polarizing filter having a 90 ° (S) transmission axis. Is provided.

  FIG. 57 is a diagram illustrating processing of the image separation unit 260, and the upper, lower, left, and right images on the imaging surface 4203 are separated. By separation into four images, unpolarized images 5701 and 5702 having parallax, a P-polarized image 5703, and an S-polarized image 5704 are obtained for one type of illumination light irradiation. These processes are as described in the fifth embodiment.

(Eighth embodiment)
FIG. 58 is a diagram schematically illustrating an overall configuration of an image processing device according to the eighth embodiment of the present disclosure. As in the second, fourth, and sixth embodiments, this image processing apparatus irradiates white light and performs color imaging with a single-plate color imaging device. Even in the present embodiment, stereoscopic viewing using parallax is possible while having the function of observing the texture of the surface of the subject.

  A feature point of this embodiment is that a compound eye lens type color imaging unit 5502 is mounted instead of the micro lens array type color polarization imaging unit.

  FIG. 59 (A) is a perspective view of the distal end portion of the endoscope, and FIG. 59 (B) is an enlarged front view of the distal end portion of the endoscope. FIG. 59C is an enlarged view of the compound eye lens portion 5601. Except for the configuration for emitting illumination light, the configuration is the same as that of the seventh embodiment described with reference to FIG.

  FIG. 60 is a diagram illustrating processing of the image separation unit 260, and the upper, lower, left, and right images on the imaging surface 4203 are separated, and the processing proceeds to the subsequent color mosaic interpolation unit 208. The processing of the color mosaic interpolation unit 208 is also as described above.

  In the first embodiment and the seventh embodiment, the illumination light irradiation of the frame sequential method is performed, and in the embodiments other than those embodiments, the white polarized illumination is performed, and the orthogonal Nicol image (P⊥), (S⊥) has been acquired. The polarization direction of the polarized illumination is rotated by 45 degrees, and images (0 ° ⊥), (45 ° ⊥), (90 ° ⊥), (135 ° ⊥) are obtained for the four types of polarized illumination. May be. In this case, a polarization mosaic array is used in which the polarization direction of the polarized illumination can be rotated by 45 °, and at the same time, the polarization imaging device is also rotated by 45 °. Since four types of polarized images are acquired, it takes time for one imaging, but the surface uneven texture can be acquired with higher accuracy.

(Ninth embodiment)
FIG. 61 is a diagram schematically illustrating an overall configuration of an image processing device according to the ninth embodiment of the present disclosure. This embodiment basically has a function of observing the same surface uneven texture as that of the fifth embodiment of the present disclosure, and is also capable of stereoscopic viewing using parallax. It is a mirror.

  This embodiment is different from the fifth embodiment in that it includes a configuration of an opening area of a microlens array type monochrome broadband polarization imaging unit 6101 and a pan focus image display unit 6102. The micro lens array type monochrome broadband polarization imaging unit 6101 is basically the same as the micro lens array type monochrome broadband polarization imaging unit 3501 shown in FIG. Also in this embodiment, the imaging surface 3704 is not provided with a color mosaic filter, and the four regions of the objective lens 3502 correspond to the four pixels arranged in two rows and two columns of the imaging surface 3704.

  The image processing apparatus according to the present embodiment can obtain a pan-focus image having a deep depth of field while having a characteristic capable of acquiring orthogonal Nicol and parallel Nicol monocular stereoscopic images. In color image observation with a normal endoscope, an optical system having a pan-focus characteristic that focuses regardless of the distance to the subject is used. However, with such optical characteristics, the light beam that has passed through the left and right regions of the objective lens does not deviate significantly on the imaging surface. Therefore, monocular stereoscopic vision itself becomes difficult.

  In the present embodiment, a pan-focus image having a deep depth of field can be obtained while using an objective lens having a focal blur that enables monocular stereoscopic vision. In order to display such a pan-focus image, the present embodiment includes a pan-focus image display unit 6102 that receives the output of the image composition unit 206. The pan focus image display unit 6102 is a display device for performing normal two-dimensional display. When the stereoscopic display unit 4503 has a two-dimensional display function, the stereoscopic display unit 4503 displays the pan focus display 6102. Can be combined.

  FIG. 62 is a diagram illustrating a configuration example of the endoscope distal end portion in the present embodiment. FIG. 62 (A) is a perspective view of the distal end portion of the endoscope, and describes a forceps 4604 protruding from the forceps opening 4603 during the operation. FIG. 62 (B) is an enlarged view of the front end portion of the endoscope as seen from the front, and shows the illumination lens 107, the opening 4602, and the forceps port 4603. An objective lens may be disposed in the opening 4602. FIG. 62C is an enlarged view of the opening 4602. The opening 4602 is provided with 2 × 2 = 4 regions. This 2 × 2 region basically has the same configuration as that shown in FIG. 46, but the upper left region UL and the upper right region UR are formed of a light shielding portion, and these regions UL, UR Are different in that a minute opening (transparent part) separated by a parallax distance DIS is provided in the vicinity of the center of each. The lower left region DL and the lower right region DR are a P (0 °) polarized light transmission region and an S (90 °) polarized light transmission region, respectively. The polarized light transmission region can be formed from, for example, a wire grid.

  According to the above configuration, among the light beams that have passed through the four regions, the light beams are narrowed by the minute openings in the UL and UR regions, so that the pan focus image (UL image, UR image) of the subject observed from each position, That is, a 2D image having a deep depth of field is obtained, and if these two images are observed with the left and right eyes, 3D stereoscopic vision is also possible.

  FIG. 63 shows image processing of the pixel selection re-integration unit 210. The image of the light rays that have passed through the regions UL, UR, DL, and DR are separated and generated by the operation of the pixel selective re-integration unit 210 in FIG. As a result, left and right parallax images are obtained from the light rays that have passed through the fine openings in the regions UL and UR. A P-polarized image and an S-polarized image are obtained from the light rays that have passed through the regions DL and DR.

  In the pan focus imaging mode, either the UL image or the UR image shown in FIG. 63 is selected and a 2D image is displayed. In addition, although the light shielding part and the fine opening part may be provided in the regions DL and DR, in that case, the light quantity reduction by the light shielding part and the light quantity reduction by the polarizing filter overlap.

In the present embodiment, illumination is performed in a frame sequential manner, but an illumination unit (FIG. 51) provided in the sixth embodiment may be employed. In that case, the correspondence relationship between the 2 × 2 region on the imaging surface and the color mosaic may be changed to the configuration shown in FIG. The imaging mode in this embodiment, the illumination light in that case, and the obtained image can be summarized as follows.

(Modification of the ninth embodiment)
FIG. 64 shows the configuration of the opening 4603 in a modification of the ninth embodiment. In this modification, there is no change other than the configuration of the opening 4603 in the configuration of the ninth embodiment. In the example of FIG. 64A, the light shielding portion and the minute opening portion are provided only in the region UR among the four regions obtained by dividing the lens opening portion. The region UL is a transparent portion, and a polarizing filter having a transmission axis of 0 ° is provided in the regions DL and DR. In this example, the illumination light to be irradiated is switched between (1) non-polarized light in the normal imaging mode, (2) 0 ° linearly polarized light in the polarized imaging mode, and (3) 90 ° linearly polarized light in the polarized imaging mode. Different images are obtained from light rays that have passed through each region as follows. However, both P / S cannot be obtained for parallel and orthogonal Nicol images. Further, it is desirable that the stereoscopic imaging is executed using the R image and the L image of the parallel Nicol image obtained in the polarization imaging mode of (2). In this example, the following image is obtained.

In the example of FIG. 64B, the light shielding portion and the opening are provided only in the region R of the two regions obtained by dividing the lens opening. In the region L, a polarizing filter having a transmission axis of 0 ° is arranged. In this case, the illumination light is switched between (1) non-polarized light in the normal imaging mode, (2) 0 ° linearly polarized light in the polarized imaging mode, and (3) 90 ° linearly polarized light in the polarized imaging mode as follows. Different images are obtained from the light rays transmitted through each region. The stereoscopic imaging can be realized using the pan-focus image in the region R and the polarization filter transmission image in the region L. In this example, the following image is obtained.

(Tenth embodiment)
FIG. 65 shows a microlens array type monochrome broadband polarization imaging unit 6101 according to the tenth embodiment. The basic configuration of the present embodiment is the same as the configuration of the ninth embodiment described with reference to FIG. The difference is that the micro lens 3703 does not correspond to the 2 rows and 2 columns of pixels on the imaging surface 3704 in the 2 rows and 2 columns area where the objective lens 3502 is located on the top, bottom, left and right, but instead of the 3 rows and 3 columns of the objective lens 3502. The area 6501 to 6503 corresponds to the 3 × 3 pixels 6504 on the imaging surface 3704. In FIG. 65, only three regions 6501 to 6503 are shown for simplicity.

  66 (A) to 66 (C) are views showing the endoscope distal end portion of the present embodiment. As shown in FIG. 66C, the objective lens 4602 is divided into nine regions a to i arranged in 3 rows and 3 columns. Regions a and b are transparent regions and form a parallax with a center-of-gravity distance DIS. Region e is located at the center of the lens. The region g and the region i are polarization filter regions having different polarization transmission axes by 90 °, and form a parallax with a center-of-gravity distance DIS.

  FIG. 67 shows image processing of the pixel selection / re-integration unit 210. The microlens 3703 and the pixel selection / re-integration unit 210 function to separate and generate images of light rays that have passed through the nine regions a to i. Left and right parallax images are obtained from the areas a and c, and P-polarized and S-polarized captured images are obtained from the areas g and i.

  In the pan focus imaging mode, an image generated by a light beam that has passed through only the center of the lens in the region e is selected and a 2D image is displayed.

  In this embodiment, since the lens aperture area is divided into 3 × 3 = 9 areas on the imaging surface, the individual image resolution is reduced, but the pan-focus image that is transmitted through the center of the lens, that is, in a sufficiently narrowed state. An image in focus on all images can be obtained.

(Eleventh embodiment)
FIG. 68 is a block diagram illustrating an eleventh embodiment of the present disclosure. In this embodiment, the microlens array type color polarization image pickup device and the compound eye lens type color image pickup device are not mounted, but a normal color image pickup device is provided, thereby realizing stereoscopic viewing. In the present embodiment, as in the second embodiment, the subject is irradiated with white light and color imaging is performed with a single-plate color imaging device (simultaneous method). The present embodiment includes a polarizing filter 6802 provided in the lens opening and a color polarization imaging element 6801.

  As shown in FIG. 69, at the distal end of the endoscope, a large number of irradiation ports for emitting illumination lights having different polarization planes of 0 ° (P) and 90 ° (S) are alternately arranged. By alternately lighting two sets of eight LEDs of the same type that are not adjacent to each other, lighting of P-polarized light and S-polarized light is realized alternately, and color imaging is performed in synchronization with the lighting.

  The opening of the objective lens 3502 is divided into right and left, and a 0 ° polarizing filter is provided in the L region and a 90 ° polarizing filter is provided in the R region. These polarizing filters may be formed of a polymer film, a wire grid, or the like.

  If linearly polarized light is irradiated onto a very smooth surface such as a mucous membrane, the linearly polarized light returns almost as it is. FIG. 70 illustrates this situation. When the lens surface on the light receiving side is divided into two regions, an image by a light beam that has passed through only the region L on the left side of the lens when the linearly polarized light is 0 °, and a region on the right side of the lens when the linearly polarized light is 90 ° Images by light rays that have passed through only R are alternately obtained in a time-sharing manner. Actually, since the non-polarized light component included in the reflected light from the subject passes through the opposite region, half of the field of view does not become dark. Therefore, a stereoscopic image can be displayed by displaying the L image and the R image obtained alternately in time division.

  In the present embodiment, since polarization imaging is used only for stereoscopic vision, it is not possible to perform image processing on a crossed Nicols image or the like. It is also effective when the subject to be imaged has a smooth surface that does not break the polarization. However, the configuration of the present embodiment can be applied to a device having a small or simple structure such as a capsule endoscope. According to the configuration of the present embodiment, even when the amount of illumination light is small, the image becomes bright, so that effective monocular stereoscopic vision can be realized even with an endoscope that is likely to generate defocus.

  Embodiments of the present disclosure include medical endoscopes for gastrointestinal internal medicine, cameras for medical use such as dermatology, dentistry, ophthalmology, and surgery, industrial endoscopes, fingerprint imaging devices, and surface inspection devices in factories, etc. The present invention is widely applicable to the field of image processing that requires observation, inspection, and recognition of surface irregularities of a subject. According to the embodiment of the present disclosure, it is possible to correctly detect irregularities on the surface of a smooth transparent object or a semi-transparent object, and it is possible to perform highlighting in a form that is easy for humans to distinguish. For this reason, the embodiment of the present disclosure is optimal for inspection of unevenness that is difficult in luminance observation.

  In addition, the image processing apparatus of the present disclosure can be applied to a digital camera, a video camera, a surveillance camera, and the like, and can be widely used for improving contrast on a water surface or aerial photography, photographing through glass, and the like.

DESCRIPTION OF SYMBOLS 101 Endoscope 102 Control apparatus 103 Insertion part 104 Light source device 105 Light guide 106 Front-end | tip part 107 Illumination lens 108 Video signal line 109 Objective lens 110 Image processor 111 Visceral mucosal surface (with unevenness)
112 Synchronizer 113 Reflected Light 114 Display Unit 115 Monochrome Broadband Polarized Image Sensor 116 Color Wheel 117 Polarized or Unpolarized Illumination Light 118 Light Source 119 Color Polarized Image Sensor 120 Illumination Rotation Control Unit 200 Illumination Filter 202 Polarization Mosaic Processing Unit 204 Concave Area Detection Unit 206 image composition unit 208 color mosaic interpolation unit 210 pixel selective re-integration unit 220 microlens 222 first wire grid layer 224 second wire grid layer 226 planarization layer 228 planarization layer 230 wiring 232 PD (photodiode)
240 color filter

Claims (19)

In the polarization imaging mode, the first illumination light that is polarized in the first direction and the second illumination light that is polarized in the second direction intersecting the first direction, sequentially, In the non-polarization imaging mode, the illumination unit irradiates the subject with non-polarized illumination light, and the wavelength range of the first illumination light does not overlap the wavelength range of the second illumination light. An illumination unit that sequentially emits the first and second illumination lights to have a portion;
An aperture region provided with a polarizer and a transparent region;
An image pickup device having a light detection element array that receives a light transmitted through a polarizer or a transparent region in the aperture region and outputs a signal, each having a plurality of microlenses covering a plurality of light detection devices; ,
The polarized imaging mode is configured by a signal of light transmitted through a polarizer having the polarization transmission axis in a direction crossing the first direction when the subject is irradiated with the first illumination light. A first polarization image and / or a signal of light transmitted through a polarizer having the polarization transmission axis in a direction intersecting the second direction when the subject is illuminated with the second illumination light. A second polarized image obtained is obtained, and in the non-polarized imaging mode, it is constituted by a light signal transmitted through a polarizer or a transparent region when the subject is irradiated with the non-polarized illumination light. A pixel selective re-integration unit that obtains a non-polarized image and obtains a plurality of parallax images composed of light signals transmitted through a polarizer or a transparent region in the stereoscopic imaging mode;
A concave area detection unit that detects a concave area on the surface of the subject based on at least one of the first and second polarization images;
An image composition unit that forms an image that highlights and shows the concave region on the surface of the subject;
A parallax image synthesizing unit that alternately reproduces the parallax images in time, and
An image processing apparatus comprising: The opening region is provided with a plurality of polarizers and a plurality of transparent regions having different directions of the polarization transmission axis,
In the polarization imaging mode, the pixel selective re-integration unit includes a polarizer having the polarization transmission axis in a direction intersecting the first direction when the subject is irradiated with the first illumination light. A polarizer having a first polarization image composed of a signal of transmitted light and the polarization transmission axis in a direction intersecting the second direction when the subject is illuminated with the second illumination light The image processing apparatus according to claim 1, wherein a second polarization image configured by a signal of light transmitted through the image is obtained. Each of the plurality of transparent regions of the opening region is surrounded by a light shielding region to form a minute opening,
The image processing apparatus according to claim 2, wherein the pixel selective re-integration unit obtains a pan-focus image composed of a signal of light transmitted through each minute opening. The transparent area of the opening area is surrounded by a light shielding area to form one minute opening,
The image processing apparatus according to claim 1, wherein the pixel selective re-integration unit obtains a pan-focus image configured by a signal of light transmitted through the minute opening. A part of the transparent region of the opening region extends so as to include the center of the opening region, and each of the plurality of microlenses covers a plurality of light detection element arrays arranged in three rows and three columns. Are arranged so that
The image processing apparatus according to claim 1, wherein the light transmitted through the center of the opening region is incident on a light detection element array positioned at the center of the plurality of light detection element arrays arranged in the 3 rows and 3 columns. In the polarization imaging mode, the first white illumination light in a state polarized in a first direction and the second white illumination light in a state polarized in a second direction intersecting the first direction, Sequentially illuminating the subject, and in the non-polarization imaging mode, an illumination unit that irradiates the subject with non-polarized white illumination light; and
An aperture region provided with a plurality of polarizers and transparent regions having different directions of polarization transmission axes;
Light that has a color mosaic filter in which color filters having different light transmission characteristics are arranged, passes through each polarizer or each transparent region of the aperture region, and then receives light transmitted through each color filter and outputs a signal An imaging device having a sensing element array, each having a plurality of microlenses covering a plurality of light sensing elements;
In the polarization imaging mode, it is configured by a signal of light transmitted through a polarizer having the polarization transmission axis in a direction intersecting the first direction when the subject is irradiated with the first white illumination light. And a signal of light transmitted through a polarizer having the polarization transmission axis in a direction crossing the second direction when the subject is illuminated with the second white illumination light. And forming a second polarized image based on signals from a part of the light detection elements selected from the plurality of light detection elements covered by each microlens, in the non-polarization imaging mode , To obtain a non-polarized image composed of a signal of light transmitted through each polarizer when the subject is illuminated with the non-polarized white illumination light, and in the stereoscopic imaging mode, the light transmitted through each transparent region signal Thus a pixel selection re stacking unit to obtain a plurality of parallax images constituted,
A concave area detection unit that detects a concave area on the surface of the subject based on at least one of the first and second polarization images;
An image composition unit that forms an image that highlights and shows the concave region on the surface of the subject;
A parallax image synthesizing unit that alternately reproduces the parallax images in time, and
An image processing apparatus comprising: In the polarization imaging mode, the first illumination light that is polarized in the first direction and the second illumination light that is polarized in the second direction intersecting the first direction, sequentially, In the non-polarization imaging mode, the illumination unit irradiates the subject with non-polarized illumination light, and the wavelength range of the first illumination light does not overlap the wavelength range of the second illumination light. An illumination unit that sequentially emits the first and second illumination lights to have a portion;
A compound eye lens unit provided with a plurality of polarizers and a plurality of transparent regions having different directions of polarization transmission axes, each having a plurality of lenses each provided with a polarizer or a transparent region; and
An image pickup device having a light detection element array that receives light transmitted through each polarizer or each transparent region of the compound eye lens unit and outputs a signal;
The polarized imaging mode is configured by a signal of light transmitted through a polarizer having the polarization transmission axis in a direction crossing the first direction when the subject is irradiated with the first illumination light. And a signal of light transmitted through a polarizer having the polarization transmission axis in a direction crossing the second direction when the subject is illuminated with the second illumination light. A non-polarized image composed of a signal of light transmitted through each polarizer when the subject is illuminated with the non-polarized illumination light in the non-polarized imaging mode. Obtaining a plurality of parallax images composed of light signals transmitted through the transparent regions in the stereoscopic imaging mode;
A concave area detection unit that detects a concave area on the surface of the subject based on at least one of the first and second polarization images;
An image composition unit that forms an image that highlights and shows the concave region on the surface of the subject;
A parallax image synthesizing unit that alternately reproduces the parallax images in time, and
An image processing apparatus comprising: In the polarization imaging mode, the first white illumination light in a state polarized in a first direction and the second white illumination light in a state polarized in a second direction intersecting the first direction, Sequentially illuminating the subject, and in the non-polarization imaging mode, an illumination unit that irradiates the subject with non-polarized white illumination light; and
A compound eye lens unit provided with a plurality of polarizers and a plurality of transparent regions having different directions of polarization transmission axes, each having a plurality of lenses each provided with a polarizer or a transparent region; and
It has a color mosaic filter in which color filters having different light transmission characteristics are arranged, and after passing through each polarizer or each transparent region of the compound eye lens unit, further receives light transmitted through each color filter and outputs a signal. Image sensor light having a light sensing element array;
The polarized imaging mode is configured by a signal of light transmitted through a polarizer having the polarization transmission axis in a direction crossing the first direction when the subject is irradiated with the first illumination light. And a signal of light transmitted through a polarizer having the polarization transmission axis in a direction crossing the second direction when the subject is illuminated with the second illumination light. A non-polarized image composed of a signal of light transmitted through each polarizer when the subject is illuminated with the non-polarized illumination light in the non-polarized imaging mode. Obtaining a plurality of parallax images composed of light signals transmitted through the transparent regions in the stereoscopic imaging mode;
A concave area detection unit that detects a concave area on the surface of the subject based on at least one of the first and second polarization images;
An image composition unit that forms an image that highlights and shows the concave region on the surface of the subject;
A parallax image synthesizing unit that alternately reproduces the parallax images in time, and
An image processing apparatus comprising: In the polarization imaging mode, the first white illumination light in a state polarized in a first direction and the second white illumination light in a state polarized in a second direction intersecting the first direction, Sequentially illuminating the subject, and in the non-polarization imaging mode, an illumination unit that irradiates the subject with non-polarized white illumination light; and
An aperture region provided with a plurality of polarizers having different directions of polarization transmission axes;
A photo-detecting element array having a color mosaic filter in which color filters having different light transmission characteristics are arranged, and further transmitting light after passing through each polarizer in the aperture region and receiving light transmitted through each color filter. An imaging device having
A pixel selective re-integration unit that obtains a plurality of time-division parallax images configured by light signals transmitted through each polarizer;
A parallax image synthesizing unit that alternately reproduces the parallax images in time, and
An image processing apparatus comprising: The wavelength range of the first illumination light is included in at least one wavelength range of B (blue) and G (green),
The image processing apparatus according to claim 1, wherein a wavelength range of the second illumination light is included in at least one of a wavelength range of B (blue) and G (green). The wavelength range of the first illumination light is included in a part of the B (blue) wavelength range and a part of the G (green) wavelength range,
The image processing apparatus according to claim 1, wherein the wavelength range of the second illumination light is included in another part of the B (blue) wavelength range and another part of the G (green) wavelength range. The wavelength range of the first illumination light is included in a part of the wavelength range of B (blue), a part of the wavelength range of G (green), and a part of the wavelength range of R (red),
The wavelength range of the second illumination light includes another part of the B (blue) wavelength range, another part of the G (green) wavelength range, and another part of the R (red) wavelength range. And an image processing apparatus according to claim 1.   The illumination unit includes, as the first illumination light, light included in a part of a B (blue) wavelength range, light included in a part of a G (green) wavelength range, and a wavelength of R (red). The light included in a part of the region is emitted at different timings, and as the second illumination light, the light included in the other part of the B (blue) wavelength region, the wavelength of G (green) The image processing apparatus according to claim 8, wherein light included in another part of the region and light included in the other part of the R (red) wavelength region are emitted at different timings.   The illuminating unit emits the first illumination light and the second illumination light alternately, and sequentially includes light included in each wavelength region of R (red), G (green), and B (blue). The image processing apparatus according to claim 13, which emits light. An endoscope used in the image processing apparatus according to any one of claims 1 to 5,
In the polarization imaging mode, the first illumination light that is polarized in the first direction and the second illumination light that is polarized in the second direction intersecting the first direction, sequentially, In the non-polarization imaging mode, the illumination unit irradiates the subject with non-polarized illumination light, and the wavelength range of the first illumination light does not overlap the wavelength range of the second illumination light. An illumination unit that sequentially emits the first and second illumination lights to have a portion;
An aperture region provided with a polarizer and a transparent region;
An image pickup device having a light detection element array that receives a light transmitted through a polarizer or a transparent region in the aperture region and outputs a signal, each having a plurality of microlenses covering a plurality of light detection devices; ,
The polarized imaging mode is configured by a signal of light transmitted through a polarizer having the polarization transmission axis in a direction crossing the first direction when the subject is irradiated with the first illumination light. A first polarization image and / or a signal of light transmitted through a polarizer having the polarization transmission axis in a direction intersecting the second direction when the subject is illuminated with the second illumination light. A second polarized image obtained is obtained, and in the non-polarized imaging mode, it is constituted by a light signal transmitted through a polarizer or a transparent region when the subject is irradiated with the non-polarized illumination light. A pixel selective re-integration unit that obtains a non-polarized image and obtains a plurality of parallax images composed of light signals transmitted through a polarizer or a transparent region in the stereoscopic imaging mode;
An endoscope comprising: An endoscope used in the image processing apparatus according to claim 6,
In the polarization imaging mode, the first white illumination light in a state polarized in a first direction and the second white illumination light in a state polarized in a second direction intersecting the first direction, Sequentially illuminating the subject, and in the non-polarization imaging mode, an illumination unit that irradiates the subject with non-polarized white illumination light; and
An aperture region provided with a plurality of polarizers and transparent regions having different directions of polarization transmission axes;
Light that has a color mosaic filter in which color filters having different light transmission characteristics are arranged, passes through each polarizer or each transparent region of the aperture region, and then receives light transmitted through each color filter and outputs a signal An imaging device having a sensing element array, each having a plurality of microlenses covering a plurality of light sensing elements;
In the polarization imaging mode, it is configured by a signal of light transmitted through a polarizer having the polarization transmission axis in a direction intersecting the first direction when the subject is irradiated with the first white illumination light. And a signal of light transmitted through a polarizer having the polarization transmission axis in a direction crossing the second direction when the subject is illuminated with the second white illumination light. And forming a second polarized image based on signals from a part of the light detection elements selected from the plurality of light detection elements covered by each microlens, in the non-polarization imaging mode , To obtain a non-polarized image composed of a signal of light transmitted through each polarizer when the subject is illuminated with the non-polarized white illumination light, and in the stereoscopic imaging mode, the light transmitted through each transparent region signal Thus a pixel selection re stacking unit to obtain a plurality of parallax images constituted,
An endoscope comprising: An endoscope used in the image processing apparatus according to claim 7,
In the polarization imaging mode, the first illumination light that is polarized in the first direction and the second illumination light that is polarized in the second direction intersecting the first direction, sequentially, In the non-polarization imaging mode, the illumination unit irradiates the subject with non-polarized illumination light, and the wavelength range of the first illumination light does not overlap the wavelength range of the second illumination light. An illumination unit that sequentially emits the first and second illumination lights to have a portion;
A compound eye lens unit provided with a plurality of polarizers and a plurality of transparent regions having different directions of polarization transmission axes, each having a plurality of lenses each provided with a polarizer or a transparent region; and
An image pickup device having a light detection element array that receives light transmitted through each polarizer or each transparent region of the compound eye lens unit and outputs a signal;
The polarized imaging mode is configured by a signal of light transmitted through a polarizer having the polarization transmission axis in a direction crossing the first direction when the subject is irradiated with the first illumination light. And a signal of light transmitted through a polarizer having the polarization transmission axis in a direction crossing the second direction when the subject is illuminated with the second illumination light. A non-polarized image composed of a signal of light transmitted through each polarizer when the subject is illuminated with the non-polarized illumination light in the non-polarized imaging mode. Obtaining a plurality of parallax images composed of light signals transmitted through the transparent regions in the stereoscopic imaging mode;
An endoscope comprising: An endoscope used in the image processing apparatus according to claim 8,
In the polarization imaging mode, the first white illumination light in a state polarized in a first direction and the second white illumination light in a state polarized in a second direction intersecting the first direction, Sequentially illuminating the subject, and in the non-polarization imaging mode, an illumination unit that irradiates the subject with non-polarized white illumination light; and
A compound eye lens unit provided with a plurality of polarizers and a plurality of transparent regions having different directions of polarization transmission axes, each having a plurality of lenses each provided with a polarizer or a transparent region; and
It has a color mosaic filter in which color filters having different light transmission characteristics are arranged, and after passing through each polarizer or each transparent region of the compound eye lens unit, further receives light transmitted through each color filter and outputs a signal. Image sensor light having a light sensing element array;
The polarized imaging mode is configured by a signal of light transmitted through a polarizer having the polarization transmission axis in a direction crossing the first direction when the subject is irradiated with the first illumination light. And a signal of light transmitted through a polarizer having the polarization transmission axis in a direction crossing the second direction when the subject is illuminated with the second illumination light. A non-polarized image composed of a signal of light transmitted through each polarizer when the subject is illuminated with the non-polarized illumination light in the non-polarized imaging mode. Obtaining a plurality of parallax images composed of light signals transmitted through the transparent regions in the stereoscopic imaging mode;
An endoscope comprising: An endoscope used in the image processing device according to claim 9,
In the polarization imaging mode, the first white illumination light in a state polarized in a first direction and the second white illumination light in a state polarized in a second direction intersecting the first direction, Sequentially illuminating the subject, and in the non-polarization imaging mode, an illumination unit that irradiates the subject with non-polarized white illumination light; and
An aperture region provided with a plurality of polarizers having different directions of polarization transmission axes;
A photo-detecting element array having a color mosaic filter in which color filters having different light transmission characteristics are arranged, and further transmitting light after passing through each polarizer in the aperture region and receiving light transmitted through each color filter. An imaging device having
A pixel selective re-integration unit that obtains a plurality of time-division parallax images configured by light signals transmitted through each polarizer;
An endoscope comprising:

Images