WO2017090229A1

WO2017090229A1 - Image processing apparatus, image processing method, image capturing apparatus, polarization control unit, image capturing system, and program

Info

Publication number: WO2017090229A1
Application number: PCT/JP2016/004686
Authority: WO
Inventors: Yoshinori Kimura
Original assignee: Canon Kabushiki Kaisha
Priority date: 2015-11-26
Filing date: 2016-10-25
Publication date: 2017-06-01
Also published as: JP2017098831A

Abstract

A polarization control unit for use with an image capturing apparatus includes a first polarizer, an adjuster configured to adjust a polarization direction of a light flux that has passed the first polarizer, and a second polarizer which the light flux from the adjuster is to pass. At least one of the first polarizer and the second polarizer include a plurality of areas which light fluxes having mutually different polarization directions can pass. The light fluxes that have passed the plurality of areas enter the same pixel in an image sensor in the image capturing apparatus.

Description

IMAGE PROCESSING APPARATUS, IMAGE PROCESSING METHOD, IMAGE CAPTURING APPARATUS, POLARIZATION CONTROL UNIT, IMAGE CAPTURING SYSTEM, AND PROGRAM

The present invention relates to an image processing apparatus, an image processing method, an image capturing apparatus, a polarization control unit, an image capturing system, and a program.

An image processing technology for generating one high resolution image from one or more low resolution images is referred to as a super resolution technology. A pixel shift super resolution is one type of the super resolution technology and generates a high resolution image by micro moving a low resolution image sensor and by performing interpolating processing for a plurality of low resolution images obtained at locations different from one another. The interpolation processing is processing for smoothly connecting rough sampling points and for estimating values among the sampling points.

PLT1 generates a high resolution image through the pixel shift super resolution using a parallel crystal plate configured to divide incident light into normal light and abnormal light, a liquid crystal polarization filter that is controllable so as to alternately transmit the normal light and the abnormal light, and time-divided normal and abnormal optical images. PLT1 acquire a plurality of mutually shifted low resolution images (normal and abnormal optical images) using the birefringence of the crystal without mechanical driving.

Another type of super resolution technology is a method for using an iteration calculation represented by a MAP (Maximum a Posteriori) estimation. The MAP estimation performs the iteration calculation with one or more low resolution images, and generates a high resolution image. More specifically, the MAP initially defines an objective function that is a sum of a likelihood term representative of the likelihood of the estimation and a prior term representative of prior knowledge concerning the estimation, such as the smoothness. Next, the MAP maximizes or minimizes a value of the objective function through the iteration calculation, and generates a high resolution image that provides a maximum posterior probability or the highest probability under a given condition.

[PLT1] Japanese Patent Laid-open No. 6-275804

Since the pixel shift super resolution needs micro driving of the low resolution image sensor and uses the simple interpolation processing, the super resolution accuracy is low. The image processing disclosed in PTL1 is also the simple interpolation processing, and thus the super resolution accuracy is low. The iteration calculation represented by the MAP estimation needs a complex calculation, causing a heavy calculation burden for the image processing.

The present invention provides an image processing apparatus, an image processing method, an image capturing apparatus, a polarization control unit, an image capturing system, and a program, which can reduce an image processing burden and highly precisely generate a high resolution image.

A polarization control unit according to the present invention for use with an image capturing apparatus includes a first polarizer, an adjuster configured to adjust a polarization direction of a light flux that has passed the first polarizer, and a second polarizer which the light flux from the adjuster is to pass. At least one of the first polarizer and the second polarizer include a plurality of areas which light fluxes having mutually different polarization directions can pass. The light fluxes that have passed the plurality of areas enter the same pixel in an image sensor in the image capturing apparatus.

Further features and aspects of the present invention will become apparent from the following description of exemplary examples with reference to the attached drawings.

The present invention can provide an image processing apparatus, an image processing method, an image capturing apparatus, a polarization control unit, an image capturing system, and a program, which can reduce an image processing burden and highly precisely generate a high resolution image.

FIG. 1 is a block diagram illustrating an illustrative configuration of an image capturing apparatus according to the present invention. FIG. 2A is a view for explaining how the image capturing apparatus illustrated in FIG. 1 forms an arbitrary transmittance distribution. FIG. 2B is a view for explaining how the image capturing apparatus illustrated in FIG. 1 forms an arbitrary transmittance distribution. FIG. 3 is a flowchart for explaining a super resolution method executed by the image capturing apparatus according to the present invention. FIG. 4 is a block diagram of an infrared camera to which the image capturing apparatus according to a first embodiment of the present invention is applied. FIG. 5 is a block diagram of an endoscope to which the image capturing apparatus according to a second embodiment of the present invention is applied. FIG. 6 is a block diagram of a microscope to which the image capturing apparatus according to a third embodiment of the present invention is applied. FIG. 7 is a view illustrating a transmittance distribution according to the present invention. FIG. 8A is a view illustrating a numerical calculation result (high resolution image by the pixel shift super resolution) as an effect of the present invention. FIG. 8B is a view illustrating a numerical calculation result (high resolution image by the pixel shift super resolution) as an effect of the present invention. FIG. 8C is a view illustrating a numerical calculation result (true image) as an effect of the present invention.

Description of Example

Exemplary examples of the present invention will be described below with reference to the accompanying drawings.

FIG. 1 is a block diagram of an illustrative configuration of an image capturing apparatus 100 according to this embodiment. The image capturing apparatus 100 includes an imaging optical system 101, a first polarization plate 102, a polarization control element 103, a second polarization plate 104, an image sensor 105, an image processor 150, a driver 155, a controller 160, a storage unit 162, and a display unit 164.

The image capturing apparatus may be a lens integrated type or an image capturing apparatus body which a lens apparatus is attached to and detached from. In the latter case, the present invention is comprised by an image capturing system that includes the lens apparatus and the image capturing apparatus body. The lens apparatus may include the image capturing optical system 101, the first polarization plate 102, the polarization control element 103, the second polarization plate 104, and the driver 155, and the image capturing apparatus body may include the image sensor 105, the image processor 150, the controller 160, the storage unit 162, and the display unit 164. The lens apparatus includes a lens controller configured to control each component in the lens apparatus, and communicates with the controller 160 that serves as the camera controller. The camera controller controls the driver 155 via the lens controller. Alternatively, the lens apparatus may include an imaging optical system 101, and the image capturing apparatus body may include the components other than the imaging optical system 101.

The first polarization plate 102, the polarization control element 103, and the second polarization plate 104 may be integrated into one polarization control unit and attached to and detached from the optical axis of the image capturing apparatus 100. As a result, when these components are retreated from the optical axis of the imaging optical system 101, the image capturing apparatus generates a normal low resolution image. When these units are inserted into the optical axis of the imaging optical system 101, the image capturing apparatus generates a normal high resolution image. The polarization control unit used for the image capturing apparatus 100 may be housed in the image capturing apparatus or the lens apparatus, or may be attached to and detached from the image capturing apparatus 100.

The imaging optical system 101 includes, for example, one or more lenses (lens unit), and forms an optical image of an object via a polarization control unit, on an image capturing plane (pixels) of the image sensor 105. The imaging optical system 101 may include a focus lens used for focusing, a zoom lens (magnification-varying lens) configured to change a focal length, an image stabilization lens used to shift the optical axis, an aperture stop configured to adjust a light quantity, etc.

The first polarization plate (first polarizer) 102 is provided on the image plane of the imaging optical system 101. Alternatively, the first polarization plate 102, the polarization control element 103, and the second polarization plate 104 may be provided in the imaging optical system 101. The polarization plate is an optical element which only specific polarized light (linearly polarized light that is polarized in a specific direction) in incident light passes, and is equivalent with the polarizer in this embodiment. A polarized light flux having an orientation parallel to the transmission axis or reflection axis of the polarizer passes the polarization plate. The polarizer may be a transmission type or a reflection type.

The first polarization plate 102 may include a wire grid, a diffractive grating, and a prism. The wire grid is made by arranging metal thin wires in a predetermined azimuth, and exhibits a polarization transmission or reflection characteristic depending on the azimuth. The diffractive grating has a polarization azimuth different for each area. The fine polarization plates having polarization transmission characteristics different from one another between at least two areas can be integrated by applying the lithography technology so as to produce metal thin wires.

On at least one of the first and

second polarization plates

102 and 104, the area corresponding to one pixel in the image sensor 105 in the image capturing apparatus 100 is divided into a plurality of areas (or sub areas), which the polarized light fluxes having different polarization directions can pass. In FIG. 1, the polarization plate 102 has this function, and thus has a plurality of areas. At least two of the plurality of areas have polarization characteristics that are different from one another, and all of the plurality of areas may have polarization characteristics that are different from one another. The polarization characteristic means a polarization direction of a light flux to pass or a direction of the transmission axis or the reflection axis which the polarized light flux is to pass (transmit through or reflect on). The light fluxes to pass enters the same pixel in the image sensor 105.

The polarization control element (adjuster) 103 includes, for example, a liquid crystal element, and is disposed behind the first polarization plate 102. The polarization control element 103 serves as the adjuster configured to adjust the orientation of the liquid crystal molecule through an electric field generated by applying the external power (voltage or current) from the outside. FIG. 1 illustrates a transmission type polarization control element 103, but a reflection type polarization control element may be used. The adjuster is not limited to the liquid crystal element as long as it can adjust the polarization direction. FIG. 1 omits an Indium-Tin Oxide (ITO) electrode configured to apply the voltage to the polarization control element 103, an orientation layer for the liquid crystal molecule, and the like.

The second polarization plate 104 (second polarizer) is made, for example, by orienting and absorbing iodine compound molecules onto a polyvinyl alcohol (PVA) film and located behind the polarization control element 103. The light that has passed the adjuster passes the second polarization plate 104. More specifically, predetermined light fluxes pass the second polarization plate 104 among light fluxes whose polarization directions have been adjusted by the adjuster. The second polarization plate 104 is a polarizer (or analyzer) that has a uniform polarization characteristic. The polarized light flux having a polarization direction in one direction passes the entire surface of the second polarization plate 104. In general, an apparatus that combines a polarizer and an analyzer with each other is referred to as a polarization unit. If the polarization characteristic is uniform, the realized transmittance distribution can be easily supposed by a numerical calculation. Even if the polarization characteristic is not uniform, a relationship between the control voltage and the transmittance distribution may be estimated through a calibration before the apparatus is used or numerical calculation. It is now assumed for simple description purposes that the polarization transmittance characteristic of the second polarization plate 104 is perfectly uniform over all areas. The iodine compound molecule has an elongated shape, and exhibits a polarization transmittance characteristic depending on the molecular orientation. The second polarization plate 104 is not limited to a component that uses the iodine compound molecule.

The image sensor 105 includes a photoelectric conversion element, such as a CMOS image sensor and a CCD image sensor, configured to receive and photoelectrically convert light from the second polarization plate 104 (optical image of the object formed by the imaging optical system 101) and to output an electric signal according to a light intensity. An A/D converter configured to convert an analog electric signal output from the image sensor 105 into a digital electric signal, and electric wires are omitted. The image sensor 105 has a plurality of pixels.

The order of the first polarization plate 102 and the second polarization plate 104 with respect to the polarization control element 103 may be reversed. In this case, the imaging optical system 101, the second polarization plate 104, the polarization control element 103, the first polarization plate 102 and the image sensor 105 may be arranged in this order.

The longitudinal and lateral dividing numbers may be different in the first polarization plate 102. For example, the first polarization plate 102 may be divided into m areas in the longitudinal direction and n areas in the lateral direction (m ≠ n). Each of the first polarization plate 102 and the second polarization plate 104 may be divided into a plurality of areas and have mutually different polarization characteristics.

Instead of the first polarization plate 102 having a plurality of areas, the area of the polarization control element 103 corresponding to one pixel in the image sensor 105 may be divided into a plurality of areas (sub areas). In this case, the polarized light fluxes having mutually different polarization directions can pass two or all of the plurality of areas, and the polarization directions may be independently adjusted. The first polarization plate 102 and the adjuster may be integrated together and the liquid crystal element may further serve as the first polarization plate 102. This function may be provided to the second polarization plate 104, and thus the polarization control unit for use with the image capturing apparatus is as follows. The polarization control unit includes a first polarizer which specific polarized light in incident light can pass, an adjuster configured to adjust a polarization direction of the specific polarized light, and a second polarizer which a predetermined polarized light flux in the specific polarized light whose polarization direction has been adjusted by the adjuster can pass. The adjuster includes a polarization control element in which an area corresponding to one pixel in the image sensor is divided into a plurality of areas, which the polarized light fluxes having mutually different polarization directions can pass. The adjuster further includes a direction adjuster configured to independently adjust the polarization directions for the plurality of areas.

The present invention premises the monochromatic light, and thus this embodiment does not include a color filter. However, the color filter may be included. In addition, the present invention premises the non-polarized light.

The image processor 150 performs signal processing, such as the gamma process and white balance, and a super resolution process, which will be described later, for the signal obtained from the image sensor 105. The controller 160 controls each component in the image capturing apparatus 100, executes the following control method, and includes a microcomputer. The storage unit 162 stores a program for the following control method, data used for the program, and images processed by the image processor 150 (including the super resolution result), and includes a variety of memories and detachable media. The storage unit 162 may be part of a computer (server) on a network, such as the Internet. The display unit 164 may be a liquid crystal display configured to display information stored in the storage unit 162 and various control information, and has a resolution corresponding to the high resolution image.

FIGs. 2A and 2B are schematic perspective views of how the image capturing apparatus 100 forms arbitrary transmittance distributions on the plurality of areas in one pixel in the image sensor.

The first polarization plate 202 includes 2×2 areas 202a to 202d, and each area has a different polarization transmittance characteristic. For example, when they are viewed from the object side, the upper left area 202a transmits the polarized light flux having a polarization direction of 135°. The upper right area 202c transmits the polarized light flux having a polarization direction of 45°. The lower left area 202b transmits the polarized light flux having a polarization direction of 90°. The lower right area 202d transmits the polarized light flux having a polarization direction of 0°. The second polarization plate 204 has such a uniform polarization transmittance characteristic that it transmits a polarized light flux having a polarization direction of 90°. The polarization control element 203 is made of a transmission type, which does not change the polarization direction of the polarized light flux that has transmitted the first polarization plate 202 in FIG. 2A, and slightly changes its polarization direction in the arrow direction in FIG. 2B. Reference numeral 205 denotes one pixel in the image sensor, and reference numerals 205a to 205d denote areas in one pixel.

The “transmittance distribution,” as used herein, means an attenuation rate distribution of incident light to the plurality of areas 205a to 205d in each pixel in the image sensor, to the incident light to the first polarization plate 202. Each pixel in the image sensor has a side, for example, of 10μm in length in its square shape. Although the imaging optical system 201 is conveniently drawn small, the actual imaging optical system 201 is much larger than other components (in the millimeter to centimeter order). The structures illustrated in FIGs. 2A and 2B are merely illustrative, and the present invention is not limited to this embodiment. For example, the first polarization plate 202 is not limited to 2×2 divisions and may be a 3×3 or 4×4 divisions.

In FIG. 2A, the polarized light flux in the polarization direction of 135° that has passed the upper left area 202a in the first polarization plate 202 transmits the polarization control element 203 and reaches the second polarization plate 204 while its polarization direction is maintained. Since the second polarization plate 204 has a characteristic that transmits the polarized light flux having the polarization direction of 90°, the polarized light flux having the polarization direction of 135° that has passed the upper left area 202a in the first polarization plate 202 reaches the image sensor 205 although its intensity attenuates by 50%. This is similarly applied to each area in the first polarization plate 202, and the transmittance distribution illustrated in the right table in FIG. 2A is finally produced.

In FIG. 2B, the polarized light flux having the polarization direction of 135° that has transmitted the upper left area 202a in the first polarization plate 202 slightly changes its polarization direction due to the polarization control element 203 and reaches the second polarization plate 204. Now assume that the polarization direction of the polarized light flux is 150° after the polarized light flux transmits the polarization control element 203. Then, the polarized light that has transmitted the upper left area 202a in the first polarization plate 202 attenuates its intensity by about 75%, and reaches the image sensor 205. This is similarly applied to each area in the first polarization plate 202, and the transmittance distribution illustrated in the right table in FIG. 2B is finally produced.

It is thus understood that the image capturing apparatus 100 can realize an arbitrary transmittance distribution by controlling the electric power (current or voltage) applied to the polarization control element 203 from the outside.

FIG. 3 is a flowchart illustrating a super resolution method (control method or image processing method) executed by the controller 160 in the image capturing apparatus 100, and “S” stands for the step. The flowchart illustrated in FIG. 3 is implemented as a program that enables a computer to execute each step. This program can be stored in the storage unit, such as a non-transitory computer readable storage medium, in the image capturing apparatus 100.

In S301, the controller 160 stores information of transmittance distributions Pi (i=1 to N) in the storage unit 162 provided by control driving of the polarization control element 103. A relationship between the voltage applied to the polarization control element 103 and the obtained transmittance distributions is previously measured or calculated by a simulation, and stored in the storage unit 162 before the image capturing apparatus is shipped from the factory.

In S302, the controller 160 stores information of the transmittance distributions Pi, information of the light intensity distributions Ii observed at each of all pixels in the image sensor 105, and their relationship in the storage unit 162.

In S303, the controller 160 repeats S301 and S302 a plurality of times (N times), and stores a plurality of mutually different transmittance distributions P1 to PN and a plurality of corresponding light intensity distributions I1 to IN in the storage unit 162. This embodiment sets the repetition number to the division number of the first polarization plate 202. This is to obtain the resolution multiple times as many as the division number of the first polarization plate 202, but the repetition number may be at least two.

In this embodiment, as illustrated in FIGs. 2A and 2B, when the first polarization plate 202 is divided into 2×2, the repetition number N becomes 2×2=4. Similarly, the repetition number N becomes 9 or 16 for the 3×3 or 4×4 divisions. More specifically, the repetition number N may be at least 4 in case of the 2×2 divisions. However, for simple description purposes, the following description assumes that the repetition number N is 4 in case of the 2×2 divisions. This is applied to the 3×3 or 4×4 divisions.

The transmittance distributions P1 to PN may be made of mutually different distributions, such as the transmittance distribution P1 being as illustrated in FIG. 2A and the transmittance distribution P2 being as illustrated in FIG. 2B, as long as they are orthogonal to one another. In other words, when the transmittance distributions P1 to PN are expressed by vectors, P1 is set orthogonal to each of P2 to P4 so that the inner product between them can be 0. Similarly, P2 is set orthogonal to each of P3 and P4, and P3 is set orthogonal to P4.

In S303, the controller 160 consecutively captures images the repetition times N, and changes the applied voltage through the driver 155 for each capture. Thus, the image capturing apparatus 100 has a consecutive capturing function, and the controller 160 controls the exposure for the image sensor 105. The image capturing apparatus 100 can operate both in a still image capturing mode and in a motion image capturing mode.

In S304, the controller 160 solves via the image processor 150 the inverse problem based on the information of the transmittance distributions and the light intensity distributions stored in the storage unit 162 for each pixel of the image sensor 105, and provides the super resolution. A solution will be given as follows.

Herein, Pi is a matrix that describes a plurality of transmittance distributions P1i to PNi in each row in an i-th pixel in the image sensor 105. Ii is a column vector that has, in respective rows, light intensities I1i to INi observed in the i-th pixel in the image sensor 105. “inv” is an operator for calculating an inverse matrix (pseudo inverse matrix). In this embodiment, a matrix and a vector are written by thick letters and a scalar is written by a thin letter. The matrix Pi has N rows and N columns. The column vector Ii has N rows and one column. A N×1 column vector yi obtained by solving the inverse problem is a super resolution pixel value in the i-th pixel in the image sensor 105. In other words, it is the vector that represents the light intensity distribution observed when the i-th pixel in the image sensor 105 is divided into √N×√N.

A description will be given of how the super resolution image value is obtained by changing the transmittance distribution in one pixel in the image sensor 105 and by capturing images a plurality of times. A light intensity I observed in one pixel in the image sensor 105, a corresponding transmittance distribution P, and a corresponding super resolution pixel value y have the following relationship.

The transmittance distribution P is a row vector, and the super resolution pixel value y is weighted by a linear combination coefficient determined by the transmittance distribution P. The light intensity I is the obtained linear sum. The super resolution pixel value y as the N rows one column vector cannot be solved by this equation alone with the light intensity I as a scalar. Accordingly, the super resolution pixel value is calculated through simultaneous equations after a plurality of observations are made with different conditions (transmittance distributions).

In S305, the controller 160 generates a high resolution image by integrating solution results of the inverse problems via the image processor 150. Information necessary for the integration is previously stored in the storage unit 162. In S306, the controller 160 stores the high resolution image in the storage unit 162, or displays it on the display unit 164. In S305, the controller 160 may store information acquired in S304 in the storage unit 162 and end the process without making the image processor 150 integrate the information. In displaying the high resolution image, the storage capacity for the storage unit 162 can be saved when the controller 160 makes the image processor 150 generate the information. Similarly, the controller 160 may end the process when the result is Yes in S303. In displaying the high resolution image, S304 to S306 may be executed.

It is thus understood that the apparatus configuration of the present invention can provide a super resolution. The present invention needs to control the polarization control element 103, but does not require mechanical driving which is necessary for the pixel shift method. In addition, the calculation load for solving the inverse problem in S304 is much lighter than the MAP estimation, and a complex calculation is unnecessary in the image processing.

PLT1 discloses an image capturing apparatus that provides the pixel shift super resolution without mechanical driving. Now, image qualities obtained by the image capturing apparatus according to the present invention and the pixel shift super resolution will be compared with each other by numerical calculations. The low resolution image has 64×64 pixels, one pixel of the low resolution image has a side of 10μm in length in a square shape, and the high resolution image has 128×128 pixels. Since one pixel for the low resolution image is divided into 2×2, the total pixel number of the high resolution image is four times as many as that of the low resolution image. Since one pixel size becomes half (5μm), the longitudinal and lateral lengths of the image do not change. On the other hand, the pixel shift super resolution generates a high resolution image based on four low resolution images obtained by random shifts and spline interpolation.

FIG. 7 are views illustrating four transmittance distributions P1 to P4 generated by numerical calculation models of the polarization control element for use with the present invention. The four transmittance distributions P1 to P4 correspond to one certain pixel in the image sensor. In other words, since one pixel is divided into 2×2 this time, each of the transmittance distributions P1 to P4 forms a 2×2 matrix. A black color means a low transmittance, and a white color means a high transmittance. This embodiment generates a high resolution image in accordance with the super resolution method illustrated in the flowchart in FIG. 3 with the four transmittance distributions P1 to P4 illustrated in FIG. 7, and uses a Moore-Penrose pseudo inverse matrix as a solution of the inverse problem in S304.

The quality of the high resolution image is evaluated by a root mean square error (RMSE) with a true image. RMSE is given as follows.

Herein, P and Q are arbitrary M rows and 1 column vectors, and pi and qi are i-th elements for P and Q. As RMSE between the high resolution image and the true image is closer to zero, P and Q becomes more similar to each other. As RMSE between the high resolution image and the true image is closer to zero, the high resolution image is more similar to the true image and the super resolution is highly properly achieved.

FIGs. 8A to 8C illustrate super resolution process results using the numerical calculations. FIG. 8A illustrates a high resolution image obtained by the pixel shift super resolution. FIG. 8B illustrates a high resolution image obtained by the image capturing apparatus according to the present invention. FIG. 8C is the true image. RMSE between the high resolution image obtained by the pixel shift super resolution and the true image is 0.0145, and RMSE between the high resolution image obtained by the image capturing apparatus according to the present invention and the true image is 0.000. Since RMSE between the high resolution image obtained by the image capturing apparatus according to the present invention and the true image is closer to zero than RMSE between the high resolution image obtained by the pixel shift super resolution and the true image, it is understood that the present invention can provide a better super resolution than the prior art.

The present invention drives the polarization control element 103, but may include means for rotating the first polarization plate 102 instead of providing the polarization control element 103. In this case, the mechanical driving occurs but the super resolution method according to this embodiment is more precise than the pixel shift method.

While the image processor 150 provides a super resolution process in this embodiment, a personal computer (PC) or a dedicated image processing apparatus in which the super resolution process program (image processing method) is installed may perform the super resolution process. In this case, the storage unit 162 may be attached to and detached from the image capturing apparatus 100, or data may be transferred to the PC or the dedicated image processing apparatus by a cable (wire), such as a USB cable, or a wireless communication. In this case, the communication may be performed through a network, such as the Internet and LAN. The image capturing apparatus 100 may include a wire or wireless communication unit.

The image processing apparatus at this time includes an acquirer configured to acquire a pixel value for an area in each pixel in the image sensor 105 by solving the inverse problem of a plurality of mutually different transmittance distributions and light intensity distributions corresponding to the transmittance distributions for respective pixels in the image sensor 105. The image processing apparatus includes a generator configured to generate a high resolution image by integrating the pixel values. The plurality of transmittance distributions are formed for the plurality of areas in each pixel in the image sensor 105 by introducing the light to the image sensor 105 via the optical elements (such as components 102 to 104). Each transmittance distribution is an attenuation rate distribution of the incident light to each pixel in the image sensor, to the incident light on the optical elements.

Similarly, the image processing method includes the steps of acquiring a pixel value for an area in each pixel in the image sensor 105 by solving the inverse problem of the above transmittance distributions and the light intensity distributions, and of generating a high resolution image by integrating the pixel values.

First Embodiment

FIG. 4 is a block diagram of an illustrative configuration of an infrared camera 400 as an image capturing apparatus according to a first embodiment. The infrared camera 400 includes an infrared imaging optical system 401, a first polarization plate 402, a liquid crystal element 403 as a polarization controller, a second polarization plate 404, and an infrared image sensor 405 as an image pickup element. These elements correspond to the imaging optical system 101, the first polarization plate 102, the polarization control element 103, the second polarization plate 104, and the image sensor 105, respectively.

All of the elements in the infrared camera 400 work in the infrared wavelength region. The infrared imaging optical system 401 is made, for example, of sapphire glass. The first polarization plate 402 is made, for example, of an aluminum wire grid and a diffractive grating. The infrared image sensor 405 is made, for example, of gallium arsenide. A wavelength filter may be installed, if necessary, in front of the infrared image sensor 405. Making the first polarization plate 402 of the wire grid and the diffractive grating is especially effective when the present invention is applied to the infrared camera.

The above configuration and the super resolution method illustrated by the flowchart in FIG. 3 can provide a high resolution infrared image using a low resolution infrared image sensor. The illustrative configuration in FIG. 4 is merely one example, and thus is applicable to a normal camera etc. in which components work in the visible light range.

Second Embodiment

FIG. 5 is a block diagram illustrating an illustrative configuration of an endoscope 500 as an image capturing apparatus according to a second embodiment. The endoscope 500 includes an illumination unit configured to illuminate a target, and an image capturing unit configured to form an image by collecting reflected light from the target illuminated by the illumination unit.

The illumination unit includes a light source 510 configured to emit parallel illumination light, an optical fiber 512 configured to transmit and make uniform light from the light source 510, and an illumination optical system 514 configured to illuminate the target using parallel light from the optical fiber 512. The light source 510 is separately necessary because the endoscope 500 is usually used in a dark location, such as an abdominal cavity. The light source 510 may emit a single wavelength or multiple wavelengths in time divisions. In the multiple wavelengths, low resolution images are acquired with different wavelengths at each time and a super resolution process is performed at each time. The light source 510 may emit non-polarized light, but may emit light having different polarization directions by time divisions. Without using the optical fiber 512 or the illumination optical system 514, the light source 510 may directly illuminate the target.

The image capturing unit includes an objective and imaging optical system 501, an optical fiber bundle 506, a first polarization plate 502, a liquid crystal element 503 as a polarization controller, a second polarization plate 504, and an image sensor 505 as an image pickup element. The components 501 to 505 correspond to the imaging optical system 101, the first polarization plate 102, the polarization control element 103, the second polarization plate 104, and the image sensor 105, respectively. The optical fiber bundle 506 transmits the optical image formed by the objective and imaging optical system 501, and may transmit it via another means, such as a rod lens.

S301 to S304 illustrated in FIG. 3 are performed for each wavelength, and the super resolution results for respective wavelengths are integrated as illustrated in S305.

When it is necessary to store the polarization state of the illumination light from the light source 510 to the target and the polarization state of the light from the target to the image sensor 505, it is necessary for the optical fiber 512 and the optical fiber bundle 506 to maintain the polarization. Fluorescent light emitted when the fluorescent material in the target is excited may be observed using the excitation light of the fluorescent light as the light source 510. In that case, it is necessary to separately provide a filter configured to separate the excitation light from the fluorescent light on the front surface of the image sensor 505, etc.

Due to the above structure and the super resolution method illustrated by the flowchart in FIG. 3, a high resolution endoscope image can be obtained with a low resolution image sensor.

Third Embodiment

FIG. 6 is a block diagram illustrating an illustrative configuration of a microscope 600 as an image capturing apparatus according to a third embodiment. The microscope 600 includes a stage 620, an illumination unit configured to illuminate a sample S on a stage, and an image capturing unit configured to collect reflected light from the sample S illuminated by the illumination unit and to form an image.

The illumination unit includes a light source 610 configured to emit parallel illumination light, and an illumination optical system 614 configured to illuminate the sample S using parallel light from the light source 610. The separate light source 610 is necessary to maintain a light amount because the microscope observes a micro portion. The light source 610 may emit a single wavelength or multiple wavelengths with time divisions. In case of the multiple wavelengths, similar to the second embodiment, low resolution images are acquired with different wavelengths at each time and a super resolution process is performed at each time. The light source 610 may emit non-polarized light or light having different polarization directions by time divisions.

The image capturing unit includes an objective and imaging optical system 601 configured to collect transmission or reflection light from the sample and to form an image, a first polarization plate 602, a liquid crystal element 603 as a polarization controller, a second polarization plate 604, and an image sensor 605 as an image pickup element. The components 601 to 605 correspond to the imaging optical system 101, the first polarization plate 102, the polarization control element 103, the second polarization plate 104, and the image sensor 105, respectively.

Fluorescent light emitted when the fluorescent material in the sample is excited may be observed using the excitation light of the fluorescent light as the light source. In that case, similar to the second embodiment, it is necessary to separately provide a filter configured to separate the excitation light from the fluorescent light on the front surface of the image sensor 605, etc.

Other Embodiments

Embodiment(s) of the present invention can also be realized by a computer of a system or apparatus that reads out and executes computer executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be referred to more fully as a 'non-transitory computer-readable storage medium') to perform the functions of one or more of the above-described embodiment(s) and/or that includes one or more circuits (e.g., application specific integrated circuit (ASIC)) for performing the functions of one or more of the above-described embodiment(s), and by a method performed by the computer of the system or apparatus by, for example, reading out and executing the computer executable instructions from the storage medium to perform the functions of one or more of the above-described embodiment(s) and/or controlling the one or more circuits to perform the functions of one or more of the above-described embodiment(s). The computer may comprise one or more processors (e.g., central processing unit (CPU), micro processing unit (MPU)) and may include a network of separate computers or separate processors to read out and execute the computer executable instructions. The computer executable instructions may be provided to the computer, for example, from a network or the storage medium. The storage medium may include, for example, one or more of a hard disk, a random-access memory (RAM), a read only memory (ROM), a storage of distributed computing systems, an optical disk (such as a compact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)^TM), a flash memory device, a memory card, and the like.

While the present invention has been described with reference to exemplary examples, it is to be understood that the invention is not limited to the disclosed exemplary examples. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

For example, the present invention is not limited to an embodiment that forms a transmittance distribution using a polarization control. In this case, the image capturing apparatus includes an image sensor configured to capture an object, a producer configured to produce a plurality of mutually different transmittance distributions for a plurality of areas in the same pixel in the image sensor, a storage unit, and a controller. The controller stores information of a light intensity distribution for each pixel of the image sensor in the storage unit, by correlating the light intensity distribution with each of the plurality of transmittance distributions. Each transmittance distribution is an attenuation distribution of the incident light on the plurality of areas, to the incident light on the producer. The producer is made, for example, by providing a plurality of types of dimming filters in which the areas have mutually different dimming rates so that these filters can be detachable or rotatable.

This application claims the benefit of Japanese Patent Application No. 2015-230722, filed on November 26, 2015 which is hereby incorporated by reference herein in its entirety.

100 image capturing apparatus
102 first polarization plate (first polarizer)
103 polarization control element (adjuster)
104 second polarization plate (second polarizer)
105 image sensor
155 driver (adjuster)
202a-d area
205 one pixel in image senor

Claims

A polarization control unit for use with an image capturing apparatus, the polarization control unit comprising:
a first polarizer;
an adjuster configured to adjust a polarization direction of a light flux that has passed the first polarizer; and
a second polarizer which the light flux from the adjuster is to pass,
wherein at least one of the first polarizer and the second polarizer include a plurality of areas which light fluxes having mutually different polarization directions can pass, and
wherein the light fluxes that have passed the plurality of areas enter the same pixel in an image sensor in the image capturing apparatus.
The polarization control unit according to claim 1, wherein the adjuster includes:
a liquid crystal element; and
an electric power applier configured to apply an electric power so as to adjust an orientation of a liquid crystal molecule in the liquid crystal element.
The polarization control unit according to claim 1, wherein the adjuster rotates the first polarizer.
The polarization control unit according to any one of claims 1 to 3, wherein at least one of the first polarizer and the second polarizer includes a wire grid.
The polarization control unit according to any one of claims 1 to 3, wherein at least one of the first polarizer and the second polarizer includes a diffractive grating.
A polarization control unit for use with an image capturing apparatus, the polarization control unit comprising:
a first polarizer;
an adjuster configured to adjust a polarization direction of a light flux that has passed the first polarizer; and
a second polarizer which the light flux from the adjuster is to pass,
wherein the adjuster includes:
a polarization control element that includes a plurality of areas which light fluxes having mutually different polarization directions can pass, and
a direction adjuster configured to adjust the polarization directions for the plurality of areas, and
wherein the light fluxes that have passed the plurality of areas enter the same pixel in an image sensor in the image capturing apparatus.
An image capturing apparatus comprising:
a polarization control unit according to any one of claims 1 to 6; and
an image sensor configured to receive light from the polarization control unit.
The image capturing apparatus according to claim 7, further comprising an imaging optical system configured to form an image of an object on a pixel in the image sensor via the polarization control unit.
The image capturing apparatus according to claim 7, further comprising a controller configured to store, in the storage unit, information of a light intensity distribution of each pixel in the image sensor and a corresponding one of the mutually different transmittance distributions for each of the plurality of areas in each pixel in the image sensor, and
wherein each transmittance distribution is an attenuation rate distribution of incident light on the plurality of areas, to incident light on the polarization control unit.
An image capturing apparatus comprising:
an image sensor configured to capture an image of an object;
a producer configured to produce a plurality of mutually different transmittance distributions for a plurality of areas in the same pixel in the image sensor; and
a controller configured to configured to store, in the storage unit, information of a light intensity distribution of each pixel in the image sensor and a corresponding one of the plurality of transmittance distributions,
wherein each transmittance distribution is an attenuation rate distribution of incident light on the plurality of areas, to incident light on the producer.
The image capturing apparatus according to claim 9 and 10, further comprising an image processor configured to generate a high resolution image by solving an inverse problem based on information of the transmittance distribution and the light intensity distribution, by acquiring pixel values the areas in each pixel, and by integrating the pixel values.
The image capturing apparatus according to claim 11, wherein vectors of the plurality of transmittance distributions are orthogonal to one another.
The image capturing apparatus according to claim 11 or 12, wherein the image processor solves the inverse problem using a pseudo inverse matrix of a matrix representative of the plurality of transmittance distributions.
The image capturing apparatus according to claim 11 or 12, further comprising a display unit configured to display the high resolution image.
An image capturing system comprising:
an imaging optical system configured to form an optical image of an object;
a lens apparatus that includes a polarization control unit according to any one of claims 1 to 6; and
an image capturing apparatus to which the lens apparatus is attachable and from which the lens apparatus is detachable.
An image processing apparatus comprising:
an acquirer configured to solve an inverse problem based on information of a plurality of mutually different transmittance distributions and corresponding light intensity distributions in respective pixels in an image sensor, for a plurality of areas in the same pixel in the image sensor which is made by introducing light to the image sensor via an optical element and to acquire pixel values in the plurality of areas in each pixel in the image sensor; and
a generator configured to generate a high resolution image by integrating the pixel pixels,
wherein each transmittance distribution is an attenuation rate distribution of incident light on the plurality of areas, to incident light on the optical element.
An image processing method comprising the steps of:
solving an inverse problem based on information of a plurality of mutually different transmittance distributions and corresponding light intensity distributions in respective pixels in an image sensor, for a plurality of areas in the same pixel in the image sensor which is made by introducing light to the image sensor via an optical element and acquiring pixel values in the plurality of areas in each pixel in the image sensor; and
generating a high resolution image by integrating the pixel pixels,
wherein each transmittance distribution is an attenuation rate distribution of incident light on the plurality of areas, to incident light on the optical element.
A program for enabling a computer to execute the image processing method according to claim 17.