US9191560B2 - Image capturing apparatus that performs photoelectric conversion on incident light that has passed through an imaging lens and outputs an electric signal - Google Patents

Image capturing apparatus that performs photoelectric conversion on incident light that has passed through an imaging lens and outputs an electric signal Download PDF

Info

Publication number
US9191560B2
US9191560B2 US13/738,198 US201313738198A US9191560B2 US 9191560 B2 US9191560 B2 US 9191560B2 US 201313738198 A US201313738198 A US 201313738198A US 9191560 B2 US9191560 B2 US 9191560B2
Authority
US
United States
Prior art keywords
pixels
photoelectric conversion
conversion unit
image
areas
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Expired - Fee Related
Application number
US13/738,198
Other versions
US20130182170A1 (en
Inventor
Satoshi Suzuki
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Canon Inc
Original Assignee
Canon Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Canon Inc filed Critical Canon Inc
Assigned to CANON KABUSHIKI KAISHA reassignment CANON KABUSHIKI KAISHA ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: SUZUKI, SATOSHI
Publication of US20130182170A1 publication Critical patent/US20130182170A1/en
Application granted granted Critical
Publication of US9191560B2 publication Critical patent/US9191560B2/en
Expired - Fee Related legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • H04N5/2254—
    • G—PHYSICS
    • G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
    • G03B—APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR
    • G03B27/00—Photographic printing apparatus
    • E—FIXED CONSTRUCTIONS
    • E04—BUILDING
    • E04D—ROOF COVERINGS; SKY-LIGHTS; GUTTERS; ROOF-WORKING TOOLS
    • E04D13/00—Special arrangements or devices in connection with roof coverings; Protection against birds; Roof drainage ; Sky-lights
    • E04D13/04—Roof drainage; Drainage fittings in flat roofs, balconies or the like
    • E04D13/064—Gutters
    • E04D13/072—Hanging means
    • E04D13/0722—Hanging means extending mainly under the gutter
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/60—Noise processing, e.g. detecting, correcting, reducing or removing noise
    • H04N25/63—Noise processing, e.g. detecting, correcting, reducing or removing noise applied to dark current
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/60—Noise processing, e.g. detecting, correcting, reducing or removing noise
    • H04N25/63—Noise processing, e.g. detecting, correcting, reducing or removing noise applied to dark current
    • H04N25/633—Noise processing, e.g. detecting, correcting, reducing or removing noise applied to dark current by using optical black pixels
    • H04N5/361—

Definitions

  • the present invention relates to an image capturing apparatus that captures, records and reproduces still images and moving images, and more particularly to an image capturing apparatus that has a micro lens array in front of an image sensor, which is a constituent element of the image capturing apparatus.
  • image capturing apparatuses such as electronic cameras, that record and reproduce still images and moving images captured with a solid-state image sensor such as a CCD or a CMOS by using a memory card having a solid-state memory element as a recording medium.
  • a solid-state image sensor such as a CCD or a CMOS
  • an image capturing apparatus has been proposed that has a configuration in which a micro lens array where the lenses are arranged such that each corresponds to a plurality of pixels is arranged in front of a solid-state image sensor, whereby information regarding the incidence direction of light rays that are incident on the image sensor is also obtained (see, for example, Ren. Ng et al. “Light Field Photography with a Hand-Held Plenoptic Camera”, Stanford Tech Report CTSR 2005-02).
  • Such an image capturing apparatus is capable of, in addition to generating ordinary captured images based on an output signal from each pixel, reconfiguring images focused at arbitrary distances by performing predetermined image processing on the captured images.
  • image capturing apparatuses not only those having a micro lens array described above, have the following problems. Specifically, it is often the case that image sensors used in commonly-used image capturing apparatuses have an OB area that is used to determine a reference level of the output signal and that includes a plurality of optical black pixels (OB pixels) that are optically shielded from light.
  • OB pixels optical black pixels
  • the OB pixels are arranged within an effective pixel area that receives light from the subject, the light signal of the pixel area cannot be received, and therefore the OB pixels are often arranged in the periphery of the effective pixel area such as the upper side, the left side of the effective pixel area, and so forth, rather than within the effective pixel area.
  • the reference level varies two-dimensionally within the capture surface, it is difficult to determine the reference level of the effective pixel area from the output signal from the OB area in the periphery, and as a result it may cause degradation of the obtained images.
  • the present invention has been made in consideration of the above situation, and provides an image capturing apparatus that includes a micro lens array comprising micro lenses, each of which corresponds to a plurality of pixels, and in which a detailed reference level in an effective pixel area can be acquired without adding a new constituent element.
  • an image capturing apparatus comprising: a photoelectric conversion unit that performs photoelectric conversion on incident light that has passed through an imaging lens and outputs an electric signal, the photoelectric conversion unit including a plurality of two-dimensionally arranged pixels; and a plurality of micro lenses that are formed between the imaging lens and the photoelectric conversion unit, the plurality of micro lenses respectively corresponding to a plurality of areas in the photoelectric conversion unit, each of the areas including a predetermined number of pixels, wherein among the pixels included in the plurality of areas, pixels whose receiving quantity of light is small in each area due to a characteristic of an optical system including the imaging lens and the micro lenses and a characteristic of the photoelectric conversion unit are configured as pixels for use in calculation of a reference value for the electric signals output from the photoelectric conversion unit.
  • an image capturing apparatus comprising: a photoelectric conversion unit that performs photoelectric conversion on incident light that has passed through an imaging lens and outputs an electric signal, the photoelectric conversion unit including a plurality of two-dimensionally arranged pixels; and a plurality of micro lenses that are formed between the imaging lens and the photoelectric conversion unit, the plurality of micro lenses respectively corresponding to a plurality of areas in the photoelectric conversion unit, each of the areas including a predetermined number of pixels, wherein among the pixels included in the plurality of areas, pixels that are arranged in an outermost region of each of the plurality of areas are configured as pixels for use in calculation of a reference value of an electric signal output from the photoelectric conversion unit.
  • FIG. 1 is a block diagram showing a schematic configuration of an image capturing apparatus according to an embodiment of the present invention
  • FIG. 2 is a block diagram showing constituent elements in the periphery of an optical system of the image capturing apparatus according to the embodiment of the present invention
  • FIGS. 3A and 3B are diagrams showing a schematic configuration of an image sensor according to the embodiment of the present invention.
  • FIGS. 4A and 4B are diagrams showing a pixel arrangement in the image sensor according to the embodiment of the present invention.
  • FIG. 5 is a ray tracing diagram from subjects in the image capturing apparatus according to the embodiment of the present invention.
  • FIG. 6 is a ray tracing diagram from subjects at different positions in the image capturing apparatus according to the embodiment of the present invention.
  • FIG. 7 is a diagram two-dimensionally showing ray tracing in the image capturing apparatus according to the embodiment of the present invention.
  • FIGS. 8A and 8B are diagrams illustrating a pixel arrangement in the image sensor according to the embodiment of the present invention.
  • FIG. 9 is a diagram illustrating an example of a reference level calculation method according to the embodiment of the present invention.
  • FIG. 1 is a block diagram showing a schematic configuration of an image capturing apparatus according to the present embodiment.
  • reference numeral 101 denotes an optical system including an imaging lens, a diaphragm and a micro lens
  • 102 is a mechanical shutter.
  • An image sensor 103 converts incident light to an electric signal and includes a photoelectric conversion unit 104 that converts incident light to an electric signal and an amplifier circuit 105 that amplifies the electric signal.
  • An analog signal processing circuit 106 performs analog signal processing on an image signal output from the image sensor 103 .
  • the analog signal processing circuit 106 includes a CDS circuit 107 that performs correlated double sampling, a signal amplifier 108 that amplifies an analog signal, a clamping circuit 109 that performs horizontal OB clamping, and an A/D converter 110 that converts an analog signal to a digital signal.
  • a timing signal generation circuit 111 generates a signal for operating the image sensor 103 and the analog signal processing circuit 106 .
  • a driving circuit 112 drives the optical system 101 and the mechanical shutter 102 .
  • a digital signal processing circuit 113 includes an image correction circuit 114 that performs necessary correction processing on image data, a signal amplifying circuit 115 that amplifies the digital signal that has been corrected by the image correction circuit 114 , and an image processing circuit 116 that performs necessary image processing on image data. With these constituent elements, the digital signal processing circuit 113 performs necessary digital signal processing on the captured image data.
  • Reference numeral 117 is an image memory that stores the processed image data
  • 118 is a recording medium removable from the image capturing apparatus
  • 119 is a recording circuit that records the image data that has undergone signal processing in the recording medium 118
  • Reference numeral 120 is an image display device that displays image data that has undergone signal processing
  • 121 is a display circuit that displays an image on the image display device 120 .
  • a system control unit 122 performs overall control on the image capturing apparatus.
  • a non-volatile memory (ROM) 123 stores a program in which a control method executed by the system control unit 122 is written, control data, such as parameters and tables, for use in executing the program, and correction data such as defect addresses.
  • the program, control data and correction data stored in the non-volatile memory 123 are transferred to and stored in a volatile memory (RAM) 124 which are used by the system control unit 122 when controlling the image capturing apparatus.
  • RAM volatile memory
  • the optical system 101 drives the optical system 101 such as a lens and forms on the image sensor 103 an image of the subject controlled so as to have an appropriate brightness.
  • the mechanical shutter 102 is driven such that the image sensor 103 is shielded from light along with the operation of the image sensor 103 so as to obtain the required exposure period when capturing a still image.
  • the required exposure period may be secured by using electrical shutter function of the image sensor 103 together with the mechanical shutter 102 .
  • the mechanical shutter 102 is maintained in a full-open state such that the image sensor 103 is constantly exposed with light during image capturing.
  • the image sensor 103 is driven with a driving pulse based on an operation pulse generated by the timing signal generation circuit 111 that is controlled by the system control unit 122 .
  • the photoelectric conversion unit 104 converts the image of the subject into electric signals by using photoelectric conversion, and the amplifier circuit 105 multiplies an electric signal by a gain that is an amplification factor that has been set according to the quantity of incident light and outputs the product as an analog image signal.
  • clock synchronization noise is removed from the analog image signal output from the image sensor 103 .
  • the signal amplifier 108 multiplies the image signal by a gain that is an amplification factor that has been set according to the quantity of incident light, and the clamping circuit 109 clamps a signal output of a horizontal OB area as a reference voltage, and the signal is converted to a digital image signal by the A/D converter 110 .
  • the image correction circuit 114 performs various types of image correction processing such as defect correction and dark shading correction on the image signal that has been converted to a digital signal.
  • image correction processing such as defect correction and dark shading correction on the image signal that has been converted to a digital signal.
  • an output signal from a reference value calculation pixel (OB pixel) is used, which will be described later.
  • the signal amplifying circuit 115 multiplies the signal by a gain that is an amplification factor that has been set according to the quantity of incident light, and the image processing circuit 116 performs various types of image processing including image processing such as color conversion, white balance and gamma correction, resolution conversion processing and image compression processing. Also, the image processing circuit 116 can perform, for example, re-focus processing by using the method disclosed by Ren. Ng et al. (processing for generating a plurality of images each focused at an arbitrary distance based on the electric signals output from the image sensor 103 ). At this time, the image memory 117 is used to temporarily store a digital image signal under signal processing or to store image data that is a digital image signal that has undergone signal processing.
  • the image data that has undergone signal processing in the digital signal processing circuit 113 and the image data stored in the image memory 117 are converted to data (for example, file system data having a hierarchical structure) suitable for the recording medium 118 by the recording circuit 119 and then recorded in the recording medium 118 .
  • the image data that has undergone resolution conversion processing in the digital signal processing circuit 113 is converted to a signal (for example, NTSC type analog signal or the like) suitable for the image display device 120 by the display circuit 121 and then displayed on the image display device 120 , for example.
  • the digital image signal in response to a control signal from the system control unit 122 , the digital image signal may be output to the image memory 117 and the recording circuit 119 as image data, without performing signal processing. Also, if there is a request from the system control unit 122 , the digital signal processing circuit 113 outputs a digital image signal generated during image processing and image data information to the system control unit 122 .
  • the image data information includes, for example, information regarding the spatial frequency of an image, the average value of a specific area, the amount of data of a compressed image and the like, or information extracted therefrom.
  • the recording circuit 119 outputs information regarding the type, free space and the like of the recording medium 118 to the system control unit 122 .
  • the recording circuit 119 reads out image data from the recording medium 118 .
  • the digital signal processing circuit 113 performs image decompression processing on the read image data and stores the image data in the image memory 117 in the case where the read image data is a compressed image.
  • the image data stored in the image memory 117 is subjected to resolution conversion processing by the digital signal processing circuit 113 , thereafter converted to a signal suitable for the image display device 120 by the display circuit 121 and then displayed on the image display device 120 .
  • FIG. 2 is a block diagram illustrating constituent elements in the periphery of the optical system of the image capturing apparatus according to the embodiment of the present invention.
  • an imaging lens 201 and a diaphragm 204 together constitute the optical system 101 .
  • a micro lens array 202 and a sensor array 203 are constituent elements of the image sensor 103 .
  • Reference numeral 205 denotes a subject. The other constituent elements shown in FIG. 2 are the same as those described in connection with FIG. 1 , and thus a description thereof is not given here.
  • an image of the subject 205 is formed on the image sensor 103 by the imaging lens 201 .
  • Light signals incident on the image sensor 103 are further converged by each micro lens of the micro lens array 202 , and incident on each pixel of the sensor array 203 .
  • the configurations of the micro lens array 202 and the sensor array 203 will be described later with reference to FIGS. 4A and 4B .
  • the light signals incident on the sensor array 203 are subjected to photoelectric conversion in each pixel and output as electric signals. The processing after this is the same as that described with reference to FIG. 1 .
  • FIGS. 3A and 3B are diagrams showing a schematic configuration of the image sensor 103 of the image capturing apparatus of the present embodiment.
  • the image sensor 103 includes a pixel area 302 in which a plurality of pixels 301 are arranged in an array and peripheral circuitry.
  • the peripheral circuitry includes a vertical shift register 303 , a readout circuit 304 , a horizontal shift register 305 , an output amplifier 306 , driving signal lines 307 , vertical signal lines 308 , and so forth.
  • a driving pulse is sent out from the vertical shift register 303 through a driving signal line 307 for each of the pixels arranged in each row.
  • the driving signal lines 307 are illustrated such that a single driving signal line is provided for each row, but in fact, a plurality of driving signal lines are connected for each row. Also, the pixels aligned in the same column are connected to the same vertical signal line 308 .
  • the pixel signals of a row selected by the vertical shift register 303 are output to the readout circuit 304 through the vertical signal lines 308 , and sequentially output from the pixel signals of a column selected by the horizontal shift register 305 through the output amplifier 306 .
  • FIGS. 3A and 3B show a configuration in which the pixel area 302 has pixels 301 in 8 columns and 8 rows, but in fact, the pixel area 302 is formed with pixels in several thousands of columns and rows.
  • FIG. 3B is a circuit diagram showing a configuration of a pixel 301 shown in FIG. 3A .
  • the pixel 301 includes a photodiode 401 , a floating diffusion unit 402 , an amplifier unit 403 , a transfer switch 404 , a reset switch 405 and a selection switch 406 .
  • the photodiode 401 functions as a photoelectric conversion unit that receives light that is incident on the pixel 301 and generates a signal charge according to the quantity of the received light.
  • the floating diffusion unit 402 functions as a charge voltage conversion unit that temporarily stores the charge generated by the photodiode 401 and transferred, and converts the held charge to a voltage signal.
  • the transfer switch 404 is driven with a transfer pulse signal PTX and transfers the signal charge generated by the photodiode 401 to the floating diffusion unit 402 .
  • the reset switch 405 is driven with a reset pulse signal PRES, and resets the potential of the floating diffusion unit 402 to a reference potential VDD.
  • the amplifier unit 403 is a source follower MOS transistor, and amplifies the voltage signal based on the charge held by the floating diffusion unit 402 and outputs the signal as a pixel signal.
  • the selection switch 406 is driven with a vertical selection pulse signal PSEL, and outputs the pixel signal amplified by the amplifier unit 403 to the vertical signal line 308 .
  • FIGS. 4A and 4B are diagrams showing a relative arrangement between the pixels of the sensor array 203 and the micro lens array 202 constituting the image sensor 103 of the present embodiment.
  • reference numeral 501 denotes a recording pixel unit corresponding to a single pixel in a reconfigured image
  • 502 denotes pixels arranged in 8 columns and 8 rows that constitute the recording pixel unit 501 , each pixel provided for each photoelectric conversion unit and corresponding to the pixel 301 described with reference to FIGS. 3A and 3B
  • Reference numeral 503 denotes micro lenses arranged in one to one correspondence with the recording pixel units 501 .
  • FIGS. 4A and 4B a sensor array 203 in which recording pixel units 501 are arranged in 4 columns and 4 rows, each recording pixel unit 501 including pixels 502 arranged in 8 columns and 8 rows.
  • FIG. 4B is an enlarged diagram showing a recording pixel unit 501 .
  • 64 pixels are labeled with numbers from a 00 to a 77 .
  • FIG. 5 is a diagram showing ray tracing of light rays that are incident on the image capturing apparatus of the present embodiment from the subjects at various distances.
  • a subject 701 a is the subject located at a position at which an image of the subject 701 a is formed on a plane A including the micro lens array 202 by the imaging lens 201 .
  • the light rays from the subject 701 a those that pass through the outermost periphery of the imaging lens 201 and are incident on the sensor array 203 via the micro lens on the optical axis are indicated by solid lines.
  • a subject 701 b is the subject located further away from the subject 701 a as viewed from the imaging lens 201 .
  • An image of the subject 701 b formed by the imaging lens 201 is formed on a plane B that is closer to the imaging lens 201 than the plane A including the micro lens array 202 is.
  • the light rays from the subject 701 b those that pass through the outermost periphery of the imaging lens 201 and are incident on the sensor array 203 via the micro lens on the optical axis are indicated by broken lines.
  • a subject 701 c is the subject located closer to the imaging lens 201 than the subject 701 a is, as viewed from the imaging lens 201 .
  • An image of the subject 701 c formed by the imaging lens 201 is formed on a plane C that is further away from the imaging lens 201 than the plane A including the micro lens array 202 is.
  • the light rays from the subject 701 c those that pass through the outermost periphery of the imaging lens 201 and are incident on the sensor array 203 via the micro lens on the optical axis are indicated by dashed and two-dotted lines.
  • FIG. 6 is a diagram showing ray tracing of light rays that are incident on the image capturing apparatus of the present embodiment from the subjects at different positions.
  • a subject 801 a is the subject located at a position at which an image of the subject 801 a is formed on a micro lens 802 a on the plane A including the micro lens array 202 by the imaging lens 201 .
  • the light rays from the subject 801 a those that pass through the outermost periphery of the imaging lens 201 and are incident on the sensor array 203 via the micro lens 802 a are indicated by solid lines. These light rays arrive at a recording pixel unit 803 a on the sensor array 203 .
  • a subject 801 b is the subject located at a position that is away from the imaging lens 201 at the same distance as the subject 801 a , as viewed from the imaging lens 201 , and at which an image of the subject 801 b is formed on a micro lens 802 b on the plane A including the micro lens array 202 by the imaging lens 201 .
  • the light rays from the subject 801 b those that pass through the outermost periphery of the imaging lens 201 and are incident on the sensor array 203 via the micro lens 802 b are indicated by broken lines. These light rays arrive at a recording pixel unit 803 b via the sensor array 203 .
  • FIG. 7 is a diagram two-dimensionally showing ray tracing of incident light rays in the image capturing apparatus according to the present embodiment.
  • reference numeral 902 denotes one of the micro lenses of the micro lens array 202
  • 903 denotes the sensor array surface.
  • a light ray passing through a center p 0 of the diaphragm 204 is incident on a point p 0 ′ at which the sensor array surface 903 and the optical axis intersect.
  • a light ray from the subject located at a position at which an image of the subject is formed on a plane including the micro lens array 202 by the imaging lens 201 that passes through a point p 1 on the outer periphery of the aperture of the diaphragm 204 is incident on p 1 ′ on the sensor array surface 903 .
  • light rays passing through other points p 2 and p 3 on the outer periphery of the aperture of the diaphragm 204 are respectively incident on points p 2 ′ and p 3 ′ on the sensor array surface 903 .
  • the quantity of light rays that are received by the pixels a 00 , a 07 , a 70 and a 77 on the sensor array surface 903 and are effective in the generation of a refocused image is smaller than that of the pixels around the optical axis of the micro lens, and thus there is a small influence on the subsequent re-focus processing.
  • the pixels a 00 , a 07 , a 70 and a 77 located at four corners of the recording pixel unit 501 have a relatively small quantity of received light due to the characteristics (i.e., D, F, d, f and the like described above) of the image sensor 103 and the optical system including the imaging lens 201 and the micro lens array 202 . Accordingly, it can be said that these pixels are ineffective for re-focus processing.
  • FIGS. 8A and 8B are diagrams showing a pixel arrangement in the image capturing apparatus of the present embodiment.
  • FIG. 8A is a layout diagram in the sensor array 203 described with reference to FIG. 4A
  • FIG. 8B is an arrangement diagram of a recording pixel unit 501 and pixels 502 , which are constituent elements of the sensor array 203 described with reference to FIG. 4B .
  • the pixels a 00 , a 07 , a 70 and a 77 located at four corners of the recording pixel unit 501 are, as described with reference to FIG. 7 , pixels that are arranged in an area that is not suitable for re-focus processing, which is performed in a later stage, and therefore these pixels are used as OB pixels.
  • the pixels that are used as OB pixels are shaded in gray and the pixels that are used as effective pixels are shown in white.
  • the four pixels that are used as OB pixels are shielded from light with aluminum wiring in order to enhance the light blocking effect.
  • FIG. 9 is a diagram illustrating an example of a reference level calculation method in the image capturing apparatus of the present embodiment. The method for calculating a reference level of a pixel amn by using output signal values of the OB pixels a 00 , a 07 , a 70 and a 77 in the pixel arrangement shown in FIG. 8B will be described with reference to FIG. 9 .
  • This calculation method is a method in which calculation is performed assuming that the reference levels of the OB pixels transition linearly in proportion to the distance from the OB pixel.
  • the output signal values of the OB pixels a 00 , a 07 , a 70 and a 77 are respectively expressed as B(0, 0), B(0, 7), B(7, 0) and B(7, 7).
  • a pixel a 0 n is the pixel located at the position at which the distance between the OB pixel a 00 and the OB pixel a 07 is internally divided in n:7-n.
  • the pixel amn is the pixel located at a position at which the distance between the OB pixel a 0 n and the OB pixel a 7 n is internally divided in m:7 ⁇ m, and therefore the reference level B(m, n) of the pixel amn is represented as follows:
  • B ⁇ ( m , n ) ( 1 - m / 7 ) ⁇ ( 1 - n / 7 ) ⁇ B ⁇ ( 0 , 0 ) + ( 1 - m / 7 ) ⁇ n / 7 ⁇ B ⁇ ( 0 , 7 ) + m / 7 ⁇ ( 1 - n / 7 ) ⁇ B ⁇ ( 7 , 0 ) + m / 7 ⁇ n / 7 ⁇ B ⁇ ( 7 , 7 ) .
  • the image capturing apparatus of the present embodiment by arranging OB pixels within an effective pixel area, it is possible to acquire reference levels that are finer and more accurate than the conventional technology. Accordingly, by performing dark level correction or the like by using such reference levels, it is possible to achieve an image capturing apparatus that can provide images of good image quality.
  • a single recording pixel unit is configured with 8-by-8 pixels.
  • the present invention is not limited thereto, and a single recording pixel unit may be configured with any number of pixels with any shape.
  • the pixels that are located at specific positions of the outermost region and whose receiving quantity of light rays, passed through the diaphragm 204 and the micro lens 902 , that can be used in re-focus processing later are used as OB pixels.
  • the present invention is not limited thereto, and more pixels may be used as OB pixels such as using a configuration in which the pixels whose receiving quantity of light rays that can be used for re-focus processing and the pixels surrounding such pixels are used OB pixels. In this case, it is desirable that the pixels used as OB pixels are shield from light with aluminum such that light rays do not arrive at the pixels.
  • the pixel configuration of the embodiment of the image capturing apparatus according to the present invention was described taking an example in which the OB pixels are shielded from light with aluminum wiring, but the present invention is not limited thereto.
  • the OB pixels may be shielded from light by any other methods.
  • a configuration may be possible in which a black color filter is disposed in front of the OB pixels so as to produce the light blocking effect.
  • the OB pixels of the present invention for shading correction of a circuit system, for example, instead of using the light blocking member as described above, it is possible to use a configuration in which the charges generated in the photoelectric conversion units are not transferred or a configuration that does not include the photoelectric conversion units.
  • the pixels that are used as OB pixels may be configured such that by causing the transfer switch 404 shown in FIG. 3B to be constantly open, the signal charge generated by the photodiode 401 is not transferred. It is also possible to use a configuration in which the photodiode 401 and the transfer switch 404 are not provided and a reset level corresponding to the reference potential VDD is constantly read out.
  • the method for calculating a reference level for each pixel was described with reference to FIG. 9 taking an example in which a reference value of each pixel is calculated based on the distance from four OB pixels.
  • the present invention is not limited thereto.
  • the digital signal processing circuit 113 which is a constituent element of the image capturing apparatus, performs image processing such as image reconfiguration, but the image processing is not necessarily performed inside the image capturing apparatus.
  • image processing such as image reconfiguration

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Architecture (AREA)
  • Civil Engineering (AREA)
  • Structural Engineering (AREA)
  • Transforming Light Signals Into Electric Signals (AREA)
  • Studio Devices (AREA)
  • Solid State Image Pick-Up Elements (AREA)

Abstract

An image capturing apparatus comprises a photoelectric conversion unit that performs photoelectric conversion on incident light that has passed through an imaging lens and outputs an electric signal, the photoelectric conversion unit including a plurality of two-dimensionally arranged pixels, and a plurality of micro lenses formed between the imaging lens and the photoelectric conversion unit. The plurality of micro lenses respectively corresponds to a plurality of areas in the photoelectric conversion unit and each of the areas includes a predetermined number of pixels. Among the pixels, those whose receiving quantity of light is small in each area due to a characteristic of an optical system including the imaging lens and the micro lenses and a characteristic of the photoelectric conversion unit are configured as pixels for use in calculation of a reference value for the electric signals output from the photoelectric conversion unit.

Description

BACKGROUND OF THE INVENTION
1. Field of the Invention
The present invention relates to an image capturing apparatus that captures, records and reproduces still images and moving images, and more particularly to an image capturing apparatus that has a micro lens array in front of an image sensor, which is a constituent element of the image capturing apparatus.
2. Description of the Related Art
Conventionally, there are many image capturing apparatuses, such as electronic cameras, that record and reproduce still images and moving images captured with a solid-state image sensor such as a CCD or a CMOS by using a memory card having a solid-state memory element as a recording medium.
As an example of a technique for such image capturing apparatuses, an image capturing apparatus has been proposed that has a configuration in which a micro lens array where the lenses are arranged such that each corresponds to a plurality of pixels is arranged in front of a solid-state image sensor, whereby information regarding the incidence direction of light rays that are incident on the image sensor is also obtained (see, for example, Ren. Ng et al. “Light Field Photography with a Hand-Held Plenoptic Camera”, Stanford Tech Report CTSR 2005-02).
Such an image capturing apparatus is capable of, in addition to generating ordinary captured images based on an output signal from each pixel, reconfiguring images focused at arbitrary distances by performing predetermined image processing on the captured images.
However, image capturing apparatuses, not only those having a micro lens array described above, have the following problems. Specifically, it is often the case that image sensors used in commonly-used image capturing apparatuses have an OB area that is used to determine a reference level of the output signal and that includes a plurality of optical black pixels (OB pixels) that are optically shielded from light. When the OB pixels are arranged within an effective pixel area that receives light from the subject, the light signal of the pixel area cannot be received, and therefore the OB pixels are often arranged in the periphery of the effective pixel area such as the upper side, the left side of the effective pixel area, and so forth, rather than within the effective pixel area.
However, no consideration has been given to the OB area in image capturing apparatuses having a micro lens array comprising micro lenses, each of which corresponds to a plurality of pixels, such as those described in the embodiments given in the present specification.
Also, if the reference level varies two-dimensionally within the capture surface, it is difficult to determine the reference level of the effective pixel area from the output signal from the OB area in the periphery, and as a result it may cause degradation of the obtained images.
SUMMARY OF THE INVENTION
The present invention has been made in consideration of the above situation, and provides an image capturing apparatus that includes a micro lens array comprising micro lenses, each of which corresponds to a plurality of pixels, and in which a detailed reference level in an effective pixel area can be acquired without adding a new constituent element.
According to the present invention, provided is an image capturing apparatus comprising: a photoelectric conversion unit that performs photoelectric conversion on incident light that has passed through an imaging lens and outputs an electric signal, the photoelectric conversion unit including a plurality of two-dimensionally arranged pixels; and a plurality of micro lenses that are formed between the imaging lens and the photoelectric conversion unit, the plurality of micro lenses respectively corresponding to a plurality of areas in the photoelectric conversion unit, each of the areas including a predetermined number of pixels, wherein among the pixels included in the plurality of areas, pixels whose receiving quantity of light is small in each area due to a characteristic of an optical system including the imaging lens and the micro lenses and a characteristic of the photoelectric conversion unit are configured as pixels for use in calculation of a reference value for the electric signals output from the photoelectric conversion unit.
According to the present invention, provided is an image capturing apparatus comprising: a photoelectric conversion unit that performs photoelectric conversion on incident light that has passed through an imaging lens and outputs an electric signal, the photoelectric conversion unit including a plurality of two-dimensionally arranged pixels; and a plurality of micro lenses that are formed between the imaging lens and the photoelectric conversion unit, the plurality of micro lenses respectively corresponding to a plurality of areas in the photoelectric conversion unit, each of the areas including a predetermined number of pixels, wherein among the pixels included in the plurality of areas, pixels that are arranged in an outermost region of each of the plurality of areas are configured as pixels for use in calculation of a reference value of an electric signal output from the photoelectric conversion unit.
Further features of the present invention will become apparent from the following description of exemplary embodiments (with reference to the attached drawings).
BRIEF DESCRIPTION OF THE DRAWINGS
The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention, and together with the description, serve to explain the principles of the invention.
FIG. 1 is a block diagram showing a schematic configuration of an image capturing apparatus according to an embodiment of the present invention;
FIG. 2 is a block diagram showing constituent elements in the periphery of an optical system of the image capturing apparatus according to the embodiment of the present invention;
FIGS. 3A and 3B are diagrams showing a schematic configuration of an image sensor according to the embodiment of the present invention;
FIGS. 4A and 4B are diagrams showing a pixel arrangement in the image sensor according to the embodiment of the present invention;
FIG. 5 is a ray tracing diagram from subjects in the image capturing apparatus according to the embodiment of the present invention;
FIG. 6 is a ray tracing diagram from subjects at different positions in the image capturing apparatus according to the embodiment of the present invention;
FIG. 7 is a diagram two-dimensionally showing ray tracing in the image capturing apparatus according to the embodiment of the present invention;
FIGS. 8A and 8B are diagrams illustrating a pixel arrangement in the image sensor according to the embodiment of the present invention; and
FIG. 9 is a diagram illustrating an example of a reference level calculation method according to the embodiment of the present invention.
 DESCRIPTION OF THE EMBODIMENTS
Exemplary embodiments of the present invention will be described in detail in accordance with the accompanying drawings. The dimensions, shapes and relative positions of the constituent parts shown in the embodiments should be changed as convenient depending on various conditions and on the structure of the apparatus adapted to the invention, and the invention is not limited to the embodiments described herein.
FIG. 1 is a block diagram showing a schematic configuration of an image capturing apparatus according to the present embodiment. In FIG. 1, reference numeral 101 denotes an optical system including an imaging lens, a diaphragm and a micro lens, and 102 is a mechanical shutter. An image sensor 103 converts incident light to an electric signal and includes a photoelectric conversion unit 104 that converts incident light to an electric signal and an amplifier circuit 105 that amplifies the electric signal. An analog signal processing circuit 106 performs analog signal processing on an image signal output from the image sensor 103. The analog signal processing circuit 106 includes a CDS circuit 107 that performs correlated double sampling, a signal amplifier 108 that amplifies an analog signal, a clamping circuit 109 that performs horizontal OB clamping, and an A/D converter 110 that converts an analog signal to a digital signal.
A timing signal generation circuit 111 generates a signal for operating the image sensor 103 and the analog signal processing circuit 106. A driving circuit 112 drives the optical system 101 and the mechanical shutter 102. A digital signal processing circuit 113 includes an image correction circuit 114 that performs necessary correction processing on image data, a signal amplifying circuit 115 that amplifies the digital signal that has been corrected by the image correction circuit 114, and an image processing circuit 116 that performs necessary image processing on image data. With these constituent elements, the digital signal processing circuit 113 performs necessary digital signal processing on the captured image data.
Reference numeral 117 is an image memory that stores the processed image data, 118 is a recording medium removable from the image capturing apparatus, and 119 is a recording circuit that records the image data that has undergone signal processing in the recording medium 118. Reference numeral 120 is an image display device that displays image data that has undergone signal processing, and 121 is a display circuit that displays an image on the image display device 120.
A system control unit 122 performs overall control on the image capturing apparatus. A non-volatile memory (ROM) 123 stores a program in which a control method executed by the system control unit 122 is written, control data, such as parameters and tables, for use in executing the program, and correction data such as defect addresses. The program, control data and correction data stored in the non-volatile memory 123, are transferred to and stored in a volatile memory (RAM) 124 which are used by the system control unit 122 when controlling the image capturing apparatus.
An image capturing operation of the image capturing apparatus having the above configuration will be described. Prior to the image capturing operation, at the start of the operation of the system control unit 122 such as when the image capturing apparatus is powered on, the necessary program, control data and correction data are transferred from the non-volatile memory 123 to the volatile memory 124 and stored. The program and data are used by the system control unit 122 when controlling the image capturing apparatus. Also, where necessary, additional programs and data are transferred from the non-volatile memory 123 to the volatile memory 124, or the system control unit 122 reads out data directly from the non-volatile memory 123 and uses the data.
First, in response to a control signal from the system control unit 122, the optical system 101 drives the optical system 101 such as a lens and forms on the image sensor 103 an image of the subject controlled so as to have an appropriate brightness. Next, in response to a control signal from the system control unit 122, the mechanical shutter 102 is driven such that the image sensor 103 is shielded from light along with the operation of the image sensor 103 so as to obtain the required exposure period when capturing a still image. At this time, in the case where the image sensor 103 has an electronic shutter function, the required exposure period may be secured by using electrical shutter function of the image sensor 103 together with the mechanical shutter 102. When capturing a moving image and when driven in live view, in response to a control signal from the system control unit 122, the mechanical shutter 102 is maintained in a full-open state such that the image sensor 103 is constantly exposed with light during image capturing.
The image sensor 103 is driven with a driving pulse based on an operation pulse generated by the timing signal generation circuit 111 that is controlled by the system control unit 122. The photoelectric conversion unit 104 converts the image of the subject into electric signals by using photoelectric conversion, and the amplifier circuit 105 multiplies an electric signal by a gain that is an amplification factor that has been set according to the quantity of incident light and outputs the product as an analog image signal.
In response to an operation pulse generated by the timing signal generation circuit 111 that is controlled by the system control unit 122, in the CDS circuit 107 of the analog signal processing circuit 106, clock synchronization noise is removed from the analog image signal output from the image sensor 103. Furthermore, the signal amplifier 108 multiplies the image signal by a gain that is an amplification factor that has been set according to the quantity of incident light, and the clamping circuit 109 clamps a signal output of a horizontal OB area as a reference voltage, and the signal is converted to a digital image signal by the A/D converter 110.
Next, in the digital signal processing circuit 113 that is controlled by the system control unit 122, processing is performed on the digital image signal output from the analog signal processing circuit 106. First, the image correction circuit 114 performs various types of image correction processing such as defect correction and dark shading correction on the image signal that has been converted to a digital signal. For correction to achieve a uniform dark level such as the dark shading correction, an output signal from a reference value calculation pixel (OB pixel) is used, which will be described later.
After that, the signal amplifying circuit 115 multiplies the signal by a gain that is an amplification factor that has been set according to the quantity of incident light, and the image processing circuit 116 performs various types of image processing including image processing such as color conversion, white balance and gamma correction, resolution conversion processing and image compression processing. Also, the image processing circuit 116 can perform, for example, re-focus processing by using the method disclosed by Ren. Ng et al. (processing for generating a plurality of images each focused at an arbitrary distance based on the electric signals output from the image sensor 103). At this time, the image memory 117 is used to temporarily store a digital image signal under signal processing or to store image data that is a digital image signal that has undergone signal processing.
The image data that has undergone signal processing in the digital signal processing circuit 113 and the image data stored in the image memory 117 are converted to data (for example, file system data having a hierarchical structure) suitable for the recording medium 118 by the recording circuit 119 and then recorded in the recording medium 118. Alternatively, the image data that has undergone resolution conversion processing in the digital signal processing circuit 113 is converted to a signal (for example, NTSC type analog signal or the like) suitable for the image display device 120 by the display circuit 121 and then displayed on the image display device 120, for example.
Here, in the digital signal processing circuit 113, in response to a control signal from the system control unit 122, the digital image signal may be output to the image memory 117 and the recording circuit 119 as image data, without performing signal processing. Also, if there is a request from the system control unit 122, the digital signal processing circuit 113 outputs a digital image signal generated during image processing and image data information to the system control unit 122. The image data information includes, for example, information regarding the spatial frequency of an image, the average value of a specific area, the amount of data of a compressed image and the like, or information extracted therefrom. Furthermore, if there is a request from the system control unit 122, the recording circuit 119 outputs information regarding the type, free space and the like of the recording medium 118 to the system control unit 122.
Next, a reproduction operation performed when image data is recorded in the recording medium 118 will be described. In response to a control signal from the system control unit 122, the recording circuit 119 reads out image data from the recording medium 118. Then, similarly, in response to a control signal from the system control unit 122, the digital signal processing circuit 113 performs image decompression processing on the read image data and stores the image data in the image memory 117 in the case where the read image data is a compressed image. Furthermore, the image data stored in the image memory 117 is subjected to resolution conversion processing by the digital signal processing circuit 113, thereafter converted to a signal suitable for the image display device 120 by the display circuit 121 and then displayed on the image display device 120.
FIG. 2 is a block diagram illustrating constituent elements in the periphery of the optical system of the image capturing apparatus according to the embodiment of the present invention.
In FIG. 2, an imaging lens 201 and a diaphragm 204 together constitute the optical system 101. A micro lens array 202 and a sensor array 203 are constituent elements of the image sensor 103. Reference numeral 205 denotes a subject. The other constituent elements shown in FIG. 2 are the same as those described in connection with FIG. 1, and thus a description thereof is not given here.
In a state in which the mechanical shutter 102 and the diaphragm 204 are open by the driving circuit 112, an image of the subject 205 is formed on the image sensor 103 by the imaging lens 201. Light signals incident on the image sensor 103 are further converged by each micro lens of the micro lens array 202, and incident on each pixel of the sensor array 203. The configurations of the micro lens array 202 and the sensor array 203 will be described later with reference to FIGS. 4A and 4B. The light signals incident on the sensor array 203 are subjected to photoelectric conversion in each pixel and output as electric signals. The processing after this is the same as that described with reference to FIG. 1.
FIGS. 3A and 3B are diagrams showing a schematic configuration of the image sensor 103 of the image capturing apparatus of the present embodiment. As shown in FIG. 3A, the image sensor 103 includes a pixel area 302 in which a plurality of pixels 301 are arranged in an array and peripheral circuitry. The peripheral circuitry includes a vertical shift register 303, a readout circuit 304, a horizontal shift register 305, an output amplifier 306, driving signal lines 307, vertical signal lines 308, and so forth. A driving pulse is sent out from the vertical shift register 303 through a driving signal line 307 for each of the pixels arranged in each row. For simplification of the drawing, the driving signal lines 307 are illustrated such that a single driving signal line is provided for each row, but in fact, a plurality of driving signal lines are connected for each row. Also, the pixels aligned in the same column are connected to the same vertical signal line 308. The pixel signals of a row selected by the vertical shift register 303 are output to the readout circuit 304 through the vertical signal lines 308, and sequentially output from the pixel signals of a column selected by the horizontal shift register 305 through the output amplifier 306. FIGS. 3A and 3B show a configuration in which the pixel area 302 has pixels 301 in 8 columns and 8 rows, but in fact, the pixel area 302 is formed with pixels in several thousands of columns and rows.
FIG. 3B is a circuit diagram showing a configuration of a pixel 301 shown in FIG. 3A. As shown in FIG. 3B, the pixel 301 includes a photodiode 401, a floating diffusion unit 402, an amplifier unit 403, a transfer switch 404, a reset switch 405 and a selection switch 406.
The photodiode 401 functions as a photoelectric conversion unit that receives light that is incident on the pixel 301 and generates a signal charge according to the quantity of the received light. The floating diffusion unit 402 functions as a charge voltage conversion unit that temporarily stores the charge generated by the photodiode 401 and transferred, and converts the held charge to a voltage signal. The transfer switch 404 is driven with a transfer pulse signal PTX and transfers the signal charge generated by the photodiode 401 to the floating diffusion unit 402. The reset switch 405 is driven with a reset pulse signal PRES, and resets the potential of the floating diffusion unit 402 to a reference potential VDD. The amplifier unit 403 is a source follower MOS transistor, and amplifies the voltage signal based on the charge held by the floating diffusion unit 402 and outputs the signal as a pixel signal. The selection switch 406 is driven with a vertical selection pulse signal PSEL, and outputs the pixel signal amplified by the amplifier unit 403 to the vertical signal line 308.
FIGS. 4A and 4B are diagrams showing a relative arrangement between the pixels of the sensor array 203 and the micro lens array 202 constituting the image sensor 103 of the present embodiment. In FIG. 4A, reference numeral 501 denotes a recording pixel unit corresponding to a single pixel in a reconfigured image, and 502 denotes pixels arranged in 8 columns and 8 rows that constitute the recording pixel unit 501, each pixel provided for each photoelectric conversion unit and corresponding to the pixel 301 described with reference to FIGS. 3A and 3B. Reference numeral 503 denotes micro lenses arranged in one to one correspondence with the recording pixel units 501.
The present embodiment will be described using, as shown in FIGS. 4A and 4B, a sensor array 203 in which recording pixel units 501 are arranged in 4 columns and 4 rows, each recording pixel unit 501 including pixels 502 arranged in 8 columns and 8 rows. FIG. 4B is an enlarged diagram showing a recording pixel unit 501. In the following description of the present embodiment, as shown in FIG. 4B, 64 pixels are labeled with numbers from a00 to a77.
FIG. 5 is a diagram showing ray tracing of light rays that are incident on the image capturing apparatus of the present embodiment from the subjects at various distances.
In FIG. 5, a subject 701 a is the subject located at a position at which an image of the subject 701 a is formed on a plane A including the micro lens array 202 by the imaging lens 201. Among the light rays from the subject 701 a, those that pass through the outermost periphery of the imaging lens 201 and are incident on the sensor array 203 via the micro lens on the optical axis are indicated by solid lines.
A subject 701 b is the subject located further away from the subject 701 a as viewed from the imaging lens 201. An image of the subject 701 b formed by the imaging lens 201 is formed on a plane B that is closer to the imaging lens 201 than the plane A including the micro lens array 202 is. Among the light rays from the subject 701 b, those that pass through the outermost periphery of the imaging lens 201 and are incident on the sensor array 203 via the micro lens on the optical axis are indicated by broken lines.
A subject 701 c is the subject located closer to the imaging lens 201 than the subject 701 a is, as viewed from the imaging lens 201. An image of the subject 701 c formed by the imaging lens 201 is formed on a plane C that is further away from the imaging lens 201 than the plane A including the micro lens array 202 is. Among the light rays from the subject 701 c, those that pass through the outermost periphery of the imaging lens 201 and are incident on the sensor array 203 via the micro lens on the optical axis are indicated by dashed and two-dotted lines.
As indicated by the ray tracing of each light ray shown in FIG. 5, light is incident on different pixels of the sensor array 203 depending on the distance from the imaging lens 201 to the subject 701. Based on this fact, with the image capturing apparatus having the above configuration, image signals after image capturing are reconfigured, whereby images focused on the subjects at various distances can be generated.
FIG. 6 is a diagram showing ray tracing of light rays that are incident on the image capturing apparatus of the present embodiment from the subjects at different positions.
In FIG. 6, a subject 801 a is the subject located at a position at which an image of the subject 801 a is formed on a micro lens 802 a on the plane A including the micro lens array 202 by the imaging lens 201. Among the light rays from the subject 801 a those that pass through the outermost periphery of the imaging lens 201 and are incident on the sensor array 203 via the micro lens 802 a are indicated by solid lines. These light rays arrive at a recording pixel unit 803 a on the sensor array 203.
A subject 801 b is the subject located at a position that is away from the imaging lens 201 at the same distance as the subject 801 a, as viewed from the imaging lens 201, and at which an image of the subject 801 b is formed on a micro lens 802 b on the plane A including the micro lens array 202 by the imaging lens 201. Among the light rays from the subject 801 b, those that pass through the outermost periphery of the imaging lens 201 and are incident on the sensor array 203 via the micro lens 802 b are indicated by broken lines. These light rays arrive at a recording pixel unit 803 b via the sensor array 203.
In the embodiment of the image capturing apparatus according to the present invention, in order to make the most effective use of the pixels of the sensor array 203, the micro lens array 202 is arranged such that the F number, which is one of the characteristics of a lens, matches between the imaging lens 201 and the micro lens. If it is assumed here that the opening diameter of the imaging lens 201 is denoted by D, the distance from the imaging lens 201 to the micro lens array 202 is denoted by F, the length of one side of the recording pixel unit 501 is denoted by d, and the distance from the micro lens array 202 to the sensor array 203 is denoted by f, then the following relationship holds:
D/F=d/f.
In the case of such an arrangement, among the light rays from the subject 801 a, those that pass through the outermost periphery of the imaging lens 201 and are incident on the recording pixel unit 803 a of the sensor array 203 via the micro lens 802 a arrive at a position in a distance of d/2 from the center of the recording pixel unit 803 a as indicated by the solid lines in FIG. 6. Likewise, among the light rays from the subject 801 b, those that pass through the outermost periphery of the imaging lens 201 and are incident on the recording pixel unit 803 b of the sensor array 203 via the micro lens 802 b arrive at a position in a distance of d/2 from the center of the recording pixel unit 803 b as indicated by the broken lines in FIG. 6. At this time, any of the light rays does not arrive at a position away from the recording pixel unit corresponding to each micro lens, and therefore a situation can be prevented in which reconfiguration (refocusing) of the image is not possible due to the overlapping of light signals on the sensor array.
FIG. 7 is a diagram two-dimensionally showing ray tracing of incident light rays in the image capturing apparatus according to the present embodiment. In FIG. 7, reference numeral 902 denotes one of the micro lenses of the micro lens array 202, and 903 denotes the sensor array surface.
A light ray passing through a center p0 of the diaphragm 204 is incident on a point p0′ at which the sensor array surface 903 and the optical axis intersect. A light ray from the subject located at a position at which an image of the subject is formed on a plane including the micro lens array 202 by the imaging lens 201 that passes through a point p1 on the outer periphery of the aperture of the diaphragm 204 is incident on p1′ on the sensor array surface 903. Likewise, light rays passing through other points p2 and p3 on the outer periphery of the aperture of the diaphragm 204 are respectively incident on points p2′ and p3′ on the sensor array surface 903.
As described above, theoretically, among light rays from the subject located at a position at which an image of the subject is formed on a plane including the micro lens array 202 by the imaging lens 201, all of those passing through the diaphragm 204 and the micro lens 902 are incident on the inside of a circle 904 on the sensor array surface 903 and do not arrive at the outside of the circle 904. Accordingly, no light rays, excluding a small quantity of stray light and ghost light, arrive at pixels a00, a07, a70 and a77 on the sensor array surface 903.
The same is true for light rays from the subject located at a position other than the position at which an image of the subject is formed on a plane including the micro lens array 202 by the imaging lens 201. In other words, the quantity of light rays that are received by the pixels a00, a07, a70 and a77 on the sensor array surface 903 and are effective in the generation of a refocused image is smaller than that of the pixels around the optical axis of the micro lens, and thus there is a small influence on the subsequent re-focus processing.
Specifically, the pixels a00, a07, a70 and a77 located at four corners of the recording pixel unit 501 have a relatively small quantity of received light due to the characteristics (i.e., D, F, d, f and the like described above) of the image sensor 103 and the optical system including the imaging lens 201 and the micro lens array 202. Accordingly, it can be said that these pixels are ineffective for re-focus processing.
FIGS. 8A and 8B are diagrams showing a pixel arrangement in the image capturing apparatus of the present embodiment. FIG. 8A is a layout diagram in the sensor array 203 described with reference to FIG. 4A, and FIG. 8B is an arrangement diagram of a recording pixel unit 501 and pixels 502, which are constituent elements of the sensor array 203 described with reference to FIG. 4B.
In FIG. 8B, the pixels a00, a07, a70 and a77 located at four corners of the recording pixel unit 501 are, as described with reference to FIG. 7, pixels that are arranged in an area that is not suitable for re-focus processing, which is performed in a later stage, and therefore these pixels are used as OB pixels. In FIGS. 8A and 8B, the pixels that are used as OB pixels are shaded in gray and the pixels that are used as effective pixels are shown in white. The four pixels that are used as OB pixels are shielded from light with aluminum wiring in order to enhance the light blocking effect.
FIG. 9 is a diagram illustrating an example of a reference level calculation method in the image capturing apparatus of the present embodiment. The method for calculating a reference level of a pixel amn by using output signal values of the OB pixels a00, a07, a70 and a77 in the pixel arrangement shown in FIG. 8B will be described with reference to FIG. 9.
This calculation method is a method in which calculation is performed assuming that the reference levels of the OB pixels transition linearly in proportion to the distance from the OB pixel. The output signal values of the OB pixels a00, a07, a70 and a77 are respectively expressed as B(0, 0), B(0, 7), B(7, 0) and B(7, 7). A pixel a0 n is the pixel located at the position at which the distance between the OB pixel a00 and the OB pixel a07 is internally divided in n:7-n. Accordingly, the reference level B(0, n) of the pixel a0 n is represented as follows:
B(0,n)=(1−n/7)·B(0,0)+n/7·B(0,7).
Similarly, a pixel a7 n is the pixel located at the position at which the distance between the OB pixel a70 and the OB pixel a77 is internally divided in n:7−n, and therefore the reference level B(7, n) of the pixel a7 n is represented as follows:
B(7,n)=(1−n/7)·B(7,0)+n/7·B(7,7).
The same calculation applies to the vertical direction. The pixel amn is the pixel located at a position at which the distance between the OB pixel a0 n and the OB pixel a7 n is internally divided in m:7−m, and therefore the reference level B(m, n) of the pixel amn is represented as follows:
B ⁡ ( m , n ) = ( 1 - m / 7 ) · ( 1 - n / 7 ) · B ⁡ ( 0 , 0 ) + ( 1 - m / 7 ) · n / 7 · B ⁡ ( 0 , 7 ) + m / 7 · ( 1 - n / 7 ) · B ⁡ ( 7 , 0 ) + m / 7 · n / 7 · B ⁡ ( 7 , 7 ) .
As described above, with the image capturing apparatus of the present embodiment, by arranging OB pixels within an effective pixel area, it is possible to acquire reference levels that are finer and more accurate than the conventional technology. Accordingly, by performing dark level correction or the like by using such reference levels, it is possible to achieve an image capturing apparatus that can provide images of good image quality.
The foregoing has described the embodiment of the image capturing apparatus according to the present invention with reference to FIGS. 1 to 9, but the present invention is not limited thereto, and the present invention can take various forms.
For example, in the pixel configuration of the embodiment of the image capturing apparatus according to the present invention, in order to facilitate the understanding of the pixel structure, a single recording pixel unit is configured with 8-by-8 pixels. However, the present invention is not limited thereto, and a single recording pixel unit may be configured with any number of pixels with any shape.
Also, in the pixel configuration of the embodiment of the image capturing apparatus according to the present invention, the pixels that are located at specific positions of the outermost region and whose receiving quantity of light rays, passed through the diaphragm 204 and the micro lens 902, that can be used in re-focus processing later are used as OB pixels. However, the present invention is not limited thereto, and more pixels may be used as OB pixels such as using a configuration in which the pixels whose receiving quantity of light rays that can be used for re-focus processing and the pixels surrounding such pixels are used OB pixels. In this case, it is desirable that the pixels used as OB pixels are shield from light with aluminum such that light rays do not arrive at the pixels.
The pixel configuration of the embodiment of the image capturing apparatus according to the present invention was described taking an example in which the OB pixels are shielded from light with aluminum wiring, but the present invention is not limited thereto. The OB pixels may be shielded from light by any other methods. For example, a configuration may be possible in which a black color filter is disposed in front of the OB pixels so as to produce the light blocking effect.
Alternatively, in the case of using the OB pixels of the present invention for shading correction of a circuit system, for example, instead of using the light blocking member as described above, it is possible to use a configuration in which the charges generated in the photoelectric conversion units are not transferred or a configuration that does not include the photoelectric conversion units.
To describe it more specifically, the pixels that are used as OB pixels may be configured such that by causing the transfer switch 404 shown in FIG. 3B to be constantly open, the signal charge generated by the photodiode 401 is not transferred. It is also possible to use a configuration in which the photodiode 401 and the transfer switch 404 are not provided and a reset level corresponding to the reference potential VDD is constantly read out.
Also, the method for calculating a reference level for each pixel was described with reference to FIG. 9 taking an example in which a reference value of each pixel is calculated based on the distance from four OB pixels. However, the present invention is not limited thereto. For example, it is possible to use a configuration in which with respect to the OB pixels 502 belonging to a recording pixel unit 501, the average value of the output values of four OB pixels belonging to the recording pixel unit 501 is set uniformly as the reference level of the pixels of the recording pixel unit 501. With this configuration, the amount of computation can be reduced.
Also, in the embodiment of the present invention described with reference to FIG. 1, the digital signal processing circuit 113, which is a constituent element of the image capturing apparatus, performs image processing such as image reconfiguration, but the image processing is not necessarily performed inside the image capturing apparatus. Specifically, it is possible to use a configuration in which an image processing unit is provided in an apparatus separate from the image capturing apparatus such as a PC (personal computer), captured data obtained by the image capturing apparatus is transferred to the PC, and image processing is performed in the PC.
While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.
This application claims the benefit of Japanese Patent Application No. 2012-005663, filed on Jan. 13, 2012 which is hereby incorporated by reference herein in its entirety.

Claims (9)

What is claimed is:
1. An image capturing apparatus comprising:
a photoelectric conversion unit that performs photoelectric conversion on incident light that has passed through an imaging lens and outputs an electric signal, the photoelectric conversion unit including a plurality of two-dimensionally arranged pixels;
a plurality of micro lenses that are formed between the imaging lens and the photoelectric conversion unit, the plurality of micro lenses respectively corresponding to a plurality of areas in the photoelectric conversion unit, each of the areas including a predetermined number of pixels, wherein the pixels included in each of the plurality of areas include effective pixels on which a relatively large quantity of light is incident for generating an image signal, and pixels on which a relatively small quantity of light is incident due to a characteristic of an optical system including the imaging lens and the micro lenses and a characteristic of the photoelectric conversion unit,
a reference value calculation unit that calculates for each of the plurality of areas a reference value for processing the image signal output from the effective pixels in the same area for which the reference value is calculated using an electric signal of the pixels on which a relatively small quantity of light is incident in the area,
wherein the pixels on which a relatively small quantity of light is incident are shielded from light, have a configuration in which the electric signal generated by photoelectric conversion is not output, or do not have a configuration that performs photoelectric conversion.
2. The image capturing apparatus according to claim 1, wherein the pixels for use in calculation of the reference value are arranged in an outermost region of each of the plurality of areas.
3. The image capturing apparatus according to claim 1, further comprising a refocused image generation unit that performs re-focus processing for generating an image focused at an arbitrary distance based on the electric signal output from the photoelectric conversion unit.
4. The image capturing apparatus according to claim 1, wherein the characteristic of the optical system is F number of the micro lens, and the F number matches between the imaging lens and the micro lens.
5. An image capturing apparatus comprising:
a photoelectric conversion unit that performs photoelectric conversion on incident light that has passed through an imaging lens and outputs an electric signal, the photoelectric conversion unit including a plurality of two-dimensionally arranged pixels;
a plurality of micro lenses that are formed between the imaging lens and the photoelectric conversion unit, the plurality of micro lenses respectively corresponding to a plurality of areas in the photoelectric conversion unit, each of the areas including a predetermined number of pixels, and wherein the pixels included in each of the plurality of areas include pixels that are arranged in an outermost region of each of the plurality of areas and effective pixels that are arranged in a region except for the outermost region of each of the plurality of areas for generating of the image signal; and
a reference value calculation unit that calculates for each of the plurality of areas a reference value for processing the image signal output from the effective pixels in the same area for which the reference value is calculated using an electric signal of the pixels that are arranged in the outermost region in the area,
and wherein the pixels that are arranged in the outermost region are shielded from light, have a configuration in which the electric signal generated by photoelectric conversion is not output, or do not have a configuration that performs photoelectric conversion.
6. An image capturing apparatus comprising:
a photoelectric conversion unit that performs photoelectric conversion on incident light that has passed through an imaging lens and outputs an electric signal, the photoelectric conversion unit including a plurality of two-dimensionally arranged pixels; and
a plurality of micro lenses that are formed between the imaging lens and the photoelectric conversion unit, the plurality of micro lenses respectively corresponding to a plurality of areas in the photoelectric conversion unit, each of the areas including a predetermined number of pixels,
wherein among the pixels included in the plurality of areas, pixels on which a relatively large quantity of light is incident are configured as effective pixels for generating an image signal, and pixels on which a relatively small quantity of light is incident due to a characteristic of an optical system including the imaging lens and the micro lenses and a characteristic of the photoelectric conversion unit are configured as pixels for use in calculation of a reference value for the image signals output from the effective pixels of the photoelectric conversion unit,
and wherein the pixels on which a relatively small quantity of light is incident are shielded from light, have a configuration in which the electric signal generated by photoelectric conversion is not output, or do not have a configuration that performs photoelectric conversion, and
wherein an F number of each of the micro lenses is equal to an F number of the imaging lens.
7. The image capturing apparatus according to claim 6, wherein the pixels for use in calculation of the reference value are arranged in an outermost region of each of the plurality of areas.
8. The image capturing apparatus according to claim 6, further comprising a refocused image generation unit that performs re-focus processing for generating an image focused at an arbitrary distance based on the electric signal output from the photoelectric conversion unit.
9. An image capturing apparatus comprising:
a photoelectric conversion unit that performs photoelectric conversion on incident light that has passed through an imaging lens and outputs an electric signal, the photoelectric conversion unit including a plurality of two-dimensionally arranged pixels; and
a plurality of micro lenses that are formed between the imaging lens and the photoelectric conversion unit, the plurality of micro lenses respectively corresponding to a plurality of areas in the photoelectric conversion unit, each of the areas including a predetermined number of pixels,
wherein among the pixels included in the plurality of areas, pixels that are arranged in an outermost region of each of the plurality of areas are configured as pixels for use in calculation of a reference value of an image signal output from effective pixels of the photoelectric conversion unit, and pixels that are arranged in a region except for the outermost region of each of the plurality of areas are configured as the effective pixels for generating of the image signal,
and wherein the pixels on which a relatively small quantity of light is incident are shielded from light, have a configuration in which the electric signal generated by photoelectric conversion is not output, or do not have a configuration that performs photoelectric conversion, and
wherein an F number of each of the micro lenses is equal to an F number of the imaging lens.
US13/738,198 2012-01-13 2013-01-10 Image capturing apparatus that performs photoelectric conversion on incident light that has passed through an imaging lens and outputs an electric signal Expired - Fee Related US9191560B2 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2012005663A JP5963448B2 (en) 2012-01-13 2012-01-13 Imaging device
JP2012-005663 2012-01-13

Publications (2)

Publication Number Publication Date
US20130182170A1 US20130182170A1 (en) 2013-07-18
US9191560B2 true US9191560B2 (en) 2015-11-17

Family

ID=48756371

Family Applications (1)

Application Number Title Priority Date Filing Date
US13/738,198 Expired - Fee Related US9191560B2 (en) 2012-01-13 2013-01-10 Image capturing apparatus that performs photoelectric conversion on incident light that has passed through an imaging lens and outputs an electric signal

Country Status (4)

Country Link
US (1) US9191560B2 (en)
JP (1) JP5963448B2 (en)
KR (1) KR101582134B1 (en)
CN (1) CN103209301A (en)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160316218A1 (en) * 2014-04-24 2016-10-27 Lytro Inc. Compression of Light Field Images
US20170280136A1 (en) * 2014-04-24 2017-09-28 Lytro, Inc. Predictive light-field compression
US20170366768A1 (en) * 2013-12-13 2017-12-21 Bio-Rad Laboratories, Inc. Digital imaging with masked pixels
US11031424B2 (en) 2018-07-25 2021-06-08 Samsung Electronics Co., Ltd. Image sensor with selective light-shielding for reference pixels

Families Citing this family (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2015072982A (en) * 2013-10-02 2015-04-16 株式会社ニコン Image sensor and imaging apparatus
CN104580877B (en) * 2013-10-29 2018-03-06 华为技术有限公司 The device and method that image obtains
JP6555863B2 (en) * 2013-12-25 2019-08-07 キヤノン株式会社 IMAGING DEVICE AND IMAGING DEVICE CONTROL METHOD
FR3040798B1 (en) * 2015-09-08 2018-02-09 Safran PLENOPTIC CAMERA
US11030724B2 (en) 2018-09-13 2021-06-08 Samsung Electronics Co., Ltd. Method and apparatus for restoring image
WO2021212486A1 (en) * 2020-04-24 2021-10-28 达闼机器人有限公司 Lens module and object detection device
US11616924B2 (en) * 2020-10-30 2023-03-28 SmartSens Technology (HK) Co., Ltd. CMOS image sensor with image black level compensation and method
CN112600994B (en) * 2020-12-02 2023-04-07 达闼机器人股份有限公司 Object detection device, method, storage medium, and electronic apparatus
CN113419394B (en) * 2021-06-23 2022-12-20 合肥联宝信息技术有限公司 Lens assembly, camera and imaging method

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20060262209A1 (en) * 2005-05-17 2006-11-23 Takafumi Kishi Image sensing device and control method therefor
US20060268358A1 (en) * 2005-05-30 2006-11-30 Fujitsu Limited Image signal processing unit and method
US20100277629A1 (en) * 2009-05-01 2010-11-04 Samsung Electronics Co., Ltd. Photo detecting device and image pickup device and method thereon
KR20100119712A (en) 2009-05-01 2010-11-10 삼성전자주식회사 Photo detecting device and image pickup device
US20110069189A1 (en) * 2008-05-20 2011-03-24 Pelican Imaging Corporation Capturing and processing of images using monolithic camera array with heterogeneous imagers
CN102090053A (en) 2008-07-10 2011-06-08 佳能株式会社 Image pickup apparatus and its control method
CN102292974A (en) 2009-03-17 2011-12-21 佳能株式会社 Image capturing apparatus
CN102318335A (en) 2009-02-06 2012-01-11 佳能株式会社 Image capturing apparatus

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7452743B2 (en) * 2005-09-01 2008-11-18 Aptina Imaging Corporation Microelectronic imaging units and methods of manufacturing microelectronic imaging units at the wafer level
JP4875691B2 (en) * 2007-12-17 2012-02-15 オリンパスメディカルシステムズ株式会社 Imaging device, image display device, and image display system
JP2010093759A (en) * 2008-10-10 2010-04-22 Canon Inc Imaging sensor and imaging apparatus
JP2010147785A (en) * 2008-12-18 2010-07-01 Fujifilm Corp Solid-state image sensor and imaging apparatus, and image correction method of the same
JP4706882B2 (en) * 2009-02-05 2011-06-22 ソニー株式会社 Imaging device
US8872953B2 (en) * 2009-10-30 2014-10-28 Sony Corporation Solid-state imaging device, manufacturing method thereof, camera, and electronic device

Patent Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20060262209A1 (en) * 2005-05-17 2006-11-23 Takafumi Kishi Image sensing device and control method therefor
US20060268358A1 (en) * 2005-05-30 2006-11-30 Fujitsu Limited Image signal processing unit and method
CN1874429A (en) 2005-05-30 2006-12-06 富士通株式会社 Image signal processing unit and method
US20110069189A1 (en) * 2008-05-20 2011-03-24 Pelican Imaging Corporation Capturing and processing of images using monolithic camera array with heterogeneous imagers
CN102037717A (en) 2008-05-20 2011-04-27 派力肯成像公司 Capturing and processing of images using monolithic camera array with hetergeneous imagers
CN102090053A (en) 2008-07-10 2011-06-08 佳能株式会社 Image pickup apparatus and its control method
CN102318335A (en) 2009-02-06 2012-01-11 佳能株式会社 Image capturing apparatus
CN102292974A (en) 2009-03-17 2011-12-21 佳能株式会社 Image capturing apparatus
US20100277629A1 (en) * 2009-05-01 2010-11-04 Samsung Electronics Co., Ltd. Photo detecting device and image pickup device and method thereon
KR20100119712A (en) 2009-05-01 2010-11-10 삼성전자주식회사 Photo detecting device and image pickup device

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
Ren Ng et al. "Light Field Photography with a Hand-Held Plenoptic Camera", (Stanford Tech Report CTSR Feb. 2005) 11 pages.
The above references were cited in a Jul. 1, 2015 Chinese Office Action, a copy of which is enclosed with an English Translation, that issued in Chinese Patent Application No. 201310011240.7.
The above references were cited in a Mar. 26, 2015 Korean Office Action, which is enclosed without an English Translation, that issued in Korean Patent Application No. 10-2013-0001874.

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170366768A1 (en) * 2013-12-13 2017-12-21 Bio-Rad Laboratories, Inc. Digital imaging with masked pixels
US10326952B2 (en) * 2013-12-13 2019-06-18 Bio-Rad Laboratories, Inc. Digital imaging with masked pixels
US20160316218A1 (en) * 2014-04-24 2016-10-27 Lytro Inc. Compression of Light Field Images
US20170280136A1 (en) * 2014-04-24 2017-09-28 Lytro, Inc. Predictive light-field compression
US10038909B2 (en) * 2014-04-24 2018-07-31 Google Llc Compression of light field images
US10531082B2 (en) * 2014-04-24 2020-01-07 Google Llc Predictive light-field compression
US11031424B2 (en) 2018-07-25 2021-06-08 Samsung Electronics Co., Ltd. Image sensor with selective light-shielding for reference pixels

Also Published As

Publication number Publication date
CN103209301A (en) 2013-07-17
JP5963448B2 (en) 2016-08-03
JP2013145981A (en) 2013-07-25
KR101582134B1 (en) 2016-01-04
US20130182170A1 (en) 2013-07-18
KR20130083847A (en) 2013-07-23

Similar Documents

Publication Publication Date Title
US9191560B2 (en) Image capturing apparatus that performs photoelectric conversion on incident light that has passed through an imaging lens and outputs an electric signal
JP6041495B2 (en) Imaging apparatus and defective pixel determination method
JP5128889B2 (en) Solid-state imaging device and solid-state imaging system using the same
US20100259663A1 (en) Image pickup apparatus
JP5013811B2 (en) Imaging apparatus and correction method
JP2006197393A (en) Solid-state imaging device, driving method thereof and camera
JP2010062638A (en) Image capturing apparatus
US9392193B2 (en) Image pickup apparatus, control method therefor and image pickup system
JP6053329B2 (en) Imaging apparatus and control method thereof
JP5917158B2 (en) Imaging apparatus, control method thereof, and imaging system
JP2010028488A (en) Imaging apparatus and control method for the same
JP5627728B2 (en) Imaging apparatus and imaging system
JP2014107739A (en) Imaging device and control method therefor
JP2020057883A (en) Imaging apparatus
JP2016103513A (en) Imaging element and imaging device
JP5222029B2 (en) IMAGING DEVICE, IMAGING SYSTEM, AND IMAGING DEVICE CONTROL METHOD
JP2007143067A (en) Image sensing device and image sensing system
KR20170077784A (en) Solid-state image pickup element and image pickup apparatus
JP6701269B2 (en) Image processing apparatus and image processing method
JP6046912B2 (en) Imaging apparatus and control method thereof
JP5789109B2 (en) Solid-state imaging device and imaging apparatus
JP6389693B2 (en) Imaging apparatus, control method therefor, program, and storage medium
JP2013106231A (en) Imaging apparatus
JP2018093301A (en) Image pick-up device and control method of the same
JP2018019268A (en) Imaging apparatus, imaging system, and driving method of imaging apparatus

Legal Events

Date Code Title Description
AS Assignment

Owner name: CANON KABUSHIKI KAISHA, JAPAN

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:SUZUKI, SATOSHI;REEL/FRAME:030193/0070

Effective date: 20121225

STCF Information on status: patent grant

Free format text: PATENTED CASE

MAFP Maintenance fee payment

Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

Year of fee payment: 4

FEPP Fee payment procedure

Free format text: MAINTENANCE FEE REMINDER MAILED (ORIGINAL EVENT CODE: REM.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

LAPS Lapse for failure to pay maintenance fees

Free format text: PATENT EXPIRED FOR FAILURE TO PAY MAINTENANCE FEES (ORIGINAL EVENT CODE: EXP.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

STCH Information on status: patent discontinuation

Free format text: PATENT EXPIRED DUE TO NONPAYMENT OF MAINTENANCE FEES UNDER 37 CFR 1.362

FP Lapsed due to failure to pay maintenance fee

Effective date: 20231117