EP3217660A1 - Dispositif de traitement d'image - Google Patents

Dispositif de traitement d'image Download PDF

Info

Publication number
EP3217660A1
EP3217660A1 EP17167696.8A EP17167696A EP3217660A1 EP 3217660 A1 EP3217660 A1 EP 3217660A1 EP 17167696 A EP17167696 A EP 17167696A EP 3217660 A1 EP3217660 A1 EP 3217660A1
Authority
EP
European Patent Office
Prior art keywords
polarization
pixel
pixels
color
polarizer
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
EP17167696.8A
Other languages
German (de)
English (en)
Other versions
EP3217660B1 (fr
Inventor
Katsuhiro Kanamori
Satoshi Sato
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Panasonic Intellectual Property Management Co Ltd
Original Assignee
Panasonic Intellectual Property Management Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Panasonic Intellectual Property Management Co Ltd filed Critical Panasonic Intellectual Property Management Co Ltd
Publication of EP3217660A1 publication Critical patent/EP3217660A1/fr
Application granted granted Critical
Publication of EP3217660B1 publication Critical patent/EP3217660B1/fr
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/80—Camera processing pipelines; Components thereof
    • H04N23/84—Camera processing pipelines; Components thereof for processing colour signals
    • H04N23/843—Demosaicing, e.g. interpolating colour pixel values
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/10—Circuitry of solid-state image sensors [SSIS]; Control thereof for transforming different wavelengths into image signals
    • H04N25/11—Arrangement of colour filter arrays [CFA]; Filter mosaics
    • H04N25/13—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements
    • H04N25/134—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements based on three different wavelength filter elements
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N2209/00—Details of colour television systems
    • H04N2209/04—Picture signal generators
    • H04N2209/041—Picture signal generators using solid-state devices
    • H04N2209/042—Picture signal generators using solid-state devices having a single pick-up sensor
    • H04N2209/045—Picture signal generators using solid-state devices having a single pick-up sensor using mosaic colour filter
    • H04N2209/046—Colour interpolation to calculate the missing colour values

Definitions

  • the present invention relates to an image processing apparatus that allows the user to obtain both color information and polarization information alike from the object.
  • an active sensor for projecting either a laser beam or a light beam emitted from an LED onto the object or a rangefinder system such as a differential stereo vision system is needed.
  • a sensor or system is not just bulky but also imposes various restrictions.
  • such a sensor or system allows a distance of at most several meters between the camera and the object.
  • such a sensor or system cannot be used unless the object is a solid and bright diffusive object. Under such a restriction, the sensor or system cannot be used to shoot an object located at a distance outdoors or take a close up photo of a person with his or her hair and clothes shot as beautifully as possible.
  • polarization may be used according to some conventional technique. This technique takes advantage of the fact that the light reflected (which may be either specular reflected or diffuse reflected) from an object irradiated with non-polarized natural light will have various types of partial polarization due to geometrical factors such the surface direction and the viewpoint. To collect those sorts of information, however, the degrees of partial polarization of the respective pixels of the object should be obtained in the form of polarized images.
  • Non-Patent Document No. 1 discloses a camera that captures a moving picture while controlling a liquid crystal polarizer that is arranged in front of the lens of the camera. If such a camera is used, a color moving picture and information about the degrees of partial polarization of the object can be obtained in the form of images. However, this camera cannot obtain a single piece of polarization information until several frames are shot with the polarization main axes of the polarizer changed by applying a voltage to the liquid crystal layer. That is why a time lag of several frames is inevitable between the color image and the polarized image. In other words, these two images cannot be obtained at the same time. This technique is equivalent to a technique for rotating a polarizer with some mechanism.
  • Non-Patent Documents Nos. 2 and 3 disclose that a patterned polarizer with multiple different polarization main axes is spatially arranged in an image capture device in order to obtain light intensity images and images representing the degrees of partial polarization of the object.
  • the patterned polarizer either a photonic crystal or a structure birefringence wave plate array may be used. According to these techniques, however, just a monochrome image and a polarized image can be obtained at the same time.
  • Patent Document No. 1 teaches arranging a polarization filter for some of G (green) pixels in a Bayer color mosaic, thereby giving polarization property to a part of an image capture device and obtaining a color image and polarization information at the same time. According to this technique, an image with reduced specular reflection components is obtained from a color image. However, since a difference between two different polarization pixels is simply calculated, information about the degree of partial polarization of the object cannot be obtained perfectly.
  • a color image is controlled with parts of polarization information added to the color image, thereby reducing the specular reflection.
  • pixels are arranged as a Bayer mosaic as shown in FIG. 37 , in which G1 and G2 both denote G pixels and in which polarizer units are arranged at the G2 locations but are not arranged at the G1 locations.
  • R1 and B1 denote R (red) pixels and B (blue) pixels, respectively, and no polarizer units are provided for these pixel locations. That is to say, in this pixel arrangement, only the G2 pixels function as polarization pixels.
  • the difference in light intensity between the G1 and G2 pixel locations is multiplied by a constant and the product is added to non-polarization pixels R1, G1 and B1.
  • the color intensity value can be controlled.
  • polarizer units are arranged at the G2 pixel locations, and therefore, accurate G light intensity cannot be obtained.
  • the interpolated color image will have some degree of color difference.
  • the partial polarization information obtained is so imperfect that it is not clear what degree of partial polarization has been produced in what direction at each pixel location. That is to say, a perfect polarized image cannot be obtained.
  • the present invention has an object of providing an apparatus and method for image processing that can obtain not only a color image but also a polarized image representing the direction and degree of polarization at the same time.
  • Another object of the present invention is to provide a patterned polarizer that can be used effectively in such an image processing apparatus and a single-chip image capture device including such a patterned polarizer.
  • An image processing apparatus includes: a color and polarization obtaining section including a single-chip color image capture device that has a color mosaic filter and a patterned polarizer in which a number of polarizer units, having polarization transmission planes defining mutually different angles, are provided for multiple pixels of the same color in the color mosaic filter; a polarization information processing section for approximating, as a sinusoidal function, a relation between the intensities of light rays that have been transmitted through the polarizer units for the multiple pixels of the same color and the angles of the polarization transmission planes of the polarizer units; and a color mosaic interpolation section for generating a color intensity image by performing color intensity interpolation and getting a color intensity that cannot be obtained at a pixel of interest.
  • the patterned polarizer has a structure in which four types of polarizer units, having polarization transmission planes defining mutually different angles, are arranged adjacent to each other.
  • the color mosaic filter has a structure in which pixels in a single particular color are arranged in a checkerboard pattern, and the polarizer units are provided for those pixels that are arranged in the checkerboard pattern.
  • the particular color is G (green) among R (red), G (green) and B (blue).
  • An image processing method includes the steps of: measuring the intensities of respective pixels using a single-chip color image capture device, in which polarizer units, having polarization transmission planes that define mutually different angles, are provided for multiple pixels in a single particular color included in a color pixel arrangement; obtaining polarization information based on the intensities that have been measured in the multiple pixels in the single particular color; and performing a spatial interpolation on a color intensity using the intensities in the single particular color and the two colors other than the single particular color, thereby generating a color intensity image.
  • the color mosaic filter has a structure in which pixels in a single particular color are arranged in a checkerboard pattern.
  • Four types of polarizer units, having polarization transmission planes defining mutually different angles, are arranged adjacent to each other for the pixels that are arranged in the checkerboard pattern, while at least two types of polarizer units, having polarization transmission planes defining mutually different angles, are arranged for the pixels in the other colors.
  • the particular color is G (green) among R (red), G (green) and B (blue).
  • the color mosaic interpolation section calculates the average of intensities that have been measured in pixels of the same color, for which (180/ ⁇ ) types of polarizer units are provided in the neighborhood of the pixel of interest, thereby canceling the effect of polarization and performing interpolation on the color intensity.
  • the (180/ ⁇ ) types of polarizer units have polarization transmission planes that define angles that are different from each other by ⁇ degrees.
  • the color mosaic interpolation section estimates the color intensity based on the intensities that have been measured in pixels of different colors, for which the polarization transmission planes define the same angle, in the neighborhood of the pixel of interest.
  • the polarization wavelength dependences of the polarizer units for use in the pixels of the same color are approximately equal to each other within a wavelength range associated with that color.
  • Another image processing method includes the steps of: measuring the intensities of respective pixels using a single-chip color image capture device in which polarizer units, having polarization transmission planes defining mutually different angles, are provided for multiple pixels in multiple different colors in a color pixel arrangement; obtaining polarization information based on the intensities that have been measured in the multiple pixels in the same color, for which the polarization transmission planes define mutually different angles; obtaining polarization information by correcting the intensities of the multiple pixels in the multiple different colors, for which the polarization transmission planes define different angles, in the neighborhood of the pixel of interest; and estimating a color intensity that cannot be obtained at the pixel of interest based on the intensities of the pixels of the same color, for which (180/ ⁇ ) types of polarizer units, having polarization transmission planes defining angles that are different from each other by ⁇ degrees, are provided in the neighborhood of the pixel of interest, and based on the intensities of the pixels in the different colors, for which the polarization
  • a patterned polarizer according to the present invention includes a number of polarizer units that are arranged two-dimensionally for pixels of at least one color that are included in a color pixel arrangement.
  • the polarizer units include polarizer units having polarization transmission planes defining mutually different angles.
  • the polarizer units are arranged in a checkerboard pattern and no polarizer units are present between adjacent ones of the polarizer units.
  • the polarizer units have a pattern in which multiple sets are arranged periodically.
  • Each set consists of (180/ ⁇ ) types of polarizer units, the polarization transmission planes of which define angles that are different from each other by ⁇ degrees.
  • the polarizer units include: polarizer units, which are provided for pixels in a particular color that are arranged in a checkerboard pattern; and polarizer units, which are provided for pixels other than the pixels in the particular color that are arranged in the checkerboard pattern.
  • polarizer units that are provided for pixels in a particular color that are arranged in a checkerboard pattern have a pattern in which multiple sets are arranged periodically.
  • Each set consists of (180/ ⁇ ) types of polarizer units, the polarization transmission planes of which define angles that are different from each other by ⁇ degrees.
  • Polarizer units that are provided for pixels other than the pixels in the particular color have a pattern in which multiple sets are arranged periodically.
  • Each set consists of two types of polarizer units, the polarization transmission planes of which define mutually different angles.
  • a single-chip color image capture device includes: a patterned polarizer according to any of the preferred embodiments of the present invention described above; and a color mosaic filter, which is arranged such that the respective polarizer units of the patterned polarizer are superposed on its respective pixels.
  • Still another image processing apparatus includes: a color and polarization obtaining section including a single-chip color image capture device that has a color mosaic filter and a patterned polarizer in which a number of polarizer units, of which the polarization transmission planes define four or more different angles, are provided for multiple pixels of the same color in the color mosaic filter; a polarization information processing section for approximating, as a sinusoidal function, a relation between the intensities of light rays that have been transmitted through the polarizer units for the multiple pixels of the same color and the angles of the polarization transmission planes of the polarizer units; and a color mosaic interpolation section for generating a color intensity image by performing color intensity interpolation using, as a pixel unit, a triangular area comprised of three polarizer units that have polarization transmission planes defining mutually different angles and that are adjacent to each other.
  • multiple types of polarizer units are used in combination with a pixel arrangement of a color mosaic filter, and therefore, information about a color intensity and information about polarization can be obtained and processed at the same time.
  • FIG. 1 is a block diagram illustrating a basic configuration that applies to every preferred embodiment of the present invention to be described herein.
  • the apparatus of this preferred embodiment obtains not only color image information from an object in real time but also polarized image information at the same time, and outputs those pieces of information as two different types of polarized images (i.e., a degree-of-polarization image ⁇ and a polarization phase image ⁇ ).
  • a degree-of-polarization image ⁇ and a polarization phase image ⁇ i.e., a degree-of-polarization image ⁇ and a polarization phase image ⁇ .
  • Each of the degree-of-polarization image ⁇ and the polarization phase image ⁇ could be either a still picture or a moving picture.
  • the incident light After having passed through the lens 100 shown in FIG. 1 , the incident light enters a color and polarization obtaining section 101. From this incident light, the color and polarization obtaining section 101 can obtain data about a color moving picture and data about a polarization information image at the same time. The data about the color mosaic image is output from the color and polarization obtaining section 101 and stored in a mosaic image frame memory 107.
  • the mosaic image data is sequentially read out from the mosaic image frame memory 107 and then passed to, and subjected to interpolation processing by, a color mosaic interpolation section 102.
  • the results of the interpolation processing are stored in a color image frame memory 104, in which the three planes of R, G and B are provided for each pixel, and are read when necessary.
  • pixel signals are also read sequentially from the mosaic image frame memory 107 and supplied to a polarization information processing section 103. These pixel signals are processed by the polarization information processing section 103 and then stored in a degree-of-polarization image frame memory 105 and a polarization phase image frame memory 106.
  • the degree-of-polarization image frame memory 105 outputs data about the degree-polarization image ( ⁇ ) and the polarization phase image frame memory 106 outputs data about the polarization phase image ( ⁇ ).
  • FIG. 2 is a schematic representation illustrating a basic arrangement for the color and polarization obtaining section 101.
  • a color mosaic filter 201 and a patterned polarizer 202 are stacked one upon the other in front of image capture device's pixels 203.
  • the color mosaic filter 201 and the patterned polarizer 202 may also be stacked in reverse order.
  • the incident light 204 is transmitted through the color mosaic filter 201 and the patterned polarizer 202 to reach the image capture device and have its intensity monitored by the image capture device's pixels 203 on a pixel-by-pixel basis.
  • color information and polarization information can be obtained at the same time using a single-chip color image capture device.
  • the patterned polarizer 202 consists of a lot of polarizer units including ones (or segments), of which the polarization transmission planes define mutually different angles. Each of those polarizer units typically has a size corresponding to one pixel. However, there is no need to provide the polarizer unit for every pixel of the color mosaic filter 201.
  • Such a patterned polarizer 202 may be realized by using the photonic crystal disclosed in Non-Patent Document No. 2, for example.
  • the photonic crystal When the photonic crystal is used, light, of which the electric field vector oscillating plane is parallel to the groove that has been cut on its surface, becomes a TE wave (transverse electric wave, of which the electric field components are transverse with respect to the incident plane), and light, of which the electric field vector oscillating plane is perpendicular to the groove on its surface, becomes a TM wave (transverse magnetic wave, of which the magnetic field components are transverse with respect to the incident plane).
  • each polarizer unit exhibits such a polarization property as to transmit the TM wave and reflect (not transmit) the TE wave in each wavelength range.
  • FIG. 3 shows the meanings of oblique lines that are schematically drawn on each polarizer unit of the patterned polarizer 202.
  • the oblique lines drawn within each pixel shown in FIG. 3 indicate the polarization main axis direction of a micro-polarizer (i.e., the polarizer unit) that is arranged over that pixel.
  • the "polarization main axis" is an axis that is defined parallel to the polarization plane (i.e., polarization transmission plane) of the light to be transmitted through a polarizer.
  • Each polarizer unit transmits light, of which the oscillation plane is parallel to the main-axis direction of the polarizer unit, and cuts off light, of which the oscillation plane is perpendicular to the main-axis direction of the unit.
  • the polarizer unit is designed according to the transmission wavelength range of the color mosaic filter.
  • FIG. 4(a) illustrates the arrangement of 16 pixels, which consists of eight vertical pixels by eight horizontal pixels, in the color and polarization obtaining section 101 of this preferred embodiment.
  • four types of polarizer units G1 through G4 are provided for the respective G pixels of a Bayer type color mosaic filter but no polarizer units are arranged for the R and B pixels.
  • a huge number of pixel blocks, each having the same arrangement as the block consisting of 64 pixels shown in FIG. 4(a) are arranged periodically within a plane that is parallel to the image capturing plane.
  • the R and B pixels are omitted from the block of pixels shown in FIG. 4(a) and numerals indicating the polarizer unit main axis directions of the G pixels are shown instead. That is to say, the numerals "1" through “4" shown in FIG. 4(b) denote pixels for which the polarizer units G1 to G4 are respectively provided.
  • the other blank pixels, to which none of these numerals "1” through “4" are added, are either R pixels or B pixels. Since no polarizer units are provided for those pixels as described above, those pixels may be referred to herein as "non-polarization pixels".
  • FIGS. 5(a) and 5(b) show the features of such a pixel arrangement from a different point of view.
  • FIG. 5(a) shows the same pixel arrangement as the one shown in FIG. 4 .
  • FIG. 5(b) shows another exemplary pixel arrangement with a similar property.
  • two types of polarization pixels are alternately arranged in different phases on each of multiple diagonal lines that are parallel to each other.
  • the diagonal lines 505 through 508 1-2-1-2-..., 3-4-3-4-..., 2-1-2-1-..., and 4-3-4-3-... are arranged repeatedly.
  • two types of pixels are alternately arranged with the phases changed such as 1--3-1-3-..., 2-4-2-4-..., 3-1-3-1-..., and 4-2-4-2-....
  • FIGS. 6(a) and 6(b) show pixel arrangements that are similar to the one shown in FIG. 4 .
  • the arrangement shown in FIGS. 6(a) and 6(b) has less than four types of polarization pixels at the neighboring eight-connected positions surrounding each polarization pixel, and therefore, cannot be adopted in this first preferred embodiment. That is to say, as for the features (1-1) to (1-3) described above, the arrangements shown in FIGS. 6(a) and 6(b) do have the features (1-1) and (1-3) but do not have the feature (1-2).
  • FIG. 7 shows another pixel arrangement, which is also similar to the one shown in FIG. 4 but cannot be adopted to carry out the present invention. This is because there are an insufficient number of types of polarization pixels at the neighboring eight-connected positions surrounding each polarization pixel.
  • the set 502 shown in FIG. 7 there are only two types of polarization pixels identified by the numerals "2" and "4" at the neighboring eight-connected positions surrounding the polarization pixel identified by the numeral "3".
  • polarization information about a non-polarization pixel (represented by a blank square) can be obtained appropriately from polarization pixels surrounding that non-polarization pixel.
  • Polarization information about a polarization pixel cannot be obtained appropriately from the polarization pixels surrounding that polarization pixel.
  • the example shown in FIG. 4 or 5 can be used effectively as a pixel arrangement that has all of the three features described above.
  • the pixel arrangement shown in FIG. 4 is supposed to be adopted.
  • the subscripts x and y are integers of 1 through 4 that are used to indicate the column and row numbers in the matrix.
  • B 22 , B 24 , B 42 and B 44 indicate B pixels with pixel coordinates (2, 2), (2, 4), (4, 2) and (4, 4), respectively. All of these R and B pixels are non-polarization pixels.
  • (G1) 12 , (G3) 14 , (G2) 21 , (G4) 23 , (G3) 32 , (G2) 34 , (G4) 41 , and (G1) 43 indicate a G1 pixel with pixel coordinates (1, 2), a G3 pixel with pixel coordinates (1, 4), a G2 pixel with pixel coordinates (2, 1), a G4 pixel with pixel coordinates (2, 3), a G3 pixel with pixel coordinates (3, 2), a G2 pixel with pixel coordinates (3, 4), a G4 pixel with pixel coordinates (4, 1), and a G1 pixel with pixel coordinates (4, 3), respectively.
  • the G1 through G4 pixels are four types of polarization pixels, of which the polarization main axes have mutually different directions.
  • R 11 and the other signs shown in FIG. 8 are used to indicate the pixel locations and show the intensities measured at those pixel locations.
  • the intensities 901 through 904 associated with the four G pixel samples ( ⁇ i , I i ) are shown.
  • the relation between the angle ⁇ i of the polarization main axis and the intensities 901 through 904 is represented by a sinusoidal function.
  • a sinusoidal function In FIG. 9 , all of the four points representing the intensities 901 through 904 are illustrated as being located right on a single sinusoidal curve. However, the sinusoidal function is determined as the optimum values by a minimum square method with respect to the four points representing the intensities that were actually measured.
  • the "polarization information” means information about the degree of modulation of the amplitude ⁇ of such a sinusoidal curve, representing the degree of dependence of the intensity on the angle of the polarization main axis, and the phase information ⁇ thereof.
  • a and B are given by the following Equations (3) and (4), respectively:
  • A a 2 + b 2
  • sin ⁇ 2 B b a 2 + b 2
  • Equation (5) can be obtained directly based on the measured intensity without using the minimum square method as represented by the following Equations (6) to (8):
  • A 1 2 I 1 ⁇ I 3 2 + I 2 ⁇ I 4 2
  • B 1 2 tan ⁇ 1 I 1 ⁇ I 3 I 2 ⁇ I 4
  • C 1 4 I 1 + I 2 + I 3 + I 4
  • the three parameters A, B and C can be approximated by the sinusoidal function.
  • a degree-of-polarization image representing the degree of polarization ⁇ and a polarization phase image representing the polarization phase ⁇ can be obtained for each pixel.
  • the degree of polarization ⁇ represents how much the light in a pixel of interest has been polarized
  • the polarization phase ⁇ represents an angular position at which the approximated sinusoidal function shown in FIG. 9 has the maximum value.
  • This angle represents the angle of a plane, including a surface normal to the object, within an image plane in a situation where the light reflected from the object has produced internal diffuse reflection.
  • the polarization main axis angles of 0 and 180 degrees ( ⁇ ) are the same as each other.
  • the color intensities to be obtained by interpolation are the intensities of B and G.
  • the B intensity the average of the intensities of the B pixels that are located at the four neighboring eight-connected positions surrounding R 33 shown in FIG. 8 is regarded as an interpolated B intensity.
  • B ⁇ 33 B 22 + B 24 + B 42 + B 44 / 4
  • the average of the intensities of the four G pixels (polarization pixels) that are located at the neighboring four-connected positions surrounding R 33 is calculated. If the intensities of all (180/ ⁇ ) types of (i.e., 180 degrees divided by the gap angle ⁇ ) polarization pixels are all added up and then the average thereof is calculated, then the intensities at two sample points with two opposite phases on the sinusoidal function (such as the intensities at the sample points 901 and 903 or at the sample points 902 and 904 in FIG. 9 ) are added together and averaged. That is why the average becomes equal to the average C of the sinusoidal function.
  • this average intensity may be regarded as the interpolated G intensity. Nevertheless, as the intensity of light that has been transmitted through a polarizer decreases to a half compared to the light that has not been transmitted through the polarizer, the intensity should be doubled to say the least.
  • the Bayer type mosaic filter may also be interpolated by a conventional method using an intensity gradient, for example. That is why the interpolation can be done with even higher precision by appropriately adopting any of those alternative methods.
  • the B pixel B 22 is surrounded with four polarization pixels (G1) 12 , (G2) 21 , (G3) 32 and (G4) 23 , which have mutually different polarization main axis directions. That is why the polarization information can be obtained by the same method as the one already described for an R pixel.
  • the color intensities to be obtained by interpolation are the intensities of R and G.
  • (G4) 23 is surrounded with four polarization pixels (G1) 12 , (G3) 14 , (G3) 32 and (G2) 34 , which have mutually different polarization main axis directions, at the neighboring eight-connected positions. That is why the polarization information can be obtained by plotting the intensity curve shown in FIG. 9 based on the intensities measured on these polarization pixels.
  • two polarization pixels G3 of the same type are included, and therefore, two pieces of the same polarization information are contained. For that reason, one of the following two methods may be adopted.
  • the other method is to determine the intensity curve by the minimum square method of Equation (5) based on the intensities measured at the five points including the two redundant ones.
  • a G pixel is a pixel with a polarizer unit. That is why at that pixel location, not just R and B intensities but also its own G intensity should be calculated as well.
  • each G pixel is always surrounded with R and B pixels, which are respectively located over, under, on the right and on the left of that G pixel.
  • R and B pixels which are respectively located over, under, on the right and on the left of that G pixel.
  • Step S1101 not only the next pixel of interest in the mosaic image but also the intensities of the pixels located at the neighboring four-connected positions thereof are read.
  • Step S1102 it is determined whether or not the pixel of interest is an R pixel. If the answer is YES, the process advances to Step S1103 , in which the B and G intensities are obtained by interpolation using Equations (11) and (12) and then the interpolated intensities are stored on the B and G planes at the location of the pixel of interest in the color image frame memory 104. At the same time, the R intensity of the pixel of interest is stored as it is on the R plane. As a result, the full color intensities of (RGB) are all stored in the color image frame memory 104.
  • Step S1104 it is determined whether or not the pixel of interest is a B pixel. If the answer is YES, the process advances to Step S1105, in which the R and G intensities are obtained by interpolation using Equations (13) and (14) and then the interpolated intensities are stored on the R and G planes at the location of the pixel of interest in the color image frame memory 104. At the same time, the B intensity of the pixel of interest is stored as it is on the B plane. As a result, the full color intensities of (RGB) are all stored in the color image frame memory 104.
  • the R and G intensities are obtained by interpolation using Equations (13) and (14) and then the interpolated intensities are stored on the R and G planes at the location of the pixel of interest in the color image frame memory 104.
  • the B intensity of the pixel of interest is stored as it is on the B plane.
  • the full color intensities of (RGB) are all stored in the color image frame memory 104.
  • Step S1106 the process advances to Step S1106, in which the R and G intensities are obtained by interpolation using Equation (15) and then the interpolated R and G intensities are stored.
  • this processing step as the intensities of the surrounding G pixels have not been interpolated, the interpolation cannot be made on G.
  • This series of processing steps is repeatedly performed until it is determined in Step S1107 that every pixel has been processed.
  • Step S1108 a pixel, of which the G intensity is yet to be determined, and the fixed intensities of its surrounding G pixels are obtained from the color image frame memory 104 after the interpolation has been done.
  • Step S1109 the G pixel intensity is interpolated by Equation (16) and the interpolated G pixel intensity is stored in the color image frame memory 104 again.
  • Step S1201 the color mosaic image data that has been obtained by the color and polarization obtaining section 101 is supposed to be stored in the mosaic image frame memory 107.
  • Step S1202 not only the next pixel of interest in the mosaic image but also the pixel intensities at the neighboring four-connected positions are read at the same time.
  • Step S1202 it is determined whether or not the pixel of interest is an R pixel. If the answer is YES, the process advances to Step S1203, in which using the G1 through G4 pixels located at the neighboring four-connected positions surrounding that R pixel, sinusoidal function parameters are calculated by Equations (6) to (8).
  • Step S1204 it is determined whether or not the pixel of interest is a B pixel. If the answer is YES, the process advances to Step S1205, in which using the G1 through G4 pixels located at the neighboring four-connected positions surrounding that B pixel, sinusoidal function parameters are also calculated by Equations (6) to (8).
  • sinusoidal function parameters may also be determined at the G pixel location by Equations (6) to (8) using the neighboring G1 through G4 pixels in Step S1206.
  • an intensity variation curve may also be calculated by the minimum square method of Equation (5) based on information about the five points, including the two redundant points.
  • Step S1207 the degree of polarization ⁇ and the polarization phase ⁇ are obtained by Equations (9) and (10) using the sinusoidal function parameters A, B and C that have been determined for each pixel, and then stored in the degree-of-polarization image frame memory 105 and the polarization phase image frame memory 106, respectively, in association with that pixel.
  • This series of processing steps is repeatedly performed until it is determined in Step S1208 that every pixel of the mosaic image frame memory has been processed.
  • FIGS. 11 and 12 are independent of each other, and therefore, may be performed in parallel with each other. Also, these processing procedures may be carried out either by executing a software program or using hardware components.
  • a patterned polarizer made of photonic crystals is supposed to be used.
  • a film type polarizer, a wire grid type polarizer, or a polarizer operating on any other principle may also be used.
  • FIG. 13(a) is an input image representing a plastic ball as a spherical object.
  • FIGS. 13(b) and 13(c) show an exemplary degree-of-polarization image ⁇ (x, y) and an exemplary polarization phase image ⁇ (x, y) with respect to the object shown in FIG. 13(a) .
  • the images shown in FIGS. 13(a) and 13(b) are illustrated so as to increase its lightness as their degree of polarization ⁇ or polarization phase ⁇ rises.
  • FIGS. 14(a) and 14(b) are schematic representations illustrating the meanings of the images shown in FIGS. 13(b) and 13(c) .
  • ⁇ degree-of-polarization image
  • the degree-of-polarization image ⁇ (x, y) the more distant from the approximate center, where the camera's viewing direction agrees with the surface normal defined with respect to the ball, a pixel location becomes in the orientations 1401, the higher the degree of polarization ⁇ of that pixel.
  • the degree of polarization ⁇ becomes maximum.
  • these degrees of polarization ⁇ are schematically represented by contours.
  • the polarization phase image ⁇ (x, y) shown in FIG. 14(b) the polarization phase increases monotonically counterclockwise in the directions indicated by the arrows 1402 and 1403 around the sphere in regular periods of 180 degrees with respect to the perpendicular line representing the vertical direction of the image that indicates that the phase is zero degrees.
  • the degree of polarization ⁇ and the polarization phase ⁇ correspond to the directions of the two degrees of freedom of the surface normal to the object. That is to say, the object's shape can be estimated based on the polarization information.
  • a degree-of-polarization image and a polarization phase image are supposed to be output based on the diffuse reflection components of the light reflected from the object.
  • these images may also be output based on the specular reflection components thereof.
  • the polarization phases ⁇ are different by 90 degrees from the ones described above.
  • the polarization information does not have to be the combination ( ⁇ , ⁇ ) described above but may also be any other combination of pieces of information as long as those pieces of information are obtained based on the sinusoidal function shown in FIG. 5 or may even be images generated based on those pieces of information.
  • ⁇ (x, y) should be multiplied by a constant factor and then the product should be subtracted from the original image. Examples of such images generated in this manner are shown in FIGS. 15(a) through 15(d) .
  • the basic configuration of this preferred embodiment is also as shown in the block diagram of FIG. 1 . That is why this preferred embodiment will also be described with reference to FIG. 1 when necessary.
  • the image processing apparatus of this preferred embodiment is characterized in that polarizer units that form the patterned polarizer are provided for all of R, G and B pixels.
  • FIG. 16(a) illustrates a pixel arrangement for the color and polarization obtaining section 101 of this preferred embodiment.
  • polarizer units are provided for not only G pixels but also R and B pixels as well.
  • FIG. 16(b) among the R and B pixels that are adjacent to the G pixels, a group of line segments, each connecting a G pixel to another pixel with a polarization main axis that is parallel to that of the G pixel, are added to the arrangement shown in FIG. 16(a) .
  • This arrangement is based on a Bayer type color mosaic filter.
  • G pixel locations arranged are four types of polarization pixels G1 through G4, of which the polarization main axis directions are identified by the numerals "1" through "4", respectively.
  • the arrangement of those G1 through G4 pixels is the same as the one shown in FIG. 4 .
  • the pixel arrangement of this preferred embodiment is different from the one shown in FIG. 4 in that in addition to providing polarizer units for those G pixels, R1 and R3 with polarization main axis directions of 0 degrees and 90 degrees, respectively, are arranged at the R pixel locations and B2 and B4 with polarization main axis directions of 45 degrees and 135 degrees, respectively, are arranged at the B pixel locations.
  • This pixel arrangement has the following features.
  • the effect to be achieved by the feature (2-1) is that a normal color intensity can be obtained because if the intensities measured on two polarization pixels, of which the polarization phases are different from each other by 90 degrees, are added together and averaged, the polarization effect is canceled.
  • the effect to be achieved by the feature (2-3) is important particularly when the polarization information needs to be figured out with G pixels.
  • FIGS. 17(a) and 17(b) illustrate another exemplary arrangement having such features.
  • the signs, numerals and groups of line segments shown in FIGS. 17(a) and 17(b) are just as already described with reference to FIGS. 16(a) and 16(b) .
  • FIGS. 18(a) through 18(c) are graphs schematically showing the wavelength characteristics of the B, G, R polarization pixels, respectively.
  • the ordinate represents the intensity of the transmitted light and the abscissa represents the wavelength.
  • Each of these B, G and R polarization pixels has such a polarization property that transmits a TM wave in the B, G or R wavelength range and reflects (i.e., not transmits) a TE wave in that wavelength range.
  • the TM wave is a wave in which magnetic field components are transverse to the incident plane
  • the TE wave is a wave in which electric field components are transverse to the incident plane.
  • FIG. 18(a) shown are the polarization properties 1802 and 1803 of the B polarization pixel and the transmission property 1801 of a B color filter.
  • the polarization properties 1802 and 1803 represent the transmittances of the TM and TE waves, respectively.
  • FIG. 18(b) shown are the polarization properties 1805 and 1806 of the G polarization pixel and the transmission property 1804 of a G color filter.
  • the polarization properties 1805 and 1806 represent the transmittances of the TM and TE waves, respectively.
  • FIG. 18(c) shown are the polarization properties 1808 and 1809 of the R polarization pixel and the transmission property 1807 of an R color filter.
  • the polarization properties 1808 and 1809 represent the transmittances of the TM and TE waves, respectively.
  • FIGS. 18(a) through 18(c) are realized by using the photonic crystal disclosed in Non-Patent Document No. 2, for example.
  • the photonic crystal When the photonic crystal is used, light, of which the electric field vector oscillating plane is parallel to the groove that has been cut on its surface, becomes a TE wave, and light, of which the electric field vector oscillating plane is perpendicular to the groove on its surface, becomes a TM wave.
  • What counts in this preferred embodiment is to use a patterned polarizer that exhibits polarization separation ability in each of the B, G and R transmission wavelength ranges as shown in FIGS. 18(a) through 18(c) .
  • the property of a polarization pixel will be represented by a combination (such as “R1” or “G1") of one of the four numerals “1", “2", “3” and “4" representing the polarization main axis direction of the polarization pixel and one of the three color codes “R", "G” and “B” representing the color of that polarization pixel.
  • the polarization pixels R1 and G1 have the same numeral, and therefore, their polarization main axis directions agree with each other.
  • RGB codes are different from each other, these are polarization pixels, of which the wavelength ranges of the light to be transmitted are different from each other.
  • the arrangement of such polarization pixels is realized by the combination of the color filter 201 and the patterned polarizer 202 shown in FIG. 2 .
  • These signs are supposed to represent not only pixels as described above but also the magnitudes of intensities measured at those pixels in mathematical equations.
  • polarization information at (R1) 33 can be obtained based on the intensities measured at those polarization pixels. This statement applies not only to the R pixel at the location identified by (R1) 33 but also to every R pixel at the location identified by R1 or R3.
  • each R pixel is provided with a polarizer unit. That is why at each R pixel, all color intensities, i.e., not just G and B intensities but also R intensity as well, need to be estimated.
  • the G intensity can be obtained using the four pixels G1 to G4 that surrounds the R pixel.
  • the intensity measured at each of these four pixels G1 to G4 is the intensity of a G ray that has been transmitted through a polarizer unit.
  • the polarization main axis angles of these four pixels G1 through G4 are different from each other by 45 degrees, the polarization effect is canceled and the average intensity of the G ray can be obtained by performing the operation of averaging the intensities measured at those pixels.
  • the B intensity can also be obtained with the polarization effect canceled by the averaging operation because there are B2 and B4 pixels, of which the phases are different from each other by 90 degrees, at the neighboring eight-connected pixels surrounding the R pixel.
  • the maximum transmittance T of the polarizer unit is designed to be the same for R, G and B. However, this is not an essential requirement for the present invention.
  • the R pixel is surrounded with no R polarization pixels available for interpolation. For that reason, in this preferred embodiment, the polarization information distributed in colors is integrated. Hereinafter, it will be described how to get that integration done.
  • the curves 2001, 2002 and 2003 shown in FIG. 20 represent R, G and B intensity variations, respectively.
  • the ordinate represents the intensity I
  • the abscissa represents the polarization main axis angle ⁇ .
  • the solid circle R1 indicates the value of the intensity measured at (R1) 33 shown in FIG. 19 and the open circles G1, G2, G3 and G4 indicate the values of intensities measured at the four G pixels surrounding the (R1) 33 pixel.
  • the maximum transmittance of the polarizer unit is supposed to be 100%.
  • the lines 2004, 2005 and 2006 represent average R, G and B intensities (i.e., the average intensities to be measured in a situation where no polarizers are provided).
  • Equation (9) defining the degree of polarization ⁇ , I R ( ⁇ ), I G ( ⁇ ) and I B ( ⁇ ) representing the R, G and B intensity variations are given by the following Equations (19):
  • the refractive index ⁇ and the angles of incidence and emittance of light are substantially constant between R, G and B, and therefore, the degree of polarization ⁇ is also constant between R, G and B. That is why the variable parts of the three sinusoidal functions in Equations (19) are common between R, G and B.
  • the intensity ratio of the three polarized light intensity curves 2001, 2002 and 2003 shown in FIG. 20 becomes equal to the ratio of the average intensities 2004, 2005 and 2006. That is to say, the intensity ratio of R1 to G1 is equal to the intensity ratio of R to G.
  • the average R intensity (i.e., ideal intensity in a situation where there is no patterned polarizer) can be estimated by multiplying the interpolated G intensity value already obtained (i.e., the value given by Equation (17)) by the ratio of the intensity value of the R1 pixel to that of the G1 pixel, which is adjacent to the R1 pixel.
  • the intensities of surrounding G and B pixels, of which the polarization main axis angles are different from that of the R3 pixel by 90 degrees, may be obtained by spatial interpolation and the R intensity may be obtained as the ratio of the intensity of the R3 pixel to that of the G3 pixel.
  • G intensity can be obtained with the polarization effect canceled by averaging the intensities measured at the four surrounding pixels G1 through G4 as represented by the following Equation (22):
  • G ⁇ 22 ( G 1 ) 12 + ( G 2 ) 21 + G 3 32 + ( G 4 ) 23 / 4 T ⁇ 2
  • the (B2) 22 pixel is surrounded with no B pixels that can be used to obtain the B intensity by spatial mosaic interpolation.
  • the average B intensity i.e., ideal intensity in a situation where there is no patterned polarizer
  • B ⁇ 22 B 2 22 G 2 21 ⁇ G ⁇ 22
  • the intensities of surrounding G and R pixels each of which is a pair of pixels with polarization main axis angles that are different from that of the B4 pixel by 45 and 90 degrees, respectively, may be obtained by spatial interpolation and the B intensity may be obtained as the ratio of the intensity of the B4 pixel to that of the G4 pixel.
  • FIG. 22 shows the intensities measured at four polarization pixels that are located at the neighboring four-connected positions surrounding the (G4) 23 pixel with theoretic sinusoidal function curves.
  • the two solid circles indicate the intensities of R1 and R3 pixels and the two shadowed circles indicate the intensities of (B2) 22 and (B4) 24 pixels.
  • the intensities are corrected by multiplying the measured intensities of (B2) 22 and (B4) 24 pixels by a coefficient K:
  • B 2 ′ 22 K ⁇ B 2 22
  • B 4 ′ 24 K ⁇ B 4 24
  • the four points R1, R3, B2' and B4' can be put on the R curve as indicated by the arrows 2207 shown in FIG. 22 .
  • polarization information can be obtained based on this sinusoidal curve.
  • the color intensities of R, G and B are estimated by interpolation in the following manner.
  • the B intensity at the (G4) 23 pixel can be obtained as an interpolated value with the polarization effect canceled by averaging the intensities measured at the B2 and B4 polarization pixels, which are adjacent to the (G4) 23 pixel on right and left-hand sides:
  • B ⁇ 23 B 2 22 + B 4 24 / 2 T ⁇ 2
  • the case of the G4 pixel has been described as an example. However, basically the same method is applicable to the other G1 to G3 pixels, too. That is to say, if R and B pixel intensities are obtained by spatial interpolation and if the interpolated R or B intensity is multiplied by a predetermined ratio, then the G pixel intensity can be obtained by interpolation. That ratio is the ratio of the intensity of the G1 pixel to that of the R1 pixel as for the G1 pixel, the ratio of the intensity of the G2 pixel to that of the B2 pixel as for the G2 pixel, and the ratio of the intensity of the G3 pixel to that of the R3 pixel as for the G3 pixel.
  • the color mosaic image data that has been obtained by the color and polarization obtaining section 101 is supposed to be stored in the mosaic image frame memory 107.
  • Step S2301 not only the first pixel of interest but also the intensities of the pixels located at the neighboring four-connected positions thereof are read at the same time from the mosaic image frame memory 107.
  • Step S2302 it is determined whether or not the pixel of interest is an R pixel. If the answer is YES, the process advances to Step S2303, in which the G and B intensities are obtained by interpolation using Equations (17) and (18).
  • Step S2304 using the G intensity just obtained by interpolated, the R intensity is obtained by interpolation using Equation (20).
  • Step S2305 it is determined whether or not the pixel of interest is a B pixel. If the answer is YES, the process advances to Step S2306, in which the R and G intensities are obtained by interpolation using Equations (21) and (22). Next, in Step S2307, using the G intensity just obtained by interpolation, the B intensity is obtained by interpolation using Equation (23).
  • Step S2308 If it is determined that the pixel of interest is a G pixel, then the process advances to Step S2308, in which the R and B intensities are obtained by interpolation using Equations (25) and (26).
  • Step S2309 using the B or R intensity just obtained by interpolation, the G intensity is obtained by interpolation using Equation (27), for example.
  • Step S2310 the interpolated intensity values of those R, G and B pixels are stored in the color image frame memory 104.
  • Step S2311 This series of processing steps is repeatedly performed until it is determined in Step S2311 that every pixel of the mosaic image frame memory 107 has been processed.
  • the color mosaic image data that has been obtained by the color and polarization obtaining section 101 is supposed to be stored in the mosaic image frame memory 107.
  • Step S2401 not only the next pixel of interest in the mosaic image but also the pixel intensities at the neighboring four-connected positions are read at the same time.
  • Step S2402 it is determined whether or not the pixel of interest is an R pixel. If the answer is YES, the process advances to Step S2403, in which based on the intensities of the G1 through G4 pixels located at the neighboring four-connected positions surrounding that R pixel location, sinusoidal function parameters are calculated by Equations (6) to (8).
  • Step S2404 it is determined whether or not the pixel of interest is a B pixel. If the answer is YES, the process advances to Step S2405, in which based on the intensities of the G1 through G4 pixels located at the neighboring four-connected positions surrounding that B pixel, sinusoidal function parameters are also calculated by Equations (6) to (8). And if the pixel of interest is a G pixel, the sinusoidal function parameters of Equations (6) to (8) may also be determined at the G pixel location by Equation (24) using the intensities of the R1 and R3 pixels located at the neighboring four-connected positions and the interpolated intensities of B2' and B4' pixels, i.e., the intensities of four samples in total.
  • Step S2407 the degree of polarization ⁇ and the polarization phase ⁇ are obtained by Equations (9) and (10) using the sinusoidal function parameters that have been determined for each pixel, and then stored in the degree-of-polarization image frame memory 105 and the polarization phase image frame memory 106, respectively, in association with that pixel.
  • Step S2408 This series of processing steps is repeatedly performed until it is determined in Step S2408 that every pixel of the mosaic image frame memory has been processed.
  • FIGS. 23 and 24 are independent of each other, and therefore, may be performed by executing two different software programs in parallel with each other. Alternatively, these processing procedures may also be carried out by using hardware components.
  • a patterned polarizer made of photonic crystals is supposed to be used.
  • a film type polarizer, a wire grid type polarizer, or a polarizer operating on any other principle may also be used.
  • the basic configuration of this preferred embodiment is also as shown in the block diagram of FIG. 1 . That is why this preferred embodiment will also be described with reference to FIG. 1 when necessary.
  • the image processing apparatus of this preferred embodiment is characterized in that polarizer units that form the patterned polarizer are provided for either only G and R pixels or only G and B pixels.
  • FIG. 25 illustrates a pixel arrangement for the color and polarization obtaining section 101 of this preferred embodiment.
  • This arrangement is based on a Bayer type color mosaic filter.
  • At G pixel locations arranged are four types of polarization pixels G1 through G4, of which the polarization main axis directions are identified by the numerals "1" through "4", respectively.
  • the arrangement of those G1 through G4 pixels is the same as the one shown in FIG. 4 .
  • the pixel arrangement of this preferred embodiment is different from the one shown in FIG. 4 in that in addition to providing polarizer units for those G pixels, R1 and R3 with polarization main axis directions of 0 degrees and 90 degrees, respectively, are arranged at the R pixel locations.
  • the arrangement of this preferred embodiment is also different from that of the second preferred embodiment described above in that no polarizer units are arranged at any B pixel locations.
  • This pixel arrangement has the following features:
  • polarizer units may also be provided for B pixels, not just the G pixels.
  • polarizer units identified by the numerals "2" and "4" may be used as the ones with polarization main axis angles that are different from each other by 90 degrees.
  • FIG. 26 illustrates an exemplary arrangement in such a situation.
  • the polarizer units for the G and R pixels are supposed to have the wavelength characteristic shown in FIG. 18 .
  • FIG. 27 or 28 the patterned polarizer is supposed to have the arrangement shown in FIG. 27 or 28 .
  • R1 33 shown in FIG. 27 Take (R1) 33 shown in FIG. 27 as an example. At the neighboring four-connected positions surrounding the R pixel (R1) 33 , arranged are four types of polarization pixels (G1) 43 , (G2) 34 , (G3) 32 and (G4) 23 having the same color but mutually different polarization main axis angles. As in the first preferred embodiment, polarization information can be derived from these polarization pixels. A detailed description thereof will be omitted herein. This statement applies to every R pixel at the location identified by R1 or R3.
  • Each R pixel is a polarization pixel provided with a polarizer unit. That is why at each R pixel location, all color intensities of R, G and B need to be estimated.
  • the G intensity can be obtained using the four pixels G1 to G4 that surrounds the R pixel.
  • the intensity measured at each of these four pixels G1 to G4 is the intensity of a G ray that has been transmitted through a polarizer unit.
  • the polarization main axis angles of these four pixels G1 through G4 are different from each other by 45 degrees, the polarization effect is canceled and the average intensity of the G ray can be obtained by performing the operation of averaging the intensities measured at those pixels.
  • the interpolated G intensity value needs to be corrected with the maximum transmittance T of the polarizer unit with respect to the G ray.
  • G intensity can be obtained with the polarization effect canceled by averaging the intensities measured at the four surrounding pixels G1 through G4:
  • G ⁇ 22 ( G 1 ) 12 + ( G 2 ) 21 + G 3 32 + ( G 4 ) 23 / 4 T ⁇ 2
  • the other method is to determine the intensity curve by the minimum square method of Equation (5) based on the intensities measured at the five points including the two redundant ones.
  • a G pixel is a pixel with a polarizer unit. That is why at that pixel location, not just R and B intensities but also its own G intensity should be calculated as well. First, it will be described how to obtain the R and B intensities by interpolation.
  • each G pixel is always surrounded with R and B pixels, which are respectively located over, under, on the right and on the left of that G pixel.
  • R and B pixels which are respectively located over, under, on the right and on the left of that G pixel.
  • R and B intensities can be obtained by linear interpolation as represented by the following Equation (33):
  • R ⁇ 23 R 3 2 + R 1 33 / 2 T ⁇ 2
  • B ⁇ 23 B 22 + B 24 / 2
  • the G intensity can be obtained by interpolation using the intensities that have already been calculated at the surrounding G pixels as represented by the following Equation (34):
  • G ⁇ 23 G ⁇ 13 + G ⁇ 22 + G ⁇ 33 + G ⁇ 24 / 4
  • a color mosaic filter with a Bayer arrangement is supposed to be used.
  • the color mosaic filter that can be used in the present invention does not have to have such a Bayer type arrangement but may also have any other type of arrangement.
  • the color of the pixels with the highest arrangement density does not have to be G (green) but may also be any other color.
  • FIG. 29(a) illustrates a pixel arrangement for the color and polarization obtaining section 101 of this preferred embodiment.
  • This arrangement is also based on a Bayer type color mosaic filter.
  • At G pixel locations arranged are four types of polarization pixels G1 through G4, of which the polarization main axis directions are identified by the numerals "1" through "4", respectively.
  • the arrangement of those pixels G1 through G4 is the same as the one shown in FIG. 4 .
  • the plane can be entirely covered with triangular pixel sets, each consisting of three different types of polarizer units, as shown in FIG. 29(b) .
  • polarization information is supposed to be obtained and color interpolation is supposed to be done using such a triangular pixel set as a unit.
  • FIG. 30 shows the same pixel arrangement as the one shown in FIG. 5
  • FIG. 31 shows the same pixel arrangement as the one shown in FIG. 6
  • the pixel arrangement shown in FIG. 6 is regarded as inappropriate for obtaining polarization information.
  • each triangular set of pixels needs to include three types of polarizer units, and therefore, the arrangement shown in FIG. 31 may be adopted.
  • each triangular pixel set consists of all four types of polarizer units #1 through #4 but one missing polarization pixel.
  • the sinusoidal function parameters A, B and C are obtained based on the polarization information collected from the three types of polarizer units that are located at the respective vertices of each triangular pixel set.
  • the parameter C is obtained by adding together and averaging two intensities at two different angles, at which the polarization phases are mutually inverted to each other. That is why the parameter C can be calculated by the equation shown on the right column of the following Table 1 with respect to each missing polarization pixel: Table 1 Missing polarization pixel C is calculated by 1 (I 2 +I 4 )/2 2 (I 1 +I 3 )/2 3 (I 2 +I 4 )/2 4 (I 1 +I 3 )/2
  • the intensities I 1 through I 4 of the polarization pixels can be obtained.
  • the other parameters A and B can be calculated by Equations (6) to (8).
  • the degrees of polarization and the polarization phases are obtained on a triangular pixel set basis. This means that the images stored in the degree-of-polarization image frame memory 105 and the polarization phase image frame memory 106 are comprised of such triangular pixel sets as units.
  • FIG. 32 illustrates the multiple triangular pixel sets shown in FIG. 29(a) on a larger scale. Each vertex (a) through (i) of these triangles is associated with one of the G polarization pixels #1, #2, #3 and #4. According to this preferred embodiment, the color intensity interpolation is also carried out on a triangular pixel set basis as shown in FIG. 32 .
  • G U which is the G intensity of the triangle (a)(b)(d) defined by vertically splitting the oblique square (a)(b)(e)(d) shown in FIG. 32
  • G d which is the G intensity of other triangle (b)(e)(d)
  • Equations (35) and (36) are calculated by the following Equations (35) and (36), respectively:
  • G U 2 I a + I b + I d 3 T
  • G d 2 I b + I d + I e 3 T where I a through I e are the intensities measured at the respective vertex pixels.
  • Equations (35) and (36) the constant 2 is a correction term to cope with the phenomenon that the intensity of non-polarized light decreases to a half when transmitted through a linear polarizer.
  • T denotes the transmittance of the polarizer and needs to be used as a correction term because T is not 100%.
  • the G intensities G U and G d can also be obtained in the same way in every other triangular area shown in FIG. 32 . It should be noted that each of these pixels in the triangular area becomes a unit with a single intensity value.
  • FIG. 33 illustrates pixels that need to be used to obtain the B color intensities of the triangular pixel sets shown in FIG. 32 .
  • Each of the nine vertices (a) through (i) shown in FIG. 33 is located at the vertex of its associated triangle and is associated with one of the G polarization pixels #1, #2, #3 and #4.
  • shown as pixel locations for obtaining necessary data for interpolation are not only four outside vertices ( ⁇ ), ( ⁇ ), ( ⁇ ) and ( ⁇ ) of the triangular pixel sets consisting of the vertices (a) through (i) but also two more outside vertices ( ⁇ ) and ( ⁇ ).
  • the square areas 3301 through 3304 which are enclosed with the dashed squares, represent square pixels with B color intensities. That is to say, B pixels are located within these square areas 3301 through 3304 and the area of each B pixel accounts for 50% of the overall area of its associated triangular pixel set. This value is also adopted for the upper and lower triangular pixel sets. When this idea is adopted, the B values are present in the shadowed areas shown in FIG. 33 .
  • the B values are present in the oblique squares (b)(c)(f)(e) and (e)(h)(g)(d) and in the triangles ( ⁇ )(b)(a) , (a)(d)( ⁇ ), (f)( ⁇ )(i) and (h)(i)( ⁇ ).
  • the triangular areas with missing B intensities are two open triangles that are included in each of the oblique squares (a)(b)(e)(d) and (e)(f)(i)(h).
  • the B color intensity should be generated by interpolation.
  • each oblique square area needs to be vertically split into two triangular pixel sets.
  • the B color intensities B U and B d are calculated on a triangular pixel set basis by the following Equations (38):
  • B U 2 G u G u + G d
  • B B d 2 G d G u + G d B
  • FIG. 34 illustrates pixels that need to be used to obtain the R color intensities of the triangular pixel sets shown in FIG. 32 .
  • four vertices ( ⁇ ), ( ⁇ ), ( ⁇ ) and ( ⁇ ), which are G pixel locations outside of the triangular pixel sets defined by the vertices (a) through (i), are also drawn.
  • the R pixels are present in the shadowed areas as in the B pixel described above.
  • the R pixels are present in the oblique squares (a)(b)(e)(d) and (e)(f)(i)(h) and in the triangles ( ⁇ )(c)(b), (c)( ⁇ )(f), (d)(g)( ⁇ ) and (g)(h)( ⁇ ).
  • the triangular pixel sets with no R pixels may be interpolated by the following Equations (39) and (40):
  • R R 3401 + R 3402 + R 3403 + R 3404 4
  • R u 2 G u G u + G d
  • R R d 2 G d G u + G d R
  • a degree-of-polarization image, a polarization phase image and an (RGB) full-color intensity image are generated for each of the triangular pixel sets shown in FIG. 29 .
  • the image data should be processed appropriately in conformity with the pixel arrangement of the display device. Such processing could be performed by the display device. However, such processing may be done inside the image processing apparatus of this preferred embodiment and then the result may be output.
  • Step S3501 a set of pixels with G intensities (G1-G2-G3) in the mosaic image is read as a unit.
  • Step S3502 the G intensity is estimated by interpolation using one of Equations (35) and (36).
  • Step S3503 G intensities are stored as a triangular pixel set on the G plane of the color image frame memory 104.
  • Step S3504 it is determined whether or not every set of pixels with G intensities (G1-G2-G3) has finished being processed. If the answer is NO, then the series of processing steps S3501 through S3504 is performed repeatedly. On the other hand, if the answer to the query of Step S3504 is YES, then the process advances to Step S3505.
  • Step S3505 pixels with B intensities and pixels with R intensities are obtained from the mosaic image frame memory, which may be carried out in parallel with each other. If the B and R pixels account for 50% of the area of the triangular pixel set, their values are adopted as the B and R pixel intensities in the triangular pixel set in Step S3506.
  • Step S3507 interpolation is carried out on a triangular pixel set with a missing B intensity by Equations (37) and (38) and on a triangular pixel set with a missing R intensity by Equations (39) and (40), respectively.
  • Step S3508 the triangular pixel sets with the interpolated B and R intensities are respectively stored on the B and R planes of the color image frame memory 104.
  • Step S3509 it is determined whether or not every pixel with the B or R intensity, stored in the mosaic image frame memory, has finished being processed. If the answer is NO, then the series of processing steps S3505 through S3509 is repeatedly carried out.
  • Step S3601 a set of pixels with G intensities (G1-G2-G3) is read from the mosaic image frame memory. Then, based on the polarized light intensities G1, G2 and G3, sinusoidal function parameters A, B and C are determined by Equations (6) through (8) and with reference to Tables 1 and 2.
  • Step S3603 the degree of polarization ⁇ and the polarization phase ⁇ are obtained by Equations (9) and (10).
  • Step S3604 the values generated are stored at the triangular pixel locations in the degree-of-polarization image frame memory 105 and the polarization phase image frame memory 106.
  • Step S3605 This series of processing steps is repeatedly carried out until it is determined in Step S3605 that every pixel in the mosaic image frame memory has finished being processed.
  • data about the degree-of-polarization image ⁇ (x, y) is generated in the degree-of-polarization image frame memory 105 and data about the polarization phase image ⁇ (x, y) is generated in the polarization phase image frame memory 106.
  • These images have a triangular pixel structure, which is different from the pixel structure of the color mosaic image, as can be seen from its generation process.
  • polarization information can be obtained with the same pixel structure and at the same resolution as a full-color image I (x, y).
  • FIGS. 35 and 36 are independent of each other, and therefore, may be performed in parallel with each other. Also, these processing procedures may be carried out either by executing a software program or using hardware components.
  • a patterned polarizer made of photonic crystals is supposed to be used.
  • a film type polarizer, a wire grid type polarizer, or a polarizer operating on any other principle may also be used.
  • polarization information and color intensity information are obtained with a triangular pixel set used as a minimum unit.
  • the number of types of polarizers to arrange does not have to be four but may also be five or more. The point is that three different types of polarizer units need to be arranged at the respective vertices of each triangular pixel set.
  • a single image processing apparatus may selectively perform either the processing of the first preferred embodiment or that of the fourth preferred embodiment. If an image processing apparatus with the same pieces of hardware is provided with a software program to carry out the processing of the first preferred embodiment and a software program to carry out the processing of the fourth preferred embodiment, one of these two types of processing may be carried out selectively.
  • the apparatus and method for image processing according to the present invention can obtain color information and polarization information from an object at the same time, and therefore, can be used in various types of digital still cameras, digital movie cameras, monitor cameras and cameras for cars.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Physics & Mathematics (AREA)
  • Spectroscopy & Molecular Physics (AREA)
  • Color Television Image Signal Generators (AREA)
  • Polarising Elements (AREA)
  • Optical Filters (AREA)
  • Solid State Image Pick-Up Elements (AREA)
  • Transforming Light Signals Into Electric Signals (AREA)
  • Image Processing (AREA)
  • Studio Devices (AREA)
EP17167696.8A 2007-06-15 2008-05-14 Dispositif de traitement d'image Active EP3217660B1 (fr)

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
JP2007158962 2007-06-15
JP2007317097 2007-12-07
PCT/JP2008/001210 WO2008155873A1 (fr) 2007-06-15 2008-05-14 Dispositif de traitement d'image
EP08751728.0A EP2040477B1 (fr) 2007-06-15 2008-05-14 Dispositif de traitement d'image

Related Parent Applications (2)

Application Number Title Priority Date Filing Date
EP08751728.0A Division EP2040477B1 (fr) 2007-06-15 2008-05-14 Dispositif de traitement d'image
EP08751728.0A Division-Into EP2040477B1 (fr) 2007-06-15 2008-05-14 Dispositif de traitement d'image

Publications (2)

Publication Number Publication Date
EP3217660A1 true EP3217660A1 (fr) 2017-09-13
EP3217660B1 EP3217660B1 (fr) 2018-10-31

Family

ID=40156037

Family Applications (2)

Application Number Title Priority Date Filing Date
EP17167696.8A Active EP3217660B1 (fr) 2007-06-15 2008-05-14 Dispositif de traitement d'image
EP08751728.0A Active EP2040477B1 (fr) 2007-06-15 2008-05-14 Dispositif de traitement d'image

Family Applications After (1)

Application Number Title Priority Date Filing Date
EP08751728.0A Active EP2040477B1 (fr) 2007-06-15 2008-05-14 Dispositif de traitement d'image

Country Status (5)

Country Link
US (1) US8130293B2 (fr)
EP (2) EP3217660B1 (fr)
JP (2) JP4500360B2 (fr)
CN (2) CN102611896B (fr)
WO (1) WO2008155873A1 (fr)

Families Citing this family (59)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP3828519A3 (fr) * 2007-05-31 2021-10-06 Panasonic Intellectual Property Management Co., Ltd. Dispositif de traitement d'images
CN101868979B (zh) * 2007-12-07 2013-03-27 松下电器产业株式会社 拍摄装置
JP5057389B2 (ja) * 2008-03-31 2012-10-24 富士フイルム株式会社 撮像装置
EP2293541B1 (fr) * 2008-06-26 2018-09-12 Panasonic Intellectual Property Management Co., Ltd. Appareil de traitement d'image, programme de division d'image et procédé de synthèse d'image
US8748799B2 (en) 2010-12-14 2014-06-10 Zena Technologies, Inc. Full color single pixel including doublet or quadruplet si nanowires for image sensors
US20160111562A1 (en) * 2008-09-04 2016-04-21 Zena Technologies, Inc. Multispectral and polarization-selective detector
JP2010183494A (ja) * 2009-02-09 2010-08-19 Hitachi Ltd 光ネットワークシステムおよび暗号化鍵の切り替え方法
KR101368606B1 (ko) * 2009-04-14 2014-02-28 서울대학교산학협력단 컬러 코드화된 자성 구조물
JP5428509B2 (ja) * 2009-05-11 2014-02-26 ソニー株式会社 ２次元固体撮像装置、及び、２次元固体撮像装置における偏光光データ処理方法
JP5391914B2 (ja) * 2009-08-06 2014-01-15 ソニー株式会社 撮像装置および映像記録再生システム
IL201110A (en) 2009-09-22 2014-08-31 Vorotec Ltd Device and method of navigation
EP2539853B1 (fr) * 2010-02-25 2019-01-09 Lirhot Systems Ltd. Filtre de lumière avec angles de polarisation variables et algorithme de traitement
JP5713002B2 (ja) * 2010-03-11 2015-05-07 日本電気株式会社 発光素子および該発光素子を用いた画像表示装置
JP2011239241A (ja) * 2010-05-11 2011-11-24 Panasonic Corp ３次元撮像装置
TWI435118B (zh) * 2010-12-15 2014-04-21 Chung Shan Inst Of Science A multi - channel optical image capture device
GB201101237D0 (en) * 2011-01-25 2011-03-09 Trw Ltd Method of processing images and apparatus
US8587751B2 (en) 2011-02-14 2013-11-19 Samsung Electronics Co., Ltd. Display panel and display apparatus having the same
EP2487530B1 (fr) 2011-02-14 2015-02-25 Samsung Electronics Co., Ltd. Panneau d'affichage comprenant un polariseur sélectif de couleur type à grille métallique
US8964012B2 (en) 2011-02-14 2015-02-24 Samsung Electronics Co., Ltd. Display panel having a polarizing layer and display apparatus having the same
JP5681555B2 (ja) * 2011-04-27 2015-03-11 パナソニックＩｐマネジメント株式会社 光沢品の外観検査装置、プログラム
US9479743B2 (en) * 2011-06-01 2016-10-25 Gvbb Holdings S.A.R.L. Single lens 3-D camera
US20120320164A1 (en) * 2011-06-16 2012-12-20 Lenny Lipton Stereoscopic camera with polarizing apertures
JP2013011827A (ja) * 2011-06-30 2013-01-17 Seiko Epson Corp 偏光素子、液晶装置及び電子機器
JP5764747B2 (ja) 2011-09-02 2015-08-19 パナソニックＩｐマネジメント株式会社 偏光撮像素子および内視鏡
US20130070140A1 (en) * 2011-09-21 2013-03-21 Robert Gove Image sensors with multiple lenses of varying polarizations
JP5624529B2 (ja) 2011-09-27 2014-11-12 株式会社東芝 手振れ補正装置及び撮像装置
JP6010895B2 (ja) 2011-11-14 2016-10-19 ソニー株式会社 撮像装置
JP5990953B2 (ja) * 2012-03-16 2016-09-14 株式会社リコー 撮像装置、物体検出装置、車両走行支援画像処理システム、及び車両
JP5702893B2 (ja) * 2012-07-06 2015-04-15 富士フイルム株式会社 カラー撮像素子および撮像装置
CN103826530B (zh) 2012-07-25 2015-12-30 松下电器产业株式会社 摄像系统
CN103703770B (zh) 2012-08-02 2016-11-02 松下知识产权经营株式会社 偏振彩色摄像装置
WO2014087807A1 (fr) * 2012-12-05 2014-06-12 富士フイルム株式会社 Dispositif d'acquisition d'image, procédé et programme de détection de lumière anormale incidente en direction oblique, et support d'enregistrement
CN104838646B (zh) * 2012-12-07 2016-11-23 富士胶片株式会社 图像处理装置、图像处理方法、程序及记录介质
JP5749409B2 (ja) * 2012-12-07 2015-07-15 富士フイルム株式会社 撮像装置、画像処理方法及びプログラム
JP2014123787A (ja) * 2012-12-20 2014-07-03 Sony Corp 画像処理装置、画像処理方法、および、プログラム
WO2014136570A1 (fr) * 2013-03-05 2014-09-12 富士フイルム株式会社 Dispositif de formation d'image, dispositif de traitement d'image, procédé de traitement d'image, et programme
KR102127762B1 (ko) 2013-10-02 2020-06-30 삼성디스플레이 주식회사 평판 표시 장치
JP6214594B2 (ja) * 2014-04-25 2017-10-18 日東電工株式会社 偏光子、偏光板および画像表示装置
JP6215864B2 (ja) * 2014-04-25 2017-10-18 日東電工株式会社 偏光子、偏光板および画像表示装置
JP6215865B2 (ja) * 2014-04-25 2017-10-18 日東電工株式会社 偏光子の製造方法
JP6215261B2 (ja) 2014-06-27 2017-10-18 日東電工株式会社 長尺状の偏光子、長尺状の偏光板および画像表示装置
JP6713189B2 (ja) 2014-06-27 2020-06-24 日東電工株式会社 長尺状の偏光フィルム積層体
CN104298024B (zh) * 2014-10-24 2017-04-26 华中科技大学 一种液晶基红外波束偏振控制芯片
JP2016127333A (ja) * 2014-12-26 2016-07-11 株式会社リコー 撮像素子および撮像装置および撮像情報認識システム
JP2016171368A (ja) * 2015-03-11 2016-09-23 株式会社東芝 撮像装置、撮像素子、及び撮像方法
JP2017005111A (ja) * 2015-06-10 2017-01-05 ソニー株式会社 固体撮像装置及び電子機器
JP7163000B2 (ja) 2015-06-25 2022-10-31 日東電工株式会社 非偏光部を有する偏光子
JP6422415B2 (ja) 2015-09-28 2018-11-14 日東電工株式会社 偏光子、偏光板および画像表示装置
US10460422B2 (en) 2015-11-10 2019-10-29 Sony Corporation Image processing device and image processing method
WO2018074064A1 (fr) * 2016-10-17 2018-04-26 ソニー株式会社 Dispositif de traitement d'image, procédé de traitement d'image, et dispositif d'imagerie
US10638054B2 (en) * 2017-01-25 2020-04-28 Cista System Corp. System and method for visible and infrared high dynamic range sensing
JP6843682B2 (ja) * 2017-04-07 2021-03-17 キヤノン株式会社 撮像素子および撮像装置
TWI672493B (zh) * 2018-03-07 2019-09-21 由田新技股份有限公司 用於檢測面板斑紋的光學檢測系統及其方法
KR102534247B1 (ko) * 2018-10-01 2023-05-18 삼성전자주식회사 편광판을 포함한 3d 이미지 센서, 및 그 3d 이미지 센서를 기반으로 한 깊이 보정방법과 3d 영상 생성방법
US10667693B1 (en) * 2018-11-16 2020-06-02 Perkinelmer Health Sciences, Inc. Systems, methods, and apparatus for interference filter correction based on angle of incidence
WO2020213238A1 (fr) 2019-04-19 2020-10-22 ソニー株式会社 Dispositif de capture d'images, dispositif de traitement d'images, et procédé de traitement d'images
US11758290B2 (en) 2020-01-09 2023-09-12 Sony Group Corporation Image processing device, image processing method, and image pickup device
CN111212247B (zh) * 2020-01-13 2022-08-05 Oppo广东移动通信有限公司 一种偏振式图像传感器、信号处理方法及存储介质
CN112788314B (zh) * 2021-01-20 2022-05-31 西北工业大学深圳研究院 基于边缘变化感应的偏振去马赛克方法

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5257092A (en) * 1990-06-27 1993-10-26 Asahi Kogaku Kogyo Kabushiki Kaisha Apparatus for measuring polarization and birefringence
USRE37752E1 (en) * 1992-10-29 2002-06-18 Equinox Corporation Polarization viewer
JP2006254331A (ja) 2005-03-14 2006-09-21 Fuji Photo Film Co Ltd 被写体の反射光成分を検出する画像処理方法およびこれを実行する装置
US7149262B1 (en) * 2000-07-06 2006-12-12 The Trustees Of Columbia University In The City Of New York Method and apparatus for enhancing data resolution
DE102005031966A1 (de) * 2005-07-08 2007-01-18 Grau, Günter, Dr.-Ing. Herstellung von Polarisationsfiltern sowie entsprechende Anwendungen

Family Cites Families (16)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CA2178325A1 (fr) * 1993-12-21 1995-06-29 Andrew J. Ouderkirk Polariseur reflechissant a luminosite amelioree
FR2808891B1 (fr) * 2000-05-12 2003-07-25 Nemoptic Dispositif bistable d'affichage en reflexion
JPWO2004008196A1 (ja) * 2002-07-13 2005-11-10 有限会社オートクローニング・テクノロジー 偏光解析装置
US6850329B2 (en) * 2002-10-15 2005-02-01 Mitutoyo Corporation Interferometer using integrated imaging array and high-density polarizer array
JP2005287073A (ja) * 2003-04-07 2005-10-13 Matsushita Electric Ind Co Ltd 固体撮像装置、信号処理装置、カメラ及び分光装置
US20050133879A1 (en) * 2003-04-07 2005-06-23 Takumi Yamaguti Solid-state imaging device, signal processing device, camera, and spectral device
JP4287320B2 (ja) 2003-04-07 2009-07-01 パナソニック株式会社 固体撮像装置、信号処理装置、カメラ及び分光装置
US7411677B2 (en) * 2003-09-17 2008-08-12 Photonic Lattice Inc. Driverless ellipsometer and ellipsometry
US7150945B2 (en) * 2003-11-18 2006-12-19 Micron Technology, Inc. Polarized reticle, photolithography system, and method of forming a pattern using a polarized reticle in conjunction with polarized light
KR20060022135A (ko) * 2004-09-06 2006-03-09 주식회사 하이닉스반도체 편광 레티클
US7193214B1 (en) * 2005-04-08 2007-03-20 The United States Of America As Represented By The Secretary Of The Army Sensor having differential polarization and a network comprised of several such sensors
JP4974543B2 (ja) 2005-08-23 2012-07-11 株式会社フォトニックラティス 偏光イメージング装置
US7522278B2 (en) * 2006-03-10 2009-04-21 University Of Washington Real-time linear-birefringence-detecting polarization microscope
WO2007139070A1 (fr) * 2006-05-29 2007-12-06 Panasonic Corporation Dispositif, système et procédé d'évaluation de source de lumière, dispositif à résolution d'image accrue, et procédé d'augmentation de résolution d'image
EP3828519A3 (fr) * 2007-05-31 2021-10-06 Panasonic Intellectual Property Management Co., Ltd. Dispositif de traitement d'images
CN101868979B (zh) * 2007-12-07 2013-03-27 松下电器产业株式会社 拍摄装置

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5257092A (en) * 1990-06-27 1993-10-26 Asahi Kogaku Kogyo Kabushiki Kaisha Apparatus for measuring polarization and birefringence
USRE37752E1 (en) * 1992-10-29 2002-06-18 Equinox Corporation Polarization viewer
US7149262B1 (en) * 2000-07-06 2006-12-12 The Trustees Of Columbia University In The City Of New York Method and apparatus for enhancing data resolution
JP2006254331A (ja) 2005-03-14 2006-09-21 Fuji Photo Film Co Ltd 被写体の反射光成分を検出する画像処理方法およびこれを実行する装置
DE102005031966A1 (de) * 2005-07-08 2007-01-18 Grau, Günter, Dr.-Ing. Herstellung von Polarisationsfiltern sowie entsprechende Anwendungen

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
KAWASHIMA; SATO; KAWAKAMI; NAGASHIMA; OTA; AOKI: "Development of Polarization Imaging Device and Applications by Using Patterned Polarizer", INSTITUTE OF ELECTRONICS, INFORMATION AND COMMUNICATION ENGINEERS OF JAPAN, NATIONAL CONFERENCE 2006, March 2006 (2006-03-01), pages 52
KIKUTA; IWATA: "Polarized Image Measuring System", OPLUS E, vol. 25, no. 11, 2003, pages 1241 - 1247
LAWRENCE B. WOLFF: "Image and Vision Computing", vol. 15, 1997, ELSEVIER SCIENCE B. V, article "Polarization Vision: A New Sensory Approach to Image Understanding", pages: 81 - 93

Also Published As

Publication number Publication date
JP4965615B2 (ja) 2012-07-04
CN101569205B (zh) 2012-05-23
JP4500360B2 (ja) 2010-07-14
US20090278954A1 (en) 2009-11-12
EP3217660B1 (fr) 2018-10-31
US8130293B2 (en) 2012-03-06
EP2040477A4 (fr) 2015-11-04
JP2009290895A (ja) 2009-12-10
WO2008155873A1 (fr) 2008-12-24
CN102611896A (zh) 2012-07-25
CN101569205A (zh) 2009-10-28
JPWO2008155873A1 (ja) 2010-08-26
EP2040477B1 (fr) 2018-08-29
EP2040477A1 (fr) 2009-03-25
CN102611896B (zh) 2015-01-07

Similar Documents

Publication Publication Date Title
EP2040477B1 (fr) Dispositif de traitement d'image
EP2216999B1 (fr) Dispositif de traitement d'image, procédé de traitement d'image, et dispositif d'imagerie
US7760256B2 (en) Image processing apparatus that obtains color and polarization information
KR101517704B1 (ko) 이미지를 기록하기 위한 이미지 기록 장치 및 방법
RU2535640C2 (ru) Формирование многоспектральных изображений
EP2263373B1 (fr) Systèmes et procédés pour appareil photographique à pixels à classification générale
JP6725526B2 (ja) 分散して空間的符号化画像を作成する画像を伴う撮像素子
CN103501416A (zh) 使用具有异构成像器的单片相机阵列的图像拍摄和图像处理
US9008412B2 (en) Image processing device, image processing method and recording medium for combining image data using depth and color information
CN106679807B (zh) 一种基于lctf高光谱成像系统的图像压缩与重构方法
US10951817B2 (en) Compound-eye imaging device, image processing method, and recording medium
US10212401B2 (en) Image generation device and imaging device
JP2018040976A (ja) フィルタ、画像撮影装置および画像撮影システム
US10360661B2 (en) Determining multispectral or hyperspectral image data
Kriss Tradeoff between aliasing artifacts and sharpness in assessing image quality
Kim The three-dimensional evolution of hyperspectral imaging
EP4356172A2 (fr) Procédé et système de fabrication et d'utilisation d'un filtre de base spectrale
Tian et al. New color filter arrays and performance analysis

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION HAS BEEN PUBLISHED

AC Divisional application: reference to earlier application

Ref document number: 2040477

Country of ref document: EP

Kind code of ref document: P

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MT NL NO PL PT RO SE SI SK TR

AX Request for extension of the european patent

Extension state: AL BA MK RS

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20180131

RBV Designated contracting states (corrected)

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MT NL NO PL PT RO SE SI SK TR

REG Reference to a national code

Ref country code: DE

Ref legal event code: R079

Ref document number: 602008057759

Country of ref document: DE

Free format text: PREVIOUS MAIN CLASS: H04N0009070000

Ipc: H04N0009040000

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

RIC1 Information provided on ipc code assigned before grant

Ipc: H04N 9/07 20060101ALI20180518BHEP

Ipc: H04N 9/04 20060101AFI20180518BHEP

INTG Intention to grant announced

Effective date: 20180607

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

AC Divisional application: reference to earlier application

Ref document number: 2040477

Country of ref document: EP

Kind code of ref document: P

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MT NL NO PL PT RO SE SI SK TR

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: AT

Ref legal event code: REF

Ref document number: 1060810

Country of ref document: AT

Kind code of ref document: T

Effective date: 20181115

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602008057759

Country of ref document: DE

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: NL

Ref legal event code: MP

Effective date: 20181031

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG4D

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 1060810

Country of ref document: AT

Kind code of ref document: T

Effective date: 20181031

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181031

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181031

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181031

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190228

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181031

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190131

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181031

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181031

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181031

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190131

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190201

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181031

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181031

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190301

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181031

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181031

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181031

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602008057759

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181031

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181031

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181031

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

26N No opposition filed

Effective date: 20190801

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181031

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

GBPC Gb: european patent ceased through non-payment of renewal fee

Effective date: 20190514

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20190531

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181031

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20190531

REG Reference to a national code

Ref country code: BE

Ref legal event code: MM

Effective date: 20190531

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20190514

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: TR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181031

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GB

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20190514

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20190514

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20190531

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: FR

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20190531

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181031

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181031

Ref country code: HU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO

Effective date: 20080514

REG Reference to a national code

Ref country code: DE

Ref legal event code: R079

Ref document number: 602008057759

Country of ref document: DE

Free format text: PREVIOUS MAIN CLASS: H04N0009040000

Ipc: H04N0023100000

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20230519

Year of fee payment: 16