WO2019102698A1 - Dispositif de traitement d'image, procédé de traitement d'image, programme, et dispositif d'imagerie à semi-conducteur - Google Patents

Dispositif de traitement d'image, procédé de traitement d'image, programme, et dispositif d'imagerie à semi-conducteur Download PDF

Info

Publication number
WO2019102698A1
WO2019102698A1 PCT/JP2018/034389 JP2018034389W WO2019102698A1 WO 2019102698 A1 WO2019102698 A1 WO 2019102698A1 JP 2018034389 W JP2018034389 W JP 2018034389W WO 2019102698 A1 WO2019102698 A1 WO 2019102698A1
Authority
WO
WIPO (PCT)
Prior art keywords
polarization
pixel
color
specific color
pixels
Prior art date
Application number
PCT/JP2018/034389
Other languages
English (en)
Japanese (ja)
Inventor
哲平 栗田
穎 陸
雄飛 近藤
康孝 平澤
Original Assignee
ソニー株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by ソニー株式会社 filed Critical ソニー株式会社
Priority to US16/763,543 priority Critical patent/US11605661B2/en
Priority to JP2019556113A priority patent/JP7230824B2/ja
Priority to CN201880073836.0A priority patent/CN111357282B/zh
Publication of WO2019102698A1 publication Critical patent/WO2019102698A1/fr

Links

Images

Classifications

    • H—ELECTRICITY
    • H01—ELECTRIC ELEMENTS
    • H01L—SEMICONDUCTOR DEVICES NOT COVERED BY CLASS H10
    • H01L27/00—Devices consisting of a plurality of semiconductor or other solid-state components formed in or on a common substrate
    • H01L27/14—Devices consisting of a plurality of semiconductor or other solid-state components formed in or on a common substrate including semiconductor components sensitive to infrared radiation, light, electromagnetic radiation of shorter wavelength or corpuscular radiation and specially adapted either for the conversion of the energy of such radiation into electrical energy or for the control of electrical energy by such radiation
    • H01L27/144—Devices controlled by radiation
    • H01L27/146—Imager structures
    • H01L27/14601—Structural or functional details thereof
    • H01L27/14625—Optical elements or arrangements associated with the device
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/70—SSIS architectures; Circuits associated therewith
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/10—Circuitry of solid-state image sensors [SSIS]; Control thereof for transforming different wavelengths into image signals
    • H04N25/11—Arrangement of colour filter arrays [CFA]; Filter mosaics
    • H04N25/13—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/10—Circuitry of solid-state image sensors [SSIS]; Control thereof for transforming different wavelengths into image signals
    • H04N25/11—Arrangement of colour filter arrays [CFA]; Filter mosaics
    • H04N25/13—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements
    • H04N25/134—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements based on three different wavelength filter elements
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/70—SSIS architectures; Circuits associated therewith
    • H04N25/702—SSIS architectures characterised by non-identical, non-equidistant or non-planar pixel layout

Definitions

  • This technology relates to an image processing device, an image processing method, a program, and a solid-state imaging device, and enables to obtain an output image with good image quality and polarization characteristics for each color.
  • Patent Document 1 polarizers having transmission axes of at least three different angles are provided in pixels of a single specific color component in pixels for each color component, and polarization characteristics are obtained from pixel values of the single specific color component. Acquisition is performed, and interpolation is performed using pixel values for each color component to obtain a captured image for each color component.
  • the specular reflection component of a color component different from a single specific color component is calculated on the assumption that the single specific color component and the degree of polarization are equal.
  • patent document 1 assumes that the polarization degree for every color is equal, when the polarization degree for every color is different, calculation etc. of a specular reflection component can not be performed accurately.
  • an object of the present technology to provide an image processing device, an image processing method, a program, and a solid-state imaging device capable of detecting polarization characteristics for each color.
  • the first aspect of this technology is Based on the polarization characteristic of a specific color, the pixel signal of a non-polarization pixel of a non-specific color different from the specific color, and the pixel signal of a polarization pixel of the non-specific color and having a polarization direction smaller than the specific color.
  • the image processing apparatus further includes a nonspecific color polarization characteristic detection unit that detects the nonspecific color polarization characteristic.
  • a non-polarization component image signal of a specific color using a pixel signal of a non-polarization pixel and a non-polarization component image signal of a non-specific color using a pixel signal of a non-polarization pixel of a non-specific color Is generated.
  • the polarization component image signal of the specific color is generated using the pixel signal of the polarization pixel of the specific color and the pixel signal of the polarization pixel of the non-specific color is used to generate the polarization component image signal of the non-specific color.
  • the non-polarization component image signal and the polarization component image signal for each color are generated by filtering using pixel signals of pixels having the same polarization direction as the color.
  • the filter processing for example, the color and the low frequency component of the target pixel position for each polarization direction calculated by performing the filter processing using pixels having the same color and polarization direction from the pixel of the target pixel position and peripheral pixels
  • the pixel signal for each color and polarization direction at the target pixel position is calculated based on the high frequency component of the target pixel position calculated using the low frequency component of the target pixel position having the same color and polarization direction as the pixel signal of Generate
  • the image processing apparatus detects the polarization characteristic of a specific color using, for example, pixel signals of polarization pixels of three or more polarization directions of a specific color or non-polarization pixels and polarization pixels of the two polarization directions. Further, in the image processing apparatus, the polarization characteristics of a specific color, a pixel signal of a non-polarization pixel of a non-specific color different from the specific color, and a polarization pixel of a non-specific color having a polarization direction smaller than the specific color. The polarization characteristics of non-specific colors are detected based on the pixel signals.
  • the image processing apparatus calculates the degree of polarization of the non-specific color based on the polarization characteristics of the non-polarization component image signal, the polarization component image signal, and the specific color. Detect the polarization characteristics of a specific color.
  • the image processing apparatus calculates the color balance coefficient by setting specular reflection as white based on the polarization characteristic of the specific color and the polarization characteristic of the non-specific color, and using the calculated color balance coefficient, the non-polarization component image signal or The white balance adjustment of the polarization component image signal is performed.
  • the color balance coefficient is adjusted based on the difference between the color balance coefficient based on the polarization characteristics and the color balance coefficient based on the color information that reproduces a white object as white, or the reflection amount of specular reflection, and the color balance after adjustment.
  • the white balance adjustment may be performed using a coefficient.
  • the second aspect of this technology is Based on the polarization characteristic of a specific color, the pixel signal of a non-polarization pixel of a non-specific color different from the specific color, and the pixel signal of a polarization pixel of the non-specific color and having a polarization direction smaller than the specific color.
  • Another aspect of the present invention is an image processing method including detecting a polarization characteristic of the non-specific color by the non-specific color polarization characteristic detection unit.
  • the third aspect of this technology is A program that causes a computer to execute processing of an image including polarized pixels and non-polarized pixels, Based on the polarization characteristic of a specific color, the pixel signal of a non-polarization pixel of a non-specific color different from the specific color, and the pixel signal of a polarization pixel of the non-specific color and having a polarization direction smaller than the specific color.
  • the program causes the computer to execute the procedure of detecting the polarization characteristic of the non-specific color.
  • the program of the present technology is, for example, a storage medium, communication medium such as an optical disc, a magnetic disc, a semiconductor memory, etc., provided in a computer readable format to a general-purpose computer capable of executing various program codes. It is a program that can be provided by a medium or a communication medium such as a network. By providing such a program in a computer readable form, processing according to the program is realized on the computer.
  • the fourth aspect of this technology is Solid-state imaging having a pixel of a specific color configured to allow detection of polarization characteristics, a non-polarization pixel, and a pixel of a non-specific color different from the specific color configured of a polarization pixel having a type of polarization smaller than the specific color.
  • a pixel of a specific color configured to allow detection of polarization characteristics, a non-polarization pixel, and a pixel of a non-specific color different from the specific color configured of a polarization pixel having a type of polarization smaller than the specific color.
  • this technique is a polarization characteristic of a specific color, a pixel signal of a non-polarization pixel of a non-specific color different from the specific color, and a polarization which is the non-specific color and whose polarization direction is smaller than the specific color.
  • the polarization characteristic of the non-specific color is detected based on the pixel signal of the pixel. Therefore, polarization characteristics can be detected with high accuracy for each color.
  • the effects described in the present specification are merely examples and are not limited, and additional effects may be present.
  • FIG. 2 is a diagram showing pixels of a solid-state imaging device and readout signal lines. It is a figure for demonstrating the read-out operation of a pixel signal. It is a block diagram showing an example of rough composition of a vehicle control system. It is explanatory drawing which shows an example of the installation position of a vehicle exterior information detection part and an imaging part.
  • First Embodiment 1-1 Configuration of First Embodiment 1-2. Specific example of the first embodiment and its operation Second Embodiment 2-1. Configuration of Second Embodiment 2-2. Specific example of second embodiment and operation thereof 3. Third Embodiment 3-1. Configuration of Third Embodiment 3-2. Specific example of third embodiment and operation thereof 4. Fourth Embodiment 4-1. Configuration of Fourth Embodiment 4-2. Specific example of the fourth embodiment and its operation 5. Fifth Embodiment 5-1. Configuration of Fifth Embodiment 5-2. Specific example of fifth embodiment and operation thereof 6. Other Embodiments 7. Another Configuration of Solid-State Imaging Device 7-1. About pixel array 7-2. About signal readout 8. Application example
  • FIG. 1 illustrates the configuration of the first embodiment.
  • the imaging system 10-1 includes a solid-state imaging device 20 and an image processing device 30-1.
  • the image processing apparatus 30-1 includes a polarization component division unit 31, a non-polarization component image generation unit 32-1, a polarization component image generation unit 33-1, a specific color polarization characteristic detection unit 34-1, a non-specific color polarization characteristic detection unit It has 35.
  • the solid-state imaging device 20 is a pixel of a non-specific color different from the specific color of the pixel of a specific color configured to be capable of detecting the polarization characteristic, and a non-polarization pixel and a polarization pixel of a type having a smaller polarization direction than the specific color. have.
  • the solid-state imaging device 20 reads a pixel signal from each pixel, performs analog / digital conversion (A / D conversion), and the like, and outputs it to the image processing device 30-1 as an image signal of a polarized RAW image. Also, the solid-state imaging device 20 may perform white balance adjustment on the image signal output to the image processing device 30-1.
  • the polarization component division unit 31 of the image processing device 30-1 divides the image signal of the polarized RAW image supplied from the solid-state imaging device 20 into the signal of the non-polarization pixel and the polarization pixel to generate the non-polarization image signal and the polarization image signal Generate
  • the polarization component division unit 31 outputs the non-polarization pixel signal to the non-polarization component image generation unit 32-1, and outputs the polarization image signal to the polarization component image generation unit 33-1.
  • the non-polarization component image generation unit 32-1 uses the signal of the specific color from the non-polarization image signal and uses the non-polarization component image signal of the specific color and the non-specific color signal from the non-polarization image signal. A non-polarization component image signal is generated.
  • the non-polarization component image generation unit 32-1 outputs the non-polarization component image signal of the specific color to the specific-color polarization characteristic detection unit 34-1, and detects the non-specific color polarization characteristic signal of the non-specific color Output to section 35.
  • the polarization component image generation unit 33-1 generates a polarization component image signal of a specific color using a signal of a specific color from the polarization image signal and a polarization component image signal of a non-specific color using a signal of a non-specific color from the polarization image signal Generate The polarization component image generation unit 33-1 outputs the polarization component image signal of the specific color to the specific color polarization characteristic detection unit 34-1, and the polarization component image signal of the nonspecific color to the nonspecific color polarization characteristic detection unit 35. Output.
  • the specific color polarization characteristic detection unit 34-1 detects the polarization characteristic of the specific color based on the non-polarization component image signal and the polarization component image signal of the specific color. Also, the detected polarization characteristic of the specific color is output to the non-specific color polarization characteristic detection unit 35.
  • the non-specific color polarization characteristic detection unit 35 is a specific color, which is the non-specific color non-specific pixel pixel signal and the non-specific color non-specific pixel polarization signal detected by the specific color polarization characteristic detection unit 34-1.
  • the polarization characteristic of the non-specific color is detected based on the pixel signal of the polarization pixel of a type having a smaller polarization direction than that of the other.
  • FIG. 2 shows a part of the configuration of the solid-state imaging device.
  • the solid-state imaging device 20 is provided with a color mosaic filter 22 and a polarizer 23 on the light incident surface of the image sensor 21.
  • the image sensor 21 performs photoelectric conversion for each image on incident light incident through the color mosaic filter 22 and the polarizer 23 to generate a pixel signal. Further, the image sensor 21 reads out a pixel signal from each pixel, performs analog / digital conversion and the like, and outputs it to the image processing device 30-1 as an image signal of a polarized RAW image.
  • the color mosaic filter 22 is provided with, for example, red, green and blue color unit areas as a Bayer arrangement.
  • the color unit area is not limited to one pixel area, and may be a plural pixel area (for example, an area of 2 ⁇ 2 pixels).
  • the polarizer 23 is provided so as to be able to detect polarization characteristics for pixels of a specific color. For example, they are provided so as to be configured to use polarization pixels in three or more polarization directions or non-polarization pixels and polarization pixels in two polarization directions. In addition, the polarizer 23 is provided such that pixels of non-specific color have non-polarization pixels and polarization pixels in one polarization direction.
  • FIG. 3 exemplifies a pixel arrangement in which the specific color is green and the non-specific colors are red and blue.
  • 2 ⁇ 2 pixels are set as a color unit area
  • 4 ⁇ 4 pixel areas are set as a pixel arrangement repeating unit of Bayer arrangement consisting of red, green and blue color unit areas.
  • an area of 2 ⁇ 2 pixels composed of two green pixels, one red pixel and one blue pixel in the central part in the pixel arrangement repeating unit is regarded as a polarization pixel, and the upper two pixels and the lower two pixels
  • the polarization directions are different, and for example, the upper two pixels have a transmission axis of 90 degrees, and the lower two pixels have a transmission axis of 135 degrees.
  • a green pixel which is a specific color is composed of two polarization pixels and six non-polarization pixels which have different polarization directions, and a red pixel and a blue pixel which are non-specific colors are one polarization pixel. And three non-polarization pixels.
  • the solid-state imaging device 20 performs white balance adjustment and the like on pixel signals generated by non-polarization pixels and polarization pixels, and outputs the result to the image processing device 30-1.
  • white balance adjustment a color balance coefficient based on color information that reproduces a white subject as white is used.
  • the equations (1) to (3) are calculated using color balance coefficients (white balance gains) g R , g G and g B preset according to the illumination light source.
  • the signal DRs is a red pixel signal before white balance adjustment
  • the signal DRw is a red pixel signal after white balance adjustment.
  • signals DGs and DBs are a green pixel signal and a blue pixel signal before white balance adjustment
  • signals DGw and DBw are a green pixel signal and a blue pixel signal after white balance adjustment.
  • the white balance adjustment may be performed by the image processing device 30-1.
  • DRw g R ⁇ DRs (1)
  • DGw g G ⁇ DGs (2)
  • DBw g B ⁇ DBs (3)
  • the polarization component division unit 31 of the image processing device 30-1 divides the image signal of the polarized RAW image supplied from the solid-state imaging device 20 into the signal of the non-polarization pixel and the signal of the polarization pixel to obtain the non-polarization image signal and the polarization. Generate an image signal. Further, the polarization component division unit 31 equalizes resolutions (number of pixels) of the non-polarization image signal and the polarization image signal.
  • the color unit area of 2 ⁇ 2 pixels is a non-polarization pixel of 3 pixels since the polarization pixel is 1 pixel and the non-polarization pixel is 3 pixels. Is integrated into one pixel. For example, a pixel average value of three non-polarized pixels in a color unit area of 2 ⁇ 2 pixels is calculated to be a pixel value of one pixel.
  • FIG. 4 shows an image before division of polarization components and an image after division; (a) of FIG. 4 is a polarization RAW image, (b) of FIG. 4 is a non-polarization image, and (c) of FIG. The image is shown.
  • the polarization component dividing unit 31 integrates non-polarization pixels shown by circles for each color unit area of 2 ⁇ 2 pixels in the polarization RAW image into one pixel of the non-polarization image.
  • the polarization component dividing unit 31 extracts a polarization pixel for each color unit area of 2 ⁇ 2 pixels in the polarization RAW image to be one pixel of the polarization image.
  • a non-polarization image and a polarization image of Bayer arrangement can be obtained from the polarization RAW image in which the resolution (number of pixels) in the vertical direction and the horizontal direction is 1/2.
  • the polarization component division unit 31 outputs the non-polarization image signal to the non-polarization component image generation unit 32-1, and outputs the polarization image signal to the polarization component image generation unit 33-1.
  • the non-polarization component image generation unit 32-1 generates a non-polarization component image signal for each color using the non-polarization image signal supplied from the polarization component division unit 31.
  • the non-polarization component image generation unit 32-1 performs, for example, interpolation processing, and calculates the pixel value of the target pixel position using the pixel values of the pixels of the same color located in the periphery.
  • FIG. 5 is a diagram for explaining the operation of the non-polarization component image generation unit 32-1, and (a) of FIG. 5 shows a non-polarization image.
  • the non-polarization component image generation unit 32-1 calculates the pixel value DR0 of the red pixel PR0 located in the periphery and the pixel values DR1 of the red pixels PR1 to PR3.
  • the equation (4) is calculated using ⁇ DR3.
  • the non-polarization component image generation unit 32-1 sets the average pixel value of the red pixels located in the periphery as the pixel value DRs1 of the red component of the target pixel position Ps1.
  • the non-polarization component image generation unit 32-1 calculates the pixel value DG0 of the green pixel PG0 and the pixel value DG1 of the green pixel PG1 located in the periphery.
  • the equation (5) is calculated using The non-polarization component image generation unit 32-1 sets the average pixel value of the green pixels located in the periphery as the pixel value DGs2 of the green component of the target pixel position Ps2.
  • DRs1 (DR0 + DR1 + DR2 + DR3) / 4
  • DGs2 (DG0 + DR1) / 2 (5)
  • the interpolation process is not limited to the method shown in FIG.
  • the non-polarization component image generation unit 32-1 when calculating the pixel value DRs1 of the red component of the target pixel position Ps1, the non-polarization component image generation unit 32-1 performs interpolation processing using two pixels of the red component facing each other across the target pixel position Ps1. Good.
  • the non-polarization component image generation unit 32-1 calculates the pixel value DGs2 of the green component at the target pixel position Ps2, the non-polarization component image generation unit 32-1 selects four pixels of the green component adjacent vertically and horizontally. Interpolation processing may be performed using this.
  • the non-polarization component image generation unit 32-1 performs interpolation processing for each color using the non-polarization image signal, and as shown in (b) of FIG. 5, for each color having the same resolution (number of pixels) as the non-polarization image. Generate a non-polarization component image.
  • the non-polarization component image generation unit 32-1 outputs the non-polarization component image signal of the specific color to the specific-color polarization characteristic detection unit 34-1, and detects the non-specific color polarization characteristic signal of the non-specific color Output to section 35.
  • the polarization component image generation unit 33-1 generates polarization component image signals for each color using the polarization image signal supplied from the polarization component division unit 31. Further, in the color including a plurality of polarization directions, polarization component image signals are generated for each polarization direction.
  • the polarization component image generation unit 33-1 performs, for example, filter processing, and calculates the pixel value of the target pixel position using pixel values of the same color and polarization direction located in the periphery.
  • FIG. 6 is a diagram for explaining the operation of the polarization component image generation unit 33-1.
  • (A) of FIG. 6 illustrates a polarization image
  • (b) of FIG. 6 illustrates filter coefficients. Also, as shown in (c) to (f) of FIG.
  • the polarized image is subjected to filter processing using pixel values of the target pixel position of the target pixel position and pixel values of the same color and polarization direction located in the periphery. Generate polarization images for each and every polarization direction.
  • the polarization component image generation unit 33-1 outputs the polarization component image signal for each polarization direction of the specific color to the specific color polarization characteristic detection unit 34-1, and the polarization component image signal of the nonspecific color is output as the nonspecific color polarization characteristic Output to the detection unit 35.
  • the specific color polarization characteristic detection unit 34-1 detects the non-polarization component image signal of the specific color obtained by the non-polarization component image generation unit 32-1 and the polarization direction of the specific color obtained by the polarization component image generation unit 33-1. Based on each polarization component image signal, the polarization characteristic of a specific color is detected.
  • Equation (6) a polarization model showing polarization characteristics is expressed as Equation (6).
  • Equation (6) “y” is the observed luminance (pixel value) when the polarization angle is “x”, and “c” is the pixel value Ic of the non-polarization pixel as shown in equation (7). It is.
  • “a” and “b” are parameter values calculated based on the equations (8) and (9).
  • “K” in the equations (8) and (9) is a gain that absorbs the difference in sensitivity between the non-polarization pixel and the polarization pixel.
  • the pixel value I0 is a pixel value when the transmission axis is a predetermined angle degree
  • the pixel value I1 is a pixel value when the transmission axis has an angle difference of 45 degrees with respect to the predetermined angle.
  • the gain for absorbing the sensitivity difference is set in advance for each pixel or for each region composed of a plurality of pixels by calibration or the like.
  • a green non-polarization component image is obtained by the non-polarization component image generation unit 32-1, and a green polarization component image whose transmission axis is at an angle of 90 degrees and 135 degrees by the polarization component image generation unit 33-1.
  • Is obtained. 7 shows a green component image
  • (a) of FIG. 7 is a non-polarization component image
  • (b) of FIG. 7 is a polarization component image of which the transmission axis angle is 90 degrees
  • the specific color polarization characteristic detection unit 34-1 is a polarization pixel in which the pixel value of the green non-polarization pixel is Ic, the transmission axis angle is 135 degrees, the pixel value of the polarization pixel is I0, and the transmission axis angle is 90 degrees.
  • the polarization model is detected for each pixel, assuming that the pixel value of the pixel is I1.
  • the specific color polarization characteristic detection unit 34-1 outputs the detected polarization characteristic of the specific color to the non-specific color polarization characteristic detection unit 35.
  • the non-specific color polarization characteristic detection unit 35 generates the non-specific color non-polarization component image signal obtained by the non-polarization component image generation unit 32-1 and the non-specific color polarization signal obtained by the polarization component image generation unit 33-1. Based on the component image signal and the polarization characteristic of the specific color detected by the specific color polarization characteristic detection unit 34-1, the polarization characteristic of the non-specific color is detected.
  • the nonspecific color polarization characteristic detection unit 35 detects the nonspecific color polarization characteristics on the assumption that the azimuth angle ⁇ is the same between the specific color and the nonspecific color. If the azimuth angle ⁇ is the same for the specific color and the non-specific color, the unknown parameters in equation (12) are the parameter c and the degree of polarization ⁇ . Therefore, the nonspecific color polarization characteristic detection unit 35 uses the pixel value Ic of the nonspecific color nonpolarization pixel obtained by the nonpolarization component image generation unit 32-1 as the parameter c, and the degree of polarization based on Equation (10). Calculate ⁇ .
  • the non-specific colors are red and blue.
  • a non-polarization component image of red and blue is obtained by the non-polarization component image generation unit 32-1, and a polarization component image of red having a transmission axis angle of 90 degrees by the polarization component image generation unit 33-1.
  • a blue polarization component image is obtained in which the angle of the transmission axis is 135 degrees.
  • 8 shows a red component image and a blue component image
  • FIG. 8 (a) shows a red non-polarization component image
  • FIG. 8 (b) shows a red polarization component image
  • FIG. 8 (c) Is a blue non-polarization component image
  • (d) of FIG. 8 is a blue polarization component image.
  • the nonspecific color polarization characteristic detection unit 35 sets the red non-polarization pixel as the pixel value Ic, the transmission axis (90 degrees) of the red polarization pixel as the polarization angle x, and the pixel value of the red polarization pixel as the observation luminance y.
  • the degree of polarization ⁇ ⁇ ⁇ ⁇ for is calculated for each pixel.
  • the nonspecific color polarization characteristic detection unit 35 sets the blue non-polarization pixel as the pixel value Ic, the transmission axis of the blue polarization pixel (135 degrees) as the polarization angle x, and the pixel value as the observation luminance y for blue.
  • the degree of polarization ⁇ is calculated for each pixel. Therefore, the non-specific color polarization characteristic detection unit 35 can detect polarization characteristics of non-specific colors even if the polarization direction of non-specific colors is smaller than the specific color.
  • FIG. 9 illustrates a polarization characteristic (polarization model), and FIG. 9A shows a polarization model of a specific color (green component) detected by the specific color polarization characteristic detection unit 34-1. Further, FIG. 9 (b) shows a polarization model of the non-specific color (for example, red component) detected by the non-specific color polarization characteristic detection unit 34-1, and FIG. 9 (c) shows non-specific color polarization. The polarization model of the non-specific color (for example, blue component) detected by the characteristic detection unit 34-1 is shown. In FIG. 9, black circles indicate pixel values obtained for the polarization pixels. Further, the green non-polarized pixel has a pixel value c G , the red non-polarized pixel has a pixel value c R , and the blue non-polarized pixel has a pixel value c B.
  • FIG. 10 is a flowchart showing the operation of the first embodiment.
  • the image processing apparatus divides the polarization component.
  • the image processing device 30-1 divides an image signal of a polarized RAW image supplied from the solid-state imaging device 20 into a signal of a non-polarization pixel and a signal of a polarization pixel to generate a non-polarization image signal and a polarization image signal. Proceed to step ST2.
  • step ST2 the image processing apparatus generates a non-polarization component image.
  • the image processing device 30-1 generates a non-polarization component image signal for each color from the non-polarization image signal obtained in step ST1, and proceeds to step ST3.
  • step ST3 the image processing apparatus generates a polarization component image.
  • the image processing device 30-1 generates polarization component image signals for each color and each polarization direction from the polarization image signal obtained in step ST1, and proceeds to step ST4.
  • step ST4 the image processing apparatus detects polarization characteristics of a specific color.
  • the image processing device 30-1 detects the polarization characteristic of the specific color based on the polarization component image signal of the specific color generated in step ST3 and the non-polarization component image signal of the specific color generated in step ST2, and executes step ST5. Go to
  • step ST5 the image processing apparatus detects polarization characteristics of non-specific colors.
  • the image processing device 30-1 detects the polarization characteristic of the specific color detected in step ST4, the non-polarization component image signal of the non-specific color generated in step ST2, and the polarization component of the non-specific color generated in step ST3. Polarization characteristics of non-specific colors are detected based on the image signal.
  • the polarization direction of polarization pixels can be made smaller than that of specific colors, so degradation of image quality due to the provision of polarization pixels can be reduced.
  • the resolution (number of pixels) of the non-polarization image output from the image processing device 30-1 is vertical and horizontal with respect to the image acquired by the solid-state imaging device 20 It is 1/2) times.
  • the resolution of the non-polarization image and the polarization image output from the image processing apparatus is made equal to the polarization RAW image supplied from the solid-state imaging device.
  • FIG. 11 illustrates the configuration of the second embodiment.
  • the imaging system 10-2 includes a solid-state imaging device 20 and an image processing device 30-2.
  • the image processing device 30-2 includes a non-polarization component image generation unit 32-2, a polarization component image generation unit 33-2, a specific color polarization characteristic detection unit 34-2, and a non-specific color polarization characteristic detection unit 35. .
  • the solid-state imaging device 20 is configured in the same manner as in the first embodiment, and outputs an image signal of a polarized RAW image on which white balance adjustment is performed to the image processing device 30-2.
  • the non-polarization component image generation unit 32-2 of the image processing device 30-2 performs a filtering process using the image signal of the polarized RAW image supplied from the solid-state imaging device 20 to generate a non-polarization component image signal for each color. Do. Further, the non-polarization component image generation unit 32-2 outputs the non-polarization component image signal of the specific color to the specific-color polarization characteristic detection unit 34-2, and the non-polarization component image signal of the non-specific color is non-specific color polarization Output to the characteristic detection unit 35.
  • the polarization component image generation unit 33-2 performs filter processing using the image signal of the polarized raw image supplied from the solid-state imaging device 20, and generates polarization component image signals for each color and each polarization direction. Further, the polarization component image generation unit 33-2 outputs the polarization component image signal of each polarization direction of the specific color to the polarization characteristic detection unit 34-2 of the specific color, and the polarization component image signal of each polarization direction of the nonspecific color Are output to the nonspecific color polarization characteristic detection unit 35.
  • the specific color polarization characteristic detection unit 34-2 detects the polarization characteristic of the specific color based on the non-polarization component image signal and the polarization component image signal of the specific color. Also, the detected polarization characteristic of the specific color is output to the non-specific color polarization characteristic detection unit 35.
  • the non-specific color polarization characteristic detection unit 35 is a specific color, which is the non-specific color non-specific pixel pixel signal and the non-specific color non-specific pixel polarization signal detected by the specific color polarization characteristic detection unit 34-1.
  • the polarization characteristic of the non-specific color is detected based on the pixel signal of the polarization pixel of a type having a smaller polarization direction than that of the other.
  • the solid-state imaging device 20 has the configuration shown in FIG. 2 described above, and has, for example, a pixel array shown in FIG.
  • the non-polarization component image generation unit 32-2 of the image processing device 30-2 performs a filtering process using the image signal of the polarized RAW image supplied from the solid-state imaging device 20 to generate a non-polarization component image signal for each color. Do.
  • the non-polarization component image generation unit 32-2 calculates non-polarization low-frequency components for each color at the target pixel position, using pixel values of non-polarization pixels for each color from the peripheral region based on the target pixel position. Further, the non-polarization component image generation unit 32-2 divides the pixel value of the target pixel position by the non-polarization low frequency component of the same color as the target pixel position to calculate the high frequency component of the target pixel position.
  • the non-polarization component image generation unit 32-2 multiplies the high frequency component of the target pixel position by the non-polarization low frequency component for each color of the target pixel position to obtain the non-polarization pixel value for each color at the target pixel position. calculate.
  • FIG. 12 is a diagram for explaining the operation of the non-polarization component image generation unit 32-2, and (a) of FIG. 12 shows the pixel arrangement of the polarization RAW image supplied from the solid-state imaging device 20.
  • (B) of FIG. 12 illustrates filter coefficients. For example, when the non-polarized blue pixel shown in (a) of FIG. 12 is the target pixel position Ps, the pixel value of the non-polarized pixel of the same color is used from the peripheral region based on the target pixel position. Filter processing is performed using the filter coefficients shown in (b) to calculate non-polarization low-frequency components for each color.
  • the filter coefficient corresponding to the green non-polarization pixel with respect to the pixel value of the green non-polarization pixel in the peripheral region based on the target pixel position And divide the sum of the multiplication results by the sum of the filter coefficients used for the multiplication to calculate the low frequency component of the green non-polarization pixel at the target pixel position.
  • the non-polarization component image generation unit 32-2 divides the pixel value of the target pixel position by the low frequency component of the same color and the same polarization direction as the target pixel position to calculate the high frequency component of the target pixel position. For example, when the non-polarized blue pixel shown in (a) of FIG. 12 is the target pixel position Ps, the pixel value of the non-polarized blue pixel which is the target pixel position Ps is calculated as no blue pixel calculated at the target pixel position. The high frequency component is calculated by dividing by the polarized low frequency component.
  • the non-polarization component image generation unit 32-2 individually multiplies the high frequency component of the target pixel position and the non-polarization low frequency component of each color of the target pixel position to calculate the non-polarization pixel value of each color. For example, when calculating the green non-polarization pixel value of the target pixel position, the high-frequency component calculated at the target pixel position is multiplied by the green non-polarization low frequency component calculated at the target pixel position to obtain the green non-polarization pixel value. calculate.
  • the non-polarization component image generation unit 32-2 multiplies the non-polarization low-frequency component for each color calculated at the target pixel position by the high-frequency component calculated at the target pixel position to obtain horizontal and vertical resolutions.
  • a non-polarization component image signal for each color is generated without reducing (the number of pixels).
  • the non-polarization component image generation unit 32-2 outputs the non-polarization image signal of the specific color to the specific-color polarization characteristic detection unit 34-2, and the non-polarization image signal of the non-specific color is Output to
  • the polarization component image generation unit 33-2 performs a filtering process using the image signal of the polarized RAW image supplied from the solid-state imaging device 20, and generates a polarization image signal for each color and each polarization direction.
  • the polarization component image generation unit 33-2 uses the pixel value of the polarization pixel for each color and for each polarization direction from the peripheral region with respect to the target pixel position, and generates polarization low frequency for each color and for each polarization direction at the target pixel position. Calculate the components.
  • the polarization component image generation unit 33-2 divides the pixel value of the target pixel position by the low frequency component of the same color and the same polarization direction as the target pixel position to calculate the high frequency component of the target pixel position.
  • the polarization component image generation unit 33-2 multiplies the high frequency component of the target pixel position by the polarization low frequency component of each color and polarization direction of the target pixel position to obtain each color and polarization direction at the target pixel position. Calculate the polarization pixel value of.
  • the pixel values of polarization pixels of the same color and the same polarization direction are used from the peripheral region based on the target pixel position.
  • Filter processing is performed using the filter coefficients shown in (b) of FIG. 12 to calculate polarization low frequency components for each color and each polarization direction.
  • the pixel value of the red polarization pixel in the peripheral region based on the target pixel position is multiplied by the filter coefficient corresponding to the red polarization pixel.
  • the low frequency component of the red polarization pixel at the target pixel position is calculated by dividing the sum of the multiplication results by the sum of the filter coefficients used for the multiplication.
  • the polarization component image generation unit 33-2 divides the pixel value of the target pixel position by the low frequency component of the same color and the same polarization direction as the target pixel position to calculate the high frequency component of the target pixel position. For example, when the non-polarized blue pixel shown in (a) of FIG. 12 is the target pixel position Ps, the pixel value of the non-polarized blue pixel which is the target pixel position Ps is calculated as no blue pixel calculated at the target pixel position. The high frequency component is calculated by dividing by the polarized low frequency component.
  • the polarization component image generation unit 33-2 individually multiplies the high frequency component of the target pixel position and the polarization low frequency component of each color and polarization direction of the target pixel position to obtain polarization pixels for each color and polarization direction. Calculate the value. For example, when calculating the red polarization pixel value of the target pixel position, the red polarization pixel value is calculated by multiplying the high frequency component calculated at the target pixel position and the red polarization low frequency component calculated at the target pixel position.
  • the polarization component image generation unit 33-2 multiplies the polarization low frequency components for each color and for each polarization direction calculated at the target pixel position by the high frequency components calculated at the target pixel position, and generates the horizontal and vertical directions.
  • the polarization component image generation unit 33-2 outputs the polarization image signal of the specific color to the specific color polarization characteristic detection unit 34-2, and outputs the polarization image signal of the nonspecific color to the nonspecific color polarization characteristic detection unit 35.
  • the specific color polarization characteristic detection unit 34-2 generates the non-polarization image signal of the specific color obtained by the non-polarization component image generation unit 32-2 and the polarization image signal of the specific color obtained by the polarization component image generation unit 33-2. Based on the above, processing similar to that of the first embodiment is performed to detect the polarization characteristic of a specific color.
  • the specific color polarization characteristic detection unit 34-2 outputs the detected polarization characteristic of the specific color to the non-specific color polarization characteristic detection unit 35.
  • the non-specific color polarization characteristic detection unit 35 generates the non-specific color non-polarization image signal obtained by the non-polarization component image generation section 32-2 and the non-specific color polarization image obtained by the polarization component image generation section 33-2. Based on the signal and the polarization characteristic of the specific color detected by the specific color polarization characteristic detection unit 34-2, the same processing as in the first embodiment is performed to detect the polarization characteristic of the non-specific color.
  • FIG. 13 is a flow chart showing the operation of the second embodiment.
  • the image processing apparatus generates a non-polarization component image.
  • the image processing device 30-2 performs a filtering process using the signal of the non-polarization pixel in the same color from the image signal of the polarized RAW image supplied from the solid-state imaging device 20, and generates the non-polarization component image signal for each color Then, the process proceeds to step ST12.
  • the image processing apparatus generates a polarization component image in step ST12.
  • the image processing device 30-2 performs a filtering process using the signal of the polarization pixel having the same color and the same polarization direction from the image signal of the polarized RAW image supplied from the solid-state imaging device 20, for each color and each polarization direction. Is generated, and the process proceeds to step ST13.
  • step ST13 the image processing apparatus detects the polarization characteristic of a specific color.
  • the image processing device 30-2 detects the polarization characteristic of the specific color based on the polarization component image signal of the specific color generated in step ST12 and the non-polarization component image signal of the specific color generated in step ST11, and executes step ST14. Go to
  • step ST14 the image processing apparatus detects polarization characteristics of non-specific colors.
  • the image processing device 30-2 detects the polarization characteristic of the specific color detected in step ST13, the non-polarization component image signal of the non-specific color generated in step ST11, and the polarization component of the non-specific color generated in step ST12.
  • the polarization characteristics of non-specific colors are detected based on the image signal.
  • the degree of polarization for each color is different, as in the first embodiment. Even in this case, polarization characteristics can be detected accurately for each color.
  • the polarization direction of polarization pixels can be made smaller than that of specific colors, so degradation of image quality due to the provision of polarization pixels can be reduced. Furthermore, it is possible to generate non-polarization images for each color and polarization images for each color and polarization direction without causing a reduction in resolution.
  • FIG. 14 illustrates the configuration of the third embodiment.
  • the imaging system 10-3 includes a solid-state imaging device 20 and an image processing device 30-3.
  • the image processing device 30-3 includes a non-polarization component image generation unit 32-3, a polarization component image generation unit 33-3, a specific color polarization characteristic detection unit 34-3, and a non-specific color polarization characteristic detection unit 35. .
  • the solid-state imaging device 20 is configured in the same manner as in the first embodiment, and the pixel arrangement is different from that in the first embodiment.
  • FIG. 15 illustrates the pixel array of the solid-state imaging device, and illustrates the case where the polarization direction of the green pixel is four directions, and the polarization direction of the red pixel and the blue pixel is one direction.
  • the solid-state imaging device 20 outputs, to the image processing device 30-3, an image signal of a polarized raw image on which white balance adjustment is performed.
  • the image processing device 30-3 generates a non-polarization image for each color, a polarization image for each color and each polarization direction, and detects polarization characteristics for each color regardless of the pixel arrangement of the solid-state imaging device 20.
  • the non-polarization component image generation unit 32-3 of the image processing device 30-3 performs a filtering process using the image signal of the polarized RAW image supplied from the solid-state imaging device 20 to generate a non-polarization component image signal for each color. Do.
  • the non-polarization component image generation unit 32-3 outputs the non-polarization component image signal of the specific color to the specific-color polarization characteristic detection unit 34-3, and detects the non-specific color polarization characteristic signal of the non-specific color Output to section 35.
  • the polarization component image generation unit 33-3 performs a filtering process using the image signal of the polarization RAW image supplied from the solid-state imaging device 20, and generates a polarization component image signal for each color and each polarization direction.
  • the polarization component image generation unit 33-3 outputs the polarization component image signal of the specific color to the specific color polarization characteristic detection unit 34-3, and the polarization component image signal of the nonspecific color to the nonspecific color polarization characteristic detection unit 35. Output.
  • the specific color polarization characteristic detection unit 34-3 detects the polarization characteristic of the specific color based on the polarization component image signal of the specific color or the non-polarization component image signal of the specific color and the polarization component image signal. Also, the detected polarization characteristic of the specific color is output to the non-specific color polarization characteristic detection unit 35.
  • the non-specific color polarization characteristic detection unit 35 detects the non-specific color polarization characteristic based on the non-specific color non-polarization component image signal, the polarization component image signal, and the specific color polarization characteristic.
  • the solid-state imaging device 20 has the configuration shown in FIG. 2 described above, and has, for example, a pixel array shown in FIG.
  • the non-polarization component image generation unit 32-3 of the image processing device 30-3 performs a filtering process using the image signal of the polarized RAW image supplied from the solid-state imaging device 20 to generate a non-polarization component image signal for each color. Do.
  • the non-polarization component image generation unit 32-3 uses the same color and non-polarization pixel value located around the target pixel position as the target pixel position with the center position of the color area unit which is 2 ⁇ 2 pixels as the target pixel position. A non-polarization pixel value is calculated for each color at the position.
  • FIG. 16 is a diagram for explaining the operation of the non-polarization component image generation unit 32-3.
  • FIG. 16 (a) shows the pixel arrangement of the image signal supplied from the solid-state imaging device 20.
  • (B) of FIG. 16 exemplifies the filter coefficient.
  • the filter coefficient shown in (a) of FIG. 16 when the center position of the blue color area unit of 2 ⁇ 2 pixels is the target pixel position Ps, the same color and no position located in the periphery with respect to the target pixel position The filter process is performed using the filter coefficient shown in FIG. 16B using the polarization pixel value to calculate the non-polarization pixel value of the non-specific color pixel at the target pixel position.
  • the pixel value of the non-polarized blue pixel located in the periphery is multiplied by the filter coefficient corresponding to the non-polarized blue pixel, and the sum of the multiplication results is used for multiplication.
  • the pixel value of the non-polarized blue pixel at the target pixel position is calculated by dividing by the sum of the filter coefficients.
  • the same process is performed using a non-polarized red pixel to calculate a non-polarized pixel value of the red pixel at the target pixel position.
  • FIG. 17 shows a non-polarization component image and a polarization component image.
  • the non-polarization component image generation unit 32-3 calculates non-specific color non-specific color pixel values at the target pixel position to set the resolution (number of pixels) in the horizontal and vertical directions to 1 ⁇ 2 (see FIG. 17).
  • the red non-polarized component image shown in (a) and the blue non-polarized component image shown in (b) of FIG. 17 are generated.
  • the non-polarization component image generation unit 32-3 outputs the non-specific color non-polarization component image signal to the non-specific color polarization characteristic detection unit 35.
  • the polarization component image generation unit 33-3 performs filter processing using the image signal of the polarized raw image supplied from the solid-state imaging device 20, and generates polarization component image signals for each color component and each polarization direction. Similar to the non-polarization component image generation unit 32-3, the polarization component image generation unit 33-3 has the same center position as the target pixel position with the center position of the color area unit of 2 ⁇ 2 pixels as the target pixel position. The polarization pixel value of each color and polarization direction at the target pixel position is calculated using the color and polarization pixel values of the same polarization direction.
  • the polarization component image generation unit 33-3 references the target pixel position.
  • Filter processing is performed using the filter coefficients shown in (b) of FIG. 16 using the pixel values of polarization pixels of the same color and the same polarization direction located in the periphery, and the polarization pixel values of the target pixel position for each color and Calculated for each polarization direction.
  • the polarization pixel value of the blue pixel in the same polarization direction located in the periphery is multiplied by the filter coefficient corresponding to the blue pixel of the same polarization, and the sum of the multiplication results is obtained.
  • the polarization pixel value of the blue pixel at the target pixel position is calculated by dividing by the sum of the filter coefficients used for the multiplication. The same process is performed using red and green pixels to calculate polarization pixel values of the red pixel at the target pixel position and the green pixel for each polarization direction.
  • the polarization component image generation unit 33-3 calculates the pixel value of the polarization pixel for each color and each polarization direction at the target pixel position, so that the resolution (the number of pixels) in the horizontal and vertical directions is 1 ⁇ 2.
  • Polarized images are generated for each color and each polarization direction shown in (c) to (h) of FIG. Note that (c) to (f) in FIG. 17 show polarization component images for each polarization direction of green, (g) in FIG. 17 shows polarization components images in red, and (h) in FIG. ing.
  • the polarization component image generation unit 33-3 outputs the green polarization component image signal of the specific color to the specific color polarization characteristic detection unit 34-3, and the non-specific color red and blue polarization component image signals are not transmitted. Output to the specific color polarization characteristic detection unit 35.
  • the specific color polarization characteristic detection unit 34-3 detects the polarization characteristic of the specific color based on the polarization component image signal of the specific color obtained by the polarization component image generation unit 33-3.
  • the polarization component image generation unit 33-3 generates polarization component images of four polarization directions as shown in (c) to (h) of FIG. Therefore, the parameters a, b and c of the above equation (6) are calculated based on the equations (13) to (15).
  • a (I45-I135) / 2 (13)
  • b (I0-I90) / 2 (14)
  • c (I0 + I45 + I90 + I135) / 4 (15)
  • the polarization degree ⁇ is calculated based on Expression (10), and the azimuth angle ⁇ is calculated based on Expression (11).
  • the polarization characteristic shown in the above-mentioned equation (12) can be obtained.
  • the specific color polarization characteristic detection unit 34-3 outputs the detected polarization characteristic of the specific color to the non-specific color polarization characteristic detection unit 35.
  • the non-specific color polarization characteristic detection unit 35 generates the non-specific color non-polarization component image signal obtained by the non-polarization component image generation section 32-3 and the non-specific color polarization signal obtained by the polarization component image generation section 33-3. Based on the component image signal and the polarization characteristic of the specific color detected by the specific color polarization characteristic detection unit 34-3, the same processing as in the first embodiment is performed to detect the polarization characteristic of the non-specific color.
  • non-polarization component image signals of non-specific colors are generated, and non-polarization component image signals of specific colors are not generated.
  • the average value of polarization pixel values is calculated for each pixel position using polarization images of four polarization directions with the transmission axis having an angle difference of 45 degrees, even if non-polarization pixels of a specific color are not provided It is possible to generate non-polarization component images of a specific color.
  • the same processing as the flowchart shown in FIG. 13 is performed, and the above-described filter processing different from the second embodiment is performed to generate a non-polarization component image and a polarization component image. And detect polarization characteristics for each color.
  • the third embodiment as in the first embodiment, even when the degree of polarization for each color is different, it is possible to accurately detect the polarization characteristic for each color. Become.
  • the polarization direction of polarization pixels can be made smaller than that of specific colors, so degradation of image quality due to provision of polarization pixels can be reduced.
  • generation of non-polarization component images and polarization component images and polarization characteristics for each color can be detected for each color region unit, it is possible to cope with various pixel arrangements of a solid-state imaging device.
  • FIG. 18 illustrates the configuration of the fourth embodiment.
  • the imaging system 10-4 includes a solid-state imaging device 20 and an image processing device 30-4.
  • the image processing device 30-4 includes a non-polarization component image generation unit 32-4, a polarization component image generation unit 33-4, a specific color polarization characteristic detection unit 34-4, an non-specific color polarization characteristic detection unit 35, and a color balance coefficient calculation.
  • the color balance adjustment unit 38 is provided.
  • the solid-state imaging device 20 has the same pixel arrangement as in the first to third embodiments, and outputs an image signal of a polarized RAW image on which white balance adjustment is not performed to the image processing device 30-4.
  • the non-polarization component image generation unit 32-4 of the image processing device 30-4 performs a filtering process using the image signal of the polarized RAW image supplied from the solid-state imaging device 20 to generate a non-polarization component image signal for each color. Do.
  • the non-polarization component image generation unit 32-4 outputs the non-polarization component image signal of the specific color to the specific-color polarization characteristic detection unit 34-4, and detects the non-specific color polarization characteristic signal of the non-specific color Output to section 35.
  • the polarization component image generation unit 33-4 performs filter processing using the image signal of the polarized raw image supplied from the solid-state imaging device 20, and generates polarization component image signals for each color and each polarization direction.
  • the polarization component image generation unit 33-4 outputs the polarization component image signal of the specific color to the specific color polarization characteristic detection unit 34-4, and the polarization component image signal of the nonspecific color to the nonspecific color polarization characteristic detection unit 35. Output.
  • the specific color polarization characteristic detection unit 34-4 detects the polarization characteristic of the specific color based on the non-polarization component image signal and the polarization component image signal of the specific color, and outputs the polarization characteristic detection unit 35 to the non-specific color polarization characteristic detection unit 35.
  • the non-specific color polarization characteristic detection unit 35 detects the non-specific color polarization characteristic based on the non-specific color non-polarization component image, the polarization component image signal, and the specific color polarization characteristic.
  • the nonspecific color polarization characteristic detection unit 35 outputs the detected polarization characteristics of the nonspecific color and the polarization characteristics of the specific color detected by the specific color polarization characteristic detection unit 34-4 to the color balance coefficient calculation unit 36.
  • the color balance coefficient calculation unit 36 calculates the color balance coefficient based on the polarization characteristic of the specific color and the polarization characteristic of the non-specific color, and outputs the color balance coefficient to the color balance adjustment unit 38.
  • the color balance adjustment unit 38 performs white balance adjustment of the non-polarization component image and the polarization component image using the color balance coefficient calculated by the color balance coefficient calculation unit 36.
  • the solid-state imaging device 20 has the configuration shown in FIG. 2 described above, and has, for example, a pixel array shown in FIG. 3 and FIG.
  • the non-polarization component image generation unit 32-4 of the image processing device 30-4 performs the same processing as any of the non-polarization component image generation units 32-2 and 32-3 of the second or third embodiment.
  • the polarization component image generation unit 33-4, the specific color polarization characteristic detection unit 34-4, and the non-specific color polarization characteristic detection unit 35 perform the same processing as any of the second to third embodiments.
  • the polarization component division unit 31 of the first embodiment may be provided to perform the same processing as that of the first embodiment.
  • the nonspecific color polarization characteristic detection unit 35 outputs the detected polarization characteristics of the nonspecific color and the polarization characteristics of the specific color detected by the specific color polarization characteristic detection unit 34-4 to the color balance coefficient calculation unit 36.
  • the color balance coefficient calculation unit 36 calculates the color balance coefficient based on the polarization characteristic of the specific color and the polarization characteristic of the non-specific color. When calculating the color balance coefficient based on the polarization characteristic, the color balance coefficient calculator 36 sets the specular reflection component to white.
  • the amplitude c ⁇ of the polarization characteristic shown in the above equation (12) indicates the specular reflection component, and when the specular reflection component is white, the amplitude of the polarization characteristic of a specific color and the amplitude of the polarization characteristic of the nonspecific color are equal. . Therefore, the color balance coefficient calculation unit 36 calculates, as the color balance coefficient, a gain that makes the amplitude of the polarization characteristic of the non-specific color equal to the amplitude of the polarization characteristic of the specific color.
  • FIG. 19 is a diagram for explaining the color balance coefficient.
  • the specular reflection component is white
  • the amplitudes of the polarization characteristics of the specific color and the non-specific color are equal, and the specific color is green as described above
  • the red for the green polarization image amplitude c G G G
  • the amplitude c R R R of the polarized light image and the amplitude c B B B of the blue polarized light image have a relationship of equation (16).
  • a green non-polarization pixel is pixel value c G
  • a red non-polarization pixel is pixel value c R
  • a blue non-polarization pixel is pixel value c B
  • green is polarization degree G G
  • red is polarization degree R R
  • Blue is the degree of polarization B B. Therefore, the color balance coefficient calculation unit 36 calculates the gains g R and g B as color balance coefficients.
  • the color balance coefficient calculation unit 36 calculates the color balance coefficients g R and g B by performing calculations of equations (17) and (18) based on the polarization characteristics of the specific color and the polarization characteristics of the non-specific color. Output to the color balance adjustment unit 38.
  • g R (c G G G ) / (c R R R ) (17)
  • g B (c G G G ) / (c B B B ) (18)
  • the color balance adjustment unit 38 adjusts the gain of the non-specific color non-polarization image or the polarization image using the color balance coefficient calculated by the color balance coefficient calculation unit 36, and performs no white balance adjustment for each color.
  • a polarization component image signal or polarization component image signal for each color and for each polarization direction is generated.
  • FIG. 20 is a flowchart showing the operation of the fourth embodiment.
  • the image processing apparatus generates a non-polarization component image.
  • the image processing device 30-4 performs a filtering process using the signal of the non-polarized pixel in the same color from the image signal of the polarized RAW image supplied from the solid-state imaging device 20 to generate the non-polarized component image signal for each color. Then, the process proceeds to step ST22.
  • the image processing apparatus generates a polarization component image in step ST22.
  • the image processing device 30-4 performs a filtering process using the signal of the polarization pixel having the same color and the same polarization direction from the image signal of the polarized RAW image supplied from the solid-state imaging device 20, and for each color and each polarization direction. Is generated, and the process proceeds to step ST23.
  • step ST23 the image processing apparatus detects the polarization characteristic of a specific color.
  • the image processing device 30-4 detects the polarization characteristic of the specific color based on the polarization component image signal of the specific color generated in step ST22 and the non-polarization component image signal of the specific color generated in step ST21, and executes step ST24. Go to
  • step ST24 the image processing apparatus detects polarization characteristics of non-specific colors.
  • the image processing device 30-4 controls the polarization characteristics of the specific color detected in step ST23, the non-polarization component image signal of the non-specific color generated in step ST21, and the polarization components of the non-specific color generated in step ST22.
  • the polarization characteristic of the non-specific color is detected based on the image signal, and the process proceeds to step ST25.
  • step ST25 the image processing apparatus calculates a color balance coefficient.
  • the image processing device 30-4 determines the amplitude of the polarization characteristic of the nonspecific color and the polarization characteristic of the specific color based on the polarization characteristic of the specific color detected in step ST23 and the polarization characteristic of the nonspecific color detected in step ST24.
  • the gain to be equalized is calculated as a color balance coefficient, and the process proceeds to step ST26.
  • step ST26 the image processing apparatus performs color balance adjustment.
  • the image processing device 30-4 performs gain adjustment of the non-specific color non-polarization component image signal generated in step ST21 using the color balance coefficient calculated in step ST25. Further, the image processing device 30-4 may perform gain adjustment of the non-specific color polarization component image signal generated in step ST22 using the color balance coefficient calculated in step ST25.
  • the fourth embodiment as in the first embodiment, even when the degree of polarization for each color is different, it is possible to accurately detect the polarization characteristic for each color. become.
  • the polarization direction of polarization pixels can be made smaller than that of specific colors, so degradation of image quality due to the provision of polarization pixels can be reduced.
  • the color balance coefficient can be calculated based on the polarization characteristic, it is possible to obtain an image signal on which white balance adjustment is performed with high accuracy even when the illumination light source is unknown or a white subject is not included in the imaging region. It will be.
  • FIG. 21 illustrates the configuration of the fifth embodiment.
  • the imaging system 10-5 includes a solid-state imaging device 20 and an image processing device 30-5.
  • the image processing device 30-5 includes a non-polarization component image generation unit 32-5, a polarization component image generation unit 33-5, a specific color polarization characteristic detection unit 34-5, an non-specific color polarization characteristic detection unit 35, and a color balance coefficient calculation.
  • a section 36, a color balance coefficient adjustment section 37, and a color balance adjustment section 38 are provided.
  • the solid-state imaging device 20 has the same pixel arrangement as in the first to third embodiments, and outputs an image signal of a polarized RAW image on which white balance adjustment is not performed to the image processing device 30-5.
  • the non-polarization component image generation unit 32-5 of the image processing device 30-5 performs a filtering process using the image signal of the polarized RAW image supplied from the solid-state imaging device 20, and generates a non-polarization component image signal for each color. Do.
  • the non-polarization component image generation unit 32-5 outputs the non-polarization component image signal of the specific color to the specific-color polarization characteristic detection unit 34-5, and detects the non-specific color polarization characteristic signal of the non-specific color Output to section 35.
  • the polarization component image generation unit 33-5 performs a filtering process using the image signal of the polarized raw image supplied from the solid-state imaging device 20, and generates a polarization component image signal for each color and each polarization direction.
  • the polarization component image generation unit 33-5 outputs the polarization component image signal of the specific color to the specific color polarization characteristic detection unit 34-5, and the polarization component image signal of the nonspecific color to the nonspecific color polarization characteristic detection unit 35. Output.
  • the specific color polarization characteristic detection unit 34-5 detects the polarization characteristic of the specific color based on the non-polarization component image signal and the polarization component image signal of the specific color, and outputs the polarization characteristic detection unit 35 to the non-specific color polarization characteristic detection unit 35.
  • the non-specific color polarization characteristic detection unit 35 detects the non-specific color polarization characteristic based on the non-specific color non-polarization component image, the polarization component image signal, and the specific color polarization characteristic.
  • the nonspecific color polarization characteristic detection unit 35 outputs the detected polarization characteristics of the nonspecific color and the polarization characteristics of the specific color detected by the specific color polarization characteristic detection unit 34-5 to the color balance coefficient calculation unit 36.
  • the color balance coefficient calculation unit 36 calculates the color balance coefficient based on the polarization characteristic of the specific color and the polarization characteristic of the non-specific color, and outputs the color balance coefficient to the color balance adjustment unit 38.
  • the color balance coefficient adjustment unit 37 adjusts the color balance coefficient using the color balance coefficient based on the color information for reproducing a white object as white and the color balance coefficient calculated by the color balance coefficient calculation unit 36, and adjusts the color balance Generate coefficients.
  • the color balance coefficient based on the color information may be acquired from the solid-state imaging device 20 or may be stored in the image processing device 30-5.
  • the color balance coefficient adjustment unit 37 outputs the adjusted color balance coefficient to the color balance adjustment unit 38.
  • the color balance adjustment unit 38 performs white balance adjustment of a non-polarization image or a polarization image using the adjustment color balance coefficient generated by the color balance coefficient adjustment unit 37.
  • the solid-state imaging device 20 has the configuration shown in FIG. 2 described above, and has, for example, a pixel array shown in FIG. 3 and FIG.
  • the non-polarization component image generation unit 32-5 of the image processing device 30-5 performs the same processing as any of the non-polarization component image generation units 32-2 and 32-3 of the second or third embodiment.
  • the polarization component image generation unit 33-5, the specific color polarization characteristic detection unit 34-5, and the non-specific color polarization characteristic detection unit 35 perform the same processing as any of the second to third embodiments.
  • the polarization component division unit 31 of the first embodiment may be provided to perform the same processing as that of the first embodiment.
  • the nonspecific color polarization characteristic detection unit 35 outputs the detected polarization characteristics of the nonspecific color and the polarization characteristics of the specific color detected by the specific color polarization characteristic detection unit 34-5 to the color balance coefficient calculation unit 36.
  • the color balance coefficient calculation unit 36 calculates the color balance coefficient based on the polarization characteristic of the specific color and the polarization characteristic of the non-specific color, as in the fourth embodiment.
  • the color balance coefficient calculation unit 36 outputs the color balance coefficient based on the calculated polarization characteristic to the color balance coefficient adjustment unit 37.
  • the color balance coefficient adjustment unit 37 adjusts the color balance coefficient using the color balance coefficient based on color information and the color balance coefficient based on the polarization characteristic calculated by the color balance coefficient calculation unit 36, and generates an adjusted color balance coefficient Do.
  • color balance coefficient adjusting unit 37 calculates a difference absolute value between the red color balance coefficients g R2 based on polarization characteristic calculated by the red color balance coefficients g R1 and color balance coefficient calculating unit 36 based on the color information Do.
  • Color balance coefficient adjusting unit 37 if the absolute value difference is within the threshold value, the red color balance coefficients g R2 based on the polarization characteristics as an adjustment color balance factor g R, if the absolute value difference exceeds the threshold value, the color information the red color balance coefficients g R1 based and adjusting color balance factor g R.
  • the color balance coefficient adjusting unit 37 performs the same processing with respect to blue, also set similarly adjusted color balance factor g B.
  • the adjustment color balance coefficient can be generated based on the polarization characteristic even when the illumination light source is unknown or the white subject is not included in the imaging region.
  • the color balance coefficient based on the color information is used as the adjustment color balance coefficient, so the accuracy of the adjustment color balance coefficient significantly decreases. Can be prevented.
  • the color balance coefficient adjustment unit 37 may use an average value of the color balance coefficient based on color information and the color balance coefficient based on the polarization characteristic as the adjustment color balance coefficient.
  • the blend ratio ⁇ approaches 1 as the ratio of the color balance coefficient based on the polarization characteristic increases.
  • the blend ratio ⁇ is set according to, for example, specular reflection, and as the amount of specular reflection increases, the blend ratio ⁇ approaches 1 as the reliability of the color balance coefficient based on the polarization characteristic is high.
  • the color balance adjustment unit 38 performs gain adjustment of the non-specific color non-polarization image or polarization image using the adjustment color balance coefficient generated by the color balance coefficient adjustment unit 37, and performs the white balance adjustment for each color.
  • a non-polarization component image signal or polarization component image signal for each color and for each polarization direction is generated.
  • the color balance coefficient adjustment unit 37 presents the user with a color balance coefficient based on color information and a color balance coefficient based on the polarization characteristic calculated by the color balance coefficient calculation unit 36, and adjusts the color balance coefficient based on a user instruction. May be generated.
  • FIG. 22 is a flow chart showing the operation of the fifth embodiment.
  • the image processing apparatus generates a non-polarization component image.
  • the image processing device 30-5 performs a filtering process using the signal of the non-polarization pixel in the same color from the image signal of the polarized RAW image supplied from the solid-state imaging device 20 to generate the non-polarization component image signal for each color. Then, the process proceeds to step ST32.
  • step ST32 the image processing apparatus generates a polarization component image.
  • the image processing device 30-5 performs a filtering process using the signal of the polarization pixel having the same color and the same polarization direction from the image signal of the polarized RAW image supplied from the solid-state imaging device 20, for each color and each polarization direction. Is generated, and the process proceeds to step ST33.
  • the image processing apparatus detects the polarization characteristic of a specific color.
  • the image processing device 30-5 detects the polarization characteristic of the specific color based on the polarization component image signal of the specific color generated in step ST32 and the non-polarization component image signal of the specific color generated in step ST31, and executes step ST34. Go to
  • step ST34 the image processing apparatus detects polarization characteristics of non-specific colors.
  • the image processing device 30-5 detects the polarization characteristics of the specific color detected in step ST33, the non-polarization component image signal of the non-specific color generated in step ST31, and the polarization components of the non-specific color generated in step ST32.
  • the polarization characteristic of the non-specific color is detected based on the image signal, and the process proceeds to step ST35.
  • the image processing apparatus calculates a color balance coefficient in step ST35.
  • the image processing device 30-5 determines the amplitude of the polarization characteristic of the nonspecific color and the polarization characteristic of the specific color based on the polarization characteristic of the specific color detected in step ST33 and the polarization characteristic of the nonspecific color detected in step ST34.
  • the gain to be equalized is calculated as a color balance coefficient, and the process proceeds to step ST36.
  • the image processing apparatus adjusts the color balance coefficient in step ST36.
  • the image processing device 30-5 generates an adjusted color balance coefficient using the color balance coefficient based on the color information and the color balance coefficient based on the polarization characteristic calculated in step ST35, and proceeds to step ST37.
  • the image processing apparatus performs color balance adjustment in step ST37.
  • the image processing device 30-5 performs gain adjustment of the non-specific color non-polarization component image signal generated in step ST31 using the adjusted color balance coefficient generated in step ST36. Further, the image processing device 30-5 may perform gain adjustment of the non-specific color polarization component image signal generated in step ST32 using the adjustment color balance coefficient generated in step ST36.
  • the fifth embodiment As described above, according to the fifth embodiment, as in the first embodiment, even when the degree of polarization for each color is different, it is possible to accurately detect the polarization characteristic for each color. become.
  • the polarization direction of polarization pixels can be made smaller than that of specific colors, so degradation of image quality due to the provision of polarization pixels can be reduced.
  • the color balance coefficient can be calculated based on the polarization characteristic, it is possible to obtain an image signal on which white balance adjustment is performed with high accuracy even when the illumination light source is unknown or a white subject is not included in the imaging region. It will be. Further, even when the specular reflection component is small and the accuracy of the color balance coefficient based on the polarization characteristic is lowered, the white balance adjustment can be performed with high accuracy using the color balance coefficient based on the color information.
  • the image signal used by the image processing apparatus is not limited to the image signal generated by the solid-state imaging device of the above-described pixel array.
  • the solid-state imaging device will be described later in ⁇ 7.
  • Another configuration of the solid-state imaging device may be a pixel array shown in FIG.
  • the pixel color is not limited to the primary color system but may be a complementary color system.
  • the configuration is shown in which the polarization characteristics of the specific color and the non-specific color, the non-polarization component image signal, and the polarization component image signal are output.
  • the fourth and fifth embodiments show the configuration in which the image signal after white balance adjustment is output, the configuration may be such that the image signal before color balance coefficient and white balance adjustment is output.
  • the filtering process is not limited to the filtering process described in the description of the embodiment, and the filtering process described in the other embodiments may be used.
  • filter processing in the generation of a polarization image or a non-polarization image of the second embodiment may be used.
  • the non-polarization component image signal and the polarization component image signal having better image quality than the above-described embodiment can be generated, the polarization characteristics of the specific color and the non-specific color can be detected more accurately.
  • the flowchart showing the above-described operation is not limited to the order process of performing the process in the order of steps shown in the figure.
  • either the non-polarization image or the polarization image may be generated first, or may be performed in parallel.
  • pipeline processing may be performed, and for example, filter processing for each pixel or calculation of polarization characteristics may be performed each time a pixel signal necessary for the processing is obtained.
  • the solid-state imaging device 20 is not limited to the configuration of sequentially reading out the signal of each pixel in the pixel array shown in FIG. 3 and FIG. Next, as another configuration of the solid-state imaging device, a pixel array and readout of pixel signals will be described.
  • the pixel array of the solid-state imaging device 20 is specified by the pixel of the specific color configured to be able to detect the polarization characteristic, and the non-polarized pixel and the polarized pixel of a type having a smaller polarization direction than the specific color. It may be a configuration having non-specific color pixels different from the color.
  • FIGS. 23 and 24 illustrate a part of the pixel array (a pixel array repeat unit of 4 ⁇ 4 pixels).
  • the specific color is green
  • the color pixel area of 2 ⁇ 2 pixels is configured by polarization pixels of four directions having different polarization directions.
  • non-specific colors are red and blue
  • a 2 ⁇ 2 pixel color pixel area is configured of a polarization pixel in one direction and three non-polarization pixels.
  • a specific color is green, and a color pixel area of 2 ⁇ 2 pixels is configured of polarization pixels in three directions with different polarization directions and one non-polarization pixel. Further, non-specific colors are red and blue, and a 2 ⁇ 2 pixel color pixel area is configured of a polarization pixel in one direction and three non-polarization pixels.
  • the specific color is green
  • the color pixel area of 2 ⁇ 2 pixels is configured of polarization pixels in two directions different in polarization direction and two non-polarization pixels.
  • non-specific colors are red and blue
  • a 2 ⁇ 2 pixel color pixel area is configured of a polarization pixel in one direction and three non-polarization pixels.
  • FIG. 23 illustrate the case where polarization pixels are provided only in one of the two color area units of the specific color included in the pixel array repeating unit of 4 ⁇ 4 pixels.
  • the image processing device 20 performs the processing as described above to accurately perform polarization characteristics for each color. Can be detected. Also, the image processing apparatus can acquire a non-polarization image and a polarization image for each color.
  • the pixel array may have a configuration shown in FIG. (A) of FIG. 24 illustrates the case where the specific color is green and red and the non-specific color is blue.
  • (B) of FIG. 24 exemplifies the pixel arrangement in the case where the color area unit is one pixel unit.
  • (C) (d) of FIG. 24 exemplifies the case where white pixels are provided in addition to the pixels of the three primary colors (red, green and blue).
  • the specific color is white and the non-specific colors are red, green and blue.
  • the pixel array direction may have an inclination of 45 degrees, for example.
  • the pixel array of the solid-state imaging device 20 can calculate the polarization characteristic of a specific color, and the non-specific color has a configuration having polarized pixels and non-polarized pixels which are types having a smaller polarization direction than the specific color. Just do it.
  • the polarization characteristic is detected using the gain K that absorbs the sensitivity difference. Therefore, in another configuration of the solid-state imaging device, a case where an image signal in which a sensitivity difference does not occur between a non-polarization pixel and a polarization pixel is generated will be described.
  • FIG. 25 illustrates the pixel configuration of the solid-state imaging device.
  • the image sensor 21 includes a pixel array unit 211 in which a plurality of pixels are arranged in an array, for example, a two-dimensional matrix, and a vertical scanning circuit 212 and a horizontal scanning circuit 213 that perform drive control of the pixel array unit 211. There is.
  • the pixel array unit 211 only a part of pixels in the row direction and the column direction are illustrated in order to simplify the description.
  • the pixels of the pixel array unit 211 indicate the pixel color and the polarization direction.
  • the pixels of the pixel array unit 211 have photodiodes and charge transfer and reset transistors. Each pixel is connected to the vertical scanning circuit 212 via a reset line and a selection line, and is connected to the horizontal scanning circuit 213 via a signal line.
  • the vertical scanning circuit 212 outputs a reset signal to the reset pixel transistor of the polarization pixel via the reset line of the polarization pixel to discharge the accumulated charge. Also, the vertical scanning circuit 212 outputs a reset signal to the reset transistor of the non-polarization pixel via the reset line of the non-polarization pixel to discharge the accumulated charge. Thereafter, the vertical scanning circuit 212 outputs the readout signal to the charge transfer transistor of the polarization pixel and the non-polarization pixel through the selection line, and an exposure period from the output of the reset signal to the output of the readout signal. The charge accumulated therein is output as a signal current to the signal line.
  • the horizontal scanning circuit 213 performs processing of converting the signal current read from each pixel into a digital pixel signal, noise removal and the like, and outputs the processed pixel signal to the image processing apparatus 30 in the order of pixels in the horizontal direction. Further, the vertical scanning circuit 212 and the horizontal scanning circuit 213 perform the above-described processing for each line. Furthermore, the vertical scanning circuit 212 controls the exposure period of the polarization pixel or the non-polarization pixel to match the sensitivity of the polarization pixel and the non-polarization pixel. For example, the vertical scanning circuit 212 controls the timing of the reset signal of the non-polarization pixel to adjust the exposure period of the non-polarization pixel so that the non-polarization pixel becomes equal to the sensitivity of the polarization pixel. Match the pixel sensitivity.
  • FIG. 26 is a diagram for explaining the operation of the image sensor in the solid-state imaging device.
  • the solid-state imaging device 20 independently controls the outputs of the polarization pixel reset signal and the non-polarization pixel reset signal, so that, for example, when imaging a non-polarization object, the signal levels of the polarization pixel and the non-polarization pixel become equal. Adjust the exposure period of the non-polarization pixel.
  • FIG. 26 shows a readout signal SCa for reading out a signal corresponding to the accumulated charge from the polarization pixel and the non-polarization pixel.
  • (B) of FIG. 26 shows the reset signal SCbp for the polarization pixel, and
  • (c) of FIG. 26 shows the accumulated charge amount in the polarization pixel.
  • charge accumulation is performed in response to incident light in the polarization pixel from time t1 when the reset signal SCbp ends.
  • FIG. 26 shows a reset signal SCbn for a non-polarization pixel
  • (e) of FIG. 26 shows the accumulated charge amount in the non-polarization pixel.
  • the non-polarization pixel after accumulated charges are discharged by the reset signal SCbn, accumulation of charges is performed according to incident light in the non-polarization pixel from time t2 when the reset signal SCbn ends.
  • the polarization pixel and the non-polarization pixel read out the charge accumulated at time t3 by the readout signal SCa. That is, in the polarization pixel, the period from time t1 to t3 is the exposure time Tep, and in the non-polarization pixel, the period from time t2 to t3 is the exposure time Ten. Therefore, by shortening the exposure time Ten with respect to the exposure time Tep according to the sensitivity of the non-polarization pixel to the polarization pixel, even if the polarization pixel and the non-polarization pixel have a sensitivity difference, the signal level SLp of the polarization pixel And a non-polarization pixel signal level SLn can generate an image signal equal.
  • gain adjustment of the pixel signal may be performed by the horizontal scanning circuit 213 so that the sensitivity of the polarization pixel and that of the non-polarization pixel coincide.
  • an image signal of a captured image in which a sensitivity difference does not occur between a polarization pixel and a non-polarization pixel can be generated. Therefore, the image processing device has a sensitivity difference between the polarization pixel and the non-polarization pixel. It is not necessary to take account of the above, and detection of polarization characteristics can be easily performed.
  • the solid-state imaging device has a configuration using a CMOS image sensor, it is possible to add and read out pixel signals.
  • FIG. 27 shows pixels of the solid-state imaging device and readout signal lines.
  • A) of FIG. 27 illustrates a part of the pixel array in the solid-state imaging device 20.
  • (b) of FIG. 27 shows a read signal line.
  • the pixel of the solid-state imaging device 20 has a photodiode and a transistor for charge transfer and reset. Each pixel drives a reset transistor by a reset signal to discharge stored charge. Thereafter, each pixel drives a transistor for charge transfer by the read signal, and outputs the charge accumulated during the exposure period from the end of reset to the start of read as a pixel signal to the read signal line.
  • the read signal line is connected to an analog / digital conversion (A / D) unit, and converts an analog pixel signal read from each pixel into a digital pixel signal and performs noise removal and the like on the image processing apparatus 30.
  • a / D analog / digital conversion
  • FIG. 28 is a diagram for explaining the readout operation of the pixel signal.
  • the solid-state imaging device 20 sequentially adds and reads pixel signals from pixels of the same color and polarization direction in pixels of the same column connected to the same readout signal line.
  • (A) of FIG. 28 illustrates a part of the pixel array in the solid-state imaging device 20.
  • the solid-state imaging device 20 adds pixel signals from two red pixels which are non-polarized pixels via the read signal line VSL0 and reads them out to be non-polarized pixels. Generate a digital pixel signal of one red pixel.
  • the solid-state imaging device 20 adds and reads pixel signals from two red pixels which are non-polarization pixels via the read signal line VSL1, and generates digital pixel signals of one red pixel which is a non-polarization pixel. Do. Also, the solid-state imaging device 20 adds and reads pixel signals from two green pixels that are non-polarization pixels via the read signal line VSL2, and generates a digital pixel signal of one green pixel that is a non-polarization pixel. Do. Furthermore, the solid-state imaging device 20 adds and reads pixel signals from two green pixels that are non-polarization pixels via the read signal line VSL3 and generates a digital pixel signal of one green pixel that is a non-polarization pixel. Do. In this way, pixel signals for one line are generated.
  • the solid-state imaging device 20 generates a pixel signal of the next line. For example, as shown in (c) of FIG. 28, the solid-state imaging device 20 adds and reads pixel signals from two red pixels that are non-polarization pixels via the read signal line VSL0, and Generate a digital pixel signal of one red pixel. In addition, the solid-state imaging device 20 adds and reads pixel signals from two red pixels that are polarization pixels (the angle of the transmission axis: 90 degrees) via the readout signal line VSL1, and reads the polarization pixels (the angle of the transmission axis Digital pixel signal of one red pixel which is 90 degrees).
  • the solid-state imaging device 20 adds and reads pixel signals from two red pixels that are polarization pixels (the angle of the transmission axis: 90 degrees) via the readout signal line VSL1, and reads the polarization pixels (the angle of the transmission axis Digital pixel signal of one red pixel which is 90 degrees).
  • the solid-state imaging device 20 adds and reads pixel signals from two green pixels that are polarization pixels (the angle of the transmission axis: 90 degrees) via the readout signal line VSL2, and reads the polarization pixels (the angle of the transmission axis 90) to generate a digital pixel signal of one green pixel. Furthermore, the solid-state imaging device 20 adds and reads pixel signals from two green pixels which are non-polarization pixels via the read signal line VSL3 and generates a digital pixel signal of one green pixel which is a polarization pixel. . In this way, pixel signals for one line are generated.
  • the solid-state imaging device 20 includes two pixels, which are non-polarization pixels, via read signal lines VSL0 to VSL3.
  • the signals are added and read out to generate a digital pixel signal of one pixel which is a non-polarization pixel, and the pixel signals are added and read out from two pixels which are polarization pixels of the same polarization direction, which is a polarization pixel Generate a digital pixel signal of one pixel.
  • the solid-state imaging device 20 sequentially moves the pixels to be subjected to the signal addition in the vertical direction, and generates a pixel signal for each line.
  • the resolution of the image signal output from the solid-state imaging device 20 in the vertical direction is halved as shown in (f) of FIG.
  • the resolution in the horizontal direction may be set to 1/2 so that the aspect ratio of the output image does not change.
  • the image signal of the polarized RAW image having a high frame rate and a good S / N ratio from the solid-state imaging device 20 to the image processing device 30-1 (30-2 to 30-5 Can be output to
  • the technology according to the present disclosure can be applied to various products.
  • the technology according to the present disclosure is realized as a device mounted on any type of mobile object such as a car, an electric car, a hybrid electric car, a motorcycle, a bicycle, personal mobility, an airplane, a drone, a ship, a robot May be
  • FIG. 29 is a block diagram showing a schematic configuration example of a vehicle control system that is an example of a mobile control system to which the technology according to the present disclosure can be applied.
  • Vehicle control system 12000 includes a plurality of electronic control units connected via communication network 12001.
  • the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an external information detection unit 12030, an in-vehicle information detection unit 12040, and an integrated control unit 12050.
  • a microcomputer 12051, an audio image output unit 12052, and an in-vehicle network I / F (Interface) 12053 are illustrated as a functional configuration of the integrated control unit 12050.
  • the driveline control unit 12010 controls the operation of devices related to the driveline of the vehicle according to various programs.
  • the drive system control unit 12010 includes a drive force generation device for generating a drive force of a vehicle such as an internal combustion engine or a drive motor, a drive force transmission mechanism for transmitting the drive force to the wheels, and a steering angle of the vehicle. It functions as a control mechanism such as a steering mechanism that adjusts and a braking device that generates a braking force of the vehicle.
  • Body system control unit 12020 controls the operation of various devices equipped on the vehicle body according to various programs.
  • the body system control unit 12020 functions as a keyless entry system, a smart key system, a power window device, or a control device of various lamps such as a headlamp, a back lamp, a brake lamp, a blinker or a fog lamp.
  • the body system control unit 12020 may receive radio waves or signals of various switches transmitted from a portable device substituting a key.
  • Body system control unit 12020 receives the input of these radio waves or signals, and controls a door lock device, a power window device, a lamp and the like of the vehicle.
  • Outside vehicle information detection unit 12030 detects information outside the vehicle equipped with vehicle control system 12000.
  • an imaging unit 12031 is connected to the external information detection unit 12030.
  • the out-of-vehicle information detection unit 12030 causes the imaging unit 12031 to capture an image outside the vehicle, and receives the captured image.
  • the external information detection unit 12030 may perform object detection processing or distance detection processing of a person, a vehicle, an obstacle, a sign, characters on a road surface, or the like based on the received image.
  • the imaging unit 12031 is an optical sensor that receives light and outputs an electrical signal according to the amount of light received.
  • the imaging unit 12031 can output an electric signal as an image or can output it as distance measurement information.
  • the light received by the imaging unit 12031 may be visible light or non-visible light such as infrared light.
  • In-vehicle information detection unit 12040 detects in-vehicle information.
  • a driver state detection unit 12041 that detects a state of a driver is connected to the in-vehicle information detection unit 12040.
  • the driver state detection unit 12041 includes, for example, a camera for imaging the driver, and the in-vehicle information detection unit 12040 determines the degree of fatigue or concentration of the driver based on the detection information input from the driver state detection unit 12041. It may be calculated or it may be determined whether the driver does not go to sleep.
  • the microcomputer 12051 calculates a control target value of the driving force generation device, the steering mechanism or the braking device based on the information inside and outside the vehicle acquired by the outside information detecting unit 12030 or the in-vehicle information detecting unit 12040, and a drive system control unit A control command can be output to 12010.
  • the microcomputer 12051 controls the driving force generating device, the steering mechanism, the braking device, and the like based on the information around the vehicle acquired by the outside information detecting unit 12030 or the in-vehicle information detecting unit 12040 so that the driver can Coordinated control can be performed for the purpose of automatic driving that travels autonomously without depending on the operation.
  • the microcomputer 12051 can output a control command to the body system control unit 12020 based on the information outside the vehicle acquired by the external information detection unit 12030.
  • the microcomputer 12051 controls the headlamp according to the position of the preceding vehicle or oncoming vehicle detected by the external information detection unit 12030, and performs cooperative control for the purpose of antiglare such as switching the high beam to the low beam. It can be carried out.
  • the audio image output unit 12052 transmits an output signal of at least one of audio and image to an output device capable of visually or aurally notifying information to a passenger or the outside of a vehicle.
  • an audio speaker 12061, a display unit 12062, and an instrument panel 12063 are illustrated as output devices.
  • the display unit 12062 may include, for example, at least one of an on-board display and a head-up display.
  • FIG. 30 is a diagram illustrating an example of the installation position of the imaging unit 12031.
  • imaging units 12101, 12102, 12103, 12104, and 12105 are included.
  • the imaging units 12101, 12102, 12103, 12104, and 12105 are provided, for example, on the front nose of the vehicle 12100, a side mirror, a rear bumper, a back door, an upper portion of a windshield of a vehicle interior, and the like.
  • the imaging unit 12101 provided in the front nose and the imaging unit 12105 provided in the upper part of the windshield in the vehicle cabin mainly acquire an image in front of the vehicle 12100.
  • the imaging units 12102 and 12103 included in the side mirror mainly acquire an image of the side of the vehicle 12100.
  • the imaging unit 12104 provided in the rear bumper or the back door mainly acquires an image of the rear of the vehicle 12100.
  • the imaging unit 12105 provided on the top of the windshield in the passenger compartment is mainly used to detect a leading vehicle or a pedestrian, an obstacle, a traffic light, a traffic sign, a lane, or the like.
  • FIG. 30 shows an example of the imaging range of the imaging units 12101 to 12104.
  • the imaging range 12111 indicates the imaging range of the imaging unit 12101 provided on the front nose
  • the imaging ranges 12112 and 12113 indicate the imaging ranges of the imaging units 12102 and 12103 provided on the side mirrors
  • the imaging range 12114 indicates The imaging range of the imaging part 12104 provided in the rear bumper or the back door is shown. For example, by overlaying the image data captured by the imaging units 12101 to 12104, a bird's eye view of the vehicle 12100 viewed from above can be obtained.
  • At least one of the imaging units 12101 to 12104 may have a function of acquiring distance information.
  • at least one of the imaging units 12101 to 12104 may be a stereo camera including a plurality of imaging devices, or an imaging device having pixels for phase difference detection.
  • the microcomputer 12051 measures the distance to each three-dimensional object in the imaging ranges 12111 to 12114, and the temporal change of this distance (relative velocity with respect to the vehicle 12100). In particular, it is possible to extract a three-dimensional object traveling at a predetermined speed (for example, 0 km / h or more) in substantially the same direction as the vehicle 12100 as a leading vehicle, in particular by finding the it can. Further, the microcomputer 12051 can set an inter-vehicle distance to be secured in advance before the preceding vehicle, and can perform automatic brake control (including follow-up stop control), automatic acceleration control (including follow-up start control), and the like. As described above, it is possible to perform coordinated control for the purpose of automatic driving or the like that travels autonomously without depending on the driver's operation.
  • automatic brake control including follow-up stop control
  • automatic acceleration control including follow-up start control
  • the microcomputer 12051 converts three-dimensional object data relating to three-dimensional objects into two-dimensional vehicles such as two-wheeled vehicles, ordinary vehicles, large vehicles, pedestrians, telephone poles, and other three-dimensional objects. It can be classified, extracted and used for automatic avoidance of obstacles. For example, the microcomputer 12051 identifies obstacles around the vehicle 12100 into obstacles visible to the driver of the vehicle 12100 and obstacles difficult to see.
  • the microcomputer 12051 determines the collision risk indicating the degree of risk of collision with each obstacle, and when the collision risk is a setting value or more and there is a possibility of a collision, through the audio speaker 12061 or the display unit 12062 By outputting a warning to the driver or performing forcible deceleration or avoidance steering via the drive system control unit 12010, driving support for collision avoidance can be performed.
  • At least one of the imaging units 12101 to 12104 may be an infrared camera that detects infrared light.
  • the microcomputer 12051 can recognize a pedestrian by determining whether a pedestrian is present in the images captured by the imaging units 12101 to 12104.
  • pedestrian recognition is, for example, a procedure for extracting feature points in images captured by the imaging units 12101 to 12104 as an infrared camera, and pattern matching processing on a series of feature points indicating the outline of an object to determine whether it is a pedestrian or not
  • the procedure is to determine
  • the audio image output unit 12052 generates a square outline for highlighting the recognized pedestrian.
  • the display unit 12062 is controlled so as to display a superimposed image. Further, the audio image output unit 12052 may control the display unit 12062 to display an icon or the like indicating a pedestrian at a desired position.
  • the example of the vehicle control system to which the technology according to the present disclosure can be applied has been described above.
  • the solid-state imaging device 20 according to the technology of the present disclosure may be applied to the imaging unit 12031 and the like among the configurations described above.
  • the image processing apparatus 30 of the technique which concerns on this indication may be applied to the vehicle exterior information detection unit 12030 among the structures demonstrated above.
  • polarization characteristics can be accurately obtained for each color. Therefore, driver fatigue can be reduced or the driver's fatigue can be reduced by performing reflection removal or recognizing a three-dimensional shape of an object. It becomes possible to obtain information required for automatic driving with high accuracy.
  • the series of processes described in the specification can be performed by hardware, software, or a combination of both.
  • a program recording the processing sequence is installed and executed in a memory in a computer incorporated in dedicated hardware.
  • the program can be installed and executed on a general-purpose computer that can execute various processes.
  • the program can be recorded in advance on a hard disk or a solid state drive (SSD) as a recording medium, or a read only memory (ROM).
  • the program may be a flexible disk, a compact disc read only memory (CD-ROM), a magneto optical (MO) disc, a digital versatile disc (DVD), a BD (Blu-Ray Disc (registered trademark)), a magnetic disc, a semiconductor memory card Etc.
  • CD-ROM compact disc read only memory
  • MO magneto optical
  • DVD digital versatile disc
  • BD Blu-Ray Disc
  • magnetic disc a semiconductor memory card Etc.
  • Such removable recording media can be provided as so-called package software.
  • the program may be installed from the removable recording medium to the computer, or may be transferred from the download site to the computer wirelessly or by wire via a network such as a LAN (Local Area Network) or the Internet.
  • the computer can receive the program transferred in such a manner, and install the program on a recording medium such as a built-in hard disk.
  • the effect described in this specification is an illustration to the last, is not limited, and may have an additional effect which is not described.
  • the present technology should not be construed as being limited to the embodiments of the above-described technology.
  • the embodiments of this technology disclose the present technology in the form of exemplification, and it is obvious that those skilled in the art can modify or substitute the embodiments within the scope of the present technology. That is, in order to determine the gist of the present technology, the claims should be taken into consideration.
  • the image processing apparatus of the present technology can also have the following configuration.
  • An image processing apparatus comprising: a non-specific color polarization characteristic detection unit configured to detect the non-specific color polarization characteristic based on a signal.
  • the non-specific color polarization characteristic detection unit detects polarization characteristics of the non-specific color assuming that the azimuth of the specific color and the non-specific color are equal.
  • the non-specific color polarization characteristic detection unit includes the polarization characteristic of the specific color, the pixel signal of the non-polarization pixel of the non-specific color, and the non-specific color having a polarization direction smaller than that of the specific color.
  • a specific color polarization characteristic detection unit for detecting the polarization characteristic of the specific color using pixel signals of polarization pixels of three or more polarization directions of the specific color or non-polarization pixels and polarization pixels of two polarization directions An image processing apparatus according to any one of 1) to 3).
  • a non-polarization component image generation unit that generates a polarization component image signal;
  • a polarization component image for generating a polarization component image signal of a non-specific color using a polarization component image signal of the specific color using a pixel signal of the polarization pixel of the specific color and a pixel signal of a polarization pixel of the non-specific color The image processing apparatus according to any one of (1) to (4), further including: a generation unit.
  • the non-polarization component image generation unit and the polarization component image generation unit perform filter processing using pixel signals of pixels having the same polarization direction as the color, and the non-polarization component image signal and the polarization component image signal
  • the image processing apparatus according to (5) which generates.
  • the non-polarization component image generation unit and the polarization component image generation unit calculate the color and polarization direction calculated by performing filter processing using pixels having the same polarization direction as the color from the pixel of the target pixel position and peripheral pixels
  • the target pixel position calculated using the low frequency component of each target pixel position and the low frequency component of the target pixel position whose color and polarization direction are the same as the pixel signal of the target pixel position and the pixel of the target pixel position
  • the image processing apparatus according to (6), wherein a pixel signal for each color and polarization direction at the target pixel position is generated based on a high frequency component of (8)
  • the image processing apparatus according to any one of (5) to (7), in which pixel signals of the polarization pixel and the non-polarization pixel of the specific color and the non-specific color are signals after white balance adjustment.
  • the image processing apparatus according to any one of (5) to (7), further including: a color balance coefficient calculation unit that calculates a color balance coefficient based on the polarization characteristic of the specific color and the polarization characteristic of the nonspecific color.
  • a color balance coefficient calculation unit that calculates a color balance coefficient based on the polarization characteristic of the specific color and the polarization characteristic of the nonspecific color.
  • the color balance coefficient calculation unit calculates the color balance coefficient with specular reflection as white.
  • the image processing apparatus according to 10).
  • a color balance that adjusts the color balance coefficient according to the color balance coefficient based on the polarization characteristic calculated by the color balance coefficient calculation unit and the color balance coefficient based on color information that reproduces a white object as white. It further comprises a coefficient adjustment unit, The image processing apparatus according to (11), wherein the color balance adjustment unit performs the white balance adjustment using the color balance coefficient adjusted by the color balance coefficient adjustment unit. (13) The color balance coefficient adjustment unit adjusts the color balance coefficient based on the difference amount between the color balance coefficient based on the polarization characteristic and the color balance coefficient based on the color information, or the reflection amount of specular reflection ( The image processing apparatus according to 12).
  • the polarization characteristics of a specific color, a pixel signal of a non-polarization pixel of a non-specific color different from the specific color The polarization characteristic of the non-specific color is detected based on the pixel signal of the polarization pixel of a type having a smaller polarization direction than that of the other. Therefore, polarization characteristics can be detected with high accuracy for each color. Therefore, it is suitable for the apparatus which can perform control etc. using polarization information, for example, a vehicle control system etc.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Physics & Mathematics (AREA)
  • Spectroscopy & Molecular Physics (AREA)
  • Power Engineering (AREA)
  • Condensed Matter Physics & Semiconductors (AREA)
  • General Physics & Mathematics (AREA)
  • Electromagnetism (AREA)
  • Computer Hardware Design (AREA)
  • Microelectronics & Electronic Packaging (AREA)
  • Color Television Image Signal Generators (AREA)
  • Polarising Elements (AREA)

Abstract

Selon la présente invention, une unité de génération d'image à composante non polarisée 32-1 génère des signaux d'image à composante non polarisée de couleur spécifique et de couleur non spécifique. Une unité de génération d'image à composante polarisée 33-1 génère des signaux d'image à composante polarisée de couleur spécifique et de couleur non spécifique. Une unité de détection de caractéristique de polarisation de couleur spécifique 34-1 détecte une caractéristique de polarisation de couleur spécifique à l'aide de signaux de pixel de pixels de polarisation de couleur spécifique ayant trois directions de polarisation ou plus ou des pixels de non polarisation et des pixels de polarisation ayant deux directions de polarisation. Une unité de détection de caractéristique de polarisation de couleur non spécifique 35 détecte une caractéristique de polarisation de couleur non spécifique sur la base de la caractéristique de polarisation de couleur spécifique détectée par l'unité de détection de caractéristique de polarisation de couleur spécifique 34-1, un signal de pixel d'un pixel de non-polarisation de couleur non spécifique, et des signaux de pixel de pixels de polarisation de couleur non spécifique dont le nombre de directions de polarisation est inférieur au nombre de pixels de polarisation de couleur spécifique. La caractéristique de polarisation peut ainsi être détectée couleur par couleur sans utiliser l'hypothèse selon laquelle des degrés de polarisation couleur par couleur sont égaux.
PCT/JP2018/034389 2017-11-21 2018-09-18 Dispositif de traitement d'image, procédé de traitement d'image, programme, et dispositif d'imagerie à semi-conducteur WO2019102698A1 (fr)

Priority Applications (3)

Application Number Priority Date Filing Date Title
US16/763,543 US11605661B2 (en) 2017-11-21 2018-09-18 Image processing apparatus, image processing method, and solid-state imaging apparatus to detect polarization properties of each color
JP2019556113A JP7230824B2 (ja) 2017-11-21 2018-09-18 画像処理装置、画像処理方法およびプログラム
CN201880073836.0A CN111357282B (zh) 2017-11-21 2018-09-18 图像处理装置、图像处理方法、程序和固态成像装置

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2017-223423 2017-11-21
JP2017223423 2017-11-21

Publications (1)

Publication Number Publication Date
WO2019102698A1 true WO2019102698A1 (fr) 2019-05-31

Family

ID=66631530

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2018/034389 WO2019102698A1 (fr) 2017-11-21 2018-09-18 Dispositif de traitement d'image, procédé de traitement d'image, programme, et dispositif d'imagerie à semi-conducteur

Country Status (4)

Country Link
US (1) US11605661B2 (fr)
JP (1) JP7230824B2 (fr)
CN (1) CN111357282B (fr)
WO (1) WO2019102698A1 (fr)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2022019025A1 (fr) * 2020-07-20 2022-01-27 ソニーグループ株式会社 Dispositif, système, procédé et programme de traitement d'informations
CN114930800A (zh) * 2020-01-09 2022-08-19 索尼集团公司 图像处理装置、图像处理方法和成像装置
WO2022196038A1 (fr) * 2021-03-19 2022-09-22 ソニーグループ株式会社 Dispositif de traitement d'informations, procédé de traitement d'informations et programme
WO2023105961A1 (fr) * 2021-12-08 2023-06-15 ソニーグループ株式会社 Dispositif de traitement d'image, procédé de traitement d'image et programme

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11405601B2 (en) * 2019-09-17 2022-08-02 Meta Platforms Technologies, Llc Polarization capture device, system, and method
DE102022201523A1 (de) * 2021-09-10 2023-03-16 Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung eingetragener Verein Ein Bildsensor, ein Verfahren zum Betreiben eines Bildsensors, ein Verfahren zum Herstellen eines Bildsensors und ein stationäres Gerät oder ein Fahrzeug oder eine Drohne mit einem Bildsensor

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2010263158A (ja) * 2009-05-11 2010-11-18 Sony Corp ２次元固体撮像装置、及び、２次元固体撮像装置における偏光光データ処理方法
JP2013066166A (ja) * 2011-08-30 2013-04-11 Ricoh Co Ltd 撮像装置、並びに、これを用いる画像解析装置及び移動装置
WO2016136085A1 (fr) * 2015-02-27 2016-09-01 ソニー株式会社 Dispositif de traitement d'image, procédé de traitement d'image et élément de capture d'image
WO2017085993A1 (fr) * 2015-11-19 2017-05-26 ソニー株式会社 Appareil de traitement d'images, et procédé de traitement d'images

Family Cites Families (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4833525A (en) 1985-03-29 1989-05-23 Canon Kabushiki Kaisha High speed color balance adjusting device capable of compensating for a flickering light source
US5090807A (en) * 1990-01-10 1992-02-25 Environmental Research Institute Of Michigan Real time optical pre-detection processing of multispectral image data
US8054371B2 (en) 2007-02-19 2011-11-08 Taiwan Semiconductor Manufacturing Company, Ltd. Color filter for image sensor
US8564700B2 (en) 2007-12-07 2013-10-22 Panasonic Corporation Image processing device, image processing method, and imaging device using a patterned polarizer
JP2013031054A (ja) * 2011-07-29 2013-02-07 Ricoh Co Ltd 撮像装置及びこれを備えた物体検出装置、並びに、光学フィルタ及びその製造方法
JP5783929B2 (ja) * 2012-02-20 2015-09-24 キヤノン株式会社 撮像装置
EP2957215B1 (fr) * 2013-02-15 2020-05-06 Panasonic Intellectual Property Management Co., Ltd. Dispositif de traitement d'image et endoscope
JP6192428B2 (ja) * 2013-08-20 2017-09-06 オリンパス株式会社 撮像装置
US10460422B2 (en) 2015-11-10 2019-10-29 Sony Corporation Image processing device and image processing method
JP6486288B2 (ja) * 2016-02-09 2019-03-20 キヤノン株式会社 撮像装置、制御方法、プログラム、記憶媒体

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2010263158A (ja) * 2009-05-11 2010-11-18 Sony Corp ２次元固体撮像装置、及び、２次元固体撮像装置における偏光光データ処理方法
JP2013066166A (ja) * 2011-08-30 2013-04-11 Ricoh Co Ltd 撮像装置、並びに、これを用いる画像解析装置及び移動装置
WO2016136085A1 (fr) * 2015-02-27 2016-09-01 ソニー株式会社 Dispositif de traitement d'image, procédé de traitement d'image et élément de capture d'image
WO2017085993A1 (fr) * 2015-11-19 2017-05-26 ソニー株式会社 Appareil de traitement d'images, et procédé de traitement d'images

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114930800A (zh) * 2020-01-09 2022-08-19 索尼集团公司 图像处理装置、图像处理方法和成像装置
CN114930800B (zh) * 2020-01-09 2024-05-28 索尼集团公司 图像处理装置、图像处理方法和成像装置
WO2022019025A1 (fr) * 2020-07-20 2022-01-27 ソニーグループ株式会社 Dispositif, système, procédé et programme de traitement d'informations
WO2022196038A1 (fr) * 2021-03-19 2022-09-22 ソニーグループ株式会社 Dispositif de traitement d'informations, procédé de traitement d'informations et programme
WO2023105961A1 (fr) * 2021-12-08 2023-06-15 ソニーグループ株式会社 Dispositif de traitement d'image, procédé de traitement d'image et programme

Also Published As

Publication number Publication date
US11605661B2 (en) 2023-03-14
US20200350353A1 (en) 2020-11-05
CN111357282A (zh) 2020-06-30
JP7230824B2 (ja) 2023-03-01
JPWO2019102698A1 (ja) 2020-12-03
CN111357282B (zh) 2023-04-18

Similar Documents

Publication Publication Date Title
WO2019102698A1 (fr) Dispositif de traitement d'image, procédé de traitement d'image, programme, et dispositif d'imagerie à semi-conducteur
CN109804621B (zh) 图像处理装置、图像处理方法和图像拾取装置
CN108353153B (zh) 图像处理装置和图像处理方法
JP7040447B2 (ja) 画像処理装置および情報生成装置と情報生成方法
JP6693530B2 (ja) 画像処理装置と画像処理方法
US20180013988A1 (en) Image processing apparatus, image processing method, and image pickup element
US11394913B2 (en) Solid-state imaging element, electronic device, and method for controlling correction of luminance in the solid-state imaging element
WO2020105314A1 (fr) Élément d'imagerie à semi-conducteurs et dispositif d'imagerie
JP6981410B2 (ja) 固体撮像装置、電子機器、レンズ制御方法および車両
WO2017175492A1 (fr) Dispositif de traitement d'image, procédé de traitement d'image, programme informatique et dispositif électronique
TWI842952B (zh) 攝像裝置
JP7500798B2 (ja) 固体撮像装置、補正方法、および電子装置
JP7147784B2 (ja) 画像処理装置と画像処理方法およびプログラム
JP7144926B2 (ja) 撮像制御装置、撮像装置、および、撮像制御装置の制御方法
US20200402206A1 (en) Image processing device, image processing method, and program
JP6981416B2 (ja) 画像処理装置と画像処理方法
JP2020107932A (ja) 固体撮像素子および撮像装置
WO2017122395A1 (fr) Circuit de traitement de données d'image et circuit d'imagerie
WO2020246250A1 (fr) Élément d'imagerie, dispositif de traitement de signal, procédé de traitement de signal, programme et dispositif d'imagerie
WO2020166284A1 (fr) Dispositif de capture d'image
WO2017169079A1 (fr) Dispositif de traitement d'image, procédé de traitement d'image et élément de capture d'image
US20200014899A1 (en) Imaging device, imaging system, and method of controlling imaging device
WO2023286297A1 (fr) Élément d'imagerie à semi-conducteurs, dispositif d'imagerie, et procédé de contrôle d'élément d'imagerie à semi-conducteurs
US20230394847A1 (en) Image processing system, image processing method, and storage medium
WO2020100399A1 (fr) Élément d'imagerie à semi-conducteurs, dispositif d'imagerie, et procédé de contrôle d'élément d'imagerie à semi-conducteurs

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18881337

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2019556113

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 18881337

Country of ref document: EP

Kind code of ref document: A1