EP4254935A1 - Bildverarbeitungsvorrichtung, bildverarbeitungsverfahren und programm - Google Patents

Bildverarbeitungsvorrichtung, bildverarbeitungsverfahren und programm Download PDF

Info

Publication number
EP4254935A1
EP4254935A1 EP21897546.4A EP21897546A EP4254935A1 EP 4254935 A1 EP4254935 A1 EP 4254935A1 EP 21897546 A EP21897546 A EP 21897546A EP 4254935 A1 EP4254935 A1 EP 4254935A1
Authority
EP
European Patent Office
Prior art keywords
image
reflection
black
color
characteristic information
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
EP21897546.4A
Other languages
English (en)
French (fr)
Other versions
EP4254935A4 (de
Inventor
Taishi ONO
Yuhi Kondo
Legong SUN
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sony Group Corp
Original Assignee
Sony Group Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sony Group Corp filed Critical Sony Group Corp
Publication of EP4254935A1 publication Critical patent/EP4254935A1/de
Publication of EP4254935A4 publication Critical patent/EP4254935A4/de
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/60—Extraction of image or video features relating to illumination properties, e.g. using a reflectance or lighting model
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/80—Camera processing pipelines; Components thereof
    • H04N23/81—Camera processing pipelines; Components thereof for suppressing or minimising disturbance in the image signal generation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/77—Retouching; Inpainting; Scratch removal
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/60—Analysis of geometric attributes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/56—Extraction of image or video features relating to colour
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/10—Circuitry of solid-state image sensors [SSIS]; Control thereof for transforming different wavelengths into image signals
    • H04N25/11—Arrangement of colour filter arrays [CFA]; Filter mosaics
    • H04N25/13—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements
    • H04N25/133—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements including elements passing panchromatic light, e.g. filters passing white light
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/10—Circuitry of solid-state image sensors [SSIS]; Control thereof for transforming different wavelengths into image signals
    • H04N25/11—Arrangement of colour filter arrays [CFA]; Filter mosaics
    • H04N25/13—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements
    • H04N25/135—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements based on four or more different wavelength filter elements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image

Definitions

  • the present technology relates to an image processing device, an image processing method, and a program, and enables a reflection-removed color image with high sensitivity to be obtained.
  • NPL 1 discloses a method of removing reflection reflected on a glass surface or water surface using a polarized image.
  • PTL 1 discloses removing a reflection component using an angle of a reflecting surface such as glass or a water surface for an imaging device that acquires a polarized image.
  • an imaging device that acquires a polarized image
  • subject light is incident on an imaging element via a polarizing element such as a polarizing plate.
  • the imaging device causes subject light to enter the imaging element via the polarizing element and a color filter. For this reason, not only the light attenuation by the polarizer but also the light attenuation by the color filter occurs at the same time. Therefore, when a reflection-removed color image from which specular reflection is removed is generated based on a color polarized image, it is difficult to obtain a high-sensitivity reflection-removed color image.
  • an object of the present technology is to provide an image processing device, an image processing method, and a program capable of obtaining a high-sensitivity reflection-removed color image.
  • a first aspect of the present technology provides an image processing device including a reflection-removed image generator that calculates characteristic information related to black-and-white reflection removal results generated based on a black-and-white polarized image, and generates a reflection-removed color image in which specular reflection is removed from a color image using the calculated characteristic information.
  • the reflection-removed image generator calculates characteristic information related to black-and-white reflection-removal results generated based on a black-and-white polarized image.
  • the reflection-removed image generator calculates information indicating a relationship between the black-and-white polarized image and an average luminance calculated from an image for each polarization direction in the black-and-white polarized image as the characteristic information.
  • the reflection-removed image generator When calculating such characteristic information, for example, the reflection-removed image generator generates the color polarized image by multiplying the color image by the characteristic information indicating a ratio of a luminance of the black-and-white polarized image to the average luminance to generate the reflection-removed color image in which the specular reflection calculated based on the generated color polarized image is removed from the color image.
  • the color image is an image in an RGB color space, or a color space in which luminance and color are separated.
  • the reflection-removed image generator calculates information used for obtaining the black-and-white reflection removal results from the black-and-white polarized image as the characteristic information, optimizes the characteristic information using the calculated characteristic information as an initial value, and generates the reflection-removed color image based on the optimized characteristic information and the color image.
  • the reflection-removed image generator calculates information related to specular reflection in the black-and-white polarized image as the characteristic information.
  • the reflection-removed image generator uses the Stokes parameters indicated by the characteristic information as initial values, and optimizes the characteristic information to minimize a geometric correlation or a correlation in a feature amount space between a luminance component and a specular reflection component of the color image.
  • the reflection-removed image generator generates the reflection-removed color image using the color image and the optimized characteristic information.
  • the image processing device further includes a reflecting surface information acquirer that acquires reflecting surface information indicating an angle of a reflecting surface of a subject that causes the specular reflection, and the reflection-removed image generator calculates information indicating polarization characteristics of the black-and-white polarized image using together with the reflecting surface information acquired by the reflecting surface information acquirer as the characteristic information. For example, when a black-and-white reflection image and a black-and-white transmission image are calculated as the characteristic information, the reflection-removed image generator uses the black-and-white reflection image and the black-and-white transmission image as initial values to optimize the characteristic information to minimize a geometric correlation or a correlation in a feature amount space between a color reflection image and a color transmission image based on the color image. The reflection-removed image generator generates the reflection-removed color image using the color image, the optimized characteristic information, and the reflecting surface information acquired by the reflecting surface information acquirer.
  • An output unit that outputs the reflection-removed color image generated by the reflection-removed image generator outputs an evaluation index used in optimization of the characteristic information together with the reflection-removed color image. For example, the output unit superimposes a display based on the evaluation index on a corresponding position of the reflection-removed color image.
  • a second aspect of the present technology provides an image processing method including allowing a reflection-removed image generator to calculate characteristic information related to black-and-white reflection removal results generated based on a black-and-white polarized image and generate a reflection-removed color image in which specular reflection is removed from a color image using the calculated characteristic information.
  • a third aspect of the present technology provides a program for causing a computer to execute image processing, the computer executing: calculating characteristic information related to black-and-white reflection removal results generated based on a black-and-white polarized image; and generating a reflection-removed color image in which specular reflection is removed from a color image using the calculated characteristic information.
  • the program of the present technology is a program that can be provided in a general-purpose computer capable of executing various program codes by a storage medium provided in a computer-readable format or a communication medium, for example, a storage medium such as an optical disc, a magnetic disk or a semiconductor memory, or a communication medium such as a network.
  • a storage medium such as an optical disc, a magnetic disk or a semiconductor memory, or a communication medium such as a network.
  • the image processing device of the present technology images a subject that causes specular reflection to obtain a black-and-white polarized image and a color image, obtains characteristic information related to black-and-white reflection removal results generated based on the black-and-white polarized image, and generates a reflection-removed color image in which specular reflection is removed from the color image using the characteristic information.
  • the image processing device uses information indicating a relation between a luminance component of a black-and-white polarized image and a luminance component of black-and-white reflection removal results as characteristic information, generates a color polarized image based on the characteristic information and the color image, and generates a reflection-removed color image in which a specular reflection component calculated based on the generated color polarized image is removed from the color image.
  • FIG. 1 illustrates the configuration of the first embodiment.
  • An image processing system 10 has an image acquirer 20, a reflection-removed image generator 40 and an output unit 50.
  • the image acquirer 20 has a black-and-white polarized image acquirer 21 that acquires a black-and-white polarized image and a color image acquirer 22 that acquires a color image. Note that in the following description, a case will be described in which an image in the RGB color space indicating the luminance of each color component of the red, green, and blue components is acquired as a color image.
  • the black-and-white polarized image acquirer 21 acquires black-and-white polarized images in a plurality of polarization directions.
  • Fig. 2 illustrates the configuration of the black-and-white polarized image acquirer.
  • the black-and-white polarized image acquirer 21 acquires polarized images by arranging a polarizing filter 212 composed of a plurality of pixels with a plurality of polarization directions in an image sensor 211 such as a CMOS (Complementary Metal Oxide Semiconductor) or a CCD (Charge Coupled Device).
  • CMOS Complementary Metal Oxide Semiconductor
  • CCD Charge Coupled Device
  • the polarizing filter 212 can extract linearly polarized light from subject light, and uses a wire grid, photonic liquid crystal, or the like, for example.
  • the polarizing filter 212 is composed of pixels with three or more different polarization directions, for example, pixels with polarization directions of 0°, 45°, 90°, and 135°, so that the polarization characteristics can be calculated by cosine fitting or the like. Therefore, the black-and-white polarized image acquirer 21 can acquire images in four polarization directions in one imaging operation.
  • the arrows illustrated in Figs. 2(a), 2(b), and 2(c) and Fig. 3 indicate the polarization directions.
  • the black-and-white polarized image acquirer 21 may generate a plurality of polarized images with different polarization directions using a multi-lens array configuration, as illustrated in Fig. 2(b) .
  • a plurality of lenses 213 (four in the figure) are provided in front of the image sensor 211, and each lens 213 forms an optical image of a subject on the imaging surface of the image sensor 211.
  • a polarizing plate 214 is provided in front of each lens 213, and a plurality of polarized images having different polarization directions are generated by setting the polarization directions of the polarizing plate 214 to be different directions.
  • polarizing plates 216-1 to 216-4 having different polarization directions may be provided in front of the imaging units 215-1 to 215-4 to generate a plurality of polarized images with different polarization directions from a plurality of different viewpoints.
  • a polarizing plate 217 may be provided in front of the imaging unit 215.
  • the polarizing plate 217 is rotated to capture images in a plurality of different polarization directions, thereby obtaining a plurality of polarized images with different polarization directions.
  • the polarized images with different polarization directions are aligned according to the amount of parallax, and the luminance of the polarized images after alignment is averaged, whereby an image equivalent to the non-polarized normal luminance image can be obtained.
  • Fig. 2(d) by averaging the luminance of the polarized images with different polarization directions for each pixel, an image equivalent to the non-polarized normal luminance image can be obtained.
  • the color image acquirer 22 performs imaging by providing, for example, a three-primary-color mosaic filter on the imaging surface of the image sensor, and generates a color image.
  • the black-and-white polarized image acquirer 21 and the color image acquirer 22 may be integrated.
  • Fig. 3 illustrates a pixel configuration when the black-and-white polarized image acquirer and the color image acquirer are integrated.
  • a pixel block of 4 pixels ⁇ 4 pixels is composed of a red pixel block R, a green pixel block G and a blue pixel block B each of which is 2 ⁇ 2 pixels.
  • the remaining pixel block W of 2 ⁇ 2 pixels is not provided with a color filter, and is a polarized pixel block provided with a polarizing filter.
  • the polarized pixel block is composed of four polarized pixels with different polarization directions.
  • the pixel values of each color of the color image and the pixel values for each of the four polarization directions of the black-and-white polarized image can be obtained for each pixel block of 4 pixels ⁇ 4 pixels.
  • the reflection-removed image generator 40 calculates, in units of pixels or pixel blocks, information indicating a relationship between a luminance of the black-and-white polarized image and an average luminance calculated from images for each polarization direction of the black-and-white polarized image acquired by the black-and-white polarized image acquirer 21 as characteristic information related to the black-and-white reflection removal results generated based on the black-and-white polarized image, generates a color polarized image based on the calculated characteristic information abnormality detection signal the color image acquired by the color image acquirer 22, and generates a reflection-removed color image in which a specular reflection component calculated based on the generated color polarized image is removed from the color image.
  • the output unit 50 outputs the reflection-removed color image generated by the reflection-removed image generator 40 to an image display device or the like for display.
  • the output unit 50 may output the reflection-removed color image to a recording device to record it on a recording medium, or may transmit the reflection-removed color image to an external device or the like via a network or the like.
  • Fig. 4 is a flowchart illustrating the operation of the first embodiment.
  • the image processing system acquires a black-and-white polarized image and a color image.
  • the black-and-white polarized image acquirer 21 of the image processing system 10 acquires a black-and-white polarized image.
  • the color image acquirer 22 acquires a color image, and then, the process proceeds to step ST2.
  • step ST2 the image processing system calculates characteristic information.
  • the reflection-removed image generator 40 of the image processing system 10 generates a black-and-white image representing the average value of luminance changes due to the difference in the polarization direction from the black-and-white polarized image.
  • the reflection-removed image generator 40 calculates the ratio (I gray,pol /I gray,intensity ) of the pixel value I gray,pol of the black-and-white polarized image to the pixel value I gray,intensity of the generated black-and-white image as characteristic information for each pixel (or for each pixel block), and then, the process proceeds to step ST3.
  • step ST3 the image processing system generates a color polarized image.
  • the reflection-removed image generator 40 of the image processing system 10 considers that the relationship between the color image and the color polarized image is the same as the relationship between the black-and-white image and the black-and-white polarized image, and calculates the pixel values (I R,pol, I G,pol , I B,pol ) of the color polarized image from the characteristic information and the pixel values (I R,intensity , I G,intensity , I B,intensity ) of the color image based on Equations (1) to (3), and then, the process proceeds to step ST4. [Math.
  • step ST4 the image processing system generates a reflection-removed color image.
  • the reflection-removed image generator 40 of the image processing system 10 calculates a specular reflection component based on the polarization characteristics of the color polarized image, removes the specular reflection component from the color image to generate the reflection-removed color image, and then, the process proceeds to step ST5.
  • step ST5 the image processing system outputs a reflection-removed color image.
  • the image output unit of the image processing system 10 outputs the reflection-removed color image generated in step ST4 to an external device such as a display device, a recording device, or a control device that performs various types of control using the reflection-removed color image.
  • a color polarized image with high sensitivity can be generated without performing imaging via a polarizing element and a color filter. Therefore, a reflection-removed color image can be generated with higher sensitivity than when a reflection-removed color image is generated from a color polarized image acquired by performing imaging using a polarizing element and a color filter.
  • the color polarized image is calculated from the color image and the characteristic information indicating the relationship between the black-and-white image and the black-and-white polarized image by considering that the relationship between the color image and the color polarized image is the same as the relationship between the black-and-white image and the black-and-white polarized image.
  • the pixel values (I R,pol , I G,pol , I B,pol ) of the color image calculated based on Equations (1) to (3) are different from the true values and artifacts (color change regions) may occur in the reflection-removed color image.
  • the image processing device calculates information used for obtaining black-and-white reflection removal results from black-and-white polarized images as the characteristic information and generates a reflection-removed color image based on the characteristic information and the color image.
  • the image processing device optimizes the characteristic information and generates the reflection-removed color image based on the optimized characteristic information and the color image.
  • the image processing device calculates information related to specular reflection in the black-and-white polarized image as characteristic information, and optimizes the characteristic information to minimize the correlation between the luminance component and the specular reflection component of the color image.
  • the image processing device uses the color image and the optimized characteristic information to generate an artifact-free reflection-removed color image.
  • the image processing device uses the calculated characteristic information as an initial value in optimizing the characteristic information.
  • the second embodiment has the same configuration as the first embodiment, and differs from the first embodiment in the operation of the reflection-removed image generator 40.
  • Fig. 5 is a flowchart illustrating the operation of the second embodiment.
  • the image processing system acquires a black-and-white polarized image and a color image.
  • the black-and-white polarized image acquirer 21 of the image processing system 10 acquires a black-and-white polarized image.
  • the color image acquirer 22 acquires a color image, and then, the process proceeds to step ST12.
  • step ST12 the image processing system calculates characteristic information. It is known that the change in luminance observed when the polarization direction (polarization angle) is changed in the imaging unit that acquires the polarized image can be fitted to a cosine curve, and the varying portion of the cosine curve corresponds to the specular reflection component.
  • the polarization characteristics of the subject can be represented by a four-dimensional vector called the Stokes vector Sv illustrated in Equation (4).
  • the Stokes parameter S 0 indicates the non-polarized luminance or average luminance.
  • the Stokes parameter S 1 is the difference in intensity between the polarization directions of 0° and 90°
  • the Stokes parameter S 2 is the difference in intensity between the polarization directions of 45° and 135°
  • the Stokes parameter S 3 is the degree of polarization of circularly polarized light.
  • the Stokes parameter S gray,0 of the black-and-white polarized image can be calculated based on Equation (5).
  • the Stokes parameter S gray,1 of the black-and-white polarized image can be calculated based on Equation (6).
  • the Stokes parameter S gray,2 of the black-and-white polarized image can be calculated based on Equation (7).
  • S gray , 0 I gray , 0 + I gray , 45 + I gray , 90 + I gray , 135 4
  • S gray , 1 I gray , 0 ⁇ I gray , 90 / 2
  • S gray , 2 I gray , 45 ⁇ I gray , 135 / 2
  • the reflection-removed image generator 40 of the image processing system 10 calculates information used for obtaining the black-and-white reflection removal result from the black-and-white polarized image, for example, the Stokes parameter S gray,1 and the Stokes parameter S gray,2 as the characteristic information, and then, the process proceeds step ST13.
  • step ST13 the image processing system optimizes the characteristic information.
  • the reflection removal result for a color image is given by Equation (11), similarly to Equation (10).
  • the Stokes parameters S RGB,1 and S RGB,2 are unknown. Therefore, the reflection-removed image generator 40 of the image processing system 10 uses the Stokes parameter S RGB,0 corresponding to the luminance component of the color image and the Stokes parameters S RGB,1 and S RGB,2 to optimize the Stokes parameters which are the characteristic information to minimize the correlation with the specular reflection component illustrated in Equation (12).
  • I RGB , min S RGB , 0 ⁇ S RGB , 1 2 + S RGB , 2 2 S RGB , 1 2 + S RGB , 2 2
  • the reflection-removed image generator 40 calculates the correlation between the luminance component and the specular reflection component of the color image using the Stokes parameters S gray,1 and S gray,2 when calculating the pixel value (I gray,min ) of the black-and-white reflection removal result in the reflection removal of the black-and-white image as the initial values of the Stokes parameters S RGB,1 and S RGB,2 .
  • the reflection-removed image generator 40 may calculate the geometric correlation by utilizing, for example, the fact that the image of the luminance component and the image of the specular reflection component are different. Alternatively, the reflection-removed image generator 40 may calculate the correlation in a feature amount space by using different feature amounts for the luminance component and the specular reflection component.
  • PSNR Peak Signal-to-Noise Ratio
  • ZNCC Zero-mean Normalized Cross-Correlation
  • SSIM Structuretural Similarity
  • a method such as CNN (Convolutional Neural Network) is used for the correlation of the feature amount space.
  • the reflection-removed image generator 40 calculates the correlation while changing the Stokes parameters S RGB,1 and S RGB,2 from the initial values, and optimizes the Stokes parameters so that the correlation is minimized.
  • the reflection-removed image generator 40 calculates the Stokes parameters S RGB,1 and S RGB,2 that minimize the correlation using a method such as regression or CNN using a gradient method or ADMN (Alternating Direction Method of Multipliers), and then, the process proceeds to step ST14.
  • a method such as regression or CNN using a gradient method or ADMN (Alternating Direction Method of Multipliers)
  • step ST14 the image processing system generates a reflection-removed color image.
  • the reflection-removed image generator 40 of the image processing system 10 performs the calculation of Equation (11) using the Stokes parameters S RGB,1 and S RGB,2 optimized in step ST13 to generate a reflection-removed color image from which specular reflection is removed, and then, the process proceeds to step ST15.
  • the image processing system outputs a reflection-removed color image.
  • the output unit 50 of the image processing system 10 outputs the reflection-removed color image generated in step ST14 to an external device such as a display device, a recording device, or a control device that performs various types of control using the reflection-removed color image.
  • the output unit 50 may use the correlation, which is an evaluation index used in the optimization of the characteristic information in the reflection-removed image generator 40, as the accuracy of the reflection removal result, for example, and superimpose a display based on the correlation when the characteristic information is optimized on the corresponding position of the reflection-removed color image.
  • the output unit 50 may divide the image acquired by the image acquirer 20 into a plurality of regions, calculate the correlation for each region, and superimpose a display based on the correlation when the characteristic information is optimized on the corresponding region of the reflection-removed color image.
  • Fig. 6 is a diagram illustrating an operation example of the second embodiment.
  • Fig. 6(a) illustrates polarized images for each of the four polarization directions acquired by the black-and-white polarized image acquirer 21.
  • Fig. 6(b) illustrates a color image acquired by the color image acquirer 22.
  • the reflection-removed image generator 40 calculates Stokes parameters S gray,1 and S gray,2 from the polarized images of 0°, 45°, 90°, and 135° illustrated in Fig. 6(a) .
  • the reflection-removed image generator 40 uses the Stokes parameters S gray,1 and S gray,2 as initial values of the Stokes parameters S RGB,1 and S RGB,2 .
  • the reflection-removed image generator 40 calculates the correlation between the luminance component and the specular reflection component of the color image, and calculates Stokes parameters S RGB,1 and S RGB,2 that minimize the correlation.
  • the reflection-removed image generator 40 generates a reflection-removed color image illustrated in FIG. 6(c) in which the specular reflection is removed from the color image illustrated in Fig. 6(b) using the optimized Stokes parameters S RGB,1 and S RGB,2 .
  • Fig. 7 illustrates the output of a reflection-removed color image.
  • the output unit 50 outputs the reflection-removed color image and the evaluation index used in the optimization of the characteristic information.
  • the reflection-removed color image can be confirmed and the occurrence state of the specular reflection can be grasped.
  • a reflection-removed color image can be generated with higher sensitivity than when a reflection-removed color image is generated from a color polarized image acquired by performing imaging via a polarizing element and a color filter. According to the second embodiment, it is possible to generate an artifact-free reflection-removed color image.
  • the occurrence state of specular reflection can be grasped by the display based on the evaluation index.
  • the case where the angle of the reflecting surface with respect to the image acquirer 20 of the subject that causes specular reflection is not known has been described.
  • the case where the angle of the reflecting surface is known will be described.
  • the transmission image and the reflection image can be calculated as illustrated in PTL 1. Therefore, the black-and-white reflection image and the black-and-white transmission image calculated based on the black-and-white polarized image are used as characteristic information.
  • Fig. 8 illustrates a configuration of a third embodiment.
  • the image processing system 10 has an image acquirer 20, a reflecting surface information acquirer 30, a reflection-removed image generator 40 and an output unit 50.
  • the image acquirer 20 has a black-and-white polarized image acquirer 21 that acquires a black-and-white polarized image and a color image acquirer 22 that acquires a color image, as in the first embodiment.
  • the black-and-white polarized image acquirer 21 acquires black-and-white polarized images in a plurality of polarization directions.
  • the color image acquirer 22 generates a color image by providing, for example, a three-primary-color mosaic filter on the imaging surface of the image sensor.
  • the reflecting surface information acquirer 30 acquires the reflecting surface information indicating the angle of the reflecting surface of the subject that causes specular reflection in the image acquired by the image acquirer 20, as in PTL 1, and inputs the reflecting surface information to the reflection-removed image generator 40.
  • angle data input by the user in a memory according to the photographing angle of a camera that images a subject that causes specular reflection, and use the stored data to calculate the angle of the reflecting surface.
  • information from an angular sensor or the like provided in a camera that images a subject that causes specular reflection may be set attribute information of an image and be stored in a memory, and the stored data may be used for calculating the angle of the reflecting surface.
  • specified angle information such as 45° may be stored in advance in a memory and the angle information may be used as the angle of the reflecting surface.
  • the angle of the reflecting surface may be calculated as a strict reflection angle for each pixel that is different for each pixel that constitutes an image. However, the same angle in common may be simply set and used for all of the constituent pixels of the image, or for each pixel group of each predetermined region.
  • the reflecting surface information acquirer 30 acquires the reflecting surface information indicating the angle of the reflecting surface of the subject included in the image acquired by the image acquirer 20, for example, by any of the above methods, and outputs the reflecting surface information to the reflection-removed image generator 40.
  • the reflection-removed image generator 40 calculates information, for example, the black-and-white reflection image and the black-and-white transmission image, indicating the polarization characteristics of the black-and-white polarized image using together with the reflecting surface information acquired by the reflecting surface information acquirer 30 as the characteristic information.
  • the reflection-removed image generator 40 uses the characteristic information calculated based on the black-and-white polarized image as an initial value to optimize the characteristic information according to the color image, and generates the reflection-removed color image using the color image, the optimized characteristic information, and the reflecting surface information acquired by the reflecting surface information acquirer 30.
  • the output unit 50 outputs the reflection-removed color image generated by the reflection-removed image generator 40 to an image display device or the like for display.
  • the output unit 50 may output the reflection-removed color image to a recording device to record it on a recording medium, or may transmit the reflection-removed color image to an external device or the like via a network or the like.
  • the output unit 50 may use the correlation, which is an evaluation index used in the optimization of the characteristic information in the reflection-removed image generator 40, as the accuracy of the reflection removal result, for example, and superimpose a display based on the correlation when the characteristic information is optimized on the corresponding position of the reflection-removed color image.
  • the output unit 50 may divide the image acquired by the image acquirer 20 into a plurality of regions, calculate the correlation for each region, and superimpose a display based on the correlation when the characteristic information is optimized on the corresponding region of the reflection-removed color image.
  • Fig. 9 is a flowchart illustrating the operation of the third embodiment.
  • the image processing system acquires a black-and-white polarized image and a color image.
  • the black-and-white polarized image acquirer 21 of the image processing system 10 acquires a black-and-white polarized image.
  • the color image acquirer 22 acquires a color image, and then, the process proceeds to step ST22.
  • step ST22 the image processing system acquires reflecting surface information.
  • the reflecting surface information acquirer 30 of the image processing system 10 receives the input of the reflecting surface angle, and outputs reflecting surface information indicating the zenith angle ⁇ and the azimuth angle ⁇ of the reflecting surface to the reflection-removed image generator 40.
  • step ST23 the image processing system calculates characteristic information. It is known that the reflection-removed image generator 40 of the image processing system 10 can fit the luminance change observed when the polarization angle is changed to a cosine curve, as described above. Here, when the angle of the reflecting surface is known, the cosine curve can be expressed as Equation (13) as disclosed in PTL 1.
  • the reflection-removed image generator 40 calculates the pixel value I gray,reft of the black-and-white reflection image and the pixel value I gray,trans of the black-and-white transmission image by solving the simultaneous equations obtained by setting the polarization direction ⁇ pol in Equation (13) to two different polarization directions.
  • the reflection-removed image generator 40 proceeds to step ST24 using the calculated pixel value I gray,reft of the black-and-white reflection image and the calculated pixel value I gray,trans of the black-and-white transmission image as characteristic information.
  • step ST24 the image processing system optimizes the characteristic information.
  • the reflection removal result for a color image is given by Equation (14), similarly to Equation (13).
  • Equation (14) the pixel value I RGB,reft of the color reflection image and the pixel value I RGB,trans of the color transmission image, which are characteristic information, are unknown. Therefore, the reflection-removed image generator 40 of the image processing system 10 uses the pixel value I gray,reft of the black-and-white reflection image and the pixel value I gray,trans of the black-and-white transmission image as initial values to optimize the characteristic information to minimize the correlation between the pixel value I RGB,reft of the color reflection image representing the reflection component and the pixel value I RGB,trans of the color transmission image representing the transmission component.
  • the reflection-removed image generator 40 may calculate the geometric correlation as in the second embodiment, or may calculate the correlation in the feature amount space.
  • the reflection-removed image generator 40 optimizes the characteristic information so that the calculated correlation is minimized.
  • the reflection-removed image generator 40 calculates the characteristic information that minimizes the correlation using the same method as in the second embodiment, and then, the process proceeds to step ST25.
  • step ST25 the image processing system generates a reflection-removed color image.
  • the reflection-removed image generator 40 of the image processing system 10 performs the calculation of Equation (14) using the characteristic information optimized in step ST24 and the angle indicated by the reflecting surface information acquired in step ST22 to generate a reflection-removed color image in which specular reflection is removed, and then, the process proceeds to step ST26.
  • the image processing system outputs a reflection-removed color image.
  • the output unit 50 of the image processing system 10 outputs the reflection-removed color image generated in step ST25 to an external device such as a display device, a recording device, or a control device that performs various types of control using the reflection-removed color image.
  • the output unit 50 may use the correlation, which is an evaluation index used in the optimization of the characteristic information in the reflection-removed image generator 40, as the accuracy of the reflection removal result, for example, and superimpose a display based on the correlation when the characteristic information is optimized on the corresponding position of the reflection-removed color image.
  • the output unit 50 may divide the image acquired by the image acquirer 20 into a plurality of regions, calculate the correlation for each region, and superimpose a display based on the correlation when the characteristic information is optimized on the corresponding region of the reflection-removed color image.
  • a reflection-removed color image can be generated with higher sensitivity than when a reflection-removed color image is generated from a color polarized image acquired by performing imaging via a polarizing element and a color filter. According to the third embodiment, it is possible to generate an artifact-free reflection-removed color image.
  • the occurrence state of specular reflection can be grasped by the display based on the evaluation index.
  • a reflection-removed color image can be generated using reflecting surface information that indicates the angle of the reflecting surface of the subject.
  • a case where the color image obtained by the color image acquirer 22 is in the RGB color space, but a color image in another color space may be used.
  • a color image in another color space may be used.
  • color artifacts can be made less likely to occur by optimizing the characteristic information so that the color correlation is reduced.
  • the technology according to the present disclosure can be applied to various fields.
  • the technology according to the present disclosure may be realized as a device equipped in any type of moving body such as an automobile, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, a personal mobility device, an airplane, a drone, a ship, and a robot.
  • the technology may be realized as a device mounted in equipment that is used in a production process in a factory or equipment that is used in a construction field.
  • the series of processing described in the specification can be executed by hardware, software, or a composite configuration of both.
  • a program in which a processing sequence has been recorded is installed in a memory in a computer embedded in dedicated hardware and executed.
  • the program can be installed in a general-purpose computer capable of executing various types of processing and executed.
  • the program can be recorded in advance in a hard disk, SSD (Solid State Drive), or ROM (Read Only Memory) as a recording medium.
  • the program can be temporarily or permanently stored (recorded) in a removable recording medium such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), a MO (Magneto optical) disk, a DVD (Digital Versatile Disc), a BD (Blu-Ray Disc (registered trademark)), a magnetic disk, and a semiconductor memory card.
  • a removable recording medium can be provided as so-called package software.
  • the program may be transferred from a download site to the computer wirelessly or by wire via a network such as a local area network (LAN) or the Internet, in addition to being installed in the computer from the removable recording medium.
  • the computer can receive the program transferred in this way and install the program in a recording medium such as a built-in hard disk.
  • the signal processing device of the present technology can also have the following configuration.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Software Systems (AREA)
  • Spectroscopy & Molecular Physics (AREA)
  • Geometry (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Image Processing (AREA)
  • Color Television Image Signal Generators (AREA)
EP21897546.4A 2020-11-30 2021-10-19 Bildverarbeitungsvorrichtung, bildverarbeitungsverfahren und programm Pending EP4254935A4 (de)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2020198069 2020-11-30
PCT/JP2021/038543 WO2022113568A1 (ja) 2020-11-30 2021-10-19 画像処理装置と画像処理方法およびプログラム

Publications (2)

Publication Number Publication Date
EP4254935A1 true EP4254935A1 (de) 2023-10-04
EP4254935A4 EP4254935A4 (de) 2024-04-10

Family

ID=81754298

Family Applications (1)

Application Number Title Priority Date Filing Date
EP21897546.4A Pending EP4254935A4 (de) 2020-11-30 2021-10-19 Bildverarbeitungsvorrichtung, bildverarbeitungsverfahren und programm

Country Status (5)

Country Link
US (1) US20230410464A1 (de)
EP (1) EP4254935A4 (de)
JP (1) JPWO2022113568A1 (de)
CN (1) CN116547984A (de)
WO (1) WO2022113568A1 (de)

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP5428509B2 (ja) * 2009-05-11 2014-02-26 ソニー株式会社 ２次元固体撮像装置、及び、２次元固体撮像装置における偏光光データ処理方法
JP5853719B2 (ja) * 2012-01-20 2016-02-09 株式会社リコー 画像処理システム及びそれを備えた車両
JP6485078B2 (ja) * 2014-02-18 2019-03-20 パナソニックＩｐマネジメント株式会社 画像処理方法および画像処理装置
JP6447121B2 (ja) * 2014-12-26 2019-01-09 株式会社リコー 画像処理装置、画像処理方法、撮像システム、画像処理システム、およびプログラム
EP3544283B1 (de) * 2016-11-15 2023-07-19 Sony Group Corporation Bildverarbeitungsvorrichtung
JPWO2020213238A1 (de) * 2019-04-19 2020-10-22

Also Published As

Publication number Publication date
US20230410464A1 (en) 2023-12-21
JPWO2022113568A1 (de) 2022-06-02
CN116547984A (zh) 2023-08-04
EP4254935A4 (de) 2024-04-10
WO2022113568A1 (ja) 2022-06-02

Similar Documents

Publication Publication Date Title
JP4787871B2 (ja) カラー偏光撮像装置
EP2268043B1 (de) Bildverarbeitungsvorrichtung sowie abbildungsvorrichtung, -verfahren und -programm
US20150278996A1 (en) Image processing apparatus, method, and medium for generating color image data
CN102685511B (zh) 图像处理设备和图像处理方法
US9848118B2 (en) Phase detection autofocus using opposing filter masks
EP3937476A1 (de) Bildaufnahmesystem, bildverarbeitungsvorrichtung und bildverarbeitungsverfahren
US9429834B2 (en) Three-dimensional imaging device
US20170257584A1 (en) Image processing device, image processing method, and image processing system
JP6422924B2 (ja) 撮像装置および被写体情報取得方法
US20120230549A1 (en) Image processing device, image processing method and recording medium
JP7024736B2 (ja) 画像処理装置、および画像処理方法、並びにプログラム
US9544570B2 (en) Three-dimensional image pickup apparatus, light-transparent unit, image processing apparatus, and program
US20160035099A1 (en) Depth estimation apparatus, imaging device, and depth estimation method
EP4254935A1 (de) Bildverarbeitungsvorrichtung, bildverarbeitungsverfahren und programm
JP2013044597A (ja) 画像処理装置および方法、プログラム
US20210142446A1 (en) Image processing device, image processing method, and non-transitory computer-readable medium
EP3944182A1 (de) Verfahren und vorrichtung zur bildwiederherstellung
US11930160B2 (en) Image processing apparatus, image processing method, and program
KR20220040025A (ko) 딥러닝 기반 열 영상 재구성 장치 및 방법
US20230239580A1 (en) Image processing apparatus, image processing method, and program
Ahmed et al. Polarization imaging for remote sensing
US20230316708A1 (en) Signal processing device, signal processing method, and program
KR20200043778A (ko) 영상 처리 장치 및 방법
JP2022074293A (ja) 情報処理装置と情報処理方法およびプログラム
Herwig et al. Spatial Gaussian filtering of Bayer images with applications to color segmentation

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20230619

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: DE

Ref legal event code: R079

Free format text: PREVIOUS MAIN CLASS: H04N0005232000

Ipc: H04N0023810000

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)
A4 Supplementary search report drawn up and despatched

Effective date: 20240311

RIC1 Information provided on ipc code assigned before grant

Ipc: H04N 5/265 20060101ALI20240304BHEP

Ipc: G06V 10/56 20220101ALI20240304BHEP

Ipc: G06V 10/60 20220101ALI20240304BHEP

Ipc: H04N 25/13 20230101ALI20240304BHEP

Ipc: H04N 25/133 20230101ALI20240304BHEP

Ipc: H04N 23/81 20230101AFI20240304BHEP