JP2013044597A - Image processing device and method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To estimate a distance to a subject accurately even if components of specular reflection light are included in captured image data.SOLUTION: On the basis of plural polarization image data pieces captured with plural cameras having polarization filters with different angles of polarization planes, at least one of the intensity of specular reflection light of a subject and the angle of the polarization plane is estimated, and a distance to the subject is estimated on the basis of the results.

Description

  The present invention relates to an image processing apparatus, method, and program for processing a plurality of image data captured by a plurality of cameras.

Conventionally, there is a distance estimation technique for estimating a distance to a target object using a stereo camera or a multi-view imaging apparatus in which a plurality of cameras are arranged on a lattice. This distance estimation technique is a method of measuring using parallax. In this measurement, it is necessary to search for corresponding points between images in a region where a plurality of images overlap with the position of the camera.
However, the error in searching for corresponding points increases due to the influence of imaging conditions. For example, since a night-time captured image has little visible light information, an error in searching for corresponding points increases. In order to reduce the error of searching for corresponding points for conditions with little visible light information at night, the visible light camera and infrared camera are divided among multiple cameras, and the vehicle is made from the binarized visible image and infrared image. A method for recognizing is proposed. (Patent Document 1)

JP 10-255019 A

However, Patent Document 1 is a method of synthesizing images with different wavelengths, and does not perform processing for polarization of light. Therefore, the specular reflection portion that occurs on the surface of glass, water, plastic, or the like cannot be matched between images of different wavelengths, and there is a problem that the accuracy of distance estimation is reduced.
Therefore, an object of the present invention is to estimate the distance to a subject with high accuracy even when captured image data includes a component of specular reflection light.

  In order to achieve the above object, an image processing apparatus of the present invention is an image processing apparatus that processes a plurality of image data captured by a plurality of cameras, and is provided with the polarization filters having different polarization plane angles. Polarized image acquisition means for acquiring a plurality of polarization image data captured by a plurality of cameras, and estimating at least one of the intensity of the specular reflection light and the angle of the polarization plane of the subject based on the plurality of polarization image data Specular reflection light estimation means, and distance estimation means for estimating the distance to the subject based on the estimation result by the specular reflection light estimation means.

  According to the present invention, it is possible to estimate the distance to the subject with high accuracy even when the captured image data includes a component of specular reflection light.

It is an image figure which attached the polarizing filter to the multi-eye imaging device. 1 is a system configuration diagram of Embodiment 1. FIG. It is a system block diagram of the imaging parts 101-109. 1 is a block diagram illustrating Example 1. FIG. It is a figure which shows the effect of Example 1. FIG. 3 is a flowchart showing the operation of the first embodiment. It is a flowchart which shows operation | movement of the depth calculation part 410 of specular reflection light. 3 is an image diagram of a captured image of Example 1. FIG. It is a flowchart which shows operation | movement of the specular reflection light estimation part 406. It is an image figure of the polarization image of Example 1, and a ranging image. It is a figure which shows the property of the incident / transmitted light of the specular reflection light with respect to a polarizing filter. 5 is a flowchart showing an operation of a ranging image generation unit 407.

[Example 1]
In this embodiment, N (N = 3 in this embodiment) polarizing filters having different polarization planes are arranged in a multi-lens imaging device. The distance estimation accuracy can be improved by estimating the intensity of the specular reflection light and the angle of the polarization plane in the subject with high accuracy using the properties of the incident / transmitted light of the polarized light with respect to the polarizing filter. Further, as the number of cameras increases, the number of polarizing filters that can be mounted also increases, and the estimation accuracy of the intensity of the specular reflection light and the angle of the polarization plane is improved.

  FIG. 1 is an image diagram in which a polarizing filter is attached to a multi-lens imaging device. FIG. 1A shows an example of a multi-lens imaging device including a plurality of cameras. The nine cameras 101 to 109 are arranged on a 3 × 3 square lattice. All the cameras are arranged on the same plane of the casing 110, and their optical axes are all substantially parallel and perpendicular to the arranged plane. In this embodiment, a distance measuring camera and an imaging camera are distinguished from each other in a plurality of cameras. FIG. 1A shows an example of the arrangement of ranging cameras and imaging cameras. In this embodiment, as shown in FIG. 1A, the ranging camera is equipped with a polarizing filter in order to improve the distance estimation accuracy. The oblique line at the center of the distance measuring camera in FIG. 1A is a line representing the angle of the polarization plane of the attached polarizing filter.

  1B and 1C are image diagrams of images obtained by photographing a subject. FIG. 1B shows an image diagram of a captured image acquired by an imaging camera without a polarizing filter. FIG. 1C shows an image diagram of a polarization image acquired by a ranging camera with a polarization filter. When there is no polarizing filter in FIG. 1B, there is a region (specular reflection region) corresponding to the reflected light of the subject on the captured image. On the other hand, when the polarizing filter of FIG. 1C is provided, the specular reflection area can be removed by the polarizing filter, and there is no specular reflection area on the captured image.

  The system configuration of this embodiment will be described with reference to FIG. The imaging units 101 to 109 receive light information from the subject with a sensor, perform A / D conversion, and output digital data to the bus 214 which is a data transfer path. The flash 201 irradiates the subject with light. A ROM 202 and a RAM 203 provide the CPU 204 with programs, data, work areas, and the like necessary for imaging and image processing. The CPU 204 executes a program stored in the ROM 202 or the RAM 203 using the RAM 203 as a work memory, and controls each component via the bus 214. Thereby, various processes described later are executed. The imaging device control unit 205 controls the imaging system instructed by the CPU 204 such as focusing, opening a shutter, and adjusting an aperture. The operation unit 206 corresponds to buttons, mode dials, and the like, and receives user instructions input via these buttons. The CG generation unit 207 generates characters and graphics. Generally, a liquid crystal display is widely used as the display unit 208, and displays images and characters indicated by captured image data received from the CG generation unit 207, a digital signal processing unit 209, which will be described later, and the image processing unit 213. . Further, it may have a touch screen function. In that case, a user instruction can be handled as an input of the operation unit 206. The digital signal processing unit 209 performs white balance processing, gamma processing, noise reduction processing, and the like on the digital value to generate digital image data. The compression / decompression unit 210 performs a process of converting the digital value into a file format such as Jpeg or Mpeg. The external memory control unit 211 is an interface for connecting to a PC or other media 212 (for example, hard disk, memory card, CF card, SD card, USB memory). The image processing unit 213 generates new image data using the digital image data obtained from the imaging units 101 to 109 or the digital image data group output from the digital signal processing unit, and the result is sent to the bus 214. Output.

  Details of the imaging units 101 to 109 will be described with reference to FIG. The imaging units 101 to 103 have a mounting filter 301 (a polarizing filter in this embodiment), while the imaging units 104 to 109 do not have a polarizing filter 301. Furthermore, the imaging units 101 to 109 include a lens 302, a diaphragm 303, a shutter 304, an optical filter 305, a sensor 306, and the like, and detect the amount of light of the subject. The A / D conversion unit 307 converts the light amount of the subject into a digital value and outputs the digital data to the bus 214.

  Although there are other components of the apparatus than the above, the description thereof is omitted because it is not the main point of the present embodiment.

  FIG. 4 is a block diagram showing an embodiment of an image processing apparatus to which this embodiment can be applied. The input polarization image data group is input from the input terminal 401 to the image processing unit 213. Polarization filter angle information corresponding to the input polarization image data group is input from the input terminal 402. Similarly, the captured image data group is input from the input terminal 403 to the image processing unit 213. Camera parameters of the polarization image data group and the captured image data group are input from the input terminal 404 to the image processing unit 213.

  The polarization image acquisition unit 405 acquires polarization image data input from the input terminal 401. Similarly, the captured image acquisition unit 409 acquires captured image data input from the input terminal 403. The acquired captured image data is input to the depth calculation unit 410 of the high luminance part, and the depth when the high luminance part is focused is calculated. The calculated depth of the high luminance part and the polarization image data acquired by the polarization image acquisition unit 405 are input to the specular reflection light estimation unit 406, and the specular reflection light intensity and the angle of the polarization plane are estimated. The estimated specular reflection light intensity and the angle of the polarization plane are input to the distance measurement image generation unit 407, and image data from which the specular reflection light has been removed is generated as distance measurement image data. The distance estimation unit 408 performs distance estimation from the distance measurement image data generated by the distance measurement image generation unit 407. The result is input to the image synthesis unit 411 and synthesized with the captured image data acquired by the captured image acquisition unit 409 to generate highly accurate three-dimensional image data (synthesized image data). The combined image data combined by the image combining unit 411 is output from the output terminal 412.

  When this embodiment is applied, as shown in FIG. 5, ranging image data obtained by removing or reducing specular reflection light having different polarization plane angles on the polarization image (FIG. 5A) indicated by the polarization image data. Can be obtained (FIG. 5B). According to the distance measurement image data from which the influence of the specular reflection light is removed or reduced, it is possible to realize distance estimation with very high accuracy. The depth calculation unit 410 of the high luminance part can also estimate the depth of the high luminance part. However, since distance estimation is based on captured image data that remains affected by specular reflected light, the distance estimation by the depth calculation unit 410 of the high-luminance part is not as accurate as the distance estimation by the distance estimation unit 408. In this embodiment, since the specular reflection light intensity and the angle of the polarization plane are estimated, the specular reflection light on the image can be added / removed with arbitrary weighting.

[Operation of Image Processing Unit 213]
FIG. 6 is a flowchart showing the image processing method of this embodiment. More specifically, after a computer-executable program describing the procedure shown in the flowchart of FIG. 6 is read from the ROM 202 onto the RAM 203, the CPU 204 executes the program to execute the process.

  Hereinafter, each process shown in FIG. 6 will be described. The captured image acquisition unit 409 inputs the captured image data to a storage area such as the ROM 202 or the RAM 203 (S601). The polarization image acquisition unit 405 inputs the polarization image data to the storage area (S602). In step S603, the camera parameters for capturing the captured image and the polarization image input in steps S601 and S602 are input to the storage area. The camera parameters are composed of internal parameters representing the focal length and optical axis deviation of each camera, external parameters representing the three-dimensional coordinates of each camera (parameters including arrangement and orientation information of each camera), and the like. The depth calculation unit 410 of the high luminance part calculates depth data when the high luminance part on the captured image indicated by the captured image data input in step S601 is focused (S604). Depth data is the distance between the position of the subject corresponding to the high luminance portion in real space and the camera. Details of the method of calculating the depth data of the high luminance part will be described later with reference to FIG. The calculated depth data of the high luminance part is stored in the storage area in association with the pixel area of the high luminance part. The specular reflection light estimation unit 406 determines whether there is a high luminance part on the captured image by checking whether the depth data of the high luminance part calculated in step S604 is in the storage area (S605). If there is depth data of the high luminance part, it is determined that there is a high luminance part, and the process proceeds to step S606. If there is no depth data of the high luminance part, it is determined that there is no high luminance part, and the process proceeds to step S609. Further, the specular reflection light estimation unit 406 inputs the polarization filter mounting angle corresponding to the polarization image data input in step S602 to the storage area (S606). The specular reflection light intensity and the angle of the polarization plane are estimated from the polarization image data input in step S602 (S607). The specular reflection light is estimated by using the depth data of the high brightness portion calculated in step S604 and the polarization filter angle input in step S606. Details of the specular reflection light estimation method will be described later with reference to FIG. The estimation result is stored in the storage area. The ranging image generation unit 407 generates ranging image data in which the influence of specular reflection light is removed / reduced from the polarization image data input in step S602 (S608). Details of the ranging image generation method will be described later with reference to FIG. The generated ranging image data is stored in the storage area. The distance estimation unit 408 estimates the distance using the distance measurement image data generated in step S608 (S609). If there is no high brightness portion in step S605, the distance is estimated using the polarization image data input in step S602. The distance estimation result is stored in the storage area. The image composition unit 411 performs image composition processing using the captured image data input in step S601 and the distance estimation result estimated in step S609, and ends the processing (S610).

[Operation of Depth Calculation Unit 410 of High Brightness Part]
Hereinafter, details of the depth calculation of the specular reflection light of the high-luminance part in step S604 will be described using the flowchart of FIG. Here, it is assumed that there are M imaging cameras (M = 6 in this embodiment), and the captured images captured by the respective imaging cameras 104 to 109 are numbered as captured image 1, captured image 2,. It shall be.

In step S701, i = 1 is initialized for the number i of the captured image. In step S702, the RGB value of the captured image i input in step S601 is converted to YCbCr. The converted image data is stored in the storage area. In step S703, it is determined whether or not there is a pixel in which the Y value satisfies Y> _Yth in the captured image i converted in step S702. In the present embodiment, for example, Y _th = 150. If _Y> pixels satisfying _{Y th} is present, the process proceeds to step S704. If there is no pixel satisfying Y> _Yth , it is determined that there is no high luminance part, and the process proceeds to step S707. Step S704 in the captured image i in, it is determined whether the region satisfies the _{S> S th} there. S is the number of pixels in the region where the pixels determined to satisfy Y> Y _th in step S703 are gathered. That is, S is a pixel region composed of pixels in which pixels satisfying Y> _Yth are adjacent to each other either vertically or horizontally. In this embodiment, for example, S _th = 10 [pixel]. If a pixel area satisfying S> S _th is present, it is determined that the high luminance portion exists, the process proceeds to step S705. If there is no region satisfying S> _Sth , it is determined that there is no high luminance part, and the process proceeds to step S707. Pixel noise on the captured image i can be removed by the processing in step S704. In step S705, high luminance portions R _i1 , R _i2 ,... _Are assigned to the pixel areas determined as the high luminance portions in step S704 as shown in FIG. In step S706, the high luminance portions R _i1 , R _i2 ,... Assigned to the numbers in step S705 are associated with the pixel areas and stored in the storage area. In step S707, the number i of the captured image is updated to i = i + 1. In step S708, it is determined whether the number i of the captured image satisfies i ≧ M. If i ≧ M is satisfied, it is determined that the processing has been completed for all M captured images, and the process proceeds to step S709. If i ≧ M is not satisfied, the process proceeds to step S702. In step S709, the pixel regions of the high luminance portions R _i1 , R _i2 ,... Of each captured image i stored in step S706 are aligned using template matching. The amount of pixel shift between the captured images that have been aligned is stored in the storage area in association with the number of the high luminance part. In step S710, as shown in FIG. 8B, R ′ ₁ , R ′ ₂ ,... For the high luminance portions R _i1 , R _i2 ,. ... R ′ _L (L = 2 in this embodiment) and numbers are assigned again. In step S711, the high luminance portions R ′ ₁ , R ′ ₂ ,... R ′ _L assigned with the numbers in step S710 and the pixel area and the pixel shift amount stored in step S709 are associated with each other and stored in the storage area. . Further, the number L of high-luminance portions that can be aligned is also stored in the storage area. In step S712, each high brightness portion is focused from the pixel shift amount corresponding to the high brightness portions R ′ ₁ , R ′ ₂ ,... R ′ _L stored in step S711 and the camera parameters input in step S603. Depth data when combined is calculated. The calculated depth is stored in the storage area as the depth of the high luminance portion in association with the high luminance portions R ′ ₁ , R ′ ₂ ,... R ′ _L, and the depth calculation processing of the high luminance portion is terminated.

  In this embodiment, a method of removing / reducing specular reflection light that appears on a plurality of captured images and greatly affects distance estimation is handled. For specular reflection light that appears in a single image in the captured image data group, for example, the specular reflection light may be removed using the pixel value of the captured image in which the specular reflection light is not reflected.

[Operation of Specular Reflection Light Estimation Unit 406]
Hereinafter, details of the specular reflection light estimation processing in step S607 will be described with reference to the flowchart of FIG. Here, it is assumed that there are N distance measuring cameras (N = 3 in this embodiment), and the polarization images taken by the distance measurement cameras 101 to 103 are the polarization image 1, the polarization image 2,. Shall be attached. As shown in FIG. 10A, each polarized image has high-luminance portions R ′ ₁ , R ′ ₂ ,... R ′ _L appearing on the image. However, it differs from the captured image of FIG. 8B in that the brightness of the high brightness portion changes depending on the angle of the polarization filter.

In step S901, j = 1 is initialized for the number j of the high-luminance portion R ′ _j . In step S902, the pixel shift amount of the polarization image 1, the polarization image 2,..., And the polarization image N is calculated from the depth of the high luminance portion calculated in step S712 and the camera parameter input in step S603. The calculated pixel shift amount is stored in the storage area in association with each polarization image. In step S903, the high in the polarization image 1, the polarization image 2,..., The polarization image N is calculated from the pixel region corresponding to the high luminance part R ′ _j stored in step S711 and the pixel shift amount calculated in step S902. The pixel area of the luminance part R ′ _j is calculated. The pixel area of the high luminance portion R ′ _j on the calculated polarization image is stored in the storage area in association with each polarization image. In step S904, the polarization image 1, the polarization image 2,..., And the polarization image N are aligned based on the polarization image input in step S602 and the pixel shift amount calculated in step S902. As shown in FIG. 10B, the aligned polarization image is in focus on the high luminance portion R ′ _j (j = 2 in FIG. 10B), and the other high luminance portions are blurred images. It becomes. Whether or not the alignment can be performed may be determined by checking whether or not the pixel area of the high-luminance portion R ′ _j for each polarization image calculated in step S903 is within the polarization image. The aligned polarization image is stored in the storage area in association with the high luminance portion R ′ _j . In step S905, the high luminance portion _R'j, counts the number _{k j} polarized images that could alignment in step S904. The count number k _j is stored in the storage area in association with the high luminance part R ′ _j . In step S906, it is determined whether k _j counted in step S905 satisfies k _j ≧ 3. When k _j ≧ 3 is satisfied, the process proceeds to step S907. When k _j ≧ 3 is not satisfied, it is determined that the specular reflection light of the high-luminance portion R ′ _j cannot be estimated, and the process proceeds to step S908. In step S907, the polarization image stored in step S904, the estimating the angle of the specular reflection light intensity and the polarization plane for each pixel in the high luminance portion _R'j. The estimated intensity of the specular reflection light and the angle of the polarization plane are stored in the storage area in association with each pixel position in the high luminance portion R ′ _j . Details of the specular reflection light estimation method will be described later. In step S908, the number j of the high luminance part R ′ _j is updated to j = j + 1. In step S909, it determines the number j of the high luminance portion _R'j whether meet j ≧ L. When i ≧ L is satisfied, it is determined that the process has been executed for all L high-luminance parts, and the specular reflection light estimation process is terminated. When j ≧ L is not satisfied, the process proceeds to step S902.

Details of the specular reflection light estimation method in step S907 will be described below. The specular reflection light is estimated by using the properties of incident / transmitted light of the specular reflection light with respect to the polarizing filter shown in FIG. In general, specular reflection light includes a lot of light (s-polarized light) polarized at an angle parallel to the reflection surface. Therefore, when the polarizing filter is attached to the camera, the intensity of the specular reflected light reflected on the camera changes depending on the attaching direction (FIG. 11A). When the angle between the polarization plane of the specular reflection light incident on the polarization filter and the polarization plane of the polarization filter is θ, the electric field amplitude of the specular reflection light transmitted through the polarization filter is the cosine component of the electric field of the incident light. (FIG. 11B). When the incident light intensity of the specular reflected light incident on the polarizing filter is I ₀ , and the transmitted light intensity of the specular reflected light that passes through the polarizing filter and reaches the camera is I ₁ , the following equation (1) is established.

By using the relationship of the expression (1), the specular reflection light intensity of the high luminance part R ′ _j and the angle of the polarization plane can be estimated with high accuracy.

Hereinafter, with reference to FIG. 10B, the specular reflection light estimation method using the point P in the high luminance portion R ′ _j (j = 2 in FIG. 10B) that satisfies k _j ≧ 3 as an example. Will be explained. The RGB values of the polarization image 0, the polarization image 1,..., The point P on the polarization image N stored in step S904 are (r ₀ , g ₀ , b ₀ ), (r ₁ , g ₁ , b ₁ ), ... (r _N , g _N , b _N ). Also, the polarization filter angles with respect to the horizontal direction of the respective polarization images are set to 0 °, α,. In this embodiment, α = 10 °. The RGB value of the specular reflection light at the point P is (r _A , g _A , b _A ), and the RGB value mainly representing the object color other than the specular reflection light is (r _C , g _C , b _C ). Furthermore, when the angle between the polarization plane of the specular reflection light and the polarization filter corresponding to the polarization image 0 is θ, the following equation (2) is obtained for the R value.

Since there are three unknowns r _A , r _C , and θ from the equation (2), the specular reflection light intensity r _A and the angle θ of the plane of polarization are set at a point P on the high-luminance portion R ′ _j that satisfies k _j ≧ 3. It can be estimated using the least squares method. The following formulas (3) and (4) are also obtained for the G value and the B value. Similarly, the specular reflection light intensity g _A , b _{A of} G value and B value and the angle θ of the polarization plane can be estimated.

The estimated RGB values (r _A , g _A , b _A ) of the intensity of the specular reflection light and the angle θ of the polarization plane are stored in the storage area in association with the point P in the high luminance portion R ′ _j . Above, the formula (2) By performing to Formula calculations (4) relative to the pixel area all in the high luminance portion _R'j, estimates the specular reflection light in the high-luminance portion _R'j in units of pixels be able to. In order to improve the accuracy of specular reflection light estimation, a process for determining whether the high-luminance portion R ′ _j is specular reflection light or light from a light source may be provided. Specifically, the estimated specular reflection light and the RGB value of the object color are converted into YCbCr values, and the respective Y values are compared for determination. When the Y value of the specular reflection light is sufficiently smaller than the Y value of the object color, the high-luminance portion R ′ _j determines that the light is light from the light source, and the RGB value of the specular reflection light is (0, 0, 0). What should I do?

[Operation of Ranging Image Generation Unit 407]
Hereinafter, details of the ranging image generation processing in step S608 will be described with reference to the flowchart of FIG.

In step S1201, j = 1 is initialized for the number j of the high-luminance portion R ′ _j . In step S1202, it is determined whether k _j counted in step S905 satisfies k _j ≧ 3. When k _j ≧ 3 is satisfied, the process proceeds to step S1203. When k _j ≧ 3 is not satisfied, it is determined that a ranging image of the high luminance portion R ′ _j cannot be generated, and the process proceeds to step S1205. In step S1203, the RGB value (r _A , g _A , b _A ) of the specular reflection light intensity corresponding to each pixel position in the high luminance portion R ′ _j and the angle θ of the polarization plane estimated in step S907 are obtained. Enter in the storage area. In step S1204, from the specular reflection light intensity input in step S1203 and the angle of the polarization plane, from the high brightness portion R ′ _j corresponding to the specular reflection light of the subject in the image indicated by each polarization image stored in step S904, the specular surface is obtained. Subtract the effect of reflected light. The polarized image 1, the polarized image 2,..., And the polarized image N obtained by subtracting the specular reflection component in the high luminance portion R ′ _j are overwritten and stored in the storage area. In step S1205, the number j of the high brightness portion R ′ _j is updated to j = j + 1. In step S1206, it determines the number j of the high luminance portion _R'j whether meet j ≧ L. If i ≧ L is satisfied, it is determined that the difference processing has been completed for all L high-luminance portions, and the process proceeds to step S1207. When j ≧ L is not satisfied, the process proceeds to step S1202. In step S1207, the polarization image 1, polarization image 2,..., Polarization image N overwritten and stored in step S1204 are stored in the storage area as distance measurement image 1, distance measurement image 2,. Then, the distance measurement image generation process ends. As shown in FIG. 10 (c), the generated ranging image includes the high-luminance portions R ′ ₁ , R ′ ₂ , which satisfy k _j ≧ 3 from the polarization image 1, the polarization image 2,. ... R ′ _L (N = 3, L = 2 in this embodiment) is removed.

  By performing the specular reflection light estimation process using the polarization filter described above, it is possible to improve the distance estimation accuracy in the multi-view imaging apparatus.

  In this embodiment, the polarizing filter angles are set to 0 °, 10 °, and 20 °. However, they need not be equally spaced and may take any value as long as three or more different angles exist. For example, a total of six polarization filter angles of 0 °, 10 °, and 20 °, and 90 °, 100 °, and 110 ° orthogonal thereto may be provided so that the pixel values of the specular reflection light reflected in the polarization image are not saturated. .

  Further, in the present embodiment, θ obtained by the equations (2) to (4) is theoretically equal, so that the specular reflection light can be estimated with high accuracy by repeatedly calculating so that the values of θ are equal. Also good.

Furthermore, in order to estimate specular reflection light with a large change in depth, the high-luminance portion R ′ _j may be divided for each depth and specular reflection light estimation processing may be performed. For example, the depth of focus in the captured image is scanned by Δz [mm]. A high-luminance area in focus for each depth, and extracted as a high luminance portion _R'j, may be estimated specular reflection light.

  As described above, according to the present embodiment, it is possible to estimate the distance to the subject with high accuracy even when the component of the specular reflection light is included in the captured image data.

<Modification>
In the present embodiment, each process is performed by the imaging apparatus as shown in FIG. 2, but the present invention is not limited to this. For example, although polarization image data and captured image data are captured by a multi-lens imaging device, each process of the image processing unit 213 may be realized by an information processing device (for example, a personal computer).
In the present embodiment, the image composition unit 411 performs image composition processing using the captured image data and the distance estimation result. However, when performing image composition, polarization image data may be used in addition to the captured image data.
In the present embodiment, only polarization image data is used for distance measurement image estimation. However, ranging image estimation may be performed using captured image data.

Claims (7)

An image processing apparatus that processes a plurality of image data captured by a plurality of cameras,
Polarization image acquisition means for acquiring a plurality of polarization image data imaged by the plurality of cameras equipped with polarization filters having different polarization plane angles;
Specular reflection light estimation means for estimating at least one of the intensity of the specular reflection light of the subject and the angle of the polarization plane based on the plurality of polarization image data;
Distance estimation means for estimating the distance to the subject based on the estimation result by the specular reflection light estimation means;
An image processing apparatus comprising:   The image processing apparatus according to claim 1, wherein the plurality of cameras are mounted with polarization filters at angles of three or more different polarization planes. Depth calculation means for calculating a distance from the captured image data captured by the plurality of cameras to a region corresponding to the specular reflection light in the subject;
The specular reflection light estimation means estimates at least one of the intensity of specular reflection light and the angle of the polarization plane of the subject based on the distance calculated by the depth calculation means and the plurality of polarization image data. The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus.   The distance estimation unit subtracts the influence of the specular reflection light from the region corresponding to the specular reflection light of the subject in the image indicated by the polarization image data based on the estimation result by the specular reflection light estimation unit. The image processing apparatus according to claim 1.   An imaging apparatus comprising the image processing apparatus according to claim 1. An image processing method for processing a plurality of image data captured by a plurality of cameras,
A polarization image acquisition step of acquiring a plurality of polarization image data captured by the plurality of cameras equipped with polarization filters having different polarization plane angles;
A specular reflection estimation step of estimating at least one of the intensity of the specular reflection light of the subject and the angle of the polarization plane based on the plurality of polarization image data;
Based on the estimation result of the specular reflection light estimation step, a distance estimation step of estimating the distance to the subject;
An image processing method comprising:   The program for functioning a computer as an image processing apparatus of any one of Claims 1 thru | or 4.