JP2015046776A - Imaging device and image processing device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging device that can perform proper imaging even when brightness image information is small.SOLUTION: An imaging device has an imaging optical system for forming an imaging target image, a light receiving unit for outputting primary brightness image information: Sand polarized image information, a polarized image component generator for dividing the polarized image information by secondary brightness image information containing the primary brightness image information: Sto obtain a polarized image component, and a brightness polarized superimposing unit for superimposing a superimposition polarized image component using the polarized image component as brightness variation of each pixel on brightness image information giving the primary brightness image information and outputting a brightness polarized superimposed image. The secondary brightness image information as a divisor of the division in the polarized image component generator is a function: f(S) containing the primary brightness image information: Sas a variable, and set to be equal to a predetermined definite value for the primary brightness image information: S=0 or S≒0.

Description

  The present invention relates to an imaging device and an image processing device.

  In an imaging apparatus such as a camera, it is known that a “region-divided polarizing filter” is arranged on an imaging element and imaging is performed by changing the polarization direction of light acquired by a pixel.

  In this way, by obtaining the polarization image information of the captured image, it is possible to determine an object that cannot be determined only by the luminance image information.

  As polarization image information, two-dimensional array information of “polarization degree and differential polarization degree” in each pixel of an image sensor is often used (Patent Document 1).

  The degree of polarization or the degree of differential polarization is not obtained directly by the image sensor, but is an amount obtained by executing an operation for dividing the polarization image information by the luminance image information.

  As described above, the degree of polarization and the difference degree of polarization are obtained through an operation called “division”. Therefore, when the luminance image information in the denominator becomes minute, the degree of polarization becomes extremely large.

  Such a large value is not an “appropriate value” but an expanded noise component, and an appropriate captured image cannot be obtained using “polarization degree or the like with an expanded noise component”.

  The case where the luminance image information is small is, for example, when photographing at night or when the road surface of the road is black and the reflectance of the subject is low.

In such a case, if the noise component is enlarged, proper imaging becomes difficult.
As far as the inventor is aware, there is no prior art document dealing with the problem of expansion of the noise component due to such a small “brightness image information”.

  This invention makes it a subject to implement | achieve the imaging device which can image appropriately, even when luminance image information is small.

Imaging apparatus of the present invention includes an imaging optical system for forming an image of the object to be shot, it receives an image of the shooting target according to the imaging optical system, the primary luminance image information: the S ₀ and the polarization image information A light receiving unit that outputs, a polarized image component generating unit that obtains a polarized image component by dividing the polarized image information by the secondary luminance image information including the primary luminance image information: S ₀ , and the polarized image component A luminance polarization superimposing unit that superimposes the polarization image component for superimposition to be used on luminance image information that gives the primary luminance image information as a change in luminance of each pixel, and outputs luminance polarization superimposed image information; secondary luminance image information as a divisor of the division in the polarization image component generation unit, the primary luminance image information: function including S _0: a f (S _0), the function: f (S ₀₎ is The primary luminance image _{information: S} 0 = 0 or _{S 0} ≒ 0 odor , Characterized in that it is set to a predetermined finite value.

Imaging apparatus of the present invention, when generating the polarized image components, secondary luminance image information by dividing the polarized image information, primary luminance image information: even if the S ₀ is 0 remains in a finite value.

  Therefore, the noise component in the polarization image component is not expanded without limit.

It is a figure for demonstrating the model of the reflection by an object. It is a figure explaining one Embodiment of an imaging device. It is a figure for demonstrating the expansion prevention effect of a noise component. It is a figure for demonstrating the expansion prevention effect of a noise component.

  Prior to describing the embodiment of the invention, first, the handling of polarized light will be described.

  In general, an imaging apparatus performs imaging by forming an image of reflected light from an object to be photographed.

  Therefore, first, the reflection characteristics of an object that is an object to be imaged will be described.

The reflection characteristic of the object surface is expressed by a bi-directional reflection distribution function (hereinafter referred to as “BRDF”).
“BRDF at a certain point” on the object surface depends on the two directions of the incident direction and the reflection direction, and is defined as the ratio of the intensity of the reflected light in the observation direction to the intensity of the incident light from the illumination direction.

  Various reflection models that describe the “polarization phenomenon” have been proposed.

  In the following description, the “Torrance Sparrow model” is used as the specular reflection component, and the “Lambert model” is used as the diffuse reflection component.

  FIG. 1 is a diagram for explaining a model of reflection by an object OB. In FIG. 1, “N” represents an “outward normal vector” at the reflection position of the object OB.

  “L” is a vector from the reflection position toward the light source, and is referred to as “light source vector”. “V” is a vector from the reflection position to the observation position and is referred to as a “gaze direction vector”.

  “H” is a vector in a direction that bisects the angle formed by the light source vector: L and the line-of-sight direction vector: V, and is simply referred to as “bisected direction vector”.

  Each of these vectors: N, L, V, and H are located on the “same plane”.

In this same plane, angles: ψ, θ _L , α, θ _i , and θ are determined as shown in the figure.

“S-polarized light BRDF: ρ _S (ψ, θ)” by the object OB is given as follows.
ρ _S (ψ, θ) = k _S R _S (θ _i ) D + k _d cos θ _L (A)
Similarly, “P-polarized light BRDF: ρ _P (ψ, θ)” is given as follows.
ρ _P (ψ, θ) = k _P R _P (θ _i ) D + k _d cos θ _L (B)
In the expressions (A) and (B), the first term on the right side is a specular reflection model term (Torrance Sparrow model), and the second term is a diffuse reflection model term (Lambert model).

In these equations (A) and (B), “k _S ” represents the specular reflection coefficient of S-polarized light, “k _P ” represents the specular reflection coefficient of P-polarized light, and “k _d ” represents the diffuse reflection coefficient.

“R _S (θ _i ) and R _P (θ _i )” are “Fresnel reflectivities” for S-polarized light and P-polarized light, respectively, and are given by the following equations.

R _S (θ _i ) = {(n ₁ cos θ _i −n ₂ cos θ _i ′) / (n ₁ cos θ _i + n ₂ cos θ _i ′)} ²
R _P (θ _i ) = {(n ₁ cos θ _i '−n ₂ cos θ _i ) / (n ₁ cos θ _i ' + n ₂ cos θ _i )} ²
Here, n ₁ and n ₂ are the refractive indexes of air and the object OB, respectively.

Further, the angle: θ _i ′ has the following relationship with the angle: θ _i and the refractive index: n ₁ , n ₂ .

sin θ _i ′ = (n ₁ / n ₂ ) sin θ _i
Accordingly, “θ _i ′” is given by the following equation.
θ _i ′ = arcsin {(n ₁ / n ₂ ) sin θ _i }.

  The above-mentioned Fresnel reflectance term reflects the polarization dependence as the behavior of the reflection model.

  “D” in the above formulas (A) and (B) is a “normal distribution term” of a minute area at the reflection position.

  Normal distribution term: D is expressed as follows.

D = exp (−α ² / 2a ² )
Here, “α” is the center coordinate value of the distribution function on the right side, and corresponds to the angle parameter α in FIG. “A” is a standard deviation in the distribution function.

  “A” is a “parameter regarding angular distribution” of a minute area. Normal distribution term: D is a “Gaussian distribution” representing a normal distribution.

  Next, “polarization information based on polarization degree” and “polarization information based on differential polarization degree” will be described as polarization information.

“Degree of polarization” is an amount defined using well-known Stokes parameters regarding polarization phenomenon: S ₀ , S ₁ , S ₂ , S ₃ , and is hereinafter abbreviated as “DOP”.

DOP is defined by the following equation using Stokes parameters: S _{0 to} S ₃ .

DOP = {√ (S ₁ ² + S ₂ ² + S ₃ ² )} / S ₀ .

“Differential degree of polarization (Sub-DOP)” is an amount defined as follows using Stokes parameters: S ₀ and S ₁ , and is abbreviated as “SDOP”.

SDOP = S ₁ / S ₀ = {I (0, φ) −I (90, φ)} / {I (0, φ) + I (90, φ)}
Here, I (0, φ) is the “light intensity through the polarizing filter” when the axial angle is 0 degree, and I (90, φ) is the polarizer when the axial angle is 90 degrees. Is the light intensity transmitted through.

When the Stokes parameter: S ₂ = S ₃ = 0, the following relationship holds.
DOP = | SDOP |
That is, the differential polarization degree: SDOP is a special case of the polarization degree: DOP.

  In this specification, polarization information will be described below in accordance with these DOP and SDOP.

In the defining equation for defining the DOP and SDOP, Stokes parameters become denominator (divisor): the amount given by S ₀ is referred to as "luminance information".

In addition, the numerator in the above definition formula: {√ (S ₁ ² + S ₂ ² + S ₃ ² )} is referred to as “polarization information based on DOP”, and S ₁ is referred to as “polarization information based on SDOP”.

  When these are not distinguished, they are simply called “polarization information”.

  FIG. 2 is a diagram for describing one embodiment of an imaging apparatus.

  FIG. 2A is a conceptual diagram for explaining a basic part of the imaging apparatus.

  In FIG. 2A, the part indicated by reference numeral 10 is “imaging optical system”, the part indicated by reference numeral 11 is “imaging element”, the part indicated by reference numeral 14 is “separation part”, and the part indicated by reference numeral 15 is “generation part”. Is.

  Further, a part indicated by reference numeral 20 is a “luminance polarization superimposing part”, and a part indicated by reference numeral 30 is an “image processing / image recognition part”.

  Among these, the imaging optical system 10, the imaging device 11, the separation unit 14, the generation unit 15, and the luminance polarization superimposing unit 20 constitute an “imaging device”, and the imaging device 11 and the separation unit 14 constitute a “light receiving unit”. .

  The image processing / image recognition unit 30 may be incorporated in the imaging device to form a part of the imaging device, or may be separately attached to and detached from the imaging device separately from the imaging device.

  The imaging optical system 10 forms an image of the photographing object on the imaging element 11. Although the imaging optical system 10 is depicted as “one lens”, it is of course generally configured as a plurality of lens systems.

  The image sensor 11 will be described with reference to FIGS.

  As shown in FIGS. 2B and 2C, the image sensor 11 includes an image sensor unit 12 and an optical filter 13.

  The image sensor section 12 is a two-dimensional array of pixels on the substrate ST. In the example in the description, the pixel array has a square matrix arrangement.

  The optical filter 13 has a configuration in which a polarizing filter 132 is sandwiched between a transparent substrate 131 and a filling layer 133.

  In FIG. 2 (c), the optical filter 13 and the image pickup device portion 12, which are drawn away from each other, are in close contact with each other.

  In other words, the filling layer 133 is filled as an “adhesive” in the vicinity of the polarizing filter 132 and the image pickup device portion 12 to integrate them.

  The polarizing filter 132 is a so-called “region-dividing polarizing filter” in which a plurality of types of polarizing filter elements having different polarization directions are two-dimensionally arranged along with the pixels of the image sensor 11.

  In FIG. 2B, polarization filter elements POL1, POL2, POL3, and POL4 are superimposed corresponding to the four adjacent pixels PC1, PC2, PC3, and PC4.

  There are two types of “polarizing filter elements” used for the polarizing filter 132: one that transmits P-polarized light and one that transmits S-polarized light.

  One of these two types of polarizing filter elements, POL1 and POL2, has a strip shape whose longitudinal direction is the vertical direction (column direction of the two-dimensional matrix) of the pixel array of the image sensor section 12 in FIG.

  In addition, the other POL3 and POL4 of the polarizing filter elements are in the form of a strip whose longitudinal direction is the horizontal direction (row direction of the two-dimensional matrix) of the pixel array of the image pickup element unit 12 in FIG.

  These two types of “strip-shaped polarizing filter elements” are alternately arranged in the horizontal direction (row direction).

  As shown in FIG. 2B, polarization filter elements POL1 and POL2 that transmit S-polarized light are arranged in the vertical direction in which the pixels PC1 and PC2 are arranged, and correspond to these pixels.

  In the vertical direction in which the pixels PC3 and PC4 are arranged, polarization filter elements POL3 and POL4 that transmit P-polarized light are arranged and correspond to these pixels.

  The image of the object to be imaged formed by the imaging optical system 10 is formed on the polarization filter 132, passes through each polarization filter element, and is received by the imaging element unit 12 as P or S polarized light.

  In this manner, an image of P-polarized light or S-polarized light is captured for each pixel by the image sensor unit 12.

  That is, the polarizing filter 132 has a two-dimensional array of a polarizing filter element POL3 that transmits P-polarized light and a polarizing filter element POL1 that transmits S-polarized light.

Therefore, the imaging region of the imaging device unit 12 is represented by two-dimensional x and y coordinates, and the intensities of S and P polarized light received by the pixel at the position: (x, y) are respectively “I _S (x, y), I _P. (x, y) ".

Using these intensities, the SDOP equation shown above:
SDOP = S ₁ / S ₀ = {I (0, φ) −I (90, φ)} / {I (0, φ) + I (90, φ)}
Can be expressed as:

SDOP = {I _S (x, y) −I _P (x, y)} / {I _S (x, y) + I _P (x, y)} (C)
In this equation (C), the denominator on the right side: I _S (x, y) + I _P (x, y) is referred to as “primary luminance image information”. The primary luminance image information is “S ₀ ”.

Further, the molecule on the right side of the formula (C): I _S (x, y) −I _P (x, y) is referred to as “polarized image information”.

  In addition, luminance image information is generally defined as follows.

Luminance image information = {I _S (x, y) + I _P (x, y)} / 2.

That is, the luminance image information, said primary luminance image information: one half of S _0.

  That is, if luminance image information is given, the primary luminance image information is given by doubling the luminance image information. This is called “luminance image information gives primary luminance image information”.

As described above, “primary luminance image information: S ₀ ” is defined by I _S (x, y) + I _P (x, y).

That is, the primary luminance image information S ₀ is configured as an “x, y image” in which the luminance information described above is distributed two-dimensionally.

Similarly, polarization image information: I _S (x, y) −I _P (x, y) is configured as an “x, y image” in which the polarization information described above is distributed two-dimensionally. .

Polarized image information: I _S (x, y) −I _P (x, y) is determined on the basis of “SDOP” according to the equation (C). Therefore, this is referred to as “polarized image information based on SDOP”. Call.

2B and 2C, light intensity signals from each pixel: I _S (x, y), I _P (x, y) are output.

  As shown in FIG. 2A, the output light intensity signal is input to the separation unit 14 as a signal SG1.

Based on the input light intensity signal SG1 (= I _S (x, y), I _P (x, y)), the separation unit 14 obtains the “polarized image information based on SDOP” and the “primary luminance image information”. Generate and output.

That is, the separation unit 14 separates and outputs the input light intensity signal SG1 into primary luminance image information SG2 (= S ₀ ) and polarization image information SG3 based on SDOP.

That is, the image pickup device 11 shown in FIG. 2 (a) and the separation section 14 receives the image of the shooting target by the imaging optical system 10, a primary luminance image information: S ₀ and a light receiving unit for outputting a polarized image information Configure.

  The primary luminance image information SG <b> 2 output from the separation unit 14 is branched, one is input to the luminance polarization superimposing unit 20, and the other is input to the generation unit 15.

  The polarization image information SG3 based on SDOP output from the separation unit 14 is input to the generation unit 15.

The generation unit 15 is a “polarization image component generation unit”, and obtains a “polarization image component” by dividing the polarization image information SG3 by the secondary luminance image information including the primary luminance image information: S ₀ (SG2).

The generation unit 15, which is a “polarization image component generation unit”, receives the input of the primary luminance image information SG 2 (= S ₀ ) and the polarization image information SG 3, calculates the polarization image component, and outputs it as a signal SG 4.

  Then, the “polarized image component for superimposition” using the “polarized image component” obtained in this way is superimposed on “luminance image information giving primary luminance image information”.

  The “superimposition” will be described later.

In the generation unit 15, a function form: f (X) for calculating the secondary luminance image information in accordance with the primary luminance image information: S ₀ (= SG2) is set in advance.

That is, the function form: f (X) is a function in which the primary luminance image information: S ₀ is a variable: X. When the primary luminance image information: S ₀ is received, X = S ₀ and f (S ₀ ) is calculated.

The f (S ₀ ) calculated in this way is “secondary luminance image information”. The secondary luminance image information is, of course, a “two-variable function with x and y as variables” and corresponds to a two-dimensional image.

The generation unit 15 calculates secondary luminance image information: f (S ₀ ) based on the primary luminance image information: S ₀ input as the signal SG2.

Then, division is performed by dividing the polarized image information: I _S (x, y) + I _P (x, y) input as the signal SG3 by the secondary luminance image information: f (S ₀ ), and the result is set as a signal SG4. Output.

  The image expressed by the signal SG4 output in this way is a “polarized image component”, and of course is a function of pixel coordinates: (x, y).

The function: f (X) is set to be a “predetermined finite value” when the primary luminance image information: S ₀ = 0 or S ₀ ≈0, which is the variable: X.

It is a condition imposed on the function form of the function: f (X) to be “predetermined finite value” at S ₀ = 0 or S ₀ ≈0, and can be arbitrarily set as long as this condition is satisfied.

The predetermined finite value is a value that can effectively suppress the expansion of the noise component of the polarization image component when S ₀ = 0 or S ₀ ≈0, and the polarization image component does not become excessively small when S0 is sufficiently large.

Accordingly, when the secondary luminance image information: f (S ₀ ) is “0 or a primary luminance image information as small as approximately 0: S ₀ ”, the result is excessive even if the division is performed on the polarization image information. There is no.

  That is, it is possible to effectively prevent the noise component included in the polarization image information from expanding.

The function form: f (X) is used to obtain the secondary luminance image information: f (S ₀ ) using the primary luminance image information: S ₀ as the variable: X. In this specification, the function: f (X) (S ₀ ).

Functions satisfying the condition: f (S ₀₎ as "f (S ₀₎ monotonically increasing function of S ₀ at ≠ 0" is one of those suitable.

As a specific example in such a case, “c” is set as a “positive constant”.
f (S ₀ ) = log (S ₀ +1) + c
And so on.

Alternatively, let c "be a" positive constant "
f (S ₀ ) = S ₀ + c
Is also a suitable example.

Polarization image information as a numerator of division: I _S (x, y) −I _P (x, y) is also used as primary luminance image information as a denominator: S ₀ (= I _S (x, y) + I _P (x , Y)) is also a function of pixel coordinates.

Further, in response to the fact that the primary luminance image information: S ₀ is a function of pixel coordinates, the secondary luminance image information: f (S ₀ ) is also a function of pixel coordinates.

Division: {I _S (x, y) + I _P (x, y)} / f (S ₀ ) is performed for each pixel coordinate corresponding to the numerator / denominator, and the calculated polarization image information is also the pixel coordinate. It becomes a function of (x, y).

  A specific example will be described later, and the superposition of the “polarization image component for superimposition” on “luminance image information giving primary luminance image information” will be described here.

First, “luminance image information giving primary luminance image information” is the above-described “{I _S (x, y) + I _P (x, y)} / 2” in the example described above. next luminance image information: twice the _{S 0.}

The “polarization image component for superimposition” to be superimposed on the luminance image information is an image component using the polarization image component and may or may not be the “polarization image component itself”.

  The case where the polarization image component for superimposition is not the polarization image component itself is a case where “additional calculation is performed” on the polarization image component.

  Examples of additional operations will be described later.

  Regarding "superimposition", the following three types are given.

The luminance image information is represented as “I _bright (x, y)”.
The “superimposed polarization image component” to be superimposed is represented as “I _por (x, y)”.

The superimposed image is represented by “I _bright-por (x, y)”.

The first superimposition mode is a mode in which the polarization image component for superimposition: I _POR (x, y) is superimposed on the luminance image information: I _bright (x, y) as a “linear sum” as shown in the following equation. .

I _bright-por (x, y) = I _bright (x, y) + k · I _por (x, y) (D)
The second superimposition mode is a mode in which the polarization image component for superimposition: I _POR (x, y) is superimposed on luminance image information: I _bright (x, y) as “multiplication” as shown in the following equation.

I _bright-por (x, y) = I _bright (x, y) × k · I _por (x, y) (E)
In the above formulas (D) and (E), “k” is a constant representing the degree of superposition, and “can be appropriately set according to specific superposition conditions”.

  The linear sum operation / product operation is performed for each pixel having the same coordinate (x, y).

  The third aspect of superposition is as follows.

I _bright-por (x, y) = k · I _bright (x, y) (F)
k = k ₁ if I _por (x, y) ≧ TH (G)
k = k ₂ if I _por (x, y) <TH (H).

That is, the threshold value TH for the superimposed polarization image component: I _por (x, y) is set, and the magnitude of both is determined for each pixel.

And, for the I _por (x, y) is larger pixels than TH (above (G)), the luminance image information: I _bright (x, y) constant: _{k 1} factor is.

Also, the I _por (x, y) is smaller pixels than TH (above (H)), the luminance image information: I _bright (x, y) constant: _{k 2-fold.}

That is, in the third mode of superimposition, the polarization image component for superimposition: I _por (x, y) is not directly superimposed on the luminance image information: I _bright (x, y).

However, since the polarization image component for superimposition: I _por (x, y) is added to the calculation of the required image: I _bright-por (x, y) by the determinations (G) and (H), this This is also called “superimposition”.

In the case of the third mode of superimposition, as is clear from the above (F), (G), and (H), the polarization image component for superimposition: I _por (x, y) and the threshold: a constant depending on the magnitude of TH : K ₁ and k ₂ are determined.

The “luminance image information” is modulated by constants: k ₁ and k ₂ and output.

Of the three modes of superposition described above, in the first and second modes (D) and (E), the superimposed polarization image component: I _por (x, y) is the luminance of each pixel. It is superimposed as a change.

In the third modes (F), (G), and (H), the image to be obtained: I _bright-por (x, y) is a constant multiple of the luminance image information: I _bright (x, y). is there.

Accordingly, the image obtained in the three modes of superimposition described above: I _bright-por (x, y) can be processed as an “image of luminance information”.

  When the superimposing mode is the first mode, the luminance polarization superimposing unit 20 superimposes the polarization image information and the luminance image information as a linear sum.

  When the superimposing mode is the second mode, the luminance polarization superimposing unit 20 superimposes the polarization image information and the luminance image information as multiplication.

When the superimposing mode is the third mode, the luminance polarization superimposing unit 20 modulates and outputs the luminance image information in accordance with the polarization image information and the threshold value. Embodiment shown in FIG. The separating unit 14, the generating unit 15, and the luminance polarization superimposing unit 20 are actually configured as a CPU or a microcomputer.

  Then, the luminance image information and the polarization image information for superimposition are superimposed by a program calculation according to the algorithms (D) to (H) described above.

After the luminance image information and the polarization image information for superimposition are superimposed in this manner, the output image information IMO (“I _bright-por (x, y)”) shown in FIG. 2B is output.

  This output image information IMO is “luminance polarization superimposed image information”.

  As described above, the output image information IMO can be “normal luminance image processing conventionally known”.

  Thus, the output image information IMO that can be processed by the conventional luminance image processing is input to the image processing / image recognition unit 30 in the example of FIG.

  The image processing / image recognition unit 30 receives the output image information IMO and performs desired image processing and image recognition processing.

  For example, if the imaging apparatus of FIG. 2A is mounted on the vehicle for “vehicle control system”, the image processing / image recognition unit 30 performs processing such as “recognition object extraction processing”.

  The vehicle control means that receives the result performs vehicle control such as vehicle braking and steering operation assistance.

  Since the output image information IMO can be image-processed by “conventional luminance image processing”, there is no need to “newly develop image processing and image recognition processing for polarized images”.

  By using the image processing system of FIG. 2, it is possible to improve the recognition rate and reduce false detection.

  Here, the prevention of the expansion of the noise component will be specifically described by taking as an example the superposition according to the expressions (F) to (H) described above as the “third mode of superposition”.

The imaging optical system 10 captured the “night road condition”. The luminance image information: I _bright (x, y) obtained by this imaging was superimposed by the equation (F).

At this time, “I _por (x, y)”, which is distinguished from the threshold value TH and the magnitude by the expressions (G) and (H), is the primary luminance image information: S ₀ (= I _S (x, y) + I _P (x, y)) was used.

In the formulas (G) and (H), TH = −∞, k ₁ = 5500, and k ₂ = 0. At this time, an image obtained as the left side of the formula (F): I _bright-por (x, y) is shown in FIG.

Since the image shown in FIG. 3A is a “night image”, the primary luminance image information: I _bright (x, y) is close to “almost 0”, and therefore the noise component of the polarization image information is significantly enlarged. Has been.

  An image as shown in FIG. 3A is not suitable for practical use.

Therefore, in the case of the present invention, secondary luminance image information is obtained using “S ₀ + c” as the function: f (S ₀ ), and superposition is performed according to equations (F) to (H).

That is, f (S ₀ ) = {I _S (x, y) + I _P (x, y) + c}.

Specifically, TH = −∞, k ₁ = 5500, k ₂ = 0, and c = 1000.

The image: I _bright-por (x, y) obtained as the left side of the formula (F) in this way is shown in FIG.

In spite of “primary luminance image information at night photography: S ₀ is extremely small”, a good and practical image is obtained.

  The supplementary explanation will be given below.

  The combination of “imaging optical system and image sensor” is not limited to that shown in FIG.

  For example, “a configuration in which a polarizer plate in which polarizers of a plurality of polarization directions are combined is arranged in front of an imaging lens and imaging is performed while rotating” described in Patent Document 1 may be used.

  Alternatively, one of the two imaging optical systems may be combined with a “polarizing filter that transmits S-polarized light” and the other with a “polarizing filter that transmits P-polarized light”.

  That is, “image information having S-polarization information” is captured by one camera, and “image information having P-polarization information” is captured by the other camera, and the image information of each camera is combined into polarization image information.

  As described above, the above-described image processing (superimposition of luminance image information and superimposing polarization image components) can be performed on “polarized images” acquired by various image capturing units (imaging optical system and image sensor).

  The polarizing filter 132 described above can be configured by, for example, a “sub-wavelength structure (SWS)”.

  In the above example, a “monochrome image sensor” is assumed as the imaging element unit 12, but not limited to this, a color image sensor can also be used.

In the example described above, the polarization image information for superimposition is “polarization image information itself”.
The polarization image information is “polarization image information based on differential polarization degree (SDOP)”.

  However, the present invention is not limited to this, and the polarization image information may be “polarization image information based on the degree of polarization (DOP)”.

  Hereinafter, the handling when the polarization image component for superimposition is “different from the polarization image component” will be described.

  In the model described with reference to FIG. 1, the polarization state of the reflected light can be expressed by using the material of the object OB (refractive index, surface state, internal scattering coefficient), the angle between the light source and the object, and the angle between the object and the observation point as variables. .

  That is, the polarization state can be expressed by the following formula (K).

Polarization state = F (light source state, material, angle) (K)
F represents a function.

  In this equation, each variable is as follows.

Angle distribution of light source state = ψ
Material = K _S , k _P , k _d , n ₂ etc.
Angle = θ.

  Expression (K) has a relationship in which the remaining one is determined if any two of the light source state, material, and angle are determined.

  The description according to FIG. 1 is based on the premise that the light source state is unchanged.

  Therefore, the superimposing method described above is effective for superimposing “a polarized image obtained by imaging in a room where the illumination state is unchanged”.

  In the reflection model described above, terms (Rs and Rp terms) representing “specular light due to regular reflection” on the surface of the object OB are important quantities that characterize the polarization state.

  When the imaging optical system 10 and the light source are at the same angle with respect to the object OB that is the object to be imaged, the specular light hardly returns and “polarization information” cannot be obtained.

  When the state of the light source cannot be controlled, for example, when considering application to an in-vehicle application where the light source condition cannot be fixed, it is necessary to consider the following points.

Since the “polarization state” has the relationship represented by the equation (K), the “polarization information held by the polarization image information” changes depending on the light source state in an in-vehicle camera that does not project the irradiation light.
For example, in an actual use situation in the case of an “on-vehicle imaging device”, it is considered that “almost all of the object to be imaged receives direct light illumination in the case of normal light”.

  Further, it is considered that “in the case of backlight, substantially all of the imaged image is subjected to backlight illumination”.

  Considering this point, it can be considered that polarization information such as SDOP and DOP is “normal light and backlight, and the value is biased as an average” when various objects are imaged.

  This bias in polarization information is called “offset”.

  In consideration of such points, the influence of the state of the light source such as forward light and backlight can be substantially canceled by removing the “offset of polarized image information such as SDOP of the entire screen”.

  In this way, it is preferable to perform “superposition in consideration of offset”.

  In the case of the “first mode” in which the superimposition is performed by a linear sum, the superimposition formula in consideration of the offset is as the following formula (M).

I _bright-por (x, y)
= I _bright (x, y) + k [I _por (x, y) − {ΣΣI _por (x, y) / X · Y}] (M)
In formula (M), I _bright (x, y) and I _por (x, y) are the same as those in formula (D) described above.

The part in square brackets on the right side of the formula (M), that is,
[I _por (x, y) − {ΣΣI _por (x, y) / X · Y}]
Gives a "polarized polarization image component".

The second term in square brackets, ie the part in curly brackets,
{ΣΣI _por (x, y) / X · Y}
Is the “offset” of the polarization image component: I _por (x, y).

  This “offset” is an average value in the polarization image component.

  That is, it is an average value of “polarized image components” obtained by dividing “polarized image information by secondary luminance image information” within the “predetermined range determined by X and Y” of the image formed on the image sensor 11.

  The polarization image information is “polarization image information based on SDOP”.

  The “predetermined range” determined by X and Y is not unique and is arbitrary.

  For example, considering that “the influence of the state of the light source is removed using the entire screen”, an image area determined by X and Y can be set to “an area obtained by removing an empty area from the entire screen”.

  The average in the predetermined area calculated in this way is “offset”, and this offset is subtracted from the polarization image information.

In this way, the “polarized image component from which the offset has been removed” becomes a superimposed polarization image component and is superimposed on the luminance image information: I _bright (x, y) by a linear sum.

  At this time, the coefficient k before the square bracket in the second term on the right side of the equation (M) is “multiplied by the amount in the parenthesis (superimposed polarization image component)”.

  The coefficient: k is determined as in the following (M1) and (M2).

k = k ₁ if {ΣΣI _por (x, y) / X · Y} ≧ TH (other than forward light) (M1)
k = k ₂ if {ΣΣI _por (x, y) / X · Y} <TH (forward light) (M2).

Here, k ₁ and k ₂ are “polarization image component superposition coefficients for superposition”.
“TH” is a “threshold value”, and is set to “recognize the follow light state”.

  In the reflection model theory according to the above-described equations (A) and (B), SDOP (see the above-described equation (C)) approaches 0 as “the imaging state is closer to the follow light state”.

  Accordingly, the “offset” is also smaller as the imaging state is closer to the follow light state.

  Therefore, the magnitude relationship between the threshold value TH and the offset is determined, and when the offset is smaller than the threshold value TH, it is determined that the state of the light source (imaging state) is the follow light state.

  Further, when the offset amount is larger than the threshold value TH, it is determined that the state of the light source is “not in the following light state” such as backlight.

Then, according to these determinations, the “polarization image component superposition coefficient for superposition: k” is switched between k ₁ and k ₂ .

In general, k ₁ <k ₂ is set, for example, “TH = −∞, k ₁ = −300, k ₂ = 0”.

Polarization image component for superimposition obtained by removing the offset: {ΣΣI _por (x, y) / X · Y} from the polarization image component: I _por (x, y) as in the expressions (M), (M1), and (M2) Is superimposed on the luminance image information.

  In this way, even when the luminance image information is 0 or ≈0, the noise component of the polarization image information is not enlarged, and an output image from which the influence of forward / backlight is removed can be obtained.

  As another example of the top in consideration of the offset, the following formulas (S), (M1), and (M2) can be given.

I _bright-por (x, y)
= I _bright (x, y) · k [I _por (x, y) / {ΣΣI _por (x, y) / X · Y}] (S)
In the formula (S), I _bright (x, y) and I _por (x, y) are the same as those in the formula (D) described above.

The part in square brackets on the right side of the formula (S), that is,
[I _por (x, y) / {ΣΣ I _por (x, y) / X · Y}]
Gives a "polarized polarization image component".

The denominator part in square brackets,
{ΣΣI _por (x, y) / X · Y}
Is the “offset” of the polarization image component: I _por (x, y) as in the above case.

In this superposition mode, the polarization image component: I _por (x, y) is divided by the offset to obtain a “superimposed polarization image component” from which the offset has been removed.

The “polarization image component superposition coefficient for superposition: k” is switched between k ₁ and k ₂ according to the above-described equations (M1) and (M2) depending on the state other than the normal light and the normal light.

  In this way, even when the luminance image information is 0 or ≈0, it is possible to obtain an output image from which the influence of forward / backlight is removed without enlarging the noise component of the polarization image information.

  As described above, the offset can be removed by subtracting or dividing the offset from the polarization image component.

  The imaging conditions also differ in “light source state” between daytime and nighttime.

That is, it is illuminated by sunlight during daytime, and illuminated by car headlights or street lights at night.
Considering this point, it is preferable that the values of the above-described luminance polarization superposition coefficients k ₁ and k ₂ are “changed between daytime and nighttime”.

The discrimination between daytime and nighttime can be made by detecting the ambient brightness from the amount of exposure when information on on / off of the headlight switch or automatic exposure adjustment is performed.
Furthermore, even if the weather is “under direct sunlight or cloudy”, a better effect can be obtained by switching the values of the luminance polarization superposition coefficients k ₁ and k ₂ .
Since the illuminance of the object to be photographed is high under direct sunlight, when the automatic exposure adjustment is performed, the exposure amount can be detected to determine “under direct sunlight or cloudy”.

  In addition, when there is a “region saturated in the acquired image”, it is preferable that the value of k is set to 0 so that “overlay is not performed” for the region, or a constant constant is output so that calculation is impossible.

That is, by setting the threshold value TH to an appropriate constant, it is possible to exclude “an object other than a detection object using polarization information” from the output image.
In this way, primary luminance image information at night shooting: If S ₀ is close to 0 or 0, it is possible to perform good imaging by suppressing the expansion of a noise component of the polarized image components.

  In the embodiment shown in FIG. 2A, the removal of the offset can be performed, for example, by arranging an “offset removal calculating unit” between the generation unit 15 and the luminance polarization superimposing unit 20.

A signal SG4 that is a polarization image component is input to such an “offset removal calculation unit”,
What is necessary is just to make calculation of Formula (M) and (S) perform.

A function for calculating an offset according to the definition {ΣΣI _por (x, y) / X · Y} is given to the “offset removal calculation unit”.

  Then, equations (M) and (S) are calculated using the calculated offset.

  Further, in order to detect the time of following light and other than the time of following light, as described above, the threshold value TH is set so that “the following light state can be recognized”.

  Then, the magnitude relationship between the threshold value TH and the offset is determined. When the offset is smaller than the threshold value TH, it is determined that the state of the light source (imaging state) is the follow light state.

  Further, when the offset amount is larger than the threshold value TH, it is determined that the state of the light source is “not in the following light state” such as backlight.

Then, according to these judgments, the “polarized image component superposition coefficient for superposition: k” is switched between k ₁ and k ₂ . This switching may be performed by the luminance polarization superimposing unit 20.

  In addition, when switching the luminance polarization superposition coefficient between daytime and nighttime, information on on / off of a headlight switch and an exposure amount adjusted by automatic exposure adjustment are used.

  That is, these may be input to the luminance polarization superimposing unit 20 as a signal for distinguishing between daytime and nighttime, and the luminance polarization superposition coefficient may be switched.

  In addition, when there is a “region saturated in the acquired image”, it is preferable that the value of k is set to 0 so that “overlay is not performed” for the region, or a constant constant is output so that calculation is impossible.

That is, by setting the threshold value TH to an appropriate constant, it is possible to exclude “an object other than a detection object using polarization information” from the output image.
Such an operation can also be incorporated into the luminance polarization superimposing unit 20.

  As described above, the imaging apparatus shown in FIG. 2A offsets the primary polarization image information by subtracting or dividing the primary polarization image information by the average value of the prescribed range on the image of the primary polarization image. It is possible to have an “offset removing unit” that removes.

  Then, the polarization image component generation unit can perform division by the secondary luminance image information on the primary polarization image information from which the offset is removed by the offset removal unit.

Further, the generation unit 15 in the imaging apparatus of FIG. 2A has primary luminance image information: a function of S ₀ : f (S ₀ ) set in advance, and f (S ₀ ) as secondary luminance image information. It is a polarization image component generation unit that performs an operation to divide polarization image information and generates a polarization image component.

f (S ₀ ) is a function including S ₀ as a variable and having a predetermined finite value when S ₀ = 0 or S ₀ ≈0.

  Accordingly, the generation unit 15 constitutes an “image processing apparatus”.

10 Imaging optical system
11 Image sensor
12 Image sensor
13 Optical filter
132 Polarizing filter
15 generator (polarized image component generator)

JP 2011-29903 A

Claims (11)

An imaging optical system that forms an image of the object to be imaged;
Receiving an image of the shooting target according to the imaging optical system, the primary luminance image information: a light receiving section for outputting the S ₀ and the polarization image information,
A polarized image component generation unit that obtains a polarized image component by dividing the polarized image information by the secondary luminance image information including the primary luminance image information: S ₀ ;
A luminance polarization superimposing unit that superimposes the polarization image component for superimposition using the polarization image component on luminance image information giving the primary luminance image information as a change in luminance of each pixel, and outputs luminance polarization superimposed image information; Have
Secondary luminance image information as a divisor of the division in the polarized image component generation unit, the primary luminance image information: function including S _0: A f (S _0), The function: f (S ₀₎ is Primary luminance image information: An imaging apparatus that is set to have a predetermined finite value when S ₀ = 0 or S ₀ ≈0. The imaging device according to claim 1,
Function: f (S ₀₎ is the f (S ₀₎ ≠ _0, an imaging apparatus which is a monotonically increasing function of S _0. The imaging apparatus according to claim 1 or 2,
Function: f (S ₀ ) is f (S ₀ + c), where c (≠ 0) is a constant. The imaging apparatus according to any one of claims 1 to 3,
An imaging apparatus, wherein the polarization image information is polarization image information based on a degree of polarization. The imaging apparatus according to any one of claims 1 to 3,
An imaging apparatus, wherein the polarization image information is polarization image information based on a differential polarization degree. In the imaging device according to any one of claims 1 to 5,
The luminance polarization superimposing unit superimposes the polarization image component for superimposition and the luminance image information as a linear sum. In the imaging device according to any one of claims 1 to 5,
The luminance polarization superimposing unit superimposes the polarization image component for superimposition and the luminance image information as multiplications. In the imaging device according to any one of claims 1 to 7,
The luminance polarization superimposing unit modulates and outputs luminance image information in accordance with the polarization image component for superimposition and the threshold value. In the imaging device according to any one of claims 1 to 8,
The polarization image information output from the light receiving unit is set as primary polarization image information, and the primary polarization image information is subtracted or divided from the primary polarization image information by an average value of a prescribed range on the image of the primary polarization image. Having an offset removal unit for removing the offset;
The polarization image component generation unit performs division by secondary luminance image information on the primary polarization image information from which the offset has been removed by the offset removal unit. In the imaging device according to any one of claims 1 to 9,
An image pickup apparatus, wherein the light receiving portion includes an image pickup device portion on which a region-dividing polarizing filter is disposed. In the imaging device according to any one of claims 1 to 10,
The primary luminance image information obtained from the light receiving unit: S ₀ is included as a variable, and when S ₀ = 0 or S ₀ ≈0, a function that becomes a predetermined finite value: f (S ₀ ) is preset, and the function: An image processing apparatus comprising a polarization image component generation unit that performs a calculation to divide polarization image information using f (S ₀ ) as secondary luminance image information and generates a polarization image component.

Images