JP2017201303A - Image processing method and image processor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To increase the accuracy of iris verification.SOLUTION: The image processing method includes: a luminance value information acquisition step for acquiring an effective radiant luminance value from an object; and an image generation step for removing at least a part of regular reflection optical components in the surface of the object from the effective radiant luminance value and generating the taken image as a collection of unit regions having the obtained luminance value.SELECTED DRAWING: Figure 16

Description

  The following disclosure relates to an image processing method, an image processing apparatus, and an image processing program for performing processing on an image used for iris authentication.

  In recent years, a technique for separating a diffuse reflection component and a specular reflection component included in reflected light from a subject in an image obtained by photographing the subject has been developed. This is because most image processing algorithms are based on diffuse reflection, and specular reflection can be a factor in reducing the performance of the image processing algorithm. An example of a technique for separating the two reflection components is disclosed in Patent Document 1.

Patent Document 1 discloses an image processing method including the following steps (1) to (5).
(1) A plurality of images of a subject are photographed by a camera through polarizing elements having different principal axis directions under arbitrary illumination.
(2) For a pixel group in which specular reflection occurs in a plurality of captured images, an incident surface is specified for each pixel from the normal vector and the line-of-sight vector of the subject.
(3) The incident angle is specified for each pixel from the normal vector and the line-of-sight vector of the subject.
(4) Pixels having similar incident surfaces and incident angles are clustered to form a pixel set.
(5) In the pixel set, the reflection component is separated assuming a stochastic independence between the diffuse reflection component and the specular reflection component.

  According to the image processing method of Patent Literature 1, it is possible to separate the diffuse reflection component and the specular reflection component even in a general illumination environment.

Japanese Patent No. 3955616 (registered on May 11, 2007)

  However, Patent Document 1 does not disclose that iris recognition is performed by reducing the regular reflection light component included in the reflected light from the subject.

  An object of one embodiment of the present disclosure is to realize an image processing method or the like that can reduce specular reflection light components.

  In order to solve the above problem, an image processing method according to an aspect of the present disclosure uses a plurality of pixels associated with a plurality of polarization elements having different principal axis directions as one pixel unit, and the pixel unit includes two pixel units. Luminance value information acquisition step of acquiring from the subject an effective radiance value that is an effective radiance value of the subject in the image sensor, and an effective radiance value acquired from the subject. An image generation step of generating a captured image including the image of the subject by using a plurality of pixel units included in the pixel unit corresponding to at least a part of the subject in the image generation step. A unit area having a luminance value obtained by removing at least part of the specularly reflected light component on the surface of the subject from the effective radiance value of the pixel. Generating the captured image as a set of.

  The image processing method according to an aspect of the present disclosure includes a plurality of pixels associated with a plurality of polarizing elements having different principal axis directions as one pixel unit, and imaging in which the pixel units are two-dimensionally arranged. A step of acquiring an image of a subject photographed by an element, and a luminance distribution of S-polarized light depending on an incident angle to the subject determined by a position on the subject corresponding to a two-dimensional position of the pixel unit in the image sensor Calculating using the output of the image sensor.

  In addition, an image processing apparatus according to an aspect of the present disclosure includes a plurality of pixels associated with a plurality of polarizing elements having different principal axis directions as one pixel unit, and imaging in which the pixel units are two-dimensionally arranged. An image processing apparatus including an element, the luminance value information acquisition unit that acquires from the subject an effective radiance value that is an effective radiance value of the subject in the imaging element by the imaging element, and acquired from the subject An image generation unit that generates a captured image including an image of the subject using an effective radiance value, and the image generation unit includes a pixel unit for each pixel unit corresponding to at least a part of the subject. At least a part of the specularly reflected light component on the surface of the subject is removed from the effective radiance values of a plurality of pixels included, and a collection of unit regions having the obtained brightness values is removed. Generating the captured image as.

  According to the image processing method and the image processing apparatus according to an aspect of the present disclosure, it is possible to realize an image processing method or the like that can reduce the specular reflection light component.

It is a flowchart which shows an example of the flow of a process in the image processing method of this embodiment. It is a figure for demonstrating the outline | summary of an image processing apparatus. It is a figure for demonstrating the outline | summary of the image processing in an image processing apparatus. It is a block diagram which shows the structure of a portable information terminal provided with the image processing apparatus of this embodiment. (A) And (b) is a figure which shows another example of the integrated polarizer with which a camera is provided. It is a figure for demonstrating the value used for the process in an image processing apparatus. It is a graph which shows the light reception intensity | strength of the light received in a pixel with respect to a polarization angle. It is a graph which shows the reflection coefficient of S polarized light and P polarized light with respect to an incident angle. It is a flowchart which shows another example of the flow of a process in the image processing method of this embodiment. (A) And (b) is a flowchart which shows another example of the flow of the process in the image processing method of this embodiment. It is a graph which shows the reflection intensity of S polarized light with respect to the incident angle of the light to a rubber ball. It is a figure which shows the outline of the image which shows the experimental result of the image processing by the image processing apparatus of this embodiment. FIG. 10 is a block diagram illustrating a configuration of an image processing apparatus according to a third embodiment. 10 is a flowchart illustrating a flow of processing in the image processing apparatus according to the third embodiment. FIG. 10 is a block diagram illustrating a configuration of an electronic information device including an image processing device according to a fourth embodiment. 14 is a flowchart illustrating a flow of processing in the image processing apparatus according to the fourth embodiment. It is a graph which shows the example of the luminance value of the pixel in a unit field. FIG. 10 is a block diagram illustrating a configuration of an electronic information device including an image processing device according to a fifth embodiment. 10 is a flowchart illustrating a processing flow in the image processing apparatus according to the fifth embodiment. It is a graph which shows the example of the luminance value of the pixel in a unit field. It is a graph which shows the parameter contained in a formula (5-1). (A) is a figure which shows the example of a lookup table, (b) is a figure which shows the example of the table which a minimum luminance value estimation part produces, (c) is an example of the signal strength of a luminance value. (D) is a graph showing an example of the signal intensity shown in (c) and an example of the signal intensity indicating the luminance value estimated by the minimum luminance value estimation unit. FIG. 10 is a block diagram illustrating a configuration of an electronic information device including an image processing apparatus according to a sixth embodiment. 14 is a flowchart illustrating a processing flow in the image processing apparatus according to the sixth embodiment. It is a graph which shows the example of the effective radiance value of the pixel contained in a pixel unit, the estimated value of the effective radiance value which the pixel contained in the said pixel unit can take, and the estimated value of a diffused light component, (a) FIG. 6 is a graph of a pixel unit in which light having an incident angle to the eyeball of 30 ° is reflected by the eyeball, and FIG. 5B is a graph in which light having an incident angle to the eyeball of 20 ° is reflected by the eyeball. It is a graph of the pixel unit in which reflected light injects, (c) is a graph of the pixel unit in which the reflected light which the light whose incident angle to the eyeball is 10 degrees reflected by the eyeball enters. FIG. 10 is a block diagram illustrating a configuration of an electronic information device including an image processing apparatus according to an eighth embodiment. 19 is a flowchart illustrating a processing flow in the image processing apparatus according to the eighth embodiment. It is a figure which shows arrangement | positioning of a pixel unit, (a) is a figure which shows the state arrange | positioned so that the pixel unit which consists of nine pixels may not mutually overlap, (b) is a pixel unit which consists of four pixels Is a diagram showing a state in which the pixel units are arranged so as not to overlap each other, (c) is a diagram showing a state in which pixel units composed of nine pixels are arranged so as to partially overlap, and (d) It is a figure which shows the state arrange | positioned so that the pixel unit which consists of four pixels may overlap partially. 14 is a flowchart illustrating a flow of processing in an image processing apparatus that executes image processing of Embodiment 6 and other image processing in parallel. It is a figure for demonstrating the case where a to-be-photographed object contains the eyeball of both eyes.

Embodiment 1
Hereinafter, Embodiment 1 of the present invention will be described in detail. The image processing apparatus of this embodiment is an apparatus that performs authentication based on an iris image of a human eyeball.

(Outline of the image processing apparatus 10)
First, an outline of the image processing apparatus 10 will be described. FIG. 2 is a diagram for explaining an overview of the image processing apparatus 10. The image processing apparatus 10 performs image processing on an image obtained by photographing a subject. Specifically, the image processing apparatus 10 performs a process of separating the diffuse reflection component and the specular reflection component included in the reflected light from the subject. The image processing apparatus 10 is mounted on the portable information terminal 1 in this embodiment.

  For example, the portable information terminal 1 is a terminal that can separate the two reflection components from an image of the user's eyeball E (subject) and perform the iris authentication of the user using the separated image. As shown in FIG. 2, the image processing apparatus 10 and the camera 20 described above are provided. Details of the image processing apparatus 10 will be described later. Note that the subject in one embodiment of the present invention is not limited to an eyeball, and may be any subject as long as it is likely to be reflected.

  The camera 20 captures a subject based on a user operation. In the present embodiment, as shown in FIG. 2, the camera 20 captures an image of the eyeball E by photographing the user's eyeball E. The camera 20 is mainly provided with an integrated polarizer 21 and a light receiving element 22 (imaging element) (see FIG. 3). The integrated polarizer 21 and the light receiving element 22 are viewed from the direction in which light enters the camera 20. Are stacked in this order.

  The integrated polarizer 21 is composed of a plurality of polarizing elements, and the principal axis directions of the plurality of polarizing elements are different. In the present embodiment, the integrated polarizer 21 corresponds to one polarizing element per pixel. In this embodiment, as shown in FIG. 2, nine polarizing elements 21 a to 21 i respectively corresponding to nine adjacent pixels are provided. Specifically, the nine polarizing elements 21a to 21i have polarization angles of 0 °, 20 °, 40 °, 60 °, 80 °, 100 °, 120 °, 140 °, and 160 °, respectively, in each pixel. .

  The light receiving element 22 includes a plurality of pixels associated with the nine polarizing elements 21a to 21i as one pixel unit, and the pixel units are two-dimensionally arranged.

  When the user's iris authentication is performed in the portable information terminal 1, the user's eyeball E is photographed by the camera 20. As shown in FIG. 2, when the user's eyeball E is irradiated with external light or room light, the light is reflected on the surface of the eyeball E, and the reflected light Lr enters the camera 20.

  The camera 20 irradiates the user's eyeball E with external light (sunlight) or room light, and obtains reflected light Lr that reflects the external light or room light in the iris, thereby including an image including the user's iris image. The mobile information terminal 1 performs user authentication by analyzing the iris image. On the other hand, when the object O exists in the user's field of view, the object O is reflected in the eyeball E due to the influence of outside light or room light, and a reflected image Ir is formed in the eyeball E as shown in FIG. That is, an external image or room light that is irradiated onto the object O and reflected by the object O is irradiated onto the eyeball E, whereby a reflected image Ir is formed on the eyeball E. Then, the camera 20 acquires an image including the reflected image Ir by acquiring the reflected light Lr obtained by reflecting external light or room light in the reflected image Ir. If the mobile information terminal 1 does not perform the process of removing the reflection image Ir from the acquired image including the iris image and the reflection image Ir, the portable information terminal 1 is affected by the reflection image Ir in the iris image analysis. Might not be able to perform iris authentication.

  In particular, since strong reflection occurs in the user's eyeball E under sunlight, accurate iris authentication outdoors is difficult. By irradiating the user's eyeball E with light having an intensity higher than the intensity of sunlight, the influence of sunlight in iris authentication can be reduced, but such high intensity light is applied to the eyeball E or the skin. When irradiated, the state of the eyeball E or the skin may deteriorate.

  The portable information terminal 1 of the present embodiment includes the integrated polarizer 21 and the image processing device 10, so that the reflection in the image analysis of the iris is performed without irradiating the eyeball E with the above-described high-intensity light. It is possible to reduce the influence of the image Ir and perform accurate iris authentication.

  Next, an outline of processing in the image processing apparatus 10 for reducing the influence will be described with reference to FIG. FIG. 3 is a diagram for explaining an outline of image processing in the image processing apparatus 10. In FIG. 3, the illustration of the integrated polarizer 21 is simplified.

  In the example shown in FIG. 3, the user's eyeball E is photographed for iris authentication under sunlight irradiation. A reflected image Ir of the object O is formed on the user's eyeball E.

  When the user's eyeball E is photographed by the camera 20 in order to perform iris authentication, the reflected light Lr from the user's eyeball E is received by the light receiving element 22 via the integrated polarizer 21.

  Here, in general, most of the intensity of light forming an image used for image processing (here, reflected light Lr indicating an iris used for authentication processing) is composed of a diffuse reflection component. In the present embodiment, the light is processed as indicating surface information indicating the surface of the eyeball E (specifically, the iris) necessary for the authentication process. On the other hand, most of the intensity of light forming an image that becomes noise to be removed in the image processing (here, the reflected light Lr indicating the object O that adversely affects the authentication processing) is composed of a specular reflection component. This specular reflection component includes an S-polarized light source component (S-polarized light component) and a P-polarized light source component (P-polarized light component).

  In FIG. 3, since the eyeball E including the reflected image Ir and the iris is photographed, the reflected light Lr including the three types of reflection components is received by the light receiving element 22 through the integrated polarizer 21. Therefore, it is necessary to remove the S polarization component and the P polarization component from the reflected light Lr received by the light receiving element 22.

  In the present embodiment, a digital conversion process is performed on the reflected light Lr received by the light receiving element 22, and an S-polarized component and a P-polarized component are removed from the reflected light Lr converted into a digital signal (S-wave P). Wave removal). Thereby, as shown in FIG. 3, the reflected image Ir is removed from the image of the eyeball E necessary for the authentication process. The portable information terminal 1 generates data for the eyeball E for authentication by coding the image from which the reflected image Ir has been removed (that is, an image including only the iris image of the eyeball E). Thereby, the portable information terminal 1 can perform accurate iris authentication.

  In order to perform iris authentication with higher accuracy, a known independent component analysis (ICA) process may be performed after the process of removing the S-polarized component and the P-polarized component.

(Configuration of portable information terminal 1)
FIG. 4 is a block diagram illustrating a configuration of the portable information terminal 1 including the image processing apparatus 10 that executes the image processing method of the present embodiment. As shown in FIG. 4, the portable information terminal 1 measures the distance from the image processing device 10, the camera 20, and each point on the surface of the user's eyeball E to the S polarization calculation unit 12 described later. And a distance measuring device 30 for transmission.

  In the present embodiment, the portable information terminal 1 that includes the image processing device 10, the camera 20, and the distance measuring device 30 will be described as an example. However, these members do not need to be provided integrally. That is, the image processing apparatus 10 can acquire an image captured from the camera 20 that is a separate apparatus from the image processing apparatus 10 and is measured from the distance measuring apparatus 30 that is a separate apparatus from the image processing apparatus 10. It is sufficient that the above distance can be acquired.

  The camera 20 of this embodiment uses a CCD (Charge Coupled Device) as a pixel. The camera 20 may use a CMOS (Complementary Metal Oxide Semiconductor) as a pixel.

  FIGS. 5A and 5B are diagrams showing another example of the integrated polarizer included in the camera 20. The camera 20 of this embodiment is provided with the integrated polarizer 21 provided with nine types of polarizing elements 21a-21i as demonstrated in FIG. However, the camera used in one embodiment of the present invention includes an integrated polarizer 21A including four types of polarizing elements 21j, 21k, 21l, and 21m, for example, as shown in FIG. Also good. In addition, the camera used in one embodiment of the present invention may include an integrated polarizer 21B including two types of polarizing elements 21n and 21o, for example, as illustrated in FIG.

(Image processing apparatus 10)
The image processing apparatus 10 includes a black eye detection unit 11, an S polarization calculation unit 12, a P polarization calculation unit 13, a diffused light calculation unit 14, and an authentication unit 15.

  The black eye detection unit 11 acquires an image captured by the camera 20 and identifies an area corresponding to the user's black eye included in the image. The processing in the black-eye detection unit 11 is well known in the field of authentication using, for example, an iris image, and will not be described in this specification.

  The S-polarized light calculation unit 12 calculates the luminance distribution of S-polarized light included in the image. The processing by the S polarization calculation unit 12 will be described later.

  The P-polarized light calculation unit 13 calculates the luminance distribution of P-polarized light included in the image. The P-polarized light calculation unit 13 calculates the luminance distribution of P-polarized light based on the luminance distribution of S-polarized light and Fresnel's law.

  The diffused light calculation unit 14 calculates the luminance distribution of the diffused light included in the image. Specifically, the diffused light calculation unit 14 subtracts the luminance values of S-polarized light and P-polarized light at the pixel from the luminance value of each pixel. By the subtraction process, an image of only the diffused light component from which the S-polarized light and P-polarized light components are removed can be obtained.

  The authentication unit 15 performs user authentication using an iris image included in an image of only diffused light. Since the authentication by the iris in the authentication unit 15 is a known technique, the description is omitted in this specification.

(Processing in the S polarization calculation unit 12)
FIG. 6 is a diagram for explaining values used for processing in the image processing apparatus. In the following description, each numerical value is defined as follows.
R: Distance from the lens of the camera 20 to the center of the eyeball E r: Radius of the eyeball E θ: Incident angle of light to the eyeball E ((i) Position P on the eyeball where light is incident and the center of the eyeball And (ii) an angle formed by a straight line L2 (second imaginary line) connecting the position P and the center of the lens of the camera 20).
Φ: (i) an angle formed by the straight line connecting the center of the lens of the camera 20 and the center of the eyeball E, and (ii) the straight line L1 The S-polarized light calculation unit 12 corresponds to the two-dimensional position of the pixel unit. The luminance distribution of S-polarized light depending on the incident angle θ determined by the position P on the eyeball E is calculated using the luminance value (output) of the pixel of the light receiving element 22. Processing in the S polarization calculation unit 12 will be described below.

(First processing by the S-polarized light calculation unit 12)
First, the S-polarized light calculation unit 12 specifies a pixel in the light receiving element 22 corresponding to a point on the eyeball where the incident angle θ becomes the Brewster angle (hereinafter, referred to as “Brewster point” as necessary). Specifically, the angle φ is calculated based on the following equation (1-1). Expression (1-2) is a modification of Expression (1-1).

  In the expressions (1-1) and (1-2), R is measured by the distance measuring device 30. More precisely, the distance measuring device 30 measures the distance from the lens of the camera 20 to the surface of the eyeball E, that is, the distance obtained by subtracting r from R. In the case of an eyeball in a human body, the value of r is 7.4 mm and is almost constant. Therefore, R can be calculated from the measurement data obtained by the distance measuring device 30. Furthermore, since the refractive index n of the cornea of the eyeball is 1.376, the Brewster angle (= θ) is 53.1 ° in the present embodiment. From these numerical values, the S-polarized light calculation unit 12 can calculate the angle φ using the formula (1-1) or (1-2).

  Here, the pixel corresponding to the surface of the eyeball E existing on the straight line connecting the camera 20 and the center of the eyeball E is the pixel closest to the camera in the region of the eyeball E. From the position of the pixel and the distance, the pixel corresponding to the Brewster point in the image of the eyeball E can be specified. A plurality of pixels corresponding to the Brewster point are specified in the image.

  Next, the S-polarized light calculation unit 12 specifies a pixel unit including each specified pixel included in the light receiving element 22 of the camera 20. The pixel unit is a group of pixels associated with a plurality of polarizing elements. In the present embodiment, the pixel unit is associated with nine polarizing elements having different principal axis directions. The pixel units are two-dimensionally arranged on the light receiving element 22 of the camera 20. In FIG. 6, light incident on the position P is received at a two-dimensional position of the pixel unit in the light receiving element 22 on a straight line L2 connecting the position P on the eyeball and the center of the lens of the camera 20.

  Next, the S-polarized light calculation unit 12 calculates the luminance value of S-polarized light at the Brewster point by subtracting the minimum value from the maximum value of the luminance values of the pixels included in the specified pixel unit. At this time, for example, an average value of differences obtained by subtracting the minimum value from the maximum value of the luminance values of the pixels for all the specified pixel units may be used as the luminance value of the S-polarized light at the Brewster point. Alternatively, only for an arbitrary pixel unit, the minimum value may be subtracted from the maximum value of the luminance value of the pixel, and the calculated difference may be used as the luminance value of S-polarized light at the Brewster point.

  FIG. 7 is a graph showing the received light intensity of light received by the pixel with respect to the polarization angle (angle in the principal axis direction). In the graph shown in FIG. 7, the horizontal axis is the polarization angle, and the vertical axis is the received light intensity. The received light intensity includes a diffuse reflection component (k-term) indicating an iris image necessary for the authentication process and a specular reflection component indicating a reflection image Ir of external light that becomes noise in the authentication process. As described above, the specular reflection component includes the S polarization component (f-term) and the P polarization component (g-term). The S-polarized component forms a substantially cosine function (cos (θ + ψ) (ψ is an arbitrary constant)) or a substantially sine function.

  The diffuse reflection component is constant regardless of the polarization angle. Therefore, in the graph shown in FIG. 7, the fluctuation of the received light intensity with respect to the polarization angle is the fluctuation of the specular reflection component, particularly the S polarization component.

  FIG. 8 is a graph showing reflection coefficients (Fresnel's law) of S-polarized light and P-polarized light with respect to the incident angle θ. In the graph shown in FIG. 8, the horizontal axis represents the incident angle θ of light to the eyeball E (see FIG. 6), and the vertical axis represents the reflection coefficient. The reflection coefficient is a relative value indicating the reflection intensity. Further, the reflection intensity of the S-polarized light and the P-polarized light in the eyeball E corresponds to each received light intensity in the pixel on a one-to-one basis. Also, the following formulas (2-1) and (2-2) are formulas showing the relationship between the incident angle θ of light to the eyeball E and the reflection coefficients of S-polarized light and P-polarized light included in the reflected light. . In the formulas (2-1) and (2-2), n = 1.376 as described above.

  As shown in FIG. 8, the reflection coefficient of P-polarized light is almost 0 when the incident angle θ is near the Brewster angle. That is, in the pixel included in the pixel unit including the pixel corresponding to the Brewster point, the variation in the received light intensity with respect to the polarization angle can be regarded as the variation in S polarization.

  It is known that the intensity of S-polarized light exhibits a sinusoidal change with respect to fluctuations in the polarization angle. Therefore, the reflection intensity (amplitude) of S-polarized light in the pixel unit can be obtained by subtracting the minimum value from the maximum luminance value of the pixels included in the specified pixel unit.

  Thereafter, the S-polarized light calculation unit 12 determines the S-polarization in the pixel units other than the specified pixel unit from the reflection coefficient and the reflection intensity of the S-polarization at the Brewster angle and the reflection coefficient of the S-polarization in other than the pixel unit corresponding to the Brewster point. Calculate the reflection intensity of the polarized light. Specifically, the ratio between the reflection coefficient of S-polarized light and the reflection intensity at the Brewster angle is calculated, and the reflection intensity corresponding to the reflection coefficient at the incident angle θ other than the Brewster angle is calculated using the ratio. That is, the S-polarized light calculation unit 12 calculates the luminance value of the S-polarized light at the incident angle θ other than the Brewster angle, starting from the luminance value of the S-polarized light at the pixel whose incident angle θ is the Brewster angle.

  Note that when no external light is reflected in the region of the eyeball E in the image, the luminance value of each pixel does not include a specular reflection component. In this case, since the luminance value for each pixel in the pixel unit does not include the S-polarized light component, it does not show a sinusoidal change with respect to the polarization angle.

  Therefore, the image processing apparatus according to one aspect of the present invention determines whether the luminance value for each pixel in the pixel unit shows a sinusoidal change with respect to the polarization angle before the processing by the S-polarization calculation unit 12. It may further include a reflection determination unit that determines the above. The manufacturer of the image processing apparatus may arbitrarily set the pixel unit to be determined by the reflection determination unit.

  When the luminance value for each pixel in the pixel unit does not show a sinusoidal change with respect to the polarization angle, it is considered that external light is not reflected on the eyeball E. In this case, the image processing apparatus 10 does not calculate S-polarized light and P-polarized light, and performs authentication in the authentication unit 15 using the minimum luminance value of the pixels included in each pixel unit as the luminance value of the diffuse reflection component. It's okay.

(Second process in the S-polarized light calculation unit 12)
The pixels corresponding to the above Brewster points are often located on the outer periphery of the black eye region in the image. For this reason, the pixel corresponding to the Brewster point may not be included in the black eye region depending on the degree of eye opening of the user or the angle of shooting.

  In such a case, since the first process cannot be performed normally, authentication using the iris fails. Therefore, when the authentication using the iris image generated by the first process described above fails, the S polarization calculation unit 12 includes the pixel corresponding to the Brewster angle in the region of the eyeball E on the light receiving element 22. As a result, the second process described below is performed.

  In the second process, the calculation of the S-polarized luminance value by subtracting the minimum value from the maximum luminance value of the pixels included in the pixel unit is repeated for a plurality of pixel units included in the light receiving element 22. Then, the luminance value of S-polarized light in the pixel unit is calculated by subtracting the minimum value from the maximum luminance value of the pixels included in each identified pixel unit. Thereafter, fitting is performed with respect to the relationship between the incident angle θ corresponding to each pixel unit and the luminance value of the S-polarized light using Equation (2-1), and the value of n is calculated.

  In the image processing apparatus according to the present embodiment, the S-polarized light calculation unit 12 may perform either the first process or the second process described above first. Further, only one of the processes may be performed.

(Processing in the P polarization calculation unit 13)
As shown in FIG. 8 and equations (2-1) and (2-2), the intensity ratio between S-polarized light and P-polarized light is determined according to the incident angle θ according to Fresnel's law. For this reason, when the S-polarized light calculating unit 12 performs the first processing described above, the P-polarized light calculating unit 13 determines the luminance of the P-polarized light based on the luminance distribution of S-polarized light and Fresnel's law as described above. Distribution can be calculated.

  In addition, when the S-polarized light calculation unit 12 performs the second process described above, the value of n is calculated. Therefore, the P-polarized light calculation unit 13 can calculate the intensity of the P-polarized light by the above-described equation (2-2).

(First processing by the image processing apparatus 10)
FIG. 1 is a flowchart illustrating an example of a processing flow in the image processing method of the present embodiment.

  First, the camera 20 captures an image including the user's eyeball E, and the distance measuring device 30 measures the distance from the camera 20 to the eyeball E (S1). The image processing apparatus 10 acquires data indicating the image and the distance. First, the black eye detection unit 11 performs a process of detecting a black eye region on the photographed image (S2).

  Next, the S-polarized light calculation unit 12 calculates the luminance value of the S-polarized light at the point where the incident angle θ of the light reflected toward the camera 20 is the Brewster angle in the detected black eye region (SA1). . Further, the S-polarized light calculating unit 12 is based on the brightness value of S-polarized light calculated in step SA1 and Fresnel's law, in that the incident angle θ of the light reflected toward the camera 20 is an angle other than the Brewster angle. The luminance value (luminance distribution) of S-polarized light is calculated (SA2).

  Thereafter, the P-polarized light calculation unit 13 calculates a P-polarized light luminance value (luminance distribution) based on the S-polarized light luminance distribution calculated in steps SA1 and SA2 and Fresnel's law (SA3). Further, the diffused light calculation unit 14 calculates the luminance distribution of the diffused light based on the luminance distribution of S-polarized light and the luminance distribution of P-polarized light (SA4).

  The authentication unit 15 performs authentication using the image showing the luminance distribution of the diffused light obtained in step SA4 (sometimes referred to as an image of diffused light) (SA5). Thereafter, the authentication unit 15 determines whether or not the user can be authenticated by the image of the diffused light (S3). If the authentication is successful (S3, Y), the image processing apparatus 10 ends the image processing.

  On the other hand, if the authentication cannot be performed (S3, N), the image processing apparatus 10 tries to authenticate the user again by a process different from steps SA1 to SA5 in order to improve the accuracy of the authentication. Specifically, the S-polarized light calculation unit 12 first calculates an S-polarized luminance value for each pixel unit in the detected black eye area (SB1). Next, the P polarization calculation unit 13 calculates a P polarization luminance value based on the S polarization luminance value calculated in step SB1 (SB2). Then, the diffused light calculation unit 14 calculates a diffused light luminance value based on the S-polarized luminance value and the P-polarized luminance value (SB3). Using the diffused light image obtained in step SB3, the authentication unit 15 performs authentication (SB4). If the authentication fails in step SB4, the authentication unit 15 notifies the user that the authentication has failed via, for example, a display unit (not shown) included in the portable information terminal 1.

  Of the processes in the flowchart shown in FIG. 1, the processes in steps SA <b> 1 to SA <b> 4 correspond to the first process in the S-polarized light calculation unit 12 described above. In addition, the processing in steps SB1 to SB3 corresponds to the second processing in the S-polarized light calculation unit 12 described above.

(Second processing by the image processing apparatus 10)
FIG. 9 is a flowchart showing another example of the processing flow in the image processing method of the present embodiment. According to the process shown in FIG. 9, it can be expected that user authentication is performed at a higher speed than the process shown in FIG.

  In the process shown in FIG. 9, first, steps S1 and S2 are executed as in the example shown in FIG. Next, the image processing apparatus 10 performs authentication in steps SB1 to SB4 prior to step SA1. Then, it is determined whether the authentication part 15 has authenticated the user (S3). If the authentication cannot be performed (S3, N), the image processing apparatus 10 performs authentication in steps SA1 to SA5.

  The authentication in steps SB1 to SB4 can be performed in a shorter time than the authentication in steps SA1 to SA5. Therefore, according to the process shown in FIG. 9, when the user can be authenticated by steps SB1 to SB4, the user can be authenticated at a higher speed than the process shown in FIG.

(Third processing by the image processing apparatus 10)
FIG. 10A is a flowchart showing still another example of the processing flow in the image processing method of the present embodiment. According to the process shown in FIG. 10A, the user authentication can be performed at a higher speed than the process shown in FIG. 10 instead of the decrease in reliability compared to the process shown in FIGS. .

  In the process shown in FIG. 10A, as in the process shown in FIG. 9, the image processing apparatus 10 performs authentication in steps SB1 to SB4 after steps S1 and S2. Thereafter, the authentication unit 15 does not perform the determination in step S3.

  According to the process shown in FIG. 10 (a), even if authentication is successful if authentication is performed in steps SA1 to SA5, authentication itself fails if authentication in steps SB1 to SB4 fails. , The reliability of authentication decreases. On the other hand, the user can quickly know the success or failure of authentication using the captured image.

(Fourth processing by the image processing apparatus 10)
FIG. 10B is a flowchart showing still another example of the processing flow in the image processing method of the present embodiment. The process illustrated in FIG. 10B is different from the process illustrated in FIG. 10A in that step SB2 is omitted. That is, according to the process shown in FIG. 10B, the image processing apparatus 10 removes only S-polarized light and does not remove P-polarized light. Therefore,
According to the process shown in FIG. 10B, in step SB4, the authentication unit 15 performs authentication using an image from which the P-polarized component of the reflected image Ir has not been removed. For this reason, compared with the process shown in FIG. 10A, the processing speed can be further increased instead of further reducing the reliability of authentication.

(Experimental example 1)
An experiment using a rubber ball as to why the image processing apparatus 10 can remove an image (reflection image) that is reflected in the image by the second processing by the S-polarization calculation unit 12 described above. An example will be described below. The camera 20 used in the experiment is a camera using a CCD sensor, and the number of pixels is 1900 × 900 (about 1.3 million pixels). The integrated polarizer included in the camera 20 has four types of polarizing elements having different polarization angles. In the following description, the polarization angles of the four types of polarizing elements are 0 °, 45 °, 90 °, and 135 °, respectively.

The manufacturing method of a polarizing element is as follows. First, an AlCu film having a thickness of 40 nm is formed on a photodiode constituting a CCD sensor via an SiO ₂ interlayer film, and a slit (a band-like region having no AlCu) is formed by dry etching at a pitch of 300 nm. . Thereafter, SiO _{2 with} a film thickness of 50 nm and AlCu with a film thickness of 40 nm are sequentially formed, and slits are formed in new AlCu so as to alternate with previously formed AlCu.

  The diameter of the rubber ball is 10 cm, and a letter “G” is written on the surface as a pattern image. In the photograph that is the basis of each figure, the reflection occurred at the position overlapping the pattern image. The distance between the rubber ball and the lens of the camera 20 is 30 cm to 50 cm.

  In an image formed by polarized light having a polarization angle of 45 °, the pattern image was relatively clear. On the other hand, the pattern image was unclear in the image formed by the polarized light having the polarization angles of 0 ° and 90 °. In an image formed by polarized light having a polarization angle of 135 °, the pattern image was further unclear. That is, in this integrated polarizer, in the light transmitted through the four polarizing elements, the luminance of the light transmitted through the polarizing element having the polarization angle of 45 ° is the highest, and the luminance of the light transmitted through the polarizing element having the polarization angle of 135 ° is the highest. Low. Therefore, the result of calculating the difference in luminance between these two lights is used for the following actual measurement values.

  FIG. 11 is a graph showing the reflection coefficient of S-polarized light with respect to the incident angle of light on the rubber ball. In FIG. 11, the horizontal axis represents the ratio of the distance from the pixel where light from the center of the rubber ball is incident to the pixel to the radius of the rubber ball in each pixel where light from the rubber ball is incident. It is a value corresponding to the incident angle of light. The vertical axis represents the reflection coefficient of S-polarized light.

  In FIG. 11, each data point is an actual measurement value. On the other hand, the solid line is a graph showing the theoretical value of the reflection coefficient at the refractive index (n = 2.2) of the rubber ball using the equation (2-1). The actual measurement value by the S-polarized light calculation unit 12 matches the theoretical value. Furthermore, the luminance distribution of P-polarized light with respect to the incident angle θ can be obtained from the above-described equation (2-2) and the value of n calculated by fitting.

  As described above, in this experiment, the S-polarized component and the P-polarized component included in the image can be calculated based on the images taken through the four polarizing elements having different principal axis directions. By subtracting the S-polarized component from the original image, an image in which the pattern image is clearer than the original image can be obtained. Further, by subtracting both the S-polarized light component and the P-polarized light component from the original image, an image with a clearer pattern image can be obtained.

  Note that when the user actually takes an image of the eye with the portable information terminal 1, the eyeball E may be deviated from the center of the image. When the eyeball E is deviated from the center of the image, a simple relationship as shown in equations (1-1) and (1-2) is not established.

  In such a case, the above-described deviation can be determined by specifying the position of the pixel whose incident angle is the Brewster angle.

  Specifically, the dependence of the luminance of the pixel in each pixel unit on the polarization angle is calculated. As described above, in the pixel unit corresponding to the Brewster angle, the P-polarized component is 0, so that the ratio of the S-polarized component in the luminance of the pixel unit is increased. As a result, the luminance of the pixels included in the pixel unit varies greatly depending on the polarization angle.

  Therefore, a pixel unit having a particularly large dependency on the polarization angle can be regarded as a pixel corresponding to the Brewster angle. Then, the displacement of the position of the camera 20 can be determined from the position of the pixel corresponding to the Brewster angle.

(Experimental example 2)
FIG. 12 is a diagram showing an outline of an image showing an experimental result of image processing by the image processing apparatus of this embodiment. In this experiment, an integrated polarizer including polarizing elements having seven different polarization angles was used. In this experiment, image processing was performed in steps SB1 to SB3 in the flowchart described above.

  Prior to image processing, the image used in this experiment included a reflected image Ir of external light on the black eye region, as indicated by the arrow in FIG. On the other hand, after the image processing, as shown after the arrow in FIG. 12, the reflected image Ir of the external light on the black eye region has been removed.

(Effect of the image processing apparatus 10)
Prior to the present invention, the inventor conducted an authentication experiment using a conventional iris authentication system under various conditions. In the experiment, shooting was performed with a smartphone camera. There were two types of smartphone positions: “front” and “bottom” relative to the eyeball. In addition, when the smartphone position is “front”, the opening degree of the eyes is three types: “large”, “medium”, and “small”, and when the smartphone position is “down”, “medium” One type was used.

  There were two types of shooting environments: “indoor” and “outdoor”. “Indoor” was further divided into “no window”, “with window (shade)”, “with window (direct sunlight)” and “dark room”. Further, “outdoor” was further classified into four types, “sunny sky (reverse light)”, “sunny sky (forward light)” and “sunny sky (side light)”. Note that “reverse light” means that sunlight enters the subject from the side opposite to the camera. Further, “forward light” means that sunlight is incident on the same side as the camera. Further, “side light” means that sunlight enters the subject from the side of the camera.

  The results of performing the authentication experiment 10 times under each condition will be described below. When the position of the smartphone is “front”, if the degree of eye opening is “medium” or more, there was no authentication failure except “with window (direct sunlight)” in “indoor”. On the other hand, with “with window (direct sunlight)”, authentication was hardly successful. In addition, when the degree of opening of the eyes was “small”, the authentication was hardly successful in any environment.

  On the other hand, when the position of the smartphone is “down”, there was no authentication failure in “indoor” except for “with window (direct sunlight)”. In addition, there were few failures in “with windows (direct sunlight)”.

  In this way, there was almost no authentication failure indoors unless the eye opening degree was small unless the place was exposed to direct sunlight from a window.

  However, in “outdoors”, if the smartphone was in the “front” position, the authentication was not successful at all. When the smartphone is in the “down” position, authentication was successful with “clear sky (side beam)”, but the number of successful authentications with “sunny sky (reverse beam)” and “clear sky (forward beam)” Was less than half.

  As described above, there has been a problem that authentication by the conventional iris authentication system is difficult to succeed outdoors. As described above, this is caused by reflection of external light or the like on the eyeball.

  Japanese Patent Application Laid-Open No. 2004-228688 discusses separating the reflection component by obtaining the luminance distribution of the specular reflection component, and thus performing iris authentication by removing an image of another object reflected in the eyeball as the subject. There is no disclosure. There is no disclosure about performing processing for iris authentication other than processing for removing the image of the object.

  According to the image processing apparatus of this embodiment, the S-polarized light component resulting from the reflected image Ir included in the image of the eyeball is obtained by obtaining the luminance distribution of the S-polarized light component depending on the incident angle θ by the S-polarized light calculating unit 12. Can be calculated and removed. Further, according to the image processing apparatus of the present embodiment, it is possible to calculate and remove the P-polarized light component resulting from the reflected image Ir based on the S-polarized light component.

  That is, according to the image processing apparatus of the present embodiment, it is possible to remove the reflection of external light on the eyeball. By performing iris authentication using an image from which the reflection of external light is removed, high-accuracy authentication can be performed regardless of the environment.

  Further, in the image processing method of Patent Document 1, it is necessary to perform the above-described processes (1) to (4) in order to separate the diffuse reflection component and the specular reflection component in the above-described process (5). Therefore, in the image processing method of Patent Document 1, an algorithm for separating the two reflection components is complicated, and as a result, the calculation speed of the image processing may be reduced. In the image processing apparatus 10 of the present embodiment, as described above, the two reflection components can be separated without performing the processes (1) to (4). Therefore, the calculation speed of image processing can be improved as compared with the image processing method of Patent Document 1.

[Embodiment 2]
Embodiment 2 of the present invention will be described below with reference to FIGS. 4 and 5. For convenience of explanation, members having the same functions as those described in the embodiment are given the same reference numerals, and descriptions thereof are omitted.

  The camera 20 of the first embodiment has nine polarizing elements 21 a to 21 i as the integrated polarizer 21. On the other hand, as shown to (a) of FIG. 5, the camera 20 of this embodiment is provided with the integrated polarizer 21A which has the four polarizing elements 21j-21m respectively corresponding to four adjacent pixels. The four polarizing elements 21j, 21k, 21l and 21m have polarization angles of 135 °, 90 °, 45 ° and 0 ° in each pixel. As described above, in this embodiment, the integrated polarizer 21 </ b> A is used instead of the integrated polarizer 21. Also in this case, as in the first embodiment, the S-polarized luminance value can be calculated in steps SA1 and SB1 shown in FIGS. That is, as in the first embodiment, the specular reflection component can be removed from the reflected light Lr.

  However, the integrated polarizer 21A of the present embodiment has four polarizing elements 21j to 21m and the number thereof is smaller than that of the first embodiment.

  Generally, in the characteristics of the polarization angle of the integrated polarizer and the received light intensity of the reflected light Lr received by the pixel, the received light intensity forms a substantially sine function or a substantially cosine function. In the first embodiment, the received light intensity of the reflected light Lr in the nine polarizing elements 21a to 21i is acquired in one pixel unit. Therefore, as shown in FIG. 7, the waveform indicated by the received light intensity in the above characteristics can be fitted to the above approximate sine function or approximate cosine function. Therefore, for example, the maximum value (maximum luminance value) and the minimum value (minimum luminance value) of the received light intensity in one pixel unit used in step SA1 shown in FIG. 1 are the maximum luminance value and the minimum luminance value in the pixel unit. It is highly possible that they are almost identical.

  On the other hand, in the present embodiment, the interpolation is performed to fit the approximate sine function or the approximate cosine function. However, the accuracy of the fitting is lower than in the first embodiment, and there is a possibility that the accuracy is far from the maximum brightness value and the minimum brightness value. There is.

  However, in the present embodiment, one pixel unit corresponding to the integrated polarizer 21A is composed of four pixels. That is, the number of integrated polarizers 21A is larger than that in the first embodiment. Therefore, for example, the pixel corresponding to the Brewster angle can be obtained by being subdivided as compared with the first embodiment. That is, the pixel can be specified with higher accuracy.

  As described above, in the present embodiment, compared with the first embodiment, although the fitting accuracy may be reduced, the pixel corresponding to the Brewster angle can be specified with high accuracy. Therefore, it is only necessary to determine which of the integrated polarizer 21 of the first embodiment and the integrated polarizer 21A of the present embodiment is used by verifying the accuracy when performing iris authentication.

  By performing the above verification, instead of the integrated polarizer 21A, the integrated polarizer 21B shown in FIG. 5B (a polarizing element 21n having a polarization angle of 90 ° and a polarizing element having a polarization angle of 0 °) 21o and 2) may be used. Further, the polarization angle and the number of polarizing elements corresponding to one pixel unit can be appropriately changed according to the verification.

[Embodiment 3]
The third embodiment of the present invention will be described below with reference to FIGS. 13 and 14. In authentication using an iris image, a case where authentication is performed by impersonating the user himself by using a counterfeit of the eyeball E can be considered. Therefore, it is necessary to consider preventing this impersonation.

  FIG. 13 is a block diagram illustrating a configuration of a portable information terminal 1A including the image processing apparatus 10A according to the present embodiment. As illustrated in FIG. 13, the image processing apparatus 10 </ b> A is different from the image processing apparatus 10 in that the authentication unit 15 includes an impersonation determination unit 15 a.

  In general, the refractive index of the cornea of a living eye is n = 1.376. On the other hand, in the case of a counterfeit eye, the refractive index is different from that of the living eye because the components constituting the eye of the living body and the material thereof are different. The impersonation determination unit 15a uses this difference to determine whether the eye is a living body or a forgery.

  Specifically, the impersonation determination unit 15a refers to the value of n calculated by the S polarization calculation unit 12 in step SB1. If the value of n is equal to or close to 1.376 (that is, a value within a predetermined range, for example, within ± 5%), the impersonation determination unit 15a determines that the captured image is a living eyeball E. judge. On the other hand, if the calculated value of n is a value that deviates from 1.376 (that is, a value that is outside a predetermined range, for example, a value that exceeds ± 5%), the impersonation determination unit 15a applies an eyeball to the captured image. It is determined that it is a forgery of E.

  FIG. 14 is a flowchart showing the flow of processing in the image processing apparatus 10A. As shown in FIG. 14, the processing in the image processing apparatus 10A differs from the processing shown in FIG. 1 only in that step SB5 is executed between step SB3 and step SB4.

  In step SB5, the impersonation determining unit 15a determines whether the value of n calculated by the S-polarized light calculating unit 12 in step SB1 is within a predetermined range. When it is within the predetermined range (SB5, Y), the impersonation determination unit 15a determines that the captured image is the eyeball E of the living body, and performs the process of SB4. On the other hand, when it is not within the predetermined range (SB5, N), the impersonation determination unit 15a determines that the captured image is a counterfeit of the eyeball E, and ends the authentication without performing the process of SB4.

  As described above, according to the image processing method according to the present embodiment, the luminance distribution of the S-polarized light component depending on the incident angle θ is obtained by the S-polarized light calculation unit 12, and the luminance distribution of the S-polarized light is determined by the subject's eyeball. It can be used to determine whether it is a counterfeit or not.

  The spoofing determination unit 15a may regard the angle at which the luminance value is minimum as the Brewster angle for the luminance distribution of the P-polarized light component calculated by the P-polarized light calculation unit 13 in step SB2. Then, whether the angle is a value close to 53.1 ° or not may be determined as the eyeball E of the living body or a forgery of the eyeball E. That is, the impersonation determination unit 15a may perform the determination based on whether or not the angle is within a predetermined range including the Brewster angle estimated to be the eyeball E of the living body.

  Also in the processes of FIGS. 9 and 10, the above determination can be made by executing step SB5 between steps SB3 and SB4.

(Modification)
Another method for preventing impersonation due to counterfeits is described below.

  If a Brewster's angle is set to 53.1 ° for a counterfeit whose refractive index is different from that of the living eyeball E, the reflection of external light or the like on the counterfeit is not properly removed. . Therefore, when there is a reflection of external light or the like, there is a high possibility that the iris pattern changes due to the reflection and authentication fails without calculating the refractive index.

  However, when authentication is performed in an environment where no reflection occurs, such as in a dark room, the iris pattern does not change, and authentication may be successful.

  Therefore, as a method for preventing impersonation by a forgery, the image processing apparatus may be provided with a light source for intentionally generating reflection in the eyeball E. The generated reflection (image of the light source) is appropriately removed by the image processing apparatus in the case of the eyeball E of the living body, and thus does not affect the authentication. On the other hand, if it is a counterfeit of the eyeball E, the image processing apparatus cannot remove the reflection, and authentication fails. Note that the intensity of light emitted from such a light source only needs to be reflected. Therefore, for example, it is not necessary to have a high strength that adversely affects the human body.

[Embodiment 4]
Embodiment 4 of the present invention will be described below with reference to FIGS. 15 to 17.

  FIG. 15 is a block diagram illustrating a configuration of the electronic information device 2 including the image processing apparatus 10B according to the present embodiment. As illustrated in FIG. 15, the electronic information device 2 includes an image processing device 10 </ b> B, a camera 20, and a storage unit 90. In addition, the image processing apparatus 10B includes a luminance value information acquisition unit 16, a minimum luminance value selection unit 17 (image generation unit), a black eye detection unit 11, and an authentication unit 15.

  The brightness value information acquisition unit 16 acquires an effective radiance value, which is an effective radiance value of the subject in the image sensor, from the subject by the image sensor provided in the camera 20. Specifically, the luminance value information acquisition unit 16 acquires the intensity of the light when the light reflected by the subject is received in each of the pixels of the imaging element included in the camera 20 as an effective radiance value in each pixel. To do. In this embodiment and the following embodiments, the “subject” may be either the eyeball E of one eye or the eyeball E of both eyes, and may include an object around the eyeball E.

  The minimum luminance value selection unit 17 generates a captured image including an image of a black eye using an effective radiance value acquired from the black eye of the subject. Specifically, for each pixel unit corresponding to the subject's black eye, the minimum luminance value selection unit 17 performs regular reflection light on the surface of the subject's black eye from the effective radiance values of a plurality of pixels included in the pixel unit. At least a part of the components is removed, and a captured image is generated as a set of unit regions having the obtained luminance value. The same applies to the diffuse light calculation unit 14 described above, and the minimum luminance value estimation unit 17A and the diffuse reflection light component calculation unit 18 described later.

  More specifically, the minimum luminance value selection unit 17 of the present embodiment uses the minimum value among the effective radiance values of a plurality of pixels included in the pixel unit as the luminance value in the unit region corresponding to the pixel unit. decide. The pixel unit corresponding to the black eye of the subject is specified in advance by the black eye detection unit 11. The unit area is an area in the image of the subject corresponding to each pixel unit included in the camera 20.

  In the present embodiment and later-described embodiments, a component of reflected light that forms a reflected image is referred to as a regular reflected light component. The specularly reflected light component includes a component that depends on the principal axis direction of the polarizing element provided corresponding to the pixel. It can be said that the minimum value of the effective radiance value in the pixel unit is an effective radiance value in which the component depending on the principal axis direction of the polarizing element is most removed in the pixel unit. Accordingly, the minimum luminance value selection unit 17 determines the minimum value of the effective radiance value in the pixel unit as the luminance value of the unit region, thereby removing at least a part of the specularly reflected light component on the surface of the cornea of the subject. Then, a captured image as a set of unit areas having the obtained luminance value is generated.

  The storage unit 90 is a storage device that stores information necessary for processing executed by the image processing apparatus 10B. The electronic information device 2 may not be provided with the storage unit 90 and may be configured to be able to communicate with a storage device provided outside the electronic information device 2.

  FIG. 16 is a flowchart showing processing in the image processing apparatus 10B. In the image processing apparatus 10B, first, the luminance value information acquisition unit 16 acquires the effective radiance value of the subject including the black eye using the camera 20 (SC1, luminance value information acquisition step). Next, the black eye detection unit 11 detects a black eye region from the effective radiance value (SC11). The minimum luminance value selection unit 17 determines the black eye area detected by the black eye detection unit 11 as a range for executing image processing (S12). The minimum luminance value selection unit 17 determines the minimum value among the effective radiance values of a plurality of pixels included in the pixel unit in the black-eye region as the luminance value of the unit region corresponding to the pixel unit, whereby the cornea At least a part of the specularly reflected light component on the surface of the image is removed (SC2, image generation step). Further, the minimum luminance value selection unit 17 generates a captured image as a set of unit areas having the luminance value determined in step SC2 (SC3, image generation step).

  The black eye detection unit 11 performs a process of detecting a black eye region on the captured image generated by the minimum luminance value selection unit 17 (SC4). The black eye detection unit 11 may use the detection result of the black eye region in step SC11 as it is as the detection result in step SC4. The authentication unit 15 authenticates the person using the detected black eye area (SC5). Thereafter, the authentication unit 15 determines whether or not the user has been authenticated (SC6). If the authentication is successful (Y in SC6), the image processing apparatus 10B ends the process. If the authentication fails (N in SC6), the black eye detection unit 11 detects the black eye region again.

  When N is determined in step SC6, there is a possibility that the black eye detection unit 11 cannot correctly detect the user's black eye region. Therefore, in step SC4 after having become N in step SC6, it is preferable that the black eye detection unit 11 makes the detected black eye region different from the black eye region detected in the previous step SC4. As a specific example of the method of making the black eye region different, enlarging or reducing the diameter of the black eye region may be mentioned. As described above, it is preferable to apply feedback to the detection process of the black eye region by the black eye detection unit 11 so that the hamming distance between the code of the registered user's eyeball and the code of the captured eyeball E is minimized.

  FIG. 17 is a graph showing an example of the effective radiance value of the pixel in the pixel unit. In the graph shown in FIG. 17, the horizontal axis represents the angle of the main axis of the polarizing element corresponding to each pixel included in the pixel unit of the camera 20, and the vertical axis represents the effective radiance value of the pixel in the pixel unit. In the example shown in FIG. 17, the pixel unit includes nine pixels, and each pixel corresponds to nine polarizing elements whose principal axis directions are different from each other by 20 ° starting from 10 °.

  In the example shown in FIG. 17, the effective radiance value of the pixel in the pixel unit has a minimum value in a pixel in which the main axis direction of the corresponding polarizing element is 90 °, and the pixel in which the main axis direction of the corresponding polarizing element is 170 °. Take the maximum value at. In this case, the minimum luminance value selection unit 17 selects the effective radiance value of the pixel corresponding to the polarizing element whose principal axis direction is 90 ° as the luminance value of the unit region corresponding to the pixel unit.

  In the image processing apparatus 10B, the minimum value of the effective radiance values of the pixels included in each pixel unit is determined as the luminance value of the unit area corresponding to the pixel unit. As a result, it is possible to easily determine the luminance value of the unit region and generate a captured image without requiring complicated calculation processing.

  Note that the image processing apparatus 10B may omit steps SC11 and SC12 among the steps of the flowchart described above. In this case, the minimum luminance value selection unit 17 may determine the luminance value of the corresponding unit area for each of all the pixel units. Further, steps SC11 and SC12 may also be omitted in the image processing apparatus 10C of the fifth embodiment described later. However, steps SC11 and SC12 cannot be omitted in image processing apparatuses 10D and 10E according to Embodiments 6 to 8 described later. Further, from the viewpoint of processing speed, it is preferable to determine the luminance value of the unit area corresponding to the pixel unit only for the pixel unit corresponding to the black eye area. Therefore, the image processing apparatuses 10B and 10C also perform steps SC11 and SC12. Is preferably performed.

  Further, the image processing apparatuses 10B and 10C according to the fourth and fifth embodiments specify not a black eye but another region, for example, a region wider than the black eye by a predetermined range, or a region corresponding to the entire eyeball E before step SC2. Processing may be performed. In this case, the minimum luminance value selection unit 17 or the minimum luminance value estimation unit 17A (see Embodiment 5) may perform image processing for the specified area. In other words, the image processing apparatuses 10B and 10C can determine the luminance value in the unit region corresponding to the pixel unit based on the effective radiance value of the plurality of pixels included in the pixel unit corresponding to at least the black eye (at least part) of the subject. Can be determined.

[Embodiment 5]
The fifth embodiment of the present invention will be described below with reference to FIGS.

  FIG. 18 is a block diagram illustrating a configuration of the electronic information device 2 including the image processing apparatus 10C according to the present embodiment. As shown in FIG. 18, the image processing apparatus 10 </ b> C is different from the image processing apparatus 10 </ b> B in that a minimum luminance value estimation unit 17 </ b> A (image generation unit) is provided instead of the minimum luminance value selection unit 17. It matches the processing device 10B.

  Based on the effective radiance values of a plurality of pixels included in each pixel unit, the minimum luminance value estimator 17A estimates the minimum effective radiance value that can be taken in the pixel unit. An example of a specific method for estimating the minimum value will be described later.

  Further, the minimum luminance value estimation unit 17A determines the luminance value of the unit region from the estimated minimum value of the effective radiance value. Here, the minimum luminance value estimation unit 17A may use the estimated minimum value as the luminance value of the unit region, or may use a value calculated by performing a predetermined calculation on the estimated luminance value as the luminance value of the unit region. For example, the minimum luminance value estimation unit 17A may use a value obtained by adding, subtracting, multiplying, or dividing a predetermined value to the minimum value of the estimated effective radiance value as the luminance value of the unit region.

  FIG. 19 is a flowchart showing processing in the image processing apparatus 10C. The processing in the image processing apparatus 10C is the same as the processing in the image processing apparatus 10B, except that step SC7 (image generation step) is executed instead of step SC2. In step SC7, the minimum luminance value estimation unit 17A estimates the minimum effective radiance value that can be taken for each of the pixel units corresponding to the eyeball E.

  FIG. 20 is a graph showing an example of the effective radiance value of the pixel in the pixel unit. In the graph shown in FIG. 20, the horizontal axis indicates the angle of the main axis of the polarizing element corresponding to the pixels included in the pixel unit, and the vertical axis indicates the effective radiance value. In the example illustrated in FIG. 20, the pixel unit includes four pixels corresponding to four polarizing elements whose main axis directions are different from each other by 45 ° starting from 10 °.

  In the example illustrated in FIG. 20, the effective radiance value acquired by the pixels included in the pixel unit is the smallest in the pixel in which the main axis direction of the corresponding polarizing element is 100 °. However, when the relationship between the principal axis direction and the effective radiance value is a function indicated by a broken line in FIG. 20, for example, the pixel included in the pixel unit is smaller depending on the principal axis direction of the polarizing element corresponding to the pixel. It can be said that an effective radiance value can be taken. In this case, when the minimum value of the actual effective radiance value of the pixel in the pixel unit is determined as the luminance value of the unit area, a specularly reflected light component depending on the main axis direction of the polarizing element remains in the luminance value. It becomes. 10C of image processing apparatuses estimate the minimum value of the effective radiance value which a pixel unit can take by the minimum luminance value estimation part 17A, Compared with the captured image which image processing apparatus 10B produces | generates, a regular reflection light component A small number of captured images can be generated.

  In the example shown below, the minimum luminance value estimation unit 17A is a triangle that uses the angle in the principal axis direction of the polarizing element corresponding to the pixel as a variable with respect to the effective radiance value of the plurality of pixels included in each pixel unit. The minimum value is estimated by applying a function. Therefore, the minimum luminance value estimation unit 17A can estimate the minimum luminance value that the unit area can take from the minimum value of the trigonometric function.

(First estimation method)
An example of a method of estimating the minimum value of the luminance value in the unit area by the minimum luminance value estimating unit 17A will be described below. When the effective radiance value of each pixel included in a certain pixel unit is Y, it is assumed that Y can be expressed by the following equation (5-1).
Y = Asin (2x + B) + C (5-1)
Here, A and B are constants specific to the pixel unit. x is the angle of the principal axis of the polarizing element corresponding to the pixel. Further, when the principal axis directions of the polarizing elements corresponding to the pixels included in the pixel unit are equiangular intervals, C matches the average E (Y) of the effective radiance values of the pixels included in the pixel unit.

If the root mean square of Y is E (Y ² ), E (Y ² ) is represented by the following formula (5-2).
E (Y ² ) = A ² × E (sin ² (2x + B)) + 2A × E (C) × E (sin (2x + B)) + (E (C ² )) (5-2)
Here, from the Wallis formula, the following equations (5-3) and (5-4) hold.
E (sin ² (2x + B)) = 1/2 (5-3)
E (sin (2x + B)) = 1 / π (5-4)
Therefore, the equation (5-2) can be transformed into the following equation (5-5).
^{E (Y 2) = A 2} /2 + 2A × E (C) / π + (E (C 2)) (5-5)
Since E (Y ² ) is equal to the mean square of the effective radiance values of each pixel included in the pixel unit, E (C) is the minimum effective radiance value of each pixel, and E (C ² ) is each pixel. The value of A can be calculated by approximating the square value of the minimum value of the effective radiance value and solving Equation (5-5) for A.

From the value of A calculated by solving Expression (5-5) and Expression (5-1), the minimum value Ymin that Y can take can be obtained by the following Expression (5-6).
Ymin = E (Y) -A (5-6)
Therefore, the minimum luminance value estimation unit 17A can estimate the minimum value possessed by the effective radiance values of the pixels included in the pixel unit only by the calculations of the above equations (5-5) and (5-6). it can.

  FIG. 21 is a graph showing parameters included in the equation (5-1). In the graph shown in FIG. 21, the horizontal axis represents the angle of the main axis of the polarizing element in the pixel unit of the camera 20, and the vertical axis represents the effective radiance value of the pixel in the pixel unit. In the example shown in FIG. 21, the pixel unit includes four pixels corresponding to four polarizing elements whose main axis directions are different from each other by 45 ° starting at 10 °.

  As shown in FIG. 21, the value C in Expression (5-1) is the median value of the effective radiance values of the pixels of the pixel unit, and is substantially equal to the average value of the effective radiance values of the pixels in the pixel unit. . Further, the value A in the equation (5-1) is the difference between the above average value and the minimum value of luminance that can be taken by the pixels in the unit area. Further, the value B in the equation (5-1) is a value that determines the position of the graph in the horizontal axis direction. That is, when the value of B varies, the entire graph shifts parallel to the horizontal axis. Since the value of B does not affect Ymin, the value of B is not calculated in the first estimation method.

(Second estimation method)
A second estimation method of the minimum luminance value estimation method in the unit area by the minimum luminance value estimation unit 17A will be described below. Also in the second estimation method, it is assumed that the effective radiance value Y of each pixel included in a certain pixel unit can be expressed by the above equation (5-1).

In the second estimation method, a look-up table indicating a correspondence relationship between the value of x and the value of sin2x is stored in the storage unit 90 in advance. The minimum luminance value estimation unit 17A sets the initial value of A in the above equation (5-1) to the average of the following (i) and (ii), and refers to the lookup table to set the value of x. A corresponding Y table is created.
(I) The difference between the maximum and average effective radiance values of each pixel included in the pixel unit (ii) The difference between the average and minimum effective radiance values of each pixel included in the pixel unit was created. The correspondence relationship between x and Y in the table can be shifted by changing the value of B in equation (5-1).

  Next, the minimum luminance value estimation unit 17A compares the created table with the effective radiance value of each pixel included in the pixel unit. The minimum luminance value estimation unit 17A changes the value of B so that the sum of the differences between the effective radiance value of each pixel included in the pixel unit and the table value at the angle of the polarizing element corresponding to each pixel is the minimum. To be. When the difference between the table value and the effective radiance value is the minimum, the minimum luminance value estimation unit 17A is the ratio of the difference between the table value and the effective radiance value of each pixel with respect to the effective radiance value of each pixel. If the average of is less than or equal to a predetermined ratio, it is estimated that the value of A is appropriate. In this case, the minimum luminance value estimation unit 17A estimates the minimum value by the above equation (5-6). The predetermined ratio is preferably 10%, more preferably 5%, and further preferably 1%.

  If the difference between the table value and the effective radiance value is not less than or equal to the predetermined ratio, the minimum luminance value estimation unit 17A changes the value of A in the equation (5-1) from the above-described initial value and performs the same operation. Make a comparison.

  FIG. 22A is a diagram illustrating an example of a lookup table. FIG. 22B is a diagram illustrating an example of a table created by the minimum luminance value estimation unit 17A. (C) of FIG. 22 is a figure which shows the example of the signal strength which shows an effective radiance value. 22D is a graph showing an example of the signal intensity shown in FIG. 22C and an example of the signal intensity indicating the effective radiance value estimated by the minimum luminance value estimation unit 17A.

  In the graph shown in FIG. 22D, the horizontal axis indicates the angle of the main axis of the polarizing element in the pixel unit of the camera 20, and the vertical axis indicates the signal intensity indicating the effective radiance value of the pixel in the pixel unit. In the example shown in (d) of FIG. 22, the pixel unit includes four pixels corresponding to four polarizing elements whose main axis directions are different by 45 ° starting from 0 °. In the example shown in FIG. 22D, light is incident on the interface between air (refractive index n1 = 1.0) and cornea (refractive index n2 = 1.376) at an incident angle of 15 °. Is.

  The storage unit 90 stores in advance a lookup table as shown in FIG. The minimum luminance value estimation unit 17A sets the value of A in Equation (5-1) as described above, and creates a Y table as shown in FIG. Further, the minimum luminance value estimation unit 17A extracts a value closest to each of the effective radiance values of the pixels included in the pixel unit, as shown in FIG. The extracted value and the principal axis direction of the polarizer corresponding to the value are indicated by “*” in FIG. On the other hand, the actual effective radiance value of each pixel included in the pixel unit and the principal axis direction of the polarizer corresponding to each pixel are indicated by “◯” in FIG.

  In FIG. 22 (d), the positions of “*” in the vicinity of 0 ° and 90 ° are in good agreement with the positions of “◯”, while the positions of “*” in the vicinity of 65 ° and 155 ° are , Greatly deviated from the positions of “◯” at 45 ° and 135 °. For this reason, said ratio becomes larger than 10%. In this case, the minimum luminance value estimation unit 17A changes the value of A in Equation (5-1) to create a table of Y again, and determines the values closest to each of the actual effective radiance values from the table of Y. Extract and make similar comparisons.

(Other estimation methods)
The minimum luminance value estimation unit 17A may estimate the minimum effective radiance value that can be taken by the pixels in the pixel unit by an estimation method other than the first and second estimation methods described above. For example, the minimum luminance value estimation unit 17A applies a polynomial in which the angle in the principal axis direction of the polarizing element corresponding to the pixel is a variable with respect to the effective radiance value of the plurality of pixels included in each pixel unit. The minimum value may be estimated. In this case, an example of a polynomial is a polynomial obtained by Taylor expansion of a trigonometric function. According to Taylor expansion, sinx and cosx can be expressed as in the following equations (5-7) and (5-8), respectively.

In formulas (5-7) and (5-8), x is the principal axis direction of the polarizing element, and n is an arbitrary integer. n is preferably 3 or more.

  Further, for example, the minimum luminance value estimation unit 17A may estimate a function indicating an effective radiance value of each pixel included in the pixel unit by an arbitrary interpolation method, and estimate the minimum value using the function. Examples of interpolation methods include spline interpolation, polynomial interpolation, trigonometric function interpolation, cubic interpolation, and the like.

[Embodiment 6]
Embodiment 6 of the present invention will be described below with reference to FIGS.

  FIG. 23 is a block diagram illustrating a configuration of the electronic information device 2 including the image processing apparatus 10D according to the present embodiment. As shown in FIG. 23, the image processing apparatus 10 </ b> D is different from the image processing apparatus 10 </ b> B in that it includes a diffuse reflection light component calculation unit 18 (image generation unit) instead of the minimum luminance value selection unit 17.

The diffuse reflection light component calculation unit 18 specifies the maximum value Ioutmax and the minimum value Ioutmin of the effective radiance values in each pixel unit. Further, the diffuse reflection light component calculation unit 18 calculates the diffuse reflection light component Ik in each of the pixel units by the following equation (6-1).
Ik = (Ioutmin−a × Ioutmax) / (1-a) (6-1)
Here, Ioutmin and Ioutmax are the minimum value and the maximum value of the effective radiance values of the pixels included in the pixel unit, respectively. Ioutmin and Ioutmax may be values estimated in the same manner as in the fifth embodiment, or may be actual effective radiance values of pixels.

  Further, a in Expression (6-1) is the ratio of the reflectance of the P-polarized component to the reflectance of the S-polarized component. As described above, a depends on the angle of incidence of light on the eyeball E. From the distance R from the lens of the camera 20 to the center of the eyeball E and the radius r of the eyeball E, the incident angle at each point on the image of the eyeball E can be calculated. In the present embodiment, the distance R when the eyeball E is in focus is stored in the storage unit 90 in advance, and the diffuse reflection light component calculation unit 18 calculates the incident angle and a using the distance R.

  The method for obtaining the distance R is specifically as follows. When the camera 20 is a fixed focus lens camera, the user's face is designed to be in focus when the user looks at the screen with the electronic information device 2 in hand. In this case, the distance R between the lens of the camera 20 and the eyeball E when the user images an area including his / her eyes so that the image of the eyeball E is included in a predetermined range of the image captured by the camera 20. Is substantially equal to the focal length of the lens of the camera 20. In addition, when the camera 20 has an autofocus function, two types of distances between lenses at which the eyeball E forms an image can be obtained, and the distance R can be calculated using the Bessel method. Further, when the camera 20 includes two image sensors and can capture a plurality of images simultaneously from different positions, the distance R can be calculated by triangulation from the images captured by the respective image sensors.

  Here, when the incident angle is small, the difference in reflectance between the S-polarized component and the P-polarized component is as small as several tens of percent. For this reason, the value of a cannot be calculated correctly due to the influence of variation between pixels or shots, and the calculation result in equation (6-1) becomes Ik ≦ 0 (diffuse reflected light component disappears). is there.

  In this case, the value of a may be changed to the value of a corresponding to an angle larger than the actual incident angle, and the calculation according to equation (6-1) may be executed. For example, what is necessary is just to perform calculation by Formula (6-1) again in order about the value of a corresponding to an angle larger by a predetermined angle (for example, 10 degrees) than an actual incident angle. In this case, however, the accuracy of removing the reflected image is lowered. Such recalculation by changing the value of a may be executed, for example, when authentication by the authentication unit 15 fails. In such recalculation, the value of a may be changed to a value corresponding to a radius or distance smaller than the actual radius r or distance R instead of changing to a value corresponding to an angle larger than the actual incident angle. Good.

  Further, for the distance R, the incident angle and a may be calculated using values assumed in advance. For example, if the electronic information device 2 is a smartphone, it is assumed that the user performs authentication while holding the electronic information device 2 in his / her hand. In that case, the distance R is expected to be about 20 cm to 30 cm. Therefore, the incident angle can be calculated using the expected distance R without actually measuring the distance R with a distance measuring device or the like.

  In this case, a plurality of types of distances R may be prepared. That is, the incident angle and a may be calculated for each of a plurality of types of R, and the diffuse reflected light component Ik may be calculated to perform authentication. In the above example, the image processing apparatus 10D may execute authentication for each of the case where the distance R is 20 cm and 30 cm, and further execute authentication for the case where the distance R is 25 cm. Also good.

Formula (6-1) will be described below. The effective radiance value of the pixel included in the pixel unit includes a diffuse reflected light component and a regular reflected light component. In the present embodiment, the regular reflection light component includes a first regular reflection light component and a second regular reflection light component. In the present embodiment, both the first regular reflection light component and the second regular reflection light component have components that depend on the principal axis direction of the polarizing element. In this case, the first regular reflection light component Is and the second regular reflection light component Ip are expressed by the following equations (6-2) and (6-3), respectively.
Is = Is0 + Is0 × cos2x (6-2)
Ip = Ip0 + Ip0 × cos2 (x−π / 2) (6-3)
Here, Is0 and Ip0 are components of the first specular reflection light component and the second specular reflection light component that do not depend on the main axis direction of the polarizing element, respectively. Further, since the phase of the second regular reflection light component is delayed by 90 ° from the phase of the first regular reflection light component, π / 2 is subtracted from x in Expression (6-3). In equations (6-2) and (6-3), the cos function may be replaced with a sin function.

The effective radiance value Iout of the pixel included in the pixel unit is expressed by the following equation (6-4).
Iout = Ik + Is + Ip = Ik + Is0 + Ip0 + (Is0−Ip0) cos2x (6-4)
Iout becomes maximum when cos2x = 1 and becomes minimum when cos2x = −1. Therefore, Ioutmax and Ioutmin in formula (6-1) are represented by the following formulas (6-5) and (6-6), respectively.
Ioutmax = Ik + 2 × Is0 (6-5)
Ioutmin = Ik + 2 × Ip0 (6-6)
Further, from Fresnel's law, the relationship between Is0 and Ip0 is expressed by the following equation (6-7).
Ip0 = a × Is0 (6-7)
Therefore, the following formula (6-8) is obtained by substituting the formula (6-7) into the formula (6-6).
Ioutmin = Ik + 2 × a × Is0 (6-8)
Equation (6-1) is obtained by solving simultaneous equations consisting of Equations (6-5) and (6-8) for Ik.

  FIG. 24 is a flowchart showing processing in the image processing apparatus 10D. The processing in the image processing device 10D is the same as the processing in the image processing device 10B, except that step SC8 (image generation step) is executed instead of step SC2. In step SC8, the diffuse reflection light component calculation unit 18 calculates the diffuse reflection light component Ik for each of the pixel units using Equation (6-1), and determines the calculated value as the luminance value of the unit region corresponding to the pixel unit. To do.

  Further, the above-described recalculation by changing the value of a may be executed as follows, for example. That is, the hamming distance between the code of the eyeball image from which the reflection image is removed and the registered code is greater than a threshold used for authentication (hereinafter referred to as an authentication threshold) (hereinafter referred to as a branch threshold). ) Is provided. In step SC6 of FIG. 24, if the hamming distance is equal to or greater than the branch threshold, the diffuse reflection light component calculation unit 18 executes the process of step SC8 again. At this time, the above-described recalculation is performed by changing the value of a. On the other hand, if the hamming distance is greater than or equal to the authentication threshold and less than the branch threshold in step SC6, the black eye detection unit 11 executes the process of step SC4 again. If the hamming distance is less than the authentication threshold, the authentication is successful.

  As described above, the diffuse reflection light component calculation unit 18 of the present embodiment, based on the effective radiance values of a plurality of pixels included in each pixel unit, the maximum value and the minimum value of the effective radiance value in the pixel unit. Is identified. The diffuse reflection light component calculation unit 18 uses the specified maximum value and minimum value and the ratio a of the reflectance of the P-polarized component to the reflectance of the S-polarized component on the surface of the eyeball E corresponding to the pixel unit. Then, the diffuse reflection light component Ik in the pixel unit is calculated. Further, the diffuse reflection light component calculation unit 18 determines the calculated diffuse reflection light component as the luminance value of the unit area corresponding to the pixel unit. Thereby, the image processing apparatus 10D can determine the diffuse reflection light component from which at least a part of the reflection light component is removed as the luminance value of each unit region.

  FIG. 25 is a graph illustrating an example of an effective radiance value of a pixel included in a pixel unit, an estimated value of an effective radiance value that can be taken by a pixel included in the pixel unit, and an estimated value of a diffused light component. FIG. 25A is a graph of a pixel unit in which reflected light obtained by reflecting light having an incident angle of 30 ° to the eyeball E with the eyeball E is incident. FIG. 25B is a graph of a pixel unit in which reflected light obtained by reflecting light having an incident angle of 20 ° to the eyeball E with the eyeball E is incident. (C) of FIG. 25 is a graph of a pixel unit in which reflected light obtained by reflecting light having an incident angle to the eyeball E of 10 ° reflected by the eyeball E enters. In the graphs shown in FIGS. 25A to 25C, the horizontal axis represents the principal axis direction of the polarizing element corresponding to each pixel included in the pixel unit, and the vertical axis represents the effective radiance value of the pixel. In the example shown in FIG. 25, the refractive index of air is 1.0, and the refractive index of the cornea of the eyeball E is 1.376.

  First, consider a pixel unit corresponding to reflected light of light having an incident angle of 30 ° to the eyeball E shown in FIG. At this time, when the effective radiance value was normalized with Ioutmax = 1, Ioutmin = 0.60. In addition, in the reflected light of light having an incident angle of 30 °, a = 0.40.

In this case, the diffuse reflected light component Ik is calculated as follows using Expression (6-1).
Ik = (Ioutmin−a × Ioutmax) / (1−a)
= (0.60-0.40 × 1.0) / (1.0-0.40)
≒ 0.33
Next, consider the pixel unit corresponding to the reflected light of light having an incident angle of 20 ° to the eyeball E shown in FIG. At this time, when the effective radiance value was normalized with Ioutmax = 1, Ioutmin = 0.79 was obtained. In addition, in the reflected light of the light having an incident angle of 20 °, a = 0.688.

In this case, the diffuse reflected light component Ik is calculated as follows using Expression (6-1).
Ik = (Ioutmin−a × Ioutmax) / (1−a)
= (0.79-0.688 * 1.0) / (1.0-0.688)
≒ 0.33
Next, consider a pixel unit corresponding to reflected light of light having an incident angle to the eyeball E of 10 ° shown in FIG. At this time, when the effective radiance value was normalized with Ioutmax = 1, Ioutmin = 0.94. Further, in the reflected light of light having an incident angle of 20 °, a = 0.91.

In this case, the diffuse reflected light component Ik is calculated as follows using Expression (6-1).
Ik = (Ioutmin−a × Ioutmax) / (1−a)
= (0.94-0.91 × 1.0) / (1.0-0.91)
≒ 0.33
Therefore, in the example shown in FIG. 25, the diffuse reflected light component Ik is normalized when Ioutmax = 1, regardless of whether the incident angle of light on the eyeball E is 10 °, 20 °, or 30 °. 0.33. Since the value of Ik is substantially the same at any angle, the value of Ik is considered to be correct.

[Embodiment 7]
Embodiment 7 of the present invention will be described below. The image processing apparatus according to the present embodiment is the same as the configuration of the image processing apparatus 10D except for the content of the processing performed by the diffuse reflection light component calculation unit 18, and thus the block diagram and the flowchart are not shown.

The diffuse reflection light component calculation unit 18 of the present embodiment calculates the diffuse reflection light component Ik in each of the pixel units by the following equation (7-1).
Ik = Ioutmin−Ip (7-1)
Equation (7-1) will be described below.

In the present embodiment, the first regular reflection light component Is has a component that depends on the principal axis direction of the polarizing element, while the second regular reflection light component Ip does not depend on the principal axis direction of the polarization element. Specifically, in the present embodiment, the diffuse reflection light component calculation unit 18 calculates Is by the following equation (7-2).
Is = Ioutmax−Ioutmin (7-2)
Ik is represented by the following formula (7-3).
Ik = Ioutmax− (Is + Ip) (7-3)
The following formula (7-4) is obtained by modifying the formula (7-3) using the formula (7-2).
Ik = Ioutmin−Ip (7-4)
Ip is a value that can be calculated using Equation (6-7). Therefore, the diffuse reflection light component calculation unit 18 can calculate the diffuse reflection light component Ik using the equations (7-4) and (6-7).

  As described above, the diffuse reflection light component calculation unit 18 of the present embodiment calculates two types of specular reflection light components, and uses the calculated specular reflection light components as the maximum value of the effective radiance value of the pixels included in the pixel unit. The value subtracted from is determined as the luminance value of the unit area corresponding to the pixel unit. Even when the diffuse reflection light component calculation unit 18 executes such processing, the image processing apparatus 10D uses the diffuse reflection light component from which at least a part of the regular reflection light component is removed as the luminance value of each unit region. Can be determined as

[Embodiment 8]
The following describes Embodiment 8 of the present invention with reference to FIGS. 26 and 27. FIG.

  FIG. 26 is a block diagram illustrating a configuration of the electronic information device 3 including the image processing apparatus 10E according to the present embodiment. As shown in FIG. 26, the electronic information device 3 is different from the electronic information device 2 of the seventh embodiment in that the electronic information device 3 includes an image processing device 10E instead of the image processing device 10D, and a polarization irradiation unit 40.

  The polarized light irradiation unit 40 irradiates the subject with polarized light. For example, the polarized light irradiation unit 40 may include a light source such as an LED and a polarization filter that transmits only polarized light in a specific direction.

  The image processing apparatus 10E is different from the image processing apparatus 10D in that a reflection presence / absence determination unit 19 is provided. The reflection presence / absence determination unit 19 determines whether or not the regular reflection light component on the surface of the subject is equal to or less than a predetermined intensity. In other words, the reflection presence / absence determination unit 19 determines whether reflection has occurred in the subject.

  When the reflection presence / absence determination unit 19 determines that no reflection has occurred in the subject, the polarization irradiation unit 40 irradiates the subject with polarized light. At this time, the luminance value information acquisition unit 16 (luminance value information reacquisition unit) acquires the effective radiance value of the subject again in a state where the polarized light irradiation unit 40 irradiates the subject with polarized light.

  In the present embodiment, the reflection presence / absence determination unit 19 calculates the ratio of the minimum value to the maximum effective radiance value for each pixel unit corresponding to the eyeball E. Furthermore, the reflection presence / absence determination unit 19 calculates the average of the above ratios for the entire pixel unit corresponding to the eyeball E, and determines that no reflection has occurred if the average is equal to or greater than a predetermined value. If it is less than the value of, it is determined that reflection has occurred.

  When the incident angle of light on the subject is small, the ratio of the minimum value to the maximum value of the effective radiance value of the pixel included in the pixel unit corresponding to the incident angle increases. The predetermined value may be a ratio of a minimum value to a maximum value of an effective radiance value at a minimum angle assumed as an incident angle to the subject. Specifically, the predetermined value may be 0.94, for example. This value is the ratio of the minimum value to the maximum value of the effective radiance value calculated using Fresnel's law for the reflected light having an incident angle to the eyeball E of 10 °.

  FIG. 27 is a flowchart showing processing in the image processing apparatus 10E. As shown in FIG. 27, in the image processing apparatus 10E, after the luminance value information acquisition unit 16 acquires the effective radiance value of the subject (SC1), the reflection presence / absence determination unit 19 determines whether reflection has occurred. (SD1, reflection determination step). When the reflection has occurred (Y in SD1), the image processing apparatus 10E performs the processes after step SC2 in the same manner as the image processing apparatus 10B.

  On the other hand, when no reflection occurs (N in SD1), the luminance value information acquisition unit 16 acquires the effective radiance value of the subject again with the polarized light irradiation unit 40 irradiating the subject with polarized light. (SD2, luminance value information reacquisition step). In this case, the image processing apparatus 10E performs the processes in and after step SC2 using the effective radiance value of the subject acquired in step SD2.

  In the image processing apparatus 10E, when there is no reflection on the eyeball E, the luminance value information acquisition unit 16 has the effective radiance of the subject including the eyeball E in a state in which the reflection is generated by the polarized light irradiation unit 40. Get the value. The diffuse reflection light component calculation unit 18 generates a captured image by removing the regular reflection light component from the effective radiance value acquired by the luminance value information acquisition unit 16 using the ratio a calculated by Fresnel's law, Authenticate. Therefore, when the eyeball E is a counterfeit made of a material different from the real eyeball, specifically a material having a refractive index different from that of the cornea (n = 1.376), Prone to failure.

  The electronic information device 3 does not necessarily need to include the polarized light irradiation unit 40. For example, another device that is communicably connected to the electronic information device 3 by wireless or wired communication may include the polarized light irradiation unit 40. In the above-described processing, the luminance value information acquisition unit 16 acquires the effective radiance value of the subject in both steps SC1 and SD2. However, the functional block (luminance value information reacquisition unit) that acquires the effective radiance value of the subject in step SD2 may be different from the luminance value information acquisition unit 16.

  Further, as described above, the image processing apparatus 10E is different from the image processing apparatus 10D in that it includes a reflection presence / absence determination unit 19. In other words, the image processing apparatus 10E has a configuration in which a reflection presence / absence determination unit 19 is added to the image processing apparatus 10D. However, the image processing apparatus according to the present embodiment may have a configuration in which the reflection presence / absence determination unit 19 is added to another image processing apparatus that uses Fresnel's law, for example, the image processing apparatuses 10 and 10A.

[Embodiment 9]
The image processing apparatus according to an aspect of the present invention may execute two or more processes among the processes described above.

  For example, the image processing apparatus according to an aspect of the present invention executes one of the image processes described in the first, second, and fourth to seventh embodiments, and if the authentication fails, Authentication may be performed again by executing image processing other than the executed image processing. If the authentication fails even after re-authentication, the image processing apparatus may acquire the effective radiance distribution of the subject again, or take a picture of the subject again, and measure the distance to the subject again if necessary. . In addition, the image processing apparatus executes another image processing in parallel or in a forward manner, acquires the effective radiance distribution of the subject a plurality of times, or captures an image of the subject, and if the authentication is not successful, You may authenticate with the captured image produced | generated by the other image processing.

  Further, in each of the above-described embodiments, when authentication fails, the black eye range detected by the black eye detection unit 11 is changed and authentication is performed again. However, the image processing apparatus according to one aspect of the present invention is configured such that the luminance value information acquisition unit 16 acquires (captures) the effective radiance value of the subject again when authentication fails once or more. Also good. In addition, the image processing apparatus executes other image processing in parallel or in a forward manner, and if the authentication is not successful even if the effective radiance distribution of the subject is acquired a plurality of times, the imaging generated by the other image processing is performed. Authentication may be performed using an image.

  An image processing apparatus according to an aspect of the present invention includes image processing according to Embodiment 6 and image processing according to another embodiment (that is, any one or more of Embodiments 1, 2, 4, 5, and 7). May be executed in parallel.

  FIG. 29 is a flowchart illustrating a flow of processing in an image processing apparatus that executes image processing according to the sixth embodiment and other image processing in parallel. In the process shown in FIG. 29, the processes in steps SC1, SC5, and SC6 are the same as those described in the above-described embodiment, and thus will not be described in detail here.

  In the process shown in FIG. 29, the image processing apparatus acquires the luminance value information of the subject (SC1), and then the first post-removal image whose luminance value of each unit region has been determined by the image processing of the sixth embodiment. The generation (SD1) and the generation (SD2) of the second post-removal image in which the luminance value of each unit region has been determined by the image processing of another embodiment are performed in parallel. Thereafter, the image processing apparatus determines whether the luminance value is greater than or equal to a predetermined threshold for each unit region of the first post-removal image (SD3). When the luminance value is equal to or greater than the threshold value (Y in SD3), the luminance value of the unit area is not changed. On the other hand, when the luminance value is not equal to or greater than the threshold value (N in SD3), the luminance value of the unit area is changed to the luminance value of the unit area at the corresponding position in the second removed image. The image processing apparatus executes the determination in step SD3 for all the unit areas, executes the process in step SD4 according to the determination result, and then performs authentication using the obtained image (SC5). Further, the image processing apparatus determines whether or not the authentication is successful (SC6). If the authentication is successful (Y in SC6), the process ends. If not (N in SC6), the process returns to step SD1. Returning, the first post-removal image is generated again. Specifically, in the case of N in Step SC6, the image processing apparatus executes again the processing after Step SC11 shown in FIG. In the case of N in step SC6, the process returns to only step SD1 and does not generate the first post-removal image again, but returns to both steps SD1 and SD2, and returns to the first post-removal image and the second post-removal image. Both images may be generated again.

  As described above, in the image processing of the sixth embodiment, there is a possibility that the diffuse reflected light component disappears in the unit region where the incident angle of light to the eyeball is small. Therefore, after determining the luminance value of each unit region by the image processing of the sixth embodiment, the image processing apparatus determines whether the luminance value of each unit region is equal to or greater than a predetermined threshold value. For a unit region whose luminance value is equal to or greater than the threshold value, the luminance value is employed as it is. On the other hand, for a unit region whose luminance value is not equal to or greater than the threshold value, the luminance value obtained by the image processing of the other embodiment is adopted as the luminance value of the unit region. Thereby, this image processing apparatus can prevent the disappearance of the diffuse reflected light component.

[Additional Notes]
Usually, the term “specular reflection light” is used only for visible light emitted in the direction of a viewpoint (for example, a user's eyes or an image sensor). However, in this specification, the term “specular reflection light” is used as a term representing a concept including infrared light in addition to visible light emitted in the direction of the viewpoint.

  In the first to third embodiments, the image processing apparatus 10 and the like are described as being mounted on the portable information terminal 1 and the like. However, the image processing apparatus 10 and the like according to these embodiments may be mounted on an electronic information device 2 or the like that is different from the portable information terminal 1 or the like, similarly to the image processing apparatus 10B or the like according to another embodiment. The electronic information devices 2 and 3 may be an interphone having a personal authentication function or an automobile door, in addition to the above-described smartphone.

  In Embodiments 1 to 3 described above, the S-polarization calculation unit 12 applies the distance from the lens of the camera 20 to the surface of the eyeball E measured by the distance measuring device 30 to the eyeball E corresponding to the pixel. The incident angle of light was specified. However, the S-polarized light calculation unit 12 calculates the distance based on, for example, the size of the image of the eyeball E when the eyeball E is in focus, and uses the distance to transmit light to the eyeball E corresponding to the pixel. The incident angle may be specified. In addition, for example, the S-polarized light calculation unit 12 assumes a distance from the lens of the camera 20 to the surface of the eyeball E (for example, 20 cm to 30 cm when the mobile information terminal 1 is a smartphone), and uses the assumed distance as a pixel. The incident angle of the light to the eyeball E corresponding to may be specified. In this case, a plurality of the above-mentioned distances may be assumed, and the reflection removal may be performed by specifying the incident angle of light to the eyeball E corresponding to the pixel using each distance. In this case, the portable information terminal 1 does not necessarily need to include the distance measuring device 30.

  Further, in the description with reference to FIG. 3, there is a description of “removing” the reflected image. Further, FIG. 3 itself also has a description “S wave P wave removal”. However, in one aspect of the present invention, the image processing apparatus does not necessarily need to completely remove the reflected image. If the reflected light component that forms the reflected image is removed, that is, the reflection can be reduced. Good.

  FIG. 28 is a diagram illustrating an arrangement of pixel units in the camera 20. FIG. 28A is a diagram illustrating a state where pixel units including nine pixels are arranged so as not to overlap each other. FIG. 28B is a diagram illustrating a state in which pixel units including four pixels are arranged so as not to overlap each other. FIG. 28C is a diagram illustrating a state where pixel units including nine pixels are arranged so as to partially overlap. FIG. 28D is a diagram illustrating a state in which pixel units including four pixels are arranged so as to partially overlap. In FIGS. 28A to 28D, a region surrounded by a dotted line, a region surrounded by a broken line, and a region surrounded by an alternate long and short dash line indicate separate pixel units.

  In each embodiment described above, in the camera 20, one pixel is included in only one pixel unit. In other words, as shown in FIGS. 28A and 28B, the pixel units are arranged so as not to overlap each other. However, in one embodiment of the present invention, one pixel may be included in two or more pixel units. That is, as shown in FIGS. 28C and 28D, the pixel units may be arranged so as to partially overlap.

  When the pixel units partially overlap, the number of pixel units increases as compared with the case where the pixel units do not overlap. Therefore, when the pixel units partially overlap, it can be expected that more specularly reflected light components are removed.

  Further, the more the number of types of polarizing elements corresponding to the pixels included in one pixel unit in the principal axis direction, the more the specular reflection light component can be appropriately removed. However, since at least one pixel is required corresponding to one kind of main axis direction, if there are many kinds in the main axis direction, the number of pixels included in the pixel unit increases.

  When the number of pixels included in the pixel unit increases, as shown in FIGS. 28A and 28B, when the pixel units are arranged so as not to overlap each other, the resolution of the captured image decreases. Occurs. Practically, it is preferable that the types of the polarizing elements corresponding to the pixels included in one pixel unit are 2 types or more and 9 types or less in the principal axis direction.

  Moreover, in each embodiment mentioned above, the image processing apparatuses 10, 10A-10E were provided with the black eye detection part 11 which specifies a black eye area | region. However, the image processing apparatuses 10, 10 </ b> A to 10 </ b> E may include a cornea / iris / pupil detection unit that specifies the user's cornea, iris, or pupil region instead of the black eye detection unit 11. The process in which the iris / pupil detection unit identifies the cornea, iris, or pupil region is the same as the process in which the black eye detection unit 11 detects the black eye region, and is known in the field of iris authentication and the like. The cornea / iris / pupil detection unit obtains the position information of the pixel corresponding to the specified cornea, iris or pupil region, depending on the embodiment, the minimum luminance value selection unit 17 / 17A or the diffuse reflection component calculation unit 18, and It transmits to the authentication part 15 etc. Processing in other blocks included in the image processing apparatuses 10, 10A to 10E is the same as the processing described in each of the above-described embodiments.

  An example of processing in which the black eye detection unit 11 (or cornea / iris / pupil detection unit) detects a black eye region (or cornea, iris, or pupil region) will be briefly described below. First, the black eye detection unit 11 (or the cornea / iris / pupil detection unit) performs sharpening, edge detection, and binarization on the effective radiance value acquired by each pixel of the image sensor. For edge detection, for example, a sobel filter can be used. For binarization, for example, a moving average method or a partial image division method can be used. The black eye detection unit 11 (or the cornea / iris / pupil detection unit) performs a Hough transform on the binarized effective radiance value and detects a circular region as a black eye (or cornea / iris / pupil region). .

  In the above-described embodiments, the radius r is described as the radius of the eyeball E. However, the radius r may be the radius of curvature of the cornea.

A case where the subject includes the eyeballs E of both eyes in the first embodiment will be described below. In this case, each numerical value is defined as follows.
Rd: distance between the midpoint of the line connecting the centers of the two eyeballs E and the lens of the camera 20 R1: distance between the center of one eyeball E and the lens of the camera 20 L: one Here, the distance between the center of the eyeball E and the intermediate point is a known value measured by the distance measuring device 30 for Rd and cannot be directly measured by the distance measuring device 30 for R1. A method for calculating L will be described later. Note that Rd is not necessarily measured by the distance measuring device 30 and may be calculated by a calculation method using only the camera 20 as in R described in the sixth embodiment.

FIG. 30 is a diagram for explaining a case where the subject includes the eyeballs E of both eyes. In this case, the value of R1 can be calculated by the following equation (10-1).
^{R1 = sqrt (Rd 2 + (} L / 2) 2) (10-1)
Here, sqrt (Rd ² + (L / 2) ² ) represents the square root of Rd ² + (L / 2) ² . By substituting R1 for R in the equations (1-1) and (1-2) described above, and calculating R1 by equation (10-1), the subject is placed in both eyes in the image processing apparatus 10 of the first embodiment. Processing for the eyeball E is possible.

A method for calculating L will be described below. In calculating L, each numerical value is defined as follows.
Α: angle formed by the center and end of the cornea with respect to the center of the eyeball E H1: number of pixels of the radius of the cornea of one eyeball E in the image H2: cornea of both eyeballs E in the image Of these values, α can be set to 35 °, for example, based on the proportion of the cornea generally occupied by a circle with a radius r. Note that the value of α may be another value, for example, an arbitrary value within a range of 20 ° to 50 °. H1 and H2 can be obtained from the image.

In this case, the value of L can be calculated by the following equation (10-2).
L = H2 × (r × sin α) / H1 (10-2)
The value of R1 can be calculated by substituting the value of L calculated by Expression (10-2) into Expression (10-1).

  Further, in each of the above-described embodiments, the refractive index of the cornea has been described as n = 1.376. However, the value of the refractive index of the cornea is not limited to the above example, and may be 1.335, 1.337, 1.3375, 1.37, or 1.38, for example, or may be another value.

[Example of software implementation]
The image processing apparatuses 10, 10A to 10E may be realized by a logic circuit (hardware) formed in an integrated circuit (IC chip) or the like, or may be realized by software using a CPU (Central Processing Unit). Good.

  In the latter case, the image processing apparatuses 10, 10 </ b> A to 10 </ b> E include a CPU that executes instructions of a program that is software that realizes each function, and a ROM (the CPU (or CPU) in which the program and various data are recorded so as to be readable) A Read Only Memory) or a storage device (these are referred to as “recording media”), a RAM (Random Access Memory) for expanding the program, and the like are provided. The computer (or CPU) reads the program from the recording medium and executes the program, thereby achieving the object of one embodiment of the present invention. As the recording medium, a “non-temporary tangible medium” such as a tape, a disk, a card, a semiconductor memory, a programmable logic circuit, or the like can be used. The program may be supplied to the computer via an arbitrary transmission medium (such as a communication network or a broadcast wave) that can transmit the program. Note that one embodiment of the present invention can also be realized in the form of a data signal embedded in a carrier wave, in which the program is embodied by electronic transmission.

[Summary]
In the image processing method according to the first aspect of the present invention, a plurality of pixels associated with a plurality of polarizing elements (21a to 21o) having different principal axis directions are used as one pixel unit, and the pixel units are two-dimensionally arranged. A step (S1) of acquiring an image of a subject (eyeball E) photographed by the image sensor (light receiving element 22), and a position on the subject corresponding to a two-dimensional position of the pixel unit in the image sensor. A step (SA1, SA2, SB1) of calculating a luminance distribution of S-polarized light depending on the determined incident angle (θ) to the subject using the output of the imaging device.

  According to the above method, for example, when the subject is an eyeball, the incident angle to the eyeball when photographing the eyeball is determined by the position on the eyeball. That is, a second imaginary line (straight line L1 shown in FIG. 6) that connects the position on the eyeball and the center of the eyeball, and the second position that connects the position on the eyeball and the center of the lens of the camera equipped with the imaging device. When a virtual line (straight line L2 shown in FIG. 6) is considered, an angle formed by the first virtual line and the second virtual line is an incident angle to the eyeball.

  On the other hand, in the imaging device, pixel units each having a plurality of pixels associated with a plurality of polarizing elements having different principal axis directions are two-dimensionally arranged. The position on the eyeball, the angle of incidence on the eyeball at that position, and the two-dimensional position of the pixel unit in the image sensor correspond to each other. In addition, since the pixel unit is associated with a plurality of polarizing elements having different principal axis directions, the output of the plurality of pixels constituting the pixel unit is a distribution state of polarized light included in the reflected light from the eyeball on the eyeball. It depends on. In particular, the luminance of S-polarized light is reflected in the output of a plurality of pixels associated with a plurality of polarizing elements having different principal axis directions. That is, the output state of the image sensor reflects the distribution state of the S-polarized light included in the reflected light from the eyeball on the eyeball. Therefore, the luminance distribution of S-polarized light depending on the incident angle to the eyeball can be calculated using the output of the image sensor.

  The luminance distribution of the S-polarized light thus obtained shows different distributions between the eyeball of the living body and the imitation product of the eyeball. As a result, for example, when an imitation of an eyeball is photographed as a subject, the luminance distribution of the S-polarized light is different from the distribution of the eyeball, so that the subject can be determined not to be an eyeball. Therefore, the image processing method of aspect 1 can be used to determine whether the subject is an eyeball or not, for example.

  The luminance distribution of S-polarized light calculated as described above can also be used to remove an image of an object reflected on the eyeball due to the influence of external light, for example.

  Therefore, according to the image processing method of aspect 1, it is possible to improve the accuracy of iris authentication.

  The image processing method according to aspect 2 of the present invention preferably further includes the step of subtracting the luminance distribution of the S-polarized light from the luminance distribution of the image in the aspect 1.

  In the above method, when another object such as a scene or a person is reflected in the eyeball, the reflected light corresponding to the other object out of the reflected light from the eyeball is mainly specular reflection light. The S-polarized light is included in the reflected light. Therefore, according to the above method, unnecessary noise images reflected in the eyeball can be reduced by subtracting the S-polarized light luminance distribution from the luminance distribution of the eyeball image.

  The image processing method according to Aspect 3 of the present invention is the step of calculating the P-polarized luminance distribution depending on the incident angle to the subject from the S-polarized luminance distribution according to Fresnel's law from the S-polarized luminance distribution. It is preferable that it is further included.

  In the above method, the specular reflection light from the eyeball when another object is reflected on the eyeball often includes not only S-polarized light but also P-polarized light. The luminance distribution of P-polarized light also shows different distributions between the eyeball of the living body and the counterfeit eyeball. Therefore, according to the above method, for example, it is possible to improve the accuracy of determining whether the subject is an eyeball or not.

  The image processing method according to aspect 4 of the present invention preferably further includes the step of subtracting the luminance distribution of the S polarization and the luminance distribution of the P polarization from the luminance distribution of the image in the aspect 3.

  According to the above method, the unnecessary noise image reflected in the eyeball can be further reduced by subtracting the luminance distribution of S-polarized light and the luminance distribution of P-polarized light from the luminance distribution of the image of the eyeball.

  An image processing method according to an aspect 5 of the present invention provides the image processing method according to any one of the above aspects 1 to 4, wherein, in the step of calculating the luminance distribution of the S-polarized light, the Brewster angle is set when the incident angle is the Brewster angle. The corresponding pixel unit is specified, and the luminance value of S-polarized light at the Brewster angle is calculated by subtracting the minimum value from the maximum value of the luminance values of the pixels included in the specified pixel unit, and according to Fresnel's law, It is preferable to calculate the luminance value of S-polarized light at an incident angle other than the Brewster angle, starting from the luminance value of S-polarized light at the Brewster angle.

  In the above method, it is known that the specular reflected light hardly includes P-polarized light and mainly includes S-polarized light at the Brewster angle. Accordingly, the output of the plurality of pixels constituting the pixel unit corresponding to the Brewster angle, that is, the luminance value reflects the change in the transmittance of S-polarized light with respect to the plurality of polarizing elements having different principal axis directions. Therefore, the luminance value of S-polarized light at the Brewster angle can be calculated by subtracting the minimum value from the maximum luminance value.

  Since the function representing the luminance distribution of S-polarized light with respect to the incident angle in the eyeball is known according to Fresnel's law, if the luminance value of S-polarized light at the Brewster angle can be calculated, the luminance value of S-polarized light at an incident angle other than the Brewster angle. Can be calculated based on theory.

  The image processing method according to aspect 6 of the present invention is the image processing method according to any one of aspects 1 to 5, wherein the luminance value of S-polarized light is calculated by subtracting the minimum value from the maximum value of the luminance values of the pixels included in the pixel unit. Repeating the step for a plurality of pixel units included in the image sensor, and by associating each calculated luminance value of S-polarized light with the incident angle corresponding to the plurality of pixel units, Is preferably calculated.

  According to the above method, as already described, the luminance value of S-polarized light can be calculated by subtracting the minimum value from the maximum value of the luminance values of the pixels included in the pixel unit. This step is repeated for a plurality of pixel units included in the image sensor, and each of the calculated S-polarized luminance values is associated with an incident angle corresponding to the plurality of pixel units, whereby the S-polarized luminance distribution is measured. Can be calculated based on

  It should be noted that the method of this aspect for calculating the luminance distribution of S-polarized light based on the actual measurement value may be executed when the method of the aspect for calculating the luminance distribution of S-polarized light based on the theory does not work.

  In the image processing device (10) according to the aspect 7 of the present invention, a plurality of pixels associated with a plurality of polarization elements having different principal axis directions are used as one pixel unit, and the pixel units are two-dimensionally arranged. With respect to the image of the subject (eyeball E) photographed by the image sensor (light receiving element 22), the S-polarized light depending on the incident angle on the subject determined by the position on the subject corresponding to the two-dimensional position of the pixel unit. An S-polarized light calculation unit (12) that calculates the luminance distribution using the output of the image sensor.

  According to said structure, there exists an effect similar to aspect 1.

  An image processing program according to aspect 8 of the present invention is an image processing program for causing a computer to function as the image processing apparatus according to aspect 7, and is an image processing program for causing a computer to function as the S-polarized light calculation unit. .

  The image processing apparatus according to each aspect of the present invention may be realized by a computer. In this case, the image processing apparatus is operated on each computer by causing the computer to operate as each unit (software element) included in the image processing apparatus. An image processing program to be realized in this manner and a computer-readable recording medium on which the image processing program is recorded also fall within the category of one embodiment of the present invention.

  One aspect of the present invention is not limited to the above-described embodiments, and various modifications can be made within the scope of the claims, and the technical means disclosed in different embodiments can be appropriately combined. Such embodiments are also included in the technical scope of one aspect of the present invention. Furthermore, a new technical feature can be formed by combining the technical means disclosed in each embodiment.

10, 10A, 10B, 10C, 10D, 10E Image processing device 15 Authentication unit 16 Luminance value information acquisition unit 17 Minimum luminance value selection unit (image generation unit)
17A Minimum luminance value estimation unit (image generation unit)
18 Diffuse / reflected light component calculation unit (image generation unit)
19 Reflection presence / absence determination unit 40 Polarized light irradiation unit

Claims (19)

A plurality of pixels associated with a plurality of polarizing elements having different principal axis directions are used as one pixel unit, and the effective radiation of the subject in the imaging element is obtained by the imaging element in which the pixel units are two-dimensionally arranged. A luminance value information acquisition step of acquiring an effective radiance value that is a luminance value from the subject;
Generating a captured image including an image of the subject using an effective radiance value acquired from the subject, and
In the image generation step, for each pixel unit corresponding to at least a part of the subject, at least one of the specularly reflected light components on the surface of the subject is obtained from the effective radiance values of a plurality of pixels included in the pixel unit. An image processing method, wherein the captured image is generated as a set of unit regions having the obtained luminance value by removing a portion.   2. The image generation step according to claim 1, wherein a minimum value among effective radiance values of the plurality of pixels included in each of the pixel units is determined as a luminance value of each of the unit regions. The image processing method as described. In the image generation step,
Based on the effective radiance values of the plurality of pixels included in each of the pixel units, estimate the minimum effective radiance value that can be taken in the pixel unit,
The image processing method according to claim 1, wherein a luminance value of each of the unit areas is determined from the estimated minimum value.   In the image generation step, by applying a trigonometric function or a polynomial having an angle in the principal axis direction of the polarizing element as a variable, to the effective radiance value of the plurality of pixels included in each of the pixel units, The image processing method according to claim 3, wherein the minimum value is estimated. In the image generation step,
Based on the effective radiance values of the plurality of pixels included in each of the pixel units, the maximum value and the minimum value of the effective radiance values in the pixel unit are specified,
Using the specified maximum and minimum values and the ratio of the reflectance of the P-polarized component to the reflectance of the S-polarized component calculated based on Fresnel's law on the surface of the subject corresponding to the pixel unit. The image processing method according to claim 1, wherein a diffuse reflected light component in each of the pixel units is calculated, and the diffuse reflected light component is determined as a luminance value of each of the unit regions. In the image generation step,
Based on the effective radiance values of the plurality of pixels included in each of the pixel units, the maximum value and the minimum value of the effective radiance values in the pixel unit are specified,
Calculating a first regular reflection light component by subtracting the minimum value from the specified maximum value;
The calculated first regular reflection light component and the ratio of the reflectance of the P polarization component to the reflectance of the S polarization component calculated based on Fresnel's law on the surface of the subject corresponding to the pixel unit, To calculate the second regular reflection light component,
The image processing method according to claim 1, wherein a value obtained by subtracting the first and second specular light components from the identified maximum value is determined as a luminance value of each of the unit regions. A reflection determination step for determining whether or not a specularly reflected light component on the surface of the subject included in the effective radiance value acquired in the luminance value information acquisition step is less than or equal to a predetermined intensity;
In the reflection determination step, when it is determined that the specularly reflected light component on the surface of the subject is equal to or less than a predetermined intensity, the effective radiance value is acquired in a state where the subject is irradiated with polarized light. A luminance value information reacquisition step,
When the luminance value information reacquisition step is executed, in the image generation step, specularly reflected light on the surface of the subject included in the effective radiance value acquired in the luminance value information reacquisition step The image processing method according to claim 1, wherein at least a part of the components is removed. A plurality of pixels associated with a plurality of polarizing elements having different principal axis directions as one pixel unit, and obtaining an image of a subject imaged by an imaging element in which the pixel units are two-dimensionally arranged;
Calculating a luminance distribution of S-polarized light depending on an incident angle to the subject determined by a position on the subject corresponding to a two-dimensional position of the pixel unit in the imaging device, using an output of the imaging device; ,
An image processing method comprising:   The image processing method according to claim 8, further comprising a step of subtracting the luminance distribution of the S-polarized light from the luminance distribution of the image obtained from the output of the imaging device.   10. The image processing method according to claim 8, further comprising a step of calculating a P-polarized light luminance distribution depending on an incident angle to the subject from the S-polarized light luminance distribution according to Fresnel's law. .   The image processing method according to claim 10, further comprising a step of subtracting the luminance distribution of the S polarization and the luminance distribution of the P polarization from the luminance distribution of the image. An image processing apparatus comprising a plurality of pixels associated with a plurality of polarizing elements having different principal axis directions as one pixel unit, and an imaging device in which the pixel units are two-dimensionally arranged,
A luminance value information acquisition unit that acquires, from the subject, an effective radiance value that is an effective radiance value of the subject in the image sensor by the image sensor;
An image generation unit that generates a captured image including an image of the subject using an effective radiance value acquired from the subject,
For each pixel unit corresponding to at least a part of the subject, the image generation unit is configured to obtain at least one of specularly reflected light components on the surface of the subject from effective radiance values of a plurality of pixels included in the pixel unit. The image processing apparatus is characterized in that the captured image is generated as a set of unit areas having the obtained luminance value by removing a portion.   The image generating unit determines a minimum value among effective radiance values of the plurality of pixels included in each of the pixel units as a luminance value of each of the unit regions. The image processing apparatus described. The image generation unit
Based on the effective radiance values of the plurality of pixels included in each of the pixel units, estimate the minimum effective radiance value that can be taken in the pixel unit,
The image processing apparatus according to claim 12, wherein a luminance value of each of the unit areas is determined from the estimated minimum value.   The image generation unit applies a trigonometric function or a polynomial having an angle in a principal axis direction of the polarizing element as a variable to an effective radiance value of the plurality of pixels included in each of the pixel units. The image processing apparatus according to claim 14, wherein a minimum value is estimated. The image generation unit
Based on the effective radiance values of the plurality of pixels included in each of the pixel units, the maximum value and the minimum value of the effective radiance values in the pixel unit are specified,
Using the specified maximum and minimum values and the ratio of the reflectance of the P-polarized component to the reflectance of the S-polarized component calculated based on Fresnel's law on the surface of the subject corresponding to the pixel unit. The image processing apparatus according to claim 12, wherein a diffuse reflected light component in each of the pixel units is calculated, and the diffuse reflected light component is determined as a luminance value of each of the unit regions. The image generation unit
Based on the effective radiance values of the plurality of pixels included in each of the pixel units, the maximum value and the minimum value of the effective radiance values in the pixel unit are specified,
Calculating a first regular reflection light component by subtracting the minimum value from the specified maximum value;
The ratio of the reflectance of the P-polarized component to the reflectance of the S-polarized component calculated based on Fresnel's law formula on the surface of the subject corresponding to the pixel unit and the calculated first regular reflected light component. To calculate the second regular reflection light component,
The image processing apparatus according to claim 12, wherein a value obtained by subtracting the first and second specularly reflected light components from the specified maximum value is determined as each luminance value of the unit region. A reflection determination unit that determines whether or not a specularly reflected light component on the surface of the subject included in the effective radiance value acquired by the luminance value information acquisition unit is a predetermined intensity or less;
Luminance for acquiring the effective radiance value in a state where polarized light is irradiated to the subject when the reflection determination unit determines that the specularly reflected light component on the surface of the subject is equal to or less than a predetermined intensity A value information reacquisition unit,
When the luminance value information reacquisition unit acquires the effective radiance value of the subject, the image generation unit includes the surface of the subject included in the effective radiance value acquired by the luminance value information reacquisition unit. The image processing apparatus according to claim 12, wherein at least a part of the regular reflection light component is removed.   The image according to any one of claims 12 to 18, further comprising an authentication unit that performs identity authentication based on a captured image generated by the image generation unit, wherein a part of the subject is an iris. Processing equipment.

Images