BR122018007964B1 - METHOD IMPLEMENTED BY COMPUTER AND SYSTEM - Google Patents

Abstract

COMPUTER AND SYSTEM IMPLEMENTED METHOD The present invention relates to biometric enrollment and verification techniques for ocular-vascular, periocular and facial regions. Periocular image regions can be defined based on the dimensions of an ocular region identified in an image of a facial region. Feature descriptors can be generated for points of interest in the ocular and periocular regions using a combination of standardized histogram feature descriptors. Quality metrics for regions can be determined based on region value scores calculated based on texture surrounding points of interest. A biometric matching process for calculating a match score based on ocular and periocular regions may progressively include additional periocular regions to achieve greater match certainty.

Description

[0001] Split from BR112018004755-4 deposited on 09/09/2016.

CROSS REFERENCE TO RELATED ORDER

[0002] The present application claims priority to and the benefit of U.S. Provisional Patent Application 62/217,660, filed on September 11, 2015 and entitled “Image Enhancement, Quality Metrics, Feature Extraction, Information Fusion, Pose Estimation and Compensation, and Template Updates for Biometric Systems”, the entirety of which is incorporated herein by reference.

BACKGROUND

[0003] The present invention relates to biometric authentication and, more specifically, to systems and methods for image enhancement, quality metrics, feature extraction and information fusion for multiple region biometric systems (subocular to face) or based on another image.

[0004] Biometric systems can be used to authenticate an individual's identity to either grant or deny access to a resource. For example, image scanners may be used by a biometric security system to identify an individual based on unique structures on the individual's face, e.g., the individual's eye and its surrounding regions. Biometric data captured from an individual, such as that captured during an enrollment process, can be stored as a template that is used to verify the individual's identity a next time. Biometric scanning technology would benefit, however, from improved techniques when using periocular features instead of or in addition to ocular features for enrollment and verification.

BRIEF SUMMARY

[0005] Systems and methods relating to biometric techniques applicable to ocular-vascular, periocular and facial regions are disclosed. In one aspect, a computer-implemented method comprises the steps of: receiving an image of a facial region of a user, the facial region including an eye and an area surrounding the eye; processing the image to define an eye image region including at least a portion of the eye in the image of the facial region; defining a plurality of periocular image regions each including at least a portion of the area surrounding the eye in the facial region image, wherein the periocular regions are defined based on dimensions of the defined ocular region; calculating one or more biometric matching scores based on the ocular image region and at least one of the periocular regions; and designating the image of the facial region as authentic or inauthentic based on one or more biometric matching scores.

[0006] In one embodiment, the plurality of periocular imaging regions comprises at least four periocular imaging regions. The at least four periocular imaging regions may comprise a periocular imaging region disposed below the ocular imaging region, a periocular imaging region disposed to the right of the ocular imaging region, a periocular imaging region disposed to the left of the ocular imaging region. and a periocular imaging region disposed above the ocular imaging region.

[0007] Defining the plurality of periocular imaging regions may comprise defining a lower periocular imaging region disposed below the ocular imaging region, the lower periocular imaging region having a width substantially equal to a width of the width of the ocular imaging region. and a height in the range of 10% to 300% of a height of the ocular imaging region. Defining the plurality of periocular imaging regions may also comprise defining a right periocular imaging region disposed to the right of the ocular imaging region, the right periocular imaging region having a width in the range of 10% to 80% of a width of the region. of an ocular image region and a height of 120% to 550% of a height of the ocular image region. Defining the plurality of periocular imaging regions may further comprise defining a left periocular imaging region disposed to the left of the ocular imaging region, the left periocular imaging region having a width in the range of 10% to 50% of a width of the ocular imaging region. eye image and a height of 120% to 550% of a height of the eye image region. Defining the plurality of periocular imaging regions may also comprise defining an upper periocular imaging region disposed above the ocular imaging region, the upper periocular imaging region having a width substantially equal to a width of the ocular imaging region and a height at the range from 10% to 150% of a height of the ocular imaging region.

[0008] In another embodiment, calculating the one or more biometric matching scores comprises: calculating a first biometric matching score based on the ocular image region and an ocular inscription model; and in response to determining that the first biometric matching score does not satisfy a first matching threshold, calculating a second biometric matching score based on the ocular imaging region, a first of the periocular imaging regions, in the ocular enrollment model, and a periocular enrollment model. Calculation of the one or more biometric match scores may further comprise, in response to determining that the second biometric match score does not satisfy a second match threshold, calculating one or more additional biometric match scores by iteratively including more of the image regions. periocular in calculating additional biometric match scores until either a particular additional biometric match score satisfies a corresponding match threshold or any additional periocular image regions are available for inclusion. The plurality of periocular image regions may be classified based on at least discriminative ability and/or quality, and additional periocular image regions may be iteratively included based on respective classifications of the additional periocular image regions. One or more of the plurality of periocular image regions may be divided into subregions based on one or more facial features derived from the periocular image regions, and additional periocular image regions may be iteratively included based on clustering importance or the respective classifications of additional periocular imaging regions.

[0009] In an additional embodiment, calculating one or more biometric matching scores comprises: identifying a first set of matching point pairs based on the ocular image region and an inscription template; and identifying a second set of corresponding point pairs based on at least one of the periocular image regions and the inscription template. Calculation of the one or more biometric match scores may further comprise: determining one or more inlier match points by entering a combination of the first and second sets of matching point pairs into an outlier detection algorithm; determining that a number of the corresponding inlier points corresponding to the ocular image region satisfies a minimum ocular inlier count; and calculating a particular biometric matching score based at least in part on the lower corresponding points. The minimum ocular inlier count can be equal to 3.

[0010] In yet another embodiment, calculating the one or more biometric matching scores further comprises: determining one or more first inlier matching points by entering the first set of pairs of matching points into an outlier detection algorithm; determining one or more second corresponding inlier points by entering the second set of corresponding point pairs into an outlier detection algorithm; and calculating a particular biometric matching score based at least in part on an output of an outlier detection algorithm using a combination of the first and second inlier matching points as input. Calculation of the one or more biometric matching scores may further comprise determining that a number of corresponding inlier points, obtained from the output of the outlier detection algorithm, that correspond to the ocular image region satisfy a minimum ocular inlier count. The minimum ocular inlier count can be equal to 3.

[0011] In another aspect, a computer-implemented method comprises the steps of: receiving an image of a user's facial region; the facial region including an eye and an area surrounding the eye; image processing to: (i) define an ocular image region including at least a portion of the eye in the facial region image and (ii) define one or more periocular image regions each including at least a portion of the area surrounding the eye in the image of the facial region; identifying a plurality of points of interest in at least one of the ocular imaging region and the one or more periocular imaging regions; generating, for each point of interest, a feature descriptor based on a combination of a plurality of standardized histogram feature descriptors; and storing the characteristic descriptors generated in a biometric model.

[0012] In one embodiment, image processing comprises enhancing at least a portion of the image using local gradient Gabor pattern (LGGP). Enhancement of at least a portion of the image using LGGP may comprise: calculating, at each of a plurality of angles, a Gabor phase image for the at least a portion of the image, aggregating the computed Gabor phase images to form an image combined Gabor phase; calculating, at each of a plurality of angles, a local gradient of the combined Gabor phase image; and retaining a maximum value of each local gradient to form an improved image.

[0013] A particular standardized histogram feature descriptor may comprise a standardized histogram of extended multi-ray local binary patterns (PH-EMR-LBP), a standardized histogram of extended multi-ray central symmetric local binary patterns (PH-EMR- CSLBP) or a standardized histogram of extended multi-ray local ternary patterns (PH-EMR-LTP). Generation of the feature descriptor for a particular point of interest may comprise generating a PH-EMR-LBP feature descriptor by: defining an image region comprising the particular point of interest; calculating a plurality of local binary pattern (LBP) codes for each pixel in the image region to form a multi-ray LBP (MR-LBP) image; division of the MR-LBP image into a plurality of subregions; deriving a plurality of histograms comprising frequencies of each MR-LBP bit location within each subregion; and combining the plurality of histograms to form the PH-EMR-LBP feature descriptor. Generation of the feature descriptor for a particular point of interest may comprise generating a PH-EMR-CSLBP feature descriptor by: defining an image region comprising the particular point of interest; calculating a plurality of central symmetric local binary pattern (CSLBP) codes for each pixel in the image region to form a multi-ray CSLBP image (MR-CSLBP); division of the MR-LPB image into a plurality of subregions; deriving a plurality of histograms comprising frequencies of each MR-CSLBP bit location within each subregion; and combining the plurality of histograms to form the PH-EMR-CSLBP feature descriptor. Generation of the feature descriptor for a particular point of interest may comprise generating a PH-EMR-LTP feature descriptor by: defining an image region comprising the particular point of interest; calculating a plurality of local ternary pattern (LTP) codes for each pixel in the image region to form a multi-ray LTP (MR-LTP) image; division of the MR-LBP image into a plurality of subregions; deriving a plurality of histograms comprising frequencies of each MR-LTP bit location within each subregion; and combining the plurality of histograms to form the PH-EMR-LTP feature descriptor.

[0014] In another embodiment, generating the feature descriptor for a particular point of interest comprises: calculating a PH-EMR-LBP feature descriptor, PH-EMR-CS-LBP feature descriptor and PH-EMR feature descriptor -LTP for the particular point of interest; and combining the PH-EMR-LBP feature descriptor, PH-EMR-CS-LBP feature descriptor and PH-EMR-LTP feature descriptor to form a combined feature descriptor. Generation of the feature descriptor for a particular point of interest may further comprise applying an analysis of variance to the combined feature descriptor to form a main feature descriptor comprising a subset of features of the combined feature descriptor. Generating the feature descriptor for a particular point of interest may further comprise performing dimensionality reduction on the main feature descriptor to form the feature descriptor for the particular point of interest.

[0015] In a further embodiment, in generating a particular standardized histogram feature descriptor, local binary pattern or local ternary pattern feature descriptors are calculated using a neighborhood defined by a square with or without corners.

[0016] In another aspect, a computer-implemented method comprises the steps of: receiving an image of a facial region of a user, the facial region including an eye and an area surrounding the eye; defining an eye image region including at least a portion of the eye in the image of the facial region; defining one or more periocular image regions each including at least a portion of the area surrounding the eye in the image of the facial region; identifying a plurality of points of interest in at least one of the ocular imaging region and in the one or more periocular imaging regions; calculating, for each point of interest, a region value for texture surrounding the point of interest; and determining at least one quality metric for at least a portion of the facial region image based on the points of interest and respective calculated region values.

[0017] In one embodiment, calculating the region value for a particular point of interest comprises: calculating at least one local binary pattern (BP) in a square-shaped neighborhood for the particular point of interest; and calculating at least one BP for one or more offset points of the particular point of interest. Calculating the region value for the particular point of interest may further comprise adjusting the region value to an average of region values calculated for the particular point of interest and a plurality of offset points. Calculating at least one BP for the particular point of interest may comprise calculating a plurality of BPs, each having a different neighborhood, for the particular point of interest, and calculating at least one BP for the offset points may comprise calculating of a plurality of BPs, each having a different neighborhood, for each compensation point. Calculating the plurality of BPs for a particular point of interest or compensation point may comprise: reducing the plurality of BPs to a Noise Binary Pattern (NBP); and creating a general binary standard (genBP) from the NBP. Calculating a plurality of BPs for a particular point of interest or compensation point may further comprise: creating an H-weighted pattern from the genBP; and calculating a region value for the particular point of interest or compensation point as: where L comprises a maximum length of continuous zeros in genBP. The compensation points may comprise a plurality of uniformly shifted pixel locations in directions other than the particular point of interest.

[0018] In another embodiment, determining the quality metric comprises: creating an ordered list of points of interest based on the respective region values of the points of interest; and calculating distances between consecutive points of interest in the order listed. Determining the quality metric may also include calculating the quality metric as: where p comprises the number of points of interest, sn comprises the region value calculated for point of interest n, swn comprises a weighted index for point of interest n and dwn comprises a weight for the distance corresponding to point n in the order listed.

[0019] In a further embodiment, at least a portion of the image of the facial region comprises the ocular region or at least one of the periocular regions. The method may further comprise classification of the periocular image regions based on respective quality metrics separately calculated for each of the periocular image regions. The method may further comprise classifying the periocular image regions based on at least one of a respective discriminative ability of each periocular image region. The method may further comprise using the periocular image regions in a progressive biometric matching device based at least in part on the respective quality metric ratings and/or discriminative ability of the periocular image regions.

[0020] In yet another embodiment, the method further comprises: determining that a difference in calculated quality metrics between a first of the ocular or periocular image regions and a second of the ocular or periocular image regions exceeds a threshold; and indication of the likely presence of fraud based on the difference determined in the calculated quality metric.

[0021] Other aspects of the above include corresponding systems and non-transitory computer readable means. Details for one or more embodiments of the subject matter described in the present application are shown in the accompanying drawings and in the description below. Other characteristics, aspects and advantages of the subject matter will become apparent from the description, drawings and claims.

BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In drawings, like reference characters generally refer to the same parts in different views. Also, the drawings are not necessarily to scale, the emphasis instead being generally placed on illustrating the principles of the modality. In the following description, various embodiments are described with reference to the following drawings, in which:

[0023] FIG. 1 shows an exemplary method for creating an ocular and periocular inscription model, in accordance with one embodiment.

[0024] FIGS. 2A and 2B show an exemplary method for defining ocular and periocular regions, and corresponding facial images, in accordance with one embodiment.

[0025] FIG. 3 shows an example of a modified local binary pattern process (genBP), according to one embodiment.

[0026] FIG. 4 shows an exemplary technique for biometric progressive matching with enrollment and verification phases, in accordance with one embodiment.

[0027] FIG. 5 shows an exemplary system for performing biometric scanning and analysis in accordance with one embodiment.

DETAILED DESCRIPTION

[0028] The white of the eye includes complex patterns (primarily due to vascular structures), which are not only readily visible and scannable, but are also unique to each individual. In some cases, other non-vascular formations may also be visible and may be taken into account by a biometric matching algorithm. In this way, these visible structures seen in the white of the eye, mainly due to the vasculature of the conjunctiva and episclera, can be scanned and advantageously used as a biometric. This biometric can be used to authenticate a particular individual or identify an unknown individual from a larger pool of candidates. Implementations of solutions for imaging and pattern matching of blood vessels in the white of the eye and for vascular point detection, feature extraction and matching are described, for example, in U.S. Patent No. 8,369,595, issued February 5 of 2013, entitled “Texture Features for Biometric Authentication” and U.S. Patent No. 9,390,327, issued on July 12, 2016 and entitled “Feature Extraction and Matching for Biometric Authentication”, the entirety of which is incorporated herein by reference. To add robustness to these and other biometric verification techniques, described here are methods for progressively including non-ocular-vascular structures visible outside the white of the eye, such as periocular or partial or even complete facial patterns or other visible identifiable, in an ocular-vascular biometric matching scheme in case, for example, additional information is needed to form a more accurate analysis.

[0029] Ocular features are those that arise from patterns seen external to the limbus and bounded by the eyelids, which are mainly due to vascular patterns seen in the whites of the eyes. Hereinafter, such features are interchangeably referred to herein as ocular or ocular-vascular. The white of the eye has several layers. The sclera is an opaque, fibrous, protective layer of the eye containing collagen and elastic fiber. The sclera is covered by the episclera, which has a particularly large number of blood vessels and veins running through and over it. The episclera is covered by the bulbar conjunctiva, which is a thin transparent membrane that interfaces the eyelid or the environment when the eyelid is opened, which also contains unique and rich vascular structures. Blood vessels run through all of these layers of the white of the eye and can be detected in images of the eye. The eye also includes eyelashes and sometimes falling eyelashes that can sometimes obscure portions of the white of the eye in an image.

[0030] Periocular refers to the immediate region of the face surrounding the eye (external to, in some cases contacting, the edges of the eyelid), which is textured and can be used for biometric applications. The periocular region may include one or more regions that may be, but are not necessarily, contiguous and which may take various forms with respect to shape and size. While the periocular region does not have any well-defined boundaries, standardization of the region of interest can minimize registration issues, further improving the accuracy of the biometric system. Such standardization can be achieved by using certain measurable value of the user's face, such as distance between the corners of the eye, iris radius, interocular distance and/or derived eye slice width and height (while using eye detection algorithms). During user authentication, one or more digital images of a user's eye and periocular region are captured, one or more verification templates are generated from the captured image or images, and the user's identity can be verified by matching the ocular structures and corresponding perioculars as expressed in the registration and verification templates.

[0031] In other embodiments, additional regions of the face extending beyond the periocular region may be used in addition to or instead of regions immediately surrounding the eye. For example, when heavy facial expressions are observed, one or more regions in and/or surrounding the nose and/or around the face may be considered as extended periocular regions. In one embodiment, facial features may complement eye features for an eye-based biometric matching device. In another embodiment, the ocular features used by an eye-based biometric matching device are complemented by both extended periocular and facial features. In another embodiment, an entire face may be used in addition to or instead of the regions immediately surrounding the eye. The face can complement ocular features for an eye-based biometric matching device. Both the periocular and face regions can complement the ocular features used by an eye-based biometric matching device.

[0032] In various other embodiments, other information such as the skin tone of the periocular region can also be measured and used as a soft biometric. Skin tone can be estimated using histograms of the intensity values of different image color ranges, such as normalized red, green, and blue (RGB) channels. During verification, the histogram information of the enrolled model can be used to stop a verification process when the distance between the enrollment histograms and verification images is greater than a threshold.

[0033] In one embodiment, skin tone can be estimated through a grouping technique that identifies the most prominent colors. As an example, the RGB pixels of the periocular region are converted into the Lab color space and grouped into the number of classes N. The optimal N can be determined using AIC (Akaike Information Criterion) or BIC (Bayesian Information Criterion). These clusters are then sorted in ascending order based on their occurrence, and the top three clusters are grouped with the predefined skin tones. The closest skin tone can be determined with a distance measurement and/or correlation measurement. In one embodiment, the distance measure is Euclidean distance, although other distance measures are understood. Then, the verification procedure can be stopped if the distance and/or correlation between the enrollment and verification image is greater than a threshold. The same process can be applied to other colored parts of the image, such as the iris.

[0034] FIG. 1 shows an exemplary method for creating an ocular and periocular inscription model. In step 102, one or more images of a user's facial region(s) are captured using an image sensor, e.g., a camera, which may be associated with a device to which access by the user must be authenticated ( for example, a smartphone, a smart watch, smart glasses, a notebook computer, a desktop computer, etc.). By way of illustration, the camera may be a digital camera, a three-dimensional (3D) camera, a light field sensor and/or a near-infrared sensor or other monochromatic and/or multispectral image sensor. Images may include one or more ocular regions (regions defining an area of the user's eye) and/or one or more periocular regions, as well as other facial regions, in some cases. Images can be captured in either still mode or video mode or a combination thereof. Images can be captured at multiple wavelengths. In some embodiments, the user is induced (through visual, auditory, or tactile feedback) to move the device closer/further to trigger the optimal distance to find the eyes and/or face.

[0035] In step 104, an initial region of interest (ROI) is identified, for example, one or both eyes can be located. Viola-Jones algorithms or similar learning algorithms trained on several different forward-looking eye images can be used for this purpose. In another embodiment, Viola-Jones algorithms trained on multiple eye images with different gazes can be used. Then, a gaze tracking algorithm, such as one using Haar filters, can be used to quantize the amount of gaze to acquire one or more images once a selected gaze direction is detected.

[0036] Once the initial ROI is located on the face shown in the captured image(s), additional computation can be performed to obtain the final ROI in the image. Typically, the final ROIs are cropped to obtain RGB and/or near-infrared images of one or more eyes. Thus, as used herein, “image” or “captured image” may also refer to a cropped image of a final ROI.

[0037] With reference to FIG. 1, in the Ocular and Periocular Stages, one or more averaged or otherwise enhanced or naturally captured images, which may be inscribed images or verification images, are processed in image enhancement substage 106 and image quality substage 108 and characteristics of the pre-processed images are extracted in the feature extraction sub-stage110, to generate one or more ocular and periocular models. Preprocessing substages 106 and 108 may include image and image quality enhancement techniques, explained further below. Feature extraction substage 110 may include point of interest detection and local descriptor extraction, which are described in detail below. The operations performed during substages 106, 108, and 110 in the Ocular and Periocular Stages need not be the same. On the contrary, different operations that account for the particular characteristics of ocular and periocular regions can be used at these respective stages. The enrollment ocular and/or periocular models, prior to storage, may be encrypted or otherwise protected in step 112.

Periocular Limits

[0038] Referring now to FIGS. 2A and 2B, periocular regions can be extracted using various methods. When acquiring an image of a user's face (step 200), one or more regions of the eye are detected (step 202). Based on the size of the eye slices identified in step 202, the size of the eye region can be recalculated (step 204). Proceeding to the ocular stage, in one embodiment, an integro-differential algorithm, Hough circles or Hessian bubble detector is used to detect the iris boundary (iris region segmentation in step 208). Similarly, a Gabor filter-based filtering algorithm can be used to detect the upper and lower eyelids, and more whites of the eye can be separated after removing the iris region (scleral region segmentation in step 206). The periocular region can be derived after subtracting both the white of the eye and iris regions from the captured images (step 210). In some embodiments, color-based methods are used to segment the regions mentioned above.

[0039] In one embodiment, as shown in the periocular region stage 212 in FIGs. 2A and 2B, periocular regions around the final ocular ROI of size width W x height H (where W and H are the number of pixels in the horizontal and vertical directions, respectively) can be defined as described below. The ocular ROI is not necessarily a rectangular shape, rather, the ROI can take a shape similar to the shape of the eye, as shown in FIG. 2B, and W and H can be defined as the width and height, respectively, of a bounding box encompassing the ocular ROI. There may be multiple numbers of regions; however, the embodiment described below includes four periocular regions (1 to 4) classified by discriminative ability, which essentially refers to how useful the region is in a biometric matching operation. More specifically, periocular region 1 represents a facial region with more distinctive, or discriminatory, characteristics than region 2, which has greater discriminatory capacity compared to region 3, and so on. Various sizes of periocular region are understood (here, defined by a bounding box of width W x height H). The ranges and values of the periocular regions disclosed here were experimentally determined based on large-scale testing of which periocular regions have sufficient discriminative capacity for the purposes of performing biometric matching operations. As used below, the term “substantially equal” means equal to a particular value or within +/- 10% of the value.

[0040] In one embodiment, the four periocular regions are defined as follows, where W and H refer to the width and height, respectively, of the ocular region. • Region 1: o Width is substantially equal to W. o Height in the range of 0.1*H to 3*H below the eye section of the ocular region, inclusive. In one embodiment, height is substantially equal to 0.3*H. • Region 2: Width is in the range of 0.1*W to 0.8*W, inclusive. In one embodiment, the width is substantially equal to 0.2*W. o Height is in the range of 1.2*H (0.1*H above the eye section of the eye region and 0.1*H below the eye section) to 5.5*H (1.5*H above of the eye cut and 3*H below the eye cut), inclusive. In one embodiment, height is substantially equal to 1.45*H (0.15* above the body of the eye and 0.3*H below the cut of the eye). • Region 3: Width is in the range of 0.1*W to 0.5*W, inclusive. In one embodiment, the width is substantially equal to 0.15*W. o Height is in the range of 1.2*H (0.1*H above the eye section of the eye region and 0.1*H below the eye section) to 5.5*H (1.5*H above of the eye cut and 3*H below the eye cut), inclusive. In one embodiment, height is substantially equal to 1.45*H (0.15*H above the eye slice and 0.3*H below the eye slice). • Region 4: o Width is substantially equal to W. o Height is in the range of 0.1*H to 1.5*H above the eye section of the ocular region, inclusive. In one embodiment, height is substantially equal to 0.15*H.

[0041] Other non-rectangular or even non-contiguous periocular (or other facial) subregions may be used based on the differentiating ability or reliability of these region(s) by individual, certain population or all users (s). In some embodiments, when drooping eyelids are frequently observed, an optimal periocular region around the tight cut of size WxH can be defined as follows: • The widths of regions 2 and 3 are substantially equal to 0.2*W and 0.15 *W, respectively. • The heights of regions 2 and 3 are substantially equal to 1.45*H each (which starts from the top of the eye slice). • The widths of regions 1 and 4 are substantially equal to W. • The heights of regions 1 and 4 are substantially equal to 0.45*H and 0.1*H, respectively.

[0042] The range of values for heights and widths for various regions is the same as explained above. Other periocular region sizes are understood. W and H values may vary based on image sensor resolution.

Image improvement

[0043] Image enhancement techniques, such as those practiced in step 106 of FIG. 1, will now be explained. Images that are captured from an image sensor may have variable quality due to, for example, exposure and motion blur artifacts. In step 106, which is optional, multiple images can be registered (i.e., spatially aligned) and averaged to reduce image noise. In some embodiments, image correlation methods are used to measure the dissimilarity between obtained images in order to align the images to average and discard those that are most dissimilar (e.g., due to motion blur or eye blinking). and, as such, are not suitable for recording and averaging. For example, n consecutive structures with minimal movement can be averaged after recording. The number of consecutive structures (disregarding the occasional dropped structures explained above) that are averaged may depend on the structure rate, noise level of the image sensor in the ambient data, environmental conditions in which the images are being captured.

[0044] In one example, if ambient light is greater than 800 lumens during image acquisition, two consecutive structures are recorded and averaged. If the ambient light is in the range of 450-800 lumens, 3 consecutive frames are recorded and averaged. If the ambient light is in the range of 0-450 lumens, 4 consecutive frames are recorded and averaged. Other combinations of selecting the number of structures based on ambient light are possible for the whites of the eyes, periocular region, and face, including the non-averaging option if the region of interest is bright enough for sensor noise to be negligible in the ambient data.

[0045] In another embodiment, the number of structures used for averaging can also be adapted to motion artifacts in consecutive structures. For example, movement may be caused due to changing expressions, eye and body movements, and/or environmental aberrations. Motion can be measured using correlation in structures, mutual information in structures, etc. In one embodiment, the number of structures used for averaging is based on both environmental light and observed motion artifacts.

[0046] In some embodiments, image enhancement algorithms include Self Quotient Image or SQI. A simple version of SQI image (Q) can be represented as follows: where I is the smoothest version of I and F is the smoothing kernel. The SQI method is known for its light invariance property, which is useful for periocular regions with variable illumination due to shadows and other light conditions. Other image enhancements such as Weber Linear Descriptors (WLD), Coherence Diffuse Filters, Atrous Wavelet, Retinex (and its variants), single or multiple structural bilateral filtering, deep learning methods and other sparse enhancement can be used.

[0047] In one embodiment, local gradient Gabor pattern (LGGP) can be used as an image enhancement technique for a particular image or image region, as described in the steps that follow.

[0048] Step 1: Calculate the Gabor phase of the green channel of the image region using even and odd Gabor responses as follows:

[0049] Step 1a: a 2D even Gabor kernel, similar to low-pass filters, is a cosine function modulated by a 2D Gaugassian envelope and a 2D odd Gabor is a sinusoidal function modulated by a 2D Gaugassian envelope. The even and odd Gabor kernels can be derived as follows:

[0050] Any Gabor kernel scale and orientation can be obtained through the following coordinate transformation: where Ox and Gy define spread of the Gaugassian envelope along the x and y axes, respectively, f is the sine and cosine frequency of modulation and 0 is kernel orientation. In one embodiment, the frequency is set to 6 and Gaussian propagation along the x and y axes is set to 2.5. In one embodiment the choice of orientations is suppressed to 6, ranging from 0 to 5 p/6, each separated by p/6 radians.

[0051] Step 1b: Gabor image phase is calculated as follows: where is the convolution operator. In any given orientation, as an example, Ifase (0 = 0) is an image of size equal to the size of the green channel of the image region.

[0052] Step 2: The phase Gabor responses (images) in various orientations are aggregated to form an output image. Three exemplary methods for obtaining this area are: (1) retaining the maximum intensity value of a pixel at a given location (x, y) in all orientations; (2) adjust the intensity of a pixel at a given location (x, y) for the weighted average of responses across all orientations at a particular location; and (3) multiply the maximum intensity value of a pixel at a given location (x, y) in all orientations with the original image, followed by normalization.

[0053] Step 3: Local gradients in four directions (0, 45, 90 and 135 degrees) are then calculated for the aggregated responses and the maximum gradient value is retained to form an output image.

[0054] In one embodiment, color normalization such as white balance using a gray world algorithm can be used before the enhancement techniques mentioned above.

[0055] In one embodiment, additional processing of the enhanced image includes image histogram and contrast adjustments such as Contrast Limited Adaptive Histogram Equalization (CLAHE). CLAHE generally operates on small regions of the image referred to as tiles. Typically, each tile contrast is enhanced such that the output histogram approximately matches the specified histogram across a particular distribution (e.g., uniform, exponential, or Rayleigh distribution). Neighboring tiles are then combined using an interpolation (e.g. bilinear interpolation) to eliminate any artificially induced divisions. In some embodiments, selecting a linear or nonlinear combination of red, green, or blue color components that have the best contrast between the ocular vessels or periocular features and the background can enhance the imaging region. For example, the green component may be preferred in an ocular-vascular RGB image because it can provide better contrast between vessels and the background.

Image Quality Metric and Feature

[0056] Referring again to step 108 in FIG. 1, the quality of each enhanced image can be measured, and those that meet a certain quality threshold are retained for further processing. This quality can be measured together or separately for the ocular and periocular regions. An image quality metric can also act as a predictive match quality metric (a measure of the respective biometric sample's ability to match) and, in some scenarios, can be merged into a final match score to improve image performance. biometric system.

[0057] In one embodiment, points of interest detected in an image can determine the quality of the image. A point-based quality metric, which is a non-reference image quality metric, will be referred to here as EV_QM. EV_QM can be calculated using three steps: Point of Interest Detection, Region Value Calculation and Quality Metric Score Generation. Points of interest can be detected using vascular point detection (VPD), explained further below. Region Value refers to the amount of texture around each point of interest. The texture around each point of interest can be determined using Local Ternary Pattern (LTP). Quality metric score generation is measured using corner location and region value.

[0058] With reference to FIG. 3, in one embodiment, region value may be calculated using the modified local binary pattern (LBP) process that follows. Three squares (with or without corners) with corresponding different halves of length (neighborhoods) are computed around a point of interest, with their pixel loci computed (as shown in pixel grid 304). For example, these length halves could be 2, 4, 6, or another number of pixels. The point of interest can be further displaced (changed position) in 8 different locations (as shown in pixel grid 302), creating 3 more squares (minus their corners) around each displacement point (similar to that shown in the pixel grid). pixel 304, with the offset point serving as the center point). LBP for each point of interest and offset are calculated in each of the three squares, which are further referred to as Binary Patterns (BP). In this way, each point of interest and corresponding offset points have three different BPs associated with them.

[0059] The three BPs can be reduced further to obtain a final region value (texture score) as follows:

[0060] Step 1: each BP is evaluated to determine the uniformity score. If the BP is not uniform (e.g., more than 4 bits of change or less than 2 bits of change), then the BP is discarded and the score (end region value) at the corresponding offset or point of interest is adjusted to zero.

[0061] Step 2: If all BPs are uniform, the filtering process, as shown graphically in step 306 of FIG. 3, is accomplished. Each element in BP1, BP2 and BP3 is added using the following formula: There are three possible values (1, 0 and 2) that can be seen in each element of Resultadon. Each of these bits is further mapped according to the corresponding values (-1, 0, and 1, respectively) in Value, as shown in step 308. This result is further referred to as Noise Binary Pattern (NBP).

[0062] Step 3: If there are more than four pixels of noise, the score for the corresponding point of interest is set to zero. Otherwise, each noise pixel in the NBP is replaced with its nearest neighbor.

[0063] Step 4: The final result is a single binary pattern with length BP to half length 2. This final binary pattern is further referred to as genBP.

[0064] Step 5: genBP is further weighted based on the formula described in step 310: the result is further referred to as a weighted or H pattern.

[0065] Step 6: Calculate the maximum length of continuous zeros in genBP and allow it to be further referred to as L.

[0066] Step 7: The final region value sn can be calculated using the formula described in step 312:

[0067] Step 8: If at least three of the points including the point of interest and corresponding offset points compute a region value, the region values of the point of interest and those of the corresponding offset points computing a region value have the average taken and called sn. Otherwise, sn is set to zero.

[0068] In another embodiment, the region value can be calculated using entropy of the region surrounding the points of interest and the corresponding displacement points around each point of interest. The entropy of the region around the point of interest is calculated as follows:

[0069] Step 1: Initialize a threshold to quantize the texture in the region of interest around the point of interest and corresponding offset points.

[0070] Step 2: Determine the number of levels (level_map) in the region of interest by subtracting the center pixel intensity with all other intensities in the region of interest.

[0071] Step 3: level_map is further quantized by grouping the pixel intensities using the threshold determined in Step 1.

[0072] Step 4: The number of unique elements in each level_map group is calculated.

[0073] Step 5: The entropy-based region value around a point of interest is calculated as follows: where N is the length of elements in group 'i'.

[0074] Step 6: take average of sn at the point of interest and corresponding displacement points and assign the average value to sn.

[0075] In another embodiment, the region value can be calculated using the gradient information entropy around the points of interest in 'D' directions. In one embodiment, D includes 0, 45, 90 and 135 degrees. Gradient information can be calculated at multiple scales around the point of interest.

[0076] In one embodiment, corner locations and corresponding region values can be used to determine quality metric score generation. Quality metric score generation can be implemented as follows:

[0077] Step 1: Sort the points of interest based on their region values in descending order and note the corresponding point of interest locations.

[0078] Step 2: Let the distances between corresponding points of interest after rearranging them based on Step 1 be d = {d1, d2, d3, ..., dn-2, dn-1, dn}, where d1 is the distance between the first and second points of interest. dn will be zero.

[0079] Step 3: Let the distance weights be computed as the logarithm of distances: dwn = log (d).

[0080] Step 4: Compute the weighted index as: , where n is the number of points of interest

[0081] The final quality metric score can be computed using the following formula: where p is the number of points of interest

[0082] Other point of interest detection algorithms can be used, such that they can be calculated using Accelerated Segment Test (FAST) or Speeded Up Robust Features (SURF) algorithms.

[0083] EV_QM can be calculated separately for ocular and periocular regions and, furthermore, each ROI can be classified based on calculated EV_QM. ROIs can be matched in the order in which they were ranked if a progressive matching device (described below) is being used. Of note, the quality metrics described here can be used in fraud detection techniques. Reproductions of facial features of an eye or face, such as physical photographs or hard copies or digital images or videos of the genuine user replayed on a screen for malicious intent, are often of reduced quality (e.g., lower resolution, distorted, discoloration, with noise, blur, etc.) compared to a real physical presence at the time of scanning. After filtering out low-quality points of interest during the verification process, a low-quality reproduction will generally not have a sufficient number of recognizable points of interest and thus will not pass verification.

[0084] Similar techniques can be used to detect partial fraud as well. For example, a person may place a valid eye print over their eye in an attempt to pass the eye check process. When performing verification, the determined quality of the eye impression (e.g., recognizable points of interest) can be compared with the determined quality of periocular and other facial regions. If the difference in quality between the eye and one or more other regions exceeds a threshold, this may indicate the presence of partial fraud, and the check may fail. Other fraud detection techniques using the revealed quality metrics are understood.

Point of Interest Detection

[0085] Various point of interest detection algorithms can be used within a captured ocular-vascular, periocular and facial image region separately. For example, the Speeded Up Robust Features (SURF) algorithm is a “bubble” feature detection algorithm that can be used to identify areas within an image region centered on a point of interest. The Features from Accelerated Segment Test (FAST) algorithm is a corner detection algorithm that can also be used to identify points of interest within an image region. The vascular point detector (VPD) can also be used to identify points that interlock in the vasculature within an image region. Furthermore, VPD can also be used to detect points in the periocular region and face. In some embodiments, candidate points can also be identified at multiple image scales. For example, if the original image size is 100x100 (Scale 0), points can be identified from the original 100x100 image and also when the original image is resized to 50x50 (Scale 1) and 25x25 (Scale 2). Other region-specific configurations of point detectors can be considered.

Point Suppression

[0086] The number of candidate points that are produced using an interest point detection algorithm can vary based on the amount of texture and quality of the image. Furthermore, such algorithms can get stuck on noisy or irrelevant information (with respect to a given application), especially with non-PVD algorithms. Such noisy or irrelevant candidate points can be removed by candidate point suppression algorithms.

[0087] In one embodiment, a non-vascular point suppression (NVS) algorithm is used to determine candidate point quality. The algorithm is explained in the following steps:

[0088] Step 1: extract a Region (R) around a point of interest. Let the size of R be MxM.

[0089] Step 2: extract Local Patches (LP) within R of size NxN, where N<M. Designate the R-centered LP as Central Patch (CP).

[0090] Step 3: Local patches within the R region are sparsely populated.

[0091] Step 4: Calculate the histogram distributions of all LPs included in the R region and subtract the CP histogram from each histogram calculated for LPs.

[0092] Step 5: for each subtraction, calculate the spread score as the number of bins being occupied divided by the total available bins.

[0093] Step 6: Derive a combination of bandpass filters based on the distribution of pixels in the original image (Bandpass Filter Computing). These bandpass filters are used to measure the amount of false pixels such as glare against the vascular information pixels.

[0094] Step 7: Patches subtracted from Step 4 are filtered based on the derived bandpass filter and a corner response is calculated.

[0095] Step 8: Use scatter score and corner response in cascade main structure. Scatter score is a binary classifier: it either rejects or accepts the point. Corner response provides a normalized score between zero and one. Zero indicates non-vascularity while one indicates vascularity.

[0096] In another embodiment, a Bandpass Filter Computation is used to determine candidate point resistance. The bandpass filter is dynamically generated based on segmented whites from the eye region statistics. The band pass filter can be generated as follows:

[0097] Step 1: extract the green layer of the scleral region from an RGB image.

[0098] Step 2: Derive a histogram of the region. For example, using “N” bins to extract a histogram from a uint8 image.

[0099] Step 3: Calculate the inverse normalized histogram exponential derived in Step 2.

[0100] Step 4: suppress the exponential by a factor k. The k factor typically ranges from 0.1 to 0.3 and can be adjusted based on the application or data set.

[0101] Step 5: Calculate the inverse exponential response with integer values ranging from 1 to N.

[0102] Step 6: Concatenate the first 5 elements from Step 4 and the rest from Step 5. This provides Bandpass Filter 1.

[0103] Step 7: for Bandpass Filter 2, construct a Gaussian with normal distribution of scleral pixels (green channel).

[0104] Step 8: use Bandpass Filter 1 and Bandpass Filter 2 in parallel to establish the resistance of the vascular point.

[0105] This filter construction process is adaptive to image pixels. If the image is dark, most of the scleral pixels are near the lower tail of the histogram. In this way, Bandpass Filter 1 would have a greater response in relation to Bandpass Filter 2. This reduces the point score. Similarly, a saturated brightness image would have all its pixels in the upper tail and Band Pass Filter 2 will have a higher response, eliminating the point with a low score.

[0106] In one embodiment, the Local Binary Pattern (LBP) uniformity may be generated to determine the quality of a candidate point. Using the 8-bit LBP code, the number of transitions in bit form from 0 to 1 or vice versa determines the uniformity of the LBP code. An LBP code is considered uniform if it has less than or equal to ‘n’ transitions. The range of transitions is 0 to 8. In one embodiment, n equals 3. As an example, the LBP codes 00000000 with 0 transitions, 01111100 with 2 transitions, 01000001 with 3 transitions are uniform. Similarly, 0.1010000 with 4 transitions and 01010010 with 6 transitions are non-uniform. Candidate points that are uniform are retained in the model.

[0107] The above-mentioned quality algorithms for candidate points can be used in combination or separately for a given application or verification process.

Local Feature Descriptors

[0108] A feature vector around each point of interest can be generated using an image patch around the point of interest for the ocular-vascular, periocular and facial regions separately. Descriptors for a point of interest can be generated using single or multiple feature descriptor algorithms. The Fast Retina Keypoint (FREAK) algorithm is an example of a feature descriptor algorithm that can be used, for example, to generate descriptors for candidate points identified by FASTA algorithms. A FREAK descriptor can be, for example, a string of binary numbers that defines visual features surrounding a candidate point. Local Binary Patterns (LBP) and its variants such as Central Symmetric Local Binary Patterns (CSLBP) around candidate points are examples of feature descriptors that can be used to describe image patches in a neighborhood of a candidate point. Oriented Gradients Histograms (HoG), LBP Histograms (HLBP), CSLBP Histograms (HCS-LBP), Extended Multi-Radius LBP Standardized Histograms (PH-EMR-LBP), Extended Multi-Radius CSLBP Standardized Histograms, Histograms Patterns of Extended Multi-ray Local Ternary Patterns (PH-EMR-LTP) and Patterned Histograms of Binary Patterns after Dimensionality Reduction (PHBP-DR) are other examples of feature descriptors that can be used to describe image neighborhoods around candidate points. Other feature descriptor algorithms or combinations of algorithms can be used to generate local image descriptors for candidate points in an image region.

[0109] In one embodiment, the following steps are involved in generating PH-EMR-PBP descriptors:

[0110] Step 1: Calculate both LBP codes for concentric squares of 3x3 pixels and 5x5 pixels around each pixel to form a multi-ray LBP (MR-LBP) image (note that in contrast to regular LBP, the technique mentioned above uses a square or rectangular neighborhood, which may include or exclude corners) rather than circular pixel loci around the central pixel for deriving the LBP code). In a 3x3 pixel region, comparison of the center pixel with its eight immediate neighboring pixels generates a 3x3 LBP code. The result is an 8-bit code, whose bit values are either 0 or 1 (1 if the intensity value of the neighboring pixel is greater than the intensity of the central pixel, otherwise 0). Similarly, in a 5x5 pixel region, comparison of the central pixel with its eight immediate neighboring pixels (i.e., sixteen pixels) generates a 5x5 LBP code (results in a 16-bit code). In this way, MR-LBP has a 24-bit code (8 of 3x3 LBP code and 16 of 5x5) for a given pixel in an image.

[0111] Step 2: An MxM patch (outside of step 1) around each point of interest is segmented into NxN subregions that may have K overlapping pixels.

[0112] Step 3: Histograms for each 24-bit MR-LBP within each subregion are derived separately and concatenated to manage PH-EMR-LBP (note that in contrast to regular LBP, this histogram is calculated based on frequency of LBP bit locations in the patch, rather than that of the decimal equivalent of the binary code).

[0113] Parameter values for M, N, and K can be adjusted based on spatial frequency, resolution, and image noise.

[0114] In one embodiment, the following steps are involved in generating PH-EMR-CSLBP descriptors:

[0115] Step 1: Calculate both 3x3 pixel and 5x5 pixel central locus binary pattern codes (CSLBP) at each pixel to form a multi-ray CS-LBP image (MR-CSLBP). In a 3x3 region, comparing the intensity values of the diagonal pixels using the 8 boundary pixels generates a 4-bit 3x3 CS-LBP code (starting from the top left pixel, consider the boundary pixels as numbered 1-8 in a clockwise direction, the 4-bit CS-LBP code is generated by comparing pixels 1, 2, 3 and 4 with 5, 6, 7 and 8, respectively). Values are 0 or 1 (1 if the intensity value of 1 is greater than 5, otherwise 0 - similarly, for other combinations). Similarly, in a 5x5 region, the 16 diagonal pixels of the outer ring are compared to generate an 8-bit 5x5 CS-LBP code. In this way, an MR-CS-LBP has a 12-bit code (4 of 3x3 CS-LBP code and 8 of 5x5) for a given pixel in an image.

[0116] Step 2: An MxM patch (result of step 1) around each point of interest is segmented into NxN subregions that may have K overlapping pixels.

[0117] Step 3: Histograms for each 12-bit MR-CS-LBP within each subregion are derived separately and concatenated to manage PH-EMR-CS-LBP.

[0118] Similar to the previous descriptor, the parameter values for M, N and K can be adjusted based on the spatial frequency, resolution and noise of the image.

[0119] In one embodiment, the following steps are involved in generating PH-EMR-LTP descriptors.

[0120] Step 1: Calculate both 3x3 pixel and 5x5 pixel local ternary pattern codes at each pixel to form a multi-ray LTP (MR-LTP) image with a Step Size (SS). In a 3x3 region, comparing the center pixel with its eight immediate neighboring pixels generates a 3x3 LTP code. The result is a 16-bit code, whose values are 0 or 1 (1 if the intensity value of each neighboring pixel is greater than the intensity of the central pixel plus SS, otherwise 0; and 1 if the intensity value of each neighboring pixel is less than the intensity of the central pixel minus SS, if not 0). Similarly, in a 5x5 region, comparison of the central pixel with its next to immediate eight neighboring pixels (i.e., 16 pixels) generates a 5x5 LTP code (result is a 32-bit code). In this way, an MR-LTP has a 48-bit code (16 of 3x3 LBP code and 32 of 5x5) for a given pixel in an image.

[0121] Step 2: An MxM patch (result of step 1) around each point of interest is segmented into NxN subregions that may have K overlapping pixels.

[0122] Step 3: Histograms for each 48-bit MR-LTP within each subregion are derived separately and concatenated to manage PH-EMR-LTP.

[0123] Similar to the previous descriptor, the parameter values for M, N and K can be adjusted based on the spatial frequency, resolution and noise of the images.

[0124] In one embodiment, a PHBD-DR can be derived using the following steps:

[0125] Step 1: PH-EMR-LBP for a given point of interest is derived by setting the values of M, N and K to 9, 3 and 1, respectively. These settings manage a feature vector of length 384 (24x16; histograms of each of the 24-bit codes in the 16 subregions. Note that 3x3 subregions are possible in a 9x9 patch with 1 pixel overlap) with values varying from 0 to 9 (since there are 9 pixels in the 3x3 subregion).

[0126] Step 2: PH-EMR-CS-LBP for a given point of interest is derived by setting the values of M, N and K to 7, 3 and 1, respectively. These settings manage a feature vector of length 108 (12x9; histograms of each of the 12-bit codes in the 9 subregions). Note that 9 3x3 subregions are possible in a 7x7 patch with 1 pixel overlap) with values ranging from 0 to 9 (since there are 9 pixels in the 3x3 subregion).

[0127] Step 3: PH-EMR-LTP PH-EMR-LBP for a given point of interest is derived by setting values of M, N and K to 9, 3 and 1, respectively. These settings managed a feature vector of length 768 (48x16; histograms of each of the 48-bit codes in the 16 subregions. Note that 16 3x3 subregions are possible in a 9x9 patch with 1 pixel overlap) with values varying from 0 to 9 (since there are 9 pixels in the 3x3 subregion). SS is set to 5 after image intensity values are normalized to 0-255.

[0128] Step 4: Feature vectors 1, 2 and 3 are concatenated to form a feature vector of length 1260.

[0129] Step 5: Using an analysis of variance, only the 720 best features are retained. In some cases, since these highly variant features are pre-calculated, only these 720 features are generated during the enrollment and verification process in the interest of reducing computational complexity.

[0130] Step 6: Finally, a dimensionality reduction method such as principal component analysis (PCA) is used to extract PHBP-DR. A separate PCA analysis can be performed for ocular and periocular regions to produce 103 and 98 long feature vectors, respectively. For face, any periocular features can be used as is or a separate PCA analysis can be performed. Other feature vector lengths for ocular, periocular, and face are possible.

[0131] At the end of the biometric enrollment process, an enrollment model may include a set of candidate points and descriptors for an ocular-vascular ROI and a set of candidate points and those described for a periocular region. In some embodiments, an inscription model may also include a set of candidate points and descriptors for a face and/or a set of candidate points and descriptors for extended periocular regions. Note that the ocular-vascular and periocular ROI and model for left and right eyes are treated separately. Multiple descriptors for a set of points of interest are understood. Original and processed images that are used to produce models for the ocular-vascular face and periocular region may be discarded for security and privacy reasons.

Progressive Matching Device

[0132] In one embodiment of a biometric matching process, a final matching score is generated using a progressive matching device. As an initial step, local image descriptors are matched to find Corresponding-Point-Pairs between enrollment and verification models for both the ocular and periocular regions using a distance measure. As an example, a Euclidean distance between interest point descriptors can be calculated between the enrollment and verification descriptor vectors of ocular and periocular regions separately, and pairs below a certain distance threshold can be retained as Pairs-of-Interest. Point-Correspondents.

[0133] In the presence of noise or other aberrations, Corresponding-Point-Pairs may have multiple outliers or otherwise erroneous extra correspondences. By assuming a plausible homography between the inscription and verification image locations of Corresponding-Point-Pairs, outliers (spatially non-overlapping corresponding points after alignment under the assumed homography) can be removed from the Corresponding-Point-Pairs.

[0134] In some embodiments, a random sample consensus (RANSAC) or other outlier detection method may be used to determine the transformation necessary to align candidate points in a verification image with points in an inscription image, while rejecting outliers which do not fit into a hypothesized transformation between genuine correspondences, in terms of geometries of ocular regions of interest encoded in inscription and verification models. In some embodiments, RANSACs of different ROIs (such as ocular-vascular and periocular ROIs) may be performed separately, and collection of surviving inlier points may be performed for a final RANSAC for final score calculations and other related computations. In some embodiments, a minimum number of inlier points from certain or all subregions of interest (e.g., vascular patches and points seen on top of the scleral ROI) may be required before proceeding to final matching. Eventually, the score is generated by combining the number of inliers that are found after RANSAC (N), scale recovered from the transformation matrix (RS) and angle recovered from the transformation matrix (RA), using the homography fitted to the pair-of-pair locations. Point-Correspondents through RANSA or equivalent.

[0135] In one embodiment, the match score is calculated using the following formula: where Cx and Cy are correlations between x and y coordinate vectors of corresponding inlier points between enrollment and verification models respectively, N is the number of these aligned points, RA is the recovered angle representing the change in angle resulting from the transformation of point locations corresponding inlier checkpoints for the enrollment points for registration and RS is the recovered scale that represents the change in scale resulting from the transformation mentioned above. RA and RS are derived from the similarity or similar geometric transformation matrix resulting from RANSAC or similar operation. Other measures such as distance from the identity transformation/registration matrix are understood, especially if the regions of interest (such as ocular-vascular and periocular) are spatially pre-normalized.

[0136] In some embodiments, M-SAC, Group-SAC and/or Optima-RANSAC may replace RANSAC.

[0137] Correspondence scores can be generated for both the whites of the eye (SV) and the periocular region (SP) separately, combined (SVP) or in a sequential manner. In the progressive approach, different ROIs are progressively included in the matching device if there is insufficient information or quality for a robust decision in the initial region of interest. For example, if a certain source quality or definitive decision cannot be obtained using vascular information in the whites of the eye, the matching device can progressively add more information from locations belonging to the periocular (and potentially beyond, such as the nose and face). as necessary to achieve the desired certainty in asserting a biometric match.

[0138] In one embodiment, the Corresponding Point-Pairs of ocular and periocular regions are calculated separately and then combined for RANSAC to generate the final correspondence score.

[0139] In another embodiment, the Corresponding Point-Pairs of ocular and periocular regions are calculated separately and then combined for RANSAC to generate the final correspondence score. However, the final inlier generation is required to have at least N points from the eye region.

[0140] In an additional embodiment, the Corresponding Point-Pairs of ocular and periocular regions are calculated separately, their respective inliers are found through ROI-specific RANSACs separately, and then the collection of model elements filtered by specific RANSAC of region is combined for a final RANSAC to generate the final match score. However, the final inlier generation is required to have at least N RANSAC points from the ocular-vascular region. In some embodiments, a typical minimum value for N is 3.

[0141] In some embodiments, a progressive matching device is implemented to make a final decision based on comparing matching scores against thresholds. As an example: Step 1: if SV > 3.2, the user is authenticated and steps 2 and 3 are skipped. Step 2: If SVP > 3.4, then the user is authenticated and step 3 is skipped. Step 3: If SP > 3.2, the user is authenticated. End of process

[0142] In case the above progressive process fails to authenticate the user, another enrollment template from the enrollment bank may be recalled again, or a new verification image may be acquired, until a certain exit condition (such as a threshold time) is reached. Other combinations of SV, SVP and SP thresholds are understood. If one or more of the regions of interest are not available for scanning due to aberrations resulting from, for example, sunglasses or glare artifacts, the systems can use other available regions.

[0143] FIG. 5 shows an embodiment of a method for biometric enrollment and authentication using progressive matching, including an enrollment phase in which enrollment templates are created based on ocular-vascular and periocular regions, and a verification phase in which progressive matching can be performed. based on captured images and entry templates. The steps illustrated in FIG. 5 can be performed according to the various techniques described here.

[0144] In some cases, a single biometric characteristic may be divided into multiple ROIs, which are then progressively matched. For example, significant regions of a periocular region can be divided into n parts which are then progressively matched.

[0145] In one embodiment, the quality of some or all of the periocular ROIs may be measured and classified and the matching process may progressively add each ROI based on its respective classification as necessary to achieve the desired certainty in asserting a biometric match. .

[0146] In another embodiment, an image-based distance metric or similar detects user poses, lighting conditions or facial gestures that could distort some biometric regions of interest such as periocular. These variations can be induced during signup or added to a rolling model bank based on a model renewal policy. At verification time, the matching device may attempt to retrieve the most relevant models from the database based on the image similarity metric mentioned above.

[0147] In another embodiment, if the matching process finds a new pose or facial expression, the closest expression is used and a special case of progressive matching device is applied. For example, a special case may divide the periocular region into several segments using a clustering process, and within each segment the score and transformation matrices are generated. The final score can be determined through a weighted fusion of all individual segment scores. These weights are determined by the amount of deformation observed in all transformation matrices. Other score mergers are possible.

[0148] FIG. 5 illustrates one embodiment of a localized system for generating secure biometric templates and performing user verification in accordance with the techniques described herein. A user device 500 may include an image sensor 530, processor 540, memory 550, biometric hardware and/or software 560, and a system, but which couples various system components, including memory 550 for the processor 540. The user device 500 may include, but is not limited to, smart phone, smart watch, smart glasses, desktop computer, portable computer, television, gaming device, music player, mobile phone, laptop, palmtop, terminal with processing capability (smart ) or without processing power (dumb), network computer, personal digital assistant, wireless device, information device, workstation, minicomputer, mainframe computer, or other computing device that is operated as a general purpose computer or a special purpose hardware device that can perform the functionality described here.

[0149] Biometric hardware and/or software 560 includes an image processing module 562 for performing operations on images captured by image sensor 530. For example, image processing module 562 can perform segmentation and enhancement on images of the eye and surrounding facial area of a user 510 to assist in isolating vascular structures and other features of interest. The template module 564 creates biometric templates based on the vasculature image and can perform various obfuscation and scrambling operations on the templates. The verification module 566 validates the identity of a user 510 by performing matching operations between a biometric verification template formed when capturing a biometric reading and a previously stored enrollment template. In some embodiments, certain functionality may be performed on devices other than the user device 500. For example, a user device may instead include only a biometric sensor, such as a camera, and image processing and verification functions may be performed on a remote server accessible to user device 500 on a network, such as the internet.

[0150] Most commonly, the systems and techniques described herein can be implemented in a computer system that includes a subordinate component (e.g., such as a data server) or that includes a bridge component (e.g., an application server). ) or that includes a front-end component (e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an embodiment of the systems and techniques described herein) or any combination of such subordinate components, bridge and front. System components can be interconnected through any form or means of digital data communication (e.g. a communication network). Examples of communication networks include a local area network (“LAN”), a wide area network (“WAN”), and the Internet.

[0151] The computing system may include clients and servers. A client and a server are usually remote from each other and can interact via a communication network. The client and server relationship arises by virtue of computer programs running on the respective computers and having a client-server relationship with each other. Several modalities have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the invention.

[0152] The embodiments of the subject matter and the operations described in the present application can be implemented in a digital electronic circuit or in computer software, firmware or hardware, including the structures disclosed in the present application and their structural equivalents or in combination of one or more of them. Embodiments of the subject matter described in the present application may be implemented as one or more computer programs, that is, one or more computer program instruction modules, encoded on computer storage media for execution by, or to control the operation of data processing apparatus. Alternatively or in addition, program instructions may be encoded in an artificially generated propagated signal, for example, a machine-generated electrical, optical or electromagnetic signal, which is generated to encode information for transmission to receiving apparatus suitable for execution by a data processing device. A computer storage medium may be, or be included in, a computer-readable storage device, a computer-readable storage substrate, a serial or random access memory array or device, or a combination of one or more from them. Furthermore, although a computer readable medium is not a propagated signal, a computer storage medium can be a source or destination of computer program instructions encoded in an artificially generated propagated signal. The computer storage medium may also be, or be included in, one or more separate components or physical media (e.g., CDs, disks or other multiple storage devices).

[0153] The operations described in the present application may be implemented as operations performed by a data processing apparatus on data stored in one or more computer-readable storage devices or received from other sources.

[0154] The term “data processing apparatus” comprises all types of apparatus, devices and machines for processing data, including, by way of example, a programmable processor, a computer, a system on a chip, or multiple, or combinations, of the above. The apparatus may include special purpose logic circuitry, for example, an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus may also include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, a operating environment, a cross-platform execution environment, a virtual machine, or a combination of one or more thereof. The appliance and execution environment can realize several different computing model structures, such as web services, distributed computing infrastructures, and grid computing.

[0155] A computer program (also known as a program, software, software application, script, or code) may be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and may be implemented in any form, including a program alone or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program can, but does not have to, match a file on a file system. A program may be stored in a portion of a file that contains other programs or data (for example, one or more scripts stored in a markup language resource), in a single field dedicated to the program in question, or in multiple coordinated files ( for example, files that store one or more modules, sub-programs or portions of code). A computer program can be implemented to run on a computer or multiple computers that are located at one site or distributed across multiple sites and interconnected by a communications network.

[0156] Embodiments of the subject matter described in the present application can be implemented in a computer system that includes a subordinate component, for example, as a data server, or that includes a bridge component, for example, an application server , or which includes a front-end component, for example, a client computer having a graphical user interface or a Web browser through which a user can interact with an embodiment of the subject matter described in the present application or any combination of one or more such components subordinate, bridge or frontal. The components of the system can be interconnected through any form or means of digital data communication, for example, a communication network. Examples of communication networks include a local area network (“LAN”) and a wide area network (“WAN”), an internetwork (e.g., the Internet), and person-to-person networks (e.g., ad hoc person-to-person networks).

[0157] The computing system may include clients and servers. A client and a server are usually remote from each other and can interact via a communication network. The client and server relationship arises by virtue of computer programs running on the respective computers and having a client-server relationship with each other. In some embodiments, a server transmits data (e.g., an HTML page) to a client device (e.g., for purposes of displaying data to and receiving user input from a user interacting with the client device) . Data generated on the client device (e.g. a result of user interaction) may be received from the client device on the server.

[0158] A system of one or more computers may be configured to perform particular operations or actions by virtue of having software, firmware, hardware or a combination thereof installed on the system that in operation performs or causes the system to perform the actions. One or more computer programs may be configured to perform particular operations or actions by virtue of including instructions that, when executed by data processing apparatus, cause the apparatus to perform the actions.

[0159] Although the present report contains many embodiment-specific details, these should not be considered as limiting the scope of any inventions or what can be claimed, but rather as descriptions of specific features for particular embodiments of particular inventions. Certain features that are described in the present application in the context of separate embodiments may also be implemented in combination in a single embodiment. On the other hand, various features that are described in the context of a single modality can also be implemented in multiple modalities separately or in any suitable subcombination. Furthermore, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features of a claimed combination may in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0160] Similarly, although operations are shown in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to obtain desired results. . In certain circumstances, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments and it should be understood that the program components and systems described can generally be integrated together into a single software product or grouped into multiple software products.

[0161] In this way, particular modalities of object matter were described. Other embodiments are within the scope of the claims that follow. In some cases, the actions mentioned in the claims can be performed in a different order and still obtain desirable results. Furthermore, the processes shown in the accompanying figures do not necessarily require the particular order shown, or sequential order, to obtain desirable results. In certain embodiments, multitasking and parallel processing may be advantageous.

Claims (18)

1. Computer-implemented method, characterized by the fact that it comprises: receiving an image of a facial region of a user, the facial region including an eye and an area surrounding the eye; process the image to: (i) define an ocular image region including at least a portion of the eye in the facial region image and (ii) define one or more periocular image regions, each including at least a portion of the area surrounding the eye in the image of the facial region, wherein processing the image comprises enhancing at least a portion of the image using local gradient Gabor pattern (LGGP) by: calculating, at each of the plurality of angles, a Gabor phase image for at least a portion of the image; aggregating the calculated Gabor phase images to form a combined Gabor phase image; calculating, at each of the plurality of angles, a local gradient of the combined Gabor phase image; and retaining a maximum value of each local gradient to form an enhanced image; identifying a plurality of points of interest in at least one of the ocular imaging region and one or more periocular imaging regions; generating, for each point of interest, a feature descriptor based on a combination of a plurality of standardized histogram feature descriptors; and store the generated characteristic descriptors in a biometric model. 2. Method according to claim 1, characterized by the fact that generating the specific standardized histogram feature descriptor comprises generating a standardized histogram of extended multi-radius local binary patterns (PH-EMR-LBP), a standardized histogram. standardized histogram of extended multi-ray central symmetric local binary patterns (PH-EMR-CSLBP) or a standardized histogram of extended multi-ray local ternary patterns (PH-EMR-LTP). 3. The method of claim 2, wherein generating a feature descriptor for a specific point of interest comprises generating a PH-EMR-LBP feature descriptor by: defining an image region comprising the specific point of interest ; calculating a plurality of local binary pattern (LBP) codes for each pixel in the image region to form a multi-ray LBP (MR-LBP) image; dividing the MR-LBP image into a plurality of subregions; deriving a plurality of histograms comprising frequencies of each MR-LBP bit location within each subregion; and combining the plurality of histograms to form the PH-EMR-LBP feature descriptor. 4. Method according to claim 2, characterized by the fact that generating the feature descriptor for a specific point of interest comprises generating a PH-EMR-CSLBP feature descriptor by: defining an image region comprising the point of interest. resespecific; calculating a plurality of central symmetric local binary pattern (CSLBP) codes for each pixel in the image region to form a multi-ray CSLBP image (MR-CSLBP); dividing the MR-LBP image into a plurality of subregions; deriving a plurality of histograms comprising frequencies of each MR-CSLBP bit location within each subregion; and combining the plurality of histograms to form the PH-EMR-CSLBP feature descriptor. 5. Method according to claim 2, characterized by the fact that generating the feature descriptor for a specific point of interest comprises generating a PH-EMR-LTP feature descriptor by: defining an image region comprising the point of interest. resespecific; calculating a plurality of local ternary pattern (LTP) codes for each pixel in the image region to form a multi-ray LTP (MR-LTP) image; dividing the MR-LBP image into a plurality of subregions; deriving a plurality of histograms comprising frequencies of each MR-LTP bit location within each subregion; and combining the plurality of histograms to form the PH-EMR-LTP feature descriptor. 6. Method according to claim 2, characterized by the fact that generating the feature descriptor for a specific point of interest comprises: calculating a PH-EMR-LBP feature descriptor, PH-EMR-CS-LBP feature descriptor and PH-EMR- LTP feature descriptor for the specific point of interest; and combining the PH-EMR-LBP feature descriptor, PH-EMR-CS-LBP feature descriptor and PH-EMR-LTP feature descriptor to form a combined feature descriptor. 7. The method of claim 6, wherein generating the feature descriptor for a specific point of interest further comprises applying an analysis of variance to the combined feature descriptor to form a superior feature descriptor comprising a subset of features of the combined feature descriptor. 8. The method of claim 7, wherein generating the feature descriptor for a specific point of interest further comprises performing dimensionality reduction on the upper feature descriptor to form a feature descriptor for the specific point of interest . 9. Method according to claim 1, characterized by the fact that when generating a specific standardized histogram feature descriptor, local binary pattern or local ternary pattern feature descriptors are calculated using a neighborhood defined by a square with or without corners . 10. System, characterized by the fact that it comprises: at least one memory to store computer-executable instructions; and at least one processing unit for executing instructions stored in the at least one memory, wherein executing the instructions programs the at least one processing unit to perform operations comprising: receiving an image of a facial region of a user, the region facial including an eye and an area surrounding the eye; process the image to: (i) define an ocular image region including at least a portion of the eye in the facial region image and (ii) define one or more periocular image regions, each including at least a portion of the area surrounding the eye in the image of the facial region, wherein processing the image comprises enhancing at least a portion of the image using local gradient Gabor pattern (LGGP) by: calculating, at each of the plurality of angles, a Gabor phase image for at least one portion of the image; aggregating the calculated Gabor phase images to form a combined Gabor phase image; calculating, at each of the plurality of angles, a local gradient of the combined Gabor phase image; and retaining a maximum value of each local gradient to form an improved image; identifying a plurality of points of interest in at least one of the ocular imaging region and one or more periocular imaging regions; generating, for each point of interest, a feature descriptor based on a combination of a plurality of standardized histogram feature descriptors; and store the generated characteristic descriptors in a biometric model. 11. System according to claim 10, characterized by the fact that a specific standardized histogram feature descriptor comprises a standardized histogram of extended multi-ray local binary patterns (PH-EMR-LBP), a standardized histogram of binary patterns extended multi-ray central symmetric locals (PH-EMR-CSLBP) or a standardized histogram of extended multi-ray local ternary patterns (PH- EMR-LTP). 12. System according to claim 11, characterized by the fact that generating the feature descriptor for a specific point of interest comprises generating a PH-EMR-LBP feature descriptor by: defining an image region comprising the point of interest. resespecific; calculating a plurality of local binary pattern (LBP) codes for each pixel in the image region to form a multi-ray LBP (MR-LBP) image; dividing the MR-LBP image into a plurality of subregions; deriving a plurality of histograms comprising frequencies of each MR-LBP bit location within each subregion; and combining the plurality of histograms to form the PH-EMR-LBP feature descriptor. 13. System according to claim 11, characterized by the fact that generating the feature descriptor for a specific point of interest comprises generating a PH-EMR-CSLBP feature descriptor by: defining an image region comprising the point of interest. resespecific; calculating a plurality of central symmetric local binary pattern (CSLBP) codes for each pixel in the image region to form a multi-ray CSLBP image (MR-CSLBP); dividing the MR-LBP image into a plurality of subregions; deriving a plurality of histograms comprising frequencies of each MR-CSLBP bit location within each subregion; and combining the plurality of histograms to form the PH-EMR-CSLBP feature descriptor. 14. System according to claim 11, characterized by the fact that generating the feature descriptor for a specific point of interest comprises generating a PH-EMR-LTP feature descriptor by: defining an image region comprising the point of interest. resespecific; calculating a plurality of local ternary pattern (LTP) codes for each pixel in the image region to form a multi-ray LTP (MR-LTP) image; dividing the MR-LBP image into a plurality of subregions; deriving a plurality of histograms comprising frequencies of each MR-LTP bit location within each subregion; and combining the plurality of histograms to form the PH-EMR-LTP feature descriptor. 15. System according to claim 11, characterized by the fact that generating the feature descriptor for a specific point of interest comprises: calculating a PH-EMR-LBP feature descriptor, PH-EMR-CS-LBP feature descriptor and PH-EMR- LTP feature descriptor for the specific point of interest; and combining the PH-EMR-LBP feature descriptor, PH-EMR-CS-LBP feature descriptor and PH-EMR-LTP feature descriptor to form a combined feature descriptor. 16. The system of claim 15, wherein generating the feature descriptor for a specific point of interest further comprises applying an analysis of variance to the combined feature descriptor to form a superior feature descriptor comprising a subset of features. of the combined feature descriptor. 17. System according to claim 16, characterized by the fact that generating the feature descriptor for a specific point of interest further comprises performing dimensionality reduction on the upper feature descriptor to form a feature descriptor for the specific point of interest . 18. System according to claim 10, characterized by the fact that when generating a specific standardized histogram feature descriptor, local binary pattern or local ternary pattern feature descriptors are calculated using a neighborhood defined by a square with or without corners .