GB2574919A

GB2574919A - Biometric user authentication

Info

Publication number: GB2574919A
Application number: GB201903784A
Authority: GB
Inventors: Nikitidis Symeon; Kurcius Jan; Angel Garcia Rodriguez Francisco
Original assignee: Yoti Holding Ltd
Current assignee: Yoti Holding Ltd
Priority date: 2017-12-21
Filing date: 2017-12-21
Publication date: 2019-12-25
Anticipated expiration: 2037-12-21
Also published as: GB2574919B; GB201903784D0

Abstract

A method of authenticating a user 102 of a device 104 comprises: receiving data from a device motion sensor 124 (e.g. accelerometer or gyroscope) captured during a user-induced interval of motion of the device; receiving image data captured using an image capture device 126 of the device during the motion; and authenticating the user by determining whether characteristics of the user-induced motion match characteristics of an expected pattern, previously learned from and uniquely associated with the authorized user, and by analysing the image data to determine whether three-dimensional facial structure is present therein. The method may be combined with facial recognition. The image and motion data may be compared to verify that movement of the facial structure corresponds to the device motion. The motion data may be domain transformed to form a feature vector comprising cepstral coefficients to be inputted to a neural network trained to distinguish between feature vectors from different users, the training being based on feature vectors captured either from the authorized user, or from a group of users not including the authorized user. The authentication may include inputting the neural network output vector to a support vector machine one-class or binary classifier.

Description

Biometric User Authentication

Field

This disclosure relates to biometric user authentication.

Background

There are many situations in which a user of a device may be authenticated. For example, a local authentication may be performed in order to “unlock’' the device, to ensure that only an authorized user can access the device’s core functions and data. Another example is “online” authentication, in which, a user is authenticated with a remote authentication server in order to gain access to data or services etc.

Traditional authentication is based on usernames, passwords, PINs and the like, where the user is authenticated based on secret knowledge, which it is assumed is only known to that user.

Increasingly, biometric authentication is being used in various contexts. With biometric authentication, users are autnenucated eased on their human characteristics, such as their face, eye structure, fingerprint and the like.

Summary

The invention provides various authentication mechanisms based on device motion, and in particular on a user’s ability to replicate, by moving a user device, an expected device motion pattern that is uniquely associated with an authorized user. The characteristics of the device motion pattern that make it unique are referred to herein as “behavioural biometrics”, and the authentication process is ultimately checking for the presence or absence of matching behavioural biometrics.

A first aspect of the invention provides a method of authenticating a user of a user device, the method comprising: receiving a time series of device motion values captured using a motion sensor of the user device during an interval of motion of the user device induced by the user;

applying a tlomain transform to at least a portion of the time series to defe-mme at least one device motion spectrum in the frequency domain; and authenticating the user of the user device by analysing the said at least one device motion spectrum to determine whether the user-induced device motion matdies an expected device motion pattern uniquely associated 5 with an authorized user.

Here, it is the unique characteristics of the device motion spectrum that are used as a basis for the authentication. These are not necessarily measured directly from the spectrum (though that is not excluded), and in preferred embodiments cepstral coefficients are extracted from 10 the spectrum and used to perform the authentication.

In embodiments, the analysing step may comprise determining a set of logarithmic values from the said at least one device motion spectrum, the set of logarithmic values being used to determine whether the user-induced device motion matches the expected device motion .1.5 pattern, each of the logarithmic values being determined as a logarithm of a respective spectral coefficient of the said at least one device motion spectrum.

The analysing step may comprise determining a set. of cepstral coefficients by applying a second domain transform to the set of logarithmic values, the set. of cepstral coefficients 2.0 being used to determine whether the user-induced motion matches the expected device motion pattern.

The analysing step may comprise inputting to a neural network a device motion feature vector comprising the cepstral coefficients, and using a resulting vector output of the neural 25 network to determine wnether the user-induced motion matches the expected device motion pattern, the neural network having been trained to distinguish between such device motion feature vectors captured from different users.

Accordingly, the device motion spectrum may, in embodiments, be subject to a quite extensive series of transformations before the final result is arrived at. Each transformation stage- is ped oi rued in a way that retains enoughs intomianon about tne behavioural biometrics captured in. the original device motion data, will· the aim of placing that information in a form that makes it possible to reliably determine whether the captured behavioural biometrics match those of the authorised user.

More generally, the analysing step may comprise inputting to a neural network a device motion feature vector comprising coefficients of the said at. least one device motion spectrum or values derived from coefficients of the said at least one device motion spectrum. For example, the spectrum or parts of the spectrum (directly), or values derived from the spectrum in some other way could be inputted to the neural network as the device motion feature vector.

More generally still, a second aspect of the invention provides a method of authenticating a user of a user device, the method comprising: receiving motion data captured using a motion sensor of the user device during an interval of motion of the user device induced by the user; processing the motion data io generate a device motion feature vector; inputting the device motion feature vector to a neural network, the neural network having been trained to distinguish between device motion feature vectors captured from different users; and authenticating the user of the user device, by using a resulting vector output of the neural network to determine whether the user-induced device motion matches an expected device motion pattern uniquely associated with an authorized user.

For example, in embodiments of the second aspect, the device motion feature vector may comprise temporal motion values (motion values in the time domain) of or derived from the device motion data (without requiring calculation of the spectrum;. These could be raw values, or other time-domain values such as time averages.

All of the following applies equally to any of the above-mentioned neural networks in embodiments of the first and second aspect., whatever the form of the device motion feature vector it receives. That is, ‘the neural network’ in the following can be the neural network in any embodiment of the first and second aspect.

The neural network may be a convolutional neural network (CNN).

The analysing step may comprise classifying, by a classifier, the vector output of the neutral network as matching or not matching the expected device motion pattern (as a means of analysing the input device motion feature vector and thus the motion data from, which it is obtained).

The neural network may have been trained based, on device motion feature vectors captured from a group of training users, which does not include the authorized user.

The classifier may have been trained based on one or more earlier device motion feature vectors captured from the authorized user and corresponding to the expected device motion pattern.

The classifier may be a one-class classifier trained without using any data captured from any unauthorized user.

Alternatively, the classifier may be a binary classifier trained using one or more earlier device motion feature vectors captured from at least one unauthorised user.

The classifier may be a support vector machine iSVM).

Alternatively, the neural network has been trained based on one or more earlier device motion feature vectors captured from the authorized user and corresponding to rhe expected device motion pattern; wherein the analysing step may comprise determining whether the vector output exhibits a significant discrepancy from an expected vector output.

The neural netwoik may have been trained to reproduce, as vector outputs, inputted device motion feature vectors captured from the authorized user and corresponding to the expected device motion pattern; wherein the analysing step may comprise determining whether the vector output exhibits a significant discrepancy from the device motion feature vector inputted to the neural network.

The second domain transform may be a discrete eosine transform (DOT) or an inverse Fourier transform (IFT'j applied to the set. of logarithmic values.

The domain transform may be a Fourier transform. For example, the device motion spectrum may be a power spectrum, which is determined by determining a power of each spectral coefficient of a Fourier spectrum determined by the Fourier transform.

Multiple device motion spectra may be determined, each by applying a domain transform io a respective portion of the time series of device motion values, and analysed to determine whether the user-induced motion of the user device matches the expected device motion pattern.

The or each device motion spectrum may be filtered by a bank of filters, wherein each of the logarithmic values is a logarithm of an energy of a respective filtered version of the device motion spectrum determined by a respective one of those filters.

In embodiments, the method of the first or second aspect comprise: receiving image data captured using an image capture device of the user device during the interval of user-induced device motion; wherein the authentication step may comprise analysing die image data to determine whether three-dimensional facial structure is present therein.

The authentication step may comprise comparing the image data with at least some of the device motion values, to verify that movement of the three-dimensional facial structure, if present, corresponds to the user-induced device, motion.

Note that in embodiments of the first aspect, a neural network may not be needed at all. For example, it may be possible to achieve the desired result by training a classifier on any of the above-mentioned type of feature vectors, so that the classifier can classify the user based on the. feature vector directly.

Another aspect of the invention provides a user authentication system for authenticating a user of a user device, the user authentication system comprising: an input configured to receive- motion data captured using a motion sensor of the user device during, an interval of motion of the user device induced by the user; a one or more processors configured to execute computer readable instructions, and thereby implement the steps of the first or second aspect or any embodiment thereof.

Another aspect of the invention provides a computer program product for authenticating a user of a user device, the computer program product comprising computer-readable instructions stored on a non-transitory computer-readable storage medium and configured, when executed, to implement the steps of the first or second aspect or any embodiment thereof.

Brief Description of Figures

For & better understanding of the invention, and to show how embodiments of the same may be carried into effect, reference is made by way of example only to the following figures, in which:

Figure 1 shows a schematic block diagram of a system for authenticating a user;

Figure 2 shows functional blocks of a user authentication system;

Figure 3 shows functional blocks of a cepstrum determination component;

Figure 4 A shows the functions of an example user authentication system in greater detail;

Figure 4B provides a graphical illustration of some of the functions of Figure -1A;

Figure '5 shows a schematic block diagram of a first neural network in a training phase;

Figure 6 shows a schematic block diagram of the first neural network when used to authenticate a user;

Figure 7 shows a schematic block diagram of a second neural network for use in authenticating a user;

Figures 8 and 9 illustrate one example mechanism for detecting the presence of 3D facial structure on a moving image based on facial landmarks.

Detailed Description of Example Embodiments

Figure 1 shows a schematic block diagram of a system 100 for authenticating a user 102. The user 102 is a user ot a moveable user device 104, which is a mobile device such as a smartphone. In general, the mobile device 104 can be a hand-held or wearable device or anv other device which can be easily moved by the user 102. In accordance with the invention, the user 102 is authenticated based at least in part on his ability to replicate, with sufficient accuracy, an expected device motion pattern that is uniquely associated with an authorised user of the device 104, in order to determine whether or not the user 102 is the authorised user (motion-based authentication).

As described below, characteristics of the device motion pattern that make it unique to the authorized user are learned from the authorized user by training a classifier. It may be that any user, when setting-up the motion-based authentication, is asked to perform a specific type 10 of device motion as part of the process in which that user becomes associated with a unique device motion pattern. This could for example be moving a device from side to side or up and down a predetermined number of times. In that event, multiple users may be associated with broadly similar device motion patterns (e.g. up-down-up-down, left-right-left-right etc.), but which exhibit subtle but measurable variations that are unique to the user in question 15 Alternatively, a user may be given more freedom in setting -heir own device motion pattern (e.g. the user may be more or less free to move the. device- however be sees lit), in which case device motion patterns can be expected to have more pronounced differences between different users. Whatever constraints are placed on the form of the device motion pattern, characteristics that make it unique to a particular user, and that are consistently exhibited by 20 that user such that they can be used as a basis for identifying that user, constitute that user’s behavioural biometrics as that term is used herein. The. motion-based authentication can be seen as a form of biometric authentication based on such behavioural biometrics.

In an extension of the core concept, the motion-based authentication can be combined with an 25 image-based “liveness test”. For the liveness lest, image data is recorded simultaneously with the motion data, preferably rising a front-facing camera of the user device so as to capture a video image of the user’s face. As the device is moved, the user’s face - assuming it Is a real three-dimensional face -- exhibits a form of parallax due to its 3D structure, whereby features appear to move, at different speeds and by different amounts in the plane of 30 the image in dependence on their depth within the image capture device’s field of view (i.e, their distance away from the image capture device along its optical axis). Not only can this be used to verify the presence of actual three-dimensional facial structure (as opposed to a two-dimensional image held in front of the image capture device in an attempt to spoof the system - a form of “replay attack”), bin by comparing the movement exhibited by the facial structure in the image with the same motion data that is used to authenticate the user to check those data match, it can be determined with a high level of confidence whether the device motion has been induced by an actual living human holding the user device 104, in addition to authenticating the user 102, This is described in further detail later,

The user device 104 is shown to comprise at least one processor 122, comprising one or more CPU(s) and/or GPU(s) or other specialized processing hardware, such as FPGA(s) etc, In addition, the user device 104 is shown to comprise at least one motion sensor 124, an image capture device 126, a display .128 and a network interface 130 via which the user device 104 10 can connect to a computer network 106. such as the Internet. The processors) 122 fetches computer-readable instructions from a memory (not shown) and executes those instructions in order to carry out the functionality of the user device 104 that is described herein. The motion sensor 124, image-capture device 126, display 128 and network interface 130 are shown connected io the processor 122, which allows the process 122 to use these components 15 io carry out the functions described later. In some implementations, the user 102 is authenticated at a remote authentication server 108 (or other back-end authentication system) by way of a remote user authentication process between the user device 102 and the authentication server 108. The authentication server 108 is shown to comprise one or more processing units 109, such as CPU(s) and/or GPIJ(s) or other specialized processing hardware, such as FPGA(s) etc. The processing unit(s) 109 fetch computer-readable instructions from a memory (not shown) and execute those instructions m order to carry out the functionality of the authentication server 108 that is described herein. The user device 104 communicates with the remote authentication server 108 via the computer network 106 in order to carry out the user authentication process. In other implementations, the authentication can be a local user authentication that, is carried out at the user device 1.04 itself. In this case, the user authentication functions can be executed by the at least one processor 122 of the-user device 104.

The motion sensor 124 is used to capture device motion data, in the form of a time series of 30 device velocity or acceleration values, motion values. The motion sensor 124 can be any type of sensor that can be used to measure or estimate the instantaneous velocity and/or acceleration of the user device 104 at a particular time instant, either on its own or in combination with another sensor(s). Here, velocity refers to angular velocity, i.e. the rate with which the device rotates about up to three axes (linear velocity could be estimated by integrating acceleration it desired). That is, ά single motion sensor, such as an accelerometer, can be used for this purpose, or alternatively the user device. 104 may comprise multiple sensors which can be used in conjunction with each other to achieve the same outcome·. For example, a combination of an accelerometer and a gyroscope could be used to generate the 5 device motion data. Each device motion value can be a vector of up to six values (three acceleration, three angular velocity).

Figure 2 shows a high-level schematic function block diagram showing functional components of a user authentication system, which co-operate to perform the user authentication. In particular, Figure 2 shows a cepstrum determination component 202, a neural network 204 and a classifier 206. In the case of a remote authentication, the components 202, 204 and 206 represent respective parts of the functionality of the authentication service 108. That is, functionality implemented by computer-readable instructions when executed at. the back-end authentication system 108. However, in other implementations some or all of this functionality could be imprinted at. the user device 104 instead, tor example. Accordingly the authentication system of Figure 2 can be implemented at the back-end system 108. the user device 104 or as a distributed system formed of the user device 104 and the back-end 108.

The cepstrum determination component 202. is shown having an input connected to receive the device motion data (labelled 201) as generated using the at least one motion sensor 126 of the user device 104. The cepstrum determination component 202 determines, from the device motion data 201, a set of cepstral coefficients 203. With vector motion values, this can be done for each dimension separately or the vectors can be converted to scalar (e.g.

magnitude) values first. Cepstral coefficients are a way of representing a short-term power spectrum of a time-varying signal. This is described in further detail later but for now suffice it to say that, in order to determine the cepstral coefficients 203, the device motion data 201, which as noted is in the form of a time series of motion values, is transformed into a set. of short-term spectra in the frequency domain, from which the cepstral coefficients are then derived.

There are various ways of defining cepstral coefficients and the manner in which they are computed, depends on how they are defined. However, with reference to Figure 3, what these have in common is that a time-varying signal (in this case the time series of device motion values 201> or a portion thereof is transformed into a spectrum in the frequency domain, at block 302 in Figure 3, by applying a first domain transform such as a Fourier transform, e.g. Fast Fourier Transform (FFT;. A set of logarithmic values is determined from spectral coefficients of the frequency-domain spectrum, at block 304, and a second domain transform 5 is applied to the logarithmic values, at block 306, in order to determine the cepstral coefficients 203. Theoretically, an Inverse Fourier Transform 1 [FT) would be appropriate where the first domain transform is a Fourier transform, however in practice it may be equally viable to implement the second domain transform as a Discrete Cosine Transform (DCT) or similar.

Returning to Figure 2, the cepstral coefficients 203 are used to form a feature vector that is input to the neural network 204. The feature vector is denoted v_m and is also referred to as the input vector. The dimension of the input vector v,_n (the number of components it has) is denoted D_itJ. Each component of the feature vector v_in may simply be one of the cepstral IS coefficients, or additional processing may be applied to the cepstral coefficients to form the feature vector t?_jn. The input vector v_m embodies device motion information, in a form suitable for interpretation by the neural network 204. That, information has ultimately been derived from the captured motion data 201. Such a feature· vector is referred to as a device motion feature vector herein.

A resulting vector output 205 of the neural network. 204 is then used, by the classifier 206. to classify the user 102 as authorised or not authorised. To achieve this, the vector output 205 needs to capture sufficient information about the motion of the user de vice 104 as induced by the user 102 in a .form that can be interpreted by the classifier 206. so that classified 206 can 25 determine whether the user-induced motion of the device 104 matches the expected device motion pattern that is uniquely associated with the authorised user. In other words, the vector output 205 must, adequately capture the behavioural biometrics of the user 102. Various example architectures which can achieve this are described later.

Figure 4A is a function block diagram illustrating how the cepstral coefficients 203 can be generated in one embodiment. To aid illustration. Figure 4B provides a graphical illustration of some of the functions performed by (lie components of Figure 4A.

With reference to Figures 4A and 4B, at block 402 a Discrete Fourier Transform (DFT), which is implemented as a Fast Fourier Transform (FFT), is applied to the device motion data 201. In this example, a FFT is in fact applied on a per-frame basis. That, is, the device motion data 201 is divided into short temporal frames 412 and a FFT is applied to each frame

412 in order to generate a spectrum for that frame 412 (short-term spectrum). The output of the FFT block 402 is a set of (complex) spectral coefficients which are converted to a power spectrum 414 for the frame in question, at block 404, for example by taking the magnitude or (in this example) the magnitude² of each of these spectral coefficients. Block 402 together with block 404 corresponds to block 302 in Figure 3, in that they transform the time-domain device motion data 201 into a power spectrum in the frequency domain (device motion spectrum) 414.

After dividing the signal into frames and before calculating FFT windowing can be performed. Simply cutting the frames out introduces discontinuities at the beginning and the 15 end of the frame which in turn can result in undesirable artefacts in the frame spectrum. In order to avoid that, the frame can be multiplied element-wise by a specially designed window which effectively fades in and fades out the signal. An example of a suitable window is the Harm window. Windowing in the context of digital signal processing is known per se, and. it is not described in fuller detail for that reason.

In order to determine the logarithmic values, in this example, the device motion spectrum 414 is filtered by a filter bank 406 as shown in Figure 4B. The fjher bank 406 is a bank of filters (406A to 406D) each of which is tuned to a different frequency band and is designed to filter out the parts of the power spectrum 414 outside of that band. Four individual filters are shown but in practice there may be more filters than this, in this example, each of the filters 406A to 408D is a triangular filter, whose output is a multiplication of the power spectrum 414 with a triangular function centred on a particular frequency. The output of each of the individual filters 406A to 406D is a respective tillered version of the device motion spectrum 414, denoted 416A to 416D for filters 406A to 406D respectively.

At block 408, the energy of each of the filtered spectra 416A to 416D is computed. The result is a set of energy values 418, one for each filtered spectrum 416A to 416d, denoted El ίο E4 in Figure 4B. Each of the energy values El to E4 represents an overall energy of the filtered spectrum 416A to 4I 6D from, which it is computed. Each can be computed by summing at the spectral coefficients of the filtered spectrum in question, or some other computation to determine an overall value representative of the spectrum energy (or magnitude).

At block 410 m Figure 4A, a logarithm of each of the energy values El to E4 is computed, denoted log(El) to log(E4).

Blocks 406, 408 and 410 in conjunction correspond to block 304 in Figure 3.

Finally, in order to compute the cepstral coefficients 203, in this example a DCT transform is applied, at block 306, to the set of logarithmic values to obtain the set of cepstral coefficients 203.

A viable option is to compute the so-called Mel-frequency cepstral coefficients (MFCCs). These are used in audio processing, and a characteristic of MFCCs is that the filter-bank is chosen so as to transform the power spectrum to the so-called Mel scale. This has the effect of weighting different frequencies according to how responsive the human ear is to those frequencies. There is no particular need to do this in the present context. However, in practice it has been found that the device motion spectrum is confined to a relatively narrow part of the spectrum, such that the effect of the Mel transformation is negligible.

Accordingly, although in principle there is no particular benefit to using MFCCs, they are nonetheless a viable choice, and a practical benefit is that, due to their adoption in audio processing, they can be computed using existing library functions and the like.

These operations are performed for each flame 412 of the time series of device motion values 201 to obtain a set of cepstral coefficients 2.03 for each frame 412. As indicated above, it is these cepstral coefficients that are used to form, the input vector that can be understood by the neural network 204. The input vector can for example comprise cepstral coefficients determined for multiple frames of the device motion data 20.1 as predetermined dimensions of the input vector iq.,. and could potentially have a large number of values (high dimensionality).

There are various viable neural network architectures that can provide a meaningful vector output 205 based on such an input vector v._n.

A first neural network 204A, having a first architecture, will now be described with reference to Figures 5 and 6.

Figure 5 schematically illustrates some of the principles according to which the first neural network 204A is trained. The first neural network 204A is shown io comprise an input layer 502, at least one hidden layer 504 and an output layer 506. The hidden layer 504 is shown connected to receive inputs from the input layer 502. and the output layer is shown connected to receive outputs from the hidden layer 504. Example arrangements of these connections are described below. For a neural network with multiple hidden layers, the first of the hidden layers connects to the input layer 502 and the last of the hidden layers connects to the output layer 506, with each but the first hidden layer connected to receive inputs from the previous hidden layer. Accordingly, with three or more hidden layers, at least one (intermediate) hidden layer is connected to both receive input from the previous hidden layer and to provide outputs to the next hidden layer.

The neural network layers 502, 504 and 506 are shown to comprise respective nodes, with each node of the hidden layer 504 having an input connected to an output of each of the nodes of the input layer, and each node of Ute output layer 506 having an input connected to an output of each of the nodes of the hidden layer 504. This is an example of a ‘‘feedforward'’ arrangement, and alternative arrangements may be viable (see below). In this example, there is only one hidden layer but as will be appreciated with multiple hidden layers intermediate hidden layers may be connected to each other in a similar manner.

Purely by way of example the input layer 502. is shown as having nine nodes, the hidden layer 504 three nodes and the output layer 506 five nodes, however is expected that in practice each layer could have significantly more nodes than this. As will become apparent, in this particular example, it may be beneficial for the hidden layer 504 io have fewer nodes than the input layer 502 in order to achieve a level of dimensionality reduction, particular where the dimension of the input vector is high.

Herein the input vector is denoted:

where denotes the i¹· component of the feature vector v_in and corresponds to one of the cepstral coefficients. The dimension of the feature vector Vj_n is denoted £>_ilv Nodes within the neural network layers 502, 504. 506 are labelled as (l,j) - denoting the /th node in layer - with / — 0,1,2 respectively; hence:

· The j^ib node in the input layer 502 is labelled (0,/):

* The A node in hidden layer 504 is labelled (1, /);

• The j* node in the output layer 506 labelled (2,jY

Further details of the specific architecture of Figure 5 are described below.

First., a more general point is made, which is that, ultimately, a neural network is a combination of an algorithm executed on one or more processing units and a set of electronicallv-stored weights. with the algorithm being configured to process innut data, i.e. the input vector v_in, based on the electronically-stored weights, in order to generate output data, in the form of an output vector denoted:

of dimension D_out. The algorithm processes the input vector v,_n in stages, which are represented by the layers of the neural network. At each stage, the algorithm applies weights in the matter described below to the data being processed at that stage. The weights are stored in memory and selected during training in a recursive process performed with the objective of minimizing a loss function applied to die input vector ιη_η and the output vector v_ouj- and denoted:

U/Wn/.knri where ,f (*) is an objective (target) function such that, when L(r(v_in), v_out) is sufficiently minimized, v_out & /(V;_n). 'That is, is the desired output of the neural network in response to v_out, or to put it another way the function the neural network is trained to model, which for conciseness is denoted:

A-w: = (yo»->yD_0ut-i) and which also has dimension D_out.

The training terminates when selected stopping criteria are met. The choice of stopping criteria is context-dependent, and they are chosen so as to minimise the loss function enough to achieve the objective but not so much as to cause significant over-fitting. In a nutshell, over-fitting means that the neural network has “memorized” the training data very well, but to the detriment of its ability to generalize that, learning to data it has not seen before. At. this point, the neural network can be said to have learned the target function f (*), and a key benefit of a properly-trained neural network is that it is able to approximate the objective function /'(*) for input vectors that exist in the same vector space as the training vectors but which the neural network has never encountered before; that is, the natural network is capable of generalizing from a relatively narrow set of training data when properly trained.

The loss function is evaluated at the output layer, in that it is applied to the output of the output layer v_out (viewed the other way round, the output layer can be defined at the layer at which the loss function is evaluated in training). By contrast, the output of the hidden layer(s) is not considered explicitly during training and, unlike the output layer output layer v_out, the output of the hidden layer(s) does not directly contribute to the loss function L(f(v_ia),v_ont), but only indirectly contributes to it in that the output i?_out of the output layer varies in dependence on the output of the hidden layerfs).

Ultimately, a trained neural network means the algorithm m combination with a set of electronically-stored weights that have been selected to achieve an objective with respect to a loss function evaluated at. die output layer. Once the weights have been learned on a particular computing platform, they can be moved to or duplicated between different computing platforms in order to make use of those weights. Moreover, in some cases, a subset ot the weights might be discarded altogether once training is complete; as will become apparent, this is viable in the following example, as it is a vector output of the hidden layer that, is used once training is complete to determine whether or not the user from which v_h has been captured is authorized.

A common choice for the loss function is mean squared error (MSE);

Iff O^fin)· ^ourJ jX i^out 1 tAiis)i '-''out

However, as discussed below, there are viable alternative loss functions, and alternative loss functions may give a performance benefit in some cases in the present context.

Each node of tire neural network represents a computation that is performed by the algorithm on a particular set of inputs using a particular set of weights. Each node can thus be seen as a functional module of a computer program embodying the algorithm.

Each hidden layer and output layer node (l, j) computes its respective output as:

// \~^Ί \ \ °i.j — 9 I I / * ii.j.i | + bjj j (1)

That is, as a function$(*) (the activation function) of a sum of the inputs (0-,-.-1,0-./.2,.,.) received at that node (i,/), weighted according to that node’s set of weights (Wjj ,, W;j ...), and an (optional) bias term &,·>. The activation function $(*) is generally a non-linear function, to allow the neural network to learn non-linear functions. It is relatively unusual in modem neural networks for the activation function to differ between layers, and even less usual for it to differ between nodes in the same layer because that can reduce the extent to which the neural network can be implemented efficiently using matrix computations; nevertheless, there is no requirement for the activation function to he fixed in this way and it can vary between layers and/or neurons within layers. Generally a differentiable activation function (or at. least a function that is differentiable over a desired set of input values) is chosen to allow' training based on back propagation, in which the loss function is minimized by iteratively computing partial differentials of the loss function with respect to the weights and tuning the weights based on the results.

The bias term 0/, for each node (j, I) can be zero or a ηυη-zero value and which can vary between different nodes. The bias, term h,,· can he fixed or it can. be adjusted during training along with the weights.

For the first hidden layer, the inputs are simply the (unweighted) components of the input vector Vj_n. By convention, a neural network is often represented as having an input layer that transparently ’relays’ the input vector to each node of the (first) bidden layer, and the first layer at which processing is actually performed; art equally viable perspective is to think of the input layer as the input vector itself; either way, the input layer represents the starting point for the algorithm. This is immaterial but, when it comes to terminology, what is material is that, herein, the first neural network layer at which the input vector ι?_ίη is processed in accordance with equation (1) (corresponding to the first substantive stage of the algorithm) is referred to as the first hidden layer, or simply the hidden layer if there is only one; that is layer I = 1 in the notation introduced above.

Accordingly , the inputs to each node of the (first) hidden layer (I = 1) are simply:

_έ = x-i for j ~ 0,..., Di_n 1 where x_f is the /.th component of the input feature vector i?_iri as noted above.

In a simple “feedforward” network, each node (j, I) of each layer t > 1 is connected to each node of the previous layer I — 1 (only.}. This is a viable arrangement m the present context, 10 but alternative arrangements, e.g. which incorporate feedback connections, may also be viable, and for the avoidance of doubt the term neural network is not restricted to a simple feedforward arrangement but encompasses oilier arrangements too.

For instance, a viable alternate is a convolutional neural network (CNN), and anv of the described technology can be implemented with a CNN. CNNs are known per se and have various distinguishing characteristics that ate also known per se. A CNN can be trained to perform any of the neural network functions disclosed herein, as will be appreciated.

The first neural network 204A is trained based on training data captured from a group of N 2.0 training users 512 (user U to user N -- 1). Ln this example, the dimension of the output layer (i = 2) is D_Oilt ~ N, i.e. one node per training user such that node (2,?i) corresponds to the n^en training user.

The training data for each training user is device motion data (recording device motion induced by that user) from which cepstral coefficients are derived, and used to form device motion feature vectors to be inputted to the neural network 204A, in the manner described above.

The device motion feature vectors are inputted at the input layer 502. of the first neural network 204A, in. a training phase, and the first neural network 204.A is trained by adjusting the weights applied at the hidden and input layers 504, 506 in a systematic way to minimise the loss function L(j‘(υ·_ιη), v_out) until the stopping criteria are met. In this example, the objective function is a “one-hot’’ categorisation function, i.e·. such that:

for any input vector captured from user n (denoted v? ). That is, is an Af-dimensional vector having all zero components, except for the nth component which is 1. In other words, the neutral network is trained to strictly classify each device motion feature vectors according 5 to which user it has been captured from.

Accordingly, when training is complete, for each of the N training users, any device motion feature vector captured for that user results in an output o₂,_n ^x 1 at the output layer node (2,n) corresponding to that user, and an output o_Zj_xn » 0 at all other output layer nodes,

In this context, although MSE may be a viable loss function, it is expected that it may be possible to achieve a performance, benefit, in some cases, with a cross-entropy error (CEE) loss function. With a one-hot objective function /’’(*), the cross entropy loss function reduces ~——/ tog(j'i) , I.·' Q J j t AaaanwI

As indicated, training can be performed using back propagation to reduce the loss function (however it is defined) until the stopping criteria are met. Any suitable backpropagation algorithm can be used, for example gradient descent. Backpropagation and gradient descent are well known per xe so further details are not discussed herein.

Regarding the architecture of Figure 5. it is important to note that there is no requirement for the user who is being authenticated (user 102 in Figure 1) to be part of the training group 512, and in practice there will be far more end-users than there are training users 512. That is, in general any user being authenticated will not have provided any of the training data used to train the first, neural network 204A. This exploits the ability of the first neural network 20-1 A, once trained, to understand” feature vectors from users it has never encountered before, i.e. the ability of the first neural network 204A to generalize from a narrow set of training data For users it has not seen before, the output vector x»_out of the neural network 502A could be seen as a measure of how' 'similar’' the unknown user is to each of the training users.

However, it is also important to note that, in the present example, the output vector need not be computed at all and the output layer can be discarded altogether after training, for reasons that will now be explained with reference to Figure 6,

With reference to Figure 6, once the first neural network 204A has been trained in this manner, the user 102 is actually classified as authorized/unauthorized, by the classifier 206, using the output of the hidden layer 504, which is a vector i?_b of dimension (the dimension of the hidden layer 504, i.e, the number of nodes it contains), which would normally form the input to the output layer 506, and which is denoted:

n = (4)

As indicated, preferably < D_in which has the effect of dimensionality reduction, i.e, reducing the number of vector components that represent the device motion.

The classifier 206 classifies the user 102 as authorized or not authorized based on this output vector v_ouj:.

The classifier 206 may be a one-class classifier. A one-class classifier refers to a classifier hat is trained, io perform a binary classification (here: autborized/not authorized), using data from a single class only (here- authorized). In other words, a one-class classifier in this context can be trained using only device motion data from a user who is authorized to use the user device 104, without any training data from any other (unauthorized) user. This means that the classifier can recognize a user as unauthorized, even though n has only ever seen” training data from the authorized user and has never seen training data from an unauthorized user. For example, the classifier 206 may be a one-class support vector machine (SVM).

Alternatively, the classifier 206 can be a binary classifier that is trained using data of other users, who are unauthorised users in this context and then fall into the second class. Where data for a large number of users is available, a selection of the data can be selected for this purpose, at random or using some other suitable selection mechanism.

This classification based on the hidden layer vector output is the means by which it is determined whether or not the unique characteristics of the device motion match those of the expected device motion pattern associated with the authorized user, using information about that device motion captured in the hidden layer vector output 1+-.Figure 6 shows a second neural network 204B having an alternative architecture. As before, the neural network 204B has an input layer 702, at least one hidden layer 704 and an output layer 706. All of the description above applies equally to the second neural network 204B. but modified as described below.

The underlying training model is quite different in this example. Here, the neural network

204B is trained using training data captured from the authorized user only. That, is, a neural network is trained independently for each authorized user.

The objective function is this second exampie is an identity function:

/(Vi_n) ~

That is., a function which simply returns its input as its output for ail values. Accordingly, training is performed until the neutral network in able to reproduce each training input vector at the output layer 706, such that v_out « v_in.

Once trained, any time the authorized user provides an. input vector subsequently, the output 15 of the output layer 706 of the second neural network 204B is expected to approximately match that input vector. However, if some other unauthorized user provides an input vector, the output of the output layer 706 is expected to exhibit a discrepancy from that input vector that is significantly greater than any discrepancy between the authorized user's input and output vectors, i.e. between v_in and v_out. By detecting the presence or absence of such 20 significant discrepancies in the output vector v_out, it can determined whether or not the user is question is the authorized user. It is the. presence of absence of any significant discrepancy that indicates whether or not the device motion matches the expected device motion pattern associated with the authorized user.

Note that, in this example, the full neural network 204B, including the output layer 706, is used once trained as part of the authentication process.

This could be a classification, by a suitably trained classifier 306. e.g. a one-class classifier (e.g. SVM) as in the above, bat. in this case trained based on the vector output v_ouf- of the 30 output layer 706 for the authorized user. The principle here is essentially the same as the first example, in that the classifier is used to classify vectors as matching the authorized user or not using only training data from the authorized. Here, the input to the classifier could simply be t?_oul, on the assumption that v_out alone is sufficient, io convey such discrepancies, or the classifier could receive data of both v_out and V|_n as an input. For example, the input could be v = v_OiJi -- v_in or some function thereof, or some other measure of the difference between and v_in.

An extension of the above techniques will now be described, in which the behavioural biometrics authentication is combined with a form of liveness detection to provide a secure authentication mechanism that is robust to spoofing, whilst minimizing the amount of input needed from a user. This is because the data needed to perform the authentication and the liveness check (device motion data and image data) are captured simultaneously as the user 10 moved the device 104, and the. device motion data is used for both authentication and liveness detection. One benefit of the technique is that it can provide biometric authentication in circumstances where it would not be possible to use facial recognition due. to the motion of the face in the image data. However the technology is not limited in this respect, and could for example be combined with facial recognition where possible to provide an additional layer of authentication securi ty.

As indicated, the motion-based authentication can be combined with what is referred to herein as structure based face anti-spoofing”, which refers to a set of methods based on the observation that a 2-dimensional still picture of a face is the result of the projection of the 3D 20 face on the 2D image plane. This observation enables a technique to be used where, given as few as two video frame images from different angles of the same face, it is possible to distinguish robustly between a static print attack and genuine input. Additional techniques to solve problems that arise when trying to automate frame selection from a video stream are also described below. A different method is also introduced, appropriate for different types of 25 malicious input, such as replay attacks, to be countered.

Consider two still pictures of an individual’s face where, the camera is at a different angle in both frames, for example as shown in Figure 9. A technique, referred to herein as landmark detection, is used to robustly locate key points on the face. In this section, only the following 30 set of points is exploited:

* Pl: Left eye centre * P2: Right eye centre » P3: Tip of the nose • P4: Left lip commissure (i.e. left corner of the mouth) • P5: Right lip commissure (i.e. right corner of the mouth)

An additional point P6 is defined, as the intersection between P1P5 andP2P4 (i.e. the line between points Pl to P5 and the line between points P2P4). This point P6 defines the centre 5 of the face.

A distance metric in the form of a displacement vector P=P6P3 (posture vector), i.e. a vector separation between points P6 and P3, is determined. Vector P6P3 behaves very differently when drawn upon a static printed picture of a person's face.

Figure 8 illustrates how vectors can be drawn between the various reference points previously discussed. Figure 9 illustrates these techniques applied to an image of a real face, shown along two different hnes of sight, in side-by-side images. This allows a vector difference such as the posture vector P to be used to determine- whether the captured movement of a face in a 15 moving image is that of a 3-dimensional object or a flat 2-dimensional image of a face. The moving image is capturing user device camera 108.

Reference points on either side of the eyes Rl, R2 (right eye) and Li, L2 (left eye) are identified and tracked as the face in the moving image exhibits motion, as are mouth reference points P4 and P5 or. either side of the mouth (left and right respectively). Central eye points Pl, P2 are determined for each eye as the mid-point between RI and R2, and LI and L2 respectively. A point P6 is then determined as the intersection of the line P1-P5 and the line P2-P4. That is, the lines from the approximately centre of each eye to the point lying approximately on the opposite corner of the month.

The intersection point P6 lies near to point P3, on the tip of the user's nose. A difference vector, which is the posture vector P ~ P6P3 ~ P3 - P6 (or P6 -- P.3 — the choice is immaterial is determined and. tracked. As noted, for a moving image of a 3-dimensional human face, i.e. a real face, this vector difference P is expected to vary in a certain way as a user moves their 30 head from side-to-side or tilts it up and down. This is because the point P6 corresponds to a point on. the nose in 3-dimensional space that is forward of the point I along the user’s facing direction. However, if what is captured in the moving image is itself an image of a face, i.e. <

2-dirnensional representation of a face on a flat surface, any variations in the difference vector P will be measurable different from those of a 3-dimensional face. Thus, by looking in variations at. the difference vector P as the user moves his head or (preferably) his device as the moving image is captured it is possible to distinguish between a real 3-dimensional face and a 2-dimensional facial image. As noted above, this can be natural movement that is not explicitly requested, or the movement may be requested by outputting suitable instructions at the user device 104.

Figure 9 shows two static images of the same face captured from different angles, which are video frames F0 and Fl respectively. The points P1-P6 are shown in each of the images, and the changes in their relative locations due to die rotation of (in this case) the user's camera is evident.

The posture vector P in frames FO and Fl is denoted P0 and Pl respectively.

An additional vector S --- P0 - Pl is determined. A vector norm of the vector S is determined as a score for the inputs. By introducing an empirically calculated threshold Ί, is possible io determine whether or not the input is genuine, where the entity only passes the test if norm(S) > T. The value norm(S) is a measure of a scalar difference between vectors P0 and PI.

I his represents an extremely efficient anti-spooling test, which can be implemented tisirm as few as two video frames FO, FT and computationally efficient image processing.

Scale and Rotation Invariance

A problem can arise where, if the face is significantly closer to the camera in one of the frames than the other, a printed static might fool the anti-spoofing detector. For this purpose, a concept of scale invariance is introduced.

To implement scale invariance, a respective distance between the centres of the eyes is determined for each of the frames FO, Fl, and applied to the posture vector of that frame as a scale factor. That is, the posture vector for that frame P is scaled by that factor to generate a new vector P' for that frame that is insensitive to the distance between the subject and the camera:

P'-P/|P1P2|

So for frames FO and Fl respectively, PO and Pl are scaled by the distance between the eyes in that frame respectively.

Another problem that can arise is when the input is static printed picture where the subject nas his head turned to the side by -30 degrees. Then an attacker could, tool the anti-spoofing system by rotating the primed picture by 90 degrees between the two frames and get a score S greater than the. threshold T. To address this, rotation invariance is introduced, where an angle of the line (P1P2) relative to a horizontal direction of the image is measured and the posture vector P is rotated by that angle:

P' = rotate(angle(P 1P2), P)

The points Pl, P2 at the respective centres of the eyes are suitable reference points to provide both the scalar and rotational invariance because, although they rnav move as the user moves his head, their positions on the face are fixed (unlike, say, points P4 and P5 on the mouth corners, whose positions on the face may vary due to facial expressions). Other such points on the face (i.e. at fixed locations on the face) can also be used.

Orientation Normalisation

It might also be the case that if the angle between the two frames is too high then even a static, printed input might get a score S large enough to pass the anti-spoofing test.

This problem can be solved by normalizing depending on the actual change in the orientation angle between the two frames before thresholding.

This requires an estimation of the orientaiion/posture of the user's face in some way, for example as described in the next section.

Frame Selection based on 3D face pose estimation

If the two frames FO, Fl are selected from a continuous input stream derived from a single camera, one of the main problems that arises is how to select the appropriate frames.

This is important since if the orientation change between the two selected frames is too small then all the inputs will be incorrectly classified as attacks instead of genuine faces. A robust pose estimation method allows an appropriate pair of frames to be selected.

3D Face Pose Estimation

To estimate the 3D face posture from a single 2D image (video frame} a generic 3D face shape model is used as a reference. Using this 3D reference shape model, a projection matrix that was used to capture the 2D image is estimated, by seeking the 2D-3D correspondences between the same landmark points in the 2D image and on the surface of the 3D face model.

The 3D to 2D projection matrix, once determined, is exploited in. order to compute the Euler angles that specify the 3D face pose on every video frame in terms of its roll, pitch and yaw angles. Euler angles are a known mathematical construct, which are also used to describe the orientation of one frame of reference relative, to another e.g. as a triplet of angular values (such as pitch, roll and yaw).

Having obtained the 3D face pose on every video frame, two 2D images are selected as having an orientation change significant enough to facilitate the two-frame structure antispoofing method. That is, the two frames that are selected are such that 3 difference between the respective orientations of the face in those frames exceeds a suitably threshold chosen to match the parameters of the test itself.

Continuity of input .A potential attack against the anti-spoofing system might be to have two separate static photos of the same person from two different angles and substitute one for the other during video capture.

This can be prevented by introducing additional checks in the image processing, namely to make sure that the face-detection box position and landmarks are typical of a single object moving at reasonable speeds throughout the input. This means that the face can never be occluded, there can never be more than a single face at a time and it cannot teleport from one point to another during the input teed. To this end, a speed measure (e.g. scalar speed or velocity vector) can be determined for the face across the video frames F, and compared with a speed threshold. The entity fails the test if the speed threshold is exceeded at any time.

Extension to replay attacks

So far. a relatively cheap and simple attack whereby the attacker displays a video of a live subject from a device/screen in front of the camera has not been addressed.

To address this type of attack, an additional requirement can be introduced, namely that the user's head remains almost still while he is capturing his face from different view angles as he is moving the hand held camera 108. That is, the user is required to rotate the camera 108 whilst keeping his head still.

Thus it is possible to discriminate between a genuine login attempt and a replay attack by measuring a correlation between the face pose change (that is, changes in the posture vector P) in time relative to the way the device is moving in the actual world by the user. In order to estimate the device location and orientation in the real world, available inertial sensor(s) 126 on the user device 104 are exploited. The vast majority of modern mobile devices are equipped with such sensors (e.g. an accelerometer, gyroscope and magnetometer).

Advantageously, the same motion data 201 that is used to authenticate the user 102 can be used to detect reply attacks in this context.

1'he 3D race pose is estimated on every irame by the method described above. Thus it is possible to practically transform the problem of liveness detection against replay attacks as a time series similarity measurement problem where it is determined whether the input video stream shows a genuine login attempt (live face) if the similarity score is higher than an experimentally defined threshold.

Finally, in order to ensure that the change in the face pose is caused by the displacement of the capture device 108 and not due io head movement, a variance of the respective sensor data is determined, and a requirement can be introduced that the determined variance is above a specified threshold for the entity to pass the test.

2b

As will be appreciated, this is just one example and there are various ways in which the device motion data 201 can be used, both to authenticate the user 102 and to verify that if observed motion in a captured video correlates with the sensed device, motion sufficiently

Although specific embodiments of the inventions have been described, variants of the described embodiments will be apparent. The scope is not defined by the described embodiments but only by the accompanying claims.

Claims

1. A method of authenticating a user of a user device, the method comprising: receiving motion data captured using a motion sensor of the user device during an interval of motion of the user device induced by the user;

receiving image data captured using an image capture device of the user device during the interval of user-induced device motion; and authenticating the user of the user device by (i) analysing the motion data to determine whether characteristics of the user-induced device motion match characteristics of an expected device motion pattern which are uniquely associated with an authorized user and which have been learned from the authorized user prior to the interval of user-induced device motion, and (ii) analysing the image data to determine whether three-dimensional facial structure is present therein.

2. A method according to claim I, wherein the user of the device is authenticated based on the analysis of the motion data in combination with facial recognition,

3. A method according to claim 1 or 2, wherein the authentication step comprises comparing the image data with the motion data, to verify that, movement of the threedimensional facial structure, if present, corresponds io the user-induced device motion.

4. A method according to any preceding claim, wherein the motion data is processed to generate a device motion feature vector, and the device motion feature vector is inputted to a neural network, the neural network having been trained to distinguish between device motion feature vectors captured from different users, wherein a resulting vector output of the neural network is used to determine whether the user-induced device motion matches the expected device motion pattern.

5. A method according to claim 4, wherein the neural network has been been trained based on device motion feature vectors captured from a group of training users, which does not include the authorized user.

6. A method according to claim 4 or 5, wherein the authentication step comprises classifying, by a classifier, the vector output of the neutral network as matching or not matching the expected device motion pattern.

5

7. A method according to claim 6, wherein the classifier has been trained for classifying the vector output of the neural network based on one or more, earlier device motion feature vectors captured from the authorized user prior to the interval of user-induced device motior. and corresponding to the expected device motion pattern.

10

8. A method according to claim 7, wherein the classifier is a one-class classifier trained without using any data captured from any unauthorized user.

9. A method according to claim 7, wherein the classifier is a binary classifier trained using one or more earlier device motion feature vectors captured from at least, one

15 unauthorised user.

10. A method according to claim 7, 8 or 9, wherein the classifier Is a support vector machine (SVM).

20

11. A method according to claim 4, wherein the neural network has been trained, based on one or more earlier device motion feature vectors captured from the authorized user and corresponding to the expected device motion pattern, wherein said determination of whether the user-induced device motion matches the expected device motion pattern is made bydetermining whether the vector output exhibits a significant discrepancy from an expected

25 vector output.

12. A method according to claim 11, wherein the neural network has been trained to reproduce, as vector outputs, inputted device motion feature vectors captured from die authorized user and corresponding to the expected device motion pattern;

30 wherein the authentication step comprises determining whether the vector output exhibits a significant discrepancy from the device motion feature vector inputted to the netir network.

13 A method according to any preceding claim, wherein if the user-induced device motion is determined to match the expected device motion pattern and 3D facial structure is determined to be present in the image data, the user of the user device is granted access to at least one of: a function of the user device, a service, and data to which the authorized user is

5 permitted access.

14. A method according to any preceding claim, wherein the vector output is an output of a hidden layer of the neural network.

10

15. A method according to claim 4 or any claim dependent thereon, wherein the neural network is a convolutional neural network (CNN).

16. A method according to claim 4 or any claim dependent thereon, wherein the device motion feature vector comprises temporal motion values of or derived from the motion data.

17. A method according to claim 4 or any claim dependent thereon, wherein the motion data comprises a time series of device motion values, and the method comprises a step of applying a domain transform to at least a portion of the time series to determine at least one device motion spectrum in the frequency domain, wherein the device motion feature vector

20 comprises coefficients of the said at least one device motion spectrum or values derived from coefficients of the said at least one device motion spectrum.

18. A method according to claim 17, wherein the device motion feature vector comprises logarithmic values and/or cepstral coefficients determined from the device motion spectrum.

19. A method according to claim 17 or 18, wherein multiple device motion spectra are determined, each by applying a domain transform to a respective portion of the time series of device motion values, and analysed to determine whether the. user-induced motion of the user device matches the expected device motion pattern.

20. A user authentication system for authenticating a user of a user device, the user authentication system comprising:

an input configured to receive motion data captured using a .motion sensor of the user device during an interval of motion of the user device induced by the user;

one or more processors configured to execute computer readable instructions, which, when executed, cause the one or more processors to implement the steps of any preceding claim.

21. A computer program product for authenticating a user of a user device, the computer program product comprising computer-readable instructions stored on a non-transitory computer-readable storage medium and configured, when executed, implement the steps of any of claims 1 to 19.