KR102059517B1 - User authentication system and method based on eye responses evoked from visual stimuli - Google Patents

Abstract

Disclosed are a user authentication system and method using a pupil reaction. The user authentication system using the pupil reaction may include a controller configured to predict the user by using the pupil reaction of the user according to the visual stimulus; And a storage unit for storing user information, wherein the controller comprises: a visual stimulus presentation unit configured to present a visual stimulus to the user; A pupil response acquiring unit for acquiring a pupil response of the user according to the presented visual stimulus; A user predicting unit predicting a user by comparing a user feature extracted from the user pupil response information obtained by tracking the eyes of the user based on the obtained pupil response with information stored in the storage unit; And a visual stimulus determiner configured to determine a new visual stimulus based on a result obtained from at least one of the visual stimulus presentation unit, the eye response obtainer, and the user predictor.

Description

Authentication system using user's eye response induced by visual stimulation {USER AUTHENTICATION SYSTEM AND METHOD BASED ON EYE RESPONSES EVOKED FROM VISUAL STIMULI}

The present invention relates to a user authentication system and method using the pupil reaction. More specifically, an embodiment of the present invention relates to a user authentication system and method using a pupil reaction using the user's unconscious eye movement response to the visual stimulus for authentication.

The human brain is known to generate unconsciousness or other brain waves and gaze movements unconsciously in the visible image, depending on whether it is familiar or unfamiliar, or called attention (called 'brain fingerprinting'). Because different people have different past experiences, knowledge, and interests, they react differently to the same video. This is a very good biosignal that can be used for user authentication because it is more impossible for others to duplicate because it cannot be controlled by themselves. .

On the other hand, the current user authentication system, even the latest technology, such as fingerprint or iris recognition, there is a risk of leakage and the safety is not guaranteed. More specifically, even biometrics, such as fingerprints and iris recognitions, which are known to be relatively high in security, can be replicated with special materials or cameras, and even liveness detection, which is considered in the recent FIDO, can be replicated with cameras. This can be avoided by using the video.

In addition, the most basic password or device (OTP, mobile phone, etc.) may be lost, and sacrificing convenience to compensate for this.

In particular, in the aging society, an elderly population who is not familiar with digital devices needs a user authentication system that greatly enhances safety and convenience in order to use various Internet services such as banking or joining a social network (SNS).

There is a need for developing a user authentication system that is absolutely secure and convenient in computer and Internet services.

An object of the present invention is to provide a user authentication system and method using a pupil reaction.

Another technical problem to be solved by the present invention is to provide a user authentication system and method using a more accurate and lightly loaded eye response that uses the user's unconscious eye movement response to the visual stimulus for authentication.

Continuous authentication system using a pupil reaction according to an embodiment of the present invention for solving the above problems is a storage unit for storing user information including pupil response data, the pupil response of the user according to the visual stimulus and the user information A control unit for predicting a user using the control unit; a response obtaining unit configured to obtain a pupil response of the user according to the visual stimulus; and the control unit stores the user feature extracted based on the acquired pupil response in the storage unit. And a user determination unit for determining a matching probability of a user ID and an authentication unit for determining authentication and access according to the matching probability of the user ID, and repeatedly performing user re-authentication after allowing the user access. have.

According to an embodiment of the present disclosure, the storage unit may authenticate past the user's pupil response, the user's personal information, the data obtained by collecting the user's daily information, and the user's pupil response. At least one of the process or result can be stored.

According to an embodiment of the present disclosure, the visual stimulus may be any or all of a screen presented to the device in use, a still image or a video of the screen presented to the device in use, and a change in the screen according to a user's motion. It can be composed of one.

According to an embodiment of the present disclosure, the controller may assign different authentication thresholds and rejection thresholds according to areas to be accessed by the user.

According to an embodiment of the present disclosure, when the user ID matching probability is equal to or greater than an authentication threshold, the authentication unit may continuously perform authentication in a state in which a user access is allowed.

According to an embodiment of the present disclosure, the authentication unit may block the user when the user ID matching probability is less than or equal to the rejection threshold.

According to an embodiment of the present disclosure, when the user ID matching probability is equal to or less than an authentication threshold and equal to or greater than a rejection threshold, the authentication unit may request another method of additional authentication to determine whether to permit or deny access.

According to an embodiment of the present disclosure, the response obtaining unit may acquire the pupil response of the user using a camera mounted on the device or connected to the device.

According to an embodiment of the present disclosure, the camera may be configured as any one of a visible light camera, an infrared light source and a camera, a near infrared light source and a camera mounted or connected to the device.

According to an embodiment of the present disclosure, the response obtaining unit may obtain the eye response of the user at a low time resolution.

According to an embodiment of the present disclosure, the response obtaining unit may obtain a pupil response of the user at a spatial low resolution.

According to an embodiment of the present disclosure, the authentication unit may perform reauthentication at a predetermined time interval.

According to an embodiment of the present invention, the pupil response may be a signal derived from at least one of eye position, intermittent movement, pupil size, and eye blink.

According to the user authentication system and method using the pupil reaction according to the present invention, by using the user's unconscious eye movement response to the visual stimulus for authentication, the user can conveniently use the user without having to remember a password or carry a separate device. Can be authenticated. It is also an unconscious response of the user's brain and can therefore have stability that no one else can copy or imitate. In addition, by using the existing PC and the camera can track the eye using only the software can make the user authentication using the eye movement universal.

According to the user authentication system and method using the pupil reaction according to the present invention, it is possible to increase the accuracy of the user authentication using the pupil reaction.

In addition, it is possible to implement a simple system because the load of the system is less by using the user's eye response by lowering the resolution.

Also, iterative reauthentication may prohibit a user from accessing by mistake or imitation.

1 is a schematic diagram showing a user authentication system using the pupil reaction according to an embodiment of the present invention.
2 illustrates an algorithm of a user authentication method using pupil reaction according to an embodiment of the present invention.
3 is a flowchart illustrating a user authentication method using pupil reaction according to an embodiment of the present invention.
4 is a block diagram of a user authentication system using pupil reaction according to an embodiment of the present invention.
5 is a diagram illustrating a display unit used in a user authentication system using a pupil reaction according to an embodiment of the present invention.
6 is a diagram illustrating eye tracking for three users according to an exemplary embodiment of the present invention.
FIG. 7 is a diagram comparing test accuracy of a user authentication system using a pupil reaction and a user authentication system using a conventional pupil reaction according to an exemplary embodiment of the present invention.
8 is a view comparing session-specific accuracy of a user authentication system using a pupil reaction and a user authentication system using a conventional pupil reaction according to an embodiment of the present invention.
9 is a view showing the performance of the user authentication system using the pupil reaction according to an embodiment of the present invention.
10 illustrates a selection probability of an object according to a user's gaze time.
11 is a graph of average preference accuracy according to a user's gaze time.
12 is a table showing a user recognition result of a single feature.
13 is a graph showing the accuracy of user recognition by each single feature.
14 is a table showing the performance of user recognition by each single feature.
15 is a graph showing the performance of user recognition by a plurality of features.
16 is a table showing the performance of user recognition by a plurality of features.
17 is a table showing the performance of user recognition by one or more features.
18 is a graph illustrating user authentication performance of a user authentication system using pupil reaction according to an embodiment of the present invention.
19 shows a normalized distribution of dissimilarities from the same user and different users for each feature.
20 shows user authentication performance according to temporal and spatial resolution.
21 shows a normalized distribution of scan path dissimilarity within and between users at spatial resolutions of 2x2 and 4x4.
Fig. 22 shows the distribution of dissimilarity according to the sample level.
23 is a diagram of an iterative authentication algorithm of a user authentication system using pupil reaction according to an embodiment of the present invention.
24 is an example of an iterative authentication system of a user authentication system using pupil reaction according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. First, in adding reference numerals to the elements of each drawing, it may be noted that the same elements are denoted by the same reference numerals as much as possible even though they are displayed on different drawings. In addition, in the following description of the present invention, detailed descriptions of related well-known configurations or features are apparent to those skilled in the art or may be considered to obscure the gist of the present invention.

This system is a continuous authentication system using pupil response.

The system consists of a part for acquiring a pupil response, a part for storing user authentication information, a part for determining whether a user is authenticated, and a part for determining visual stimulus presentation.

The system has the following features.

1) User authentication using pupil response obtained from the front camera of the mobile device

2) user authentication during the user's daily work

3) User authentication that requires different security level according to the task performed

First, in the acquiring pupil response, the pupil response information of the user is obtained using the front camera of the mobile device. The pupil reaction information includes information such as eye movement, iris shape, and pupil size.

The storing part of the user authentication information continuously stores and updates information such as work to be performed, user reaction information of the user, whether the authentication is successful, and reliability information of the authentication.

In the part of determining whether the user is authenticated, it is determined whether the user is authenticated from the user's pupil reaction information and the user's information stored in the system in consideration of the security level required by the current task. Here, the authentication of the user is a decision about acceptance, rejection, additional authentication request, and the like.

The part of presenting the visual stimulus further determines the visual stimulus to be presented to the user when authentication is required in the part of determining whether the user is authenticated.

It is known that the human brain unconsciously generates other brain waves and gaze movements depending on whether it is familiar or unfamiliar with visible images, or whether there is interest. Because different people have different past experiences, knowledge, and interests, they react differently to the same video. This is a very good biosignal that can be used for user authentication because it is more impossible for others to duplicate because it cannot be controlled by themselves. .

Accordingly, the present invention is proposed by the necessity of developing a user authentication system that is absolutely secure and convenient by authenticating a user using eye movement.

The visual stimulus suggested by the present invention may be separately presented for user authentication, or may be displayed naturally in driving the system or executing the program. The visual stimulus may be configured as any or all of a screen presenting the device in use, a still image or a video of the screen present in the device in use, and a change in the screen according to a user's motion.

In the present invention, the user response may include a conscious response and an unconscious response as the response of the user by the visual stimulus.

1 is a schematic diagram showing a user authentication system using the pupil reaction according to an embodiment of the present invention.

Referring to FIG. 1, a user's pupils respond to a visual stimulus, and the user's pupil responses are compared with information stored in a database. As such, using the comparison result, the user authentication system according to the present invention may predict the user, verify the user, or verify the identity of the user. In addition, by comparing the user's pupil response to the visual stimulus and information stored in the database, it is possible to control the displayed visual stimulus.

2 illustrates an algorithm of a user authentication method using pupil reaction according to an embodiment of the present invention.

Referring to FIG. 2, a user authentication method using a pupil reaction according to an embodiment of the present invention may start a trial for user authentication. To begin the test, the user may present his ID. Starting the test may determine whether the ID matches the user. A visual stimulus may be presented to the user to determine whether the ID matches the user. The pupil response of the user according to the visual stimulus may be obtained to determine whether the ID matches the user.

The test may be started to determine whether or not the ID matches the user to determine or obtain the matching probability. The matching probability determined through the test may be updated, and whether or not the user and ID are matched according to the matching probability may be determined.

If the match probability is greater than or equal to the first threshold value, the user may be authenticated as a legitimate user matching the ID and allowed access.

If the match probability is less than or equal to the second threshold, the user may determine that the user is an unauthorized user who does not match the ID and block access.

The first threshold value may be greater than the second threshold value.

If the match probability has a value between the first threshold value and the second threshold value, the determination of whether the user matches the ID may be suspended and an additional test or session may be performed.

At the start of the test, it is possible to determine a match probability with a plurality of authorized user IDs stored in the system according to the user's pupil reaction without inputting the user's ID.

When the probability of matching the ID is equal to or greater than a first threshold value, the user may be authenticated as a legitimate user who matches the ID and allow access.

If the probability of matching the ID is equal to or less than a second threshold, the user may determine that the user is an unauthorized user who does not match the ID and block access.

When the match probability has a value between a specific ID and the first threshold value and the second threshold value, the determination of whether the user matches the ID may be suspended and an additional test or session may be performed.

If the match probability is a specific ID and the probability is equal to or greater than the first threshold value, the user may authenticate as a legitimate user and end the session and the test. When all of the plurality of IDs and the matching probability are less than or equal to the first threshold value, a test or session for re-authentication may be additionally performed on whether or not the IDs having a probability greater than or equal to the second threshold value match.

3 is a flowchart illustrating a user authentication method using pupil reaction according to an embodiment of the present invention.

Referring to FIG. 3, the user authentication method using the pupil reaction according to an embodiment of the present invention may include starting ID authentication (S110).

In step S110, the user authentication system using the pupil reaction may start ID verification of the user. The ID may be information for authenticating the identity of a legitimate user. The ID may be pre-stored in a user authentication system using the pupil reaction. The ID may be stored in plural. By right of the ID, the user may have the right to access the system.

According to an embodiment of the present disclosure, a user may input an ID for authenticating to start ID authentication. According to an embodiment of the present disclosure, authentication may be started to determine whether or not all the IDs stored in the user authentication system using a pupil reaction match the user without inputting an ID of the user.

A user authentication method using a pupil reaction according to an embodiment of the present invention may include providing a visual stimulus to a user (S120).

In operation S120, the user authentication system using the pupil reaction may present a visual stimulus to the user. The visual stimulus may be visually recognized by the user and may include an image, a text, an image, and the like. In providing the visual stimulus, the user authentication system using the pupil reaction may align the eyes of the user or focus on a specific position. Prior to providing the visual stimulus, the line of sight of the user may be aligned or focused on a specific position to initialize the position of the line of sight so that the user's response may be observed or acquired. The data presented to the visual stimulus may use data previously stored in the user authentication system using the pupil reaction. Data presented to the visual stimulus may be received by the user authentication system using the pupil reaction from outside. The data presented to the visual stimulus may be updated by the user authentication system using the pupil reaction by the user's instruction or a predetermined period.

The visual stimulus may be provided by placing a plurality of images on one screen. The visual stimulus may be classified and provided according to a specific subject. The visual stimulus may be classified to include at least one or more visual stimulus data according to a category, and may be provided according to the category.

According to an embodiment of the present invention, a total of 359 images may be collected from the Internet, a face recognition technology (FERET) database, and an ImageNet database. In addition, three different selfies may be collected.

According to an embodiment of the present invention, an important part for identifying the category of the visual stimulus for each visual stimulus may be larger than 250x250 pixels. An important part of every visual stimulus can be cut out and resized to 350x350 pixels. They can change from color visual stimulus to gray scale visual stimulus. Lighting can also be standardized. It is possible to reduce the protruding effects that affect the bottom-up interest of the user.

Four visual stimuli may be randomly selected to constitute one set of visual stimuli of the plurality of visual stimuli. Some sets of visual stimuli are composed of visual stimuli of the same category and others are composed of visual stimuli of different categories. Each visual stimulus set may be 800 x 800 pixels in size. According to an embodiment of the present disclosure, each visual stimulus may be disposed at each corner of the screen. The background may be black and a yellow cross is displayed at the center to initialize the user's visual response, such as focusing the user's gaze before the visual stimulus is presented.

The user is free to observe each set of visual stimuli during the session. The user may select a preferred visual stimulus after observing each visual stimulus set during the session. According to an embodiment of the present invention, a short break may be set between sessions. The order of presenting the set of visual stimuli in each session may be random. The cross in the center of the screen may be displayed for 0.5 seconds and the visual stimulus may be displayed for 2 seconds. There was a short interval after each attempt, and the user can adjust each interval.

The user authentication method using the pupil reaction according to an embodiment of the present invention may include obtaining a user response (S130).

In operation S130, the user authentication system using the pupil reaction may acquire a response indicated by the user with respect to the visual stimulus presented to the user. The response may include a pupil response including the gaze of the user, a gaze time, a scan path, and the like. The response may include a preference of the user. The user authentication system using the pupil response may obtain a pupil response of the user through a gaze tracker, and obtain a preference using data input by the user or the pupil response.

In step S130, the pupil reaction can be obtained by lowering the resolution. The resolution may include spatial resolution and temporal resolution. The spatial resolution may refer to the number of points for measuring a change in position of the pupil reaction. The spatial resolution may be represented by n × n, and the smaller the number corresponding to n, the lower the spatial resolution. According to an embodiment of the present disclosure, the spatial resolution may be 4 × 4 or more. If the spatial resolution is 2 × 2 or less, the performance of user authentication may be degraded.

The temporal resolution may mean the number of times per hour for measuring the change in the pupil response. The time resolution may be displayed in units of Hz. The lower the time resolution, the longer the measurement interval and the fewer the number of measurements. According to an embodiment of the present invention, the time resolution may be 10 Hz or more. If the time resolution is less than 10 Hz, the performance of user authentication may be degraded.

A user authentication method using a pupil reaction according to an embodiment of the present invention may include a step (S140) of determining a match probability of an ID and a user.

In operation S140, the matching probability between the ID and the user may be determined using the user response obtained in operation S130. The matching probability may be determined for each test. The matching probability may be determined for each test to calculate or determine the matching probability as a whole session including a plurality of tests. The matching probability may be a matching probability between the ID input by the user in step S110 and the user. The matching probability may be determined by determining the matching probability of all IDs and users stored in the user authentication system using the pupil reaction. The determination of the matching probability may be determined based on the similarity of the user response to the ID.

A user authentication method using a pupil reaction according to an embodiment of the present invention may include determining whether to access according to the matching probability (S150).

In operation S150, it may be determined or determined whether the user permits access according to the matching probability. If the match probability is greater than or equal to the first threshold value, the user may determine that the user matches the ID and allow access. When the match probability is less than or equal to the second threshold, the user may determine that the user is an unauthorized user who does not match the ID, and may restrict or deny access. When the match probability has a value between the first threshold value and the second threshold value, an additional test or session may be performed to perform user authentication. That is, when the match probability has a value between the first threshold value and the second threshold value, the user authentication system using the pupil reaction may return to step S120 to provide a new visual stimulus.

In the case of determining a matching probability with all IDs previously stored in the user authentication system using the pupil reaction, if there is an ID having a value equal to or greater than a first threshold value among the plurality of matching probabilities, the user is a legitimate user whose ID has the largest matching probability. Authenticate users and allow access.

In the case of determining a match probability with all IDs stored in the user authentication system using the pupil reaction, if there is no ID having a value equal to or greater than a first threshold value among the plurality of match probabilities, a value equal to or greater than a second threshold value among the plurality of match probabilities is determined. Can be determined. If there is a value equal to or greater than a second threshold value among the plurality of matching probabilities, an additional test or session may be performed to perform user reauthentication. Return to step S120 for re-authentication can present a visual stimulus. Preferably, the reauthentication may be performed only for IDs whose matching probability is greater than or equal to the second threshold.

In the case of determining a match probability with all IDs previously stored in the user authentication system using the pupil reaction, if there is no ID having a value greater than or equal to a second threshold value among the plurality of match probabilities, the user is determined to be an unauthorized user, and access is made. You can block or reject it.

The user authentication method using the pupil reaction according to an embodiment of the present invention may include determining whether to access through repeated ID authentication (S160).

In operation S160, the user authentication system using the pupil reaction may repeatedly perform reauthentication for the user who is allowed to access. Returning to step S120 for the re-authentication may provide a visual stimulus. The reauthentication may be performed by a predetermined time unit. If it is determined that the user is a legitimate unauthorized user as a result of the reauthentication, the access may be released and the access may be blocked or denied. As a result of the re-authentication, if the user has the right authority, the access can be continued.

In step S160, since the existing computer system authenticates the user only during the initial login session, it may cause a serious security flaw. That is, the authentication of the user by tracking the movement of the user's eyes can show high accuracy, but the possibility that another person who imitates the eyes of the legitimate user can be authenticated as the legitimate user can not be ignored. This allows the system to continuously authenticate the user, which constantly monitors and authenticates the user according to the user's biometric characteristics.

In an embodiment of the present invention, when the program is executed, a loading time may be required for the system to prepare to use the program, and a logo image may appear in the center of the screen while loading.

According to an embodiment of the present invention, the user authentication system using the pupil reaction may increase security of user authentication through continuous user authentication instead of one-time user authentication.

4 is a block diagram of a user authentication system using pupil reaction according to an embodiment of the present invention.

Referring to FIG. 4, a user authentication system using a pupil reaction according to an embodiment of the present invention may include a response obtaining unit 100, a control unit 200, an authentication unit 300, and a storage unit 400. .

The response obtaining unit 100 may obtain a response of the user. The obtained user response may be transmitted to the controller 200. The response obtaining unit 100 may obtain a response indicated by the user with respect to the visual stimulus presented to the user. The response may include a pupil response including the gaze of the user, a gaze time, a scan path, and the like. The response may include a preference of the user. The user authentication system using the pupil response may obtain a pupil response of the user through a gaze tracker, and obtain a preference using data input by the user or the pupil response.

The response obtaining unit may obtain the eye response of the user using a camera mounted on the device or connected to the device.

The reaction acquisition unit 100 may include a camera, and the camera may be configured as any one of a visible light camera, an infrared light source and a camera, a near infrared light source and a camera mounted or connected to the device.

The response obtaining unit 100 may obtain the pupil response by lowering the resolution. The resolution may include spatial resolution and temporal resolution. The spatial resolution may refer to the number of points for measuring a change in position of the pupil reaction. The spatial resolution may be represented by n × n, and the smaller the number corresponding to n, the lower the spatial resolution. According to an embodiment of the present disclosure, the spatial resolution may be 4 × 4 or more. If the spatial resolution is 2 × 2 or less, the performance of user authentication may be degraded.

The temporal resolution may mean the number of times per hour for measuring the change in the pupil response. The time resolution may be displayed in units of Hz. The lower the time resolution, the longer the measurement interval and the fewer the number of measurements. According to an embodiment of the present invention, the time resolution may be 10 Hz or more. If the time resolution is less than 10 Hz, the performance of user authentication may be degraded.

The reaction obtaining unit 100 or the control unit 200 may change the time and spatial resolution as in the following embodiment.

The response acquiring unit 100 may change and measure the resolution at the time of measurement, and the controller 200 may change the resolution of the obtained data.

1) time resolution

The sampling frequency of the raw data was originally 120 Hz and was reduced to 60, 30, 20, 10 and 5 Hz. Temporal down sampling was performed using the resample function of MATLAB (The MathWorks, Inc., Natick, MA, USA).

The resample function removes samples from the raw data according to the target sampling frequency and performs anti-aliasing with low pass filtering.

2) spatial resolution

Quantization is used to reduce the spatial resolution of the scanpath. There are two scan path quantization methods, radius-based and grid-based. In an embodiment of the present invention, spatial resolution may be changed using radius-based quantization. An error uniformly distributed at the line of sight was added. In contrast, lattice-based quantization can superimpose a fixed lattice on visual stimuli and discretize the gaze position values. Since we presented the image in a 2 Х 2 layout in our experiment, we use grid-based quantization.

와 x 좌표 에서의 양자화된 시선 위치는 각각 다음 수학식 1 및 수학식 2와 같이 표현 될 수 있다.To implement grid-based quantization, uniform quantization is applied using mid-riser at the line of sight of x and y coordinates. Using the above method, the gaze position is quantized to the center position of the region to which the gaze belongs. Quantization step size when the target resolution is r Х r (r is the number of quantization levels)

The quantized gaze positions at and x coordinates may be expressed as Equations 1 and 2, respectively.

[Equation 1]

[Equation 2]

Where is the horizontal size of the screen and is the raw gaze position at the x coordinate. Since the horizontal and vertical sizes of the screen are the same, the quantization of the y coordinate is the same as the x coordinate. The original spatial resolution of the raw data was 800 Х 800 and scaled down to 256 Х 256, 64 Х 64, 16 Х 16, 4 Х 4 and 2 Х 2.

Scan Path Comparison

1) Euclidean distance

Euclidean distance can represent dissimilarity between two sequences through pairwise comparison. Scan paths of the same user may be similar, but may not be similar to scan paths of other users. The Euclidean distance between two scan paths is calculated as in Equation 3 below.

[Equation 3]

Where 1 () and 2 () are the gaze positions of the primary and secondary scan paths in the x coordinate, and 1 () and 2 () are the gaze positions of the y coordinate. On the other hand, represents a sample index, and is the total number of samples in the scan path.

2) Dynamic Time Warping

Dynamic time warping (DTW) is an algorithm used to compare two time sequences with different speeds. It is proposed to compare speech sequences at different speeds for automatic speech recognition. Compared with Euclidean distance, DTW is robust to movement in the time domain. In the DTW algorithm, the cost matrix is first obtained by calculating the distances for the sample pairs that make up each sequence. Dynamic programming is then used to find the optimal warping path in the cost matrix to minimize the sum of distance values on the path. The minimum cost is called the DTW score of the two sequences.

User identification

Using the nearest neighbor method, identification is performed at each attempt. Given the rating data, we calculate the difference for the development data of 27 users and assign the index of the rated user's identity and the lowest dissimilarity. If the identified user matches the evaluated user, the identification is successful. The identification process is expressed by the following equations (4) and (5).

[Equation 4]

[Equation 5]

Where and are the identification (ID) of development and evaluation data. Identification performance was measured with an average Rank-1 identification rate of 110 trials.

The response obtaining unit 100 may obtain preference information of the selected visual stimulus when the user selects a specific visual stimulus with a remote control button or the like. The response obtaining unit 100 may display a number for each visual stimulus so that the user can select a preference.

The response obtaining unit 100 may have a time zone in which a preference and a gaze are closely related to each other and a time zone in which a response is not closely related to each other in a test or a session. For example, the beginning of each test or session is a process of moving the gaze from a fixed point in the center of the screen to an item, so the gaze during this interval may be less relevant to preference. The final gaze index and gaze bias can be calculated by dividing the test or session into several time intervals. The lengths of the segments were 1/4, 1/2 and 3/4 of the test and the starting point of each segment could be the beginning, 1/4, 1/2 and 3/4 of the test. The preference prediction results of the gaze characteristics can be compared, and the characteristics with excellent preference prediction results can be used for user authentication and recognition.

Affinity Index (Index)

The index obtained by the response obtaining unit 100 may be an index of an image input by the user through a keyboard button after the user has tested the session. For example, indexing the preferred index of four images may be the same as indexing the low resolution scan path data. Tests in which the user has not pressed a button may be excluded from the analysis. If more than one index is entered in a test, the last index pressed may be considered as a preference index.

The controller 200 may present a visual stimulus to a user and determine a probability of matching the ID with the user using a user response. The ID may be pre-stored in the storage unit 400. The ID may include information of an authorized user. The controller 200 may determine a probability that the ID coincides with the user.

Each user has various preferences for the presented candidates, and when selecting a preference, the pupil response may include information about the preference and the subject identity. The user may be authenticated or recognized through explicit preferences, implicit gaze information, scan paths, or a combination thereof.

According to an embodiment of the present invention, the control unit 200 sets up a test in which four visual stimuli are coupled to each other and continuously presents differently configured tests to the user. Visual stimuli can be collected in multiple categories and can provide tests organized in the same category and tests organized in different categories. Various tests may provide various preference options for the user.

According to an embodiment of the present invention, a preference feature may be extracted from keyboard buttons or eye movement data. The controller 200 may extract some gaze features known to be related to the preferences (gaze bias, initial and final gaze indexes). Because preferences are expressed implicitly, they are contrasted with explicit preferences expressed with button presses. The response acquirer 100 may extract a scanpath which is a trajectory of the pupil reaction in a time and space region as a feature representing a target user. The scan path contains spatiotemporal information about how the user sees the visual stimulus. The process of making a decision in a situation in which the user must select one of the candidates varies from person to person and the process can be indicated by the scan path. Also, the scan path for observing one image varies from person to person, and the scanpath for observing multiple images is the same. Therefore, in addition to image preference, scan paths can be extracted from eye movement patterns.

The controller 200 or the authenticator 300 may authenticate and recognize a user through explicit preference, implicit preference, scan path, and fusion thereof. Since preferences and scan paths are expected to include information about the user's identity, authentication and recognition results can be obtained using classic biometric assessments (Rank-1 identification rate and same error rate). In the case of an unknown user, the authentication result may indicate whether the person is a truly authorized user and the recognition result may indicate the identity of the user. Performance can be improved by using information obtained from various modalities or information with different characteristics. The controller 200 may perform 'preferred fusion' on the implied preference obtained from the eye movement and the explicit preference obtained through the keyboard, and the 'preferability and scan path fusion' for the combination of the preference information and the scan path information. 'Can be performed.

The controller 200 may extract features such as a scan path, a gaze index, a gaze bias, and the like from the eye response data and extract a preference index from the button press data.

1) Scan Path-High Resolution and Low Resolution

The scan path may represent a time sequence of the user's gaze points within the test. The controller 200 may obtain two types of scan paths according to the resolution of the spatial coordinates. The high resolution scan path is a sequence of x, y pixel coordinates of the user's gaze position, and the low resolution scan path may be a series of AOIs (area of interest) obtained by replacing the user's gaze position with quadrant numbers.

The controller 200 may obtain a high resolution scan path by interpolating and smoothing raw raw gaze points. The raw gaze point collected from the response obtaining unit 100 may include two-dimensional coordinates of a position on the screen at which the user views each viewpoint. The center of the screen is the origin and the x and y coordinates can range from -480 to 480. A gaze trajectory can be recorded for 2 seconds with a sampling frequency of 120 Hz for each trial. The controller 200 may reject a test for missing further 20% of the sample for further analysis to deal with a defect contaminated by the device or the user. The controller 200 may reduce the difference between the existing data and the generated data by generating a sample of the section in which data is missing through linear interpolation and applying smoothing when the missing rate is less than 20%.

The controller 200 may generate the low resolution scan path data by quantizing the high resolution scan path data. The low-resolution scan path data may be classified into one of four quadrants of the user's gaze position divided by the origin, and may be assigned a value corresponding to the quadrant. A value of 1 can be assigned to the top left image, 2 to the top right, 3 to the bottom left, and 4 to the bottom right.

2) Eye Features-Initial / Final Eye Index and Eye Bias

The control unit 200 displays the image of the user viewed from the gaze feature, a specific time segment (gaze bias), or a specific time point (gaze index) to indicate which of the four images the user was viewing at each time point. The index of the most viewed image can be extracted from the low resolution scan path.

The gaze index may be an index of an image in which the user's gaze falls at a specific point in time. The controller 200 may extract an initial gaze index and a final gaze index. After the visual stimulus is presented by the controller 200, the first response may be related to a sign, and the final gaze index may be used to indicate a point viewed by the user for the first time since the final gaze index is extracted to show that the individual's final gaze goes to a preferred item. The index can be extracted.

Gaze bias is the index of the visual stimulus seen by the user for the longest period of time when performing the test. The gaze cascade model may tend to be more interested in items that people prefer. Thus, gaze bias may be closely related to preference. The number of samples in which the user's gaze remains in each of 1, 2, 3, and 4 images can be calculated, and the index of the largest number of images can be determined as the gaze bias of the test.

The controller 200 may calculate dissimilarity using the preference information and the pupil reaction.

The controller 200 may compare similarity or dissimilarity between main user response data and pre-stored IDs in order to authenticate and recognize a user. The controller 200 may collect data on an evaluation and development session, calculate dissimilarity for each test, and calculate the total dissimilarity between the two sessions. The dissimilarity calculation method of the controller 200 is a calculation in a test step, and other calculation methods described below may be applied according to characteristics of data used.

1) Euclidean Distance-High Resolution Scan Path

Euclidean distances of two temporal sequences of gaze positions may be calculated to investigate the difference between two gaze sequences in the space-time region. The Euclidean distance between two sequences created by the same user may be relatively smaller than the distance of other users and the similarity of gaze patterns may be higher.

Given a development session and an evaluation session, the Euclidean distance for the kth test can be calculated as: The Euclidean distance between the i th sample of the development data and the evaluation data is calculated on the two-dimensional plane, this calculation is performed for all the samples making up the k th test, and the distance values are summed to calculate the Euclidean distance of the test. Can be. The dissimilarity of the k th test can be described as in Equation 6 below.

[Equation 6]

및

는 개발 세션에서 k 번째 시도의 i 번째 샘플의 x 및 y 좌표에서 시선 위치를 나타낸다. 유사하게,

및

는 평가 세션의 시선 위치를 나타낸다.Where i is the index of the sample in the k th test and is the number of samples in the k th trial. sign

And

Denotes the gaze position at the x and y coordinates of the i th sample of the k th trial in the development session. Similarly,

And

Represents the gaze position of the evaluation session.

2) Hamming distance-low resolution scan path, gaze characteristics and preference index

The same user may have the same preference index or gaze feature in different sessions. Hamming distance can be used for the measurement. The Hamming distance specifies a binary value of 0 if the two samples to be compared are equal and a binary value of 1 if they are different. The hamming distance at the k th trial may be calculated as shown in Equation 7 below.

[Equation 7]

와

는 각각 개발 세션과 평가 세션의 k 번째 시험에서 시선 특징 또는 선호 지수다. 모든 시험에 대한 해밍 거리의 합계는 두 세션에서 선호 지수가 일치하지 않는 시도 횟수를 나타낸다.here

Wow

Is the gaze characteristic or preference index in the kth trial of the development and evaluation sessions, respectively. The sum of Hamming distances for all tests represents the number of trials where the preference index did not match in both sessions.

Comparing two low-resolution scan paths through Hamming distance consists of checking whether the user stared in the same quadrant at each time point and combining the results for all time points. For sample pairs, the Hamming distance between development and evaluation data is calculated as zero if the gaze positions are in the same quadrant, and the Hamming distance is calculated as one if it is in the other quadrant. This calculation is performed for all samples that make up the test. After the sum, the number of samples with different quadrant numbers is represented by the distance value. The Hamming distance at the kth trial is calculated as

[Equation 8]

및

는 각각 개발 세션 및 평가 세션에서 k 번째 시험의 저해상도 스캔 경로 지수, 고해상도 스캔 경로 지수를 각각 보여준다.Where _k is the Hamming distance at the i th sample at the k th trial, i is the exponent of the sample, is the number of samples in the test,

And

Shows the low resolution scan path indices and the high resolution scan path indices of the k th test in the development and evaluation sessions, respectively.

3) fusion rules

Dissimilarities calculated with various features are fused to improve authentication and recognition performance. Two main types of fusion can be performed: 'preferred fusion' and 'preferred and scan path fusion'. Preference fusion is a fusion of explicit and implicit preferences, and a user can be authenticated and recognized by fusion of dissimilarity properties calculated from preference index and gaze characteristics. Dissimilarity of explicit or implicit preferences in preference and scan path fusion can be combined with scan path dissimilarity.

When two dissimilarities are combined, the dissimilarity of each feature may first be normalized to fit the scale of the two dissimilarities. The sum of the weights of two dissimilarities where the sum of the two weights is 1 may yield a fusion distance. For example, in the case of fusion between dissimilarity of the scan path and dissimilarity of gaze deflection, the distance calculated through fusion is expressed by Equation 9 below.

[Equation 9]

Is the dissimilarity of the scan path, is the dissimilarity of the gaze deflection, and w is the weight multiplied by the scan path dissimilarity. The value of w varies in 0.1 intervals from 0 to 1.

User recognition method

In the user recognition, the control unit 200 may identify the evaluation data for the unknown user by the ID of the user most similar among the development data in the system. The data from one session and the data from one session can consist of 110 tests, and the dissimilarity of each test can be calculated and the session dissimilarity can be calculated by combining the values of eoks dissimilarity for each test. In development data, you can be authenticated as the user with the lowest average session mismatch with respect to evaluation data. If the recognition result matches the user ID of the evaluation data, the recognition is successful. Otherwise it can be considered a failure. The recognition process may be expressed as in Equations 10 and 11 below.

[Equation 10]

[Equation 11]

는 평가 세션과 J 번째 사용자 데이터 사이의 k 번째 시험에 대해 계산 된 비유사도다.Where and are the identity of the development and evaluation session, k is the test index of the session, and is the number of trial sessions. value

Is a dissimilarity figure calculated for the k th test between the evaluation session and the j th user data.

Recognition performance may be measured at Rank-1 identification speed. Given the test data for each session combination, the identification speed can be calculated as the rate of successful recognition in all cases.

User Authentication Method

Unlike the user recognition, the authentication unit 300 may obtain a user authentication result through measured evaluation data, development data of a user who actually claims stored in the system, and a system threshold value.

Given the evaluation data of stranger users attempting to access, we use the difference from the development data to calculate the probability that the user attempting to access each test will be hit, and then accumulate them to calculate the probability of successive tests. Can be. The authentication method of the present embodiment may reject a user whose cumulative probability is higher than a threshold or a user whose matching probability is lower than a second threshold. The authentication process may be expressed as in Equations 12 and 13 below.

[Equation 12]

,

,

[Equation 13]

는 주장하는 사용자,

는 개발 데이터의

사용자 (

, N: 시스템에 저장된 전체 사용자 수)의 안구 반응과 인증을 할 평가(Evaluation)데이터와의 차이,

는 t 번째로 자극을 제시하고 획득한 안구 반응이며,

는 t 번째까지 자극을 제시하고 안구 반응들을 획득했을 때 접근하려는 사용자가 주장하는 사용자일 확률로, 각각의 자극에 의해 유도된 안구 반응으로 구한 주장하는 사용자 확률의 누적값이다.here

Is the claimed user,

Of development data

user (

, N : the difference between the eye response of the total number of users stored in the system and the evaluation data to be authenticated,

Is the ocular response obtained by presenting the stimulus at t-th,

Is the probability of the user claiming to approach when the stimulus is presented up to the t th time and the eye responses are obtained.

)은 다음의 식을 이용할 수 있다.The authenticator 300 may determine or determine whether to allow the user to access, based on the probability of matching the claimed user data. If the match probability is greater than or equal to the first threshold value, the user may determine that the user matches the ID and allow access. When the match probability is less than or equal to the second threshold, the user may determine that the user is an unauthorized user who does not match the ID, and may restrict or deny access. When the match probability has a value between the first threshold value and the second threshold value, an additional test or session may be performed to perform user authentication. That is, when the match probability has a value between the first threshold value and the second threshold value, the user authentication system using the pupil reaction may allow the controller 200 to provide a new visual stimulus. The visual stimulus to present first

) Can use the following equation.

는 개발데이터에서 주장하는 사용자(

)와 주장하는 사용자 외에서 가장 높은 확률을 갖는 사용자(

) 데이터 간의 차이이다. 즉, 현재 가능성이 높은 타 사용자와의 차이가 큰 안구 움직임을 획득 할 수 있는 시각 자극을 우선적으로 제시하는 것이다.here

Is the user who claims in development data (

) And the user with the highest probability (other than the claimant)

) Is the difference between the data. In other words, the present invention first suggests visual stimuli that can acquire eye movements that are highly different from other users.

However, using this method alone can slow down the decision to deny authentication. Therefore, if the probability of a user other than the asserting user is high, a method of rapidly increasing the probability of the user and rapidly lowering the claiming user can be added as follows to accelerate the decision to deny the authentication.

The authentication unit 300 may repeatedly perform reauthentication to a user who is allowed to access the user authentication system using a pupil reaction. The controller 200 may allow the user to provide a visual stimulus for the reauthentication. The reauthentication may be performed by a predetermined time unit. If it is determined that the user is a legitimate unauthorized user as a result of the reauthentication, the access may be released and the access may be blocked or denied. As a result of the re-authentication, if the user has the right authority, the access can be continued.

The authenticator 300 may continuously authenticate the user for continuously monitoring and authenticating the user according to the biometric characteristics of the user.

In an embodiment of the present invention, when the program is executed, a loading time may be required for the system to prepare to use the program, and a logo image may appear in the center of the screen while loading.

According to an embodiment of the present invention, the user authentication system using the pupil reaction may increase security of user authentication through continuous user authentication instead of one-time user authentication.

The controller 200 uses the data of the storage unit to visually induce a highly consistent pupil response from the ID presented by the user or a pupil response having a large difference from other users from the ID presented by the user. Can be selected and presented.

The controller 200 may use at least one or more other user data other than the ID presented by the user to obtain a difference between the pupil response of the ID presented by the user and the pupil response from another ID.

The control unit 200 is similar to the pupil response of an ID other than the ID presented by the user, if the acquired probability of the user's pupil response is higher than a certain threshold, the user may quickly reject the authentication. A visual stimulus may be selected and presented that may induce a pupil response having a high consistency from the other ID other than the presented ID or a large difference between the user and the ID presented by the user.

The visual stimulus may be a combination of one or more still images or moving images.

The pupil response may be a signal derived from at least one of gaze position, intermittent movement, pupil size, and eye blink.

5 is a diagram illustrating a display unit used in a user authentication system using a pupil reaction according to an embodiment of the present invention.

Referring to FIG. 5, when the user authentication system using the front camera and the pupil reaction is operated, a fixation and a specific visual stimulus are sequentially displayed on the screen, and gaze tracking responds differently to each user who views the displayed screen. It can be seen that the information is used to authenticate the user. At this time, the green and red circles and lines of the second and third stimulus pictures are overlapped with the measured eye tracking information for understanding as there is no actual stimulus.

In addition, only visual stimuli may be sequentially displayed on the screen without gaze fixing, and the number of visual stimuli presented may be variously set.

As such, the user authentication system using the pupil reaction may authenticate the user through eye gaze that reacts differently to each person.

More specifically, a user authentication system using a pupil reaction capable of authenticating a user according to the present invention measures eye gaze movements of a visual stimulus and looks at a user built in a database and a plurality of other people. It may be a system that "authenticates" or "denies" a user by comparing movements.

A person's eye movements vary depending on their memory, interest, and what they want to do (for example, a lot of attention goes to the face they are interested in), but even the same person can change their memory or interest over time. Shows sequentially, and by using the eye movements according to the appropriate authentication reliability is secured.

That is, one image is one question, and the response to the question is one answer, and the question and answer is continued until the reliability of Accept / Reject for the user reaches a specific value. If you binarize one answer to "right" and "wrong", the answer from the N video questions corresponds to an N-bit cipher. In other words, the more video questions, the higher the reliability.

For example, one visual stimulus may be composed of four independent images at a 2x2 position, and a 2x2 class ultra-low resolution gaze tracker may be used to detect eye differences.

In addition, the stimulus image may be selected as an image having high discrimination power every time. For example, if the probability of being a user is 70% so far, but the second highest probability of being A is 20%, a visual stimulus that contributes to discriminating user and A is selected and used.

6 is a diagram illustrating eye tracking for three users according to an exemplary embodiment of the present invention.

Referring to FIG. 6, as an example of eye tracking for three users, the size of the circle is proportional to the time of stay, and the number in the circle indicates the order of eye gaze. The solid line indicates that the eyes are connected in order. It can be seen that a different response to the same image is unconsciously different for each person.

More specifically, by showing a specific image on the computer screen and eye tracking, it can be seen that several local attention is formed on the same image, and different gaze patterns are sequentially formed for each person. In this case, a face, a building, and an animal photograph may be used as the image.

While a general gaze tracking system shows one complex image, the present invention ultimately presents different images at four corners of the screen to use the low resolution (2x2) gaze tracking system for authentication, and gaze movements for them. Was measured. More eye gaze will stay on the favorite image more often. As a result, as shown in FIG. 6, movements of different gazes are observed for each person even for the same image, and user authentication can be performed using the same.

FIG. 7 is a diagram comparing test accuracy of a user authentication system using a pupil reaction and a user authentication system using a conventional pupil reaction according to an exemplary embodiment of the present invention.

Referring to FIG. 7, first, user recognition of each eye data for one visual stimulus was attempted. One development session and another evaluation session can be used per user. In the present embodiment, the Euclidean distance between one evaluation data and the entire valid development data may be calculated at each time point. The closest identity id can be compared to the actual identity. The same can be considered correct. If they are different, they can be considered wrong. The procedure may repeat the entire validity assessment data. 7 shows the accuracy of test identification at each time interval. At 30M intervals, the accuracy of the work session was about 17%. That's a better figure than accidental levels (about 3.7%).

8 is a result of attempting user authentication by selecting a visual stimulus. The method used the final accumulated user probability using all the eye data obtained by measuring the false recognition rate (FAR) and the false rejection rate (FRR) as the first threshold baseline. It can be seen that the authentication performance is improved when the visual stimulus that selects the claiming user can be selected first (Method 1). In addition, we can see that the false recognition rate is lowered when we add a method to speed up access denial (Method 2).

9 is a diagram showing how the number of visual stimuli used is reduced compared to when all the measured data are used. Compared with the baseline, even less than 1/4 visual stimulus can provide higher performance.

10 illustrates a selection probability of an object according to a user's gaze time.

Referring to FIG. 10, the results of observing the time course of the gaze position in order to determine how the gaze of a person is related to the preference during the test. Through the gaze order data and the preference selection of clinical trials, the present invention can use a binary value sequence to indicate whether a subject has seen a preference item at each time zone (1 when viewing a preferred item, 0 otherwise). 10 shows the average time course for all trials of all user sessions. The curve is similar to that of the gaze cascade model. The closer the decision time is, the more likely the user is to see the item selected by the user. Since four images have been presented to the subject, there is a 1/4 chance that the gaze will accidentally turn to a favorite item. If the line of sight is greater than one-fourth the probability of being placed in a preference item, the line of sight is deliberately viewed as an eye, rather than accidentally pointing to the item. Approximately 0.5 seconds after the onset of stimulation, the probability exceeds 1/4. The first quarter of the trial is not closely related to preference.

11 is a graph of average preference accuracy according to a user's gaze time.

Referring to FIG. 11, the preference is classified by using features extracted from the gaze data in order to examine the relationship between the preference and the gaze. Here the button press result is considered the ground truth label of the symbol and the gaze feature is considered the expected label. The gaze characteristics were the initial gaze index, the final gaze index, the gaze bias for the demonstration, and the final gaze index and gaze bias within the various time domains of the demonstration. The length and starting point of the segment were used differently to obtain various time segments. There were three interval lengths, 1/4, 1/2, and 3/4 of each test, with four starting points for the first, quarter, third and third points of each test. .

As a result, the classification accuracy of the initial gaze index was 25.9% and the final gaze index was 47.6%, so the latter part of the test was related to the preference. Also, the longer the length of the time section, the higher the accuracy of preference classification. However, there was no significant difference between the coincidence ratios of gaze bias between 0-2 seconds and 0.5-2 seconds. This result indicates that gaze data up to 0.5 seconds is not related to preference and that the results are consistent with the gaze time-lapse observations. Therefore, the gaze bias extracted from the data of 0.5 to 2 seconds was used as the gaze characteristic to authenticate and recognize the user.

12 is a table showing a user recognition result of a single feature, FIG. 13 is a graph showing the accuracy of user recognition by each single feature, and FIG. 14 is a table showing the performance of user recognition by each single feature.

Single feature versus multiple feature fusion

The features extracted from the pupil response data are high resolution and low resolution scan paths and gaze bias, and the features extracted from the button press data may be a preference index. In order to see the effect of the fusion of the features, a single feature and a feature fusion were used to compare the user recognition and authentication results. Certification and recognition results were obtained for all combinations of development and evaluation sessions. Results were averaged over the time interval (30 minutes, 1 week and 2 weeks) between the development session and the evaluation session.

Referring to FIG. 12, a user recognition result of a single feature may be confirmed. The preference index showed the best performance. For pupil response features, the best performance was in the order of the high resolution scan path, the low resolution scan path, and the gaze bias. Performance using gaze bias was lower than other features, but significantly higher than 1.75% probability. As the time interval increases, the recognition performance of the feature extracted from the eye movement data decreases, but the feature extracted by pressing the button shows stable performance regardless of the time interval. 13 and 14, it is possible to confirm the authentication result of each single feature similar to the recognition result. The values of EER, FAR @ 1/10 FRR, and FAR @ 1/20 FRR obtained by the certification of preference index were smaller than other features. The low resolution scan path only stores limited information compared to the high resolution scan path, but there is no difference in recognition or authentication performance between the high resolution and low resolution scan paths.

As the ratio of FRR to FAR increases, the difference in authentication performance between high resolution and low resolution scan paths decreases. Low resolution scan paths are well-performed features that provide protection against cover observation.

15 is a graph showing the performance of user recognition by a plurality of features.

Referring to FIG. 15 (a), recognition performance is improved through fusion. All fusions and time intervals perform better than no fusion. Convergence improved the user identification rate from 1.8% to 10.5% with an average of 5.8%. Increasing performance through convergence was greater in preference and scan path fusion than in priority fusion. The maximum performance was 96.5% at 30-minute intervals when using the Preference and High Resolution Scan Paths. The best performance results in the convergence of gaze deflection and high resolution scan paths were 86.8% at 30 minute intervals and 9.6% smaller than the preference index and high resolution scan path fusion results.

The converged target authentication results are illustrated in Figure 16 (b). As with the recognition results, convergence improves the authentication performance in all cases. On average, EER improved by 1.33%, FAR @ 1/10 FRR 0.38%, and FAR @ 1/20 FRR 0.22%. The fusion of a preference index and a high resolution scan path resulted in a peak performance of 2.6% EER every 30 minutes. When the gaze bias and the high resolution scan pass were fused, the EER was 5.4%, which was 2.8% higher than the preference index and the high resolution scan path fusion result.

In most cases, better performance is achieved by giving more weight to a single feature in both user recognition and authentication results. The performance difference between the low resolution scan path and the high resolution scan path was greater in the gaze bias and the scan path fusion than in the convergence of the preference and the scan path. Acceptable recognition and authentication performance can be achieved even when only pupil response data is used without explicitly revealed preference information.

FIG. 16 is a table illustrating performance of user recognition by a plurality of features, and FIG. 17 is a table illustrating performance of user recognition by a single or plural features.

Temporal robustness of user authentication and recognition performance

The proposed method should guarantee reliable performance over time. Persistence is one means of determining whether biometrics is suitable for an authentication system. If the time interval between development and evaluation sessions is one year, authentication and recognition performance is verified. In both years, we used the data for the users involved in the analysis. Referring to FIGS. 16 and 17, the result of the authentication and recognition of the single feature and the result of fusing the feature may be confirmed. The results of feature fusion show the best performance between results using different weights.

At yearly intervals, authentication and recognition performance decreased for most single features except gaze bias. The reduction in the identification rate between the two-week and one-year intervals was greater than the identification rate between the 30-minute and one-week intervals and between the one- and two-week intervals. Performance deteriorated at 1 year intervals but remained within 30% of performance at 30 minute intervals. This result indicates that the proposed method is effective even one year after registration. In addition, the degradation in feature fusion was much smaller than the reduction of a single feature. The performance degradation was within 10% at 30 minute intervals. This result indicates that over time, performance degradation can be reduced due to fusion characteristics.

18 is a graph illustrating user authentication performance of a user authentication system using pupil reaction according to an embodiment of the present invention.

Reduce the number of exams displayed

Referring to FIG. 18, as the recognition performance of each case is evaluated, the result of increasing the number of attempts constituting the subset from 5 to 110 may be confirmed. The results listed above were calculated using the pupil response and button press response for the entire session presented to the subject. However, it takes time for the user to see all the exams in the session and it can be cumbersome to press the button for each exam. Performance improves because more data is available as the test progresses, but authentication and recognition must be performed quickly for field use. Therefore, one embodiment of the present invention observed how the recognition performance changes when only a few experiments are seen in the session, that is, when the number of tests decreases.

A 110-102 subset can be constructed by randomly dividing a session of 110 trials into k trials. Using the recorded data from the tests corresponding to each subset, the user was recognized and the results averaged over all subsets. This result can be influenced by how the subset is divided, so repeat this process several times to create a random subset. The preference index, eye bias, and fusion of these two features were used as features.

As shown, the user recognition rate increases as the number of attempts to construct a subset increases. The performance improvement is relatively fast in the subset but slower in the subset. Performance was higher in the order of convergence, preference index, and gaze bias, and the order remained the same even though the number of trials varied. The results of 110 trials were 91.4%, the results of 55 trials were 87.1%, and the results of 25 trials were 75.4%. Even a 50 percent reduction in performance was only 5 percent. The performance degradation was 16.8% when the number of tests was 75%. The results show that performance is guaranteed even if the number of tests is cut in half and three quarters in the current paradigm, and the user recognition is possible within a short time.

19 shows a normalized distribution of dissimilarities from the same user and different users for each feature.

Subject identification of a single feature

Referring to FIG. 19, the single feature showed different performance in user recognition and authentication. Performance is highest in order of preference index, high resolution scan path, low resolution scan path and gaze bias. The high performance of subject recognition and authentication is due to the discrimination between dissimilarities calculated by the same subject and other subjects. For higher performance features, two distinct distribution values can be distinguished.

To evaluate the user identity of each feature, Kullback-Leibler (KL) divergence between the distribution of criticism within and between users was calculated. The KL divergence of the distribution among users in the user distribution was 0.243 for the preference index, 0.189 for the gaze bias, 0.153 for the high resolution scan path and 0.208 for the low resolution scan path. Similarly, the KL divergence for the intra-user distribution in the inter-user distribution was 0.207 for the preference index, 0.075 for the gaze bias, 0.171 for the high resolution scan path and 0.082 for the low resolution scan path. If we average KL divergence from user to user and from user to user, the rank of average KL divergence is the same as that of performance.

Change preferences over time

Preference can change due to multiple personal experiences, and the decision process itself can change it. This may raise questions about whether the configuration is stable enough to provide user authentication over time. The results of this embodiment confirmed that the authentication and recognition performance is guaranteed even after 1 week, 2 weeks and 1 year after registration.

In the proposed method, the idea of presenting several experiments to the user was to ensure stability over time. If you use multiple tests with multiple categories of images, the preferences are retained for other tests, even if the preferences are changed for some tests. If authentication is successful, the system can be configured to continuously update user preference information.

When constructing a test, selecting an image that is less affected by time changes can improve the temporal robustness of the system. Images directly related to the user, such as themselves, familiar faces or past experiences, have a higher preference than images not related to the user. Thus, when a test is constructed using self-diagnostic images, the preference remains more stable over time.

Scan path resolution

The present invention has generated two types of scan path data: high resolution scan path and low resolution scan path. The high resolution scan path represents the gaze position at the x and y coordinates of the screen, and the low resolution scan path represents information of which of the four images presented by the user.

The low resolution scan path data is meaningful even when the authentication and recognition results using the low resolution scan path data are lower than the results using the high resolution scan path. In the experiments of the present invention, eye tracking devices with high spatial accuracy were used to track the eyes of the user, but it is difficult to use such expensive equipment in everyday life. Less expensive and simple eye movement recording systems have been developed, but so far the accuracy of these systems is relatively low. Authentication and recognition results using low resolution scan paths mean that the proposed method exhibits acceptable levels of performance even when the security system is operating on devices with limited spatial accuracy.

Intruder test mimics scan path

When the intruder steals information from the scam path and exercises with that information, eye movement speed is determined by the inherent characteristics of other eye movement devices in relation to whether the system is safe, so the gaze order in both time and space domains. It is difficult to follow. Two intruders were recruited to test this scenario. In the test, the intruder must first look at the unique user's eye sequence video and imitate as much as possible. Two intruders performed imitations on 20 attempts, but succeeded in creating one or three similar scan path shapes. For the test they successfully imitate, there was a time delay in eye movement between the true user and the intruder. This recall process delays the time it takes for an attacker to reclaim memory in the scan path. For this reason, the intruder has stolen the actual user's eye movement information, but it is impossible to copy the scan path.

This embodiment proposes a system for authenticating, recognizing, and evaluating performance by using preference and eye movement data. Since preferences vary from person to person, they have been proposed as a means to overcome the disadvantages of memory-based authentication. In an embodiment of the present invention, the present invention showed a series of tests for a user and performed the preferred selection task in each test. The preferences were entered on the keyboard and the pupils were recorded during the image observation. For authentication and recognition, a combination of preferences and scan path information derived from button presses (explicit preferences), eye movements (implicit preferences) were used.

20 shows user authentication performance according to temporal and spatial resolution.

Referring to FIG. 20, as described above, the identification performance increases as the spatial resolution increases. Average identification results were obtained for each spatial and temporal resolution. The time interval between development data and evaluation data also affects performance, so identification results are grouped and averaged over time intervals. The increase in identification speed was greatest when the spatial resolution increased from 2x2 to 4x4. This increase in performance was statistically significant when tested with a paired t-test (p <0.005).

21 shows a normalized distribution of scan path dissimilarity within and between users at spatial resolutions of 2x2 and 4x4.

Referring to FIG. 21, the present embodiment also shows that the difference in scan paths between users in a user can be quantitatively evaluated by using KL (Kullback-Leibler) divergence. KL divergence can be calculated by comparing the relative entropy of one distribution with another entropy to calculate the difference between the two distributions. The KL divergence for the inter-user distribution in the intra-user distribution increased to 0.199 for 2 Х 2 and 0.290 for 4 Х 4. Similarly, the KL divergence from the inter-user distribution to the intra-user distribution was 0.163 for 2 Х 2 and increased to 0.215 for 4 Х 4. When the spatial resolution was higher than 4 Х 4, the KL divergence did not increase and therefore the identification performance did not increase. 10 shows a normalized distribution of scan path dissimilarity within and between users for 2x2 and 4x4.

Fig. 22 shows the distribution of dissimilarity according to the sample level.

Referring to FIG. 22, since scan path dissimilarity is obtained from the sum of sample-level dissimilarities, differences in scan path dissimilarity between users and users according to different spatial resolutions can be obtained from dissimilarity calculations at the sample level. . This example analyzes the distribution of user scan path dissimilarities within the sample-level for different spatial resolutions. When the spatial resolution decreases to 16 Х 16 and 4 Х 4, as shown in FIG. 22, the overall distribution of the sample-level scan path dissimilarity is similar to that of the original data, but at 2 Х 2, the sample-level ratio The distribution of similarity was too sparse to reflect the original distribution. In the distribution at 2 Х 2 resolution, since there are only three dissimilar values, the ability to distinguish users using the calculated dissimilarity was limited. Spatial resolutions higher than 4x4 can yield a wider variety of sample level scan path differences to better distinguish users.

For time resolution, identification performance is maintained when the sampling frequency is reduced to 10 Hz, which is 1/12 of the original resolution. If the sampling frequency is reduced from 10 Hz to 5 Hz, the identification is greatly reduced. The results show that user identification performance is guaranteed when the sampling frequency is greater than 10 Hz.

In addition, as the time interval between development and evaluation data increases, the identification rate decreases. However, the trend between spatial or temporal resolution and identification results was similar for different time intervals.

Effect of Measuring Mismatch

Identification results from Euclidean distance and DTW scores were compared for each spatial and temporal resolution. Even if the same person observes the same image repeatedly, the search paths are not the same. The DTW calculates the scan path difference taking into account some differences in the time domain, so the DTW score was expected to be superior to the Euclidean distance. In FIG. 20, the identification performance of the DTW score was in most cases higher than the identification performance of Euclidean distance as expected. However, when the spatial resolution is 2 Х 2, the performance of the DTW score is worse than that of the Euclidean distance.

For 2 Х 2 resolution, the gaze position is encoded with 4 indexes, which corresponds to the display image of the 2 Х 2 layout. The order of the gazed item index is too simple to reveal the individual characteristics of eye movement. Significant performance degradation from 4 Х 4 resolution to 2 Х 2 resolution and the sparseness of the sample-level scan path mismatch distribution at 2 Х 2 resolution support this analysis. Therefore, 4 Х 4 is the minimum resolution at which individual characteristics of eye movements are expressed, and dissimilarity measurements can distinguish subjects.

Eye movement data can be obtained using a variety of devices having various spatial and temporal resolutions. However, the optimal specification for an eye movement based user identification system has not been investigated yet. The present invention implements a scan path based user identification system and evaluates the performance at various spatial and temporal resolutions of visual acuity data. The results show that the effect of spatial resolution is large, but the effect of temporal resolution is relatively low. If the stimulus is displayed in a 2x2 layout, using 4x4 spatial resolution and a sampling frequency of 10Hz ensures user identification. This result implies that the optimal spatial and temporal resolution should be determined in consideration of the stimulus representation and the criticism algorithm. The impact of the time interval between development and evaluation was important in all cases, so development data should be updated regularly to ensure performance. The present invention can help to select the specification of the eye tracking device when building the gaze-based input system.

23 is a diagram of an iterative authentication algorithm of a user authentication system using pupil reaction according to an embodiment of the present invention.

Referring to FIG. 23, the user authentication system may determine whether the current user is a legitimate user by continuously and repeatedly performing authentication after the user logs in through an authentication system such as gaze, fingerprint, face recognition, and password. . The user authentication system may exclude the invalid user through continuous re-authentication when an invalid user is authenticated by imitating a legitimate user's eye movement or authenticated with an error or a coincidence.

The user authentication system may include different authentication programs depending on the security level. The security level of the user authentication system may be divided into a plurality of security levels. The security level may include two or more security levels.

According to an embodiment of the present disclosure, a low security level may perform user authentication using information including a biometric authentication system, a motion, a password, a face, and a motion.

According to an embodiment of the present invention, a high security level may perform user authentication using information including a PIN, a face, a fingerprint, a voice, and the like.

According to an embodiment of the present disclosure, the user authentication system may drive an authentication program according to the security level, and collect user gaze motion data according to the program to perform user authentication.

According to an embodiment of the present disclosure, the user authentication system may collect user's gaze movement data according to the security level and repeatedly perform user authentication. The repetition of the user authentication may be determined according to the security level and the authentication method.

According to an embodiment of the present disclosure, the user authentication system may continuously store and update information on a trust level of a user.

According to an embodiment of the present disclosure, if it is determined that the current user is not a legitimate user by repetitive authentication, the user authentication system may deny access of the current user.

24 is an example of an iterative authentication system of a user authentication system using pupil reaction according to an embodiment of the present invention.

Referring to FIG. 24, continuous user authentication based on pupil response (including logo)

1) Different programs may have different logos. The program can be used according to the security level.

2) During the loading time, people wait until it is ready to display eye movements and obtain them for use in user authentication.

The first threshold may be an authentication threshold and the second threshold may be a reject threshold.

Although the preferred embodiments of the present invention have been shown and described above, the present invention is not limited to the specific preferred embodiments described above, and the present invention belongs to the present invention without departing from the gist of the present invention as claimed in the claims. Various modifications can be made by those skilled in the art, and such changes are within the scope of the claims.

Claims (10)

A storage unit for storing user information including eye response data;
A control unit predicting a user by using the user's eye response and the user information according to a visual stimulus;
A response acquiring unit for acquiring a pupil response of the user according to the visual stimulus;
The control unit,
A user determination unit which compares the user feature extracted based on the acquired pupil response with the user information stored in the storage unit and determines a matching probability with an ID presented by the user; And
And an authentication unit for determining whether to authenticate and access according to a probability of matching with the ID.
The authentication unit,
If the probability of matching with the ID is greater than or equal to the authentication threshold, the user is allowed to access,
If the probability of matching the ID presented by the user is a value between the authentication threshold and the rejection threshold, it is determined to perform a further authentication by presenting a new visual stimulus,
The matching probability is determined for each test, and the user authentication system using the pupil reaction, characterized in that the matching probability in a plurality of tests are accumulated and updated. The method of claim 1,
The authentication unit,
And the user's access is denied when the probability that the user's ID matches the ID presented by the user is less than or equal to the rejection threshold by comparing the acquired pupil response with the stored information.
delete The method of claim 1,
The authentication unit,
User authentication system using the pupil reaction to continuously perform a new authentication in the state of allowing the user access.
The method of claim 1,
The control unit,
The user using the pupil response using the data of the storage to present a visual stimulation that can induce a highly consistent pupil response, or a pupil response with a large difference from other user IDs from the ID presented by the user Authentication system.
The method of claim 5,
The control unit,
And a pupil response that obtains each of at least one other user data and an ID presented by the user.
The method of claim 2,
The control unit,
User authentication system using the pupil response that gives different authentication threshold and rejection threshold according to the area that the user wants to access.
The method of claim 1,
The pupil response is a user authentication system using the pupil response, characterized in that the signal derived from at least one of eye position, intermittent movement, pupil size, eye blink.
The method of claim 1,
The authentication unit,
If the user ID matching probability is less than the authentication threshold and more than the rejection threshold, the user authentication system using the pupil response to request the additional authentication of another method to determine the access or rejection.
The method according to claim 1 or 7,
The visual stimulus is a user authentication system using a pupil reaction that is a combination of one or more still or moving images.

Images