KR102111858B1 - Method and system for authenticating stroke-based handwritten signature using machine learning - Google Patents

Abstract

A method and system for stroke-based handwritten signature authentication using machine learning are presented. A stroke-based handwritten signature authentication method according to an embodiment includes extracting a stroke image sequence and a stroke time sequence from collection points of a signature received from a user; Extracting a feature of each stroke image from the stroke image sequence, and extracting a feature vector by extracting features of each stroke time from the stroke time sequence; And determining whether the signature is authentic by using the extracted feature vector.

Description

METHOD AND SYSTEM FOR AUTHENTICATING STROKE-BASED HANDWRITTEN SIGNATURE USING MACHINE LEARNING}

The following embodiments relate to the authentication of handwritten signatures, and more particularly, the method and system for authenticating handwritten signatures that distinguish between authentic signatures of different people using machine learning, and also authentic signatures and counterfeit signatures imitating them. It is about.

Recently, biometric techniques for authenticating themselves using biological characteristics that are distinct from each other are being developed. Fingerprints, irises, and handwritten signatures are used as general biometric means. Of these, handwritten signatures have the advantage that they can be easily applied compared to other methods because they are already widely used in society.

However, in the biometric method using a handwritten signature, there is a problem in that the true signature of the same person is not completely matched every time, and the true signature and the signature imitating it are not easily distinguishable. Since this causes the security problem of the authentication system, in the case of the authentication system using a handwritten signature, a method of distinguishing the user's true signature from the signature imitating it or another's true signature must be accompanied.

Conventional techniques for handwritten signature authentication include methods using a Dynamic Time Warping (DTW) algorithm or a Recurrent Neural Network (RNN), which is a type of neural network.

The DTW-based method includes collecting the original signatures of the user, dynamically comparing the original signatures of the new and existing collected users to measure similarity, and if the obtained similarity is lower than a predetermined threshold, the signature is a forged signature. If it is high, it consists of a step of judging by the signature that is lost. This signature authentication method has a risk that users' signature information can be leaked from the database at any time because all original signatures are databased for comparison.

In order to solve the above problem, feature extraction based signature authentication methods have been proposed. Among these, a signature authentication method using a recurrent network includes collecting signatures, pre-processing the collected signatures, extracting a feature vector of a lost signature from the collected signatures, and using the extracted feature vector It consists of a process of discriminating by a signature of a falsity or a forgery. The step of extracting the feature vector from the signature uses a recurrent network that outputs a signature feature vector as an input and a sequence of collection points constituting the signature.

The feature extraction-based signature authentication method has an advantage in that there is no risk of the user's original signature being leaked because the original signatures are not databaseized.

However, the above method has a problem of decreasing gradient that occurs as the sequence length of input data of the recurrent network increases. Because handwritten signatures generally consist of a sequence of hundreds of collection points, the network does not converge correctly during the learning process of the recurrent network, and as a result, the accuracy of signature authentication decreases.

Korean Patent Publication No. 10-2017-0007068 relates to such a handwritten signature authentication system and method, in which a handwritten signature image registered in advance for a user's handwritten signature, and a traced handwritten signature image to be tracked at the time of handwriting signature, are collected at the time of handwriting signature. Describes a technique for performing handwritten signature authentication by comparing one or more of handwritten signature behavior characteristics and one or more of the handwritten signature images restored by the handwritten signature behavior characteristics.

Korean Patent Publication No. 10-2017-0007068

Embodiments describe a method and system for stroke-based handwritten signature authentication using machine learning, and more specifically, a technique for distinguishing between real people's true signatures and also true signatures and counterfeit signatures using machine learning. Gives

The embodiments convert a collection point sequence constituting a handwritten signature into a stroke sequence, and then extract a feature vector to distinguish between a true signature and a forged signature, thereby using stroke-based handwriting using machine learning to improve the performance of signature authentication. It provides a signature authentication method and system.

A stroke-based handwritten signature authentication method according to an embodiment includes extracting a stroke image sequence and a stroke time sequence from collection points of a signature received from a user; Extracting a feature of each stroke image from the stroke image sequence, and extracting a feature vector by extracting features of each stroke time from the stroke time sequence; And determining whether the signature is authentic by using the extracted feature vector.

The step of extracting the stroke image sequence and the stroke time sequence from the collection points of the signature is to search for a collection point where the intensity of pressure applied to the screen is 0 in the sequence of the collection points of the received signature and search for a stroke sequence. Extracting; Generating the stroke image sequence using coordinate information on a screen of a collection point included in each extracted stroke sequence; And generating the stroke time sequence by estimating the time taken to draw the stroke through the number of collection points included in each of the stroke sequences.

Extracting the feature vector may include extracting features of each stroke image from the stroke image sequence; Extracting features of each stroke time from the stroke time sequence; And extracting a feature vector from the extracted feature of the stroke image and the feature of the stroke time.

In the step of extracting features of each stroke image from the stroke image sequence, features of each stroke image may be extracted from the stroke image sequence through a single or a plurality of convolution layers.

Extracting a feature vector from features of the stroke image and features of the stroke time may extract feature vectors from features of the stroke image and features of the stroke time through a single or multiple recurrent layers. have.

In the step of extracting features of each stroke time from the stroke time sequence, a time period including the stroke time may be searched to output a representative vector of the previously learned time period.

Extracting features of each stroke time from the stroke time sequence may further include learning a representative vector for the stroke time interval. Here, the step of learning the representative vector for the stroke time interval comprises: learning an auto-encoder to output the same stroke image when one stroke image is input from the training data; Encoding a stroke image of the training data using a trained auto-encoder encoder network; Dividing the range from 0 second to the maximum stroke time of the learning data into natural time K time intervals that can be selected as optimal values in a conventional machine learning discrimination process; Assigning each stroke of the learning data to a stroke time interval according to the stroke time; And generating a representative vector of the corresponding time period by obtaining an average of the encoded stroke images of the strokes included in each stroke time period.

In the step of determining whether the signature is authentic using the feature vector, the feature vector, which is an output, may be input and compared with the feature vectors of the user's true signature to determine whether the corresponding signature is a true signature or a fake signature.

According to another embodiment, a method for authenticating a handwriting signature based on a stroke may include receiving a signature of a user consisting of a sequence of collection points received by hand through a user terminal; Determining whether the signature is authentic; And transmitting the determined result to the user terminal. At this time, determining whether the signature is authentic includes: extracting a stroke image sequence and a stroke time sequence from collection points of the signature received from the user; Extracting a feature of each stroke image from the stroke image sequence, and extracting a feature vector by extracting features of each stroke time from the stroke time sequence; And determining whether the signature is authentic by using the extracted feature vector.

A stroke-based handwritten signature authentication system according to another embodiment includes: a signature pre-processor for extracting a stroke image sequence and a stroke time sequence from collection points of a signature received from a user; A feature extraction unit for extracting features of each stroke image from the stroke image sequence, and extracting a feature vector by extracting features of each stroke time from the stroke time sequence; And a signature determining unit that determines whether a signature is authentic by using the extracted feature vector.

An input unit that receives a user's signature consisting of a sequence of collection points received by hand through a user terminal; And an output unit that transmits the determined result to the user terminal.

The signature pre-processing unit extracts a stroke sequence by finding a collection point where the intensity of pressure applied to the screen becomes 0 in the sequence of the collection points of the received signature, and collects the stroke sequence included in each extracted stroke sequence The stroke image sequence may be generated using coordinate information on a screen of a dot, and the stroke time sequence may be generated by estimating the time taken to draw a stroke through the number of collection points included in each of the stroke sequences.

The feature extracting unit may include a convolution layer unit for extracting features of each stroke image from the stroke image sequence through a single or a plurality of convolution layers; A stroke time interval embedding layer for extracting features of each stroke time from the stroke time sequence; And a recurrent layer that extracts a feature vector from features of the stroke image and features of the stroke time extracted through a single or multiple recurrent layers.

The stroke time period embedding layer unit may search for a time period including the stroke time and output a representative vector of the pre-trained corresponding time period.

The signature determining unit may receive the output feature vector and compare it with the feature vectors of the user's true signature to determine whether the corresponding signature is a true signature or a fake signature.

According to embodiments, a stroke-based handwritten signature authentication method and system using machine learning to solve a decreasing gradient problem of a recurrent network by converting a sequence of collection points of a signature into a stroke image and a time sequence to reduce the length of the sequence Can provide

In addition, according to embodiments, a stroke-based handwritten signature authentication method and system using machine learning to improve the accuracy of signature authentication by distinguishing the difference in temporal characteristics between a stroke from a stroke time interval embedding layer and a stroke of a forged signature Can provide

1 is a view showing the configuration of a signature authentication system according to an embodiment.
2 is a view for explaining a feature extraction unit of a handwritten signature authentication system according to an embodiment.
3 is a view showing an example of a stroke time interval according to an embodiment.
4 is a flowchart of a method for authenticating a stroke-based handwritten signature according to an embodiment.
5 is a flow chart for learning a stroke time interval embedding layer according to an embodiment.
6 is a diagram for describing a user-independent feature learning model according to an embodiment.
7 is a diagram for explaining reconstruction (K = 5) of a time interval vector according to an embodiment.

Hereinafter, embodiments will be described with reference to the accompanying drawings. However, the described embodiments may be modified in various other forms, and the scope of the present invention is not limited by the embodiments described below. In addition, various embodiments are provided to more fully describe the present invention to those skilled in the art. The shape and size of elements in the drawings may be exaggerated for a more clear description.

The following embodiments can improve the performance of the signature authentication system by dividing the user's signature into a stroke sequence and extracting a feature vector so that the stroke sequence of the forged signature and the stroke sequence of the forged signature are distinguished.

The configuration of the signature authentication system according to these embodiments will be described with reference to FIG. 1.

1 is a view showing the configuration of a signature authentication system according to an embodiment.

Referring to FIG. 1, the signature authentication system may include a stroke-based handwritten signature authentication system 200, and may further include a user terminal 100 according to an embodiment. Here, the stroke-based handwritten signature authentication system 200 may be a signature authentication server.

The user terminal 100 receives the user's handwritten signature and transmits it to the stroke-based handwritten signature authentication system 200, and when the authenticity of the signature is determined from the stroke-based handwritten signature authentication system 200, the user terminal displays the result. 100).

In addition, the signature collected by the user terminal 100 may consist of a sequence of collection points. The collection points are collected at regular time intervals, and may include information such as coordinates on the screen of the user terminal 100 and intensity of pressure applied to the screen.

Here, the user terminal 100 according to an embodiment of the present invention is described as a representative example of a mobile communication terminal capable of providing a launcher service connected to a communication network, but the terminal is not limited to a mobile communication terminal, and all information communication devices, It can be applied to various terminals such as multimedia terminals, wired terminals, fixed terminals, and IP (Internet Protocol) terminals. Also, the terminal

Here, the user terminal 100 includes a mobile phone, a portable multimedia player (PMP), a mobile Internet device (MID), a smart phone, a desktop, a tablet PC, a notebook, and a netbook. It can be advantageously used when a mobile terminal has various mobile communication specifications such as (Net book) and information communication equipment.

When the signature is transmitted to the stroke-based handwritten signature authentication system 200, the signature may be determined through the signature pre-processing unit 210, the feature extraction unit 220, and the signature determination unit 230 in order to determine whether the signature is authentic.

The signature pre-processing unit 210 of the stroke-based handwritten signature authentication system 200 may extract a stroke sequence by finding a collection point where the intensity of pressure applied to the screen is 0 in the sequence of the received collection points. In addition, the signature pre-processing unit 210 may generate a stroke image using coordinate information on the screen of a collection point included in each extracted stroke. In addition, since the collection points are collected at regular time intervals, it is possible to estimate the time taken to draw the strokes by the number of collection points included in each stroke.

The feature extraction unit 220 is composed of a neural network, and may include a convolutional layer unit 221, a recurrent layer unit 222, and a stroke time embedding layer unit 223.

The stroke-based handwritten signature authentication system will be described in more detail below.

The stroke-based handwritten signature authentication system according to an embodiment may include a signature pre-processing unit 210, a feature extraction unit 220, and a signature determination unit 230. According to an embodiment, the stroke-based handwritten signature authentication system may further include an input unit and an output unit, and may further include a user terminal 100 as described above. Here, the stroke-based handwritten signature authentication system may be a signature authentication server that wirelessly communicates with a user terminal through a network as described above.

The signature pre-processing unit 210 may extract a stroke image sequence and a stroke time sequence from collection points of signatures received from the user. Here, the signature may be a signature received from a user through the user terminal 100, and may be made of a sequence of collection points.

The signature pre-processing unit 210 finds a collection point where the intensity of pressure applied to the screen becomes 0 in the sequence of the collection points of the received signature, extracts a stroke sequence, and collects included in each extracted stroke sequence A stroke image sequence may be generated using coordinate information on a screen of a dot, and a stroke time sequence may be generated by estimating the time taken to draw a stroke through the number of collection points included in each stroke sequence.

The feature extraction unit 220 may extract features of each stroke image from the stroke image sequence, and extract a feature vector by extracting features of each stroke time from the stroke time sequence.

More specifically, the feature extraction unit 220 is composed of a neural network, and may include a convolutional layer unit 221, a recurrent layer unit 222, and a stroke time embedding layer unit 223.

The convolution layer unit 221 may extract characteristics of each stroke image from the stroke image sequence through a single or multiple convolution layers.

The recurrent layer 222 may extract a feature vector from features of a stroke image and features of a stroke time extracted through a single or multiple recurrent layers.

The stroke time section embedding layer unit 223 may extract characteristics of each stroke time from the stroke time sequence, and search for a time period including the stroke time to output a representative vector of the previously learned time period. .

In particular, the stroke time section embedding layer 223 trains an auto-encoder to output the same stroke image when one stroke image is input from the training data, and learns the stroke image of the training data. Encoding is performed using the encoder network of the auto-encoder, and the range from 0 second to the maximum stroke time of the learning data is divided into K number of natural time periods that can be selected as the optimal value in the discrimination process of conventional machine learning. , Each stroke of the learning data is allocated to a stroke time interval according to the stroke time, and a representative vector of the corresponding time interval can be generated by obtaining an average of the encoded stroke images of strokes included in each stroke time interval.

The signature determining unit 230 determines whether the signature is authentic by using the extracted feature vector, receives the feature vector as an output, compares it with the feature vectors of the user's true signature, and determines whether the corresponding signature is a true signature or a counterfeit signature. Can be determined.

Meanwhile, according to an embodiment, the stroke-based handwritten signature authentication system may further include an input unit and an output unit. The input unit may receive a signature of a user consisting of a sequence of collection points received by hand through the user terminal 100, and the output unit may transmit the determined result to the user terminal 100.

Hereinafter, the feature extraction unit 220 will be described in more detail with reference to FIGS. 1 and 2.

2 is a view for explaining a feature extraction unit of a handwritten signature authentication system according to an embodiment.

The signature transmitted to the stroke-based handwritten signature authentication system 200 may be converted into a stroke image sequence and a stroke time sequence through the signature pre-processing unit 210. Each stroke image of the stroke image sequence may be converted into a real vector containing characteristics of the image through a convolution layer 221 commonly used to extract characteristics of the image. In addition, each stroke time of the stroke time sequence may be converted into a real vector by passing through the stroke time interval embedding layer 223 to be described below. The sequences of the two real vectors may be combined according to the sequence index and transmitted to the recurrent layer unit 222. The recurrent layer unit 222 may receive the combined sequence as input and finally output a signature feature vector.

The stroke time section embedding layer 223 may receive a stroke time sequence as an input. Prior to the description, the time period refers to a section in which real sections from 0 to the maximum stroke time indicated in the training data are divided into K hyper-parameters to be learned without overlapping. Therefore, each stroke time is always included in one time interval. Also, the time period may be divided such that the number of strokes included in each time period is similar or equal.

3 is a view showing an example of a stroke time interval according to an embodiment.

). Referring to FIG. 3, it is a diagram showing an example of a stroke time period when K = 3 to help understanding the stroke time period. Here, the x-axis means the stroke time, and the y-axis means the number of corresponding stroke times in the training data. Stroke time intervals may be divided such that the number of strokes included in each stroke time interval is the same (ie,

).

The stroke time section embedding layer unit 223 searches for a time section including the stroke time, and outputs a representative vector of the pre-trained corresponding time section. The learning process of the representative vector of the stroke time interval will be described in more detail below.

The signature determining unit 230 may receive the feature vector that is the output of the feature extraction unit 220 and compare it with the user's true signature feature vectors to determine whether the corresponding signature is a true signature or a fake signature. As the signature determining unit 230, a support vector machine (SVM), a logistic regression method, and the like, which are commonly used in classification in the field of machine learning, may be used.

4 is a flowchart of a method for authenticating a stroke-based handwritten signature according to an embodiment.

Referring to FIG. 4, the stroke-based handwritten signature authentication method includes extracting a stroke image sequence and a stroke time sequence from collection points of a signature received from a user (S110), and each of the stroke image sequences Extracting features of the stroke image, extracting features of each stroke time from the stroke time sequence (S120), and determining whether the signature is authentic using the extracted feature vectors (S130) It can be made including.

Here, the step of extracting the feature vector (S120) includes extracting features of each stroke image from the stroke image sequence, extracting features of each stroke time from the stroke time sequence, and features of the extracted stroke image and And extracting a feature vector from features of the stroke time.

Hereinafter, a method for authenticating a stroke-based handwritten signature according to an embodiment will be described in more detail.

The stroke-based handwritten signature authentication method according to an embodiment may be described in more detail using the stroke-based handwritten signature authentication system according to an embodiment described with reference to FIGS. 1 to 3. The stroke-based handwritten signature authentication system according to an embodiment may include a signature pre-processing unit 210, a feature extraction unit 220, and a signature determination unit 230.

In step S110, the signature pre-processing unit 210 may extract the stroke image sequence and the stroke time sequence from the collection points of the signature received from the user.

More specifically, the signature pre-processing unit 210 extracts a stroke sequence by finding a collection point where the intensity of pressure applied to the screen becomes 0 in the sequence of the collection points of the received signature, and extracts each stroke sequence. A stroke image sequence may be generated by using coordinate information on a screen of a collection point included in a stroke, and a stroke time sequence may be generated by estimating the time taken to draw a stroke through the number of collection points included in each stroke sequence.

In step S120, the feature extraction unit 220 may extract features of each stroke image from the stroke image sequence, and extract feature vectors by extracting features of each stroke time from the stroke time sequence.

Here, the feature extraction unit 220 is composed of a neural network, and may include a convolutional layer unit 221, a recurrent layer unit 222, and a stroke time embedding layer unit 223.

The convolution layer unit 221 may extract characteristics of each stroke image from the stroke image sequence, and extract characteristics of each stroke image from the stroke image sequence through a single or multiple convolution layers. can do.

The stroke time section embedding layer unit 223 may extract the characteristics of each stroke time from the stroke time sequence, and at this time, search for a time period including the stroke time and output a representative vector of the previously learned time period. Can be. The stroke time interval embedding layer 223 may further include learning a representative vector for the stroke time interval, which will be described in more detail below.

The recurrent layer unit 222 may extract a feature vector from features of the extracted stroke image and features of the stroke time, wherein the feature and stroke of the stroke image extracted through a single or multiple recurrent layers Feature vectors can be extracted from features of time.

In step S130, the signature determining unit 230 may determine whether the signature is authentic using the extracted feature vector. In particular, the signature determining unit 230 may receive the output feature vector and compare it with the feature vectors of the user's true signature to determine whether the corresponding signature is a true signature or a fake signature.

On the other hand, the stroke-based handwritten signature authentication method according to another embodiment, the step of receiving a user's signature consisting of a sequence (sequence) of a collection point (point) received by hand through the user terminal, determining whether the signature is authentic And transmitting the determined result to the user terminal. At this time, the step of determining the authenticity of the signature may include: extracting a stroke image sequence and a stroke time sequence from collection points of the signature received from a user, extracting features of each stroke image from the stroke image sequence, and , Extracting a feature vector of each stroke time from the stroke time sequence, and extracting a feature vector, and determining whether the signature is authentic using the extracted feature vector. The stroke-based handwritten signature authentication method according to another embodiment has the same or similar configuration to the stroke-based handwritten signature authentication method according to the above-described embodiment, so a detailed description thereof will be omitted.

The process of learning the representative vector of the stroke time interval will be described in detail with reference to FIG. 5.

5 is a flow chart for learning a stroke time interval embedding layer unit according to an embodiment.

Referring to FIG. 5, the stroke time interval embedding layer unit 223 may further include learning a representative vector for the stroke time interval.

The step of learning the representative vector for the stroke time interval comprises: learning an auto-encoder to output the same stroke image when one stroke image is input from the training data (S210), Encoding the stroke image using the trained encoder network of the auto-encoder (S220), the range from 0 second to the maximum stroke time of the training data can be selected as an optimal value in the discrimination process of conventional machine learning. Dividing into natural K number of time periods (S230), assigning each stroke of the learning data to the stroke time period according to the stroke time (S240), and the encoded strokes of strokes included in each stroke time period It may be made by including the step (S250) of generating a representative vector of the corresponding time interval by obtaining the average of the image.

In step S210, the stroke time section embedding layer unit 223 may train an auto-encoder to output the same stroke image when one stroke image is input from the training data. The representative vector of the stroke time interval is learned using an auto-encoder, and the auto-encoder can use the collected stroke images for learning. At this time, the learning of the auto-encoder is characterized in that when one stroke image is input, the same stroke image is output.

In step S220, after learning the auto-encoder, all the stroke images of the training data may be encoded using the learned encoder network of the auto-encoder. That is, the stroke time section embedding layer unit 223 may encode the stroke image of the training data using the trained auto-encoder encoder network.

Then, in step S230, the stroke time section embedding layer unit 223 is a natural number K that can be selected as an optimal value in the range of 0 seconds to the maximum stroke time of the training data in the discrimination process of conventional machine learning. It can be divided into two time intervals. At this time, the natural number K is a hyper-parameter and may be selected as an optimal value in the process of discrimination in conventional machine learning.

When the time period is divided, in step S240, the stroke time period embedding layer unit 223 may allocate each stroke of the learning data to the stroke time period according to the stroke time.

Thereafter, in step S250, the stroke time interval embedding layer unit 223 may generate a representative vector of the corresponding time interval by obtaining an average of the encoded stroke images of the strokes included in each stroke time interval. .

As described above, according to the embodiments, the stroke-based handwritten signature authentication method using machine learning to improve the accuracy of signature authentication by distinguishing the difference between the temporal characteristics of the signature from the stroke time interval embedding layer and the forged signature stroke, and System.

The following describes a stroke-based handwritten signature authentication method using machine learning and a signature verification experiment using a system according to the present embodiments.

For online handwritten signature verification, a new network structure based on Long-Term Recurrent Convolutional Network (LRCN) can be proposed. In addition, a forgery-sensitive loss function can be proposed to distinguish between a true signature and a forged signature. Here, a signature verification experiment was conducted using a SUSIG data set containing both true and counterfeit signatures.

The handwritten signature verification system may be composed of a user-independent feature learning step and a user-dependent classification step. In the user-independent feature learning stage, the LRCN model is trained to extract global features that can be distinguished from forged and forged signatures using the signatures of many users. In the user-dependent classification step, the learned LRCN model extracts the features of the input signature and classifies whether the feature is a true signature or a counterfeit signature by using each user's own support vector machine (SVM). The reason for dividing the verification system into two stages is that it is unrealistic to acquire counterfeit signature data of newly registered users in the actual registration system.

6 is a diagram for describing a user-independent feature learning model according to an embodiment.

Referring to FIG. 6, an LRCN feature learning model including a convolutional neural networ (CNN) layer, a stroke time interval embedding layer, and a long short term memory (LSTM) layer is shown.

을 입력으로 입력으로 {y,s}을 출력으로 하여 LRCN 모델이 학습하게 된다. 이 때 x_i는 획의 이미지, t_i는 획을 그리는데 걸린 시간, m은 해당 서명을 이루는 총 획수를 의미한다. 또한, y는 해당 서명의 사용자 ID 를, s는 해당 서명이 진 서명인지 혹은 숙련된 위조 서명인지를 나타낸다.In learning,

As input, {y, s} as input, LRCN model learns. In this case, x _i is the image of the stroke, t _i is the time taken to draw the stroke, and m is the total number of strokes that make up the signature. Also, y represents the user ID of the signature, and s represents whether the signature is a true signature or a skilled counterfeit signature.

The CNN layer and the stroke time interval embedding layer respectively encode the stroke image x _i and the stroke time t _i as a real vector, and the LSTM layer extracts the sequence of the encoded real vector as a feature vector.

The feature learning model aims that the extracted feature vector has the following features. (1) The feature vectors of the same user's signature must be similar; (2) The feature vectors of different users' signatures should be distinguished; (3) Distinguished signatures from skilled signatures must be distinguished. In the following description, it is described how the proposed model satisfies the features mentioned above.

We propose a forgery-sensitive loss function consisting of a weighted sum of user classification loss and skilled counterfeiter classification loss.

First, in the case of a user classification loss function, it can be learned to minimize the following multi-label cross entropy.

[Equation 1]

는 입력 서명이 목표 사용자로 분류될 확률, n_u는 전체 사용자의 수를 의미한다. 학습 과정에서, 모델은 서로 다른 유저를 구별하도록 학습되므로 모델의 히든 레이어는 유저 내(intra-user) 서명들이 가까이 있고 유저간(inter-user) 서명들이 구별되도록 입력 서명을 특징벡터로 변환하게 된다.Here, y _i is a one-hot encoding vector where only the value at the index of the target user ID is 1,

Is the probability that the input signature will be classified as the target user, n _u is the total number of users. In the learning process, the model is trained to differentiate between different users, so the hidden layer of the model converts the input signature into a feature vector so that intra-user signatures are close and inter-user signatures are distinguished. .

However, it is not possible to extract feature vectors to distinguish skilled counterfeit signatures that are very similar to true signatures only by loss of user classification. To solve this problem, skilled counterfeit classification losses can be proposed.

[Equation 2]

이에 대한 모델의 예측 확률을 뜻한다. 모델은 숙련된 위조 분류 손실을 줄이기 위해, 진 서명들과 숙련된 위조 서명이 상대적으로 먼 거리의 특징 공간으로 투영되도록 학습될 수 있다.Here, S indicates whether the given input is a true signature or a skilled counterfeit signature

It means the predicted probability of the model. The model can be trained to reduce the loss of skilled counterfeit classification, so that the true signatures and the skilled counterfeit signature are projected into a relatively long feature space.

Finally, in the feature learning stage, the weighted sum of two proposed losses can be used as a loss function.

[Equation 3]

는

에 대한 가중치 하이퍼 파라미터이다.

The

Is the weighted hyperparameter for.

이므로 모든 획 시간을 각각 임베딩 벡터로 변환하는 것은 불가능하다. 대신, 연속되는 시간 범위를 K 개 시간 구간으로 나누고 이를 벡터로 변환한다. 또한 각 시간 구간에 포함되는 학습 데이터의 획의 숫자가 균형적으로 분포하도록 획 구간을 나누었다. 그 후, 아래와 같이 각 시간 구간에 대한 벡터 표현을 얻을 수 있다.Here, we propose a vector representation of the stroke time that maintains information about the length of the time in embedding the time taken to draw the stroke. The range of consecutive time is

Therefore, it is impossible to convert all stroke times into embedding vectors. Instead, it divides consecutive time ranges into K time periods and converts them into vectors. In addition, the stroke section is divided so that the number of strokes of the learning data included in each time section is distributed in a balanced manner. After that, a vector representation for each time interval can be obtained as follows.

[Equation 4]

는 학습 데이터에서 진 서명의 모든 획 이미지들을 이용하여 학습된 CNN 기반 오토-인코더로부터 추출된 실수 벡터이다.S _{i, j} is an image of the strokes included in the time period S _i , and n _i is the total number of strokes included in the time period S _i .

Is a real vector extracted from a CNN-based auto-encoder trained using all stroke images of a true signature from the training data.

7 is a diagram for explaining reconstruction (K = 5) of a time interval vector according to an embodiment.

Referring to FIG. 7, an image obtained by reconstructing a time interval vector using a decoder of a learned auto encoder is shown. From the sharpness of the image and the number of lines drawn, it can be seen that each time period is uniquely expressed. In addition, the sharpness of the image is higher and the number of lines is higher than that of the low time zone. This indicates that the proposed time period maintains information about the length of time.

In this experiment, SUSIG, a well-known online signature data set, was used. The data set consists of 94 signature data, with 20 signatures per person and 10 skilled counterfeits. In the pre-processing process of the signature, stroke images were extracted using the x-y coordinates and pen up / down information of each collection point of the signature, and the stroke time was estimated by the number of collection points included in each stroke. The feature learning model was trained using 54 of the 94 signatures, and 6 signatures were used to adjust the hyperparameters. The user-dependent classification model was trained and tested using 34 signatures.

Six CNN layers, two LSTM layers, and one deep neural network (DNN) for user classification and skilled counterfeiter classification were used as the feature learning model. For the CNN layer, 32X3X3 kernel functions were used.

Table 1 shows performance comparison according to whether or not to embed a stroke time interval.

[Table 1]

=0.8를 이용하였다.As a result of the experiment, both models were trained through the proposed loss function. Equal Error Rate (EER) and AREA UNDER AN ROC (AUC) evaluation scales were used for performance comparison. For SVM, Linear, RBF, Poly kernels, K = 10 for hyper parameters,

= 0.8 was used.

EER _r and AUC _r refer to experiments on random forgery and EER _s and AUC _s refer to experiments on skilled counterfeiting. When the two models were compared, both EER and AUC showed a performance improvement of more than 10% for skilled counterfeit experiments by adding embedding of stroke time intervals. These results show that the proposed stroke time embedding is very effective in capturing the different aspects between the lost signature and the skilled counterfeit signature.

The device described above may be implemented with hardware components, software components, and / or combinations of hardware components and software components. For example, the devices and components described in the embodiments include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor (micro signal processor), a microcomputer, a field programmable array (FPA), It may be implemented using one or more general purpose computers or special purpose computers, such as a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. The processing device may run an operating system (OS) and one or more software applications running on the operating system. In addition, the processing device may access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, a processing device may be described as one being used, but a person having ordinary skill in the art, the processing device may include a plurality of processing elements and / or a plurality of types of processing elements. It can be seen that may include. For example, the processing device may include a plurality of processors or a processor and a controller. In addition, other processing configurations, such as parallel processors, are possible.

The software may include a computer program, code, instruction, or a combination of one or more of these, and configure the processing device to operate as desired, or process independently or collectively You can command the device. Software and / or data may be interpreted by a processing device, or to provide instructions or data to a processing device, of any type of machine, component, physical device, virtual equipment, computer storage medium or device. Can be embodied in The software may be distributed over networked computer systems, and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, or the like alone or in combination. The program instructions recorded in the medium may be specially designed and configured for the embodiments or may be known and usable by those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs, DVDs, and magnetic media such as floptical disks. -Hardware devices specifically configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, etc., as well as machine language codes produced by a compiler.

As described above, although the embodiments have been described by a limited embodiment and drawings, those skilled in the art can make various modifications and variations from the above description. For example, the described techniques are performed in a different order than the described method, and / or the components of the described system, structure, device, circuit, etc. are combined or combined in a different form from the described method, or other components Alternatively, even if replaced or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

100: user terminal
200: stroke-based handwritten signature authentication system
210: signature preprocessor
220: feature extraction unit
230: signature discrimination unit
221: convolution layer
222: Recurrent layer
223: Stroke time section embedding layer

Claims (15)

Extracting a stroke image sequence and a stroke time sequence from collection points of a signature input from a user;
Extracting a feature of each stroke image from the stroke image sequence, and extracting a feature vector by extracting features of each stroke time from the stroke time sequence; And
Determining whether a signature is authentic using the extracted feature vector
Including,
Extracting the feature vector,
Extracting features of each stroke image from the stroke image sequence;
Extracting features of each stroke time from the stroke time sequence; And
Extracting a feature vector from the extracted feature of the stroke image and the feature of the stroke time
It includes,
Extracting a feature vector from the feature of the stroke image and the feature of the stroke time,
Extracting feature vectors from features of the stroke image and features of the stroke time through a single or multiple recurrent layers
Characterized in that, stroke-based handwritten signature authentication method. According to claim 1,
Extracting the stroke image sequence and the stroke time sequence from the collection points of the signature,
Extracting a stroke sequence by finding a collection point where the intensity of pressure applied to the screen becomes 0 in the sequence of the collection points of the received signature;
Generating the stroke image sequence using coordinate information on a screen of a collection point included in each extracted stroke sequence; And
Generating the stroke time sequence by estimating the time taken to draw the stroke through the number of collection points included in each of the stroke sequences
A stroke-based handwritten signature authentication method comprising a. delete According to claim 1,
Extracting features of each stroke image from the stroke image sequence may include:
Extracting features of each stroke image from the stroke image sequence through a single or multiple convolution layers
Characterized in that, stroke-based handwritten signature authentication method. delete Extracting a stroke image sequence and a stroke time sequence from collection points of a signature input from a user;
Extracting a feature of each stroke image from the stroke image sequence, and extracting a feature vector by extracting features of each stroke time from the stroke time sequence; And
Determining whether a signature is authentic using the extracted feature vector
Including,
Extracting the feature vector,
Extracting features of each stroke image from the stroke image sequence;
Extracting features of each stroke time from the stroke time sequence; And
Extracting a feature vector from the extracted feature of the stroke image and the feature of the stroke time
It includes,
Extracting the features of each stroke time from the stroke time sequence,
Searching for a time period that includes the stroke time and outputting a representative vector of the pre-trained time period.
Characterized in that, stroke-based handwritten signature authentication method. The method of claim 6,
Extracting the features of each stroke time from the stroke time sequence,
Further comprising the step of learning a representative vector for the stroke time interval,
Learning the representative vector for the stroke time interval,
Learning an auto-encoder to output the same stroke image when one stroke image is input from the training data;
Encoding a stroke image of the training data using a trained auto-encoder encoder network;
Dividing the range from 0 second to the maximum stroke time of the learning data into K time intervals of natural numbers that can be selected as optimal values in a conventional machine learning discrimination process;
Assigning each stroke of the learning data to a stroke time interval according to the stroke time; And
Generating a representative vector of the corresponding time period by obtaining an average of the encoded stroke images of the strokes included in each stroke time period
A stroke-based handwritten signature authentication method comprising a. According to claim 1,
The step of determining the authenticity of the signature using the feature vector,
It receives the output feature vector and compares it with the feature vectors of the user's true signature to determine whether the corresponding signature is a true or counterfeit signature.
Characterized in that, stroke-based handwritten signature authentication method. Receiving a user's signature consisting of a sequence of collection points received manually by a user terminal through a network;
Determining whether the signature is authentic; And
Transmitting the determined result to the user terminal through a network
Including,
The step of determining the authenticity of the signature,
Extracting a stroke image sequence and a stroke time sequence from collection points of the signature received from a user;
Extracting a feature of each stroke image from the stroke image sequence, and extracting a feature vector by extracting features of each stroke time from the stroke time sequence; And
Determining whether a signature is authentic using the extracted feature vector
Including,
Extracting the feature vector,
Extracting features of each stroke image from the stroke image sequence;
Extracting features of each stroke time from the stroke time sequence; And
Extracting a feature vector from the extracted feature of the stroke image and the feature of the stroke time
It includes,
Extracting a feature vector from the feature of the stroke image and the feature of the stroke time,
Extracting feature vectors from features of the stroke image and features of the stroke time through a single or multiple recurrent layers
Characterized in that, stroke-based handwritten signature authentication method. A signature pre-processing unit extracting a stroke image sequence and a stroke time sequence from collection points of a signature received from a user;
A feature extraction unit for extracting features of each stroke image from the stroke image sequence, and extracting a feature vector by extracting features of each stroke time from the stroke time sequence; And
Signature discrimination unit that determines whether a signature is authentic using the extracted feature vector
Including,
The feature extraction unit,
A convolution layer unit for extracting features of each stroke image from the stroke image sequence through a single or a plurality of convolution layers;
A stroke time interval embedding layer for extracting features of each stroke time from the stroke time sequence; And
A recurrent layer that extracts a feature vector from features of the stroke image and features of the stroke time extracted through a single or multiple recurrent layers
Including, stroke-based handwritten signature authentication system. The method of claim 10,
An input unit that receives a user's signature consisting of a sequence of collection points received by hand through a user terminal; And
An output unit that transmits the determined result to the user terminal
Further comprising, a stroke-based handwritten signature authentication system. The method of claim 10,
The signature pre-processing unit,
In the sequence of the collection points of the received signature, a stroke point is extracted by finding a collection point where the intensity of pressure applied to the screen is 0, and on-screen coordinates of the collection points included in each extracted stroke sequence Generating the stroke image sequence using information, and generating the stroke time sequence by estimating the time taken to draw the stroke through the number of collection points included in each of the stroke sequences
Characterized in that, stroke-based handwritten signature authentication system. delete The method of claim 10,
The stroke time section embedding layer unit,
Searching for a time period that includes the stroke time and outputting a representative vector of the pre-trained time period.
Characterized in that, stroke-based handwritten signature authentication system. The method of claim 10,
The signature determining unit,
It receives the output feature vector and compares it with the feature vectors of the user's true signature to determine whether the corresponding signature is a true or counterfeit signature.
Characterized in that, stroke-based handwritten signature authentication system.

Images