JP2019101927A - Learning system and image retrieval system - Google Patents

Abstract

To perform image retrieval at high speed and with high accuracy.SOLUTION: A learning system holds: image information of a pair of images; a flag indicating whether or not the pair of images are provided for a same target; and a first neural network outputting output vectors for which the image information is inputted and a value of each element is included in a predetermined range, inputs the image information of each of the pair of images into the first neural network and outputs a first output vector for each image, calculates an expected value of a distance between the first output vectors, and updates, with reference to the flag, a parameter of the first neural network in a manner such that the calculated expected value is small in a case where the pair of images are provided for the same target while the calculated expected value is large in a case where the pair of images are provided for different targets.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a learning system and an image search system.

  BACKGROUND ART In recent years, biometric authentication technology for authenticating an individual using biometric information such as fingerprints, faces, and veins has become widespread. In biometric authentication, personal information of an individual is registered in advance, and the individual is identified by collating the biological information input at the time of authentication with the registered biological information. In this biometric authentication, the ID of the user is input to identify registered biometric information, and then verification is performed. 1: 1 authentication and the verification with all registered biometric information without inputting the user ID is performed 1: N. There is an authentication. In 1: N authentication, since the user does not need to input the ID, the user does not have to store the ID or input the ID, and the convenience is high.

  On the other hand, in 1: N authentication, since collation is performed between biometric information to be authenticated and all registered biometric information, it takes longer to perform comparison than 1: 1 authentication. For example, in the case where one million registered biometric information items are simply compared with all the items, it takes one million times the verification time of 1: 1 authentication.

  As a technique for performing such large-scale 1: N authentication at high speed, there is JP-A-2013-206187 (Patent Document 1). In this publication, "the information retrieval apparatus 1 sets a feature amount vector of data to be searched for using the Hamming distance, a wildcard symbol whose Hamming distance to the binary symbol is 0, and the binary symbol. The information search device 1 converts the query data into binary data, and searches for a string whose Hamming distance to the binaryized query data is equal to or less than a predetermined value, thereby generating neighborhood data of the query data. Search for "(see summary).

JP, 2013-206187, A

  The technique described in Patent Document 1 applies binary vectorization by simple projection to 1: N authentication. According to the method, even if the authentication speed is improved, the accuracy of the authentication may be reduced. One aspect of the present invention aims to perform image search at high speed and with high accuracy, such as verification of biometric information in 1: N authentication.

  In order to solve the above-mentioned subject, one mode of the present invention adopts the following composition. A learning system for learning parameters of a neural network includes a processor and a memory, and the memory includes image information of an image pair, a flag indicating whether the image pair is the same target image, and image information , And holds network information indicating a first neural network that outputs an output vector in which the value of each element is included in a predetermined range, and the processor stores the respective images of the image pair in the first neural network. Image information is input, a first output vector is output for each image, an expected value of the distance between the first output vectors is calculated, and the image is an image of the same target as the image pair with reference to the flag. If the calculated expected value is small if the image pair is an image of a different target, the calculated expected value is large, To update the parameters of the serial first neural network learning system.

  According to one aspect of the present invention, image search can be performed at high speed and with high accuracy. Problems, configurations, and effects other than those described above are clarified by the following embodiments.

FIG. 1 is a block diagram illustrating an exemplary configuration of an authentication system according to a first embodiment. It is a flowchart which shows an example of the parameter learning process in Example 1, a biometric information registration process, and 1: N authentication process. 7 is a flowchart illustrating an example of a parameter learning process according to the first embodiment. FIG. 6 is an explanatory view showing an example of a data structure of learning data in the first embodiment. FIG. 6 is an explanatory view showing an example of a data structure of parameters in Example 1; 5 is a flowchart illustrating an example of the details of registration data generation processing according to the first embodiment; 7 is a flowchart showing an example of details of authentication data generation processing in the first embodiment. 7 is a flowchart showing an example of the details of the narrowing-down collation process in the first embodiment. FIG. 8 is an explanatory view showing an example of the data structure of registration data in the first embodiment. It is a block diagram which shows the hardware structural example of the learning machine in Example 1, an authentication client, and an authentication server. It is a flowchart which shows an example of the parameter learning process in Example 2, a biometric information registration process, and 1: N authentication process. FIG. 18 is a flowchart illustrating an example of parameter learning processing according to the third embodiment. FIG. 18 is a flowchart illustrating an example of the details of the narrow-down matching process according to the third embodiment.

  Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. It should be noted that the present embodiment is merely an example for realizing the present invention, and does not limit the technical scope of the present invention. The same reference numerals are given to the same configuration in each drawing.

  The authentication system will be described below. The authentication system learns the parameters of the neural network using the learning data. The authentication system registers binary vectors obtained by inputting biological information into a neural network. In addition, the authentication system performs high-speed 1: N authentication by narrowing down the match target by matching the binary vector obtained by inputting biological information to the neural network at the time of 1: N authentication and the registered binary vector. Do.

  The authentication system of this embodiment can realize high-speed and high-precision 1: N authentication by utilizing machine learning called Deep Learning for binary vectorization of biometrics. In the present embodiment, an example in which information of an image such as a fingerprint, an iris, a vein, or a face (for example, an image feature amount) is used as biological information will be described, but other biological information such as voice is used, for example. It may be done.

  FIG. 1 is a block diagram showing an exemplary configuration of an authentication system. The authentication system includes, for example, a learning device 1000, an authentication client 1100, and an authentication server 1200. For example, the learning device 1000 includes a parameter initializing unit 1010, a learning data acquiring unit 1020, a neural network calculating unit 1030, an inter-vector distance calculating unit 1040, a loss calculating unit 1050, a parameter updating unit 1060, a learning data storage unit 1090, and parameters. And a storage unit 1091.

  The parameter initialization unit 1010 initializes the parameters of the neural network. The learning data acquisition unit 1020 acquires learning data from the learning data storage unit 1090. A neural network calculation unit 1030 inputs biological information to a neural network to calculate a vector. An inter-vector distance calculation unit 1040 calculates the distance between two vectors.

  The loss calculating unit 1050 calculates the output of the loss function based on the distance between the vectors. The parameter updating unit 1060 updates the parameters of the neural network based on the output of the loss function calculated by the loss calculating unit 1050. The learning data storage unit 1090 stores learning data. The parameter storage unit 1091 stores the parameters learned by the parameter updating unit 1060.

  The authentication client 1100 includes, for example, a biometric information acquisition unit 1110, a neural network calculation unit 1120, a vector binarization unit 1130, a registration vector fusion unit 1140, a mask vector generation unit 1150, a template conversion unit 1160, a sample conversion unit 1161, template matching. And a communication unit 1180 and a parameter storage unit 1190.

  The biometric information acquisition unit 1110 acquires biometric information input from the user at the time of registration or authentication. The neural network calculation unit 1120 inputs the biological information acquired by the biological information acquisition unit 1110 to a neural network to calculate an output vector.

  The vector binarization unit 1130 binarizes the output vector calculated by the neural network calculation unit 1120 to generate a binary vector. The registration vector fusion unit 1140 fuses the plurality of binary vectors generated by the vector binarization unit 1130 to generate a single registration vector.

  The mask vector generation unit 1150 generates a mask vector for the binary vector generated by the vector binarization unit 1130. The template conversion unit 1160 converts the biometric information acquired by the biometric information acquisition unit 1110 into a registered template. The sample conversion unit 1161 converts the biological information acquired by the biological information acquisition unit 1110 into an authentication sample.

  The template collating unit 1170 collates the registered template generated by the template converting unit 1160 with the authentication sample generated by the sample converting unit 1161 to determine the person. The communication unit 1180 communicates with the authentication server 1200 via the network 1300. The parameter storage unit 1190 stores the same parameters as the learned parameters stored in the parameter storage unit 1091 of the learning device 1000.

  The authentication server 1200 includes, for example, an inter-vector distance calculation unit 1210, a communication unit 1220, and a registration template storage unit 1290. The inter-vector distance calculation unit 1210 calculates the Hamming distance between the binary vectors calculated by the neural network calculation unit 1120 and the vector binarization unit 1130. The communication unit 1220 communicates with the authentication client 1100 via the network 1300. The registration template storage unit 1290 stores the registration template generated by the template conversion unit 1160.

  In the example of FIG. 1, the learning device 1000 is not connected to the authentication client 1100 and the authentication server 1200, but may be connected to at least one of the authentication client 1100 and the authentication server 1200 via the network 1300. .

  FIG. 10 is a block diagram showing an example of the hardware configuration of the learning device 1000, the authentication client 1100, and the authentication server 1200. The learning device 1000, the authentication client 1100, and the authentication server 1200 are configured by, for example, a computer shown in FIG. The computer includes a central processing unit (CPU) 1001, a main storage device 1002, an auxiliary storage device 1003, an input device 1004, an output device 1005, and a communication device 1006.

  The CPU 1001 includes a processor and executes a program stored in the main storage device 1002. The main storage device (memory) 1002 includes a ROM, which is a non-volatile storage element, and a RAM, which is a volatile storage element. The ROM stores an immutable program (for example, BIOS). The RAM is a high-speed and volatile storage element such as a dynamic random access memory (DRAM), and temporarily stores a program executed by the CPU 1001 and data used when the program is executed.

  The auxiliary storage device 1003 is, for example, a large-capacity and non-volatile storage device such as a hard disk drive (HDD) or a solid state drive (SSD), and stores programs executed by the processor and data used when executing the programs. Do. That is, the program is read from the auxiliary storage device 1003, loaded into the main storage device 1002, and executed by the processor.

  The computer has an input interface and an output interface. The input interface is an interface to which an input device 1004 such as a keyboard, a mouse, a camera, and a scanner is connected and which receives an input from an operator. The output interface is an interface to which an output device 1005 such as a display device or a printer is connected, and which outputs the execution result of the program in a format that can be viewed by the operator.

  The communication device 1006 is a network interface device that controls communication with another device according to a predetermined protocol, and is used for communication by the communication unit. Also, the communication device 1006 includes, for example, a serial interface such as USB.

  A program executed by the CPU 1001 is provided to a computer via removable media (CD-ROM, flash memory, etc.) or a network, and stored in a non-volatile auxiliary storage device 1003 which is a non-transitory storage medium. For this reason, the computer should have an interface for reading data from removable media.

  The learning device 1000, the authentication client 1100, and the authentication server 1200 are computer systems configured respectively on physically one computer or on a plurality of logically or physically configured computers. It may operate on a separate thread on a computer, or may operate on a virtual computer built on multiple physical computer resources.

  The CPU 1001 of the learning device 1000 includes a parameter initializing unit 1010, a neural network calculating unit 1030, an inter-vector distance calculating unit 1040, a loss calculating unit 1050, and a parameter updating unit 1060. The CPU 1001 of the authentication client 1100 includes a neural network calculation unit 1120, a vector binarization unit 1130, a registration vector fusion unit 1140, a mask vector generation unit 1150, a template conversion unit 1160, a sample conversion unit 1161, a template comparison unit 1170, and a communication unit. Including 1180. The CPU 1001 of the authentication server 1200 includes an inter-vector distance calculation unit 1210 and a communication unit 1220.

  For example, the CPU 1001 of the learning device 1000 operates as the parameter initializing unit 1010 by operating according to the parameter initializing unit program loaded in the main storage device 1002 of the learning device 1000, and the main storage device 1002 of the learning device 1000. By operating according to the loaded neural network calculation program, it functions as a neural network calculation unit 1030. The same applies to the other units included in the CPU 1001 of the learning device 1000, the units included in the CPU 1001 of the authentication client 1100, and the units included in the CPU 1001 of the authentication server 1200.

  The auxiliary storage device 1003 of the learning device 1000 includes a learning data storage unit 1090 and a parameter storage unit 1091. The auxiliary storage device 1003 of the authentication client 1100 includes a parameter storage unit 1190. The auxiliary storage device 1003 of the authentication server 1200 includes a registration template storage unit 1290.

  Hereinafter, an example of the process sequence which an authentication system performs is demonstrated. FIG. 2 is a flowchart showing an example of parameter learning processing, biometric information registration processing, and 1: N authentication processing.

  First, the learning device 1000 performs parameter learning processing (S2010). In this process, the learning device 1000 learns the parameters of the neural network based on the learning data stored in advance in the learning data storage unit 1090, and stores the learned parameters in the parameter storage unit 1091. Details of parameter learning in step S2010 will be described later with reference to FIG.

  The parameters acquired in step S2010 are stored in the parameter storage unit 1091 and copied to the parameter storage unit 1190 of the authentication client 1100 after the end of the learning process. When the learning device 1000 and the authentication client 1100 are not connected via the network 1300, for example, the parameters are provided to the authentication client 1100 via removable media.

  Next, the authentication client 1100 and the authentication server 1200 perform biometric information registration processing. First, the biometric information acquisition unit 1110 of the authentication client 1100 acquires biometric information (for example, a fingerprint, a face, an iris, a vein, etc.) of the user from, for example, the input device 1004 or other devices via the network 1300 S2110).

  The authentication client 1100 generates registration data from the acquired biometric information (S2120), and transmits it to the authentication server 1200 (S2130). By the registration data generation process of step S2120, three of a registration vector, a mask vector, and a registration template are obtained. Details of the registration data generation process in step S2120 will be described later with reference to FIG.

  Next, the authentication server 1200 receives the registration data (S2210), and stores the received registration data in the registration template storage unit 1290 (S2220). The registration data includes registration templates, binary vectors, and mask vectors described later. By executing the above processing for each user, biometric information registration of each user is completed, and 1: N authentication becomes available.

  Next, the authentication client 1100 and the authentication server 1200 perform 1: N authentication processing. The biometric information acquisition unit 1110 of the authentication client 1100 acquires biometric information from the user (S2140). The authentication client 1100 generates authentication data from the obtained biometric information (S2150), and transmits it to the authentication server 1200 (S2160). Note that the binary vector and the authentication sample are obtained by the authentication data generation in step S2150. Authentication data includes binary vectors. The details of step S2150 will be described later with reference to FIG.

  The authentication server 1200 receives the authentication data (S2230), and performs narrowing-down collation using the binary vector included in the authentication data (S2240). The details of step S2240 will be described later with reference to FIG. In step S2240, candidate template IDs are obtained. The authentication server 1200 acquires a candidate template from the registration template storage unit 1290 based on the candidate template ID (S2250), and transmits the candidate template to the authentication client 1100 (S2260).

  If no candidate template is acquired in step S2250, authentication fails at this point, and the process from step S2260 is not performed. In this case, for example, the authentication server 1200 notifies the authentication client 1100 that the authentication has failed, and the authentication client 1100 outputs an authentication result indicating that the authentication has failed to, for example, the output device 1005 or the like.

  If one or more candidate templates are obtained in step S2250, the processes in step S2260 and subsequent steps are executed. At this time, since the number of candidate templates is smaller than the number of registered templates stored in the registered template storage unit 1290, the number of template matching can be reduced. As a result, in the authentication system of this embodiment, high-speed processing can be realized as compared with the case where the registration templates of all the cases and the authentication sample are simply compared.

  The authentication client 1100 receives the candidate template transmitted in step S2260 (S2170). The template collating unit 1170 collates the candidate template with the authentication sample generated in the authentication data generation process using, for example, an existing collation method (S2180).

  As a result of template matching in step S2180, it is determined whether or not there is a registered template having the same biological information as the authentication sample. If it is determined that there is a registered template having the same biometric information as the authentication sample, the template collating unit 1170 outputs an authentication result indicating that the authentication is successful, for example, to the output device 1005 or the like in step S2190.

  On the other hand, when it is determined that there is no registered template having the same biometric information as the authentication sample, in step S2190, an authentication result indicating that the authentication has failed is output to, for example, the output device 1005 or the like. By the above, 1: N authentication which introduced the narrowing down of the template by the binary vector is completed.

  FIG. 3 is a flowchart showing an example of the parameter learning process in step S2010. In this embodiment, parameter learning using a Siamese Network is performed. The Siamese Network includes two identical networks (both in the present embodiment as neural networks).

  The learning data in the Siamese network includes input information (for example, image information) input to each of the two networks, and a flag (identity indicating whether the two input information are in the same class or in different classes). Information) and. Each of the same networks is a multilayer (for example, three or more layers) network.

  The classes are the same only when the input information is the same biometric information of the same person. Specifically, for example, when the two pieces of input information are both image information of the vein of the forefinger of the right hand of the user A (the image itself may be different), the classes in these pieces of input information are equal. For example, the classes of the image information of the forefinger vein of the right hand of the user A and the image information of the veins of the forefinger of the right user B differ. Also, for example, the classes of the image information of the vein of the forefinger of the right hand of the user A and the image information of the veins of the forefinger of the left hand of the user A are different. That is, when the input information is image information, classes are equal only in image information obtained from the same object (subject).

  In the learning in Siamese Network, the parameters of the neural network are updated so that the distance between output vectors becomes smaller if the class of input information is the same, and the distance between output vectors becomes larger if the class of input information is different. Do.

  First, the parameter initialization unit 1010 of the learning device 1000 initializes the parameters stored in the parameter storage unit 1091 (S3010). The parameter initialization unit 1010 initializes parameters by, for example, substituting 0, substituting a random number according to a probability distribution such as a normal distribution, or the like. Note that, as a parameter initialization method, other methods generally performed in learning of a neural network may be used.

  The learning data acquisition unit 1020 acquires data to be used for learning from the learning data storage unit 1090 (S3020). Specifically, for example, the learning data acquisition unit 1020 selects M pieces of learning data (that is, data including two pairs of biological information and a flag). A method in which M pieces of learning data are selected at one time and parameter updating is performed is called Mini Batch. Note that, as a method of selecting learning data for updating parameters, another method generally performed in learning of a neural network may be used.

  The neural network calculation unit 1030 inputs the biological information in the learning data to the neural network to acquire an output vector output from the output layer (S3030). In the present embodiment, a convolutional neural network often used in the field of image recognition can be used.

  Also, a function that converts the value of each element of the output vector of the neural network into a value included in a predetermined range is used as a function of the final layer of the neural network. The predetermined range is, for example, a range in which two values that each element of the binary vector can take are the maximum value and the minimum value, respectively. In this embodiment, each element of the binary vector is 0 or 1, so the predetermined range is 0 or more and 1 or less.

  Therefore, in the present embodiment, the Sigmoid function that converts the value of each element to 0 or more and 1 or less is an example of the function. Hereinafter, it is assumed that the Sigmoid function is used. By using such a function as a function of the final layer, each element of the output vector has a value of 0 or more and 1 or less, and can be regarded as a probability value.

  The inter-vector distance calculation unit 1040 calculates the distance between the output vectors acquired in step S3030 for each pair of input information included in the learning data acquired in step S3020 (S3040). In addition to the vector-to-vector distance, the vector-to-vector distance represented by the following equation 1 in addition to the sum of squares of differences of elements of a vector (Euclidean distance) and the sum of absolute values of differences of elements of a vector (Manhattan distance) Functions can be applied.

In the inter-vector distance function of Equation 1, x and x ′ represent vectors of which the distances are to be calculated. Also, x _i is the ith element of the output vector x, x ′ _i is the _ith element of the output vector x ′, and n is the order of the output vector x and the output vector x ′. Since the final layer of the neural network is a Sigmoid function, the value of each element of the output vector is normalized to a value of 0 to 1. Therefore, in the binary vector obtained from the output vector (the details of the binary vector obtained from the output vector will be described later), each element of the output vector has a value of 1 corresponding to each element of the output vector. It can also be interpreted as a probability. Interpreted in this way, the distance obtained by the inter-vector distance function shown in Equation 1 represents the expected value of the Hamming distance between two binary vectors.

  Subsequently, the loss calculating unit 1050 calculates a loss based on the inter-vector distance obtained in step S3040 (S3050). The loss is a value indicating how ideal the inter-vector distance is. The closer to 0, the more the optimization progresses. Both Contrastive Loss shown in Equation 2 below and the loss function shown in Equation 3 below are both examples of equations used to calculate the loss.

  However, y in Equation 2 and Equation 3 is a flag indicating whether the class is the same or different for the pair of output vectors, 1 when the two output vectors are the same class, and 0 when the output vectors are different classes is there. Also, Margin is a predetermined value, for example, one.

  The loss function including Equation 2 and Equation 3 is a class in which biological information corresponding to two output vectors is in the same class, and a distance between the output vectors is small, and biological information corresponding to two output vectors is different. And take a small value when the distance between the output vectors is large. The loss function is a class in which biological information corresponding to two output vectors is in the same class and a distance between the output vectors is large, and biological information corresponding to the two output vectors is different from one another, and When the distance between the output vectors is small, it takes a large value.

  Subsequently, the parameter updating unit 1060 updates each parameter of the neural network so as to reduce the loss calculated in step S3050 (S3060). As a result, the distance between vectors of the same class decreases, and learning of parameters proceeds such that the distance between vectors of different classes increases. Also, learning of parameters proceeds such that the value of each element of the output vector approaches 0 or 1.

  Subsequently, the parameter updating unit 1060 determines whether learning has ended (S3070). Specifically, for example, when the first condition indicating that the number of times of execution of Mini Batch reaches a predetermined number is satisfied, for example, the parameter updating unit 1060 determines that learning is completed, and the first condition is satisfied. If is not satisfied, it is determined that learning has not ended. Also, for example, when the second condition indicating that the value of loss falls below the predetermined value is satisfied, the parameter updating unit 1060 determines that learning has ended, and the second condition is not satisfied. It is determined that learning has not ended. Also, for example, when at least one of the first condition and the second condition is satisfied, the parameter updating unit 1060 determines that the learning is ended, and when both are not satisfied, the learning is ended. You may determine that it is not.

  If the parameter updating unit 1060 determines that learning has not ended (S3070: No), the process returns to step S3020 and learning is performed again. When it is determined that the learning is completed (S3070: Yes), the parameter updating unit 1060 stores the obtained parameter in the parameter storage unit 1091 (S3080).

  In the present embodiment, parameter learning is performed by Siamse Network including two neural networks. However, parameter learning may be performed by a network including three or more neural networks (for example, Triplet Network). Thus, learning of parameters is completed.

  FIG. 4 is an explanatory view showing an example of the data structure of learning data stored in the learning data storage unit 1090. As shown in FIG. The learning data includes learning biometric information 410 and a class name 420. The learning biometric information 410 is biometric information of the same modality as the biometric information used for 1: N authentication collected for learning, and includes, for example, an image of a fingerprint, a face, an iris, a vein, or the like.

  The class name 420 is a name of a class of learning biometric information 410, and is, for example, a character string composed of alphanumeric characters. This class indicates the part of the person or body to which the biological information belongs. The biological information having the same class name is the same type of biological information acquired from the same living body. Biological information with different class names is biological information acquired from different living organisms.

  FIG. 5 is an explanatory diagram of an example of a data structure of parameters stored in the parameter storage unit 1091. As shown in FIG. The parameters shown in FIG. 5 are parameters required to uniquely determine the calculation process of the neural network. For example, if the neural network is a convolutional neural network, the parameters include each luminance value of the convolutional kernel, the value of the bias, the value of the weight of all the combined layers, and so on.

  If these parameters are determined, the value of the output vector can be uniquely calculated for the input biometric information. In the present embodiment, since the image information is used as biological information, the parameter includes a luminance value. For example, when voice information is used as biological information, for example, voice is used instead of the luminance value. The predetermined feature amount obtained from the signal to be shown is included in the parameter.

  FIG. 6 is a flowchart showing an example of details of registration data generation processing (S2120) by the authentication client 1100. After the end of step S2010 and before the start of step S2120, the parameters stored in the parameter storage unit 1091 are copied and stored in the parameter storage unit 1190.

  First, the neural network calculation unit 1120 inputs the biological information acquired in step S2110 to the neural network defined by the parameters stored in the parameter storage unit 1190, and generates an output vector (S6010).

  The vector binarization unit 1130 binarizes the output vector obtained in step S6010 to generate a binary vector in which all elements are 0 or 1 (S6020). In step S6020, for example, when the value range of each element of the output vector is 0 to 1, the vector binarization unit 1130 makes the element less than a predetermined threshold (for example, 0.5) 0, not less than the predetermined threshold Implement binarization processing by converting elements to 1. However, this binarization process is not limited to such a simple threshold process.

  For example, with respect to the value of each element of the output vector, the vector binarization unit 1130 outputs 0 if the value is included in a ratio (for example, lower 50%) equal to or less than a predetermined threshold of the value of the other element A general binarization scheme may be applied, such as conversion to 1 if included in a ratio (for example, upper 50%) equal to or higher than the predetermined threshold value. The other output vector may include an output vector obtained by past authentication processing by the authentication client 1100, or may include an output vector obtained by learning data stored in the learning data storage unit 1090. .

  The registration information may be generated from a plurality of biometric information of the same type of the same person. That is, step S2110 is executed a plurality of times, a plurality of biological information of the same type of the same person is generated, and a plurality of binary vectors are generated by executing steps S6010 and S6020 for each of the plurality of biological information. It is generated.

  A plurality of binary vectors may be generated from one piece of biometric information. Specifically, for example, when biological information is an image, a plurality of images are mechanically displaced, tilted, distorted, added with noise, or the like with respect to one image. May be generated.

  The registration vector fusion unit 1140 fuses a plurality of binary vectors to generate a registration vector (S6030). The registration vector fusion unit 1140, for example, calculates the median of each element of the plurality of binary vectors, and determines a binary vector having the median of each element as the value of the element as the registration vector. In addition, when the number of the plurality of binary vectors is an even number, for example, the value of one (for example, the larger one) of the two central values is set as the central value. In addition, statistics (for example, average or mode etc.) other than median may be used.

  Note that the registration vector fusion unit 1140 may determine the plurality of binary vectors as the registration vector as it is, or when only one binary vector is generated, the binary vector may be determined as the registration vector. Good.

  Subsequently, the mask vector generation unit 1150 generates a mask vector corresponding to the registration vector (S6040). When matching a binary vector, the accuracy of matching is improved by applying a mask vector. The biological information is not uniformly distributed on the feature space, and there is a specific bias for each individual. Therefore, individual differences occur in the errors included in each element of the binary vector.

  The mask vector generation unit 1150 analyzes, for example, a plurality of binary vectors acquired at the time of registration, and a plurality of binary vectors generated from biological information with arbitrary perturbations and noises added thereto. Calculate the ease of blurring (ie low reliability).

  The mask vector generation unit 1150 calculates the easiness of blurring, for example, by totaling how often the value of each element of the generated registered vector is inverted. Specifically, for each element of a plurality of binary vectors to be analyzed, for example, the mask vector generation unit 1150 is the difference between the number of binary vectors whose element is 0 and the number of binary vectors whose element is 1 Is set to a lower value, and as the difference is smaller, the easiness of the element is set to a higher value.

  Also, the mask vector generation unit 1150 may calculate the ease of blurring from an output vector obtained from one piece of biological information. Specifically, for example, when a binary vector is generated that is 0 when each element of the output vector is less than a predetermined threshold and is 1 when the output vector has a predetermined threshold or more, the mask vector generation unit 1150 As the value of the element is closer to the predetermined threshold, the easiness of blurring is set to a higher value, and as the value of the element is farther from the predetermined threshold, the easiness of blurring is set to a lower value.

  Since an element is more likely to include an error when the blurring easiness is a predetermined value or more, accuracy may be improved if data matching is performed using a vector excluding the element. It is the mask vector that realizes the exclusion of the element that is likely to include this error.

  Specifically, the mask vector generation unit 1150 has a value of an element that is likely to include an error (for example, the blur easiness is a predetermined value or more) 0, and an error is less likely to be included (for example, the blur easiness is less than a predetermined value) Generate a mask vector whose element value is 1 and register it with the binary vector.

  Subsequently, the template conversion unit 1160 performs template conversion on the biological information acquired in step S2110 to generate a registered template (S6050). The template conversion unit 1160 may generate, as a registered template, data obtained by converting biological information into feature points or a pattern image by performing feature extraction processing on the biological information.

  Also, the template conversion unit 1160 may apply template protection technology to the converted data to perform irreversible conversion, and generate original biometric information or data concealing its features as a registered template. Thus, the registration data generation process of step S2120 by the authentication client 1100 is completed.

  FIG. 7 is a flowchart showing an example of the details of the authentication data generation process of step S2150 by the authentication client 1100. First, the neural network calculation unit 1120 inputs the biological information acquired in step S2140 into the neural network defined by the parameters stored in the parameter storage unit 1190, and generates an output vector (S7010).

  The vector binarization unit 1130 binarizes the output vector obtained in step S7010 to generate a binary vector (S7020). The processes of step S7010 and step S7020 are the same as the processes of step S6010 and step S6020, respectively, and thus detailed description will be omitted.

  The sample conversion unit 1161 generates an authentication sample by subjecting the biological information acquired in step S2140 to sample conversion similar to the conversion in the registration template generation processing (S7030). The sample conversion unit 1161 may generate, as an authentication sample, data obtained by converting biological information into feature points or a pattern image by performing feature extraction processing from biological information as in the case of the registration template.

  Further, when the registration template is generated by applying the template protection technology, the sample conversion unit 1161 applies the template protection technology to the converted data to generate an authentication sample. Thus, the authentication data generation process (S2150) of step S2150 by the authentication client 1100 is completed.

  FIG. 8 is a flowchart showing an example of details of the narrowing-down verification process of step S2240 by the authentication server 1200. First, the inter-vector distance calculation unit 1210 substitutes 1 into a variable i (S8010).

  The inter-vector distance calculation unit 1210 refers to the registered vector that is a binary vector included in the i-th registration data stored in the registration template storage unit 1290, and calculates the distance to the binary vector generated in step S7020. (S8020). The inter-vector distance calculation unit 1210 calculates, for example, a Hamming distance as a distance between these vectors.

  Furthermore, when the mask vector is generated in step S6040, the inter-vector distance calculation unit 1210 can calculate the Hamming distance from only the element whose mask vector value is 1. Specifically, for example, the vector-to-vector distance calculating unit 1210 calculates a logical product of each of the binary vectors to be calculated for the distance calculation with the mask vector, calculates an exclusive OR between the logical products, and the obtained vector By counting the number of elements whose value is 1 among the elements of, it is possible to calculate the Hamming distance from which the element to be masked is excluded.

  Since the distance between binary vectors tends to be smaller as the number of elements to be masked (that is, the number of elements having a value of 1 in the mask vector) increases, the inter-vector distance calculation unit 1210 calculates the calculated distance, for example. It is desirable to calculate a normalized distance by dividing by the number of non-masked elements.

  Subsequently, the inter-vector distance calculation unit 1210 determines whether the distance calculated in step S8020 is smaller than a predetermined threshold T (S8030). If it is determined that the distance is smaller than the threshold T (S8030: Yes), the inter-vector distance calculation unit 1210 adds the ID of the ith registered template as a candidate template ID (S8040).

  The fact that the distance is smaller than the threshold value T indicates that the i-th registration data is for the person to be authenticated with a certain probability or more, so the registration template itself of the registration data is to be collated. When the inter-vector distance calculation unit 1210 determines that the distance is equal to or more than the threshold T (S8030: No), the inter-vector distance calculation unit 1210 transitions to step S8050.

  Subsequently, the inter-vector distance calculation unit 1210 increments i by 1 (S8050), and compares it with the total number N of registered data (S8060). If the vector-to-vector distance calculation unit 1210 determines that i after the increase is less than the total number N (S8060: No), it returns to step S8020, and determines that i is the total number N or more (S8060: Yes), Complete the narrowing verification process. Thus, the narrowing-down verification process of step S2240 is completed.

  FIG. 9 is an explanatory view showing an example of the data structure of registration data stored in the registration template storage unit 1290. As shown in FIG. The registration data includes, for example, a user ID column 910, a registration vector column 920, a mask vector column 930, and a registration template n 940. The user ID column 910 stores a user ID which is information uniquely identifying each user. The user ID is composed of, for example, alphanumeric characters. The user ID may be generated by the authentication client 1100 in the registration data generation process (S2120), or may be generated by the authentication server 1200 in the registration data storage process (S2220).

  The registration vector column 920 stores the registration vector generated in step S6030. The mask vector column 930 stores the mask vector generated in step S6040. The registration template field 940 stores the registration template generated in step S6050.

  As described above, the authentication system of this embodiment uses a plurality of identical neural networks so that the distance between output vectors obtained from the same type of biological information of the same person is short, and from different people or different biological information. The parameters of the neural network are learned so that the distance of the obtained output vector is long. In addition, the authentication system generates and registers a binary vector from biometric information to be registered by a neural network to which the parameter is applied.

  Further, the authentication system acquires biometric information to be authenticated, generates a binary vector from the biometric information by the neural network to which the parameter is applied, and compares the generated binary vector with the registered binary vector. Narrow down candidates for biometric information to be authenticated and biometric information on the same target. That is, the authentication system of the present embodiment can accelerate the authentication by narrowing down targets to which the registration template is directly compared. Furthermore, the authentication system of the present embodiment can suppress a decrease in authentication accuracy by learning using a plurality of identical neural networks.

  The biometric authentication system of the present embodiment is not limited to biometric information, and can be applied to an image search system that searches for a registered image identical to the input image. In this case, the authentication client 1100 functions as a search client, and the authentication server 1200 functions as a search server. That is, the image search system compares the binary vector generated from the input image input to the search client with the registered binary vector to narrow down the candidate (similar image) of the same registered image as the input image. be able to.

  In this embodiment, in 1: N authentication, the authentication server 1200 performs template matching processing instead of the authentication client 1100. Hereinafter, differences from the first embodiment will be described. Although illustration is omitted, the authentication server 1200 of this embodiment includes a template matching unit.

  FIG. 11 is a flowchart illustrating an example of the parameter learning process, the biometric information registration process, and the 1: N authentication process. The differences from FIG. 2 will be described. In step S2150, the authentication client 1100 includes the authentication sample in the authentication data in addition to the binary vector, and transmits the authentication data in step S2160.

  Following step S2250, the template matching unit of the authentication server 1200 performs matching processing between the candidate template and the authentication sample (S11260). Since the process of step S11260 is the same as the process of step S2180, the description will be omitted. Subsequently, the authentication server 1200 transmits the authentication result in the matching process to the authentication client 1100 (S11270). That is, in the present embodiment, the authentication server 1200 does not transmit the candidate template to the authentication client 1100.

  In the authentication system of this embodiment, since the authentication server 1200 completes the 1: N collation, the processing is simplified and the load on the authentication client 1100 can be reduced. However, in the authentication system of the first embodiment, since the authentication sample is not transmitted from the authentication client 1100 to the outside, when the template protection technology is applied, the authentication system of the first embodiment is the second embodiment. It may be more secure than an authentication system. Therefore, it is desirable that the authentication system of the first embodiment and the authentication system of the second embodiment be used properly according to the use case of the user.

  The authentication system of this embodiment generates a binary vector of a lower order than one of the binary vectors described in the first embodiment for one registration data. Further, the authentication system of the present embodiment generates a binary vector with a low degree of the order for biometric information to be authenticated in the narrowing-down collation process, performs collation with the registered low-order binary vector, and performs primary narrowing-down processing. Conduct.

  Furthermore, in the narrowing-down collating process, the authentication system is a binary vector having a high degree between the biometric information to be authenticated and the registered data narrowed down by the first-order narrowing-down process (the binary vector generated also in the first embodiment). We narrow down the comparison to obtain candidate templates. That is, the authentication system of the present embodiment implements faster primary 1: N authentication by executing primary narrowing using a binary vector with a lower degree and narrowing down the target of secondary narrowing in advance. Hereinafter, differences from the first embodiment will be described.

  FIG. 12 is a flowchart showing an example of the parameter learning process in step S2010. The differences from FIG. 3 will be described. If the parameter updating unit 1060 determines that learning has been completed (S3070: Yes), the neural network calculation unit 1120 determines whether the same neural network (hereinafter also referred to as a first neural network) included in the above-mentioned Siamese Network is selected. Reference is made to a new Siamese Network consisting of a network (hereinafter also referred to as a second neural network) to which the same output layer is added (S12000). That is, assuming that the first neural network consists of N layers, the second neural network consists of N + 1 layers, and the first layer of the second neural network,. The layers coincide with the Nth layer.

  Hereinafter, the learning device 1000 relates to the output layer of the second neural network in a state in which the parameter common to the first neural network in the second neural network is fixed to the value at the time of determining that learning is completed in step S3070. It learns parameters (that is, parameters not common to the first neural network).

  First, the parameter initialization unit 1010 of the learning device 1000 initializes parameters related to the output layer of the second neural network (S12010). The method of parameter initialization is the same as step S3010. The learning data acquisition unit 1020 acquires data to be used for learning from the learning data storage unit 1090 (S12020). The learning data acquisition method is the same as in step S3020.

  The neural network calculation unit 1030 inputs the biological information in the learning data to the second neural network to obtain an output vector output from the output layer (S12030). In addition, it is desirable that a function (for example, a Sigmoid function) that normalizes the value of each element of the output of the second neural network to a value of 0 or more and 1 or less is used as a function of the output layer of the second neural network.

  The inter-vector distance calculation unit 1040 calculates the distance between the output vectors acquired in step S12030 for each pair of input information included in the learning data acquired in step S12020 (S12040). The method of calculating the inter-vector distance is the same as in step S3040.

  Subsequently, the loss calculation unit 1050 calculates a loss based on the inter-vector distance obtained in step S12040 (S12050). The method of calculating the loss is the same as in step S3050. Subsequently, the parameter updating unit 1060 updates each parameter of the neural network so as to reduce the loss calculated in step S12050 (S12060).

  Subsequently, the parameter updating unit 1060 determines whether learning has ended (S3070). The method of learning end determination is the same as step S3070. If the parameter updating unit 1060 determines that learning has not ended (S12070: No), the process returns to step S12020 to execute learning again. When the parameter updating unit 1060 determines that the learning is completed (S12070: Yes), the process moves on to step S3080.

  Hereinafter, differences from the first embodiment will be described with reference to FIG. In step S6010, the neural network calculation unit 1120 inputs the biological information acquired in step S2110 to the first neural network to generate a first output vector, and inputs the biological information acquired in step S2110 into the second neural network. Generate a second output vector.

  In step S6020, the vector binarization unit 1130 performs a binarization process on the first output vector to generate a first binary vector, and performs a binarization process on the second output vector to perform a second process. Generate binary vector.

  In step S6030, the registration vector fusion unit 1140 fuses the plurality of first binary vectors to generate a first registration vector, and fuses the plurality of second binary vectors to generate a second registration vector. In step S6040, the mask vector generation unit 1150 generates a first mask vector corresponding to the first registration vector and a second mask vector corresponding to the second registration vector.

  Hereinafter, differences from the first embodiment will be described with reference to FIG. In step S7010, the neural network calculation unit 1120 inputs the biological information acquired in step S2110 to the first neural network to generate a first output vector, and inputs the biological information acquired in step S2140 into the second neural network. Generate a second output vector.

  In step S7020, the vector binarization unit 1130 performs a binarization process on the first output vector to generate a first binary vector, and performs a binarization process on the second output vector to generate a second binary vector. Generate binary vector.

  FIG. 13 is a flowchart showing an example of the details of the narrowing-down checking process of step S2240 by the authentication server 1200. The differences from FIG. 8 will be described. Subsequent to step S8010, the inter-vector distance calculation unit 1210 refers to the second registration vector included in the i-th registration data stored in the registration template storage unit 1290, and generates the second binary vector generated in step S7020. And the distance between them (S13010). The method of calculating the distance between the vectors is the same as in step S8020.

  Subsequently, the inter-vector distance calculation unit 1210 determines whether the distance calculated in step 13010 is smaller than a predetermined threshold T2 (S13020). If the inter-vector distance calculation unit 1210 determines that the distance is greater than or equal to the threshold value T2 (S13020: No), the inter-vector distance calculation unit 1210 transitions to step S8050.

  When it is determined that the distance is smaller than the threshold value T2 (S13020: Yes), the inter-vector distance calculation unit 1210 refers to the first registration vector included in the i-th registration data stored in the registration template storage unit 1290. The distance to the first binary vector generated at step S7020 is calculated (S13030).

  Subsequently, the inter-vector distance calculation unit 1210 determines whether the distance calculated in step 13030 is smaller than a predetermined threshold T1 (S13040). If the inter-vector distance calculation unit 1210 determines that the distance is greater than or equal to the threshold value T1 (S13040: No), the inter-vector distance calculation unit 1210 transitions to step S8050. When it is determined that the distance is smaller than the threshold T1 (S13040: Yes), the inter-vector distance calculation unit 1210 transitions to step S8040 and adds the ID of the i-th registered template as a candidate template ID.

  The present invention is not limited to the embodiments described above, but includes various modifications. For example, the embodiments described above are described in detail in order to explain the present invention in an easy-to-understand manner, and are not necessarily limited to those having all the configurations described. In addition, part of the configuration of one embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of one embodiment. In addition, with respect to a part of the configuration of each embodiment, it is possible to add, delete, and replace other configurations.

  Further, each of the configurations, functions, processing units, processing means, etc. described above may be realized by hardware, for example, by designing part or all of them with an integrated circuit. Further, each configuration, function, etc. described above may be realized by software by the processor interpreting and executing a program that realizes each function. Information such as a program, a table, and a file for realizing each function can be placed in a memory, a hard disk, a recording device such as an SSD (Solid State Drive), or a recording medium such as an IC card, an SD card, or a DVD.

  Further, control lines and information lines indicate what is considered to be necessary for the description, and not all control lines and information lines in the product are necessarily shown. In practice, almost all configurations may be considered to be mutually connected.

  1000 learning machines, 1001 CPUs, 1002 main storage devices, 1006 communication devices, 1030 neural network calculation units, 1040 inter-vector distance calculation units, 1050 loss calculation units, 1060 parameter update units, 1090 learning data storage units, 1110 biometric information acquisition units , 1120 neural network calculation unit, 1130 vector binarization unit, 1140 registration vector fusion unit, 1150 mask vector generation unit, 1190 parameter storage unit, 1100 authentication client, 1200 authentication server, 1290 registration template storage unit

Claims (14)

A learning system for learning parameters of a neural network,
Including processor and memory,
The memory is
Image information of image pair,
A flag indicating whether the image pair is an image of the same object;
Network information representing a first neural network which receives image information and outputs an output vector in which the value of each element is included in a predetermined range;
The processor is
Inputting image information of each image of the image pair into the first neural network, and outputting a first output vector for each image;
Calculating an expected value of the distance between the first output vectors;
With reference to the flag, the calculated expected value is small when the image pair is an image of the same target, and the calculated expected value is large when the image pair is an image of a different target. A learning system for updating parameters of the first neural network. The learning system according to claim 1, wherein
The predetermined range is 0 or more and 1 or less,
The processor calculates an expected value of the Hamming distance between the first output vectors using the following equation 1 as the expected value of the distance between the first output vectors:

In Equation 1, x is a first output vector corresponding to one image of the image pair, x ′ is a first output vector corresponding to the other image of the image pair, and n is a first output vector A learning system, wherein x _i is the i th element of x, and x ′ _i is the i th element of x ′. The learning system according to claim 2, wherein
The value of the flag is 1 when the image pair is an image of the same object,
When the image pair is an image of a different target, the value of the flag is 0,
The processor calculates the loss of the first neural network using Equation 2 below:

In the above equation 2, y is the flag, and Margin is a constant,
The learning system, wherein the processor updates the parameters of the first neural network so as to reduce the calculated loss. The learning system according to claim 2, wherein
The value of the flag is 1 when the image pair is an image of the same object,
When the image pair is an image of a different target, the value of the flag is 0,
The processor calculates the loss of the first neural network using Equation 3 below:

In the above equation 3, y is the flag, and Margin is a constant,
The learning system, wherein the processor updates the parameters of the first neural network so as to reduce the calculated loss. The learning system according to claim 1, wherein
A learning system, wherein a Sigmoid function is used in an output layer of the first neural network. The learning system according to claim 1, wherein
The network information indicates an output vector in which the value of each element is included in a predetermined range, and indicates a second neural network in which an output layer is added to the first neural network.
The order of a second output vector, which is an output vector of the second neural network, is lower than the order of the first output vector,
The processor is
In the second neural network, the parameter common to the first neural network is fixed to the updated parameter of the first neural network,
Inputting image information of each image of the image pair into the second neural network, and outputting a second output vector for each image;
Calculating an expected value of the distance between the second output vectors of each image of the image pair;
With reference to the flag, the calculated expected value is small when the image pair is an image of the same target, and the calculated expected value is large when the image pair is an image of a different target. A learning system, updating parameters related to an output layer of the second neural network. An image search system comprising a learning system, a search client and a search server,
The learning system is
Image information of image pair,
A flag indicating whether the image pair is an image of the same object;
Holding first network information indicating a first neural network which receives image information and outputs an output vector in which the value of each element is included in a predetermined range;
Inputting image information of each image of the image pair into the first neural network, and outputting a first output vector for each image;
Calculating an expected value of the distance between the first output vectors;
With reference to the flag, the calculated expected value is small when the image pair is an image of the same target, and the calculated expected value is large when the image pair is an image of a different target. Updating the parameters of the first neural network;
The search client is
Holding second network information indicating information of the first neural network having updated parameters;
Inputting image information of an input image into the first neural network indicated by the second network information, and outputting a first output vector of the input image;
Threshold processing to convert a first output vector of the input image into a first binary vector for transmission to the search server;
Each element of the first binary vector is a maximum value or a minimum value of the predetermined range,
The search server is
Hold a first binary vector corresponding to each of a plurality of registered images,
The first binary vector of the input image and the first binary vector of the plurality of registered images are compared to determine a first candidate of an image of the same target as the input image from the plurality of registered images , Image search system. The image search system according to claim 7, wherein
The predetermined range is 0 or more and 1 or less,
The learning system calculates an expected value of the Hamming distance between the first output vectors using the following Equation 4 as the expected value of the distance between the first output vectors:

In the above Equation 4, x is a first output vector corresponding to one image of the image pair, x ′ is a first output vector corresponding to the other image of the image pair, and n is a first output vector An image retrieval system, wherein x _i is the i th element of x, and x ′ _i is the i th element of x ′. The image search system according to claim 8, wherein
The value of the flag is 1 when the image pair is an image of the same object,
When the image pair is an image of a different target, the value of the flag is 0,
The learning system calculates the loss of the first neural network using Equation 5 below:

In the above equation 5, y is the flag, and Margin is a constant,
The image search system, wherein the learning system updates the parameters of the first neural network so as to reduce the calculated loss. The image search system according to claim 8, wherein
The value of the flag is 1 when the image pair is an image of the same object,
When the image pair is an image of a different target, the value of the flag is 0,
The learning system calculates the loss of the first neural network using Equation 6 below:

In the above equation 6, y is the flag, and Margin is a constant,
The image search system, wherein the learning system updates the parameters of the first neural network so as to reduce the calculated loss. The image search system according to claim 7, wherein
An image retrieval system, wherein a Sigmoid function is used in an output layer of the first neural network. The image search system according to claim 7, wherein
The first network information indicates a second neural network which outputs an output vector in which the value of each element is included in a predetermined range, and an output layer is added to the first neural network;
The order of a second output vector, which is an output vector of the second neural network, is lower than the order of the first output vector,
The learning system is
In the second neural network, the parameter common to the first neural network is fixed to the updated parameter of the first neural network,
Inputting image information of each image of the image pair into the second neural network, and outputting a second output vector for each image;
Calculating an expected value of the distance between the second output vectors of each image of the image pair;
With reference to the flag, the calculated expected value is small when the image pair is an image of the same target, and the calculated expected value is large when the image pair is an image of a different target. Updating the parameters related to the output layer of the second neural network;
The second network information indicates information of the second neural network having updated parameters,
The search client is
Inputting image information of the input image into the second neural network indicated by the second network information, and outputting a second output vector of the input image;
Threshold processing to convert a second output vector of the input image into a second binary vector for transmission to the search server;
Each element of the second binary vector is a maximum value or a minimum value of the predetermined range,
The search server is
Holding a second binary vector corresponding to each of the plurality of registered images;
The second binary vector of the input image and the second binary vector of the plurality of registered images are compared to determine a second candidate of an image of the same target as the input image from the plurality of registered images ,
The first candidate is determined from the registered image included in the second candidate by comparing the first binary vector of the input image and the first binary vector of the registered image included in the second candidate. Image search system. The image search system according to claim 7, wherein
The first binary vector corresponding to each of the plurality of registered images is
The threshold processing is performed on a first output vector obtained from the first neural network indicated by the second network information and a plurality of images of the same target as the registered image corresponding to the first binary vector. An image retrieval system, wherein the binary vector obtained is a binary vector fused by predetermined statistical processing. The image search system according to claim 7, wherein
The search server is
Holding a mask vector corresponding to each of the plurality of registered images;
Each element of the mask vector is
The value of the element of the first binary vector obtained from the first neural network indicated by the second network information and the plurality of images of the same target as the registered image corresponding to the mask vector has the same value Show the confidence to take
The search server is
For each of the plurality of registered images, an element with high reliability is identified as an element to be masked with reference to a mask vector,
An image that determines a registered image to be included in the first candidate by comparing the non-masked element of the first binary vector of the input image with the non-masked element of the first binary vector of the registered image Search system.

Images