KR20220036250A - Method for weakly supervised learning of neural network using verification information of identification card and system for the method - Google Patents

Abstract

The present invention relates to a learning method of a neural network. The neural network learning method according to the present invention includes the steps of outputting a recognition result using a neural network that recognizes an object included in a user's identification card image; receiving a counselor's inspection result for at least part of the recognition result or inputting identity information collected about the user; And a step of training a neural network by using the inspection result or the identity information as the ground truth value of the corresponding recognition result. According to the present invention, the overall prediction performance of the neural network can be improved by performing additional learning with only the recognition results for some objects of the neural network for recognizing various objects included in the ID card.

Description

{Method for weakly supervised learning of neural network using verification information of identification card and system for the method}

The present invention relates to a learning method of a neural network, and to a simplified learning method of a neural network for ID recognition.

Thanks to the development of mobile communication terminals, especially smartphones, and mobile communication technology, functions that were previously performed offline can now be easily and conveniently used anytime, anywhere through mobile applications.

In particular, in recent years, in order to improve user convenience, technology has been developed and serviced to process financial transactions non-face-to-face through applications, and various methods have been devised to verify the user's identity in non-face-to-face situations. there is.

In addition, with the development of artificial intelligence technology, artificial intelligence technologies are being applied to various fields, and methods of detecting and tracking objects in images through neural network models are being developed instead of existing image processing methods. By applying this artificial intelligence technology to ID recognition, the user recognizes the information in the ID card image taken from the smartphone camera instead of directly entering the ID information, and through this, the methods of authenticating the user's identity are actually applied in financial services. there is.

However, since various objects to be recognized and various objects such as coatings and holograms are expressed in the ID card image to prevent contamination or counterfeiting of the ID card itself, it is necessary to learn the neural network by reflecting errors that may occur depending on the actual usage environment. .

The purpose of the present invention is to propose a neural network learning method for ID recognition.

In addition, the purpose of the present invention is to propose an efficient learning method that uses information generated when verifying recognized information of an ID card during actual use.

A neural network learning method according to the present invention for solving the above technical problem includes outputting a recognition result using a neural network that recognizes an object included in a user's ID card image; receiving a counselor's inspection result for at least part of the recognition result or inputting identity information collected about the user; And a step of training a neural network by using the inspection result or the identity information as the ground truth value of the corresponding recognition result.

The neural network preferably outputs a first recognition result that defines the area in which the object exists according to regression analysis and a second recognition result that defines the recognition value of the object according to classification.

In the input receiving step, it is preferable to receive the counselor's inspection result for the second recognition result.

It is preferable that the input receiving step receives the counselor's error judgment result for the first recognition result.

In the learning step, it is preferable to train the neural network by freezing the loss of the first recognition result to a constant value.

In the learning step, it is preferable to fix the loss for the first recognition result and train the neural network using the loss calculated by using the inspection result or identity information for the second recognition result as the correct value.

In the learning step, it is preferable to train the neural network based on a learning set consisting of pairs of the recognition result and the correct value of the ID card image collected according to a predetermined cycle.

The cycle is preferably determined by the batch size.

A neural network learning server according to the present invention for solving the above technical problem includes a recognition result output unit that outputs a recognition result using a neural network that recognizes an object included in a user's ID card image; an inspection result input unit that receives a counselor's inspection result for at least part of the recognition result or identity information collected about the user; and a learning unit that trains a neural network by using the inspection result or the identity information as a ground truth value of the corresponding recognition result.

The neural network preferably outputs a first recognition result that defines the area in which the object exists according to regression analysis and a second recognition result that defines the recognition value of the object according to classification.

Preferably, the inspection result input unit receives the counselor's inspection result for the second recognition result.

Preferably, the inspection result input unit receives the counselor's error judgment result for the first recognition result.

It is preferable that the learning unit trains the neural network by fixing the loss for the first recognition result to a constant value.

Preferably, the learning unit trains the neural network based on a learning set consisting of pairs of the recognition result and the correct answer value of the ID card image collected according to a predetermined cycle.

According to the present invention, a neural network capable of recognizing various information contained in an identification card can be efficiently trained.

According to the present invention, the overall prediction performance of the neural network can be improved by performing additional learning with only the recognition results of some objects of the neural network for ID recognition.

In addition, the present invention can streamline the learning process by using the inspection results obtained during the identification process as the correct answer for learning in areas where supervised learning is difficult when recognizing an ID card.

Figure 1 is an exemplary diagram showing a neural network learning system according to an embodiment of the present invention.
Figure 2 is a flowchart showing a neural network learning method according to an embodiment of the present invention.
3 and 4 are diagrams illustrating output data for learning of a neural network according to an embodiment of the present invention.
Figure 5 is a diagram illustrating the inspection results of the recognition results of the neural network according to an embodiment of the present invention.
Figure 6 is a diagram showing the configuration of a loss function for learning a neural network according to an embodiment of the present invention.
Figure 7 is a diagram illustrating a learning data set for learning a neural network according to an embodiment of the present invention.
Figure 8 is a block diagram showing the configuration of a learning server according to an embodiment of the present invention.

The following merely illustrates the principles of the invention. Therefore, a person skilled in the art can invent various devices that embody the principles of the invention and are included in the concept and scope of the invention, although not clearly described or shown herein. In addition, all conditional terms and embodiments listed in this specification are, in principle, clearly intended only for the purpose of ensuring that the inventive concept is understood, and should be understood as not limiting to the specifically listed embodiments and states. .

The above-mentioned purpose, features and advantages will become clearer through the following detailed description in relation to the attached drawings, and accordingly, those skilled in the art in the technical field to which the invention pertains will be able to easily implement the technical idea of the invention. .

Additionally, when describing the invention, if it is determined that a detailed description of the known technology related to the invention may unnecessarily obscure the gist of the invention, the detailed description will be omitted. Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the attached drawings.

1 is a diagram showing the configuration of a learning system in which a neural network learning method is performed according to an embodiment of the present invention.

Referring to FIG. 1, the learning system according to this embodiment includes a user terminal 1000 that the user 10 uses to use a non-face-to-face financial service, and the identity information of the user 10 input from the user terminal 1000. It may be configured as a learning server 100 that receives information and performs identity authentication through it.

The learning server 100 receives the ID card image 50 from the user 10 in order to perform non-face-to-face identity authentication, and directly uses the learned neural network to identify objects required for authentication included in the ID card image 50. It can be recognized and output as a result (70).

For example, the text included in the name, resident registration number, and address information of the user 10 included in the ID card image 50 can be recognized using a neural network.

Specifically, the learned neural network can output location information about the area where the object to be recognized exists as a recognition result.

As an example, information about the border of the ID card can be output to the entire area of the ID card image 50 as coordinates on the image. Alternatively, a value for an area where text such as the owner's name or resident registration number exists as an object in the ID card image 50 can be detected and output.

At this time, the area can be defined as a rectangular bounding box. Specifically, the bounding box can be defined as a 4-dimensional vector value of the x- and y-axis coordinates of the center point and the width w and height h of the border.

In addition, anti-counterfeiting devices such as coatings and holograms are applied to ID cards, and such devices can hinder recognition of objects by causing reflection or blocking when photographing ID cards in non-face-to-face situations. Therefore, it is also possible for the neural network to determine whether the captured ID image can be used by determining the presence or absence of these hindering factors.

Additionally, the ID image 50 may include a printed facial photo for identification of the same person, and feature information extracted from the facial photo may also be generated as output.

In other words, the ID card image 50 contains various information for identity authentication, and therefore, it can be used for authentication only when all objects included in the neural network are accurately recognized.

However, even in the case of a well-trained neural network, errors may occur in object recognition due to the influence of the environment in which the ID card image 50 is taken and devices to prevent forgery and tampering, so in a business environment that provides actual non-face-to-face services, Errors are being corrected through additional inspection procedures.

Therefore, in this embodiment, the learning server 100 seeks to perform neural network learning using the result value of the verification procedure for the recognition result. The learning server 100 receives the inspector's inspection results and performs learning using the inspection results as the ground truth of the learning data.

However, for the purpose of the original task of inspection, the inspector does not need to check for errors on all objects recognized by the neural network. Since the inspector checks for errors in objects that affect identity authentication, the learning method of the neural network accordingly suggests.

Hereinafter, the learning method according to this embodiment will be described in more detail with reference to FIG. 2.

First, the recognition result is output using a neural network that recognizes the object included in the ID card image 50 of the user 10 (S100).

The neural network may be pre-trained to output object recognition results for the ID card image 50.

Specifically, the neural network may output a first recognition result regarding the location information of objects included in the ID card image 50 and a second recognition result as a classification result of the objects, respectively.

Since the neural network can classify objects located within the area on the premise of accurate judgment of the area in which the objects exist in the image, the first recognition result and the second recognition result have a correlation with each other in the output process and are derived from a common neural network. can be printed.

For this purpose, the neural network performs supervised learning using a training data set in which the regions and classifications of images and objects included in the images are pre-labeled as training data.

However, in this embodiment, the purpose is to provide more robust performance in the real environment by additionally performing learning using information collected during the inspection process of the ID card image 50 in addition to pre-labeled data in learning the neural network.

Referring to FIG. 3, the neural network 30 provides a first recognition result (70-1) that defines the presence area of the object according to regression analysis and a second recognition result (70-1) that defines the recognition value of the object according to classification. Recognition results (70-2) can be output.

In this embodiment, the neural network 30 may be implemented as a CNN (Convolution Neural Network) model composed of a stacked structure of layers that perform convolution operations.

The convolution layers within the CNN model perform a convolution operation on the feature values in the input image data according to the set filters, and can generate a feature map according to the dimension determined as the output.

Additionally, the output feature map may be output as a first recognition result (70-1) as a result of regression analysis according to this embodiment and a second recognition result (70-2) as a classification result while passing through a fully connected layer.

At this time, in addition to the CNN model described above, various algorithms such as R-CNN (Region with CNN), RNN (Recurrent Neural Network), and LSTM (Long Short-Term Memory) can be applied to the neural network depending on the properties of the object to be recognized. The implementation method is not limited to this.

Hereinafter, the results output from the ID card image according to this embodiment will be described in more detail with reference to FIG. 4.

First, the first recognition result 70-1 is location information of objects included in the ID card image, and may be coordinate values defined as a bounding box.

Accordingly, information on the border area of the ID card within the ID image and area information on the text portion, such as name or resident registration number, can be output as the first recognition result (70-1).

On the other hand, the second recognition result (70-2) may be a classification result value of objects included in the ID card image.

For example, text classification results in which the last name in the name is recognized as 'Hong' can be directly output, and numeric values of the resident registration number can be output as classification results in the numeric domain.

In addition, it is possible to determine whether a shadow is included or whether a hologram is present as factors that affect identity authentication and output them as recognition results. In addition, facial information can be used to output judgment information about whether the user is wearing glasses and hair style in the photo included in the ID card as a classification result.

At this time, the first recognition result (70-1) is the area of the object, and the coordinate information of the bounding box is a value that does not affect identity authentication, so checking for errors in the coordinate value itself may not be necessary for the agent's original work purpose. there is.

Therefore, setting the correct value for the first recognition result (70-1), for example, correcting the coordinate value of the area to the correct position, may be an additional process, so the agent's minimum true/false judgment Weakly supervised learning can be performed using the results.

In other words, during the inspection process, instead of setting the correct answer to the location coordinate value in the ID card image where the objects exist, a dichotomous judgment can be requested as to whether the extracted area is correct or incorrect.

For example, instead of determining the error of each of the coordinates of the four corners of the ID border extracted from the ID image, the judgment of whether the border itself is correct or incorrect can be used for weak map learning.

In addition, if a part of the ID card image with a quality problem is output as a recognition result as an extractable area value, instead of the counselor directly correcting the coordinates of the part, a weak map is provided by judging the rightness or error of the judgment of the quality itself. It is also possible to use it for learning.

On the other hand, since the second recognition result (70-2) is a direct value for the user's name, resident registration number, etc., it can be a standard for judging a normal user depending on whether it is the same as the resident registration number assigned by the ID issuing agency. In other words, since identity verification is determined based on the recognition result, the agent inspects the second recognition result (70-2) and generates the inspection result.

In addition, the second recognition result (70-2) may include facial feature information in the ID card image or a classification result for the normal state based on the presence or absence of light occlusion, hologram, or light reflection of the ID card image, and the agent's inspection results for this may be included. You can use supervised learning to perform supervised learning.

The inspection result at this time is the correct value for the classification result of the neural network, so it can be used directly for supervised learning. In other words, the agent determines whether there is an error in the neural network's recognition result, thereby lowering the false rejection rate in which normal users are rejected for identity authentication. At the same time, in this embodiment, the neural network is trained using the generated inspection results.

Accordingly, the learning server receives the agent's inspection result for at least part of the recognition result or the identity information collected about the user 10 (S200).

The inspection result may be received from a counselor's inspection device (not shown) that inspects the recognition result of the ID card image 50 received from the learning server 100.

The counselor recognizes the objects printed on the ID card image 50 with the naked eye through an inspection device and compares them with the recognition results of the neural network to determine whether there is a misrecognition.

At this time, the counselor does not judge whether all recognition results are errors, but judges whether some of the recognition results are errors.

Specifically, referring to Figure 5, if the last name in the name is recognized as 'Hang' as a result of recognition of the ID card image, but the result visually confirmed by the counselor is 'Hong', correction of the error can be generated as an inspection result. .

Additionally, if the hair style shown in the user's photo as facial information is recognized as having no parting, but is determined to have parting as a result of the counselor's confirmation, it is also possible to generate this as an inspection result.

However, since the agent does not perform a separate error check on the area information about the border of the ID card, the check result may be generated in a different form from the output of the neural network.

In other words, for weakly supervised learning, the agent can generate an error judgment about the truth/false of the detected border area as a result of inspection, although it is not a specific correct answer regarding the area information about the border of the ID card, and it is fixed according to the judgment result. It is used for learning neural networks by defining the constant value as the loss.

In other words, the learning server 100 can use the agent's inspection result for the second recognition result (70-2) among the recognition results output from the neural network to calculate the loss as the correct answer and use it for learning, and the first recognition result (70-2) Although it is not the correct answer to 1), it can be used for learning as a fixed loss value determined through the agent's error judgment results.

In addition, the learning server may additionally directly receive information entered by the user 100 through the user terminal 1000 and determine whether there is an error. For example, a user can input personal information such as his or her name or social security number using the input interface of the user terminal. It is also possible to receive the input information and use it for learning.

Alternatively, it is also possible to receive and use information such as the ID card holder's name or issuance number as ID issuance information from an external server of an organization that manages ID cards.

Next, the learning server uses the inspection result or identity information as the ground truth value of the corresponding recognition result to learn the neural network (S300).

In other words, learning can be performed by defining the difference from the correct answer value as the agent's inspection result (80) for the second recognition result (70-2), which appears as a classification result among the recognition results, as the loss function of the neural network. there is.

Referring to FIG. 6, two recognition results can be output from the neural network. In this embodiment, for the second recognition result (70-2), the inspection result (80) is received from the counselor as the correct value during the inspection process. You can update the layers inside the neural network by calculating the difference between results and back-propagating the error.

However, in this embodiment, in order to perform weakly supervised learning in which the correct answer value is not set for some recognition results, among the recognition results 70, those that do not correspond to the inspection result or identity information (the correct answer is not defined) The neural network 30 is trained by freezing the loss for the (unknown) values.

Specifically, for the first recognition result (70-1) among the recognition results (70), instead of defining the correct answer value for the output value, a rough judgment of whether the entire output value is correct or incorrect is made, so this is defined as a loss and learned. It is available for use.

For example, as a result of the agent's inspection, if the location coordinate values of the recognized area are similar to the correct answer and are judged to be roughly correct, the loss for the first recognition result (70-1) is set to a constant value of 0 as if there is no error. Regardless of the output of the first recognition result 70-1, the parameters of the internal layers of the neural network 30 that affect the second recognition result 70-2 can be selected and updated.

Conversely, if the position coordinate values of the recognized object are judged to be incorrect, the internal layer sets the loss to a constant value of 1 to affect the error of the first recognition result (70-1) and the second recognition result (70-2). You can have them updated.

As above, the learning server 100 can efficiently learn the neural network 30 that can recognize various information contained in the ID card, and uses the inspection result 80 obtained during the inspection process of the ID card as the correct answer for learning. By doing so, the learning process can be streamlined.

In addition, since the neural network is trained to correct errors that occur during the actual identity authentication process, it can have more robust performance in the use environment.

Furthermore, in order to further increase the efficiency of learning, the learning server 100 may train a neural network based on a learning set consisting of pairs of recognition results and correct answer values of the ID card image 50 collected according to a predetermined cycle.

Specifically, the cycle can be determined by the batch size, and the batch size can be determined by the number of training data collected.

Referring to FIG. 7, N or more ID card image 50' and inspection result pairs determined by the minimum number of learning data can be accumulated and the neural network 30 can be trained in batch units.

For example, through a buffer that collects learning data, the agent can accumulate the results of labeling the correct answers as inspection results and perform the learning step when the accumulated learning data exceeds N. It is also possible to re-train the entire data by additionally loading learning data that was accumulated after or before accumulation.

However, depending on the determined threshold N, the accumulation cycle of learning data can be very fast, so it is possible to set N pieces of data as one set (basic unit for learning) and re-learn by mixing multiple sets with existing data.

Additionally, additional learning criteria can be set considering time. For example, by not accumulating training data for more than one day, meaningful features between training data sets can be learned.

Additionally, for more efficient learning, the feature values in the collected learning data set can be normalized and input into the neural network 30 by performing batch normalization on a determined batch basis. A neural network can evenly update the parameters of internal layers by back-propagating the error between the normalized input value and the correct answer value.

At this time, the batch size may be determined according to the type or number of the second recognition results 70-2, or may be determined according to a specific period or update cycle of the neural network 30, etc.

Hereinafter, the configuration of the learning server 100 that performs the neural network learning method according to an embodiment of the present invention will be described.

Referring to FIG. 8 , the learning server 100 may include a recognition result output unit 110, an inspection result input unit 120, and a learning unit 130.

First, the recognition result output unit 110 outputs a recognition result using the neural network 30 that performs object recognition on the ID card image 50.

The recognition result output unit 110 may output an analysis result of the presence area of the object to be recognized among the ID card image 50 and a classification result of the object.

Specifically, the neural network may output a first recognition result 70-1 that defines the area in which the object exists according to regression analysis and a second recognition result 70-2 that defines the recognition value of the object according to classification.

The inspection result input unit 120 receives the counselor's inspection result for at least part of the recognition result or the identity information collected about the user.

In this embodiment, the inspection result 80 may be generated from the agent's inspection device that inspects the recognition result of the ID card image 50.

However, at this time, the counselor may judge whether some of the recognition results are error-free rather than determining whether all recognition results are error-free.

Specifically, the first recognition result (70-1) based on regression analysis is the coordinate information of the bounding box in the area of the object, so it does not affect identity authentication, but the second recognition result (70-2) based on classification is the user's name, Since it is a direct value required for authentication of resident registration numbers, etc., it is determined whether there is an error.

Furthermore, the inspection result input unit 120 receives the personal information value directly entered through the user terminal in addition to the agent's inspection result, or receives information such as the name or issuance number of the ID card holder as issuance information from an external server and uses it as the correct value. possible.

Next, the learning unit 130 trains the neural network 30 by using the inspection result or the received identity information as the correct answer value of the corresponding recognition result.

The learning unit 130 performs learning by using the agent's inspection result (80) for the second recognition result (70-2), which appears as a classification result among the recognition results, as the correct value and defining the difference between the recognition results as the loss function of the neural network. can do.

In addition, in order to increase the efficiency of learning, the learning unit 130 trains the neural network 30 for each generation of inspection results, and instead performs learning consisting of pairs of recognition results of ID images collected cumulatively according to a predetermined cycle and the correct answer value. By configuring the set, you can efficiently train a neural network.

Specifically, the cycle may be determined by the minimum number of learning data determined by the batch size, or may be determined by the period or distribution cycle of the neural network 30.

Additionally, for more efficient learning, the feature values in the collected learning data set can be normalized and input into the neural network by normalizing the determined batch as a unit.

According to the present invention, the overall prediction performance of the neural network can be increased by performing additional learning with only the recognition results for some objects of the neural network for recognizing various objects included in the ID card.

In addition, the present invention can prevent the occurrence of additional procedures by obtaining learning data for learning during the inspection of ID cards.

Furthermore, various embodiments described herein may be implemented in a recording medium readable by a computer or similar device, for example, using software, hardware, or a combination thereof.

According to hardware implementation, the embodiments described herein include application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), and field programmable gate arrays (FPGAs). In some cases, it may be implemented using at least one of processors, controllers, micro-controllers, microprocessors, and other electrical units for performing functions. The described embodiments may be implemented as a control module itself.

According to software implementation, embodiments such as procedures and functions described in this specification may be implemented as separate software modules. Each of the software modules may perform one or more functions and operations described herein. Software code can be implemented as a software application written in an appropriate programming language. The software code may be stored in a memory module and executed by a control module.

The above description is merely an illustrative explanation of the technical idea of the present invention, and various modifications, changes, and substitutions can be made by those skilled in the art without departing from the essential characteristics of the present invention. will be.

Accordingly, the embodiments disclosed in the present invention and the accompanying drawings are not intended to limit the technical idea of the present invention, but are for illustrative purposes, and the scope of the technical idea of the present invention is not limited by these embodiments and the attached drawings. . The scope of protection of the present invention should be interpreted in accordance with the claims below, and all technical ideas within the equivalent scope should be construed as being included in the scope of rights of the present invention.

Claims (15)

In the neural network learning method,
Outputting a recognition result using a neural network that recognizes an object included in the user's ID card image;
receiving a counselor's inspection result for at least part of the recognition result or inputting identity information collected about the user; and
A neural network learning method comprising the step of training a neural network by using the inspection result or the identity information as a ground truth value of the corresponding recognition result. According to claim 1,
The neural network learning method is characterized in that the neural network outputs a first recognition result defining the area in which the object exists according to regression analysis and a second recognition result defining the recognition value of the object according to classification. According to claim 2,
The neural network learning method characterized in that the input receiving step involves receiving a counselor's inspection result for the second recognition result. According to claim 3,
The neural network learning method characterized in that the input receiving step involves receiving an error judgment result of the counselor regarding the first recognition result. According to claim 3,
The learning step is a neural network learning method characterized in that the neural network is trained by freezing the loss for the first recognition result to a constant value. According to claim 5,
The learning step is characterized by fixing the loss for the first recognition result and training the neural network using the loss calculated by using the inspection result or identity information for the second recognition result as the correct value. Neural network learning methods. According to claim 1,
The learning step is a neural network learning method, characterized in that the neural network is trained on a learning set consisting of pairs of the recognition result and the correct value of the ID card image collected according to a predetermined cycle. According to claim 7,
A neural network learning method, characterized in that the cycle is determined by the batch size. In a server that performs neural network learning,
a recognition result output unit that outputs recognition results using a neural network that recognizes objects included in the user's ID card image;
an inspection result input unit that receives a counselor's inspection result for at least part of the recognition result or identity information collected about the user; and
A server that performs neural network learning, comprising a learning unit that trains a neural network by using the inspection result or the identity information as the correct answer value of the corresponding recognition result. According to clause 9,
The neural network outputs a first recognition result defining a region of the object according to regression analysis and a second recognition result defining a recognition value of the object according to classification. According to claim 10,
A server performing neural network learning, wherein the inspection result input unit receives a counselor's inspection result for the second recognition result. According to claim 11,
A server performing neural network learning, wherein the inspection result input unit receives a counselor's error judgment result for the first recognition result. According to claim 12,
A server that performs neural network learning, wherein the learning unit trains the neural network by fixing the loss for the first recognition result to a constant value. According to clause 9,
A server that performs neural network learning, wherein the learning unit trains a neural network based on a learning set consisting of pairs of the recognition result and the correct answer value of the ID card image collected according to a predetermined cycle. A computer-readable recording medium storing a program for performing the neural network learning method according to any one of claims 1 to 8.

Images