KR102145132B1 - Surrogate Interview Prevention Method Using Deep Learning - Google Patents

Abstract

The present invention studies a deep learning learning model in a face recognition system so that real face recognition can be used to solve the problem of blind interview.
In the blind interview, applicants check the identity of each individual in the interview waiting room and enter the interview room with only the provisional number information.
At this time, the evaluator conducts an interview with all information that may impair fairness in evaluating the applicants in the state of being blinded in the application data.
Therefore, since the interview evaluation committee judges that the applicant is the same person with only the provisional number, various problems may arise. An object of the present invention is to solve these problems.

Description

Surrogate Interview Prevention Method Using Deep Learning}

The present invention relates to a proxy interview prevention using deep learning during a blind interview, and the embodiment relates to a blind interview evaluation management technology.

Face recognition technology is a technology that identifies whose face extracted from an image is, and is a biometric recognition technology used along with iris and fingerprint recognition. Referring to FIG. 1, a general face recognition system is largely composed of four steps as follows.
1. The step of detecting a face in a given image (Face Detection)
2. Aligning the detected face size and direction (Face Alignment)
3. Feature Extraction from the aligned face
4. Step of searching who is the extracted feature in the database (Feature Matching)
Face detection and feature extraction, which are the main steps of face recognition, have been studied in the existing image processing fields such as Histogram of Oriented Gradient (HOG), Local Binary Pattern (LBP), and Scale Invariant Feature Transform (SIFT). Features are extracted based on the pattern of image pixel values. In addition, recently, a landmark-based feature extraction technology is also being studied, and through this, research on facial expression recognition technology that classifies emotions in addition to face recognition is being conducted. However, in the case of feature extraction based on a pattern of pixel values, it is not consistent with the surrounding environment such as lighting, and since the feature extraction based on landmarks tends to be classified according to facial expressions, it is not suitable for face recognition. As related prior art, Patent No. 10-0983346 (name of invention: face recognition system and method using IR illumination), Registration Patent No. 10-2005150 (name of invention: face expression recognition system using machine learning) And method).

Recently, deep learning algorithms based on artificial intelligence technology are being applied to various industries, and in the field of face recognition technology, it has higher accuracy than existing face recognition technology. Therefore, using this technology, it is possible to solve the problems that appear as blind interviews gradually expand in college entrance exams.

When conducting a blind interview, applicants are individually identified in the interview waiting room and enter the interview room with a temporary number assigned.

The interview evaluation committee conducts an interview based on the screening data that blinded information that may compromise fairness in evaluating applicants.

Therefore, there is a problem that it is difficult for the interviewer to check whether the applicants are the same person as the screening data to be reviewed, and we want to solve these problems.

delete

Korean Patent Publication No. 10-2002-0032010 (2002.05.03.) Korean Patent Publication No. 10-0983346 (2010.09.14.)

The blind interview evaluation is conducted as a way to secure fairness by covering areas where prejudice against applicants may be intervened when conducting an interview evaluation for college admissions.

Since there are often school uniforms in the photos on the application form, there is a possibility that the school uniform can be used to infer the applicant's school, so the applicant's photo cannot be provided to the evaluation committee.

For this reason, the following problems may occur.

First, there is a possibility of confusion with other applicants because the interview room checks whether the applicant and the application data match with only the provisional number.

Second, if an applicant intentionally attempts a misconduct such as a proxy interview, the evaluation committee cannot immediately notice it.

Third, after the interview evaluation was completed, there are no evidence of applicants who took the actual interview.

The present invention seeks to solve the above problems.

The present invention studies a deep learning learning model in a face recognition system so that real face recognition can be used to solve the problem of blind interview.

The present invention proceeded with focus on face detection and feature extraction, which are key steps among the four steps of face recognition. It was confirmed that the parameters of the model can be lowered and the amount of computation can be reduced by applying the cascade model design and the inception model, and the objective function setting for applying this model in the image processing field was derived.

In the evaluation system according to the embodiment of the present invention, a QR code is given to an examination ticket of the examinee's terminal 100, the examinee on the day of the interview brings the examinee's terminal 100 and an identification card, and the supervisor in the waiting room from the examinee's terminal 100 The process of transmitting identification and photo-taking information to the management server 300 through the supervisor terminal 200 that scans the QR code and performs attendance processing, and the result of identification from the supervisor terminal 200 to the interviewer terminal 500 Including the process of transmitting.

The interviewer terminal 500 detects a face from an application reception photo in the management server 300 and performs image processing on an area under the face to blind the school uniform information.

The interviewer terminal 500 may calculate the similarity between the student's application reception photo and the photo transmitted from the supervisor terminal 200, and if the similarity is lower than the reference value, the similarity result is guided to the interviewer terminal 500. Can give.

This process will be a solution to the problems identified above.

Facial recognition system using deep learning learning model plays an important role in solving various problems of blind interview.

First, there was a possibility of confusion with other applicants because the interview room checks whether the applicant and the screening data match with only the provisional number. However, if the applicant's picture covered in school uniform is provided, the applicant and the screening data are matched based on the applicant's face. Whether or not you can intuitively notice.

Second, if the applicant intentionally attempts a fraudulent act such as a proxy interview, the suspicious behavior is determined based on the similarity between the photograph of the application in the management server 300 and the photograph of the applicant's face taken in the interview waiting room through the supervisor terminal 200. It can be notified in advance from the interviewer terminal 500, and the interviewer can immediately notice it.

Third, after the interview evaluation was completed, there was no data remaining to prove that the person who took the actual interview was the person who took the interview, but the photograph in which the uniform was blinded in the reception photo on the management server 300 and the supervisor terminal 200 in the examination room. ), the similarity analysis result of the applicant's photos taken with) is recorded in the management server 300, so that the photos and information of the person who took the interview can be viewed even after the screening is over.

Lastly, while the process for attendance verification and non-ID card holders (i.e., identification and attendance verification, photographing, archiving, and post ID verification work for non-ID card holders) is complicated, attendance and identification can be performed with a smartphone carried by almost all students. This has the effect of drastically reducing the work of the supervisor.

Figure 1: Structure diagram of a face recognition system
Figure 2: Structure diagram of a face detection system
Figure 3: Layer structure diagram by step
Figure 4: Inception model structure diagram
5: Structure diagram of feature extraction network
Figure 6: Triplet Loss conceptual diagram
Figure 7: Case of application of school uniform blinds
Figure 8: Case of measuring the Cosine Distance between facial feature vectors
Figure 9: System configuration diagram

Advantages and features of the present invention, and a method of achieving them will be sufficiently explained through examples to be described later in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below and may be implemented in various different forms. Unless otherwise defined, all terms used in the present specification may be used with meanings that can be commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in a commonly used dictionary are not interpreted ideally or excessively unless defined.

Hereinafter, the technical features of the present invention will be described in detail with reference to the accompanying drawings.

The following is a description of the face recognition system.

Referring to FIG. 1, a general face recognition system is largely composed of four steps as follows.

1. The step of detecting a face in a given image (Face Detection)

2. Aligning the detected face size and direction (Face Alignment)

3. Feature Extraction from the aligned face

4. Step of searching who is the extracted feature in the database (Feature Matching)

Face detection and feature extraction, which are the main stages of face recognition, are fields that have been studied a lot in the existing image processing fields such as Histogram of Oriented Gradient (HOG), Local Binary Pattern (LBP), and Scale Invariant Feature Transform (SIFT). In the present invention, the papers on Face Detection and Feature Extraction using deep learning, not the existing image processing methods, are analyzed, and face detection and feature extraction are performed using the deep learning learning model used in each paper. It was applied to feature extraction.

Referring to FIG. 2, the face detection system proposed in the paper connects three deep convolutional networks step by step, and converts the input image into various sizes and uses it as the first step input. In each step, the position of the face in the input image is guessed, and non-maximum suppression is used to integrate candidate regions that overlap a lot. The processed area is extracted and used as an input for the next step, and the area is predicted more precisely as the step progresses. The layer structure of each step is shown in FIG. 3.

Referring to FIG. 3, all three stages are composed of a Convolutional Layer, a Max-Pooling Layer, and a Fully Connected Layer, and the final output is divided into three types: face classification, area prediction, and landmark coordinate prediction. The later the step, the deeper the structure and the number of filters increases, enabling more precise and accurate prediction. In addition, by passing a layer of a simple structure at first and then passing a layer of a complex structure while narrowing the area, it is possible to reduce the amount of computation for the entire image rather than passing a complex layer from the beginning.

delete

The face detection system used in the present invention is called MTCNN (Multi-task Cascaded Convolutional Networks) because it learns the model connected step by step as described above using the following three objective functions (face classification, region prediction, and landmark coordinate prediction), Below is an explanation of the learning method.

. y는 정답 라벨에 해당되며, p는 네트워크를 통해 예측된 확률이다.Face classification can be viewed as a classification problem of two classes whether the corresponding area is a face or not. Therefore, cross-entropy was used as the objective function, and the formula is as follows.

. y corresponds to the correct answer label, and p is the predicted probability through the network.

Region prediction predicts a rectangular area with a face, and the rectangular area may be expressed by x, y coordinates, height and width of the left vertex. Therefore, in the case of region prediction, it can be seen as a regression problem that predicts four values, and for the objective function, a Euclidean distance is used.

Landmark coordinate prediction is a regression problem that predicts 10 values as a problem of predicting x and y of five feature points of a face (left eye, right eye, nose, left mouth vertex, right mouth vertex). Therefore, similar to the area prediction, the objective function used the Euclidean distance.

The following is an explanation of how to extract facial features.

In the case of a general convolutional neural network, the deeper the network, the better the performance. However, continuous convolution has the disadvantage of lowering the resolution of the image, and as the number of channels is multiplied, the parameters of the model are also greatly increased.

The DCNN (Deep Convolitional Neural Network) model structure used for facial feature extraction in the present invention is based on the inception model published by GoogLeNet, and the structure of the inception model is shown in FIG. 4. Referring to FIG. 4, the Inception model is a model proposed to solve this problem, and a method of increasing the width of the network is selected.

The overall network structure using the Inception model is shown in FIG. 5.

Referring to FIG. 5, in the overall network structure, inception layers are stacked several times, and a structure in which the depth and the width are increased together was selected. By selecting such a structure, features of various sizes can be effectively extracted, and the effect of reducing the amount of computation appears rather than simply deeply stacking the same number of convolution filters. At the end, embedding is performed by connecting Fully Connected, and the length of the embedding vector is matched through L2 normalization.

The following is a description of a model learning method that extracts facial features.

The final output through the network is a 128-dimensional feature vector, and Triplet Loss is introduced for face recognition. As shown in FIG. 6, the feature vectors of the same person should be learned similarly, and the feature vectors of others should be learned in a distance.

로

는 anchor의 특징 벡터,

는 anchor와 동일인의 특징 벡터,

는 타인의 특징 벡터이다.If this is expressed by an equation with reference to FIG. 6,

in

Is the anchor's feature vector,

Is the feature vector of the same person as the anchor,

Is another person's feature vector.

The present invention analyzes a deep learning learning model and applies these models to face recognition to solve problems that may occur in the current blind interview.

를

위치 픽셀의 빨강, 초록, 파랑 값,

를 검출된 얼굴 영역,

를 검출된 얼굴 영역의 픽셀 개수라 할 때, 검출된 얼굴 영역내 평균 픽셀값(

) 은 다음과 같이 계산된다.

얼굴 아래 영역에 대해 가우시안 블러를 적용하며 수식은 아래와 같다.

는 2차원 가우시안 함수로 수식은 아래와 같다.

이때,

는

에서의 표준 편차이며, 아래와 같다.

해당 픽셀 값과 평균 값의 차이가 클수록 표준 편차가 커지며 더 흐릿하게 블라인드 처리가 된다.

는 사용자 설정 값으로 전체적인 블라인드 정도를 조정한다.The following is an explanation of how to blind school uniform information by performing image processing.

To

The red, green, blue values of the location pixel,

The detected face area,

When is the number of pixels in the detected face region, the average pixel value in the detected face region (

) Is calculated as follows.

Gaussian blur is applied to the area under the face, and the formula is as follows.

Is a two-dimensional Gaussian function, and the formula is as follows.

At this time,

Is

Is the standard deviation from, and is as follows.

The larger the difference between the pixel value and the average value, the larger the standard deviation and the more blurry the blinding process is.

Is the user set value to adjust the overall blindness.

Referring to FIGS. 7 and 8, these models play an important role in solving the blind problem when the uniform removal of the applicant's photo and the similarity check of the same person as the applicant and the interviewee are applied to the blind interview in the blind interview system.

9 is a block diagram showing in detail an embodiment of a proxy interview prevention using deep learning of the present invention.

Referring to FIG. 9, the present invention includes a test taker terminal 100, a supervisor terminal 200, a management server 300, an electronic authentication server 400, and an interviewer terminal 500. Each terminal can use a number of terminals for each test room, and is a terminal capable of wired or wireless communication with each other and transmits and receives real-time information.

The examinee's terminal 100 receives an electronic test ticket through the management server 300. The examinee's terminal 100 requests the electronic authentication server 400 to process the user's electronic authentication using the electronic test ticket issued from the management server 300.

The electronic authentication server 400 informs the user of the electronic authentication processing result requested from the examinee's terminal 100.

The examinee's terminal 100 temporarily issues the examinee's identification code and the electronic authentication encryption token using the result of the user's electronic authentication confirmed from the electronic authentication server 400 and stores it in the management server 300, and then the examinee's terminal 100 If you take a picture of the test room QR code attached to the test room desk or the wall, the attendance status is made using the examinee identification code, electronic authentication encryption token, and test room code, and a QR code is generated in the examinee's terminal 100. This is the process of verifying attendance and processing of non-ID card holders by verifying their identity using a smart phone (i.e., identification and attendance verification, photographing, storing, and post identification verification work for non-ID cards), even if students do not bring their ID card. ) Is drastically reduced.

The supervisor's terminal 200 scans the QR code output on the examinee's terminal 100 with a camera recognition module when additional face-to-face attendance verification is required in addition to the attendance and identification of the examinee's terminal 100. Using an authentication encryption token, the examinee's identification code is one-way encrypted hash value, and the digital authentication encryption token in the management server 300 is used as an encryption key to verify the authenticity by whether the hash value of one-way encryption of the examinee's identification code is matched. Check.
At this time, the examinee identification code read by the supervisor terminal 200 from the QR code of the examinee terminal 100 is encrypted with one-way hash salt. The salt key value of the one-way hash salt is used as an electronic authentication encryption token issued when authentication is completed from the electronic authentication server 400. Therefore, even if the examinee or a third party captures or copies the screen as an image from the examinee's terminal 100 through carelessness or hacking, the salt key temporarily given by the management server 300 (that is, the electronic authentication encryption token) is different. The QR code image held by the third party is no longer valid and authentication fails. In addition, the electronic authentication server 400 used here uses a service of a trusted institution that legally guarantees the user's electronic authentication, thereby preventing an attempt to verify an identity using manipulated photo information or an identification card at the time of application. When the authenticity is confirmed in this way, the supervisor terminal takes a picture of the applicant's attendance and transmits it to the management server 300.

The management server 300 temporarily stores the electronic authentication encryption token generated from the examinee's terminal 100 and receives a request for confirmation using the examinee's identification code and the electronic authentication encryption token from the supervisor's terminal 200. After checking whether this is valid, the result is notified to the supervisor terminal 200. At this time, the management server 300 destroys the electronic authentication encryption token in a state that can no longer be used. In addition, the management server 300 updates to the attendance processing state when the encryption token transmitted from the supervisor terminal 200 is valid, and when not valid, guides the identification card to be checked offline and photographs of attendance can be taken.

The management server 300 calculates the cosine similarity between the attendance photo received from the supervisor terminal 200 and the feature vector extracted from the reception photo submitted by the applicant when the application is received when the applicant is updated to the attendance processing state, and the interviewer terminal 500 ).

At this time, the management server 300 transmits the photograph of the application submitted at the time of application as a photograph obtained by blinding the school uniform in advance.

100: student terminal
200: Supervisor terminal
300: management server
400: electronic authentication server
500: Interviewer terminal

Claims (3)

If the test room QR code attached to the test room desk or wall is photographed with the test examinee's terminal 100, an attendance state is established using the test examinee identification code, an electronic authentication encryption token, and the test room code, and generating a QR code on the test examinee's terminal 100;
In addition to the attendance and identification of the examinee terminal 100, the supervisor terminal 200 scans the QR code output on the examinee's terminal 100 with a camera recognition module when a face-to-face attendance confirmation is required, and the identification code and electronic authentication of the examinee in the QR code Identifying the encryption token and confirming the authenticity with the management server 300;
When the authenticity is confirmed, the supervisor terminal takes a picture of the applicant's attendance and transmits it to the management server 300;
The management server 300 includes a step of detecting a face from the photo submitted at the time of application by using Multi-task Cascaded Convolutional Networks (MTCNN);
The management server 300 calculates an average pixel value of the detected face area;
The management server 300 performs blind processing by applying a Gaussian blur according to the difference from the average pixel value in the area under the detected face.
The management server 300 includes the steps of detecting a face by using the MTCNN of the picture taken by the supervisor terminal 200;
A step of the management server 300 aligning the size and direction of the face detected in the application reception photo and the taken photo;
The management server 300 extracting features from the aligned faces using DCNN;
A proxy interview prevention method using deep learning, comprising the step of measuring, by the management server 300, a cosine similarity between an application acceptance picture and a feature vector extracted from the taken picture.
delete delete

Images