KR20230017454A - Method, Device and Computer Program For Preventing Cheating In Non-face-to-face Evaluation - Google Patents

Abstract

Disclosed is a method for preventing cheating in a non-face-to-face evaluation. The method comprises: a first cheating detection step of checking the identity of an examinee based on a facial image extracted from the photographed image of the examinee; and a second cheating detection step of calculating the screen coordinates of the pupil gaze of the examinee from the photographed image of the examinee and determining correspondence to cheating based on the calculated screen coordinates. In the second cheating detection step, non-correspondence to cheating is determined if the calculated screen coordinates are placed in a region inside a screen, and correspondence to cheating is determined if the calculated screen coordinates are placed in a region outside the screen. Therefore, examinees can be prevented from cheating.

Description

Method, device and computer program for preventing cheating in non-face-to-face evaluation

The present invention relates to a method for preventing cheating in a non-face-to-face evaluation, and more particularly, detects cheating of an examinee using at least one of a facial image, a pupil image, joint data, and sound data extracted from a photographed image of an examinee. By doing so, it relates to a method, device, and program for preventing cheating in non-face-to-face evaluation.

As computer technology and Internet technology gradually develop, online education or online evaluation of various educational courses can be performed for the public.

Specifically, the online education or online evaluation may be performed by having the examinee connect to the online education website or online evaluation website through a computer platform or smart mobile application and use the online education function or online evaluation function provided by the administrator. there is.

On the other hand, the non-face-to-face evaluation method, in which the examinee conducts the online evaluation without the need to visit the designated place, has better accessibility to the evaluation and the convenience of the examinee, compared to the face-to-face evaluation method in which the examinee visits the designated place and conducts the evaluation. It has the advantage that it can be increased, and it has the effect that the risk of accidents, disease infection, etc. that can occur in face-to-face evaluation can be prevented.

However, compared to a face-to-face evaluation method in which personnel such as supervisors can directly observe test-takers at the evaluation site, the non-face-to-face evaluation method, in which only the examinee himself participates in the evaluation, is prone to cheating in various ways due to its unique high openness. However, there is a lack of measures to objectively prevent such cheating.

One of the various tasks of the present invention is to provide a method for preventing cheating in a non-face-to-face evaluation based on data extracted from a photographed image of an examinee.

One of the various tasks of the present invention is to provide a device for preventing cheating in a non-face-to-face evaluation based on data extracted from a photographed image of an examinee.

One of the various tasks of the present invention is to provide a program stored in a recording medium including a program code for executing a method of preventing cheating in a non-face-to-face evaluation based on data extracted from a photographed image of an examinee.

One of the various problems of the present invention is to provide a method for detecting cheating by analyzing a facial image of a test taker.

One of the various problems of the present invention is to provide a method for detecting cheating by analyzing a pupil image of an examinee.

One of the various problems of the present invention is to provide a method for detecting cheating by analyzing joint data of an examinee.

One of the various problems of the present invention is to provide a method for detecting cheating by analyzing test-taker's sound data.

A method for preventing cheating in a non-face-to-face evaluation according to an embodiment of the present invention for achieving the above object is a first cheating detection step of confirming an identity based on a facial image extracted from a photographed image of a test taker and the test taker A second cheating detection step of calculating screen coordinates at which the examinee's pupil gazes from the captured image and determining whether cheating is applicable based on the calculated screen coordinates, wherein the second cheating detecting step comprises: , If the calculated screen coordinates are located in the inner region of the screen, it is determined that they do not correspond to cheating, and if the calculated screen coordinates are located in the outer region of the screen, it is determined that they correspond to cheating.

In addition, the first cheating detection step may include a first facial region extracted from the photographed image of the examinee, a second facial region extracted from the ID of the examinee, and a third extracted from a recent photo submitted by the examinee. It may be performed by comparing "at least one of the facial regions".

In the first misbehavior detection step, the identity of the test taker may be confirmed by extracting vector values from the first to third facial regions, respectively, and comparing Euclidean distances of the extracted vector values. .

In the second cheating detection step, the screen coordinates may be calculated using an artificial neural network model learned to determine which region of the screen the pupil of the test taker who is looking at the screen is located.

In addition, the area gazed by the examinee's pupil is divided into a central area of the screen, an edge area of the screen, and an outer area outside the screen, and the central area of the screen and the edge area of the screen are safe areas that do not correspond to cheating. , The external area outside the screen may be set as a non-safe area corresponding to cheating.

In addition, the artificial neural network model is a softmax layer of a plurality of classes for whether the examinee's pupil gazes at a safe area located inside the screen or a non-safe area located outside the screen, and whether the examinee's eyes are closed. By setting to , it is possible to calculate the screen coordinates at which the pupils of the examinee gaze.

The method may further include a third misconduct detection step of detecting a third misconduct based on object recognition data recognized from the photographed image of the examinee, wherein the third misbehavior detection step includes detecting a person from the captured image of the examinee. It may be performed by extracting object recognition data under recognition, extracting joint data of the examinee from the object recognition data, and determining that the examinee's joint data corresponds to cheating when the joint data of the examinee is out of a preset safe action range.

And, further comprising a fourth cheating detection step of determining whether or not cheating is applicable based on the sound data extracted from the photographed image of the examinee, wherein the fourth cheating detection step comprises the sound data from the photographed image of the examinee. It can be performed by measuring , and determining that it corresponds to cheating when the decibel (dB) of the sound data is out of a preset safe decibel range.

In addition, the fourth misbehavior detection step may be performed by determining that the measured sound data corresponds to a misbehavior when it is detected that a voice is included in the measured sound data.

In addition, if at least one of the first misconduct detection step, the second misbehavior detection step, the third misbehavior detection step, and the fourth misbehavior detection step is not satisfied, the examinee may be determined to have cheated. there is.

Meanwhile, an apparatus for preventing cheating in a non-face-to-face evaluation according to an embodiment of the present invention for achieving the above object includes a face detection module for detecting a facial image of the examinee from a photographed image of the examinee, and a photographed image of the examinee. A pupil detection module for detecting the pupil image of the examinee, a joint detection module for detecting the joint data of the examinee from the photographed image of the examinee, a sound detection module for detecting sound data from the photographed image of the examinee, and the above and a control unit controlling the face detection module, the pupil detection module, the joint detection module, and the sound detection module to determine cheating of the examinee, wherein the control unit determines whether the pupil of the examinee is detected from the photographed image of the examinee. Calculate the coordinates of the staring screen, and if the calculated screen coordinates are located in the inner area of the screen, it is determined that they do not correspond to cheating, and if the calculated screen coordinates are located in the outer area of the screen, it is determined that they correspond to cheating. can judge

In addition, a program stored in a recording medium according to an embodiment of the present invention for achieving the above object may include a program code for executing the above-described cheating prevention method in non-face-to-face evaluation.

A method for preventing cheating in a non-face-to-face evaluation according to exemplary embodiments of the present invention detects cheating of a test taker through identity authentication confirmation, eye tracking, action recognition, and sound detection based on data extracted from a photographed image of the test taker. It can be prevented.

Accordingly, the occurrence of cheating in non-face-to-face evaluation can be prevented, evaluation reliability can be improved, and fairness of evaluation results can be established.

1 is a flowchart illustrating a method of performing non-face-to-face evaluation according to exemplary embodiments of the present invention.
2 is a flowchart for explaining the evaluation acceptance step in the non-face-to-face evaluation.
3 is a flowchart for explaining a preliminary inspection step in the non-face-to-face evaluation.
4 is a flowchart for explaining the evaluation preparation step in the non-face-to-face evaluation.
5 is a flowchart for explaining the test progress step in the non-face-to-face evaluation.
6 is a flowchart illustrating a fraud detection step using a user authentication process according to exemplary embodiments of the present invention.
7 is a diagram for explaining in more detail AI-based facial authentication according to exemplary embodiments of the present invention.
8 is a diagram for explaining a cheating detection step using gaze tracking according to exemplary embodiments of the present invention.
9 is an exemplary diagram for explaining a process of extracting a face region and both eye regions using landmarks according to an embodiment of the present invention.
10 is a diagram for explaining an arrangement structure of screen coordinates for classifying regions gazed by the examinee's pupils.
11 and 12 are diagrams for explaining a cheating detection step using action recognition according to exemplary embodiments of the present invention.
13 is a flowchart illustrating a fraud detection step using sound detection according to exemplary embodiments of the present invention.
14 is a block diagram illustrating a configuration for executing a method for preventing cheating in non-face-to-face evaluation according to exemplary embodiments of the present invention.

Hereinafter, specific embodiments of the present invention will be described. The detailed descriptions that follow are provided to provide a comprehensive understanding of the methods, devices and/or systems described herein. However, this is only an example and the present invention is not limited thereto.

In describing the embodiments of the present invention, if it is determined that the detailed description of the known technology related to the present invention may unnecessarily obscure the subject matter of the present invention, the detailed description will be omitted. In addition, terms to be described later are terms defined in consideration of functions in the present invention, which may vary according to the intention or custom of a user or operator. Therefore, the definition should be made based on the contents throughout this specification. Terminology used in the detailed description is only for describing the embodiments of the present invention and should in no way be limiting. Unless expressly used otherwise, singular forms of expression include plural forms. In this description, expressions such as "comprising" or "comprising" are intended to indicate any characteristic, number, step, operation, element, portion or combination thereof, one or more other than those described. It should not be construed to exclude the existence or possibility of any other feature, number, step, operation, element, part or combination thereof.

Also, terms such as first, second, A, B, (a), and (b) may be used to describe components of an embodiment of the present invention. These terms are only used to distinguish the component from other components, and the nature, order, or order of the corresponding component is not limited by the term.

1 is a flowchart illustrating a method of performing non-face-to-face evaluation according to exemplary embodiments of the present invention.

1, the non-face-to-face evaluation according to exemplary embodiments of the present invention may include an evaluation acceptance step (S1), a preliminary inspection step (S2), an evaluation preparation step (S3), and a test progress step (S4). The evaluation acceptance step (S1), pre-inspection step (S2), evaluation preparation step (S3), and test progress step (S4) may be sequentially performed.

Non-face-to-face evaluation according to exemplary embodiments of the present invention may be performed using a computer such as a desktop PC, notebook, netbook, etc., together with a portable electronic device such as a tablet PC, smart phone, etc., but the concept of the present invention is essential. Not limited to this. That is, the non-face-to-face evaluation according to exemplary embodiments of the present invention may be performed only using at least one of a computer and a portable electronic device, and in this case, the process related to the test taker's access to the App can be omitted. .

Hereinafter, each of the evaluation acceptance step (S1), preliminary inspection step (S2), evaluation preparation step (S3), and test progress step (S4) will be described in detail based on FIGS. 2 to 5.

Figure 2 is a flow chart for explaining the evaluation acceptance step in the non-face-to-face evaluation, Figure 3 is a flow chart for explaining the pre-inspection step in the non-face-to-face evaluation, Figure 4 is the evaluation preparation step in the non-face-to-face evaluation It is a flowchart for explaining, and FIG. 5 is a flowchart for explaining the test progress step in the non-face-to-face evaluation.

Referring to FIG. 2 , in the evaluation reception step (S1) according to exemplary embodiments of the present invention, personal information is registered (S11) from the examinee, and the examinee's consent to the use of personal information is collected (S12). It can be performed by receiving the ID photo of the examinee registered (S13) and confirming the authenticity of the ID (14).

Although not shown, the evaluation reception step (S1) further includes a process of providing guidance on the use of personal information while the test taker accesses the application page before the process of consenting to the use of personal information (S12) is performed. If the process of consenting to the use of personal information (S12) is not completed, it can be controlled to return to the process of providing information on the use of personal information. In addition, the evaluation acceptance step (S1) may further include a process of collecting consent to the security pledge from the examinee, and when the process of consent to use of personal information (S12) is completed, the process of collecting consent to the security pledge this can be done

In exemplary embodiments, the evaluation acceptance step (S1) may further include a process (S15) of registering a recent photo of the examinee and then matching it with the ID photo after the authenticity confirmation of the ID card (S14) is completed. .

The ID photo registration process (S13) is performed by receiving a file in which the ID photo is taken from the examinee, verifying the authenticity of the ID card, and extracting the facial area (or, in other words, the facial area) of the ID photo. It can be.

At this time, the submission of the ID photographing file may be performed by guiding the examinee's ID card to enter the guide line through the ID photographing guide and then guiding the student to photograph the ID card so that there is no reflected light.

When the submission of the ID photographing file is completed, a pre-processing process may be performed on the ID photographing file.

The pre-processing process may be performed by removing noise from the ID photographing file, performing dead zone detection and flattening, and performing ROI (Region Of Interest) division set for each type of ID card.

When the pre-processing of the ID photographing file is completed, a feature extraction process may be performed on the preprocessed ID photographing file.

The feature extraction process may be performed by extracting an image area and a text area of the ID photographing file, respectively.

In exemplary embodiments, the image region extraction may be performed by extracting the examinee's face region using an artificial neural network model, then extracting a facial landmark, and generating an embedding. In one embodiment, the image area extraction may be performed by further detecting whether the number of extracted face areas is one or two or more.

In exemplary embodiments, the character region extraction may be performed through OCR (Optical Character Recognition).

When the feature extraction process is completed, a process of confirming the authenticity of the ID card based on the ID information extracted through a specific institution may be performed (S14). The identity confirmation process (S14) may be performed by receiving an authenticity confirmation result from the specific institution after transmitting the extracted ID card information to the specific institution through the server, but the concept of the present invention is necessarily limited thereto. It doesn't work. That is, the identity confirmation process (S14) may be performed by comparing the extracted identity card information with previously stored information in a database (DB) of the server. The identification information may include a name, resident registration number, issuance date, driver's license number, passport number, alien registration number, and the like. The specific agency may include a Korean information management portal such as Government 24, a portal for the Korea Road Traffic Authority under the National Police Agency, and a foreigner information management portal such as Hi Korea.

If the identity verification process (S14) is not completed, it can be controlled to return to the identification card photo registration process (S13), and if the identity verification process (S14) is completed, the latest photo registration and identification photo A matching process (S15) may be performed.

Similar to the process of registering an ID photo (S13), the recent photo registration may be performed by receiving a recent photo from the test taker and then extracting a face area or facial area of the recent photo. In this case, extraction of the face region or facial region of the latest photo may be performed substantially the same as or similar to the extraction of the image region of the preprocessed ID file. That is, the extraction of the face region or facial region of the latest photo may be performed by extracting the examinee's face region through an artificial neural network model, extracting a facial landmark, and generating an embedding.

Thereafter, vector values are extracted from each of the facial area of the latest photo and the facial area of the ID photo, respectively, and the Euclidean distance of the vector values extracted from the facial area of the recent photo and the ID photo By comparing Euclidean distances of vector values extracted from the face region, the process of registering the latest photo and matching it with the ID photo (S15) can be completed. In contrast, in the process of registering the latest photo and matching it with the ID photo (S15), the Euclidean distance between the specific vector value extracted from the facial area of the latest photo and the specific vector value extracted from the facial area of the ID photo is calculated. It can also be done by comparing. In this case, the Euclidean distance (d) for completing matching between the latest photo and the ID photo may be less than 1, and preferably less than 0.4.

Face authentication based on the artificial neural network model will be described in more detail with reference to FIGS. 6 and 7 hereinafter.

Meanwhile, when the process of registering the latest photo and matching the ID photo (S15) is completed, a matching rate determination process (S16) of calculating a matching rate between the recent photo and the ID photo may be performed.

In exemplary embodiments, when the matching rate between the recent photo of the examinee and the ID photo of the examinee does not reach a specific matching rate (A), the test taker's acceptance confirmation (S17) is performed through an administrator. In this case, the evaluation acceptance step (S1) may be suspended in an incomplete state. In this case, the specific matching rate A may be greater than 60% and less than 80%, for example, 70%.

In one embodiment, if it is determined that the person in the ID photo and the recent photo is the same person through the evaluation acceptance confirmation process (S17) of the manager, the evaluation acceptance step (S1) may be completed by normally processing the DB flag. In contrast, if it is determined that the ID photo and the person in the recent photo are not the same person through the administrator's evaluation acceptance confirmation process (S17), the process of requesting an explanation from the test taker through text or phone after processing the DB Flag as abnormal is further carried out. can be performed

On the other hand, if the matching rate between the recent photo of the examinee and the ID photo of the examinee exceeds the specific matching rate (A), the evaluation acceptance step (S1) may be completed, and the preliminary check step (S2) Guidance for can be performed.

Referring to FIG. 3, in the pre-inspection step (S2) according to exemplary embodiments of the present invention, after the examinee accesses the Test Page (S21), installs an essential program (S22), and installs the essential program. It can be performed by confirming that the is completed (S23).

At this time, the process of confirming the installation completion of the essential program (S23) may be performed through device inspection.

More specifically, the installation completion confirmation process of the required program (S23) may be performed by checking whether the evaluation security program, the camera control program, and the microphone control program operate normally through the device inspection, and among the programs properly It can be done by guiding the installation of programs that do not work. At this time, if the evaluation security program, camera control program, and microphone control program do not function properly even though they are properly installed, the administrator may guide the examinee to solve the problem through Q/A or administrator chatting.

In addition, the process of confirming the installation completion of the essential program (S23) may be performed by further confirming whether an illegal program, a screen sharing program, a keyboard control program, and a monitor control program are detected through the device inspection, and the illegal program, screen sharing program, When a keyboard control program and a monitor control program are detected, they may be guided to terminate. In addition, in relation to monitor control, forced locking may be set to use only a single monitor, and when such forced locking does not function properly, a guide to stop using the dual monitor may be performed. In addition, when the camera does not function properly in relation to camera control, a camera inspection message may be displayed, and in case the microphone does not function properly in relation to microphone control, the microphone operation is turned on and the microphone volume is forced to a preset value. You can also perform a fixing process. At this time, if the setting values of the negative program, screen sharing program, keyboard control program, and monitor control program are different from the pre-stored non-face-to-face evaluation manual, the administrator will guide the examinee to solve the problem through Q/A or administrator chat. can

The type of camera used in the non-face-to-face evaluation is not particularly limited. In example embodiments, the camera may include a charge coupled device (CCD) image sensor, a complementary metal oxide semi-conductor (CMOS) image sensor, a charge priming device (CPD) image sensor, and a charge injection device (CID). ) may be implemented as any one of image sensors such as an image sensor and an infrared image sensor.

When the installation completion confirmation process (S23) of the required program is completed, a non-face-to-face evaluation access link is transmitted to the test taker's cell phone, and then the pre-check step (S2) may be completed when the test taker's access to the App is confirmed. At this time, if the examinee's access to the App is not confirmed, the administrator may guide the examinee to solve the problem through Q/A or administrator chatting.

Referring to FIG. 4, in the evaluation preparation step (S3) according to exemplary embodiments of the present invention, the examinee accesses the evaluation site N hours ago (S31), and checks whether the device inspection and setting are completed (S32). It can be performed by confirming the test taker's access to the App (S33), completing the test taker's identification (S34), and correcting the examinee's gaze from the captured image (S35).

The device inspection and setting confirmation process (S32) may be performed in substantially the same or similar manner as the pre-inspection step (S2) described with reference to FIG. The only difference can be whether or not.

The App access confirmation process (S33) transmits a link to the evaluation page through the mobile phone app of the examinee, checks whether the examinee has accessed the evaluation page, and provides the examinee with a non-face-to-face evaluation mode such as a do not disturb mode. After guiding to execute, it can be performed by checking the authentication code between the computer and the portable electronic device.

The process of verifying the examinee's identity (S34) outputs a webcam guide on the screen of the computer, guides the examinee so that his/her face is located within the webcam guide, and checks whether the examinee's face is located within the webcam guide, After photographing the examinee's face, it may be performed by comparing the photographed image of the examinee's face with the ID photo of the examinee and/or the recent photo of the examinee to determine whether the matching rate is greater than or equal to a specific matching rate (B). there is.

In exemplary embodiments, when the matching rate between the face-up image of the examinee and the ID photo of the examinee and/or the recent photo of the examinee does not reach a specific matching rate (B), the identity verification process of the examinee (S34) may be guided to be performed from the beginning. At this time, even though the identity verification process (S34) of the examinee was repeated three times, the matching rate between the photographic image of the examinee's face and the ID photo of the examinee and/or the recent photo of the examinee was at a specific matching rate (B). If it is not reached, the identity verification process (S34) of the examinee may be performed through an administrator. In this case, the specific matching rate B may be greater than 90% and less than 100%, for example, 95%. Here, the matching process may be performed using an artificial neural network model, and will be described in more detail with reference to FIGS. 6 and 7 hereinafter.

If the matching rate between the face-up image of the examinee and the ID photo of the examinee and/or the recent photo of the examinee exceeds a specific matching rate (B), the identity verification process of the examinee (S34) may be completed. , and then a process of correcting the examinee's gaze from the photographed image (S35) may be performed.

The process of correcting the examinee's gaze (S35) outputs the webcam guide on the screen of the computer, guides the examinee to place his or her face in the webcam guide, exposes the corrected screen, and then displays the examinee's face. After confirming whether it is located in the webcam guide, extracting the pupil coordinates of the examinee, sequentially exposing the screen before and after correction, perspective transformation between the pupil coordinates before and after correction (Perspective Projection ) can be achieved by performing

Through the gaze correction process (S35) as described above, the positions of the coordinates of the examinee's pupils can be more clearly extracted, and accordingly, the examinee's eye gaze tracking described later can be performed more effectively.

Referring to FIG. 5 , the test progress step (S4) according to exemplary embodiments of the present invention includes a cheating detection start step (S40), a user authentication step (S41), an eye tracking step (S43), and an action recognition step ( S45), and a sound detection process (S47), including a cheating detection process (S40), an identity authentication process (S41), an eye tracking process (S43), an action recognition process (S45), and a sound detection process. (S47) constitutes one cycle and can be controlled to be performed at each specific cycle while the non-face-to-face evaluation according to the present invention is in progress. At this time, the specific period may consist of a period in seconds, for example, various periods from several seconds to several tens of seconds, and, if necessary, a period in minutes, for example, various periods from several minutes to several tens of minutes. may be

If the identity authentication process (S41), eye tracking process (S43), action recognition process (S45), and sound detection process (S47) are all normally performed until the non-face-to-face evaluation is completed in the test progress step (S4), A test completion step (S49) of normally ending the non-face-to-face evaluation may be performed.

However, at least one of the identity authentication process (S41), eye tracking process (S43), action recognition process (S45), and sound detection process (S47) until the non-face-to-face evaluation is completed in the test progress step (S4). If not satisfied, it may be judged that the examinee is in a cheating situation. And, until the non-face-to-face evaluation is completed in the test progress step (S4), at least one of the identity authentication process (S41), the eye tracking process (S43), the action recognition process (S45), and the sound detection process (S47) If the process is not normally performed, after notifying the test taker of the cheating detection fact, the dashboard is labeled (S42), and whether the test taker's behavior corresponds to cheating may be confirmed (S44) through the manager. At this time, if the cheating of the test taker is confirmed, a process of confiscating the evaluation (S46) may be performed, and if the cheating of the test taker is not confirmed, it may be controlled to return to the cheating detection start process (S40). .

Hereinafter, the identity authentication process (S41), eye tracking process (S43), action recognition process (S45), and sound detection process (S47) will be described in detail based on FIGS. 6 to 13 .

6 is a flowchart illustrating a fraud detection step using a user authentication process according to exemplary embodiments of the present invention.

Referring to FIG. 6, the identity authentication process (S41) according to exemplary embodiments of the present invention outputs a webcam guide to the screen (S410), guides the examinee's face to be located in the guide (S411), and It is confirmed whether the examinee's face is located in the guide (S412), and after guiding the stop for a specific time, the examinee's face is photographed (S413), and the photographed examinee's face image is matched with the previously submitted photograph of the examinee. (S414), and comparing the matching rate of the photographed examinee's face image and the pre-submitted photograph of the examinee with a preset specific matching rate (B) (S415). At this time, the specific matching rate (B) may be a value greater than 90% and less than 100%, for example, 95%, and the pre-submitted photo of the examinee is the ID image of the examinee and / or the test taker's recent It may be a photographic image, and the specific time may be within 3 seconds.

Specifically, the process of matching the photographed examinee's facial image with the previously submitted photograph of the examinee (S414) includes a first facial region extracted from the examinee's photographed image, a second facial region extracted from the examinee's identification card, and/or Alternatively, third facial regions extracted from the recent photo submitted by the examinee are matched, vector values are extracted from the first facial region to the third facial region, respectively, and Euclidean distances of the extracted vector values are compared. It can be done by doing At this time, the Euclidean distance (d) for completing the process of matching the photographed examinee's face image with the previously submitted photograph of the examinee (S414) may be less than 1, and preferably less than 0.4.

The identity authentication process (S41) according to the present invention may be performed using an artificial neural network model. Specifically, according to the present invention, an artificial neural network model capable of recognizing a face in real time and estimating the position of a pupil in the recognized face may be used. Here, the artificial neural network model may be, for example, a residual network (ResNet) based on a VGG neural network, but is not limited thereto.

Facial authentication using the artificial neural network model according to the present invention may include steps of face detection, landmark detection, and facial feature point extraction.

Specifically, the artificial neural network model may perform face detection using a learning model capable of real-time face detection for real-time face detection in an image input from a camera. Here, the learning model may be, for example, Dlib-model, but is not limited thereto.

In addition, the artificial neural network model can detect landmarks representing the eyebrows, eyes, nose, mouth, and facial outline through shape prediction. For example, the detected landmarks may include 68 landmarks as shown in FIG. 7 .

In addition, the artificial neural network model may extract face recognition vector information by receiving the detected landmark parameter and converting it into a vector form through an encoded vector. Here, the extracted facial recognition vector information may be used for authentication of the test taker.

On the other hand, when it is determined that glasses are worn in the face region detected using the artificial neural network model, the glasses are removed through a CycleGAN-based deblurring module, and landmarks can be extracted from the removed image. .

On the other hand, the above-mentioned identity verification process (S14), recent photo registration and ID photo matching (S15-16), identity verification (S34), and identity authentication process (S41) are real-time Authentication success may be determined by comparing the matching rate of the examinee's face photographed with . Accordingly, the artificial neural network model according to the present invention can perform the above-described various examinee authentication processes through comparison of facial recognition vector information.

For example, in the identity authentication process ( S41 ), a video image of an ID card is captured, a face region is extracted, and face recognition vector information may be extracted through an artificial neural network model. In addition, in the test taker authentication process (S41), facial recognition vector information may be extracted from the examinee's face according to the photographing from the camera through an artificial neural network model. In addition, it is possible to perform authentication of the examinee himself by calculating the Euclidean distance based on the vector information generated from the video image of the ID card and the vector information generated from the face of the examinee taken by the camera. In one embodiment, when the Euclidean distance of the generated vector is 0.4 or more, it may be determined that the face image captured from the video and the ID capture image are not identical, and authentication failure may be determined. On the other hand, if the Euclidean distance of the generated vector is less than 0.4, it may be determined that the face image captured from the video and the ID card capture image are identical to each other, and authentication may be determined as success.

On the other hand, according to one example of the present invention, in the process of verifying the identity of the examinee, since the learning image used to learn the same person is only one ID card image, in order to confirm whether the facial image of the ID card and the facial image extracted from the camera are identical. One-shot learning is available. Here, one-shot learning may be a method of learning with one data, and after learning a similarity function (d(img1,img2) = return the difference between two images) using an artificial neural network, the value of the similarity function is greater than a preset reference value. It may be set to recognize that the two images are different if they are large, and to recognize them as similar if the value of the similarity function is smaller than a preset reference value.

As such, according to the present invention, by performing face recognition using an artificial neural network model, image analysis is possible through a general camera without separate equipment or sensors, so it is possible to have a cost reduction effect due to equipment, and not a single image. Real-time analysis of large-capacity image data is possible, so processing efficiency and accuracy can be improved compared to the input manpower.

In one embodiment, the identity authentication process (S41) may be performed by further detecting whether the number of extracted face regions is one or two or more, and if the number of extracted face regions is one, negative If it is determined that it does not correspond to an act, and the number of the extracted face regions is two or more, it can be controlled to determine that it corresponds to a cheating act.

Then, in the matching rate comparison process (S415), if the matching rate of the photographed examinee's face image and the pre-submitted photograph of the examinee does not reach the specific matching rate (B), the test taker notification and dashboard through the administrator A labeling process (S42) and a cheating check process (S44) may be performed, and depending on the result of the cheating check process (S44), an evaluation confiscation process (S46) is performed or a return to the cheating detection start process (S40) is performed. It can be.

Unlike this, in the matching rate comparison process (S415), when the matching rate between the photographed examinee's facial image and the pre-submitted photograph of the examinee exceeds the specific matching rate (B), the identity verification authentication of the examinee is completed. (S416), and then eye tracking process (S43), action recognition process (S45), and sound detection process (S47) may be performed.

8 is a flowchart illustrating a cheating detection step using gaze tracking according to exemplary embodiments of the present invention. Referring to FIG. 8 , in the gaze tracking process (S43) according to exemplary embodiments of the present invention, the examinee's face and pupil are extracted (S430), and it is determined whether the number of extracted faces is one (S431), It can be performed by detecting the pupil coordinates of the examinee (S432), calculating screen coordinates at which the pupil gazes (S433), and determining whether the calculated screen coordinates are located in the safe area (S434).

Here, the process of extracting the examinee's face and pupils (S430) will be described in more detail with reference to FIG.

9 is an exemplary diagram for explaining a process of extracting a face region and both eye regions using landmarks according to an embodiment of the present invention. Referring to FIG. 9 , in the pupil extraction process, a face area and both eye areas may be extracted using landmarks extracted using the ResNet of FIGS. 6 to 7 .

Meanwhile, since the determination of the number of extracted faces may be performed through substantially the same method as the above-described identity authentication process (S41), a detailed description thereof will be omitted.

After the process of extracting the pupil of the examinee (S430) is completed, the process of detecting the coordinates of the examinee's pupil (S432) may be performed. In this case, the pupil coordinate detection process ( S432 ) may be performed through a model-based technique of estimating a pupil position according to a modeling method after performing preset 2D or 3D eye modeling. Alternatively, the pupil coordinate detection process ( S432 ) may be performed through an appearance-based technique for estimating the center of the pupil by extracting a predetermined feature from the pupil or iris of the examinee through the Myeongdong method or the Ampupil method. In one embodiment, the pupil coordinate detection process ( S432 ) may be extracted through a pupil detection algorithm such as circle fitting and ellipse fitting.

Here, the pupil coordinates calculated through the pupil coordinate detection process (S432) may be used to correct the examinee's gaze. That is, according to an embodiment of the present invention, gaze correction may be performed by performing reference angle error correction between the central axis of the camera and the central axis of the pupil using a transformation matrix.

Meanwhile, in the screen coordinate calculation process (S433), screen coordinates at which the pupil of the examinee gazes may be calculated. Here, the area where the examinee's pupil gazes may have the same meaning as the area toward which the examinee's gaze is directed.

The area that the examinee's pupil gazes may be divided into a central area of the screen, an edge area of the screen, and an area outside the screen, and screen coordinates for the examinee's gaze may be calculated by assigning screen coordinates to each of the divided areas. there is.

Here, according to an embodiment of the present invention, the screen coordinates of the examinee may be calculated using the learned “gaze position tracking artificial neural network model” to determine which region of the screen the pupil of the examinee is gazing at. can

This will be described in more detail with reference to FIG. 10 . 10 is a diagram for explaining an arrangement structure of screen coordinates for classifying regions gazed by the examinee's pupils. Referring to FIG. 10 , the area where the examinee's pupil gazes may be divided into a central area 510 of the screen, an edge area 520 of the screen, and an outer area 530 outside the screen.

According to one embodiment of the present invention, the screen coordinate "1" is assigned to the central region, the screen coordinate "2" is assigned to the upper left region, and the screen coordinate "3" is assigned to the upper region of the middle region 510. The screen coordinate "4" is allocated to the upper right area, the screen coordinate "5" is allocated to the left side of the middle area 510, the screen coordinate "6" is allocated to the right side of the middle area 510, and the screen coordinate "6" is allocated to the left side of the middle area 510. Screen coordinates "7" are assigned to the lower area, screen coordinates "8" are assigned to the lower area of the middle area (510), screen coordinates "9" are assigned to the lower right area, and screen coordinates are assigned to the outer area (530). Coordinate "10" can be assigned.

Here, the central area 510 and the edge area 520 of the screen to which screen coordinates “1 to 9” are assigned are collectively referred to as a safe area, and the outer area 530 of the screen to which screen coordinates “10” is assigned is It can be collectively referred to as a non-safe area.

The gaze position tracking artificial neural network model according to an embodiment of the present invention determines whether the examinee gazes at nine safe areas 510 and 520 located inside the screen or one non-safe area located outside the screen with respect to the pupil images as shown in FIG. It may be a model learned by labeling whether or not the examinee gazes at 530 and whether the examinee closes his or her eyes.

For example, i) the test taker clicks the areas of “1” to “10” displayed on the screen one by one with a mouse according to the guidance, and ii) sequentially clicks “1” to “10” with the mouse at the time of the click. Matching screen coordinates “1” to “10” corresponding to the pupil image captured by the camera, and iii) randomly clicking after sequential clicks, screen coordinates corresponding to the pupil image captured by the camera at the time of the click “ 1" to "10" are matched, and iv) a gaze position tracking artificial neural network model may be generated by using matching data of the screen coordinates and the pupil image as learning data.

The gaze position tracking artificial neural network model according to the present invention determines whether the examinee gazes at nine safe areas 510 and 520 located inside the screen or one non-safe area 530 located outside the screen, and whether the examinee gazes at one non-safe area 530 located outside the screen, and the examinee closes his/her eyes. By setting the 11 classes for knowledge as a softmax layer, it is possible to calculate the screen coordinates of the pupils of the examinee's pupils.

Meanwhile, according to the above example, although the number of screen coordinates inside the screen has been described as an example, the concept of the present invention is not necessarily limited thereto, and the number of screen coordinates may be smaller or larger than this. In addition, the number of screen coordinates for the outside of the screen may also be one or more or less than this number as needed.

Meanwhile, the process of determining whether the examinee's gaze is located in the safe area (S434) may be performed by determining whether the examinee's pupil gazes at the safe area 510 or 520 or the non-safe area 530. Specifically, according to the present invention, when the calculated screen coordinates are "numbers 1 to 9" corresponding to the inner area of the screen, it can be determined that the examinee's pupil is located in the safe area 510 or 520. However, if the calculated screen coordinates are “No. 10” corresponding to the area outside the screen, it may be determined that the pupil of the examinee is located in the non-safe area 530 .

If the pupil of the examinee is located within the safe area (510, 520), it is determined that it does not correspond to cheating, and a gaze tracking completion process (S435) may be performed, and the pupil of the examinee is located in the non-safe area (530). If so, it can be judged to constitute misconduct.

If it is determined that the test taker's behavior corresponds to cheating through the process of determining whether the pupil is located in the safe area (S434), the test taker is notified through the manager and the dashboard labeling process (S42) and cheating check A process (S44) may be performed, and depending on the result of the cheating check process (S44), an evaluation confiscation process (S46) may be performed or a cheating detection start process (S40) may be performed.

11 and 12 are diagrams for explaining a cheating detection step using action recognition according to exemplary embodiments of the present invention.

Referring to FIG. 11, in the action recognition process (S45) according to exemplary embodiments of the present invention, a human object is recognized from an image of the examinee (S450), and it is determined whether the number of recognized persons is one or more. (S451), after setting a bounding box for the recognized human object, the image is cropped (S452), joint data is extracted from the cropped image (S453), and the examinee's behavior is based on the extracted joint data. It can be performed by determining whether this corresponds to a safety action (S454).

In the process of recognizing a human object from the image of the examinee (S450), after recognizing various objects such as a person, a pencil, and a notebook through an object recognition artificial neural network model, the location of the object is estimated and bounding box It may include a process of classifying into.

In particular, the object recognition artificial neural network model according to the present invention can be learned through a dataset to effectively extract a person who is a fraud monitoring target.

On the other hand, the process of determining whether it corresponds to a safe action (S454) may be performed by detecting whether the action radius of the extracted joint data of the examinee is out of a preset safe action range.

Specifically, the process of determining whether or not it corresponds to a safe behavior (S454) compares the extracted joint data of the examinee extracted from the joint data extraction process (S453) with the previously extracted joint data based on the safe behavior posture. By doing so, it can be performed by detecting whether the action radius of the examinee's joint data is out of a preset safe action range.

According to an embodiment of the present invention, in the process of determining whether the behavior corresponds to a safe behavior (S454), the movement form of the hand in the joint data of the examinee performing the non-face-to-face evaluation is determined by using an artificial neural network model to determine normal behavior and cheating behavior. It can be performed by determining as . Here, the artificial neural network model is a Recurrent Neural Network (RNN), which is an effective deep learning technique for recognizing regular patterns in data and identifying current information through past information while learning the order through a structure in which certain parts are repeated. It can discriminate between normal behavior and cheating behavior through RNN.

For example, as shown in FIG. 12 , the artificial neural network model according to the present invention may determine that a safe action is out of order when a situation in which the examinee's hand leaves the screen occurs.

If it is determined that the behavior of the examinee corresponds to cheating through the process of determining whether the behavior corresponds to safety (S454), the process of notifying the examinee through the administrator, labeling the dashboard (S42), and confirming cheating (S44) This may be performed, and depending on the result of the cheating confirmation process (S44), the evaluation confiscation process (S46) may be performed or the cheating detection start process (S40) may be returned.

In one embodiment, the action recognition process (S45) is performed before the process (S454) for determining whether the action corresponds to a safe action is performed, and may further include a step of guiding the examinee about a safe action posture. can

13 is a flowchart illustrating a fraud detection step using sound detection according to exemplary embodiments of the present invention.

13, in the sound detection process (S47) according to exemplary embodiments of the present invention, after checking the operation of the webcam microphone, the decibel is measured (S470), and the decibel of the measured sound is within the safe decibel range (D). (S471), obtaining a section in which the sound occurred (S472), performing voice operation detection for the obtained sound section (S473), and determining whether a voice is detected in the obtained sound section. It may be performed by determining whether or not (S474).

In this case, the safe decibel range (D) may be a range obtained by adding and subtracting a pre-stored error range from an average value of collected values of living noise, and the sound section may be obtained in units of 10 ms such as 10 ms, 20 ms, and 30 ms. can

In exemplary embodiments, when the decibel (dB) of the sound data measured from the webcam microphone is out of a preset safe decibel range, or it is detected that the sound data measured from the webcam microphone contains voice, cheating is committed. In this case, the test taker notification and dashboard labeling process (S42) and cheating check process (S44) can be performed through the administrator, and the evaluation is based on the results of the cheating check process (S44). The confiscation process (S46) may be performed or the cheating detection start process (S40) may be returned.

In the above, in the fraud prevention method in the non-face-to-face evaluation, it has been described that the identity authentication process (S41), the eye tracking process (S43), the action recognition process (S45), and the sound detection process (S47) are sequentially performed. , the concept of the present invention is not necessarily limited thereto. That is, the identity authentication process (S41), eye tracking process (S43), action recognition process (S45), and sound detection process (S47) may be performed simultaneously or in a different order from the previously described order.

As described above, the method for preventing cheating in a non-face-to-face evaluation according to exemplary embodiments of the present invention includes identity authentication confirmation (S41), gaze tracking (S43), behavior based on data extracted from a photographed image of an examinee. Through recognition (S45) and sound detection (S45), cheating by test takers can be prevented, and accordingly, cheating in non-face-to-face evaluation can be prevented, reliability of evaluation can be improved, and fairness of evaluation results can be improved. this can be established.

14 is a flowchart illustrating a configuration for executing a method for preventing cheating in non-face-to-face evaluation according to exemplary embodiments of the present invention.

Referring to FIG. 14 , an apparatus 100 for performing a method for preventing cheating in a non-face-to-face evaluation according to exemplary embodiments of the present invention includes a face detection module 110, a pupil detection module 120, and joint detection. It can be performed through the module 130, and the face detection module 110, the pupil detection module 120, and the joint detection module 130 are cameras integrally installed in the computer or cameras installed separately from the computer. It may be provided to detect the examinee's face region, pupil region, and joint region from the photographed image of the examinee outputted from the test taker and generate data for each region. In this case, the face detection module, the pupil detection module, and the joint detection module may each use a face detection algorithm such as adaboost, a support vector machine (SVM), or a neural network. . In particular, the pupil detection module may detect a pupil area using a pupil detection algorithm such as circle fitting and ellipse fitting.

In addition, the apparatus 100 for performing the method for preventing cheating in non-face-to-face evaluation according to exemplary embodiments of the present invention may further include a sound detection module, and the sound detection module is integrally installed in the computer. It may be provided to measure decibels (dB) from sound data output from a webcam microphone or a webcam microphone installed separately from the computer and detect human voice.

In exemplary embodiments, the face detection module 110, the pupil detection module 120, the joint detection module 130, and the sound detection module 140 each perform a self-authentication process (S41), a gaze tracking process (S43), It may be provided to perform an action recognition process (S45) and a sound detection process (S47).

Meanwhile, an apparatus 100 for performing a method for preventing cheating in non-face-to-face evaluation according to exemplary embodiments of the present invention includes a face detection module 110, a pupil detection module 120, a joint detection module 130, and storing data generated from the control unit 150 for controlling the sound detection module 140, the face detection module 110, the pupil detection module 120, the joint detection module 130, and the sound detection module 140. A storage unit 160 for storage, and a communication unit 170 for transmitting the data to a server or receiving information from the server may be further included.

The above description is merely an example of the technical idea of the present invention, and those skilled in the art can make various modifications, changes, and substitutions without departing from the essential characteristics of the present invention. will be.

Therefore, the embodiments disclosed in the present invention and the accompanying drawings are not intended to limit the technical idea of the present invention, but to explain, and the scope of the technical idea of the present invention is not limited by these embodiments and the accompanying drawings. . The protection scope of the present invention should be construed according to the following claims, and all technical ideas within the equivalent range should be construed as being included in the scope of the present invention.

Meanwhile, the method according to various embodiments of the present invention described above may be implemented as a program and provided to servers or devices. Accordingly, each device may access a server or device in which the program is stored and download the program.

In addition, the method according to various embodiments of the present invention described above may be implemented as a program and stored in various non-transitory computer readable media to be provided. A non-transitory readable medium is not a medium that stores data for a short moment, such as a register, cache, or memory, but a medium that stores data semi-permanently and can be read by a device. Specifically, the various applications or programs described above may be stored and provided in non-transitory readable media such as CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, and the like.

In addition, although the preferred embodiments of the present invention have been shown and described above, the present invention is not limited to the specific embodiments described above, and the technical field to which the present invention belongs without departing from the gist of the present invention claimed in the claims. Of course, various modifications are possible by those skilled in the art, and these modifications should not be individually understood from the technical spirit or perspective of the present invention.

Claims (12)

A first misconduct detection step of confirming an identity based on a facial image extracted from a photographed image of a test taker; and
A second cheating detection step of calculating screen coordinates at which the pupils of the examinee gaze at from the photographed image of the examinee and determining whether cheating is applicable based on the calculated screen coordinates; includes,
The second cheating detection step,
If the calculated screen coordinates are located in the inner area of the screen, it is determined that it does not correspond to cheating;
Cheating prevention method in non-face-to-face evaluation, characterized in that when the calculated screen coordinates are located in the area outside the screen, it is determined that they correspond to cheating. According to claim 1,
The first misconduct detection step,
Performed by comparing the first facial region extracted from the photographed image of the examinee with "at least one of a second facial region extracted from the ID of the examinee and a third facial region extracted from a recent photo submitted by the examinee" Method for preventing cheating in non-face-to-face evaluation, characterized in that. According to claim 2,
The first misconduct detection step is characterized in that the identity of the examinee is confirmed by extracting vector values from the first to third facial regions, respectively, and comparing Euclidean distances of the extracted vector values. Methods to prevent cheating in non-face-to-face evaluations. According to claim 1,
The second cheating detection step,
Cheating prevention method in non-face-to-face evaluation, characterized in that the screen coordinates are calculated using an artificial neural network model learned to determine in which area of the screen the pupil of the examinee looking at the screen is located. According to claim 4,
The area gazed by the pupil of the examinee is divided into a central area of the screen, an edge area of the screen, and an outer area outside the screen;
Preventing cheating in non-face-to-face evaluation, characterized in that the central area of the screen and the edge area of the screen are set as safe areas that do not correspond to cheating, and the outer area outside the screen is set as a non-safe area that corresponds to cheating method. According to claim 5,
The artificial neural network model sets a plurality of classes for whether the examinee's pupil gazes at a safe area located inside the screen, a non-safe area located outside the screen, and whether the examinee closes their eyes as a softmax layer. By doing so, a cheating prevention method in non-face-to-face evaluation, characterized in that for calculating the screen coordinates that the examinee's pupil gazes at. According to claim 1,
A third misconduct detection step of detecting a third misconduct based on object recognition data recognized from the photographed image of the examinee; further comprising,
The third fraud detection step,
Recognizing a person from the photographed image of the examinee and extracting object recognition data;
extracting joint data of the examinee from the object recognition data;
A method for preventing cheating in non-face-to-face evaluation, characterized in that performed by determining that the examinee's joint data is out of the preset safe behavior range. According to claim 7,
A fourth cheating detection step of determining whether or not cheating is applicable based on sound data extracted from the photographed image of the examinee; further comprising,
The fourth fraud detection step,
Measuring sound data from the photographed image of the examinee,
Cheating prevention method in non-face-to-face evaluation, characterized in that performed by determining that the sound data corresponds to cheating when the decibel (dB) of the sound data is out of the preset safe decibel range. According to claim 8,
The fourth fraud detection step,
A method for preventing cheating in a non-face-to-face evaluation, characterized in that performed by determining that the measured sound data contains a voice. According to claim 9,
If at least one of the first misconduct detection step, the second misconduct detection step, the third misconduct detection step, and the fourth misbehavior detection step is not satisfied, it is characterized in that the test taker is judged as cheating. Methods to prevent cheating in non-face-to-face evaluations. a face detection module for detecting a face image of the examinee from a photographed image of the examinee;
a pupil detection module for detecting a pupil image of the examinee from the captured image of the examinee;
a joint detection module for detecting joint data of the examinee from the captured image of the examinee;
a sound detection module for detecting sound data from the photographed image of the examinee; and
Including; a controller for controlling the face detection module, the pupil detection module, the joint detection module, and the sound detection module to determine the cheating of the examinee,
The control unit,
Screen coordinates at which the examinee's pupil gazes are calculated from the photographed image of the examinee, and if the calculated screen coordinates are located in the inner area of the screen, it is determined that the act does not correspond to cheating, and the calculated screen coordinates are located outside the screen. An anti-cheating device in non-face-to-face evaluation, characterized in that it is determined that it corresponds to cheating when located in the area. A program stored in a recording medium including a program code for executing the method for preventing cheating in a non-face-to-face evaluation according to any one of claims 1 to 10.

Images