KR20230030599A - Cathexis learning system and method using AI in an untact learning based on Web browser - Google Patents

Abstract

Disclosed are a cathexis learning system and method using artificial intelligence (AI) in web browser-based non-contact learning. The cathexis learning system using AI in web browser-based non-contact learning comprises: a learning content server providing non-contact lecture learning content; and a user terminal connect to the learning content server to run an educational content viewer program, and having an educational content viewer installed thereon so that a user learns, in an AI cathexis manner, learning content by using AI face recognition/motion recognition and sound playback techniques for image data from a front camera of the user terminal during non-contact learning. For non-contact learning, the educational content viewer of the user terminal includes a media playback unit and a facial outline recognition unit module. The cathexis learning system recognizes the outline of a face and the features of the face such as two eyes, a nose, and two ears on a five-point scale so that the user learns, in the AI cathexis manner, non-contact lecture learning content focused on the camera of the user terminal linked to the learning content server (ubcloud server) while gazing at the non-contact lecture learning content, generates an alarm or warning message to the user terminal by using sound and messaging techniques if the gaze is out of the non-contact lecture learning content by a predetermined angle, and provides cathexis learning using web browser-based ubcloud AI during non-contact learning.

Description

{Cathexis learning system and method using AI in an untact learning based on Web browser}

The present invention relates to an attentional learning system and method using artificial intelligence in non-face-to-face learning (ubcloud), and more particularly, in online lecture non-face-to-face learning, an educational content viewer of a user terminal is a media playback unit and a facial contour recognition unit module. , and five facial feature points of eyes 2/nose/ear 2 so that attention is focused on the learning content focused on the front camera of the user terminal linked to the learning content server (ubcloud server). If the eyes looking at the user terminal at a certain angle are deviating through scale facial contour recognition, an alarm/warning message is generated to the user terminal using sound and messaging technology, and in non-face-to-face learning, using web browser-based ubcloud artificial intelligence It relates to an attentional learning system and method using artificial intelligence in web browser-based non-face-to-face learning (ubcloud) that provides attentional learning.

As face recognition technology, an appearance based matching method and feature based face recognition are mainly used. Face recognition is differently recognized according to a photographing angle of a camera, a direction of lighting, a pose, a change in expression, and a change in face over time.

For feature-based face recognition, a detection method using Haar-like features and a detection method using MCT (Modified Census Transform) images are used for image data captured by a digital camera, IoT device camera, or smartphone camera. . Using the face and eye detectors learned as Haar-like features from the input images of the smartphone camera, the contours of the face and feature points of the forehead/eyes/nose/mouth are detected, and the region of interest (ROI, ROI, The eye area set as Region of Interest) is converted to grayscale, and the histogram of the eye image [pixels of each pixel on the x-axis] is used by using a statistical threshold by experiment in which the pupil and the outline of the eye are extracted from the eye area. value, the number of pixel values corresponding to the y-axis], binarizes the image of the eye, and pre-processes the eye area photo through histogram equalization. In the face area, the eyebrows, eyes, nose, The face is recognized by detecting face data having facial feature points of the mouth and outline, extracting texture features and shape features, and comparing similarity with feature points of face photos stored in the face recognition DB.

The feature values of the eyebrows, eyes, nose, mouth, and chin of the face area are expressed as the difference between the sum of pixels included in the white area of the Haar-like feature and the sum of pixels included in the black area.

For example, the distance from the eye area detected in a standard horizontal and vertical face area picture to the endpoints of the right and left eyes, the size of the iris extracted using a hough circle transform algorithm value is used as the feature value.

1A is a configuration diagram of an existing face recognition device.

The face recognition device 100 includes an image display device, an image recording device, a face recognition server, a tablet PC, a laptop, a personal PC, a smartphone, a personal digital assistant (PDA), a mobile communication terminal, and It may be any one of an intelligent robot and the like.

The face recognition apparatus 100 includes an input image acquisition unit 112 acquiring an input image from a camera; A front pose image is generated by detecting a face region in the input image and normalizing a face pose. a normalization unit 114 for normalizing ) to generate a normalized image; a feature vector extraction unit 116 extracting a feature vector representing the face of the subject from the normalized image; and a face recognition unit 118 recognizing the face of the subject included in the input image by applying the feature vector to the previously learned classification model.

The input image acquisition unit 112 acquires an input image from a camera. The camera can be a depth-aware camera, a stereo camera, and a color camera (eg, a Kinect camera, etc.). In addition, the input image is an image including a face of a subject to be recognized, and includes a 2D still image and a moving image. The input image may include a color image, a depth image, and a color-depth (RGB-D) image.

The normalization unit 114 detects a face region from an input image and normalizes a face pose and perspective to generate a normalized image. When there is a change in the face pose, the face recognition rate is lowered because the gray scale, shape, location of feature points, etc. are changed. In addition, if the distance between the camera and the subject is different, perspective distortion (distortion) occurs differently for each photographed position even for the same subject, so it may seem as if a different subject has been photographed. Therefore, it is necessary to normalize the face pose and perspective of the input image in order to improve the face recognition rate.

The normalization unit 114 inputs face images for learning in various poses to the input layer of the first artificial neural network, and trains the first artificial neural network so that face images for learning in front poses are output from the output layer of the first artificial neural network. face pose normalization learning unit; And inputting the data output from the output layer of the first artificial neural network to the input layer of the second artificial neural network, and learning the second artificial neural network so that a face image for learning without perspective distortion is output from the output layer of the second artificial neural network. Includes a perspective normalization learning unit.

The normalization unit inputs face images for learning in various poses with various perspective distortions to the input layer of the integrated artificial neural network in which the first artificial neural network and the second artificial neural network are integrated, and the front pose is free of perspective distortion. The integrated artificial neural network is trained so that a facial image for learning is output from an output layer of the integrated artificial neural network.

The feature vector extractor 116 is determined through machine learning and extracts a feature vector representing a face of a subject from a normalized image.

The feature vector is a vector having feature values used for face recognition as elements. Filters used to extract feature vectors include Gabor filter, Haar filter, LBP (Local Binary Pattern) - including DLBP (Discriminative LBP), ULBP (Uniform LBP), NLBP (Number LBP), etc. - but must be limited to these and other filters may be used.

The face recognition unit 118 recognizes the face of a subject included in the input image by applying the feature vector extracted from the feature vector extraction unit 116 to the previously learned classification model. The pre-learned classification model may include, but is not limited to, Support Vector Machine (SVM), Linear Discriminant Analysis (LDA), and Softmax.

The virtual face image generation unit 124 may generate a plurality of virtual face images used for learning by the normalization unit 114 , the feature vector extraction unit 116 , and the face recognition unit 118 .

The plurality of virtual face images refer to face images generated by transforming a 3D face model synthesized by the virtual face image generation unit 124 using one or more 2D reference images obtained from a camera.

As prior art 1 related to this, Patent Registration No. 10-2103521 has registered “a video object recognition system and method based on artificial intelligence deep learning”.

1B is a block diagram of a conventional video object recognition system based on artificial intelligence deep learning. Various images are pre-learned through deep learning of artificial intelligence, and images are analyzed in frame units for distributed video materials by reflecting the above-mentioned pre-learning results, and in some cases, the audio information of the video material is also analyzed in chronological order. After deriving keywords, we provide a video recognition system and method based on artificial intelligence deep learning that compares them with the keywords accumulated in advance with the time-series keywords of video scripts.

A system implemented by artificial intelligence includes a dataset learning unit for deep learning using a plurality of object images as keywords; a DB unit for extracting keywords from video scripts, chronologically arranging them, and storing them; a video object analysis unit that extracts objects on the image and the relationship between the objects as keywords through frame-by-frame image analysis of the video object, and sequentially arranges them; and a comparison determination unit that compares keywords of the DB unit and the video analysis unit to determine similarity.

However, existing e-learning or u-learning systems simply provided online learning content, and a function that generates an alarm and enables attention-focused learning when a learner carelessly loses his or her facial gaze by using face recognition technology, sound, and messaging technology. did not provide

Patent Registration No. 10-2103521 (Registration Date: April 16, 2020), "Video object recognition system and method based on artificial intelligence deep learning", Sangmyung University Industry-University Cooperation Foundation

An object of the present invention to solve the above problem is that in online lecture non-face-to-face learning, the educational content viewer of the user terminal has a media playback unit and a facial contour recognizing unit module, and the user terminal linked to the learning content server (ubcloud server). Directly look at the learning content focused on the front camera and focus on the learning content of the non-face-to-face lecture, so that the gaze is directed at a certain angle through facial contour recognition on a 5-point scale of face contour and eye 2/nose/ear 2 facial feature points. In case of deviation, an alarm/warning message is generated to the corresponding user terminal using sound and messaging technology, and in non-face-to-face learning, artificial intelligence It provides an attention learning system using intelligence.

Another object of the present invention is to provide an attention learning method using artificial intelligence in web browser-based non-face-to-face learning.

In order to achieve the object of the present invention, an attention learning system using artificial intelligence in web browser-based non-face-to-face learning includes a learning content server that provides non-face-to-face online lecture learning content; And accessing the learning content server through a wired/wireless communication network to drive the educational content viewer program, and in non-face-to-face online learning, AI face recognition/motion recognition sound reproduction technology of the front camera image data of the user terminal is used to display learning content through artificial intelligence. Including a user terminal in which an educational content viewer is installed for attention-focused learning,

The user terminal is a VOD media player for playing non-face-to-face lecture learning content; A chat module that transmits and receives chat data between the instructor and the learner; a face recognition module for recognizing a facial behavior pattern of the front camera image of the user terminal and recognizing a 5-point scale for facial feature points of eyes 2/nose/ear 2; and when it is more than a certain angle away from the screen (depending on whether the 3 eye/nose points get closer to the 2 ears at both ends, when the distance in the corresponding direction of the eye-ear distance or nose-ear distance exceeds a certain standard value) or the learner's voice includes a learner's alarm generating unit that generates an alarm or outputs a warning message to a user terminal of the learner so that attention is learned when learning is recognized, and the user terminal further includes a voice recognition module that recognizes the learner's voice,

The facial recognition module uses the posenet algorithm as an AI-based facial contour recognition technology,

The educational content viewer has a VOD media player, a chat screen, and an online learning memo space,

In non-face-to-face online learning, the instructor's terminal transmits and receives question-and-answer chatting data to a plurality of learner terminals in a 1:N manner through the learning content server.

In order to directly look at the non-face-to-face online lecture learning content focused on the front camera of the user terminal and receive attention training, the eye 2/nose/ear 2 facial feature point 5-point scale facial contour recognition is used to prevent the gaze from deviating at a certain angle. alarm/warning messages using sound and messaging technology.

In order to achieve another object of the present invention, the attentional learning method in web browser-based non-face-to-face learning includes (a) receiving and storing learner information and a standard-sized frontal face picture from a user terminal in a learning content server; (b) issuing, by the learning content server, a QR code corresponding to user information and a front face picture; (c) In a user terminal equipped with an educational content viewer program and a face recognition module, the recognition result of the front face photo of the camera is received by the learning content server using the front face recognition algorithm of the front camera image, and the previously registered learner information and Checking whether the learner is himself by comparing a standard size frontal face photo with facial feature points; (d) providing learning content from the learning content server to a user terminal of a pre-registered member; and (e) the learning content server analyzes the online learner's face image detected by the camera at each learner's user terminal, and sends the learning content server to the corresponding user terminal for attention-focused learning when the gaze is out of a certain angle or when the learner's voice is recognized. generating a warning message or alarm;

In non-face-to-face online learning, the educational content viewer has a VOD media player, a chatting screen, and an online learning memo space, and the instructor's terminal has question and answer chatting with a plurality of learner terminals in a 1:N manner through the learning content server. Further comprising the step of transmitting and receiving data,

The user terminal is a VOD media player for playing non-face-to-face lecture learning content; A chat module that transmits and receives chat data between the instructor and the learner; a face recognition module recognizing a facial behavior pattern of the front camera image of the user terminal and recognizing feature points of the face on a 5-point scale of eye 2/nose/ear 2; and when it is more than a certain angle away from the screen (depending on whether the 3 eye/nose points get closer to the 2 ears at both ends, when the distance in the corresponding direction of the eye-ear distance or nose-ear distance exceeds a certain standard value) or the learner's voice includes a learner's alarm generating unit that generates an alarm or outputs a warning message to the learner's user terminal so that attention is learned when recognizing , and the user terminal further includes a voice recognition module for recognizing the learner's voice,

The facial recognition module uses the posenet algorithm as an AI-based facial contour recognition technology.

In the web browser-based non-face-to-face learning system and method of the present invention, in the non-face-to-face learning, the educational content viewer of the user terminal is provided with a media playback unit and a facial contour recognition module, and a learning content server ( ubcloud server), face contour and eye 2 / nose / ear 2 facial feature points 5-point scale facial contour recognition to direct attention to the non-face-to-face lecture learning content focused on the front camera of the user terminal linked to the user terminal When the eyes are diverted at a certain angle through this, an alarm/warning message is generated to the user terminal using sound and messaging technology, and it has the effect of providing attention-focused learning using web browser-based ubcloud artificial intelligence in non-face-to-face learning.

1A is a configuration diagram of an existing face recognition device.
1B is a block diagram of a conventional video object recognition system based on artificial intelligence deep learning.
2a and 2b are implementation examples of an attention-focused learning system using web browser-based ubcloud artificial intelligence in non-face-to-face online learning.
3A is a diagram showing the concept of a facial contour recognition artificial intelligence platform having a tablet PC, a smart phone, and a PC-based learning content server during online learning.
3B is a diagram showing a face recognition function on the ubiquitous-based NSDAI platform.
3c and 3d show the goals and User Experience (1. User registration -> 2. Learning -> 3. QR Code generation -> 4. UBT App/Web login (QR code/passwd, ID/passwd) -> 5. Supervisor confirmation and test -> 6.7 UBT App/Web camera facial contour recognition 8. End of test) It is a picture.
4 shows the process from learner/test taker registration to test taking department, 1) candidate registration, 2) learning, 3) QR code issuance, 4) QR code and face recognition, 5) supervisor confirmation (applicant face/test taker information), 6) Taking the test - This is a picture showing the process.
5 is a screen for registering/learning/QR code issuing a candidate-face matching code issuing unit-at the time of an online test/UBT test based on machine learning and learning results of a registered candidate's face.
6 is a test taker identification unit screen in the UBT system that measures the distance and similarity of facial feature points of a standard-sized frontal face picture based on the vertex of the nose through AI-based facial contour recognition of the test taker's face.
7 is a rule user definition unit and a program test screen when determining a learner.
8 is a learning content server access/login/left-QR code recognition/recognition of QR code during non-face-to-face online learning (TTS)/right-face recognition (smartphone/tablet PC front camera)/face in the background This is the demonstration screen including the process of starting area recognition/recognition rate display - UBT authentication process user definition part using the generated user code.
9 is a block diagram of an attentional learning system using artificial intelligence in tablet PC, smartphone, and PC-based web browser-based non-face-to-face learning using facial contour recognition artificial intelligence in web browser-based non-face-to-face learning according to the present invention.
10 is a diagram explaining the functions of an AI-based facial contour recognition module during online learning and testing.
11 is a flowchart showing a tablet PC, smartphone, and PC-based attention learning method using facial contour recognition artificial intelligence in web browser-based non-face-to-face learning according to the present invention.

Hereinafter, the configuration and operation of a preferred embodiment of the present invention will be described in detail with reference to the accompanying drawings. In the description of the present invention, if it is determined that a detailed description of a related known technology or configuration may unnecessarily obscure the subject matter of the present invention, the detailed description will be omitted. In addition, the attached drawing numbers refer to the same drawing numbers in other drawings when indicating the same configuration.

In the web browser-based non-face-to-face learning system and method of the present invention, in the non-face-to-face learning, the educational content viewer of the user terminal is provided with a media playback unit and a facial contour recognition module, and a learning content server ( ubcloud server), face contour and eye 2 / nose / ear 2 facial feature points 5-point scale facial contour recognition to direct attention to the non-face-to-face lecture learning content focused on the front camera of the user terminal linked to the user terminal When the gaze is diverted at a certain angle through the system, an alarm/warning message is generated to the user terminal using sound and messaging technology, and attentional learning using web browser-based ubcloud artificial intelligence is provided in non-face-to-face learning.

* Same facial features and contour recognition technology (ubcloud)

* Added learning content server and exam server function

* It is characterized by being browser-based and being able to use both online (learning data-based online reading and online storage) / offline (library browser built-in and web page built-in and recognition data stored on the user's device).

* Utilizes sound and messaging to draw attention.

The attention learning system and method using artificial intelligence in web browser-based non-face-to-face learning of the present invention is an educational content viewer in which a user terminal (tablet PC, smartphone, PC) is linked with a learning content server during non-face-to-face online learning. is installed, using AI face recognition / motion recognition / sound and messaging technology, using AI-based face recognition (posenet algorithm, machine learning model which allows for real-time face pose estimation) technology using the front camera of the user terminal A face recognition module is installed that recognizes the contour of the face and the feature points of the face on a 5-point scale of eyes 2/nose/ear 2, and when recognizing a face, it learns a frontal face picture taken using the front camera (C) of the test taker terminal. Detect whether the learner is present by comparing the facial feature points of the standard-sized front face photo of the learner in the face photo DB of the content server, and in non-face-to-face online learning, AI-based real-time using the front camera image data of the user terminal By recognizing the facial contours and detecting the behavior pattern of the learner's face in the video, the user's eyes are directly focused on the non-face-to-face online lecture learning content focused on the front camera of the user terminal linked to the learning content server (ubcloud server), so that attention is focused on the education. 2/Nose/Ear 2’s facial feature points 5-point scale Facial contour recognition generates an alarm/warning message to the learner’s user terminal when the gaze is deviated at a certain angle, and uses sound and messaging technology to focus attention do.

User terminals include not only smart phones and tablet PCs that receive online learning contents from the learning contents server through wired/wireless communication networks, but also laptops capable of accessing the Internet.

The user terminal is installed with an educational content viewer that is linked with the learning content server, and uses AI face recognition/motion recognition/sound and messaging technology to detect the contours of the learner's face and facial feature points on a 5-point scale of eyes 2/nose/ear 2. It has a face recognition module that recognizes.

2a and 2b are implementation examples of an attention-focused learning system using web browser-based ubcloud artificial intelligence in non-face-to-face online learning.

3A is a diagram showing the concept of a facial contour recognition artificial intelligence platform having a tablet PC, a smart phone, and a PC-based learning content server during online learning.

3B is a diagram showing a face recognition function on the ubiquitous-based NSDAI platform.

3c and 3d show the goals and User Experience (1. User registration -> 2. Learning -> 3. QR Code generation -> 4. UBT App/Web login (QR code/passwd, ID/passwd) -> 5. Supervisor confirmation and test -> 6.7 UBT App/Web camera facial contour recognition 8. End of test) It is a picture.

4 shows the process from learner/test taker registration to test taker registration, 1) test taker registration, 2) learning, 3) QR code issuance, 4) QR code and face recognition, 5) supervisor confirmation (applicant face/test taker information), 6) Taking the test - This is a picture showing the process.

5 is a screen for registering/learning/QR code issuing a candidate-face matching code issuing unit-at the time of an online test/UBT test based on machine learning and learning results of a registered candidate's face.

6 is a test taker identification unit screen in the UBT system that measures the distance and similarity of facial feature points of a standard-sized frontal face picture based on the vertex of the nose through AI-based facial contour recognition of the test taker's face.

7 is a rule user definition unit and a program test screen when determining a learner.

8 is a learning content server access/login/left-QR code recognition/recognition of QR code during non-face-to-face online learning (TTS)/right-face recognition (smartphone/tablet PC front camera)/face in the background This is the demonstration screen including the process of starting area recognition/recognition rate display - the UBT authentication process user definition part using the generated user code.

(1) The educational content viewer is installed on the user terminal, and the educational content viewer is equipped with a 5-point scale facial recognition module that recognizes facial feature points of eyes 2/nose/ear 2 of the learner's face image using artificial intelligence face recognition. .

QR codes were used as recognition codes for online learning and testing.

In the embodiment, the user terminal uses a tablet PC, and the recognition code uses a QR code.

Artificial intelligence learning the registration photos of learners/test takers generates a recognition code (QR code) for each user, and the test taker holds the test taker with the corresponding recognition code (QR code) attached to the camera of the test center PC or tablet PC, and then the user The result of the recognized AI module pre-learning the candidate information and front face photo (standard-size front face photo with the nose as the vertex), pre-registered candidate information in the face DB of the learning content server, and standard-size front face photo The model is compared, and the similarity of facial feature points is calculated, and if a certain value is exceeded, it is recognized as the person, moved to the learning screen, and the learner learns non-face-to-face online. In case the similarity of facial feature points does not reach a certain level (completely insufficient, pending judgment), the information is transmitted to the supervisor's terminal through the server, and the supervisor conducts online learning after confirming the actual face and personal information.

<System Configuration>

1) Non-face-to-face online learning: test taker terminal (PC/smartphone/tablet) user program -> wired/wireless communication network (WAN, LTE 4G/5G) and internal network (LAN, Wi-Fi) -> learning content server (learning curriculum information / learner information), the front face picture of each user terminal and learner information are transmitted, and the supervisor terminal determines whether to participate in learning.

2) When using offline: The AI face recognition module is installed in the user program (PC/Smartphone/Tablet), and only the final face face recognition result is transmitted to the learning content server after recognizing the facial contour of the candidate's face by itself.

<Partial APK of the app prototype of the learner's device - Attachment for Android>

https://we.tl/t-wFdXexsors File Download Password: nsdevil

Install the app on your Android smartphone or tablet PC

<web version function address>

1) Face Recognition Module: https://facere.nsdai.org (id: nsdevil, passwd: nsdevil) > Take a picture of the QR code with your smartphone and place it on the recognition screen on the PC camera to activate.

2) Facial contour recognition module of face contour and 5-point scale face feature points: https://headpos.ublrandd.com.np (id: nsdevil, passwd: nsdevil) Facial recognition of facial feature points on a 5-point scale

3) Voice recognition module

Speech Recognition is a man-machine interface technology that removes noise from an acoustic speech signal input by a microphone, extracts features of the speech signal, and converts it into a set of words or text of a sentence (mapping). It is a process, microphone -> AMP -> LPF -> ADC -> it is stored in the voice database.

The speech recognition module uses a method using vector quantization, a method using dynamic time warping (DTW), a method using a neural network, and a method using HMM (Hidden Markov Model). method is used The voice recognition module uses a voice recognition module of hardware and software and a voice recognition API of STT (Speech To Text) technology.

9 is a block diagram of an attentional learning system using artificial intelligence in tablet PC, smartphone, and PC-based web browser-based non-face-to-face learning using facial contour recognition artificial intelligence in web browser-based non-face-to-face learning according to the present invention.

10 is a diagram explaining the functions of an AI-based facial contour recognition module during online learning and testing.

The learning content server 100 having a test management unit includes a WWW server 101, a control unit 102, a member registration unit 103, a user authentication unit 104, a learning content providing unit 105, a QR code management unit 106 , supervisor identification unit, face recognition unit 107, examination management unit 109, examination information DB 120, candidate DB 121, and face photo DB 123.

In the web browser-based non-face-to-face learning of the present invention, the attention learning system

A learning content server 100 that provides non-face-to-face online lecture learning content authored by an authoring tool; and

The learning content server 100 is accessed through a wired/wireless communication network to drive the educational content viewer program, and in non-face-to-face online learning, learning content is displayed using AI face recognition/motion recognition sound reproduction technology of the front camera image data of the user terminal. Including user terminals (300, 310, 311) with an educational content viewer installed for artificial intelligence attention-focused learning,

The user terminal (300, 310, 311) is a VOD media player for playing non-face-to-face lecture learning content; A chat module that transmits and receives chat data between the instructor and the learner; a face recognition module recognizing a facial behavior pattern of the front camera image of the user terminal and recognizing facial outlines and 5-point scales of facial feature points of eyes 2/nose/ear 2; And if it is out of the screen by more than a certain angle (depending on whether the 3 eye/nose points get closer to the 2 ears at both ends, if the distance in the corresponding direction of the eye-to-ear distance or the nose-to-ear distance exceeds a certain standard value) or the user of the learner When the terminal recognizes the learner's voice, a learner's alarm generator generates an alarm or outputs a warning message to the user terminal of the learner so that attention is learned, and the user terminal recognizes the learner's voice as a microphone STT ( It further includes a voice recognition module using Speech To Text) technology,

The facial recognition module uses the posenet algorithm as an AI-based facial contour recognition technology,

In order to directly look at the non-face-to-face online lecture learning content focused on the front camera of the user terminal and focus attention, the gaze is directed at a certain angle through face recognition on a 5-point scale of facial contours and eye 2/nose/ear 2 facial feature points. In the case of a miss or when the learner's voice is recognized by the learner's user terminal, an alarm / warning message is generated to the corresponding user terminal using sound and messaging technology so that attention is learned,

The educational content viewer has a VOD media player, a chatting screen, and a memo space for online learning. In non-face-to-face online learning, the instructor's terminal (supervisor's terminal) connects a plurality of learners in a 1:N manner through the learning content server 100. Q&A chatting data is transmitted and received to terminals.

In non-face-to-face online learning, the learning content server 100 provides learning content regardless of time and place and underlines or texts, pictures, and photos in the online learning memo space (shared workspace) of the educational content viewer. , Sending/receiving files with attached reports/learning files and sending/receiving notes are provided to the instructor's terminal (supervisor terminal) and a plurality of learner terminals according to the instructor's floor control through voice recording,

In the case of non-face-to-face lectures, the instructor's terminal (supervisor terminal) is a plurality of learner terminals in a 1:N manner through the learning content server 100. Writing / graphic writing / memo in the shared workspace of the educational content viewer writing is displayed.

In non-face-to-face online learning, the instructor's terminal (supervisor terminal) transmits and receives question-and-answer chatting data to a plurality of learner terminals in a 1:N manner through the learning content server 100.

In non-face-to-face online learning, regardless of time and place, during online learning through the learning content server, the instructor provides learning content and underlines or texts, pictures, and photos in the online learning memo space (shared workspace) of the educational content viewer. , It provides sending/receiving files with attached reports/learning files and sending/receiving notes to the instructor's terminal (supervisor's terminal) and a plurality of learner terminals according to the instructor's floor control through voice recording.

In non-face-to-face online learning, during non-face-to-face lectures, the instructor's terminal (supervisor terminal) is a 1:N method through the learning content server 100 to a plurality of learner terminals in the shared workspace of the educational content viewer. Text/Graphic handwriting/Memo writing is displayed.

The learning content server 100 provides a function of recognizing a QR code as a recognition code when using a smartphone.

The user terminal uses any one of a tablet PC, a smartphone, and a PC having a front camera, and the educational content viewer downloaded from the learning content server 100 is installed, and the educational content viewer is VOD media. A chat screen with the player is provided.

The user terminal

A VOD media player that plays non-face-to-face lecture learning content;

A chat module that transmits and receives chat data between the instructor and the learner; and

a face recognition module for recognizing a facial behavior pattern of the front camera image of the user terminal, and recognizing face outlines and facial feature points on a 5-point scale of eyes 2/nose/ear 2; and

If the line of sight deviates from the screen by more than a certain angle (when the distance between the eyes and ears and the distance between the nose and ears in the corresponding direction exceeds a certain standard depending on whether the 3 points of the eye/nose get closer to the 2 points of the ears at both ends), the user of the learner and a learner's alarm generating unit generating an alarm or outputting a warning message to the terminal.

The face recognition module of the user terminal uses the posenet algorithm as an AI-based face contour recognition technology.

The face recognition module extracts a face object using a face recognition algorithm for a front face photo of a learner of a standard size that has been resized/rotated/angle corrected during non-face-to-face online learning, recognizes a facial action pattern, and recognizes the learner's face. The contour line and facial feature points of Eye2/Nose/Ear2 are extracted, and the Euclidean distance between the facial feature points of Eye2/Nose/Ear2 and the center point (pupil) of the left/right ear and the left/right eye, respectively ( d) and similarity are calculated, and compared with specific points of the user's face photo in the face photo DB, it is checked whether the line of sight is deviating, and depending on whether the 3 eye/nose points are closer to the 2 ears at both ends, the right/left/right/nose points are calculated. Detects head movement to the left, detects face patterns related to gaze deviation, and detects facial contours when recognizing faces, or when it is out of the screen by more than a certain angle (3 points of eyes/nose are closer to 2 points of both ears If the eye-to-ear distance or the nose-to-ear distance in the corresponding direction exceeds a certain standard value), the facial behavior pattern captured by the camera of the user terminal is recognized, and the right eye and the right ear are Since the distance and the distance between the left eye and the left ear are different, the learner's user terminal generates an alarm or outputs a warning message so that attention is learned.

The learning content server 100

WWW server 101;

a control unit 102 controlling to provide online learning content to the user terminal;

A member registration unit 103 connected to the control unit 102 and receiving registered member information and storing and managing ID/Passwd;

A user authentication unit 104 that is connected to the control unit 102 and authenticates using a QR code/Passwd or ID/Passwd or a personal certificate;

A QR code management unit 106 connected to the control unit 102 and issuing and managing a QR code corresponding to user information;

a learning content provider 105 connected to the controller 102 and providing the online learning content to the user terminal;

A chatting server 111 connected to the control unit 102 and transmitting and receiving chatting data between the instructor and the learner;

It is connected to the control unit 102, receives the front face picture of the learner/applicant from the camera from the user terminal, converts it into a standard size front face picture through size correction/rotation/angle correction of the standard size, and stores it in the database of the server. a supervisor checking unit that compares the stored user information with a standard-size frontal face picture and confirms eligibility for online learning/online exam by checking the supervisor's terminal at the supervisor's terminal;

Connected to the control unit 102, the user terminal uses a face recognition module to recognize the outline of the learner's face and facial feature points of the eyes 2/nose/ear 2, and face recognition receiving the face recognition result from the user terminal. part 107; and

a test information DB for storing test papers, prepared answers, and scoring results of user terminals; It includes a candidate DB and a face DB that store candidate information and a standard size frontal face picture.

Additionally, the learning content server 100 is connected to the controller 102 and further includes a chatting server 111 that transmits and receives chatting data between the instructor and the learner.

Additionally, the learning content server 100 is connected to the control unit 102 and further includes a send/receive note providing unit for transmitting/receiving note data to which the instructor and the learner can attach a report file.

Additionally, the learning content server 100 is connected to the controller 102 and further includes an FTP server for transmitting and receiving learning data files.

In addition, during the online exam or UBT exam, the test program (App) and test papers are provided to the candidate terminal and the supervisor terminal, and the candidate information, the candidate's on-site face picture, and supervisor information are managed, and the online exam or UBT exam A test management unit that stores test paper writing answers in each test taker terminal within a certain test time, transmits them to the test server at the end of the test, and stores and manages test takers' test paper writing answers, scoring results, supervisor information, and test taker status information. do.

Additionally, the AI-based facial recognition technology used in the facial recognition module of the test taker terminal and the anti-cheating module using the eye 2/nose/ear 2 face feature points on a 5-point scale recognizes the facial outline and facial feature points of the camera image of the tablet PC. To do this, the posenet algorithm was used. During an online test or UBT test, AI-based face recognition technology is used to identify the tablet PC test taker through a proctor terminal linked to the test server, and to enable focused learning during online learning or to prevent proxy testing during the test. In online learning, online exams, or UBT exams, substitute exams or visual/auditory cheating are prevented during attention-focused learning or exams by test taker face recognition and voice recognition technology through the server.

The test paper questions of the test program (App) include subjective and/or multiple-choice test questions, and for each question, not only text and images, but also multimedia test questions including text, images, VR/AR content, voice and video are presented and server It is displayed on the user terminal through

1. Artificial Intelligence Hub Platform

ㆍ NSD-AI (NSDevil's Artificial intelligence) platform-based artificial intelligence hub platform

ㆍ Instructor scoring guide through syntax recognition for language and medical education (collaborative research with Inje University College of Medicine and others)

ㆍ Experiential learning support system through object recognition (Fun Edu Lab and Korean Folk Village, etc.)

ㆍ Disease analysis function in dental education sector through tooth image analysis (joint research with Yonsei University College of Dentistry and other institutions in 3 countries)

ㆍ Based on UBT connect platform

ㆍ Tablet PC-based evaluation platform

ㆍ Language evaluation section (Speaking/Listening/Reading/Writing)

ㆍ Health and Medical Evaluation Division (Listening/Reading/Writing)

ㆍ Other job evaluation areas (read/write)

Additionally, the learning content server 100 having a test management unit receives and stores test taker information and front face photos registered from a user terminal through a wired/wireless communication network, issues a QR code attached to a test taker to the test taker, and issues QR codes for each test taker. After recognizing the code, a greeting is provided through TTS conversion technology, and online test papers and a test program for conducting an online test or UBT test are provided to test taker terminals.

Additionally, the online exam or UBT exam offers subjective exams as well as 2/3/4/5 choice multiple-choice exams. The subjective test questions provide a subjective test in which the handwritten text of the stylus pen is recognized using the handwriting recognition unit of the test taker's terminal having a touch sensor and a display and converted into text to recognize the handwritten text.

Online test papers can be presented in a combination of all multiple-choice and multiple-choice/subjective forms.

Candidate terminals (300, 310, 311) are connected to the learning content server 100 having a test management unit through wired and wireless communication networks (LAN, Wi-Fi, LTE 4G/5G), drive test programs, and AI face recognition/motion recognition/ Using sound recognition technology, the front camera (C) of the test taker's terminal is used to perform facial recognition/motion recognition of the test taker's face, and auditory misconduct is monitored through sound recognition through the microphone and voice recognition module of the test taker's terminal. The artificial intelligence facial recognition module of the front face photograph of the candidate taken by the front camera C of the terminal, and the face outline constituting the feature points of the face by recognizing the facial behavior pattern taken by the front camera C of the candidate terminal Equipped with an eye 2 / nose / ear 2 5-point scale cheating prevention module, a voice recognition module that detects auditory cheating through voice recognition of the test taker, and interlocks with the learning content server 100 equipped with a test management unit A recording/recording program is installed in the test taker terminal.

During an online test or UBT test, the proctor terminal receives face photos and cheating information from a plurality of test taker terminals through the learning content server 100 having a test management unit, checks them, and learns content server having a test management unit. Through (100), an anti-cheating alarm or message is transmitted to the corresponding test taker terminal.

The AI-based face contour recognition technology used in the face recognition module of the user terminal uses a posenet algorithm (machine learning model which allows for real-time face pose estimation).

In the embodiment, during online learning/online exam or UBT exam, the user terminal uses a tablet PC.

The test taker terminals 300, 310, and 311 are equipped with a facial contour line of the learner's face image and a face recognition module with a 5-point scale for eye2/nose/ear2 facial feature points.

In addition, the test taker terminal further includes a handwriting recognition unit of the test taker terminal having a display and a touch sensor for recognizing and converting handwritten text of a stylus pen into text for subjective test questions.

The candidate terminals 300, 310, and 311 use the posenet algorithm as the AI-based facial contour recognition technology used in the face recognition module during online learning/online exam.

During the online test or the UBT test, the eye2/nose/ear2 face feature point 5-point scale face recognition module detects the ROI of the face image captured in real time by the front camera of the test taker terminal and detects the apex of the nose. Based on the horizontal x vertical standard size, a face object is extracted using a face recognition algorithm for a standard size frontal face photo generated through size correction/rotation/angle correction, and facial behavior patterns are recognized to determine the size of the test taker's face. The outline and facial feature points of Eye 2/Nose/Ear 2 are extracted, and through feature extraction and classification, the facial feature points of Eye 2/Nose/Ear 2 are connected to the left/right ear and the central point (pupil) of the left/right eye, respectively. Calculate the Euclidean distance (d) and similarity, compare with the photo in the face photo DB of the online test or UBT test server to check whether it is a proxy test, and in the online test or UBT test, 3 points on the eyes/nose are positive Detects head movement to the right/left according to the proximity to the two end ears, detects abnormal facial behavior patterns related to cheating, and when facial contours are not recognized during face recognition, strays from the test screen by more than a certain angle In this case (when the distance between the eye and the ear or the distance between the nose and the ear in the corresponding direction exceeds a certain standard value depending on how close the 3 eye/nose points are to the 2 ear points at both ends) The distance between the right eye and the right ear and the distance between the left eye and the left ear change according to the angle at which the front face is turned left and right by recognizing the facial behavior pattern. is transmitted to the learning content server 100 having the test management unit, and after checking by the supervisor terminal 200 linked to the learning content server 100 having the test management unit, through the learning content server 100 having the test management unit. A warning message or an alarm is generated to the corresponding test taker terminal.

(Example)

TensorFlow was used for pose estimation of a human face on Android on a tablet PC using the Posenet algorithm model. For reference, the Posenet algorithm is a vision model that estimates a person's pose from an image or video by detecting the position of a person's face, body, arms and legs, and major body parts for action recognition. At the time of face recognition, TensorFlow Lite uses the front camera of the test taker's terminal to identify the facial contour and eye 2/nose/ear 2 facial feature points in real time during face recognition, and creates a facial contour pose estimation model for the face. implemented.

The posenet algorithm used in the artificial intelligence face recognition module uses a five-point scale anti-fraud module of leftEye, rightEye, nose, leftEar, and rightEar. implemented,

After finding the x,y coordinates of leftEar and the x,y coordinates of rightEar, you can find the diameter of a circle based on these and create an angel ring with ellipse( ). The diameter can be easily obtained with dist( ) in p5js without having to use the Pythagorean theorem.

<Connecting ml5js and Webcam>

<script src="https://unpkg.com/ml5@0.3.1/dist/ml5.min.js"></script>

let video;

let poseNet;

function setup() {

createCanvas(400, 400);

video = createCapture(VIDEO);

// Used to prevent the video from appearing twice

video.hide( );

poseNet = ml5.poseNet(video, modelLoaded);

poseNet.on('pose', gotPoses);

console.log(ml5);

}

// Pose-related callbacks are entered

function gotPoses(poses) {

}

// Used to check if loading was successful

function modelLoaded() {

console.log('Model Loaded');

}

function draw() {

// Draw the webcam at the 0,0 position.

image(video,0,0);

}

The face region is extracted in real time from the face image of the front camera of the test taker device, and the standard size is adjusted to the standard size of the front face photo through size correction/rotation/angle correction based on the vertex of the nose, and the standard size pre-stored in the test server Comparison with learning data of face photo DB -> Learning model of face photo -> Extract and classify face outline and eye 2/nose/ear 2 feature points, face outline of face recognition data, eye 2/nose/ear 2 Ear 2 performs group clustering (center of cluster k-means algorithm) density estimation of facial feature points, and calculates distance (Euclidean distance) and similarity of facial feature points to online test or UBT test server's face photo DB By comparing the distance (d) and similarity of the facial feature points of the standard size front face photograph of the candidate stored in the standard size, the identity of the candidate photograph is checked to prevent proxy testing, and visual irregularities during the online test or UBT test. judge the action.

For example, if the direction of the face photographed by the front camera of the test taker terminal (tablet PC) is changed or the test taker turns the face at a certain angle, the eyes 2/nose/ear 2 and the outline of the face picture are changed. If the facial feature points are not recognized or the angle toward the learning content or test program changes within the effective distance between the user's face and the smartphone (20 to 30 cm), attentional learning is not performed or it is recognized as cheating during the test. and transmits the cheating detection result to the learning content server 100 having an online test or UBT test management unit, transmits it to the supervisor terminal 200, and after the supervisor confirms it, the learning content server 100 having the test management unit Through this, an alarm is generated or a warning message is transmitted to the corresponding candidate terminal so that the candidate is prevented from cheating.

In addition, [Text A] As soon as the warning threshold set in the predefined test taker program with reference to data becomes 0, a warning message is displayed to the test taker without server connection and can be transmitted back to the supervisor/cheating detection server/other test management server, Or, after the test is over, the raw data is sent to the cheating detection server/other test management server for post-processing.

In addition, the candidate terminal includes a face recognition module, an eye 2/nose/ear 2 facial feature point 5-point scale cheating prevention module, and the cheating prevention module recognizes the candidate's speech to prevent the candidate's visual cheating. Further comprising a voice recognition module,

During online learning, online exam, or UBT exam, the voice recognition module recognizes the speech of the candidate related to the cheating input through the microphone of the candidate terminal, and immediately recognizes the voice and the learning content server 100 having a test management unit. After the supervisor checks the cheating notification, an alarm is generated or a warning message is sent to the candidate terminal through the learning content server, and cheating is prevented. Alternatively, the test taker program itself directly displays an alarm or warning message to the corresponding learner / test taker terminal without a server command by referring to the threshold of the predefined distance of the standard size frontal face photograph of the eye-to-ear distance and the nose-to-ear distance, It transmits cheating information to the test server and the supervisor's terminal.

11 is a flowchart showing a tablet PC, smartphone, and PC-based attention learning method using facial contour recognition artificial intelligence in web browser-based non-face-to-face learning according to the present invention.

In the web browser-based non-face-to-face learning of the present invention, the tablet PC, smartphone, and PC-based attentional learning method using facial contour recognition artificial intelligence (a) registers learner information and standard-sized frontal face photos as a learning content server storing; (b) issuing, by the learning content server, a QR code corresponding to user information and a standard size frontal face picture; (c) In a user terminal equipped with an educational content viewer program and a face recognition module, the learning content server receives the recognition result of the front face photo of the camera using the front face recognition algorithm of the front camera image and corrects the size of the standard size. / Creating a standard size frontal face picture through rotation / angle correction and comparing the learner information pre-registered in the database of the server with the standard size frontal face picture and the facial feature points to confirm whether the learner is himself or not; (d) providing learning content from the learning content server to a user terminal of a pre-registered member; and (e) the learning content server analyzes the online learner's face image detected by the camera in each learner's user terminal, and when the gaze looking at the screen of the user terminal deviates from a certain angle or the learner's voice in the learner's user terminal In case of recognizing, generating a warning message or an alarm to the user terminal of the learner so that attention is learned,

In the method, in non-face-to-face online learning, the educational content viewer has a VOD media player, a chat screen, and an online learning memo space, and the instructor's terminal provides a plurality of 1:N methods through the learning content server 100. Further comprising transmitting and receiving question and response chatting data to learner terminals,

The user terminal is a VOD media player for playing non-face-to-face lecture learning content; A chat module that transmits and receives chat data between the instructor and the learner; A face recognition module for recognizing a facial behavior pattern of the front camera image of the user terminal and recognizing face outlines and face feature points on a 5-point scale of eyes 2/nose/ear 2; and when it is more than a certain angle away from the screen (depending on whether the 3 eye/nose points get closer to the 2 ears at both ends, when the distance in the corresponding direction of the eye-ear distance or nose-ear distance exceeds a certain standard value) or the learner's voice includes a learner's alarm generating unit that generates an alarm or outputs a warning message to the corresponding user terminal so that when it recognizes, the user terminal is a voice recognition module using STT technology that recognizes the learner's voice from a microphone Including more,

The facial recognition module uses the posenet algorithm as an AI-based facial contour recognition technology.

In non-face-to-face online learning, regardless of time and place, during online learning, the instructor provides learning content and underlines or texts the online learning memo space (shared workspace) of the educational content viewer. Sending/receiving files with report/learning files attached to the instructor's terminal (supervisor's terminal) and multiple learner terminals according to the instructor's floor control through drawings, photos, and voice recordings, and sending/receiving notes. provide,

In the case of non-face-to-face lectures, the instructor's terminal (supervisor terminal) is a plurality of learner terminals in a 1:N manner through the learning content server 100. Writing / graphic writing / memo in the shared workspace of the educational content viewer writing is displayed.

The learning content server 100 provides a function of recognizing a QR code as a recognition code when using a smartphone.

The user terminal uses any one of a tablet PC, a smartphone, and a PC having a front camera, and the educational content viewer downloaded from the learning content server is installed, and the educational content viewer chats with a VOD media player. screen and space for online learning notes.

The user terminal

A VOD media player that plays non-face-to-face lecture learning content;

A chat module that transmits and receives chat data between the instructor and the learner; and

a face recognition module for recognizing a facial behavior pattern of the front camera image of the user terminal and recognizing face outlines and feature points of the eye2/nose/ear2 face;

If the screen of the user device is out of a certain angle or more (depending on whether the 3 eye/nose points get closer to the 2 ears at both ends, if the distance in the corresponding direction of the eye-to-ear distance or nose-to-ear distance exceeds a certain standard value), an alarm is triggered. Include a learner's alarm generating unit that generates or outputs a warning message.

The facial recognition module uses the posenet algorithm as an AI-based facial contour recognition technology.

The face recognition module extracts a face object using a frontal face recognition algorithm for a face photo of a learner of a standard size corrected for size/rotation/angle during non-face-to-face online learning, recognizes a facial action pattern, and recognizes the face of the learner. Euclidean distance (d) between the facial feature points of the eyes 2/nose/ear 2 and the central point (pupil) of the left/right ears and the left/right eyes respectively. and similarity are calculated, and compared with face specific points of the user's standard-sized frontal face photo of the face photo DB, whether or not the line of sight is deviating is checked, and whether the 3 eye/nose points get closer to the 2 ear points at both ends is determined. Detects head movement to the right/left along the line, detects face patterns related to gaze deviation, and detects facial contours when recognizing faces, or when the user's screen is deviating from the user's screen by more than a certain angle (3 points on eyes/nose at both ends Depending on how close the two ears are, if the eye-to-ear distance or the nose-to-ear distance in the corresponding direction exceeds a certain standard value) Recognizes the facial behavior pattern captured by the camera of the user terminal and adjusts the angle at which the front face is turned left and right Accordingly, since the distance between the right eye and the right ear and the distance between the left eye and the left ear change, an alarm is generated or a warning message is output to the corresponding user terminal.

The learning content server 100

WWW server;

a control unit controlling to provide online learning content to the user terminal;

a member registration unit that is connected to the control unit and stores and manages ID/Passwd by receiving registered member information;

A user authentication unit that is connected to the control unit and authenticates using a QR code/Passwd or ID/Passwd or personal certificate;

a QR code management unit connected to the control unit and issuing and managing a QR code corresponding to user information;

a learning content provider connected to the controller and providing the online learning content to the user terminal;

a chatting server connected to the control unit and transmitting and receiving chatting data between the instructor and the learner;

It is connected to the control unit, receives a picture taken by a camera of a candidate from a user terminal, generates a standard size front face picture through size correction/angle correction/rotation to a standard size, and user information and front face previously stored in a server database. a supervisor confirmation unit that compares face photos and confirms eligibility for online learning by a supervisor checking the supervisor's terminal;

a face recognition unit connected to the control unit and recognizing the outline of the learner's face and facial feature points of eyes 2/nose/ear 2 using a face recognition module in the user terminal and receiving a face recognition result from the user terminal; and

Test information DB for storing test papers, written answers, and scoring results of test taker terminals; It includes a candidate DB and a face DB that store candidate information and a standard size frontal face picture.

Additionally, the learning content server 100 is connected to the controller 102 and further includes a chatting server 111 that transmits and receives chatting data between the instructor and the learner.

Additionally, the learning content server 100 is connected to the control unit 102 and further includes a send/receive note providing unit for transmitting/receiving note data to which the instructor and the learner can attach a report file.

Additionally, the learning content server 100 is connected to the controller 102 and further includes an FTP server for transmitting and receiving learning data files.

In non-face-to-face online learning, the instructor's terminal (supervisor terminal) transmits and receives question-and-answer chatting data to a plurality of learner terminals in a 1:N manner through the learning content server 100.

In non-face-to-face online learning, regardless of time and place, during online learning through the learning content server 100, the instructor provides learning content and underlines or texts the online learning memo space (shared workspace) of the educational content viewer. Sending/receiving files with report/learning files attached to the instructor's terminal (supervisor's terminal) and multiple learner terminals according to the instructor's floor control through drawings, photos, and voice recordings, and sending/receiving notes. provide,

In non-face-to-face online learning, during non-face-to-face lectures, the instructor's terminal (supervisor terminal) is a shared workspace of the educational content viewer with multiple learner terminals simultaneously in a 1:N manner through the learning content server 100 Text/Graphic handwriting/Memo writing are displayed on .

In addition, during the online exam or UBT exam, the candidate information, the front face photo of the candidate's field, and the supervisor information are stored and managed in the database of the learning content server equipped with the test management unit, and the schedule is tested during the online exam or UBT exam. During the time, after storing the test paper writing answers in each test taker terminal, at the end of the test, the test taker terminal receives and stores the test taker terminal to the learning content server 100 having a test management unit, and provides the scoring results of the test takers' test paper preparation answers to the test taker terminal Include more steps.

In addition, the online and UBT tests include a 2nd/3rd/4th/5th multiple-choice multiple-choice test and a subjective test in which the handwriting of a stylus pen is recognized and converted into characters using the handwriting recognition unit of the test taker terminal to recognize handwritten characters. include

In addition, the above method stores the motion of the user's face in the camera image of the user terminal in frame units as in [Text A] and subtracts a defined frame or time or number of times, and then sends a warning message to the learner/applicant terminal at the time when it becomes 0. After displaying/exposing or sending a supervisory device or cheating detection image to the server, initializing the number of warning triggers to start deduction again from the next movement.

[Text A] - Facial behavior pattern information stored on the tablet PC of the test taker terminal (Ex. Face pattern data of Seoul National University dental school learners): information such as date, time, movement angle/distance, and phase (left and right) It is stored every frame or hourly, and based on this, an alarm or warning message is output to the user terminal, and supervisor confirmation and candidate face recognition misconduct detection - The corresponding RAW data is transmitted to the learning content server system for post-processing.

200519094106,0.02,R

200519094111,0.03,R

200519094120,0.42,R

200519094148,0.01,R

200519094150,0.21,R

200519094155,0.22,R

200519094158,0.09,R

200519094209,0.11,L

200519094214,0.09,R

200519094216,0.10,R

200519094219,0.09,R

200519094224,0.01,L

In the web browser-based non-face-to-face learning system and method of the present invention, an attention-focused learning system and method using artificial intelligence in non-face-to-face online learning includes an educational content viewer of a user terminal equipped with a media playback unit and a facial contour recognition module, and a learning content server. 5-point scale face contour recognition for facial contours and eye 2/nose/ear 2 facial feature points to direct attention to the non-face-to-face lecture learning content focused on the front camera of the user terminal linked to the (ubcloud server) When the gaze is deviated at a certain angle through, an alarm/warning message is generated to the user terminal using sound and messaging technology, and attention-focused learning/testing using web browser-based ubcloud artificial intelligence is provided in non-face-to-face online learning. .

Embodiments according to the present invention are implemented in the form of program instructions that can be executed through various computer means, and can be recorded in a computer readable recording medium. The computer readable recording medium may include program instructions, data files, and data structures alone or in combination. Computer-readable recording media include server storage, hard disks, magnetic media such as floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and floptical disks. A hardware device configured to store and execute program instructions in storage media such as magneto-optical media, ROM, RAM, flash memory, etc. may be included. Examples of program instructions may include those produced by a compiler, machine language codes as well as high-level language codes that can be executed by a computer using an interpreter. The hardware device may be configured to operate as one or more software modules to perform the operations of the present invention.

As described above, the method of the present invention is implemented as a program and can be read using computer software on a recording medium (CD-ROM, RAM, ROM, memory card, hard disk, magneto-optical disk, storage device, etc.) ) can be stored in

Although described with reference to specific embodiments of the present invention, the present invention is not limited to the same configuration and operation as the specific embodiments to illustrate the technical idea as described above, and within the limit that does not deviate from the technical spirit and scope of the present invention It can be implemented with various modifications, and the scope of the present invention should be determined by the claims described later.

100: learning content server 101: WWW server
102: control unit 103: member registration unit
104: user authentication unit 105: learning content providing unit
106: QR code providing unit 107: face recognition unit (facial contour recognition)
109: test management department 111: chat server
300, 310, 311: user terminal

Claims (12)

A learning content server that provides non-face-to-face online lecture learning content; and
Connect to the learning content server and run the educational content viewer program. In non-face-to-face online learning, AI facial recognition/motion recognition sound reproduction technology of the front camera image data of the user terminal is used to educate learning content with artificial intelligence attention. Including a user terminal with a content viewer installed,
The user terminal is a VOD media player for playing non-face-to-face lecture learning content; A chat module that transmits and receives chat data between the instructor and the learner; a face recognition module recognizing a facial behavior pattern of the front camera image of the user terminal and recognizing facial outlines and 5-point scales of facial feature points of eyes 2/nose/ear 2; and when it is more than a certain angle away from the screen (depending on whether the 3 eye/nose points get closer to the 2 ears at both ends, when the distance in the corresponding direction of the eye-ear distance or nose-ear distance exceeds a certain standard value) or the learner's voice includes a learner's alarm generating unit that generates an alarm or outputs a warning message to the learner's user terminal so that attention is learned when recognizing , and the user terminal further includes a voice recognition module for recognizing the learner's voice,
The facial recognition module uses the posenet algorithm as an AI-based facial contour recognition technology,
The educational content viewer has a VOD media player, a chat screen, and an online learning memo space,
In non-face-to-face online learning, the instructor's terminal transmits and receives question-and-answer chatting data to a plurality of learner terminals in a 1:N manner through the learning content server. According to claim 1,
The learning content server provides a function of recognizing a QR code as a recognition code when using a smartphone, an attentional learning system in web browser-based non-face-to-face learning. According to claim 1,
The user terminal uses any one of a tablet PC, a smartphone, and a PC having a front camera, and focuses attention in web browser-based non-face-to-face learning in which the educational content viewer downloaded from the learning content server is installed. learning system. According to claim 1,
The face recognition module extracts a face object using a face recognition algorithm for a front face photo of a learner of a standard size that has been resized/rotated/angle corrected during non-face-to-face online learning, recognizes a facial action pattern, and recognizes the learner's face. The contour line and facial feature points of Eye2/Nose/Ear2 are extracted, and the Euclidean distance between the facial feature points of Eye2/Nose/Ear2 and the center point (pupil) of the left/right ear and the left/right eye, respectively ( d) and similarity are calculated, and compared with specific points of the user's face photo in the face photo DB, it is checked whether the line of sight is deviating, and depending on whether the 3 eye/nose points are closer to the 2 ears at both ends, the right/left/right/nose points are calculated. Detects head movement to the left, detects face patterns related to gaze deviation, and detects facial contours when recognizing faces, or when it is out of the screen by more than a certain angle (3 points of eyes/nose are closer to 2 points of both ears If the eye-to-ear distance or nose-to-ear distance in the corresponding direction exceeds a certain standard value), the facial behavior pattern captured by the camera of the user terminal is recognized and the right eye and right ear are separated according to the angle at which the front face is turned left and right. An attentional learning system in web browser-based non-face-to-face learning that generates an alarm or outputs a warning message to a corresponding user terminal because the distance and the distance between the left eye and the left ear are different. According to claim 1,
The learning content server
WWW server;
a control unit controlling to provide online learning content to the user terminal;
a member registration unit that is connected to the control unit and stores and manages ID/Passwd by receiving registered member information;
A user authentication unit that is connected to the control unit and authenticates using a QR code/Passwd or ID/Passwd or personal certificate;
a learning content provider connected to the controller and providing the online learning content to the user terminal;
a QR code management unit connected to the control unit and issuing and managing a QR code corresponding to user information;
a chatting server connected to the control unit and transmitting and receiving chatting data between the instructor and the learner;
It is connected to the control unit, receives a photo taken by a camera of a test taker from a user terminal, converts it into a frontal face photo through size correction/rotation/angle correction in a standard size, and compares user information previously stored in a database of a server with a frontal face photo. a supervisor verification unit for confirming eligibility for online learning by checking the supervisor at the supervisor's terminal;
a face recognition unit connected to the control unit, recognizing the outline of the learner's face and feature points of the eyes 2/nose/ear 2 using the face recognition module of the user terminal, and receiving a face recognition result from the user terminal; and
Test information DB for storing test papers, written answers, and scoring results of test taker terminals; a candidate DB and a face DB that store candidate information and a standard size frontal face picture;
Attentional learning system in non-face-to-face learning based on web browsers. According to claim 5,
It provides the test program (App) and test papers to the candidate terminal and the supervisor terminal, manages the candidate information, the candidate's on-site face picture, and supervisor information, and sends the test paper to each candidate terminal within a certain test time during the online test or UBT test. After saving the written answer, it is transmitted to the learning content server having the test management unit at the end of the test, and further includes a test management unit that stores and manages test takers' test paper written answers, scoring results, supervisor information, and test taker status information. Attentional learning system in non-face-to-face learning. (a) receiving and storing learner information and a standard-sized frontal face photo from a user terminal in a learning content server;
(b) issuing, by the learning content server, a QR code corresponding to user information and a front face picture;
(c) In a user terminal equipped with an educational content viewer program and a face recognition module, the recognition result of the front face photo of the camera is received by the learning content server using the front face recognition algorithm of the front camera image, and the previously registered learner information and Checking whether the learner is himself by comparing a standard size frontal face photo with facial feature points;
(d) providing learning content from the learning content server to a user terminal of a pre-registered member; and
(e) The learning content server analyzes the online learner's face image detected by the camera at each learner's user terminal so that when the gaze is off at a certain angle or when the learner's user terminal recognizes the learner's voice, attention is learned. Generating a warning message or alarm to the user terminal of the learner,
In non-face-to-face online learning, the educational content viewer has a VOD media player, a chatting screen, and an online learning memo space, and the instructor's terminal has question and answer chatting with a plurality of learner terminals in a 1:N manner through the learning content server. Further comprising the step of transmitting and receiving data,
The user terminal is a VOD media player for playing non-face-to-face lecture learning content; A chat module that transmits and receives chat data between the instructor and the learner; A face recognition module for recognizing a facial behavior pattern of the front camera image of the user terminal and recognizing face outlines and face feature points on a 5-point scale of eyes 2/nose/ear 2; and when it is more than a certain angle away from the screen (depending on whether the 3 eye/nose points get closer to the 2 ears at both ends, when the distance in the corresponding direction of the eye-ear distance or nose-ear distance exceeds a certain standard value) or the learner's voice includes a learner's alarm generating unit that generates an alarm or outputs a warning message to a corresponding user terminal so that attention is learned when recognizing , and the user terminal further includes a voice recognition module that recognizes the learner's voice,
The face recognition module is an attentional learning method in web browser-based non-face-to-face learning using the posenet algorithm as an AI-based facial contour recognition technology. According to claim 7,
The learning content server further comprises providing a function of recognizing a QR code as a recognition code when using a smartphone. According to claim 7,
The user terminal uses any one of a tablet PC, a smartphone, and a PC having a front camera, and focuses attention in web browser-based non-face-to-face learning in which the educational content viewer downloaded from the learning content server is installed. learning method. According to claim 7,
The face recognition module extracts a face object using a frontal face recognition algorithm for a face photo of a learner of a standard size corrected for size/rotation/angle during non-face-to-face online learning, recognizes a facial action pattern, and recognizes the face of the learner. The contour line and facial feature points of Eye2/Nose/Ear2 are extracted, and the Euclidean distance between the facial feature points of Eye2/Nose/Ear2 and the center point (pupil) of the left/right ear and the left/right eye, respectively ( d) and similarity are calculated, and compared with specific points of the user's frontal face photo in the face photo DB, whether or not the line of sight is displaced is checked, and according to whether the 3 eye/nose points are closer to the 2 ears at both ends, the right / Detects head movement to the left, detects face patterns related to gaze deviation, and detects facial contours when recognizing faces, or when it is out of a certain angle or more from the screen (3 points of eyes/nose are close to 2 points of ears at both ends) If the eye-to-ear distance or the nose-to-ear distance in the corresponding direction exceeds a certain standard value) Recognizes the facial behavior pattern captured by the camera of the user terminal and detects the right eye and right eye according to the angle at which the front face is turned left and right An attentional learning method in web browser-based non-face-to-face learning that generates an alarm or outputs a warning message at the user terminal because the distance between the ear and the distance between the left eye and the left ear is different. According to claim 7,
The learning content server
WWW server;
a control unit controlling to provide online learning content to the user terminal;
a member registration unit that is connected to the control unit and stores and manages ID/Passwd by receiving registered member information;
A user authentication unit that is connected to the control unit and authenticates using a QR code/Passwd or ID/Passwd or personal certificate;
a learning content provider connected to the controller and providing the online learning content to the user terminal;
a QR code management unit connected to the control unit and issuing and managing a QR code corresponding to user information;
It is connected to the control unit, receives a photo taken by a camera of a test taker from a user terminal, converts it into a frontal face photo through size correction/rotation/angle correction in a standard size, and compares user information previously stored in a database of a server with a frontal face photo. a supervisor verification unit for confirming eligibility for online learning by checking the supervisor at the supervisor's terminal;
a face recognition unit that is connected to the control unit and recognizes the outline of the learner's face and eye2/nose/ear2 feature points using a face recognition module in the user terminal and receives a face recognition result from the user terminal; and
Test information DB for storing test papers, written answers, and scoring results of test taker terminals; a candidate DB and a face DB that store candidate information and a standard size frontal face picture;
Attentional learning method in non-face-to-face learning based on web browsers. According to claim 7,
The learning content server stores and manages the candidate information, the candidate's on-site face picture, and the supervisor information in the database of the test server, and stores the test paper preparation answer in each candidate terminal during a certain test time during the online test or UBT test. Concentration of attention in web browser-based non-face-to-face learning, further comprising receiving and storing the test taker terminal to the learning content server having a test management unit at the end of the test, and providing the scoring results of the test taker's answers to the test taker terminal to the test taker terminal. learning method.

Images