KR20220126051A - System for providing real-time streaming based non face-to-face learning service - Google Patents

Abstract

The present invention provides a real-time streaming-based non-face-to-face education service providing system. The system comprises: a professor terminal that receives voice input through a microphone and outputs a web browser, a blackboard, a clock, and a voting window on a screen using at least one plug-in, collects input writing data to stream the input writing data in real time, and outputs a class atmosphere measured by a sentiment analysis model; at least one student terminal that receives real-time streaming of the screen and voice transmitted from the professor terminal and transmits images taken in opposite directions; and an education service providing server that comprises a storage unit that modularize a web browser capable of transmitting and receiving interactive audio and video between the professor terminal and at least one student terminal and at least one plug-in and forms a database, a count unit that starts counting to check a preset class time when class starts at the professor terminal, a streaming unit that real-time streams the screen and voice input from the professor terminal to at least one student terminal, and an analysis unit that inputs the video transmitted from at least one student terminal into a pre-stored sentiment analysis model and transmits and records the output class atmosphere to the professor terminal. Accordingly, the present invention can minimize time lag or delay in conversation.

Description

Real-time streaming-based non-face-to-face education service provision system {SYSTEM FOR PROVIDING REAL-TIME STREAMING BASED NON FACE-TO-FACE LEARNING SERVICE}

The present invention relates to a real-time streaming-based non-face-to-face education service providing system, and provides a web browser-based real-time streaming platform capable of providing various convenient functions and low delay for non-face-to-face classes.

The type and method of distance education can be divided into three types: a real-time interactive class type, a content utilization-oriented type, and a task performance-oriented type. Looking at the class statistics of each educational institution that has switched to non-face-to-face classes due to the recent Corona (COVID-19) situation, real-time video class type, lecture content type using the Learning Management System, and use of email or SNS It was found that there were many in the order of one task presentation type. LMS users prefer a system that can be reconfigured and usefully utilized rather than a large number of functions. Considering that, we want a system that is simple yet interactive and realistic. The factors that should be considered the most in online classes are interaction, learning content, class conditions, and evaluation. They provide information and feedback, and provide immediate responses to student requests for support.

At this time, a platform was researched and developed to divide the online class screen or diversify the blackboard for interaction and to minimize the delay due to traffic. Published on August 29, 2008) and Korean Patent Registration No. 10-2177691 (notice on November 11, 2020), output an online lecture window group, select and move at least one online lecture window, A plurality of lecture screens and electronic blackboard screens in online lectures provided in real time by outputting to the distribution browser area to output the target target slot corresponding to the distribution selection signal among the lecture windows in which the screen personalization arrangement is performed on the lecture window arrangement browser area and a configuration to allow mutual sharing of screen information written by students, and real-time measurement of traffic transmitted to a plurality of terminals. Each configuration is disclosed for adjusting the traffic capacity of the frames per second of the video transmitted to the .

However, among the above configurations, even in the former case, the electronic blackboard or screen separation is provided, but it lacks a sense of reality such as writing on a blackboard using a plurality of colors or highlights. As a tool developed for real-time meetings, it is consistent with the point that it lacks additional functions and convenience other than the sharing function of video and audio. In addition, when one teacher has to grasp the class attitude of dozens of students, and people are arranged on a divided screen, the gallery-type screen causes the central vision of the brain to recognize too many people at once, thereby accumulating fatigue. . If this split of attention lasts for a long time, you feel a sense of embarrassment that you can't do anything, and your brain is over-focused on finding non-verbal cues it can't find and is overwhelmed by unfamiliar excessive stimuli. Even with the latter configuration, the human brain can detect up to a fraction of a second, which is a fraction of a second, so if the screen stutters or slows down even a little, the brain panics and finds the cause, and burns with frequent jet lag in conversation In a meeting or class, the brain is constantly tense and reacting, so it is easy to feel fatigued. Accordingly, research and development of a platform capable of minimizing delay or jet lag to reduce brain fatigue while increasing various functions and convenience for education rather than a tool for meetings is required.

An embodiment of the present invention enables to minimize the time difference or delay of conversation through WebRTC while conducting a class through real-time streaming, and considering that one teacher cannot see all the class attitudes of dozens of students Using the emotion analysis model to measure the current class atmosphere and to alert inactive users through warnings. In this case, it is automatically detected and immediately notified to the professor's terminal, so that in Korea, where the vertical sequence culture is underlying, students cannot point out to the instructor, so it is possible to zero , a real-time streaming-based non-face-to-face education service provision method can be provided. However, the technical task to be achieved by the present embodiment is not limited to the above-described technical task, and other technical tasks may exist.

As a technical means for achieving the above-described technical problem, an embodiment of the present invention receives a voice input through a microphone and outputs a web browser, a blackboard, a clock and a voting window on the screen using at least one plug-in, At least one that collects and streams the written writing data in real time, receives a real-time stream of the screen and voice transmitted from the professor terminal and the professor terminal that outputs the class atmosphere measured by the emotional analysis model, and transmits the image taken in the opposite direction A web browser capable of transmitting and receiving two-way audio and video between the student terminal and the professor terminal and at least one student terminal, and at least one plug-in modularizing and database storage unit, checking the preset class time when starting a class from the professor terminal A count unit that starts counting to do so, a streaming unit that streams the screen and audio input from the professor terminal in real time to at least one student terminal, and the image transmitted from at least one student terminal is inputted into a pre-stored emotion analysis model and then output It includes an education service providing server including an analysis unit for transmitting and recording the class atmosphere to the professor terminal.

According to any one of the above-described problem solving means of the present invention, it is possible to minimize the time difference or delay of conversation through WebRTC while conducting a class in real-time streaming, and one teacher can view all the class attitudes of dozens of students. Considering that it cannot In case the screen is not transmitted, it is automatically detected and immediately notified to the professor's terminal. The situation can be zeroed out.

1 is a diagram for explaining a system for providing a real-time streaming-based non-face-to-face education service according to an embodiment of the present invention.
FIG. 2 is a block diagram illustrating an education service providing server included in the system of FIG. 1 .
3 and 4 are diagrams for explaining an embodiment in which a real-time streaming-based non-face-to-face education service according to an embodiment of the present invention is implemented.
5 is an operation flowchart illustrating a method of providing a real-time streaming-based non-face-to-face education service according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily carry out the present invention. However, the present invention may be embodied in several different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part is "connected" with another part, this includes not only the case of being "directly connected" but also the case of being "electrically connected" with another element interposed therebetween. . Also, when a part "includes" a certain component, it means that other components may be further included, rather than excluding other components, unless otherwise stated, and one or more other features However, it is to be understood that the existence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded in advance.

The terms "about," "substantially," and the like, to the extent used throughout the specification are used in or close to the numerical values when manufacturing and material tolerances inherent in the stated meaning are presented, and are intended to enhance the understanding of the present invention. To help, precise or absolute figures are used to prevent unfair use by unscrupulous infringers of the stated disclosure. As used throughout the specification of the present invention, the term "step of (to)" or "step of" does not mean "step for".

In this specification, a "part" includes a unit realized by hardware, a unit realized by software, and a unit realized using both. In addition, one unit may be implemented using two or more hardware, and two or more units may be implemented by one hardware. Meanwhile, '~ unit' is not limited to software or hardware, and '~ unit' may be configured to be in an addressable storage medium or may be configured to reproduce one or more processors. Thus, as an example, '~' denotes components such as software components, object-oriented software components, class components, and task components, and processes, functions, properties, and procedures. , subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays and variables. The functions provided in the components and '~ units' may be combined into a smaller number of components and '~ units' or further separated into additional components and '~ units'. In addition, components and '~ units' may be implemented to play one or more CPUs in a device or secure multimedia card.

In this specification, some of the operations or functions described as being performed by the terminal, apparatus, or device may be performed instead of in a server connected to the terminal, apparatus, or device. Similarly, some of the operations or functions described as being performed by the server may also be performed in a terminal, apparatus, or device connected to the server.

In this specification, some of the operations or functions described as mapping or matching with the terminal means mapping or matching the terminal's unique number or personal identification information, which is the identification data of the terminal. can be interpreted as

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings.

1 is a diagram for explaining a system for providing a real-time streaming-based non-face-to-face education service according to an embodiment of the present invention. Referring to FIG. 1 , the real-time streaming-based non-face-to-face education service providing system 1 may include at least one student terminal 100 , an education service providing server 300 , and at least one professor terminal 400 . . However, since the real-time streaming-based non-face-to-face education service providing system 1 of FIG. 1 is only an embodiment of the present invention, the present invention is not limitedly interpreted through FIG. 1 .

At this time, each component of FIG. 1 is generally connected through a network (Network, 200). For example, as shown in FIG. 1 , at least one student terminal 100 may be connected to the education service providing server 300 through the network 200 . In addition, the education service providing server 300 may be connected to at least one student terminal 100 and at least one professor terminal 400 through the network 200 . Also, the at least one teaching terminal 400 may be connected to the education service providing server 300 through the network 200 .

Here, the network refers to a connection structure in which information exchange is possible between each node, such as a plurality of terminals and servers, and an example of such a network includes a local area network (LAN), a wide area network (WAN: Wide Area Network), the Internet (WWW: World Wide Web), wired and wireless data communication networks, telephone networks, wired and wireless television networks, and the like. Examples of wireless data communication networks include 3G, 4G, 5G, 3rd Generation Partnership Project (3GPP), 5th Generation Partnership Project (5GPP), Long Term Evolution (LTE), World Interoperability for Microwave Access (WIMAX), Wi-Fi (Wi-Fi) , Internet, LAN (Local Area Network), Wireless LAN (Wireless Local Area Network), WAN (Wide Area Network), PAN (Personal Area Network), RF (Radio Frequency), Bluetooth (Bluetooth) network, NFC ( Near-Field Communication) networks, satellite broadcast networks, analog broadcast networks, Digital Multimedia Broadcasting (DMB) networks, and the like are included, but are not limited thereto.

In the following, the term at least one is defined as a term including the singular and the plural, and even if at least one term does not exist, each component may exist in the singular or plural, and may mean the singular or plural. it will be self-evident In addition, that each component is provided in singular or plural may be changed according to embodiments.

The at least one student terminal 100 may be a terminal of a student participating in a class using a real-time streaming-based non-face-to-face educational service related web page, an app page, a program, or an application. At this time, the student terminal 100 uses the platform according to an embodiment of the present invention for class, receives the screen and voice of the professor terminal 400 streaming in real time, and transmits chatting or participates in voting. can

Here, the at least one student terminal 100 may be implemented as a computer capable of accessing a remote server or terminal through a network. Here, the computer may include, for example, navigation, a laptop equipped with a web browser, a desktop, and a laptop. In this case, the at least one student terminal 100 may be implemented as a terminal capable of accessing a remote server or terminal through a network. At least one student terminal 100, for example, as a wireless communication device that guarantees portability and mobility, navigation, PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS(Personal Handyphone System), PDA(Personal Digital Assistant), IMT(International Mobile Telecommunication)-2000, CDMA(Code Division Multiple Access)-2000, W-CDMA(W-Code Division Multiple Access), Wibro(Wireless Broadband Internet) ) terminal, a smartphone, a smart pad, a tablet PC, etc. may include all kinds of handheld-based wireless communication devices.

The education service providing server 300 may be a server that provides a real-time streaming-based non-face-to-face education service web page, an app page, a program, or an application. In addition, the education service providing server 300 allows two-way audio and video to be transmitted and received between the student terminal 100 and the professor terminal 400 and modularizes at least one plug-in to each student terminal 100 and the professor terminal 400 . It can be a server that allows you to use the plugin you want in . In addition, the education service providing server 300 can stream the screen and audio input from the professor terminal 400 to the student terminal 100 in real time, and the image transmitted from the student terminal 100 is pre-stored as an emotional analysis model. It may be a server that outputs the class atmosphere after inputting it as .

Here, the education service providing server 300 may be implemented as a computer capable of accessing a remote server or terminal through a network. Here, the computer may include, for example, navigation, a laptop equipped with a web browser, a desktop, and a laptop.

The at least one professor terminal 400 may be a terminal of an instructor using a real-time streaming-based non-face-to-face education service related web page, an app page, a program, or an application. The professor terminal 400 may be a terminal that receives writing data by using at least one plug-in, shares a web page, outputs lecture materials, and receives and transmits voice utterances.

Here, the at least one teaching terminal 400 may be implemented as a computer capable of accessing a remote server or terminal through a network. Here, the computer may include, for example, navigation, a laptop equipped with a web browser, a desktop, and a laptop. In this case, the at least one teaching terminal 400 may be implemented as a terminal capable of accessing a remote server or terminal through a network. At least one teaching terminal 400, for example, as a wireless communication device that guarantees portability and mobility, navigation, PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS(Personal Handyphone System), PDA(Personal Digital Assistant), IMT(International Mobile Telecommunication)-2000, CDMA(Code Division Multiple Access)-2000, W-CDMA(W-Code Division Multiple Access), Wibro(Wireless Broadband Internet) ) terminal, a smartphone, a smart pad, a tablet PC, etc. may include all kinds of handheld-based wireless communication devices.

2 is a block diagram for explaining an education service providing server included in the system of FIG. 1, and FIGS. 3 and 4 are an embodiment in which a real-time streaming-based non-face-to-face education service is implemented according to an embodiment of the present invention. It is a drawing for explaining.

Referring to FIG. 2 , the education service providing server 300 includes a storage unit 310 , a count unit 320 , a streaming unit 330 , an analysis unit 340 , a delay prevention unit 350 , and an LMS unit 360 . ), a flip running unit 370 , a notification unit 380 , a readjustment unit 390 , and a concentration induction unit 391 may be included.

The education service providing server 300 according to an embodiment of the present invention or another server (not shown) operating in conjunction with at least one student terminal 100 and at least one professor terminal 400 is non-face-to-face based on real-time streaming. When transmitting an education service application, program, app page, web page, etc., at least one student terminal 100 and at least one professor terminal 400, a real-time streaming-based non-face-to-face education service application, program, app page, You can install or open web pages, etc. In addition, the service program may be driven in at least one student terminal 100 and at least one professor terminal 400 by using a script executed in a web browser. Here, the web browser is a program that enables the use of a web (WWW: World Wide Web) service, and refers to a program that receives and displays hypertext written in HTML (Hyper Text Mark-up Language), for example, Netscape. , Explorer, Chrome, and the like. In addition, the application means an application on the terminal, for example, includes an app (App) executed in a mobile terminal (smartphone).

Referring to FIG. 2 , the storage unit 310 may modularize a web browser capable of transmitting and receiving interactive audio and video between the professor terminal 400 and at least one student terminal 100 and at least one plug-in to form a database. The storage unit 310 may modularize at least one plug-in including a web browser, a blackboard, a clock, and a voting window. In this case, the blackboard plug-in is a plug-in that supports a collaborative blackboard capable of simultaneously taking notes in at least one student terminal 100 and the professor terminal 400, and at least one student terminal 100 or a professor terminal in the blackboard window. When writing is performed at 400 , the nickname or real name of at least one student terminal 100 or the professor terminal 400 may be output over a mouse cursor or output on a line or text to be written. The blackboard window may be divided by a Split function.

The counting unit 320 may start counting to check a preset class time when starting a class in the professor terminal 400 . In most cases, since the class time and time are fixed, attendance is checked according to the start time, and the class time is output as a timer from the start time. so that you can check whether

The streaming unit 330 may stream the screen and voice input from the professor terminal 400 to at least one student terminal 100 in real time. The professor terminal 400 receives a voice input through a microphone, outputs a web browser, a blackboard, a clock and a voting window on the screen using at least one plug-in, collects the input writing data and streams it in real time, and performs a sentiment analysis model The measured class atmosphere can be output. At least one student terminal 100 may receive a screen and a voice transmitted from the professor terminal 400 in real time, and may transmit an image taken in opposite directions. In this case, HTML5-based streaming can also be performed so that it can be streamed based on a web browser without installing a program or application.

Here, HTML5 is the latest standard of HTML (Hyoer Text Markup Language), which is a web document standard, and includes a function of executing pictures, videos, and music in the center of document creation. HTML5 eliminates the need to separately install programs such as plug-in-based Active-X and Flash, and most of the problems caused by this can be solved. At this time, the biggest difference between the conventional HTML standard and HTML5 is that semantic markup is supported and an additional API is provided. Semantic markup is a new element that can semantically structure web documents. For example, if a document is created with elements such as <head>, <section>, and <footer> so that the contents of the document can be divided into header, body, and footer, the contents of the document can be clearly understood structurally. In addition, HTML5 provides various APIs. The API provided by HTML5 allows you to add various functions to web applications, such as supporting 2D graphics or the ability to check the geographic location of a device. In addition, HTML5 is a markup language used to create web pages, and can express and provide HTML, which used to be displayed as hyperlinks and texts, to various applications such as multimedia.

HTML5 can be implemented including HTML, a markup language for designing web document structure, CSS for design expression, JavaScript for expressing interactive action, etc., and new tags are added to existing HTML tags. More can be added, and the structure to contain contents can be improved to even perform the role of a platform. In addition, although HTML5 is not a standard for the mobile web, it provides many functions necessary for the implementation of the mobile web among the various functions described above. 2D graphics, audio, and various web forms can enrich and diversify the UI (User Interface) of the mobile web. That is, it is possible to provide multimedia information or support a user's dynamic interaction even in a mobile web environment. In the mobile web environment, there is a limitation in transmitting large amounts of data due to the nature of the mobile environment, and an environment in which the Internet is not connected frequently occurs. Alternatively, you can store the DB locally and query it with an SQL statement.

The analysis unit 340 may input the image transmitted from the at least one student terminal 100 into a pre-stored emotion analysis model, and then transmit and record the output class atmosphere to the professor terminal 400 . The pre-stored emotion analysis model may be a model that measures the polarity and intensity of emotion to measure positive reaction, negative reaction, and energy. In the platform according to an embodiment of the present invention, only the face of the instructor is output in the case of a one-to-many class, so that the instructor cannot see the students. Even if all the students' faces are printed in a gallery format, it is not easy for the instructor to examine the students' faces one by one. Accordingly, the analysis unit 340 captures the student's face and collects it, determines whether the students are concentrating, whether the current seat is not vacant, whether they are sleeping, etc. This can be done to call attention to a student who is currently not concentrating, is absent, or is dozing off.

For facial emotion recognition, techniques such as LBP (Local Binary Pattern) and NMF (Non-Negative Matrix Factorization) may be used, and image classification may be performed using a Convolutional Neural Network (CNN). At this time, although VGGnet or Resnet structures show excellent classification accuracy, they are not suitable for real-time processing in an environment where the use of hardware resources is limited due to a large number of parameters and high computational complexity. Accordingly, in an embodiment of the present invention, a CNN structure made to enable real-time processing by reducing the amount of computation and the number of parameters while maintaining high facial emotion recognition accuracy may be used.

A lightweight CNN that reduces the amount of computation and the number of parameters uses a new hierarchical structure to improve model performance while reducing the number of parameters (Efficient Structure) and efficient computation to reduce the convolution operation that requires the largest amount of computation for training and inference. Techniques (Efficient Operation) and the like may be used. The Fire Module used in Squeezenet can be configured as a basic operation block, and the Fire Module operates as follows. For the input feature map, it passes through the Squeeze Layer that reduces the channel and the Expand Layer that increases the channel in turn, and performs the operation (Channel-Wise Concatenation) of stacking the feature map in units of channels. can The number of parameters and the amount of computation can be reduced when compared with a structure in which a convolutional filter with a large filter size is continuously used. In addition, in order to reduce the consumption of additional hardware resources, a structure can be created to remove the pooling layer and reduce the size of the feature map in the convolution filter having a stride of 2.

By adding the Bypass Connection used in the Resnet structure only to the feature map with a stride of 1, so that the network can learn the residual between the input and the output, it is located in the last layer of the CNN and is responsible for image classification. A large number of parameters can be reduced by replacing the fully connected layer, which is a layer that does not And to alleviate the overfitting problem and improve the accuracy of CNN, a BN (Batch Normalization) layer can be added between the convolution filter and the activation function. In this case, as the experimental dataset, the FERPLUS dataset supplemented with the FER2013 dataset may be used, and methods such as bilateral filtering, histogram smoothing, and CutMix may be further used for training data enhancement.

The delay prevention unit 350 may perform P2P image and data communication using Web Real-Time Communication (WebRTC). As for the protocol for streaming, various protocols such as RTMP, RTSP, MPEG Dash, and WebRTC exist and may be used. WebRTC is a protocol proposed by W3C that helps enable real-time communication such as voice and video without a separate plug-in. It does not require a separate plug-in and has the advantage of providing low latency. WebRTC is an open source project and has a mesh, SFU, and MCU structure. Like the P2P method, the mesh structure is a method in which each node connects one-to-one to exchange data. SFU and MCU have a structure in which multiple nodes communicate, and a central server exists. The SFU structure is a method of distributing and communicating data without processing in the central server. The MCU structure is a method of processing and communicating data in a central server. Since the communication method is different for each model, it is possible to find and utilize the appropriate application method for each model. In the case of real-time streaming, it is necessary to communicate with a large number of users through a central server and delay time is an important factor, so in an embodiment of the present invention, real-time streaming that can be accessed by two or more users can be used by utilizing the SFU method. Of course, other methods are not excluded.

The efficiency of the HTTP-based adaptive media streaming system can also be increased by using P2P communication based on WebRTC. This is called the Hive system and uses DASH among HTTP-based adaptive media streaming methods. In the Hive system, when a client needs a DASH segment, before requesting it to the CDN server, it requests the corresponding DASH segment from another peer who is watching the same content from the CDN server or HTTP cache. traffic can be significantly reduced. This method can significantly reduce the cost of the CDN operator without lowering the QoS (Quality of Service) when streaming media. In addition, compared to the general P2P method, in the case of a browser supporting the WebRTC standard, a standard API can be used, so the web program development cost using JavaScript is reduced, and there is no need to install a special plug-in or additional application. Therefore, user's accessibility and convenience are increased.

After the P2P connection is successfully completed, the connected peers inform each other of their media data information. When peers use this information to request specific media data, they request media data from a randomly selected peer among peers having the media data. If the data request is rejected or the predetermined time is exceeded, the request is sent to the CDN media server to receive the media data. In this process, if media data is requested from peers in a low-speed network environment, requests to the CDN server increase due to poor P2P connection, resulting in CDN cost reduction (the amount of data received through P2P/total amount of data received) ) may be lowered. Therefore, the client peer accumulates the media data delivery history of peers in the peer list and keeps it as a statistical value, and uses it when requesting media data through P2P communication to give preference to the peer expected to have the highest P2P success rate Algorithms for requesting media data can be used.

However, in order to maximize the success rate of segment acquisition from the client's point of view, there is a disadvantage of concentrating the load on such peers because it tends to continuously request segments from peers that communicate with them the best. When the load is concentrated on one peer, not only the resources of that peer are used excessively, but also when peer-to-peer communication is paid, the owner of the peer is dissatisfied because the owner of the peer bears the cost of P2P traffic on his own in order to reduce CDN cost. can have Therefore, it is preferable to distribute the burden to several peers at an appropriate level rather than placing the burden on one peer in this extreme way, and a new peer selection algorithm that distributes the burden to multiple peers while properly maintaining the CDN cost reduction level can be used. .

For example, instead of selecting the peer with the highest priority, a list including all the peers with the highest priority and the peers with a priority 1 and 2 steps lower than the priority is constructed, and from this One peer can be selected randomly. It can be configured to avoid as much as possible the case of receiving two segment requests in succession. Of course, if there is only one peer remaining in the list, send a segment request even if it is the peer that received the previous segment request. In general, the priority of peers maintained in a good P2P communication state for a certain period of time easily reaches the upper limit. This is to give an opportunity to select up to a peer with a priority of 3 through a random selection process.

The LMS unit 360 extracts a problem with an incorrect answer from solving a problem in class and a problem similar to the problem and transmits it to the user terminal, and provides a messenger function between at least one student terminal 100, multiple choice or You can automatically grade open-ended assignments and provide results. In this case, if the subjective task is an essay type, automatic essay scoring may be used. This is a problem in which the system automatically scores human-written essays, and in order to evaluate human-written essays, grammar, closeness and relevance of content must be considered. A large amount of essay data is required. However, since the amount of graded essay data is small and it is difficult to collect, the problem of data shortage can be solved through Data Augmentation, which increases the amount of data by applying slight modifications to the data.

At this time, in order to apply a transformation to the essay data, the data expansion is limited because care must be taken not to impair the flow of the content. . A general generative adversarial network creates a new sentence by predicting the next candidate word on a word-by-word basis. In addition, since general generative adversarial networks cannot generate essay data in the desired score range, CS-GAN is used to input score information to the generator and add a classifier so that the generator matches the score information. After calculating a reward for generating the essay, the user can generate the essay data of the desired score by inputting the score information to be obtained to the generator as an input.

The flipped learning unit 370 transmits the scoring result provided from the LMS unit 360 to the professor terminal 400 and provides at least one statistical data so that the professor terminal 400 can refer to the lecture plan. can do. Flipped learning does not mean reviewing after taking a class, but changing the course in reverse so that the instructor lectures on the parts that are difficult or difficult for students to understand. way to do it

The notification unit 380, when a voice or a screen is input from the professor terminal 400, but the voice or screen is not output from any one of the at least one student terminal 100 or the at least one student terminal 100, the professor terminal It is possible to automatically readjust the settings of 400 or transmit a message indicating that voice or screen is not output to the professor terminal 400 .

The readjustment unit 390 is not output when a voice or a screen is input from the professor terminal 400, but a voice or a screen is not output from any one of the at least one student terminals 100 or at least one of the student terminals 100 It is possible to automatically readjust the settings of the student terminal 100 that is not. Since it is not difficult to extract the setting value of the terminal and readjust it according to each model or specification, or according to the device or OS, it is not difficult to change the setting value at once if the setting value is saved in advance, so a detailed description is omitted.

The concentration inducing unit 391 may synthesize or delete a background image excluding a subject in the image as a preset background when collecting images captured by the at least one student terminal 100 . In this case, the background may be replaced by performing foreground separation and extracting only the subject. Most students take classes in their own rooms or study cafes, etc. If a dog or cat passes by or their room is dirty, they focus only on the area where the student's face appears rather than on the instructor, so their concentration is low. . Therefore, by separating the foreground and extracting only the student's face or body as a subject, and then deleting the background, you can focus on the instructor rather than your own figure, or you can read the student's facial expression using a character, but the student's face itself. may be set not to appear.

In addition, in the case of one-on-one instruction, voice signals may be used together. Of course, since a student's voice signal can be input to a one-to-many class or a discussion class, a method for recognizing emotions can be further used. In the case of voice signals, another data that can reveal human emotions, it is the most natural medium for communication between humans and contains information that contains the emotions of the speaker, such as verbal content, intonation size, and speed. Therefore, the most important problem in the speech emotion recognition system is to extract appropriate acoustic features, such as pitch and formant energy, from the user's speech signal and use them in parallel with an appropriate classification engine.

In this case, MelFrequency Cepstrum Coefficients (MFCC) can be used as a method for extracting acoustic features, but there is no clear link between the user's emotional state and the acoustic features extracted from the voice signal, so it is different from the facial emotion recognition method. The recognition rate is relatively low compared to the recognition method. Therefore, it is possible to increase the recognition rate by extracting appropriate acoustic features and reflecting them in the model. In general, people recognize the emotions of others using words and facial expressions, such as happiness, sadness, anger, and neutrality. Facial expressions are a representative example of non-verbal elements. In terms of human perception and perception, it is natural that when a computer affects human emotions, using voice signals and facial images at the same time can help in more accurate and natural emotion recognition. Therefore, it is possible to increase the recognition rate in emotion recognition by fusing the voice signal and the face image sequence using the Joint Fine-Tuning method.

To solve the problem of combining two inputs, three deep networks are available. The first model, based on images, is trained using a sequence of facial images to capture changes in facial expressions. The second model based on the image is trained to reflect the movement of the face using the face landmark, and the third model based on the voice signal is converted into acoustic features while synchronizing the image sequence and learned. In this case, RAVDESS (Ryerson Audio-Visual Database of Emotional Speech and Song) may be used as the database.

Hereinafter, an operation process according to the configuration of the education service providing server of FIG. 2 will be described in detail with reference to FIGS. 3 and 4 as an example. However, it will be apparent that the embodiment is only one of various embodiments of the present invention and is not limited thereto.

Referring to FIG. 3A , an embodiment of the present invention provides various plug-ins and allows users to use them customized in a way that they may or may not apply as desired, and a split function as shown in FIG. 3B provides a problem on one side. You can also use the online blackboard function on the page, and as shown in Figure 3c, by using the dynamic view with A4 size as the default, the enlargement or reduction of the screen of the instructor and the student during class is synchronized to see which part of the page the student is currently working on. Make sure you don't get lost in your search. Also, as shown in FIG. 3D, various factors such as report function, problem participation, correct answer rate, or participation rate are provided as a report using the AI lecture auxiliary module as shown in FIG. 3D, and problem solving and discussion type as shown in FIG. It can provide a platform for studying, checking assignments, reporting functions, and preparing class materials. A platform such as that of Figure 3g can be built and provides a solution for high-value education.

As shown in Fig. 4a, the student terminal 100 and the teacher terminal 400, after registering as a member and logging in, create a new blackboard to output notes or e-books as shown in Fig. 4b. As shown in Fig. 4c, the types of notes or An e-book may be selected, and the blackboard may be divided and used as shown in FIG. 4D . A small class lecture video may be conducted as shown in Fig. 4e, a large-scale lecture may be conducted as shown in Fig. 4f, a note may be added as shown in Fig. 4g, and a test paper or textbook as shown in Figs. 4h to 4j is placed on the right side You can write on the left side, or you can write directly on the test paper or textbook. The writing function may provide a function with a shortcut key as shown in FIG. 4K, select a textbook as shown in FIG. 4L, use a link as shown in FIG. 4M, or use a voting function or a clock as shown in FIG. 4N.

The matters not described with respect to the real-time streaming-based non-face-to-face education service providing method of FIGS. 2 to 4 are the same as those described for the real-time streaming-based non-face-to-face education service providing method through FIG. 1 or from the described content. Since it can be easily inferred, the following description will be omitted.

5 is a diagram illustrating a process in which data is transmitted/received between components included in the real-time streaming-based non-face-to-face education service providing system of FIG. 1 according to an embodiment of the present invention. Hereinafter, an example of a process in which data is transmitted/received between each component will be described with reference to FIG. 5, but the present application is not limited to such an embodiment, and the example shown in FIG. 5 according to the various embodiments described above will be described. It is apparent to those skilled in the art that the data transmission/reception process may be changed.

Referring to FIG. 5 , the education service providing server modularizes a web browser and at least one plug-in capable of transmitting and receiving interactive audio and video between a professor terminal and at least one student terminal (S5100).

Then, the education service providing server starts counting to check the preset class time when starting a class at the professor terminal (S5200), and streams the screen and voice input from the professor terminal to at least one student terminal in real time ( S5300).

In addition, the education service providing server, after inputting the image transmitted from at least one student terminal into a pre-stored emotion analysis model, transmits and records the output class atmosphere to the professor terminal (S5400).

The order between the above-described steps (S5100 to S5400) is only an example, and is not limited thereto. That is, the order between the above-described steps ( S5100 to S5400 ) may be mutually changed, and some of these steps may be simultaneously executed or deleted.

Matters not described with respect to the real-time streaming-based non-face-to-face education service providing method of FIG. 5 are the same as those described for the real-time streaming-based non-face-to-face education service providing method through FIGS. 1 to 4 or from the described content Since it can be easily inferred, the following description will be omitted.

The real-time streaming-based non-face-to-face education service providing method according to an embodiment described through FIG. 5 may also be implemented in the form of a recording medium including instructions executable by a computer, such as an application or program module executed by a computer. can Computer-readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. Also, computer-readable media may include all computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

The method for providing a real-time streaming-based non-face-to-face education service according to an embodiment of the present invention described above is performed by an application basically installed in a terminal (which may include a program included in a platform or operating system, etc. that is basically installed in the terminal). may be executed, and may be executed by an application (ie, a program) directly installed in the master terminal by a user through an application providing server such as an application store server, an application, or a web server related to the corresponding service. In this sense, the above-described real-time streaming-based non-face-to-face education service providing method according to an embodiment of the present invention is implemented as an application (ie, a program) installed by default in a terminal or directly installed by a user, and is transmitted to a computer such as a terminal. It may be recorded on a readable recording medium.

The foregoing description of the present invention is for illustration, and those of ordinary skill in the art to which the present invention pertains can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a distributed manner, and likewise components described as distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the following claims rather than the above detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

Claims (10)

Receives voice input through a microphone, outputs a web browser, blackboard, clock and voting window on the screen using at least one plug-in, collects the input writing data and streams it in real time, and outputs the class atmosphere measured by the emotional analysis model a professor terminal;
at least one student terminal that receives the screen and audio transmitted from the professor terminal in real time and transmits an image taken in opposite directions; and
A web browser capable of transmitting and receiving interactive audio and video between the professor terminal and at least one student terminal and a storage unit for modularizing and converting at least one plug-in into a database, counting to check a preset class time when starting a class from the professor terminal A count unit to start the , a streaming unit for real-time streaming the screen and audio input from the professor terminal to the at least one student terminal, and output after inputting the image transmitted from the at least one student terminal into a pre-stored emotion analysis model an education service providing server including an analysis unit that transmits and records the class atmosphere to the professor terminal;
A real-time streaming-based non-face-to-face education service providing system that includes.
The method of claim 1,
The education service providing server,
a delay preventing unit performing P2P image and data communication using WebRTC (Web Real-Time Communication);
Real-time streaming-based non-face-to-face education service providing system, characterized in that it further comprises.
The method of claim 1,
The pre-stored emotion analysis model is a real-time streaming-based non-face-to-face education service providing system, characterized in that it is a model that measures the polarity and intensity of emotion to measure positive reaction, negative reaction, and energy.
The method of claim 1,
The blackboard plug-in is a plug-in supporting a collaborative blackboard capable of simultaneously taking notes in the at least one student terminal and the professor terminal,
When writing is performed on the at least one student terminal or the professor terminal in the blackboard window, the nickname or real name of the at least one student terminal or the professor terminal is output over the mouse cursor or output on a line or text to be written A system for providing real-time streaming-based non-face-to-face education services.
5. The method of claim 4,
The blackboard window is a real-time streaming-based non-face-to-face education service providing system, characterized in that the split (Split) function.
The method of claim 1,
The education service providing server,
In class problem solving, a problem with an incorrect answer and a problem similar to the problem are extracted and transmitted to the user terminal, a messenger function between the at least one student terminal is provided, and multiple-choice or subjective-type assignments are automatically scored. an LMS unit providing results;
Real-time streaming-based non-face-to-face education service providing system, characterized in that it further comprises.
7. The method of claim 6,
a flipped learning unit that transmits the scoring result provided by the LMS unit to the teaching terminal and provides at least one statistical data so that the teaching terminal can refer to a lecture plan;
Real-time streaming-based non-face-to-face education service providing system, characterized in that it further comprises.
The method of claim 1,
The education service providing server,
When a voice or screen is input from the professor terminal, but no voice or screen is output from any one or at least one student terminal of the at least one student terminal, the setting of the professor terminal is automatically readjusted, or the setting of the professor terminal is automatically read or voice or a notification unit that transmits a message that the screen is not output;
Real-time streaming-based non-face-to-face education service providing system, characterized in that it further comprises.
The method of claim 1,
The education service providing server,
a readjustment unit for automatically re-adjusting settings of a student terminal that is not output when a voice or a screen is input from the professor terminal, but no audio or a screen is output from any one of the at least one student terminal or at least one student terminal;
Real-time streaming-based non-face-to-face education service providing system, characterized in that it further comprises.
The method of claim 1,
The education service providing server,
a concentration inducing unit for synthesizing or deleting a background image excluding a subject in the image into a preset background when collecting the images captured by the at least one student terminal;
Real-time streaming-based non-face-to-face education service providing system, characterized in that it further comprises.

Images