KR102446970B1 - An electronic podium system that performs AI control technology in cloud environment and includes a self-sterilization function - Google Patents

Abstract

The present invention relates to an electronic teaching desk, which includes a video camera for continuously photographing and storing the motion of a user for each unit time and a voice sensor for recognizing the voice of the user, comprising: a microphone array for receiving voice signals through the voice sensor from the user, separating sound and canceling noise; a voice recognition engine for receiving the voice signals, which are analog signal having noise removed therefrom, from the microphone array and converting the same into digital data, which are text signals; a database for repeatedly storing the received voice signals and converted text signals; a command interpretation part for receiving the converted text signals from the speech recognition engine and interpreting the same as a command through machine learning using the database; and a control part for performing a query search or internal/external control functions with an external cloud service according to the interpreted command, wherein the voice recognition engine includes a deep learning-based correction timing determining software for determining the correction timing of the voice signals inputted from the user.

Description

An electronic podium system that performs AI control technology in cloud environment and includes a self-sterilization function}

The present invention relates to an electronic classroom, and more particularly, to an electronic classroom system capable of artificial intelligence (AI)-based internal function control processing of voice recognition data and control of external devices connected to the Internet of Things (IoT), an image camera and It is a technology related to a system that detects the movement and sound of the speaker by using a sound sensor, automatically determines whether the speaker’s voice is abnormal through deep learning technology, and calculates the correction time. It relates to an electronic classroom system that controls or monitors connected devices through an optimal path in controlling peripheral external devices, builds a database using simulated data to protect user privacy in a cloud environment, and strengthens security by authenticating the speaker will be.

In recent years, as the classroom or meeting room changes to a digital environment, various classroom or meeting tools are replaced with digital equipment, and the communication method has also changed to two-way digital communication. For example, with the spread of digital textbooks and the spread of educational environments using various multimedia devices, in the case of an electronic classroom, documents of lecture contents or presentation contents are displayed on the screen, and various kinds of lectures necessary for lectures are provided, such as writing and writing functions. It is possible to perform a function, and at this time, it is possible to directly touch a human hand or to detect a touch with a dedicated touch pen.

In addition, recently, with the development of artificial intelligence (AI) equipment and processes, a specific command is transmitted to the device by the user's audio voice recognition instead of a traditional input device (eg, keyboard or mouse, etc.), and various searches of the device are performed. And the technology to control the control function appeared.

The following prior art 1 (KR 2015-0087687 A) is an interactive system, a display apparatus, and a controlling method thereof, and the disclosed display apparatus includes an input unit for receiving a user's utterance voice, an utterance A communication unit that transmits a voice signal for a voice to the voice recognition device, a voice recognition unit that performs voice recognition on the spoken voice, and a reliability value for the first voice information recognized through the voice recognition unit and the second value recognized through the voice recognition device and a controller for determining, as an execution command for the spoken voice, voice information having a reliability value greater than or equal to a preset threshold among reliability values for the second voice information. Accordingly, when the display device and the external device simultaneously recognize the user's spoken voice, the display device may select a voice recognition result that is close to the user's intention among the two voice recognition results. On the other hand, another prior art 2 (US 10198246 B2) below is a method and apparatus for voice-activated control of an interactive display, in which communication is possible with an interactive display device. Although it is possible to receive the user's voice data through the coupled voice input device, the technical elements of the interactive electronic classroom including a touch panel screen having a touch sensor are not disclosed.

Therefore, in the present invention, a video camera and a sound sensor are used in a touch sensor electronic classroom using a capacitive (PCAP), electromagnetic induction (EMR) or infrared (IR) method to detect the movement and sound of the speaker, and deep learning technology It is intended to propose an electronic classroom that includes a system that automatically determines whether the speaker's voice is abnormal and calculates the correction time, and includes a self-sterilization function.

Prior Patent 1: Korean Patent Publication No. 10-20152015-0087687 (interactive system, display apparatus and controlling method thereof) Prior Patent 2: US Patent Registration Publication No. 10198246 (Methods and apparatus for voice-activated control of an interactive display) Prior Art 3: Republic of Korea Patent Registration Publication No. 10-0919680 (Apparatus and method for delivering data based sensor network)

It is an object of the present invention for each speaker or speaker when using a separate mixing system to output the speaker's voice as audio of a certain balance in conducting a face-to-face or non-face-to-face lecture or speech using an electronic classroom at the pulpit The technician directly observes the speaker's voice response and accordingly mixes the components of the voice signal, such as adjusting the volume and frequency, in an appropriate combination. The purpose is to solve the problem that quality voice correction is not guaranteed.

In addition, even if a person detects an abnormality in the voice signal in a specific section, a mistake may occur in determining the correct correction time, and human labor is required to catch an audio signal with an optimal balance through various trials and errors. The purpose is to solve

Another object of the present invention is to perform an artificial intelligence (AI)-based cloud search or device control function by processing a user's voice recognition information in an electronic classroom including a large-area touch screen.

Another object of the present invention is to perform a function of controlling an external device connected to an Internet of Things (IoT) network by wire or wireless according to voice recognition information in an electronic classroom including a large-area touch screen.

Another object of the present invention is to control or monitor connected devices through an optimal path in controlling various external devices around IoT, and to build a database to protect user privacy in a cloud environment and to strengthen security by authenticating the user The purpose is to perform

The present invention includes a panel unit capable of touch recognition and a sensor unit that determines a touch method when the panel unit is touched and performs touch recognition according to the touch method, and continuously captures and stores the user's movement for each unit time. An electronic classroom comprising a camera and a voice sensor for recognizing the user's voice, the electronic classroom comprising: a microphone array that receives a voice signal from the user through the voice sensor, separates a sound source, and removes noise; a voice recognition engine that receives a voice signal that is an analog signal from which noise has been removed from the microphone array and converts it into digital data that is a text signal; a database for repeatedly storing the received voice signal and the converted text signal; a command interpretation unit that receives the text signal converted from the speech recognition engine and interprets it as a command through machine learning using the database; and a control unit that performs a query search or internal/external control function with an external cloud service according to the interpreted command, wherein the voice recognition engine includes a deep learning-based correction timing for determining the correct timing of the voice signal input from the user. Software may be included.

and a voice pattern matching unit that compares the command interpreted by the command interpretation unit with an internal command list to search for a pattern of the voice signal, wherein the voice pattern matching unit recognizes the voice by a predetermined voice pattern in the first sleep mode An internal control signal for changing the engine to a wake-up mode is transmitted to the control unit.

On the other hand, the deep learning-based corrective timing determination software may include: a data acquisition module for collecting movement and voice data of a speaker who is the user; a data pre-processing module for removing noise from data through an algorithm that divides image and sound signals from the acquired data; an abnormal situation detection module that detects an abnormal situation when an analysis value that is out of a certain range from a standard set in the learning data through deep learning learning is detected and uses the abnormal situation to determine the right time to correct the speaker's voice; and a post-processing module having an interface for outputting the result calculated through the corrective timing determination software.

The abnormal situation detection module uses an artificial intelligence neural network model, but replaces insufficient data in neural network learning with pre-made fake image data and fake sound data, and learns Generative Adversarial Nets (GAN) of an unsupervised learning neural network model. It is characterized in that it is determined as an abnormal point of the speaker's voice signal when it is determined as an analysis value out of a predetermined numerical range in the deep learning learning data standard.

The deep learning-based corrective timing determination software further comprises a calculation step of determining a corrective timing of the voice signal from the abnormal timing when the abnormal situation detection module determines that the speaker's voice signal is an abnormal timing, the calculation Comparing the result value of the corrected correction bandit with the next similar abnormal situation, the method further includes a correction step of learning by feeding back the error data.

When performing a control function of the connected external peripheral devices, the control unit performs the control function through an optimal route recommendation AI algorithm, the algorithm includes a data collection step in which the data collection module receives a data set, a preprocessing module A preprocessing step of preprocessing the input data set, a prediction calculation step in which a prediction module configured through machine learning of an artificial neural network (ANN) applies the preprocessed data set to a prediction algorithm, and an optimization module predicts a model according to an objective function It consists of an optimal route recommendation step that finds an optimal route using and constraints.

The voice signal and the converted text signal are stored in a database according to the result of processing the hash function process based on the timestamp, which is the generation time of the data, using the data of each original signal and k-1 copies of the simulated data. By doing so, it is possible to protect sensitive information related to the user.

The hash function uses a Hash-based Message Authentication Code algorithm (HMAC), and a result value Q may be generated by Equation 1 below.

<Equation 1>

(TS is timestamp information, K_CNT is a count number based on the k value, and PK is previously shared secret key information)

The system generates a two-class user model by using a support vector machine (SVM), which is a binary linear classification model, from the conversion information converted by the IP band or user agent, which is environmental information that the user logs into the system, so that the user can Decide whether you are

The electronic classroom system includes a main body on which a touch display is disposed, a cover that covers the touch display and is arranged to be movable on the main body, and is disposed on one side of the cover facing the touch display to irradiate the touch display with ultraviolet rays Further comprising a sterilization unit including a UV LED and a control unit for setting the operation of the sterilization unit, wherein the control unit applies power to the sterilization unit when the cover covers the touch display to apply power to the touch display for a preset time The display operates to irradiate UV rays, and the UV LEDs are arranged at intervals on the one surface, and the UV LEDs are separated from the touch display by 35 to 45 mm and irradiate the UV LEDs at an angle of 110 to 130 degrees. It is characterized in that the interval between the UV LEDs is 60 to 80 mm.

The electronic classroom of the present invention detects abnormalities in the speaker's voice signal in real time through a voice sensor and a video camera and determines the appropriate time to correct the voice signal, thereby risking a sense of heterogeneity due to irregular tremors of the voice signal or the introduction of abnormal sound It is possible to increase the quality of audio quality of lectures or speeches by minimizing the

In addition, by automatically detecting voice abnormalities in real time, it is possible to reduce instability according to the success or failure of calibration performed by humans and the burden of labor costs caused by the input of manpower for monitoring.

The present invention has the effect of performing an artificial intelligence (AI)-based cloud search or control function by learning, interpreting, and processing the user's voice recognition information in an electronic classroom including a touch sensor.

In addition, the present invention has an effect of controlling an external device connected to an Internet of Things (IoT) network through a user's voice command in an electronic classroom including a large-area touch screen.

In addition, in performing these AI-based input/output processing and IoT control functions, voice data processing is possible through its own independent processing algorithm without being dependent on the dedicated server of a third-party Internet service provider (communication company). It has the effect of being easy to install and use independently regardless of overseas.

On the other hand, in controlling various external devices around the IoT, it is possible to control or monitor a number of connected devices through an optimal path.

On the other hand, to protect privacy in the cloud environment, security can be further strengthened by building a database using simulated data and authenticating the speaking user using SVM.

1 shows a structure in which an artificial intelligence (AI)-based cloud search or control function can be performed by processing a user's voice command 200 information in an electronic classroom 300 of a touch screen according to an embodiment of the present invention. It is a drawing.
FIG. 2 is a diagram illustrating a situation in which the cloud service 240 is used using the existing communication company's dedicated speaker 210 , the communication network 250 , and the AI server 230 .
3 is a diagram illustrating a situation in which the cloud service 330 is used by using its own individual speaker 310 regardless of a communication company according to an embodiment of the present invention.
4 is an artificial intelligence (AI)-based cloud search 450 or control function 480, 490 by processing the user's touch input or voice input (400, 410) information in the electronic classroom as an embodiment of the present invention. It is a diagram showing a flow chart to perform.
5 is a block diagram of a neural network model for producing fake image data in deep learning-based corrective timing judgment software according to an embodiment of the present invention.
6 is a diagram illustrating an electronic classroom in which a UI screen including a menu configuration such as an AI-based control function is displayed according to an embodiment of the present invention.
7 is a flowchart illustrating an example of applying a process for determining whether a voice is abnormal and correcting timing using a video camera and an audio sensor regarding a location and distance from a speaker according to an embodiment of the present invention.
8 is a schematic perspective view showing an electronic classroom with a cover including a sterilization function according to an embodiment of the present invention.
9 is a view schematically showing the inside of the electronic classroom of FIG. 8, and in particular, a cross-sectional view showing the side of the sterilization unit through a perspective view.

Advantages and features of the present invention and a method of achieving it will become more apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings.

However, the present invention is not limited to some limited embodiments disclosed below, but may be implemented in various different forms within substantially the same scope, and these embodiments are also considered to be included in the technical matters of the present invention. will be able

Examples in the present specification are provided to make the disclosure of the present invention clearer, and to fully understand the scope of the invention to those of ordinary skill in the art to which the present invention pertains.

And the scope of the present invention will be defined by the claims to be described later.

Accordingly, in some embodiments, well-known components, well-known operations, and well-known details are not specifically described to avoid obscuring the present invention.

In addition, the same reference numerals refer to the same elements throughout the specification, and the terms used herein are for the purpose of describing the embodiments only and are not intended to limit the present invention.

In this specification, the singular shall be construed to include the plural as well, unless the phrase specifically states otherwise, and elements or operations referred to as including (or having) do not exclude the presence and addition of one or more other elements or operations. .

Unless otherwise defined, all terms (including technical and scientific terms) used herein may be used with the meaning commonly understood by those of ordinary skill in the art to which the present invention belongs.

In addition, terms defined in general used dictionary shall not be interpreted excessively or ideally unless otherwise defined.

Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings.

1 shows a structure in which an artificial intelligence (AI)-based cloud search or control function can be performed by processing a user's voice command 200 information in an electronic classroom 300 of a touch screen according to an embodiment of the present invention. It is a drawing.

The electronic classroom 300 including a panel unit (not shown) capable of touch recognition and a sensor unit (not shown) that performs touch recognition according to the touch method by determining a touch method when the panel unit is touched is an external source A touch input signal 100 or a voice command signal 200 is received from (100, 200).

At this time, according to the input touch method, a pattern of a touch recognized by the touch sensor 310 is displayed on the screen of the electronic classroom 300 and a response according to other UI input is output.

If the voice command 200 signal is input, the quality of the analog voice signal signal input by the array type microphone module 320 that separates the sound source and removes noise is improved. In this case, the microphone array 320 may implement a gain compensation or echo cancellation function using a plurality of microphones and perform acoustic model machine learning for compensating for the distorted voice signal. Through this, voice recognition can be smoothly processed even at a distance from the microphone, and the voice recognition function must be operated even when there is another audio function (eg, audio reproduction of presentation materials) in the electronic classroom.

Then, a voice signal, which is an analog signal from which noise has been removed, is received from the microphone array 320 and converted into digital data, which is a text signal, in the voice recognition engine 340 (Speech to Text), and the voice signal and the converted text signal are converted into a database. It is stored repeatedly at 360. The data stored in this way are used as learning data for machine learning based on artificial intelligence for continuous upgrade of S/W in the voice recognition system including the voice recognition engine 340 and the command interpretation unit 350 using its own speaker.

Meanwhile, in the command interpretation unit 350 that receives the converted text signal from the speech recognition engine and interprets it as a command through machine learning using the database 360, the algorithm includes an algorithm for analyzing a voice command pattern, and command control according to the pattern. A protocol matching algorithm, a voice pattern array matching algorithm, and a matching pattern algorithm for granting the same meaning in heterogeneous languages may be performed together.

According to the command interpreted in this way, the control unit 330 may perform a query search function to an external cloud service, such as an Internet search, or perform an internal/external control function.

At this time, taking an example of an internal/external control function, first, as an internal function control, the power of the electronic classroom 300 is turned on or off, the volume of the electronic classroom 300 is adjusted, or an on-screen control control (announcement screen size adjustment, announcement) Page movement, etc.) or an input source (USB, HDMI, etc.) conversion, the electronic classroom 300 can perform an internal control function.

And, taking the external function control as an example, there may be a situation in which the electronic classroom 300 and peripheral devices (eg, classroom lamps, door locks, other electronic classrooms, etc.) located in the Internet of Things (IoT) network connected by wire or wireless are controlled. That is, by using an IoT-related relay and an IR controller, it is possible to switch the lamp lights around and turn them on and off.

On the other hand, the command interpreted by the command interpretation unit 350 further includes a voice pattern matching unit (not shown) that searches for the pattern of the voice signal by comparing it with the internal command list 370, and the voice pattern matching unit is in the first sleep mode ( In the sleep mode), an internal control signal for changing the voice recognition engine 340 to a wake-up mode according to a predetermined voice pattern is generated. In addition, when there is no additional voice command input for a predetermined time in the wake-up mode, it is switched back to the sleep mode to minimize power and other resource consumption.

In addition, the present invention aims to provide an optimal path for a control process that can reduce data transmission costs for external devices connected to the cloud and maximize IoT data (input, sensing, monitoring) collection.

The optimal path (root) recommendation algorithm can be configured including data acquisition module (data acquisition), preprocessing module (system processes), prediction module (prediction), predictive model learning module (learning module) and optimization module (optimization module) In addition, the collection module receives an input dataset that is actual usage data of the peripheral device, and the preprocessing module may preprocess the inputted data set. In addition, the preprocessing module divides the data set into a training data set and a test data set, and the prediction module applies the preprocessed data set to a prediction algorithm, more specifically, the prediction module converts the test data set into a prediction algorithm can be predicted by applying

On the other hand, the predictive model configuration unit can configure the prediction module by machine learning in an artificial neural network (ANN) using the training data set, and the optimization module uses the prediction model and constraints according to the objective function to determine the optimal route can be found

The preprocessing module preprocesses the input data set. In this case, null input can be removed and fields necessary for prediction can be derived during preprocessing, and the preprocessed data can be supplied to the prediction module. Therefore, in the prediction module, the usage prediction model can be calculated by applying the prediction algorithm having the highest accuracy among various prediction algorithms to the data, and at this time, the prediction algorithm with the best performance is stored and applied to the next step. Accordingly, the optimization module can use some constraints and predictive models and find the most suitable path (root) according to the objective function.

A data set can be created based on the actual peripheral usage frequency and power consumption, and the cost according to the communication cost of peripheral devices (distance between devices and data amount) can be calculated through the model and constraints predicted using the objective function. Minimize and maximize IoT data collection for a specific grid.

FIG. 2 is a diagram illustrating a situation in which the cloud service 240 is used using the existing communication company's dedicated speaker 210 , the communication network 250 , and the AI server 230 .

Unlike FIG. 3, which will be described later, in FIG. 2, in addition to the smartphone 200 and the computer 220, the cloud service ( 240) can be used.

That is, it is necessary to subscribe to a specific communication company and use the AI speaker dedicated to the communication company, and the specialized artificial intelligence service must also be developed by each communication company, so it is not suitable for use for special purposes such as electronic classrooms. In addition, AI functions with specialized characteristics such as electronic tutoring consume resources due to frequent and many signal requests compared to other services. And in exceptional circumstances, such as when the telecommunication company's AI server goes down, the entire AI function may not be available. Also, foreign language-based export has disadvantages, and internationalization is difficult due to the interests (differences in language implementation, communication service methods, etc.) between telecommunication operators in other countries. In comparison with this, the cloud service usage method of the present invention of FIG. 3 will be described.

3 is a diagram illustrating a situation in which the cloud service 330 is used by using its own individual speaker 310 regardless of a communication company according to an embodiment of the present invention.

In FIG. 3 , there is no dedicated speaker 210 developed and provided by an Internet service provider (communication company) in addition to the smartphone 300 and the computer 320 , and an Internet communication network 340 independent of the communication company that is not the communication company dedicated communication network 250 . We assume the situation of using . In addition, the cloud service 240 can be directly used by using the individual speaker 310 equipped with the voice recognition engine. In other words, there is no need to subscribe to a specific telecommunication company and use an AI speaker dedicated to the telecommunication company. Therefore, it is suitable for use for special purposes such as electronic teaching tables. In addition, specialized AI functions such as electronic classrooms can consume fewer resources by installing dedicated S/W to handle frequent and many signal requests compared to other services. Since there is no limitation that it cannot be applied when using other carriers, there is no need to worry about whether the carrier is a carrier or not, and there is no need to consider exceptions such as a downfall of the telecommunication company's AI server.

In addition, even if it is used for overseas export based on a foreign language other than Korean, it can be developed as a customized service for each exporting country as needed. .

As shown in FIG. 3 , when using the cloud service 330 , the control unit connects communication to an external cloud server according to the command interpreted by the command interpretation unit, and searches for SaaS (Software as a Service)-based cloud data by querying. The corresponding content can be displayed on the screen of the electronic classroom.

On the other hand, various types of services can be provided on the cloud platform to which the electronic classroom is connected. Among them, the storage-based cloud service is the most widely used service type. For this purpose, a deduplication function is applied that does not store the same data as the existing data. However, if the deduplication technology is used, there is a structural privacy problem.

That is, in order to apply the deduplication technology, the mapping structure between users and files is stored as meta information, and in this process, a list of users who uploaded a specific file can be secured through meta-analysis on the server.

In addition, with the development of big data technology, the technology for tracking sensitive information of a specific individual or device is also becoming important. Specific sensitive information of individuals or devices uses location information expressed in GPS coordinates (latitude/longitude), intellectual property rights information such as copyright of presentation materials used for speeches or presentations, and other private personal information that can identify the speaker. In the case of applications that do this, if these applications access the cloud, send and receive various sensitive information data, and continuously store various sensitive information output values, serious privacy violations may occur.

As a security technique that can be used in such a number of cases, a hash function value of Q can be calculated by Equation 1 using the K-anonymity model and data perturbation. For K-anonymization, Q values that are HMAC values for time information may be calculated based on a preset k value (eg, k=3).

<Equation 1>

(HMAC is a Hash-based Message Authentication Code algorithm, TS is the timestamp information of time information, K_CNT is the count number based on the k value, and PK is the previously shared secret key information)

K-anonymity is one of the privacy protection models of data de-identification, and since there is a risk of individual identification when only specific single data is stored, it is a method of lowering the possibility of identification by simultaneously storing data with a value of k or more when storing data. For example, if k is 3, when saving 1 IoT data, 3 data are stored at the same time. At this time, one piece of data is real data, and the other two pieces of data are virtual fake data. At this time, you need to know the PK corresponding to the pre-shared secret key, such as an authorized administrator, so that you can determine which one is the real data.

TS means a time stamp, and as mentioned above, when data is input at a preset period, it means a specific time at which data is input or the time at which IoT data is generated. K_CNT corresponds to a number counted according to a preset value of k. For example, when k is 3, the value of K_CNT has values of 1, 2, and 3. In this case, when the K_CNT value is 1, it is original data, and when it is 2 or 3, it can be treated as fake (imitation) data.

As in Equation 1 above, specific timestamp (TS) information is converted into Q values of character data using the HMAC algorithm based on PK information, and at the same time, all output values corresponding to k K_CNT values are calculated and k number of Q value can be calculated. Time information is not included in the database stored in this way, and only k Q values are stored. Therefore, even if the database is hacked, the time information is not exposed, and even if the Q-value information is hacked, if the secret key (PK) information is not known, it is impossible to know which values are the original data, so it is practically impossible to hack the original data. will do

4 is an artificial intelligence (AI)-based cloud search 450 or control function 480, 490 by processing the user's touch input or voice input (400, 410) information in the electronic classroom as an embodiment of the present invention. It is a diagram showing a flow chart to perform.

Referring to FIG. 4, first, it is determined whether there is an input 400 of a touch input or a voice input (410), and when a voice signal is input, the voice recognition engine provided in the electronic classroom converts the voice signal, which is an analog signal, to a digital signal. Converts to a text signal (speech to text) (420).

Then, in interpreting the converted text data (430), the previously analyzed data and the previously classified voice signal through machine learning are called from the database and compared repeatedly, so that as time passes and the amount of data to be processed increases, more Sophisticated interpretation is possible. In this way, by using AI-based learning and interpretation, the speech recognition rate can be increased to more than 90%, and the operation according to the speech recognition can be implemented.

Next, it is determined whether the interpreted command is a request such as an Internet search using a cloud service or a request to control a device (440), and in the case of an Internet search request (450), the content screen corresponding to the search result is displayed (450). The electronic classroom may be marked (460).

Or, in the case of a request to control a device (470), is it a command to control the internal functions (power, volume, screen, source, etc.) of the electronic classroom or control functions of external devices connected by wire or wireless (classroom lamps, curtains, door locks, etc.) The external device is controlled (480) or an internal function of the electronic classroom is executed (490) depending on whether the command is to be performed (e.g., electronic classroom, etc.).

On the other hand, as a method for the electronic classroom to authenticate the user who has uttered the voice signal, the following steps may be performed.

First, the user's login environment information is extracted. The user's login environment information may be an IP band and a user-agent when the user logs in to an Internet site. Next, the extracted log-in environment information is converted into a One-Hot-Encoding method to generate conversion information, and a number is assigned to unique values for each environment information from 1 to label encoding.

Next, a two-class user model is generated using the conversion information using a support vector machine (SVM). SVM (Support Vector Machine) is one of the machine learning algorithms and is mainly used for pattern recognition.

SVM (Support Vector Machine) creates a non-stochastic binary linear classification model that determines which category the new data belongs to based on the given data set when a set of data belonging to one of two categories is given. In the user authentication method, a two-class user model is generated using SVM based on conversion information of the entire system user, and when a user logs in, the two-class user model is applied to determine whether the user is the user. Therefore, when an illegal user steals the ID and password of the original owner to log in to an IoT peripheral device connected to the electronic classroom or to an Internet site through the electronic classroom, it is possible to detect and deny access, thereby improving security.

5 is a block diagram of a neural network model for producing fake image and voice data while performing a deep learning-based corrective timing determination process in an electronic classroom according to an embodiment of the present invention.

A large amount of data is required for neural network learning to determine and further predict the corrective timing of output signals, especially audio signals, such as voice, based on artificial intelligence from electronic devices that have received video and audio signals.

Therefore, in an embodiment of the present invention, a neural network model is used as an artificial intelligence model as an electronic classroom including an image camera and a voice recognition sensor, but image data insufficient for neural network learning is pre-fabricated as fake image and voice data , and apply the Generative Adversarial Nets (GAN) model, an unsupervised learning neural network model, to neural network learning to increase the accuracy of judgment and prediction of the corrective timing of speech signals.

That is, an artificial intelligence neural network model is used, but the data lacking in neural network learning is replaced with pre-made fake image data and fake voice sound data, and the GAN of the unsupervised learning neural network model is applied to the learning in the deep learning data standard. When it is determined that the analysis value is out of a predetermined numerical range, it can be determined as the time when an abnormal situation occurs in the speaker's voice.

More specifically, in order to increase the accuracy of the prediction and judgment of the corrective timing of speech, the advantages of the Variational Auto Encoder (VAE) and the characteristics of the GAN model are used. In addition, the GAN, which is often judged by visualizing the results because the comparison criteria are not clear, is placed at the rear so that a high-accuracy neural network model can be used.

Referring to FIG. 5 , a neural network model for producing fake image/voice data may largely include a filter unit 510 , a VAE unit 520 , a CNN unit 530 , and a GAN unit 540 , and input ( As input), the previously secured speaker's motion image data and audio data may be used. The input data is subjected to a pre-processing process such as noise removal through the filter unit 510 so that various distortions can be corrected, and a data augmentation (Crop, Rotate, Flip, Translate, Resize, etc.) technique can be used.

The distortion-corrected image/audio data or data generated by the augmentation technique is input to the VAE unit 520 composed of multiple inputs, and is processed by the encoder (Encoder, 521), the latent space (523) and the decoder (Decoder, 525). By passing through , a plurality of latent feature vector data may be output.

Here, the Augmentation (data modulation) technique is a method of increasing the accuracy of prediction and judgment of the corrective timing of speech with artificial intelligence. Examples of various preprocessing techniques, selection of appropriate learning models, selection of appropriate neural networks, and various ensemble models can be seen.

In the present invention, to solve the data shortage phenomenon necessary for learning the AI model, when sufficient data is obtained by sufficiently observing and photographing the movement of the speaker, particularly the distance and position from the microphone, the data obtained in the pre-processing step is used or , other algorithm methods can be mixed to learn for higher accuracy.

A plurality of potential feature vector data passes through the CNN unit 530 and is used as input data of P(x) 541 of the neural network GAN model, and random vector data can be used as input data of G(x) 543. have.

In the D(x) 545 of the GAN unit 540, a loss function is obtained through the mutual difference between the P(x) 541 and the G(x) 543, and a backproper using the obtained loss function The fake image/audio data and the real image/audio data are corrected through the backpropagation method.

By repeating this process, more precise prediction is possible, and the output data of D(x) 545 can be used for final training or model validation.

In this case, the CNN unit 530 may configure multiple neural network layers, and other neural networks of supervised learning, unsupervised learning, or semi-supervised learning may be configured in the layer to configure various learning data.

6 is a diagram illustrating an electronic classroom in which a UI screen including a menu configuration such as an AI-based control function is displayed according to an embodiment of the present invention.

Basic menus such as File (610), E-board (620), Internet (630) and more various application menus (640) are displayed on the screen of the electronic classroom, and the user can use such UI touch input, motion input or Through voice input, it is possible to control not only the internal control operation of the electronic classroom, but also other peripherally connected devices (classroom lighting, indoor air conditioner, classroom beam projector, window curtains, door locks, etc.).

7 is a flowchart illustrating an example of applying a process for determining whether a voice is abnormal and correcting timing using a video camera and an audio sensor according to an embodiment of the present invention.

Each step will be described in detail with reference to FIG. 7 as follows.

First, as a preparation step, as described above, a neural network model is used as an artificial intelligence model because a large amount of image data is required for neural network learning to determine and predict the correct timing of audio based on video images and audio signals. Decision and prediction of corrective timing of voice signals by replacing insufficient image data for learning with pre-made fake image and audio data, and applying Generative Adversarial Nets (GAN) model, an unsupervised learning neural network model, to neural network learning can increase the accuracy of

That is, an artificial intelligence neural network model is used, but the data lacking in neural network learning is replaced with pre-made fake image data and fake voice sound data, and the GAN of the unsupervised learning neural network model is applied to the learning in the deep learning data standard. When it is determined that the analysis value is out of a predetermined numerical range, it is determined that an abnormal situation occurs in the speaker's voice, and immediate voice correction can be applied.

Next, by using the video camera installed in the electronic classroom, data is acquired by measuring the position/posture changes such as the distance to the speaker's microphone, position, and movement posture continuously for each unit time.

The next step is data pre-processing. Data pre-processing is performed to remove noise from data through an algorithm that divides distance images and audio signals, and then, the acquired image or audio and sound data is learned with a deep learning algorithm to perform repetitive movements. By forming a model for judging the correction timing as a sound and a sound, it goes through the steps of detecting abnormal situations in the observer’s and speaker’s voice. If the abnormality detection module determines that it is an abnormal timing of the speaker's voice, the final result value is derived through the step of calculating the corrective time of the voice, and the speaker's voice abnormality detection result and the corrective timing calculation result are calculated. record

On the other hand, the result of the correction time calculated in this way is compared with a similar abnormal situation later, and the error data is fed back to the correction timing judgment software. have.

8 is a schematic perspective view showing an electronic classroom with a cover including a sterilization function according to an embodiment of the present invention, and FIG. 9 schematically shows the inside of the sterilization electronic classroom of FIG. It is a view showing through a sectional view of the .

8 and 9, the electronic classroom 1 including a sterilization function according to the present embodiment includes a main body 10, a cover 20, a sterilization unit 30a, and a control unit 40, and It is possible to protect the user from various viruses by sterilizing the user's hand and the part that is easily touched by splashes.

The main body 10 forms the appearance of the sterilization electronic classroom 1, and the touch display 11 is disposed on the upper surface, and the storage space (not shown) in which the office equipment, the computer main body, etc. are located inside the main body 10 ) is formed, and the computer body is connected to the touch display 11 .

In addition, a storage space 12 in which a drawer 13 that is opened toward a user is located is formed inside the main body 10 . A keyboard, a mouse, etc. connected to the computer body may be disposed in the drawer 13 .

The cover 20 is disposed on the upper surface of the main body 10 to be linearly movable. The cover 20 faces the touch display 11 while covering the upper surface of the main body 10, and protects it from external requirements such as impact and contamination. However, the cover 20 may deviate from the upper surface of the cover 20 by linear movement to expose the touch display 11 . In this case, the user may use the touch display 11 .

The detailed configuration of the main body 10 and the cover 20 of the sterilization electronic classroom is described by the applicant of the known electronic classroom system (Patent Registration No. 10-1047040) and the dual monitor-based electronic classroom (patent registration). No. 10-1304520), the configuration of the electronic tutoring system (Patent Registration No. 10-1094193) can be applied.

The sterilizing part 30a is disposed on the lower surface (one surface) of the cover 20 . The sterilization unit 30a faces the touch display 11 in a state in which the cover 20 covers the upper surface of the main body 10 . The sterilization unit 30a includes a UV LED capable of irradiating ultraviolet rays with a short wavelength of 100 nm to 280 nm.

A plurality of UV LEDs of the sterilization unit 30a are arranged at intervals on the lower surface of the cover 20 . Referring to the drawings, in a state in which the cover 20 covers the touch display 11, the UV LEDs are spaced apart from the touch display 11 by 35 to 45 mm (L1). UV LEDs may irradiate ultraviolet rays at an angle (D) of 110 to 130° to the touch display 11 . And the interval L2 between the adjacent UV LEDs may be 60 to 80 mm. The interval between adjacent UV LEDs may be 69.2 mm, and the UV irradiation angle may be 120°. Accordingly, the sterilization unit 30a may irradiate the entire surface of the touch display 11 with ultraviolet rays.

The ultraviolet rays generated by the sterilization unit 30a are irradiated to the touch display 11 to sterilize the surface of the touch display 11 . Accordingly, the touch display 11 can remove 99% or more of viruses (bacteria) without residual substances.

Although the sterilization unit 30a has been described as a UV LED, the sterilization unit 30a may be formed of a UV lamp that generates ultraviolet rays.

On the other hand, a sub-sterilization unit 30b is disposed in the storage space 12 . The sub-sterilization unit 30b has the same configuration as the sterilization unit 30a, and is sterilized by irradiating ultraviolet rays with a keyboard, mouse, or the like.

The control unit 40 is disposed on the cover 20 and is electrically connected to the sterilization unit 30a and the sub sterilization unit 30b. The control unit 40 applies power so that the sterilization unit 30a operates when the cover 20 covers the upper surface of the main body 10 . In addition, the control unit 40 applies power to operate the sub-sterilization unit 30b when the drawer 13 is closed.

The sterilization unit 30a and the sub sterilization unit 30b sterilize the touch display 11, the keyboard, the mouse, etc. by generating ultraviolet rays for a preset time set in the control unit 40 . The sterilization time may be variously changed according to a user's setting.

The sterilization unit 30a is connected to the control unit 40 and operates an operation switch 32 that operates the control unit 40 to operate the sterilization unit 30a and the sub sterilization unit 30b, the sterilization unit 30a and the sub sterilization unit 30a. It may include an information display window 31 indicating the operating state of (30b). The operation switch 32 and the information display window 31 are disposed on the cover 20 and are exposed so that the user can operate and check them.

On the other hand, the information display window 31 may display a guide phrase such as "Do not move the cover because it may be exposed to UV rays during the operation of the touch display sterilization function and the operation of the keyboard sterilization function".

When the cover 20 or the drawer 13 is opened during the operation of the sterilization unit 30a and the sub sterilization unit 30b, the control unit 40 cuts off the power applied to the sterilization unit 30a and the sub sterilization unit 30b to prevent ultraviolet rays. You can prevent this from happening.

And a human body detection sensor (not shown) is disposed on the body 10 . The human body detection sensor detects when the user approaches the sterilization electronic classroom 1 during the operation of the sterilization unit 30a. The control unit 40 receives the signal from the human body detection sensor and cuts off the power applied to the sterilization unit 30a and the sub sterilization unit 30b to prevent ultraviolet rays from being generated. Accordingly, it is possible to protect a user approaching the sterilization electronic classroom 1 from ultraviolet rays during the operation of the sterilization unit 30a and the sub-sterilization unit 30b.

Accordingly, according to the present embodiment, after using the sterilization electronic classroom 1, the cover 20 is closed and the drawer 13 is put in, the sterilization unit 30a is operated to emit ultraviolet rays for a preset time. Ultraviolet rays can be irradiated to touch displays, control pads, terminal interfaces, keyboards, mice, etc. that are easily touched by the user's hands and splashes to sterilize the surface. Accordingly, by sterilizing viruses that can inhabit the surface of the touch display, users such as speakers and speakers can be protected from viruses.

The present invention is not limited to the specific preferred embodiments described above, and various modifications can be made by anyone with ordinary skill in the art to which the invention pertains without departing from the gist of the invention as claimed in the claims. And it will be apparent that such changes are within the scope of the claims.

100: touch input
200 : voice command
300: electronic classroom
310: touch sensor
320: microphone array
330: control unit
340: speech recognition engine
350: command interpretation unit
360: database
370 : command list

Claims (10)

An image camera comprising a panel unit capable of recognizing a touch and a sensor unit performing touch recognition according to the touch method by determining a touch method when the panel unit is touched, and continuously photographing and storing the user's movement for each unit time, and the In the electronic classroom comprising a voice sensor for recognizing a user's voice,
a microphone array that receives a voice signal from the user through the voice sensor, separates a sound source, and removes noise;
a voice recognition engine that receives a voice signal that is an analog signal from which noise has been removed from the microphone array and converts it into digital data that is a text signal;
a database for repeatedly storing the received voice signal and the converted text signal;
a command interpretation unit that receives the text signal converted from the speech recognition engine and interprets it as a command through machine learning using the database;
Includes a control unit that performs a query search or internal and external control function to an external cloud service according to the interpreted command,
The voice recognition engine includes deep learning-based corrective time determination software for determining the correct time to correct the voice signal input from the user,
In the panel unit and the sensor unit, according to the input touch method, a pattern of a touch recognized by the touch sensor is displayed on the screen of the electronic classroom and outputs a response according to another UI input,
The microphone array implements a gain compensation or echo cancellation function using a plurality of microphones, and performs a function of compensating for a distorted portion in the voice signal by far-field voice recognition and other audio signals,
The control unit creates a data set based on the frequency of use and power consumption of the actual peripheral device, and minimizes the cost according to the distance between devices and the data amount of the peripheral device through the model and constraints predicted using the objective function. characterized by maximizing the data collection of
When using the cloud service, the control unit connects the communication to the external cloud server according to the command interpreted by the command interpretation unit, queries the SaaS-based cloud data and displays the contents on the screen of the electronic classroom,
The cloud service is a storage-based cloud service, and the storage-based cloud service applies a deduplication function that does not store the same data as the existing data, and extracts the user's login environment information as a method of authenticating the user, An electronic classroom system, characterized in that the extracted log-in environment information is converted into a One-Hot-Encoding method to generate conversion information, and labels are encoded by assigning numbers to unique values for each environment information in turn.
delete delete delete delete delete delete delete delete The method according to claim 1,
The electronic classroom system includes a main body on which a touch display is disposed, a cover that covers the touch display and is arranged to be movable on the main body, and is disposed on one side of the cover facing the touch display to irradiate the touch display with ultraviolet rays Further comprising a sterilization unit including a UV LED and a control unit for setting the operation of the sterilization unit,
When the cover covers the touch display, the control unit applies power to the sterilizer and operates to irradiate the touch display with ultraviolet rays for a preset time,
A human body detection sensor is disposed on the main body, and the human body detection sensor detects when a user approaches during the operation of the sterilization unit, and the control unit receives the signal from the human body detection sensor and cuts off the power applied to the sterilization unit to emit ultraviolet rays. Electronic teaching system, characterized in that it does not occur and protects an approaching user from UV rays.

Images