KR102652008B1 - Method and apparatus for providing a multimodal-based english learning service applying native language acquisition principles to a user terminal using a neural network - Google Patents

Abstract

Embodiments present a method and device in which a server provides a multimodal-based English learning service to a user terminal using a neural network. The method according to one embodiment includes receiving a first request message for learning English words from a user terminal, the first request message including personal information about the user and information about the user terminal, and personal information about the user. Based on the information and the information on the user terminal, proceeding method information for the user terminal is determined through a method decision model using the first neural network, and a learning start message including the proceeding method information is transmitted to the user terminal. And, multimodal-based English word learning content displayed on the user terminal is progressed based on the progress method information, and the multimodal-based English word learning content includes information about the English word, information about the definition of the English word, and information related to the English word. Contains information about animation, receives information about the progress result of the multimodal-based English word learning content from the user terminal, and generates a second neural signal based on the information about the progress result of the multimodal-based English word learning content. It may include determining a learning score for the user terminal through a learning achievement evaluation model using a network, and transmitting a learning reward message including the learning score for the user terminal to the user terminal. For example, a plurality of English word games may be activated for the user terminal based on the learning reward message.

Description

In performing English education applying the principles of native language acquisition, a method and device for providing a multimodal-based English learning service to a user terminal using a neural network {METHOD AND APPARATUS FOR PROVIDING A MULTIMODAL-BASED ENGLISH LEARNING SERVICE APPLYING NATIVE LANGUAGE ACQUISITION PRINCIPLES TO A USER TERMINAL USING A NEURAL NETWORK}

Embodiments of the present disclosure relate to technology for providing a multimodal-based English learning service to a user terminal, and to a method and device for providing a multimodal-based English learning service to a user terminal using a neural network.

Online English education services that generally start with learning English words mostly start with phonics. In other cases, it is mostly a simple composition of word-picture-Korean meaning. Most of these services focus on how to memorize better. In other words, it does not matter what meaning the word has in the sentence, and we try to match the English word and the Korean interpretation 1:1 to enable memorization quickly and easily.

It is widely acknowledged that such phonics-centered English education is ineffective. However, as they have not been able to find an alternative, they continue to use phonics, and due to this type of education, Korea, which is said to be the country that takes TOEIC and TOEFL the most in the world, eventually took the TOEFL in 2019. ) It ranks 87th out of 171 countries, and ranks only 132nd in TOEFL speaking.

The reason why we stick to phonics education despite this situation is because phonics education itself is centered on memorization. In Korean education, memorization is an important tool. Education in English and other subjects is mostly done through memorization, and even in mathematics, it is done through memorizing formulas to quickly solve problems. The math subject of the SAT test, a college entrance exam in the United States, is called the Reasoning Test, and is very different from the test that tests logic and reasoning skills. Therefore, phonics education, which is a memorization-based education, has easily taken its place in Korean children's English education.

The Korean method of learning words through memorization focuses on how quickly one can memorize the meaning of a Korean word that matches an English word. Words memorized by matching English and Korean are not effective when their meaning is expanded and derived in various situations. For example, we often use the expression “mammoth-scale” to mean something like “very large” or “on a large scale.” And animals similar to elephants that lived during the Ice Age are called mammoths. By the way, mammoth and mammoth are the same word, “Mammuthus”. Sometimes they are called mammoth and sometimes they are called mammoth in Korean and they are recognized as different. This is a phenomenon that occurs because people are accustomed to the Korean way of interpreting and memorizing things. If an image of the fundamental meaning of the word itself is formed, it is easy to infer and understand its meaning even if it is used in a modified manner.

Based on this theoretical foundation, we would like to suggest that the most effective early English education method is a native language acquisition method that utilizes a holistic language approach consisting of "words + word definitions + short animations."

Accordingly, based on personal information about the user and information about the user terminal, proceeding method information for the user terminal is determined through a method decision model using the first neural network, and based on the proceeding method information, the multimedia display is displayed on the user terminal. Modal-based English word learning content is progressed, and based on information about the progress of the multimodal-based English word learning content, a learning score for the user terminal is determined through a learning achievement evaluation model using a second neural network and sent to the user terminal. A method and device for providing a multimodal-based English learning service is needed.

Embodiments of the present disclosure can provide a method and device for providing a multimodal-based English learning service to a user terminal using a neural network.

The technical challenges to be achieved in the embodiments are not limited to the matters mentioned above, and other technical challenges not mentioned may be considered by those skilled in the art from the various embodiments described below. You can.

A method in which a server provides a multimodal-based English learning service to a user terminal using a neural network according to an embodiment includes receiving a first request message for English word learning from the user terminal, The first request message includes personal information about the user and information about the user terminal, and the user determines how to use the first neural network based on the personal information about the user and information about the user terminal. Determine progress method information for the terminal, transmit a learning start message including the progress method information to the user terminal, and proceed with multimodal-based English word learning content displayed on the user terminal based on the progress method information. The multimodal-based English word learning content includes information about the English word, information about the definition of the English word, and information about the animation related to the English word, and the progress result of the multimodal-based English word learning content is sent from the user terminal. receives information about the multimodal-based English word learning content, determines a learning score for the user terminal through a learning achievement evaluation model using a second neural network based on information about the progress of the multimodal-based English word learning content, and sends the user terminal to the user terminal. It may include transmitting a learning reward message including a learning score for the user terminal. For example, a plurality of English word games may be activated for the user terminal based on the learning reward message.

According to one embodiment, the server may receive a second request message for one English word game among a plurality of English word games from the user terminal. The second request message is a message requesting the server to participate in one English word game or to create one English word game. For example, the second request message may include either a value indicating game participation or a value indicating game creation, and identification information about the game. Identification information about the game may include an identification value indicating the type of game.

According to one embodiment, the server may receive a third request message for learning English words from the user terminal. The third request message may be a message requesting the user terminal to learn English words after the learning score for the user terminal is determined through the learning achievement evaluation model. For example, the third request message may include information about the user terminal.

According to one embodiment, a plurality of animation images constituting animation included in multi-modal-based English word learning content may be generated through an animation generation model using a neural network based on a generative adversarial network.

According to embodiments, the server can improve the user's association ability through English words and short animations by providing multimodal-based English word learning content to the user terminal. Through this, the user's creativity can be improved, memory for English words can be increased, and even meaning inference ability can be improved.

According to embodiments, the server determines and provides proceeding method information for the user terminal through a method decision model using the first neural network based on personal information about the user and information about the user terminal, Depending on basic information, you can proceed with multimodal-based English vocabulary learning content in an appropriate way.

According to embodiments, the server determines and provides a learning score for the user terminal through a learning achievement evaluation model using a second neural network based on information about the progress of multimodal-based English word learning content, and provides it to the user. Not only can it motivate learning,

The effects that can be obtained from the examples are not limited to the effects mentioned above, and other effects not mentioned can be clearly derived and understood by those skilled in the art based on the detailed description below. It can be.

The accompanying drawings, which are included as part of the detailed description to aid understanding of the embodiments, provide various embodiments and together with the detailed description describe technical features of the various embodiments.
1 is a diagram showing the configuration of an electronic device according to an embodiment.
Figure 2 is a diagram showing the configuration of a program according to one embodiment.
Figure 3 shows a method in which a server provides a multimodal-based English learning service to a user terminal using a neural network according to one embodiment.
Figure 4 is an example of multimodal-based English word learning content according to an embodiment.
Figure 5 is an example of a settings screen for multimodal-based English word learning content according to an embodiment.
Figure 6 is an example of a reward screen of multimodal-based English word learning content according to an embodiment.
Figure 7 is an example of a team-based game of multimodal-based English word learning content according to an embodiment.
Figure 8 is a diagram showing an example of an animation generation model using a neural network based on a generative adversarial network according to an embodiment.
Figure 9 is a block diagram showing the configuration of a server according to an embodiment.

The following embodiments combine elements and features of the embodiments in a predetermined form. Each component or feature may be considered optional unless explicitly stated otherwise. Each component or feature may be implemented in a form that is not combined with other components or features. Additionally, various embodiments may be configured by combining some components and/or features. The order of operations described in various embodiments may change. Some features or features of one embodiment may be included in other embodiments or may be replaced with corresponding features or features of other embodiments.

In the description of the drawings, procedures or steps that may obscure the gist of the various embodiments are not described, and procedures or steps that can be understood at the level of a person with ordinary knowledge in the relevant technical field are not described. did.

Throughout the specification, when a part is said to “comprise or include” a certain element, this means that it does not exclude other elements but may further include other elements, unless specifically stated to the contrary. do. In addition, terms such as "... unit", "... unit", and "module" used in the specification refer to a unit that processes at least one function or operation, which refers to hardware, software, or a combination of hardware and software. It can be implemented as: Additionally, the terms “a or an,” “one,” “the,” and similar related terms are used herein in the context of describing various embodiments (particularly in the context of the claims below). Unless otherwise indicated or clearly contradicted by context, it may be used in both singular and plural terms.

Hereinafter, embodiments according to various embodiments will be described in detail with reference to the attached drawings. The detailed description set forth below in conjunction with the accompanying drawings is intended to describe exemplary embodiments of various embodiments and is not intended to represent the only embodiment.

In addition, specific terms used in various embodiments are provided to aid understanding of the various embodiments, and the use of such specific terms may be changed to other forms without departing from the technical spirit of the various embodiments. .

1 is a diagram showing the configuration of an electronic device according to an embodiment.

1 is a block diagram of an electronic device 101 in a network environment 100, according to various embodiments. Referring to FIG. 1, in the network environment 100, the electronic device 101 communicates with the electronic device 102 through a first network 198 (e.g., a short-range wireless communication network) or a second network 199. It is possible to communicate with at least one of the electronic device 104 or the server 108 through (e.g., a long-distance wireless communication network). According to one embodiment, the electronic device 101 may communicate with the electronic device 104 through the server 108. According to one embodiment, the electronic device 101 includes a processor 120, a memory 130, an input module 150, an audio output module 155, a display module 160, an audio module 170, and a sensor module ( 176), interface 177, connection terminal 178, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, subscriber identification module 196 , or may include an antenna module 197. In some embodiments, at least one of these components (eg, the connection terminal 178) may be omitted or one or more other components may be added to the electronic device 101. In some embodiments, some of these components (e.g., sensor module 176, camera module 180, or antenna module 197) are integrated into one component (e.g., display module 160). It can be. The electronic device 101 may also be referred to as a client, terminal, or peer.

The processor 120, for example, executes software (e.g., program 140) to operate at least one other component (e.g., hardware or software component) of the electronic device 101 connected to the processor 120. It can be controlled and various data processing or calculations can be performed. According to one embodiment, as at least part of data processing or computation, the processor 120 stores commands or data received from another component (e.g., sensor module 176 or communication module 190) in volatile memory 132. The commands or data stored in the volatile memory 132 can be processed, and the resulting data can be stored in the non-volatile memory 134. According to one embodiment, the processor 120 includes a main processor 121 (e.g., a central processing unit or an application processor) or an auxiliary processor 123 that can operate independently or together (e.g., a graphics processing unit, a neural network processing unit ( It may include a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor). For example, if the electronic device 101 includes a main processor 121 and a secondary processor 123, the secondary processor 123 may be set to use lower power than the main processor 121 or be specialized for a designated function. You can. The auxiliary processor 123 may be implemented separately from the main processor 121 or as part of it.

The auxiliary processor 123 may, for example, act on behalf of the main processor 121 while the main processor 121 is in an inactive (e.g., sleep) state, or while the main processor 121 is in an active (e.g., application execution) state. ), together with the main processor 121, at least one of the components of the electronic device 101 (e.g., the display module 160, the sensor module 176, or the communication module 190) At least some of the functions or states related to can be controlled. According to one embodiment, co-processor 123 (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module 180 or communication module 190). there is. According to one embodiment, the auxiliary processor 123 (eg, neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models.

Artificial intelligence models can be created through machine learning. For example, such learning may be performed in the electronic device 101 itself on which the artificial intelligence model is performed, or may be performed through a separate server (e.g., server 108). Learning algorithms may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but It is not limited. An artificial intelligence model may include multiple artificial neural network layers. Artificial neural networks include deep neural network (DNN), convolutional neural network (CNN), recurrent neural network (RNN), restricted boltzmann machine (RBM), belief deep network (DBN), bidirectional recurrent deep neural network (BRDNN), It may be one of deep Q-networks or a combination of two or more of the above, but is not limited to the examples described above. In addition to hardware structures, artificial intelligence models may additionally or alternatively include software structures.

The memory 130 may store various data used by at least one component (eg, the processor 120 or the sensor module 176) of the electronic device 101. Data may include, for example, input data or output data for software (e.g., program 140) and instructions related thereto. Memory 130 may include volatile memory 132 or non-volatile memory 134.

The program 140 may be stored as software in the memory 130 and may include, for example, an operating system 142, middleware 144, or application 146.

The input module 150 may receive commands or data to be used in a component of the electronic device 101 (e.g., the processor 120) from outside the electronic device 101 (e.g., a user). The input module 150 may include, for example, a microphone, mouse, keyboard, keys (eg, buttons), or digital pen (eg, stylus pen).

The sound output module 155 may output sound signals to the outside of the electronic device 101. The sound output module 155 may include, for example, a speaker or a receiver. Speakers can be used for general purposes such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part of it.

The display module 160 can visually provide information to the outside of the electronic device 101 (eg, a user). The display module 160 may include, for example, a display, a hologram device, or a projector, and a control circuit for controlling the device. According to one embodiment, the display module 160 may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of force generated by the touch.

The audio module 170 can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module 170 acquires sound through the input module 150, the sound output module 155, or an external electronic device (e.g., directly or wirelessly connected to the electronic device 101). Sound may be output through the electronic device 102 (e.g., speaker or headphone).

The sensor module 176 detects the operating state (e.g., power or temperature) of the electronic device 101 or the external environmental state (e.g., user state) and generates an electrical signal or data value corresponding to the detected state. can do. According to one embodiment, the sensor module 176 includes, for example, a gesture sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, It may include a temperature sensor, humidity sensor, or light sensor.

The interface 177 may support one or more designated protocols that can be used to connect the electronic device 101 directly or wirelessly with an external electronic device (eg, the electronic device 102). According to one embodiment, the interface 177 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal 178 may include a connector through which the electronic device 101 can be physically connected to an external electronic device (eg, the electronic device 102). According to one embodiment, the connection terminal 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

The haptic module 179 can convert electrical signals into mechanical stimulation (e.g., vibration or movement) or electrical stimulation that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module 180 can capture still images and videos. According to one embodiment, the camera module 180 may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module 188 can manage power supplied to the electronic device 101. According to one embodiment, the power management module 188 may be implemented as at least a part of, for example, a power management integrated circuit (PMIC).

The battery 189 may supply power to at least one component of the electronic device 101. According to one embodiment, the battery 189 may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

Communication module 190 is configured to provide a direct (e.g., wired) communication channel or wireless communication channel between electronic device 101 and an external electronic device (e.g., electronic device 102, electronic device 104, or server 108). It can support establishment and communication through established communication channels. Communication module 190 operates independently of processor 120 (e.g., an application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module 190 is a wireless communication module 192 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (e.g., : LAN (local area network) communication module, or power line communication module) may be included. Among these communication modules, the corresponding communication module is a first network 198 (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network 199 (e.g., legacy It may communicate with an external electronic device 104 through a telecommunication network such as a cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., LAN or WAN). These various types of communication modules may be integrated into one component (e.g., a single chip) or may be implemented as a plurality of separate components (e.g., multiple chips). The wireless communication module 192 uses subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module 196 within a communication network such as the first network 198 or the second network 199. The electronic device 101 can be confirmed or authenticated.

The wireless communication module 192 may support 5G networks after 4G networks and next-generation communication technologies, for example, NR access technology (new radio access technology). NR access technology provides high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and access to multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low latency). -latency communications)) can be supported. The wireless communication module 192 may support high frequency bands (eg, mmWave bands), for example, to achieve high data rates. The wireless communication module 192 uses various technologies to secure performance in high frequency bands, for example, beamforming, massive array multiple-input and multiple-output (MIMO), and full-dimensional multiplexing. It can support technologies such as input/output (FD-MIMO: full dimensional MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module 192 may support various requirements specified in the electronic device 101, an external electronic device (e.g., electronic device 104), or a network system (e.g., second network 199). According to one embodiment, the wireless communication module 192 supports Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mmTC, or U-plane latency (e.g., 164 dB or less) for realizing URLLC. Example: Downlink (DL) and uplink (UL) each of 0.5 ms or less, or round trip 1 ms or less) can be supported.

The antenna module 197 may transmit or receive signals or power to or from the outside (eg, an external electronic device). According to one embodiment, the antenna module 197 may include an antenna including a radiator made of a conductor or a conductive pattern formed on a substrate (eg, PCB). According to one embodiment, the antenna module 197 may include a plurality of antennas (eg, an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network such as the first network 198 or the second network 199 is connected to the plurality of antennas by, for example, the communication module 190. can be selected Signals or power may be transmitted or received between the communication module 190 and an external electronic device through the at least one selected antenna. According to some embodiments, in addition to the radiator, other components (eg, radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module 197.

According to various embodiments, the antenna module 197 may form a mmWave antenna module. According to one embodiment, a mmWave antenna module includes: a printed circuit board, an RFIC disposed on or adjacent to a first side (e.g., bottom side) of the printed circuit board and capable of supporting a designated high frequency band (e.g., mmWave band); And a plurality of antennas (e.g., array antennas) disposed on or adjacent to the second side (e.g., top or side) of the printed circuit board and capable of transmitting or receiving signals in the designated high frequency band. can do.

At least some of the components are connected to each other through a communication method between peripheral devices (e.g., bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and signal ( (e.g. commands or data) can be exchanged with each other.

According to one embodiment, commands or data may be transmitted or received between the electronic device 101 and the external electronic device 104 through the server 108 connected to the second network 199. Each of the external electronic devices 102 or 104 may be of the same or different type as the electronic device 101. According to one embodiment, all or part of the operations performed in the electronic device 101 may be executed in one or more of the external electronic devices 102, 104, or 108. For example, when the electronic device 101 needs to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device 101 may perform the function or service instead of executing the function or service on its own. Alternatively, or additionally, one or more external electronic devices may be requested to perform at least part of the function or service. One or more external electronic devices that have received the request may execute at least part of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device 101. The electronic device 101 may process the result as is or additionally and provide it as at least part of a response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology can be used. The electronic device 101 may provide an ultra-low latency service using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device 104 may include an Internet of Things (IoT) device. Server 108 may be an intelligent server using machine learning and/or neural networks. According to one embodiment, the external electronic device 104 or server 108 may be included in the second network 199. The electronic device 101 may be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

The server 108 is connected to the electronic device 101 and can provide services to the connected electronic device 101. In addition, the server 108 may perform a membership registration process, store and manage various information of users who have registered as members, and provide various purchase and payment functions related to the service. Additionally, the server 108 may share execution data of service applications running on each of the plurality of electronic devices 101 in real time so that services can be shared between users. This server 108 may have the same hardware configuration as a typical web server or service server. However, in terms of software, it may be implemented through any language such as C, C++, Java, Python, Golang, and Kotlin and may include program modules that perform various functions. In addition, the server 108 is generally connected to an unspecified number of clients and/or other servers through an open computer network such as the Internet, and receives work performance requests from clients or other servers and derives and provides work results in response. It refers to a computer system and the computer software (server program) installed for it. In addition, in addition to the server program described above, the server 108 includes a series of application programs running on the server 108 and, in some cases, various databases (DBs) built internally or externally, hereinafter " It should be understood as a broad concept including “DB”). Accordingly, the server 108 classifies membership registration information and various information and data about games, stores them in a DB, and manages this DB, which may be implemented inside or outside the server 108. In addition, the server 108 can be implemented using a variety of server programs provided on general server hardware and operating systems such as Windows, Linux, UNIX, and Macintosh. , Representative examples include IIS (Internet Information Server) used in a Windows environment and CERN, NCSA, APPACH, and TOMCAT used in a Unix environment, etc., to implement web services. Additionally, the server 108 may be linked with an authentication system and payment system for user authentication of the service or payment for purchases related to the service.

The first network 198 and the second network 199 are a connection structure that allows information exchange between each node, such as terminals and servers, or a network connecting the server 108 and the electronic devices 101 and 104. It means (Network). The first network 198 and the second network 199 are the Internet, LAN (Local Area Network), Wireless LAN (Wireless Local Area Network), WAN (Wide Area Network), PAN (Personal Area Network), and 3G. , 4G, LTE, 5G, Wi-Fi, etc., but are not limited to these. The first network 198 and the second network 199 may be closed, such as a LAN or WAN, but are preferably open, such as the Internet. The Internet includes protocols such as TCP/IP protocol, TCP, and UDP (user datagram protocol), as well as various services that exist at the upper layer, such as HTTP (HyperText Transfer Protocol), Telnet, FTP (File Transfer Protocol), and DNS (Domain Name System). ), a worldwide open computer primary network (198) and secondary network (199) that provides Simple Mail Transfer Protocol (SMTP), Simple Network Management Protocol (SNMP), Network File Service (NFS), and Network Information Service (NIS). ) refers to the structure.

A database can have a general data structure implemented in the storage space (hard disk or memory) of a computer system using a database management program (DBMS). A database may have a data storage format that allows for free search (extraction), deletion, editing, addition, etc. of data. Databases are relational database management systems (RDBMS) such as Oracle, Infomix, Sybase, and DB2, or object-oriented database management such as Gemston, Orion, and O2. It can be implemented according to the purpose of an embodiment of the present disclosure using a system (OODBMS) and an XML native database such as Excelon, Tamino, and Sekaiju, and has its own functions. To achieve this, you can have appropriate fields or elements.

Figure 2 is a diagram showing the configuration of a program according to one embodiment.

Figure 2 is a block diagram 200 illustrating program 140 according to various embodiments. According to one embodiment, the program 140 includes an operating system 142, middleware 144, or an application 146 executable on the operating system 142 for controlling one or more resources of the electronic device 101. It can be included. Operating system 142 may include, for example, AndroidTM, iOSTM, WindowsTM, SymbianTM, TizenTM, or BadaTM. At least some of the programs 140 are preloaded into the electronic device 101, for example, at the time of manufacture, or are stored in an external electronic device (e.g., the electronic device 102 or 104, or a server) when used by a user. It can be downloaded or updated from 108)). All or part of the program 140 may include a neural network.

The operating system 142 may control management (eg, allocation or retrieval) of one or more system resources (eg, process, memory, or power) of the electronic device 101 . Operating system 142 may additionally or alternatively operate on other hardware devices of electronic device 101, such as input module 150, audio output module 155, display module 160, and audio module 170. , sensor module 176, interface 177, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, subscriber identification module 196, or It may include one or more driver programs for driving the antenna module 197.

The middleware 144 may provide various functions to the application 146 so that functions or information provided from one or more resources of the electronic device 101 can be used by the application 146. The middleware 144 includes, for example, an application manager 201, a window manager 203, a multimedia manager 205, a resource manager 207, a power manager 209, a database manager 211, and a package manager 213. ), connectivity manager (215), notification manager (217), location manager (219), graphics manager (221), security manager (223), call manager (225), or voice recognition manager (227). You can.

The application manager 201 may, for example, manage the life cycle of the application 146. The window manager 203 may, for example, manage one or more GUI resources used on the screen. For example, the multimedia manager 205 identifies one or more formats required for playing media files, and encodes or decodes the corresponding media file using a codec suitable for the selected format. It can be done. The resource manager 207 may, for example, manage the source code of the application 146 or the memory space of the memory 130. The power manager 209 manages, for example, the capacity, temperature, or power of the battery 189, and may use this information to determine or provide related information necessary for the operation of the electronic device 101. . According to one embodiment, the power manager 209 may interface with a basic input/output system (BIOS) (not shown) of the electronic device 101.

Database manager 211 may create, search, or change a database to be used by application 146, for example. The package manager 213 may, for example, manage the installation or update of applications distributed in the form of package files. The connectivity manager 215 may manage, for example, a wireless connection or direct connection between the electronic device 101 and an external electronic device. For example, the notification manager 217 may provide a function for notifying the user of the occurrence of a designated event (eg, an incoming call, message, or alarm). The location manager 219 may, for example, manage location information of the electronic device 101. The graphics manager 221 may, for example, manage one or more graphic effects to be provided to the user or a user interface related thereto.

Security manager 223 may provide, for example, system security or user authentication. The telephony manager 225 may manage, for example, a voice call function or a video call function provided by the electronic device 101. For example, the voice recognition manager 227 transmits the user's voice data to the server 108 and provides a command corresponding to a function to be performed in the electronic device 101 based at least in part on the voice data, Alternatively, text data converted based at least in part on the voice data may be received from the server 108. According to one embodiment, the middleware 244 may dynamically delete some existing components or add new components. According to one embodiment, at least a portion of the middleware 144 may be included as part of the operating system 142 or may be implemented as separate software different from the operating system 142.

The application 146 includes, for example, home 251, dialer 253, SMS/MMS (255), instant message (IM) 257, browser 259, camera 261, and alarm 263. , Contacts (265), Voice Recognition (267), Email (269), Calendar (271), Media Player (273), Album (275), Watch (277), Health (279) (such as exercise amount or blood sugar) It may include applications that measure biometric information) or environmental information 281 (e.g., measure atmospheric pressure, humidity, or temperature information). According to one embodiment, the application 146 may further include an information exchange application (not shown) that can support information exchange between the electronic device 101 and an external electronic device. The information exchange application may include, for example, a notification relay application configured to deliver designated information (e.g., calls, messages, or alarms) to an external electronic device, or a device management application configured to manage the external electronic device. there is. The notification relay application, for example, transmits notification information corresponding to a specified event (e.g., mail reception) generated in another application (e.g., email application 269) of the electronic device 101 to an external electronic device. You can. Additionally or alternatively, the notification relay application may receive notification information from an external electronic device and provide it to the user of the electronic device 101.

The device management application, for example, controls the power (e.g., turn-on or turn-off) of an external electronic device or some component thereof (e.g., a display module or camera module of the external electronic device) that communicates with the electronic device 101. ) or functions (such as brightness, resolution, or focus). A device management application may additionally or alternatively support installation, deletion, or update of applications running on external electronic devices.

Throughout this specification, neural network, neural network, and network function may be used with the same meaning. A neural network may consist of a set of interconnected computational units, which may generally be referred to as “nodes.” These “nodes” may also be referred to as “neurons.” A neural network is composed of at least two or more nodes. Nodes (or neurons) that make up neural networks may be interconnected by one or more “links.”

Within a neural network, two or more nodes connected through a link can relatively form a relationship as an input node and an output node. The concepts of input node and output node are relative, and any node in an output node relationship with one node may be in an input node relationship with another node, and vice versa. As described above, input node to output node relationships can be created around links. One or more output nodes can be connected to one input node through a link, and vice versa.

In a relationship between an input node and an output node connected through one link, the value of the output node may be determined based on data input to the input node. Here, nodes connecting the input node and the output node may have weights. Weights may be variable and may be varied by a user or algorithm in order for the neural network to perform a desired function. Here, the edges or links that interconnect the input nodes and output nodes have weights that can be variably applied by the user or algorithm to perform the function desired by the neural network. For example, when one or more input nodes are connected to one output node by respective links, the output node is set to the values input to the input nodes connected to the output node and the links corresponding to each input node. The output node value can be determined based on the weight.

As described above, in a neural network, two or more nodes are interconnected through one or more links to form an input node and output node relationship within the neural network. The characteristics of the neural network may be determined according to the number of nodes and links in the neural network, the correlation between the nodes and links, and the value of the weight assigned to each link. For example, if there are two neural networks with the same number of nodes and links and different weight values between the links, the two neural networks may be recognized as different from each other.

Meanwhile, in various embodiments of the present disclosure, the server may be a server that provides a multimodal-based English learning service to a user terminal using a neural network. For example, the server may include server 108 of FIG. 1 .

For example, the server can determine a method of proceeding for the user terminal through a neural network in order to provide a multimodal-based English learning service suitable for the user. For example, the server may provide multimodal-based English word learning content to the user terminal depending on the progress method. For example, the server may reflect the progress results of multimodal-based English word learning content from the user terminal and determine a new progress method for the user terminal through a neural network.

The user terminal may be a terminal for learners who use a multimodal-based English learning service. For example, the user terminal may include the electronic device 101 of FIG. 1 .

Hereinafter, a multimodal-based English learning service will be described.

Multimodal-based English learning services may include an English teaching method based on native language acquisition, also known as the mother's English teaching method. This native language acquisition method of English education refers to an educational method that matches the process of learning Korean since Koreans were born with the process of learning English and teaches them to naturally acquire English as a language as if learning their native language. Therefore, it may be logically contradictory to say that English education based on native language acquisition is based on reading or based on the grammar system.

The process of acquiring one's native language can follow the following process. This process can occur almost identically when learning one's native language, including Korean, English, Japanese, Chinese, and most other languages. However, only the terms used are different, and when learning one's native language, there are three stages: Pre-Linguistic stage, First-word infant syntax stage, Multi-words infant syntax stage, It goes through the language system acquisition stage, language settlement stage, and language learning start stage - school age.

In the pre-linguistic stage, learners only hear sounds and may not recognize human sounds as meaningful speech sounds. Learners (e.g. babies) themselves begin to make sounds other than crying, and at this time, the baby tries to imitate the sounds they have heard a lot. This is called babbling, and the term grounding is used for this process in the English education method of learning the native language. The prelingual stage is a period in which the child tries to imitate sounds and can repeat watching and listening indefinitely.

The first-word infant syntax stage is the time when proper speech begins. Typically, a learner may start an utterance with the words “mom” and “dad.” First-time learners start with “~~ma” and “~~ppa” and gradually become able to pronounce mom and dad clearly. There is research showing that learners hear the words approximately 10,000 times before they can pronounce them and understand their meaning, and the pronunciations of children around the world at this age are similar. Korean has a sound form that develops from mom and dad, English has mama and papa, Japanese has ママ、パパ (mama, papa), and Chinese both have sound forms that develop from “um~~ma” and “um~~pa.” Mainly, in the single-word language stage, learners pronounce the titles of people close to them or exclamations as one word.

The multi-word infant syntax stage is a period in which words are learned by grouping them together. Learners begin to express more and more complex thoughts by adding simple meaningful words such as "Mom, eat" and "Dad, let's play" to "Mom, I'm hungry, feed me" and "Dad, let's play together." From this point on, it is the period when the learner's language skills increase in earnest, and is also called the immersion speaking period.

The language system acquisition stage is a period in which the learner's verbal communication gradually develops into reading. Reading can begin with picture books read by parents and gradually progress to storybooks. At this stage, you can read whole words when reading a book. The most important thing about this period is that the learner has become quite proficient at communicating with parents verbally. Phonics is not used or may not be taught in early book reading. Additionally, in the United States, the activity of reading books to children at bedtime, referred to as bedtime stories, is considered very important. Through this, along with the role of nurturing children's creativity and imagination, children can become interested in reading and become motivated to approach books. If learners go through this period of bedtime story well, they fall in love with reading, and phonics is taught from the basics at the end when the desire to read books on their own becomes stronger. Therefore, the most important part of this period is to make reading books fun for children. In order to read books, you need training to read words. Names are written on paper and pasted on televisions, refrigerators, washing machines, desks, etc., and children are taught to read the words. This method can be used in the same way in both Korea and English-speaking countries.

The language settlement stage may be the time when the learner writes a diary. Learners at this age have natural verbal communication and can read basic books. This is the time when learners find out words they do not know by asking them, and they gradually begin to write. Writing progresses first through the stage of writing letters and then into the stage of writing sentences. The step of writing a sentence also progresses through copying and writing on your own. During this period, learners begin to write diaries, which in Korea is called a diary, but in the United States it is usually expressed as a journal. The language settlement period shows some differences between Korea and the English-speaking world. The difference in meaning of terms is also reflected in the actual writing instruction method. In other words, in Korea, the terms diary (a record of each day) and diary (a record of each day) are used, and learners' early writing is done by recording their daily lives, and whether children's diaries are written every day at school. Check . In the English-speaking world, as the term journal (recording opinions on a specific topic) suggests, when something special happens that is different from usual, writing is taught on that topic. Daily life is not the subject of journal writing. However, in that writing is taught in earnest during the language settlement stage, it is along the same line as the process of learning one's native language.

At the beginning of language learning (school age), the concept of learning is incorporated in earnest. Learning refers to learning and mastering something consciously or intentionally. Learning in the process of acquiring one's native language can lead to learning the grammatical system of one's native language in earnest.

There are several things to consider as the mother tongue acquisition process progresses. At the time of birth, a child does not have the ability to understand and recognize objects. Children begin to learn their native language in that state, but in the case of children who encounter English in an environment such as Korea, they are already quite capable of communicating in Korean and have formed an understanding of corresponding objects and cognitive abilities. Therefore, from the point when meaningless sounds such as noise or sound begin to sound meaningful, the process of learning words can be done in parallel with listening to sounds. In other words, word acquisition progresses from the single-word language stage to the language system acquisition stage. However, children are not forced to read books on their own during this period. The main purpose of learning vocabulary during this period is to understand the meaning of words heard by sound rather than reading books. Reading at this time should not be about forcing children to read on their own, but rather should be about reading to them, and when reading to them, children should be able to understand much of the content of the book just by hearing what they hear. According to several studies, children can become more comfortable reading books when they select books in which they can hear and understand more than 90% of the words. Therefore, during the time of word learning, book reading, rather than reading, should be carried out, and this can be understood as the same as the bedtime story activity in English-speaking countries. The claim that just because you can understand sounds if you listen to them a lot, you can understand texts if you read them unconditionally is a false claim. Even when there was no writing, communication was possible through words. Even people who cannot read can communicate verbally. Even animals that cannot read or write can at least communicate using sounds. Sounds and images of objects exist naturally and are natural abilities that exist naturally as part of nature, while letters are functional inventions intentionally created by humans and must be learned through a separate learning process. am. For this reason, the letters written in books can only be understood and written after intentionally learning the words. Therefore, rather than having children read a lot, we should enable children to match sounds and words by reading a lot, and after going through that process, we should be able to access reading.

In the end, expressions such as the native language acquisition method based on reading and the native language acquisition method based on learning the grammar system are incorrect expressions that can be used because the principles of native language acquisition itself are not understood. The reason these expressions are used is because of the vicarious satisfaction provided by terms such as reading and grammar. Parents believe that reading or learning grammar is the absolute best way to improve children's grades. Therefore, in order to deceive parents, it is powerful to say that children will learn reading or grammar. Therefore, even though they are completely different approaches, the expressions 'English acquisition method through native language acquisition method' and 'Education method based on reading/grammar' are used together, and even though these words do not make logical sense, parents are deceived. What is accurate is that reading must be treated as important during the reading stage of acquiring one's native language. In other words, when it comes to language acquisition through the native language acquisition method, the correct perspective on reading is “reading in the process of native language acquisition,” rather than “a native language acquisition method based on reading.”

Online English education services that generally start with learning English words mostly start with phonics. In other cases, it is mostly a simple composition of word-picture-Korean meaning. Most of these services focus on how to memorize better. In other words, it does not matter what meaning the word has in the sentence, and we try to match the English word and the Korean interpretation 1:1 to enable memorization quickly and easily.

The multimodal-based English learning service of this disclosure is a service that focuses on “natural memorization methods.” The method of memorizing well is easy to remember and forget easily, and it is difficult to respond to even the slightest change in meaning, whereas the method of memorizing naturally remembers the meaning by inferring it through association, so once it is remembered, it is difficult to forget, and the meaning changes in various ways. It can also be easily applied.

An idiomatic expression is an expression in which several words have a meaning different from that of the original words in a specific situation. These idiomatic expressions are usually learned through memorization. However, idiomatic expressions are sometimes used while retaining the original meaning of the words. As a simple example, big fish is an expression meaning a boss or leader. However, in a sentence it can actually mean a large fish. If you have memorized this expression as a boss, the meaning of the sentence becomes completely strange. In other words, learning through memorization has limitations in understanding the meaning of actual spoken words or writing.

Meanwhile, in the case of phonics, which aims to improve word reading skills by practicing pronunciation through the phonetic sounds of English letters, children in countries where English is their native language read books while their listening and speaking skills have been developed. Since it was developed as an educational method to apply, it is correct to apply it from the beginning to children in ESL (English as a Second Language) or EFL (English as a Foreign Language) environments who have not developed the ability to listen and speak in English. No. The reason is that children who do not have English listening and speaking skills are taught to memorize letters, the sounds they make, and practice reading the words in books out loud. This is in line with the original purpose of phonics education. does not match However, in most private education institutions, phonics is the gateway to learning English, and speaking begins with phonics, and most children's English education sites provided through the Internet also use phonics. ) and starts with This teaching method has problems in three major aspects. First, phonics was not initially developed as a pronunciation training method to practice speaking, but to teach how to pronounce unfamiliar words when reading a book. In other words, a program developed to read books aloud was wrongly transformed into a program to improve basic skills needed for English speaking training. Second, the educational stage that children whose native language is English undergoes after their listening and speaking skills are developed is phonics. This is because it is applied through indoctrination to children whose native language is not English and whose listening and speaking skills have not been developed. It is difficult to see the effect of education in the first place. Third, looking at the process of learning one's native language, it does not start with learning the phonetic value of each letter like phonics. This is common to all countries. For example, looking at the word 'apple', in Korea, we teach how to read the shape of the letter written 'apple' as a whole. Then, after the reading ability is formed to a certain extent, 'sap' is broken down into 'ㅅ(su), ㅏ(ah), ㄱ(he), ㅗ(oh), ㅏ(ah)', and 'suagwowa'. Teach them that if you pronounce it quickly, it becomes ‘apple’. This course focuses on the period corresponding to school age in the process of acquiring the native language. The same goes for English. First, learn the word 'apple' itself. Afterwards, when children reach school age, they learn the sounds of the letters 'a (ae), pp (ppu), l (ㄹ), e (silent sound)' and are taught how to pronounce them. In other words, according to the native language acquisition procedure for both Korean and English, the process of teaching is taught by connecting the 'form' of the word itself and its 'meaning'. This is called the Whole Word Approach, which is part of the Whole Language Approach. Therefore, it is not right to start with a program that practices phonics while advocating English education using the method of learning one's native language.

It is known that even if you learn phonics, you can only read 80-85% of words. This means that about one word out of five-word sentences cannot be read according to the principles of phonics. Some claim that as grades go up, less than 50% of words can be read using phonics. In addition, there are an increasing number of cases where loan words originating from languages such as French and German have become established as English words. Except for Spanish, which is pronounced exactly as it is spelled, English words derived from many languages are most often not pronounced according to the principle of phonics. This means that there are more and more words that can be pronounced by learning phonics.

When learning the meaning of words, the efficiency of writing English words and memorizing their meanings as in the past is very low. According to the forgetting curve proposed by German psychologist Hermann Ebbinghaus, our memory for newly learned information declines rapidly over time, with up to 80% of the information being forgotten within the first 24 hours. As proof of this, efforts are made to increase the efficiency of memory by repeating learning again before being forgotten.

A theory to overcome this is the Dual Coding Theory proposed by Allan Paivio in 1971. The theory is that the learning effect is much higher when the left brain, which is responsible for the verbal area such as letters and words, and the right brain, which is responsible for the non-verbal area such as images and emotions, are utilized simultaneously. The theory is that an image is defined as another code, and that the learning effect is high when these two codes are combined. For example, memorizing words (characters) and their meaning (characters) while mumbling (speaking) is a learning method that uses only one code, a linguistic element. When learning together using words (characters), which are linguistic elements, and images, which are non-verbal elements that directly or indirectly explain their meaning, the learning effect and lasting effect are high. According to a study by Chun and Plass (1996), learners who studied words with pictures had better word memory than learners who studied words without pictures.

An extension of the Dual Coding Theory is the Multimodal Learning Theory. This theory is that if information is provided through various sensory channels such as kinesthetics in addition to the visual and auditory elements responsible for verbal/non-verbal codes, learners will understand and remember it more easily. The kinesthetic sense mentioned in this theory does not mean watching videos (audiovisual), but rather includes the learner's individual moving activities. In other words, it refers to direct activities that involve moving the body together, such as imitating actions while watching a favorite video or imitating songs and movements. Multimodal learning is being actively studied by various researchers in the field of artificial intelligence based on human behavior recognition and emotion recognition, and is equally applied to children's learning.

A study published in the Journal of Science Education and Technology (2009), “The effects of animated and static visuals on students' understanding and recall of chemical experiments,” found that animated videos depicting scientific experiments were more effective than static images or We found that it was more effective in improving students' understanding and memory of the experiment compared to text-based explanations.

Another reference theory is the “Cognitive Load Theory” proposed by John Sweller. According to this theory, people have limited working memory capacity, so presenting too much information at once can overload memory and interfere with learning. Short videos reduce cognitive load and allow learners to process information more easily. It is explained that it is more effective in learning because it can be maintained. The results of this study show that short videos can lead to better learning outcomes compared to longer videos, especially in terms of information retention and engagement.

Figure 3 shows a method in which a server provides a multimodal-based English learning service to a user terminal using a neural network according to one embodiment. Figure 4 is an example of multimodal-based English word learning content according to an embodiment. Figure 5 is an example of a settings screen for multimodal-based English word learning content according to an embodiment. Figure 6 is an example of a reward screen of multimodal-based English word learning content according to an embodiment. Figure 7 is an example of a team-based game of multimodal-based English word learning content according to an embodiment. The embodiments of FIGS. 3 to 7 can be combined with various embodiments of the present disclosure.

Referring to FIG. 3, in step S310, the server may receive a first request message for English word learning from the user terminal.

The first request message is a message that the user terminal requests the server to receive multimodal-based English word learning content. For example, the first request message may include personal information about the user and information about the user terminal. Personal information about a user may include the user's name, the user's date of birth, and the user's gender. Information about the user terminal may include identification information about the terminal, information about the type of terminal, and information about the capacity of the terminal.

For example, the identification information for the terminal may include at least one of an identifier (ID) of the user terminal or an international mobile equipment identity of the user terminal. The ID of the user terminal may be the ID that subscribed to the English learning service. The international mobile phone identification number is a 15-digit number assigned by the manufacturer when manufacturing the terminal, and may consist of the certification authority's unique number, terminal manufacturer, model name, and terminal serial number.

For example, information about the terminal model is information indicating the terminal model and may include a value for the product name of the terminal.

For example, information about the capacity of the terminal is information about the space where data can be stored in the terminal and may include the current storage capacity of the terminal.

Referring to FIG. 4, the screen 400 of the multimodal-based English word learning content includes an alphabet thumbnail menu 410, an English word 420, an English word definition 430, an output button 440, an animation 450, and a reward. It may include an area 460 and a menu bar 470.

The alphabet thumbnail menu 410 may be a menu that categorizes English words starting with each alphabet (eg, A to Z). The output button 440 may be a button that outputs a recorded voice for at least one of the English word 420 or the English word definition 430. For example, the English word definition 430 is an area where the definition of the English word 420 is written in English, and the hypertext 431, which displays important words in different colors, and a highlight indicating the text in which the recorded voice is output. It may include (432). The animation 450 may be an area where a dynamic image explaining the English word 420 is displayed. For example, when animation 450 is activated, the image may move for a certain period of time. At this time, the specific time may be different for each animation 450 matched to the English word 420, and the specific time may be within the range of 3 seconds or more and 7 seconds or less. For example, when the animation 450 is inactive, the image may be displayed in a stopped state. At this time, the image displayed in the stopped state may be referred to as a representative image. The reward area 460 may be an area that displays the reward that the user terminal obtains according to the learning progress rate for the corresponding English word. For example, the reward may be a sticker expressing an English word 420, and the act of obtaining the reward may be referred to as a sticker hunt. The menu bar 470 may be an area containing a menu that can control multimodal-based English word learning content. For example, the menu bar 470 may include a button to proceed to the previous step, a button to proceed to the next step, a record/play button, a settings button, a home button, and an end button.

Additionally, for example, in multimodal-based English word learning content, the order in which English word learning progresses may be selected by the user terminal. The progress order of English word learning may include at least one of the order of frequency of English words, alphabet order, or classification by subject. For example, when the server collects a plurality of English words from source information such as English subtitles of movies, English textbooks, and English children's books, it can count the number of times each English word is exposed. The server determines the number of exposed English words as the frequency of the English word, and determines the order of English word learning in the order of the frequency of the English word. For example, the frequency of each English word may be displayed in multimodal-based English word learning content. For example, the alphabetical order may be from A to Z. For example, if both the frequency order and the alphabet order of English words are selected, the server can proceed with learning in alphabetical order from A to Z, but can proceed with learning English words by alphabet in the order of high frequency of English words. . For example, if classification by topic is selected, you can proceed with learning about English words classified by home, school, vehicle, etc. You can select classification by topic, frequency order, or alphabetical order.

Referring to Figure 5, settings 500 for multimodal-based English word learning content include basic settings 510, progress method 520, learning time 530, language pronunciation 540, and detailed settings 550. It can be included.

For example, the user terminal can turn on or off each function of auto-play, text highlighting, repeated reading, user interface sound, automatic page turning, and dictation for multimodal-based English word learning content through the default settings 510. You can. For example, the user terminal can automatically proceed with each learning step of multimodal-based English word learning content by setting auto play to on. For example, the user terminal sets text highlighting to on, so that the highlight 432 for the definition of an English word in multimodal-based English word learning content can be displayed in accordance with the voice for the definition of the English word. For example, the user terminal can set repeated reading to on, so that the English word itself or the voice for the definition of the English word can be played repeatedly. For example, the user terminal sets the user interface sound to on, so that the sound set in the user interface can be output for a preset interaction. For example, by setting automatic page turning to on, the user terminal can automatically move on to the next English word after all learning steps for one English word are completed. For example, by setting dictation to on, the user terminal may display a window for entering the spelling or definition of an English word in multimodal-based English word learning content.

For example, the user terminal can turn on or off the fast progress, basic progress, and full progress functions for multimodal-based English word learning content through the preference method 520. For example, fast progress can be the fast progress of the flash technique, which only looks at the image of the word and animation and then moves on to the next word. To progress quickly, you can choose between seeing the word, listening to the sound of the word, watching the animation video, only viewing the animation image, seeing the definition, or listening to the sound of the definition.

For example, the basic process is set to look at the word, listen to the sound of the word, watch the animation, look at the definition, and listen to the sound of the definition.

For example, the entire process involves seeing the word, hearing the sound of the word, recording and playing the word, viewing the animation, seeing the definition, hearing the sound of the definition, and recording and playing the definition. It is set, and additional copying can be optionally performed.

For example, the user terminal can check the learning time to date through the learning time 530.

For example, the user terminal can set the voice played through the language pronunciation 540 to either English pronunciation or British pronunciation.

For example, the user terminal can select either automatic progress or manual progress through detailed settings 550. For example, when manual progress is selected, the user terminal can set each of the English word, definition of the English word, and animation operation through detailed settings 550.

In step S320, the server may determine proceeding method information for the user terminal through a method decision model using the first neural network based on personal information about the user and information about the user terminal.

The server may generate a user vector through data preprocessing on personal information about the user and information about the user terminal. The user vector may include a value for the user's age, a value for the user's gender, and a value related to the capacity of the user terminal.

The value for the user's age is a value representing the user's age. For example, the value for the user's age may be determined based on the user's date of birth and current date. For example, if the user's date of birth is June 5, 2020, and the current date is July 17, 2023, the value for the user's age is the current year minus the year the user was born, multiplied by 12, and It can be calculated by adding the month minus the month in which the user was born, and can be determined to be 37. That is, for example, the value of the user's age may be expressed in months.

The value for the user's gender is a value representing the user's gender. For example, the value for the user's gender may have a value of 1 if the user is male and a value of 2 if the user is female.

The value related to the capacity of the user terminal is a value representing the storage capacity of the user terminal. For example, the value related to the capacity of the user terminal may be in units of megabytes.

The server may determine the average achievement vector of the group with the highest similarity to the user vector among the plurality of groups set in the server as the basic achievement vector for the user terminal. For example, the average achievement vector may be the average value of multiple conscientiousness, multiple speech rates, multiple pronunciation accuracy, and multiple performance scores included in the group with the highest similarity to the user vector.

Additionally, for example, the server multiplies values for the user's age, values for the user's gender, values related to the capacity of the user's terminal, sincerity, speech rate, pronunciation accuracy, and performance over a preset period of time. Modal-based English word learning content can be obtained from each of multiple user terminals. Here, sincerity, speech rate, pronunciation accuracy, and performance may be values output by the learning achievement evaluation model described later. That is, the server can store user vectors and learning achievement vectors for each of a plurality of user terminals that have performed a multimodal-based English learning service during a preset period. For example, the preset period may be a period of 6 months or longer. For example, the server may update multiple groups at preset intervals.

For example, the server can determine n groups through a clustering technique using a neural network based on a plurality of user vectors. Clustering may refer to unsupervised learning that groups data with similar properties into a certain number of clusters. For example, n groups can be determined through DBSCAN (Density-Based Spatial Clustering of Applications with Noise) technique based on a plurality of user vectors. For example, DBSCAN assumes that if a specific element (point) belongs to a cluster, it must be close to many other elements in the cluster, and for this calculation, the radius and minimum points are required. can be used For example, the diameter may be a radius based on a specific data element, which may be referred to as a dense area. For example, the minimum element can indicate how many elements are needed around the core point to specify it. Additionally, each element in the data set can be divided into core, border, and outlier points.

For example, the server can check the size of the diameter of each element and discover how many surrounding elements there are. Afterwards, if there are more than m elements within the diameter range, the server may determine that element to be a key element. Additionally, the server may determine an element included within the diameter range from the core element as a boundary element. Additionally, the server may determine elements that are not included within the diameter range from the core element as outlier elements, and the outlier elements may be excluded from the corresponding cluster. Additionally, the server can classify core elements into the same cluster if the distance between them is less than the diameter. Through this, the server can determine n groups classified based on age, gender, and capacity of the user terminal for a plurality of user vectors.

For example, the server may set an average achievement vector for each of the n groups by averaging the learning achievement vectors for a plurality of user terminals included in each of the n groups.

For example, the group with the highest similarity to the user terminal may be the group with the shortest distance between the user vector for the user terminal and the center vector of the group among the plurality of groups. Here, the center vector may be a vector obtained by averaging a plurality of user vectors included in the corresponding group.

For example, progress method information for the user terminal may be output based on the user vector and the basic achievement vector being input to the method decision model. Here, the progress method may be the initial progress method of multimodal-based English word learning content for the user terminal. For example, the progress method information includes the number of learning steps, the number of repetitions for each learning step, the total progress time for each learning step, and values for at least one action related to each of the English words, definitions of English words, and animations for each learning step, It may include the playback time related to the English word, the maximum recording time related to the English word at each learning stage, the playback time related to the definition of the English word at each learning stage, and the maximum recording time related to the definition of the English word at each learning stage.

The number of learning steps may be the number of learning steps to be performed for one English word in multimodal-based English word learning content. The number of repetitions at each learning stage may indicate the number of repetitions of each learning stage of multimodal-based English word learning content. The total progress time for each learning step may represent the time taken for each learning step of multimodal-based English word learning content.

At each learning stage, the value for at least one operation related to each English word, English word definition, and animation may be a value representing each operation of the English word, English word definition, and animation in multimodal-based English word learning content.

For example, the value for at least one operation related to an English word may include at least one value for recording the English word among the values for the display operation of the English word, the value for the playback operation of the English word, and the value for the reading operation of the English word. You can. For example, the value for at least one operation related to the definition of an English word may include a value for a display operation of the definition of an English word, a value for a playback operation of the definition of an English word, a value for a reading operation of the definition of an English word, and a value for the operation of reading the definition of an English word. It may include at least one of a value for a highlight display operation or a value for recording a definition of an English word. For example, the value for at least one action related to animation may include at least one of a value for a display action or a value for an activation action.

Here, the display operation may be an operation of displaying multimodal-based English word learning content. For example, if the value for the display action is 1, the action of displaying an English word or definition of an English word is instructed in multimodal-based English word learning content, and if the value for the display action is 2, a window for entering text is indicated. You can instruct an action to display .

Here, the playback operation may be an operation of playing a voice for an English word or a definition of an English word in multimodal-based English word learning content. Voices for English words or definitions of English words may be pre-recorded and pre-stored on a server. For example, the value for the playback operation may have a value of 0 or 1. For example, if the value for the playback action is 1, the action of playing back an English word or a voice for the definition of an English word can be instructed in multimodal-based English word learning content. For example, if the value for the playback operation is 0, the audio for the English word or the definition of the English word may not be played in the multimodal-based English word learning content.

Here, the reading operation may be an operation that leads to reading an English word or a definition of an English word displayed in multimodal-based English word learning content. For example, the value for a read operation may have a value of 0 or 1. That is, for example, if the value for a read operation is 1, the corresponding operation can be indicated. For example, an action to encourage reading may include an action of displaying the phrase “read” or “read” around an English word or a definition of an English word. For example, if the value for the reading operation is 0, reading of English words or definitions of English words displayed in multimodal-based English word learning content may not be encouraged.

Here, the highlight display operation may be an operation of displaying a highlight in real time for a word being played in the definition of an English word while a voice for the definition of an English word is played in multimodal-based English word learning content. For example, the value for the highlight display operation may have a value of 0 or 1. For example, if the value for the highlight display operation is 1, the corresponding operation can be indicated. For example, if the value for the highlight display operation is 0, it may indicate not performing the corresponding operation.

Here, the display operation may be an operation of displaying animation on multimodal-based English word learning content. For example, the value for display operation may have a value of 0 or 1. For example, if the value for the display operation is 1, the corresponding operation can be indicated. For example, if the value for the display operation is 0, it may indicate not performing the corresponding operation.

Here, the activation operation may be an operation of activating an animation displayed in multimodal-based English word learning content. For example, if the value for the activation motion is 1, it can indicate a motion in which the animation is moving, and if the value for the activation motion is 2, it can indicate a state in which the animation is stopped.

Here, the recording operation may be an operation of recording and listening to an English word or definition of an English word displayed in multimodal-based English word learning content. For example, the value for the recording operation may have a value of 0 or 1. That is, for example, if the value for the recording operation is 1, the corresponding operation can be indicated. For example, if the value for the recording operation is 0, it may indicate not performing the corresponding operation.

The playback time related to an English word at each learning stage may be the time required to play the English word. The maximum recording time related to an English word at each learning stage may be the maximum time allocated to the recording operation of the English word. The playback time related to the definition of an English word at each learning stage may be the time required to play the definition of the English word. The maximum recording time related to the definition of an English word at each learning stage may be the maximum time allocated to the recording operation of the definition of an English word.

For example, a first neural network may include a first input layer, one or more first hidden layers, and a first output layer. One or more first hidden layers may include one or more convolutional layers and one or more pooling layers.

For example, each user vector and basic achievement vector may be filtered in a convolutional layer, and a feature map may be formed through the convolutional layer.

For example, in a pooling layer, fixed vectors related to features are selected for dimensionality reduction based on the formed feature map, and sub-sampling is performed on the formed feature map, thereby representing a coloring pattern in the vectorized time series data. can be extracted. For example, the pooling layer may be a max pooling layer that extracts the largest value. For example, the pooling layer may be an average pooling layer that extracts average values. For example, at this time, the parameters of the first neural network may include parameters (size of feature map, size of filter, depth, stride, zero padding) related to the convolutional layer and the pooling layer.

Data for learning of each first neural network consisting of a plurality of user vectors, a plurality of learning achievement vectors, and a plurality of correct answer progress method information is input to the first input layer of the first neural network, and the one or more first neural networks 1 A first output vector is output through a hidden layer and a first output layer, the first output vector is input to a first loss function layer connected to the first output layer, and the first loss function layer is the first loss function layer. 1 A first loss value is output using a first loss function that compares the output vector and the first correct answer vector for each training data, and the parameters of the first neural network are adjusted in a direction in which the first loss value becomes smaller. It can be learned.

For example, a plurality of user vectors, a plurality of learning achievement vectors, and a plurality of correct answer progress method information may be obtained in advance by the server. For example, the learning achievement vector may include sincerity, speech rate, pronunciation accuracy, and performance. For example, one user vector and one learning achievement vector used as learning data may consist of one correct answer progress method information and one set. For example, multiple sets may be pre-stored on a server.

Through this, the server can train a method decision model to determine an appropriate learning method according to the initial information of the user terminal.

In step S33O, the server may transmit a learning start message including progress method information to the user terminal.

The learning start message may be a message that allows the user terminal to start learning about multimodal-based English word learning content.

For example, multimodal-based English word learning content displayed on the user terminal may be progressed based on progress method information. The user terminal may transmit a first request message to the server while the program for the installed multimodal-based English learning service is running. Based on the user terminal receiving a learning start message from the server, multimodal-based English word learning content may be executed according to the progress method information.

For example, multimodal-based English word learning content includes information about the basic interface, information about English words, information about the definition of English words, information about animations related to English words, multiple reward information, and information about multiple English word games. may include. Information about the basic interface may include information about the basic interface for multimodal-based English word learning content. The plurality of reward information may include information about rewards paid to the user terminal according to the learning achievement rate of multimodal-based English word learning content. The information about the plurality of English word games may include information about the plurality of English word games that the user terminal plays to learn English words. Information about English words may include a plurality of English words and audio information for each of the plurality of English words. Information about the definition of an English word may include definitions of a plurality of English words, audio information for each definition of the plurality of English words, and highlight information for each definition of the plurality of English words. Information about animations related to English words may include a plurality of animation images matching each of a plurality of English words.

In step S340, the server may receive information about the progress of multimodal-based English word learning content from the user terminal.

Information about the progress results of the multimodal-based English word learning content may include results learned by the user terminal through the multimodal-based English word learning content. For example, information about the progress of multimodal-based learning content may include the achievement rate of each learning stage for each English word and a plurality of recorded voice files for each English word. Here, the achievement rate for each learning step may be the number of repetitions included in the progress method information divided by the number of repetitions by the user terminal for each learning step. The plurality of recorded voice files for each English word may include a file recording the utterance of the English word itself and a file recording an utterance about the definition of the English word according to the recording operation performed by the user terminal in the learning stage. That is, multiple voice files can be recorded for one English word.

In step S350, the server may determine the learning achievement level for the user terminal through a learning achievement evaluation model using a second neural network based on information about the progress results of multimodal-based learning content.

For example, the server may generate a first evaluation vector composed of the achievement rate of each learning stage for each English word through data preprocessing of information on the progress results of multimodal-based learning content. For example, the achievement rate for each learning stage may have a value of 0 or more.

For example, the server may generate a plurality of second evaluation vectors and a plurality of third evaluation vectors for each English word through data preprocessing of a plurality of recorded voice files. A plurality of second evaluation vectors for one English word may be arranged in the order in which the voice file was recorded first. A plurality of third evaluation vectors for one English word may be arranged in the order in which the voice file was recorded first.

For example, the server divides the waveform of the voice file into certain frames and converts it from the time domain to the frequency domain, thereby converting it into a sum of different frequencies. For example, the server uses linear prediction coefficient (LPC) technology, which predicts the current sample value by multiplying a certain number of past sample values by a coefficient and predicting the current sample value using the sum of the coefficients, or the energy of the output that passed through each band-pass filter. Data preprocessing for voice files can be performed through various voice feature extraction technologies, such as energy technology for each frequency band that uses values as voice features.

At this time, the server preprocesses data for each voice file to determine values related to pitch, values related to amplitude, values related to voice high-pitched sound, values related to energy of phase sounds, and the energy ratio between phase sounds and noise. The value associated with can be determined. Pitch refers to the fundamental frequency, that is, F0, and can represent the periodic characteristics of the voice. Here, amplitude is a factor that determines the loudness of sound. Here, the voice high-pitched band represents the resonance of the vocal tract, and refers to lines connecting the peaks of frequency energy by converting the voice signal into the frequency domain. At this time, it is expressed in the order of F1, F2, and F3, starting from the lowest frequency peak, and generally, for vowels, high energy may appear in the F1 to F3 frequency range. Here, the phase tone is a frequency component that is a multiple of the fundamental frequency, and the phase tone energy is the energy for a frequency that is a multiple of the fundamental frequency.

For example, a value related to pitch may include the average rate of change of the pitch period. Values related to amplitude may include the average value of the rate of change of amplitude. Values related to the voice high-pitched band may include the amplitude of the voice high-pitched band and the bandwidth of the voice high-pitched band. Values related to phase sound energy may include the rate of decrease of phase sound energy as frequency increases, the ratio of phase sound energy at low frequencies to energy at high frequencies, and the amplitude of phase sound. The energy ratio of the matched sound and the noise may include the energy ratio of the matched sound and the noise for a plurality of frequency ranges. The utterance time may be the time when the user uttered the English word among the recorded times. The average number of syllables per second can be determined by dividing the total number of syllables by the utterance time for the definition of an English word.

For example, the second evaluation vector is a value related to the pitch period, a value related to the amplitude, a value related to the fundamental frequency, a value related to the energy of the consonant, a value related to the consonant and noise, for a voice file in which the user recorded the English word itself. It can be composed of a value related to the energy rate and ignition time. For example, if the English word itself is "airplane", the second evaluation vector is a value related to the pitch period, a value related to the amplitude, a value related to the fundamental frequency, and a phase for a voice file recorded by the user of "airplane". It may include values related to voice energy, values related to the energy ratio of phased sounds and noise, and speech time.

For example, the third evaluation vector is a value related to the pitch period, a value related to the amplitude, a value related to the fundamental frequency, a value related to the energy of the consonant, a value related to the consonant and the consonant, for a voice file in which the user recorded the definition of an English word. It may include values related to the energy rate of the noise and the average number of syllables per second. For example, if the English word itself is "airplane", the third evaluation vector is the definition of "airplane" for the voice file recorded by the user, a value related to the pitch period, a value related to the amplitude, and a value related to the fundamental frequency. , it may include values related to the energy of phase consonants, values related to the energy ratio of phase consonants and noise, and the average number of syllables per second.

For example, the fidelity, speech rate, and pronunciation for the user terminal are based on the first evaluation vector, the plurality of second evaluation vectors, and the plurality of third evaluation vectors being input to the learning achievement evaluation model using the second neural network. Accuracy and performance can be output for each English word.

For example, a second neural network may include a second input layer, one or more second hidden layers, and a second output layer. Data for training each second neural network consisting of a plurality of first evaluation vectors, a plurality of second evaluation vectors, a plurality of third evaluation vectors, a plurality of reference vectors, and a plurality of correct answer learning achievement vectors are the second neural network. is input to the second input layer of a network, passes through the one or more second hidden layers and a second output layer, and outputs a second output vector, and the second output vector is connected to the second output layer. is input to a function layer, and the second loss function layer outputs a second loss value using a second loss function that compares the second output vector with the second correct vector for each training data, and the second loss function layer Parameters of the neural network may be learned in a direction that reduces the second loss value.

A plurality of first evaluation vectors, a plurality of second evaluation vectors, and a plurality of third evaluation vectors used for learning may be obtained based on data collected through multimodal-based English word learning content. The plurality of first evaluation vectors used for learning may be configured one for each English word, and the plurality of second evaluation vectors and the plurality of third evaluation vectors may be configured for each of the plurality of English words. Here, the reference vector is a vector that serves as a standard for evaluating the accuracy of speech rate and pronunciation compared to the second and third evaluation vectors, and the plurality of reference vectors are reference vectors for the English word itself for each English word and definitions of the English word. It can be pre-stored on the server as a reference vector for . The reference vectors for the English word itself are: a reference value related to the pitch period, a reference value related to the amplitude, a reference value related to the fundamental frequency, a reference value related to the energy of the overtone, a reference value related to the energy ratio of the overtone to the noise, and a reference time of speech. may include. The reference vectors for the definition of an English word are: a reference value related to the pitch period, a reference value related to the amplitude, a reference value related to the fundamental frequency, a reference value related to the energy of the phased consonant, a reference value related to the energy ratio of the phased consonant and the noise, and a reference value per second. Can include number of syllables. For example, one first evaluation vector used for learning may be composed of a set with a plurality of second evaluation vectors and a plurality of third evaluation vectors. For example, multiple sets may be pre-stored on a server.

The correct answer learning achievement vector may include sincerity, speech speed, pronunciation accuracy, and performance. For example, a correct answer learning achievement vector can be set for each English word.

For example, sincerity may be determined as the average of the achievement rates for each learning step included in the first evaluation vector.

For example, the speech rate may be determined by comparing the speech time of the second evaluation vector and the average number of syllables per second of the third evaluation vector with the reference vector. For example, from a base value for speech rate, a first absolute value for the difference value of the speech time of the reference vector and the speech time of the second evaluation vector and the reference syllables per second of the reference vector and the average per second of the third evaluation vector. It may be a value obtained by subtracting the sum of the second absolute value for the difference value of the number of syllables.

For example, pronunciation accuracy is determined by using the second evaluation vector located last among a plurality of second evaluation vectors for one English word and the third evaluation vector located last among a plurality of third evaluation vectors for one English word as a reference vector. It can be determined by comparing with .

For example, the performance level is calculated by dividing the pronunciation accuracy for the second evaluation vector located last among a plurality of second evaluation vectors for one English word by the pronunciation accuracy for the second evaluation vector located first, and one value. It may be a third value obtained by dividing the pronunciation accuracy for the third evaluation vector located last among the plurality of third evaluation vectors for an English word by the pronunciation accuracy for the third evaluation vector located first, and adding up the second value. The second evaluation vector located last among the plurality of second evaluation vectors for one English word may be a vector generated for one English word based on the voice file in which the English word itself was recorded last. The third evaluation vector located last among the plurality of third evaluation vectors for one English word may be a vector generated based on the voice file in which the definition of the English word was recorded last.

Additionally, for example, pronunciation accuracy can be determined by Equation 1 below.

In Equation 1, the AP _score is the accuracy of the pronunciation, the diff1 is a comparison value between the reference vector and the second evaluation vector located last among the plurality of second evaluation vectors for the corresponding English word, and the diff2 is the corresponding It is a comparison value between the third evaluation vector located last among a plurality of third evaluation vectors for an English word and a reference vector, where AP1 is the default value for the second evaluation vector, and AP2 is the default value for the third evaluation vector. You can.

For example, the default value for the second evaluation vector and the default value for the third evaluation vector may be pre-stored in the server.

For example, the comparison value between the second evaluation vector located last among the plurality of second evaluation vectors for the corresponding English word and the reference vector may be determined by Equation 2 below.

In Equation 2, diff1 is a comparison value between the second evaluation vector located last and a reference vector, and F0 _diff is a value obtained by subtracting the lowest value of FO from the highest value of F0 of the second evaluation vector, The F0 _ref is a reference value for the maximum change in F0 included in the reference vector matched to the second evaluation vector, the F21 _diff is a value obtained by subtracting the F1 value from the F2 value of the second evaluation vector, and the F21 _ref is a reference value for the amount of change between the F2 value and the F1 value included in the reference vector matched to the corresponding second evaluation vector, and v _add is the average change rate of the pitch period and the average value of the amplitude change rate of the corresponding second evaluation vector. It is a summed value, and v _ref is a sum of the reference value for the average rate of change of the pitch period and the reference value for the average value of the amplitude change rate included in the reference vector matched to the second evaluation vector, and t _s is the utterance time of the corresponding second evaluation vector, and t _ref may be the reference utterance time included in the reference vector matched to the corresponding second evaluation vector.

For example, the comparison value between the third evaluation vector located last among the plurality of third evaluation vectors for the corresponding English word and the reference vector can be determined by Equation 3 below.

In Equation 3, diff2 is a comparison value between the third evaluation vector located last and a reference vector, and F0 _diff is a value obtained by subtracting the lowest value of FO from the highest value of F0 of the third evaluation vector, The F0 _ref is a reference value for the maximum change in F0 included in the reference vector matched to the third evaluation vector, the F21 _diff is a value obtained by subtracting the F1 value from the F2 value of the third evaluation vector, and the F21 _ref is a reference value for the amount of change between the F2 value and the F1 value included in the reference vector matched to the third evaluation vector, and v _add is the average change rate of the pitch period and the average change rate of amplitude of the third evaluation vector. It is a summed value, and v _ref is a sum of the reference value for the average rate of change of the pitch period and the average value of the amplitude change rate included in the reference vector matched to the third evaluation vector, and n _s is the average number of syllables per second of the corresponding third evaluation vector, and n _ref may be the reference number of syllables per second included in the reference vector matched to the corresponding third evaluation vector.

Through this, the server can train a learning achievement evaluation model to more accurately determine the accuracy of pronunciation based on the difference value compared to the reference value for the voice file recorded by the user.

For example, the learning score for the user terminal may be determined based on the sincerity, speech rate, pronunciation accuracy, and performance output for each English word.

For example, the learning score for the user terminal may be the average of the sum of sincerity, speech speed, pronunciation accuracy, and performance for each English word.

For example, based on the fact that the sincerity for a specific English word is higher than a preset first standard score, the state of the first object related to the specific English word displayed on the user terminal may be changed. Here, the first object related to a specific English word may be a silhouette of an object expressed in the specific English word. For example, based on the sincerity for a specific English word being higher than a preset first standard score, the silhouette for a specific English word may be changed to a preset image.

Referring to FIG. 6, the compensation collection area 600 may include a plurality of silhouette objects. Here, the reward collection area 600 is an area where English words are classified by topic and images corresponding to the English words for each topic are collected, and may be referred to as a sticker hunt. For English words corresponding to sticker hunts, an image icon representing the sticker hunt (e.g., animation 450 or reward area 460) may appear on the screen 400 of the multimodal-based English word learning content. If the number of repetitions for each learning step for the corresponding English word is satisfied, the user terminal can obtain an image corresponding to the image icon for the corresponding English word. The compensation collection area 600 initially contains only silhouette objects for the corresponding English words, and each time an image for the corresponding English word is acquired, the silhouette object may be changed into an image. For example, when a specific English word is 'dog', based on the sincerity for 'dog' being greater than or equal to the preset first standard score, the silhouette object 610 for 'dog' is It can be changed to image 620.

In step S360, the server may transmit a learning reward message including a learning score for the user terminal to the user terminal.

The learning reward message may be a message informing the user terminal of compensation according to the results of the multi-modal-based English word learning content of the user terminal. For example, the learning reward message may further include a learning achievement vector for each English word and a file recording the user's utterance in each learning step.

For example, a plurality of English word games may be activated for the user terminal based on a learning reward message. For example, based on the server receiving information about the progress of multimodal-based English word learning content, the server may activate a plurality of English word games for the user terminal.

Here, the plural English word game may be a game for learning various English words. For example, multiple English word games may include one-on-one games, survival games, and team games. For example, the difficulty level for each of the plurality of English word games may be determined based on the learning score for the user terminal. For example, the higher the learning score for the user terminal, the higher the level of difficulty for each of the plurality of English word games. The multiple English word game can be performed in a separate menu by separating sight words, exclamations, conjunctions, prepositions, interrogative words, and relative pronouns. Here, Sight Words may be a collection of words listing the words that appear most frequently in English-speaking children's publications in order of frequency.

For example, a team-based game may be a game in which multiple teams participate and play bingo consisting of multiple English words on a team-by-team basis (hereinafter referred to as a bingo game).

Referring to FIG. 7, the corresponding bingo game may represent an example in which a menu of prepositions indicating location and direction is selected. Bingo game 700 may be composed of 5*5 boxes. The 5*5 box is centered, in front of it, behind it, above it, below it, between, next to, frontmost, backmost, highest, lowest, to the right, to the left, in, and above. Both can be expressed, and the shape of the box can be expressed as both ~inside and ~outside. Users whose learning scores are lower than the preset score or who are of a younger age can play a bingo game in the form of a 3*3 box. Through this bingo game, various education about locations is possible, and the alphabet or numbers in each space can be applied in a fixed or changing form. Examples of some expressions using boxes in the bingo game include "J is in the middle", "3 is next to 2", "K is above L", "F is between C and K", etc., in various locations. The expression for can be applied.

In bingo games, the number of team players can range from a minimum of 1 to a maximum of 5 people. At this time, a minimum of 2 teams and a maximum of 4 teams can participate in the game. For example, words used in a bingo game may be randomly placed among 1,000 preset English words. For example, the words used in a bingo game may be the same 25 preset English words arranged differently at random. For example, 100 English words for a bingo game are given, and each user selects one word in turn, and the selected words can be arranged to disappear from the screen.

The game score is 1) a method in which the average of the scores of participating teams in the range of 10 to 100 is selected as the game score, and 2) a method in which one number is randomly determined as the roller rotates in increments of 10 from 10 to 100. , 3) There may be a way for the person who created the room to determine the score of the game from the beginning.

In team games, multiple weights may be applied. The plurality of weights may include team member weights, team weights, progress weights, and victory weights. For example, individual team member weights are given depending on each team member's level. Here, the level is determined according to the learning score and may include 10 levels ranging from 1 to 10. In other words, the higher the learning score, the higher the level can be determined. Team member weights can be calculated for each individual. For example, the team member weight may be determined to be 1 for team members at the same level as the team member. When forming a team with a team member of a lower level than oneself, the team member weight can be calculated by multiplying the difference in weight with each team member of a lower level. Team member weight may not be calculated in relationships with team members higher than oneself. In other words, in relationships with team members higher than oneself, the team member weight can be determined to be 1. Therefore, high-level users should team up with low-level users as much as possible to obtain more points in the final result. A team can consist of a minimum of 1 person and a maximum of 5 people. At this time, the larger the number of people, the higher the team weight can be applied. To make the game progress quickly, a higher speed weight can be given to the team that selects words quickly. For example, if the number of users in a given order exceeds 20 seconds during their team's turn, the word selection can be transferred to the next team. A higher victory weight may be given in the order in which the team played Bingo quickly.

At this time, separate from the team score, each user can accumulate individual scores.

For example, a one-on-one game may be a one-on-one game in which the correct answer is first selected from a plurality of examples related to an English word. For example, in a one-on-one game, the person who answers the most questions correctly in one minute wins. For example, a one-to-one game can be composed of five items: 1) English text, 2) audio of the English word, 3) image, 4) definition of the English word, and 5) audio of the definition of the English word. For example, the user terminal selects an answer from a view for the displayed word, and the view may be given as one of the remaining four items. In order to ensure fairness in one-on-one games, low-level participants (users) may be asked questions with a higher frequency than the preset frequency, mainly for questions involving combinations of 1) English words and 3) images. In addition, participants with the highest level may be asked questions that are a combination of 5) audio and 3) images for definitions of English words with a higher frequency than the preset frequency. In other words, the difficulty level can be automatically adjusted depending on the level difference of the participants. The question type of the one-on-one game can be composed of five items mixed together in a complex manner. For example, a text is given and four example images are given, a sentence is given as a definition and four words are given as examples, a sentence is given as a definition of an English word and images are given as examples, or an English word is given as an example. It can take a complex form, such as being given a voice and a definition of an English word given as an example. In a one-on-one game, the overall (number of questions answered correctly - number of questions answered incorrectly) may be the determining factor in victory. A user terminal that creates a one-on-one game can set the score to be acquired, that is, a target score, and can proceed with the game without knowing any information about the opposing user terminal. The purpose of the one-on-one game is to put you in a state of immersion with a high level of concentration for one minute, so even if you repeat it only three times, you can be immersed for three minutes. In this case, a higher educational effect can be achieved than when exposed to English academically for one hour.

For example, a survival game may be played in the form of two or more teams consisting of one or more user terminals. For example, a survival game can have less than 100 total participants. The survival game continues until the last person emerges as the winner, and the winning team can earn the same score as all team members.

For example, the survival game is a first game in which a team survives by completing the spelling of an English word and collecting the alphabet through exploration of a specific virtual space using a game character corresponding to the one or more user terminals, and a second game in which the team that matches English words with collected alphabets survives, and a third game in which the team that first matches all of the plural objects hidden in a specific picture with examples consisting of plural English words survives. For example, in a survival game, team members may be able to discuss each other through chat or voice.

For example, the first game may be an assembly game, in which team members continuously guess words by completing spellings of words during a preset period of time. In the first game, team members can discuss each other through chat or voice, and team members may be required to participate in the game in order. For example, team members may be required to participate in a game in order.

For example, the second game may be a treasure hunt game where the game character moves around a specific space, finds the appearing alphabet, and creates a given word.

For example, the third game is the Eye Spy game, which can be a game in which you search for various objects hidden in pictures by matching them with words in the picture. For example, participants in the third game may have to match objects and words in order to find them.

For example, the user terminal can create a character to play the game with, and can also select one from the characters provided by default. Additionally, for example, the user terminal may create a character imaged by taking a photo of the user's face (uploading a photo or taking a photo). The user terminal can also decorate or develop game characters using points earned through games.

For example, in multimodal-based English word learning content, the server can provide the user terminal with a movement in the form of the game character it created turning pages when switching from each word to the next word. Additionally, motivating actions can be implemented through game characters throughout the game. When motivating, images, sounds, and voices expressing various exclamations can be provided.

According to one embodiment, the server may receive a second request message for one English word game among a plurality of English word games from the user terminal.

The second request message is a message requesting the server to participate in one English word game or to create one English word game. For example, the second request message may include either a value indicating game participation or a value indicating game creation, and identification information about the game. Identification information about the game may include an identification value indicating the type of game.

For example, the server may transmit the access address for one English word game to the user terminal. For example, the user terminal may participate in an English word game requested by the user terminal based on the access address. For example, when receiving a value indicating game participation, the server selects the access address of the English word game in which the user terminal can participate according to the learning score of the user terminal among a plurality of English word games corresponding to the type of game requested by the user terminal. You can decide. For example, when receiving a value indicating game creation, the server creates an English word game corresponding to the type of game requested by the user terminal according to the learning score of the user terminal, and determines the access address of the created English word game. .

For example, the server may receive result information about one English word game from the user terminal. Here, the result information about the English word game may include the total score obtained by the user terminal through the English word game and the English words for which the user terminal played the English word game. For example, when an English word game in which the user terminal participates ends, the user terminal may transmit the results of the ended English word game to the server. The server can increase the achievement rate for each learning stage for an English word played by the user terminal in an English word game in proportion to the total score obtained through the English word game.

According to one embodiment, the server may receive a third request message for learning English words from the user terminal. The third request message may be a message requesting the user terminal to learn English words after the learning score for the user terminal is determined through the learning achievement evaluation model. For example, the third request message may include information about the user terminal.

For example, the server may identify the user terminal based on identification information about the terminal. For example, the server may obtain previously stored personal information about the user and a learning achievement vector for the identified user terminal.

For example, the server may generate a user vector by performing data preprocessing on personal information about the user and information about the user terminal included in the third request message.

For example, the server may obtain new progress method information for the user terminal by inputting the user vector and the learning achievement vector into the method decision model. Here, the new progression method may be a progression method after completing learning through multimodal-based English word learning content for the user terminal. In other words, the new progress method may be a proceeding method for cases where there are progress results for multimodal-based English word learning content for the user terminal.

For example, the server may transmit a learning start message containing new progress method information to the user terminal.

For example, multimodal-based English word learning content displayed on the user terminal may be progressed based on new progress method information.

For example, the server may determine new progress method information whenever the user terminal receives information about the progress results of multimodal-based learning content and determines the learning score.

For example, the learning start message may further include review information about English words. Here, the review information may include information about all English words previously learned by the user terminal. For example, multimodal-based learning content may provide a review function to the user terminal according to review information. For example, the review function may be a function that displays a screen consisting of only a combination of English words and animation images on the user terminal for between 1 and 1.5 seconds. At this time, the process can be performed automatically for all previously learned English words. For example, if the user terminal has learned 30 English words in previous learning, the previous learning can be quickly reviewed for a short period of time between 30 and 45 seconds through a combination of one English word and an animation image displayed on the screen of the user terminal. can do. Through this, the user examines English words in a highly immersive state for a short period of time, which can have the effect of helping the user remember English words for a longer period of time.

According to one embodiment, a plurality of animation images constituting animation included in multi-modal-based English word learning content may be generated through an animation generation model using a neural network based on a generative adversarial network.

For example, for each of a plurality of English words, the server can collect a plurality of picture images associated with the English word through web crawling. For example, the server may transmit a plurality of picture images to a plurality of user terminals and receive preferences for each of the plurality of picture images from the plurality of user terminals. The server classifies a plurality of user terminals into n groups based on age and gender, and for each group, the most preferred picture image among the plurality of picture images can be determined by English word as the preferred picture image. For example, n groups may be n groups classified based on the plurality of user vectors described above. At this time, the plurality of user terminals may include user terminals corresponding to user vectors constituting n groups.

For example, the server may collect, for each of a plurality of English words, a plurality of dynamic images related to the English word through web crawling. For example, the server may determine the animation playback time for each of a plurality of dynamic images. For example, the server may determine the reference playback time for the animation according to the playback time of the definition of the English word and the number of texts of the definition of the English word for each of a plurality of English words. Here, the reference playback time may be between 3 and 7 seconds. At this time, the standard playback time may be determined to be closer to 7 seconds as the playback time of the English word definition is longer and the number of texts in the English word definition is greater. For example, the server may determine the dynamic image with the animation playback time closest to the standard playback time among the plurality of dynamic images as the standard dynamic image for each English word. At this time, if it is determined that there are a plurality of dynamic images with animation playback times closest to the reference playback time among the plurality of dynamic images, the server determines that the number of images constituting the dynamic image among the plurality of dynamic images with the closest animation playback times is greater. Many dynamic images can be resolved into standard dynamic images.

For example, the server inputs a plurality of preferred picture images and a plurality of standard dynamic images into an animation creation model using a neural network based on a generative adversarial network, thereby creating an animation included in multi-modal-based English word learning content. Images can be created for each English word. At this time, one preferred picture image and one standard dynamic image may be matched for each English word. An animated image may be an image in which a preferred drawing image is synthesized with the actions of a standard dynamic image.

For example, the server inputs each of a plurality of constituent images, which constitute one standard dynamic image for each preferred picture image, into an animation generation model using a neural network based on a generative adversarial network, so that one preferred picture image is converted into a standard dynamic image. A composite image can be obtained through the motion of a dynamic image.

Figure 8 is a diagram showing an example of an animation generation model using a neural network based on a generative adversarial network according to an embodiment. The embodiment of FIG. 8 can be combined with various embodiments of the present disclosure.

Referring to FIG. 8, the animation generation model may use a neural network 800 based on a generative adversarial network. A neural network based on a generative adversarial network is a backbone network that can effectively separate data distributions and construct latent vectors using only unsupervised learning. A latent vector may refer to a pair of independent latent variables. A model that generates high-quality images from disentangled latent vectors can be used.

Additionally, a neural network based on a generative adversarial network may be composed of a network structure of AdaIN (adaptive instance normalization) using style transfer. Here, style transfer refers to a method of extracting content from a first input image, extracting a style from a second input image, and combining the style and content. For example, a neural network based on a generative adversarial network may be a style transfer network consisting of a VGG (Visual Geometry Group) encoder, an AdaIN layer, and a decoder. Here, the VGG encoder is a basic CNN composed of a convolutional layer and a pooling layer, and may be composed of 16 layers or 19 layers. For example, the AdaIN layer of a neural network based on a generative adversarial network subtracts the style of the first input image from the first input image including the content through Equation 4 below in the feature space, and then inputs the second input. You can combine image styles.

In Equation 4, σ(x) is the average for the second input image including the style, σ(y) is the standard deviation for the second input image including the style, and μ(x) is the content. is the average for the first input image including, and μ(y) may be the standard deviation for the first input image including the content.

For example, feature t generated through the AdaIN layer can be determined by Equation 5 below.

In Equation 5, the f(c) function is a function indicating that the first input image including content is input to the VGG encoder, and the f(s) is a function indicating that the second input image including style is input to the VGG encoder. It can be a function that represents something.

Afterwards, the two feature maps encoded through the VGG encoder are input to a randomly initialized decoder to train the decoder, so that the decoder can generate a style-synthesized image T(c, s).

For example, the plurality of first input images 810 may be a plurality of favorite picture images. For example, the plurality of second input images 820 may include a plurality of standard dynamic images. For example, the server uses a plurality of first input images and a plurality of second input images as learning data to use a generative adversarial network model to obtain an image 830 in which the preferred picture image is synthesized with the operation of a standard dynamic image. It can be learned.

Through this, an animation generation model can be trained so that preferred picture images naturally generate images synthesized with the actions of standard dynamic images.

That is, the server inputs each of a plurality of constituent images, which constitute one standard dynamic image for each preferred picture image, into an animation generation model using a neural network based on a generative adversarial network, so that one preferred picture image is converted into a standard dynamic image. A synthesized image can be obtained through the operation of .

Figure 9 is a block diagram showing the configuration of a server according to one embodiment. One embodiment of FIG. 9 may be combined with various embodiments of the present disclosure.

As shown in FIG. 9, the server 900 may include a processor 910, a communication unit 920, and a memory 930. However, not all of the components shown in FIG. 9 are essential components of the server 900. The server 900 may be implemented with more components than those shown in FIG. 9, or the server 900 may be implemented with fewer components than those shown in FIG. 9. For example, the server 900 according to some embodiments may further include a user input interface (not shown), an output unit (not shown), etc. in addition to the processor 910, the communication unit 920, and the memory 930. .

The processor 910 typically controls the overall operation of the server 900. The processor 910 may include one or more processors and control other components included in the server 900. For example, the processor 910 can generally control the communication unit 920 and the memory 930 by executing programs stored in the memory 930. Additionally, the processor 910 may perform the functions of the server 900 shown in FIGS. 3 to 8 by executing programs stored in the memory 930.

The communication unit 920 may include one or more components that allow the server 900 to communicate with other devices (not shown) and servers (not shown). The other device (not shown) may be a computing device such as the server 900 or a sensing device, but is not limited thereto. The communication unit 920 may receive a user input from another electronic device or receive data stored in an external device from an external device through a network.

For example, the communication unit 920 may transmit and receive a message to establish a connection with at least one device. The communication unit 920 may transmit information generated by the processor 910 to at least one device connected to the server. The communication unit 920 may receive information from at least one device connected to the server. The communication unit 920 may transmit information related to the received information in response to information received from at least one device.

The memory 930 may store programs for processing and control of the processor 910. For example, the memory 930 may store information input to a server or information received from another device through a network. Additionally, the memory 930 may store data generated by the processor 910. The memory 930 may store information input to or output from the server 900.

The memory 930 is a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (for example, SD or XD memory, etc.), and RAM. (RAM, Random Access Memory) SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic disk , and may include at least one type of storage medium among optical disks.

The embodiments described above may be implemented with hardware components, software components, and/or a combination of hardware components and software components. For example, the devices, methods, and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, and a field programmable gate (FPGA). It may be implemented using one or more general-purpose or special-purpose computers, such as an array, programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. A processing device may execute an operating system (OS) and one or more software applications that run on the operating system. Additionally, a processing device may access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, a single processing device may be described as being used; however, those skilled in the art will understand that a processing device includes multiple processing elements and/or multiple types of processing elements. It can be seen that it may include. For example, a processing device may include a plurality of processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are possible.

Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing unit to operate as desired, or may be processed independently or collectively. You can command the device. Software and/or data may be used on any type of machine, component, physical device, virtual equipment, computer storage medium or device to be interpreted by or to provide instructions or data to a processing device. , or may be permanently or temporarily embodied in a transmitted signal wave. Software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., singly or in combination. Program instructions recorded on the medium may be specially designed and configured for the embodiment or may be known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -Includes optical media (magneto-optical media) and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions include machine language code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

Although the embodiments have been described with limited drawings as described above, those skilled in the art can apply various technical modifications and variations based on the above. For example, the described techniques are performed in a different order than the described method, and/or components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or other components are used. Alternatively, appropriate results may be achieved even if substituted or substituted by an equivalent.

Therefore, other implementations, other embodiments, and equivalents of the claims also fall within the scope of the claims described below.

Claims (5)

In a method for a server to provide a multimodal-based English learning service to a user terminal using a neural network,
Receiving a first request message for learning English words from a user terminal;
The first request message includes personal information about the user and information about the user terminal,
determining proceeding method information for the user terminal through a method decision model using a first neural network based on personal information about the user and information about the user terminal;
Transmitting a learning start message including the progress method information to the user terminal;
Multimodal-based English word learning content displayed on the user terminal is progressed based on the progress method information,
The multimodal-based English word learning content includes information about English words, information about definitions of English words, and information about animations related to English words,
Receiving information about the progress of the multimodal-based English word learning content from the user terminal;
determining a learning score for the user terminal through a learning achievement evaluation model using a second neural network based on information about the progress of the multimodal-based English word learning content; and
Including transmitting a learning reward message containing a learning score for the user terminal to the user terminal,
A plurality of English word games are activated for the user terminal based on the learning reward message,
A user vector is generated through data preprocessing of personal information about the user and information about the user terminal,
The user vector includes a value for the user's age, a value for the user's gender, and a value related to the capacity of the user terminal,
Among the plurality of average achievement vectors set in the server, an average achievement vector matching the user vector is determined as a basic achievement vector for the user terminal,
Progress method information for the user terminal is output based on the user vector and the basic achievement vector being input to the method decision model,
The progress method information includes the number of learning steps, the number of repetitions for each learning step, the total progress time for each learning step, and a value for at least one action related to each English word, definition of English word, and animation for each learning step, and information related to the English word for each learning step. It includes playback time, maximum recording time related to English words at each learning stage, playback time related to definitions of English words at each learning stage, and maximum recording time related to definitions of English words at each learning stage,
Information on the progress results of the multimodal-based learning content includes the achievement rate of each learning stage for each English word and a plurality of recorded voice files for each English word,
Through data preprocessing of information on the progress results of the multimodal-based learning content, a first evaluation vector consisting of the achievement rate of each learning stage is generated for each English word,
A plurality of second evaluation vectors and a plurality of third evaluation vectors are generated for each English word through data preprocessing of the plurality of recorded voice files,
The fidelity and speech rate for the user terminal are based on the first evaluation vector, the plurality of second evaluation vectors, and the plurality of third evaluation vectors being input to the learning achievement evaluation model using the second neural network. , Pronunciation accuracy and performance are displayed for each English word,
The learning score for the user terminal is determined based on the sincerity output for each English word, the speech rate, the pronunciation accuracy, and the performance,
Based on the sincerity for a specific English word being more than a preset first standard score, the state of the first object related to the specific English word displayed on the user terminal is changed,
method.
delete delete According to clause 1,
Receiving a second request message for one English word game among the plurality of English word games from the user terminal;
Transmitting an access address for the one English word game to the user terminal;
The user terminal participates in one of the English word games through the access address,
Further comprising receiving result information for the one English word game from the user terminal,
The plurality of English word games include one-on-one games, survival games, and team games,
The difficulty level for each of the plurality of English word games is determined based on the learning score for the user terminal,
method.
According to clause 4,
The one-on-one game is a one-on-one game in which the correct answer is first selected from a plurality of examples related to English words,
The team-based game is a game in which multiple teams participate and play bingo consisting of multiple English words on a team-by-team basis,
The survival game is a form in which two or more teams consisting of one or more user terminals participate,
The survival game is a first game in which a team survives by completing the spelling of an English word, collecting the alphabet through exploration of a specific virtual space using a game character corresponding to the one or more user terminals, and collecting the collected alphabets. A second game in which the team that matches English words using the alphabet survives, and a third game in which the team that first matches all of the plural objects hidden in a specific picture with examples of multiple English words survives.
method.

Images