KR102396263B1 - A System for Smart Language Learning Services using Scripts - Google Patents

Abstract

The present invention relates to a smart language learning service (hereinafter 'smart language learning service') that uses a script synchronized with audio over time, and more particularly, to a personal computer, mobile/home/learning/office information device, server, It is operated in an information environment environment such as a network, and by using a script synchronized with audio, it is possible to distinguish from descriptive text, listen to sections at the level of fingerprints, sentences, and words, listen repeatedly, skip, and simultaneously listen while watching scripts, and interactive text It is related to a smart language learning service capable of listening to sections at the level of dialogue, speaker, problem, question, and answer at the level of dialogue, repeated listening, skipping, and simultaneous listening to the script and the management method of script data synchronized with audio. Here, the audio includes audio provided simultaneously with visual information such as an image or video.

Description

{A System for Smart Language Learning Services using Scripts}

The present invention relates to a system for providing a smart language learning service (hereinafter 'smart language learning service') using a script synchronized with audio over time. It is possible to listen to sections at the level of classification, fingerprint, sentence, and word, listen repeatedly, skip, and listen at the same time while watching the script. And it is related to a smart language learning service that allows simultaneous listening while watching a script, and a method for managing script data synchronized with audio. Here, the audio includes audio provided simultaneously with visual information such as an image or video.

Conventional general language learning services (including 'language learning machine') provide functions such as forward, backward, stop, resume, section setting, and repeat listening by providing an audio or video file.
Korean Patent Application Laid-Open No. 2013-0129034 discloses a method for repeated learning of English and foreign languages centered on speaking and listening based on a tree structure, a method for linking learning of sentences and words, and a storage medium for a program. In No. 2013-0129034, the learner directly designates the repetition section, links the specified information to sentences and words, stores it in a tree structure, and uses it for repeated learning. Here, in order to play the audio of a specific sentence part of the script, the learner has to find the corresponding point on the tree structure previously designated, and if it is not designated, it is inconvenient to designate it again.
Korean Patent Application Laid-Open No. 2013-0076852 discloses a method for generating language content and a terminal for the same. In an audio file, a sentence is separated based on a space having an amplitude close to '0', and the text is segmented into sentence units to provide audio and text are synchronizing In this method, when there is a space between words depending on the speaker, there is a high possibility of errors in associating with the sentence. And the exact word-to-audio synchronization is not disclosed.

In the conventional language learner, functions such as division, fingerprint, sentence, and word level in descriptive text, repeated listening, skipping, and simultaneous listening while watching script, conversation in interactive text, speaker, problem, question, answer level Although the functions such as section listening, repeat listening, skipping, and simultaneous listening while watching the script are important from the learner’s point of view, the conventional language learning services set the part to proceed while listening to the audio waveform graphic and sound, and the script structure or interactive Users were inconvenient because a function to support language learning using characteristics such as sentences was not provided.

The present invention, which was devised to solve the above problems, includes a method for managing script data synchronized with audio, using the script structure or characteristics such as interactive sentences, and for personal computer, mobile/home/study/office use. An object of the present invention is to present a method of providing a smart language learning service and service characterized in that it is operated in an information environment such as information devices, servers, and networks.

The system for providing a smart language learning service according to the present invention, which was created to achieve the above object, is a user interface used by a user, an application unit using characteristics such as a structure of a script or an interactive sentence, and information such as a script and audio database containing . It consists of a data management unit such as a script synchronized with audio, and can be operated alone or distributedly in an information environment such as a personal computer, mobile/home/learning/office information device, server, and network.

The user interface is provided with a user menu for executing language audio, and the application unit is composed of basic functions related to audio execution and script setting functions supporting narrative or interactive text.

The data management unit is composed of the steps of synchronizing audio and script, editing data, verifying whether data is suitable for a smart language learning service, and managing a plurality of verified data.

The basic audio functions of the application unit include run, stop, resume, low-speed or high-speed skip forward or reverse, mark setting, section listening, repeat listening, and the like.

The script function of the application unit includes functions such as division, fingerprint, sentence, and word level in descriptive text, repeated listening, skipping, and simultaneous listening while watching the script, from interactive text to dialogue, speaker, problem, question, and answer level It includes functions such as section listening, repeat listening, skipping, and simultaneous listening while watching a script.

The present invention configured as described above solves the conventional inconvenience of learners having to find the starting point and ending point of a sentence or word by using functions such as forward, backward, stop, and resume according to the temporal flow of audio data, and the characteristics of the script It provides useful effects such as classification, fingerprints, sentences, words, conversations, and problems that allow learners to learn in a more friendly way.

[Figure 1] is a system configuration diagram of a smart language learning service.
[Fig. 2a] is an example of a user interface of a conventional language learning service.
[Fig. 2b] is an example of a user interface of a smart language learning service.
[Figure 3] is an example of the script configuration of the smart language learning service.
[Figure 4] is an example of synchronization of audio and script in the smart language learning service.
[Fig. 5] is an example of script editing in the smart language learning service.

Objects, effects and features of the present invention, and techniques for achieving these, etc. will become clear with reference to the embodiments described in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below and may be implemented in various different forms. This embodiment may be provided so that the disclosure of the present invention is complete, and to fully inform those of ordinary skill in the art to which the present invention belongs, the scope of the invention.

The terms used in this specification are for the purpose of describing the embodiment and are not intended to limit the present invention. In this specification, the singular also includes the plural unless specifically stated otherwise in the phrase. As used herein, 'comprise' and/or 'comprising' means that a referenced component, step, operation and/or element is the presence of one or more other components, steps, operations and/or elements. or addition is not excluded.

Hereinafter, the configuration and operating principle of the present invention will be described in more detail with reference to the accompanying drawings.

1 is a system configuration diagram of a smart language learning service, and is composed of a user interface 100 , an application unit 200 , a database 300 , a data management unit 400 , and the like.

The user interface is provided with a user menu for executing audio for language study. 2A is an example of a user interface of a conventional language learning service, and is composed of a display unit 110 , a status unit 120 , and an execution unit 130 . The display unit 110 displays the audio waveform so that the learner can visually see the speaker's voice, and can set an arbitrary point or repeat section. The state unit 120 displays the start and end time of the time, and also displays the current progress time. By moving the display of the current progress point forward or backward on the time axis, audio execution is also possible at any progress point. The execution unit 130 includes various menu buttons, for example, play 131, stop 132, reverse skip 133, forward skip 134, section mark 135, repeat play 136 ), initialization 137, double speed setting 138, and the like menus are provided. Here, play (131) and pause (132) are indicated as intersecting each other and are operated.

In a user interface of a system for providing a conventional language learning service, first, a script synchronized with audio is not provided, but an audio waveform is provided. Second, in order to reproduce the audio of a specific sentence part of a script, a desired playback point could be found through listening and adjustment in a trial and error method using forward or backward skip while viewing a separate script in print or file format. Third, there was the inconvenience of setting by trial and error while listening to audio waveforms and sounds to set the section for repeated learning.

Figure 2b is a user interface of a system that provides a smart language learning service, and is composed of a script display unit 140, a status unit 150, a setting unit 160, a search unit 170, an execution unit 180, etc. It has the same main characteristics as First, audio and synchronized scripts are provided. Second, it is possible to find a desired text or problem in a script including a plurality of language problems and reproduce it from the beginning of a specific sentence or word. Third, in the script, it is possible to select units such as division, fingerprint, sentence, word, speaker, problem, question, answer, etc., or select arbitrary text within the script, and play or repeat audio.

The script display unit 140 includes a display window 141 , an upward operation button 142 for moving text up and down, a downward operation button 143 , and a scroll bar 144 . The learner may switch to the problem solving mode and the commentary mode by a specific touch (eg, touching two fingers at the same time) on the display window 141 . In the problem solving mode, the guide text 211 and the body 212 of the fingerprint are not visible as needed in the example of FIG. 3 .

The status unit 150 indicates the script progress state as a start point, a current point, and an end point, and can be displayed as a text progress rate (%) or a temporal progress state (hours:minutes:seconds).

The setting unit 160 provides a function of setting a unit item for selecting or reproducing a script, and includes a division 161, a fingerprint 162, a sentence 163, a word 164, a speaker 165, a question ( 166), questions (167), and answers (168). Here, division 161 indicates the type of problem. For example, the type that shows a picture and chooses the correct explanation, the type that selects the correct answer to a question in a short conversation type, the type that listens to a conversation between multiple people and selects each answer from multiple questions There may be a type of telling a story and selecting each answer from multiple questions.

3 is an example of a script configuration for language learning. In FIG. 2B , the fingerprint 162 is a common text for a plurality of problems, and in the example of FIG. 3 , the fingerprint is composed of a guide 211 and a main body 212 , and a plurality of related problems 215 , 217 , 218 . information presented in As shown in FIG. 3 , the main body 212 may be presented in a dialogue format between a male and a female speaker, and may be presented in a dialogue format of three or more unspecified characters, or as an explanatory text by one person. In addition, the fingerprint may be provided in the form of an image or video.
The sentence 163 is composed of a subject and a predicate. A word 164 is a basic unit constituting a sentence.
The speaker 165 is expressed as a man 213, a woman 214, or a specific name in the body 212 and refers to a speaking subject. In addition to this, grammatical elements such as main clauses, subordinate clauses, and syntax in paragraphs and sentences are also included in the scope, although not expressed in the examples of the present invention. Question 166 refers to question parts 215 , 217 , 218 in question that are appended to passage 162 . The question 167 means an option 216 for the question, and the like. The answer 168 means an option that is an answer among the options.

In FIG. 2B , when one item button is selected in the setting unit 160 , it is selected by a touch on the display window 141 of the display unit 140 . For example, when the 'fingerprint 161' button is selected and the fingerprint displayed on the display window is touched, the entire fingerprint is selected. Similarly, when the 'sentence 163' or 'word 164' button is selected, it is accumulated and selected in units of sentences or words by touch. By repeating this function, you can easily select arbitrary paragraphs, multiple sentences and phrases. To select an arbitrary section, if you touch and hold the screen, a section display appears and you can select a desired text section by moving the front and rear adjusters.

In FIG. 2B , the search unit 170 includes a search window and a search button. Information input into the search window provides priority of search results according to the item selected by the setting unit 160 . For example, if the information '10' is entered in the search box and the search button is pressed while the 'fingerprint 162' button is selected, the fingerprint of question 10 and incidental problems are displayed on the display window 141 of FIG. 2B. are presented and the current point is moved to the starting point of the fingerprint. If the 'question 166' button is selected here, the contents of the display window are the same and the current point is moved to the starting point of the '10' question.

As another example, when the word 'class' is input in the search window and 'fingerprint 161' is selected in the setting unit 160, the search is performed only within the fingerprint. In this way, the selected button item of the setting unit serves as a filter to filter information of the search result. If the 'word 164' button is selected, the entire script is searched. These functions can be extended to a set of scripts stored in a specific information device or server.

In Figure 2b, the execution unit 180 plays (181), stop (182), reverse skip (183), forward skip (184), section mark (185), repeat play (186), initialization (187), double speed setting 188 and the like. When the play 181 button is pressed, audio is reproduced from the current point in progress of the status unit 150 , and the current point in progress is displayed in the text of the status unit 150 and the display window 141 in real time. At this time, while the audio is being played, the play (181) button is replaced with the stop (182) button and displayed. When the stop 182 button is pressed, audio reproduction is stopped and the button is replaced with a play button 181 . The reverse or forward skip buttons 183 and 184 can be skipped in units of fingerprints, sentences, words, speakers, questions, questions, and answers according to the button item selected in the setting unit 160 . Section mark setting is performed by selecting the desired text using the functions of the display unit 140 and setting unit 160 and then pressing the section mark 185 button. If the repeat play 186 button is pressed after the section mark is set, the set section is repeatedly reproduced. When the initialization (187) button is pressed, it moves to the initial state. In addition, when the double speed setting 188 button is pressed, the double speed of the playback speed can be adjusted.

The application unit 200 of FIG. 1 is composed of a script setting module 210 and a search and execution module 220 . The script setting module 210 is an entity that implements the function of the setting unit 160 of FIG. 2B in the user interface 100 and performs each function from the division 161 to the answer 168, and the contents of each function are described above. As described in , it is omitted. Similarly, the search and execution module 220 is an entity that implements the functions of the search unit 170 and the execution unit 180 of FIG. 2B , and the description of each function has been described above.

The database 300 of FIG. 1 includes meta information 310 , a script 320 , an audio 340 , and a video/image 350 .

Meta information 310 has attribute information such as textbook for learning, copyright holder, name server, data/schema/form address, time, version, script 320, audio 340, video/image 350, etc. there is.

The script 320 has text information synchronized with the audio or sound part of the audio or video in terms of time. In addition, it has information such as classification, fingerprint (including guide, dialogue, or narrative text), speaker, and problem (including questions and items) that express the characteristics of the textbook. And it distinguishes the sentences in the script from a grammatical point of view. It can be distinguished simply by punctuation, and recently, using natural language processing technology, it is possible to distinguish sentences, sentence constituents, phrases, and even parts of speech of words.

4 is a representation of an example of a script configuration for language learning presented in FIG. 3 in an XML (Extensible Markup Language) format. This includes problem groups 321, parts 322, passages 323, announcements 324, dialogues 325, speaker dialogues 326,327, questions 328,332, questions 329, options 330,331, etc. is composed of The problem group 321 means a group consisting of two or more problems. Part 322 is a type of problem that shows a picture and finds a correct explanation, a type that answers a short question, a type that presents a dialogue between two or more people and presents multiple problems, a type that shows a picture and finds a correct explanation There may be various types of presenting the problems of As the fingerprint 323 , a guide 324 , a dialogue text 325 , or an explanatory text may be presented. The guide 324 is a guide sentence for solving a problem. The dialogue sentence 325 includes a plurality of speaker dialogue sentences 326 and 327 , and the gender or name of the speaker is presented. The passage 323 is followed by singular or plural questions 328 and 332 , each consisting of a question 329 and options 330 and 331 . Here, it has time stamp information synchronized with the script with respect to sounds expressed by voice or sound, such as the fingerprint 323 and the problem 328 .

The audio 340 is data that can be expressed by voice or sound. The audio is synchronized with the script 320 and can be managed in a database or file format. In the case of an audio file, the format of the stored information is presented.

The video/image 350 includes image information that is temporally synchronized with a video including voice or sound. The video/image may be synchronized with the script 320 and managed in a database or file format. In the case of a file format, the format of each stored information is presented.

The data management unit 400 of FIG. 1 is composed of an editing module 410 , a synchronization module 420 , a verification module 430 , and a management module 440 , and the entire system becomes one system or a combination of modules. configurable. The editing module 410 supports an editing function that can understand the XML-formatted sentences shown in the example of the script data of FIG. 4 .

The synchronization module 420 supports a function of managing synchronization time information between the script and audio, video, and image information. A method of providing synchronization time information may exist in various ways. First, it is a method of manually matching timetable information such as a script and audio using the editing module. Although partial editing is possible, it is not suitable for processing a large audio file. Second, as a case in which timetable information such as script and audio exists, it can be processed by a function of converting the script data of FIG. 4 according to the format. In this case, the number of converters is required as many as the number of formats representing the existing timetable information. Third, in the case where the script and audio exist independently but there is no timetable information, synchronized timetable information is generated through the same procedure as in FIG. 5 . In Fig. 5, the steps of generating text and time expression from the audio (step 1), segmenting the script to fit the size of the generated text (step 2), replacing the generated text with the segmented script (step 3), Configure the conversion step (step 4) to fit the script data format.
In step 1, an audio file is input to generate timetable information synchronized with text, and speech to text (STT) related tools are available. With a few modifications to these tools, time expressions can be created in units such as sentences or number of words. The example of FIG. 5 shows an example composed of a timetable and text. Here, the text may include various types of error data due to differences in converter performance. Step 2 is a step of segmenting the authentic script into the generated text size with synchronized timetable information. In step 3, the generated text containing errors is replaced with segmented text. Through these steps, a synchronized timetable and error-free text file can be created. Step 4 is a step of converting the generated synchronization script to fit the format of FIG. 4 .

In addition to the above three cases, a case in which only a script exists and there is no data such as audio or only data such as audio exists and no script may be considered. The former may apply the third method after generating audio information using a text to speech (TTS) function or recording using a voice actor. In the latter, a method of generating synchronized timetable information and text using the speech-to-text (STT) function using audio data, and removing errors in the text by a human is possible. Thereafter, the process of step 4 of FIG. 5 may be performed.

The verification module 430 verifies the script format and meaning shown in an example of the script data of FIG. 4 . From the formal point of view, the composition of the problem defined for language learning and the XML grammar are verified. The former verifies whether the script text, including division, fingerprint, sentence, word, speaker, question, sentence, and answer, is composed in accordance with form and meaning. In addition, it is possible to verify whether the numerical value has meaning in accordance with the text order of the timetable information.

The management module 440 includes a function of managing so that the learner can smoothly learn a language. This may include all or selectively functions such as available lectures, course registration, fee payment, lecture data, usage environment (PC, mobile, etc.), usage rights, and authentication.

100 Smart language learning service system user interface
110 Display unit in the user interface of the conventional language learning service system
120 State part in the user interface of the conventional language learning service system
130 Execution unit in the user interface of the conventional language learning service system
131 play
132 stop
133 Skip backwards
134 Skip Forward
135 section mark
136 repeat play
137 Reset
138 speed setting
140 Script display unit in the user interface of the smart language learning service system
141 display
142 Up/down button to move text up and down
143 Down button to move text up and down
144 Scroll bar to move text up and down
150 The status part in the user interface of the smart language learning service system
160 The setting unit in the user interface of the smart language learning service system
161 Classification of types of language learning problems
Fingerprints of 162 language learning problems
Sentences from 163 language learning problems
164 words in language learning problems
165 speakers in language learning problems
166 questions from language learning problems
167 questions from language learning problems
Answers to 168 language learning questions
170 Search unit in the user interface of the smart language learning service system
180 Execution unit in the user interface of the smart language learning service system
181 play
182 stop
183 Skip backwards
184 Skip Forward
185 section mark
186 Repeat Play
187 Reset
188 speed setting
200 Application part of smart language learning service
210 script setting module
211 Guidance in an example of a language learning problem
212 Text from an example of a language learning problem
213 Male speaker in an example of a language learning problem
214 Female speakers in an example of a language learning problem
215 The question in question in an example of a language learning problem
216 Choices in an example of a language learning problem
217 The question in question in an example of a language learning problem
218 The question in question in an example of a language learning problem
220 Search and Execute Module
Database of 300 Smart Language Learning Service System
310 meta information
320 scripts
321 problem groups in language learning scripts
Part of 322 Language Learning Script
Fingerprint of 323 language learning script
Guide to 324 Language Learning Scripts
325 Dialogue of Language Learning Script
326 Male speaker dialogue in language learning script
327 Female speaker dialogue in language learning script
328 Problems with language learning scripts
329 Questions in Language Learning Script
Choice of 330 language learning scripts
331 Language Learning Script Options
332 Language Learning Script Problem
340 audio
350 videos/images
400 Data management unit of smart language learning service system
410 editing module
420 Synchronization Module
430 Verification Module
440 management module

Claims (2)

In a system for providing a smart language learning service including a user interface 100, an application unit 200, a database 300, and a data management unit 400,
A user interface (100) that supports the menus of setting the status of learning problems, searching, and playing while the user sees a script synchronized with audio with characteristics including classification, fingerprint, sentence, word, speaker, problem, question, and answer on the screen (100) );
an application unit 200 that sets a movement unit, search range, and execution range of the script, and supports movement, search, and reproduction according to the script setting;
a database 300 including audio or video information synchronized with the script and meta information that is characteristic information thereof;
It includes a data management unit 400 having a script editing, script and audio unit or batch automatic synchronization, verification and management functions;
It is operated in an information environment including personal computers, mobile/home/study/office information devices, servers, and networks.
In the user interface 100, the script is provided on the display window 141 on the screen, and the setting unit 160 is disposed at the lower end of the display window 141, and the user distinguishes through the operation of the setting unit 160, Selects at least one of the characteristics of the script including a fingerprint, a sentence, a word, a speaker, a problem, a question, and an answer, and responds when an area of the script corresponding to the selected item in the script including a plurality of language problems is touched all of the script parts related to the item to be performed are selected on the display window 141, so that the user can play from the beginning of the selected sentence or word, and play or repeat audio;
The application unit 200 provides a function of setting a movement unit, a search range, and an execution range by using the characteristics of the script while displaying the script on the screen in the setting module 210; The execution module 220 includes a search function reflecting the characteristics of the script; providing functions including play, pause, reverse skip, forward skip, section mark, repeat play, initialization, and double speed setting reflecting the characteristics of the script;
The database 300 is configured to include meta information 310 for learning, a script 320 , an audio 340 , and a video/image 350 ; The meta information 310 includes property information of textbooks, copyright holders, name servers, addresses of data/schema/forms, time, version, script, audio, and video/image; The script 320 includes text information synchronized in time with the audio or sound part of the audio or video, a division expressing the characteristics of a textbook, a fingerprint, a sentence, a word, a speaker, a problem, a question, and an answer; Audio 340 or video/image 350 is synchronized with the script as voice or sound; The video/image 350 is synchronized with the script text;
The data management unit 400 receives an audio file as an input from the editing module 410 and applies a function such as Speech To Text (STT) to generate text and a timetable; dividing the original script by the synchronization module 420 into a fingerprint, a sentence, a constituent element of a sentence, and a part-of-speech of a word; generating the timetable of the original script divided by using the generated text and the timetable; converting and verifying in the verification module 430 according to the format and meaning including the characteristics of the script; provides a data management method including synchronized data management between audio and script in the management module 440;
A system that provides a smart language learning service through the user interface 100, the application unit 200, the database 300, and the data management unit 400 functions. delete

Images