KR102552857B1 - Subtitle processing method for language education and apparatus thereof - Google Patents

Abstract

The present invention relates to a method and apparatus for processing captions of images for language education. The method for processing captions of images for language education includes the steps of receiving and recognizing voices of a video and receiving them as text; determining difficulty or grammatical elements of words by referring to a database for at least a portion of the text; applying a difficulty or grammatical element of a word to at least some of the texts to be distinguished from other texts; and outputting at least a part of the applied text as a subtitle on the video. Accordingly, it is possible to provide a caption service with high satisfaction to the user.

Description

Subtitle processing method and apparatus for video for language education {SUBTITLE PROCESSING METHOD FOR LANGUAGE EDUCATION AND APPARATUS THEREOF}

The present invention relates to a method and apparatus for processing video subtitles for language education, and more particularly, based on speech recognition, audio of a video is received as text, and difficulty and grammatical elements of words are determined to distinguish them from other texts. The present invention relates to a method and apparatus for processing subtitles for a video for language education, thereby increasing the efficiency of a user's language learning.

Recently, foreigners' interest in the Korean language has greatly increased. Therefore, most foreigners who want to learn Korean use various contents such as dramas, songs, and entertainment programs. At this time, foreigners aim to speak Korean freely, and for this, they start language learning from very basic characters and pronunciation. In addition, various methods are used to learn a language, such as lectures at private institutes, language training, and listening to broadcasts. However, lectures at private institutes and language training have cost and time constraints. Therefore, people who want to learn Korean try to find an effective way to learn the language that does not require a lot of cost and time, and among such methods, there is a language learning method through video.

Compared to language learning methods such as private institute lectures or language training, the video-based language learning method has the advantage of not having significant limitations in terms of cost and time.

However, although it is possible to learn through subtitles by loading pre-stored subtitle information for videos such as dramas and songs, there is a disadvantage in that some other contents that do not have pre-stored subtitle information can only be roughly understood.

In addition, it is not easy for foreigners to understand variations of derived words, transformed words, compound words, etc. at once, even for video content with pre-stored subtitle information. For example, there is a need to alleviate the degree of confusion in changing endings, such as -do, -did, -did have been derived from '-had'.

In this way, in order for learners learning through video contents to increase language learning efficiency through video, it is necessary to provide subtitles for all video contents through subtitle recognition technology through voice recognition.

An object of the present invention is to provide a method and apparatus for processing subtitles for language education in order to solve the above problems.

Specifically, an object to be solved by the present invention is to provide a video caption processing method and apparatus for language education that outputs caption information by automatically recognizing audio of a video and converting the recognized voice into text to a user.

Another problem to be solved by the present invention is video subtitles for language education that can increase the user's language learning efficiency by utilizing video subtitle recognition technology that automatically recognizes the difficulty and grammatical elements of words in output text. It is to provide a processing method and apparatus.

The tasks of the present invention are not limited to the tasks mentioned above, and other tasks not mentioned will be clearly understood by those skilled in the art from the following description.

In order to solve the above problems, a caption processing method for a video for language education according to an embodiment of the present invention includes the steps of receiving and recognizing the audio of a video and receiving it as text; determining difficulty or grammatical elements of words by referring to a database for at least a portion of the text; applying a difficulty or grammatical element of a word to at least some of the texts to be distinguished from other texts; and outputting at least a part of the applied text as a subtitle on the video. Accordingly, it is possible to provide a caption service with high satisfaction to the user.

According to another feature of the present invention, a video may be output in response to a user's input on at least one of difficulty, classification, vocabulary, and grammar for a subject the user wants to learn.

According to another feature of the present invention, the step of determining a modified word for a word consisting of at least any one of a derivative word, a synonym, a compound word, a compound word, and a single word to be distinguished from other texts in at least a part of the text may further include. there is.

According to another feature of the present invention, the grammatical element may include determining the grammatical element based on a text formed in the form of at least one of a particle, an ending, and an affix.

According to another feature of the present invention, the method may further include outputting, separately from subtitles, dictionary contents corresponding to texts having difficulty or grammatical elements of words among texts.

According to another feature of the present invention, the postpositional case further includes at least one modification of the nominative case, objective case, tubular case, adverbial case, and descriptive case, and the ending is a declarative type, interrogative type, imperative type, clarification type, exclamation type, negative type, exalted, tense, aspect , may further include at least one of the passive and passive variants.

According to another feature of the present invention, the step of distinguishing the difficulty or grammatical elements of words from other texts in at least some of the text,

A step of changing properties of text including difficulty or grammatical elements of words may be included.

According to another feature of the present invention, a step of selectively outputting prior information on the first region may be further included.

According to another feature of the present invention, the changing of the property of the text may include applying a first property to the text including the difficulty level of the word; and applying the second property to text including the grammatical element.

According to another feature of the present invention, the first property and the second property may be applied to text to be distinguished by at least one of size, color, shape, and highlight.

According to another feature of the present invention, when the first property and the second property are simultaneously changed, different properties may be applied to the first property and the second property.

According to another feature of the present invention, the step of outputting at least a part of the text as a subtitle on a moving picture,

The method may include displaying and outputting subtitles to be distinguished for each speaker in response to audio of at least one speaker included in the video.

According to another feature of the present invention, the method may further include selectively determining a conversation partner specified in the video and simultaneously outputting images of the determined conversation partner and the user.

According to another feature of the present invention, the method may further include recognizing the audio of the conversation partner's video and the user's video, respectively, and outputting a caption on the video.

According to another feature of the present invention, when the viewing of the video is finished, the step of setting a question corresponding to the difficulty or grammatical elements of words in the text output as subtitles may be further included.

An apparatus for processing video captions for language education according to an embodiment of the present invention includes a receiving unit that receives and recognizes the audio of a video and receives it as text; A database in which texts with word difficulty and texts with grammatical elements are stored; a determination unit for determining a difficulty level or a grammatical element of a word by referring to a database for at least a portion of the received text; It may include an output unit that applies the difficulty or grammatical elements of words to at least a portion of text to be distinguished from other text, and outputs at least a portion of the applied text as a subtitle on a video.

Other embodiment specifics are included in the detailed description and drawings.

According to the present invention, a caption service with high satisfaction can be provided to a user by providing caption information to the user based on voice recognition of a video.

In addition, the present invention can increase the user's language learning efficiency by utilizing a video caption recognition technology that automatically recognizes words and grammatical elements in caption information.

In addition, the present invention can grasp the user's understanding of the video by providing problems having a plurality of problem types corresponding to the caption information after learning of the video including the caption information based on voice recognition is completed.

Effects according to the present invention are not limited by the contents exemplified above, and more various effects are included in the present specification.

1 is a schematic diagram for explaining a video caption processing system for language education according to an embodiment of the present invention.
2 is a schematic diagram illustrating a user device according to an embodiment of the present invention.
3 is a schematic flowchart illustrating a method of processing captions of images for language education according to an embodiment of the present invention.
4A is a diagram illustrating an exemplary output screen for displaying a video by receiving a grammar and/or topic that a user wants to learn through a search category of a video caption processing apparatus for language education according to an embodiment of the present invention. am.
4B is a diagram illustrating an exemplary output screen for outputting a video output window corresponding to a grammar and/or topic that a user wants to learn and a chat partner information providing window for the corresponding video from a user device according to an embodiment of the present invention. am.
4C is a diagram showing an exemplary output screen for outputting an example sentence window corresponding to an example sentence menu of an image for language education according to an embodiment of the present invention.
4D is a diagram illustrating an exemplary output screen on which a conversation partner video and a user video are simultaneously output in response to a chat request menu of a video caption processing apparatus for language education according to an embodiment of the present invention.
4E is a diagram for explaining a subject search method of a video caption processing apparatus for language education according to an embodiment of the present invention.

Advantages and features of the present invention, and methods of achieving them, will become clear with reference to the detailed description of the following embodiments taken in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but will be implemented in various different forms, only these embodiments make the disclosure of the present invention complete, and common knowledge in the art to which the present invention belongs. It is provided to fully inform the holder of the scope of the invention, and the present invention is only defined by the scope of the claims.

In addition, although first, second, etc. are used to describe various components, these components are not limited by these terms, of course. These terms are only used to distinguish one component from another. Accordingly, it goes without saying that the first element mentioned below may also be the second element within the technical spirit of the present invention.

Like reference numbers designate like elements throughout the specification.

Each feature of the various embodiments of the present invention can be partially or entirely combined or combined with each other, and as those skilled in the art can fully understand, various interlocking and driving operations are possible, and each embodiment can be implemented independently of each other. It may be possible to implement together in an association relationship.

Hereinafter, various embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a schematic diagram for explaining a video caption processing system for language education according to an embodiment of the present invention.

Referring to FIG. 1 , a caption processing system for a video for language education according to an embodiment of the present invention includes a service providing device 100 and at least one user device 200 . Here, the video caption processing system for language education receives and recognizes the audio of the video and receives it as text, determines the difficulty or grammatical elements of words by referring to a database for at least a part of the text, and determines the difficulty or grammatical elements of words. A system that applies elements to at least a portion of text to be distinguished from other text, and outputs at least a portion of the applied text as a subtitle on a video.

The service providing device 100 receives a grammar and/or topic to be learned from at least one user device 200, and a database that stores a plurality of video contents 403 corresponding to the received grammar and/or topic. At least one video may be received from 202 , audio of the received video among the plurality of video contents 403 may be received and recognized, and text may be transmitted to at least one user device 200 .

Here, the grammar and/or subject to be learned may include selection of at least one of a level, classification, grammar, and subject for the subject to be learned by the user.

In addition, the plurality of video contents 403 may be contents including travel, food, K-culture, fashion, beauty, current affairs, lectures, business, etc. provided by a video caption processing system for language education.

In addition, the service providing device 100 and at least one user device 200 exist in the same network or are connected to each other through communication so that they can perform the caption processing method of video for language education and the device using the same according to the present invention. do. A detailed configuration and function of the service providing device 100 will be described later with reference to FIG. 2 .

At least one user device 200 is a device that provides a user interface for providing a video caption service for language education to a user, and may be a user's computer or a user's portable terminal. Although FIG. 1 shows that at least one user device 200 is a smart phone or a computer, the scope of the present invention is not limited thereto. As described above, at least one user device 200 may be employed without limitation as long as an application or program for providing a video caption service for language education can be loaded. For example, it may be a smart phone, a tablet PC, or a wearable device.

A corresponding application or program loaded on at least one user device 200 may connect the at least one user device 200 and the service providing device 100 to each other.

Basically, the user is provided with subtitle information based on voice recognition corresponding to at least one video according to the user's location online through a corresponding application or program. However, the present invention is not limited thereto, and stored caption information may be provided, and the user may be provided with a plurality of types of problems for the caption information thus provided. In various embodiments, a corresponding application or program may assign, for example, an online teacher to each user's account, and provide a video chat service for subtitle information or videos learned with the assigned online teacher. In addition, a problem corresponding to a video may be provided and a reward service provided according to a correct answer rate may be provided.

Specifically, at least one user device 200 may include a display displaying a screen and an input device receiving data from a user.

In various embodiments, at least one user device 200 and the service providing device 100 may include a camera module for video chatting. Specifically, at least one user device 200 may obtain user image information (eg, image information about the user device) through a camera module and transmit the acquired image information to the service providing device 100 .

In various embodiments, at least one user device 200 and the service providing device 100 may include a communication module for short-range communication (eg, Bluetooth, BLE, Zigbee, or NFC). Specifically, at least one user device 200 receives problem information, subtitle information, or information on a compensation service from the service providing device 100 through short-range communication, requests problem information to the service providing device 100, Messages about the correct/incorrect answer to the problem can be delivered.

In FIG. 1 , for convenience of description, each service providing device 100 for language education and one user device are illustrated, but the service providing device 100 and the user device are not limited thereto, and a plurality of service providing devices 100 and user devices may be provided. and the service providing device 100 may communicate with at least one user device 200 .

Hereinafter, the configuration of the service providing apparatus 100 in the video caption processing system for language education will be described. For a more detailed description, see FIG. 2 together.

2 is a schematic diagram illustrating a user device according to an embodiment of the present invention.

Referring to FIG. 2 , the user device includes a receiving unit 201 , a database 202 , a determining unit 203 and an output unit 204 .

The receiving unit 201 connects a user device to communicate with an external device. For example, the receiving unit 201 is connected to the service providing device 100 through a wired/wireless network, transmits image information of the user to the service providing device 100, and receives data to learn from the service providing device 100. At least one video is received from the database 202 storing a plurality of video contents 403 corresponding to the received grammar and/or subject, and one of the video contents 403 is received. Text can be received by receiving and recognizing the audio of the video. In various embodiments, the receiver 201 may use cellular communication, short-range wireless communication such as WiFi, Bluetooth, near field communication (NFC), or wireless communication including a Global Positioning System (GPS).

The database 202 may store various data about subtitle services of images for language education. For example, the database 202 stores an application or program for providing a caption service for a video for language education, or a grammar to be learned received from the service providing device 100 through the receiving unit 201 and/or A plurality of video contents 403 corresponding to the subject, received grammar and/or subject, and audio of a video among the plurality of video contents 403 may be received and recognized, and the received text (ie, subtitle information) may be stored. .

The determination unit 203 may determine the level of difficulty or grammatical elements of words by referring to the database 202 for at least a portion of the received text. Specifically, the difficulty level of the word may be determined by referring to words for each level of difficulty stored in the database 202, meanings of the words, and various variations of the words. In addition, the grammatical element may be determined by referring to the shape of the grammatical element and the meaning of the grammatical element stored in the database 202 .

The output unit 204 may display various contents (eg, text, image, video, icon, banner, or symbol) to the user. Specifically, the output unit 204 may display an execution screen such as an application or program for providing a subtitle service of an image for language education executed at the user's request. The output unit 203 displays a user interface for inputting the grammar and/or subject to be learned received from the service providing device 100 through the receiver 201, or responds to the grammar and/or subject to be learned. Text can be displayed by receiving and recognizing a video and a voice of the video. In various embodiments, the output unit 203 may include a touch screen, and for example, a touch using an electronic pen or a part of the user's body, a gesture, proximity, drag, and A swipe or hovering input may be received.

Although not shown in FIG. 2, the processor may be operatively connected to the receiving unit 201, the database 202, and the output unit 204, and may perform various commands for providing a subtitle service for language education. can Hereinafter, operations for receiving and recognizing video audio as text using an application providing a video caption service for language education, and displaying video and caption information according to the audio will be described in detail.

The receiving unit 201 receives input from the user on grammar and/or subject to be learned, and receives at least one video and audio of the video among the plurality of video contents 403 in response to the input. Thereafter, the voice of the received video is recognized and received as text. In this specification, although it is specified that the receiving unit 201 receives a video, receives and recognizes the audio of the received video, and receives it as text, it may also receive only a video.

The determination unit 203 may determine the level of difficulty or grammatical elements of words by referring to the database 202 for at least a portion of the received text.

The output unit 204 may apply the difficulty or grammatical elements of words to at least a portion of the text to be distinguished from other text, and output at least a portion of the applied text as a subtitle on the video. Here, the caption means caption information matched to at least one video among the video contents 403 input from the user, and is displayed on the output unit 204 of the user device 200 .

The output unit 204 may display an execution screen of an application providing a subtitle service for a video for language education according to a user's application execution request. The execution screen may automatically recognize audio information generated in a video to provide subtitle information, convert the recognized audio information into text, and display the output subtitle information. In addition, a search category input and received from a user, a large content classification menu, a small content classification menu, a keyword menu, and the like may be displayed.

In addition, the output unit 204 may display video content corresponding to a user input and a video output window, and a word menu 406, a grammar menu 407, an example menu 408, and a question menu 409 for the corresponding video. ), a chat request menu 410, and the like can be displayed.

Subtitle information automatically converted into text through the above-described speech recognition system can distinguish text including a word difficulty or grammatical element from other text, that is, text not including a word difficulty or grammatical element.

In this case, the text including the difficulty of words or grammatical elements may be changed in properties so as to be distinguished from other texts not including the difficulty of words or grammatical elements.

Specifically, the first properties 404a' and 404a'' may be applied to text including the difficulty level of words, and the second property 404b may be applied to text including grammatical elements. The first properties 404a' and 404a'' and the second properties 404b may be displayed in any one of underlining, highlighting, and color marking, respectively. In FIG. 4A, which will be described later, the first properties 404a' and 404a'' are underlined and highlighted, and the second property 404b is displayed in red letters. Also, the order of the marks displayed in the first properties 404a' and 404a'' and the second properties 404b may be changed.

Hereinafter, based on the description of at least one user device 200 according to an embodiment of the present invention described above with reference to FIGS. 1 and 2, and with reference to FIGS. 3 to 4E, images for language education are provided. A caption processing method will be described.

3 is a schematic flowchart illustrating a method of processing captions of images for language education according to an embodiment of the present invention. 4A is a diagram illustrating an exemplary output screen for displaying a video by receiving a grammar and/or topic that a user wants to learn through a search category of a video caption processing apparatus for language education according to an embodiment of the present invention. am. 4B is a diagram illustrating an exemplary output screen for outputting a video output window corresponding to a grammar and/or topic that a user wants to learn and a chat partner information providing window for the corresponding video from a user device according to an embodiment of the present invention. am. 4C is a diagram showing an exemplary output screen for outputting an example sentence window corresponding to an example sentence menu of an image for language education according to an embodiment of the present invention. 4D is a diagram illustrating an exemplary output screen on which a conversation partner video and a user video are simultaneously output in response to a chat request menu of a video caption processing apparatus for language education according to an embodiment of the present invention. 4E is a diagram for explaining a subject search method of a video caption processing apparatus for language education according to an embodiment of the present invention.

First, the apparatus for processing video captions for language education according to an embodiment of the present invention receives and recognizes the audio of a video from a user device (user device 200) and receives it as text (S100).

Referring to FIG. 4A , the display 203 of at least one user device 200 according to an embodiment of the present invention includes a video content to be learned using at least one of a search category 401 and a content category 402. A plurality of videos corresponding to (403) can be output and displayed.

Referring to FIG. 4A , a search category 401 may be implemented to the user through a large content category menu 401a, a small content category menu 401b, and a keyword menu 401c. Through the main content classification menu 401a and the content subcategory menu 401b, the user can set a subject to be studied. The user may set a word or grammar that the user wants to learn through the keyword menu 401c. The search category 401 may receive an input for at least one of a large content classification menu 401a, a small content classification menu 401b, and a keyword menu 401c.

Referring to FIG. 4A , a content category 402 is a category-by-category classification for easy identification of a subject the user wants to learn, and can be implemented by the user. For example, the content category 402 can be broadly classified into Urirang patterns, travel, food, K-culture, fashion, beauty, current events, lectures, and business provided by a video subtitle service for language education. Here, K-culture can be subclassed into K-POP, K-DRAMA, and others.

In FIG. 4A, the search category 401 is illustrated as being composed of a large content classification menu 401a, a subcontent classification menu 401b, and a keyword menu 401c, but as shown in FIG. 4E, a more detailed search may be performed. . This will be described later with reference to FIG. 4E.

For example, when a user selects business as a subject to study, as shown in FIG. 4A , a plurality of video contents 403 corresponding to the business may be displayed on the output unit 204 . Each of the video contents 403 may display information about the level (or difficulty) of the video contents 403, playback time, coins required for playback, number of views, interest, and the like.

Hereinafter, with reference to FIGS. 4B and 4C , a case in which a user selects a video about innovation in the pizza market with a difficulty level among a plurality of video contents 403 will be described in detail.

Subsequently, the apparatus for processing captions for language education according to the present invention refers to the database 202 for at least a portion of text and determines the level of difficulty or grammatical elements of words (S200). Subsequently, the apparatus for processing captions of images for language education according to an embodiment of the present invention applies a level of difficulty or a grammatical element of a word to at least a part of text to be distinguished from other texts (S300).

Referring to FIG. 4B , among the plurality of video contents 403 output through the search category 401 as described above, a video output window 404 corresponding to a user's input may be displayed on the output unit 204. .

Referring to FIG. 4B , the video output window 404 may display text including first properties 404a' and 404a'' and second properties 404b. Here, the text means that audio information for a video is automatically recognized and the audio information is automatically converted into text information. ) may be composed of this text and the rest of the text.

Here, the first properties 404a' and 404a'' are applied to text including the difficulty level of words, and the second property 404b is applied to text including grammatical elements. When only one of the first properties 404a' and 404a'' and the second property 404b is changed in the text constituting the subtitle information, at least one property of size, color, shape, and highlight is applied to the text. However, when the first properties 404a' and 404a'' and the second property 404b are simultaneously changed in the text constituting the subtitle information, the first properties 404a' and 404a'' and the second property It is characterized in that different attributes are applied to (404b) to distinguish them.

In addition, the subtitle information is characterized in that it is divided into sentences of a certain length and output on the image of the video output window 404 .

The plurality of word forms may include variants for words formed of at least one of a derivative word, a synonym, a compound word, a compound word, and a single word, and the variant word distinguishes at least some of the text constituting the subtitle information from other texts. It may include the step of deciding to be.

In addition, the grammatical element is characterized in that it is determined based on the text formed in the form of at least one of a particle, an ending, and an affix. Here, the modal case may further include at least one modification among the nominative case, the objective case, the tubular case, the adverbial case, and the descriptive case.

In addition, the ending may further include at least one modification among declarative, interrogative, imperative, inquiry, exclamatory, indefinite, exalted, tense, modal, passive, and passive variations.

In addition, an apparatus for processing captions of images for language education according to another embodiment of the present invention may output dictionary contents corresponding to texts having difficulty or grammatical elements of words among texts separately from captions.

Specifically, when the user clicks the corresponding text for dictionary content on text having at least one of the difficulty of the word and grammatical elements, a pop-up window including the dictionary content may be generated separately from the subtitle, and a pop-up window may be generated at the top of the subtitle. And dictionary content may be displayed on either side of the bottom.

That is, by selecting a word to which the first properties 404a' and 404a'' are applied without selecting the word menu 406 of the display 203, the dictionary meaning of the word can be displayed in a new window. . At this time, the dictionary information may be displayed in a pop-up form within the video output window 404, and a corresponding site may be output by automatically connecting a link to a dictionary in which a dictionary meaning of a corresponding word is searched.

Although not shown in FIG. 4B, the apparatus for processing video captions for language education according to another embodiment of the present invention creates dictionary information for a user by separately storing dictionary contents on the difficulty level and grammatical elements of a word. can Accordingly, the user can store and display user-only dictionary information for each learned video, or store and display dictionary information for all learned videos.

Therefore, after the user completes learning of a video including subtitle information based on voice recognition, the user can effectively learn unknown words or grammar through dictionary information stored separately by the user.

In addition, in the apparatus for processing captions of images for language education according to another embodiment of the present invention, text in word form among text constituting caption information among images in the video output window 404 is converted into video images corresponding to the text. You can indicate that you are responding. For example, when the word 'pizza' is displayed in the first area and an image related to pizza is displayed in the video output window 404, a display indicating that the word and the image of pizza are the same may be provided. In this case, the display may be a block process for the corresponding image, a circle display, or a connection between an image and a word through a line.

In addition, the grammatical element may include at least one of a particle, an ending, and an affix. Here, the particle further includes at least one variant of the nominative case, objective case, tubular case, adverbial case, and descriptive case, and the ending is a declarative interrogative type, an imperative type, a hearing type, an exclamation type, an indefinite type, an exaltation type, a tense type, a mode, a passive type, and a passive type. At least one modification may be further included.

The above-described first properties 404a' and 404a'' and the second property 404b may be at least one of the size, color, shape, and highlight of characters constituting the caption information, and the first property 404a' , 404a″), the second property 404b, and the remaining areas may all have different properties. That is, the remaining area may mean characters having no attributes.

In FIG. 4B , among a plurality of components constituting subtitle information, referring to the database 202, 'huge', which is a text having a word difficulty, is indicated as a first attribute 404a', and 'excitement' is the first property 404a. 1 attribute (404'').

In addition, among a plurality of components constituting subtitle information, the database 202 is referred to, and text including grammatical elements is displayed as a second marker 404b. Although underline, highlight, and red attributes are applied to the first attributes 404a' and 404a'' and the second attributes 404b, respectively, the first attributes 404a' and 404a'' and the second attributes 404b If different properties are applied to , it may not be limited thereto.

Referring to FIG. 4B , the output unit 204 according to an embodiment of the present invention may output a conversation partner information providing window 405 . For example, the conversation partner information providing window 405 displays information about online teachers corresponding to the video output window 404 and provides profile information for each teacher, so that the user can select a teacher to talk to. The conversation request menu 410 can be output and displayed together so that conversation can be made.

Hereinafter, an exemplary output screen corresponding to a user's input corresponding to the chat request menu 410 will be described in detail with reference to FIG. 4D.

Referring to FIG. 4B , the display 203 according to an embodiment of the present invention includes a word menu 406 for learning on a video output window 404, a grammar menu 407, an example sentence menu 408, and a question menu. (409) may be further included.

The word menu 406 may be a menu for providing pronunciation display and meaning of a plurality of words included in a video corresponding to a video input that a user wants to learn. For example, as shown in FIG. 4B , a video including subtitle information output from the video output window 404 includes words such as 'huge' and 'exciting', and each word is a first mark 404a. ) and a second mark 404b. In this case, when the user selects the word menu 406 to study words indicated by the first mark 404a and the second mark 404b, derivatives, pronunciations, and meanings of each word may be provided. For example, huge, [huge], to be huge, great are displayed as the derivation, pronunciation and meaning of the word for 'huge', and [excited] as the pronunciation and meaning of the word for 'excited'. , to be turn up slightly is displayed. Here, it is based on being displayed in a foreign language studied Korean for understanding the meaning of the word, but is not limited thereto.

The grammar menu 407 may be a menu for providing grammar included in a video corresponding to a video input that the user wants to learn, that is, a Korean language pattern. For example, as shown in FIG. 4B , a video including subtitle information output from the video output window 404 has 'going', which is text including a grammatical element such as 'doing', as a second attribute. 404b. In this case, when the user selects the grammar menu 407 to learn the grammar indicated by the second attribute 404b, grammar information corresponding to 'having' may be displayed. That is, the pattern of the corresponding grammatical element, the meaning of the corresponding pattern, and example sentences using the corresponding pattern may be displayed together. As shown in FIG. 4B, 'there' is a grammar having a pattern of '(verb) + ~', which has a second property 404b, and means that it is an expression indicating that a certain action or state is in progress. And examples of using the above pattern may include, 'recently, a lot of watermelons are sold', 'I am studying', and the like.

The example sentence menu 408 may be a menu for providing information on all subtitles included in a video corresponding to a video input that the user wants to learn. Here, although the caption information according to an embodiment of the present invention is characterized in that it is output based on voice recognition, the caption data may be received from the database 202 by storing the caption data in advance in the database 202 . For example, as shown in FIG. 4C , subtitle data is received from the database 202 in which subtitle data matching the video is previously stored on the side of the video output from the video output window 404 and displayed on the example sentence window 418. can do. There is an effect that the user can grasp words and grammatical patterns included in the video that the user wants to learn at a glance through the example sentence window 418 .

Subsequently, the question menu 409 may be a menu for providing a plurality of pre-provided questions to determine the degree of understanding of the video learned by the user. Accordingly, users ask a question provided in the question menu 409 after learning about the video, for example, 'Have you eaten Korean pizza? Questions such as 'How was it?' may be provided. Also, although not shown in FIG. 4C , a recording function may be further provided so that feedback on questions provided in the question menu 409 can be made.

In addition, after thinking about answers to the questions provided in the provided question menu 409, the user can have a conversation with the online teacher about the questions through the conversation request menu 410 shown in FIG. 4C.

Referring to FIG. 4D , a conversation partner image 411 and a user image 412 corresponding to the chat request menu 410 selected by the user are simultaneously output to the output unit 204, and the output unit 203 allows the user to learn An example sentence window 418 for one video may be output together. Therefore, the user can have a conversation about the questions provided in the question menu 409 or the questions in the example sentence window 418 while viewing the example sentence window 418 displayed at the same time during the conversation with the conversation partner, that is, the teacher. there is.

In this case, the conversation partner image 411 and the user image 412 may output and display subtitle information in real time through a voice recognition system. That is, as shown in FIG. 4D , the voice of the conversation partner image 411 is recognized and the first caption 411a is output on the conversation partner image 411 . In addition, the voice of the user image 412 is recognized and the second caption 412a is output on the user image 412 . Here, the conversation partner image 411 and the user image 412 are characterized in that they are output simultaneously, but are not limited thereto, and may be output on different screens.

The user performing feedback on the video learned through the conversation partner video 411 and the user video 412 including the first subtitle 411a and the second subtitle 412a automatically recognizes and outputs the subtitle Through the information, there is an effect of understanding the conversation with the conversation partner more quickly and accurately.

Also, although not shown in FIG. 4D , text including a difficulty level or a grammatical element of a word may be divided into a first property and a second property, respectively, and displayed in the first caption 411a and the second caption 412a. Accordingly, the user can grasp the difficulty level or grammatical elements of a word at a glance while having a conversation with a teacher who is a conversation partner.

Hereinafter, with reference to FIG. 4E, a subject search method of the caption processing apparatus for a video for language education will be described in detail.

Referring to FIG. 4E , a user may perform a detailed subject search through a subject search menu in addition to a search category 401 composed of a content major category menu 401a, a content subcategory menu 401b, and a keyword menu 401c.

Referring to FIG. 4E , the user may utilize the writing system by selecting the level, classification, grammar, and subject of the subject through the subject search menu. As shown in FIG. 4E, the user selects beginner level as the level, 'something new, try' as the classification, 'see ah/uh' as the grammar to learn, and travel as the subject. When is selected, a sentence corresponding to the user's selection may be automatically provided. For example, in response to the selection, example sentences such as 'try bibimbap in Jeonju' may be output and displayed, and 'swim in Jeju Island' including 'see -hae', which is a variation of 'see -ah/uh'. An example sentence such as 'try it' may be output and displayed.

In FIG. 4E , the sentences are shown to be simultaneously output to the subject selection menu and the search sentence menu, but may be sequentially output.

Specifically, among the subject selection example sentences output in response to the user's input corresponding to classification in the subject search menu, when the user wants to write a composition in relation to 'swimming in Jeju Island', the user can enter the sentence directly in the search sentence menu. can be entered.

When the user inputs a sentence in the search sentence menu and selects the search menu, a more accurate sentence can be provided while automatically checking for misspellings and inscriptions in the sentence entered by the user. For example, when a user inputs 'try eating bibimbap with friends in Jeonju tomorrow' in the search sentence and selects the search menu, the suggested sentence menu corrects incorrect expressions in the search sentence and outputs a sentence with the correct expression. can be displayed That is, the suggestion sentence menu displays various usage examples so that the user can use the correct expression by providing expressions such as 'I will try bibimbap with a friend tomorrow in Jeonju' or 'I ate bibimbap with a friend in Jeonju today'. can do.

Accordingly, in the apparatus and method for processing captions of images for language education according to an embodiment of the present invention, an automatic caption recognition technology is used for a user to easily identify and learn grammar knowledge and words, Here, by providing examples of usage used in real life through example sentences, it has the effect of improving the learning ability for colloquial speech.

In addition, by matching the online teacher desired by the user to the learned video, feedback on the online teacher and the video can be performed in real time, so that not only grammar, reading and writing, but also speaking can be learned.

Accordingly, a user who uses the apparatus and method for processing captions for language education according to an embodiment of the present invention can quickly and efficiently identify words and grammatical elements within caption information to increase language learning efficiency.

In addition, in the apparatus and method for processing video captions for language education according to an embodiment of the present invention, after learning of a video including caption information based on voice recognition is completed, a plurality of problem types corresponding to the caption information are determined. It is possible to grasp the level of understanding of the user's video by providing problems with the user.

In this specification, each block or each step may represent a module, segment, or portion of code that includes one or more executable instructions for executing specified logical function(s). It should also be noted that in some alternative embodiments it is possible for the functions recited in blocks or steps to occur out of order. For example, two blocks or steps shown in succession may in fact be performed substantially concurrently, or the blocks or steps may sometimes be performed in reverse order depending on their function.

The steps of a method or algorithm described in connection with the embodiments disclosed herein may be directly embodied as hardware executed by a processor, a software module, or a combination of the two. A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, and the processor can read information from, and write information to, the storage medium. Alternatively, the storage medium may be integral with the processor. A processor and storage medium may reside within an application specific integrated circuit. An ASIC may reside within a user terminal. Alternatively, the processor and storage medium may reside as separate components within a user terminal.

Although the embodiments of the present invention have been described in more detail with reference to the accompanying drawings, the present invention is not necessarily limited to these embodiments, and may be variously modified without departing from the technical spirit of the present invention. . Therefore, the embodiments disclosed in the present invention are not intended to limit the technical idea of the present invention, but to explain, and the scope of the technical idea of the present invention is not limited by these embodiments. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. The protection scope of the present invention should be construed according to the claims below, and all technical ideas within the equivalent range should be construed as being included in the scope of the present invention.

100: service providing device
200 at least one user device
201: receiver
202: database
203: decision unit
204: output unit
401: search category
401a: Main Content Classification Menu
401b: content subcategory menu
401c: keyword menu
402: content category
403: video content
404: video output window
404a', 404a'': first attribute
404b: second attribute
405: Contact information provision window
406: word menu
407: grammar menu
408: example menu
409: Question menu
410: conversation request menu
411: Buddy video
411a: first subtitle
412 User image
412a: second subtitle
418: Example sentence window

Claims (16)

outputting a video related to a subject that a user wants to learn through a user input and at least one of difficulty, classification, and grammar for the subject;
obtaining text by receiving and recognizing the audio of the video;
determining at least some texts having difficulty or grammatical elements of words among the texts by referring to a database;
determining a property for distinguishing the at least some texts having the difficulty level or the grammatical elements from other texts; and
outputting at least a portion of the text, the property of which is determined, as a subtitle on the video; Including,
The step of determining the at least some text,
A word corresponding to a derivative, a synonym, a compound word, a compound word, and a modified word in the form of a single word is determined as a text having the above level of difficulty, and the propositional, declarative, and interrogative forms include at least one modification among the subject case, object case, tubular case, adverbial case, and predicate case. , Imperative type, request type, exclamatory type, indefinite type, elevation, tense, mode, passive and inactive, and determining words consisting of endings and affixes including at least one modification example as text having the grammatical elements,
In the step of outputting the subtitle,
A step of outputting an output screen including a word menu, grammar menu, and question menu area for providing a learning service through interaction with the subtitle to an area adjacent to a window for outputting the video,
The output screen is
When any one word constituting the text is selected from the word menu through a user input, it is configured to output the derivative, pronunciation, and meaning of the word,
When a text having the grammatical element is selected from among the texts through a user input in the grammar menu, a pattern for the grammatical element, the meaning of the pattern, and a usage example are output,
After the learning of the video is completed, a question related to the subject, the difficulty, or the text having the grammatical elements is output through a user input in the question menu, and a user answer is recorded for feedback on the question. configured so that
After the step of outputting the subtitle,
outputting at least a portion of text having the difficulty level or the grammatical element among texts for user answers obtained through recognition of the recording; Further comprising a subtitle processing method of video for language education. delete delete delete According to claim 1,
The method of processing video captions for language education, further comprising outputting, separately from the captions, dictionary contents corresponding to the at least some texts having the difficulty level or the grammatical element among the texts. According to claim 5,
The method of processing subtitles for a video for language education, further comprising storing the dictionary content for the difficulty level or the grammatical element as separate dictionary information. delete delete According to claim 1,
To determine the properties,
applying a first property to the text having the difficulty level; and
A method of processing video captions for language education, comprising applying a second property to text having the grammatical elements. According to claim 9,
The first property and the second property are
A method for processing video captions for language education, wherein the text is distinguished by at least one attribute of size, color, shape, and highlight. According to claim 9,
When the first property and the second property are simultaneously applied to any part of the text, the first property and the second property, which are different from each other, are applied and distinguished. According to claim 1,
In the step of outputting the subtitle,
A method of processing video captions for language education, comprising displaying and outputting captions to be distinguished for each speaker in response to audio of at least one speaker included in the video. According to claim 1,
The video caption processing method for language education further comprising the step of selectively determining a conversation partner specified in the video and simultaneously outputting images of the determined conversation partner and the user. According to claim 13,
The method of processing captions for a video for language education, further comprising recognizing the audio of the conversation partner's video and the user's video, respectively, and outputting a caption on the video. delete a receiving unit configured to receive and recognize audio of a video related to a subject to be learned by a user and at least one of difficulty, classification, and grammar of the subject through a user input, and obtain text;
a database configured to store texts with word difficulty and texts with grammatical elements;
a determination unit configured to determine at least some of the texts having difficulty or grammatical elements of words among the texts by referring to the database among the texts acquired through the reception unit;
An output unit configured to determine a property for distinguishing the at least some text having the difficulty or the grammatical element from other text, and to output at least some of the text, the property of which is determined, as a subtitle on the video; ,
The decision section,
A word corresponding to a derivative, a synonym, a compound word, a compound word, and a modified word in the form of a single word is determined as a text having the above level of difficulty, and the propositional, declarative, and interrogative forms include at least one modification among the subject case, object case, tubular case, adverbial case, and predicate case. , imperative, hearing, exclamatory, indefinite, elevated, tense, modal, passive, and passive, and determining words consisting of endings and affixes including at least one variation among the grammatical elements as text having the grammatical elements,
the output unit,
It is configured to output an output screen including a word menu, grammar menu, and question menu area for providing a learning service through interaction with the subtitle to an area adjacent to the window for outputting the video,
The output screen is
When any one word constituting the text is selected from the word menu through a user input, it is configured to output the derivative, pronunciation, and meaning of the word,
When a text having the grammatical element is selected from among the texts through a user input in the grammar menu, a pattern for the grammatical element, the meaning of the pattern, and a usage example are output,
After the learning of the video is completed, a question related to the subject, the difficulty, or the text having the grammatical elements is output through a user input in the question menu, and a user answer is recorded for feedback on the question. configured so that
the output unit,
The subtitle processing apparatus for video for language education, further configured to output at least a portion of text having the difficulty level or the grammatical element among texts for user answers obtained through recognition of the recording.

Images