KR20110034797A - Document processing apparatus and method - Google Patents

Abstract

PURPOSE: A document processing apparatus and a method thereof are provided to supply new and various example sentences by automatically and continuously collecting the example sentences without a manual work. CONSTITUTION: An example collecting unit(110) collects, from a web, a first example sentence which is written in a first language, and a second example sentence which is the translation of the first example sentence. An example providing unit(170) provides supplies a user with a parallel example including the first and second example sentences. The example providing unit extracts from an input document the expression list used for learning and searches the expression included in the expression list in order to obtain the first example sentence including the expression.

Description

Document processing apparatus and method {DOCUMENT PROCESSING APPARATUS AND METHOD}

TECHNICAL FIELD The present invention relates to a document processing apparatus and method, and more particularly, to a document processing apparatus and method for collecting a document on the web and processing the collected document.

Foreign or electronic foreign languages published in paper format are produced by the foreign language expert. Examples in the dictionary are usually written by hand, and the dictionary content is typically updated only after a long period of time by changing the dictionary's version. Recently used web dictionary is based on the data contained in the existing electronic dictionary. In the case of web dictionaries, new examples are continuously added with the translations, allowing data to be expanded and updated in a relatively short time, but this is also done manually. As it is necessary to go through human hands to construct examples in paper dictionaries, electronic dictionaries or web dictionaries, it is not only scalable but also difficult to update new examples, and is continuously maintained and managed to add new usages of new words or existing expressions. This requires a lot of capital, time and manpower.

On the other hand, the example sentences provided in sentence units are merely listed each example sentences and translations it is difficult for the user to learn in-depth about the example sentences.

The problem to be solved by the present invention is to provide a document processing apparatus and method capable of providing new and more various examples by providing automatic and continuous collection of example sentences without human intervention, and can provide example analysis information related to the example sentences. .

The document processing apparatus according to an embodiment of the present invention for solving the technical problem, a first sentence consisting of a first example and a second language consisting of a first language, the example sentence for collecting a second example sentence which is a translation of the first example sentence from the web And a collection example and an example sentence providing unit for providing the user with a parallel example sentence composed of the first example sentence and the second example sentence in response to a request from the user.

The example sentence collection unit may extract the expression list to be used for learning from an input document, and obtain the first example sentence including the expression by searching the web for an expression included in the expression list.

If the first document containing the first example sentence includes a portion written in the second language, or the document linked by a hyperlink included in the first document includes a portion written in the second language. The example sentence collection unit may extract a portion written in the second language and include it in the candidate example sentence list as a candidate example sentence of the second example sentence.

The example sentence collection unit may extract the second example sentence from the candidate example sentence by comparing the first example sentence with the candidate example sentence and determining whether the candidate example sentence is a translation sentence of the first example sentence.

The example sentence providing unit may generate a search word by analyzing a syntax structure of the query word input by the user, and search for a parallel example sentence including the search word in a database and provide the search word to the user.

An example sentence analyzing unit configured to analyze the syntax structures of the first example sentence and the second example sentence and to generate alignment information for identifying a correspondence relationship between the expression included in the first example sentence and the expression included in the second example sentence It may include.

The example sentence providing unit may provide the parallel example sentence to the user along with example sentence analysis information including the alignment information.

Example sentence analysis information including the sorting information and the database for storing the parallel example sentence may further include.

The first language may be Korean, and the second language may be English.

A foreign language education system according to another aspect of the present invention includes any of the above-described document processing apparatus.

According to another aspect of the present invention, there is provided a document processing method comprising: collecting, from a web, a second example sentence comprising a first example sentence in a first language and a second language, which is a translation of the first example sentence, and a request from a user. And providing the user with a parallel example sentence composed of the first example sentence and the second example sentence.

The collecting may include extracting a list of expressions to be used for learning from an input document, and obtaining the first example sentence including the expressions by searching the web for expressions included in the expression list. Can be.

The collecting step may include a portion written in the second language in a first document including the first example sentence or a hyperlink included in the first document in a second document. If a part is included, extracting a part written in the second language may be included in the candidate example sentence list as a candidate example sentence of the second example sentence.

The collecting step may further include extracting the second example sentence from the candidate example sentence by comparing the first example sentence with the candidate example sentence and determining whether the candidate example sentence is a translation of the first example sentence.

The providing may include analyzing a syntax structure of a query input by the user to generate a search word, and searching for a parallel example sentence including the search word in a database and providing the search word to the user.

Analyzing the syntax structures of the first example sentence and the second example sentence, and generating alignment information for identifying a correspondence relationship between the expression included in the first example sentence and the expression included in the second example sentence. can do.

The providing step may include providing the parallel example sentence to the user along with example sentence analysis information including the alignment information.

The method may further include storing example sentence analysis information including the alignment information and the parallel example sentence.

The first language may be Korean, and the second language may be English.

The foreign language teaching method according to another aspect of the present invention includes any of the above-described document processing methods.

A computer readable medium according to another aspect of the present invention records a program for causing a computer to execute any of the above methods.

As described above, according to the present invention, parallel example sentences in which a pair of example sentences written in a first language and a translation sentence written in a second language of the example sentences are paired can be continuously and automatically collected from the web without human intervention. Can provide new and more varied examples. In addition, by providing the expression of the expression included in the example sentence and the translation corresponding to the expression to the user can improve the learning efficiency of the user.

DETAILED DESCRIPTION Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art may easily implement the present invention.

First, a document processing apparatus according to an embodiment of the present invention will be described in detail with reference to FIG. 1.

1 is a block diagram illustrating a document processing apparatus according to an embodiment of the present invention.

Document processing apparatus 100 according to an embodiment of the present invention includes a sentence sentence collection unit 110, a sentence sentence analysis unit 130, a database 150, and a sentence sentence providing unit 170, through the communication network 400 The plurality of web servers 200 and the user terminal 300 are connected. The document processing apparatus 100 collects parallel example sentences from the web server 200 and stores additional information obtained by analyzing the collected parallel example sentences together with the parallel example sentences. In addition, the document processing apparatus 100 receives a query from the user terminal 300 and provides a parallel example sentence and example sentence analysis information related thereto corresponding to the query to the user terminal 300.

In this case, the parallel sentences are composed of a pair of original sentences containing a specific word or expression and translations thereof. The original example sentence means a sentence, phrase, clause, word, etc. expressed in an arbitrary language (hereinafter referred to as a 'first language'), and a translation thereof refers to an arbitrary subject language (hereinafter referred to as a 'second language'). Means expressed as. For example, if the first language is Korean and the second language is English, the parallel example sentences are Korean-English translation sentences. Of course, the parallel example sentences may be English-English translation sentence pairs, and the first and second languages may be languages different from Korean and English.

The user terminal 300 is a communication terminal device used by a user to receive a web service. The user terminal 300 accesses a web server 200, a document processing apparatus 100, and the like through the communication network 400 to exchange information. The user terminal 300 may be a desktop computer as well as a laptop computer, a workstation, a palmtop computer, an ultra mobile personal computer, a tablet PC, a personal digital assistant (PDA), a web pad, a mobile phone. It may be made of a terminal having a computing power by including a memory means and a microprocessor.

The communication network 400 may include a telephone network as well as a data communication network including a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), the Internet, and the like. It can be used, and it does not matter wired and wireless, and it does not matter which communication method is used.

The web server 200 provides a web page in response to a request from the user terminal 300 and the document processing apparatus 100 through the communication network 400. The web server 200 may provide a web page through a portal site or a general web site that provides blogs, cafes, knowledge information, specialized materials, news, and the like, and the document processing apparatus 100 may be included in a web page. You can extract parallel sentences from various documents.

Then, each component included in the document processing apparatus 100 will be described in more detail with reference to FIGS. 2 and 3.

2 is a block diagram of the example sentence collection unit 110 shown in FIG. 1, and FIG. 3 is a block diagram of the example sentence analysis unit 130 shown in FIG. 1.

Referring to FIG. 2, the example sentence collection unit 110 includes a syntax structure analyzing unit 111, an example sentence searching unit 113, a text extracting unit 115, and a sentence alignment unit 117, and includes a specific expression. The document is searched through the communication network 400 to generate parallel example sentence data, and the generated parallel example sentence data is provided to the example sentence analyzing unit 130.

The syntax structure analyzing unit 111 analyzes the input document in morpheme or word units, and analyzes words and phrase boundaries based on the morphemes, words, and the like. Create a list of expressions consisting of phrases, clauses, etc. Here, the input document may be any document extracted on the web or any document input by an administrator of the document processing apparatus 100, and a document in one specific language is preferable, but is not limited thereto. Hereinafter, it is assumed that the input document is made of the first language. And expression is a unit for conveying at least one meaning, consisting of morphemes, words or a combination thereof. In this case, the syntax structure analyzing unit 111 is researched in the field of natural language processing such as Dan Klein and Christopher D. Manning's article "Accurate Unlexicalized Parsing" and "Stanford parser", which is published as an actual implementation thereof. By using the developed technology, you can automatically parse the syntax of input documents.

The example sentence search unit 113 accesses a plurality of web servers 200 through the communication network 400 to search for a document based on a list of expressions, and generates a search document set including an original example sentence and a translation candidate document. Here, the original example text indicates that the document containing the expressions in the expression list is written in the first language, the translation candidate document indicates a document that is likely to be translated in the second language, and the search document set is Represents a set of documents composed of pairs of original text examples and corresponding translation candidate documents. The example sentence search unit 113 may search for a document using a search engine of its own or search for a document using a search engine of a web server 200 that provides a search site.

In more detail, the example sentence search unit 113 extracts a specific expression from the expression list, and retrieves a document containing the expression through the communication network 400 to extract the original sentence sentence containing the expression. When the retrieved document is written in the first language and the second language, since the portion written in the second language is more likely to be a translation of the part written in the first language, the example sentence search unit 113 may determine from the retrieved document. The part written in the second language is included in the translation candidate document list. In addition, when the searched document includes a hyperlink portion and is designed as a multilingual web page, it is highly likely that a second language web page having the same structure and content is linked to the document. ) Includes the hyperlinked document in the second candidate language in the translation candidate document list. The example sentence search unit 113 generates a search document set consisting of a plurality of original example sentences and pairs of translation candidate documents corresponding to the expressions in the expression list through the above search process.

The text extractor 115 extracts the text portion written in the first language from the original sentence examples of the search document set to generate the first text document, and extracts the text portion written in the second language from the translation candidate document corresponding to the original sentence example. By extracting and generating a second text candidate document, a text example sentence set consisting of the first text document and the second text candidate document is generated. Thus, it is possible to obtain a pair of documents in a first language and documents in a second language that may be a translation of this document or possibly contain some translation of this document.

The sentence aligning unit 117 checks whether a punctuation mark, a special symbol, a number, etc. are matched, compares sentence lengths, dictionary information, etc. by pairing the first text document of the text example sentence set and the second text candidate document corresponding thereto. The method determines whether the sentences included in the first text document and the sentences included in the second text candidate document are translations of each other by checking whether expressions included in the first text document exist in the second text candidate document. Then, the sentence aligning unit 117 extracts the first sentence and the second sentence determined to be translations of each other. Here, the first sentence represents a sentence written in the first language, and the second sentence represents a sentence written in the second language. The sentence aligning unit 117 repeatedly performs the above-described inspection process on the pairs of the first text document of the text example sentence set data and the second text candidate document corresponding thereto, and thus the plurality of first sentences and the second sentences corresponding thereto. Generate paired parallel example data. In this way, you can get parallel sentences sorted by sentence. Of course, the plurality of first sentences and the corresponding plurality of second sentences may be aligned in paragraph units to form parallel sentences, or not in sentences, but may be aligned in phrase units to form parallel sentences.

Referring to FIG. 3, the example sentence analyzing unit 130 includes a first syntax structure analyzing unit 131, a second syntax structure analyzing unit 132, and a word alignment unit 135, and analyzes parallel example sentence data to provide additional information. And stores it in the database 150 along with the parallel example data.

The first and second syntax structure analyzing units 131 and 133 analyze syntax structures of sentences written in the first and second languages, respectively. That is, the first syntax structure analyzer 131 analyzes the syntax structure of the first sentence in the parallel example sentence, and the second syntax structure analyzer 133 analyzes the syntax structure of the second sentence in the parallel example sentence. Like the syntax structure analyzing unit 111 described above, the first and second syntax structure analyzing units 131 and 133 analyze each sentence in morphemes or word units to determine morphemes, words, phrases, clauses, and the like. Generates syntax structure information. On the other hand, the first syntax structure analysis unit 131 may be implemented by borrowing the syntax structure analysis unit 111 of the example sentence collection unit 110 may be implemented independently of this.

The word alignment unit 135 generates word alignment information by performing word alignment based on syntax structure information of the first and second sentences. Here, word alignment may be performed in morpheme units for accuracy. The word sorter 135 may use a word sorting technique widely used in statistical machine translation. For example, the paper "The Mathematics of Statistical Machine Translation: Parameter Estimation" by Peter F. Brown et al. As with FJ Och's "GIZA ++", you can use publicly available software such as algorithms or word alignment algorithms published by the paper. Through such word alignment, the correspondence between the expressions included in the first sentence and the second sentence may be known.

The example sentence analysis unit 135 stores example sentence analysis information including syntax structure information and word alignment information generated based on one parallel example sentence in the database 150 together with the parallel example sentence data. In this case, the pair of first sentences and the second sentence are stored as one document, and the example sentence analysis information is stored together in each document.

On the other hand, the sentence sentence analysis unit 130 may additionally perform various analysis, such as pronunciation string, rhyme, etc. from the parallel sentence sentence in addition to the syntax structure information or word alignment information, by adding the information derived as a result to the sentence sentence analysis information database ( 150).

Referring back to FIG. 1, the database 150 stores parallel example sentence data and example sentence analysis information provided by the example sentence analysis unit 130, and provides parallel example sentence data and example sentence analysis information at the request of the example sentence provider 170. do.

The example sentence providing unit 170 performs an indexing operation so that the database 150 can be easily searched, and targets a document in which example sentence analysis information is included in the database 150. Also, when the example sentence providing unit 170 receives a query word from the user terminal 300, the sentence structure analysis unit 131 and 133 analyzes the syntax structure of the query word and extracts the most suitable search word through the analysis. After the preprocessing process, the document including the extracted search word is searched in the database 150 and the searched document is provided to the user terminal 300. Accordingly, as a result of the search, a document regarding a sentence that best matches a user query among documents included in the database 150 may be provided to the user terminal 300, and the sentence sentence analysis information may be provided as well as parallel sentences. This can be useful to the user.

As such, according to the document processing apparatus 100 according to the exemplary embodiment of the present invention, an online dictionary may be automatically constructed by automatically and continuously collecting parallel example sentences containing a specific expression on the web, thereby providing a user with more information. New and different examples can be provided. In addition, by providing not only the example sentence but also the analyzed information to the user, the learning efficiency of the user can be improved.

The document processing apparatus 100 according to the embodiment of the present invention may be implemented in an independent server form, but may be implemented in an integrated form with the user terminal 300. That is, all of the functions of the document processing apparatus 100 may be implemented in the user terminal 300 to collect and analyze the parallel example sentences in the user terminal 300, and retrieve and provide the parallel example sentences to the user according to the user's request. Of course, some of the functions of the document processing apparatus 100 may be implemented in the user terminal 300.

In addition, the document processing apparatus 100 may be used to collect parallel example sentences for foreign language education, and thus may be included in various systems (not shown) for foreign language education. Examples of such systems include, but are not limited to, online dictionary systems or interactive educational systems. In addition, the document processing apparatus 100 according to an embodiment of the present invention may be utilized in a system for statistical based and example based automatic translation.

Meanwhile, the parallel example sentences collected by the document processing apparatus 100 may be used as parallel data necessary for statistical-based and example-based automatic translation techniques, and may be utilized as learning materials in general software for foreign language learning, including a dialogue system. have. In addition, the sentence analysis information after the word alignment process may be used for automatic construction of the word and phrase dictionary.

Next, a document processing method according to an embodiment of the present invention will be described with reference to FIG. 4.

4 is a flowchart illustrating a method of collecting parallel example sentences according to an embodiment of the present invention, and FIG. 5 is a flowchart illustrating a method of providing parallel example sentences according to an embodiment of the present invention.

First, referring to FIG. 4, the document processing apparatus 100 analyzes an input document in terms of morphemes or word units, analyzes words and phrase boundaries based on this, and classifies phrases and sections based on the boundaries. To generate an expression list consisting of words, phrases, clauses, and the like (S510).

Thereafter, the document processing apparatus 100 extracts a specific expression from the expression list, retrieves a document containing the expression through the communication network 400, and extracts an original sentence including the expression. Also, when the searched document is written in the first language and the second language, the example sentence search unit 113 includes a part written in the second language from the searched document in the translation candidate document list, and the searched document is hyperlinked. If a multilingual web page is included, the hyperlinked document in the second language is included in the translation candidate document list. A search document set consisting of a plurality of original text examples and pairs of translation candidate documents corresponding thereto is generated through the above search process repeatedly for the expressions in the expression list (S520).

Next, the document processing apparatus 100 generates a first text document by extracting a text portion written in the first language from the original text example of the search document set, and is created in the second language from the translation candidate document corresponding to the original text example. By extracting the text portion and generating a second text candidate document, a text example sentence set consisting of the first text document and the second text candidate document is generated (S530).

Subsequently, the document processing apparatus 100 determines whether or not the sentences included in the first text document of the text example sentence set and the sentences included in the second text candidate document are translated from each other through a predetermined inspection, and determined to be translations of each other. The first sentence and the second sentence are extracted. The document processing apparatus 100 repeatedly checks pairs of the first text document of the text example sentence set data and the second text candidate document corresponding thereto, and converts the first text document into a pair of the first sentences and the second sentences corresponding thereto. The generated parallel example sentence data is generated (S540).

Next, the document processing apparatus 100 analyzes each sentence of the parallel example sentence in morphemes or word units to generate syntax structure information such as morphemes, words, phrases, and clauses included in the sentences (S550). The document processing apparatus 100 generates word alignment information by performing word alignment on the basis of the syntax structure information of the first and second sentences, and generates example sentence analysis information including the syntax structure information and the word alignment information. In operation S560, the sentence analysis information is stored in the database 150 together with the parallel sentence data (S570).

Referring to FIG. 5, the document processing apparatus 100 receives a query from the user terminal 300, performs a preprocessing process such as analyzing a syntax structure of the query and extracting a most suitable search word through analysis (S610). Thereafter, the document including the extracted search word is searched in the database 150 (S620), and the search result is transmitted to the user terminal 300 (S530).

The document processing method according to an embodiment of the present invention may be used to collect parallel example sentences for foreign language education, and thus may be included in various methods for foreign language education. Examples of such methods include, but are not limited to, online dictionary provision methods or interactive teaching methods. In addition, the document processing method according to an embodiment of the present invention may be utilized in a statistical based and example based automatic translation method.

Embodiments of the invention include a computer readable medium containing program instructions for performing various computer-implemented operations. This medium records a program for executing the document processing method described so far. The media may include, alone or in combination with the program instructions, data files, data structures, and the like. Examples of such media include magnetic media such as hard disks, floppy disks and magnetic tape, optical recording media such as CDs and DVDs, floppy disks and program commands such as magnetic-optical media, ROM, RAM and flash memory. Hardware devices configured to store and perform such operations. Alternatively, the medium may be a transmission medium such as an optical or metal wire, a waveguide, or the like including a carrier wave for transmitting a signal specifying a program command, a data structure, and the like. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like.

Although the preferred embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements of those skilled in the art using the basic concepts of the present invention defined in the following claims are also provided. It belongs to the scope of rights.

1 is a block diagram illustrating a document processing apparatus according to an embodiment of the present invention.

FIG. 2 is a block diagram of an example sentence collection unit shown in FIG. 1.

3 is a block diagram of an example sentence analysis unit illustrated in FIG. 1.

4 is a flowchart illustrating a method of collecting parallel example sentences in accordance with an embodiment of the present invention.

5 is a flowchart illustrating a method of providing a parallel example sentence according to an embodiment of the present invention.

<Explanation of symbols for the main parts of the drawings>

100: document processing apparatus, 110: sentence collection unit,

111: syntax structure analysis unit, 113: example sentence search unit,

115: text extracting unit, 117: sentence alignment unit,

130: example sentence analysis unit, 131: first syntax structure analysis unit,

132: second syntax structure analysis unit, 135: word alignment unit,

150: database, 170: example sentence provider,

200: web server, 300: user terminal,

400: network

Claims (21)

An example collection unit for collecting a second example sentence, which is composed of a first example sentence in a first language and a second language, which is a translation of the first example sentence from the web, and An example sentence providing unit for providing a parallel example sentence composed of the first example sentence and the second example sentence to the user according to a request from the user. Document processing apparatus comprising a. In claim 1, And the example sentence collection unit extracts a list of expressions to be used for learning from an input document, and searches the web for an expression included in the expression list to obtain the first example sentence including the expression. In claim 1, If the first document containing the first example sentence includes a portion written in the second language, or the document linked by a hyperlink included in the first document includes a portion written in the second language. And the example sentence collection unit extracts a portion written in the second language and includes it in a candidate example sentence list as a candidate example sentence of the second example sentence. 4. The method of claim 3, And the example sentence collection unit extracts the second example sentence from the candidate example sentence by comparing the first example sentence with the candidate example sentence and determining whether the candidate example sentence is a translation sentence of the first example sentence. In claim 1, The example sentence providing unit may generate a search word by analyzing a syntax structure of the query word input by the user, and search for a parallel example sentence including the search word in a database and provide the search word to the user. In claim 1, An example sentence analyzing unit configured to analyze the syntax structures of the first example sentence and the second example sentence and to generate alignment information for identifying a correspondence relationship between the expression included in the first example sentence and the expression included in the second example sentence Document processing apparatus comprising. In claim 6, And the example sentence providing unit provides the parallel example sentence to the user along with example sentence analysis information including the alignment information. In claim 6, And a database for storing the example sentence analysis information including the alignment information and the parallel example sentence. In claim 1, And the first language is Korean and the second language is English. A foreign language education system comprising the document processing device of any one of claims 1 to 9. Collecting from the web a second example sentence comprising a first example sentence in a first language and a second language, which is a translation of the first example sentence, and Providing to the user a parallel example sentence composed of the first example sentence and the second example sentence in response to a request from a user. Document processing method comprising a. In claim 11, The collecting step may include extracting a list of expressions to be used for learning from an input document, and obtaining the first example sentence including the expressions by searching the web for the expressions included in the expression list. Treatment method. In claim 11, The collecting step may include a portion written in the second language in a first document including the first example sentence or a hyperlink included in the first document in a second document. And if the portion is included, extracting the portion written in the second language and including the portion in the candidate example list as a candidate example sentence of the second example sentence. The method of claim 13, The collecting step further includes extracting the second example sentence from the candidate example sentence by comparing the first example sentence with the candidate example sentence and determining whether the candidate example sentence is a translation sentence of the first example sentence. In claim 11, The providing step may include generating a search word by analyzing a syntax structure of the query word input by the user, and searching for a parallel example sentence including the search word in a database and providing the search word to the user. In claim 11, Analyzing the syntax structures of the first example sentence and the second example sentence, and generating alignment information for identifying a correspondence relationship between the expression included in the first example sentence and the expression included in the second example sentence. How to process a document. The method of claim 16, The providing step includes providing the parallel example sentence to the user with example sentence analysis information including the alignment information. The method of claim 16, And storing the example sentence analysis information including the alignment information and the parallel example sentence. In claim 11, And the first language is Korean and the second language is English. 20. A foreign language teaching method comprising the document processing method of any one of claims 11 to 19. 20. A computer readable medium having recorded thereon a program for causing the computer to execute the method of any one of claims 11 to 19.

Images