KR100631086B1 - Method and apparatus for text normalization using extensible markup language(xml) - Google Patents

Abstract

A device and a method for normalizing text using XML(eXtensible Markup Language) are provided to facilitate maintenance of text normalization by applying the XML, facilitate a tuning process according to the feature of input data in the text normalization, and reduce errors of the text normalization and induce rules by easily applying the dynamic feature to the text normalization and verifying the data in advance. A storing part stores text sentences. A parser divides the stored text sentence into a word unit and parses the divided text data to a semantic unit to analyze semantics. An XML data converter converts the parsed text data of the word unit into XML data based on the first predefined XML DTD(Data Type Definition). An attribute setting part sets an attribute to each word unit by checking front/rear context after reading the converted XML data and considering the checked context. An XML data generator generates the final XML data for converting the text data according to the text normalization if the attribute value is set to each word unit.

Description

Method and apparatus for text normalization using XML {Method and apparatus for text normalization using extensible markup language (XML)}

1 is a flowchart illustrating a text normalization method using XML according to the present invention.

2A and 2B illustrate DTDs for generating XML data for text normalization according to the present invention.

3 is a preferred embodiment showing attribute values of word elements according to the present invention;

4 is a view showing a preferred embodiment of the final XML data for sentence transformation according to the present invention

FIG. 5 illustrates a preferred embodiment in which XML data according to the embodiment of FIG. 4 is represented in the form of a DOM (Document Object Model) tree.

* Explanation of symbols for main parts of the drawings

10: sentence (<SENT>) element 20: word (<EOJEOL>) element

30: subelement

The present invention relates to the field of speech and natural language processing, and more particularly, to a method and apparatus for text normalization using extensible markup language (XML).

The field of speech and natural language processing is to study the characteristics of each language in order to analyze and process languages with a computer, and to develop algorithms for language processing from an engineering point of view and to study language knowledge for them.

Text normalization is a technique used to generate language models or to synthesize speech in the fields of speech and natural language processing. In other words, it is a technology for converting symbols, numbers, and English characters into Korean, which is used for language model generation and speech synthesis.

Representative ways of conveying one's thoughts and meanings to others are text and voice. In particular, texts have been an effective medium that has long been used to convey one's thoughts among distant people. Now, with the development of e-mail and the Internet, it is now possible to convey one's own thoughts as well as texts or images and voices.In particular, the visual expression of the images is more implicit. Not only can you communicate effectively, but it can also make your partner feel more friendly.

In addition, with the rapid development of computers and the Internet, various services such as icons, animations, avatars, and the like have emerged, and there is a tendency to own and use various and customized images. Nevertheless, to date, a series of processes have been performed manually. In addition, the automatic conversion service has been a problem that the meaning is distorted due to the word matching and wrong matching by simple comparison because only the form of the letter is changed by simply comparing.

In particular, the nature of the language, symbols and numbers, etc. can be variously converted according to various conditions including the context, and also has a ambiguity that can be variously changed under one condition.

Therefore, the conventional text normalization method requires a tuning operation according to the characteristics of the input data. In other words, the conversion is performed by predefined data structures and rules.

However, the text normalization method according to the tuning operation is inevitably a major disadvantage in terms of maintenance of text normalization since it is inevitable to modify the code unit when a modification or addition of a rule or a data structure is required during the tuning operation. .

Accordingly, an object of the present invention is to provide a text normalization method and apparatus that can easily perform maintenance of text normalization by applying extensible markup language (XML).

Another object of the present invention is to provide a method and apparatus for text normalization which can more easily perform a tuning operation according to characteristics of input data in performing text normalization.

It is another object of the present invention to reduce the error of text normalization and to induce rules by facilitating the reflection of the dynamic characteristics of text normalization and pre-verifying the data.

A feature of the text normalization method using extensible markup language (XML) according to the present invention for achieving the above object is a first step of parsing an input text sentence into semantic units and converting it into XML data, and the converted XML And a second step of generating final XML data by setting attribute values in consideration of the context of the sentence in the data.

Preferably, the first step includes separating the input text sentence into word units, parsing the separated word units into respective semantic units capable of semantic analysis, and defining each of the parsed semantic units. And converting the data into XML data based on the first XML Document Type Definition (DTD).

Preferably, the parsing step includes parsing using at least one of morphological analysis, part-of-speech tagging, and syntax analysis of the input sentence.

Preferably, the first XML DTD uses a sentence element as a root element and forms at least one word element as a sub element of the sentence element, and at least one And forming a plurality of sub-elements according to their meanings for each of the above word elements.

Preferably, the sub element is one of Korean (<KOR>), English (<ENG>), Chinese (<CHI>), number (NUM), and symbol (<SYM>).

Preferably, the step of setting the attribute value for each of the sub-elements in consideration of the context in accordance with a second XML DTD (Document Type Definition) previously defined after reading the converted XML data, and the set And generating final XML data for conversion of text data according to text normalization according to the corresponding rule in consideration of the attribute value.

Preferably, for each of the plurality of sub-elements, when the sub-element is a Korean (<KOR>) element, attributes of yeal_han (eleven method), sip_il (ten-way method), none (if there is no information necessary for conversion) If the sub-element is an English (<ENG>) element, define it by including at least one value. If the sub-element is an English (<ENG>) element, email (e-mail), web (web address), acronym (abbreviation and English word), unit (English unit), none ( And define at least one attribute value of the alphabet), and if the sub element is a Hanja element, include at least one attribute value of none (convert to the corresponding Hangul), and define the sub element Is a numeric (<NUM>) element, decimal (decimal point), range (score), score (score), exp (formula), time (time), date (date), tel (telephone / fax), post ( Zip code), ip (IP), card (card number), account (account number), event (event date), none ( At least one attribute value, and converts with reference to the next <KOR> node). If the sub-element is a symbolic <SYM> element, one-byte (two-byte characters), two and at least one attribute value such as -byte (2 byte character).

A feature of a text normalization apparatus using extensible markup language (XML) according to the present invention for achieving the above object is a storage unit for storing an externally input text sentence, the text sentence stored in the storage unit in word units After separation, the parser parses the separated text data into semantic units capable of semantic analysis, and the word unit separated based on the first XML document type definition (DTD) previously defined by the parser. An XML data converter for converting the XML data into XML data, a property setter configured to read the XML data converted by the XML data converter to determine the context before and after and to set attribute values in units of words in consideration of the identified context; When the attribute value is set in each word unit through the final XML data for conversion of text data according to text normalization It consists of including the XML data generator that generates emitter.

Other objects, features and advantages of the present invention will become apparent from the following detailed description of embodiments with reference to the accompanying drawings.

A preferred embodiment of the method and apparatus for text normalization using extensible markup language (XML) according to the present invention will be described with reference to the accompanying drawings.

1 is a flowchart illustrating a text normalization method using XML according to the present invention.

As shown in FIG. 1, when text data is input (S100), a sentence analysis step of first parsing a sentence of the input text data into a semantic unit is converted into XML data (S200).

That is, in the sentence analyzing step, the sentence of the input text data is first stored in a storage unit, and the morphological analysis, the part-of-speech tagging, and the segmentation analysis are performed through a parser, and separated into word units. The parsed word unit is parsed into each semantic unit capable of meaning analysis (S210).

Thereafter, the text data parsed by the semantic unit is converted into XML data (S220). In this case, the XML is converted through the XML data converter based on the first XML Document Type Definition (DTD) of FIG. 2A.

Referring to FIG. 2A, the first XML DTD has a sentence (<SENT>) element 10 as a root element, and a plurality of words (<EOJEOL>) as sub-elements of the sentence element 10. Element 20. Each word element 20 may be divided into a plurality of different sub elements 30 according to their meanings.

In the present specification, for ease of explanation, five sub-elements 30, which are Korean (<KOR>), English (<ENG>), Chinese (<CHI>), and symbol (<SYM>), are separated. This is a preferred embodiment and can be expanded and changed as much as the designer desires.

Therefore, when the input text data is parsed in a semantic unit, each parsed data is passed through the sentence (<SENT>) element 10 which is the root element and the corresponding XML through the word element (<EOJEOL>) element 20 which is a child element. Convert to data. That is, the meaning of each parsed data is Korean (<KOR>), English (<ENG>), Hanja (<CHI>), Number (<NUM>), Symbol (< SYM>) is determined to be one of the sub-elements 30 and tagging the corresponding part, it is converted into XML data.

In this case, in the sentence parsing step, only XML elements corresponding to meanings are added to each parsed data, and attribute values are not set. In order to set attribute values for each parsed data, it is necessary to accurately grasp the context before and after the context analysis.

However, the attribute value of each word element 20 should be accurately determined and set as a very important role in text normalization conversion.

Therefore, after the character analysis step is completed, the attribute value considering the context is set by reading the converted XML data based on the second XML document type definition (DTD) according to the sub element 30 of FIG. The sentence conversion step of generating the XML data is performed (S300).

The reason why the attribute value of each word element can be determined in the sentence conversion step is because the parsed data is converted into XML data in the sentence analysis step so that the context can be easily identified.

That is, in the sentence conversion step, first, by reading the XML data converted in the character analysis step through an attribute setter, the context is determined before and after, and the correct attribute value is set in consideration of the identified context (S310).

At this time, the attribute value according to the second XML Document Type Definition (DTD) according to the sub element 30 of FIG. 2B has different values for each word element 20, and accordingly, each word element 20 has an attribute value. Considering the context and context, the corresponding rule can be defined as shown in FIG. 3.

3 is a preferred embodiment showing attribute values of word elements according to the present invention. In this case, Figure 3 is a preferred embodiment according to the attribute value of each word element 20 for easy description of the present invention, which can be changed and added by the designer.

Referring to FIG. 3, the word element <EOJEOL> 20 may include Korean (<KOR>), English (<ENG>), Hanja (<CHI>), numeral (NUM), and symbol (<SYM>). Are separated into sub-elements 30 of each.

The Korean element has at least one attribute value such as yeal_han (eleven method), sip_il (ten days method), none (when there is no information necessary for conversion).

In addition, the English element includes at least one attribute value such as email (e-mail), web (web address), acronym (abbreviation and English word), unit (English unit), none (pronounced alphabet). Have it.

Also, the Hanja element has at least one attribute value such as none (converted to the corresponding Hangul).

In addition, the <NUM> element may be a decimal, a range, a score, an exp, a time, a date, a tel or a fax. Attributes such as zip code, ip (IP), card (card number), account (account number), event (event date), none (number without schedule, conversion by referring to next <KOR> node) It has at least one value.

The symbol <SYM> element may include at least one attribute value such as one-byte (one byte character), two-byte (two byte character), or the like.

As such, when the attribute value is set for each word element 20 through the attribute setter, the XML data generator converts the text data according to the text normalization according to the corresponding rule in consideration of the attribute value according to the context as shown in FIG. 4. Create final XML data for (S320). At this time, since the attribute value of the word element 20 plays the most important role in the text normalization transformation, it should be accurately determined and set.

FIG. 5 is a preferred embodiment in which XML data according to the embodiment of FIG. 4 is represented in the form of a DOM (Document Object Model) tree.

As shown in Fig. 5, the data structure of the XML data is composed of a tree structure, so that there is no fixed set of tags like the database and it is not predefined so that new tags can be easily added as needed. This gives you very flexibility in data structure and data processing.

As described above, the preferred embodiment of the present invention has been disclosed through the detailed description and the drawings. The terms are used only for the purpose of describing the present invention and are not used to limit the scope of the present invention as defined in the meaning or claims. Therefore, those skilled in the art will understand that various modifications and equivalent other embodiments are possible therefrom. Therefore, the true technical protection scope of the present invention will be defined by the technical spirit of the appended claims.

As described above, the method and apparatus for text normalization using XML according to the present invention has the following effects.

First, by applying XML in performing text normalization, it is easy to reflect the dynamic characteristics of text normalization and the data can be verified in advance, which can help reduce errors in text normalization and induce rules.

Second, since the data structure of XML data is composed of tree structure, there is no fixed set of tags like the database and it is not defined in advance so that new tags can be added as needed to provide flexibility in data structure and data processing. Can provide.

Third, by using XML, it is possible to support multiple data formats and include all existing data structures.

Claims (9)

A first step of parsing the input text sentence into semantic units and converting the input text sentence into XML data; And a second step of generating final XML data by setting an attribute value in consideration of the context of the sentence in the transformed XML data. The method of claim 1, wherein the first step Dividing the input text sentence into word units and parsing the separated word units into respective semantic units capable of semantic analysis; And converting each of the parsed semantic units into XML data based on a first XML Document Type Definition (DTD) previously defined. The method of claim 2, The parsing step includes parsing the text using at least one of morphological analysis, part-of-speech tagging, and syntax analysis of the input sentence. The method of claim 2, wherein the first XML DTD is Forming a sentence element as a root element, and forming at least one word element as a sub-element of the sentence element; Forming a plurality of sub-elements for each of the at least one word element according to their meanings. The method of claim 4, wherein The sub element is one of Korean (<KOR>), English (<ENG>), Chinese (<CHI>), number (NUM), and symbol (<SYM>). The method of claim 5, wherein the second step Setting an attribute value for each of the sub elements in consideration of a context according to a second XML document type definition (DTD) defined after reading the converted XML data; And generating final XML data for conversion of text data according to text normalization according to a corresponding rule in consideration of the set attribute value. The method of claim 6, wherein for each of the plurality of sub-elements, When the sub-element is a Korean (<KOR>) element, it is defined to include at least one attribute value of yeal_han (eleven method), sip_il (ten days method), none (when there is no information necessary for conversion), If the sub-element is an English (<ENG>) element, attribute values of email (e-mail), web (web address), acronym (abbreviation and English word), unit (English unit), and none (pronounced alphabet) are included. Define one, If the sub-element is a Hanja element, it is defined to include at least one attribute value of none (converted to the corresponding Hangul), If the subelement is a number element, it is decimal, range, score, exp, time, date, or tel. , post (zip code), ip (IP), card (card number), account (account number), event (event date), none (number of unformatted, convert by referring to next <KOR> node) Define at least one attribute value of, If the sub-element is a symbol <SYM> element, at least one attribute value of one-byte (single-byte character), two-byte (two-byte character), and the like is defined. . A storage unit for storing text sentences, A parser that separates the text sentence stored in the storage unit into word units and parses the text data separated into the word units into semantic units capable of semantic analysis; An XML data converter for converting the text data parsed by the parser into XML data in units of separated words based on a first XML DTD (Document Type Definition) previously defined; An attribute setter configured to read the XML data converted by the XML data converter to grasp a context before and after and set attribute values in units of words in consideration of the grasped context; And an XML data generator for generating final XML data for converting text data according to text normalization when an attribute value is set in each word unit through the attribute setter. The method of claim 8, The first XML Document Type Definition (DTD) includes a sentence element as a root element and includes a plurality of word elements as sub-elements of the sentence element. The word normalization apparatus using XML, characterized in that the word element is composed of a tree structure composed of a plurality of different sub-elements according to their meanings.

Images