KR20160061448A - Method for extracting unknown word from raw corpus - Google Patents

Abstract

An embodiment of the present invention relates to a method for extracting unregistered words from a raw corpus. The method for extracting unregistered words according to an embodiment of the present invention comprises the steps: separating the corpus into a chunk unit; recording a word separation location with respect to a character string of the chunk unit based on at least one of a word dictionary, a character string independence information, the existence of a function word and a grammatical rule; extracting an unregistered word candidate based on the recorded word separation location; and extracting the remaining words except the words registered in the word dictionary among the extracted unregistered word candidates as an unregistered word. According to an embodiment of the present invention, it is possible to extract unregistered words consisting of a language without word spacing with a raw corpus alone, without a mass processed language resource.

Description

Method for extracting unknown word from raw corpus [

Embodiments of the present invention are directed to a method for extracting unregistered words from a raw corpus.

Unregistered word extraction technology is an important element technology in natural language processing field. Unregistered words are used as important information when determining morpheme units in the morpheme analysis stage. In a natural language application system such as automatic translation, the quality of translation quality is improved by selecting a word for an unregistered word.

The conventional unregistered word extraction method includes a method in which a linguist determines whether a high-frequency new string extracted from a raw corpus is a word, a method of extracting an unregistered word through web mining, and a method of extracting unregistered words using a parallel corpus And how to extract words.

The first method can be largely divided into N-gram and statistical methods according to the method of extracting a new string.

In the N-gram method, a sentence is sequentially scanned in units of characters, separated from each other by a character string consisting of 2 to N characters starting from an arbitrary character, . The N gram method generates too many unnecessary new strings. Also, it takes a lot of manpower and efforts to determine whether or not an unregistered word belongs to a high frequency string.

In the statistical method, a tagged corpus separated by a linguist in advance by a linguist is learned by a machine learning method, a new sentence is separated into a pre-learned model, and a new string which is not a dictionary is used as an unregistered word candidate And sorting them in order of frequency. The statistical approach requires a large amount of tagged corpus, and building such a tagged corpus requires a lot of manpower and cost. In addition, it is troublesome to construct a tagged corpus suitable for a new field.

The second method is a method of extracting an unregistered word boundary of a source language from a pattern of a form of a " source language word (target language) "in the search result by performing a web search on an already known unknown word. This method has a problem that an unregistered word must be known in advance. It also requires the collection and text processing of large amounts of Web documents.

 The third method is a method of estimating the boundary of a primitive language word corresponding to a target language if the target language is divided into words using a parallel corpus consisting of a source language sentence and a corresponding band sentence. At this time, if the estimated string is not registered in advance, it is regarded as an unregistered word candidate and extracted. In order to apply this method, the word units of the target language sentence must be well-delimited, and the performance of distinguishing the boundaries of the source language words corresponding to the target language words should be good.

On the other hand, in a language without a space such as Chinese and Japanese, since the boundary between words is ambiguous, it is not easy to separate sentences in the language into words.

Embodiments of the present invention provide a method for effectively extracting unregistered words from a corpus made up of a language without spaces.

The method of extracting unregistered words according to an embodiment of the present invention includes: separating a corpus into chunk units; Writing a word separation position for a character string of the chunk unit based on at least one of a word dictionary, string independence information, functional word presence, and grammar rules; Extracting an unregistered word candidate based on the recorded word separation position; And extracting, as unregistered words, words other than the words registered in the word dictionary, out of the extracted unregistered word candidates.

According to embodiments of the present invention, it is possible to easily extract unregistered words with only a raw corpus made up of languages without a large number of processed language resources.

1 is a flowchart for explaining an unregistered word extracting method according to an embodiment of the present invention;
FIG. 2 is an exemplary diagram for explaining a character string separation method according to an embodiment of the present invention. FIG.
FIG. 3 is a block diagram for explaining an unregistered word extracting apparatus according to an embodiment of the present invention; FIG.

In the following description of the embodiments of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear.

Embodiments of the present invention provide a method for extracting unregistered words from a raw corpus of a non-spaced language.

In describing the embodiments of the present invention, Chinese is used as a language without a space, but embodiments of the present invention may be applied to other languages that do not have a space, for example Japanese.

Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

1 is a flowchart illustrating an unregistered word extraction method according to an embodiment of the present invention.

In step 101, the unregistered word extracting apparatus receives a raw corpus and separates the inputted raw corpus in units of chunks. The chunk may be a string corresponding to one sentence or a string including a plurality of characters.

For separation into chunk units, the unregistered word extraction device can perform character-based encoding identification processing.

That is, the unregistered word extracting apparatus can discriminate which language each character corresponds to, or whether each character (hereinafter, used as a meaning including symbols and numbers) is a symbol or a number.

For example, when the raw corpus is Chinese, the unregistered word extraction device can perform filtering such as deleting encoded characters such as Korean and Japanese. In addition, characters that can handle Chinese text, such as Chinese characters, symbols, numbers, and English characters, may not be filtered. Also, for convenience of subsequent processing, a single-byte character can be converted to a double-byte character.

便宜..."라는 문자열이 있는 경우, 미등록 단어 추출 장치는, 도 2의 (a) 단계에 도시된 바와 같이, 해당 문자열을 기호들(",", "..."을 기준으로 두 개의 청크 단위로 분리할 수 있다.
And, the unregistered word extracting device can separate the raw corpus into chunks based on a specific character, for example, a symbol or a number. For example, in the raw corpus, "眞 奇怪,

Quot ;, "... ", the unregistered word extracting device extracts the corresponding character string from two or more characters (e.g.," Can be separated into chunks.

In step 103, the unregistered word extraction device refers to the word dictionary 342 and performs word-segmentation position recording on the character string separated in chunk units.

The word dictionary 342 may be a dictionary containing high frequency words commonly used in Chinese. Assuming that the word "Precise" is registered in the word dictionary 342, the unregistered word extraction device records the separation position for the word "Precision ", as shown in step (b) can do.

In step 105, the unregistered word extracting device performs word-separated position recording based on the character string independence information 344 and whether or not the word is a functional word.

)을 기준으로 단어 분리 위치를 기록할 수 있다. First, the unregistered word extraction device can check whether or not a functional word is included in a character string separated by a chunk unit, and perform word-based positional recording based on the functional word. For example, as shown in step (c) of FIG. 2, the unregistered word extracting apparatus extracts functional words (這, 的, 是,

) Can be recorded.

On the other hand, having string independence can mean, for example, that there is a clear boundary between the beginning and end of a string, or that the beginning and end of a string can be clearly distinguished.

For example, the beginning of a sentence can be thought of as the starting point of a string, and the starting point of the string is clearly identified. On the other hand, if a number or function word is adjacent to the end of the string, the character preceding the corresponding number or function word can be regarded as the end point of the corresponding string, and the end point of the corresponding string is clearly distinguished. If the starting and ending points of the string are clearly distinguished, the string can be said to have string independence.

The string independence information 344 may be constructed in advance based on an arbitrary corpus. In FIG. 2, as an example, string independence information 344 for a string of "market", "market value", "value" and "market value" is constructed.

The unregistered word extracting device can record the separation position of a word having string independence by referring to the string independence information 344. [ Step (c) of FIG. 2 shows that the string "market value" can be divided into a string of "market", "market value" and "market value".

In step 107, the unregistered word extraction device performs word segmentation location recording based on grammar rules.

A grammar rule can mean, for example, a sentence rule that is common in the language. For example, in Chinese, sentence rules such as "preposition + two", "conjunction + string", "investigation + two", "two + noun" and "adverb + verb" often appear. The unregistered word extraction device can record the word separation position based on the grammar rule.

For example, in the string "眞 奇怪", "true" is an adverb, so the "adverb + verb" rule can be applied, and in the string "chunk table" . Thus, the unregistered word extracting apparatus can record the separation position after the adverbial "true" and record the separation position after the two words "chunk", as shown in step (d) of FIG.

In step 109, the unregistered word extraction device extracts unregistered word candidates based on the recorded word separation position.

The unregistered word extracting apparatus can extract unregistered word candidates by targeting a string including two or more characters. In step (e) of FIG. 2, a character string including two or more characters ("bizarre", "table", "market", "market value", "market value", " ") Is extracted as an unregistered word candidate.

On the other hand, the longest matching technique can be applied to extract unregistered word candidates. For example, the string "market value" may be a combination of a first candidate string combination ("market" + "value"), a second candidate string combination ("market value" Quot;). ≪ / RTI > According to the longest matching technique, a candidate having the largest value among values obtained by multiplying the lengths of words included in each candidate string or candidate string combination is selected. In the above example, the first candidate character string combination has a value of 4 (2 x 2), the second candidate character string combination has a value of 3 (3 x 1), and the third candidate character string has a value of 4 (4 x 1). Therefore, the first candidate character string combination and the third candidate character string have the largest value (4), and the third candidate character string corresponding to the compound word combination is eliminated. As a result, the string "market value" can be divided into "market" + "price".

As a result, in step 109, the strings "bizarre "," table "," market ","price","

In step 111, the unregistered word extraction device extracts, as unregistered words, words not registered in the dictionary among the extracted unregistered word candidates.

In the above example, since the word "accuracy" is registered in advance, the strings "bizarre "," market "," value ", and "convenience"

3 is a block diagram for explaining an unregistered word extracting apparatus according to an embodiment of the present invention.

3, the apparatus for extracting unregistered words according to an embodiment of the present invention includes a corpus preprocessing unit 310, a character extraction unit 320, a word extraction unit 330, and a memory unit 340. Depending on the embodiment, at least one of the aforementioned components may be omitted.

The corpus preprocessing unit 310 separates a raw corpus made up of a language without a space into chunks. The corpus-preprocessing unit 310 may perform encoding identification processing to filter remaining characters excluding characters capable of processing Chinese text.

The unregistered word candidate extracting unit 320 may perform word segmentation of a character string of a chunk unit based on a word dictionary, character string independence information, whether or not a functional word exists, and a grammar rule. Then, the unregistered word candidate extracting unit 320 extracts unregistered word candidates based on the recorded word separation position. At this time, if many unregistered word candidates sharing the same character in one character string can be extracted, the unregistered word candidate extracting unit 320 can apply the longest matching technique to the corresponding character string.

The unregistered word extracting unit 330 extracts an unregistered word among the unregistered word candidates extracted by the unregistered word candidate extracting unit 320. At this time, the unregistered word extracting unit 330 can extract, as unregistered word candidates, the remaining words other than the words registered in the word dictionary, among the unregistered word candidates.

The memory unit 340 may store word dictionary and string independence information.

The embodiments of the invention described above may be implemented in any of a variety of ways. For example, embodiments of the present invention may be implemented using hardware, software, or a combination thereof. When implemented in software, it may be implemented as software running on one or more processors using various operating systems or platforms. Additionally, such software may be written using any of a number of suitable programming languages, and may also be compiled into machine code or intermediate code executable in a framework or virtual machine.

Also, when embodiments of the present invention are implemented on one or more processors, one or more programs for carrying out the methods of implementing the various embodiments of the invention discussed above may be stored on a processor readable medium (e.g., memory, A floppy disk, a hard disk, a compact disk, an optical disk, a magnetic tape, or the like).

Claims (1)

Separating the corpus into chunks;
Writing a word separation position for a character string of the chunk unit based on at least one of a word dictionary, string independence information, functional word presence, and grammar rules;
Extracting an unregistered word candidate based on the recorded word separation position; And
Extracting, as unregistered words, the remaining words other than the words registered in the word dictionary, out of the extracted unregistered word candidates
And extracting an unregistered word.

Images