KR20000038079A - Device and method for extracting example of use statistically for large amount language batch - Google Patents

Abstract

PURPOSE: A device and method for extracting example of use statistically for a large amount language batch are provided so that a whole field of a natural language process can be used for constructing a basic knowledge. CONSTITUTION: A device for extracting example of use statistically for a large amount language batch includes a device for producing a basic table in order to extract an example of use from a language batch. A candidate of example of use is determined from a device for determining an example of use. For management a memory and time efficiently, a shortened table producing device is used.

Description

Statistical Application Extraction Method and Method for Massive Coral

The present invention relates to a means and method for extracting continuous and discontinuous applications by efficiently using space and time from a large corpus, in particular to determine example candidates compared to other strings from strings extracted from corpus, Determines continuous and discontinuous use cases on a reduced table for spatial and temporal efficiency, and finally uses patterns elimination rules to eliminate unnecessary uses, reflecting corpus features available in all areas of natural language processing in large corpus. An extraction means and a method thereof.

The usage extracting means means that the user can use the pattern table and the sorting pattern table to search for phrases and words including compound nouns or idiom expressions that appear repeatedly within the corpus, which are considered to reflect the characteristics of the corpus. Means to extract in an efficient way to minimize the pattern and shorten the extraction time.

In the field of natural language processing, since the early 1990s, a series of researches on automatic knowledge extraction have made many technical advances in automatic knowledge extraction. This technology enabled fast and robust automatic extraction of each predefined language pattern from resources extracted from real-world online text or computer networks.

Phrase or pattern electronic dictionaries constructed by automatically extracted patterns may be usefully used in complex unit processing and the like in a machine translation system. In particular, for pattern-based translation, the identification and automatic extraction process of frequently used patterns should be premised. Overseas, a method using similarity of words appearing in each language and a letter-based N-gram sorting method have been suggested from relatively small corpuses. have.

Existing character-based pattern extraction generates a large number of unnecessary patterns and has a disadvantage that the extraction time is excessive compared to the corpus due to the excessive size of the table.

The present invention can be processed even if the size of the corpus to be used for extracting the usage to be universally used in natural language processing applications, and to efficiently manage the memory space and time required for usage extraction It aims to extract continuous and discontinuous usages from the extraction of existing simple keywords (verb stems or nouns).

According to the present invention for achieving the above object, the statistical usage extracting means for large-capacity corpus includes a means for generating a base table for extracting a case from the corpus, a case candidate determining means for determining example candidates in a table entry, a memory; Means for creating a reduced table that reduces the base table for efficient management of extraction time, means for extracting continuous and discontinuous use cases from use case candidates, means for removing unnecessary uses using rules from the extracted use cases, and And means for presenting the finally extracted applications.

In addition, the statistical usage extraction method for a large-capacity corpus according to the present invention for achieving the above object is as a single entry in the pointer table while a string of a constant length in the corpus input through the input device is sequentially moved by one character Inserting; copying the pointer table to create an alignment pointer table if the string to be inserted does not remain in the corpus; and obtaining an NMC and inserting into the Nth entry of the alignment pointer table; Determining an Nth NSC of N from an N-1th NMC and an Nth NMC, a larger value of the valid flag obtained for the pointer table, and an entry if the valid flag is 1 Is inserted into the collapsed pointer table and moves to the next entry if the valid flag is 0 Checking whether an entry exists, calculating a valid flag if the entry exists, and if the entry does not exist, generating a reduced sort pointer table from the reduced pointer table; Calculating NES in a sort pointer table to extract consecutive usages, and checking overlaps between successive usages appearing in the same sentence using a start pointer in the reduced sorting pointer table, and if overlapping does not occur as a result of the overlapping inspection In this case, the continuous usages belonging to the same sentence are regarded as a new non-consecutive use and the pattern elimination rule is used to remove unnecessary use from the continuous use and the non-consecutive use.

In addition, the present invention for achieving the above object is a means for generating a base table for extracting a case from the corpus, a case candidate determining means for determining the case candidates in the table entry, the base for efficient management of memory and extraction time Means for creating a reduced table that reduces the table, means for extracting continuous and discontinuous use cases from the use case candidates, means for removing unnecessary uses using pattern removal rules for the extracted use cases, and finally extracted cases. And a computer readable recording medium having recorded thereon a program for functioning the outputting means.

1 is a block diagram illustrating the concept of a statistical application extracting means according to the present invention;

2 is a flowchart illustrating a method for extracting statistical continuous and discontinuous applications according to the present invention.

Figure 3 (a) and Figure 3 (b) is an internal structure diagram of PT, SPT and RPT, RSPT used in the present invention.

4 is an embodiment of a statistical application extraction means according to the present invention.

<Description of the code | symbol about the principal part of drawing>

101: input means 102: PT generation unit

103: SPT generation unit 104: NMC calculation unit

105: NSC calculator 106: valid flag calculator

107: RPT generator 108: RSPT generator

109: NES calculation unit 110: ST calculation unit

111: unnecessary application removal unit 112: printing unit

113: printing apparatus 114: display control unit

115: display device

Continuous usage means that the extracted usage appears as a sequence of words in the sentence, and discontinuous usage means that the extracted usage appears to be separated by other words or phrases discontinuously in the sentence. Say.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram illustrating the concept of a statistical application extracting means according to the present invention.

Through the input device 101, a string of a certain length in the corpus is sequentially inserted one character into the PT generated by the pointer table (PT) generation unit 102 while sequentially moving one character. If no more strings to be inserted remain in the corpus, the sorted pointer table (SPT) generation unit 103 copies the PT to generate the SPT. The internal structure of the PT and SPT is shown in Fig. 3 (a). SPT is used to compare the same number of characters as the entry immediately above or below the current entry, sorted by PT according to the string. The Number of Matched Characters (NMC) calculation unit 104 obtains an NMC, which is the number of characters common between the Nth string and the N + 1th string, and inserts the NMC into the Nth entry of the SPT. Next, the NSC calculation unit 105 determines the N-th NSC of the SPT from a large value among the N-1 th NMC and the N th NMC. The validity flag calculation unit 106 obtains a valid flag for the PT and uses the same to determine an application candidate. The Validity flag is set to 1 if the value of the Nth NSC is greater than or equal to the value of the N-1th NSC, otherwise it is set to 0. Entries with a Validity flag of 1 are moved to the RPT generated by the reduced pointer table (reduced PT: RP) 107 to avoid unnecessary operations on entries that do not need subsequent operations. Improve space and speed. If the valid flag is 0, it moves to the next entry and if there is no entry there, PSTP is created in the same manner as the SPT is created in PT by the reduced sorted pointer table (Reduced SPT: RSPT) generator 108. Create The internal structure of the RPT and RSPT is shown in Fig. 3 (b). NES (Number of Extracted Substring) is the number of extracted applications, which increases by 1 each time a usage is found, so the number of extracted applications can be known from the final NES value. NES is incremented when the Validity flag and the value of NSC are present in RSPT and equal to the values of the previous index. Whenever the NES value is changed by the NEC calculation unit 109, it is assumed that a new continuous application is extracted, and in the same sentence by using a start pointer (ST) calculation unit 110 in RSPT. Check for overlap between successive applications shown. ST is the information of the first character in the sentence to which the string belongs when the string is first inserted into the PT from the corpus. Consecutive usages that belong to the same sentence without overlapping are considered new non-consecutive usages. Finally, the unnecessary use case removal unit 111 is applied with a pattern removal rule to remove unnecessary use cases, and is output to the printing apparatus 113 through the printing unit 112 or the display device 115 through the display control unit 114. Will output

2 is a block diagram of a statistical application extraction device comprising continuous and discontinuous application extraction means in accordance with the present invention.

A string of a certain length in the corpus 200 input through the input device is inserted as one entry into the PT while sequentially moving by one character. If no more strings to be inserted remain in the corpus (201), the PT is copied to generate an SPT (203). SPT is used to compare the same number of characters as the entry immediately above or below the current entry, sorted by PT according to the string. The NMC, which is the number of characters common between the Nth string and the N + 1th string, is obtained and inserted into the Nth entry of the SPT. Next, the Nth NSC of the SPT is determined from the large value among the N-1th NMC and the Nth NMC (206). A valid flag is obtained for the PT and used to determine a candidate for use (207). The valid flag is set to 1 if the value of the N th NSC is greater than or equal to the value of the N-1 th NSC, otherwise it is set to zero. Entries with a valid flag of 1 are moved to the RPT to improve memory space and speed by avoiding unnecessary operations on entries that do not need subsequent operations (209). If the valid flag is 0 (208), it moves to the next entry and checks whether an entry exists (211). If a check result entry exists, the process transitions to step 207. If there is no entry, generation of the RSPT is performed in the same manner as that of creating an SPT in the PT (212 and 213). The NES is the number of extracted applications, which increases by 1 each time a usage is found, so the number of applications extracted from the final NES value can be known. The NES is incremented 214 if there is a valid flag and the value of NSC in RSPT equal to the values of the previous index. Whenever the value of NES changes, it is assumed that a new continuous usage is extracted (215), and the overlap between successive usages shown in the same sentence is checked using ST in RSPT (216). ST is the information of the first character in the sentence to which the string belongs when the string is first inserted into the PT from the corpus. Consecutive usages belonging to the same sentence without overlapping are considered as new discontinuous usages (217). Finally, the usage extraction is terminated by removing unnecessary usage (218) using the pattern removal rule.

4 shows an example of presenting the examples and the sentences to which the examples belong when selecting the corpus after selecting corpus as an embodiment of the statistical example extracting means according to the present invention.

The present invention described above is capable of various substitutions, modifications, and changes within the scope without departing from the spirit of the present invention for those skilled in the art to which the present invention pertains, the foregoing embodiments and the accompanying drawings. It is not limited to this.

As described above, according to the present invention, since it is possible to extract a variety of applications in continuous and discontinuous form from a large amount of corpus, including a field for analyzing the characteristics of corpus or a field for natural language processing using a large amount of applications. It can be used extensively to build basic knowledge in all areas of natural language processing.

Claims (10)

Means for creating a base table for extracting usage from corpus, Usage candidate determination means for determining usage candidates in a table entry; Means for creating a collapsed table with a reduced base table for efficient management of memory and extraction time; Means for extracting continuous and discontinuous usage from the usage candidates; Means for eliminating unnecessary usages using pattern removal rules in the extracted usages; Statistical example extraction means for large-capacity corpus consisting of means for outputting the finally extracted applications. The apparatus of claim 1, wherein the basic table generating means comprises: a pointer table generating unit for generating a pointer table by sequentially extracting a string from a corpus; And a sorted pointer table generator for generating a pointer table arranged in synchronization with the pointer table. The application candidate determining means comprises: an NMC calculator and an NSC calculator for calculating the same number of characters between character strings; Statistical example extracting means for large-capacity corpus characterized in that it comprises a valid flag calculation unit for determining whether to use the candidate. 2. The apparatus of claim 1, wherein the reduction table generating means comprises: a reduced pointer table generation unit configured to extract only application candidates from a base table and form a reduced type table; And a sorted collapsed pointer table generator for generating a reduced pointer table arranged in synchronization with the reduced pointer table. The method according to claim 1, wherein the continuous and discontinuous application extracting means comprises: an NES calculation unit for determining whether the application candidate is selected as the longest match in the corpus; Statistical case extraction means for large-capacity corpus, characterized in that it comprises a start pointer calculation unit for examining whether the overlap between the two applications in the sentence. 2. The statistical usage extracting means according to claim 1, wherein the unnecessary usage removing means comprises an unnecessary usage portion for removing unnecessary usage using a pattern removing rule. Inserting a string of a constant length in the corpus input through the input device as an entry into the pointer table while sequentially moving one character at a time; Creating a sort pointer table by copying the pointer table when the string to be inserted does not remain in the corpus; Obtaining an NMC and inserting the same into an Nth entry of the sort pointer table and determining an Nth NSC of the sort pointer table from a larger value of N-1th NMC and Nth NMC; Checking whether the valid flag obtained for the pointer table is 1; Inserting the entry into the reduced pointer table when the valid flag is 1 and moving to the next entry when the valid flag is 0, and checking whether the entry exists; Transitioning to calculating a valid flag if there is an entry as a result of the check, and generating a reduced sort pointer table from the reduced pointer table if the entry does not exist; Calculating NES from the reduced alignment pointer table to extract consecutive usages, and checking overlaps between successive usages appearing in the same sentence using a start pointer in the reduced alignment pointer table; The overlapping check result is that the overlapping occurs, including the continuous use of the same sentence as a new non-consecutive use and using a pattern removal rule to remove unnecessary use from the continuous use and the non-continuous use Statistical application for extracting large volumes of corpus. 10. The method of claim 7, wherein the sort pointer table is used to compare the same number of characters as the entry immediately above or below the current entry in the form of sorting the pointer table according to a character string. Way. 8. The statistical example of claim 7 wherein the valid flag is set to 1 if the value of the N th NSC is greater than or equal to the value of the N-1 th NSC, otherwise it is set to 0. Way. Means for creating a base table for extracting usage from corpus, Usage candidate determination means for determining usage candidates in a table entry; Means for creating a collapsed table with a reduced base table for efficient management of memory and extraction time; Means for extracting continuous and discontinuous usage from the usage candidates; Means for eliminating unnecessary usages using pattern removal rules in the extracted usages; A computer-readable recording medium having recorded thereon a program for functioning means for outputting finally extracted examples.