JP5380566B2 - Language processing apparatus, program, and method - Google Patents

Description

  Embodiments described herein relate generally to a data structure storing a term and a term searching method.

  Many natural language processing systems represented by information retrieval systems, such as kanji conversion systems and machine translation systems, store in advance a machine-readable dictionary of information about individual words in the language to be processed, such as English and Japanese. In addition, it is often provided with a processing unit that performs a high-speed dictionary search using a machine-readable dictionary for a natural language sentence to be processed and analyzes word information of an input sentence. This dictionary search is usually performed by a process called morphological analysis.

  The Japanese-English translation system includes a Japanese morphological analysis system using a machine-readable dictionary in which Japanese terms are registered because the original text is Japanese. On the other hand, since the original text is English, the English-Japanese translation system includes an English morphological analysis system using a machine-readable dictionary in which English terms are registered.

  The following documents are disclosed as related technologies.

Japanese Patent Laid-Open No. 04-101271

  The problem to be solved by the present invention is to provide a technique capable of speeding up dictionary search.

  The language processing apparatus according to the embodiment includes a determination unit and a search unit. The determination unit determines the range of the character string in the original text. The search unit normalizes tokens included in the range determined by the determination unit for each token, creates a concatenated token string by concatenating the normalized tokens, and the concatenated token string reaches the specified length. The token string is truncated at the specified length, the hash value of the token string is calculated, and the term associated with the calculated hash value is registered in advance with the hash value and the term associated with each other. One or more are acquired from the storage unit.

It is a figure which shows an example of the span for a dictionary search while showing an example of an original sentence. It is a figure which shows the example of a response | compatibility with the original text and the headline registered into the dictionary. It is a figure explaining an example of the process which normalizes an original sentence and a dictionary heading, calculates a hash value, and searches. It is a figure which shows the example of a response | compatibility with the result of normalizing the dictionary heading and the heading. It is a figure which shows an example of the original text containing a period and a colon. It is a figure which shows the variation example of the span required when performing the process for determining whether a period is a part of dictionary headword. It is a figure which shows the structural example of the English morphological analysis system of embodiment. It is a figure which shows an example of a flat dictionary. It is a figure which shows the operation example of the search index preparation part for eliminating the 1st subject. It is a figure which shows the operation example of the span determination part for eliminating the 1st subject. It is a figure which shows the operation example of the dictionary search part for eliminating a 1st subject. It is a figure which illustrates the flat dictionary of this embodiment, and is a figure which shows an example of a dictionary heading, a normalized character string, and a truncation normalized character string. It is a figure which shows the example of reduction of the number of hash search processes at the time of applying this embodiment. It is a figure which shows the operation example of the search index preparation part for eliminating the 2nd subject. It is a figure which shows the operation example of the span determination part for eliminating the 2nd subject. It is a figure which shows the operation example of the dictionary search part for eliminating a 2nd subject. It is the figure which illustrated the variation of the span about the original text which illustrated the period including the period. It is the figure which illustrated the head of the flat dictionary regarding the original sentence shown in FIG. 17, the normalization character string, and the reverse order truncation normalization character string. It is a figure which shows the example of reduction of the number of hash search processes at the time of applying this embodiment with respect to the original text shown in FIG.

  Before starting the description of the present embodiment, first, a prerequisite technique will be described.

(Prerequisite technology)
<Dictionary search technology>
The dictionary search technique / algorithm is one of the basic techniques of computer engineering, and there are various methods for the dictionary search technique and algorithm. Many search methods are called by the name of the tree-like data structure used there. For example, hash search, balanced tree, B tree, Patricia tree, suffix tree, trie tree / trie search, double array, and the like.

  These dictionary search methods have superiority and inferiority in search speed, amount of data structure (index) necessary for search, creation speed of the index, and search method (whether partial match search can be performed, etc.). Are used according to the specifications required by the scene and system used.

<System dictionary, user dictionary>
In a machine translation system, as a machine-readable dictionary used for morphological analysis, both a system dictionary provided in advance in the system and a user dictionary created later by a user are often provided. For example, in Japanese morphological analysis, a system dictionary is a technical dictionary that contains basic dictionaries that contain words that are used on a daily basis, such as “desk” and “move”, and technical terms that are used in specific fields such as “morphological analysis”. Often composed of dictionaries. The basic dictionary can be used for translation of any document, but the technical term dictionary can be selected by the user according to the field of the document to be translated.

The number of registered terms in the system dictionary is often several tens to several millions, and a high-speed search is required. However, the system dictionary is provided by a machine translation system vendor, and it is not assumed that the user adds or deletes terms registered therein. Therefore, since terms are not added or deleted later, the following matters can be realized.
(1) The data structure and search method of the system dictionary can be reduced in weight and speed.
(2) Construction of a machine-readable dictionary, that is, creation of an index for dictionary search only needs to be performed once by the vendor. Therefore, it is preferable that the dictionary search speed by the created index is high even if the dictionary search method is slow in creating the index.
(3) The readonly mode is sufficient when accessing the system dictionary file on the computer system. As a result, the exclusion control can be omitted, the system can be simplified, and the search speed can be increased.

On the other hand, the user dictionary is used by the user after registering / deleting terms, and the required specifications differ from those of the system dictionary. Specifically, (1) even in user dictionaries, the number of registrations may be several hundreds of thousands in recent years, and it is not realistic to re-index the entire dictionary to add or delete one term. Therefore, there is a need for a dictionary search method that can partially recreate the index.
(2) Since there are both reading and writing for the user dictionary, both read and write exist for file access. When a plurality of users and translation processes access the same user dictionary, exclusive control is necessary as necessary.

<Advantage of hash search in user dictionary>
As described above, a search for a user dictionary requires a dictionary search method that can partially re-create an index. In a dictionary search method such as a double array, it is difficult to partially re-index. In Patricia tree, trie tree / trie search, etc., if the number of dictionary registrations is n, the search index has a size of n ^ 2 or more. In many cases, the search index is usually held in a memory for speeding up the search. In recent years, it is not uncommon for the number of dictionary words to reach millions of words. Therefore, if the search index is huge, a huge amount of memory is required for the search.

  Because of this situation and its logic, the user dictionary search often uses a hash search.

<Dictionary search for English morphological analysis using hash search>
Here, dictionary search for English morphological analysis using hash search will be described. The functions required for English morphological analysis include the following.
(1) Dictionary search This is a basic function of English morphological analysis. In English, there are basically symbols such as spaces and hyphens between words. On the other hand, there are also dictionary headings consisting of multiple words such as "computer science". For this reason, it is necessary to perform dictionary lookup for various ranges (spans) of the original text in order to confirm whether the portion does not constitute a dictionary headword.

Hereinafter, in this embodiment, a character string separated by a symbol such as a hyphen or a comma or a space is referred to as a token. An arbitrary continuous token string is called a span.

  An example is shown in FIG. In this example, for the original text "The computer science-based algorithm on partial lattice", the span variations required for the dictionary search are indicated by arrows. 1, the tokens are “computer”, science ”,“ based ”,“ algorithm ”,“ on ”,“ partial ”,“ lattice ”, and these tokens are often registered in the dictionary. However, spans such as “computer science”, “science-based”, and “science-based algorithm” also have one meaning, so there is a possibility that they are registered as one dictionary heading in the dictionary. Therefore, it is necessary to perform a dictionary search in these spans, where “heading” and “dictionary heading” refer to terms (character strings) registered in the dictionary, not just one token, It may be composed of multiple tokens.

  In addition, it is preferable that English factual words such as “Times flies like an arrow.” Can be searched as a single dictionary registered term. Therefore, it is desirable that the search term length or the number of tokens should not have an upper limit.

(2) Diverse dictionary registration For example, in documents related to recent patent applications, extremely long character strings such as chemical formulas, base sequences, DNA sequence character strings, and symbol strings are often treated as one technical term, and therefore extremely long. The number of cases where character strings are directly registered in the dictionary is increasing. It is desirable that such a term can be registered and searched for a user dictionary. These symbol strings may include characters that have not been conventionally used as constituent characters of dictionary headwords, such as hyphens, colons, and periods. In particular, the period symbol is an obstacle to the sentence division processing described later, and thus there are many various limitations in the conventional dictionary search system. However, the ideal dictionary search is that any character string can be registered and searched.

(3) Flexible dictionary search In English, dictionary headings and original text may be different. The main ones are the difference in capitalization and inflection. In other words, when a general word spelled in lowercase is positioned at the beginning of a sentence, the first letter is capitalized (capitalized). In addition, all characters may be capitalized, such as "ATTENTION" and "NO WARRANTY". In addition, there are notations such as fluctuations in spaces and hyphens, and the presence or absence of spaces. An example of this is shown in FIG. FIG. 2 is a diagram showing the correspondence between the original text and dictionary headings. It is desirable that such a fluctuation can be covered by a dictionary search. Furthermore, it is desirable to be able to specify a range of notation fluctuations to be covered in dictionary search. A hash search is suitable for such a flexible search.

  An example is shown in FIG. Specifically, when a dictionary heading is registered in the dictionary, the hash value (index) is calculated after normalizing the notation, and the dictionary heading term is registered in association with the hash value (the left column in FIG. 3). reference). Also in the search, the original value obtained from the user is normalized and then the hash value is calculated to search for the headline terms that match the hash value (see the right column in FIG. 3). When the relationship between the hash value and the dictionary heading is 1: N (N is a plurality), the search target is narrowed down to N by the hash search, and pattern matching or the like is performed for the N pieces. Search processing is performed.

  For normalization, a technique such as stemming is used. FIG. 4 illustrates the result after the stemming process. When performing a dictionary search using a dictionary index (hash table) in which headwords are registered in this way, normalization is also performed on the original text string that serves as a search key, and a hash value is calculated before the dictionary is searched. A search is performed (see the right column of FIG. 3). By this normalization, the same search key can be obtained even if the notation is fluctuated, so that a dictionary search can be performed without depending on the notation.

(4) Sentence division function In a system such as machine translation, when the entire original document is processed (translated), the sentence contained in the original document is divided into individual sentences and processed (translated) as a unit. It is common. In English, periods and semicolons are usually sentence separators, and individual sentences can be obtained by dividing the original sentence at periods or semicolons. However, as shown in FIG. 5, there are many exceptions such as an abbreviation with a period at the end (for example, “Corp.” indicating a company or a company). Further, as described above, if the headings that can be registered in the user dictionary are not limited, dictionary registration such as “book. A”, that is, even a period followed by a space may be part of the heading term. Therefore, it is necessary to perform a dictionary search for accurate sentence division or sentence end position determination.

(Task)
<First issue>
As explained above, when a dictionary search is performed on an English original text in an English morphological analysis, it is necessary to perform a dictionary search for every span of the original text to check whether the span constitutes a dictionary heading. . In this search process, if the number of tokens in the original text is n, the number of dictionary searches is n! Therefore, a very large number of hash dictionary searches are required for a long sentence. This leads to a decrease in the speed of the morphological analysis process. By restricting the word length and the number of entry word tokens that can be registered, the number of dictionary searches can be reduced, but this is an obstacle to the various dictionary registrations described above.

  In the aspect of the present embodiment, when registering a dictionary heading in a hash table, a character string (normalized character string) obtained by performing stemming for each token is not used as a character string serving as a hash key. A character string (censored normalized character string) from which a part exceeding a predetermined character string length is deleted is used. Similarly, when searching, instead of using the character string (normalized character string) stemmed for each token as it is, the character string (censored normalized character string) from which the part exceeding the character string length is deleted is used. Use.

<Second problem>
As described above, when sentence division is performed, it is necessary to determine whether a period or semicolon included in the original sentence is not a part of the dictionary entry word. An example is shown in FIG. In this example, an arrow indicates a variation of the dictionary search, the number of searches, and the span length necessary for determining whether the period after “rrr” is a part of the dictionary entry word. When there is no restriction on the headword length, it is necessary to go back from the period position to the beginning of the sentence as a variation of the start position of the span. Moreover, the variation of the end position of the span needs to reach from the period position to the end of the sentence. Therefore, there is a problem that the number of dictionary searches using a hash is extremely increased, and the speed of morphological analysis processing is reduced.

  In the aspect of this embodiment, in order to prevent an increase in the number of dictionary searches necessary for determining the sentence end position, the above-described normalization character string truncation process is performed on the token string heading forward from the period position. The token sequence used there is reversed. As a result, the number of dictionary searches by hash can be reduced, and the speed of morphological analysis can be increased.

(Aspect of Embodiment)
FIG. 7 shows a configuration example of a system that solves the above-described problem. In this embodiment, a morphological analysis system incorporated in a translation system that acquires an original English sentence and outputs a Japanese translation is shown as an example. However, the present invention is not limited to the translation system. In the present embodiment, processing for English terms in registration data (later flat dictionary described later) is described for dictionary registration, and processing for English original text is described for search. The target language is not limited to English, but may be applied to other natural languages, artificial languages such as programming and mathematical formulas, and formal languages.

  In the following description, it is noted that there are two different (asynchronous) operations: an operation for registering a term in the dictionary and an operation for retrieving a term from the dictionary.

  The English morpheme analysis system 100 (language processing apparatus) includes a document information management unit 10, a morpheme analysis unit 20, and a search index creation unit 30 (registration unit). The morphological analysis unit 20 further includes a sentence division processing unit 21, a span determination unit 22 (determination unit), and a dictionary search unit 23 (search unit). The dictionary search unit 23 further includes a character string normalization unit 24, a hash value The calculation unit 25, the hash table 27, and the morphological analysis dictionary 26 are included.

  The English morphological analysis system 100 is assumed to be a conventional computer having a secondary storage device such as a processor, a memory, and a hard disk drive, an input device such as a keyboard and a mouse, and an output device such as a monitor. Each unit shown in FIG. 7 may be provided by a single computer housing, or may be configured and provided by a plurality of computers according to the use frequency or the number of users. Each unit shown in FIG. 7 is realized by loading a program, which has been introduced in advance into the secondary storage device, into the memory, and executing the loaded program by the processor.

  First, a unit that operates when searching a dictionary will be described. The document information management unit 10 acquires and manages the document information of the entire original document (digitalized document), and stores the sentence division result by the morphological analysis unit 20. When the original document is an HTML or XML document, the document information management unit 10 centrally manages tag information, chapter / section structure information, and the like included in the original document. For this, a technique called DOM (Document Object Model) can be used.

  The morpheme analysis unit 20 refers to the original document information managed by the document information management unit 10 to perform partial or entire sentence division processing and morpheme analysis, and outputs the morpheme analysis result to a file, an output device, or the next step. Are output to a processing unit such as a module or a function to be executed. Morphological analysis results are usually output as part-of-speech strings or part-of-speech lattices for each sentence.

  The sentence division processing unit 21 controls the span determination unit 22 and the dictionary search unit 23 to divide the original sentence.

  The span determination unit 22 creates a span variation for the original text and performs a dictionary search for each span (a variation example of this embodiment will be described later).

  The character string normalization unit 24 generates the same character string regardless of the notation for the same term by absorbing the fluctuation of the notation by a technique such as stemming. For the stemming, a publicly-known technique can be adopted. However, in the present embodiment, the stemmed tokens are further concatenated, and if the concatenated token character string has reached the prescribed length, it is cut off from the beginning of the character string with the prescribed length. Perform processing (processing to delete character strings that exceed the specified length). Also, the character string normalization unit 24 performs processing for concatenating tokens in reverse order when the processing target includes a period. Details of these processes will be described later.

  The hash value calculation unit 25 calculates the hash value of the character string normalized by the character string normalization unit 24. A publicly known technique can be used for the hash function used for calculating the hash value.

  The hash table 27 is used when acquiring a term entry in the morphological analysis dictionary 26 from a hash value, and is usually implemented as a unidirectional link list. The morphological analysis dictionary 26 is configured to machine-readable information on individual terms in a flat dictionary (described later). Note that the hash table 27 and the morphological analysis dictionary 26 are stored in a memory or a secondary storage device (which is referred to as a storage unit).

  Next, the search index creation unit 30 which is a unit that mainly operates during dictionary registration will be described. Prior to execution of the search process (morpheme analysis process), the search index creation unit 30 analyzes the flat dictionary offline to calculate a hash value, and creates a hash table 27 and a morpheme analysis dictionary 26 for hash search. . A flat dictionary is a text file that is a source for creating a dictionary search index. An example is shown in FIG. The flat dictionary of the present embodiment is information including a headline term, a part of speech of the headword, a change form that is information of a word utilization form, translation information, and the like. The search index creation unit 30 normalizes the headword term in the flat dictionary, calculates a hash value of the normalized character string, registers the hash value in the hash table 27, and defines the headline term and part of speech defined in the flat dictionary. The change form and the translation are registered in the morphological analysis dictionary 26 line by line. In addition, the search index creation unit 30 defines the hash value in the hash table 27 to be registered as a link to the heading character string in the morpheme analysis dictionary 26 to be registered.

  The search index creation unit 30 performs the same processing as the character string normalization unit 24 when performing normalization. In other words, the search index creation unit 30 concatenates the stemmed tokens when the headline term is composed of a plurality of tokens, and defines from the beginning of the character string when the concatenated token character string has reached the specified length. A process to cut off by long is performed. Further, the search index creation unit 30 performs a process of concatenating tokens in the reverse order when a period is included in the dictionary head to be processed. Details of this processing will be described later. In addition, the search index creation unit 30 performs the same hash calculation process as the hash value calculation unit 25 in order to match the value with the hash value calculated during the search.

  In addition, the morpheme analysis unit 20 integrates the dictionary search results of each span and outputs a final morpheme analysis result. In addition, utilization information corresponding to the fluctuation of the original text is given to the morphological analysis result. For example, if the original text is "books" and the normalized string is "book" and a dictionary search is possible, the plural form will be included in the heading term ("book") information included in the morphological analysis result. Is given information.

<Mode for the first problem>
In the English morphological analysis system 100 configured as described above, a mode for preventing an increase in the number of hash dictionary searches, which is the first problem described above, will be described. In this embodiment, when registering a dictionary heading in the hash table 27, a character string obtained by performing stemming for each token is not used as a character string to be a hash key, but a character string length defined in advance is used. Use a character string with the part exceeding Hereinafter, description will be made with reference to FIGS. 9 to 13.

  First, the operation at the time of dictionary registration performed prior to the morphological analysis process will be described. The search index creation unit 30 constructs the hash table 27 and the morphological analysis dictionary 26 according to the algorithm shown in FIG.

  The search index creation unit 30 performs the processing of S102 to S110 for each dictionary entry (heading term) in the flat dictionary (S101).

  The search index creation unit 30 registers term information such as part of speech information and translation information in the morphological analysis dictionary 26 (S102). The search index creation unit 30 generates a notation fluctuation (a verb or adjective usage, a plural form of a noun, a possessive case, etc.) based on the usage information described in the dictionary entry (S103).

  The search index creation unit 30 performs the following processing for each notation fluctuation. That is, when the dictionary entry is composed of a plurality of tokens, the search index creation unit 30 divides the dictionary entry into tokens (S105) and performs stemming for each token (S106). The search index creation unit 30 creates a concatenated token character string by concatenating the stemmed token character strings (S107). When the concatenated token character string has reached the specified character string length, the search index creation unit 30 terminates the concatenated character string from the beginning with the specified character string length (S108). The search index creation unit 30 calculates a hash value for the remaining censored character string (canceled token character string, censored normalized character string) (S109).

  The search index creation unit 30 registers a link from the hash value to the term registration position of the morphological analysis dictionary 26 in the hash table 27 (S110).

  Subsequently, an operation at the time of dictionary search (morpheme analysis processing) will be described. In the execution of the morphological analysis process, when the original text is given, the span determination unit 22 generates each span of the original text according to the algorithm shown in FIG. The operation of FIG. 10 will be described. The span determination unit 22 first counts the number of tokens in the original text (S201). Here, the count number is n.

  The span determination unit 22 executes a for loop (nested configuration) until the variable s reaches 1 (from the first token of the sentence) to n and the variable e reaches (s + 1) to n (S202, S203). .

  The span determination unit 22 sets a token string from the variable s to the variable e as one span (S204). A dictionary search is performed for this span (S205). Note that the process of S205 is a process of the dictionary search unit 23, and details will be described later.

  If a matching headline is found in the dictionary search process in step S205 (S206, Y), the dictionary search result is output (S207), and the process proceeds to S208. If no matching headline is found (S206, N), or after the processing of S207 is performed, the span determination unit 22 determines whether the censored normalized character string exceeds the specified length (S208). When the specified length is exceeded (S208, Y), the span determination unit 22 terminates the inner loop (loop of variable e), increments variable s, and executes the processing from S203 again (S209). . On the other hand, when the length is equal to or less than the specified length (S208, N), the span determination unit 22 increments the value of the variable e and executes the processing from S204 again.

  By the operations of S201 to S209, spans are generated in the order as shown in FIG. The dictionary search unit 23 performs a dictionary search (the process of S205) according to the algorithm shown in FIG. 11 with respect to the original character string span given from the span determination unit 22.

  The dictionary search unit 23 divides the character string included in the span (span character string) into tokens (S301). The dictionary search unit 23 performs stemming for each token (S302), and connects the token character strings that are stemmed (S303). The dictionary search unit 23 terminates the concatenated token character string with the specified character string length from the beginning of the character string (S304). The dictionary search unit 23 calculates a hash value of the censored normalized character string (S305), and acquires a link to the term registration position of the morphological analysis dictionary 26 based on the calculated hash value and the hash table 27 (S306). .

  The dictionary search unit 23 acquires the term information (one or more (N)) registered in the morphological analysis dictionary 26 from the acquired link (S307), and performs the following processing for each term information (S308). . The dictionary search unit 23 generates notation fluctuations (verb or adjective usage, plurals of nouns, possession, etc.) based on the usage information described in each entry of the morphological analysis dictionary 26 (S309). The dictionary search unit 23 determines whether the span character string and the subsequent character string in the original text match any of the notation fluctuations (S310). When they do not match (S310, they do not match), the search processing by the dictionary search unit 23 ends, and when they match (S310, they match), the dictionary search unit 23 responds to the utilization information of the notation of the matching fluctuation. Utilization information is added to the dictionary search result (S311), and the dictionary search result is output (S312). The output destination at this time is assumed to be the function caller span determination unit 22, but may be output to a file or an output device.

  A specific example of each of the above operations will be described with reference to FIGS.

  FIG. 12 explains the operation of the search index creation unit 30 shown in FIG. 9, and is an example when registering the heading terms in the first column of FIG. If there is a heading term as shown in the first column of FIG. 12 in the flat dictionary, the search index creation unit 30 normalizes and concatenates each heading for each token (see the second column). The censored normalized character string shown in the third column is generated. The specified length of censoring is 14 in this example. The search index creation unit 30 calculates a hash value for the censored normalized character string, and generates a link from the hash table 27 to the morpheme analysis dictionary 26 based on the value. It should be noted that the processing up to the generation of the censored normalized character string may be performed by the character string normalization unit 24, and the hash value calculation unit 25 may perform the calculation of the hash value.

FIG. 13 illustrates how the span determination unit 22 and the dictionary search unit 23 perform a dictionary search for a given original sentence using the morphological analysis dictionary 26 and the hash table 27 in which the heading terms of FIG. 12 are registered. Is. As shown in FIG. 10, the span determination unit 22
(1) Extend the end of the span (shift backward one token).
(2) When the end of the span reaches the end of the sentence, the span start position is shifted backward by one token.
Each variation of the span is generated in this order.

  The dictionary search unit 23 performs a dictionary lookup for each generated span, but when the censored normalized character string reaches a specified value, subsequent span end expansion and dictionary search for it can be omitted. This is because if a headline term longer than that exists in the morphological analysis dictionary 26, the hash key thereafter becomes the same as “conputscienceb” at the time of registration shown in FIG.

In the example of FIG. 13, the search is performed in the order of “computer” and “computer science”, and the hash value of the truncated normalized character string “conputscienceb” is calculated in the character string search for the “computer science-based” span. Is done. The heading linked to this hash value is shown in FIG.
computer science-based
computer science-based algorithm
computer science-based algorithm course
computer science-based theory
The four headings are searched as being headlines linked to the hash value. Of which, the original matches
computer science-based
computer science-based algorithm
It is. Accordingly, these two are output as dictionary search results.

As shown in FIG.
"computer science-based algorithm"
"computer science-based algorithm on"
"computer science-based algorithm on partial"
"computer science-based algorithm on partial lattice"
In contrast, a dictionary search based on the hash value is unnecessary. Similarly, the dictionary search for “science”, which is the next step, can be omitted if the censored normalized character string reaches a specified value, and the subsequent span end expansion and dictionary search for it are omitted.

  When this embodiment is not applied, from the viewpoint of omission of search, for all span variations of the long token character string, tokens are normalized, concatenated to generate a character string, and a hash value is calculated. The operation is to search for linked headings.

  On the other hand, by applying this embodiment, only the processing until reaching the censored normalized character string may be performed based on the token normalization, character string concatenation, hash value calculation, and hash value. For spans longer than the censored normalized character string, it is only necessary to search from several to several tens of cases as described above. As in the present embodiment, the number of search processes using a hash value can be reduced by canceling the search using the hash value using the censored normalized character string. The price of this is that the number of hash search results may increase compared to the case of performing a hash search for all span variations, but the method of extracting from the search results by matching or the like is faster. Can be achieved. As described above, the first problem is solved by applying this embodiment.

<Mode for the second problem>
Next, a mode for preventing an increase in the number of dictionary searches necessary for determining the sentence end position, which is the second problem, will be described. In the present embodiment, the above-described normalization character string truncation process is performed on the token string ahead from the period position. The token sequence used there is reversed. Hereinafter, a description will be given with reference to FIGS.

  First, prior to the morpheme analysis process (search process), the search index creation unit 30 constructs the hash table 27 and the morpheme analysis dictionary 26 according to the algorithm shown in FIG. This process is performed for terms that include a period in the heading.

  The search index creation unit 30 performs the following process for dictionary entries that include a period in the flat dictionary as a heading term (S401).

  The search index creation unit 30 registers term information such as part-of-speech information and translation information in the morphological analysis dictionary 26 (S402). The search index creation unit 30 generates notation fluctuations (verb and adjective usage forms, plural forms of nouns, possessive case, etc.) based on the usage information described in the dictionary entry (S403).

  The search index creation unit 30 performs the following process for each notation fluctuation. That is, if the dictionary entry is composed of a plurality of tokens, the search index creation unit 30 divides the token entry into tokens (S405), deletes tokens after the period (S406), and performs stemming for each token. (S407). The search index creation unit 30 reverses the order in token units (S408), and concatenates the token character strings in the reverse order (S409). S407 and S408 may be reversed. The search index creation unit 30 creates a normalized character string that is truncated at the specified character string length from the beginning of the token character string, and creates a reverse-censored normalized character string (deletes characters that exceed the specified character string length) (S410). The search index creation unit 30 calculates a hash value of the reverse order censored normalized character string (S411).

  The search index creation unit 30 registers a link from the hash value to the term registration position of the morphological analysis dictionary 26 in the hash table 27 (S412).

  The operation during search will be described. At the time of retrieval, when the original sentence is given during execution of the sentence end determination process, the span determination unit 22 generates a span around the period position of the original sentence according to the algorithm shown in FIG. The operation of FIG. 15 will be described. In this process, the span determination unit 22 can also determine whether or not the period is a period meaning a sentence cut.

  The span determination unit 22 first searches for a period position in the original text (S501). The position of this period is n. Next, the span determination unit 22 repeatedly executes the processing of S503 to S508 until the variable s reaches 1 from (n−1) (S502). In S502, the value (n-1) means the token immediately before (immediately before) the period character, and the value 1 means the first token of the sentence.

  The span determination unit 22 sets a token string from the variable s to (n−1) as one span (S503), and a dictionary search is performed for this span (S504). Note that the process of step S504 is a process of the dictionary search unit 23, and details will be described later.

  If a matching headline is searched in the dictionary search process in step S504 (S505, Y), the period is a character used in a term registered in the dictionary, and therefore the span determination unit 22 It is not treated as a period meaning a sentence delimiter, and therefore the sentence is not cut (S506). Then, the process proceeds to S507. On the other hand, if no matching headline is found (S505, N), or after the processing of S506 is performed, the span determination unit 22 determines whether the reverse-censored normalized character string exceeds the specified length ( S508). If it exceeds the specified length (S508, Y), the span determination unit 22 cuts the sentence at the period and ends the process because the period is not a character used in the term registered in the dictionary. (S508). If the reverse-censored normalized character string does not exceed the specified length (S507, N), the span determination unit 22 decrements the value of the variable s and executes the processing from S503 again.

  The dictionary search unit 23 performs a dictionary search (the process of S504 above) on the span around the period position of the original text given from the span determination unit 22 in accordance with the algorithm shown in FIG.

  The dictionary search unit 23 divides the character string in the span (span character string) that ends with a period into tokens (S601). The dictionary search unit 23 performs stemming for each token (S602), reverses the stemmed tokens in reverse order (S603), and connects the tokens after the reverse order (S604). The order of steps S602 and S603 may be reversed, but the order is the same as S407 and S408 in FIG.

  The dictionary search unit 23 cuts off at the specified character string length from the beginning of the character string (reverse censored normalized character string) (S605). The dictionary search unit 23 calculates a hash value of the reverse censored normalized character string (S606), and acquires a link to the term registration position of the morphological analysis dictionary 26 based on the calculated hash value and the hash table 27 (S607). ).

  The dictionary search unit 23 acquires the term information (one or more (N)) registered in the morphological analysis dictionary 26 from the acquired link (S608), and performs the following processing for each term information (S609). . Based on the utilization information described in each entry of the morphological analysis dictionary 26, the dictionary search unit 23 generates notation fluctuations (utilization forms of verbs and adjectives, plural forms of nouns, possessive cases, etc.) (S610). The dictionary search unit 23 determines whether the span character string matches any of the above-described fluctuations (S611). When they do not match (S611, they do not match), the search processing by the dictionary search unit 23 ends. When they match (S611, they match), the dictionary search unit 23 indicates that the character string after the period of the original text It is determined whether or not it coincides with the portion after the period of fluctuation (S612). When they do not match (S612, they do not match), the search processing by the dictionary search unit 23 ends, and when they match (S612, matches), the dictionary search result is output (S613). The output destination at this time is assumed to be the function caller span determination unit 22, but may be output to a file or an output device.

  Hereinafter, the above operation will be described in detail with reference to FIGS. 17, 18, and 19.

  FIG. 17 shows a variation of all spans including a period with respect to a certain original including the period. Theoretically, since each of these may be a dictionary registered word, it is necessary to perform a dictionary search for every variation.

  FIG. 18 explains the operation of the search index creation unit 30 shown in FIG. 14, and is an example in the case of registering the first column of FIG. When the flat dictionary includes a heading including a period as included in the first column of FIG. 18, the search index creation unit 30 normalizes and concatenates tokens with respect to the token sequence ahead from the period position ( (Refer to the second column) In reverse order, a reverse-censored normalized character string as shown in the third column is created. In this example, the specified cutoff length is 12. A hash value is calculated for the generated reverse-censored normalized character string, the calculated hash value is registered in the hash table 27, and definition information in the flat dictionary including a heading term is hashed in the morphological analysis dictionary 26. Registered in association with the value.

  FIG. 19 illustrates how the original sentence division position is determined using the morphological analysis dictionary 26 and the hash table 27 in which the heading terms of FIG. 18 are registered.

  As shown in FIG. 15, if there is a period in the original text, the span determination unit 22 creates a span from the position to the front, and determines the range of the token string to be processed. Then, the dictionary search unit 23 performs a dictionary search for each of them. The dictionary search logic performs dictionary lookup for each generated span, but when the reverse-censored normalized character string reaches a specified value, subsequent span expansion and dictionary search by hashing can be omitted. .

  In the example of FIG. 19, the dictionary heading “G. Thompson” is searched in the dictionary search for the span “G.” (see also FIG. 18). Also, the dictionary headings “analysis method by G. Thompson” and “molecular analysis method by G.” are searched by dictionary search for the span “analysis method by G.”. At this point, since the reverse-censored normalized character string has reached the specified value, it is not necessary to perform a dictionary search by hash for the span “molecular analysis method by G.” or “optical molecular analysis method by G.”.

  If sentence division is the purpose, if even one dictionary word is searched at a period position, it can be determined that the period at that position is part of the word and should not be divided, and the process can be terminated. . When there are a plurality of periods in the original text, the method described above may be applied to each period.

  With such an aspect, sentence division can be performed with a much smaller number of dictionary searches than in FIG. As explained above in “Aspects for the first problem”, by using the reverse-censored normalized character string, the search based on the hash value is terminated, and instead, a matching item is extracted from a plurality of hits. The search is faster than the search by hash value for all spans.

  In this example, the order is reversed in normalized token units. That is, the reverse-censored normalized character string is generated in the order of “g”, “b”, “method”, “anal”, but it may be reversed in character units. In this case, the generated truncation character string is “gbdohtemsyla”. Naturally, when the order is reversed in units of one character at the time of registration, it is necessary to reverse the order in units of one character to match the registered hash value even at the time of searching.

  As described above, the term search can be speeded up by the aspect described in this embodiment.

  In addition, although embodiment of this invention was described, the said embodiment is shown as an example and is not intending limiting the range of invention. The novel embodiment can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalents thereof.

  10 document information management unit, 20 morpheme analysis unit, 21 sentence division processing unit, 22 span determination unit, 23 dictionary search unit, 24 character string normalization unit, 25 hash value calculation unit, 26 morpheme analysis dictionary, 27 hash table, 30 Search index creation unit, 100 English morphological analysis system (language processing device).

Claims (5)

A determination unit that determines the range of the character string in the original text;
The token included in the range determined by the determination unit is normalized for each token, and the normalized token is concatenated to create a concatenated token character string, and the concatenated token character string reaches a specified length. The concatenated token character string is truncated at the specified length, a hash value of the truncated token character string is calculated, and a term associated with the calculated hash value is associated with a hash value and a term in advance. A search unit that acquires one or more from the storage unit registered in
A language processing apparatus. The language processing apparatus according to claim 1,
The search unit reverses the order of tokens in front of the specified character in the token unit or the character unit before or after normalization when the specified character used as a sentence delimiter is in the original text. , A language processing apparatus for creating a concatenated token character string by concatenating the token character strings in the reverse order. The language processing device according to claim 1, further comprising:
Acquires the character string of the registered term described in the data for registering the term, normalizes the tokens included in the registered term character string for each token, concatenates the normalized tokens, and concatenates the token characters If the concatenated token character string has reached the specified length, the concatenated token character string is censored at the specified length, a hash value of the censored token character string is calculated, and the calculated hash value and A language processing apparatus including a registration unit that registers the registered term character string in association with the storage unit. On the computer,
Let the range of text in the text be determined,
Normalize tokens included in the determined range for each token,
Concatenating the normalized tokens to create a concatenated token string;
If the concatenated token character string has reached a specified length, the concatenated token character string is terminated at the specified length;
Calculating a hash value of the censored token string;
A program for acquiring one or more terms associated with the calculated hash value from a storage unit in which the hash value and the term are associated and registered in advance. Computer
Determine the range of text in the source text,
Normalize tokens included in the determined range for each token,
Concatenating the normalized tokens to create a concatenated token string,
If the concatenated token character string has reached a specified length, the concatenated token character string is truncated at the specified length;
Calculate a hash value of the censored token string,
A method of acquiring one or more terms associated with the calculated hash value from a storage unit in which the hash value and the term are associated and registered in advance.

Images