JP2010140107A - Method, apparatus, program, and computer readable recording medium for registering unknown word - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To extract with high accuracy unknown words for use in a morphological analysis apparatus. <P>SOLUTION: Unknown word candidates are extracted from a set of search keywords including differences of a search keyword obtained when one query of a query log often input as a set of search keywords consisting of one compound sentence is divided by spaces. The unknown word candidates which are used frequently are extracted as unknown words and registered in a reference dictionary. A method of extracting the unknown words includes: merging the set of unknown word candidates with the reference dictionary to create a temporary dictionary, morphologically analyzing corpus using the temporary dictionary, counting the frequencies of occurrence of the unknown word candidates, and defining the unknown word candidates whose occurrence frequencies are greater than a predetermined threshold value as unknown words. Alternatively, a search system performs search using the unknown word candidates as a search query, and defines the unknown word candidates whose search results are greater in number than a predetermined threshold value as unknown words. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an unknown word registration method, apparatus, program, and computer-readable recording medium, and more particularly to automatically collecting unknown words for use in a morphological analyzer and registering them in a dictionary of the morphological analyzer. The present invention relates to an unknown word registration method and apparatus, a program, and a computer-readable recording medium.

  As a conventional unknown word extraction technique, there is a technique using a word probability model. In this method, the character likelihood of a character string is determined by using a character string constituting the word and a probability model of a sentence in which the word appears (see, for example, Non-Patent Document 1).

In addition, there is a technique for extracting words that are not included in the word dictionary as unknown words by morphological analysis. In this method, morphological analysis is used to extract a character string that cannot be analyzed or a combination of words around it as an unknown word (see, for example, Patent Document 1).
JP 11-338863 A Masaaki Nagata, "Obtaining Vocabulary from Text Based on Probability Model of Unknown Words and Expected Frequency of Words", IPSJ Journal, vol.40. No.9, 1999

  However, the above unknown word extraction method using the probability model requires a word-division corpus that is used for learning the model, but creating this corpus is expensive. Also, new types of unknown words that do not appear in this corpus cannot be extracted.

  The unknown word extraction method based on morphological analysis mainly extracts unknown words based on character type information, such as treating the same character type character string as one word, but this method consists of many character types. It is difficult to extract words that have been used.

  The present invention has been made in view of the above points, and an object thereof is to provide an unknown word registration method and apparatus, a program, and a computer-readable recording medium that can extract unknown words with high accuracy.

  FIG. 1 is a diagram for explaining the principle of the present invention.

The present invention (Claim 1) is an unknown word registration method for registering an unknown word in a reference dictionary used for morphological analysis,
When a query log is input, the unknown word candidate acquisition means extracts an unknown word candidate from a search keyword set that is a different set of search keywords obtained by dividing one query of the query log by a space, and stores it in the storage means An unknown word candidate acquisition step (step 1),
An unknown word extraction unit that extracts frequently used unknown word candidates from among the unknown word candidates stored in the storage unit as an unknown word and stores it in the storage unit (step 2);
The unknown word registration means performs an unknown word registration step (step 3) for registering the unknown words stored in the storage means in the reference dictionary.

Further, the present invention (Claim 2) is an unknown word candidate acquisition step (Step 1).
Only when each search keyword in the search keyword set is not included in the reference dictionary, it is considered as an unknown word candidate.
Or
If each search keyword in the search keyword set is a verb stem and an inflection ending, or only one morpheme, it is not an unknown word candidate, otherwise it is an unknown word candidate.

Further, the present invention (Claim 3) provides an unknown word extraction step (Step 2).
A temporary dictionary is generated by merging a set of unknown word candidates stored in the storage means and the reference dictionary, and a morphological analysis of the corpus is performed using the temporary dictionary.
The frequency of appearance of unknown word candidates is counted, and unknown word candidates whose appearance frequency is higher than a predetermined threshold are stored in the storage means as unknown words.

Further, the present invention (Claim 4), in the unknown word extraction step (Step 2),
Perform a search with the search system using the unknown word candidate stored in the storage means as a search query,
Unknown word candidates whose number of search results is greater than a predetermined threshold are stored in the storage means as unknown words.

  FIG. 2 is a principle configuration diagram of the present invention.

The present invention (Claim 5) is an unknown word registration device for registering an unknown word in the reference dictionary 2 used for morphological analysis,
When a query log is input, an unknown word candidate is extracted from a search keyword set, which is a set of different search keywords obtained by dividing one query of the query log by a space, and is stored in a storage unit. When,
An unknown word extraction means 20 that extracts frequently-used words from unknown word candidates stored in the storage means as unknown words, and stores them in the storage means;
And an unknown word registration means 30 for registering an unknown word stored in the storage means in the reference dictionary 2.

Further, the present invention (Claim 6) provides the unknown word candidate acquisition means 10 with:
Means for making an unknown word candidate only when each search keyword of the search keyword set is not included in the reference dictionary,
Or
Each search keyword in the search keyword set includes either a verb stem and an inflection ending, or means for not using an unknown word candidate if it is only one morpheme, and otherwise making it an unknown word candidate.

Further, the present invention (Claim 7) is provided in the unknown word extraction means 20,
A temporary dictionary is generated by merging a set of unknown word candidates stored in the storage means and the reference dictionary 2, and a morphological analysis of the corpus is performed using the temporary dictionary, and the frequency of appearance of the unknown word candidates is counted. And means for storing in the storage means unknown word candidates whose appearance frequency is greater than a predetermined threshold value as unknown words.

Further, the present invention (claim 8) provides the unknown word extraction means 20 with:
A means for searching the unknown word candidate stored in the storage means using the search system as a search query, and storing in the storage means an unknown word candidate whose number of search results is greater than a predetermined threshold as an unknown word.

  The present invention (Claim 9) is an unknown word registration program for causing a computer to function as each means constituting the unknown word registration apparatus according to any one of Claims 5 to 8.

  The present invention (Claim 10) is a computer-readable recording medium storing the unknown word registration program according to Claim 9.

  In the present invention, in order to extract an unknown word with an accurate morpheme, a query log that is frequently input as a set of search keywords including one compound word phrase is used. Also, in order to make the input query the smallest unit, it is divided for each space. The search keyword obtained as a result is rarely composed of a plurality of compound word phrases, and when used frequently, it is considered to have a morpheme property. The search keyword obtained by the division is set as an unknown word candidate, and how many of the candidates can be determined as an unknown word is determined by two types of methods.

  Thus, according to the present invention, compared to the method of extracting unknown words using a probability model, there is no problem that a new type of unknown word that does not match the probability model cannot be extracted without requiring a word-divided corpus. In addition, it is possible to extract unknown words composed of various character types, compared to the method of extracting unknown words by morphological analysis.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[First Embodiment]
FIG. 3 shows the configuration of the unknown word registration device according to the first embodiment of the present invention.

  The unknown word registration apparatus shown in FIG. 1 includes a query log storage unit 1, a reference dictionary 2, a temporary dictionary 3, a corpus storage unit 4, a keyword set memory 5, an unknown word candidate set memory 6, an unknown word set memory 7, and an unknown word candidate acquisition. Unit 10, unknown word extraction unit 20, and unknown word registration unit 30. Among these, the query log storage unit 1, the reference dictionary 2, the temporary dictionary 3, and the corpus storage unit 4 are storage media such as a hard disk device, and also include a keyword set memory 5, an unknown word candidate set memory 6, and an unknown word set memory 7. Is a memory for temporarily storing processing results.

  The unknown word candidate acquisition unit 10 refers to the reference dictionary 2 based on the query log acquired from the query log storage unit 1, collects unknown word candidates that are not included in the reference dictionary 2, and stores them in the unknown word candidate set memory 6. Store.

  The unknown word extraction unit 20 acquires unknown word candidates from the unknown word candidate set memory 6, extracts unknown words from the unknown word candidates, and stores them in the unknown word set memory 7.

  The unknown word registration unit 30 registers the unknown word acquired from the unknown word set memory 7 in the reference dictionary 2.

  Hereinafter, the operation of each component will be described.

  The operation of the unknown word candidate acquisition unit 10 will be described below. The unknown word candidate acquisition unit 10 extracts unknown word candidates from the query log in the query log storage unit 1.

  FIG. 4 is a flowchart of the operation of the unknown word candidate acquisition unit in the first embodiment of the present invention.

  First, the unknown word candidate acquisition unit 10 acquires a query log from the query log storage unit 1, generates a search keyword set that is a different set of search keywords obtained by dividing one query by a space, and stores it in the keyword set memory 5. (Step 101).

  Next, a search keyword is acquired from the keyword set memory 5, an unknown word candidate set is generated, and stored in the unknown word candidate set memory 6 (step 102). In that case, two methods can be considered.

  One is a method of making an unknown word candidate only when the search keyword is not included in the reference dictionary 2. The other is a morphological analysis using the reference dictionary 2 for the search keyword. If the search keyword is a verb stem and an inflection ending, or one morpheme, it is not an unknown word candidate. This is a method of making an unknown word candidate.

  Next, the unknown word extraction unit 20 will be described.

  In the unknown word extraction unit 20, an unknown word candidate is acquired from the unknown word candidate set memory 6, and the morphological analysis is performed on the corpus of the corpus storage unit 4 with the unknown word candidate registered in the temporary dictionary 3 as one word. And an unknown word set is obtained based on the result of counting the appearance frequency of unknown word candidates.

  FIG. 5 is a flowchart of the operation of the unknown word extraction unit in the first embodiment of the present invention.

  The unknown word extraction unit 20 generates a temporary dictionary 3 in which the unknown word candidates included in the unknown word candidate set memory 6 are merged with the contents of the reference dictionary 2 assuming nouns (step 201). Next, a morphological analysis of the corpus of the corpus storage unit 4 is performed using the temporary dictionary 3 (step 202), and unknown word candidates included in the unknown word candidate set memory 6 output as one morpheme from the result are displayed. The appearance frequency is counted (step 203). An unknown word candidate whose appearance frequency is equal to or higher than a threshold is extracted as an unknown word and stored in the unknown word set memory 7 (step 204).

  Next, the unknown word registration unit 30 will be described.

  The unknown word registration unit 30 registers unknown words extracted by the unknown word extraction unit 20 and stored in the unknown word set memory 7 in the reference dictionary 2.

[Second Embodiment]
In the present embodiment, the processing of the unknown word extraction unit 20 is different from that of the first embodiment.

  FIG. 6 shows the configuration of the unknown word registration device according to the second embodiment of the present invention. In the figure, the same components as those in FIG.

  The unknown word registration device shown in FIG. 1 includes a query log storage unit 1, a reference dictionary 2, a keyword set memory 5, an unknown word candidate set memory 6, an unknown word set memory 7, an unknown word candidate acquisition unit 10, an unknown word extraction unit 25, The unknown word registration unit 30 is configured, and the unknown word extraction unit 25 is connected to an external search system 40.

  The processes of the unknown word candidate acquisition unit 10 and the unknown word registration unit 30 are the same as those in the first embodiment.

  The unknown word extraction unit 25 searches the unknown word candidate stored in the unknown word candidate set memory 6 as a search query by the search system 40, and determines whether the unknown word candidate is an unknown word based on the number of search results.

  FIG. 7 is a flowchart of the operation of the unknown word extraction unit in the second embodiment of the present invention.

  The unknown word extraction unit 25 determines an unknown word candidate X to be processed from the unknown word candidates stored in the unknown word candidate set memory 6 (step 301). A search is performed by the search system 40 using the unknown word candidate X as a search query (step 302). At this time, a search engine for the Web may be used as the search system 40. The number of search results obtained from the search system 40 is compared with a predetermined threshold, and if the number of search results is greater than or equal to the threshold (step 303, Yes), an unknown word candidate is extracted as an unknown word, and the unknown word set memory 7 To store. On the other hand, when the number of search results is smaller than the threshold (No at Step 303), the next candidate word X to be processed is determined and the above process is repeated (Step 301).

  Further, the components of FIG. 3 of the first embodiment and FIG. 6 of the second embodiment are constructed as a program, installed in a computer used as an unknown word registration device, and executed by the CPU. Alternatively, it can be distributed via a network.

  Further, the constructed program can be stored in a portable storage medium such as a hard disk, a flexible disk, or a CD-ROM, and can be installed or distributed in a computer.

  The present invention is not limited to the above-described embodiment, and various modifications and applications can be made within the scope of the claims.

  The present invention is applicable to a technique for registering an unknown word in a dictionary used for morphological analysis.

It is a figure for demonstrating the principle of this invention. It is a principle block diagram of this invention. It is a block diagram of the unknown word registration apparatus in the 1st Embodiment of this invention. It is a flowchart of operation | movement of the unknown word candidate acquisition part in the 1st Embodiment of this invention. It is a flowchart of operation | movement of the unknown word extraction part in the 1st Embodiment of this invention. It is a block diagram of the unknown word registration apparatus in the 2nd Embodiment of this invention. It is a flowchart of operation | movement of the unknown word extraction part in the 2nd Embodiment of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Query log memory | storage part 2 Reference dictionary 3 Temporary dictionary 4 Corpus memory | storage part 5 Keyword set memory 6 Unknown word candidate set memory 7 Unknown word set memory 10 Unknown word candidate acquisition means, unknown word candidate acquisition part 20 Unknown word extraction means, unknown word extraction Unit 25 unknown word extraction unit 30 unknown word registration means, unknown word registration unit 40 search system (search engine)

Claims (10)

An unknown word registration method for registering an unknown word in a reference dictionary used for morphological analysis,
When a query log is input, the unknown word candidate acquisition means extracts an unknown word candidate from a search keyword set that is a different set of search keywords obtained by dividing one query of the query log by a space, and stores it in the storage means An unknown word candidate acquisition step,
An unknown word extracting means extracts an frequently used unknown word candidate from among the unknown word candidates stored in the storage means as an unknown word, and stores the unknown word extraction step in the storage means;
An unknown word registration unit registers an unknown word stored in the storage unit in the reference dictionary, and an unknown word registration step;
The unknown word registration method characterized by performing. In the unknown word candidate acquisition step,
Only when each search keyword of the search keyword set is not included in the reference dictionary, is an unknown word candidate,
Or
The unknown word registration method according to claim 1, wherein each search keyword in the search keyword set is a verb stem and an inflection ending, or an unknown word candidate if it is only a morpheme, and an unknown word candidate otherwise. In the unknown word extraction step,
Merging the set of unknown word candidates stored in the storage means and the reference dictionary to generate a temporary dictionary, using the temporary dictionary to perform a morphological analysis of the corpus,
The unknown word registration method according to claim 1, wherein the appearance frequency of the unknown word candidates is counted, and unknown word candidates having the appearance frequency higher than a predetermined threshold value are stored in the storage unit as unknown words. In the unknown word extraction step,
Perform a search in a search system using the unknown word candidate stored in the storage means as a search query,
The unknown word registration method according to claim 1, wherein unknown word candidates whose number of search results is greater than a predetermined threshold are stored in the storage unit as unknown words. An unknown word registration device that registers unknown words in a reference dictionary used for morphological analysis,
An unknown word candidate acquiring means for extracting an unknown word candidate from a search keyword set which is a different set of search keywords obtained by dividing one query of the query log by a space when a query log is input; ,
Extracting frequently used unknown word candidates from among the unknown word candidates stored in the storage means as unknown words, and storing the unknown word extraction means in the storage means;
Unknown word registration means for registering unknown words stored in the storage means in the reference dictionary;
An unknown word registration device characterized by comprising: The unknown word candidate acquisition means includes:
Means for making an unknown word candidate only when each search keyword of the search keyword set is not included in the reference dictionary;
Or
6. The method according to claim 5, wherein each search keyword of the search keyword set includes a verb stem and an inflection ending, or a means for not using an unknown word candidate if it is only a morpheme, and an unknown word candidate otherwise. Unknown word registration device. The unknown word extraction means includes
The temporary dictionary is generated by merging the set of unknown word candidates stored in the storage means and the reference dictionary, and a morphological analysis of the corpus is performed using the temporary dictionary, and the appearance frequency of the unknown word candidates is determined. 6. The unknown word registration device according to claim 5, further comprising means for counting and storing unknown word candidates whose appearance frequency is greater than a predetermined threshold as unknown words in the storage means. The unknown word extraction means includes
A means for performing a search by a search system using the unknown word candidate stored in the storage means as a search query, and storing an unknown word candidate having a number of search results larger than a predetermined threshold as an unknown word in the storage means. 5. The unknown word registration device according to 5.   The unknown word registration program for functioning a computer as each means which comprises the unknown word registration apparatus of any one of Claim 5 thru | or 8.   A computer-readable recording medium storing the unknown word registration program according to claim 9.

Images