JP2002092017A - Concept dictionary extending method and its device and recording medium with concept dictionary extending program recorded thereon - Google Patents

Abstract

PROBLEM TO BE SOLVED: To calculate the attribute vectors of specific nouns or new words which do not exist in a concept dictionary, and to add those attribute vectors to the concept dictionary. SOLUTION: A relevancy calculating part 4 acquires the information of a user ID (or a terminal ID), retrieval words, and a retrieval time from a retrieval log in a retrieval log data base 3, and calculates relevancy among the retrieval words, and preserves relevant word data in a relevant data base 5. A new word vector calculating part 6 acquires relevant word data related with words read from a new word list 1 from the relevant word data base 5, and calculates the attribute vectors of new words by using word attribute vector information in a concept dictionary 2 and a temporarily preserved vector data base 7 based on the relevant word data, and outputs the attribute vectors of the new words to the temporarily preserved vector data base 7 or a new word dictionary 8.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a concept dictionary extending apparatus for adding a new word to a concept dictionary used for performing a semantic processing of a natural language sentence on a computer.

[0002]

2. Description of the Related Art With the recent development of the Internet and the like, a large number of digitized documents are present, and there is an increasing demand for searching and classifying them. The method of searching and classifying digitized documents includes a method using a document vector based on the frequency of occurrence of a word, and a method using a concept dictionary expressing the meaning of the word with an attribute vector, and each word appearing in the document. There is a method of using a document vector expressed by the sum of the attribute vectors.

[0003] Those utilizing the frequency of appearance of words have the advantage of not requiring a dictionary in particular, but rely only on the notation, and words with different notations are treated as completely different words. There is a drawback that closeness of meaning between words cannot be expressed. On the other hand, in the method using the concept dictionary, words having similar attribute vectors in the concept dictionary can be determined as similar words, and therefore, similarity of meaning between words can be handled.

[0004] As a method of creating a concept dictionary, an existing Japanese language dictionary or the like is used, a word of a headword is set as an attribute of a word appearing in a subsequent meaning sentence, and the number of appearances is set as a value of the attribute. There is a way to define a vector.

[0005]

However, in the method using the meaning of the dictionary, it is not possible to define an attribute vector of a word other than a word included in the dictionary. For this reason, there is a problem that proper nouns and new words appearing on WWW pages on the Internet cannot be handled.

An object of the present invention is to provide a concept dictionary extension method, a concept dictionary extension program and a concept dictionary extension method for calculating an attribute vector of a proper noun or a new word which does not exist in the concept dictionary, adding the vector to the concept dictionary, and extending the concept dictionary. An object of the present invention is to provide a recorded recording medium.

[0007]

The concept dictionary expansion apparatus of the present invention has a concept dictionary, a search log database, a related word database, a new word list, a temporary storage vector database, a new word dictionary, a relevance calculator, and a new word vector calculator. .

[0008] The relevance calculation unit is obtained from the search log.
The degree of relevance between the two search terms is calculated using the information on the time interval of each of the two search terms used by the search user, and related word data including the search terms and their relevance is created.

The new word vector calculation unit reads one word from a new word list which is a list of new words to which an attribute vector is added, and receives a related word relating to the word together with the degree of relevance from a related word database.

Next, for related words that already exist in the concept dictionary, attribute vectors are obtained from the concept dictionary,
These are weighted by the degree of relevance and then added to obtain a new word attribute vector. This is performed for each word in the new word list, and the result is stored in the temporary storage vector database.

Next, the new word vector calculation unit reads one word from the new word list again, and obtains related words and their degrees of relevance from the related word database in the same manner as described above.

[0012] Among the related words, those existing in the concept dictionary are:
For those existing in the temporary storage vector database from the concept dictionary, the attribute vectors are obtained from the temporary storage vector database, weighted by the degree of relevance, and added together to form a new attribute vector for the new word. To record.

The operation of acquiring related word data and calculating a new attribute vector by using the attribute vector data in the concept dictionary and the temporary storage vector database is performed a predetermined number of times, It repeats until the sum of the differences of the new attribute vectors falls below a predetermined threshold, and outputs the finally obtained result to the new word dictionary.

As described above, the concept dictionary can be expanded by calculating the attribute vector of a new word from the search log and the concept dictionary and adding the new word dictionary to the concept dictionary.

[0015]

Next, embodiments of the present invention will be described with reference to the drawings.

Referring to FIG. 1, an apparatus for expanding a concept dictionary according to an embodiment of the present invention includes a new word list 1, a concept dictionary 2, a search log database 3, a relevance calculator 4, a related word database 5, and a new word vector calculator 6. And a temporary storage vector database 7 and a new word dictionary 8.

The new word list 1 stores a list of words to which an attribute vector is added. The concept dictionary 2 is a dictionary expressing the meaning of a word by an attribute vector. The search log database 3 stores a search log of a WWW search engine or a search log of a database. The relevance calculator 4 obtains information on the user ID (or terminal ID), the search term, and the search time from the search log in the search log database 3, calculates the relevance between the search terms, and stores it in the related term database 5. Outputs related word data including search terms and relevance. The related word database 5 stores the related word data output by the relevance calculator 4. The new word vector calculation unit 6 acquires related word data relating to the word read from the new word list 1 from the related word database 5 and uses the concept dictionary 2 and the attribute vector information of the words in the temporary storage vector database 7 based on the word data. Is calculated and output to the temporary storage vector database 7 and the new word dictionary 8. The temporary storage vector database 7 is calculated by the new word vector calculation unit 6,
Temporarily save the intermediate results of the new word attribute vector. The new word dictionary 8 stores the final new word attribute vector calculated by the new word vector calculation unit 6.

FIG. 2 is an example of a word list in the new word list 1. A word to be newly added to the concept dictionary 2 is described one word per line.

Table 1 is an example of dictionary data in the concept dictionary 2. In the figure, "telephone", "restaurant", "graph" and the like are words, and "A", "B", "C",... "ZZZ" are attribute names. The value of each attribute is defined for each word, whereby the word is represented as an attribute vector. This is created and given in advance. The data in the temporary storage vector database 7 and the new word dictionary 8 have the same format.

[0020]

[Table 1]

Table 2 is an example of a search log in the search log database 3. A user ID representing a user or a terminal, a word input by the user, and a time at which the word was input are described.

[0022]

[Table 2]

In this example, the time is represented by the number of seconds from a certain point as a starting point. The expression format of the log is an example, and there is no limitation on the format as long as the information includes the user ID, the search time, and the search word.

FIG. 3 is an example of related word data in the related word database 5. Two words and their relevance are described. The larger the value, the higher the degree of relevance between the two words.

Next, the operation of the concept dictionary expanding apparatus will be described with reference to the flowchart shown in FIG.

In step 101, the relevance calculator 4 reads the search log in the search log database 3, calculates the relevance, and generates related word data including the related word and the relevance.
It is stored in the related word database 5. The degree of relevance V _jk between the search words w _j and w _k is determined by, for example, the following equation.

[0027]

(Equation 1)

Here, i represents a user who has used both the search words w _j and w _k ,

[0029]

[Outside 1]

Is given below.

[0031]

(Equation 2)

Here, t _ij is the time when the user i uses the search word w _j .

Further, the larger the value of x, the more the function f (x)
A function that gives a small value.

The relevance between the search terms w _j and w _k is determined by a certain user i
It becomes larger as the time interval using w _j and w _k is smaller, and becomes larger as the number of users using both w _j and w _k is larger.

With the above-described method, the degree of relevance is calculated for all combinations of search words, related word data in a format as shown in FIG. 3 is created, and stored in the related word database 5.

In step 102, the new word vector calculation unit 6
Extracts the word t from the list L in the new word list 1.

In step 103, the new word vector calculation unit 6
Acquires the related word data R of the word t from the related word database 5. The related word data R includes a related word r _i of the word t and a word t
Set of relevance v _i of related words _{r i (r i, v i} ) is the set of. Here, i is the number of the related word.

In step 104, the new word vector calculation unit 6
Is the concept dictionary 2 among the related words ri in the related word data R.
Attribute vector of each related word r _i from concept dictionary 2

[0039]

[Outside 2]

Is obtained.

In step 105, the new word vector calculation unit 6
Is the attribute vector of the word r in the related word data R, which is present in the temporary storage vector database 7, from the temporary storage database 7.

[0042]

[Outside 3]

Is obtained. In the initial state, there is no data in the temporary storage vector database 7.

In step 106, the new word vector calculation unit 6
Assumes that words having a high degree of relevance are deeply semantically related, and the attribute vector data acquired in step 104 or 105

[0045]

[Outside 4]

[0046] and by using the relevance of the value v _i obtained in step 103, attribute vector of the word t

[0047]

[Outside 5]

Is calculated by the following equation.

[0049]

(Equation 3)

Then,

Here, the subscript 1 represents the result of the first calculation of the attribute vector of the word t. In general, the n-th calculation result of the attribute vector of word t is

[0052]

[Outside 6]

Is represented by

In step 107, it is determined whether or not an unprocessed word exists in the new word list L. If there is, the process proceeds to step 102, and if the processing has been completed for all words, the process proceeds to step 108. At this point, the calculation of the attribute vector for each word in the new word list L is performed once,
It has ended.

In step 108, the new word vector calculation unit 6
Determines the termination condition. Examples of the termination condition include whether or not a predetermined number of calculations has been reached, and whether or not the sum D of the difference between the attribute vector of each word and the previous calculation result is smaller than a predetermined threshold. D is defined by the following equation.

[0056]

(Equation 4)

If the termination condition is satisfied, the process proceeds to step 110; otherwise, the process proceeds to step 109.
Proceed to.

In step 109, the new word vector calculation unit 6
Rewrites the temporary storage vector database 7 with the attribute vector of each word calculated this time, and returns to step 102.

In step 110, the new word vector calculation unit 6
Writes the attribute vector of each word calculated this time to the new word dictionary 8.

According to the present embodiment, it is possible to automatically calculate the attribute vector of a new word only by preparing the existing concept dictionary 2 and the search log.

The above-described processing of FIG. 4 is performed as a concept dictionary extension program by using a floppy disk, a CD-RO
M, recorded on a recording medium such as a magneto-optical disk, and can be executed on a computer such as a personal computer.

[0062]

As described above, the present invention obtains a search term, a time at which the search term was used, ID information of a user of the search term or a terminal used from a search log of an Internet search engine or a database. Then, based on these, the degree of relevance indicating the degree of relevance between the search words is calculated, and by using the degree of relevance and the attribute vector of the word defined in the concept dictionary, the attribute vector of the new word is automatically calculated. There is an effect that a concept dictionary corresponding to a new word or a proper noun can be easily constructed.

[Brief description of the drawings]

FIG. 1 is a block diagram of a concept dictionary extension device according to an embodiment of the present invention.

FIG. 2 is a part of an example of a word list in the new word list 1 shown in FIG.

FIG. 3 is a part of an example of related word data generated by a relevance calculator 4 shown in FIG. 1 and stored in a related word database 5;

FIG. 4 is a flowchart showing an operation of the concept dictionary extension device of FIG. 1;

[Description of Signs] 1 New word list 2 Concept dictionary 3 Search log database 4 Relevance calculator 5 Related word database 6 New word vector calculator 7 Temporary storage vector database 8 New word dictionary 101 to 110 steps

Continuation of front page (72) Inventor Hiroto Inagaki 2-3-1 Otemachi, Chiyoda-ku, Tokyo F-term in Nippon Telegraph and Telephone Corporation (reference) 5B075 ND03 NK46 NR12 PR03 PR10 QM07 UU01

Claims (5)

[Claims] A relevance between two search terms is calculated using information on time intervals of two search terms used by a search user obtained from a search log, and the search terms and the relevance are calculated. A first step of creating related word data including the related word database and storing the related word data in a related word database; reading one word from the new word list; obtaining related words related to the word together with the degree of relevance from the related word database; For words that already exist in the concept dictionary, the attribute vector is obtained from the concept dictionary, and for the acquired related words that exist in the temporary storage vector database, the attribute vector is obtained from the temporary storage vector database. Calculating the attribute vector of the word by adding the acquired attribute vectors after weighting them with the degree of association, A second step for all the words in the list, a third step for determining whether a predetermined end condition is satisfied, and an attribute vector of each word calculated this time when the predetermined end condition is not satisfied. A fourth step of rewriting the temporary storage vector database and returning to the second step, and a fifth step of writing the attribute vector of each word calculated this time to the new word dictionary when a predetermined end condition is satisfied. Concept dictionary expansion method to have. 2. The method according to claim 1, wherein the termination condition is whether the number of calculations has reached a predetermined number of calculations. 3. The method according to claim 1, wherein the end condition is whether or not a sum of differences between the attribute vector of each word and a previous calculation result is smaller than a preset threshold. 4. A concept dictionary in which the meaning of a word is expressed in a vector, a search log database that holds a search log that records a user's search, and a related word database that temporarily stores related word data including a search word and its relevance. A new word list that is a list of new words to which a new attribute vector is to be added; a temporary storage vector database that temporarily stores an attribute vector; a new word dictionary that holds the final new word attribute vector calculation result; and the search log. The search log in the database is read, and the degree of relevance between the two search terms is calculated using information on the time interval of each of the two search terms used by the search user obtained from the search log. A relevance calculator that stores related word data including a relevance in the related word database; and, for each word in the new word list, the related word data. From the database, a related word related to the word is acquired together with the degree of relevance, and among the acquired related words that already exist in the concept dictionary, an attribute vector is acquired from the concept dictionary. For those that exist in the temporary storage vector database, obtain the attribute vector from the temporary storage vector database, calculate the attribute vector of the word by adding the acquired attribute vectors after weighting them with the degree of association, It is determined whether a predetermined termination condition is satisfied.If not, the temporary storage vector database is rewritten with the attribute vector of each word calculated this time, and the process returns to the calculation of the attribute vector of the word, and the condition is satisfied. In this case, there is a new word vector calculation unit for writing the attribute vector of each word calculated this time into the new word dictionary. Concept dictionary expansion device. 5. The degree of relevancy between two search terms is calculated using information on time intervals of two search terms used by a search user obtained from a search log, and the search term and its relevance are calculated. A first process of creating related word data including the related word database and storing the related word data in a related word database; reading one word from the new word list; acquiring a related word related to the word together with the degree of relevance from the related word database; For words that already exist in the concept dictionary, the attribute vector is obtained from the concept dictionary, and for the acquired related words that exist in the temporary storage vector database, the attribute vector is obtained from the temporary storage vector database. Calculating the attribute vector of the word by adding the acquired attribute vectors after weighting them with the degree of relevance, A second process to be performed on all the words in, a third process to determine whether a predetermined end condition has been satisfied, and, if the predetermined end condition has not been satisfied, an attribute vector of each word calculated this time is used. A fourth process of rewriting the temporary storage vector database and returning to the second process, and a fifth process of writing the attribute vector of each word calculated this time to a new word dictionary when a predetermined end condition is satisfied, Storage medium storing a concept dictionary extension program to be executed by a user.