TWI290684B - Incremental thesaurus construction method - Google Patents

Abstract

An incremental thesaurus construction method is provided. The incremental thesaurus construction method first collects keywords in a document set, divides each of the documents in the document set into a plurality of logical segments, and perform related terms analysis based on the logical segments.

Description

1290684 11219twf3.doc/006 96-9-4 mei, invention description: [Technical field of invention] The present invention relates to a method for constructing a progressive association vocabulary, and in particular to a co-occurrence progressive vocabulary Construction method. [Prior Art] "Vocabulary mismatch" has long been one of the main reasons for the failure of users in information retrieval systems. The so-called "unmatched vocabulary" problem is that the vocabulary released by the user is not the same as the vocabulary used by the system to index the file. For example, different documents may have inconsistent words such as "notebook", "laptop" or "notebook". If the system directly uses the vocabulary of the original document, the purpose of indexing is to speed up. Query the speed of the comparison. When the user releases the "laptop", the file containing the same meaning but different vocabulary strings may be missed or failed due to incorrect comparison. In the theory of library science, this phenomenon has been noticed, and tools such as “authorization file” and “related vocabulary” have been proposed to solve this problem. Various synonymous words are recorded in the "authority file", so that when quoting or retrieving, words of the same meaning but different forms can be matched and treated as the same vocabulary. For example, the words "notebook" and "laptop" or "Executive Dean" and "Grandfather", or "Dementia" and "Dementia for the Elderly" can be used through the use of authorized files. Treated as the same vocabulary when indexing and retrieving. The "thesaurus" further records more relationships between vocabulary, in addition to the synonym 96-9-4 '1290684 11219twf3.doc/006, there are also antonyms, generalized words, narrow words, related words, etc. To expand or narrow the scope of the subject of the search term. For example, the "notebook computer" and the "handheld computer" are very close to each other and can be regarded as the narrow word of "portable computer". In contrast, "portable computer" can be regarded as the broad word of these two words. Through the extension of the generalized word, the query word "notebook" can be used to find the file containing "handheld computer". "Associative lexicon" lists the relationship between vocabulary and is used to query each other for the purpose of querying words to expand or narrow the scope of the query, or to prompt different linguistic terms for mourning, so that the search can be upgraded from the original string alignment level to I am doing the level of comparison. In order to construct the semantic relationship between such vocabulary, manual analysis and collation are often required. The advantages of artificially making a related lexicon are high accuracy, and the disadvantages are high cost, slow construction, difficult maintenance, and pre-selected vocabulary may be independent of subsequent or other new files. In the past, research on the information retrieval experiment pointed out that the general-purposed lexicon is used in the retrieval of documents in a specific field, and there is a situation in which the search performance cannot be improved. Although the related vocabulary captures the semantic gap between vocabulary, the vocabulary topics covered by the related lexicon may be inferior to the theme of the document, and the purpose of improving the retrieval effectiveness by the associated lexicon is not achieved. An extreme example is the application of the related lexicon of the humanities to the retrieval of engineering science literature, and its retrieval effect is of course difficult to manifest. However, creating a related vocabulary for each of the literature fields is time consuming and labor intensive. Therefore, according to the subject of the literature itself, the method of automatically and immediately generating the related lexicon is the subject of discussion. Automated methods, mostly relying on relevant vocabulary in the document often appear 1290684 11219twf3.doc/006 96-9-4 together to create a related lexicon. The related vocabulary constructed in this way can be called "co-occurrence thesaurus" or "co-occurrence related vocabulary". The relationship between vocabulary and vocabulary in the co-occurring vocabulary is not as accurate as the artificially-produced lexicon, such as generalized words and narrow words, but it has statistical correlation. This correlation often reveals knowledge that is implicit in the document and that is not easily detected, detected, or maintained. Therefore, the method of automatically constructing such a related vocabulary can also be regarded as a method of knowledge discovery from text (or text mining).

Gerard Salton was at Automatic Text Processing: The

Transformation, Analysis, and Retrieval of Information by Computer, Addison-Wesley, 1989—This paper proposes a basic method for constructing co-occurring related lexicons by first calculating the similarity between vocabularies (this similarity is based on vocabulary in each document). Depending on the situation that occurs together, then the vocabulary is clustered according to this similarity. In this method, if the file Di is represented by the file vector DiKdn, di2, ..., dit), where t represents the number of index words, and 屯 represents the weight of the vocabulary Tj in the file Di, the weight of the vocabulary η in each file can also be The vocabulary vectors TjK^j, d2j, ..., dnj) are composed, where η represents the number of all files. Then the similarity between the vocabulary Tj and Tk can be defined as = sim{TjJk) Once the similarity between any two vocabularies is calculated, various categorization techniques can be used to automatically classify vocabulary with high similarity into the same class. However, this method of construction, the calculation of similarity between any two words and the operation of classification is 1290684 11219twf3.doc/006 96-9-4. The amount of memory is too large, and it takes a lot of memory and long-term use when it is applied to a large number of files. calculating time.

Hsinchun Chen and others are also in "Automatic Thesaurus

Generation for an Electronic Community System,,, Journal of the American Society for Information Science, 46(3): 175-193, April 1995 - A related lexicon is constructed based on the statistical characteristics of lexical co-words appearing in documents. They define the weight of the vocabulary Tj in the file Di as: [ \ ^ij - tfij X l°g ~7Γ X Wj \Jj j where n is the total number of files, dfj is the number of words in the vocabulary Tj, and tfij is the vocabulary Tj The word frequency (number of occurrences) of the document Di, and Wj is the length of the vocabulary Tj. For example, "Artificial Intelligence" contains 2 English words, the length of which is defined as 2, and "digital music" contains four Chinese characters, the length of which is defined as 4. In addition, the common weights of the vocabulary Tj and Tk in the file Di are defined as: ^ , Ί f η ) dm = Uijk x log — x where dfjk is the number of pieces in which the two words co-occur in all the files' tfijk represents the two The number of occurrences of the vocabulary in the file Di is taken as a trick of tfijk that occurs less frequently. Chen et al. believe that the symmetric co-occurrence vocabulary calculation method often finds more frequent vocabulary, and this vocabulary does not help the improvement of retrieval performance, so they define The asymmetric vocabulary classification method, that is, the vocabulary Τ 对 vocabulary Tk classification weight is:

Cluster _ Weight(Tj ,Tk) = yn -^1-- x weighting _ factor {Tk) 1290684 11219twf3 .doc/006 96-9-4 and the vocabulary Tk has a categorization weight for the vocabulary Tj

Cluster _ Weight{Tk 9 ) x weighting _ factor(Tj) where the weighting factor is weighting _ factor(Tj) = log dfj /og(8) Using the above categorization formula, they are from 4,714 files (disk space totals 8 MB) ), resulting in 1,708,551 co-occurrence pairs. Equivalent to each word being associated with thousands of words. Since there are too many pairs of related words, it is very easy to cause the burden of the user to browse and browse. Therefore, for each word, the number of related words is limited to at most one. This removes 60% of the word pairs, leaving 709,659 word pairs, and these word pairs are composed of 7829 different words. Due to the large amount of computation of the above formula, in order to generate these word pairs, it takes 9.2 CPU hours on the workstation, and the disk space occupied by all word pairs is 12.3 MB, which is larger than the original file set.

Mark Sanderson et al. at ''Deriving Concept Hierarchies from Text5ff Proceedings of the 22nd International ACM SIGIR Conference on Research and Development in Information Retrieval - SIGIR '99, Aug. 15-19, Berkeley, USA·, 1999, ρρ·206-213 - In this paper, a co-occurring vocabulary with hierarchical structure is constructed in a completely different way. Their purpose is to automatically generate concept hierarchies from the retrieved files to facilitate users to understand the detected files. The approximate content. The method is divided into two steps: the first step is to make a choice of vocabulary to determine which words are to be listed in the mourning class for the user to browse. The choice of vocabulary is mainly from the first few documents of the search results of 1290684 11219twf3 .doc/006 96-9-4, in the paragraphs with better comparison, find the words that often appear together. Another option is to select the vocabulary from the paragraph that is most relevant (ie, the most approximate query condition) in each of the checked documents: the number of words in a checked file divided by the vocabulary Now the number of articles in all files is greater than 0.1, namely: (df in retrieved set ) / (df in the collection) >= 0.1 Using these two options, they use the first 500 results of each query result from the TREC file set. In the document, an average of 2430 words were taken. With these important vocabulary related to the query vocabulary, the second step is to perform lexical association analysis. They use any two words as a comparison of the subsumption relationship: that is, if Tj contains (subsirnies) Tk, the conditional probability ρ(η | Tk) = 1 and P(Tk | Tj) < 1 should To be established. That is, whenever the vocabulary Tk appears in a certain document, the vocabulary Tj also appears, but when η appears in a certain file, Tk does not necessarily appear, and this case says that η contains Tk. In general, generalized words will contain narrow words, but there are no generalized, narrowly defined words, and some will also conform to this inclusion. Although this condition is a direct description of the mathematical relationship of the inclusion relationship, this condition is too strict, and a vocabulary without strict qualifications may also have a semantic inclusion relationship. Sanderson then relaxes the above conditions: if Tj contains Tk, the conditional probability P(Tj | Tk) >= 0.8 and P(Tk | Tj) < 1 should be established. Through this test containing conditions, an average of 200 subsumption pairs are extracted from each query subject of the TREC file set. From these inclusions, the sacred class can be generated through the window tools of some hierarchical menus. In this mourning class, the inclusive is the parent node, and the included one is its 1290684 11219twD.doc/006 96-9-4 child node. After testing, it was found that a total of 67% included interest for further exploration. The so-called "related" refers to the motivation of the subject to be interested in the inclusion of the automatic extraction, so that they want to explore further. However, experiments have shown that 'only random pairing can make the correlation reach 51%. The reason may be that the selected words are all from the same search result. These words are highly likely to belong to the same subject, so even if these words are randomly matched There will also be some degree of relevance. On the whole, this method is only performed at the time of query, the query response time is affected, and the vocabulary of the prompt is limited to the first N pieces of the search result, so it is not a construction method of the global thesaurus. SUMMARY OF THE INVENTION Accordingly, it is an object of the present invention to provide a method for constructing a progressive association vocabulary that has a fast construction speed. Another object of the present invention is to provide a method for constructing a progressive association vocabulary that can retrieve related words only by requiring a single document. In order to achieve the above and other objects, the present invention provides a method for constructing a progressive association vocabulary. The automatic construction method first captures multiple keywords in a file group, and then divides the files in the file group into multiple logical paragraphs. Based on these logical paragraphs, the related words are analyzed to establish a related thesaurus. In a preferred embodiment of the present invention, the step of analyzing the related words based on the logical paragraphs first calculates the associated weight between the two keywords, and accumulates the associated right when the associated weight is greater than or equal to a threshold. 96-9-4 1290684 11219twf3.doc/006 Heavy. Finally, the accumulated correlation weights are multiplied by the normalized reverse file number to obtain the similarity. When a file length difference in a file group exceeds a predetermined predetermined time, a relatively long file can be cut into multiple files. In another preferred embodiment of the present invention, the step of calculating the associated weight is obtained by the following equation: xln(l.72 + ^-) ^gKTipTik) 2xS{Tij(^Tik) S(TV)^S( T^ where Si is the number of logical paragraphs in which document i is divided, S(Tij) is the number of logical paragraphs in which vocabulary j appears in file i, and s(Tij n Tik) is the number of logical paragraphs in which vocabulary j and vocabulary k appear together. In still another preferred embodiment of the present invention, the calculation of the similarity is obtained by the following equation: l〇g^kxn/dfk) log❸ thief, Ό) /=1 where η is the total number of files, dfk Is the number of words appearing in the word _ k, wk is the length of the keyword k, when greater than or equal to a threshold, the order / / (¥ (?;, 7;) production ¥ 〇;, 7;), otherwise ordered / ( _(7;,4))=〇. In the case of display, according to a preferred embodiment of the present invention, the word frequency and similarity associated with the related word can be displayed at the same time when the related word is displayed. It is also possible to determine the position of the display based on the similarity of the related words when displaying the related words. In addition, related words can be sorted according to word frequency, appearance time, and similarity. The invention analyzes the related words by using logical paragraphs in the article, such as sentences, paragraphs, etc., so that the corresponding related words and the correlation between the related words can be obtained by relying on only one document in a short time. 11 T290684 11219twf3.doc/006 96-9-4 When the new document is added, it only needs to analyze all the related words and new files at present, without having to re-analyze the related words. Therefore, the establishment of its associated vocabulary will be much faster than the previous technology. The above and other objects, features, and advantages of the present invention will become more apparent and understood.

[Embodiment] Z Please refer to FIG. 1 , which is a flow chart showing the execution steps of a preferred embodiment of the co-occurring association vocabulary automatic construction method according to the present invention. In this embodiment, keyword scrambling of the file is first performed (S100), and then each file is divided into a plurality of logical paragraphs, and the related words are analyzed based on the logical paragraphs (S102), and finally all the related words are accumulated. Correlation vocabulary (S104), can form a co-occurrence related vocabulary. Since the keyword is a meaningful and representative vocabulary within the document, and is the smallest unit that presents the meaning of the document, various automated document processing must first perform the steps of automatic keyword retrieval. However, since the identification of keywords is subjective judgment and is not conducive to the automatic processing of computers, it is assumed in this method that if a document explores a topic, then certain strings should be mentioned several times. The characteristics that this topic vocabulary may repeat are the rules that computers can follow and the basis on which keywords can be used. Based on this repetitive assumption, the inventor developed a set of automatic keyword extraction methods and obtained the invention patent of the Republic of China (Republic of China invention patent No. 1^789). This method can learn new words, proper nouns, person names, place names, organization names, etc.; does not require the integrity of the file, can be applied to environments with 12 1290684 96-9-4 11219twf3.doc/006 noise, such as OCR files, voice recognition and other documents; no additional resources such as dictionaries, thesaurus, grammar parser, corpus, etc., which require a lot of manpower to build or maintain in advance; the keywords are not limited in length; the speed of capture is fast The memory space consumed is small; the keywords with very low statistical characteristics (only appear twice) can also be captured; the correct rate of sampling is high, about 86%. Since the automatic extraction method of this keyword is based on a simple assumption, documents in various fields, such as engineering, humanities, medicine, patents, etc., as well as melody strings in different languages such as Chinese, English, and even music, etc. This method can be used immediately to capture keywords, key melody, or key segments of a file. As for some documents that do not seem to meet this assumption, such as bibliographic data, or short documents, because the text is streamlined, the vocabulary is unlikely to be repeated, so some techniques can be used to make these documents also meet the repetitive assumptions. One of the techniques is to collect related short files into longer files, so that important words have more chances to repeat, and then perform keyword capture actions. For example, when processing bibliographic data, you can use this technique to capture 350,000 titles using 50,000 titles as a document, then combine the keywords and compare which words come from Which titles can be used to build a related vocabulary for query and bibliographic retrieval. In addition, you can use the background file to use the search function or other reliable methods to find the file related to the short file content you want to process as the background file, and treat the short file and the background file as the same file, making the important words There is an opportunity to repeat, keyword search, and finally leave the words that appear only in short files, 13 1290684 11219twO.doc/006 96-9-4 is the keyword that is automatically extracted from the file. The ability to learn new vocabulary without relying on the thesaurus or the dictionary is very important for certain document types or cross-domain literature. For example, Chinese news files often have new vocabulary. The inventor compared the vocabulary of the news document with a vocabulary containing I2 million Chinese words, and compared it with the inventor's own development method, and found that the method developed by the inventor extracted an average of 33 news files per article. Keywords (words that appear at least twice), of which 11 (33%) of the meaningful words are words that are not recorded in the thesaurus. It is shown that if you use the previously constructed thesaurus to capture the vocabulary of news files, you may miss 1/3 of the important words. This keyword automatic extraction method can also be combined with the lexicon break word method to improve the correct rate of keyword capture. Simple lexicon grammar, for example, the longest word first method is to compare each word in the file in turn, and compare the longest vocabulary with the word in the lexicon. If there is such a vocabulary in the lexicon, Just break the word from the file and proceed to the next word after the word, repeating the above-mentioned thesaurus comparison. If there is no vocabulary in the lexicon, the word is broken into a word (or part of the unknown word), and then proceeds to the next word to continue the comparison. Repeat the comparison until the end of the file. The words that are broken out are all words in the lexicon, and their precision is higher. However, there may be words in the vocabulary that are not in the vocabulary, so the recall rate is low. Since the above keyword automatic retrieval method can extract new vocabulary, the keyword arranging method can be applied to the lexicon breaking vocabulary method, so that the advantages of the two methods complement each other, thereby obtaining high correctness and highness. The result of the recall rate. According to the experiment, after the word is broken by the vocabulary of 120,000 words, 1290684 11219 twG.doc/006 96-9-4 and the keyword automatic extraction method is used to obtain the correctness of the keyword from the original 86. % increased to 96%. After the keywords of the file are extracted according to the foregoing method or any other method, the correlation analysis can be performed. Past analytical methods have evolved based on whether vocabulary often appears together in the same document. However, in the context of full-text files, it is easy to lose accuracy by calculating the total number of documents in a single document. This is because the greater the distance between two words in the file, the closer the possibility The lower it will be. Therefore, the co-occurring units should be reduced to a logical paragraph (1〇gical ParagraPh), or a paragraph, or a sentence, or any unit of split files. In this way, it is possible to count the number of units of any two words in a single file and the number of units that appear together, and then use the similarity calculation formulas such as Dice, Cosine, Jaccard or mutual information to obtain the degree of association between the two words. After calculating the degree of association between the important words of each document, the associated words with the strength of the association exceeding a certain threshold are accumulated, and the associated thesaurus of the entire file database can be completed. In the present invention, the following formula is used to calculate the association weight of two vocabularies in the same text xln(1.72 + 5, /)

2xS(TvnTlk) wgK ip ih) SiT^+SiW where Si is the number of units that the file i is divided into. 'This unit is usually a sentence, a paragraph, or any meaningful segmentation unit. For the convenience of subsequent explanations, this unit is collectively called a sentence. "." S(Tij) represents the number of sentences in which the vocabulary j appears in the file i, and S(Tij ΓΊ Tik) is the number of sentences in which the vocabulary j and the vocabulary k appear together. So the first item in the above formula happens to be the Dice coefficient. But the light is 15 1290684 96-9-4 11219twf3.doc/006

When the Dice coefficient is calculated, the number of associated weights between words in a long file is often shorter than the number of files. Therefore, we compensate the weight of the long file vocabulary with the second term ln(1.72 + Si), so that the lexical association weights of files of different lengths have about the same range. It must be noted that the constant 1.72 used here is not the only one that can be used. In fact, as long as it is greater than or equal to the base of the natural logarithm, it can be used here. After the above calculation, the present invention accumulates the association weight greater than or equal to a certain threshold, and multiplies the normalized inverse document number (IDF) to obtain the final similarity. degree. The exact formula is as follows: l〇g{W^n)dfkl χΣ/(^^.)) where η is the total number of files, dfk is the number of words in the vocabulary k, and wk is the length of the vocabulary k. And when a is greater than or equal to a threshold, let f(a) = a, otherwise let f(a) = 0. In this formula, the number of reverse files (IDF) can be calculated after all the files have been processed, without affecting the action of the single word to retrieve the relevant words. In most cases, this threshold can be applied to most file types as long as 1.0 is selected. For example, a bibliographic database, a bibliographic record can be regarded as a logical sentence 'and a vocabulary appearing in a record, as long as it occurs once, the weight of association between the two vocabulary (2χ1/(1 + 1)χ1η(1·72+1) =1η(2·72)>1·0) can pass this threshold. For a file database with a very uneven file length allocation, it is best to cut a very long file (here, relative to other files in the file database) into several shorter files, and then use the above formula to find Its related words and similarities. 16 !29〇684 11219twD.doc/006 96-9-4 After changing the co-occurring unit from the entire document to a smaller paragraph or sentence, the immediate effect is that a single document can retrieve related words without using Wait until the full file is processed before you can see the related words. This effect has three advantages: 1. Incremental is easy to make related words. For a database that adds files every day or regularly, the processed files do not need to be processed, and only the related words of the newly imported files are retrieved. Add to the entire associated vocabulary. Second, which related words appear in which documents, easy to track records, so it is easy to sort the related words by relevance strength, or by the number of articles (df) appearing by the related words, or by the date of the file. Third, the speed of the related words is fast. Originally, the file is used as a common unit to calculate the related words. It is necessary to compare all the important words appearing in all the files to know whether they often appear together. This requires the calculation of m2n, where m is all important words. The number of η is the number of all files. Often in a document database, m is tens of thousands, or even hundreds of thousands, and η is usually thousands, hundreds of thousands, or even millions. After the change, the calculation amount is about 〇(nK2s), where Κ is the number of vocabularies used in the single document for correlation analysis, and s is the average number of sentences per file. In general, the range of K and s can be set between 10 and 100. Compared with 0 (m2n), 0 (nK2s) is much smaller, so the operation time is much shorter. According to the experiment, from the 330,000 Chinese news files (the volume of text is about 323MB), the desktop Pentium II computer computing takes about 5.6 hours, resulting in 2.5 million related word pairs (all different words) There are 250,000), and each word has 10 corresponding related words. In contrast, Chen et al. spent 9.2 hours on the Sun Spare workstation, and found 1,708,551 related words from 4,714 17 1290684 11219twD.doc/006 96_9-4 English (8MB in total). Since there are too many related words, 60% of the related word pairs are removed from them as the final result. Another experiment by Chen et al. used the 2GB English abstract database to generate 4,000,000 related word pairs from 270,000 words, which cost the supercomputer 24.5 CPU hours. Figure 2 shows the contents of a document and examples of keywords and related words that are automatically extracted by the above method. The parentheses after the keyword show the number of times the vocabulary appears in the text, while the related words show the vocabulary associations with each other in a two-dimensional representation. After the related words of each file are taken out, a related vocabulary can be built up for query prompts. Users can use this to understand the general content of the file and select useful related terms to explore the information recorded in the database. Figure 3 shows the results of an implementation in accordance with another preferred embodiment of the present invention. When the user inputs DoCoMo, the system prompts three words in the leftmost column that are similar to the DoCoMo string, one is the query vocabulary itself, and the other is the word of the vocabulary that is queried (can be regarded as a narrow word) . Each word is marked with the number of occurrences in the parentheses after it. If the vocabulary is a category, then the user can clearly know that there are 62 files belonging to the d〇CoMo category, 19 files belonging to the NTT DoCoMo category, and so on. In this way, users can click on smaller categories to quickly narrow the scope. In addition, these appearances can also help the user to know the number of articles that will appear when using this vocabulary without further inquiry, so that the user can make only one query and then learn several queries. result. The effect is similar to a catalogue, except that the catalogue will vary depending on the vocabulary of the query, and will vary from document to database. It is called "dynamic classification directory". The column labeled "Related Terms" in the middle of Figure 3 is also a prompt word constructed using the associated vocabulary. For each related word of the query term DoCoMo, the following parentheses display two data, the first is the word frequency of the vocabulary (df, that is, the number of articles appearing), and the second is the accumulated correlation strength. In this example, DoCoMo's related words are sorted by the strength of the association from large to small. From this related prompting words, it is known that DoCoMo is related to the Japanese telecommunications industry. Since "three G" and "i-Mode" are special terms in the mobile phone field, it should be precisely related to the mobile phone field of the telecommunications industry. These tips have a considerable degree of abstraction for users who want to explore DoCoMo. If the user wants to know more, click on the appropriate vocabulary, call up the relevant documents, and learn more from the descriptions. The above related words are displayed in one-dimensional space, and we can also apply related words to the environment of two-dimensional space, so that it has another interesting taste. In addition, if we can distinguish the nature of vocabulary, when displaying these vocabulary, we can also actively present more relevant information according to its nature. As shown in Figure 4, after querying MP3, the user found that there are many words related to MP3. Among them, "Central" is like a company name. After clicking the user, you can find out the articles related to MP3 and Central. Its detailed relationship. At this point, the system is also compared to the company in Central in the vendor database, so the company's details are also displayed. Thus, from unstructured documents, the associations between things can be obtained, and these associations are like 19 1290684 11219twf3. doc/006 95.9.4 Some knowledge is waiting to be explored, and some knowledge is in free sentences in the file. Disclosure, some knowledge is supplemented by structured data prepared in advance. Step by step exploration in such a two-dimensional space is like stepping through the map to find the information or knowledge you need. Therefore, such a system can be simply referred to as a "knowledge map." As mentioned earlier, when accumulating the related words of each document, we can know the strength of the association between the vocabulary (ie, similarity), the number of documents and articles that appear, and their date and time. Therefore, when displaying the related words of a query vocabulary, we can sort them according to these messages. However, no matter how the related words are sorted, the corresponding set of related terms are the same, except that the order of the words is different. However, the presentation in different orders will make the user feel different about the related words prompted by the system. One order may be better than the other, that is, the proportion of vocabulary related is higher. Please refer to Figure 5, which shows the related words of the query word "history", which are sorted by "word frequency", "time", "similarity without IDF" and "similarity". "Word frequency" is the number of documents that appear in the prompt word. "Time" means the file in which the prompt word appears together with the query word, with the date of the file being the closest, as the date of the mention of the word, and then sorting the prompt words according to the date from near to far. The intention is to take into account that some "relevant word pairs" appear in recent reports. 'Their strength or word frequency has not yet accumulated enough, but can be ranked in front. This is especially true for documents like news. "The degree of similarity without 1DF" refers to the similarity obtained by the following formula ~ win (M) = (1 W)) 20 l29〇684 112l9twf3.doc/006 96-9-4 where the definition of the function f() Same as the previous definition. This similarity formula, compared with the original one, reduces the log(wk X n/dfk)/l〇g(n) term, and the purpose is to compare the effect of the reverse file frequency (IDF). Finally, "similarity" refers to the similarity obtained by the original similarity calculation formula. In the prompt words in Figure 4, the words that are judged to be more relevant to the query word "historic" are marked with a tick. This example seems to have a better effect of sorting by similarity. According to the experimental results, if six query vocabularies are drawn up, and each query is said to check the front N (here N=50) prompt words and judge the relevance of the query words, then all 30 query words will be The results of the average results will be as shown in Figure 6. When the associated prompt words are sorted according to "similarity", the ratio determined to be relevant is 69%, followed by "no similarity with IDF": 62% 'Secondly, sorted by "time": 59%, most The difference is that the ratio is '48% according to the number of articles. This data once again confirms that "Reverse Document Numbers" (IDF) often plays an important role in information retrieval. Although sorting by time is not good in this case, it may still be important in a document database that emphasizes chronological order. Applications. As far as the correlation ratio is concerned, it can be seen that the sorting by similarity is the best. If the correlation ratio is the same, calculate the accuracy and recall rate, you can further show the difference between the different methods. For example, when the correlation ratio of two sorting methods is 60%, that is, 30 of the first 50 prompt words are judged to be related, 'the 30 related prompt words are arranged at the front and the last one. (ie the top 20 are not relevant), obviously there is still a big difference in effect. Through the TREC search competition commonly used accuracy and recall rate calculation program 21 1290684 11219twf3.doc / 006 96-9-4 trec_eval, can further distinguish the advantages and disadvantages of the two sorting methods. In addition, in the above four sorting methods, in the first 50 prompt words, the related prompt words may not be the same. Through the accuracy rate of the trec_eval and the recall rate calculation, all the related prompt words found by the four methods are collectively collected to calculate the recall. Rate, you can also roughly know that the method can often find related prompt words that other methods can not find, or related prompt words found by other methods, often a vocabulary that can be found by a certain method. Figure 7 shows the trec_eval output when the query word "history" is sorted by similarity. It can be seen from this table that the more vocabulary that is judged to be related, the more concentrated it is, the higher the average accuracy. If you find as many relevant prompts, but don't focus on the front, the average accuracy will be lower. The final average accuracy rate is to average the average accuracy of all 30 query words, and the ranking according to "similarity" is still the best way. The average accuracy is 0.5284, followed by "no IDF." Sort of similarity, average accuracy: 0.4346, followed by "time": 0.4028, and finally by number of articles: 0.3020 In summary, the method for constructing the progressive association vocabulary proposed by the present invention, Its construction speed is fast and the results are good. Compared with the past studies, although the assessment environment is different, on the other hand, the effectiveness of each other is almost the same level. In addition, only a single document is needed to retrieve related words, and progressively drawing related words is easy to make. For databases that add files every day or regularly, the processed files do not need to be processed, just need to retrieve new files. The associated words can be added to the entire associated vocabulary. Furthermore, the ordering of related words is easy. It is easy to sort the related words by the intensity of the association, or by the number of related words that appear in 22 1290684 11219twD.doc/006 95.9.4 (df), or by the date of the file. Although the present invention has been described above in terms of a preferred embodiment, it is not intended to limit the invention, and it is obvious to those skilled in the art that the present invention may be modified and retouched without departing from the spirit and scope of the invention. The scope of the invention is defined by the scope of the appended claims. BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a flow chart showing the steps of a preferred embodiment of the present invention; FIG. 2 is a view showing the contents of a document and the keywords automatically extracted by a preferred embodiment of the present invention. Example of a related word; FIG. 3 is a schematic view showing the result of performing another embodiment of the present invention; FIG. 4 is a view showing another embodiment of the present invention according to a preferred embodiment of the present invention; A preferred embodiment queries the related words of "historic monuments" and sorts the results of the results by using "word frequency", "time", "no similarity of IDF" and "similarity" respectively; Figure 6 shows A schematic diagram of the results obtained in accordance with a preferred embodiment of the present invention; and Figure 7 shows the results of the trec__eval output when the query word "history" is sorted by similarity. BRIEF DESCRIPTION OF THE DRAWINGS: _ sl〇〇~S104: Step 23 is performed in accordance with a preferred embodiment of the present invention

Claims (1)

1290684 11219twfi.doc/〇〇6 Picking up, applying for a patent garden: I. A method for constructing a progressive association vocabulary, comprising: taking a plurality of keywords in a file group; dividing each file in the file group into plural a logical paragraph; calculating an associated weight of the two keywords according to the logical paragraphs; accumulating the associated weight when the related weight is greater than or equal to a threshold; multiplying the accumulated associated weight by the normalized reverse The number of documents is obtained to obtain a similarity to establish the related vocabulary according to the similarity; and to display the related words in the related vocabulary. 2. The construction method of the progressive associative lexicon described in claim 1 of the patent application, wherein the step of calculating the associated weight is obtained by the following equation: ^gKTipTik) x 111(1.72 + 5^) 2x^(7;. ΤΤ;,) ~S(Tv)^S(Tik) where Si is the number of logical paragraphs in which document i is divided, scTij) represents the number of logical paragraphs in which vocabulary j appears in file i, and s(Tij n Tik) is the vocabulary j The number of logical paragraphs that appear with the vocabulary k. 3. The method for constructing a progressive associative vocabulary as described in claim 1, wherein the calculation of the similarity is obtained by the following equation: log ❸ /=1 where η is the total number of files, dfk is the vocabulary The number of articles appearing in k, ~!^ is the length of the keyword k. When μ^(7;·,7;) is greater than or equal to a threshold, let /(_(?;;))= _(7; ·, 7;), otherwise, /0^(7^4))=0. 4. The method for constructing a progressive associative vocabulary according to claim 1, wherein the step of displaying the related words in the associated vocabulary comprises: displaying a word frequency and a similarity associated with the related words. 24 1290684 11219twD.doc/0〇6 96-9-4 5. The method for constructing the progressive association vocabulary as described in claim 1 'the steps of displaying the related words in the associated lexicon include: according to the related words One degree of similarity to determine the position of the display. 6. The method for constructing a progressive associative vocabulary as described in claim 5, wherein the less similarly related term is displayed at a position farther from the center point. 7. The method for constructing a progressive associative vocabulary according to claim 1, wherein the step of displaying the related words in the associated lexicon comprises: sorting the related words according to the word frequency. 8. The method for constructing a progressive associative vocabulary according to claim 1, wherein the step of displaying the related words in the associated vocabulary comprises: sorting the related words according to the time of occurrence. 9. The method for constructing a progressive associative vocabulary according to claim 1, wherein the step of displaying the related words in the associated lexicon comprises: sorting the related words according to one of the similarities without the number of reversed documents. 10. The method for constructing a progressive associative vocabulary as described in claim 1 of the patent application, further comprising: cutting a relatively long file into a plurality of files when the file length difference in the file group exceeds a predetermined threshold. 25 ^ 1290684 11219twf3.doc/006 96-9-4 Abstract: A method for constructing a progressive association lexicon. This automatic construction method first captures multiple keywords in a file group, and then groups the files. The file in it is divided into multiple logical paragraphs, and the related words are analyzed based on these logical paragraphs. The incremental thesaurus construction method first collects keywords in a document set, divides each of the documents in the document set into a plurality of logical segments, and perform related terms analysis based on The logical segments. 指定, designated representative circle: (1) The representative representative of the case is: (1). (2) A brief description of the components of the representative drawings: S100 to S104: The execution steps according to a preferred embodiment of the present invention 捌 If the chemical formula is used in the present case, please disclose the chemical formula which best shows the characteristics of the invention.