JP2006318398A - Vector generation method and device, information classifying method and device, and program, and computer readable storage medium with program stored therein - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To generate a vector for precisely classifying articles wherein individual experiences and thoughts are written, such as a blog. <P>SOLUTION: Texts for study are acquired, and words are extracted from each text, and a word set including stored retrieval words is extracted, and degrees of importance of kernel words in the word set are calculated by using evaluation values by which a higher degree of importance is given for a higher rank of the retrieval words, and a feature vector of each category is calculated and stored, and a list of words constituting the feature vector is stored. Furthermore, a word included in the list of words is extracted from an inputted classification object text being in an unknown category and is converted to a vector, and similarities between this vector and stored feature vectors of respective categories are obtained, and a category having a similarity equal to or higher than a prescribed threshold is given to the classification object text. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a vector generation method and apparatus, an information classification method and apparatus, a program and a computer-readable storage medium storing the program, and information for efficiently classifying the latest document information published on a network. In particular, a vector generation method and apparatus, an information classification method, an apparatus and a program capable of efficiently classifying the latest text by generating a vector including a search term input to a search site. The present invention relates to a computer-readable storage medium storing a program.

  In recent years, with the development of computer networks such as the Internet, a large amount of electronic document information has been transmitted. In addition to the conventional bulletin board, network users can easily send their opinions and opinions on matters of their interest using a blog service. For example, news articles are classified into categories such as “society”, “international”, and “IT”, and the categories of news articles are almost the same in a plurality of news sites. Therefore, it is possible to classify information based on category information given to articles.

  However, in blog services, there are many cases where information senders select categories that classify articles from their respective viewpoints or set new categories by themselves, so articles based on the category information given to articles Is difficult to classify.

  Furthermore, most articles do not have category information such as homepage information.

  Conventionally, in the field of natural language processing and information retrieval technology, there is a technology for classifying a computer by representing digitized text as a vector of words appearing in the text and comparing the direction with a vector representing the characteristics of the category. .

  For example, a word that is important for a specific category is a word that has a large number of occurrences of text in a specific category and does not appear in other categories, and the number of categories in which the specific word appears in the total number of categories in the learning text set. A category frequency coefficient based on the reciprocal of the number of words is calculated, and the importance for each word in the learning text is calculated by multiplying the number of occurrences of words appearing in a specific category by the category frequency coefficient. There is a method of generating a vector having the importance for each word as a vector element and performing category classification (see, for example, Patent Document 1).

  Hereinafter, this method will be described in detail.

  The learning text is a collection of texts with clear categories (including cases where they belong to multiple categories), and is collected from newspaper article databases or from Web pages with clear categories (portal site directory services). . In recent years, articles classified into categories can be collected from newspaper company homepages, which are used for constantly learning new information.

  For example, consider a case where a category related to a movie is to be generated. Collect news articles of movies, and give each article a “movie” as a category and store it. “Director”, “Screenplay”, “Appearance”, and the like are extracted as important words that frequently appear only in the accumulated movie news. These words frequently appear in movie news articles in order to accurately convey information about the released movie. Since movie titles differ from movie to movie, the number of appearances is small compared to words such as “director”, “screenplay”, and “appearance”.

  Also, the number of occurrences in the learning text is small for words that have recently started to attract attention.

  A feature vector characterizing each category is generated from the learning text set.

The feature vector of the category C _j (j = 1 to M) can be expressed by the following equation, for example.

ここで、Ｗ（Ｃ_i，ｗ）は、カテゴリC_ｉにおける単語ｗの重要度であり、カテゴリＣ_iにおける単語ｗの出現文書数に、

Here, W (C _i, w) is the importance of the word w in category C _i, the appearance number of documents of the word w in category C _i,

を乗算した値をとる。つまり、特定のカテゴリにおける出現文書数が多い単語で、他のカテゴリに出現しない単語の重要度が高くなっている。

Takes the value multiplied by. In other words, words that have a large number of appearing documents in a specific category and that do not appear in other categories have a higher importance.

Therefore, uncategorized text that contains many words of high importance in category A is similar to the feature vector of category A (the similarity between the text vector and the category feature vector falls within a predetermined range). A is given.
Patent No. 3488063

  However, when an individual makes an article about his / her impression of a movie on a blog or the like, the article is usually written with the title of the movie, the name of the actor, the name of the main character, and the like. A user who continues to send a critique of a movie may use words such as “director” or “screenplay” to more accurately convey information about the movie to the reader.

  In the above prior art, the article of the user who continues to send critiques of the latter movie is similar to the feature vector of the movie category generated from the learning text, so it is classified into the movie category. Articles made up of movie titles, actor names, protagonist names, etc. are judged to have vectors not similar to the feature vectors of the movie category (similarity falls outside the predetermined range) and classified into movie categories There is a problem that can not be.

  If the movie title is composed of a plurality of words, the movie title is divided into a plurality of words when the words are extracted from the text. Furthermore, when the divided word is a word that is frequently used in another category (for example, “world”, “baseball”), the number of categories in which the word appears increases and the importance of the word decreases. As a result, there is a problem that even if a movie title is included, it cannot be classified into a movie category.

  In addition, uncategorized text that includes words that have recently attracted attention may be given a category if it contains a word that has received attention but does not contain any other high-importance words that characterize the category. There is a problem that you can not.

  The present invention has been made in view of the above points, and a vector generation method and apparatus capable of generating a vector for accurately classifying articles in which personal experiences and impressions such as blogs are written, and information classification It is an object to provide a method, an apparatus, a program, and a computer-readable storage medium storing the program.

  FIG. 1 is a diagram for explaining the principle of the present invention.

The present invention (Claim 1) provides a search word storage means, a learning text storage means, a word for classifying arbitrary text published on a network into one or more categories to which the text belongs. In the vector generation method in the vector generation device having the extraction means and the vector conversion means,
In the word extraction means, a search word for a certain period is extracted from the search word accumulation means in which a search word ranked in a certain period and a ranking set of the search word are accumulated (step 1), and the learning text is accumulated. A word is extracted from the learning text storage means, and a word is extracted from the input classification target text (step 2), a word extraction step;
In the vector conversion means, a conversion step (step 3) is performed in which the word extracted from the learning text and the classification target text in the word extraction step is converted into a vector including the search word.

The present invention (Claim 2) provides a search word storage means, a learning text storage means, a word for classifying an arbitrary text published on a network into one or a plurality of categories to which the text belongs. In the vector generation method in the vector generation apparatus having the extraction means and the importance calculation means,
In the word extracting means, a search word storage means for extracting a search word for a certain period from a search word storage means for storing a set of search words ranked in a certain period and the ranking of the search word, and storing a learning text A word extraction step of extracting a word from the means and extracting a word set including the search word;
Importance for calculating the importance of each word in the word set using an evaluation value that increases the importance of the search word only in the category assigned to the search word and converting it into a vector in the importance calculation means And a calculation step.

The present invention (Claim 3) provides a search word storage means, a learning text storage means, a word for classifying an arbitrary text published on a network into one or a plurality of categories to which the text belongs. In the vector generation method in the vector generation device having the extraction means and the importance degree calculation means,
In the word extraction means, a category is assigned, and a search word for a certain period is extracted from the search word accumulation means in which a set of search words ranked in a certain period and the ranking of the search word is accumulated, and a category is assigned in advance. A word extraction step of extracting a word set from the learning text storage means in which the learning text is stored;
An importance level calculating step of calculating the importance level of each word in each category using an evaluation value that increases the importance level of the search word only in the category assigned to the search word, and converting it into a vector; ,I do.

The present invention (Claim 4) provides a search word storage means, a learning text storage means, a word extraction means, and an importance calculation means for classifying an arbitrary text into one or a plurality of categories to which the text belongs. In the category classification method in the category classification device having the vector conversion means and the category determination means,
In the word extraction means, the search text for a certain period is extracted from the search word storage means for storing the search terms ranked in a certain period and the rankings of the search terms, and the learning text in which the learning text set is stored A word extraction step of extracting a word from the storage means and extracting a set of words including the search term;
In the importance calculation means, the importance calculation step of calculating the importance of the word set using an evaluation value that increases the importance of the search word, and generating a feature vector for each category;
In the vector conversion means, a vector conversion step for converting the classification target text whose input category is unknown to a vector;
The category determining means calculates a similarity between the vector and the feature vector for each category, and performs a category determining step of assigning a category to the classification target text.

The present invention (Claim 5) provides search word storage means, learning text storage means, word extraction means, importance calculation means for classifying an arbitrary text into one or a plurality of categories to which the text belongs. In the category classification method in the category classification device having the vector conversion means and the category determination means,
In the word extraction means, a category is assigned, and a search word for a certain period is extracted from the search word accumulation means in which a set of search words ranked in a certain period and the ranking of the search word is accumulated, and a category is assigned in advance. A word extracting step of extracting a word from the learning text storage means in which the learning text set is stored, and extracting a word set including the search word;
In the importance calculation means, the importance of the written word set is calculated using an evaluation value that increases the importance of the search word only in the category assigned to the written search word, and the feature vector for each category is generated. A degree calculation step;
In the vector conversion means, a vector conversion step for converting the classification target text whose input category is unknown to a vector;
The category determining means calculates a similarity between the vector and the feature vector for each category, and performs a category determining step of assigning a category to the classification target text.

  FIG. 2 is a principle configuration diagram of the present invention.

The present invention (Claim 6) is a vector generation device for classifying arbitrary text published on a network into one or more categories to which the text belongs,
Learning text storage means 30 in which learning text to which a category has been assigned is stored;
A search word storage means 20 in which a search word ranked in a certain period and a set of search word rankings are stored;
A word extraction unit 11 that acquires a learning text from the learning text storage unit 30, extracts a word from each learning text, and extracts a word set including a search word stored in the search word storage unit 20;
The importance level of each word in the word set is calculated using an evaluation value that increases the importance level as the ranking of the search word is higher. And an importance level calculation means 12 for storing a list of words constituting the feature vector in the word list storage means 14.

According to the present invention (Claim 7), in the above Claim 6, the search word ranking of the search word storage means 20 is performed by comparing the search word ranking received last time in the search engine with the search word ranking received this time. Determined based on the required rise,
The importance calculation means 12
Means for calculating the number of word occurrences in a category and the word appearance rate in the category for words included in one category;
Based on the number of words in the category and the word appearance rate in the category, a means for obtaining an importance level that frequently appears in a certain category and does not appear much in other categories and takes a higher value as the degree of increase is higher.

The present invention (Claim 8) is a vector generation device for classifying arbitrary text published on a network into one or more categories to which the text belongs,
Learning text storage means 30 in which learning text to which a category has been assigned is stored;
A search word storage means 20 in which a category is assigned and a set of search words ranked in a certain period and the ranking of the search words is stored;
A word extraction unit 11 that extracts a search term for a certain period from the search word storage unit 20, extracts a word from the learning text storage unit 30, and extracts a word set including the search term;
The importance of the word in each category is calculated using an evaluation value that increases the importance of the search word as the ranking of the search word is higher only in the category assigned to the written search word, and the feature vector for each category is calculated and the feature is calculated. And an importance level calculation means 12 for storing the list of words constituting the feature vector in the word list storage means as well as in the vector storage means.

According to the present invention (claim 9), in claim 8, the search word ranking of the search word storage means 20 is obtained by comparing the search word ranking received last time in the search engine with the search word ranking received this time. Determined based on the degree of increase for each category,
The importance calculation means 12
When the search term is classified into a category, the rank of increase in the category of the search term is taken, and when the search term is not classified into the category, there is an evaluation value that takes the total number of search terms in the category. The degree of importance that takes a higher value is obtained as the degree of increase in the category of the search word in the category is higher, which frequently appears in the category and does not appear much in the other categories.

The present invention (Claim 10) is an information classification apparatus for classifying arbitrary text published on a network into one or more categories to which the text belongs,
Learning text storage means 30 in which learning text to which a category has been assigned is stored;
A search word storage means 20 in which a search word ranked in a certain period and a set of search word rankings are stored;
A word extraction unit 11 that acquires a learning text from the learning text storage unit 30, extracts a word from each learning text, and extracts a word set including a search word stored in the search word storage unit 20;
The importance of each word in the word set is calculated using an evaluation value that increases the importance as the ranking of the search word is higher, the feature vector for each category is calculated and stored in the feature vector storage means 13, Importance calculation means 11 for storing a list of words constituting the feature vector in the word list storage means 14;
A vector conversion unit 41 that extracts words included in the word list storage unit 14 from the input classification target text whose category is unknown, and converts the extracted word into a vector;
A category for determining the similarity between the vector generated by the vector conversion unit 41 and the feature vector of each category stored in the feature vector storage unit 13 and adding a category having a similarity equal to or higher than a predetermined threshold to the classification target text Determining means 42.

The present invention (Claim 11) is an information classification generation apparatus for classifying an arbitrary text published on a network into one or a plurality of categories to which the text belongs.
Learning text storage means 30 in which learning text to which a category has been assigned is stored;
A search word storage means 20 in which a category is assigned and a set of search words ranked in a certain period and the ranking of the search words is stored;
A word extraction unit 11 that acquires a learning text to which a category is assigned from the learning text storage unit 30 and extracts a word set from each learning text;
The importance of each word of the word set in each category is calculated using an evaluation value that increases the importance as the search word ranking is higher only in the category assigned to the search word, and a feature vector for each category is calculated. Importance level calculating means 11 for calculating and storing in the feature vector storage means 13 and storing a list of words constituting the feature vector in the word list storage means 14;
A vector conversion unit 41 that extracts words included in the word list storage unit 14 from the input classification target text whose category is unknown, and converts the extracted word into a vector;
A category for determining the similarity between the vector generated by the vector conversion unit 41 and the feature vector of each category stored in the feature vector storage unit 13 and adding a category having a similarity equal to or higher than a predetermined threshold to the classification target text Determining means 42.

According to the present invention (Claim 12), in Claim 10 or 11, the vector conversion means 41 includes:
Means for calculating the number of occurrences for each word that matches the words of the text to be classified and the words stored in the word list storage unit, and generating a vector based on the number of occurrences;
The category determination means 42
Means for obtaining a distance between a vector generated by the vector conversion means 41 and a feature vector for each category accumulated in the feature vector accumulation means 13 and assigning a category to the classification target text if the distance is greater than a predetermined threshold including.

The present invention (Claim 13) is a program for classifying arbitrary text published on a network into one or more categories to which the text belongs,
A computer having at least learning text storage means in which learning text is stored on the storage means, and search word storage means in which a set of search terms ranked in a certain period and rankings of the search terms are stored A program for executing the respective means according to claims 6 to 12.

The present invention (Claim 14) is a computer-readable storage medium storing a program for classifying any text published on a network into one or more categories to which the text belongs. ,
A computer having at least learning text storage means in which learning text is stored on the storage means, and search word storage means in which a set of search terms ranked in a certain period and rankings of the search terms are stored Is a computer-readable storage medium storing a program that functions as a vector generation device or an information classification device having each means according to claims 6 to 12.

  As described above, in the present invention, a search term ranking in a certain period can be acquired, and a feature vector of a category including the importance of the search term as an element can be generated from the learning text set, so that it is conventionally treated as one word. It is possible to generate a vector for accurately classifying articles including search terms that could not be obtained.

  Further, by increasing the importance of the search word according to the ranking, it is possible to generate a vector for accurately classifying articles in which personal experiences and impressions such as blogs are written.

  Further, it is possible to generate a vector for classifying articles including words that have recently attracted attention (search rankings have rapidly increased) into appropriate categories.

  Furthermore, using the generated vector, it is possible to accurately classify text whose category including the search term is unknown.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[First Embodiment]
FIG. 3 shows the configuration of the information classification apparatus in the first embodiment of the present invention.

  The information classification apparatus shown in FIG. 1 includes a search word acquisition unit 200, a learning text acquisition unit 300, a feature vector generation unit 10, and a classification unit 40.

  The search word acquisition unit 200 includes a search word collection unit 21, an attention search word ranking unit 22, and a search word storage unit 20.

  The search word collection unit 21 acquires the search word rankings aggregated and ranked in a certain period from the collection destination set in the search word collection destination storage unit 23 and sends the search word ranking to the attention search word ranking unit 22. Here, it is assumed that the search term collection destination storage unit 23 stores a machine name and a directory in which search term ranking data (search terms and ranks) is stored. Assume that the search term ranking is obtained by accessing the machine. However, the present invention is not limited to this method, and there is a method in which the URL is stored in the search word collection destination storage unit 23 and the search word ranking is accessed and obtained from the URL of a Web page published on the Internet or the like. . The search term ranking is data in which the search term and the search term are paired, and is acquired in the form of an HTML document, text, data file, or the like. In the search term ranking, the search terms input by the search site user are totaled in a certain period, and the search terms are ranked from the top in descending order from the search terms with the highest number of inputs. The search terms in the top ranking are “2 Channel”, “yahoo”, “Rakuten”, “DoCoMo”, etc., and there are many search terms entered to search for the site, and the number of times of entry is always large, so it occupies the top. The ranking of search terms related to information that has become popular recently is not high, and the number of inputs tends to increase rapidly. Accordingly, the attention search word ranking unit 22 extracts attention keywords whose number of input has been increased rapidly recently.

  The attention search word ranking unit 22 receives the search word ranking from the search word collection unit 21 and compares it with the search word ranking received in the previous time and stored in the memory with respect to the search words higher than the predetermined rank of the search word received this time. Thus, the degree of increase (for example, a value obtained by subtracting the previous search word rank from the current search word rank) is obtained, and ranking is performed from the first place in descending order from the search word having the highest rise degree. Hereinafter, the ranking given in descending order of the degree of increase is referred to as the degree of increase rank. The search terms to which the increase rank is assigned are stored in the search term storage unit 20. Here, the search term for searching the previous ranking is a search term of a predetermined ranking or higher, but this is because the number of people who are interested in the search term with a low ranking this time is considered to be larger. This is to extract search terms that are attracting attention from people.

  For example, when the search word ranking is acquired in the search word collection unit 21 in one month, the attention search word ranking unit 22 receives the latest search word ranking for the search words having a predetermined rank or more in the previous process. The degree of increase is calculated in comparison with the search word ranking of one month ago and stored in the search word storage unit 20.

  In addition, there are sites that calculate and rank search word rises in search sites. When the search word collection unit 21 obtains a search word with an increase degree ranking already assigned, the search word ranking unit 22 is omitted, and the search word accumulation unit 20 is given a rise degree rank. Shall be accumulated.

  The learning text acquisition unit 300 includes a learning text collection unit 31 and a learning text storage unit 30.

  The learning text collection unit 31 has a learning collection destination list 32 in which a collection destination list is stored for each category in a memory, and periodically sends text from the collection destinations set in the learning collection destination list 32 to a network ( (Not shown). The collected text is stored in the learning text storage unit 30 in association with the collection destination and the category.

  Since the learning text (learning data) collected and stored in the learning text storage unit 30 is weighted by the search word in the importance calculation unit 12 described later, the search text ranking obtained by the search word collection unit 21 is used. The text is preferably generated / updated during the period. Therefore, the learning text collection unit 31 performs collection for each search term ranking period. Prior to starting the collection, the learning text storage unit 30 in which the previously collected learning text is stored is moved to another storage location (area), and the text collected this time was collected at the collection destination last time. The text is searched, and if the same text does not exist, it is stored in the learning text storage unit 30 as a new text.

  When the amount of learning text data is small, not only the text collected this time but also the previously collected text may be stored in the learning text storage unit 30 as learning text.

  The learning text storage unit 30 stores the category associated with the text so that it can be easily identified. For example, a method of generating a directory for each category and storing a text file therein, or including a name for identifying the category in the name of the text file is used.

  The feature vector generation unit 10 includes a word extraction unit 11, an importance calculation unit 12, a feature vector storage unit 13, and a word list storage unit 14.

  The word extraction unit 11 acquires all the learning text accumulated for each category accumulated in the learning text accumulation unit 30 after the learning text accumulation unit 30 has completed accumulation, and the word and the word are extracted from each text. The number of occurrences is extracted and recorded in the text recording unit 15 in association with the category.

  FIG. 4 is a flowchart of the process of the word extraction unit in the first embodiment of the present invention.

  First, the word extraction unit 11 reads one category from the learning text storage unit 30 (step 12), and then reads one text (step 14). In this process, the category identification numbers assigned in advance for each category are sequentially read out, and the text is also read out in the order of the text identification numbers assigned in advance.

  The read text is decomposed into morphemes by morphological analysis processing. From the morphemes, nouns, verbs, adjectives and other independent words whose meanings are understood by themselves are extracted as words (step 15). In the following, “word” refers to an independent word, and when an attached word or the like is included, it is simply written as “word”. The search word storage unit 20 is searched based on the extracted word (step 17). It is checked whether or not it matches forward by the search (step 18). If it matches forward, words following the extracted word (including not only independent words but also attached words) are checked in order to check whether or not they match the search word ( Step 19). If it matches the search word, a plurality of words that match the search word are combined into one word (step 20). When there is no forward matching search word (step 18, NO), and when the word following the forward matching does not match the search word (step 19, NO), the number of occurrences of the word extracted in step 16 is increased by one. If there is a word that matches the search word (step 19, YES), the number of occurrences of the search word as one word is increased by one. Further, it is assumed that the word matching the search word has been processed, and it is checked whether there is a subsequent unprocessed word (step 16). If there is an unprocessed word, one word is read out to see if it matches the search word (steps 17 to 20). When all the words in the text have been processed (step 16, NO), the extracted words and the number of appearances of the words are recorded in the text recording unit 15 so that the text can be identified (step 22). Subsequently, it is checked whether there is unprocessed text (step 13). If there is, the word of the unprocessed text is extracted (step 15 to step 22). When all the texts in the category have been processed (step 13, NO), it is checked whether there is an unprocessed category (step 11). If there is, a word is extracted from the text in the unprocessed category (steps 13 to 13). Step 22). When all the categories have been processed (step 11, NO), the processing of the word extraction unit 11 is terminated.

  In the processing of step 21, the number of appearances of words is counted for each text. However, the extracted words may be listed and recorded in order without counting the number of appearances of words.

  In addition, when the previously processed text is included in the learning text, the information of the text recording unit 15 recorded by the previous processing is moved to another storage location (area), and the unprocessed text in Step 13 is moved. In the checking process, it is checked whether there is any text in the previously recorded text recording unit 15. If the text exists, it is determined that the word extraction process has been completed (step 13, NO).

  An example of text processed by the word extraction unit 11 will be described with reference to FIG.

  FIG. 5 shows an example of processing the text “Introducing Winter Sonata Piano Songs” included in the category “Korean Drama”. In step 14 of FIG. "I will introduce" is read. In order to extract words from the text a, morphological analysis processing is performed, and as a result, pairs of words and parts of speech as shown in FIG. Here, verbs and adjectives are not restored to their original form, but are converted back to the original form (even if different inflected forms are used, the number of occurrences is counted as the same word for the same verb or adjective. Is possible). Based on the extracted word, any of nouns, verbs, and adjectives from this word set is examined to see if there is a word corresponding to the search word stored in the search word storage unit 20 (step 17). First, “winter” is extracted as a word, and it is checked whether or not there is a search word in front of “winter” in the search word storage unit 20 (step 18).

  As a result, “winter sonata” and “winter general” are acquired in a forward agreement (step 18, Yes). The word “no” following the word “winter” in the text is checked to see if it matches the search word, and matches “winter sonata”. Next, it is checked whether or not it matches with “sonata” following the word “no”, and in order to match, “winter sonata” is extracted as one word (step 19, Yes, step 20). The number of appearances of the extracted “Winter Sonata” is increased by 1 (step 21), and the processing for the next word is started. Here, the processing has been completed up to the word “sonata” in the text, and processing (step 16 to step 20) is performed in the same manner from the next independent word “piano” to match the search word. When “piano” is in the search word storage unit 20, “piano” is searched for by a forward match. Since the search term “piano” has no word following “piano” (step 19, NO), the number of occurrences of “piano” is increased by one. Similarly, as a result of processing the word, a pair of the word and the number of appearances of the word (FIG. 5c) is associated with the category “Korean drama” and recorded in the text recording unit 15 (step 22).

  The importance calculation unit 12 starts processing after the word extraction unit 11 finishes extracting words from all learning texts. The importance level calculation unit 12 acquires the word and the word appearance count for each category recorded in the text recording unit 15, acquires the rank ranking of the search word from the search word storage unit 20, and the word importance level for each category Is calculated.

  FIG. 6 is a flowchart of the process of the importance calculation unit in the first embodiment of the present invention.

The importance calculation unit 12 processes the words extracted by the word extraction unit 11 and stored in the text recording unit 15 for each category. First, words are read out for one category (step 31), and the number of words in the category and the word appearance rate in the category are calculated for the words included in the category (step 32). The in-category word appearance rate rc (C _i , w) is calculated by the formula (1), and is a ratio of the number of documents in which a certain word appears to the total number of documents in the category, and takes a value from 0 to 1. .

図７にステップ３２で算出したカテゴリ毎の単語出現数とカテゴリ内単語出現率の例を示す。同図において、重要度算出部１２は、カテゴリを識別するための情報１００１，１００２，１００３と各カテゴリ内の単語、単語の出現数、及びカテゴリ内単語出現率を記録したテーブル１１００，１２００、１３００を持つ。なお、本実施の形態では、当該テーブル１１００，１２００，１３００は、重要度算出部１２内のメモリに格納されるものとする。

FIG. 7 shows an example of the number of word appearances for each category and the word appearance rate within the category calculated in step 32. In the figure, an importance level calculation unit 12 is a table 1100, 1200, 1300 in which information 1001, 1002, 1003 for identifying a category and words in each category, the number of occurrences of the word, and the word appearance rate in the category are recorded. have. In this embodiment, it is assumed that the tables 1100, 1200, 1300 are stored in the memory in the importance calculation unit 12.

After the processing of step 32 is performed for all categories (step 33, NO), the importance of the word in each category is obtained by weight (C _i, w) in equation (2) (step 34). The importance level weight (C _i , w) frequently appears in the category C _i , does not appear much in other categories, and takes a higher value as the rank of the search word increases.

ここで、tf（C_i，w）は、単語ｗがカテゴリC_ｉ（ｉ＝１〜Ｍ）に出現する出現数であり、カテゴリ内によく出現する単語ほど大きな値をとり、重要度を高くする働きをする。icf（C_i，w）は、単語wのカテゴリＣ_iにおけるカテゴリ内単語出現率が、全カテゴリにおけるカテゴリ内単語出現率の和に占める割合である。

Here, tf (C _i , w) is the number of appearances of the word w in the category C _i (i = 1 to M), and the larger the word that appears more frequently in the category, the higher the importance. To work. icf (C _i , w) is a ratio of the word appearance rate in the category C _i of the word w to the sum of the word appearance rates in the categories in all categories.

If there are 2 documents in category A, 10 documents in category B, 5 documents in category C, and word w is included in 3 documents in category B and 3 documents in category C. Icf (C _i , w) is calculated as follows.

カテゴリＢとＣでは同じ３文書に単語ｗが出現するが、カテゴリ内単語出現率の高いカテゴリＣのicf（Ｃ，ｗ）が高くなり、結果として単語ｗの重要度が高くなる。

In categories B and C, word w appears in the same three documents, but icf (C, w) of category C having a high word appearance rate in the category increases, and as a result, the importance of word w increases.

In the example of FIG. 7, in the category “Korean drama” of the word “Korea”
icf (Korean drama, Korea)
icf (Korean drama, Korea) = 0.7 / (0.7 + 0.2 + 0.1) = 0.7
Is calculated.

  N is the total number of search terms in the search term storage unit 20, and rank (w) is the rank order of the word w.

を乗算することにより、上昇度順位の高い検索語ほどrank（ｗ）が大きな値をとり、重要度を高くする働きをする。例えば、図５の検索語蓄積部２０において、「冬のソナタ」は順位１５位であり、Ｎ＝１５００の場合、

By multiplying by, the search word having a higher degree of rank rank has a larger rank (w) value and works to increase the importance. For example, in the search word storage unit 20 of FIG. 5, “Winter Sonata” is ranked 15th, and N = 1500,

となり、「冬のソナタ」の重要度を３倍にすることができる。

Thus, the importance of “Winter Sonata” can be tripled.

Since rank (w) has a value only for the search words in the top N rankings of the ranks of increase, it is assumed that words not included in the N cases are processed as rank (w) = N for convenience. The importance of a word that is not included in the search word storage unit 20 is determined only by tf (C _i , w) × icf (C _i , w) in the category and is included in the search word storage unit 20 according to Expression (2). The importance level of a word to be displayed can be increased as the rank increases.

  In addition, the formula (4) can be used as the formula for calculating the importance of the word instead of the formula (2). Also in this case, words that are not included in the N rankings of the search word storage unit 20 are processed as rank (w) = N.

式（４）においては、検索語ランキングＮ件に含まれない単語の重要度は０となり、検索語ランキングＮ件に含まれる単語のみ０以外の値をとる。そのため、検索語ランキングＮ件に含まれる単語のみをベクトルの要素として、ベクトルを生成することになる。検索語Ｎ件のみのベクトルは、学習用テキスト中の自立語全てを用いるベクトルに比べ次元数が小さいため、ベクトル演算を高速化することができる。

In Expression (4), the importance of words not included in N search word rankings is 0, and only words included in N search word rankings take values other than 0. Therefore, a vector is generated with only words included in N search word rankings as vector elements. Since the vector of only N search words has a smaller number of dimensions than a vector using all independent words in the learning text, the vector operation can be speeded up.

  If the size of N is small or if you want to increase the importance of the search term, you may calculate the importance by multiplying by an appropriate coefficient.

の代わりに対数をとらずにN／rank（w）を用いても良い。

Instead of taking the logarithm, N / rank (w) may be used.

  After calculating the importance level of the word for each category, the importance level calculation unit 12 generates a feature vector for each category according to equation (5) (step 35). Feature vector for each category

は、単語抽出部１１で抽出された全ての単語をw_ｉ（ｉ＝１〜ｚ）とし、カテゴリC_ｊ（ｊ＝１〜Ｍ）における単語の重要度weight（C_ｊ，w_ｉ）を要素とするベクトルである。

Are all the words extracted by the word extraction unit 11 as w _i (i = 1 to z), and the importance level weight (C _j , w _i ) of the words in the category C _j (j = 1 to M) is an element. It is a vector.

全てのカテゴリに対して特徴ベクトルを生成し、特徴ベクトル蓄積部１３に蓄積する。

Feature vectors are generated for all categories and stored in the feature vector storage unit 13.

All the words w _i (i = 1 to z) constituting the feature vector are numbered from 1 in the order of the elements of the word, accumulated together with the number of appearances in the word list accumulation unit 14 (step 36), and the importance calculation unit The process of 12 is finished.

  FIG. 8 shows an accumulation example of the feature vector accumulation unit in the first embodiment of the present invention.

  In the figure, the category and the importance of each word are recorded in one line, the category and the importance of the word are separated by a colon “:”, and the importance of the word w1 to the word wz is separated by a comma “,” and enumerated. ing.

FIG. 9 shows an accumulation example of the word list accumulation unit in the first embodiment of the present invention. The information stored in the word list storage unit 14 shown in the figure includes a number 901, the number of appearances 902, and a word 903. The number 901 corresponds to i of the word w _i (i = 1 to z). The numerical value from z to z is taken. The number of appearances 902 is the number of appearances of words in all the learning texts. A word 903 is a word extracted from the text. In the example of FIG. 9, z = 20000, the words are arranged in descending order of the number of appearances, and numbers are assigned from 1. Although the number of appearances 902 is illustrated for explanation, the number of appearances may not be accumulated because the number of appearances is not used in the classification unit 40. The number of the word “Korea” is “100”, the number of appearances is “250”, the number of “Winter Sonata” is “1000”, and the number of appearances is “100”. Words such as “day”, “people”, and “year” that have a large number of appearances have a large number of appearance documents in any category. Therefore, when icf (C _i , w) is calculated, it takes a value close to 0 and the importance of the word is low.

  In addition, a processing example of the importance calculation unit 12 will be described with reference to FIG. In FIG. 7, there are categories of “Korean drama”, “international”, and “gourmet”, and an example of calculating the importance of the words “Korea” and “Winter Sonata” will be described.

  “Korea” appears in categories other than “Korean drama”, “international news”, and “gourmet”, and “Winter Sonata” appears only in the “Korean drama” category. Further, assuming that “Korea” and “Winter Sonata” are stored in the search word storage unit 20 as search words, and the increase rank is the rank shown in FIG. And the importance of “Winter Sonata” can be calculated as follows.

weight (Korean drama, Korea) = 200 × 0.7 / (0.7 + 0.2 + 0.1) × log (1500/150) +1
= 280
weight (Korean drama, winter sonata) = 100 × 0.5 / 0.5 × log (1500/15) + 1 = 300
In tf (C _i , w) and tf (C _i , w) × icf (C _i, w), the importance of `` Korea '' becomes high, but the importance reflects the increasing ranking of search terms By calculating, the importance of “Winter Sonata” can be made higher than “Korea”.

  The classification unit 40 includes a vector conversion unit 41 and a category determination unit 42. The classification unit 40 receives a classification target text whose category is unknown, and assigns a predetermined category (category associated with the learning text) to the classification target text. Note that the text to be classified is input from an input device (not shown) outside the device.

  The classification unit 40 starts the process when the classification target text is input after the importance level calculation unit 12 finishes the process.

  When the classification target text is input, the vector conversion unit 41 generates a vector whose element is the number of occurrences of the word stored in the word list storage unit 14.

  FIG. 10 is a flowchart of the process of the vector conversion unit in the first embodiment of the present invention.

  First, the vector conversion unit 41 decomposes the input classification target text into morphemes by morpheme analysis processing, and extracts independent words as words from the decomposed morphemes (step 41). It is checked whether or not the word subjected to morphological analysis matches forward with the word stored in the word list storage unit 14 (step 43). If there is a forward match, words following the word (including not only independent words but also associated words) are checked in order to check whether they match the forward match word (step 45). If they match, a plurality of words are combined into one word (step 46). If they do not match, it is checked whether there is an unprocessed word (step 42).

  When all the words have been processed (step 42, NO), the number of appearances is calculated for each word (step 47), and the number of appearances is obtained by referring to the words and numbers stored in the word list storage unit 14. Then, a vector is generated by equation (6) (step 48).

ここで、tf(w)は、単語ｗがテキストに出現する出現数である。

Here, tf (w) is the number of appearances of the word w in the text.

  When the processing of the vector conversion unit 41 is completed, the processing of the category determination unit 42 is subsequently started.

  The category determination unit 42 determines a category to be assigned to the classification target text using the classification target text vector and the feature vector for each category stored in the feature vector storage unit 13.

  For example, as a conventional method, there is a method of calculating a distance between vectors by calculating a cosine between two vectors. The closer the cosine value is to 1, the closer the distance between the two vectors. Class vector text

各カテゴリの特徴ベクトル

Feature vector for each category

とすると、２ベクトル間の距離

Then the distance between the two vectors

は式（７）で求めることができる。

Can be obtained by equation (7).

各カテゴリＣ_ｊ（ｊ＝１〜Ｍ）について、２ベクトル間の距離を求め、距離が予め設定しておいた閾値より大きい場合、分類対象テキストに該カテゴリを付与する。距離が閾値より大きいカテゴリが複数ある場合は、複数のカテゴリを分類対象テキストに付与する。

For each category C _j (j = 1 to M), the distance between the two vectors is obtained, and when the distance is greater than a preset threshold, the category is assigned to the classification target text. When there are a plurality of categories whose distance is greater than the threshold, the plurality of categories are assigned to the classification target text.

  FIG. 11 is a processing example of the vector conversion unit in the first embodiment of the present invention.

  Text a "Piano book" Winter Sonata "was purchased" is a text related to the Korean drama "Winter Sonata" and music, and text e "It's already winter. I bought a piano sonata book. "Is text that is only relevant to music. The result of the morphological analysis processing on the text a is b. An independent word is extracted as a word from the result of b (step 43), and the words existing in the word list storage unit 14 and the number of appearances thereof are tabulated (step 47), and the result is c. Similarly, the result of the morphological analysis processing for the text e is f, and independent words are extracted from the result of f as words (step 43), and the words existing in the word list storage unit 14 and the number of appearances are tabulated (steps). 47) The result is g. Different word vectors are generated from c and g.

  However, if “winter sonata” does not exist in the word list storage unit 14, the word generated from the text a and the number of word appearances are d. Since the words d and g and the number of word appearances are the same, the two texts are classified into the same category.

  In the present invention, since “Winter Sonata” exists in the word list storage unit 14, it is possible to generate different vectors from the two texts, and as a result, the texts can be classified into different categories.

  In this way, by expressing the learning text and the unclassified text with the vectors including the search terms higher in the search term ranking, the texts can be classified with high accuracy.

  In addition, even if a new word that has recently become a hot topic and does not have a large number of occurrences, if the search word ranking is high, the importance of the word in the category vector increases. When the text is input to the classification unit 40, the degree of similarity is calculated to be high and can be classified into an appropriate category.

[Second Embodiment]
FIG. 12 shows the configuration of the information classification apparatus in the second embodiment of the present invention. The present embodiment is an example when the search word ranking accumulated in the search word accumulation unit is categorized, and weighting that reflects the search word ranking is performed only in the category in which the search word is classified, Generate category feature vectors.

  In FIG. 12, the processing of the search word acquisition unit 200A and the importance level calculation unit 12A of the feature vector generation unit 10 is different from that of the first embodiment. It should be noted that blocks that perform the same processing as in the first embodiment are denoted by the same reference numerals, description thereof will be omitted, and only portions different in processing will be described.

  The search word acquisition unit 200A includes a search word collection unit 21A, an attention search word ranking unit 22A, and a search word storage unit 20A.

  The search word collection unit 21A is based on the collection destination set in the search word collection destination storage unit 23 for the search word ranking aggregated and ranked for each category in a certain period of time. And the search word ranking is sent to the attention search word ranking unit 22A. The search word ranking is data in which a category, a search word, and a rank of the search word are set, and is acquired in the form of an HTML document, text, a data file, or the like.

  The attention search word ranking unit 22A receives the search word ranking from the search word collection unit 21, and compares the search word ranking received in the previous search term with respect to a search word higher than a predetermined rank of the search word received this time for each category. The degree of increase in the category (for example, a value obtained by subtracting the previous search word rank from the current search word rank) is obtained, and the rank rank is assigned in order from the first place in descending order from the search word having the highest rise degree. The search terms to which the in-category increase rank is assigned for each category are stored in the search term storage unit 20A.

  FIG. 13 shows an accumulation example of the search word accumulation unit in the second embodiment of the present invention. The search word storage unit 20A stores the category 131, the search word 132, and the ascending rank 133 within the search word category in association with each other.

  Further, when the search word collection unit 21A acquires the search word ranking to which the higher rank ranking of the search word is assigned for each category, the processing of the attention search word ranking unit 22A is omitted, and the category is stored in the search word accumulation unit 20A. The search words classified by category to which the increasing degree ranking is assigned are accumulated.

  The importance calculation unit 12A starts processing after the word extraction unit 11 finishes extracting words from all learning texts. The importance level calculation unit 12A acquires the word and the word appearance count for each category recorded in the text recording unit 15, acquires the rank increase degree in the category of the search word from the search word storage unit 20A, and determines the word level for each category. Calculate importance. The process flow is the same as the process flowchart (FIG. 6) of the importance calculation unit 12 in the first embodiment, and the word importance calculation process in the category of step 34 is different.

In the present embodiment, since the weighting is performed only in the category in which the search term is classified, the evaluation value sr (C _i , w) of Expression (8) is used. The evaluation value sr (C _i , w) takes the rank increase degree in the category of the search term when the search term is classified into the category C _i, and the evaluation value sr (C _i , w) in the category when the search term is not classified in the category C _i The total number of search words L is taken. Subtracting m evaluation values sr (C _i , w), the importance of the word is calculated by equation (9).

The importance level weight (C _i , w) appears frequently in the category C _i , does not appear much in other categories, and takes a higher value as the rank ranking of the search terms in the category C _i is higher.

ここで、tf(Ci,w)は、単語ｗがカテゴリＣ_i＝（ｉ＝１〜Ｍ）に出現する出現数であり、カテゴリ内によく出現する単語ほど大きな値をとり、重要度を高くする働きをする。icf(C_i，w）は単語wのカテゴリC_iにおけるカテゴリ内単語出現率が、全カテゴリにおけるカテゴリ内単語出現率の和に占める割合であり、式（３）で求められる値である。

Here, tf (Ci, w) is the number of appearances of the word w in the category C _i = (i = 1 to M), and the larger the word that frequently appears in the category, the higher the importance. To work. icf (C _i , w) is a ratio of the word appearance rate in the category C _i of the word w to the sum of the word appearance rates in the categories in all categories, and is a value obtained by Expression (3).

  It is also possible to calculate the importance of a word using equation (10) instead of equation (9).

式（１０）においては、検索語蓄積部２０Ａに含まれない単語の重要度は全て０となり、検索語蓄積部２０Ａに含まれる単語のみをベクトルの要素として、ベクトルを生成することが可能となる。検索語蓄積部２０Ａに含まれる単語のみのベクトルは、学習用テキスト中の自立語全てを用いるベクトルに比べ次元数が小さいため、ベクトル演算を高速化することができる。

In Expression (10), the importance levels of the words not included in the search word storage unit 20A are all 0, and it becomes possible to generate a vector using only the words included in the search word storage unit 20A as vector elements. . Since the vector of only the words included in the search word storage unit 20A has a smaller number of dimensions than the vector using all the independent words in the learning text, the vector calculation can be speeded up.

  If the size of L is small or if you want to increase the importance of the search term, you may calculate the importance by multiplying by an appropriate coefficient.

の代わりに、対数をとらずにL/sr(C_i,w)を用いてもよい。

Instead of logarithm, L / sr (C _i , w) may be used without taking the logarithm.

After calculating the importance level of the word for each category, the importance level calculation unit 12A generates a feature vector for each category according to the equation (5) (step 35) and stores it in the feature vector storage unit 13. All words w _i (i = 1 to z) constituting the feature vector are numbered from 1 in the order of the word elements, and are accumulated in the word list accumulating unit 14 together with the number of appearances (step 36). The process of 12 is finished.

  As described above, when a category is assigned to the search word, the category (9) and the expression (10) are used as the importance of each word to reflect in which category the search word is focused. Importance can be assigned. That is, when a search word appears in a plurality of categories, in the first embodiment, the importance of the search word increases in all categories in which the search word appears, whereas in the second embodiment, The importance of the search term can be increased only in a specific category assigned to the search term.

  Note that the configuration including the search word acquisition unit 200, the learning text acquisition unit 300, and the feature vector generation unit 10 in the first and second embodiments is a vector generation device, and the operation of the vector generation device is programmed. And can be executed by a computer or distributed on a network. Further, a configuration including the vector generation device and the classification unit 40 can be used as an information classification device, and the operation of the information classification device can be constructed as a program and executed by a computer or distributed on a network.

  Further, the constructed program can be stored in a portable storage medium such as a hard disk, CD-ROM, or flexible disk, and can be installed or distributed on a computer used as a vector generation device and an information classification device. .

  The present invention is not limited to the above-described embodiment, and various modifications and applications can be made within the scope of the claims.

  The present invention can be applied to a technique for classifying document information disclosed on a network.

It is a figure for demonstrating the principle of this invention. It is a principle block diagram of this invention. It is a block diagram of the information classification device in the 1st Embodiment of this invention. It is a flowchart of the process of the word extraction part in the 1st Embodiment of this invention. It is an example of a process of the word extraction part in the 1st Embodiment of this invention. It is a flowchart of the process of the importance calculation part in the 1st Embodiment of this invention. It is a figure explaining the process of the importance calculation part in the 1st Embodiment of this invention. It is an example of accumulation | storage of the feature vector storage part in the 1st Embodiment of this invention. It is an example of accumulation | storage of the word list storage part in the 1st Embodiment of this invention. It is a flowchart of the process of the vector conversion part in the 1st Embodiment of this invention. It is an example of a process of the vector conversion part in the 1st Embodiment of this invention. It is a block diagram of the information classification device in the 2nd Embodiment of this invention. It is an example of accumulation | storage of the search word storage part in the 2nd Embodiment of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Feature vector generation part 11 Word extraction means, Word extraction part 12, 12A Importance calculation means, Importance calculation part 13 Feature vector storage means, Feature vector storage part 14 Word list storage means, Word list storage part 15 Text storage part 20 Search word storage means, search word storage section 20A Search word storage sections 21, 21A Search word collection sections 22, 22A Attention search word ranking section 30 Learning text storage means, learning text storage section 31 Learning text collection section 32 For learning Collection destination list 40 Classification unit 41 Vector conversion unit, vector conversion unit 42 Category determination unit, category determination unit 131 Category 132 Search term 133 In-category rise degree 200, 200A Search term acquisition unit 300 Learning text acquisition unit 801 Categories and each Column 901 indicating the importance of the word number 902 number of occurrences 903 Word 1001, 1002, 1003 Category name 1100, 1200, 1300 Table

Claims (14)

Search word storage means, learning text storage means, word extraction means, and vector conversion means for classifying arbitrary text published on the network into one or more categories to which the text belongs In a vector generation method in a vector generation device having
In the word extraction means, the search words for a certain period are extracted from the search word accumulation means in which a set of search words ranked in a certain period and rankings of the search words are accumulated, and the learning text is accumulated. A word extraction step of extracting a word from the text storage means and extracting a word from the inputted classification target text;
In the vector conversion means, a conversion step of converting the word extracted from the learning text and the classification target text into the vector including the search word in the word extraction step;
A vector generation method characterized by: Search word storage means, learning text storage means, word extraction means, and importance calculation means for classifying arbitrary text published on the network into one or more categories to which the text belongs; In a vector generation method in a vector generation device having
In the word extraction means, the search words for a certain period are extracted from the search word accumulation means in which pairs of search words ranked in a certain period and rankings of the search words are accumulated, and the learning text is stored. A word extraction step of extracting words from the text storage means and extracting a word set including the search terms;
In the importance calculation means, an importance calculation step of calculating the importance of each word of the word set using an evaluation value that increases the importance of a word having a higher search word ranking, and converting it into a vector;
A vector generation method characterized by: Search term storage means, learning text storage means, word extraction means, and importance calculation means for classifying arbitrary text published on the network into one or more categories to which the text belongs In a vector generation method in a vector generation device having
The word extraction means extracts a search term for a certain period from the search word storage means in which a category is assigned and a search word ranked in a certain period and a set of rankings of the search word are accumulated, and a category is assigned in advance. A word extraction step of extracting a word set from the learning text storage means in which the learning text is stored;
In the importance calculation means, the importance of the word in each category is calculated using an evaluation value that increases the importance of the search word only in the category assigned to the search word, and is converted into a vector A calculation step;
A vector generation method characterized by: Search word storage means, learning text storage means, word extraction means, importance calculation means, vector conversion means, category determination means for classifying arbitrary text into one or more categories to which the text belongs In the category classification method in the category classification device having
In the word extraction means, a search word for a certain period is extracted from the search word storage means in which a search word ranked in a certain period and a set of rankings of the search word are accumulated, and the learning text set is stored in the word extraction means A word extraction step of extracting words from the learning text storage means and extracting a word set including the search terms;
In the importance calculation means, an importance calculation step of calculating the importance of the word set using an evaluation value that increases the importance of the search word, and generating a feature vector for each category;
In the vector conversion means, a vector conversion step of converting a classification target text whose input category is unknown into a vector;
In the category determining means, a category determining step of calculating a similarity between the vector and a feature vector for each category, and adding a category to the classification target text;
A category classification method characterized by: Search word storage means, learning text storage means, word extraction means, importance calculation means, vector conversion means, category determination means for classifying arbitrary text into one or more categories to which the text belongs In the category classification method in the category classification device having
The word extraction means extracts a search term for a certain period from the search word storage means in which a category is assigned and a search word ranked in a certain period and a set of rankings of the search word are accumulated, and a category is assigned in advance. A word extracting step of extracting a word from the learning text storage means in which the learning text set stored is extracted, and extracting a word set including the search term;
The importance calculating means calculates the importance of the word set using an evaluation value that increases the importance of the search word only in the category assigned to the search word, and generates a feature vector for each category. Importance calculation step,
In the vector conversion means, a vector conversion step of converting a classification target text whose input category is unknown into a vector;
In the category determination means, a category determination step of calculating a similarity between the vector and a feature vector for each category, and adding a category to the classification target text;
A category classification method characterized by: A vector generation device for classifying arbitrary text published on a network into one or more categories to which the text belongs,
Learning text storage means for storing learning text with a category assigned in advance;
A search term storage means in which a search term ranked in a certain period and a set of rankings of the search terms are stored;
A word extraction unit that obtains a learning text from the learning text storage unit, extracts a word from each learning text, and extracts a word set including a search word stored in the search word storage unit;
The importance of each word of the word set is calculated using an evaluation value that increases the importance as the ranking of the search word is higher, and the feature vector for each category is calculated and stored in the feature vector storage means, Importance calculation means for storing a list of words constituting the feature vector in the word list storage means;
A vector generation device characterized by comprising: The search word ranking of the search word storage means is determined based on the degree of increase obtained by comparing the search word ranking received last time in the search engine with the search word ranking received this time,
The importance calculation means includes:
Means for calculating a word appearance number and a word appearance rate in the category for words included in one category;
Means for determining importance based on the number of words in the category and the word appearance rate in the category, which often appears in a certain category, does not appear much in other categories, and has a higher value as the degree of increase is higher. The vector generation device according to claim 6, further comprising: A vector generation device for classifying arbitrary text published on a network into one or more categories to which the text belongs,
Learning text storage means for storing learning text with a category assigned in advance;
A search term storage means in which a category is assigned and a set of search terms ranked in a certain period and the ranking of the search terms is stored;
A word extraction unit that extracts a search term for a certain period from the search word storage unit, extracts a word from the learning text storage unit, and extracts a word set including the search term;
The importance of the word in each category is calculated using an evaluation value that increases the importance as the ranking of the search word is higher only in the category assigned to the search word, and a feature vector for each category is calculated. Importance level calculating means for storing in the feature vector storage means and storing a list of words constituting the feature vector in the word list storage means;
A vector generation device characterized by comprising: The search word ranking of the search word storage means is determined based on the degree of increase for each category obtained by comparing the search word ranking received last time in the search engine with the search word ranking received this time,
The importance calculation means includes:
When the search term is classified into a category, the rank of increase in the category of the search term is taken, and when the search term is not classified into the category, an evaluation value that takes the total number of search terms in the category is used. The vector generation device according to claim 8, wherein the vector generation device obtains a degree of importance that frequently appears in a certain category and does not appear much in other categories, and takes a higher value as the degree of increase in the category of the search word in the category is higher. An information classification device for classifying arbitrary text published on a network into one or more categories to which the text belongs,
Learning text storage means for storing learning text with a category assigned in advance;
A search term storage means in which a search term ranked in a certain period and a set of rankings of the search terms are stored;
A word extraction unit that obtains a learning text from the learning text storage unit, extracts a word from each learning text, and extracts a word set including a search word stored in the search word storage unit;
The importance of each word of the word set is calculated using an evaluation value that increases the importance as the ranking of the search word is higher, and the feature vector for each category is calculated and stored in the feature vector storage means, Importance calculation means for storing a list of words constituting the feature vector in the word list storage means;
Vector conversion means for extracting words contained in the word list storage means from the classification target text whose unknown category is unknown, and converting the words into vectors,
A category for obtaining a similarity between a vector generated by the vector conversion unit and a feature vector of each category stored in the feature vector storage unit, and adding a category having a similarity equal to or higher than a predetermined threshold to the classification target text A determination means;
An information classification apparatus characterized by comprising: An information classification generation device for classifying arbitrary text published on a network into one or more categories to which the text belongs,
Learning text storage means for storing learning text with a category assigned in advance;
A search term storage means in which a category is assigned and a set of search terms ranked in a certain period and the ranking of the search terms is stored;
A word extraction unit that acquires a learning text to which a category is given from the learning text storage unit, and extracts a word set from each learning text;
The importance of each word of the word set in each category is calculated using an evaluation value that increases the importance as the ranking of the search word is higher only in the category assigned to the search word, and a feature vector for each category And calculating the degree of importance in the feature vector accumulating means, and the degree of importance calculating means for accumulating the word list constituting the feature vector in the word list accumulating means,
Vector conversion means for extracting words contained in the word list storage means from the classification target text whose unknown category is unknown, and converting the words into vectors,
A category for obtaining a similarity between a vector generated by the vector conversion unit and a feature vector of each category stored in the feature vector storage unit, and adding a category having a similarity equal to or higher than a predetermined threshold to the classification target text A determination means;
An information classification apparatus characterized by comprising: The vector conversion means includes
Means for calculating the number of occurrences for each word that matches the word in the classification target text and the word stored in the word list storage unit, and generating a vector based on the number of occurrences;
The category determining means includes
The distance between the vector generated by the vector conversion unit and the feature vector for each category stored in the feature vector storage unit is obtained. If the distance is greater than a predetermined threshold, a category is assigned to the classification target text. The information classification device according to claim 10 or 11, comprising means. A program for classifying arbitrary text published on a network into one or more categories to which the text belongs,
A computer having at least learning text storage means in which learning text is stored on the storage means, and search word storage means in which a set of search terms ranked in a certain period and rankings of the search terms are stored In addition,
13. A program that causes each means according to claim 6 to be executed. A computer-readable storage medium storing a program for classifying arbitrary text published on a network into one or more categories to which the text belongs,
A computer having at least learning text storage means in which learning text is stored on the storage means, and search word storage means in which a set of search terms ranked in a certain period and rankings of the search terms are stored The
A computer-readable storage medium storing a program that functions as a vector generation device or an information classification device having each means according to claim 6.

Images