JP2003167891A - Word significance calculating method, device, program and recording medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To calculate significance of a retrieving word. <P>SOLUTION: First of all, a category information input part 101 inputs a category being front information for an automatic classifying system to automatically classify the category and a sample document allocated to the category. Next, a retrieval recording extracting part 102 extracts information on 'a retrieving word' and 'a selected category' from retrieval input recording recorded by retrieval service. A retrieval recording totalizing part 103 totalizes an input frequency of the retrieving word with respective categories. A category information learning part 104 extracts a word existing in a document from data acquired by the category information input part 101, and totalizes the word with respective categories, and calculates significance of the word to the respective categories by also using data acquired by the retrieval recording totalizing part 103. A result output part 105 outputs a result determined by the category information learning part 104 to a display. <P>COPYRIGHT: (C)2003,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method and apparatus for calculating the importance of words necessary for automatically classifying a large amount of document data.

[0002]

2. Description of the Related Art In recent years, it has become possible to exchange a large amount of electronic documents through a computer network such as the Internet. Therefore, services that allow individuals to search for information that they need have been realized on the network.

Generally, there are a robot type search service and a directory type search service as text search services on the Internet. "Robot type" is a service that automatically collects documents on the Internet, and when you input a word as a search input, a list of documents containing the word and the full text of the actual document are output services. "Directory type" is prepared in advance. This is a service that assigns documents to a multi-layer tree structure called "directory (category)", and follows the directory to reach the target document list or documents (Fig. 3 shows "directory"). Image of "type"). Naturally, a search service that targets only the documents registered in the directory is conceivable, and a service that searches the directory itself is also conceivable.

In the case of searching the directory itself, as a method of displaying the search result, a method of displaying a category that matches the character string of the word that has been searched and input, or a plurality of words are assigned to the category in advance for search, and the search input is performed. There is a way to display the categories that contain the specified word.

As an example of searching a special directory, a search service for documents having directory information is used.
There is also a device that totals the directory information given to documents satisfying the input conditions and displays the directory information when the search result is displayed (Japanese Patent Laid-Open No. 11-306187, "Method for presenting search result of document with category and its method"apparatus").

On the other hand, conventionally, research has been conducted on a method of automatically classifying document information. As a typical method, a category for classifying like a library is known,
A method for classifying new information into appropriate categories ("Automatic classification of texts using mutual relationships of classification systems" Yamamoto, Masuyama (Toyohashi University of Technology) Naito (NT
T), The Society of Natural Language Studies 1995), or a method in which categories to be classified are unknown and similar documents are collected from a set of documents to create and allocate classification categories ("automatic division by competitive learning neural network" Kikuchi, Matsuoka et al. (Utsunomiya Univ.), IEICE Transactions 1995)
and so on.

The classification method using the dictionary generated by the dictionary learning method which is the subject of the present invention is a method when the category for classification is known in advance. Given to the system a category for pre-classification and a sample document or word that should be in it, the system calculates the importance of the word from that information, and the importance of the word and the importance of the category as a feature of the category. Produces a pair of vectors. Similarly, for a document to be classified, a word and the importance of the document are calculated, and a vector is generated (“Automatic Text Processinng” Gerard Salton, ADD).
ISON-WESLEY pub. 1989).

Feature vector generated for each category i

[0009]

[Outer 1]

[0010]

[0011]

[Equation 1]

(N is the total number of words that appear in the sample document group).

In the classification process, the distance between the feature vector of a category and the feature vector of a document is defined, and the value is used to assign each document to a similar category. Also, if it is determined that the distance is very large, that is, it is not similar to any category, it is not assigned to any category.

[0014]

In the above method for generating a feature vector, the type and importance of a learned word are determined by the sample document from which the word is extracted. In other words, if there are many sample documents that contain many inappropriate words, the learned words will also be inappropriate, and it will be difficult to find a sample document that fits that category, or to create a document manually. is there.

An object of the present invention is to provide a learning method and apparatus for calculating the importance of words necessary for automatically classifying a large amount of document data.

[0016]

A search input record recorded by a search service is used. Search service is JP-A-11
-Assuming a service as proposed in No. 306187. The feature of this search service is that "the categories are displayed in a list according to the entered word" and "the word and the category name do not necessarily match". That is,
For example, for the search input words "broadcast""broadcaststation",
If there are many "TV stations" as the category name given to the document including them, "TV station" is displayed in the search result display, and if there are many "radio stations" as the category name, "radio station" is displayed.

On the search service side, it is recorded as a search record that the search service user (hereinafter, user) has, for example, entered the word "broadcast" and selected the category "TV station"". In the same way, record the information of other "(multiple) search terms" and "selected category".

Using this search record, the number of times each search word has been input for each category is totaled. In the above example, the search term "broadcast" is entered once for the category "TV station". Also, when multiple words are input as the search input, each word is added once (the number of times the search word j is input for the category i is
c _ij ).

Next, the calculation of the feature vector of the category information
Will be described. Assigned to a category (let's say i)
Parsed sample documents that appear and appear in the sample document
Count the number of words (let's call it j) (this is tf_ijTosu
). In general, word importance w _ijIs

[0020]

[Equation 2]

It can be expressed as Here, using the number of inputs c _ij tabulated from the search record, the importance w ′ _ij of the new word is calculated, for example,

[0022]

[Equation 3]

Then, "the importance of the word to the category increases because the number of appearances increases".
An effect such as “(after normalizing the feature vector), the importance of the words in the categories not selected becomes relatively small” can be expected.

As another usage of c _ij , for example, "c
A word j of _ij > th (th is a threshold value) is an indispensable word at the time of classification processing (if word j does not exist in the document, category i
There is a usage method such as "is not classified".

The word importance is calculated before the actual document classification process.

[0026]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Next, embodiments of the present invention will be described with reference to the drawings.

Referring to FIG. 1, a word importance calculating apparatus according to an embodiment of the present invention includes a category information input unit 101, a search record extracting unit 102, a search record totaling unit 103, a category information learning unit 104, and a result output unit 105. And is used in the search service as shown in FIG.

First, the category information input unit 101 inputs a category which is pre-information for automatic classification by the automatic classification system and a sample document assigned to the category.

Next, the search record extracting section 102 extracts the information of "search word" and "selected category" from the search input record recorded by the search service. Search record totaling unit 103
Then, using the information extracted by the search record extraction unit 102,
The number of input search words when a category is selected is totaled for each category. The category information learning unit 104 extracts words existing in the sample document from the data acquired by the category information input unit 101, totals the number of appearances of each word for each category, and acquires the data acquired by the search record totaling unit 103 ( The number of inputs) is also used to calculate the importance of each word for each category. The result output unit 105 outputs the degree of importance obtained by the category information learning unit 104 to a display or the like together with the categories and words.

The processing flow of this embodiment will be described in detail.

It is assumed that the search record is recorded by a search service that is performed separately from the present invention. It is assumed that at least two points of the search word used in the present invention and the selected category are recorded in the search record.

The category information input unit 101 inputs category information and a sample document. For example, the categories are “/ entertainment / TV / TV stations” and “/ entertainment / TV / animation / animated voice actors”, and the sample documents in each category are documents related to voice actors of television stations or anime, or newspaper articles.

In the search record extracting unit 102, information necessary for the present invention (“search word” “selected category”) is obtained from the search record.
To extract. “Animation”, “Anime voice actor”, and “Broadcasting” “TV station” are extracted as “search term” and “selected category” from the example in Table 1. One of the search term and the selected category If there is only a record of, the extraction will not be performed.

[0034]

[Table 1]

The search record totaling unit 103 uses the search records extracted by the search record extracting unit 102 to input the number of words.
_Find c _ij . From the example in Table 1, the category "TV station"
The number of times the word "broadcasting" is increased by one, and the number of times the word "animation" for the category "animated voice actor" is increased by one.

The category information learning unit 104 calculates the degree of importance of each word for each category from the number of input times c _{ij of the} words totaled by the search record totaling unit 103 and the data input by the category information input unit 101.

For example, the number of appearances tf _ij in the sample document of the word “broadcast” for the category “TV station” is 10, the number of inputs c _ij is 5, the total number of categories input by the category information input unit 101 is 100, and “broadcast” If the number of appearance categories of “” is 5, the word importance w ′ _ij is w ′ _ij = (10 + 5) * log (100/5) = 19.52.

It can be seen that the value is larger than that of w _ij = 10 * log (100/5) = 13.01. Similarly, the value of w ′ _ij is calculated for other words and categories, and the result output unit 105 outputs the result “category, word, importance”.

In this embodiment, the output of this embodiment is used for classification by the automatic classification system, the results are used by the search service, and the present invention is relearned by using the information recorded by the search service. However, since the data flow circulates, the information input by the category information input unit 101 may be recorded in the recording medium in advance instead of being input each time the present invention processes. It is possible.

In addition to the functions realized by the dedicated hardware, this embodiment records a program for realizing the function in a computer-readable recording medium, and the program recorded in the recording medium. May be read by a computer system and executed. Computer-readable recording media include floppy disks, magneto-optical disks, CDs
A recording medium such as a ROM, or a storage device such as a hard disk device built into a computer system. Further, the computer-readable recording medium dynamically holds the program for a short time (transmission medium or transmission wave) such as when transmitting the program via the Internet, and serves as a server in that case. It also includes a volatile memory that holds a program for a certain period of time, such as a volatile memory inside a computer system.

[0041]

As described above, according to the present invention, the action for the user's search result is recorded, and the word for correcting the learning contents of the classification dictionary used by the automatic classification system is recorded by using the recorded action. By calculating the importance, the classification accuracy of the classification system can be improved.

[Brief description of drawings]

FIG. 1 is a block diagram showing a schematic configuration of a word importance calculation device according to an embodiment of the present invention.

FIG. 2 is a block diagram showing an overall configuration of a system including a word importance calculation device according to an embodiment of the present invention.

FIG. 3 is an image diagram of directory information of a conventional “directory type” service and document information assigned to it.

[Explanation of symbols]

101 Category information input section 102 search record extraction unit 103 Search Record Aggregation Unit 104 Category Information Learning Department 105 result output section

   ─────────────────────────────────────────────────── ─── Continued front page    (72) Inventor Masaru Miyamoto             2-3-1, Otemachi, Chiyoda-ku, Tokyo             Inside Telegraph and Telephone Corporation (72) Inventor Hiroto Inagaki             2-3-1, Otemachi, Chiyoda-ku, Tokyo             Inside Telegraph and Telephone Corporation F term (reference) 5B075 ND03 NR12 PQ02 PQ38 PR04

Claims (4)

[Claims] 1. A word calculation method for calculating a degree of importance of a word, which is necessary for automatically classifying a large amount of document information into categories, which category is a classification destination, and a sample document previously assigned to the category. The first step of inputting, the second step of acquiring the word that is the search word and the information of the selected category from the search input record, and the input of each word for each category obtained in the second step. A third step of counting the number of times, counting the number of times a word of the category to be classified appears in the sample document, and determining the importance of the word for each category, the total number of the classification destination categories, and the word The number of categories in which the word appears, the number of appearances of the word in the category, the fourth step of calculating from the number of times the word is input, the category, the word, the category Word significance calculation method having a fifth step of outputting the importance of said word for. 2. A word importance degree calculating device for calculating the degree of importance of a word, which is necessary for automatically classifying a large amount of document information into categories, and a category to be a classification destination and a word assigned in advance to the category. A category information input section for inputting a sample document, a search record extraction section for acquiring information on a word in a search word and a selected category from a search input record, and a search record extraction section for each category obtained by the search record extraction section. A search record aggregating unit for aggregating the number of input words, aggregating the number of times that a word of the category to be classified appears in the sample document, and the importance of the word for each category is the total number of the category to be classified. A number of categories in which the word appears, a number of appearances of the word in the category, and a category information learning unit that is calculated from the number of times the word is input, The importance obtained in Gori information learning unit, category, word significance calculating device and a result output unit for outputting with words. 3. A word importance calculation program for causing a computer to execute the word importance calculation method according to claim 1. 4. A recording medium recording a word importance calculation program for causing a computer to execute the word importance calculation method according to claim 1.