JP2001014330A - Storage medium stored with term evaluation program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To conduct a term analysis meeting with an operator's intention among multiple documents. SOLUTION: According to given retrieval conditions, a term statistic arithmetic means 5 specifies one or more documents as objects of term extraction among multiple document data stored in a document storage means 3 and calculates the statistics of terms present in the specified term extraction object documents as term statistics of the retrieval conditions. An evaluated value determining means 7 when supplied with the term statistics of different retrieval conditions from the term statistic arithmetic means 5 determines evaluated values of terms with a numeric vector having dimensions as many as the given retrieval conditions. The term extracted documents are narrowed down under the retrieval conditions and no terms having no relation with the retrieval conditions are extracted. The evaluated values of the respective terms are represented as the numeric vector to analyze the terms.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention relates to a term evaluation apparatus for evaluating terms extracted from document data, and more particularly to a term evaluation method.

[0002]

BACKGROUND OF THE INVENTION Today,
Many pieces of document information are stored as data, and desired information and knowledge are discovered from these pieces of document information (hereinafter, referred to as “data”).
Mining) has been attempted. As one of the methods, it has been proposed that a term is extracted from a document and the mining is performed based on the extracted term.

However, there are various terms in a document, and there is no established method for extracting terms. Therefore, the extracted terms could not be evaluated.

[0004] It is an object of the present invention to solve the above-mentioned problems and to provide a term evaluation apparatus or method capable of evaluating terms extracted from a plurality of documents. It is another object of the present invention to determine a term extraction target document from a plurality of documents.

[0005]

Means for Solving the Problems and Effects of the Invention 1) In a recording medium storing a program according to the present invention, the program causes the computer to execute the following processing. Given, for each search condition,
A term extraction target document determination process and a term statistic calculation process are executed, and a1) a term for specifying one or more documents to be a term extraction target from a plurality of document data based on a given search condition; Extraction target document determination processing, a2) term statistic calculation processing for calculating the statistic of the term present in the specified term extraction target document as the term statistic in the search condition, B) using the plurality of term statistics , A numerical vector having the same dimension as the given number of search conditions is determined as the evaluation value of each term.

As described above, by determining a document to be extracted from the document data for each search condition, terms can be extracted from a document that matches the search condition. In addition, various numerical calculations can be performed by expressing the extracted terms by a numerical vector having the same dimension as the number of search conditions. As a result, terms extracted from a plurality of documents can be evaluated. In addition, it is also possible to analyze the terms by sorting.

[0007] 2) In a recording medium storing the program according to the present invention, terms having predetermined characteristics are extracted based on the evaluation value of each term. Thus, a characteristic term can be extracted based on the vector.

[0008] 3) In the recording medium storing the program according to the present invention, when a plurality of viewpoint profiles are given as the search condition, for each viewpoint profile,
The term extraction target document is specified using a plurality of related words related to the viewpoint profile. Therefore, each term can be evaluated using the vector element corresponding to the viewpoint profile.

[0011] 4) In the recording medium storing the program according to the present invention, the plurality of documents store a creation time, and when one viewpoint profile and an extraction target period are given, each of the extraction target periods is given. Each time, the plurality of search conditions are generated using the one viewpoint profile, and the term extraction target document determination processing is executed. Thus, a time-series change in one viewpoint profile can be analyzed.

[0010] 5) In the recording medium storing the program according to the present invention, the extracted terms are terms in which only specific vector elements have values equal to or greater than a predetermined threshold value. Therefore, terms specific to the operator's particular interests can be extracted.

6) In the recording medium storing the program according to the present invention, the term to be extracted is a term in which all of a plurality of specific vector elements have values equal to or more than a predetermined threshold value. . Therefore, terms common to the operator's specific interests can be extracted.

[0012] 7) In the recording medium storing the program according to the present invention, the extracted terms have specific vector element values that are specifically larger than other terms.
Or specifically small terms. Therefore, terms that are particularly relevant to the specific interest of the operator can be extracted from the extracted terms.

8) In the recording medium storing the program according to the present invention, the plurality of search conditions are search conditions having a meaning in a permutation, and the term to be extracted is such that a value of a certain vector element has a value before and after the value of a certain vector element. The term is larger than a predetermined difference compared to the vector element. Therefore, it is possible to extract terms that change at a certain time in a time series in one viewpoint profile.

9) In a recording medium storing the program according to the present invention, a predetermined character string included in the term and expressing the intention of the document creator is classified and stored for each intention. If the operator selects one of the categories, only the terms in which the specified character string exists are extracted. Therefore, a term including a character string expressing the intention of the document creator can be extracted. This facilitates term extraction that matches the operator's intention.
As a character string expressing the intention of such a document creator,
There are auxiliary verbs, suffixes, classifiers, etc.

10) In the method for evaluating terms constituting document data according to the present invention, A) When a plurality of search conditions are given, for each search condition,
The following a1) term extraction target document determination processing and a2) term statistic calculation processing are performed. A1) Based on given search conditions, one or more documents to be subjected to term extraction are extracted from a plurality of document data. Term extraction target document determination processing that determines the term extraction target document by specifying it.a2) Term statistic calculation processing that calculates the statistic of the term present in the specified term extraction target document as the term statistic in the search condition. B) Using the plurality of term statistics, an evaluation value of each term is determined by a numerical vector having the same dimension as the given number of search conditions.

As described above, by determining the document to be extracted from the document data for each search condition, it is possible to extract terms from a document that matches the search condition. In addition, various numerical calculations can be performed by expressing the extracted terms by a numerical vector having the same dimension as the number of search conditions. As a result, terms extracted from a plurality of documents can be evaluated. In addition, it is also possible to analyze the terms by sorting.

11) In the method of evaluating terms constituting document data according to the present invention, an operator uses a character string attached to a verb or adjective, and the verb or adjective expresses the intention of the document creator. Is given, extract only the terms in which the specified character string exists. Therefore, a term including a character string expressing the intention of the document creator can be extracted. This facilitates term extraction that matches the operator's intention.

12) In the term evaluation apparatus for constructing document data according to the present invention, the term statistic calculation means extracts a document to be subjected to term extraction from a plurality of document data based on a given search condition. One or more specified terms are calculated, and the statistic of the term present in the specified term extraction target document is calculated and output as the term statistic under the search condition. When the term statistic in a plurality of different search conditions is given from the term statistic calculation means, the evaluation means determines an evaluation value of each term by a numerical vector having the same dimension as the given number of search conditions.

As described above, by determining a document to be extracted from the document data for each search condition, terms can be extracted from a document that meets the search condition. In addition, various numerical calculations can be performed by expressing the extracted terms with a numerical vector having the same dimension as the search condition. As a result, terms extracted from a plurality of documents can be evaluated. In addition, it is also possible to analyze the terms by sorting.

[0020]

BEST MODE FOR CARRYING OUT THE INVENTION Description of Functional Block Diagram One embodiment of the present invention will be described with reference to the drawings. The term evaluation apparatus 1 shown in FIG. 1 includes a document storage unit 3, a term statistic calculation unit 5, an evaluation unit 7, an analysis unit 8, and a notification unit 19.

The document storage means 3 stores a plurality of documents. The term statistic calculation means 5 specifies one or two or more documents to be a term extraction target from a plurality of document data stored in the document storage means 3 based on a given search condition, and specifies the specified term extraction target. The statistic of the term present in the document is calculated as the term statistic in the search condition. The evaluation value determining means 7 includes the term statistic calculating means 5
When term statistics for a plurality of different search conditions are given from, the evaluation value of each term is determined by a numerical vector having the same dimension as the given number of search conditions.

The analysis means 8 analyzes terms based on the evaluation value of each term. The notifying means 9 notifies the analysis result.

In the present embodiment, the term statistic calculation means 5 stores a plurality of viewpoint profile correspondence information composed of a viewpoint profile and a plurality of related words related to the viewpoint profile. When a profile is given, the term extraction target document is specified using the plurality of related words.

As described above, by narrowing down the term extraction document by the viewpoint profile, the extraction term is extracted in accordance with the viewpoint profile of interest of the operator. Also, by expressing the evaluation value of each term by a numerical vector, it becomes possible to analyze the terms.

In this embodiment, the display means is used as the notification means, but other notification means may be used.

2. 1. Hardware Configuration (2.1) Outline The hardware configuration of the term evaluation apparatus 1 shown in FIG. 1 will be described. The computer system 40 shown in FIG.
An input device 41, a control device 43, a display device 45, and a storage device 47 are provided. The input device 41 is for inputting various commands. The storage device 47 stores a program for performing a predetermined process based on the given instruction. The control device 43 performs predetermined data processing based on the program stored in the storage device 47.

(2.2) Details FIG. 3 shows a computer system 40 shown in FIG.
1 shows an example of a hardware configuration realized by using.

The computer system 40 includes the CPU 2
3, memory 27, hard disk 26, CRT 30, F
A DD 25, a keyboard 28, a mouse 31, and a bus line 29 are provided. The CPU 23 is a hard disk 2
6 according to the control program stored in the bus line 2
Each component is controlled via 9.

This control program is transmitted via the FDD 25 to the flexible disk 25 storing the program.
a and is installed on the hard disk 26. In addition to the flexible disk, a hard disk may be installed from a computer-readable recording medium in which a program such as a CD-ROM or an IC card is substantially integrated. Furthermore, you may make it download using a communication line.

In the present embodiment, the program is installed from the flexible disk to the hard disk 26 so that the computer indirectly executes the program stored in the flexible disk. However, without being limited to this, the program stored in the flexible disk is stored in the FDD25.
Alternatively, it may be executed directly from. Note that, as a program executable by a computer, not only a program that can be directly executed by simply installing it as it is, but also a program that needs to be once converted into another form or the like (for example,
Decompression of data that has been compressed, etc.), and also includes those that can be executed in combination with other module parts.

The hard disk 26 has a program storage unit 26a, a correspondence table storage unit 26b, a document storage unit 26c, and an evaluation storage unit 26d. The program storage unit 26a stores a program described later. A plurality of viewpoint profiles as shown in FIG. 4 are stored in the correspondence table storage unit 26b. Each viewpoint profile stores a plurality of corresponding keywords. Document storage unit 26
In c, a plurality of documents to be evaluated are stored. In the present embodiment, each document is composed of a creation date and time, a title of the document, and the content of each document. Evaluation storage unit 2
6d stores the evaluation result of each term. Memory 27
In addition, various calculation results and the like are stored.

3. Flow chart Next, the program storage unit 26a of the hard disk 26
Will be described with reference to the flowcharts of FIGS. In the following, a case where each term is evaluated using “good economy” and “bad economy” as viewpoint profiles will be described as an example.

First, the operator inputs a plurality of queries.
In the present embodiment, since the viewpoint profile is used for the query, the selection box 62 shown in FIG.
It is displayed on T30, and the viewpoint profile "good economy"
What is necessary is just to select “recession”. FIG. 8 shows a state in which the viewpoint profile “Boom” is selected after the viewpoint profile “Boom” is selected.

If there is no viewpoint profile in the selection box 62 shown in FIG. 8, a button 64 may be clicked to add a necessary viewpoint profile and a corresponding related keyword. As a result, FIG.
Is added to the correspondence table shown in (1)

In addition, when a related keyword is to be added to or deleted from an existing viewpoint profile, it may be added to or deleted from the related keyword box 63. As described above, by creating a viewpoint profile from a viewpoint of the operator's interest and specifying a term extraction target document, it is possible to extract a document according to the viewpoint desired by the operator.

When determining that a plurality of queries have been input (step S1 in FIG. 5), the CPU 23 initializes a query number k (step S3). Then, an extraction target document is determined for the viewpoint profile “good business” which is the 0th query (step S5). The process of determining a document to be extracted will be described with reference to FIG. In the present embodiment, as described below, a similarity with a viewpoint profile is determined, and a document having a similarity exceeding a predetermined threshold is determined as a term extraction document.

The CPU 23 initializes the document number m (Step S50). It is determined whether the vectorization processing of each document has been completed (step S51). In this case, since the vectorization process has not been completed, the process proceeds to step S53, in which morphological analysis is performed on each document, and words (terms) appearing in the document are extracted. Then, for the extracted terms, important terms are determined using the tfidf method (step S55). The tfidf method is a technique for determining a keyword in information retrieval, and is a tf (term fr) that indicates the frequency of occurrence of the term in a certain document.
idf (inverse) that indicates the rarity of the term and how rare it appears in all documents.
This is a method of weighting terms using document frequency.

Using the important terms extracted in this way, each document is vectorized as a numerical vector having the same dimension as the number of important terms. For example, when there are 100 important terms, a 100-dimensional numerical vector is obtained. In addition,
Some documents do not include the determined important terms. In this case, the value of that dimension of the document is 0
(Sparse). The CPU 23 stores the thus obtained numerical vector in the memory 27.

Next, the CPU 23 converts the viewpoint profile into a vector (step S59). In the present embodiment, all the related keywords of the k-th viewpoint profile are read out from the hard disk 26 and are vectorized as numerical vectors having the same dimension as the number of the related keywords in the viewpoint profile. In the present embodiment, the tfidf value for each related keyword is obtained from the tf of each related keyword in the viewpoint profile and the idf in all documents, and is vectorized as a numerical vector having the same dimension as the number of the related keywords.

Next, the similarity between the viewpoint profile and the m-th document is calculated (step S61). Such similarity can be obtained by calculating the inner product of the numerical value vector obtained in step S57 and the numerical value vector obtained in step S59.

Next, the CPU 23 determines whether or not the similarity calculation has been completed for all the documents (step S63).
If the similarity calculation has not been completed for all the documents, the document number m is incremented (step S65), and the processing from step S61 is performed.

When the similarity calculation is completed for all documents in step S63, a document exceeding a predetermined threshold is determined as an extracted document (step S67).

Next, the CPU 23 calculates the statistics of terms in the document to be extracted. Such calculation processing will be described with reference to FIG.

The CPU 23 initializes the noted document number i and the noted term number j (step S25). The i-th document is set as the document of interest (step S27). Noted document j
The term of interest is set as the term of interest (step S29).

The CPU 23 determines whether the term of interest is a term that first appeared in the document of interest (step S3).
1). If the term appears for the first time in the document of interest, the term occurrence document number bi is incremented (step S33).

The CPU 23 determines that the term of interest is stored in the memory 27.
It is determined whether or not it exists in the extracted term table (not shown) (step S35). If the term already exists, the term appearance frequency ti is incremented (step S3).
7). On the other hand, if it does not exist, it is added to the extracted term table (step S39).

The CPU 23 determines whether or not the term of interest is the last term (step S41). If it is not the last term, the term of interest j is incremented (step S43), and the processing from step S29 is repeated. On the other hand, if it is the last term, it is determined whether term extraction has been completed for all documents (step S45). If there remains any document for which term extraction has not been completed, the target document number i is incremented (step S47),
The term of interest j is initialized (step S49), and the processing from step S27 is repeated. As a result, the number of appearance documents bi and the appearance frequency ti are obtained for each term.

Next, the CPU 23 determines whether or not the processing has been completed for all the queries (step S9 in FIG. 5).
If processing has not been completed for all queries, the CPU
23 increments the query number k (step S
11), and repeat the processing from step S5.

When the processing is completed for all queries, the CP
U23 performs numerical vectorization processing for each query for each term (step S13). In the present embodiment, the vector elements for term vectorization include 1) term appearance frequency ti, 2) term appearance document number bi,
Any of 3) term appearance frequency ti / term appearance document number bi and 4) relevance can be selected.

The degree of relevance is a numerical value indicating the degree of relevance in a document to a related word group forming a viewpoint profile, and differs depending on the algorithm to be evaluated. In the present embodiment, the term appearance document number bi
And the rarity of all documents. Therefore, the relevance increases as the number of term-appearing documents increases, and decreases as the number of term-appearing documents decreases. Also, there is not much in documents other than the term extracted document, and the terms present in the term extracted document have a high degree of relevance. In other words, when the number of term-appearing documents is high, and there are few documents other than the term-extracted document, and there are many terms-extracted documents, the relevance increases.

Note that 1) term appearance frequency ti and 3) term appearance frequency ti / term appearance document number bi are related to the extraction of terms co-occurring with the keywords used in the query set, that is, the interest of the operator. It is thought to be useful for extracting events. Also, 2) the term occurrence document number bi is considered to be useful for extracting main topics related to the operator's interest and the number of occurrences. 4) The degree of relevance seems to be particularly useful for extracting events that are rarely related to the interests of the operator.

FIG. 9 shows an example in which each term is represented by a numerical vector. In this case, the viewpoint profile "boom economy"
Regarding “recession”, the degree of relevance is obtained as a vector element, and any one of the thresholds of 0 or more (all terms) is displayed in the display area 72. The setting of the vector element and the threshold value is performed by the threshold value setting button 7
If 1 is clicked on with a mouse, a threshold setting dialog 74 is displayed, so that the operator can set desired conditions.

FIG. 10 shows a display example when the common terms are extracted by changing the setting conditions. In FIG. 10, the term appearance document number bi as a vector element and the threshold value for all the queries being 1 or more are displayed in the display area 72. As described above, a term in which a plurality of designated vector elements all have a value equal to or greater than or equal to a threshold value designated by the operator is referred to as a common term.

The common term is a term that is extracted in common for all queries. When the query is a viewpoint profile representing the interest of the user as in this embodiment,
It is often a term (topic) common to all interests of the user.

FIG. 11 shows a display example in a case where the setting conditions are changed and the unique terms are extracted. In FIG. 11, the number of term-appearing documents bi as a vector element and a threshold value of one or more for only one query are displayed in the display area 72. As described above, a term in which only a specific vector element has a value equal to or greater than or equal to a threshold value specified by the operator is called a unique term.

The unique term is a term extracted only by the corresponding query. When the query is a viewpoint profile representing the interest of the user as in the present embodiment, the term unique to the specific interest of the user is used. Often it is.

Next, the CPU 23 extracts characteristic terms (step S15 in FIG. 5). The extraction of the feature terms may be performed by the operator giving an extraction criterion. In the present embodiment, terms including a specific character string are extracted. In this case, by setting a filtering condition, feature term extraction is performed as follows.

FIG. 12 shows a display example when only terms including the character string "company" are extracted. In this case, to set the filtering condition, the operator presses the filter button 75
Clicking displays a filtering dialog 76, so that "company" may be set as a character string filter.

As the filtering condition, in addition to the character string filter, a dictionary filter and a pattern filter can be set alone or in combination.

The dictionary filter is a glossary storing one or more terms desired by the operator. In the present embodiment, the dictionary filter stores in advance a corresponding auxiliary verb for each user's intention as shown in FIG.
The operator can select this. Specifically, when the operator selects “desired” as the intention, a plurality of terms “want” are subjected to a filtering process as a character string filter under the or condition. This makes it possible to extract terms including the term “want”, for example, terms such as “want to know” and “want to check”.

As described above, by preliminarily converting the auxiliary verb, which serves as a guideline when performing mining, into a dictionary,
Mining becomes easy.

Further, one or two or more suffixes may be stored in advance in the dictionary filter for each user's intention so that the suffix can be selected. A suffix is a non-independent word that adds meaning to the end of a word or gives certain qualifications to a part of speech. For example, "target",
"Sex" and the like. As a result, "specific" and "innovative"
Terms can be extracted. Then, it is possible to know what word the term relates to in the document.

Further, one or two or more classifiers may be stored in advance in the dictionary filter for each user's intention, and may be selected. For example, "yen",
"MHz" and the like. As a result, a term composed of a number and a classifier can be extracted, so that it is possible to know what word the term relates to in a document.
In particular, in the case of a price, it is possible to set a threshold value of the price, and extract only information on products whose prices are lower or higher.

The dictionary filter may have a hierarchical structure for each user's intention.

In this embodiment, the auxiliary verb corresponding to the user's intention, which is stored in the dictionary filter in advance, is used. However, even if such a dictionary filter is not used, the user can add the auxiliary verb to the character string filter. It may be provided and filtered.

The characteristic terms other than the above are as follows.
It can be extracted based on the following criteria.

1) Singular term: A term in which a certain vector element value is specifically larger or smaller than another term A singular term is a term in which the difference between each vector element value is larger than a predetermined threshold value. Although not as unique as terms, they are often related to a particular interest of the user. For example, three vector element values are “1,1,6”
When the threshold value is set to “5” in “5” or when the threshold value is set to “5” in “1, 3, 6”, it is extracted as a unique term.

2) Difference term: For a query having a meaning in the permutation, a term having a large difference between a certain query and a query before and after the query. The first viewpoint profile “Sedan”, the second viewpoint profile “Sports”, the third viewpoint profile “RV”
In the case of using, it becomes possible to extract new topics that co-occur with the change. Note that the difference term is also effective when limiting an extraction period described later.

As described above, in the present embodiment, a document to be extracted is determined for each viewpoint profile, the evaluation of each term in the viewpoint profile is calculated, and this is repeated for a plurality of viewpoint profiles. The evaluation was made using numerical vectors for multiple viewpoint profiles. By specifying the term extraction target document from a viewpoint desired by the operator, terms are extracted from documents related to the operator's interest, so that terms that are closely related to the operator's interest can be extracted. Also, the term analysis becomes easier because the total number of terms can be narrowed down.

Further, by expressing each extracted term by a vector for a plurality of viewpoint profiles, it is possible to evaluate the terms from another viewpoint from the viewpoint desired by the operator.
Therefore, it is easy to find unknown knowledge.

In the present embodiment, a pattern filter for designating the pattern of the vector element of each term is employed. As a result, for example, the common term, the unique term, the unique term, and the difference term can be extracted.

4. Trend Analysis In the above embodiment, different viewpoint profiles are used as a plurality of search conditions. However, a plurality of search conditions may be used depending on one viewpoint profile and a period for creating each document. For example, as shown in FIG. 14, the viewpoint profile “recession” is selected, and the weighting dialog 8
At 3, weights are set for the related keywords of this viewpoint profile. In this case, all the related keywords are set to weight “1”. Further, the start period is set to five times, the start date is 1997/1/1, and the period is every month.

FIG. 15 shows an execution result under the above conditions. In the display area 77, the term extracted documents are displayed in order of date. In the display area 72, the terms are sorted and displayed according to the relevance of the terms extracted from the term extraction document.
In the present embodiment, rearrangement is performed based on the flowchart shown in FIG.

The CPU 23 first initializes the period number n (step S71 in FIG. 16). The terms are rearranged by the n-th value (step S73). In this case, the images are rearranged in descending order by the value of the 0th period (period of 1/1 to 1/31).

Next, the CPU 23 extracts the terms whose value in the n-th period is “0”, and
Sort by the value of the first period (step S75). That is, in this case, the term whose value in the 0th period is “0” and whose value in the first period is not “0” is the first term.
The values of the period are sorted in descending order.

The CPU 23 determines whether the examination has been completed up to the final period (step S77), and if not, increments the period number n (step S7).
9), repeat the processing from step S75. On the other hand, when the examination is completed up to the final period, the processing ends.

By such rearrangement, it is easy to find a term that appears for the first time in the period of interest. For example, FIG.
In 5, the target period is set to the second period (3/1 to 3 /
31), the term “souvenir and marine products wholesale” does not appear in the previous period 2/1 to 2/28, and the term of interest 3/1
It can be seen that the term first appeared on March 31.

A character string filter or a pattern filter may be used in combination with the term analysis. FIG.
FIG. 18 shows an analysis result when the filtering condition is set. In FIG. 17, the viewpoint profile “stock” is selected, and weighting is set in the weighting dialog 63 for the related keywords of this viewpoint profile. In this case, the weights are all “1”. further,
The attention period is set in the period setting dialog 67. In this case, the start date is 1997/1/1, the period is set every three months, and the number of times is set to four.

FIG. 18 shows an execution result under the above conditions. In the display area 77, the term extracted documents are displayed in order of date. In the display area 72, terms extracted from the term extraction document are displayed. In this case, as the filtering condition, the character string "stock" is set as the character string filter, and the pattern filter "monotonically increasing" is set. Therefore, the character string "stock" is included in the extracted terms. The term where the value every March is monotonically increasing,
Is displayed.

By such a trend analysis, it is possible to extract terms that appear in a document in each period, terms that appear only in a document in a specific period, and terms that newly appear or disappear over time. For example, it is useful for extracting events that gradually increase or decrease.

For such trend analysis, a graph (line graph or the like) may be displayed for each term or to some extent collectively to facilitate visual grasp.

5. Other Embodiments In the above embodiment, one of the plurality of viewpoint profiles has a direction opposite to that of a good economy or a bad economy.
Although a pair of viewpoint profiles is adopted, the present invention is not limited to this, and the user can freely set any of a plurality of viewpoint profiles.

Although the viewpoint profile is given as a query, some terms related to the user's interest may be given directly as keywords. Further, for example, “I want to know the relationship between stock prices and economic trends” Given a natural sentence like this, semantic analysis of this natural sentence,
A query may be generated by setting a keyword. Further, constraints such as a search time range that restricts the query and terms that must be included may be added.

Further, since the document data to be subjected to term extraction and the number thereof are different for each query, the correction may be performed using statistical data. For example, the term appearance frequency may be divided by the number of documents including the term or the number of extracted documents.

Note that the numerical vector data may be classified using a conventional classification method. For example, the following three methods can be adopted.

1) Cluster Analysis If there is no classification standard in advance, a cluster analysis may be performed. After the classification, the typical meaning of each group can be extracted to extract the meaning of the group.
To extract a typical example, for example, the term closest to the center of the group may be extracted.

As a clustering method, a conventionally used method can be adopted. For example, not only a hierarchical cluster analysis but also a non-hierarchical cluster analysis may be performed.

2) Classification Several clusters are prepared in advance, and terms are classified by assigning each numerical vector to the closest cluster.

3) Simple Classification The terms in which the same vector element value is maximum are put together and
One cluster. As many classification clusters as the number of queries are generated, and each classification cluster classifies the term associated with the corresponding query.

With such a classification process, it is possible to discover new knowledge by extracting characteristic terms of a certain classification group and comparing and extracting the terms.

Further, in this embodiment, since the document to be extracted is specified by the viewpoint profile of the user's interest, the term analysis can be performed with relatively high accuracy. Also, compared to the case of extracting from all documents, the total number of terms is reduced, classification can be performed in a short time, and user analysis is facilitated.

In the above-described embodiment, a pair of viewpoint profiles having opposite directions is adopted as the plurality of viewpoint profiles. However, the present invention is not limited to this. For example,
“Politics” and “birth rate” may be used.

Further, in the present embodiment, “good economy” and “bad economy” are used as the profiles representing states as the viewpoint profiles, but “increase in production”, “increase in profit”, etc. may be used as profiles indicating changes. . By using a profile representing such a state or change, data mining becomes easier. The viewpoint profile is not limited to these as long as it is a viewpoint profile desired by the operator, and any profile may be used.

Note that the process of extracting terms from each document and converting them into a vector, which is performed when determining an extracted document, is described below.
If the contents of all the documents can be determined, they can be processed in advance, and may be executed when a new document is added and stored.

In this embodiment, the important terms are extracted from each document in the extracted document determination processing. However, the present invention is not limited to this. For example, not only important terms but also all terms May be extracted, or may be extracted based on a certain extraction criterion.

In this embodiment, the similarity between two numerical vectors is determined by calculating the inner product of the two numerical vectors, but the cosine value of both numerical vectors may be used as the similarity.

Further, in this embodiment, the case of a Japanese document has been described, but the present invention can be similarly applied to other languages, for example, English, Chinese, Korean, and the like.

In the present embodiment, the functions shown in FIG. 1 are realized by using the CPU 23 and software. However, some or all of them may be realized by hardware such as a logic circuit.

As described above, terms present in a document are converted into numerical vectors based on scores regarding a plurality of viewpoint profiles of interest to the operator. As a result, the operator can easily grasp the relationship between his / her own interest and each term as compared with a case where the term is represented by a value depending on the appearance frequency of the term. In addition, the terms converted into numerical vectors can be analyzed using various statistical analysis methods. Further, the meaning of the analysis result becomes easy.

[Brief description of the drawings]

FIG. 1 is a functional block diagram of a term evaluation device 1 according to the present invention.

FIG. 2 is a diagram showing an example of a hardware configuration of the term evaluation device shown in FIG.

FIG. 3 shows a computer system 40 shown in FIG.
FIG. 3 is a diagram illustrating an example of a hardware configuration realized using the H.23.

FIG. 4 is a diagram showing a correspondence between a viewpoint profile and a related keyword.

FIG. 5 is an overall flowchart of a term evaluation process.

FIG. 6 is a detailed flowchart of an extracted document determination process.

FIG. 7 is a detailed flowchart of a statistic determination process.

FIG. 8 is an example of a dialog for determining a query.

FIG. 9 is an example of a display of an extraction term.

FIG. 10 is an example of a display of an extraction term.

FIG. 11 is an example of a display of an extraction term.

FIG. 12 is an example of display of an extraction term.

FIG. 13 shows a data structure of a dictionary filter.

FIG. 14 shows a setting dialog for trend analysis.

FIG. 15 shows the result of a trend analysis.

FIG. 16 is a detailed flowchart of a rearrangement process.

FIG. 17 shows a setting dialog for trend analysis.

FIG. 18 shows the result of a trend analysis.

[Explanation of symbols]

 23: CPU 27: Memory

 ──────────────────────────────────────────────────続 き Continued on the front page (72) Inventor Koji Doi 108-4 Hiraishiwakamatsu, Kawauchi-cho, Tokushima City, Tokushima Prefecture Just System Co., Ltd. (72) Inventor Hiroyuki Mitsuya 108-4, Hiraishi-Wakamatsu, Kawauchi-cho, Tokushima City, Tokushima Prefecture F-term in Justsystem Corporation (reference) 5B075 ND03 PP13 PQ02 PQ32 PQ46 PR06 QM08

Claims (12)

[Claims] 1. A recording medium storing a program for causing a computer including an input device, a control device, and a storage device to function as a term evaluation device, wherein the program causes the computer to execute the following processing: A ) Given multiple search conditions, for each search condition,
A term extraction target document determination process and a term statistic calculation process are executed, and a1) a term for specifying one or more documents to be a term extraction target from a plurality of document data based on a given search condition; Extraction target document determination processing, a2) term statistic calculation processing for calculating the statistic of the term present in the specified term extraction target document as the term statistic in the search condition, B) using the plurality of term statistics And determining a numerical vector having the same dimension as the given search condition number as an evaluation value of each term. 2. A recording medium storing the program according to claim 1, wherein the program is based on an evaluation value of each of the terms.
Extracting terms having predetermined characteristics. 3. A recording medium storing the program according to claim 1, wherein, when a plurality of viewpoint profiles are given as the search condition, a plurality of related words related to the viewpoint profile are provided for each viewpoint profile. The term extraction target document is specified using: 4. The recording medium storing the program according to claim 2, wherein the plurality of documents store a creation time, and when one viewpoint profile and a plurality of extraction target periods are given, each of the plurality of documents is extracted. Generating the plurality of search conditions by using the one viewpoint profile for each period, and executing a term extraction target document determination process. 5. The recording medium storing the program according to claim 2 or 4, wherein the extracted terms are terms in which only specific vector elements have values equal to or greater than or equal to a predetermined threshold value. Characterized by,. 6. A recording medium storing the program according to claim 2 or 4, wherein the extracted terms are such that all of a plurality of specific vector elements have values equal to or greater than or equal to a predetermined threshold value. Term. 7. A recording medium storing the program according to claim 2 or 4, wherein the extracted terms have a specific vector element value that is specifically large or specifically small compared to other terms. Term. 8. The recording medium storing the program according to claim 2 or 4, wherein the plurality of search conditions are search conditions having a meaning in a permutation, and the extracted terms are those having a value of a certain vector element. A term that is larger than a predetermined difference compared to the vector elements before and after the term. 9. A recording medium storing the program according to claim 5, wherein a predetermined character string included in the term and expressing the intention of the document creator is classified and stored for each intention. In addition, if the operator selects one of the categories, only the terms in which the specified character string exists are extracted. 10. A method for evaluating terms constituting document data, comprising: A) when a plurality of search conditions are given,
The following a1) term extraction target document determination processing and a2) term statistic calculation processing are performed. A1) Based on given search conditions, one or more documents to be subjected to term extraction are extracted from a plurality of document data. Term extraction target document determination processing that determines the term extraction target document by specifying it.a2) Term statistic calculation processing that calculates the statistic of the term present in the specified term extraction target document as the term statistic in the search condition. B) using the plurality of term statistics, determining an evaluation value of each term with a numerical vector having the same dimension as the given number of search conditions. 11. The term evaluation method according to claim 10, wherein the operator is designated when the verb or the adjective is given a character string expressing the intention of the document creator, which is attached to the verb or the adjective. Extracting only the terms in which the character string exists. 12. Based on a given search condition, one or more documents from which term extraction is to be performed are specified from a plurality of document data, and the statistics of terms existing in the specified term extraction target document are calculated. Term statistic calculating means for outputting as the term statistic in the search condition, when the term statistic in the plurality of different search conditions is given from the term statistic calculating means, the term statistic has the same dimension as the given number of search conditions An evaluation means for determining an evaluation value of each term by a numerical vector.