JP4010711B2 - Storage medium storing term evaluation program - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a term evaluation device for evaluating a term extracted from document data, and more particularly to a term evaluation method.
[0002]
[Background Art and Problems to be Solved by the Invention]
Today, a large amount of document information is stored as data, and attempts have been made to discover desired information and knowledge from these document information (hereinafter referred to as mining). As one of the techniques, it has been proposed to extract a term from a document and perform the mining based on the extracted term.
[0003]
However, there were miscellaneous terms in the document, and there was no established method for term extraction criteria. Therefore, the extracted term could not be evaluated.
[0004]
SUMMARY OF THE INVENTION An object of the present invention is to provide a term evaluation apparatus or method capable of solving the above problems and evaluating terms extracted from a plurality of documents. It is another object of the present invention to determine a term extraction target document from a plurality of documents.
[0005]
[Means for Solving the Problems and Effects of the Invention]
  1) In the recording medium storing the program according to the present invention, the program causes the computer to execute the following processing. A) A plurality of corresponding related keywords are stored for each viewpoint profile. When a plurality of viewpoint profiles are given, the following term extraction target document determination process and term statistic calculation process are executed for each viewpoint profile. A1) a term extraction target document determination process for specifying one or more terms extraction target documents from a plurality of document data using a plurality of related words related to the viewpoint profile; a2) the specified Term statistic calculation processing for calculating the term statistic existing in the term extraction target document and calculating it as the term statistic in the viewpoint profile, and B) each vector for each term existing in the specified term extraction target document The value of the element is represented by the term statistic of each viewpoint profile, and the same number as the number of viewpoint profiles. We asked a number vector with,Displays the term for which the numeric vector was found, along with the numeric vector.
  In this way, by determining the extraction target document from the document data for each viewpoint profile, it is possible to extract terms from a document that matches a plurality of related words related to the viewpoint profile. Further, for each of the terms, a numerical vector having the same dimension as the number of the viewpoint profiles is obtained in which the value of each vector element is represented by the term statistic of each viewpoint profile,By displaying the term for which the numeric vector was obtained, together with the numeric vector,Extracted terms from multiple documents can be evaluated. In addition, it is possible to perform analysis by sorting and classifying the terms.
[0006]
  2) In the recording medium storing the program according to the present invention, based on the numerical vector of each term, a term in which only a specific vector element has a value greater than or less than a predetermined threshold is extracted., Display the extracted terms along with their numeric vectors. Therefore, terms specific to the operator's specific interest are extracted based on the vector.Take out and displaycan do.
[0007]
  3) In the recording medium storing the program according to the present invention, based on the numerical vector of each term, all the specific vector elements that have a value greater than or equal to a predetermined threshold value are extracted. AndDisplay extracted terms along with their numeric vectors. Therefore, terms common to the operator's specific interests are extractedDisplaycan do.
[0008]
  4) In the recording medium storing the program according to the present invention,Based on the numerical vector of each term, a specific vector element value is extracted that is specifically larger or specifically smaller than other terms, and the extracted term is displayed together with the numerical vector.. Therefore, terms that are particularly relevant to the operator's specific interests are extracted from the extracted terms., Display the extracted terms along with their numeric vectorsbe able to.
[0009]
  5) In the recording medium storing the program according to the present invention, the plurality of viewpoint profiles are viewpoint profiles that are meaningful in permutation, and based on the numerical vector of each term, the value of a certain vector element is Extract terms larger than a given difference compared to vector elements, Display the extracted terms along with their numeric vectors. Therefore, terms that change at a certain time in a time series in one viewpoint profile are extracted.Displaycan do.
[0010]
  6) In the recording medium storing the program according to the present invention, a predetermined character string included in the term and expressing the intention of the document creator is classified and stored for each intention. When the operator selects one of the categories, only the terms that have the specified character string are extracted., Display the extracted terms along with their numeric vectors. Therefore, the term including the character string that expresses the intention of the document creator is extracted.Displaycan do. This facilitates term extraction that matches the operator's intention. Examples of character strings that express the intention of the document creator include auxiliary verbs, suffixes, and classifiers.
[0011]
  7) The term evaluation method according to the present invention is a method for evaluating terms constituting document data by a computer, wherein the computer stores a plurality of corresponding keywords for each viewpoint profile. Execute the process. A) When a plurality of viewpoint profiles in which a plurality of related words are defined are given, the following a1) term extraction target document determination process and a2) term statistic calculation process are performed for each viewpoint profile. a1) Term extraction target document determination processing for determining one or more term extraction target documents from a plurality of document data using a plurality of related words related to the viewpoint profile. A2) Term statistic calculation processing for calculating a term statistic existing in the specified term extraction target document as a term statistic in the viewpoint profile. B) For each term, find a numeric vector whose vector element value is represented by the term statistic of each viewpoint profile and has the same dimensions as the number of viewpoint profiles;Displays the term for which the numeric vector was found, along with the numeric vector.
  In this way, by determining the extraction target document from the document data for each viewpoint profile, it is possible to extract terms from a document that matches a plurality of related words related to the viewpoint profile. Further, for each of the terms, a numerical vector having the same dimension as the number of the viewpoint profiles is obtained in which the value of each vector element is represented by the term statistic of each viewpoint profile,By displaying the term for which the numeric vector was obtained, together with the numeric vector,Extracted terms from multiple documents can be evaluated. In addition, it is possible to perform analysis by sorting and classifying the terms.
[0012]
  8) In the method for evaluating the terms constituting the document data according to the present invention, it is used when attached to a verb or adjective and given a character string expressing the intention of the document creator. Extract only terms that containThe extracted terms are displayed together with their numerical vectors.Therefore, it is possible to extract a term including a character string expressing the intention of the document creator. This facilitates term extraction that matches the operator's intention.
[0013]
  9) In the term evaluation device according to the present invention, the term extraction target document specifying means stores a plurality of corresponding keywords for each viewpoint profile, and when a plurality of viewpoint profiles are given, the given viewpoint is given. For each profile, one or more documents for which terms are to be extracted are specified from a plurality of document data using a plurality of corresponding keywords. The term extraction target document specifying means determines the viewpoint profile based on the given viewpoint profile.everyIn addition, one or more documents to be subject to term extraction are specified from a plurality of document data. The term statistic calculating means calculates a term statistic existing in the specified term extraction target document and outputs it as a term statistic in the viewpoint profile. The evaluation means obtains a numerical vector having the same dimension as the number of viewpoint profiles for each term, and the value of each vector element represented by the term statistic of each viewpoint profile,C) About the term which calculated | required the said numerical vector, it displays with the numerical vector.
  In this way, by determining the extraction target document from the document data for each viewpoint profile, it is possible to extract terms from a document that matches a plurality of related words related to the viewpoint profile. Further, for each of the terms, a numerical vector having the same dimension as the number of the viewpoint profiles is obtained in which the value of each vector element is represented by the term statistic of each viewpoint profile,The term for which the numerical vector is obtained is displayed together with the numerical vector.Thus, various numerical operations can be performed by expressing each term as a statistic with the plurality of viewpoint profiles as viewpoints. Thereby, the terms extracted from a plurality of documents can be evaluated. In addition, it is possible to perform analysis by sorting and classifying the terms.
10) In the term evaluation device according to the present invention, the display means extracts a term in which only a specific vector element has a value greater than or less than a predetermined threshold, based on the numerical vector of each term. The extracted terms are displayed along with their numerical vectors. Therefore, it is possible to extract and display a term specific to the operator's specific interest based on the vector.
11) In the term evaluation device according to the present invention, the display means has a term in which all of a plurality of specific vector elements have a value greater than or equal to a predetermined threshold based on the numerical vector of each term. Are extracted and the extracted terms are displayed together with their numerical vectors. Therefore, it is possible to extract and display terms that are common to the operator's specific interest.
12) In the term evaluation device according to the present invention, the display means is a term in which a specific vector element value is specifically larger or specifically smaller than other terms based on the numerical vector of each term. Are extracted and the extracted terms are displayed together with their numerical vectors. Therefore, it is possible to extract a term particularly related to the operator's specific interest from the extracted term, and display the extracted term together with its numerical vector.
13) In the term evaluation device according to the present invention, the plurality of viewpoint profiles are viewpoint profiles having meanings in a permutation, and the display means has a value of a certain vector element based on a numerical vector of each term. A term larger than a predetermined difference is extracted compared to the preceding and following vector elements, and the extracted term is displayed together with its numerical vector. Therefore, it is possible to extract and display terms that change at a certain time in a time series in one viewpoint profile.
14) In the term evaluation device according to the present invention, character strings that are predetermined character strings included in the terms and express the intentions of the document creator are classified and stored for each intention, and the display When any classification is designated by the operator, the means extracts only the term in which the designated character string exists, and displays the extracted term together with its numerical vector. Therefore, it is possible to extract and display a term including a character string expressing the intention of the document creator. This facilitates term extraction that matches the operator's intention. Examples of character strings that express the intention of the document creator include auxiliary verbs, suffixes, and classifiers.
[0020]
DETAILED DESCRIPTION OF THE INVENTION
1. Description of Functional Block Diagram An embodiment of the present invention will be described with reference to the drawings. 1 includes a document storage unit 3, a term statistic calculation unit 5, an evaluation unit 7, an analysis unit 8, and a notification unit.9It has.
[0021]
The document storage unit 3 stores a plurality of documents. The term statistic calculation means 5 specifies one or more documents to be extracted from the plurality of document data stored in the document storage means 3 based on the given search condition, and specifies the specified term extraction target A term statistic existing in the document is calculated as a term statistic in the search condition. When the term statistic in a plurality of different search conditions is given from the term statistic calculation means 5, the evaluation value decision means 7 decides the evaluation value of each term using a numerical vector having the same dimension as the given search condition number. .
[0022]
The analysis means 8 analyzes the term based on the evaluation value of each term. The notification means 9 notifies the analysis result.
[0023]
In this embodiment, the term statistic calculating means 5 stores a plurality of pieces of viewpoint profile correspondence information composed of a viewpoint profile and a plurality of related words related to the viewpoint profile, and the viewpoint profile gives a search condition. Then, the term extraction target document is specified using the plurality of related words.
[0024]
In this way, by extracting the term extraction document with the viewpoint profile, the extraction term is extracted in accordance with the viewpoint profile in which the operator is interested. Also, by representing the evaluation value of each term with a numerical vector, it is possible to analyze the term.
[0025]
In the present embodiment, the display means is employed as the notification means, but other notification means may be employed.
[0026]
2. Hardware configuration
(2.1) Outline
A hardware configuration of the term evaluation device 1 shown in FIG. 1 will be described. The computer system 40 shown in FIG. 2 includes an input device 41, a control device 43, a display device 45, and a storage device 47. The input device 41 is for inputting various commands. The storage device 47 stores a program for performing a predetermined process based on a given command. The control device 43 performs predetermined data processing based on a program stored in the storage device 47.
[0027]
(2.2) Details
FIG. 3 shows an example of a hardware configuration in which the computer system 40 shown in FIG. 2 is realized using a CPU.
[0028]
The computer system 40 includes a CPU 23, a memory 27, a hard disk 26, a CRT 30, an FDD 25, a keyboard 28, a mouse 31, and a bus line 29. The CPU 23 controls each unit via the bus line 29 according to a control program stored in the hard disk 26.
[0029]
This control program is read from the flexible disk 25 a storing the program via the FDD 25 and installed in the hard disk 26. In addition to the flexible disk, it may be installed on a hard disk from a computer-readable recording medium in which a program such as a CD-ROM or an IC card is substantially integrated. Furthermore, it may be downloaded using a communication line.
[0030]
In the present embodiment, the program stored in the flexible disk is indirectly executed by the computer by installing the program from the flexible disk to the hard disk 26. However, the present invention is not limited to this, and the program stored in the flexible disk may be directly executed from the FDD 25. Note that programs that can be executed by a computer are not only those that can be directly executed by simply installing them, but also those that need to be converted to other forms once (for example, those that have been compressed) are decompressed. Etc.), and further executable in combination with other module parts.
[0031]
The hard disk 26 includes a program storage unit 26a, a correspondence table storage unit 26b, a document storage unit 26c, and an evaluation storage unit 26d. The program storage unit 26a stores a program to be described later. The correspondence table storage unit 26b stores a plurality of viewpoint profiles as shown in FIG. Each viewpoint profile stores a plurality of corresponding keywords. A plurality of documents to be evaluated are stored in the document storage unit 26c. In the present embodiment, each document includes a creation date and time, a title of the document, and contents of each document. The evaluation storage unit 26d stores evaluation results for each term. In addition, the memory 27 stores various calculation results and the like.
[0032]
3. flowchart
Next, the program stored in the program storage unit 26a of the hard disk 26 will be described with reference to the flowcharts of FIGS. In the following, an example will be described in which each term is evaluated using “prosperous economy” and “recession” as the viewpoint profile.
[0033]
First, the operator inputs a plurality of queries. In the present embodiment, since the viewpoint profile is adopted for the query, a selection box 62 as shown in FIG. 8 may be displayed on the CRT 30 to select the viewpoint profiles “boom” and “recession”. FIG. 8 shows a state where the viewpoint profile “depression” is selected after the viewpoint profile “boom” is selected.
[0034]
If the viewpoint profile does not exist in the selection box 62 shown in FIG. 8, the button 64 may be clicked to add the required viewpoint profile and the corresponding related keyword. Thereby, it is added to the correspondence table shown in FIG.
[0035]
Note that existing viewpoint profiles may be added to or deleted from the related keyword box 63 when additional keywords are deleted. In this way, by creating a viewpoint profile from the viewpoint in which the operator is interested and specifying the term extraction target document, it is possible to extract a document in accordance with the viewpoint desired by the operator.
[0036]
When CPU 23 determines that a plurality of queries have been input (step S1 in FIG. 5), it initializes query number k (step S3). Then, an extraction target document is determined for the viewpoint profile “business boom” which is the 0th query (step S5). The extraction target document determination process will be described with reference to FIG. In the present embodiment, as described below, the similarity with the viewpoint profile is determined, and a document with a similarity exceeding a predetermined threshold is determined as a term extraction document.
[0037]
The CPU 23 initializes the document number m (step S50). It is determined whether or not the vectorization process for each document has been completed (step S51). In this case, since the vectorization process has not been completed, the process proceeds to step S53, where morphological analysis is performed for each document, and words (terms) appearing in the document are extracted. Then, for the extracted terms, important terms are determined using the tfidf method (step S55). The tfidf method is a method for determining a keyword in information retrieval, and the tf (term frequency) indicating the frequency of appearance of a term in a document and the rareness of how few the term appears in all documents. This is a method of weighting a term using idf (inverse document frequency) indicating.
[0038]
Using the important terms extracted in this way, each document is vectorized as a numerical vector having the same dimensions as the number of important terms. For example, when there are 100 important terms, a 100-dimensional numerical vector is obtained. Some documents may not include the important terms that have been determined. In this case, the dimension value of the document is 0 (sparse). The CPU 23 stores the numerical vector thus obtained in the memory 27.
[0039]
Next, the CPU 23 vectorizes the viewpoint profile (step S59). In the present embodiment, all the related keywords of the kth viewpoint profile are read from the hard disk 26 and vectorized as a numerical vector having the same dimension as the number of each related keyword in the viewpoint profile. In this embodiment, the tfidf value for each related keyword is obtained from the tf of each related keyword in the viewpoint profile and the idf in all documents, and is vectorized as a numerical vector having the same dimension as the number of the related keywords.
[0040]
Next, the similarity between the viewpoint profile and the mth document is calculated (step S61). Such similarity can be obtained by calculating the inner product of the numerical vector obtained in step S57 and the numerical vector obtained in step S59.
[0041]
Next, the CPU 23 determines whether or not the similarity calculation has been completed for all documents (step S63). If the similarity calculation has not been completed for all the documents, the document number m is incremented (step S65), and the processing from step S61 is performed.
[0042]
When the similarity calculation is completed for all documents in step S63, a document exceeding a predetermined threshold is determined as an extracted document (step S67).
[0043]
Next, the CPU 23 calculates a statistical amount of terms existing in the extraction target document. Such calculation processing will be described with reference to FIG.
[0044]
The CPU 23 initializes the target document number i and the target term number j (step S25). The i-th document is set as the document of interest (step S27). The j-th term of the document of interest is set as the term of interest (step S29).
[0045]
The CPU 23 determines whether or not the term of interest is the term that first appears in the document of interest (step S31). If the term first appears in the document of interest, the term appearing document number bi is incremented (step S33).
[0046]
The CPU 23 determines whether or not the term of interest exists in an extraction term table (not shown) in the memory 27 (step S35). If it already exists, the term appearance frequency ti is incremented (step S37). On the other hand, if it does not exist, it is added to the extraction term table (step S39).
[0047]
The CPU 23 determines whether or not the term of interest is the final term (step S41). If it is not the final term, the target term number j is incremented (step S43), and the processing from step S29 is repeated. On the other hand, if it is the final term, it is determined whether or not the term extraction has been completed for all documents (step S45). If there is a document that has not been term-extracted, the document number i of interest is incremented (step S47), the term of interest number j is initialized (step S49), and the processes in and after step S27 are repeated. Thereby, the number of appearance documents bi and the appearance frequency ti are obtained for each term.
[0048]
Next, the CPU 23 determines whether or not processing has been completed for all queries (step S9 in FIG. 5). If the processing has not been completed for all the queries, the CPU 23 increments the query number k (step S11) and repeats the processing from step S5.
[0049]
When processing is completed for all queries, the CPU 23 performs numerical vectorization processing for each query for each term (step S13). In this embodiment, the vector elements of the numerical value vectorization of terms include 1) term appearance frequency ti, 2) term appearance document number bi, 3) term appearance frequency ti / term appearance document number bi, 4) relevance degree You can choose either.
[0050]
The degree of relevance is a numerical value indicating the degree of relevance in the document with respect to the related word group constituting the viewpoint profile, and varies depending on the algorithm to be evaluated. In this embodiment, it is determined based on the number of term appearance documents bi and the rarity in all documents. Accordingly, the greater the number of term appearing documents, the higher the degree of association, and the smaller the number of term appearing documents, the lower the degree of association. Further, there are not many documents other than the term extracted document, and the terms existing in the term extracted document have a high degree of relevance. That is, when the number of term appearance documents is high and there are not many documents other than the term extraction document, and there are many documents in the term extraction document, the relevance degree is high.
[0051]
Note that 1) term appearance frequency ti and 3) term appearance frequency ti / term appearance document number bi are the extraction of terms co-occurring with the keywords used in the query set, that is, the extraction of events related to the operator's interest. It is thought that it is useful. Also, 2) the term appearance document number bi is considered to be useful for extracting main topics related to the operator's interest and the number of occurrences. 4) The degree of relevance seems to be useful for extracting events that have a rare relationship with the operator's interest.
[0052]
FIG. 9 shows an example in which each term is represented by a numerical vector. In this case, the degree of association is obtained as a vector element for the viewpoint profiles “prosperous economy” and “recessive economy”, and threshold values that have a threshold value of 0 or more (all terms) are displayed in the display area 72. Such vector elements and threshold values can be set by clicking on the threshold value setting button 71 with a mouse, and a threshold value setting dialog 74 is displayed so that the operator can set desired conditions.
[0053]
FIG. 10 shows a display example when the common conditions are extracted by changing the setting conditions. In FIG. 10, the number of term appearing documents bi as vector elements and threshold values of all queries of 1 or more are displayed in the display area 72. As described above, a term in which all of a plurality of designated vector elements have values greater than or less than a threshold value designated by the operator is referred to as a common term.
[0054]
The common term is a term that is extracted in common to all queries. When the query is a viewpoint profile that represents the user's interest as in the present embodiment, the term ( In many cases.
[0055]
FIG. 11 shows a display example when the unique conditions are extracted by changing the setting conditions. In FIG. 11, the term appearance document number bi as a vector element and a threshold value of 1 or more for only one query are displayed in the display area 72. In this way, a term in which only a specific vector element has a value greater than or less than a threshold value designated by the operator is referred to as a specific term.
[0056]
The unique term is a term that is extracted only by the corresponding query. When the query is a viewpoint profile that represents the user's interest as in this embodiment, the unique term is a term that is specific to the user's specific interest. There are many.
[0057]
Next, the CPU 23 extracts feature terms (step S15 in FIG. 5). The feature terms may be extracted by the operator providing an extraction criterion. In this embodiment, a term including a specific character string is extracted. In this case, feature term extraction is performed as follows by setting filtering conditions.
[0058]
FIG. 12 shows a display example when only terms including the character string “company” are extracted. In this case, in order to set the filtering condition, when the operator clicks the filter button 75, a filtering dialog 76 is displayed. Therefore, “company” may be set as the character string filter.
[0059]
As a filtering condition, in addition to the character string filter, a dictionary filter and a pattern filter can be set alone or in combination.
[0060]
A dictionary filter is a glossary that stores one or more terms desired by an operator. In the present embodiment, the corresponding auxiliary verb is stored in advance in the dictionary filter for each user's intention as shown in FIG. 13 so that the operator can select it. Specifically, when the operator selects “hope” as an intention, a plurality of terms “want” are subjected to filtering processing as a character string filter under the or condition. As a result, it is possible to extract terms including the term “tai”, for example, terms such as “want to know” and “want to check”.
[0061]
In this way, mining is facilitated by previously creating a dictionary of auxiliary verbs that serve as guidelines for mining.
[0062]
Further, one or more suffixes may be stored in advance in the dictionary filter for each user's intention so that they can be selected. A suffix is an independent word that adds meaning to the end of a word or qualifies a part of speech. For example, “target”, “sex”, and the like. As a result, terms such as “specific” and “innovative” can be extracted. Then, it is possible to know what word the term relates to in the document.
[0063]
Further, one or more classifiers may be stored in advance in the dictionary filter for each user's intention so that they can be selected. For example, “Yen”, “MHz”, and the like. Thereby, since a term composed of a number and a classifier can be extracted, it is possible to know what word the term relates to in the document. In particular, in the case of a price, it is possible to set a threshold value for the price and extract only information on products with a price lower than or higher than that.
[0064]
Note that such a dictionary filter may further have a hierarchical structure for each user's intention.
[0065]
In this embodiment, auxiliary verbs corresponding to the user's intentions stored in advance in the dictionary filter are used. However, even if such a dictionary filter is not used, the user gives this to the character string filter for filtering. May be.
[0066]
In addition to the above, the feature terms can be extracted based on the following criteria.
[0067]
1) Singular term: A term whose vector element value is specifically larger or smaller than other terms
A singular term is a term in which the difference between each vector element value is larger than a predetermined threshold, and although it is not as unique as a specific term, it is often a term related to a user's specific interest. For example, even if the three vector element values are “1, 1, 6” and the threshold value is “5”, the threshold value is “1, 3, 6” and the threshold value is “5”. Extracted as
[0068]
2) Difference term: For queries that have meaning in permutation, a term with a large difference between a query and the query before and after it.
The difference terms are, for example, the first viewpoint profile “sedan”, the second viewpoint profile “sports”, the third viewpoint profile “RV”, etc. When is used, it is possible to extract new topics and the like that co-occur with changes. Note that the differential term is also effective when the extraction period described later is limited.
[0069]
As described above, in this embodiment, an extraction target document is determined for each viewpoint profile, the evaluation of each term in the viewpoint profile is calculated, and this is repeated for a plurality of viewpoint profiles, and each term is converted into a plurality of viewpoints. Evaluated by numerical vector for profile. By specifying the term extraction target document from the viewpoint desired by the operator, the term is extracted from the document related to the operator's interest, so that the term deeply related to the operator's interest can be extracted. Moreover, since the total number of terms can be narrowed down, term analysis becomes easier.
[0070]
  In addition, by expressing each extracted term as a vector for multiple viewpoint profiles,MultifacetedThe term can be evaluated. Therefore, discovery of unknown knowledge is facilitated.
[0071]
In the present embodiment, a pattern filter that designates the pattern of a vector element of each term is employed. Thereby, for example, the common term, unique term, unique term, and differential term can be extracted.
[0072]
4). Trend analysis
In the above embodiment, different viewpoint profiles are used as a plurality of search conditions. However, a plurality of search conditions may be used depending on one viewpoint profile and the creation period of each document. For example, as shown in FIG. 14, the viewpoint profile “recession” is selected, and the weighting dialog 83 sets weights for the related keywords of this viewpoint profile. In this case, all the related keywords are set to weight “1”. Furthermore, the period of interest is set to 5 times with a start date of 1997/1/1 and a period of one month.
[0073]
An execution result under the above conditions is shown in FIG. In the display area 77, term-extracted documents are displayed in order of date. In the display area 72, the terms are sorted and displayed according to the degree of relevance of terms extracted from the term extraction document. In the present embodiment, rearrangement is performed based on the flowchart shown in FIG.
[0074]
First, the CPU 23 initializes the period number n (step S71 in FIG. 16). The terms are rearranged by the nth value (step S73). In this case, the values are rearranged in descending order by the value of the 0th period (1 / 1-1 / 31 period).
[0075]
Next, the CPU 23 extracts terms whose value in the nth period is “0” and sorts them by the value in the (n + 1) th period (step S75). That is, in this case, terms whose value in the 0th period is “0” and whose value in the 1st period is not “0” are rearranged in descending order of the value in the 1st period.
[0076]
The CPU 23 determines whether or not the examination has been completed up to the final period (step S77). If it is not the final period, the period number n is incremented (step S79), and the processes after step S75 are repeated. On the other hand, when the examination is completed until the final period, the process is terminated.
[0077]
Such rearrangement makes it easier to find terms that first appear during the period of interest. For example, in FIG. 15, if the period of interest is the second period (3/1 to 3/31), the term “souvenir / fishery wholesale” will not appear in the previous period 2/1 to 2/28. It can be seen that the term first appeared in the period of interest 3/1 to 3/31.
[0078]
You may make it use a character string filter and a pattern filter together for this term analysis. The analysis result at the time of setting the filtering conditions concerning FIG. 17, FIG. 18 is shown. In FIG. 17, the viewpoint profile “stock” is selected, and the weighting dialog 63 sets weights for the related keywords of this viewpoint profile. In this case, the weights are all “1”. Further, the period of interest is set in the period setting dialog 67. In this case, the start date is 1997/1/1, the period is set to 4 times every 3 months.
[0079]
An execution result under the above conditions is shown in FIG. In the display area 77, term-extracted documents are displayed in order of date. In the display area 72, terms extracted from the term extraction document are displayed. In this case, as the filtering condition, the character string “stock” is set as the pattern filter “monotonically increasing” as the character string filter, and thus the character string “stock” is included in the extracted terms. The term whose monotonically increasing value is displayed is displayed.
[0080]
By such trend analysis, it is possible to extract terms that appear in common in documents in each period, terms that appear only in documents in a specific period, and terms that newly appear or disappear over time. For example, it is useful for extracting events that gradually increase or decrease.
[0081]
Such trend analysis may be displayed as a graph (such as a line graph) for each term or to some extent in order to facilitate visual grasp.
[0082]
5. Other embodiments
In the above embodiment, a pair of viewpoint profiles having the opposite direction to the booming economy and the recession is adopted as the plurality of viewpoint profiles. However, the present invention is not limited to this, and any viewpoint profiles can be used. Anything can be set freely by the user.
[0083]
Although the viewpoint profile is given as a query, some terms related to the user's interest may be given directly as keywords. For example, “I want to know the relationship between stock prices and economic trends” When a natural sentence is given, the natural sentence may be subjected to semantic analysis, a keyword may be set, and a query may be generated. Furthermore, you may make it add constraint conditions, such as the search time range which restrict | limits a query, and the term which must be included.
[0084]
In addition, since the document data and the number of terms to be extracted are different for each query, correction using statistical data may be performed. For example, the term appearance frequency may be divided by the number of documents including the term or the number of extracted documents.
[0085]
The numerical vector data may be classified using a conventional classification method. For example, the following three methods can be employed.
[0086]
1) Cluster analysis
If there is no classification criterion in advance, a cluster analysis may be performed. After classification, the meaning of each group can be estimated by extracting typical examples of each group. For example, a term that is closest to the center of the group may be extracted.
[0087]
As a clustering method, a conventionally used method can be employed. For example, not only hierarchical cluster analysis but also non-hierarchical cluster analysis may be performed.
[0088]
2) Classification
Several clusters are prepared in advance, and the terms are classified by assigning each numerical vector to the nearest cluster.
[0089]
3) Simple classification
The terms with the same vector element value are maximized to form one cluster. The same number of classification clusters as the number of queries is generated, and the terms related to the corresponding query are classified into each classification cluster.
[0090]
By such classification processing, it becomes possible to discover new knowledge by extracting characteristic terms of a certain classification group and comparing them.
[0091]
In the present embodiment, since the extraction target document is specified by the viewpoint profile that the user is interested in, term analysis with relatively high accuracy is possible. In addition, the total number of terms is reduced compared to the case of extracting from all documents, classification is possible in a short time, and user analysis is facilitated.
[0092]
In the above embodiment, a pair of viewpoint profiles having opposite directions are adopted as the plurality of viewpoint profiles. However, the present invention is not limited to this, and for example, viewpoint profiles that are not related at first glance, for example, “ It may be “politics” and “birth rate”.
[0093]
In the present embodiment, “prosperous economy” and “recession” are used as the profile representing the state as the viewpoint profile, but “increased production”, “increased profit”, and the like may be used as profiles representing changes. By using a profile representing such a state or change, data mining becomes easier. The viewpoint profile is not limited to these as long as the viewpoint profile desired by the operator is used, and any viewpoint profile may be used.
[0094]
Note that the process of extracting and vectorizing terms from each document when determining the extracted document can be performed in advance if the contents of all documents can be confirmed, so a new document has been added. May be executed and remembered from time to time.
[0095]
In the present embodiment, the important terms are extracted from each document in the extracted document determination process. However, the term is not limited to this, and for example, not only the important terms but also all the terms are extracted. Or may be extracted based on a certain extraction criterion.
[0096]
In the present embodiment, the similarity between two numerical vectors is determined by calculating the inner product of both numerical vectors, but the cosine value of both numerical vectors may be used as the similarity.
[0097]
In the present embodiment, the case of a Japanese document has been described. However, the present invention can be similarly applied to other languages such as English, Chinese, and Korean.
[0098]
In the present embodiment, the CPU 23 is used to realize the function shown in FIG. 1, and this is realized by software. However, some or all of them may be realized by hardware such as a logic circuit.
[0099]
In this way, the terms existing in the document are converted into numerical vectors by the scores relating to a plurality of viewpoint profiles in which the operator is interested. Thereby, compared with the case where it represents with the value depending on the appearance frequency of a term, the operator can grasp | ascertain the relationship between an own interest and each term easily. In addition, it is possible to analyze a term converted into a numerical vector by using various statistical analysis methods. Furthermore, the meaning of the analysis result becomes easy.
[Brief description of the drawings]
FIG. 1 is a functional block diagram of a term evaluation device 1 according to the present invention.
FIG. 2 is a diagram showing an example of a hardware configuration of the term evaluation device shown in FIG. 1;
3 is a diagram showing an example of a hardware configuration in which the computer system 40 shown in FIG. 2 is realized by using a CPU 23. FIG.
FIG. 4 is a diagram illustrating a correspondence between viewpoint profiles and related keywords.
FIG. 5 is an overall flowchart of term evaluation processing.
FIG. 6 is a detailed flowchart of extracted document determination processing;
FIG. 7 is a detailed flowchart of a statistic determination process.
FIG. 8 is an example of a dialog for query determination.
FIG. 9 is an example of a display of an extraction term.
FIG. 10 is an example of a display of an extraction term.
FIG. 11 is an example of a display of an extraction term.
FIG. 12 is an example of a display of an extraction term.
FIG. 13 shows a data structure of a dictionary filter.
FIG. 14 shows a setting dialog for trend analysis.
FIG. 15 shows the result of trend analysis.
FIG. 16 is a detailed flowchart of the sorting process.
FIG. 17 shows a setting dialog for trend analysis.
FIG. 18 shows a result of trend analysis.
[Explanation of symbols]
23 ... CPU
27 ... Memory

Claims (14)

In a recording medium storing a program for causing a computer including an input device, a control device, and a storage device to function as a term evaluation device,
The program causes the computer to execute the following processing:
A) A plurality of corresponding related keywords are stored for each viewpoint profile. When a plurality of viewpoint profiles are given, the following term extraction target document determination process and term statistic calculation process are executed for each viewpoint profile. And
a1) Term extraction target document determination processing for specifying one or more documents to be subject to term extraction from a plurality of document data using a plurality of related words predetermined for the viewpoint profile;
a2) Term statistic calculation processing for calculating a term statistic existing in the specified term extraction target document and calculating as a term statistic in the viewpoint profile ;
B) For each term existing in the specified term extraction target document, a numerical vector in which the value of each vector element is represented by the term statistic of each viewpoint profile and has the same dimension as the number of viewpoint profiles Seeking
C) displaying the term for which the numerical vector is obtained together with the numerical vector;
A recording medium storing a program characterized by the above. In the recording medium which memorize | stored the program of Claim 1,
Based on the numerical vector of each term , extracting a term in which only a specific vector element has a value greater than or equal to a predetermined threshold value, and displaying the extracted term together with the numerical vector;
It is characterized by. A recording medium storing a program according to claim 1,
Based on the numerical vector of each term, a specific plurality of vector elements all extract a term having a value equal to or greater than or equal to a predetermined threshold, and the extracted term is displayed together with the numerical vector.
It is characterized by. In the recording medium which memorize | stored the program of Claim 1,
Extracting a term having a specific vector element value that is specifically larger or specifically smaller than other terms based on the numeric vector of each term, and displaying the extracted term together with the numeric vector;
It is characterized by. In the recording medium which memorize | stored the program of Claim 1,
Wherein the plurality of viewpoints profiles are perspective profile is meaningful permutations, on the basis of the numeric vector of each term, the value of a vector element in comparison with before and after the vector element that extracts the larger terms than a predetermined difference, Displaying the extracted terms along with their numeric vectors,
It is characterized by. In the recording medium which memorize | stored the program of Claim 1,
A specified character string included in the term, the character string expressing the intention of the document creator is classified and stored for each intention, and when the operator selects one of the classifications, the specified character string is specified. Extract only terms that have strings , and display the extracted terms along with their numeric vectors.
It is characterized by. An evaluation method for evaluating terms constituting document data by a computer, wherein the computer stores a plurality of corresponding keywords corresponding to each viewpoint profile, and executes the following processing:
A) When multiple viewpoint profiles are given, the following a1) term extraction target document determination process and a2) term statistic calculation process are performed for each viewpoint profile.
a1) Term extraction for determining a term extraction target document by specifying one or more documents to be subject to term extraction from a plurality of document data using a plurality of related words predetermined for the viewpoint profile. Target document decision processing,
a2) Term statistic calculation processing for calculating the term statistic existing in the specified term extraction target document as the term statistic in the viewpoint profile;
B) For each term, find a numeric vector whose vector element value is represented by the term statistic of each viewpoint profile and has the same dimensions as the number of viewpoint profiles;
C) displaying the term for which the numerical vector is obtained together with the numerical vector;
A term evaluation method characterized by The term evaluation method according to claim 7,
If the operator gives a character string that is used attached to a verb or adjective and expresses the intention of the document creator to the verb or adjective, the specified character string exists, and each of the terms Extracting a term having a predetermined feature based on the numerical vector of the and displaying the extracted term together with the numerical vector;
It is characterized by. A plurality of corresponding keywords are stored for each viewpoint profile, and when a plurality of viewpoint profiles are given, a plurality of corresponding keywords are used for each given viewpoint profile, from a plurality of document data, Term extraction target document specifying means for specifying one or more documents to be subject to term extraction;
Term statistic calculating means for calculating a term statistic existing in the specified term extraction target document and outputting it as a term statistic in the viewpoint profile;
Numeric vector computing means for obtaining a numeric vector having the same dimension as the number of viewpoint profiles for each term and the value of each vector element represented by the term statistic of each viewpoint profile;
C) Display means for displaying each term computed by the numeric vector computing means together with the numeric vector;
A term evaluation device comprising:   The term evaluation device of claim 9,
  The display means extracts a term in which only a specific vector element has a value greater than or equal to a predetermined threshold based on the numeric vector of each term, and displays the extracted term together with the numeric vector. ,
  It is characterized by.   The term evaluation device of claim 9,
    The display means extracts a term in which all of a plurality of specific vector elements have values greater than or equal to a predetermined threshold value based on the numeric vector of each term, and the numeric vector for the extracted term Display with,
  It is characterized by.   The term evaluation device of claim 9,
    The display means extracts, based on the numerical vector of each term, a specific vector element value that is specifically larger or specifically smaller than the other terms, and for the extracted terms, the numerical vector Display with,
  It is characterized by.   The term evaluation device of claim 9,
  The plurality of viewpoint profiles are viewpoint profiles that are meaningful in permutation,
  The display means extracts a term in which a value of a certain vector element is larger than a predetermined difference from the preceding and following vector elements based on the numeric vector of each term, and displays the extracted term together with the numeric vector. thing,
  It is characterized by.   The term evaluation device of claim 9,
  A predetermined character string included in the term, and a character string expressing the intention of the document creator is classified and stored for each intention,
    The display means, when any classification is designated by the operator, extracts only the term in which the designated character string exists, and displays the extracted term together with its numerical vector,
  It is characterized by.

Images