JP4639388B2 - Important word extraction method, important word extraction apparatus, computer program, and program storage medium in document database - Google Patents

Description

  The present invention relates to a method of extracting important words from a document database formed by aggregating a large number of documents in a specific field such as a predetermined academic field by a programmed computer.

  In a document database in which a large number of documents are aggregated, in order to identify differences between documents, there is a concept of extracting terms (important words) that frequently appear in a specific document and are biased toward a specific document. Important words, unlike particles and adverbs that appear many times in every document, the absolute number of occurrences is not large, but only in a specific document appears more than other documents, Words or collocations that can be understood to some extent.

In general, in order to determine whether each term included in a database is an important word, the appearance frequency of the term is used. Known methods for calculating the appearance frequency include the tf method, idf method, tf · idf method, and the like. Then, the appearance frequency calculated by these methods is defined as an importance level, and a term having a high importance level is defined as an important word. Also, based on the quantified importance, it is possible to determine whether or not the keyword to be referred to the database is appropriate for searching a specific document, and to identify what the important word characterizing each document is. To do. The following non-patent document 1 describes an information processing method related to extraction of important words.
Makoto Nagao, 5 other authors, “Information Processing of Characters and Sounds”, 1st edition, January 21, 2000, p29-p35

  In the document database in which a large number of documents in a specific field such as a predetermined academic field are aggregated, even when a specific term appears frequently only in a specific document, the term We know empirically that it is not necessarily an important word for documents. In other words, the importance calculation method cannot identify an appropriate keyword that characterizes a document.

 In addition, it is also known that in documents related to specialized fields such as academic papers, the contents cannot be grasped unless the importance and correlation between the terms are examined for all the terms included in the document. That is, even if an academically characteristic term is included in a very small number of specific documents, the documents do not necessarily have the same content. For example, in the document database “OMIM (Online Mendelian Inheritance in Man)” that summarizes medical and molecular genetic explanations (entries) on human genetic traits such as genetic diseases, the key words for each entry are the frequency of occurrence in the past. When trying to extract by the method based on the above, each entry in the OMIM is originally related to the same academic field, so the same term is extracted as an important word in a plurality of entries. For example, in order to search for an entry describing a specific genetic disease, a term characterizing the entry is presented based on the importance calculated by a conventional method, and the term characterizes a specific content (disease). Even if there are, there is a high possibility that a plurality of entries including the term are presented. Because multiple presented entries are different in content (disease), researchers must look through all the presented entries in the end when they try to find an entry for the disease they want to investigate. It takes a lot of labor and time to find an entry.

  In the document database in which a large number of documents in a specific field such as a predetermined academic field are aggregated, the present inventors first studied a calculation method of importance so that important words of individual documents can be accurately identified. Then, after calculating the appearance frequency of the terms, a special calculation formula for finding the importance using the appearance frequency was found. Also, based on the importance obtained by the special calculation method, we examined and found out how to present the terms contained in the document so that it can be grasped by glances at the contents of each document.

  The present invention has been made on the basis of these findings. The purpose of the present invention is to accurately identify important words that characterize each document in a document database in which a large number of documents about a specific field are aggregated, and to understand the contents of each document. It is to provide a method for extracting important words so that it can be grasped.

In order to achieve the above object, the present invention searches a document database in which n documents related to a predetermined academic field are collected by a programmed computer, calculates the importance of terms contained in the database, and identifies the specified database. A method for extracting terms that are highly relevant to a field,
A term storage step of acquiring the total number m of terms included in the database and each term T _j (j = 1, 2, 3,..., M), and identifying and managing each term T _j ;
An appearance frequency calculating step of calculating an appearance frequency W _ij for the term T _j in the document D _i ;
A variance calculation step for calculating a variance S ² _j of appearance frequency W _ij values for the term T _j ;
The number of occurrences of the term T _j in the document D _i is U _ij , and the importance V _ij of the term T _j in the document D _i is V _ij = U _ij × S ² _j
Importance calculation step calculated by
A list creation step of creating and outputting a term list in which terms T _j are listed based on V _ij ;
Extracting one or more documents D _h from n documents included in the document database ;
Obtaining the total number x of terms contained in the document D _h ;
The term T _g (g = 1, 2, 3,..., X) included in the term list for the document D _h created by the list creation step is output, and one from the term T _{g is} inputted by the user input. Receiving a designation of the above term T _k (k = 1, 2, 3,..., X);
Obtaining a designated T _k number a;
Extracting y terms T _f (f = 1, 2, 3,..., Y) based on the importance V _hg for the term T _g in the document D _h ;
Obtaining the number b of terms that match the designated term T _k out of the terms T _f while variably setting the number y of terms ;
Term extraction accuracy Zh for document D _h is
Z _h = b / a + {x− (a + y−b)} / (x−a)
A step of calculating according to the formula:
Obtaining x and y when the value of Z _h is maximum, and obtaining a function y = f (x) approximating the relationship between x and y;
Based on the function y = f (x), the number y _i of terms to be listed in the term list for the document D _i including x _i terms is calculated by y _i = f (x _i ), and the calculated Re-creating a term list with terms of the number of terms y _i ,
Including
In the occurrence frequency calculation step, the number of occurrences of term T _j in all documents and U _j, the number of occurrences of term T _j in the document D _i and U _ij, the total number of occurrences of the obtained m-number of all terms As U, the appearance frequency W _ij is _expressed as W _ij = (U _ij / U _j ) × log (U / U _j )
Calculated by
In the variance calculation step, the average value of the appearance frequencies of the term T _j for all documents is W, and the variance value S ² _j is S ² _j = {(W _1j −W) ² + (W _2j −W) ² + ... + (W _nj −W) ² } / n
Calculated by
This is a key word extraction method in a document database characterized by this.

  An important word extraction method in a document database that further includes any one of the following requirements (1) to (4) in the above important word extraction method is also within the scope of the present invention.

(1) including a step of accepting designation of the document D _i by user input, and the list creation step creates a list in which the terms T _j included in the designated document D _i are listed in order based on importance. .

(2) the list creation step, for each of all the documents, as well as create a list of terms that sequentially lists of high importance terminology, a step of accepting a designation of a keyword by the user input, each term of all documents among the list, the term corresponding to the keyword and outputting the document D _i terms list when a predetermined importance V _ij

(3) including a step of accessing a dictionary database storing specific terms, wherein the list creation step does not list the terms existing in the dictionary database in the term list.

(4) accessing a coefficient database in which a specific term and a coefficient are stored in association with each other; and a step of setting a value obtained by multiplying the importance V _ij of the term T _{j by} the corresponding coefficient as a new importance. And the list creating step creates a list based on the new importance.

  The present invention includes a keyword extraction device that is configured by a computer and executes the steps included in the above method, and is installed in the computer and executes the steps included in any of the above methods on the computer. And a program storage medium readable by a computer recording the computer program.

  According to the important word extraction method of the present invention, in a document database in which a large number of documents about a specific field are aggregated, it is possible to accurately identify important words that characterize the document from terms included in each document, and to determine the contents of each document. You can grasp at a glance.

=== Outline of the keyword extraction method ===
As one embodiment of the present invention, a computer (important word extracting device) programmed to access a document database in which a large number of documents about a specific field are aggregated and extract important words contained in a document by the method of the present invention. : Extraction device) below. According to the extraction apparatus of the present embodiment, when extracting important words, the importance of terms included in each document in the document database is measured by a special calculation formula, and as a result of the measurement, terms having high importance are classified for each document. Create and output a list of terms listed in order. The document database may be attached to the extraction apparatus or may be external. The output of the list may be to store the list itself in a predetermined storage resource, or to output the list so that it can be viewed by the user of the document database.

=== Document database ===
In the present embodiment, the extraction apparatus uses OMIM as an important word extraction target. As is well known, OMIM is a document database of “MIM (Mendelian Inheritance in Man)”, an encyclopedia that summarizes medical and molecular genetic descriptions of human genetic traits such as genetic diseases. As of January 2004, the number of articles (entries) included in the OMIM is more than 15,000. Among the entries, there are about 4,500 entries for different genetic diseases. This OMIM can be searched and viewed online by a WWW server on the Internet. In this embodiment, the extraction apparatus is configured to access the OMIM via the Internet. Of course, an OMIM may be attached to the extraction device.

=== Calculation of importance ===
1A to 1D show an outline of importance calculation processing in the extraction apparatus. The extraction device extracts terms for all entries in the OMIM, obtains all n entries and all m terms, and identifies each entry with an identifier D _i (i = 1, 2, 3,..., N). ), Identifiers T _j (j = 1, 2, 3,..., M) are assigned to the terms, and all the n entries and all the m terms are identified and managed. The number of occurrences of each term Tj is counted for each entry and stored. Then, a matrix of n rows and m columns is created, and the number of appearances of the term T _j in the entry D _i is stored in each intersection (cell) of the matrix. Therefore, if the cell matrix (i, j) is designated, the number of occurrences of a specific term in a specific entry can be known. Here, the number of appearances of a specific term T _j in that specific entry D _i (term appearance number) is U _ij , the total number of appearances of each cell included in one column, that is, the specific term T _j through all entries. Let U _j be the number of appearances (total number of appearances of terms) (A). Moreover, U represents the number of appearances for all terms in all entries (total number of appearances of all terms).

Next, the appearance frequency W _ij regarding the term T _j in the entry D _i is _expressed by the following equation (1).
W _ij = (U _ij / U _j ) × log (U / U _ij ) (1)
And the value of _Wij is stored in each cell (B).

In the present embodiment, S ² _j is further calculated as a variance value of W _ij values for each term T _j (C).
That is, in the matrix (B) in which the appearance frequency W _ij is stored in each cell, the average value of W _ij in each column is W, and the variance value S ² _j for each column is known as the following formula (2)
S ² _j = {(W _1j −W) ² + (W _2j −W) ² +... + (W _nj −W) ² } / n (C).

Next, based on S ² _j , the importance V _ij of the term T _j in the entry D _i is _expressed by the following equation (3).
^{_{V ij = U ij × S 2}} j ... Equation (3)
And the calculation result is stored in each corresponding cell (D).

  In this embodiment, in order to calculate the degree of importance with higher accuracy, the appearance frequency is calculated by the above-described equation (1). However, for the calculation of the appearance frequency, the conventional tf method, idf The tf · idf method may be employed. The idea of the present invention is to abandon the conventional concept of adopting the appearance frequency as the importance, and for the specific term in the specific document, the importance value is the product of the dispersion value of the appearance frequency and the term appearance frequency. In the point.

=== Presentation of important words ===
When the extraction device calculates the importance level of a specific term in a specific entry using the above formulas (1) to (3), it creates a term list listed for each entry in the order according to the importance level. In this embodiment, a term list is created and stored in order of the most important terms for each entry.

  The created list may be output by making it viewable on, for example, a display attached to the extraction device or a computer accessible to the extraction device. As a result, multiple terms for a specific entry are displayed according to their importance, and if an expert such as a researcher glances at the list of terms, the importance of multiple terms in a specific entry and their correlation You can understand the relationship and know the contents of the entry.

  FIG. 2 shows a term list created based on the method of this embodiment for a specific entry (entry number # 137750). The title of entry # 137750 is an entry that describes the genetic disease GLAUCOMA, PRIMARY OPEN ANGLE, JUVENILE-ONSET, 1; JOAG (juvenile open angle glaucoma), and FIG. A list 10 in which 12 high terms (11a, 11b) are listed in order from the top is shown. For reference, the variance value 13 for each term is also shown in the list 10. Even for a term having a low variance value, if the number of appearances in a specific entry is large, the importance value of the entry becomes high, and the term list for the specific entry is listed higher. Therefore, it is possible to reliably list terms that can be important words in a specific entry.

  In order to prove the reliability of the importance calculated in this embodiment, entry # 137750 is actually read by an expert such as a researcher, and the expert actually understands the contents of the entry. I was asked to specify a term 11a that could be adopted as an important word. The important word 11a recognized by the expert is listed at the top of the list 10 created by the extraction device. Therefore, it has been found that the importance calculation method employed in the method of this embodiment is an index for extracting important words with high accuracy. In addition, since the terms are listed in order of importance for each entry, the correlation of a plurality of terms included in the entry can be known. That is, the importance levels of a plurality of terms can be compared, and the contents of the entry can be accurately recognized if a high-order term is listed in the term list created by the extraction device.

=== User interface ===
As a user interface when the user actually uses the extraction device in the present embodiment, it accepts entry entry and presents a term list of the entry, or accepts keyword entry and corresponds to a keyword. A method of presenting a term list having a high importance of the terms to be considered is conceivable.

  In addition, as a method of accepting user-specified entry and keyword specification input or presenting the term extraction result originating from that input, the extraction device itself is equipped with a user interface such as a keyboard and display, Although a method of inputting / outputting via a user interface may be used, a method of mounting a function as a WWW server in the extraction apparatus and connecting it to the Internet is also conceivable. The WWW server function prepares a Web page including a form for accepting entry and keyword designation input in the extraction device, and the user can use the computer (browser terminal) such as a personal computer to implement the browser. A Web page is obtained and the entry or keyword input on the page is sent to the extraction device. The extraction device may create a term list for a specified entry or a term list for an entry having a high degree of importance of a term corresponding to a keyword on a Web page and send it back to the browser terminal.

=== About improvement of key word extraction accuracy ===
As shown in FIG. 2, when the importance is calculated by the calculation formula of the present embodiment, a small number of terms 11 b that are not so important to the researcher are listed at the top of the term list 10. It is more preferable if such term 11b can be eliminated as much as possible. Therefore, a dictionary containing terms to be deleted from the term list is prepared, and the terms described in the dictionary in the created term list may be deleted from the list. In the OMIM database targeted by this embodiment, a person name, a unique mutation name, a marker, or the like can be a deletion target.

=== Weighting of important words ===
Even terms below the term list may actually be important to researchers. Therefore, the terms included in the database are stored and managed in a predetermined database in association with a coefficient to be multiplied by the importance in advance. When the extraction device presents a term list for a certain entry, the importance obtained by the expressions (1) to (3) is multiplied by a corresponding coefficient for each term in the list, and the importance is obtained. And create a term list based on the updated importance. As a result, the terms listed below based on the importance calculated by the equations (1) to (3) are also listed higher so as to match the substantial importance. As a result, researchers can grasp the contents of the entry in more detail without having to look up the terms in the term list.

  The database in which terms and coefficients are associated with each other may be an internal database attached to the extraction device, or an external database accessible by the extraction device. In addition, in order to determine the coefficient of each term, for example, a completely unnecessary term such as a particle or adverb is set to 0 so that it is not listed, or a specific type of term that is listed in a medical dictionary For gene symbols, organs, tissues, symptoms, diseases, etc., the low variance value is a term that certainly suggests the contents of the entry, so it is uniformly listed with a high coefficient and listed at the top of the list. For terms that are automatically listed at the top, such as OMIM titles, the coefficients assigned to the terms are such as associating predetermined coefficients to the extent that they are surely listed. What is necessary is just to set suitably.

=== Optimization of the number of key words extracted ===
Of course, it is not necessary to put a completely unnecessary term in the term list. If an appropriate number of terms are listed, the contents of the entry can be grasped only by the terms. Of course, a list including useless terms naturally has a large amount of data, and the large volume of data places an excessive load on the extraction device that handles the data. However, if the number of terms to be listed is uniformly limited for all entries, important terms may be missing from the list for entries with long documents and a large number of terms. Therefore, it is necessary to optimize the number of terms listed for each entry.

  Here, a technique for the optimization is illustrated. In general, a specialist such as a researcher who can judge the importance of a term is viewed and an important word is selected from the entry. Then, the selected important word is compared with the term in the term list created by the extractor, and applied to all entries based on the degree of matching between the term in the term list and the actual important word selected by the expert. Find the laws that can be done. Then, according to the rule, the optimum number of terms to be posted in the term list is determined for each entry.

Specifically, the extraction device designates a plurality of appropriate entries D _h from all the entries from a user such as an expert, and the important word T _k (the user selects from terms included in the D _h ). k = 1, 2, 3, ... specification and) received through the user interface, separately obtains the specified entry _{D h} the number a of the specified _{T k.}

Further, the extraction device, regarding the term list for the entry D _h created based on the above formulas (1) to (3), the total number x of terms included in the list and each term T _g (g = 1) in the list. , 2, 3,..., X). Next, y is the number of terms to be included in the term list, and y terms T _f (f = 1, 2, 3,..., Y) are listed based on the importance D _hg from the term T _g. To do. Then, the number b of terms that match the term T _k specified by the user among the terms T _f is acquired.

When extracting y terms, the y value itself may be set, and the terms up to y-th may be extracted in descending order of importance V _hg in each term T _g . the value variably set, and extracted their importance values more terms T _g as T _f, such as the number of T _f and y, if the variable set as appropriate based on the value of y is the degree of importance Good.

Then, while gradually variably setting the value of y, the following equation accuracy Z _h terms T _f that lists the documents D _h (4),
Z _h = b / a + {x− (a + y−b)} / (x−a) (4)
To obtain x and y when Z _h takes the maximum value, and obtain a function y = f (x) that approximates the relationship between x and y.

This y = f (x) is also applied to the other entries D _i to determine the number of terms to be listed in the term list in all entries D _i . That is, if it contains _{x i} number of terms in the entry D _i, the entry _D above function the number _{y i} of the terms listed in terms list for _i y _i = f _(x i)
Calculated by Then, a term list including the determined term number y _i is recreated for the entry D _i .

  Therefore, once the number of terms has been determined, a term list for all entries will be created, and if this created term list is prepared so that it can be referenced, a query for database search will be accepted. In addition, it is not necessary to perform the calculation processing by the above formulas (1) to (3) again. The extraction device may present a search result for the query with reference to the created term list.

Note that the entry D _h selected in the process of obtaining Expression (4) may not be extracted by user input. It may be set in advance on the extraction device side, or may be appropriately extracted such as random. Of course D _h is the reliability of the function expressed by the formula (4) is somewhat fall, it may be one entry.

=== Application example ===
The important word extraction method of the present invention can be applied to a database in which documents in a specific field are aggregated without being limited to OMIM. In addition, the database is not limited to a collection of articles such as a collection of papers. For example, each document is used as an explanation of definitions and definitions for terms in a specific field, and the database collects explanations about the terms. It may be a dictionary (dictionary) / encyclopedia.

It is a figure explaining the concept of the important word extraction method in the Example of this invention. It is the schematic of the term list produced by the said method.

Explanation of symbols

10 Term list 11a, 11b Term 12 Importance

Claims (8)

A method of searching a document database in which n documents related to a predetermined academic field are aggregated by a programmed computer, calculating the importance of terms included in the database, and extracting terms having high importance in the specific field Because
A term storage step of acquiring the total number m of terms included in the database and each term T _j (j = 1, 2, 3,..., M), and identifying and managing each term T _j ;
An appearance frequency calculating step of calculating an appearance frequency W _ij for the term T _j in the document D _i ;
A variance calculation step for calculating a variance S ² _j of appearance frequency W _ij values for the term T _j ;
The number of occurrences of the term T _j in the document D _i is U _ij , and the importance V _ij of the term T _j in the document D _i is V _ij = U _ij × S ² _j
Importance calculation step calculated by
A list creation step of creating and outputting a term list in which terms T _j are listed based on V _ij ;
Extracting one or more documents D _h from n documents included in the document database ;
Obtaining the total number x of terms contained in the document D _h ;
The term T _g (g = 1, 2, 3,..., X) included in the term list for the document D _h created by the list creation step is output, and one from the term T _{g is} inputted by the user input. Receiving a designation of the above term T _k (k = 1, 2, 3,..., X);
Obtaining a designated T _k number a;
Extracting y terms T _f (f = 1, 2, 3,..., Y) based on the importance V _hg for the term T _g in the document D _h ;
Obtaining the number b of terms that match the designated term T _k out of the terms T _f while variably setting the number y of terms ;
Term extraction accuracy Zh for document D _h is
Z _h = b / a + {x− (a + y−b)} / (x−a)
A step of calculating according to the formula:
Obtaining x and y when the value of Z _h is maximum, and obtaining a function y = f (x) approximating the relationship between x and y;
Based on the function y = f (x), the number y _i of terms to be listed in the term list for the document D _i including x _i terms is calculated by y _i = f (x _i ), and the calculated Re-creating a term list with terms of the number of terms y _i ,
Including
In the occurrence frequency calculation step, the number of occurrences of term T _j in all documents and U _j, the number of occurrences of term T _j in the document D _i and U _ij, the total number of occurrences of the obtained m-number of all terms As U, the appearance frequency W _ij is _expressed as W _ij = (U _ij / U _j ) × log (U / U _j )
Calculated by
In the variance calculation step, the average value of the appearance frequencies of the term T _j for all documents is W, and the variance value S ² _j is S ² _j = {(W _1j −W) ² + (W _2j −W) ² + ... + (W _nj −W) ² } / n
Calculated by
A method for extracting important words from a document database. 2. The method according to claim 1, further comprising a step of accepting specification of the document D _i by user input, wherein the list creation step is based on the importance calculated by the importance calculation step for the term T _j included in the specified document D _i. An important word extraction method in a document database, characterized in that a list listed in order is created. The list creation step according to claim 1, wherein the list creation step creates a term list in which high-priority terms are listed in order for each of all documents, receives a keyword specification by user input, And outputting a document D _i term list when a term corresponding to the keyword has a predetermined importance V _ij .   2. The document database according to claim 1, further comprising a step of accessing a dictionary database containing specific terms, wherein the list creation step does not list the terms existing in the dictionary database in the term list. Word extraction method. 2. The step of accessing a coefficient database in which a specific term and a coefficient are stored in association with each other, and a step of setting a value obtained by multiplying the importance V _ij of the term T _{j by} the corresponding coefficient as a new importance. And the list creating step creates a term list based on the new importance level. It is configured by a computer, key words and executes the steps involved in the method of any of claims 1-5 extractor. A computer program which is installed in a computer and causes the computer to execute the steps included in the method of any one of claims 1 to 5 . A computer-readable program storage medium storing the computer program according to claim 7 .

Images