JP3612125B2 - Information filtering method and information filtering apparatus - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an information filtering method and an information filtering device for searching for a user's request / interest from an enormous amount of text articles and periodically providing it to the user.
[0002]
[Prior art]
In recent years, with the spread of word processors and electronic computers, and the spread of electronic mail and electronic news via computer networks such as the Internet, the digitization of documents is accelerating.
[0003]
As the term “electronic publishing” suggests, it will be common to provide information on newspapers, magazines and books electronically in the future. As a result, the amount of text information that can be obtained in real time for an individual is expected to become enormous.
[0004]
Accordingly, there is an increasing demand for an information filtering system or an information filtering service that selects and regularly provides a user with a user's request / interest from huge text articles such as newspapers and magazines.
[0005]
On the other hand, from the viewpoint of making a large amount of information compact, research has also been conducted on methods for creating document abstracts. Conventionally, a method of extracting only the first paragraph or a method of extracting only a sentence including a keyword using a keyword registered in advance has been employed.
[0006]
[Problems to be solved by the invention]
However, the method of extracting only the first paragraph does not necessarily include the information required by the user, and is not an appropriate method. Moreover, there is a problem that the connection of sentences is not understood only by arranging sentences including keywords.
[0007]
Also, once the registered keyword is not changed by the user, it is not added or modified, and it is troublesome for the user to do this.
Therefore, the present invention has been made in view of the above problems, and for an article searched by information filtering, there is no burden on the user, and information that can generate an easy-to-read abstract that matches each user's desired theme. An object is to provide a filtering method and an information filtering device.
[0008]
[Means for Solving the Problems]
The information filtering method of the present invention receives distribution of articles such as texts and images from a plurality of information sources, searches the distributed articles for articles on a theme specified in advance for each user, and distributes them to the users. In the information filtering method, an article desired by a user is searched from articles distributed from the information source based on search information based on a theme specified in advance for each user, and an abstract of the searched article is searched. By creating a summary based on information and distributing the generated abstract to the user, a profile storing information about the user's interests and interests is used when generating the abstract. It is possible to generate an abstract that includes it.
[0009]
Further, the information filtering method of the present invention receives articles such as texts and images from a plurality of information sources, searches for articles related to the theme designated in advance for each user from the distributed articles. In the information filtering method distributed to the user, based on the search information based on the theme specified in advance for each user, the article desired by the user is searched from the articles distributed from the information source. An abstract of the searched article is generated based on the search information, and the generated abstract is distributed to the user, so that the document can be generated using an abstract generation method appropriate for the document according to the feature of the article. Abstracts can be generated.
[0010]
Further, the information filtering device of the present invention receives articles such as texts and images from a plurality of information sources, searches the articles for articles on a theme specified in advance for each user, and searches the articles for users. In the information filtering device distributed on the basis of search information based on a theme specified in advance for each user, search means for searching for an article desired by the user from articles distributed from the information source, and the search information When generating an abstract, the system includes: an abstract generation unit that generates an abstract of the article searched by the search unit; and a distribution unit that distributes the abstract generated by the abstract generation unit to the user. It is possible to generate an abstract that contains information that each user needs because it uses a profile that stores information about interests That.
[0011]
Furthermore, the information filtering device of the present invention receives distribution of articles such as texts and images from a plurality of information sources, searches the distributed articles for articles on a theme specified in advance for each user, and In the information filtering device distributed to the user, based on search information based on a theme specified in advance for each user, search means for searching for an article desired by the user from articles distributed from the information source, and search by the search means A plurality of abstract generation means for generating abstracts of articles searched by the search means based on the search information, and feature detection for detecting features of articles searched by the search means A selection means for selecting one of the plurality of abstract generation means based on the feature of the article detected by the feature detection means, and a selection by the selection means A distribution means for distributing the abstract generated by the abstract generation means to the user, and generating an abstract of the document using an abstract generation method appropriate for the document according to the feature of the article. Can do.
[0012]
DETAILED DESCRIPTION OF THE INVENTION
An embodiment of the present invention will be described with reference to the drawings.
First, with reference to FIG. 1, the structure of the whole information filtering system using the information filtering apparatus of this invention is demonstrated.
[0013]
The information filtering system is an information providing system for receiving a text article from a plurality of information sources 2 such as a newspaper company, a news agency, or a publisher, and periodically transmitting it to each of the subscriber user terminals 3. The information providing service is realized by the information filtering center 1.
[0014]
The information filtering center 1 is composed of one information filtering device connected to a plurality of information sources 2 and a plurality of subscriber user terminals 3 via a communication network, and performs control and processing for information filtering here. Central processing unit 4, a storage device 5 such as a semiconductor memory for storing programs and data, a magnetic disk, an optical disc, a receiving unit 6 for receiving a text article from information source 2 via a communication network such as a line or radio wave, a line or radio wave The transmission / reception part 7 etc. which transmit a text article to the user terminal 3 via a communication network etc., or receive the reply from the user terminal 3, etc. are comprised.
[0015]
Each user terminal 3 is an information processing terminal such as a personal computer or a workstation, for example, and receives a text article transmitted from the information filtering center 1 or transmits a text data to the information filtering center 1. And a display unit 9 for displaying the received text article on the screen.
[0016]
As shown in FIG. 2, the information filtering center 1 holds a kind of search condition called a user profile 10 for each user, and searches for an article to be provided to the corresponding user according to the user profile 10. To do. The user profile 10 is configured based on a plurality of themes (topics) designated by the user, and articles matching those themes are searched and selected and sent to the user.
[0017]
Next, the configuration of the information filtering center 1 will be described with reference to FIG. In FIG. 3, the user profile storage unit 50 stores profiles corresponding to search conditions for searching for articles. Here, a profile storage format and a specific example thereof will be described with reference to FIG. As shown in FIG. 4A, the profile describes a word that is considered to be included in an article that the user wants and its weight. Specifically, as shown in FIG. 4B, a plurality of words and their weights are stored for each user. The profile creation method is not the main point of the present invention. For example, a method in which a person asks what kind of article he wants from a user and enumerates words that are supposed to be included in the corresponding article.
[0018]
The article storage unit 51 stores articles distributed from the information source 2 and received by the receiving unit 6. Here, a storage format of the article data stored in the article storage unit 51 and a specific example thereof will be described with reference to FIG.
[0019]
As shown in FIGS. 5A and 5B, the article data includes article IDs (for example, “001”, “002”, “003”) for identifying the articles for all articles managed by the center 1. …), Publishers of articles (eg “A newspaper publisher”, “B newspaper publisher”, “C publisher”…), headlines of articles (eg “multi-media PC release”, “semiconductor sales”) , And a pointer (for example, 123456, 123457, 123458) to the article storage unit 51 indicating the storage position of the article body, and stored in the article storage unit 51.
[0020]
The article search unit 52 searches for articles stored in the article storage unit 51 using the user profile stored in the user profile storage unit 50. As a search method here, for example, a publicly known method described in “SMART information search system” (edited by Gerald Salton, translated by Shinjin Koji, Planning Center) can be used. That is, according to this method (vector space method), for example, assuming the vector space of each word stored in the profile, the weight vector of each word and the appearance of the corresponding word in the article to be searched on this vector space It is possible to calculate the similarity to the profile by calculating the inner product with the frequency vector, and rank and search the articles that are ranked in descending order of the similarity.
[0021]
FIG. 6 shows a specific example of the search result output from the article search unit 52. FIG. 6 shows a case where four articles are searched for a certain user profile. That is, a search result including an article rank (for example, “1”, “2”, “3”, “4”) based on the calculated similarity and an article ID together with the ID of the user profile is the search result. It is stored in the storage unit 53.
[0022]
In the conventional filtering system, a filtering result that is finally sent to the user is created from the result stored in the search result storage unit 53 and the article data stored in the article storage unit 51. It is stored in a predetermined storage area.
[0023]
FIG. 7 shows a specific example of the conventional filtering result sent to the user, and is composed of the rank of the article, the headline of the article, the body of the article, and the like.
The abstract generation unit 100 generates an abstract of articles included in the filtering result for the filtering result stored in the search result storage unit 53. As illustrated in FIG. 3, the abstract generation unit 100 includes an abstract generation control unit 101, an article feature detection unit 102, and a plurality of (for example, three) generation units 103-1, 103-2, and 103-3. Composed.
[0024]
The article feature detection unit 102 detects article features such as the number of characters of an article and the number of paragraphs from the articles searched by the article search unit 52.
The first creation unit 103-1, the second creation unit 103-2, and the third creation unit 103-3 each generate an abstract suitable for the feature of the article detected by the article feature detection unit 102. It has become.
[0025]
The abstract generation control unit 101 controls the entire abstract generation unit 100.
Next, the processing operation of the abstract generation control unit 101 will be described with reference to the flowchart shown in FIG.
[0026]
The abstract generation control unit 101 extracts, for example, the filtering results stored in the search result storage unit 53 in descending order of article rank (n) (steps S1 to S3), and the article IDs of the filtering results in order. The following processing is performed. First, the article ID is passed to the article feature detection unit 102, and the number of the generation unit to be selected from the article feature detection unit 102 (in this case, “1” (first generation unit 103-1), “2” (second) Generation unit 103-2) and “3” (any of the third generation units 103-3)) (step S4 to step S5).
[0027]
Next, the article ID and the user profile ID are passed to the number generation unit (step S6). And the abstract produced | generated by the said production | generation part is received (step S7), the rank of the article contained in the filtering result stored in the search result storage part 53 as shown in FIG. Thus, the article headline extracted from the article storage unit 51 and the abstract of the article are stored in a predetermined storage area of the search result storage unit 53, and finally a filtering result to be distributed to the user is generated.
[0028]
Next, the processing operation of the article feature detection unit 102 will be described with reference to the flowchart shown in FIG.
The article feature detection unit 102 uses the article ID passed from the abstract generation control unit 101 to follow the pointer to the article body from the article storage unit 51 and extracts the article body (step S10). Next, the number of characters and the number of paragraphs in the extracted article body are counted (step S11). A paragraph may be the beginning of a paragraph by detecting that the beginning of the line is a white space character. Further, the processing proceeds while referring to the abstract generation condition table stored in advance.
[0029]
FIG. 10 shows the format of the abstract generation condition table stored in the article feature detection unit 102 (FIG. 10A) and its specific example (FIG. 10B). The abstract generation condition table includes an ID number (for example, “1”, “2”, “3”) for identifying a plurality of generation units, and a maximum value of the number of characters of articles applied by the generation unit (for example, “ 400 ”,“ 800 ”,“ 10000 ”) and the maximum number of paragraphs (for example,“ 5 ”,“ 10 ”,“ 100 ”) are stored.
[0030]
Now, the article feature detection unit 102 takes out the number of characters and the number of paragraphs of the first generation unit 103-1 having the ID number “1” shown in the first row of the abstract generation condition table as shown in FIG. The number of characters and the number of paragraphs of the article body are compared (step S12). If both the number of characters and the number of paragraphs of the article body are smaller than the respective values of the abstract creation unit 1, “1” is returned and the process ends (step S13).
[0031]
If this condition is not met, the next line of the abstract generation condition table, that is, the number of characters and the number of paragraphs of the second generation unit 103-2 with the ID number “2” is extracted and the same processing is performed If the condition is met, “2” is returned and the process ends (steps S14 to S15). If this condition is not met, the ID number “3” of the third generation unit 103-3 is returned and the process ends (step S16).
[0032]
The abstract generation control unit 101 selects one of the first to third generation units corresponding to the number returned from the article feature detection unit 102, and supplies the article ID and the user profile to the selected generation unit. hand over.
[0033]
Next, processing operations of the first to third generation units (103-1 to 103-3) will be described.
First, the processing operation of the first generation unit 103-1 will be described. The first generation unit 103-1 takes out the first paragraph of the article from the article storage unit 51 using the article ID passed from the abstract generation control unit 101. For articles with a small number of characters and paragraphs, main information is often written in the first paragraph, and therefore processing for extracting the first paragraph of the article extracted from the article storage unit 51 is performed.
[0034]
Next, the processing operation of the second generation unit 103-2 will be described with reference to the flowchart shown in FIG. The second generation unit 103-2 first extracts and stores the text of the article from the article storage unit 51 using the article ID passed from the abstract generation control unit 101 (step S20). The similarity is calculated (step S21).
[0035]
Here, the paragraph similarity calculation processing operation will be described with reference to the flowchart shown in FIG.
First, paragraphs are extracted one by one in order from the first paragraph of the article text extracted from the article storage unit 51 (steps S30 to S33), and a desired user profile is obtained from the user profile storage unit 50 using the user profile ID. Then, the similarity between paragraphs is calculated using this user profile (step S34), and the result is stored (step S35).
[0036]
The calculation of the similarity of a paragraph can be realized by performing the similarity calculation processing (vector space method) performed on the entire article by the article search unit 52 described above for the paragraph.
[0037]
When the similarities are calculated for all the paragraphs and the results are stored (steps S36 and S31), the stored results are finally rearranged in the order of the similarity of the paragraphs (step S32). The paragraph similarity calculation result is stored in a predetermined storage area of the search result storage unit 53 assigned to the second generation unit 103-2, for example, as shown in FIG. As shown in FIG. 13A, the calculation result of the paragraph similarity is composed of the paragraph number, the paragraph similarity, and the paragraph contents, and is arranged in order from the highest paragraph similarity (FIG. 13). 13 (b)).
[0038]
Returning to the description of FIG. 11, the second generation unit 103-2 extracts the number of the paragraph with the highest similarity based on the paragraph similarity calculation result as shown in FIG. 13 (step S <b> 22). If the number of the paragraph is “1”, only the first paragraph is returned to the abstract generation control unit 101 (steps S23 and S25). Otherwise, both the stored first paragraph and the paragraph with the highest similarity are returned to the abstract generation control unit 101 (step S23, step S24).
[0039]
As described above, the second generation unit 103-2 performs processing for extracting both the first paragraph in which main information is written and the paragraph in which the user is highly interested (the highest similarity). . This is because, for example, in an article on the budget of a national or local government, information about the entire budget is written in the first paragraph, and each detail of the budget is written in the second and subsequent paragraphs. If a person is interested in multimedia, a paragraph about the budget allocation related to multimedia can be taken out by using the profile, and sent to the user together with the first paragraph. Accordingly, the user can obtain information on the entire article and information on his / her interest from the first paragraph.
[0040]
In addition to the first paragraph, it is easy to select a plurality of paragraphs from the paragraph having the highest similarity as well as the paragraph having the highest similarity.
Next, the processing operation of the third generation unit 103-3 will be described with reference to the flowchart shown in FIG.
[0041]
The third generation unit 103-3 takes out the article text from the article storage unit 51 using the article ID received from the abstract generation control unit 101 (step S40), and is similar to the paragraph using the user profile shown in FIG. The degree calculation process is performed to calculate the similarities of all paragraphs, and stored in a predetermined storage area of the search result storage unit 53 assigned to the third generation unit 103-2, for example, as shown in FIG. (Step S41).
[0042]
Based on the result, next, for example, the number of paragraphs having the largest similarity is extracted in order from the total number of paragraphs divided by 3 (an integer). By using a value obtained by dividing the total number of paragraphs by “3”, an abstract having a length of about 1/3 can be generated (step S42). For example, if the entire article consists of 14 paragraphs, 4 paragraphs are extracted from the paragraph with the highest similarity. In this way, the third generation unit 103-3 creates an abstract by extracting a paragraph that is considered to contain information of interest to the user when the entire article is long, such as an article in a magazine. Processing to do.
[0043]
The value that divides the total number of paragraphs is not limited to “3”, and may be another value. It is also easy to set a value designated in advance by the user.
The abstract generation control unit 101 receives the abstract generated by any one of the first to third generation units 103-1 to 103-3 (step S7 in FIG. 8), and the article rank for the article. The article headline, the abstract of the article, and the like are stored in a predetermined storage area of the search result storage unit 53, and a filtering result that is finally distributed to each user is generated.
[0044]
FIG. 15 shows a specific example of the filtering result generated by the abstract generation unit 100. As shown in FIG. 15, the filtering result is composed of an article rank, an article headline, and an abstract created by extracting a part of the paragraph from the entire article.
[0045]
The filtering result stored in the search result storage unit 53 with such a configuration is sent to the user by e-mail or FAX, for example, via the text information transmission / reception unit 7 and a predetermined communication network. The user can read the sent article and send it back to the information filtering center 1 with “O” or “X” in view of whether the article is of interest to the user. That is, the answer sent back from the user terminal via the predetermined communication network and the text information transmitting / receiving unit 7 is stored in the user answer storage unit 60.
[0046]
FIG. 16 shows a specific example of user answer data stored in the user answer storage unit 60. The user keeps the articles that are neither “○” for articles that are interested, or “x” for articles that are not interested. The user's answer is further used to modify the profile so that the user can select the articles he needs.
[0047]
The profile correction unit 61 stores the user stored in the user profile storage unit 50 from the user's answer as shown in FIG. 16 and the article text extracted from the article storage unit 51 using the article ID in the user answer. Modify the profile. As a method for correcting the profile, for example, the method described in “SMART information retrieval system” (edited by Gerald Salton, translated by Shinjin Koji, Planning Center) may be applied. In other words, this method counts the appearance frequency of words from articles that the user is interested in, and newly adds the word having the largest count value to the user profile.
[0048]
As described above, according to the above-described embodiment, the profile for each user stored in the user profile storage unit 50 is always corrected and adjusted based on the user's answer, and the article search unit using this profile. 52, the similarity with the article stored in the article storage 51 is calculated to search for an article including information desired by the user, and the abstract generation unit 100 also uses the profile for the searched article. By generating the abstract by calculating the similarity, the abstract including the information required by the user can be easily created.
[0049]
Further, when the abstract of the article searched by the abstract generation unit 100 is generated, first, the article feature detection unit 102 obtains the feature of the article, specifically, for example, the number of characters and the number of paragraphs. Select one of the third generation units 103-1 to 103-3 and select an abstract generation method suitable for the feature of the article, thereby creating an abstract including information required by the user Can be done easily.
[0050]
Furthermore, by distributing the abstracts generated in this way to each user, the time for each user to obtain information from the contents of the articles searched and distributed by information filtering is reduced, and labor is reduced. Can do.
[0051]
【The invention's effect】
As described above, according to the present invention, it is possible to provide an information filtering method and an information filtering apparatus that can easily create an abstract including information required by a user for an article searched by information filtering.
[Brief description of the drawings]
FIG. 1 is a block diagram showing a configuration of an entire system of an information filtering apparatus according to an embodiment of the present invention.
FIG. 2 is a diagram conceptually showing an operation mode of the system of FIG.
FIG. 3 is a block diagram showing a configuration of an information filtering device.
FIG. 4 is a diagram for explaining a user profile storage format and a specific example thereof.
FIG. 5 is a diagram for explaining a storage format of article data stored in an article storage unit and a specific example thereof.
FIG. 6 is a diagram for explaining a storage format of search results output from an article search unit and stored in a search result storage unit and a specific example thereof.
FIG. 7 is a diagram showing a specific example of an output form of a conventional filtering device.
FIG. 8 is a flowchart for explaining the processing operation of the abstract generation control unit.
FIG. 9 is a flowchart for explaining the processing operation of the article feature detection unit of the abstract generation unit.
FIG. 10 is a diagram for explaining a storage format of an abstract generation condition table referred to by an article feature detection unit and a specific example thereof.
FIG. 11 is a flowchart for explaining the processing operation of the second generation unit of the abstract generation unit.
FIG. 12 is a flowchart for explaining paragraph similarity calculation processing;
FIG. 13 is a diagram for explaining a storage format of paragraph similarity calculation results and a specific example thereof.
FIG. 14 is a flowchart for explaining a processing operation of a third generation unit of the abstract generation unit.
FIG. 15 is a diagram illustrating a specific example of an information filtering result generated based on an abstract generated by an abstract generation unit, stored in a search result storage unit, and distributed to each user.
FIG. 16 is a diagram showing a specific example of user answer data stored in a user answer storage unit;
[Explanation of symbols]
DESCRIPTION OF SYMBOLS 1 ... Information filtering center, 2 ... Information source, 3 ... User terminal, 4 ... Central processing part, 5 ... Memory | storage part, 6 ... Text information receiving part, 7 ... Text information transmission / reception part, 50 ... User profile storage part, 51 ... Article storage unit 52 ... Article search unit 53 ... Search result storage unit 60 ... User answer storage unit 61 ... Profile correction unit 100 ... Abstract generation unit 101 ... Abstract generation control unit 102 ... Article feature detection unit 103-1 ... 1st production | generation part, 103-2 ... 2nd production | generation part, 103-3 ... 3rd production | generation part.

Claims (4)

Receiving means for receiving articles including text and images distributed from a plurality of information sources;
  First storage means for storing articles received by the receiving means;
  Second storage means for storing a search condition in which a word for searching for an article related to a theme specified in advance by a user and a weight value thereof are specified;
  A computing means for extracting an article to be distributed to the user from the articles stored in the first storage means and performing an information filtering process for creating an abstract of the article;
  A distribution means for distributing the result of the information filtering process;
  In an information filtering method in a computer comprising:
  The computing means is
  (1) Using the number of appearances of the word of the search condition included in each article stored in the first storage means and the weight value of the word, the similarity of each article to the search condition is calculated A search step for searching for a plurality of articles ranked in descending order of the similarity;
  (2) For each article searched in the search step, a first group in which the number of characters and the number of paragraphs of the article body are less than a predetermined first threshold, which is equal to or more than the first threshold and the first Classifying into any one of a second group less than a second threshold greater than a threshold and a third group greater than or equal to the second threshold;
  (3) a first abstract creation step of extracting the first paragraph as an abstract for the articles classified into the first group;
  (4) For each paragraph of the article classified into the second group, the similarity to the search condition is calculated using the number of appearances of the word of the search condition included in the paragraph and the weight value of the word. A second abstract creation step for extracting the first paragraph and at least the paragraph with the highest similarity from the article as an abstract,
  (5) For each paragraph of the article classified into the third group, the similarity to the search condition is calculated using the number of appearances of the word of the search condition included in the paragraph and the weight value of the word. A third abstract creating step for extracting a predetermined number of paragraphs from the article in descending order of the similarity,
  Run
  A delivery step in which the delivery means delivers the rank of each article searched in the search step and the abstract created in any of the first to third abstract creation steps;
  Run
An information filtering method characterized by the above.   The computing means is
  Of the articles distributed as abstracts in the distribution step, the article that the user replied with interest has the step of adding the word with the highest appearance frequency contained therein to the search condition. The information filtering method according to claim 1.   Receiving means for receiving articles including text and images distributed from a plurality of information sources;
  First storage means for storing articles received by the receiving means;
  Second storage means for storing a search condition in which a word for searching for an article related to a theme designated in advance by a user and its weight value are specified;
  Using the number of appearances of the word of the search condition included in each article stored in the first storage means and the weight value of the word, the similarity to the search condition of each article is calculated, and the similarity A search means for searching a plurality of articles ranked in descending order;
  Each article searched by the search means is a first group in which the number of characters and the number of paragraphs of the article body are smaller than a predetermined first threshold, which is equal to or more than the first threshold and is lower than the first threshold. A second group that is less than a large second threshold and a third group that is greater than or equal to the second threshold Means for classifying any one of
  A first abstract creating means for extracting the first paragraph as an abstract for the articles classified into the first group;
  For each paragraph of the article classified into the second group, the similarity to the search condition is calculated using the number of appearances of the word of the search condition included in the paragraph and the weight value of the word, A second abstract creation means for extracting the first paragraph and at least the paragraph with the highest similarity from the article as an abstract;
  For each paragraph of the articles classified into the third group, the similarity to the search condition is calculated using the number of occurrences of the word of the search condition included in the paragraph and the weight value of the word, A third abstract creation means for extracting a predetermined number of paragraphs from the article in descending order of the degree of similarity;
  Distribution means for distributing the rank of each article searched by the search means and the abstract created by any one of the first to third abstract creation means;
  An information filtering apparatus comprising:   Of the articles distributed as abstracts by the distribution means, the articles that the user replied with interest are further provided with means for adding the word with the highest appearance frequency contained therein to the search condition. 4. The information filtering device according to claim 3, wherein

Images