JP2008040985A - Document information extracting device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a document information extracting device capable of guessing future information streams by utilizing information used in the past subject extraction. <P>SOLUTION: An information upheaval detecting device 10 comprises: a processing object document collecting part 100 for collecting document information having time information; a word recognizing part 102 for performing a morpheme analysis of the document information; a syntax recognizing part 106 for performing modification analyzing of the document information; a storing part 116 for storing a stream pattern of information constituting a subject; an information stream managing part 114 for detecting words constituting the subject and a stream of modification relations from analysis results of the document information; and an information stream guessing part 118 for comparing the detected stream with a stream appearance pattern of the storing part and guessing the words constituting the subject or a modification relation on the basis of a stream appearance pattern selected from comparison results. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a technique for extracting and utilizing information bulges or currents such as words constituting a topic and dependency relations from a set of digitized document information.

  With the development of the Internet and information communication, it is possible to obtain a large amount of digitized document information. In order to easily extract or retrieve desired information from a large amount of document information, a method of classifying document information into a predetermined category or assigning similar document information to a category is used.

  Patent Document 1 discloses a technique for classifying documents for each category and viewing the transition of the number of documents for each category, regarding an information flow presentation method and apparatus. In this method, the category corresponds to the topic.

  Patent Document 2 relates to a topic change extraction method, which selects an important word from each document in a document set having time information according to the appearance frequency (TF * IDF, etc.) and extracts a dependency including the important word. The topic change is extracted by a vector obtained from a keyword.

JP-A-10-154150 JP 2004-185135 A

  However, conventional document information extraction has the following problems. Conventional document information extraction devices include a function that extracts topics from document information and informs the user of this, but accumulates information flows and transition information that have been used for topic extraction in the past. It has not been used. Also, if information trends are extracted using only words such as keywords, topics with completely different contexts may be identified. Furthermore, even if a topic is informed, if the user has no knowledge about the topic, after all, the user knows the topic and then asks the person or searches a search engine to start the topic. , You have to examine the progress.

  The present invention solves the above-described conventional problems, and provides a user-friendly document information extraction apparatus that can use information used for past topic extraction to estimate future information trends. The purpose is to do.

  The document information extraction apparatus according to the present invention includes a document information collection unit that collects document information having time information, an analysis unit that analyzes at least one of syntax analysis and semantic analysis, and information included in the document information A storage means for storing a plurality of appearance patterns of information constituting a topic, and selecting a specific appearance pattern from the plurality of appearance patterns stored in the storage means based on the analysis result of the document information by the analysis means, And inference means for inferring information constituting the topic based on the specific appearance pattern. Thereby, the information which comprises a topic can be estimated using the accumulated appearance pattern, and this can be notified to a user.

  Preferably, the information constituting the topic is a word having a certain appearance frequency in a set of document information or a dependency relationship. A word or dependency relationship with a certain frequency of appearance means a rise of information or a trend of information. In particular, the use of the dependency relationship improves the extraction accuracy of topics having the same identity.

  Preferably, the document information extraction apparatus further includes a management unit that extracts an appearance pattern of information constituting a topic from the document information analyzed by the analysis unit, and stores the extracted appearance pattern in the storage unit. The management unit compares the appearance pattern extracted from the document information with the appearance pattern stored in the storage unit, and when the two patterns are determined to be the same or similar, the management unit weights the stored appearance pattern. By performing the weighting, it is possible to improve the accuracy of the appearance pattern to be selected.

  Preferably, the inference means selects an appearance pattern similar to the appearance pattern extracted by the management means, and predicts the appearance of information constituting the topic according to the selected appearance pattern. Preferably, the appearance pattern includes any of an appearance interval of words, an appearance interval of dependency relations, or an appearance interval of feature vectors of words included in a document. Preferably, the inference means outputs a word or dependency relationship included in the selected appearance pattern as an inference result. This allows the user to anticipate future topics.

  The document information extraction program according to the present invention analyzes a document information by either a step of preparing a storage means for storing an appearance pattern of information included in the document information and constituting a topic, and a semantic analysis or a syntax analysis. A step, a step of extracting a specific appearance pattern from a plurality of appearance patterns based on the analysis result of the document information, a step of inferring information constituting the topic based on the extracted appearance pattern, and an inference result are displayed. Steps.

  According to the present invention, since the information constituting the topic that will appear next is inferred based on the appearance pattern of the information constituting the topic, the information used for topic extraction in the past can be utilized. The user can know information about the next topic from the currently processed document information.

  DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, exemplary embodiments of the invention will be described with reference to the drawings.

  FIG. 1 is a diagram illustrating a configuration example of an information bump detection apparatus according to an embodiment of the present invention. The information ridge detection device 10 according to the present embodiment extracts important words and dependency relationships that form topics from a set of document information having time information, and the amount of information of such words and dependency relationships is extracted. It is possible to extract a peak point, in other words, an uplift or tide of information, and to infer information constituting a future topic from the transition pattern of the tide of information. Furthermore, the information ridge detection device 10 can narrow down the appearance context of words using the dependency analysis result, and after presenting the information flow, present the progress from the beginning of the topic to the user. Enable.

  The information ridge detection device 10 includes an input device 12 including a keyboard, an image scanner, and other inputs, an external interface (I / F) 14 that enables transmission and reception of information to and from an external network, an output from a display device 16, a printer, and the like. The apparatus 18 includes a storage device 20 that can store various data, a program memory 22 that stores a program, and a CPU (Central Processing Unit) 24 that can control the operation of each unit in accordance with the program.

  The program memory 22 detects a bulge or tide of information constituting a topic from electronic document information, stores a tide of detected information, or infers information constituting a future topic or topic Stores the detection program. The CPU 22 controls each unit in accordance with such an information bump detection program.

  FIG. 2 shows a functional block diagram of the information bump detection device. The information ridge detection apparatus 10 includes a processing target document collection unit 100 that inputs and collects a set of digitized text information, a word recognition unit (morpheme analyzer) 102 that performs morphological analysis on a document and recognizes words, and is subjected to morphological analysis. A document-to-word database 104 that associates and stores a word and a document, a syntax recognition unit (a dependency analyzer) 106 that performs dependency analysis of the document and recognizes the syntax, and a dependency relationship and document that have been subjected to dependency analysis. The document dependency database 108 to store in association, the document management unit 110 to manage the input document, the document database 112 to store the document, the document management unit 110, or the information flow based on the information from the document-to-word database 104. An information flow management unit 114 that performs detection and the like, and a current flow pattern based on the information flow detected by the information flow management unit 114 An information flow inference unit 118 that infers information flows based on the detection result of information flow by the information flow management unit 114 and presents inferences or predicted information flows to the user. A multi-document summarizing unit 120 that presents the user with a summary of the tidal history based on the detection result of the tidal current of information by the tidal current management unit 114 is configured.

  Next, the operation of the information flow detection unit will be described with reference to the operation flow of FIG. First, the processing target document collection unit 100 collects a set of document information to be processed (step S101). The collection of document information is a document collected from the Internet at regular intervals, such as a blog, a Web (Web) log, an HTML document, a document in an intranet, and an email. These documents are accompanied by a creation date attribute.

  Next, the word recognizing unit 102 takes out text in each document from a set of periodically collected document information, performs morphological analysis processing (step S102), performs word segmentation, part of speech identification, and the like. The result of the morphological analysis process is stored in the document-to-word database 104.

  Next, the syntax recognizing unit 106 performs dependency analysis processing (step S103), and according to the dependency grammar created by manual or corpus learning, for example, between words such as “dismissing the House of Representatives” and “rescue the citizen” A dependency relationship is extracted, and a plurality of dependency relationships are acquired from one document. In addition, in order to determine an important one among the extracted dependency relationships, the subject matter is extracted from context analysis or document structure, or the importance of the word is obtained by TF * IDF of the word frequency. It may be reflected in the dependency relationship. In addition, each of the extracted dependency-related words is normalized if necessary using a synonym dictionary, a regular expression dictionary, a thesaurus, or the like. If the same dependency relationship is recognized in the normalized range, it is counted as the appearance frequency of the relationship. The result of dependency analysis processing by the syntax recognition unit 106 is stored in the document pair dependency database 108.

  The document management unit 110 manages document information subjected to morphological analysis and / or dependency analysis, and the processed document information is stored in the document database 112. Next, the information flow management unit 114 detects an uplift or flow of information based on the document supplied from the document management unit 110 (step S104). An information bulge or tidal current is one in which the amount of information increases over time or changes in the amount of information over time. Here, the dependency relationship is mainly described as information.

  The information flow management unit 114 receives the analyzed document information from the document management unit 112, and detects the flow of dependency-related information therefrom. At this time, the information flow management unit 114 may receive the document and the corresponding dependency relationship from the document dependency dependency database 108. When detecting a tide of morphologically analyzed word information, a document and a corresponding morphological analysis result may be received from the document-to-word database 104. Further, the document management unit 112 may be configured so that the user can set a period and a range in which information flow should be detected.

  FIG. 4 shows an example in which document information is collected every day for a period of two weeks and three dependency relationships (topics) change. That is, the transition of the frequency at which the dependency relationship A, the dependency relationship B, and the dependency relationship C appear during the period from October 1 to October 14 is shown for each day. For example, the dependency relationship A is an expression “a does b”, the dependency relationship B is an expression “c does d”, and the dependency relationship C is “e does f”. ".

  In the example of FIG. 4, when the appearance frequency of the dependency relationship is 10 or more and it continues for 2 days or more, it is set as a condition for identifying it as a topic. It is October 3rd and 4th for the dependency relationship A, and October 6th, 7th and 8th for the dependency relationship B, and October for the dependency relationship C. 10, 11, 12 days. In the figure, the frequency of appearance is indicated by hatching.

  When the information flow management unit 114 detects the flow, the information flow management unit 114 displays the information flow or elevation in a graph as shown in FIG. The vertical axis of the graph is the number of articles (appearance frequency), the horizontal axis is time, and the trend of topics A, B, and C is displayed. Topic A, Topic B, and Topic C correspond to dependency relationships A, B, and C, respectively.

  Preferably, a scroll bar 150 for changing the granularity of topics displayed on the graph is displayed as a user interface. If the topic granularity is made fine, the threshold value of the above-described condition is lowered and a large number of topical tides are displayed. Conversely, if the topic granularity is coarsened, the threshold value is increased and a small number of topical tides are displayed.

  In addition, when the user clicks on a graph of a desired topic, the information flow management unit 114 may search for the topic history from the Internet or the like. If the user does not have knowledge about the topic, the beginning and progress of the topic are searched. For example, when the user clicks on the topic A, the information flow management unit 114 executes a search using the dependency-related words constituting the topic A as keywords.

  The information flow management unit 114 executes a topic history search, and then the multiple document summarization unit 120 displays a summary of the topic history search results on the display. FIG. 6A shows a search result of the history of the topic A, and the multi-document summarizing unit 120 summarizes and outputs it at a constant cycle based on the document information searched for the topic A. For example, a summary is output every day from 10/01 to 10/05. On 10/01, a summary of a plurality of articles written on topic A is displayed. When the user further clicks on the summary from the summary display state, the multi-document summary unit 120 can display the entire linked original document contents as shown in FIG.

  Next, the information flow management unit 114 extracts a dependency-related power flow appearance pattern included in the document information based on the analysis result of the document information (step S106). For example, taking FIG. 4 as an example, what is detected as a tidal current is the receiving relationship A, the dependency relationship B, and the dependency relationship C. As shown in FIG. 7, the dependency relationships A, B, and C have elapsed times 160 and 162 between the dependency relationships. Preferably, the tidal current appearance pattern is an appearance interval considering the elapsed times 160 and 162. It is. The elapsed times 160 and 162 are, for example, 2 days.

  The tidal current appearance pattern is not necessarily limited to the appearance interval using the dependency relationship, and may be a word or a word vector using a morphological analysis result. For example, as shown in FIG. 8 (a), the word A, the word B, and the word C may be simply arranged, or as shown in FIG. 8 (b), the words A, B, It may be an appearance interval of words including elapsed times 170 and 172 between C and each word. Furthermore, a word vector may be used. For example, as shown in FIG. 8C, a vector formed from a document space including the word A on 10/03, 10/04, elapsed time 180, 10 It is also possible to set a vector formed from a document space including the word B at / 06 to 10/08, an elapsed time 182 and an appearance interval of a vector formed from the document space including the word C from 10/10 to 10/12. it can.

  Next, the information flow management unit 114 stores the power flow appearance pattern extracted from the document information in the power flow appearance pattern database 116 (step S107). The information flow management unit 114 determines whether the same pattern as the extracted current flow appearance pattern is stored. If a matching tidal current appearance pattern is already stored, the importance of the tidal current appearance pattern is increased. For example, as weighting, the number of matches is counted, and the count result is stored in association with the tidal current appearance pattern.

  Further, the similarity between the tidal current appearance pattern extracted from the document information and the stored tidal current appearance pattern may be calculated and reflected in the importance. For example, the similarity of patterns is “1” if the word relationship is completely the same (“A”, “=” and “B” all match if the dependency relationship “A is B”). Otherwise, it may be “0”, or a word similarity, a case particle difference score, or the like may be introduced. Also, taking into account the elapsed time between tidal currents, measures such as lowering the score as time opens between the two patterns may be taken.

  Furthermore, if the degree of similarity between patterns is high, both patterns may be merged to increase the importance. For example, as shown in FIG. 9, when there are a tidal current appearance pattern 190 composed of topic A, topic B, and topic C and a tidal current appearance pattern 192 composed of topic B, topic C, and topic D, a common topic from each pattern A tidal current appearance pattern 194 consisting of B and topic C may be generated, and the importance of the tidal current appearance pattern 194 may be increased and stored. On the other hand, if there is nothing similar to the accumulated tidal current appearance pattern, the tidal current appearance pattern extracted from the document information is accumulated in the database 116 as a new tidal current appearance pattern.

  Next, when receiving the detection result of the tidal current by the information tidal current management unit 114, the information tidal current inference unit 118 compares it with a plurality of tidal current appearance patterns stored in the tidal current appearance pattern database 116. A similar tidal current appearance pattern is selected (step S108), and this is read out. Next, the information flow inference unit 118 infers the flow of information that will appear in the future based on the read out current flow pattern (step S109). The inferred information is displayed on the display (step S110). The user can know the next topic from the trend of the displayed information, that is, from words or dependency relationships.

  For example, when a keyword at the beginning of a tidal current appearance pattern is detected as a tidal current, the information tidal current inference unit 118 refers to the selected tidal current appearance pattern and informs the user of the keyword predicted to appear next as a forecast. . For example, as shown in FIG. 10, if the word B is detected as a tide, the word C is inferred as the next tide, and as a keyword or sentence including the word C, “the topic of the word C will boom in the near future. The expected output 200 is displayed.

  For example, as shown in FIG. 10B, the information flow inference unit 118 also detects the dependency relationship B and the dependency that are predicted to appear next when the dependency relationship A at the beginning of the tidal current appearance pattern is detected. It is also possible to inform the user of the relationship C. Thereby, the user can know the tide (topic) that will occur next to the tide (or topic) extracted at the present time from the transition pattern of the tide of information accumulated in the past. Note that the prediction to be presented to the user only needs to include at least the words and dependency relationships that make up the current, and as shown in FIGS. 10 (a) and 10 (b), the sentence including the words and dependency relationships is predicted output. May be given to the user as

  Next, a second embodiment of the present invention will be described with reference to FIG. In the second embodiment, a document classification unit 130 and a document classification database 132 are newly added to the configuration of FIG. 2 shown in the first embodiment. Other configurations are the same as those in the first embodiment. Similar to the example.

  In the figure, a document classification unit 130 classifies a set of sentence information to be processed into categories. In the classification method, classification is made into a predetermined category, or similar document information is assigned to the category. The document information classified into the categories is stored in the document information database 132. Here, the information flow management unit 114 detects a flow included in the document information of a specific category, and provides the detection result to the information flow inference unit 118. Note that the information flow management unit 114 may select a specific category in accordance with an input from the user.

  The information flow inference unit 118 estimates the information flow in a specific category and informs the user of this. By classifying the document information by category, the user can know the topics in the selected specific category.

  The preferred embodiments of the present invention have been described in detail above. However, the present invention is not limited to the specific embodiments according to the present invention, and various modifications can be made within the scope of the gist of the present invention described in the claims. Deformation / change is possible.

  The present invention is used for an information collecting apparatus that selectively collects and classifies information such as sent opinions and inquiries represented by web pages scattered on the Internet.

It is a figure which shows the example of 1 structure of the document bulge detection apparatus based on the Example of this invention. It is a functional block diagram of the information uplifting device concerning the 1st example of the present invention. It is a figure which shows the operation | movement flow of the information raising apparatus which concerns on a 1st Example. It is a figure which shows the example which three dependency relationships changed. It is the figure which detected the information flow from the transition of FIG. 4, and showed it on the graph. FIG. 6 is a diagram for searching a topic history from the graph of FIG. 5 and displaying a summary thereof. It is a figure which shows the tidal current appearance pattern of a dependency relationship. 8A shows a tidal appearance pattern of words, FIG. 8B shows a tidal appearance pattern when the appearance interval is considered in FIG. 8A, and FIG. 8C shows a tidal appearance of word vectors included in the document. It is a figure which shows a pattern. It is a figure which shows the example which unites both patterns when the tidal current appearance pattern extracted from document information and the stored tidal current appearance pattern are similar. It is a figure which shows the keyword or sentence estimated from a tidal current appearance pattern. It is a functional block diagram of the information bump detection apparatus which concerns on 2nd Example of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10: Information bump detection apparatus 100: Processing object document collection part 102: Word recognition part 104: Document versus word database 106: Syntax recognition part 108: Document dependency database 110: Document management part 112: Document database 114: Information flow management part 116: Tidal current appearance pattern database 118: Information tidal current reasoning unit 120: Multiple document summarizing unit 130: Document classification unit 132: Document classification database

Claims (10)

Document information collecting means for collecting document information having time information;
An analysis means for analyzing document information by at least one of syntax analysis and semantic analysis;
Storage means for storing a plurality of appearance patterns of information included in document information and constituting a topic;
Inference means for selecting a specific appearance pattern from a plurality of appearance patterns stored in the storage means based on an analysis result of document information by the analysis means, and inferring information constituting a topic based on the specific appearance pattern;
A document information extracting apparatus having The document information extracting apparatus according to claim 1, wherein the information constituting the topic is a word having a certain appearance frequency or a dependency relationship in a set of document information. The document information extracting apparatus further includes a managing unit that extracts an appearance pattern of information constituting a topic from the document information analyzed by the analyzing unit, and stores the extracted appearance pattern in the storage unit. Document information extraction device. The management means compares the appearance pattern extracted from the document information with the appearance pattern stored in the storage means, and weights the stored appearance pattern when it is determined that both patterns are the same or similar. 3. The document information extraction device according to 3. The inference means selects an appearance pattern similar to the appearance pattern extracted by the management means, and predicts the appearance of information constituting a topic according to the selected appearance pattern. Document information extraction device described. The document information extraction according to any one of claims 1 to 5, wherein the appearance pattern includes any one of an appearance interval of words, an appearance interval of dependency relations, or an appearance interval of a feature vector of a word included in a document. apparatus. The document information extraction apparatus according to claim 1, wherein the inference means outputs a word or dependency relationship included in the selected appearance pattern as an inference result. An extraction program for extracting information constituting a topic from document information,
Providing a storage means for storing an appearance pattern of information included in document information and constituting a topic;
Analyzing document information by either semantic analysis or parsing;
Selecting a specific appearance pattern from a plurality of appearance patterns based on the analysis result of the document information;
Inferring information comprising the topic based on the selected occurrence pattern;
Displaying inference results;
Document information extraction program having 9. The document information extraction program according to claim 8, further comprising a step of writing an appearance pattern extracted from the analysis result of the document information to the storage unit. The writing step compares the appearance pattern extracted from the document information with the appearance pattern stored in the storage unit, and weights the stored appearance pattern when it is determined that both patterns are the same or similar. 9. The document information extraction program according to 9.

Images