CN108415900A - A kind of visualText INFORMATION DISCOVERY method and system based on multistage cooccurrence relation word figure - Google Patents

Abstract

The present invention provides a method for discovering visual text information based on a multi-level co-occurrence relationship word graph, the steps of which include: extracting the text content of a document, segmenting the text content to obtain text segments; segmenting the text segments, and extracting key words word, and mark the word category label; construct a multi-level co-occurrence relationship word graph according to the co-occurrence relationship of keywords in text fragments, the nodes in the graph correspond to keywords, and the edges in the graph correspond to keyword co-occurrence; Each keyword constructs a word-document inverted index to retrieve documents containing keywords; obtain visual text information through co-occurrence relationship word graphs. The present invention also provides a visual text information discovery system based on a multi-level co-occurrence relationship word graph, including a document preprocessing module, a keyword extraction module, a multi-level word graph construction module, a word-document index construction module and a visual information discovery module .

Description

A method for visual text information discovery based on multi-level co-occurrence relationship word graph and its system

technical field

The invention belongs to the fields of text mining and natural language processing, and relates to a method and system for discovering visual text information based on a multi-level co-occurrence relationship word graph.

Background technique

With the development of the Internet and office electronics, text information has shown an explosive growth trend, and the amount of text generated has surpassed any previous era. On the one hand, texts contain a lot of valuable information, on the other hand, massive texts significantly increase the discovery cost of effective information. For the vast majority of applications (such as publishing, research, and supervision), it is impossible for users to read every document in the collected documents to find effective information. How to use computers to assist in mining valuable information from massive texts ( Text mining) has become an important problem to be solved urgently.

According to the characteristics of target information, text mining can be divided into two categories: the first category is text mining where effective information can be clearly defined, such as classification or search with clear goals, and existing computers can basically meet daily needs through matching calculations; the second category It is text mining where effective information is difficult to clearly define, such as scenes with vague search requirements. Existing methods generally use "exploratory" methods for information discovery. The bottom layer of "exploratory" information discovery uses the search function: the user enters query words, manually checks the search results, forms the next query word and continues to search, and this process is repeated until the result is found. For "exploratory" information discovery, as the user understands the results, the final query used is likely to be completely different from the original query.

At present, there are three problems in the "exploratory" information discovery method: first, the efficiency of manual inspection of search results is low, and manual browsing of documents (search results) is a very time-consuming process, and the target information cannot be quickly located; second, the entire process lacks The global control of the target document collection causes users to often fall into the problem of not knowing where to come from and where to go during the discovery process, and the state of information inspection cannot be recovered and effectively used in the next inspection; Checked documents are filtered, making it difficult to avoid double checking.

Contents of the invention

In order to overcome the above shortcomings of information discovery, the present invention proposes a method and system for visual text information discovery based on a multi-level co-occurrence relationship word graph.

In order to solve the problems of the technologies described above, the present invention adopts the following technical solutions:

A method for discovering visual text information based on a multi-level co-occurrence relationship word graph, as shown in Figure 1, the steps include:

Extract the text content of the document, segment the text content, and obtain text fragments;

Segment text fragments, extract keywords, and mark word category tags;

Analyze the text fragments, construct a multi-level co-occurrence relationship word graph according to the co-occurrence relationship of keywords in the text fragments, the nodes in the graph correspond to keywords, and the edges in the graph correspond to keyword co-occurrence;

Build a word-document inverted index for each keyword in the graph to retrieve documents containing keywords;

Obtain visual text information through co-occurrence relationship word graph.

Further, before extracting the text content of the document, the document is format parsed first.

Further, use symbols for segmentation, the symbols include punctuation marks; or use a fixed window for segmentation, set the size of the window and the moving step, move from the beginning to the end of the text, and each text segment enclosed by the window is used as output.

Further, word category tags include part-of-speech tags, entity word tags, document core word tags, semantic role tags, and custom type tags.

Further, the entity word tags include compound entity words.

Further, for the document core word tags, the method of finding the document core words includes using TF-IDF or TextRank to calculate word weights, sort keywords based on word weights, and take the Topk keywords with the highest ranking as document core words.

Further, the co-occurrence relationship of keywords includes co-occurrence in the same text segment, co-occurrence in N adjacent text segments, and co-occurrence in the entire document.

Further, for a pair of keywords, it can only exist in a single co-occurrence relationship word graph with the closest co-occurrence relationship. Fragment co-occurs, co-occurs throughout the document.

Further, the method of obtaining visual text information through the co-occurrence relationship word graph is shown in Figure 2, including: online browsing of the global graph and local graph, selection browsing and extended browsing of the local graph, switching display and parallel display of the co-occurrence relationship , word graph browsing history, word node markers, and document markers.

The online browsing of the global graph and the local graph refers to: the global graph provides a display function for all words, and the user can use this function to form a general overview of the document set; the local graph provides the adjacent word nodes of the selected word node The display function, users can use this function to browse the key areas of the document set. For different co-occurrence windows, the display content of the graph is different. The functions of the global graph and the local graph are realized by displaying word graph information loaded offline by the front end on demand.

The selection browsing and extended browsing of the partial graph refer to: the selection browsing includes carrying out full-text search to the words in the overall graph, selecting the word of interest, showing the partial graph centered on the word, including searching the word in the graph according to the word type label. Nodes are selected for browsing; extended browsing means that the user can click on a neighbor node in the partial graph, and the partial graph is automatically updated to a partial graph centered on the neighbor node.

The switching display and side-by-side display of the co-occurrence relationship refer to: the switching display supports the user to take a word as the center, and loads different partial diagrams by selecting different co-occurrence levels (window size); the side-by-side display supports the user to take a word as the center , to display the partial graphs under different co-occurrence levels side by side. Switching display and side-by-side display allow users to flexibly view the context of words and find relevant clues.

The word map browsing history refers to: the user is in the process of expanding browsing, the system will record the points and related paths that the user has clicked, and the path is saved in a graph structure, and the subsequent user can load and search the historical path for easy recall and resume checking status.

The term node mark and document mark refer to: during the browsing process, the user can mark the word node and related documents. It includes two types of marks: one is the collection mark, the marked nodes and related documents can be checked later by the user; the other is the deletion mark, the marked nodes and related documents will be deleted from the document set, and the corresponding multi-level common The existing relation graph will also be updated.

A visual text information discovery system based on multi-level co-occurrence relationship word graph, as shown in Figure 3, including document preprocessing module, keyword extraction module, multi-level word graph building module, word-document index building module and visual information Discovery modules.

Document preprocessing module: the input of this module is a collection of document files, and the output is a collection of <document number, list of text fragments>. The processing of each document file includes parsing the format of the file, extracting the text content therein, and segmenting the entire text according to predefined rules to obtain an ordered list of text fragments.

Keyword extraction module: This module uses the output of the document preprocessing module as input, numbers each text segment, and further cuts the text segment to obtain a set of <word, word category>. The tagging of word categories can be done using related tools of natural language processing, or by user-defined processing.

Multi-level word graph construction module: This module uses the output of the keyword extraction module as input to construct a multi-level co-occurrence relationship word graph. Multi-level refers to the use of different window sizes to examine the co-occurrence of words, thereby generating multiple co-occurrence relationship word graphs. For example, co-occurrence in the same text segment, co-occurrence in N adjacent text segments, co-occurrence in the same document, etc.

Word-document index building module: This module builds a word-document inverted index for each word in the word graph, which is used to retrieve documents containing words.

Visual information discovery module: This module provides document browsing and discovery functions based on word categories and word co-occurrence relationship word graphs, provides document marking functions, provides state saving functions for traversing word graphs, and realizes browsing information of interest from multiple perspectives Find.

The method of the present invention conducts visual information discovery for a given document set, first uses natural language processing technology to segment and filter documents to form a keyword set, and then uses windows of different sizes to investigate the co-occurrence of words to construct multi-level co-occurrence relationship words Graph, the co-occurrence relationship word graph is also called word graph; users can discover visual information by browsing the word graph; visual information discovery supports users to search for words in the word graph; supports selecting a word center, through the co-occurrence relationship View related words; support key checks on documents containing selected words, support deletion of word nodes to delete related documents and update co-occurrence relationship word graphs, and support saving paths for users to traverse word graphs.

Using the word map to check information can improve the efficiency of document checking. The word map is equivalent to providing a summary of the content of the document; using the word map co-occurrence relationship can easily perform extended inspections, and recording the user's word map traversal path can help users control the inspection progress; Marking word nodes for deletion can reduce the number of subsequent document checks and avoid repeated checks.

The method of the present invention is flexible and convenient, which is reflected in the size of the text fragments obtained by adjusting the size of the custom window, and the word associations obtained are different if the text fragments are different in size; keywords can be customized, and which words and word categories to extract can be determined according to the found needs to be determined.

Description of drawings

Fig. 1 is a flow chart of a method for discovering visual text information based on a multi-level co-occurrence relation graph.

Fig. 2 is a schematic diagram of text visualization information discovery function.

Fig. 3 is a diagram of a visual text information discovery system based on a multi-level co-occurrence relation word graph.

Fig. 4 is a schematic diagram of document preprocessing and keyword extraction.

Fig. 5 is a schematic diagram of the co-occurrence information used by the multi-level word graph building block.

Fig. 6 is a window co-occurrence graph-global graph.

Fig. 7 is a two-window co-occurrence map-global map.

Fig. 8 is a window co-occurrence map-partial map ("Tang Dechuan" as the center).

Figure 9 is a two-window co-occurrence map-partial map ("Tang Dechuan" as the center).

Fig. 10 is a schematic diagram of extended browsing (the central word is from "Tang Dechuan" to "profitable enterprise").

Detailed ways

In order to make the above-mentioned features and advantages of the present invention more comprehensible, the following specific embodiments are described in detail in conjunction with the accompanying drawings.

This embodiment provides a method for discovering visual text information based on a multi-level co-occurrence relationship word graph, and performs information discovery on a document collection, the document collection contains 2 documents, as shown in Figure 1, the method steps include:

1. Document preprocessing:

For each document in the document set, output <document number, list of text fragments>. The specific processing process includes: (1) parsing the format of the document to extract valid text content; (2) segmenting the text content, and the segmented text fragments generally correspond to meaningful semantic units; the following two types can be used for segmentation Method: (a) use symbols for segmentation, and the symbols are specified by the user. These symbols include common punctuation marks, such as periods, commas, line breaks, paragraph indents, etc.; (b) use fixed window segmentation, set the window Two parameters, size and moving step, move from the beginning to the end of the document, and each text fragment enclosed by the window is output.

For the text content segmentation in this example, use method (a) and select comma as the delimiter to segment the document to obtain a sentence set. The result of document preprocessing is shown in Figure 4.

2. Keyword extraction:

For each text segment of each document, this step numbers the text segment and segments the text segment to obtain a <word, word category> list. Word category labels are determined by users according to their needs, and can be extracted using related natural language processing toolkits. Commonly used word category tags can include: (a) part-of-speech tags, such as nouns, verbs, etc.; (b) entity word tags, such as time, place, person name, organization name, etc. Entities also include compound entities, which are composed of multiple words The new entity referred to later, such as "Group Commendation Meeting", where "Group" and "Commendation Meeting" are entity words respectively, and the combination of the two refers to the new entity; (c) document core word tags, the implementation method includes using TF- IDF or TextRank calculates word weights, sorts words based on word weights, and takes the top-ranked Topk words as core words; (d) Semantic Rolelabeling, such as beneficiary, condition, purpose, reason, etc.; (e ) self-defined type, which can be post-processed based on the result of syntactic analysis, such as the subject, predicate, object, etc. obtained by OpenIE.

For this example, the word category tags of "noun, compound entity, person name, place name, organization name" are reserved, and information discovery is performed on documents based on these category words. The results of keyword extraction are shown in Figure 4. For example, for the sentence "When Tang Dechuan praised the southern district at the group commendation meeting", the sequence of three words and word categories of "Tang Dechuan/person's name", "group commendation meeting/compound entity", "South district/place name" was extracted .

3. Multi-level word graph (i.e. co-occurrence relationship word graph) construction:

Word graph nodes use the words output in step 2, and word graph edges are determined by the co-occurrence relationship of words. Multi-level refers to the use of different window sizes to examine the co-occurrence of words, thereby generating multiple co-occurrence relationship word graphs. For example, co-occurrence in the same text segment, co-occurrence in N adjacent text segments, co-occurrence in the entire document, etc.

For a pair of specific words, it is required that they can only appear in a single word graph, which is the co-occurrence relationship word graph corresponding to the smallest window where the keyword pair appears. The edges of words obtained through co-occurrence can also be filtered and deleted, and the filtering rules are determined by the user according to needs.

For this example, two levels of co-occurrence relationships are used: co-occurrence in the same window, co-occurrence in two adjacent windows, the window unit is a sentence, and the corresponding generated word graphs are called "one-window co-occurrence graph" and "co-occurrence graph" respectively. Two-window co-occurrence map". The obtained co-occurrence combination of words and words is shown in Figure 5, which is reflected in the word graph as connected edges. Specifically, taking co-occurrence in the same window as an example, the three words ["Tang Dechuan/person's name", "Group commendation meeting/compound entity", "South District/place name"] appear in the same sentence, then through this sentence, The edges of the obtained word map are the pairwise combinations of these three words, namely <Tang Dechuan, Group Commendation Meeting>, <Tang Dechuan, South District>, <Group Commendation Meeting, South District>.

Taking the co-occurrence of two adjacent windows as an example, the words in word list 1 ["Tang Dechuan/person's name", "group commendation meeting/compound entity", "South District/place name"] and word list 2 ["South District/place name ", "Representatives in the Southern District Group/Composite Entity"] co-occur in the two windows, then the words in the word list 1 and the words in the word list 2 can be combined in pairs to obtain the co-occurrence graph of the two windows even side. Note here that <Tang Dechuan, South District> and <Group Commendation Meeting, South District> appear in the "one-window co-occurrence map", according to "for a pair of specific words, it is required that they can only appear in a single word map", Therefore, these two connected edges are deleted in the "two-window co-occurrence graph".

4. Word-document index construction:

For each word in the word graph, build a word-document inverted index for retrieving documents containing words.

Through steps 1-4, the multi-level co-occurrence relationship word graph and the data structure of the inverted index are generated, and the subsequent visual information discovery is completed by searching and loading the data structure on demand.

5. Visual information discovery, the core functions include:

1) Online browsing of global and local maps.

The global graph provides a display function for all words, and users can use this function to form an overview of the document collection. Figure 6 shows the global graph of a co-occurrence graph in one window, and Figure 7 shows the co-occurrence graph in two windows. Global picture. The partial graph provides the display function of the adjacent word nodes of the selected word node. Users can use this function to browse the key areas of the document set. Figure 8 shows a partial graph of the co-occurrence graph in a window.

For co-occurrence windows of different sizes, the display content of the graph is different. The functions of the global graph and the local graph are realized by displaying word graph information loaded offline by the front end on demand.

2) Selected browsing and extended browsing of partial graphs.

Selective browsing includes full-text search of words in the global graph, selecting a word of interest, and displaying a local graph centered on the word, including selecting and browsing nodes in the graph according to the word type label. Extended browsing means that the user can click on a neighbor node in the partial graph, and the partial graph is automatically updated to a partial graph centered on the neighbor node.

Figure 10 shows an example of extended browsing. The user clicks "Tang Dechuan" to display a partial map centered on "Tang Dechuan". Only four neighbor nodes are highlighted in the partial map, and the user clicks the neighbor node "Profit Enterprise" to display a partial map centered on "Profit Enterprise".

3) Switching display and parallel display of co-occurrence relationship.

Switching display supports users to focus on a word, and load different partial graphs by selecting different co-occurrence levels (window size), keeping the position of the central word unchanged. Side-by-side display allows users to display side-by-side partial graphs at different co-occurrence levels centered on a word. Switching display and side-by-side display allow users to flexibly view the context of words and find relevant clues.

Figure 8 and Figure 9 show the co-occurrence words with "Tang Dechuan" as the central word, Figure 8 is a partial graph of one window, and Figure 9 is a partial graph of two windows. Switching the display will fix the position of the word "Tang Dechuan", and switch between Figure 8 and Figure 9; side-by-side display will display partial pictures of multiple levels at the same time.

4) Word map browsing history. Users click on the words in the word map to check the relevant documents, and often use the extended browsing function in function 3. During the browsing process, the system will record the points clicked by the user and the related paths. The path is saved in a tree structure. The user can load and search the historical path, which is convenient for the user to recall and restore the inspection status.

As for Figure 10, the "Tang Dechuan" and "Profit Enterprise" clicked by the user will be saved.

5) Word node marking and document marking.

During browsing, users can mark word nodes and related documents. Two types of tags are included:

One is the collection mark, and the marked nodes and related documents can be checked by users in the follow-up;

The second is to delete the mark. The marked nodes and related documents will be deleted from the document set, and the corresponding multi-level co-occurrence relationship word graph will also be updated.

This embodiment also provides a visual text information discovery system based on a multi-level co-occurrence relationship word graph, which is used to implement the above method. The composition is shown in Figure 3, including a document preprocessing module, a keyword extraction module, and a multi-level word graph. building blocks, word-document indexing building blocks, and visual information discovery modules.

The above embodiments are only used to illustrate the technical solution of the present invention and not to limit it. Those of ordinary skill in the art can modify or equivalently replace the technical solution of the present invention without departing from the spirit and scope of the present invention. The scope of protection should be determined by the claims.

Claims (10)

1. a kind of visualText INFORMATION DISCOVERY method based on multistage cooccurrence relation word figure, step include：

The content of text of abstracting document carries out cutting to content of text, obtains text fragments；

Cutting is carried out to text fragments, extracts keyword, and tagged words class label；

According to cooccurrence relation structure multistage cooccurrence relation word figure of the keyword in text fragments, the node in figure corresponds to crucial Word, the side in figure correspond to key words co-occurrence；

Word-document inverted index is built to each keyword in figure, to retrieve the document for including keyword；

VisualText information is obtained by cooccurrence relation word figure.

2. according to the method described in claim 1, it is characterized in that, before the content of text of abstracting document, first by document into Row format parses.

3. according to the method described in claim 1, it is characterized in that, using symbol or fixed window to content of text and text fragments Mouth carries out cutting, which includes punctuation mark, which is to start to ending to move from text.

4. according to the method described in claim 1, it is characterized in that, part of speech distinguishing label includes part of speech label, entity word label, text Shelves core word label, semantic role label, customization type label.

5. according to the method described in claim 4, it is characterized in that, entity word label includes solid polymer composite word.

6. according to the method described in claim 4, it is characterized in that, for document core word label, document core word is found Method includes calculating word weight using TF-IDF or TextRank, and word-based weight is ranked up keyword, takes ranking most Topk high keyword is as document core word.

7. according to the method described in claim 1, it is characterized in that, the cooccurrence relation of keyword includes total in one text segment Existing, co-occurrence, the co-occurrence in entire document in adjacent N number of text fragments.

8. the method according to the description of claim 7 is characterized in that for a pair of of keyword, cooccurrence relation can be only present in In nearest single cooccurrence relation word figure, cooccurrence relation according to sequence from the near to the distant be one text segment in co-occurrence, in phase Co-occurrence, the co-occurrence in entire document in adjacent N number of text fragments.

9. according to the method described in claim 1, it is characterized in that, obtaining visualText information by cooccurrence relation word figure Method, including：Overall situation figure and the online browse of Local map, the switching of the selection browsing of Local map and extension browsing, cooccurrence relation Displaying and side by side displaying, word figure browsing history, word vertex ticks and document markup.

10. a kind of visualText INFORMATION DISCOVERY system based on multistage cooccurrence relation word figure, including：

Document preprocessing module extracts content of text and carries out cutting, obtain text fragments for being parsed into row format to document Ordered list；

Keyword extracting module carries out further cutting for being numbered for each text fragments, and to text fragments, obtains <Word, word class>Set；

Multistage word figure builds module, for the cooccurrence relation according to keyword in text fragments, builds multistage cooccurrence relation word Figure；

Word-document index builds module, and for building word-document inverted index, retrieval includes the document of keyword；

Visual information discovery module, for realizing document browsing, label, status saving function based on cooccurrence relation word figure.