CN105389377B - Event based on Topics Crawling rolls into a ball acquisition methods - Google Patents
Event based on Topics Crawling rolls into a ball acquisition methods Download PDFInfo
- Publication number
- CN105389377B CN105389377B CN201510799309.6A CN201510799309A CN105389377B CN 105389377 B CN105389377 B CN 105389377B CN 201510799309 A CN201510799309 A CN 201510799309A CN 105389377 B CN105389377 B CN 105389377B
- Authority
- CN
- China
- Prior art keywords
- theme
- text
- event
- text data
- group
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
- G06F16/285—Clustering or classification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/35—Clustering; Classification
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a kind of, and the event based on Topics Crawling rolls into a ball acquisition methods, comprising the following steps: S1: collecting text data set C;S2: pre-processing the text data set C, removes the word that the text data concentrates not practical significance;S3: setting number of topics n and parameter run CTM, obtain CTM model;S4: in the covariance matrix Σ that the CTM model indicates the correlation degree between theme and theme, all Cliques, the group of being the theme are found out using backtracking algorithm;S5: each theme that each theme group is included is selected into a highest article of degree of correspondence in the text data set C, clusters of events corresponding to the highest article of the degree of correspondence is formed into event group.The present invention has the advantage that the degree of association information of theme level is utilized during analyzing and associating degree, the degree of association information of conventional event digging utilization word level is compared, can more promote the reasonability for calculating event correlation degree.
Description
Technical field
The invention belongs to computer literal present treatment and excavation applications, are related to hierarchical subject model digging technology, and in particular to
A kind of event group acquisition methods based on Topics Crawling.
Background technique
With the rapid development of science and technology, the circulation way of information has earth-shaking change.Especially Internet technology
It is universal and the influence of internet power be growing so that the network information becomes the main means that people obtain information.On network
Text information it is more and more, effective event information how is excavated from text information into a major challenge.Based on this
Actual demand, it is necessary to have a kind of technology that can automatically, accurately and real-time extract event.Existing machine learning techniques
It can solve and excavate event from text, they are all to excavate single thing in the level of text using topic model mostly
Part.
In real life, be mostly between event and event it is related, their correlation degree is different, association
One group of high event of degree constitutes the very high event group of a cohesion degree.For example, event 1 " air crash ", " parent of event 2
Belong to extremely grieved ", event 3 " search and rescue are carrying out ", event 4 " various countries leader makes a speech one after another ", this 4 events exist
One special date may form the very high event group of a cohesion degree, and a good event group is more preferable for us
Event information is excavated on ground apparent help.Method for digging still without event group at present.
There are many Text Mining Technologies in machine learning and information retrieval field, it is main in existing Text Mining Technology
Topic model has developed to obtain relative maturity.Topic model achieves huge success, such as social networks in various fields
[1], sentiment analysis [2], recommender system [3] etc..
Latent Dirichlet Allocation (LDA) [4] be it is a kind of it is traditional, for identifying extensive document sets
Or the non-supervisory machine learning techniques of the subject information hidden in corpus.The method that it uses bag of words, by each document
It is considered as a word frequency vector, so that text information is converted the digital information for ease of modeling.Each documents representative one
The probability distribution that a little themes are constituted, and each theme represents the probability distribution that many words are constituted.
In LDA, the mutual independence between theme is lain in the Di Li Cray distribution of theme obedience.Therefore, Blei, David M
And Lafferty, John D point out the defect of LDA in A correlated topic model of science (CTM) [5]
First is that it can not directly measure the degree of correlation between theme.
[1]Lim,Kar Wai and Chen,Changyou and Buntine,Wray.Twitter-Network
Topic Model:A Full Bayesian Treatment for Social Network and Text Modeling.
[C].NIPS workshop on Topic Modeling.(cited on page 15).2013.
[2]Lin,Chenghua and He,Yulan.Joint sentiment/topic model for
sentiment analysis.[C].Proceedings of the 18th ACM conference on Information
and knowledge management.375--384.ACM.2009.
[3]GE,Hao and YE,Yan and BAO,Xi-lin and WU,Min.The Design and
Implementation of Personal Recommendation Module in CMET Based on Topic
Model.[J].Science Technology and Industry.6:033.2013.
[4]Blei,David M and Ng,Andrew Y and Jordan,Michael I.Jordan.Latent
dirichlet allocation.[J].the Journal of machine Learning research.3:993--
1022.JMLR.org.2003.
[5]Blei,David M and Lafferty,John D.A correlated topic model of
science.[J].The Annals of Applied Statistics.17--35.JSTOR.2007.
Summary of the invention
The present invention is directed at least solve one of above-mentioned technical problem.
For this purpose, an object of the present invention is to provide a kind of, the event based on Topics Crawling rolls into a ball acquisition methods.
To achieve the goals above, the embodiment of the first aspect of the present invention discloses a kind of event based on Topics Crawling
Group's acquisition methods, comprising the following steps: S1: text data set C is collected;S2: the text data set C is pre-processed, is gone
Except the text data concentrates the word of not practical significance;S3: setting number of topics n and parameter run CTM, obtain CTM mould
Type;S4: in the covariance matrix Σ that the CTM model indicates the correlation degree between theme and theme, backtracking algorithm is utilized
Find out all Cliques, the group of being the theme;S5: each theme for being included by each theme group is in the textual data
According to a highest text of degree of correspondence is selected in collection C, by clusters of events shape corresponding to the highest text of the degree of correspondence
At event group.
Event according to an embodiment of the present invention based on Topics Crawling rolls into a ball acquisition methods, sharp during analyzing and associating degree
With the degree of association information of theme level, the degree of association information of conventional event digging utilization word level is compared, meter can be more promoted
Calculate the reasonability of event correlation degree.
In addition, event according to the above embodiment of the present invention based on Topics Crawling rolls into a ball acquisition methods, can also have as
Under additional technical characteristic:
Further, the step S1 further comprises: grabbing text information from main stream website, forms the textual data
According to collection C.
Further, it is described to the text data set C carry out pretreatment further comprise: for the text data set
Each text D in CiStop words processing is carried out, if the text DiFor English text, root is carried out to the text
Processing.
Further, the step S3 further comprises: S301: the different text data set C being set different
Number of topics n and threshold θ;S302: adjustment model parameter maximum number of run, EM algorithmic statement condition and the variable condition of convergence;
S303: running the CTM, obtains the convergent CTM model.
Further, the step S4 further comprises: S401: initialization set P, X, R, wherein X, R are empty set, and P is
The set of all themes;S402: a theme z is taken out from the set P every timei, when there is no vertex in the set P,
If the set X is empty set, the set R is the Clique, records the set R and deletes from the set of theme
It removes, repeats backtracking algorithm again and, if not finding the Clique, recalled until all themes are all deleted, entered
Step S4031;S4031: the theme z' obtained from set P for eachiIf the theme z'iWith each theme zj
Between covariance ΣijGreater than the threshold θ, then by the theme z'iIt is added in the set X, and will from the set P
The theme ziIt deletes, wherein i and j is natural number;S4032: if the element number in the set X is greater than in set R
Element number, then the set X is preferred Clique, then R:=X relative to the set R;S4033: circulation step
S4031 to S4032 does not have vertex until finding in the Clique set R or set P;S4034: by the theme ziIt is added to
In the set P, and by the theme z from the set XiIt deletes;S4035: if the element number in the set X is big
Element number in the set R, then the set X is preferred Clique relative to the set R, is denoted as R:=X;
S4036: next round circulation, return step S4031 are carried out;S404: if find Clique set R, the Clique is remembered
Record is got off, that is, the group of being the theme, and is deleted from the set of theme, return step S401, until all themes are all deleted.
Further, the step S5 further comprises: S501: for all theme zi, from all text DjMiddle choosing
Theme distribution p (z=z outi|Dj) one highest, by theme distribution p (z=zi|Dj) a highest DjIt is assigned to the theme
zi;S502: for each theme group, by each described theme z included in itiThe text D assignedjIt is corresponding
Clusters of events together, form the event group.
Additional aspect and advantage of the invention will be set forth in part in the description, and will partially become from the following description
Obviously, or practice through the invention is recognized.
Detailed description of the invention
Above-mentioned and/or additional aspect of the invention and advantage will become from the description of the embodiment in conjunction with the following figures
Obviously and it is readily appreciated that, in which:
Fig. 1 is the schematic diagram of the event group of one embodiment of the invention;
Fig. 2 is the flow chart of the event group acquisition methods based on Topics Crawling of one embodiment of the invention.
Specific embodiment
The embodiment of the present invention is described below in detail, examples of the embodiments are shown in the accompanying drawings, wherein from beginning to end
Same or similar label indicates same or similar element or element with the same or similar functions.Below with reference to attached
The embodiment of figure description is exemplary, and for explaining only the invention, and is not considered as limiting the invention.
In the description of the present invention, it is to be understood that, term " center ", " longitudinal direction ", " transverse direction ", "upper", "lower",
The orientation or positional relationship of the instructions such as "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outside" is
It is based on the orientation or positional relationship shown in the drawings, is merely for convenience of description of the present invention and simplification of the description, rather than instruction or dark
Show that signified device or element must have a particular orientation, be constructed and operated in a specific orientation, therefore should not be understood as pair
Limitation of the invention.In addition, term " first ", " second " are used for description purposes only, it is not understood to indicate or imply opposite
Importance.
In the description of the present invention, it should be noted that unless otherwise clearly defined and limited, term " installation ", " phase
Even ", " connection " shall be understood in a broad sense, for example, it may be being fixedly connected, may be a detachable connection, or be integrally connected;It can
To be mechanical connection, it is also possible to be electrically connected;It can be directly connected, can also can be indirectly connected through an intermediary
Connection inside two elements.For the ordinary skill in the art, above-mentioned term can be understood at this with concrete condition
Concrete meaning in invention.
Referring to following description and drawings, it will be clear that these and other aspects of the embodiment of the present invention.In these descriptions
In attached drawing, some particular implementations in the embodiment of the present invention are specifically disclosed, to indicate to implement implementation of the invention
Some modes of the principle of example, but it is to be understood that the scope of embodiments of the invention is not limited.On the contrary, of the invention
Embodiment includes all changes, modification and the equivalent fallen within the scope of the spirit and intension of attached claims.
The event according to an embodiment of the present invention based on Topics Crawling, which is described, below in conjunction with attached drawing rolls into a ball acquisition methods.
1) text data set is collected
In a certain section of valuable significant and reasonable time range (one month), text envelope is grabbed from main stream website
It ceases (news, blog, microblogging, forum etc.), text data set C needed for being formed.
2) data set is pre-processed
For each text D in text data set Ci, stop words operation processing is carried out respectively, and stop words includes language
Gas auxiliary word, adverbial word, preposition, conjunction etc..
If it is English text data set, also need additionally to carry out root operation processing.Rootization operation can be word
Initial root is restored back in different form variation, although grammatically there is mistake in this way, different form can be changed
Influence reduce.
3) number of topics and parameter are set, CTM is run, obtains a CTM model.
Different number of topics n and threshold θ are set for different text data set C.In addition can be joined with appropriate adjustment model
Number such as EM algorithm (EM algorithm) maximum number of run, EM algorithmic statement condition, the variable condition of convergence.CTM is run, is obtained
To a convergent CTM model.It is specific as follows:
Initialize μ(0)=md=0, ∑(0)=IT, Vd=IT
Circulating repetition until convergence
(E step) calculates each d=1,2 ..., D
WhereinSd=[s1,…,sV]
(M step) calculates
4) all Cliques are found out to be the theme group
The CTM model indicate the correlation degree between theme and theme covariance matrix Σ (covariance indicate be
The expectation of two variable global errors.Covariance matrix is a matrix, each of which element is the association between each vector element
Variance) in, all Cliques, the group of being the theme are found out using backtracking algorithm.
4.1) set P, X, R are initialized, wherein X, R are empty set, and set P is the set of all themes.
4.2) a theme z is taken out from set P every timei, when there is no vertex in set, two kinds of situations:
4.2.1) if set X is empty set at this time, set R is Clique, and set R is recorded, and from the collection of theme
It is deleted in conjunction, repeats backtracking algorithm again, until all themes are all deleted.
4.2.2 Clique) is not found, is recalled at this time.
4.3) for theme z' that each is obtained from set Pi, there is following processing:
4.3.1) consider each of set X theme zjIf theme z'iWith each theme zjBetween covariance Σij
(reading from covariance matrix Σ) is all larger than threshold θ, then by theme z'iIt is added in set X, and the not property of the group of destruction.From
By theme z' in set PiIt deletes.
4.3.2) if the element number in set X is greater than the element number in set R, R:=X at this time.
4.3.3) since repeating 4.3.2 4.3.1) up to finding in Clique set R or set P there is no vertex.
4.3.4) theme ziIt is added in set P, and by theme z from set XiIt deletes.
4.3.5) if the element number in set X is greater than the element number in set R at this time, X is one more better than R
Clique is denoted as R:=X.
4.3.6 next round circulation, return step S4031) are carried out.
4.4) when finding Clique set R, Clique is recorded, that is, the group of being the theme, and is deleted from the set of theme
It removes, repeats backtracking algorithm since 4.1 again, until all themes are all deleted.
5) find and analyze event group
5.1) for all theme zi, from all text DjIn select theme distribution p (z=zi|Dj) one highest, it will
Theme distribution p (z=zi|Dj) highest one be assigned to theme zi, p (z=zi|Dj) indicate theme distribution form, give text
DjWhen, choose the probability for arriving theme Zi.We find out one for each theme Zi and theme distribution are made to take peak
Text.
5.2) for each theme group, by each theme z included in itiThe text D assignedjCorresponding thing
Part flocks together, and forms an event group.
6) form for being formed by event group diagram is shown
Each event indicates with its corresponding text, is simply visually indicated with the title of every text or keyword
The text shows event group with exemplary form is rolled into a ball shaped like Fig. 1 event.
In addition, the embodiment of the present invention based on Topics Crawling event group acquisition methods other compositions and effect for
All be for those skilled in the art it is known, in order to reduce redundancy, do not repeat them here.
In the description of this specification, reference term " one embodiment ", " some embodiments ", " example ", " specifically show
The description of example " or " some examples " etc. means specific features, structure, material or spy described in conjunction with this embodiment or example
Point is included at least one embodiment or example of the invention.In the present specification, schematic expression of the above terms are not
Centainly refer to identical embodiment or example.Moreover, particular features, structures, materials, or characteristics described can be any
One or more embodiment or examples in can be combined in any suitable manner.
Although an embodiment of the present invention has been shown and described, it will be understood by those skilled in the art that: not
A variety of change, modification, replacement and modification can be carried out to these embodiments in the case where being detached from the principle of the present invention and objective, this
The range of invention is by claim and its equivalent limits.
Claims (6)
1. a kind of event based on Topics Crawling rolls into a ball acquisition methods, which comprises the following steps:
S1: text data set C is collected;
S2: pre-processing the text data set C, removes the word that the text data concentrates not practical significance;
S3: setting number of topics n and parameter run CTM, obtain CTM model;
S4: in the covariance matrix Σ that the CTM model indicates the correlation degree between theme and theme, backtracking algorithm is utilized
Find out all Cliques, the group of being the theme;
S5: each theme that each theme group is included is selected into a corresponding journey in the text data set C
Highest text is spent, clusters of events corresponding to the highest text of the degree of correspondence is formed into event group.
2. event according to claim 1 based on Topics Crawling rolls into a ball acquisition methods, which is characterized in that the step S1 into
One step includes: that text information is grabbed from main stream website, forms the text data set C.
3. the event according to claim 1 based on Topics Crawling rolls into a ball acquisition methods, which is characterized in that described to the text
Notebook data collection C carries out pretreatment:
For each text D in the text data set CiStop words processing is carried out,
If the text DiFor English text, root processing is carried out to the text.
4. event according to claim 1 based on Topics Crawling rolls into a ball acquisition methods, which is characterized in that the step S3 into
One step includes:
S301: different number of topics n and threshold θ are set for the different text data set C;
S302: adjustment model parameter maximum number of run, EM algorithmic statement condition and the variable condition of convergence;
S303: running the CTM, obtains the convergent CTM model.
5. event according to claim 4 based on Topics Crawling rolls into a ball acquisition methods, which is characterized in that the step S4 into
One step includes:
S401: initialization set P, X, R, wherein X, R are empty set, and P is the set of all themes;
S402: a theme z is taken out from the set P every timei, when there is no vertex in the set P,
If the set X is empty set, the set R is the Clique, records the set R and from the set of theme
It deletes, repeats backtracking algorithm again, until all themes are all deleted,
If not finding the Clique, recalled, enters step S4031;
S4031: the theme z obtained from set P for eachi', if the theme zi' and each theme zjBetween
Covariance ΣijGreater than the threshold θ, then by the theme zi' be added in the set X, and by the master from the set P
Inscribe zi' delete, wherein i and j is natural number;
S4032: if the element number in the set X is greater than the element number in set R, the set X is relative to institute
Stating set R is preferred Clique, then R:=X;
S4033: circulation step S4031 to S4032, there is no vertex until finding in the Clique set R or set P;
S4034: by the theme ziIt is added in the set P, and by the theme z from the set XiIt deletes;
S4035: if the element number in the set X is greater than the element number in the set R, the set X is opposite
It is preferred Clique in the set R, is denoted as R:=X;
S4036: next round circulation, return step S4031 are carried out;
S404: if find Clique set R, the Clique being recorded, that is, the group of being the theme, and from the set of theme
Middle deletion, return step S401, until all themes are all deleted.
6. event according to claim 5 based on Topics Crawling rolls into a ball acquisition methods, which is characterized in that the step S5 into
One step includes:
S501: for all theme zi, from all text DjIn select theme distribution p (z=zi|Dj) one highest, it will
Theme distribution p (z=zi|Dj) a highest DjIt is assigned to the theme zi;
S502: for each theme group, by each described theme z included in itiThe text D assignedjIt is corresponding
Clusters of events together, form the event group.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201510799309.6A CN105389377B (en) | 2015-11-18 | 2015-11-18 | Event based on Topics Crawling rolls into a ball acquisition methods |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201510799309.6A CN105389377B (en) | 2015-11-18 | 2015-11-18 | Event based on Topics Crawling rolls into a ball acquisition methods |
Publications (2)
Publication Number | Publication Date |
---|---|
CN105389377A CN105389377A (en) | 2016-03-09 |
CN105389377B true CN105389377B (en) | 2019-02-05 |
Family
ID=55421667
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201510799309.6A Active CN105389377B (en) | 2015-11-18 | 2015-11-18 | Event based on Topics Crawling rolls into a ball acquisition methods |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN105389377B (en) |
Families Citing this family (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN108052636B (en) * | 2017-12-20 | 2022-02-25 | 北京工业大学 | Method and device for determining text theme correlation degree and terminal equipment |
CN108763258B (en) * | 2018-04-03 | 2023-01-10 | 平安科技(深圳)有限公司 | Document theme parameter extraction method, product recommendation method, device and storage medium |
Citations (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN104199974A (en) * | 2013-09-22 | 2014-12-10 | 中科嘉速(北京)并行软件有限公司 | Microblog-oriented dynamic topic detection and evolution tracking method |
-
2015
- 2015-11-18 CN CN201510799309.6A patent/CN105389377B/en active Active
Patent Citations (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN104199974A (en) * | 2013-09-22 | 2014-12-10 | 中科嘉速(北京)并行软件有限公司 | Microblog-oriented dynamic topic detection and evolution tracking method |
Non-Patent Citations (5)
Title |
---|
Clustering Based Topic Events Detection on Text Stream;Chunshan Li et al;《ACIIDS 2014, Part I, LNAI 8397》;20141231;第42-52页 |
Exploiting Background Information Networks to Enhance Bilingual Event Extraction Through Topic Modeling;Hao Li et al;《IMMM 2011 : The First International Conference on Advances in Information Mining and Management》;20111231;第23-30页 |
LDA及主题词相关性的新事件检测;黄颖;《计算机与现代化》;20120131(第1期);第6-9、13页 |
Mining streams of short text for analysis of world-wide event evolutions;Guangyan Huang et al;《World Wide Web》;20140604;第1201-1217页 |
基于概率主题建模的新闻文本可视化综述;汤斯亮 等;《计算机辅助设计与图形学学报》;20150531;第27卷(第5期);第771-782页 |
Also Published As
Publication number | Publication date |
---|---|
CN105389377A (en) | 2016-03-09 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN110717047B (en) | Web service classification method based on graph convolution neural network | |
CN106126732B (en) | Author influence propagation capacity prediction method based on interest similarity model | |
Korenčić et al. | Document-based topic coherence measures for news media text | |
Wang et al. | User identity linkage across social networks via linked heterogeneous network embedding | |
Vulić et al. | Cross-language information retrieval models based on latent topic models trained with document-aligned comparable corpora | |
CN104268197A (en) | Industry comment data fine grain sentiment analysis method | |
CN112966091B (en) | Knowledge map recommendation system fusing entity information and heat | |
CN105138665A (en) | Online internet topic mining method based on improved LDA model | |
Thijs et al. | Do second-order similarities provide added-value in a hybrid approach? | |
Remus et al. | Three knowledge-free methods for automatic lexical chain extraction | |
Shen et al. | SsciBERT: A pre-trained language model for social science texts | |
Schinas et al. | Mgraph: multimodal event summarization in social media using topic models and graph-based ranking | |
Lu et al. | Visual topic discovering, tracking and summarization from social media streams | |
CN105389377B (en) | Event based on Topics Crawling rolls into a ball acquisition methods | |
Ning et al. | Source-aware crisis-relevant tweet identification and key information summarization | |
Chandra et al. | Collective representation learning on spatiotemporal heterogeneous information networks | |
Balouchian et al. | Context-sensitive single-modality image emotion analysis: A unified architecture from dataset construction to CNN classification | |
Priya et al. | Characterizing infrastructure damage after earthquake: a split-query based ir approach | |
Wang et al. | Emotional contagion-based social sentiment mining in social networks by introducing network communities | |
Shu et al. | Automatic extraction of web page text information based on network topology coincidence degree | |
Abulaish et al. | A layered approach for summarization and context learning from microblogging data | |
Hertling et al. | Gollum: A Gold Standard for Large Scale Multi Source Knowledge Graph Matching | |
Aziz et al. | Sentence based semantic similarity measure for blog-posts | |
Yu et al. | Mining hidden interests from twitter based on word similarity and social relationship for OLAP | |
Showrov et al. | Keyword extraction from bengali news |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
C06 | Publication | ||
PB01 | Publication | ||
C10 | Entry into substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |