CN112163071A - Unsupervised learning analysis method and system for information correlation degree of emergency - Google Patents
Unsupervised learning analysis method and system for information correlation degree of emergency Download PDFInfo
- Publication number
- CN112163071A CN112163071A CN202011043305.2A CN202011043305A CN112163071A CN 112163071 A CN112163071 A CN 112163071A CN 202011043305 A CN202011043305 A CN 202011043305A CN 112163071 A CN112163071 A CN 112163071A
- Authority
- CN
- China
- Prior art keywords
- article
- vector
- topic
- word
- input
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/33—Querying
- G06F16/3331—Query processing
- G06F16/334—Query execution
- G06F16/3344—Query execution using natural language analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/35—Clustering; Classification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/205—Parsing
- G06F40/216—Parsing using statistical methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/279—Recognition of textual entities
- G06F40/289—Phrasal analysis, e.g. finite state techniques or chunking
Abstract
The invention discloses an unsupervised learning analysis method and system for information correlation degree of an emergency, wherein the method comprises the following steps: step S1, defining the emergency topics and keywords of each topic, and calculating the topic vector of each topic; step S2, acquiring an input article, and calculating an article vector of the input article; in step S3, the relevance between the input article and the topic is calculated according to the article vector of the input article and the topic vector in step S1.
Description
Technical Field
The invention relates to the technical field of natural language processing, in particular to an unsupervised learning analysis method and system for information correlation degree of an emergency.
Background
For the relevance between the article and the topic, a classification algorithm is generally adopted at present to classify the article into the relevant topic.
In the current classification algorithm, each classification is generally mutually exclusive, that is, only the classification to which the article belongs can be calculated with the highest probability. However, sometimes an article may contain multiple topics, such as an emergency event, e.g. a rainstorm article, and may also contain related topics such as rainstorm, waterlogging, landslide, debris flow, etc. Therefore, the relevance of the articles and a plurality of topics cannot be well calculated by using the classification algorithm, and the current classification algorithm is generally supervised, and each article needs to be labeled manually and then model training is performed, so that the workload and the time are high.
Disclosure of Invention
In order to overcome the defects of the prior art, the invention aims to provide an unsupervised learning analysis method and system for the information correlation degree of an emergency, so that the correlation degree of an article and a theme can be conveniently and quickly analyzed, the article is marked with a related theme label, a plurality of themes are independent from each other, and no mutual exclusion relation exists.
In order to achieve the above and other objects, the present invention provides an unsupervised learning analysis method for correlation degree of emergency information, comprising the following steps:
step S1, defining the emergency topics and keywords of each topic, and calculating the topic vector of each topic;
step S2, acquiring an input article, and calculating an article vector of the input article;
in step S3, the relevance between the input article and the topic is calculated according to the article vector of the input article and the topic vector in step S1.
Preferably, the step S1 further includes:
step S100, defining a plurality of topics of the emergency, and defining corresponding keywords for each topic;
step S101, calculating a theme vector of each theme by using a word vector model according to the defined keywords.
Preferably, in step S101, the topic vector of each topic is equal to the sum of the word vectors of each keyword thereof.
Preferably, the step S2 further includes:
step S200, acquiring the text of an input article, and performing Chinese word segmentation on the text of the article;
step S201, calculating TF-IDF weight of each word after word segmentation;
step S202, calculating the article vector of the input article according to the TF-IDF weight of each word obtained in step S201.
Preferably, in step S202, the TF-IDF weight of each word is obtained by the following formula:
wt ═ number of times the word appears in the article —. IDF weight of the word
Preferably, in step S203, the article vector of the input article is equal to the sum of the product of the word vector of each word and the TF-IDF weight of the word vector.
Preferably, the article vector of the input article is obtained using the following formula:
vd is an article vector, Vk is a word vector of each word, and Wk is the TF-IDF weight of each word.
Preferably, in step S3, the relevance between the input article and the topic is the cosine distance between the topic vector and the article vector of the input article.
Preferably, the relevance of the input article to the topic is calculated by the following formula:
a, B represents the topic vector and the article vector of the input article, respectively.
In order to achieve the above object, the present invention further provides an unsupervised learning analysis device for correlation degree of emergency information, including:
the theme definition unit is used for defining the emergency theme and the key words of each theme and calculating the theme vector of each theme;
the article vector analysis unit is used for acquiring an input article and calculating an article vector of the input article;
and the relevancy analysis unit is used for calculating the relevancy between the input article and the theme according to the article vector of the input article and the theme vector obtained by the theme definition unit.
Compared with the prior art, the unsupervised learning analysis method and the unsupervised learning analysis system for the correlation degree of the emergency information calculate the topic vector of each topic by defining the emergency topic and the key words of each topic, then acquire the input article, calculate the article vector of the input article, and calculate the correlation degree of the input article and the topic according to the article vector of the input article and the topic vector of the step S1, so that the correlation degree of the article and the topic is conveniently and quickly analyzed, the article is marked with the relevant topic label, and a plurality of topics are independent from each other and do not have mutual exclusion relation.
Drawings
FIG. 1 is a flowchart illustrating the steps of an unsupervised learning analysis method for information relevancy of emergency events according to the present invention;
FIG. 2 is a system architecture diagram of an unsupervised learning analysis apparatus for correlation of emergency information according to the present invention;
FIG. 3 is a diagram illustrating analysis results according to an embodiment of the present invention.
Detailed Description
Other advantages and capabilities of the present invention will be readily apparent to those skilled in the art from the present disclosure by describing the embodiments of the present invention with specific embodiments thereof in conjunction with the accompanying drawings. The invention is capable of other and different embodiments and its several details are capable of modification in various other respects, all without departing from the spirit and scope of the present invention.
Fig. 1 is a flowchart illustrating steps of an unsupervised learning analysis method for information correlation of an emergency according to the present invention. The invention relates to an unsupervised learning analysis method for information correlation degree of an emergency, which comprises the following steps:
step S1, defining the topics of the emergency and the keywords of each topic, and calculating the topic vector of each topic.
Specifically, step S1 further includes:
step S100, defining a plurality of topics and defining corresponding keywords for each topic.
That is, in the present invention, the topics of the emergency may be defined in advance, and keywords of each topic may be defined, for example, defining the "rainstorm" topic of the emergency, which may include keywords of "rainstorm", "heavy rainfall", and the like.
Step S101, calculating a theme vector of each theme by using a word vector model.
In the present invention, the topic vector is equal to the sum of the word vectors for each keyword. That is, for each topic, the word vector of each keyword thereof can be calculated by using the word vector model, and the topic vector of the topic is the sum of the word vectors of each keyword thereof. In the embodiment of the present invention, the word vector model may use an open-source word vector model, or may train the word vector model by itself, which is not limited in the present invention.
In step S2, an input article is acquired, and an article vector of the input article is calculated.
Specifically, step S2 further includes:
and step S200, acquiring the text of the input article, and performing Chinese word segmentation on the text of the article. The invention adopts the existing Chinese word segmentation system, such as the Chinese lexical analysis system ICTCCLAS of the institute of computational technology of Chinese academy of sciences, and the system can mark the part of speech while segmenting words, so that words without practical meaning, such as auxiliary words, digital words, word atmosphere words and the like, can be removed through the part of speech mark, and words which can reflect the theme of an article, such as nouns, verbs and adjectives, are reserved. Since Chinese word segmentation is the existing mature technology, it is not described herein.
Step S201, calculating TF-IDF (Term Frequency-Inverse Document Frequency) weight of each word after word segmentation. Specifically, the TF-IDF weight of each word is calculated using the following formula:
wt ═ number of times the word appears in the article —. IDF weight of the word
In the present invention, the IDF weight of each word is obtained according to the existing IDF model, which may use an open source model or may use a published corpus to train the IDF model by itself, but the present invention is not limited thereto.
Step S202, calculating the article vector of the input article according to the TF-IDF weight of each word obtained in step S201.
Specifically, the article vector of the input article is equal to the sum of the product of the word vector of each word and the TF-IDF weight of the input article. For example, the following formula is used to obtain:
vd is an article vector, Vk is a word vector of each word, and Wk is the TF-IDF weight of each word. The word vector Vk for each word can be computed using an open-source word vector model.
In step S3, the relevance between the input article and the topic is calculated according to the article vector of the input article and the topic vector in step S1.
In an embodiment of the present invention, the relevance of the input article to the topic is equal to the cosine distance between the topic vector and the article vector. The formula can be used for calculation:
a, B represents a topic vector and an article vector, respectively.
Fig. 2 is a system architecture diagram of an unsupervised learning analysis apparatus for correlation of emergency information according to the present invention. The invention relates to an unsupervised learning analysis device for the information correlation degree of an emergency, which comprises:
the topic definition unit 201 is configured to define topics of the emergency and keywords of each topic, and calculate a topic vector of each topic.
In the present invention, the theme definition unit 201 is specifically configured to:
several topics are defined for the incident and a corresponding keyword is defined for each topic.
That is, in the present invention, the topics of the emergency may be defined in advance, and keywords of each topic may be defined, for example, a "rainstorm" topic is defined, which may include keywords of "rainstorm", "heavy rainfall", and the like.
A topic vector for each topic is calculated using the word vector model.
In the present invention, the topic vector is equal to the sum of the word vectors for each keyword. That is, for each topic, the word vector of each keyword thereof can be calculated by using the word vector model, and the topic vector of the topic is the sum of the word vectors of each keyword thereof. In the embodiment of the present invention, the word vector model may use an open-source word vector model, or may train the word vector model by itself, which is not limited in the present invention.
The article vector analysis unit 202 is configured to acquire an input article and calculate an article vector of the input article.
Specifically, the article vector calculation unit 202 further includes:
and the word segmentation module is used for acquiring the text of the input article and performing Chinese word segmentation on the text of the article. The invention adopts the existing Chinese word segmentation system, such as the Chinese lexical analysis system ICTCCLAS of the institute of computational technology of Chinese academy of sciences, and the system can mark the part of speech while segmenting words, so that words without practical meaning, such as auxiliary words, digital words, word atmosphere words and the like, can be removed through the part of speech mark, and words which can reflect the theme of an article, such as nouns, verbs and adjectives, are reserved. Since Chinese word segmentation is the existing mature technology, it is not described herein.
And the TF-IDF weight calculation module is used for calculating the TF-IDF (Term Frequency-Inverse Document Frequency) weight of each word after word segmentation. Specifically, the TF-IDF weight of each word is calculated using the following formula:
wt ═ number of times the word appears in the article —. IDF weight of the word
And the article vector calculation module is used for calculating the article vector of the input article according to the TF-IDF weight of each word obtained by the TF-IDF weight calculation module.
Specifically, the article vector of the input article is equal to the sum of the product of the word vector of each word and the TF-IDF weight of the input article. For example, the following formula is used to obtain:
vd is an article vector, Vk is a word vector of each word, and Wk is the TF-IDF weight of each word. The word vector Vk of each word can be calculated by using an open-source word vector model.
The relevancy analysis unit 203 is configured to calculate the relevancy between the input article and the topic according to the article vector of the input article and the topic vector obtained by the topic definition unit 201.
In an embodiment of the present invention, the relevance of the input article to the topic is equal to the cosine distance between the topic vector and the article vector. The formula can be used for calculation:
a, B represents a topic vector and an article vector, respectively.
Examples
In this embodiment, taking the topic of "waterlogging" as an example, the corresponding keywords are set as "waterlogging", "rainstorm", "heavy precipitation", "heavy rainfall" and "ponding". An article is input, the relevance of the article and the topic of 'waterlogging' is calculated, and the analysis result is shown in figure 3.
In summary, the unsupervised learning analysis method and system for the correlation degree of the emergency information calculate the topic vector of each topic by defining the emergency topic and the keyword of each topic, then obtain the input article, calculate the article vector of the input article, and calculate the correlation degree of the input article and the topic according to the article vector of the input article and the topic vector of step S1, so as to conveniently and quickly analyze the correlation degree of the article and the topic and print related topic labels for the article, and in the invention, a plurality of topics are independent from each other, and no mutual exclusion relationship exists.
The foregoing embodiments are merely illustrative of the principles and utilities of the present invention and are not intended to limit the invention. Modifications and variations can be made to the above-described embodiments by those skilled in the art without departing from the spirit and scope of the present invention. Therefore, the scope of the invention should be determined from the following claims.
Claims (10)
1. An unsupervised learning analysis method for the correlation degree of emergency information comprises the following steps:
step S1, defining the emergency topics and keywords of each topic, and calculating the topic vector of each topic;
step S2, acquiring an input article, and calculating an article vector of the input article;
in step S3, the relevance between the input article and the topic is calculated according to the article vector of the input article and the topic vector in step S1.
2. The method as claimed in claim 1, wherein the step S1 further comprises:
step S100, defining a plurality of topics of the emergency, and defining corresponding keywords for each topic;
step S101, calculating a theme vector of each theme by using a word vector model according to the defined keywords.
3. The method as claimed in claim 2, wherein the method comprises the following steps: in step S101, the topic vector of each topic is equal to the sum of the word vectors of each keyword.
4. The method as claimed in claim 2, wherein the step S2 further comprises:
step S200, acquiring the text of an input article, and performing Chinese word segmentation on the text of the article;
step S201, calculating TF-IDF weight of each word after word segmentation;
step S202, calculating the article vector of the input article according to the TF-IDF weight of each word obtained in step S201.
5. The method as claimed in claim 4, wherein in step S202, the TF-IDF weight of each word is obtained by the following formula:
wt is the number of times the word appears in the article, the IDF weight of the word.
6. The method of claim 5, wherein in step S203, the article vector of the input article is equal to the sum of the product of the word vector of each word and the TF-IDF weight.
7. The method as claimed in claim 6, wherein the article vector of the input article is obtained by the following formula:
vd is an article vector, Vk is a word vector of each word, and Wk is the TF-IDF weight of each word.
8. The method as claimed in claim 4, wherein the method comprises the following steps: in step S3, the relevance between the input article and the topic is the cosine distance between the topic vector and the article vector of the input article.
9. The method as claimed in claim 8, wherein the correlation between the input article and the topic is calculated by the following formula:
a, B represents the topic vector and the article vector of the input article, respectively.
10. An unsupervised learning analysis device for the correlation degree of emergency information comprises:
the theme definition unit is used for defining the emergency theme and the key words of each theme and calculating the theme vector of each theme;
the article vector analysis unit is used for acquiring an input article and calculating an article vector of the input article;
and the relevancy analysis unit is used for calculating the relevancy between the input article and the theme according to the article vector of the input article and the theme vector obtained by the theme definition unit.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011043305.2A CN112163071A (en) | 2020-09-28 | 2020-09-28 | Unsupervised learning analysis method and system for information correlation degree of emergency |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011043305.2A CN112163071A (en) | 2020-09-28 | 2020-09-28 | Unsupervised learning analysis method and system for information correlation degree of emergency |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112163071A true CN112163071A (en) | 2021-01-01 |
Family
ID=73861883
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011043305.2A Pending CN112163071A (en) | 2020-09-28 | 2020-09-28 | Unsupervised learning analysis method and system for information correlation degree of emergency |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112163071A (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108628825A (en) * | 2018-04-10 | 2018-10-09 | 平安科技（深圳）有限公司 | Text message Similarity Match Method, device, computer equipment and storage medium |
| CN110889443A (en) * | 2019-11-21 | 2020-03-17 | 成都数联铭品科技有限公司 | Unsupervised text classification system and unsupervised text classification method |
| US20200273064A1 (en) * | 2019-02-27 | 2020-08-27 | Nanocorp AG | Generating Campaign Datasets for Use in Automated Assessment of Online Marketing Campaigns Run on Online Advertising Platforms |
-
2020
- 2020-09-28 CN CN202011043305.2A patent/CN112163071A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108628825A (en) * | 2018-04-10 | 2018-10-09 | 平安科技（深圳）有限公司 | Text message Similarity Match Method, device, computer equipment and storage medium |
| US20200273064A1 (en) * | 2019-02-27 | 2020-08-27 | Nanocorp AG | Generating Campaign Datasets for Use in Automated Assessment of Online Marketing Campaigns Run on Online Advertising Platforms |
| CN110889443A (en) * | 2019-11-21 | 2020-03-17 | 成都数联铭品科技有限公司 | Unsupervised text classification system and unsupervised text classification method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Ghoshal et al. | Hidden Markov models for automatic annotation and content-based retrieval of images and video | |
| KR101999152B1 (en) | English text formatting method based on convolution network | |
| CN111143576A (en) | Event-oriented dynamic knowledge graph construction method and device | |
| CN109086265B (en) | Semantic training method and multi-semantic word disambiguation method in short text | |
| CN108090077B (en) | Comprehensive similarity calculation method based on natural language retrieval | |
| CN111611807B (en) | Keyword extraction method and device based on neural network and electronic equipment | |
| CN110909122A (en) | Information processing method and related equipment | |
| Zhang et al. | Real-time object detection algorithm based on improved YOLOv3 | |
| Zhang et al. | Multilingual sentence categorization and novelty mining | |
| CN108763192B (en) | Entity relation extraction method and device for text processing | |
| CN115203421A (en) | Method, device and equipment for generating label of long text and storage medium | |
| CN114548321A (en) | Self-supervision public opinion comment viewpoint object classification method based on comparative learning | |
| Bhutada et al. | Semantic latent dirichlet allocation for automatic topic extraction | |
| WO2022156822A1 (en) | Classification model training method and system | |
| Anandika et al. | A study on machine learning approaches for named entity recognition | |
| Gong et al. | A semantic similarity language model to improve automatic image annotation | |
| CN115312033A (en) | Speech emotion recognition method, device, equipment and medium based on artificial intelligence | |
| CN115062621A (en) | Label extraction method and device, electronic equipment and storage medium | |
| CN107122378B (en) | Object processing method and device and mobile terminal | |
| Kumar et al. | Multi document summarization based on cross-document relation using voting technique | |
| CN112163071A (en) | Unsupervised learning analysis method and system for information correlation degree of emergency | |
| CN110597982A (en) | Short text topic clustering algorithm based on word co-occurrence network | |
| Han et al. | Unsupervised Word Sense Disambiguation based on Word Embedding and Collocation. | |
| Yang et al. | Exploring word similarity to improve chinese personal name disambiguation | |
| CN114996400A (en) | Referee document processing method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |