CN112883167A - Text emotion classification model based on hierarchical self-power-generation capsule network - Google Patents
Text emotion classification model based on hierarchical self-power-generation capsule network Download PDFInfo
- Publication number
- CN112883167A CN112883167A CN202110292481.8A CN202110292481A CN112883167A CN 112883167 A CN112883167 A CN 112883167A CN 202110292481 A CN202110292481 A CN 202110292481A CN 112883167 A CN112883167 A CN 112883167A
- Authority
- CN
- China
- Prior art keywords
- network
- attention
- text
- hierarchical
- model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 239000002775 capsule Substances 0.000 title claims abstract description 34
- 230000008451 emotion Effects 0.000 title abstract description 10
- 238000013145 classification model Methods 0.000 title description 2
- 238000010248 power generation Methods 0.000 title description 2
- 238000003062 neural network model Methods 0.000 claims description 3
- 230000007246 mechanism Effects 0.000 abstract description 16
- 238000000034 method Methods 0.000 abstract description 8
- 238000013528 artificial neural network Methods 0.000 abstract description 6
- 230000007547 defect Effects 0.000 abstract description 4
- 230000002996 emotional effect Effects 0.000 abstract description 2
- 239000013598 vector Substances 0.000 description 5
- 238000003058 natural language processing Methods 0.000 description 4
- 238000012545 processing Methods 0.000 description 4
- 230000000306 recurrent effect Effects 0.000 description 4
- 238000004458 analytical method Methods 0.000 description 3
- 230000002457 bidirectional effect Effects 0.000 description 3
- 238000013527 convolutional neural network Methods 0.000 description 3
- 238000005096 rolling process Methods 0.000 description 3
- 241000282414 Homo sapiens Species 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 239000011159 matrix material Substances 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 238000012549 training Methods 0.000 description 2
- 230000001149 cognitive effect Effects 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 238000000151 deposition Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 230000015654 memory Effects 0.000 description 1
- 238000005065 mining Methods 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 230000001172 regenerating effect Effects 0.000 description 1
- 230000006403 short-term memory Effects 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/33—Querying
- G06F16/3331—Query processing
- G06F16/334—Query execution
- G06F16/3344—Query execution using natural language analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/35—Clustering; Classification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/30—Semantic analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Artificial Intelligence (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Health & Medical Sciences (AREA)
- Databases & Information Systems (AREA)
- General Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Computing Systems (AREA)
- Molecular Biology (AREA)
- Evolutionary Computation (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Biophysics (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Machine Translation (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
Aiming at the defects of the capsule network and the hierarchical attention network, a hybrid neural network based on the hierarchical self-attention network and the capsule network is provided. The invention aims to integrate the advantages of a capsule network, a hierarchical model and an attention mechanism, and introduces the self-attention mechanism in the hierarchical attention network to enhance the feature expression capability of the model and reduce the dependence of external parameters, so that the model can learn more key features of the text. The method comprises the steps of using a hierarchical self-attention network to carry out emotion modeling on text from two levels of words and sentences, and meanwhile selectively paying attention to important words and sentences in the text. And the capsule network coding text part and the whole space incidence relation is used, richer text emotional characteristics are extracted, and the advantages of the two are fused to improve the classification performance.
Description
Technical Field
The invention belongs to the field of natural language processing, and is applied to emotion analysis tasks.
Background
First, noun interpretation: 1. textual emotion analysis (SentimentAnalysis): the method is a process for analyzing, processing and extracting subjective texts with emotional colors by utilizing natural language processing and text mining technologies.
2. Capsule networks (connected networks): in 2017, the Hinton group proposed a capsule network, expanded the scalar type network to vectors for the first time, and used a dynamic routing algorithm to perform the transfer calculations between capsules. The method is used in the image processing field at first and is later used in natural language processing tasks such as text emotion classification, and more text semantic information is reserved compared with a convolutional neural network.
3. Bidirectional GRU network (Bidirectional Gated regenerative Unit): to solve the gradient message problem of the standard RNN (recurrent neural network), the GRU uses an update gate (update gate) and a reset gate (resetgate). These two gating mechanisms are unique in that they can preserve information in long-term sequences and do not clear over time or remove because they are not relevant to prediction.
4. Self-attention Mechanism (self-attention Mechanism): derived from studies on human vision. In cognitive science, due to the bottleneck of information processing, human beings selectively pay attention to a part of all information while ignoring other visible information; later, people applied this idea to image processing and natural language processing and achieved good results, and the purpose of introducing a self-attention mechanism is to better focus on words that are important for emotion classification.
Secondly, the prior art: 1, (1) text emotion classification method based on RNN: socher et al use a tree-structured LSTM (Long short term memory network) network to improve semantic representation, in which memory elements are able to hold associations between instances, thereby capturing relationships between words; cho et al propose GRU units, which have fewer parameters and faster training than LSTM models, and can capture global semantic features. (2) The emotion analysis method combining the hierarchical neural network and the attention mechanism comprises the following steps: tang et al builds a hierarchical model using convolutional neural networks and LSTM to obtain the feature representation of chapters from both word and sentence perspectives. Yang et al combines an attention mechanism with a hierarchical model, and pays attention to keywords in sentences and key sentences in chapters respectively by using the attention mechanism. (3) Capsule network: in 2011, GeoffreyHinton et al introduced the capsule network for the first time in a paper entitled "transform autoencoder". In 2018, Zhao et al put forward for the first time that the capsule network is applied to a text classification task, and the classification performance of the capsule network on a plurality of data sets exceeds that of a general neural network model. In the same year, Kim and the like propose a capsule network based on a static routing mechanism according to text attributes, so that the classification performance is improved while the calculation complexity is effectively reduced.
The CapsNet model: the model applies the capsule network to the text classification task, and comprises the following four layers: the input layer inputs the text into the network in a word vector form; the convolution layer uses a convolution control module, and the thought is derived from a gate control mechanism of a recurrent neural network (LSTM) and a GRU, so that noise information is screened out, and local features of a text are better extracted; rolling and depositing a capsule layer: due to the high variability of the text, the layer improves the traditional dynamic algorithm into a static routing algorithm to obtain the global semantic features of the text; classifying capsule layers: and outputting the prediction label of the text.
3. Hierarchical attention model: the method is based on a recurrent neural network model and is combined with attention to construct a hierarchical model. The model models the text from two levels of words and sentences, uses a recurrent neural network to extract the characteristics of the text, pays attention to important words and sentences respectively, and finally classifies the text.
Thirdly, the technical problem is as follows: 1. although the capsule network improves the defects of the traditional convolutional neural network to a certain extent, the local features of the text are extracted through convolutional operation essentially, important words in the text cannot be paid attention selectively, and the long-distance dependency relationship cannot be coded, so that the method has great limitation in recognizing the text with semantic turning. In the hierarchical attention model, the attention mechanism needs more parameter dependence, and the model cannot focus more on the internal sequence relation of the text.
2. Aiming at the defects of the capsule network and the hierarchical attention network, the invention aims to integrate the advantages of the capsule network, the hierarchical model and the attention mechanism, introduce the self-attention mechanism into the hierarchical attention network, reduce the dependence of external parameters and enable the model to learn more key characteristics of the text. The global semantic features of the texts are learned by using a hierarchical self-attention network, important words and sentences in the texts are concerned, and the spatial association relation between the parts of the texts and the whole text is learned by using a capsule network, so that the classification performance of the model is effectively improved.
Disclosure of Invention
1. Aiming at the defects of the capsule network and the hierarchical attention network, the invention aims to integrate the advantages of the capsule network, the hierarchical model and the attention mechanism, introduce the self-attention mechanism into the hierarchical attention network, reduce the dependence of external parameters and enable the model to learn more key characteristics of the text. The global semantic features of the texts are learned by using a hierarchical self-attention network, important words and sentences in the texts are concerned, and the spatial association relation between the parts of the texts and the whole text is learned by using a capsule network, so that the classification performance of the model is effectively improved.
2. The technical innovation points of the invention are as follows: (1) a mixed neural network based on a hierarchical self-attention network and a capsule network is designed, richer semantic feature information is extracted by utilizing the relation between the part of a rubber network coded text and the whole body, and context information is extracted from two levels of words and sentences by utilizing the hierarchical self-attention network, so that the classification performance is improved by fusing the advantages of the two levels; (2) self-attention is introduced into the hierarchical model to replace the traditional attention, so that the dependence of external parameters is reduced, the internal dependence relation of the text is captured, important words and sentences are concerned, and the characteristic expression capability of the model is enhanced;
drawings
FIG. 1 is a diagram of a hierarchical self-attention capsule network model architecture.
Detailed Description
The attached drawing is a model structure diagram of the invention, which mainly comprises two modules: the hierarchical network module consists of a bidirectional GRU network and a hierarchical network based on self attention, and learns text context information by using the BiGRU to capture important word and sentence characteristics from the attention; and the capsule network module consists of a rolling capsule layer and a classification capsule layer, encodes text semantics and structural information based on the text representation output by the hierarchical network module, learns the associated characteristic information of the text part and the integration, and finally classifies the text semantics and the structural information. According to different functions of each layer in the layered network module, the module is divided into five layers by the model: word embedding, word level coding layer, word level self-attention layer, sentence level coding layer and sentence level self-attention layer. Word embedding: and performing word embedding mapping on the text to obtain a continuous low-dimensional real-valued vector for representing semantic information of words. The model uses Glove pre-training word vectors, maps words in a text into a 300-dimensional word vector matrix, and establishes an initial characteristic matrix representing the text as an input X of the model. Word level coding layer: and taking the word X as a feature extraction object, and performing feature acquisition on the text by using the BiGRU to obtain the global semantic information hit of the text. Word level self-attention layer: the model can pay attention to important information in the text, different weights are given to each word by self-attention in the weight adjusting layer to represent the contribution degree of the words, and finally the sentence sequence representation Si is obtained. Sentence-level coding layer: the method comprises the following steps of taking a sentence Si as a feature extraction object, and carrying out feature acquisition on a text by using BiGRU to obtain a hidden feature hi in the sentence, wherein the sentence level self-attention layer comprises the following steps: and endowing each sentence with different contribution degrees to measure the importance of the sentence to the text to obtain a final sentence representation V. The capsule network module inputs the sentence representation V into the rolling capsule layer to learn the spatial association relation between the text part and the whole body through a dynamic routing algorithm to obtain the high-level feature representation Vout. And finally, carrying out normalization processing on the classification capsule layer to complete the text emotion classification task.
Claims (2)
1. A mixed neural network model based on a hierarchical self-attention network and a capsule network is provided, richer semantic feature information is extracted by utilizing the relation between parts and the whole body in a capsule network coding text, and context information is extracted from two levels of words and sentences by utilizing the hierarchical self-attention network, so that the classification performance is improved by combining the advantages of the two levels.
2. The model in claim 1 introduces self-attention to replace traditional attention, reduces external parameter dependence, captures text internal dependence, pays attention to important words and sentences, and enhances the feature expression capability of the model.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110292481.8A CN112883167A (en) | 2021-03-18 | 2021-03-18 | Text emotion classification model based on hierarchical self-power-generation capsule network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110292481.8A CN112883167A (en) | 2021-03-18 | 2021-03-18 | Text emotion classification model based on hierarchical self-power-generation capsule network |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112883167A true CN112883167A (en) | 2021-06-01 |
Family
ID=76041259
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110292481.8A Pending CN112883167A (en) | 2021-03-18 | 2021-03-18 | Text emotion classification model based on hierarchical self-power-generation capsule network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112883167A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114036366A (en) * | 2021-11-19 | 2022-02-11 | 四川大学 | Social network deception detection method based on text semantics and hierarchical structure |
| CN114118200A (en) * | 2021-09-24 | 2022-03-01 | 杭州电子科技大学 | Multi-modal emotion classification method based on attention-guided bidirectional capsule network |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109241283A (en) * | 2018-08-08 | 2019-01-18 | 广东工业大学 | A kind of file classification method based on multi-angle capsule network |
| CN110826336A (en) * | 2019-09-18 | 2020-02-21 | 华南师范大学 | Emotion classification method, system, storage medium and equipment |
| US20200210526A1 (en) * | 2019-01-02 | 2020-07-02 | Netapp, Inc. | Document classification using attention networks |
| CN112131391A (en) * | 2020-11-25 | 2020-12-25 | 江苏电力信息技术有限公司 | Power supply service client appeal text classification method based on capsule network |
-
2021
- 2021-03-18 CN CN202110292481.8A patent/CN112883167A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109241283A (en) * | 2018-08-08 | 2019-01-18 | 广东工业大学 | A kind of file classification method based on multi-angle capsule network |
| US20200210526A1 (en) * | 2019-01-02 | 2020-07-02 | Netapp, Inc. | Document classification using attention networks |
| CN110826336A (en) * | 2019-09-18 | 2020-02-21 | 华南师范大学 | Emotion classification method, system, storage medium and equipment |
| CN112131391A (en) * | 2020-11-25 | 2020-12-25 | 江苏电力信息技术有限公司 | Power supply service client appeal text classification method based on capsule network |
Non-Patent Citations (4)
| Title |
|---|
| 刘纯嘉: "面向高校舆情的中文文本情感倾向性分析", 《中国优秀硕士学位论文全文数据库(信息科技辑)》, pages 1 - 47 * |
| 王弘中 等: "一种基于混合词向量的胶囊网络文本分类方法", 《小型微型计算机系统》 * |
| 程艳 等: "基于注意力机制的多通道CNN和BiGRU的文本情感倾向性分析", 《计算机研究与发展》 * |
| 程艳 等: "融合卷积神经网络与层次化注意力网络的中文 文本情感倾向性分析", 《中文信息学报》 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114118200A (en) * | 2021-09-24 | 2022-03-01 | 杭州电子科技大学 | Multi-modal emotion classification method based on attention-guided bidirectional capsule network |
| CN114036366A (en) * | 2021-11-19 | 2022-02-11 | 四川大学 | Social network deception detection method based on text semantics and hierarchical structure |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108596039B (en) | Bimodal emotion recognition method and system based on 3D convolutional neural network | |
| CN113158875B (en) | Image-text emotion analysis method and system based on multi-mode interaction fusion network | |
| CN108830287A (en) | The Chinese image, semantic of Inception network integration multilayer GRU based on residual error connection describes method | |
| CN109918671A (en) | Electronic health record entity relation extraction method based on convolution loop neural network | |
| CN111626116B (en) | Video semantic analysis method based on fusion of multi-attention mechanism and Graph | |
| CN111339837B (en) | Continuous sign language recognition method | |
| CN112036276B (en) | Artificial intelligent video question-answering method | |
| CN101187990A (en) | A session robotic system | |
| CN110347831A (en) | Based on the sensibility classification method from attention mechanism | |
| CN111046179A (en) | Text classification method for open network question in specific field | |
| Li et al. | Sign language recognition based on computer vision | |
| CN114549850B (en) | Multi-mode image aesthetic quality evaluation method for solving modal missing problem | |
| CN112883167A (en) | Text emotion classification model based on hierarchical self-power-generation capsule network | |
| CN111598183A (en) | Multi-feature fusion image description method | |
| CN111985205A (en) | Aspect level emotion classification model | |
| CN113239153B (en) | Text and image mutual retrieval method based on example masking | |
| CN114662497A (en) | False news detection method based on cooperative neural network | |
| CN113283234A (en) | Emotion classification method based on capsule network | |
| CN117236338B (en) | Named entity recognition model of dense entity text and training method thereof | |
| CN113255602A (en) | Dynamic gesture recognition method based on multi-modal data | |
| CN115393948A (en) | Sign language video generation method based on improved Transformer model | |
| CN109508640A (en) | A kind of crowd's sentiment analysis method, apparatus and storage medium | |
| CN111445545B (en) | Text transfer mapping method and device, storage medium and electronic equipment | |
| Fan et al. | An improved yolov5 marine biological object detection algorithm | |
| Luqin | A survey of facial expression recognition based on convolutional neural network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20210601 |