CN112732920A

CN112732920A - BERT-based multi-feature fusion entity emotion analysis method and system

Info

Publication number: CN112732920A
Application number: CN202110055036.XA
Authority: CN
Inventors: 吴佳鸣
Original assignee: Beijing Minglue Zhaohui Technology Co Ltd
Current assignee: Beijing Minglue Zhaohui Technology Co Ltd
Priority date: 2021-01-15
Filing date: 2021-01-15
Publication date: 2021-04-30

Abstract

The application discloses a BERT-based multi-feature fusion entity emotion analysis method and system, wherein the method comprises the following steps: marking the text to obtain entity emotion data; performing model training based on the entity emotion data to obtain an entity emotion classification model; and performing entity emotion analysis on the predicted text by using the entity emotion classification model. By the method and the device, entity emotion classification performance can be improved.

Description

BERT-based multi-feature fusion entity emotion analysis method and system

Technical Field

The invention relates to the technical field of natural language processing. More specifically, the invention relates to a BERT-based multi-feature fusion entity emotion analysis method and system.

Background

With the rapid rise and development of a plurality of internet platforms, people are used to share the feeling of the people among all the large platforms. In many fields, the need and experience to learn about users by generating content for users has become a common need for many service providers. Entity emotion analysis, one of popular application techniques in the field of natural language processing, provides a detailed and efficient solution to this need.

Compared with the traditional sentiment classification technology, the research object of entity sentiment classification is a specific entity with finer granularity. The emotion of a specific entity can provide characteristics for different tasks, so that the application scene of the tasks is richer and more flexible.

At present, the sentence-level emotion analysis method based on the BERT pre-training model can obtain more advanced performance, usually considers a single feature (entity semantic feature) or uses a more complex model to fuse a certain feature, but still has the following disadvantages:

1. on the entity emotion analysis task, the performance is not as expected due to the fact that more entity-related characteristics cannot be utilized;

2. the incidence relation between entity semantic information and entity position information, between sentence semantic information and entity context characteristics is not comprehensively considered, and the model performance is low due to the fact that a model of the entity semantic characteristics is used only;

3. the model with multi-feature fusion is used, the feature fusion mode is complex, the calculation cost is high, the performance improvement is not obvious, and the model complexity is high.

Disclosure of Invention

The embodiment of the application provides a BERT-based multi-feature fusion entity emotion analysis method, which is used for at least solving the problem of subjective factor influence in the related technology.

The invention provides a BERT-based multi-feature fusion entity emotion analysis method, which comprises the following steps:

labeling: marking the text to obtain entity emotion data;

training: performing model training based on the entity emotion data to obtain an entity emotion classification model;

a prediction step: and performing entity emotion analysis on the predicted text by using the entity emotion classification model.

As a further improvement of the present invention, the labeling step specifically includes the following steps:

the construction steps are as follows: constructing an annotation platform;

an input step: and inputting the text into the marking platform to carry out artificial entity and entity emotion marking, and acquiring the entity emotion data.

As a further improvement of the present invention, the constructing step specifically comprises the following steps:

a rule definition step: defining an annotation rule;

and a label definition step: defining an annotation tag;

example given procedure: an instance of the annotation is given.

As a further improvement of the present invention, the training step specifically includes the following steps:

a checking step: carrying out data format verification on the entity emotion data, and dividing to obtain a training data set;

a marking step: performing position information labeling on the training data set;

a characteristic obtaining step: inputting the training data set containing the marking information into a BERT pre-training model to obtain hidden layer output characteristic vectors;

and (3) feature fusion step: acquiring a fusion feature vector based on the hidden layer output feature vector;

a parameter acquisition step: and performing probability transformation on the fusion feature vector through softmax to obtain model parameters.

As a further improvement of the present invention, the position information mark in the marking step includes a whole sentence classification identifier mark, an entity start position mark, and an entity end position mark.

As a further improvement of the present invention, the feature fusion step specifically includes the steps of:

the combination step is as follows: combining the hidden layer output characteristic vectors in a mode of adding and taking arithmetic mean and inputting the arithmetic mean to a feedforward neural network, and calculating a whole sentence classification vector, an entity upper context vector, an entity lower context vector and an entity vector;

splicing: and splicing the whole sentence classification vector, the entity upper vector, the entity lower vector and the entity vector, and then fusing to obtain the fusion characteristic vector.

As a further development of the invention, the model parameters are adjusted using a back-propagation gradient descent.

Based on the same invention idea, the invention also discloses a BERT-based multi-feature fusion entity sentiment analysis system based on any one of the invention multi-feature fusion entity sentiment analysis method disclosed by the invention creation,

the BERT-based multi-feature fusion entity emotion analysis system comprises:

the marking module is used for marking the text to acquire entity emotion data;

the training module is used for carrying out model training based on the entity emotion data to obtain an entity emotion classification model;

and the prediction module is used for carrying out entity emotion analysis on the predicted text by using the entity emotion classification model.

As a further improvement of the present invention, the labeling module specifically includes:

the construction unit is used for constructing the labeling platform;

and the input unit is used for inputting the text into the marking platform to carry out artificial entity and entity emotion marking so as to acquire the entity emotion data.

As a further improvement of the present invention, the building unit specifically includes:

a rule definition unit for defining a labeling rule;

a label definition unit that defines a label;

example given unit, given annotation example.

Compared with the prior art, the invention has the following beneficial effects:

1. a multi-feature fusion entity emotion analysis method based on BERT is provided, and entity semantic information, entity position information, sentence semantic information and entity context feature information are fused in an appropriate mode to improve the performance of an entity emotion classification task model;

2. on the entity emotion analysis task, entity emotion analysis is carried out by combining multiple feature vectors, so that the performance is effectively improved;

3. and by using multi-feature fusion, the feature fusion mode is simple, and the calculation complexity is reduced.

The details of one or more embodiments of the application are set forth in the accompanying drawings and the description below to provide a more thorough understanding of the application.

Drawings

The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the application and together with the description serve to explain the application and not to limit the application. In the drawings:

FIG. 1 is an overall flowchart of a method for analyzing emotion of a multi-feature fusion entity based on BERT according to an embodiment of the present invention;

FIG. 2 is a flowchart illustrating the overall method of the present invention;

FIG. 3 is a flowchart illustrating the overall process of step S1 disclosed in FIG. 1;

FIG. 4 is a flowchart illustrating the whole step S11 disclosed in FIG. 3;

FIG. 5 is a flowchart illustrating the whole step S2 disclosed in FIG. 1;

FIG. 6 is a diagram of entity emotion classification models disclosed in the present embodiment;

FIG. 7 is a flowchart illustrating the whole step S24 disclosed in FIG. 5;

FIG. 8 is a structural framework diagram of a BERT-based multi-feature fusion entity sentiment analysis system provided by the present embodiment;

fig. 9 is a block diagram of a computer apparatus according to an embodiment of the present invention.

In the above figures:

1. a labeling module; 2. a training module; 3. a prediction module; 11. a building unit; 12. an input unit; 111. a rule definition unit; 112. a tag definition unit; 113. instantiating a given cell; 21. a verification unit; 22. a marking unit; 23. a feature acquisition unit; 24. a feature fusion unit; 241. a combination unit; 242. A splicing unit; 25. a parameter acquisition unit; 80. a bus; 81. a processor; 82. a memory; 83. a communication interface.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the present application will be described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments provided in the present application without any inventive step are within the scope of protection of the present application.

It is obvious that the drawings in the following description are only examples or embodiments of the present application, and that it is also possible for a person skilled in the art to apply the present application to other similar contexts on the basis of these drawings without inventive effort. Moreover, it should be appreciated that in the development of any such actual implementation, as in any engineering or design project, numerous implementation-specific decisions must be made to achieve the developers' specific goals, such as compliance with system-related and business-related constraints, which may vary from one implementation to another.

Reference in the specification to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the specification. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Those of ordinary skill in the art will explicitly and implicitly appreciate that the embodiments described herein may be combined with other embodiments without conflict.

Unless defined otherwise, technical or scientific terms referred to herein shall have the ordinary meaning as understood by those of ordinary skill in the art to which this application belongs. Reference to "a," "an," "the," and similar words throughout this application are not to be construed as limiting in number, and may refer to the singular or the plural. The present application is directed to the use of the terms "including," "comprising," "having," and any variations thereof, which are intended to cover non-exclusive inclusions; for example, a process, method, system, article, or apparatus that comprises a list of steps or modules (elements) is not limited to the listed steps or elements, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus. Reference to "connected," "coupled," and the like in this application is not intended to be limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The term "plurality" as referred to herein means two or more. "and/or" describes an association relationship of associated objects, meaning that three relationships may exist, for example, "A and/or B" may mean: a exists alone, A and B exist simultaneously, and B exists alone. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship. Reference to the terms "first," "second," "third," and the like in this application merely distinguishes similar objects and is not to be construed as referring to a particular ordering of objects.

The present invention is described in detail with reference to the embodiments shown in the drawings, but it should be understood that these embodiments are not intended to limit the present invention, and those skilled in the art should understand that the functional, methodological, or structural equivalents of these embodiments or alternatives thereof fall within the scope of the present invention.

Before describing in detail the various embodiments of the present invention, the core inventive concepts of the present invention are summarized and described in detail by the following several embodiments.

The invention can carry out multi-feature fusion entity emotion analysis based on BERT, and improves the model performance by fusing entity semantic information, entity position information, sentence semantic information and entity context feature information in a proper way.

The first embodiment is as follows:

referring to fig. 1 to 7, the present example discloses a concrete implementation of a BERT-based multi-feature fusion entity sentiment analysis method (hereinafter referred to as "method").

Specifically, referring to fig. 1 and fig. 2, the method disclosed in this embodiment mainly includes three major parts, namely, entity and entity emotion labeling, model training, and model prediction, and includes the following steps:

and step S1, labeling the text to acquire entity emotion data.

Specifically, in some embodiments, step S1 shown in fig. 3 specifically includes the following steps:

s11, constructing an annotation platform;

and S12, inputting the text into the marking platform to carry out artificial entity and entity emotion marking, and acquiring the entity emotion data.

Specifically, in some embodiments, step S11 shown in fig. 4 specifically includes the following steps:

s111, defining a labeling rule;

s112, defining a label;

and S113, giving an annotation example.

Specifically, the entity emotion task corpus is labeled by using a labeling platform. For example, in the sentence "see the long-term development potential of the scientific and technological science", the entity "the scientific and technological science" is labeled and the emotion of the entity "the scientific and technological science" is labeled as the positive.

Then, referring to fig. 5 and fig. 6, step S2 is executed, and model training is performed based on the entity emotion data to obtain an entity emotion classification model.

Specifically, in some embodiments, the step S2 specifically includes the following steps:

s21, carrying out data format verification on the entity emotion data, and dividing to obtain a training data set;

s22, marking the position information of the training data set;

s23, inputting the training data set containing the marking information into a BERT pre-training model to obtain hidden layer output feature vectors;

s24, acquiring a fusion feature vector based on the hidden layer output feature vector;

and S25, performing probability transformation on the fusion feature vector through softmax to obtain model parameters.

Specifically, data is input to a model, and model training is performed. Firstly, the position information mark is carried out on the given text containing the entity, and the position information mark comprises a whole sentence classification identifier mark, an entity starting position mark and an entity ending position mark. The specific position marking process is as follows: the whole sentence classification identification symbol "[ CLS ]" is inserted at the beginning of the sentence, and "[ EL ]" and "[ ER ]" are inserted to mark the beginning and the end of the entity respectively. Let l ength be the length of the sentence, and i, j be the starting position and ending position of the entity in the sentence after inserting all the identifiers, respectively. Also taking the sentence "see the long-term development potential of the Ministry technology" as an example, the sentence marked by the text position information should be the "CLS see the long-term development potential of the EL Ministry technology [ ER ].

Specifically, in step S23, the text containing the label information is input into the BERT pre-training model, and the BERT pre-training model is used to perform fine-tuning of a single-sentence task on the given corpus, so as to obtain the output feature vector (denoted as H) of the last hidden layer. The fine-tuned BERT model has stronger representation capability and better performance.

Specifically, step S24 shown in fig. 7 specifically includes the following steps:

s241, combining the hidden layer output characteristic vectors in a mode of adding and taking arithmetic mean and inputting the arithmetic mean to a feedforward neural network, and calculating a whole sentence classification vector, an entity upper context vector, an entity lower context vector and an entity vector;

and S242, splicing the whole sentence classification vector, the entity context vector and the entity vector, and then fusing to obtain the fusion feature vector.

Specifically, the hidden layer output feature vectors are combined in a mode of adding a plurality of vectors, taking arithmetic mean and inputting the sum of the vectors into a Feed-Forward Neural Network (Feed-Forward Neural Network), and a whole sentence classification vector H is calculated_clsPhysical context vector H_LEntity context vector H_RAnd an entity vector H_entity。

Specifically, "[ CLS ] according to the index]"hidden layer vector corresponding to mark is recorded as H_oThe entity vector is denoted as H from left to right_i～H_jThe physical context vector is denoted as H from left to right₁～H_i-2Entity context vector is denoted as H from left to right_j+2～H_length-1. In the calculation order, the formulas are respectively as follows:

whole sentence classification vector H_cls：H_cls＝W_cls[tanh(H_o)]+b_cls

Entity vector H_entity:

Physical context vector H_L:

Entity context vector H_R:

Wherein, W is a parameter matrix in the feedforward neural network, the matrix size is dxd, b is an offset matrix, the sizes are dx1, and W_cls∈R^d×d，W_entity∈R^d×d，b_cls∈R^d×1，b_entity∈R^d×1。

Specifically, the whole sentence classification vector H is calculated_clsEntity vector H_entityPhysical context vector H_LAnd entity context vector H_RAnd splicing, and inputting the spliced characteristic vectors into a new feedforward neural network for feature fusion to obtain a fused feature vector.

Specifically, the vector in which the plurality of features are fused is finally subjected to probability transformation by softmax, and a final prediction result is obtained. The calculation formula is as follows:

fused feature vector H_merge＝W_merge[concat(H_cls,H_L,H_entity,H_R)]+b_merge

Probability of feature vector p ═ softmax (H)_merge)

Wherein, W_merge∈R^L×4d，b_merge∈R^L×1And L is the number of entity classification tags.

Specifically, after all the feature vectors are subjected to probabilistic transformation, the label at the position corresponding to the maximum probability value is the "prediction" result of the current model parameter for the sentence. In addition, the loss between the result and the correct label is calculated using a cross entropy loss function, and model parameters are adjusted using a back-propagation gradient descent.

And then executing step S3, and performing entity emotion analysis on the predicted text by using the entity emotion classification model.

According to the BERT-based multi-feature fusion entity emotion analysis method disclosed by the embodiment of the application, entity semantic information, entity position information, sentence semantic information and entity context feature information are fused in a proper mode, and the performance of an entity emotion classification task model is improved; on the entity emotion analysis task, entity emotion analysis is carried out by combining multiple feature vectors, so that the performance is effectively improved; and by using multi-feature fusion, the feature fusion mode is simple, and the calculation complexity is reduced.

Example two:

in combination with the method for analyzing emotion of a multi-feature fusion entity based on BERT disclosed in the first embodiment, the present embodiment discloses a specific implementation example of a system for analyzing emotion of a multi-feature fusion entity based on BERT (hereinafter referred to as "system").

Referring to fig. 8, the system includes:

the marking module 1 marks the text to acquire entity emotion data;

the training module 2 is used for carrying out model training based on the entity emotion data to obtain an entity emotion classification model;

and the predicting module 3 is used for performing entity emotion analysis on the predicted text by using the entity emotion classification model.

Specifically, in some embodiments, the labeling module 1 specifically includes:

the construction unit 11 is used for constructing an annotation platform;

and the input unit 12 is used for inputting the text into the marking platform to carry out artificial entity and entity emotion marking, and acquiring the entity emotion data.

Specifically, in some embodiments, the building unit 11 specifically includes:

a rule definition unit 111 that defines a labeling rule;

a label definition unit 112 that defines a label;

the instance given unit 113, given an annotated instance.

Specifically, in some embodiments, the training module 2 specifically includes:

the checking unit 21 is used for checking the data format of the entity emotion data and dividing the entity emotion data to obtain a training data set;

a labeling unit 22 for labeling the training data set with position information;

the feature acquisition unit 23 is configured to input the training data set containing the label information into a BERT pre-training model to acquire a hidden layer output feature vector;

a feature fusion unit 24 configured to obtain a fusion feature vector based on the hidden layer output feature vector;

and the parameter acquisition unit 25 is used for performing probability transformation on the fusion feature vector through softmax to acquire model parameters.

Specifically, in some embodiments, the feature fusion unit 24 specifically includes:

the combination unit 241 combines the hidden layer output feature vectors in a mode of adding, taking arithmetic mean and inputting the arithmetic mean to a feedforward neural network, and calculates a whole sentence classification vector, an entity upper context vector, an entity lower context vector and an entity vector;

and the splicing unit 242 splices and fuses the whole sentence classification vector, the entity context vector and the entity vector to obtain the fusion feature vector.

For reference, the embodiment a description is given to the technical solutions of the same parts in the system and the method for analyzing emotion of a BERT-based multi-feature fusion entity disclosed in the first embodiment, and details are not repeated here.

Example three:

referring to fig. 9, this embodiment discloses an embodiment of a computer device. The computer device may comprise a processor 81 and a memory 82 in which computer program instructions are stored.

Specifically, the processor 81 may include a Central Processing Unit (CPU), or A Specific Integrated Circuit (ASIC), or may be configured to implement one or more Integrated circuits of the embodiments of the present Application.

Memory 82 may include, among other things, mass storage for data or instructions. By way of example, and not limitation, memory 82 may include a Hard Disk Drive (Hard Disk Drive, abbreviated to HDD), a floppy Disk Drive, a Solid State Drive (SSD), flash memory, an optical Disk, a magneto-optical Disk, tape, or a Universal Serial Bus (USB) Drive or a combination of two or more of these. Memory 82 may include removable or non-removable (or fixed) media, where appropriate. The memory 82 may be internal or external to the data processing apparatus, where appropriate. In a particular embodiment, the memory 82 is a Non-Volatile (Non-Volatile) memory. In particular embodiments, Memory 82 includes Read-Only Memory (ROM) and Random Access Memory (RAM). The ROM may be mask-programmed ROM, Programmable ROM (PROM), Erasable PROM (EPROM), Electrically Erasable PROM (EEPROM), Electrically rewritable ROM (EAROM), or FLASH Memory (FLASH), or a combination of two or more of these, where appropriate. The RAM may be a Static Random-Access Memory (SRAM) or a Dynamic Random-Access Memory (DRAM), where the DRAM may be a Fast Page Mode Dynamic Random-Access Memory (FPMDRAM), an Extended data output Dynamic Random-Access Memory (EDODRAM), a Synchronous Dynamic Random-Access Memory (SDRAM), and the like.

The memory 82 may be used to store or cache various data files for processing and/or communication use, as well as possible computer program instructions executed by the processor 81.

The processor 81 reads and executes the computer program instructions stored in the memory 82 to implement any one of the BERT-based multi-feature fusion entity emotion analysis methods in the above embodiments.

In some of these embodiments, the computer device may also include a communication interface 83 and a bus 80. As shown in fig. 9, the processor 81, the memory 82, and the communication interface 83 are connected via the bus 80 to complete communication therebetween.

The communication interface 83 is used for implementing communication between modules, devices, units and/or equipment in the embodiment of the present application. The communication port 83 may also be implemented with other components such as: the data communication is carried out among external equipment, image/data acquisition equipment, a database, external storage, an image/data processing workstation and the like.

Bus 80 includes hardware, software, or both to couple the components of the computer device to each other. Bus 80 includes, but is not limited to, at least one of the following: data Bus (Data Bus), Address Bus (Address Bus), Control Bus (Control Bus), Expansion Bus (Expansion Bus), and Local Bus (Local Bus). By way of example, and not limitation, Bus 80 may include an Accelerated Graphics Port (AGP) or other Graphics Bus, an Enhanced Industry Standard Architecture (EISA) Bus, a Front-Side Bus (FSB), a Hyper Transport (HT) Interconnect, an ISA (ISA) Bus, an InfiniBand (InfiniBand) Interconnect, a Low Pin Count (LPC) Bus, a memory Bus, a microchannel Architecture (MCA) Bus, a PCI (Peripheral Component Interconnect) Bus, a PCI-Express (PCI-X) Bus, a Serial Advanced Technology Attachment (SATA) Bus, a Video Electronics Bus (audio Electronics Association), abbreviated VLB) bus or other suitable bus or a combination of two or more of these. Bus 80 may include one or more buses, where appropriate. Although specific buses are described and shown in the embodiments of the application, any suitable buses or interconnects are contemplated by the application.

The computer device can implement multi-feature fusion entity sentiment analysis based on BERT, thereby implementing the method described in conjunction with FIG. 1.

In addition, in combination with the BERT-based multi-feature fusion entity emotion analysis method in the foregoing embodiments, the embodiments of the present application may provide a computer-readable storage medium to implement. The computer readable storage medium having stored thereon computer program instructions; the computer program instructions, when executed by a processor, implement any of the BERT-based multi-feature fusion entity sentiment analysis methods in the above embodiments.

The technical features of the embodiments described above may be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the embodiments described above are not described, but should be considered as being within the scope of the present specification as long as there is no contradiction between the combinations of the technical features.

In summary, the method for analyzing the multi-feature fusion entity emotion based on the BERT has the advantages that entity semantic information, entity position information, sentence semantic information and entity context feature information are fused in a proper mode, and the performance of an entity emotion classification task model is improved; on the entity emotion analysis task, entity emotion analysis is carried out by combining multiple feature vectors, so that the performance is effectively improved; and by using multi-feature fusion, the feature fusion mode is simple, and the calculation complexity is reduced.

The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.

Claims

1. A BERT-based multi-feature fusion entity emotion analysis method is characterized by comprising the following steps:

labeling: marking the text to obtain entity emotion data;

2. The BERT-based multi-feature fusion entity emotion analysis method of claim 1, wherein the labeling step specifically includes the steps of:

the construction steps are as follows: constructing an annotation platform;

3. The BERT-based multi-feature fusion entity emotion analysis method of claim 2, wherein the construction step specifically includes the steps of:

a rule definition step: defining an annotation rule;

and a label definition step: defining an annotation tag;

example given procedure: an instance of the annotation is given.

4. The BERT-based multi-feature fusion entity emotion analysis method of claim 1, wherein the training step specifically includes the steps of:

5. The BERT-based multi-feature fusion entity emotion analysis method of claim 4, wherein the position information labels in the labeling step include a sentence classification identifier label, an entity start position label, and an entity end position label.

6. The BERT-based multi-feature fusion entity emotion analysis method of claim 4, wherein the feature fusion step specifically includes the steps of:

7. The method for BERT-based multi-feature fusion entity emotion analysis of claim 4, wherein the model parameters are adjusted using back-propagation gradient descent.

8. A BERT-based multi-feature fusion entity sentiment analysis system is characterized by comprising:

the marking module is used for marking the text to acquire entity emotion data;

9. The BERT-based multi-feature fusion entity emotion analysis system of claim 8, wherein the labeling module specifically comprises:

the construction unit is used for constructing the labeling platform;

10. The BERT-based multi-feature fusion entity emotion analysis system of claim 9, wherein the construction unit specifically includes:

a rule definition unit for defining a labeling rule;

a label definition unit that defines a label;

example given unit, given annotation example.