KR20240094110A - Method, apparatus and system for analyzing emotions in aspect-based sentences using adaptive masked attention mechanism - Google Patents

Abstract

A device according to an embodiment receives a sentence for which sentiment analysis is to be performed, extracts text from the received sentence, and uses the text to generate local text data and a global text including local text and aspect terms. text) data, extract the part corresponding to the local text data from the global text data, convert the text data into a vector value using BERT, and obtain the result value by applying weight to the vector value of the aspect term. , you can use the result to analyze the emotion of the text within the sentence.

Description

Aspect-based emotion analysis method, device, and system using an adaptive masked attention mechanism {METHOD, APPARATUS AND SYSTEM FOR ANALYZING EMOTIONS IN ASPECT-BASED SENTENCES USING ADAPTIVE MASKED ATTENTION MECHANISM}

The following embodiments relate to technology for analyzing emotion within an aspect-based sentence using an adaptive masked attention mechanism.

Aspect-based Sentiment Analysis (ABSA) is an important research field in natural language processing that aims to analyze the emotional polarity of aspect terms present in input sentences. In experiments in recent years, many models have focused more on local text or local text aspect relationships by designing models that act directly on local text and then fusing features from global text.

The existing model had the problem of wasting memory space by using two pre-training models (BERT) to model each text and aspect term, and when considering weight allocation in the model, it did not consider the context relationship and followed the instructions of the token. Accordingly, there was a problem in that it was not possible to initiate a detailed natural language analysis method by assigning weights to the distance between aspect terms.

Embodiments propose a masking attention mechanism that acts on the local embedding part of the global embedding, based on the global attention mechanism, and uses two pre-trained models for local text and global text to transform the global into one pre-trained model. We aim to provide an aspect-based sentiment analysis method within sentences using an adaptive masked attention mechanism that can learn both text and local text features and thus improves model training efficiency.

According to one embodiment, a method performed by an apparatus includes receiving a sentence for which emotion analysis is to be performed and extracting text from the received sentence; Using the text to generate local text data and global text data including the local text and an aspect term; extracting a portion corresponding to the local text data from the global text data; Converting the text data into a vector value using BERT; Obtaining a result value by applying a weight to the vector value of the aspect term; It may include analyzing the emotion of the text in the sentence using the result value.

The weight is generated using an attention mechanism, and processing the embedded portion of the local text data using the attention mechanism to obtain dimension; calculating weights of different token characteristics assigned to the dimension using an adaptive mechanism; The step of sorting the weights and calculating the importance of the aspect term using the sorted weights may be further included.

The step of converting to a vector value includes determining whether the length of the aspect term exceeds 1, maintaining the modeling result as is if the length of the aspect term is 1, and determining whether the length of the aspect term is 1. If it exceeds, a step of correcting the modeling result by supplementing 0 by the shortened length of the modeling result may be included.

The device according to one embodiment may be combined with hardware and controlled by a computer program stored in a medium to execute any one of the above-described methods.

Embodiments propose a masking attention mechanism that acts on the local embedding part of the global embedding, based on the global attention mechanism, and uses two pre-trained models for local text and global text to transform the global into one pre-trained model. Features of both text and local text can be learned, and therefore, it is possible to provide an aspect-based sentiment analysis method within sentences using an adaptive masked attention mechanism that improves the training efficiency of the model.

1 is a diagram for explaining the configuration of a system according to an embodiment.
Figure 2 is a diagram illustrating a method of analyzing emotions in an aspect-based sentence using an adaptive masked attention mechanism according to an embodiment.
Figure 3 is a diagram for explaining a process for calculating weights according to an embodiment.
Figure 4 is an exemplary diagram of the configuration of a device according to an embodiment.

Hereinafter, embodiments will be described in detail with reference to the attached drawings. However, various changes can be made to the embodiments, so the scope of the patent application is not limited or limited by these embodiments. It should be understood that all changes, equivalents, or substitutes for the embodiments are included in the scope of rights.

Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only and may be modified and implemented in various forms. Accordingly, the embodiments are not limited to the specific disclosed form, and the scope of the present specification includes changes, equivalents, or substitutes included in the technical spirit.

Terms such as first or second may be used to describe various components, but these terms should be interpreted only for the purpose of distinguishing one component from another component. For example, a first component may be named a second component, and similarly, the second component may also be named a first component.

When a component is referred to as being “connected” to another component, it should be understood that it may be directly connected or connected to the other component, but that other components may exist in between.

The terms used in the examples are for descriptive purposes only and should not be construed as limiting. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as “comprise” or “have” are intended to designate the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, but are not intended to indicate the presence of one or more other features. It should be understood that this does not exclude in advance the possibility of the existence or addition of elements, numbers, steps, operations, components, parts, or combinations thereof.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by a person of ordinary skill in the technical field to which the embodiments belong. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related technology, and should not be interpreted in an ideal or excessively formal sense unless explicitly defined in the present application. No.

In addition, when describing with reference to the accompanying drawings, identical components will be assigned the same reference numerals regardless of the reference numerals, and overlapping descriptions thereof will be omitted. In describing the embodiments, if it is determined that detailed descriptions of related known technologies may unnecessarily obscure the gist of the embodiments, the detailed descriptions are omitted.

Embodiments may be implemented in various types of products such as personal computers, laptop computers, tablet computers, smart phones, televisions, smart home appliances, intelligent vehicles, kiosks, and wearable devices.

1 is a diagram for explaining the configuration of a system according to an embodiment.

The system according to one embodiment may include a user terminal 10 and a device 30 that can communicate with each other through a communication network.

First, a communication network can be configured regardless of the communication mode, such as wired or wireless, and can be implemented in various forms to enable communication between servers and between servers and terminals.

The user terminal 10 may be a terminal used by a user who provides a sentence for which emotions are to be analyzed according to the present invention. The user terminal 10 may be a desktop computer, laptop, tablet, smartphone, etc. For example, as shown in FIG. 1, the user terminal 10 may be a smartphone and may be employed differently depending on the embodiment.

The user terminal 10 may be configured to perform all or part of the calculation function, storage/reference function, input/output function, and control function of a typical computer. The user terminal 10 may be configured to communicate with the device 30 wired or wirelessly.

The user terminal 10 is connected to a web page operated by a person or organization providing services using the device 30, or an application developed and distributed by a person or organization providing services using the device 30. Can be installed. The user terminal 10 may be linked to the device 30 through a web page or application.

The user terminal 10 can access the device 30 through a web page or application provided by the device 30.

The singular expressions recited in the claims may be understood to include the plural. For example, the user in a claim may refer to one user or two or more users.

The device 30 can provide an aspect-based emotion analysis method within a sentence using an adaptive masked attention mechanism.

The device 30 may be an own server owned by a person or organization that provides services using the device 30, a cloud server, or a p2p (peer-to-peer) collection of distributed nodes. It may be possible. The device 30 may be configured to perform all or part of the calculation function, storage/reference function, input/output function, and control function of a typical computer.

The device 30 may be configured to communicate with the user terminal 10 wired or wirelessly, control the operation of the user terminal 10, and control which information to display on the screen of the user terminal 10. .

Meanwhile, for convenience of explanation, only the user terminal 10 is shown in FIG. 1, but the number of terminals may vary depending on the embodiment. There are no particular restrictions on the number of terminals and the number of printer devices as long as the processing capacity of the device 30 allows.

According to one embodiment, a database may be provided within the device 30, but the present invention is not limited to this, and the database may be configured separately from the device 30. Device 30 may include multiple artificial neural networks for performing machine learning algorithms.

In the present invention, artificial intelligence (AI) refers to technology that imitates human learning ability, reasoning ability, perception ability, etc. and implements this with a computer, and includes concepts such as machine learning and symbolic logic. may include. Machine Learning (ML) is an algorithmic technology that classifies or learns the characteristics of input data on its own. Artificial intelligence technology is a machine learning algorithm that analyzes input data, learns the results of the analysis, and makes judgments or predictions based on the results of the learning. Additionally, technologies that mimic the functions of the human brain, such as cognition and judgment, using machine learning algorithms can also be understood as the category of artificial intelligence. For example, technical fields such as verbal understanding, visual understanding, reasoning/prediction, knowledge representation, and motion control may be included.

Machine learning can refer to the process of training a neural network model using experience processing data. Machine learning can mean that computer software improves its own data processing capabilities. A neural network model is built by modeling the correlation between data, and the correlation can be expressed by a plurality of parameters. A neural network model extracts and analyzes features from given data to derive correlations between data. Repeating this process to optimize the parameters of the neural network model can be called machine learning. For example, a neural network model can learn the mapping (correlation) between input and output for data given as input-output pairs. Alternatively, a neural network model may learn the relationships by deriving regularities between given data even when only input data is given.

An artificial intelligence learning model or neural network model may be designed to implement the human brain structure on a computer, and may include a plurality of network nodes with weights that simulate neurons of a human neural network. A plurality of network nodes may have a connection relationship with each other by simulating the synaptic activity of neurons in which neurons exchange signals through synapses. In an artificial intelligence learning model, multiple network nodes are located in layers of different depths and can exchange data according to convolutional connection relationships. The artificial intelligence learning model may be, for example, an artificial neural network model (Artificial Neural Network), a convolution neural network (CNN) model, etc. As an example, an artificial intelligence learning model may be machine-learned according to methods such as supervised learning, unsupervised learning, and reinforcement learning. Machine learning algorithms for performing machine learning include Decision Tree, Bayesian Network, Support Vector Machine, Artificial Neural Network, and Ada-boost. , Perceptron, Genetic Programming, Clustering, etc. can be used.

Among them, CNN is a type of multilayer perceptrons designed to use minimal preprocessing. CNN consists of one or several convolution layers and general artificial neural network layers on top of them, and additionally utilizes weights and pooling layers. Thanks to this structure, CNN can fully utilize input data with a two-dimensional structure. Compared to other deep learning structures, CNN shows good performance in both video and audio fields. CNNs can also be trained via standard back propagation. CNNs have the advantage of being easier to train and using fewer parameters than other feedforward artificial neural network techniques.

Convolutional networks are neural networks that contain sets of nodes with bound parameters. The increasing size of available training data and the availability of computational power, combined with algorithmic advances such as piecewise linear unit and dropout training, have led to significant improvements in many computer vision tasks. For extremely large data sets, such as those available for many tasks today, overfitting is not critical, and increasing the size of the network improves test accuracy. Optimal use of computing resources becomes a limiting factor. For this purpose, distributed, scalable implementations of deep neural networks can be used.

According to one embodiment, by applying the BERT language model, an emotion analysis model of an appropriate size that can sufficiently satisfy accuracy and processing speed can be implemented.

Figure 2 is a diagram illustrating a method of analyzing emotions in an aspect-based sentence using an adaptive masked attention mechanism according to an embodiment.

The BERT language model is one of the language models based on the Transformer structure, and is a method of pre-training using large amounts of data to identify various morphological and semantic elements that can be obtained from big data. This is a model that can be trained on a model.

In other words, the BERT language model can learn various lexical features that do not exist in the training data through prior learning, so the problem of heterogeneity that does not appear in the training data occurs during the fine-tuning process of adjusting the model's parameters. It also provides the advantage of being able to solve problems.

The BERT language model 400 receives a given sentence as input, first embeds it in token units through an input embedding layer, and then passes through a transformer layer to express the context through encoding that takes into account the location information of the token. (contextual representation) structure.

The transformer layer is composed of a self-attention model instead of models such as CNN or RNN, and the BERT language model embeds the language using only the encoder among the encoders and decoders of the transformer.

As this BERT language model is implemented with a model structure that performs prior learning, its performance is superior compared to models such as CNN or LSTM.

The BERT language model has higher performance as the model grows, but in situations where it is essential to compress or reduce the model to use it in a service, it is necessary to optimize the model by adjusting the thresholds of performance and processing speed.

The emotion analysis system according to the embodiment includes emotion analysis for products (e.g., shopping, movies, music, etc.), emotion analysis for businesses (e.g., restaurants, hotels, companies, etc.), and people (e.g., It can be used for emotional analysis of (celebrities, politicians, etc.), emotional analysis of speaker remarks, etc.

Local text can be defined as [CLS] + Sentence + [SEP], and global text can be defined as [CLS] + Sentence + aspect term + [SEP].

To facilitate subsequent work, the device 30 may model the sentence using the BERT model and then reduce the length of the aspect term in the content to 1, and if the length of the original aspect term is 1, it may be maintained as is.

Here, E represents the modeled content, E' represents the max pooling processed content, and 0 can be added at the end of the shortened length to ensure that the length of the input is always the same.

The device 30 may store the local text length and the side term length during data processing to cut the local embedding portion from the global embedding. Ei={e1,e2,... ,en}, as can be seen from the above formula, n represents the length of the local text, m represents the length of the side term, and the length of the side with the largest pool in the previous section is 1 Since it is shortened to , the length of local text to be processed here may be n-m+1.

Therefore, the processed local embedding is E’l={e’1, e’2, ..., e'n-m-1}. We can process Q in the same way to obtain Q'.

After BERT modeling, the device 30 can process only the local embedding part of the global embedding using an attention mechanism and obtain an alpha value, excluding the last aspect term part of the global text, and the obtained dimensions are [n-m-1, n-m-1].

At this time, n in the first dimension may represent the current sequence length, and n in the second dimension may represent the weight of each token assigned to n tokens.

Accordingly, the device 30 can find the vector of n corresponding to the side vector in the second dimension in the first dimension and sort the values therein from large to small.

Figure 3 is a diagram for explaining a process for calculating weights according to an embodiment.

αij can represent the result of attention calculating local text from global text.

The device 30 can calculate a result value that weights the local embedding obtained when processing local text using an attention mechanism that is input to the classifier as part of the final result.

Figure 4 is an exemplary diagram of the configuration of a device according to an embodiment.

Device 30 according to one embodiment includes a processor 31 and memory 32. The device 30 according to one embodiment may be the server or terminal described above. The processor 31 may include at least one device described above with reference to FIGS. 1 to 3 or may perform at least one method described with reference to FIGS. 1 to 3 . The memory 32 may store information related to the above-described method or store a program in which the above-described method is implemented. Memory 32 may be volatile memory or non-volatile memory.

The processor 31 can execute programs and control the device 30. The code of the program executed by the processor 31 may be stored in the memory 32. The device 30 is connected to an external device (eg, a personal computer or a network) through an input/output device (not shown) and can exchange data.

The embodiments described above may be implemented with hardware components, software components, and/or a combination of hardware components and software components. For example, the devices, methods, and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, and a field programmable gate (FPGA). It may be implemented using one or more general-purpose or special-purpose computers, such as an array, programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. A processing device may execute an operating system (OS) and one or more software applications that run on the operating system. Additionally, a processing device may access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, a single processing device may be described as being used; however, those skilled in the art will understand that a processing device includes multiple processing elements and/or multiple types of processing elements. It can be seen that it may include. For example, a processing device may include a plurality of processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are possible.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., singly or in combination. Program instructions recorded on the medium may be specially designed and configured for the embodiment or may be known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -Includes optical media (magneto-optical media) and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions include machine language code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing unit to operate as desired, or may be processed independently or collectively. You can command the device. Software and/or data may be used on any type of machine, component, physical device, virtual equipment, computer storage medium or device to be interpreted by or to provide instructions or data to a processing device. , or may be permanently or temporarily embodied in a transmitted signal wave. Software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

Although the embodiments have been described with limited drawings as described above, those skilled in the art can apply various technical modifications and variations based on the above. For example, the described techniques are performed in a different order than the described method, and/or components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or other components are used. Alternatively, appropriate results may be achieved even if substituted or substituted by an equivalent.

Therefore, other implementations, other embodiments, and equivalents of the claims also fall within the scope of the following claims.

Claims (3)

In a method performed by a device,
Receiving a sentence for which emotion analysis is to be performed from the user's terminal and extracting text from the received sentence;
Generating local text data and global text data including the local text and aspect terms using the text;
extracting a portion corresponding to the local text data from the global text data;
Converting the text data into a vector value using BERT;
Obtaining a result value by applying a weight to the vector value of the aspect term;
Comprising the step of analyzing the emotion of the text in the sentence using the result value.
An aspect-based emotion analysis method in sentences using an adaptive masked attention mechanism. According to paragraph 1,
The weight is,
Generated using the attention mechanism,
Processing the embedded part of the local text data using the attention mechanism to obtain dimension;
calculating weights of different token characteristics assigned to the dimension using an adaptive mechanism;
Sorting the weights and calculating the importance of the aspect term using the sorted weights.
An aspect-based emotion analysis method in sentences using an adaptive masked attention mechanism. According to paragraph 1,
The step of converting to the vector value is,
Determining whether the length of the aspect term exceeds 1,
If the length of the aspect term is 1, maintaining the modeling result as is, and
When the length of the aspect term exceeds 1, including the step of correcting the modeling result by supplementing 0 by the shortened length of the modeling result.
An aspect-based emotion analysis method in sentences using an adaptive masked attention mechanism.