EA041027B1 - METHOD AND SYSTEM FOR RETRIEVING NAMED ENTITIES - Google Patents

Description

Technical field

The present invention relates, in general, to the field of computer technology, and in particular to a method and system for extracting named entities. The technical solution finds application in many areas related to the processing of texts in natural languages and extracting information from text.

State of the art

At the moment, the task of extracting named entities from text data is required in a wide range of areas and applications. In particular, when building dialog systems (chatbots, smart assistants), the extraction of named entities is necessary for a correct understanding of the meaning of the text, user intentions and the formation of a response to user requests.

In industrial applications where document classification is required, named entity extraction can be used to generate new text features to improve classification performance in real-world applications.

The task of extracting named entities is a subtask in a number of areas, such as the analysis of text documents, classification of reviews, clustering of text collections, in dialog systems, etc. Currently, a number of approaches are used to solve the problem of extracting named entities from texts in Russian, each of which has its own advantages and disadvantages:

Named Entity Recognition (NER) by DeepPavlov, URL: http://docs.deeppavlov.ai/en/master/features/models/ner.html. The solution is based on additional training of the RuBERT language model for the task of extracting named entities. The system has more than 100 million trainable parameters. Training with this model requires large amounts of GPU memory (more than 3 GB);

Application of a Hybrid Bi-LSTM-CRF model to the task of Russian Named Entity Recognition. URL: https://arxiv.org/pdf/1709.09686.pdf. The solution is based on the concatenation of a Bi-LSTM char layer, a Token Embedding layer for encoding the input sequence, and a Bi-LSTM-CRF layer for entity prediction. This solution shows worse quality than modern models based on large language models, however, it takes up much less space in memory during training;

Deep-NER: named entity recognizer based on deep neural networks and transfer learning. The model is based on ELMO (Matthew E. Peters, Mark Neumann, Mohitlyyer, Matt Gardner, Christopher Clark, Kenton Lee, Luke Zettlemoyer. Deep contextualized word representations. 2018. URL: https://arxiv.org/abs/1802.05365) or BERT language model. Next comes the BiLSTM-CRF layer for predictions. The system has more than 100 million trainable parameters and takes up more than 3-4 GB of GPU memory in memory during training. Works worse than the model from DeepPavlov.

Tomita parser. URL: https://yandex.ru/dev/tomita/. The Tomita parser is designed to extract structured data from natural language text. Extraction of facts occurs with the help of context-free grammars and keyword dictionaries. The parser allows you to write your own grammars and add dictionaries for the desired language. It can be used to retrieve named entities. The main disadvantage is the complexity of writing rules: for each new entity, a new group of rules is required, which requires the time of a specialist. However, this tool does not require training data. The main disadvantages of the above neural network models are that they require large amounts of memory for training and prediction and the availability of appropriate equipment. Using the tomitaparser requires a highly skilled grammar writer.

Disclosure of invention

The technical problem or task posed in this technical solution is to create a new efficient, simple and reliable method for extracting named entities from the text in Russian. At the same time, the solution should provide high quality extraction of named entities and require as little GPU memory as possible (less than 2 GB). Also, the solution should be able to solve the problem of classifying named entities (at the same time - Joint learning (Bing Liu, Lane Ian. 2016. Attention-based recurrent neural network models for joint intent detection and slot filling. arXiv:1609.01454) or separately).

The technical result is to increase the accuracy of predicting named entities. The specified technical result is achieved due to the implementation of a method for extracting named entities from textual information, performed by at least one computing device, comprising the steps of:

receive text information;

split the text into words;

performing tokenization of the text to obtain a sequence of tokens;

form by means of a neural network for the received sequence of tokens a set of vectors;

based on the obtained set of vectors, a vector representation of the sequence of tokens is formed;

by comparing the indicators of the obtained vector representation of the sequence

- 1 041027 tokens with predetermined indicators of vectors obtained as a result of training the neural network carry out the prediction of named entities for the vector representation of the sequence of tokens;

recognize the named entities obtained at the previous stage by selecting the word label.

In one of the particular examples of the implementation of the method, a vector representation of a sequence of tokens based on a set of vectors is formed by calculating the weighted sum or average values of the indicators of the vectors contained in the set of vectors.

In another particular embodiment of the method, the step is additionally performed, at which the dimension of the vectors contained in the received vector representation of the sequence of tokens is corrected in order to obtain a vector representation of the sequence of tokens with a given dimension.

In another particular embodiment of the method, the steps are additionally performed, at which:

based on the indicators of the vectors contained in the vector representation of the sequence of tokens, and the indicators of the previous vectors in the said sequence, dependencies of the indicators of the token vectors are determined by means of a recurrent neural network layer;

adding information about the dependencies of indicators of vectors of tokens to the representation of the sequence of tokens by generating a new vector representation of the sequence of tokens.

In another particular embodiment of the method, the steps are additionally performed, at which:

determine for each vector in the vector representation of the sequence of tokens the dependence of its indicators on the indicators of other vectors;

based on the indicators of the vector representation of the sequence of tokens and the data on the dependencies of the indicators of the vectors determined in the previous step, a new vector representation of the sequence of tokens is generated.

In another particular embodiment of the method, the final word label can be determined by neural networks by voting.

In another particular embodiment of the method, the steps are additionally performed, at which:

on the basis of indicators of the vector representation of the sequence of tokens, the maximum values of these indicators for all vectors are determined and, on their basis, a vector is formed containing the maximum values of the indicators of the obtained vector representation of the sequence of tokens;

on the basis of indicators of the vector representation of the sequence of tokens, the average values of these indicators for all vectors are determined and, on their basis, a vector is formed containing the average values of the indicators of the obtained vector representation of the sequence of tokens;

based on the indices of the vector containing the maximum values of the indices of the vector representation of the sequence of tokens, the vector containing the average values of the indices of the obtained vector representation of the sequence of tokens, and the indices of the last vector in the vector representation of the sequence of tokens, the resulting vector is formed;

carry out the classification of the resulting vector to determine the class of the vector representation of the sequence of tokens. In another preferred embodiment of the claimed solution, a named entity retrieval system is provided, comprising at least one computing device and at least one memory device containing machine-readable instructions that, when executed by at least one computing device, perform the above method.

Brief description of the drawings

The features and advantages of the present technical solution will become apparent from the following detailed description of the invention and the accompanying drawings, in which:

in fig. 1 shows the general scheme of interaction between the elements of the system for extracting named entities;

in fig. Figure 2 shows an example of a general view of a named entity extraction system.

Implementation of the invention

The concepts and terms necessary for understanding this technical solution will be described below.

In this technical solution, a system means, among other things, a computer system, a computer (electronic computer), CNC (numerical control), PLC (programmable logic controller), computerized control systems and any other devices capable of performing a given, clearly a certain sequence of operations (actions, instructions).

A command processing device is an electronic unit, a computing device, or an integrated circuit (microprocessor) that executes machine instructions (programs).

The command processor reads and executes machine instructions (programs) from one

- 2 041027 or more storage devices. The role of a storage device can be, but not limited to, hard drives (HDD), flash memory, ROM (read only memory), solid state drives (SSD), optical drives.

Program - a sequence of instructions intended for execution by a computer control device or a command processing device.

Database (DB) - a collection of data organized in accordance with a conceptual structure that describes the characteristics of this data and the relationship between them, and such a collection of data that supports one or more areas of application (ISO / IEC 2382: 2015, 2121423 database).

An entity is any distinguishable object (an object that we can distinguish from another), information about which needs to be stored in the database. An entity instance is a specific instance of that entity. For example, Employee Ivanov can be a representative of the Employee entity. Entity instances must be distinguishable, i.e. entities must have some properties that are unique to each instance of that entity. A named entity is an object of a specific type that has a name, title, or identifier. What types the system allocates is determined within the framework of a specific task. For a news domain, these are usually people (PER), places (LOC), organizations (ORG) and miscellaneous, objects of a wide range: events, slogans, etc.

In accordance with the diagram shown in Fig. 1, the named entity extraction system 100 includes interconnected: a text preprocessing and tokenization module 10; a vector generating unit 20; a module 30 for generating a vector representation of a sequence of tokens; a module 40 for determining the dependence of indicators of token vectors; a module 50 for determining types of dependencies between tokens in a token vector representation sequence; a vector dimension correction module 60; a named entity prediction module 70 for tokens; a token vector representation sequence class prediction module 80; and a named entity recognition module 90 .

The text preprocessing and tokenization module 10 may be implemented on the basis of at least one computing device equipped with appropriate software and include a set of models for text tokenization, for example, a set of WordPiece models (see Yonghui Wu, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Mike Schuster, Zhifeng Chen 2016. Googles neural machine translation system: Bridging the gap between human and machine translation, volume arXiv:1609.08144).

The vector generation module 20 may be implemented based on the BERT language model. BERT is a neural network from Google that has shown state-of-the-art results by a wide margin on a number of tasks. With BERT, you can create AI programs for natural language processing, answer free-form questions, create chat bots, automatic translators, analyze text, and so on. The module 30 for generating a vector representation of a sequence of tokens can be implemented on the basis of at least one computing device equipped with appropriate software for determining the weighted sum or average (depending on the configuration) of the values of the indicators of vectors for generating a vector representation of a sequence of tokens.

The module 40 for determining the dependence of indicators of vectors of tokens can be implemented on the basis of a recurrent neural network (RNN) configured to determine recurrent dependencies between indicators of vectors based on the values of the indicators of the vectors contained in the vector representation of the sequence of tokens. The calculation graph of recurrent neural networks can also contain cycles that reflect the dependence of the value of a variable at the next point in time on its current value.

The module 50 for determining the types of dependencies between the tokens in the sequence of the vector representation of the tokens can be implemented on the basis of a neural network containing a Multi-Head attention neural network layer. Multi-head attention is a special layer that allows each input vector to interact with other words through the attention mechanism, instead of passing hidden state parameters as in RNNs or adjacent words as in CNNs. It is given Query vectors as input, and several Key and Value pairs (in practice, Key and Value are always the same vector). Each of them is transformed by a trainable linear transformation, and then the scalar product Q is calculated with all K in turn, the result of these scalar products is run through softmax, and with the resulting weights, all vectors are summed into a single vector. The module 60 for adjusting the dimension of the vector representation of the sequence of tokens can be implemented on the basis of the linear layers of the neural network, designed to process the indicators of the vectors to adjust their dimensions. The weighting factors for the line layers can be specified by the module 60 designer during its design. Named entity prediction module 70 for tokens may be implemented based on at least one neural network pre-trained on specific sets of named entities and their corresponding sets of vectors. In particular, the softmax layer, widely known in the art, can be used to predict named entities (see,

- 3 041027 for example, https://ru.wikipedia.org/wiki/Softmax).

Token vector representation class prediction module 80 can be implemented based on at least one neural network configured to perform subsampling operations on the maximum (Max Pooling) and average (Average Pooling) value, containing at least one linear layer for adjusting the dimension of the vectors and a softmax layer. Named entity recognition module 90 may be implemented with at least one computing device capable of converting predicted named entity data into words. Said conversion algorithm may be predetermined by the developer of said module 90 during its design.

At the first stage of operation of the named entity extraction system 100, text is received into the text preprocessing and tokenization module 10, for example, from data source 1. Data source 1 can be at least one database (DB) that contains text information, or source 1 data can be a user device, for example, a laptop or desktop computer, mobile phone or smartphone, tablet, etc. The text preprocessing and tokenization module 10 splits the text into words by the space between words and tokenizes it using the WordPiece model to obtain a sequence of tokens. For example, for the text I want to issue a debit card, the sequence of tokens will be: X', '##ochu', 'o', '##for', '##mi', '##t', 'de', '# #bet', '##new', 'card', '##y'. Also, said module 10 adds a special token [CLS] to the beginning of the token sequence.

Accordingly, the generated sequence of tokens is further transmitted by the mentioned module 10 to the vector generation module 20, which is using methods known from the prior art (see, for example, the article Anton Emelyanov, Ekaterina Artemova. Gapping parsing using pretrained embeddings, attention mechanism and NCRF. 2019. Computational Linguistics and Intellectual Technologies Papers from the Annual International Conference Dialogue (2019) Issue 18 Supplementary volume Pages 21-30 URL: http://www.dialog-21.ru/media/4870/-dialog2019scopusvolplus.pdf) forms for a sequence of tokens, a set of vectors, for example, 12 vector representations of the sequence of tokens, after which the resulting set of vectors is sent by the mentioned module 20 to the module 30 for generating a vector representation of the sequence of tokens. Based on the obtained set of vectors, the module 30 generates a vector representation of the sequence of tokens, for example, by calculating the weighted sum or average values of the indicators of the 12 mentioned vector representations of the sequence of tokens.

Next, the vector representation of the sequence of tokens obtained by module 30 is sent to the dimensionality correction module 60, which corrects the dimension of the vectors contained in the received vector representation of the sequence of tokens, for example, by multiplying the indicators of the vectors by the generated random non-integer number, to obtain a vector representation of the sequence of tokens with a given dimension.

In an alternative implementation of the proposed solution, the obtained vector representation of the sequence of tokens by the mentioned module 30 can be sent to the module 40 for determining the dependence of the indicators of the vectors of tokens, which, based on the indicators of the vectors contained in the vector representation of the sequence of tokens, and the indicators of the previous vectors in the mentioned sequence, by means of a recurrent layer defines the dependencies of indicators of vectors of tokens, which can be expressed as vectors. Information about the dependencies of indicators of token vectors is added by said module 40 to the vector representation of the sequence of tokens by generating a new vector representation of the sequence of tokens, after which the vector representation of the sequence of tokens generated by module 40 with information about the dependencies of indicators of token vectors is sent to the module 50 for determining types of dependencies between tokens, or to the module 60 for adjusting the dimension of the vector representation of the sequence of tokens for its processing in the manner described earlier. The module 50 for determining the types of dependencies between tokens, when obtaining a vector representation of a sequence of tokens for each vector by methods known from the prior art, determines the dependence of its indicators on the indicators of other vectors included in the vector representation of a sequence of tokens, which can also be expressed as vectors, after which the said module 50 corrects vector representation of the sequence of tokens by generating, based on the indicators of the vector representation of the sequence of tokens and data on the dependencies of the indicators of the vectors of a new vector representation of the sequence of tokens. Dependency data on vector indicators characterize the types of dependencies between tokens contained in the vector representation of a sequence of tokens and may indicate the presence of a semantic or morphological dependency between tokens. Accordingly, the generated vector representation of the sequence of tokens is sent by the module 50 to the module 60 for adjusting the dimension of the vector representation of the sequence of tokens for processing in the manner described earlier.

After the vector representation of the sequence of tokens with a given dimension has been generated by the mentioned module 60, it is sent to the module 70 for predicting named

- 4 041027 entities for tokens. The token named entity prediction module 70 performs named entity prediction for the vector representation of the token sequence based on the comparison of the metrics of the obtained vector representation of the token sequence with predetermined vector metrics obtained as a result of training the neural network. For example, for the word debit, the following sequence of labels of named entities in IO markup can be defined: 'I_DEB_CARD', 'I_DEB_CARD', 'I_O'. The label O means that there is no entity (in this case, the model made a mistake on one token). The final prediction of the named entity for the word debit will be DEB_CARD.

Further, the data about the named entities is sent by the mentioned module 70 to the named entity recognition module 90, which selects a word label for each named entity using methods known from the prior art, and the final label of the word can be determined by means of neural networks by voting, and the final label is selected as , which has the largest number of them. For example, the label I_DEB_CARD could have 2 votes, and the label I_O could have 1 vote. Accordingly, the final prediction of the named entity label for the word will also be DEB_CARD.

Also, additionally, a vector representation of a sequence of tokens with a given dimension can be processed by the sequence class prediction module 80 of the vector representation of tokens. The mentioned module 80, based on the indicators of the vector representation of a sequence of tokens with a given dimension, received from the previously mentioned module 60, using the Max Pooling operation, determines the maximum values of these indicators for all vectors and, based on them, forms a vector containing the maximum values of the indicators of the obtained vector representation of the sequence of tokens . Also, the mentioned module 80, based on the indicators of the vector representation of the sequence of tokens with a given dimension, using the Average Pooling operation, determines the average values of these indicators for all vectors and, on their basis, forms a vector containing the average values of the indicators of the obtained vector representation of the sequence of tokens, after which, based on the indicators of the vector , containing the maximum values of the indicators of the vector representation of the sequence of tokens, a vector containing the average values of the indicators of the obtained vector representation of the sequence of tokens, and the indicators of the last vector in the vector representation of the sequence of tokens, said module 80 forms the resulting vector, for example, by means of a concatenation operation. Next, the resulting vector is processed by a linear layer included in module 80 to adjust its dimension in the manner described earlier, and a softmax layer to classify the resulting vector and determine, based on the classification result, the class for the vector representation of the sequence of tokens for which the resulting vector was generated.

Information about the class of the vector representation of the sequence of tokens may indicate, for example, the presence or absence of named entities in the text information or the type of user request (intention) contained in the text information for which the said vector representation of the sequence of tokens was generated. For example, if the text information is text on my debit card number xxxx xxxx xxxx xxxx there were transactions that I didn't do. What to do?, in a marked-up form, which will be presented as O O B_DEB_CARD I_DEB_CARD O B_NUM_CARD I_NUM_CARD O O O O O O O O O O O, then the vector representation of the sequence of tokens generated for this textual information will be defined by module 80 with the call_operator class, indicating the type of user request - operator call. If the text information will be the text I want a debit card, then the class will be defined as want_card, indicating the type of request - card design. Also, the type of user request may indicate a desire to receive a loan or other service. The word label data by the named entity recognition module 90 and the token sequence vector representation classification data by the token vector representation sequence class prediction module 80 can then be sent to the data storage device 2 for subsequent display to the user upon request. Thus, due to the fact that the vector representation of the sequence of tokens is formed on the basis of a set of vectors obtained by the neural network as a result of processing the data of the sequence of tokens, and the prediction of named entities for the vector representation of the sequence of tokens is performed by comparing the indicators of the obtained vector representation of the sequence of tokens with predetermined indicators of vectors obtained as a result of neural network training increase the accuracy of predicting named entities. Also, the prediction accuracy of named entities is additionally ensured due to the fact that the prediction takes into account information about the dependencies of the indicators of token vectors on other vectors contained in the vector representation of the sequence of tokens.

In general terms (see Fig. 2), the system (200) for extracting named entities contains one or more processors (201), memory facilities, such as RAM (202) and ROM (203), I / O interfaces connected by a common information exchange bus (204), input/output devices (205), and

-

Claims (2)

host for networking (206). The processor (201) (or multiple processors, multi-core processor, etc.) can be selected from a range of devices currently widely used, for example, manufacturers such as: Intel™, AMD™, Apple™, Samsung Exynos™, MediaTEK ™, Qualcomm Snapdragon™, etc. Under the processor or one of the processors used in the system (200), it is also necessary to take into account the graphics processor, for example, NVIDIA GPU with a CUDA-compatible software model, or Graphcore, the type of which is also suitable for full or partial execution of the method, and can also be used to training and application of machine learning models in various information systems. RAM (202) is a random access memory and is designed to store machine-readable instructions executable by the processor (201) to perform the necessary operations for logical data processing. The RAM (202) typically contains the executable instructions of the operating system and associated software components (applications, program modules, etc.). In this case, the RAM (202) may be the available memory of the graphics card or graphics processor. ROM (203) is one or more persistent storage devices such as a hard disk drive (HDD), a solid state drive (SSD), flash memory (EEPROM, NAND, etc.), optical storage media (CD-R /RW, DVD-R/RW, BlueRay Disc, MD), etc. Various types of I/O interfaces (204) are used to organize the operation of system components (200) and organize the operation of external connected devices. The choice of appropriate interfaces depends on the particular design of the computing device, which can be, but not limited to: PCI, AGP, PS/2, IrDa, FireWire, LPT, COM, SATA, IDE, Lightning, USB (2.0, 3.0, 3.1, micro, mini, type C), TRS/Audio jack (2.5, 3.5, 6.35), HDMI, DVI, VGA, Display Port, RJ45, RS232, etc. To ensure user interaction with the computer system (200), various means (205) of I/O information are used, for example, a keyboard, a display (monitor), a touch screen, a touchpad, a joystick, a mouse manipulator, a light pen, a stylus, a touch panel, trackball, speakers, microphone, augmented reality tools, optical sensors, tablet, indicator lights, projector, camera, biometric identification tools (retinal scanner, fingerprint scanner, voice recognition module), etc. The networking means (206) provides data transmission via an internal or external computer network, for example, an Intranet, Internet, LAN, etc. As one or more means (206) can be used, but not limited to: Ethernet card, GSM modem, GPRS modem, LTE modem, 5G modem, satellite communication module, NFC module, Bluetooth and / or BLE module, Wi-Fi module and others Additionally, satellite navigation tools can also be used as part of the system (200), for example, GPS, GLONASS, BeiDou, Galileo. The specific choice of elements of the system (200) for the implementation of various software and hardware architectural solutions may vary while maintaining the required functionality provided. Modifications and improvements to the above described embodiments of the present technical solution will be clear to experts in this field of technology. The foregoing description is provided by way of example only and is not intended to be limiting in any way. Thus, the scope of the present technical solution is limited only by the scope of the appended claims. CLAIM 1. A method for extracting named entities from textual information, performed by at least one computing device, comprising the steps of: receive text information; split the text into words; performing tokenization of the text to obtain a sequence of tokens; form by means of a neural network for the received sequence of tokens a set of vectors; based on the obtained set of vectors, a vector representation of the sequence of tokens is formed; by comparing the indicators of the obtained vector representation of the sequence of tokens with predetermined indicators of the vectors obtained as a result of training the neural network, predict named entities for the vector representation of the sequence of tokens; recognize the named entities obtained at the previous stage by selecting the word label. 2. The method according to claim 1, characterized in that a vector representation of a sequence of tokens based on a set of vectors is formed by calculating a weighted sum or average values -