EA039466B1 - Method and system for classifying data in order to detect confidential information in a text - Google Patents

Abstract

The invention relates in general to the field of electronic data processing, and more particularly to methods for classifying data in order to detect confidential information. Claimed is a computerized method for classifying data in order to detect confidential information, which is carried out using at least one processor and comprises the steps of obtaining data presented in text format; processing the obtained data with the aid of machine learning algorithms, wherein each word in the text is assigned a tag corresponding to a set type of confidential information, and wherein a classification matrix is generated for each machine learning algorithm, said matrix being used as the basis on which to calculate an F-score for each type of data; classifying each word in the text on the basis of the tagged texts obtained from each machine learning algorithm and the F-score matrix corresponding to the machine learning algorithms, and generating a summary version of the text with tags assigned; classifying, according to confidentiality classes, the text with tags assigned to each word by comparing all of the tags in the text with set confidential information tags.

Description

Technical field

The present technical solution, in general, relates to the field of computational data processing, and in particular, to methods for classifying data to identify confidential information.

State of the art

Currently, the identification of confidential information from a large amount of data and its subsequent classification is a priority for many industries. The most widespread use of these technologies is observed in the financial sector, where among large volumes of various data it is necessary to separately identify and classify confidential information. To do this, various tools and technologies are used to somehow reveal confidential information from large volumes of shared data. A key feature in the operation of such tools is data analysis using machine learning algorithms.

Data is stored and processed in various automated systems and file resources that have different levels of confidentiality, access methods, and attributive composition. Checking for the presence of sensitive data is carried out by various tools. In this regard, it became necessary to create a single technical solution that allows using neural networks to automatically process a large amount of data and identify confidential information. To do this, it is necessary to train artificial intelligence to recognize the contents of documents that may contain confidential information. At the moment, solutions are known from the prior art that are aimed at storing and classifying data according to user-defined criteria.

Amazon Macie is a data monitoring service that uses several automatic content classification methods to identify and prioritize sensitive data and pinpoint the business value of the data. The Service recognizes information such as personal information or intellectual property. One classification method is regular expression classification. Classification of objects using regular expressions is based on specific data or data patterns that Amazon Macie looks for when it inspects the contents of data objects. Amazon Macie offers a set of managed regular expressions, each with a specific risk level from 1 to 10. Amazon Macie also classifies objects using support vector machines.

The disadvantages of this solution are the inability to change existing or add new regular expressions, the ability to only enable or disable the search for any existing regular expressions, the service identifies only those objects that match the rules. The disadvantages of using regular expressions are that for each type of confidential information it is necessary to write several regular expressions that do not take into account rare features of the data or may be more general, for example, contain unnecessary data.

The Google Cloud DLP solution is known to provide fast, scalable classification and editing for sensitive data such as credit card numbers, names, social security numbers, selected international identifiers, phone numbers, and GCP credentials. The DLP cloud classifies this data using over 90 predefined detectors to identify patterns, formats, and checksums.

The disadvantage of this solution is the use of only regular expressions, for each type of confidential information it is necessary to write several regular expressions that do not take into account the rare features of the data or may be more general, for example, contain unnecessary data.

The essence of the technical solution

The claimed technical solution offers a new approach in the field of identifying and classifying confidential information by creating machine learning models for processing large amounts of data.

The technical problem or technical challenge to be solved is to create a new data classification method with a high degree of accuracy and a high speed of recognition of confidential information.

The main technical result achieved in solving the above technical problem is to increase the accuracy of the classification of confidential information.

An additional technical result achieved by solving the above technical problem is to increase the speed of classifying confidential information.

The claimed results are achieved through a computer-implemented method for classifying data to identify confidential information, performed using at least one processor and comprising the steps of:

receive data presented in text format;

processing the obtained data using machine learning algorithms, during which each word in the text is assigned a tag corresponding to a given type of confidential information, and for each machine learning algorithm a classification matrix is formed, on the basis of which the F-measure is calculated for each type data;

classifying each word in the text based on the tagged texts obtained from each machine learning algorithm and the matrix of F-measures corresponding to the machine learning algorithms, and forming the final version of the tagged text;

classifying the text with tags for each word according to privacy classes based on a comparison of the totality of available tags in the text with the given tags of confidential information.

In one particular embodiment of the method, for each machine learning algorithm, F-measure scores are calculated for each type of data.

In another particular embodiment of the method, the confidential information is presented at least in the form of textual data and/or numerical data.

Also, these technical results are achieved through the implementation of a data classification system to identify confidential information, which contains at least one processor; at least one memory coupled to the processor that contains machine-readable instructions that, when executed by the at least one processor, perform the above method.

Description of drawings

The features and advantages of the present invention will become apparent from the following detailed description of the invention and the accompanying drawings, in which:

fig. 1 illustrates a block diagram of the claimed method;

fig. 2 illustrates an example of retrieved named entities;

fig. 3 illustrates an example of a CRF architecture with Bi-LSTM;

fig. 4 illustrates an example of labeled data for model training;

fig. 5 illustrates the results of model training;

fig. 6 illustrates the results of model training;

fig. 7 illustrates an example of test results on a test set;

fig. 8 illustrates a general view of the claimed system.

Implementation of the invention

In this technical solution, terms and abbreviations can be used for clarity of understanding of the work, which will be deciphered later in these application materials.

A model in machine learning is a set of artificial intelligence methods, the characteristic feature of which is not the direct solution of a problem, but learning in the process of applying solutions to many similar problems.

AI (Artificial Intelligence) - artificial intelligence.

A token is an element of a sequence of letters or words, or a punctuation mark.

Tag - the value assigned to the token.

The sequence labeling problem is the assignment of a corresponding tag to each element of a sequence (token).

A named entity is a word or phrase denoting an object or phenomenon of a certain category.

Named entity recognition, NER - extraction of named entities - selection of objects, collections of objects from the text and assigning to these objects a category that determines the value of these objects (for example, full name, names of organizations, locations).

RNN is short for Recurrent neural network, recurrent neural networks.

LSTM is short for Long Short Term Memory, long short term memory is a recurrent neural network architecture.

Bi-LSTM is short for Bidirectional Long ShortTerm Memory. Bidirectional Long Short Term Memory is a recurrent neural network architecture.

NLP is short for Natural Language Processing, natural language processing.

CRF is short for Conditional Random Field, conditional random fields.

Word embeddings - vector representation of words - comparison of objects supplied to the input of the model to vectors.

The claimed method (100) for classifying data to identify sensitive information, as shown in FIG. 1 is to perform a series of sequential steps performed by the processor of a computing device.

The initial step (101) is to receive an array of data in text format. Text data contains information that can be bank card numbers, SNILS, OKPO, OGRN, TIN, date, passport number, phone number, last name, first name, patronymic, email, address, position, website address, etc., without being limited .

The next step (102) is processing the received data using machine learning algorithms, during which each word in the text is assigned a tag corresponding to a given type of confidential information, and for each neural network a classification matrix is formed, on the basis of which F is calculated. -measure for each data type.

In order for the models to process the data supplied to them as input, it is necessary to present the text in a form understandable to neural networks. To do this, it is necessary to match all objects supplied to the input with vectors. In the claimed method, a combination of vector representation of words and symbols is used for this. A combination of methods is introduced to improve the performance of the model. The letters of each word in a sentence are fed into the Bi-LSTM network in order to reveal the characteristics of the words at the character level. A vector representation of words (token embedding) is created separately. Then the vector representation of words and characters are concatenated and then fed into the Bi-LSTM+CRF model. The standard component of a neural network for solving natural language processing problems is pre-trained vector representations of words.

Neural networks are trained on pre-labeled data. Each token in the sequence is assigned a tag (a short string that one-to-one matches types of sensitive information) from a predefined set of tags. Tags are selected in such a way that the user can intuitively understand what this tag means, for example, CARD - card number, NAME - name, etc. Tags are written in Latin, so that they have a common form in all encodings. Types of confidential information are included in one of the categories of legally regulated data, such as personal data, bank secrecy, trade secret, etc.

For the NER task, there are several common entity types that are essentially tags. To determine the confidentiality of a document, it is necessary to be able to extract the following entities: full name, date, position, postal code, etc., without being limited. An example of retrieved named entities is shown in FIG. 2. The source text is tokenized and tagged. Each token has a separate markup tag. Tags are separated from tokens by spaces. The sentences are separated by blank lines. The data set is a text file or a set of text files. The data set should be divided into three sections: training, testing and validation. The training ones are used to train the network, namely to adjust the weights with gradient descent. Validation is used to monitor learning progress and stop early.

A method for training neural networks will be disclosed later in the present application materials.

A classification matrix is a standard tool for evaluating statistical models and displays the probabilities of recognizing an actual value as a predictor for each given predictor.

Based on the classification of the test data, F-measures are calculated. F-score or (F1-score) is a joint score of precision and recall. This metric is calculated using the following formula: F-score = 2 * Precision * Recall / (Precision + Recall). The F-measure is calculated in each algorithm for each kind of data.

The next step (103) is to classify each word in the text based on the tagged texts received from each neural network and the F-measures matrix corresponding to the neural networks and form the final tagged text.

The set of tags is predefined. This set is formed to designate all categories of data that need to be checked for confidentiality in accordance with legislative, regulatory, internal or other standards. Preprocessing (vectorization) of data is aimed at speeding up and facilitating further processing, and combinations of several neural networks - to improve the accuracy of tagging. The tag is affixed to which at least one model points to with the highest probability. Models must also perform contextual sequential analysis, since addresses, full names, and some other data types generally consist of several words. To prepare the text for classification, it is tokenized and tagged. Each token has a separate markup tag. Tags are separated from tokens by spaces. The sentences are separated by blank lines. The resulting data set is a tagged text file or a set of text files.

To extract named entities, this solution uses the Yargy parser. The markup parser is a ready-made engine that can extract names, dates, locations, organizations, etc. To improve the performance of the parser, an existing library named Natasha is used. Some ready-made parsing rules are already available in the Natasha library. For the current solution, the rules for extracting entities are described using context-free grammars and dictionaries built on the basis of requirements specified by normative documents. For example, if there are several levels of information criticality, then it is listed which entities should belong to each of the levels.

As a result, a breakdown is made by proposals, each proposal is assigned a corresponding number. All available characters are matched with words in sentences. Tags are cast to a categorical type.

The first layer of the model (Embedding layer) turns sequences of numbers (words associated with numbers) into fixed-size dense vectors. Next, the TimeDistributed wrapper is used to apply the Embedding layer to each sequence of characters and get a vector representation of the words. Next, vector representations of words and characters are concatenated.

The resulting vector representations are fed into the main layer of the model (Bidirectional). This layer calculates tag probabilities for each word in a sentence. These probabilities are then fed into the CRF layer, which calculates the probability distribution of the transition from one tag to another.

At step (104), the text is classified with tags affixed to each word according to privacy classes based on a comparison of the totality of available tags in the text with the given tags of confidential information.

The data prepared in step (103) is used to analyze and assign a classification level to the entire text. To do this, prepared tables of tag combinations are used that determine the total level of confidentiality of the entire text. The level of privacy depends on the combination of tags, not just their presence.

Example: just an address in the entire text does not represent criticality, and an address with a mention of circulating funds should already be classified with increased severity. Also, the name classifies a whole group of people and cannot be considered critical. But a name with a phone number is already personal data that should be classified at the appropriate level of criticality. If the document contains only the full name, this is one level, but if, in addition to the full name, there is a phone number and date of birth, then the level of confidentiality of the document is much higher. For correct classification, it is important to track the context of the tagged word, that is, to evaluate the entities located on the left and right. Tags are prefixed with B- if this is the first occurrence of a tag of this type, and I- if it is a continuation. Example: April [B-Date] 2019 [I-Date] of [I-Date] or 117 [B-Money] million [IMoney].

Recurrent Neural Networks (RNNs) are used to solve a variety of problems including natural language processing problems due to their ability to use previous information from a sequence to calculate the current output.

In order to correctly process the current word in the text (assign a tag), the network needs to be based on an understanding of the previous context. So she must remember what was the text to the left of the current word. Traditional neural networks do not have this property, they cannot be trained for long-term dependencies. Recurrent neural networks help solve this problem. They contain feedback, thanks to which they can transfer information from one step of the network to another.

However, despite the ability to learn long-term dependencies, in practice, RNN models do not perform well and suffer from the vanishing gradient problem. This problem occurs because back-propagated error signals quickly become very small (or, conversely, excessively large). In practice, they decrease exponentially with the number of layers in the network. For this reason, a special RNN architecture called Long Short Term Memory (LSTM) was developed to deal with the vanishing gradient. One repeating module of the LSTM network consists of four layers. The LSTM replaces the hidden blocks in the RNN architecture with blocks called memory blocks, which contain four components: three kinds of filters (input filter, forget filter, output filter, and memory cell). Proper recognition of a named object in a sentence depends on the context. and subsequent words are important for tag prediction.Bidirectional recurrent neural networks have been designed to encode each element into a sequence given left and right contexts, making them one of the best choices for the NER task.The bidirectional calculation model consists of two steps: the forward layer computes representation of the left context, the back layer computes the representation of the right context The outputs of these steps are combined to obtain a full representation of the input sequence element Conditional Random Field (CRF) is a non-directional probabilistic graphical model for structured conditional prediction even probabilities of events corresponding to the vertices of some graph, given the observed data. The Bi-LSTM CRF architecture used to implement method (100) is shown in FIG. 3.

In the combined model, vector representations of words (the method for obtaining a vector representation is described in the practical part) are fed into the Bi-LSTM bidirectional neural network. This network calculates tag probabilities for each word in a sentence. Let for the input sequence of words (sentence) Х=(х ₀ , x1,.., x _n ) Р - probability matrix, which is produced by the Bi-LSTM network. This matrix is n*k, where k is the number of different tags, an is the length of the input sequence. Pi,j is the probability that the i-th word in the sentence has tag j. For a sequence of answers y = {y ₀ , y1, ..,y _N } score is calculated using the following formula:

Score(X,y) =Σ^ ₀ Α _{γυγί+1 + ZAiA} , _yi i+1 will be exactly the tag y _i+1 , provided that the previous tag is yi.

- 4 039466

To improve prediction accuracy, the CRF layer is trained to enforce constraints based on tag order. For example, in an I O B schema (I - Inside, O - Other, B - Beginning), the I tag never appears at the beginning of a sentence, or O IB O is an invalid sequence of tags. The complete set of parameters for this model consists of the Bi-LSTM layer parameters (weight matrices, biases, word vector representation matrix) and the CRF layer transition matrix. All these parameters are tuned during training of the stochastic gradient descent backpropagation algorithm.

Next, the principle of training neural networks and assessing the quality of models for the purposes of implementing the claimed method will be presented.

In FIG. 4 shows an example of labeled data for training models.

To train models, it is necessary to prepare a labeled dataset with text and tags (Fig. 4). A breakdown is made by proposals, each proposal is assigned a corresponding number.

Example: in the prepared text there are 47959 sentences containing 35179 different words. It turns out 847657 lines in the dataset.

At the first stage of training, a breakdown is made by proposals, each proposal is assigned a corresponding number. At the next stage, the tokens (each value of the Word column in Fig. 4) are brought to a vector form. To do this, dictionaries are introduced for words, for symbols and for tags. Words in sentences are matched against a sequence of numbers, and then the pad_sequensec() function is applied to the number sequences to bring the sequences to the same size. Next, all available characters are compared with words in sentences, and tags are converted to a categorical type.

At the next stage, the sample is divided into training, validation and test. The proportions used are 80% to 10% to 10%.

Example: training - 38846 sentences, validation - 4317 sentences, test - 4796 sentences.

The next step in learning is building the model. The first layer is the Embedding layer, whose task is to translate the sequences of numbers (to which the words of the markup text have been assigned) into dense vectors of a fixed size. Thus, we obtain a vector representation of words.

Next, the TimeDistributed wrapper is used to apply the Embedding layer to each character sequence. After the above processing, a vector representation of the characters is obtained. Next, the vector representations of words and symbols are concatenated.

At the next stage, the main layer of the model is activated - Bidirectional. The vector representations obtained at the previous stage are fed into the Bidirectional layer. This layer calculates tag probabilities for each word in a sentence. These probabilities are then fed into the CRF layer, which calculates the probability distribution of the transition from one tag to another. All model parameters (weight matrices, biases, word vector representation matrix, and layer CRF transition matrix) are tuned during training of the stochastic gradient descent backpropagation algorithm.

Next, the model is trained/trained.

In FIG. Figures 5 and 6 show the results of training the Bi-LSTM+CRF model. It shows the increase in training accuracy (accuracy) and the increase in accuracy on validation data (validation accuracy) with an increase in the number of epochs, and also shows how the loss decreased.

In FIG. 7 shows the results of the check on the test sample. The text that was marked up by a person is compared with the markup results obtained at the output of the model. The quality of the model is determined by how closely the model predicted the value of the tag, or, in other words, how much less deviation the result of the model has from the pre-labeled tag values. To assess the quality of work of a particular model, generally accepted methods and metrics for assessing the quality of models are used.

In FIG. 8 shows an example of a general view of a computing system (300), which provides the implementation of the claimed method (100) or is a part of a computer system, for example, a server, a personal computer, a part of a computing cluster that processes the necessary data to implement the claimed technical solution.

In the general case, the system (300) contains one or more processors (301) connected by a common information exchange bus, memory facilities such as RAM (302) and ROM (303), input/output interfaces (304), input/output devices (1105 ) and a device for networking (306).

The processor (301) (or multiple processors, multi-core processor, etc.) can be selected from a range of devices currently widely used, for example from manufacturers such as: Intel™, AMD™, Apple™, Samsung Exynos™, MediaTEK™ , Qualcomm Snapdragon™, etc. Under the processor or one of the processors used in the system (300), it is also necessary to take into account the graphics processor, such as NVIDIA GPU or Graphcore, the type of which is also suitable for full or partial execution of the method (100), and can also be used for learning and applying mods.

- 5 039466 machine learning tasks in various information systems.

RAM (302) is a random access memory and is designed to store machine-readable instructions executable by the processor (301) to perform the necessary data logical processing operations. The RAM (302) typically contains the executable instructions of the operating system and associated software components (applications, program modules, etc.). In this case, the RAM (302) may be the available memory of the graphics card or graphics processor.

The ROM (303) is one or more persistent storage devices such as a hard disk drive (HDD), a solid state data drive (SSD), flash memory (EEPROM, NAKD, etc.), optical storage media (CD-R/ RW, DVD-R/RW, BlueRay Disc, MD), etc.

Various types of I/O interfaces (304) are used to organize the operation of system components (300) and organize the operation of external connected devices. The choice of appropriate interfaces depends on the particular design of the computing device, which can be, but not limited to: PCI, AGP, PS/2, IrDa, FireWire, LPT, COM, SATA, IDE, Lightning, USB (2.0, 3.0, 3.1, micro, mini, type C), TRS/Audio jack (2.5, 3.5, 6.35), HDMI, DVI, VGA, Display Port, RJ45, RS232, etc. To ensure user interaction with the computing system (300), various means (305) of I/O information are used, for example, a keyboard, a display (monitor), a touch screen, a touchpad, a joystick, a mouse, a light pen, a stylus, a touchpad, a trackball, speakers , microphone, augmented reality tools, optical sensors, tablet, light indicators, projector, camera, biometric identification tools (retinal scanner, fingerprint scanner, voice recognition module), etc.

The networking means (306) provides data transmission via an internal or external computer network, such as an Intranet, Internet, LAN, and the like. As one or more means (306) can be used, but not limited to: Ethernet card, GSM modem, GPRS modem, LTE modem, 5G modem, satellite communication module, NFC module, Bluetooth and / or BLE module, Wi-Fi module and etc. The submitted application materials disclose preferred examples of the implementation of the technical solution and should not be interpreted as limiting other, particular, examples of its implementation that do not go beyond the scope of the requested legal protection, which are obvious to specialists in the relevant field of technology.

Claims (4)

1. A computer-implemented method for classifying data to identify confidential information, performed using at least one processor and comprising the steps of: receive data presented in text format; processing the received data using machine learning algorithms, during which each word in the text is assigned a tag corresponding to a given type of confidential information, and for each machine learning algorithm a classification matrix is formed, on the basis of which the F-measure is calculated for each type of data; classifying each word in the text based on the tagged texts obtained from each machine learning algorithm and the matrix of F-measures corresponding to the machine learning algorithms, and forming the final version of the tagged text; classifying the text with tags for each word according to privacy classes based on a comparison of the totality of available tags in the text with the given tags of confidential information. 2. The method according to claim 1, characterized in that for each machine learning algorithm, F-measure scores are calculated for each type of data. 3. The method according to claim 1, characterized in that the confidential information is presented at least in the form of textual data and/or numerical data. 4. Data classification system for detecting confidential information, containing at least one processor; at least one memory connected to the processor, which contains machine-readable instructions, which, when executed by at least one processor, perform the method according to any one of paragraphs. 1-4. - 6 039466 (101) Data presented in text format is received. (102) The received data is processed using machine learning algorithms. (SW) Perform classification of each word in the text based on step (102). (104) The text is classified with tags for each word according to privacy classes.