RU2804747C1 - Method and system for depersonalization of confidential data - Google Patents

Abstract

FIELD: data protection.

SUBSTANCE: specifically the depersonalization of confidential data while maintaining the data structure in text documents. The method for depersonalizing confidential data in text documents while preserving the data structure comprises the following steps: receiving a document with text data; segmenting and tokenizing text data; performing token vectorization; determining whether each token belongs to the category of confidential data using a machine learning model; depersonalizing data related to tokens with confidential data while maintaining the data structure; forming a list of replacements; replacing the original confidential data in the text document with depersonalized data according to the list of replacements, and in the process of replacement, the depersonalized data is formatted in accordance with the positions of formatting changes of parts of the text.

EFFECT: providing the possibility of preserving the stylistic, semantic, lexical and morphological structure of data in text documents when depersonalizing them, increasing the accuracy of depersonalizing data in text documents by identifying confidential data in text documents using a machine learning model.

10 cl, 6 dwg, 6 tbl

Description

TECHNICAL FIELD

[1] This technical solution, in general, relates to the field of data protection, and more specifically to the anonymization of confidential data while maintaining the data structure in text documents.

BACKGROUND OF THE ART

[2] Currently, in any organization there are restrictions on the storage and processing of confidential data. Thus, organizations are obliged to store and process confidential data in certain databases and ensure the complete safety of such data, for example, in accordance with legislative acts such as the Federal Law “On Personal Data”, the Federal Law “On Banks and Banking Activities”, etc. d. Confidential data contains personal data of clients and employees, information constituting banking and commercial secrets, information about health insurance, medical records, etc., which explains such high requirements for their processing and distribution, and, accordingly, does not allow the use of such data in external structures. At the same time, organizations continue to need and need to use such data, for example, when developing and testing software, for transfer to third parties, such as document translation agencies, consulting, auditing companies, for use in the development of artificial intelligence models, etc. d.

[3] One of the ways to use confidential data in external sources, known from the prior art, is to depersonalize confidential data. Thus, confidential data is subject to irreversible modification, excluding the possibility of attributing this data to a specific subject directly or indirectly.

[4] However, this approach is not applicable in areas where it is necessary to maintain the integrity of the data structure, such as high relevance, syntactic and morphological features of data, for example, when creating AI models, translating documents by contractors, prototyping an analytical application or data mart, due to the impossibility correct translation and/or accurate model training.

[5] Thus, from the prior art, the solution disclosed in US patent application No. US 2012131075 Al (MAWDSLEY GARY [GB], et al.), publ. 05/24/2012. This solution, in particular, discloses a method for reversibly anonymizing personal data stored in a database, in which, for a subset of data elements, the deviation of each of the specified data elements in the subset relative to the reference data elements is determined and deviation identifiers are assigned to each of the specified deviations in the specified data for anonymizing data elements in the specified subset of data; creating a conversion table mapping said data elements in said subset to said variance identifiers; storing said translation table; and storing said deviation identifiers defining said anonymized data elements.

[6] The disadvantages of this solution are the low accuracy and efficiency of data anonymization in areas where maintaining data integrity is required, due to the impossibility of applying the method on unstructured data and recognizing confidential information based on rules, which can lead to missing data that is not taken into account in the rules, as well as the inability to preserve data formatting and its semantic and morphological features.

[7] A common shortcoming of existing solutions is the lack of an accurate and efficient way to anonymize data in text documents, providing the ability to preserve the format of the style and structure of the data, and also allowing the discovery of data that has not been encountered before. In addition, this kind of solution must provide a semantic structure of the data and check the check bits of certain types of data. Also, such a solution should provide the ability to perform reversible anonymization.

DISCLOSURE INVENTION

[8] This technical solution is aimed at eliminating the disadvantages inherent in existing solutions known from the prior art.

[9] The technical problem solved in this technical solution is the creation of a new and effective way to anonymize confidential data in text documents.

[10] The main technical result that appears when solving the above problem is to ensure the ability to preserve the stylistic, semantic, lexical and morphological structure of data in text documents when they are anonymized.

[11] An additional technical result is to increase the accuracy of anonymizing data in text documents by identifying sensitive data in text documents using a machine learning model.

[12] The specified technical results are achieved by implementing a method for depersonalizing confidential data in text documents while preserving the data structure, containing the stages of:

a) receive a document containing text data;

b) segmenting the text data obtained at step a), and during segmentation the data is divided along the boundaries of changes in text formatting into parts of the text, and assigning an ordinal position in the text to the specified parts of the text;

c) tokenize the text data obtained in step a) and assign ordinal positions in the text to each token;

d) vectorize the tokens obtained in step c);

e) process the vector representations of the tokens obtained in step d) using a machine learning model based on a neural network trained on sensitive data sets, during which it is determined that each token belongs to the category of confidential data;

f) depersonalize data related to tokens with confidential data, and during depersonalization, the data is replaced with data from the same category while maintaining the data structure;

g) generate a list of replacements, including indications of the ordinal positions of tokens with confidential data in the text, ordinal positions of changes in the formatting boundaries of parts of the text, anonymized data corresponding to confidential data;

h) replace the original confidential data in the text document with anonymized data according to the list of replacements, and during the replacement process, the anonymized data is formatted in accordance with the positions of changing the formatting of parts of the text.

[13] In one of the particular examples of the method, text data is obtained in the form of an unstructured text document.

[14] In another particular example of the method, the categories of confidential data are at least one of:

• numerical entities;

• named entities.

[15] In another particular example of the method, the numeric entities represent at least one of: the main cardholder number, TIN number, telephone number, account number, insurance certificate number, IP address, MAC address.

[16] In another particular example of the method, named entities represent at least one of: persons, locations, organizations.

[17] In another particular example of the method, the data structure is consistent with the format, type and semantic content of confidential and anonymized data.

[18] In another particular example of the method, anonymized data for numerical entities is generated while preserving identifying characteristics.

[19] In another particular example of the method, anonymized data for named entities is reduced to a morphological form corresponding to confidential data.

[20] In another particular example of the method, duplicate confidential data is anonymized with the same data.

[21] In addition, the stated technical results are achieved through a system for depersonalizing confidential data in text documents while preserving the data structure containing:

• at least one processor;

• at least one memory coupled to the processor that contains machine-readable instructions that, when executed by at least one processor, implement a method for de-identifying sensitive data in text documents while preserving the data structure.

BRIEF DESCRIPTION OF THE DRAWINGS

[22] The features and advantages of the present technical solution will become apparent from the following detailed description and the accompanying drawings, in which:

[23] FIG. 1 illustrates a block diagram of the claimed method.

[24] FIG. 2 illustrates a flowchart of a method for reversibly anonymizing data.

[25] FIG. 3 illustrates a flowchart of the method for de-identifying data

[26] FIG. 4 illustrates an example of a system for automatic anonymization of confidential data.

[27] FIG. 5 illustrates an example of a system for automatic reversible anonymization of confidential data.

[28] FIG. 6 illustrates an example of a computing device for implementing the claimed systems.

IMPLEMENTATION OF THE INVENTION

[29] The claimed technical solution offers a new approach that ensures the depersonalization of confidential data in text documents. The main feature of the proposed solution is to provide the ability to depersonalize confidential data in text documents while preserving the structure of the anonymized data, by preserving the stylistic structure of confidential data and depersonalizing data while preserving the semantic, lexical and morphological form of the data. In addition, the implementation of this technical solution increases the accuracy of data anonymization, due to the implementation of a machine learning model based on a neural network, which ensures the search for confidential data that has not been encountered before, including in unstructured data. Also, another additional advantage achieved when using the claimed solution is the ability to perform both reversible anonymization of data and irreversible anonymization of data.

[30] The claimed technical solution can be implemented, for example, by a system, machine-readable medium, server, etc. In this technical solution, a system means, including a computer system, a computer (electronic computer), CNC (computer numerical control), PLC (programmable logic controller), computerized control systems and any other devices capable of performing a given, clearly defined sequence of operations (actions, instructions).

[31] A command processing device means an electronic unit or an integrated circuit (microprocessor) that executes machine instructions (programs).

[32] An instruction processing device reads and executes machine instructions (programs) from one or more data storage devices, such as devices such as random access memory (RAM) and/or read only memory (ROM). ROM can be, but is not limited to, hard drives (HDD), flash memory, solid-state drives (SSD), optical storage media (CD, DVD, BD, MD, etc.), etc.

[33] Program - a sequence of instructions intended for execution by a computer control device or a command processing device.

[34] The term "instructions" as used in this application may refer generally to software instructions or software commands that are written in a given programming language to perform a specific function, such as, for example, retrieving and processing data, generating a user profile , reception and transmission of signals, analysis of received data, user identification, etc. Instructions can be implemented in a variety of ways, including, for example, object-oriented methods. For example, instructions can be implemented using the programming language C++, Java, Python, various libraries (for example, "MFC"; Microsoft Foundation Classes), etc. Instructions that carry out the processes described in this solution can be transmitted over both wired and wireless data channels, for example, Wi-Fi, Bluetooth, USB, WLAN, LAN, etc.

[35] In FIG. 1 shows a block diagram of a method 100 for depersonalizing confidential data in text documents, which is disclosed step by step in more detail below. Said method 100 involves performing steps to process various digital data. The processing is typically performed by a system, which may be, for example, a server, a computer, a mobile device, a computing device, etc. The elements of the system are disclosed in more detail in Fig. 6. In one particular embodiment, may also be performed by system 400, which is discussed in more detail below.

[36] The term confidential data in this solution means data to which access is limited in accordance with the security policies of organizations and/or legislation. Thus, confidential data can include personal data, information constituting banking and commercial secrets, full names of employees and clients, partners and suppliers, addresses, telephone numbers, email addresses, social security numbers, bank card information, tax identification number, registration number car, BIC number, IP address, geolocation data, marriage document number, education document number, date, URL, MAC address, work book number, military ID number, OKPO code, etc., not limited to .

[37] Depersonalization of data - actions as a result of which it becomes impossible, without the use of additional information, to determine the ownership of confidential data to a specific subject of confidential data.

[38] At step 110, a document containing text data is received.

[39] At step 110, a document containing text data is received into a system, such as system 400. In one particular embodiment, the document may be downloaded to system 400 via a communications network such as the Internet, LAN, etc. In yet another particular embodiment, a document may be imported directly from a flash drive and/or internal memory of the system 400. A document containing text data in this solution refers to any data file containing text data. Thus, the document can be any unstructured document, for example, a Word file, PDF, text document, photograph, digitized document, email file, etc.

[40] Method 100 then proceeds to step 120.

[41] At step 120, the text data obtained at step 110 is segmented, and during the segmentation, the data is divided along text formatting boundaries into parts of text, and assigned an ordinal position in the text to the specified parts of text.

[42] As noted above, formatting style is essential in certain data processing tasks, such as developing artificial intelligence (AI) models, text translation, etc. At the same time, known data anonymization systems do not imply or contain such a possibility due to the fact that text formatting styles are not tied directly to the text, but are contained in the accompanying document data, for example, in metadata. In addition, some machine learning models also take into account the formatting of the data, which consequently increases the accuracy of the training process itself.

[43] Accordingly, to implement this possibility in the claimed method, the following approach is proposed. This approach, in one particular embodiment, solves both the problem of providing third-party developers with anonymized data and ensures the formation of an accurate and as close as possible, including formatting style, sample of data for model training.

[44] Upon receipt of a text document by system 400, said system 400 extracts text from the document in portions so as to preserve the document's original formatting styles. In this solution, extracting text from a document should be understood as extracting certain parts of the text from a text document and storing these parts in the memory of the system 400, for example, in the form of a data file, i.e. at this step 120, the source text is converted into several parts, divided along the boundaries of the formatting change. In this solution, formatted text is understood as the result of the process of formatting a page, paragraph, line, or symbol of a document. Thus, the style of text formatting can be changing the color, underlining, italics, outline, changing indents, registers, etc., of certain words, characters, paragraphs in a text document. The formatting style of the text can be defined using XML (Extensible Markup Language) markup, which can be stored, for example, in a text document file, be a machine-readable attachment to the text document, etc. It is worth noting that the peculiarity of this stage is that the text is not divided into whole words and/or tokens surrounded by XML markup, but is broken precisely along the boundaries of changes in markup formatting. This approach ensures that style violations and further incorrect application of styles to impersonal parts are avoided.

[45] Consider an example of text extraction in accordance with the specified stage 120. Thus, the following text: “Ivan Ivanov received card 4485 2911 2679 2812 at Betta LLC.” Now Ivan has access to online payments, for which Andrey congratulated him.” will be converted into several parts, divided according to the boundaries of the formatting changes (Table 1).

[46] As can be seen from Table 1, the symbol B is part of the word Betta, but has a different formatting style, and, accordingly, is placed in a separate part of the text. It will be apparent to one skilled in the art that while the example is based on case and character underlining, this approach is applicable to any text formatting change.

[47] After receiving parts of the text, the specified parts are assigned start and end indices within the overall text. These indexes are subsequently required to convert the anonymized data into the original formatting style.

[48] As a tool for determining the beginning and end of a part of text (text length), both internal document tools that provide counting of characters in the text, for example, text editors, and tools, for example, software for analyzing text documents, including including 400 built into the system.

[49] So, continuing the previous example, the following indices will be assigned to the indicated parts of the text (Table 2):

[50] Thus, at step 120, the text data is segmented and the specified portions of the text are assigned an ordinal position in the text.

[51] Method 100 then proceeds to step 130.

[52] At step 130, the text data obtained at step 110 is tokenized and ordinal positions in the text are assigned to each token.

[53] At step 130, processing of the received text data is performed. The input text is tokenized, i.e. segmented into parts, such as sentences, words or symbols. In this solution, a token should be understood as a sequence of characters in a document that is important for analysis. Thus, tokens can be, for example, individual words, parts of words, etc. Tokenization of sentences can be carried out using, for example, lexical analyzers such as razdel, rusenttokenize, NLTK, etc. Lexical analysis is the process of analytically parsing an input sequence of characters into recognized groups (tokens) in order to obtain identified sequences called "tokens" as output. In addition, tokenization of the input text can be done based on regular expressions. It is obvious to a person skilled in the art that any lexical analyzer known in the art can be used and this solution is not limited to the above examples. Following the tokenization process, similar to the assignment of start and end indices in step 120, text tokens are assigned a start and end index. As mentioned above, determining the position of a word in a text document can be done, for example, using text analyzers.

[54] At the same time, it is worth noting that in the process of text tokenization, text formatting styles will be erased, because the specified process cannot take into account the style of text in the document.

[55] Continuing with the example above, the presented text would be broken down into the following tokens, which would be assigned the following indices (Table 3):

[56] At step 140, vectorization of the tokens obtained at step 130 is performed.

[57] At step 140, each token resulting from the tokenization process is vectorized, for example, using embeddings or one hot encoding. So, for example, during tokenization, each token is represented in the dictionary by its index, which displays the position in the specified dictionary. Thus, each token represents an index in the dictionary, and accordingly the vectorization process is carried out by replacing each token with its index in the dictionary. Then the indices are grouped taking into account the sparseness of the dictionary and the semantic proximity of the tokens. It will be obvious to one skilled in the art that other vectorization algorithms can be used to vectorize tokens, such as, but not limited to, TransformersBertEmbedder, Word2vec, fastText, etc. algorithms. This vectorization process is a preparatory step for data processing by a machine learning model that performs recognition of sensitive data (step 150).

[58] At step 150, the vector representations of tokens are processed using a machine learning model based on a neural network trained on sensitive data sets, during which it is determined that each token belongs to a category of sensitive data.

[59] A machine learning model was developed and applied to process unstructured data and identify sensitive data in unstructured text. The specified model is designed to solve the text classification problem (NER problem). Named-entity recognition (NER) is an information retrieval subtask that aims to find and classify mentions of named entities in unstructured text into predefined categories such as person names, organizations, locations, monetary values, percentages, etc. d.

[60] The ML model was trained on pre-labeled data. At the time of creation of the model, a dataset (data set) of labeled confidential data was used, consisting of more than 2 million tokens. The data set was generated from original text documents containing personal data, information constituting banking and commercial secrets, etc. Training the model to recognize confidential data in text documents consisted of identifying classes of named entities in the text (full name, bank card number, social security number, etc.). The schema for data markup can be, for example, BIO/IOB schema, BILUO schema, etc., without limitation. A dataset can be represented, for example, as an ordered list of tokens separated by a separator from the named entity class, such as a space character. It is obvious to a person skilled in the art that any scheme for marking and presenting a dataset known from the prior art can be used and this solution is not limited to the examples given above. To train the ML model, 80% of the tokens from the dataset were used, and 20% were used to calculate quality metrics.

[61] The main metric of ML model quality, the class-weighted F1 measure, is about 95.3%, which exceeds the results of automatic machine learning (AutoML) by more than 15%. At the same time, the spread between completeness (recall) and accuracy (precision) does not exceed 0.8%, which indicates a high level of quality of the model both from a security point of view - low probability of missing confidential information, and from the point of view of the end user - a small number of false detection of confidential information. The degree of decline in quality metrics when augmenting texts at the character level, indicating the stability of the model, does not exceed 5%.

[62] The specified machine learning model based on a neural network was successfully implemented and tested in the organization in the work processes of departments.

[63] In one particular embodiment, the neural network may be, for example, a Transformer neural network, a recurrent neural network (RNN), etc., but is not limited to. A feature of this machine learning model is the ability to effectively recognize confidential data in text documents, including through analysis of the semantic proximity of tokens.

[64] Thus, at step 150, the provided trained machine learning model processes the data to recognize sensitive data. Continuing the example, the result of data processing will look like this (Table 4):

[65] Thus, from the specified text data, confidential data is extracted with the assigned category to which the specified data belongs. For example, as can be seen from Table 4, all data related to confidential data is marked with tags - short strings that one-to-one correspond to the types of confidential data. Tags correspond to the category assigned to the specified data, for example, CARD - card number, NAME - name, etc. Tags are written in Latin so that they have a common appearance in all encodings.

[66] In one particular embodiment, the sensitive data categories that are assigned to the text data are at least a numeric entity data category and a named entity data category. This step can be performed by a machine learning model and is a coarse classification where data is divided into numeric and named entities to subsequently identify the type of sensitive data, for example, using Multiclass classification, Named Entity Recognition (NER), etc. This division is necessary to select the correct depersonalization algorithm corresponding to the type of confidential data, and to establish more stringent requirements for preserving the morphological features of named entities. Thus, after classifying the data as numeric, the type of numeric data can be determined. Numerical entities can represent the following data: Primary cardholder number (PAN), FL INN number, LE INN number, House number, DUL number, FL account number, LE account number, Telephone, OGRNIP number, OGRN number, SNILS number, Number Compulsory Medical Insurance Policy, Vehicle Passport, BIC Number, IP Address, Social Security Number SNN, Employer Identification Number EIN, Individual Taxpayer Number ITIN, Geolocation Data, Marriage Document Number, Education Document Number, Date, Work Book Number, Military Number ticket, OKPO code, IMEI, etc. To increase the accuracy of recognition of confidential data, this solution also applies a procedure for checking check digits for numeric entities. The check bit checking algorithm checks the data against the check bits, which are typically calculated using the Luhn algorithm or other algorithms. The Luhn algorithm is an algorithm for calculating the check digit of certain types of data. It is not a cryptographic tool, but is intended primarily to identify errors caused by unintentional data corruption. The check digit is used in various numbers, such as: bank card numbers, SNILS, OKPO, OGRN, INN, etc. not limited to. The check bit is necessary in order to eliminate the possibility of incorrect recognition of the type of confidential data.

[67] Accordingly, named entities can also be divided by entity type. Thus, named entities can represent: persons, locations, organizations, for example, Last Name, First Name, Patronymic, Position, Full and short name of a legal entity, login, street, city, etc.

[68] These extracted entities may be stored, for example, in a secure memory of the system 400 for further anonymization. In one particular embodiment, tokens with the same sensitive data tags are combined into a single token (for example, tokens characterizing a bank card number). Thus, tokens belonging to the CARD tag can be combined into a single token. Additionally, in yet another particular embodiment, identical tokens with the same class can also be identified and marked for later replacement with the same anonymized values, for example, the same name occurring will be anonymized with the same value.

[69] Thus, at the specified step 150, all entities containing sensitive information are extracted from the text document.

[70] Next, at step 160, data related to sensitive data tokens is anonymized, and during anonymization, the data is replaced with data from the same category while maintaining the data structure.

[71] Thus, after determining the type of sensitive data, method 100 proceeds to step 160. At step 160, the determined sensitive data is anonymized while preserving the structure of the data. The main feature of the claimed technical solution is the ability to anonymize data while preserving morphological features, text structure and replacing identical entities with identical anonymized values. This feature allows you to use depersonalization for those tasks where the morphological features and meaning of the text are important, for example, training artificial intelligence models, text translation, etc. Text structure preservation refers to the pseudonymization of sensitive data, in which sensitive or personal data is replaced with data of the same type, gender and language/region. For example, a female name is replaced by another female name common in the local culture, a street is replaced by a street with a similar morphological and phonetic structure, for example, Catherine is replaced by Elizabeth, etc. For this process, based on the type of sensitive data in the dictionary, for example, the database of the system 400 containing the data set to be anonymized, similar attributes to the text data fragment to be replaced are determined. For example, when comparing the attributes of Ivan Ivanov, the Elena Smith attribute will have a low degree of similarity, because These data have different classes (gender, male and female) and nationalities (Smith is ethnically German). At the same time, the attribute Petrov Peter will have a high degree of similarity, because gender and nationality are the same. As discussed above, determining the type of data in a category can be performed at step 150 using a machine learning model. Additionally, in one particular embodiment, a portion of the text data may be replaced by data with an equal number of letters, for example, the word "two" may be anonymized as "three." This feature can be achieved through a pre-labeled database containing indications of the above structural features of the text (for example, the database can store sections related to personal data by nationality, gender, etc.), i.e. the database may be a table with the column name and data type identified.

[72] Additionally, in yet another particular embodiment, sensitive data represented by dates may be shifted by a fixed offset, such as 1 day, 10 days, etc. This feature allows you to obtain representative data, for example, for a training data set, where the date of the event may be critical, for example, in tasks of searching for anomalous events.

[73] It is worth noting that for these anonymization tasks, the system 400 may additionally contain data type classifiers, for example, context classifiers, morphological analysis classifiers, a combination of these classifiers, etc. These classifiers are designed to determine the specific type of data (subclass of data) to which the data belongs (nationality, geographic origin, etc.). An example of a classifier is described in more detail in an article found on the Internet, see https://arxiv.org/abs/1708.07903. For example, morphological analysis classifiers such as pymorphy2 and others may include a type classifier based on prefixes and suffixes or a type classifier based on subwords and compound words. Classification by nationality and ethnicity can be achieved, for example, using, but not limited to, NamePrism and other classifiers that use homophily patterns.

[74] Thus, continuing at step 160, anonymized values preserving indirect identifying characteristics are generated for tokens related to the type and category of sensitive data. In one particular embodiment, for numeric entities, values may be generated based on specified patterns. So, for example, for telephone numbers, the template will contain the country code and the number of digits of the number. For example, for Russia the country code is +7, respectively, phone numbers contained in confidential data and starting with +7 will be replaced while maintaining the specified code. It is obvious to a person skilled in the art that these features can be preserved for other numeric entities (TIN, social insurance, etc.).

[75] In yet another particular embodiment, as discussed above, named entities are anonymized according to the desired morphological form for tokens relating to sensitive data using an anonymized data store (eg, a database). Thus, the specified database or dictionary is initially filled with unique data in normal form and grouped by type of confidential information: First Name, Last Name, Position, etc.

[76] Thus, the data from the text document given in the example above will be replaced with the following data (Table 5):

[77] As can be seen from the table, the CARD data type was anonymized while preserving the structure, namely, the first 6 digits, indicating the bank identification number, and the last 4 digits, used to identify the card and verify its authenticity using a check digit. In addition, first names, last names, and names of legal entities were also anonymized while maintaining the data structure.

[78] It is worth noting that in one particular implementation, with a reversible data anonymization process, the replacement table is stored in a secure data store, memory 400.

[79] Thus, at the specified stage 160, data related to tokens with confidential data are anonymized, and during anonymization, the data is replaced with data from the same category while maintaining the data structure (gender, nationality, etc.).

[80] The method then proceeds to step 170.

[81] At step 170, a list of replacements is generated, including indications of the ordinal positions of tokens with confidential data in the text, ordinal positions of changing the formatting boundaries of parts of the text, anonymized data corresponding to confidential data.

[82] At step 170, based on the generated replacement table, format table and ordinal positions of tokens in the text (start and end indexes), a list of replacements with anonymized values for parts of the text is compiled. The list of replacements contains an indication of the serial number of a part of the text, the boundary replacement indices within the part of the text, the type of confidential data found, and an anonymized value. If the confidential token is represented by several parts of text, then the anonymous value is divided into the appropriate number of parts.

[83] The list may also be generated in the system 400.

[84] The list of replacements for the example above will look like this (Table 6):

[85] At step 180, the original confidential data in the text document is replaced with anonymized data according to the list of replacements, and during the replacement process, the anonymized data is formatted in accordance with the positions of changes in the formatting of parts of the text.

[86] Thus, in the source document, parts of the text are replaced with anonymized values according to the list of replacements that was generated at step 170. Thus, the original formatting of the document is preserved.

[87] For example, the text from the example after depersonalization will look like this: “Petrov Petrov received a card 4485 2952 7892 2810 at Gamma LLC.” Now Peter has access to online payments, for which Potap congratulated him.”

[88] In one particular embodiment, if the reversible anonymization mode is selected, then, as discussed above, the system 400 stores a lookup table, for example, in a protected memory area. In this case, the system 400 is configured to recover the original entities from the anonymized values.

[89] After replacing parts of the text containing confidential data with anonymized data, the system 400 is configured to generate a new text document and provide the specified document to the user of the system and/or upload the document to the Internet.

[90] Let us consider in more detail an embodiment of the claimed invention related to reversible depersonalization.

[91] In more detail, a method 200 for reversibly anonymizing sensitive data in text documents while preserving structure is shown in FIG. 2.

[92] At step 210, the system such as system 500 receives a text document. This step is similar to step 110, and accordingly the above means can be used to obtain the document.

[93] At step 220, the text data obtained at step 210 is segmented, and during the segmentation, the data is divided along text formatting boundaries into parts of text, and assigns an ordinal position in the text to the specified parts of text.

[94] At step 230, the text data obtained at step 210 is tokenized and ordinal positions in the text are assigned to each token.

[95] At step 240, vectorization of the tokens obtained at step 230 is performed.

[96] At step 250, the vector representations of the tokens obtained at step 240 are processed using a machine learning model based on a neural network trained on sensitive data sets, which determines whether each token belongs to a category of sensitive data.

[97] At step 260, data related to sensitive data tokens is anonymized, and during anonymization, the data is replaced with data from the same category while maintaining the data structure.

[98] At step 270, a lookup table is generated that characterizes the correspondence between the confidential data and the anonymized data and stores the specified table in memory. In particular, the specified lookup table is stored in a secure memory of the system 500, for example, in a data store 506. This feature allows the anonymized data to be subsequently de-identified, i.e. restore the original data in the document.

[99] At step 280, a list of replacements is generated, including indications of the ordinal positions of confidential data in the text, ordinal positions of changing the boundaries of the formatting of parts of the text, anonymized data corresponding to the confidential data. The list may also be stored in storage 406.

[100] At step 290, the original confidential data is replaced with anonymized data according to a list of replacements, and during the replacement process, the anonymized data is formatted in accordance with the positions of changes in the formatting of parts of the text. After replacing sensitive data with non-personal data, the system 500 can generate a new text document.

[101] As stated above, the main feature of reversible anonymization is the ability to restore the original confidential data in a text document. Such a need may arise, for example, when translating text outside the organization. So, if you need to translate a contract containing confidential data, the presence of such data will affect the context of the translation, and, accordingly, irrelevant anonymized data may lead to incorrect translation. In addition, after providing the translated document, it is necessary to restore the original data. Accordingly, in yet another particular embodiment, the claimed solution discloses a data de-anonymization method 300, disclosed in more detail in FIG. 3

[102] Thus, said method 300 may be a reverse process to the steps of method 200

[103] At step 310, the system, for example, system 500, receives a text document in the form of a data file, anonymized in accordance with the steps of method 200. In one particular embodiment, the document may be downloaded to system 500.

[104] At step 320, the system 500 determines the anonymized data in the text document that needs to be anonymized using a lookup table.

[105] This step 320 is based on a lookup table stored during the document anonymization process. Thus, in a text document, all entities contained in the substitution table are defined, for example, full name, name of organization, etc. and for subsequent replacement with confidential data contained in the original document. It is worth noting that the process of determining entities that are subject to de-anonymization can be performed by searching for ordinal indexes in the specified substitution table.

[106] Next, at step 330, the process of replacing the anonymized data with sensitive data is carried out in accordance with the lookup table.

[107] At this step 330, all anonymized values are replaced with the original confidential data. During the de-anonymization process, the document's style is not affected, ensuring that the original document is returned with the same formatting style.

[108] Thus, for example, after a de-identified document has been converted by third parties, said document is uploaded to the system 500. That is, a text document is obtained, anonymized in accordance with the steps of method 200. Next, based on the list of replacements stored in the system, the anonymized data in the text document that needs to be de-identified is determined using a substitution table. After this, the process of replacing anonymized data with confidential data is carried out in accordance with the substitution table.

[109] Thus, the above materials described a method for depersonalizing confidential data in text documents while preserving the data structure.

[110] Now let’s consider a scenario for using some variants of the claimed solution.

[111] Thus, one of the application scenarios may be the translation of contracts, agreements and other documents of the organization by contractors. As noted earlier, to carry out translation, the translator needs the context and semantic content of the data; in addition, the translation also takes into account the lexical and morphological features of the text.

[112] Thus, to implement the claimed invention, the document to be translated is loaded via a user interface (eg, interface 501) into a system, such as system 500. Next, system 500 segments the text data, for example, in accordance with step 120 or step 220, tokenizes and vectorizes the raw data (steps 130 and 140 or steps 230 and 240), and processes the vectorized tokens through a machine learning model trained to recognize sensitive data (step 150 or step 250). Moreover, confidential data is recognized by data types and categories (step 160 or step 260). After this, the confidential data is anonymized and a list of replacements is generated (steps 170 and 180). In addition, if reversible anonymization is selected, then the lookup table is further stored in secure memory of the system, such as system 500 (steps 270 and 280). The final step of the system 500 is to perform replacement of the sensitive data in accordance with the list of replacements in the text document (step 190 or step 290). As can be seen from the presented description of the example, this method can also be performed by system 400, in the case of irreversible depersonalization. At the same time, anonymized confidential data preserves the data structure, namely, formatting style, morphological, lexical and semantic features. The generated new document with anonymized data can then be displayed in the user interface and/or uploaded as a data file. The specified file can then be sent to translators.

[113] Accordingly, once the translation has been completed, the modified data file can be loaded into the system to replace the anonymized data with the original sensitive data.

[114] It is also worth noting that maintaining formatting styles is critical in some areas. For example, some automated systems store documents in an internal representation, and the document style is taken into account by these systems, so if the document styles differ, the system will not be able to correlate the original and anonymized document. Also, styles in documents can be used as part of standardized forms or to highlight key information; for example, without saving styles, a translator will not be able to use them in his work, and therefore will not be able to produce a correct translation. Also, formatting style can be taken into account in the field of artificial intelligence development. So, in addition to the training stage itself, there is a stage of document parsing and data preparation. If you do not save styles, they will not be taken into account during parsing, which will subsequently lead to incorrect data preparation and incorrect training of models. Those. One of the advantages of the disclosed invention is the ability to embed a new tool into existing processes where interaction with documents having certain styles occurs. Removing these styles can lead to disruption of automated systems, decreased quality of information processing, and violation of standards.

[115] In FIG. 6 shows a system 600 implementing the steps of the claimed method (100).

[116] In general, system 600 includes components such as: one or more processors 601, at least one memory 602, data storage 603, input/output interfaces 604, I/O 605, networking 606, which are connected via a universal bus.

[117] Processor 601 performs the basic computational operations necessary to process data when executing method 100. Processor 601 executes the necessary machine-readable instructions contained in main memory 602.

[118] Memory 602 is typically in the form of RAM and contains the necessary software logic to provide the required functionality.

[119] The data storage medium 603 can be in the form of HDD, SSD drives, raid array, flash memory, optical storage devices (CD, DVD, MD, Blue-Ray disks), etc. The facilities 603 allow for long-term storage of various types of information, such as lookup table history, format table, and the like.

[120] To organize the operation of system components 600 and organize the operation of external connected devices, various types of I/O interfaces 604 are used. The choice of appropriate interfaces depends on the specific design of the computing device, which can be, but is not limited to: PCI, AGP, PS/2, IrDa, FireWire, LPT, COM, SATA, IDE, Lightning, USB (2.0, 3.0, 3.1, micro, mini, type C), TRS/Audio jack (2.5, 3.5, 6.35), HDMI, DVI, VGA, Display Port , RJ45, RS232, etc.

[121] The choice of interfaces 604 depends on the specific implementation of the system 600, which can be implemented on a wide class of devices, for example, a personal computer, mainframe, laptop, server cluster, thin client, smartphone, server, etc.

[122] The following can be used as data I/O devices 605: keyboard, joystick, display (touch display), monitor, touch display, touch pad, mouse, light pen, stylus, touch pad, trackball, speakers, microphone, augmented reality tools, optical sensors, tablet, light indicators, projector, camera, biometric identification tools (retina scanner, fingerprint scanner, voice recognition module), etc.

[123] Network communication means 606 are selected from devices that provide network reception and transmission of data, for example, an Ethernet card, WLAN/Wi-Fi module, Bluetooth module, BLE module, NFC module, IrDa, RFID module, GSM modem, etc. . Using tools 606, the organization of data exchange is ensured between, for example, a system 600, represented in the form of a server, and a user’s computing device, on which the received data (anonymized text document) can be displayed via a wired or wireless data transmission channel, for example, WAN, PAN, LAN (LAN), Intranet, Internet, WLAN, WMAN or GSM.

[124] In FIG. 4 shows a special case of an implementation of system 600.

[125] The specified system 400 consists of a graphical user interface 401, a control module 402, an anonymization module 403, a confidential information recognition module 404.

[126] Modules 402-404 may be firmware and may be at least a server, computer, etc., performing a function assigned to it. In addition, it will be obvious to a person skilled in the art that said modules can be implemented using at least one processor and can be logical modules. Thus, the sensitive information recognition (CI) module 404 may contain a machine learning model and is designed to determine categories and types of sensitive information (step 150). Module 403 is accordingly configured to replace certain sensitive data with anonymized data. In one particular embodiment, the specified module 403 may also contain means for classifying types of confidential information and a database with data sets for anonymization.

[127] In FIG. 5 shows another special case of the implementation of system 600.

[128] The specified system 500 consists of a graphical user interface 501, a control module 502, an anonymization module 503, a confidential information recognition module 504, a lookup table management module 505, a data warehouse 506.

[129] Modules 502-506 may be firmware and may be at least a server, computer, etc. that performs its assigned function. In addition, it will be obvious to a person skilled in the art that said modules can be implemented using at least one processor and can be logical modules. Thus, the sensitive information recognition (CI) module 504 may contain a machine learning model and is designed to determine categories and types of sensitive information (step 250). Module 503 is accordingly configured to replace certain sensitive data with anonymized data. In one particular embodiment, the specified module 503 may also contain means for classifying types of confidential information and a database with data sets for anonymization. The storage 506 may represent a remote server, an on-chip secure memory, and/or a secure memory area, etc.

[130] The submitted application materials disclose preferred examples of implementation of a technical solution and should not be interpreted as limiting other, particular examples of its implementation that do not go beyond the scope of the requested legal protection, which are obvious to specialists in the relevant field.

Claims (22)

1. A method for depersonalizing confidential data in text documents while preserving the data structure, containing stages in which: a) receive a document containing text data; b) segmenting the text data obtained at step a), and during segmentation the data is divided along the boundaries of changes in text formatting into parts of the text, and assigning an ordinal position in the text to the specified parts of the text; c) tokenize the text data obtained in step a) and assign ordinal positions in the text to each token; d) vectorize the tokens obtained in step c); e) process the vector representations of the tokens obtained in step d) using a machine learning model based on a neural network trained on sensitive data sets, during which it is determined that each token belongs to the category of confidential data; f) depersonalize data related to tokens with confidential data, and during depersonalization, the data is replaced with data from the same category while maintaining the data structure; g) generate a list of replacements, including indications of the ordinal positions of tokens with confidential data in the text, ordinal positions of changes in the formatting boundaries of parts of the text, anonymized data corresponding to confidential data; h) replace the original confidential data in the text document with anonymized data according to the list of replacements, and during the replacement process, the anonymized data is formatted in accordance with the positions of changing the formatting of parts of the text. 2. The method according to claim 1, characterized in that the text data is received in the form of an unstructured text document. 3. The method according to claim 1, characterized in that the categories of confidential data are at least one of: numeric entities; named entities. 4. The method according to claim 3, characterized in that the numeric entities represent at least one of: the main cardholder number, TIN number, telephone number, account number, insurance certificate number, IP address. 5. The method according to claim 3, characterized in that the named entities represent at least one of: persons, locations, organizations. 6. The method according to claim 1, characterized by the fact that the data structure is consistent with the format, type and semantic content of confidential and anonymized data. 7. The method according to claim 3, characterized in that anonymized data for numerical entities is generated while preserving identifying characteristics. 8. The method according to claim 3, characterized in that anonymized data for named entities is reduced to a morphological form corresponding to confidential data. 9. The method according to claim 1, characterized in that repeated confidential data is anonymized with the same data. 10. A system for depersonalizing confidential data in text documents while preserving the data structure, containing: at least one processor; at least one memory coupled to the processor, which contains machine-readable instructions that, when executed by at least one processor, enable the method of any one of claims to be performed. 1-9.

Images