RU2775351C1 - Method and system for obtaining a vector representation of an electronic document - Google Patents

Abstract

FIELD: computing technology.

SUBSTANCE: computer-implemented method for obtaining a vector representation of an electronic document, executed by means of a processing unit and including the stages of: generating a model of placement of m-skip-n-grams by clusters, wherein the generation of said model involves the following: determining the list of used m-skip-n-grams; converting each m-skip-n-gram from the list into a vector representation; clustering the m-skip-n-grams; processing the text document using the resulting model, involving: calculating the occurrence of m-skip-n-grams in the document; determining clusters of the document based on the occurrence of m-skip-n-grams; summarizing the amount of occurrences of m-skip-n-grams from each cluster; forming a vector representation of the document.

EFFECT: possibility of preserving different semantics of words in the document by matching words to multiple clusters.

10 cl, 6 dwg, 1 tbl

Description

FIELD OF TECHNOLOGY

[0001] The present invention relates to computing systems in a broad sense. More specifically, to systems and methods for processing natural language, artificial language, any sign systems. Can be used in information processing systems, databases, electronic storage

BACKGROUND OF THE INVENTION

[0002] Automatic processing, transmission, storage of documents may include the classification of source documents, their clustering and other actions by correlating a vector representation of a document with another vector representation of a document, any set or group of documents. Embodiments of this invention may be similar to the solutions described earlier in patents RU 2701995 C2, RU 2583716 C2, RU 2254610 C2.

[0003] The main disadvantages of existing solutions are due to the following:

- when clustering words, the semantic connection between words in the text is lost;

- the ongoing clustering does not have a generalizing ability to describe a wide range of possible hidden topics of the text;

- a word can be associated with only one cluster, which leads to the loss of the meaning of the word in various contexts, for example, "key" as a stream of water and "key" as an element of cryptographic systems;

- the resulting clustering is inefficient to use for calculating distances, since there are many common words in all texts;

- it is impossible to take into account new words that were not previously in the used dictionary;

- limitation of application for classification problems.

SUMMARY OF THE INVENTION

[0004] The claimed invention is aimed at solving a technical problem, which consists in improving the quality of analysis (classification, clustering) of text data by converting them into vector form.

[0005] The technical result is to increase the accuracy of the representation of text data in vector format, by using vector representations of t-skip-n-gram words and their application for subsequent clustering of a text document to convert it into a vector form.

[0006] An additional result of the claimed solution is also the preservation of the semantic meaning of the text when it is translated into a vector representation, due to the clustering of directly m-skip-n-gram words.

[0007] The claimed technical result is achieved by performing a computer-implemented method for obtaining a vector representation of an electronic document, which is performed using a processor and contains the following steps:

- at least one model of placing m-skip-n-grams in clusters is formed, while the m-skip-n-gram represents at least a single word, and when forming the said model, the following is carried out:

• definition of the list of used m-skip-n-grams;

• conversion to vector representation of each m-skip-n-gram from the list:

• clustering of m-skip-n-grams according to their vector representations;

- processing at least one text document using the obtained m-skip-n-gram placement model, during which:

• calculate the occurrence of m-skip-n-grams in a text document;

• determine text document clusters based on the occurrence of m-skip-n-grams;

• sum the number of occurrences of m-skip-n-grams from each cluster;

• a vector representation of a text document is formed based on an ordered sequence of m-skip-n-gram sums.

[0008] In one particular embodiment of the method, a fuzzy partitioning of the list of m-skip-n-grams into clusters is used.

[0009] In another particular embodiment of the method, each m-skip-n-gram belongs to several clusters.

[0010] In another particular embodiment of the method, each m-skip-n-gram has a weight characterizing its proximity to a given cluster.

[0011] In another particular implementation of the method, the clustering of m-skip-n-grams according to their vector representations is performed more than once.

[0012] In another particular embodiment of the method, weights are additionally used for m-skip-n-gram clusters, characterizing the significance of clusters for the vector representation of the document.

[0013] In another particular embodiment of the method, vector representations of m-skip-n-grams are additionally obtained. not included in the list, and their correlation by vector representations to at least one cluster.

[0014] In another particular implementation of the method, the list or part of the list of used m-skip-n-grams is formed based on the occurrence of m-skip-n-grams in text data obtained from external data sources.

[0015] The claimed solution is also implemented using a system for obtaining a vector representation of an electronic document, which contains but at least one processor and at least one memory storing machine-readable instructions that, when executed by the processor, implement the above method.

[0016] The claimed technical result is also achieved through a computer-implemented method of searching for sources of information, performed using a processor and containing the steps in which:

- forming at least one model for placing m-skip-n-grams in clusters on a given topic, while the m-skip-n-gram represents at least a single word, and when forming the said model, the following is carried out:

• definition of the list of used m-skip-n-grams;

• conversion to vector representation of each m-skip-n-gram from the list;

• clustering of m-skip-n-grams according to their vector representations;

- receive at least one source of information in the form of an electronic text document;

- processing the received text document using the obtained m-skip-n-gram placement model, during which:

• calculate the occurrence of m-skip-n-grams in a text document;

• determine text document clusters based on the occurrence of m-skip-n-grams:

• sum the number of occurrences of m-skip-n-grams from each cluster;

• form a vector representation of a text document on the basis of an ordered sequence of sums of m-skip-n-grams;

- determine whether the vector representation of the document belongs to a given topic.

[0017] In this document, under the "m-skip-n-gram of words" (or simply "m-skip-n-gram" // http://www.machinelearning.ru/wiki/images/7/78/2017_DrapakSN .pdf) is understood as a set of a sequence of n words, which is obtained from a sequence of words from some text, keeping in it a sequence of words in texts, while m words are removed from the original sequence of words after each one of n words. For example, 0-skip-1-gram words are just single words from a text. 0-skip-2-grams are digrams of words (a pair of consecutive words in a text). 0-skip-3-grams are trigrams of words (three consecutive words in a text). To build a 1-skip-2-gram, a sequence of three words is taken, in which the second word is deleted - that is, these are the first and third words. To build a 2-skip-4-gram, a sequence of 10 words is taken, the first word is taken in it, then 2 words are removed, then the next word is taken, then the next 2 words are removed, and so on, until a sequence of 4 words is obtained. For example, we have the sentence "in accordance with one or more embodiments of the present invention." It removes the words "according to", "or more", "implementation of the present". Then the 2-skip-4-gram for this sentence would look like "in one variation of the invention". At the same time, when constructing m-skip-n-grams, a “word” can be taken as only the words of the language, as well as any punctuation mark, preposition, conjunction, or any independent unit of the language. Everywhere below, without loss of generality to simplify the presentation, by "word" ("words" and so on) we mean any possible m-skip-n-gram of words and a sequence of m-skip-n-grams of words, unless otherwise stated, for example , "individual words", "single words" and the like. In this case, we will also use the original term m-skip-n-gram itself.

[0018] By “embedding” a word (from English embedding - embedding) or a “vector representation” of a word or simply a “vector” of a word, we will understand such a numeric vector that is obtained from words or other linguistic entities, and which is defined for a word, and has a fixed dimension for the method of obtaining it. In other words, the vector representation of a word is an ordered sequence of numbers - a numeric vector of some size, when each word has its own specific numeric vector. In the simplest case, word embeddings can be obtained by numbering words in some dictionary and setting a value equal to 1 in a vector whose dimension is equal to the number of words in this dictionary. In this case, the remaining positions will contain values equal to 0. For example, for the Russian language, you can use Dahl's Explanatory Dictionary. We number all the words in it from the first to the last. So the word "lampshade" would have the value 1 at position 3, "abanat" would have the value 1 at position 7, and so on. If there are 200,000 words in the dictionary, then the embedding will have a dimension of 200,000. A similar method of building embeddings is called one-hot encoding. The description of the invention does not limit the method for obtaining word vectors. The vector data can be obtained, for example, by a neural network that implements a mathematical transformation from a space with one dimension per word to some vector space with a higher dimension. Other methods that allow each m-skip-n-gram to be associated with its own vector of numbers of a given dimension. This vector representation of words can be obtained from already known set of vectorized words (Word2Vec, Glove, FastText and others), with or without modification. In this case, clustering is understood as a grouping of a set of objects into subsets (clusters) in such a way that objects from one cluster are more similar to each other than to objects from other clusters according to a given criterion. The “weight” of an element, such as “cluster weight” or “m-skip-n-gram weight”, can be understood as a mathematical construction, coefficients, multipliers used when summing, integrating or averaging, etc., in order to give some elements greater significance in the resulting value compared to other elements. "Weight" can also be defined as an additional factor, coefficient or number associated with individual terms or other factors in the scalar product of the elements of the vector space used.

BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The present invention is illustrated by examples, without any limitation; its essence becomes clear when considering the following detailed description of the invention in combination with the drawings, while:

[0020] In FIG. 1 schematically shows an example of a process for automatically obtaining a vector representation of a document.

[0021] In FIG. 2 schematically shows an example of a process for obtaining a plurality of cluster sets, with weights for each cluster in each cluster set.

[0022] In FIG. 3 schematically shows an example of a process for obtaining a plurality of m-skip-n-gram words, where each m-skip-n-gram word is loosely related to each cluster in each set of clusters.

[0023] In FIG. 4 schematically shows an example of a document feature extraction process.

[0024] In FIG. Figure 5 schematically shows an example of matching word clusters with positions of the document vector, for the two-dimensional case of the vector representation of words and a clear correlation of the word to one cluster.

[0025] FIG. 6 illustrates the general layout of a computing device.

IMPLEMENTATION OF THE INVENTION

[0026] This document describes how to obtain a vector representation of an electronic document for further processing, transmission and storage. The invention can be applied to any natural languages, artificial languages, any sign system, and, further in this document, all of this will be referred to simply as "language". Thus, further, "language" means natural languages, artificial languages, any sign systems.

[0027] Automatic processing, transmission, storage of documents may include the classification of source documents, their clustering and other actions by correlating a vector representation of a document with another vector representation of a document, any set or group of documents.

[0028] In FIG. 1 shows a general block diagram of the claimed method for obtaining the vector form of a document. At the first stage (101) of the implementation of the invention, a model for placing m-skip-n-grams in clusters is obtained. Such a model is a list of m-skip-n-grams, each of which is assigned to at least one of the clusters. An example of such a model for a particular case, when m-skip-n-grams are symbols and words from the Russian language, and a breakdown into 1000 clusters is shown in the table below:

The process of obtaining a placement model of m-skip-n-grams by clusters.

[0029] To generate the model, a list of m-skip-n-grams to be used is first determined. In a particular case, this can be an explanatory dictionary of the language, dictionaries of word embeddings (Word2Vec. Glove, FastText and others), a list of all words from articles from the Wikipedia site. In such a case, the word list must be large to cover a sufficient number of words from the analyzed documents. In another particular case, this list goes through the normalization of words - the process when they receive lemmas (https://ru.wikipedia.org/wiki/Lemmatization), stemmings (https://ru.wikipedia.org/wiki/Stemming), morphological roots, morphological bases of words. In a particular case, capital (uppercase) and lowercase letters do not differ when forming m-skip-n-grams. In another particular case, capital letters differ from lowercase letters and their presence forms different m-skip-n-grams. In addition to the words of natural, formal, artificial languages, the list may include punctuation marks, numbers, abbreviations, common nouns, proper names, abbreviations, special characters (@";##$%%^*()_}{⎢\ and others) , mathematical symbols, and any other printable sequences of characters.In a particular case, any sequence of printable characters in the sign system can be considered with the entered word separation sign.The size of the list, specific m-skip-n-grams.The use of normalization is selected based on the problem being solved, available computational resources and other factors.

[0030] Next, each m-skip-n-gram from the list is converted into a vector representation. The present description does not limit the method of obtaining, the dimension or the form of the vector representation. To do this, you can use methods that are based on the algorithms and models of Word2Vec, Glove, FastText, Universal Sentence Encoder, addition with singular value transformation, Tf-Idf. In one particular implementation, the m-skip-n-gram vector is a concatenation of several vectors of this m-skip-n-gram obtained by different methods.

[0031] Next, the m-skip-n-gram vectors are clustered. The present description does not limit the way the m-skip-n-gram list is clustered. You can use such clustering algorithms as K-Means, Affinity propagation, Mean-shift, Spectral clustering. Ward hierarchical clustering, Agglomerative clustering, DBSCAN. OPTICS, Gaussian mixtures, Birch. The end result of clustering is the assignment of each m-skip-n-gram to at least one cluster.

[0032] Upon completion of obtaining the m-skip-n-gram placement model, processing of text documents is performed at step (102) using it. Upon completion of the application of the model, we obtain a vector view of the document (103). If there are still unprocessed documents, then go to the next document. So until each document is processed.

[0033] In FIG. Figure 2 shows an example of the stages of forming a model for placing m-skip-n-grams by clusters (101). Here, a set of cluster sets is obtained with weights for each cluster in each set. First, we select the basic list of m-skip-n-grams, then we get their vector (1011). The present description does not limit in any way the list, the form of m-skip-n-grams and the values of the parameters m and n in them. A distinctive part of the invention is the very possibility of using not only separate, single words of the language, but any m-skip-n-gram of words. In this case, the possibility of choosing such n and m is not limited. that the resulting m-skip-n-grams will coincide, for example, with individual words, bigrams of words, trigrams of words, and so on.

[0034] This stage can be implemented by using ready-made dictionaries (Word2Vec, Glove, FastText and others), defining all possible or only m-skip-n-grams of interest exclusively in the class of text documents being analyzed (according to the available database or training set) or any in a different way. Next, we obtain a vector representation of the selected m-skip-n-grams in any way (algorithms and models Word2Vec, Glove. FastText, Universal Sentence Encoder, addition with singular value transformation, Tf-Idf and others) (1011). At step 1012, we perform multiple clustering of the vector representation of m-skip-n-gram words. This clustering is performed by any known method of object clustering, in which the required number of clusters is initially fixed. A distinctive feature of the invention is that the clustering can be carried out several times. At the same time, it can be performed for a different number of clusters by the same clustering algorithm, for the same number of clusters by the same clustering algorithm, but with different initial initializations, for the same number of clusters by different clustering algorithms, or in any other way to obtain different sets of object clusterings.

задается свой весовой коэффициент q_ij, который может характеризовать значимость этого кластера для векторного представления документа (этап 1014). Описание изобретения не ограничивает методы получения весовых коэффициентов q_ij и их значения. Отличительной частью изобретения являет сама возможность использования весовых коэффициентов для кластеров, что позволяет характеризовать значимость каждого кластера для векторного представления документа. Одной из возможных реализаций является подход, когда производится выделение «мусорных» кластеров и исключение данного кластера, например, обнулением его веса. При этом «мусорным» кластером могут являться кластеры с высокой долей «стоп-слов», общих слов, не несущих информативность слов для конкретной решаемой задачи анализа текстовых данных. Значения весовых коэффициентов можно получить какими-либо автоматизированными вычислениями, определить экспертной оценкой, любым иным способом. При этом возможна ситуация, когда все кластеры являются равнозначными или, как отмечено выше, когда какие-либо кластеры вообще исключаются из использования. Результатом данного этапа является множество наборов кластеров с весами для каждого кластера в каждом наборе (этап 1015).[0035] Thus, a set of cluster sets K={k ₁ , k ₂ , ..., k _N-1 , k _N ] is formed (step 1013). For each C _ij cluster - the j-th cluster in the i-th partition into clusters, where

its own weight coefficient q _ij is set, which can characterize the significance of this cluster for the vector representation of the document (step 1014). The description of the invention does not limit the methods for obtaining the weight coefficients q _ij and their values. A distinctive part of the invention is the very possibility of using weight coefficients for clusters, which allows characterizing the significance of each cluster for the vector representation of the document. One of the possible implementations is the approach when "garbage" clusters are selected and this cluster is excluded, for example, by zeroing its weight. At the same time, a "garbage" cluster can be clusters with a high proportion of "stop words", common words that do not carry the information content of words for a specific problem of text data analysis being solved. The values of the weight coefficients can be obtained by any automated calculations, determined by an expert assessment, in any other way. In this case, a situation is possible when all clusters are equivalent or, as noted above, when any clusters are generally excluded from use. The result of this step is a plurality of cluster sets with weights for each cluster in each set (step 1015).

Тогда общее множество наборов кластеров обозначим K={k₁, k₂, …, k_n-1, k_N}. Каждая i-ая кластеризация k_i будет представлять собой набор из M_i кластеров. Каждый j-й кластер в i-м разбиении на кластеры обозначим C_ij, где

,

. Далее, для данного множество наборов кластеров K={k₁, k₂, …, k_N-1, k_N} выбираем показатели соотнесения слова к кластерам (этап 201). Показатель соотнесения слова к кластерам позволяет определить нечеткое распределение слов по кластерам. В качестве такого показателя может быть расстояние от m-skip-n-грамм до центра кластера. Данное расстояние может нормироваться по расстояниям до центра всех кластеров, или по расстояниям только до нескольких (возможен вариант ближайших) кластеров. Другой возможный вариант реализации изобретения состоит в том, чтобы определять расстояние до оправленного количества ближайших объектов при четкой кластеризации и рассчитывать долю объектов из каждого кластера в этом оправленном количестве. При этом описание изобретение никак не ограничивает способ, меру и показатель получения нечеткого соотнесения объекта к кластеру. Отличительной особенностью изобретения является возможность использовать нечеткое соотнесение m-skip-n-грамм к кластерам. Таким образом, для каждого используемого s-го слова определяется некоторый показатель w^(s) _ij, который характеризует в какой степени s-e слово относится к кластеру C_ij (этап 302).[0036] In FIG. 3 shows an example of the stages of obtaining a fuzzy correlation of words from the list. For all or part of the words from the dictionary, we perform clustering N times (step 1013) as follows: for each clustering, a certain number of clusters can be selected; some clustering method is selected that allows objects to be clustered into a given or indefinite number of clusters, and clustering is performed. Denote each clustering by k _i , where

Then the total set of sets of clusters will be denoted as K={k ₁ , k ₂ , …, k _n-1 , k _N }. Each i-th clustering k _i will be a set of M _i clusters. Each j-th cluster in the i-th partition into clusters will be denoted by C _ij , where

,

. Next, for a given set of sets of clusters K={k ₁ , k ₂ , ..., k _N-1 , k _N }, we select indicators of matching words to clusters (step 201). The word-to-cluster correlation indicator allows you to determine the fuzzy distribution of words by clusters. As such an indicator, the distance from m-skip-n-grams to the center of the cluster can be. This distance can be normalized by the distances to the center of all clusters, or by the distances to only a few (possibly closest) clusters. Another possible implementation of the invention is to determine the distance to a fixed number of nearest objects with a clear clustering and calculate the proportion of objects from each cluster in this adjusted number. At the same time, the description of the invention does not limit in any way the method, measure and indicator of obtaining a fuzzy assignment of an object to a cluster. A distinctive feature of the invention is the ability to use fuzzy mapping of m-skip-n-grams to clusters. Thus, for each s-th word used, some indicator w ^(s) _ij is determined, which characterizes to what extent se the word belongs to the cluster C _ij (step 302).

[0037] We can attribute each word to any number of clusters. Such a clustering with fuzzy clustering can be obtained, for example, using the “C-means” fuzzy clustering method, or using an expert's assessment. In this case, indicators can be chosen such that each word will belong to only one cluster, for example, in the case when the value of the correlation indicator is equal to one for one cluster (not necessarily the word closest to the vector) and equal to zero for all others. In some cases, the results of such a choice will characterize a clear clustering, but embodiments of the invention are possible when m-skip-n-gram will not always refer to the closest cluster. That is, while the results of such a choice of correlation indicators, in the general case, will not necessarily correspond to the results of the work of clear word clustering algorithms. The description of the invention does not limit the number of clusterings, which, obviously, must be greater than zero (>0). The description of the invention does not limit the metric and mathematical spaces for clustering and matching words to clusters.

[0038] Steps (201) and (202) have common preceding steps with steps (1014) and (1015). These steps are steps (1011, 1012, 1013). The description of the invention does not limit the mutual implementation of the mentioned stages of the steps and their combined use, but only provides explanations for them.

[0039] An exemplary disclosure of step (102) is shown in FIG. 4. At the input of stage (102) is a document with data in text form. In a particular implementation of the method, the m-skip-n-grams that are present in the document and that are present in the list of m-skip-n-grams of the placement model are first determined (step 1021).

[0040] When implementing step (1021), the number of occurrences of m-skip-n-grams from the generated list in the document is counted. Then it looks to which clusters these m-skip-n-grams belong. The number of m-skip-n-grams from the document within each cluster is counted.

For example, we have the following sentence: "the minister will speak this week." In it, in accordance with the example from Table 1, cluster 457 occurs 1 time (“minister”), cluster 537 - 1 time (“speak”), cluster 737 - 2 times (“on”, “this”), cluster 368 - 1 time ("week"). Then the final document vector will be a vector with the value 1 at positions 457, 537 and 368, the value 2 at positions 737, and the value 0 at all other positions. In another particular embodiment of the method, subsequent normalization of this vector is used. In the considered example m-skip-n-grams from the list occur 5 times. Then the normalized vector will be a vector with the value 0.2 at positions 457, 537 and 368, the value 0.4 at position 737, and the value 0 at all other positions.

[0041] Next, the formation of the document vector begins. Each position in this vector corresponds to a specific cluster. First, an m-skip-n-gram is taken from the document, for subsequent determination to which cluster it belongs according to the generated m-skip-n-gram placement model. The value at the position corresponding to the cluster in the document vector is then incremented by the number of that m-skip-n-gram in the document (step 1022). In one of the particular implementation options, the value in the position of the vector is increased not by the number of m-skip-n-grams, but by the product of the number and weight of the m-skip-n-gram for this cluster (ethane 1022). At the next stage (1023), the obtained values at the positions of the document vector are multiplied by the corresponding weights of the clusters. The result is a vector representation of the document (block 1024).

[0042] In accordance with one or more embodiments of the present invention, an example of a method for automatically obtaining a vector representation of an electronic document may include the following steps. A vector is selected whose dimension coincides with the total number of clusters obtained in all clusterings. This vector is initialized with arbitrary initial values. The description of the invention does not limit the values used for the initial initialization of the vector.

[0043] It is also possible to use, for example, zero values. Each position of the vector strictly corresponds to a certain cluster (Fig. 5). The words are extracted from the document. This extraction can be obtained, for example, by a sequential pass through the document, use some representation of the document, which already has the extracted words and their numbers, in any other way. For each word, ratio weights are used for each cluster in each set (step 202).

[0044] For example, consider the sequential extraction of words from text, without limiting the embodiments of the invention. The word is obtained from the text. According to the weights of the correlation of a word with each cluster, a search is made for a position in the document vector, the values in which must be increased. We increase these values in the identified positions by a certain value. This value can be fixed or change depending on the conditions during document processing. This value can already take into account the weight of the word's relationship with each cluster.

[0045] It is also possible for the invention to increase the values at a position in the vector by a fixed value, which is then multiplied by the weight of the word ratio. The description of the present solution does not limit methods for changing values at positions in a vector that are associated with a given word. It is also possible to take into account the fuzzy correlation of the word to different clusters in the final change in the values in the vector. In this case, the values of the change in values are not limited, which, in the general case, can be negative. To calculate the values by which the values in the positions of the vectors change, you can use various methods that allow you to take into account any other characteristics of the words. An example would be to use the Term Frequency - Inverse Document Frequency (TF-IDF) method, or just word frequency. Also, to calculate the values, you can take into account the weight of the cluster, which characterizes the significance of the cluster for the vector representation of the document. Passing through the entire text in this way, we increase the corresponding positions in the document vector. The result of the pass can be a vector view of the document.

[0046] The present description does not limit approaches to using cluster weights that characterize the significance of each cluster for a vector representation of a document, and word ratio weights for each cluster in each set. The claimed solution makes it possible to use weight data when compiling a vector view of a document. The obtained values can be used for the vector view of the document, or any set and subset of them can be used in data processing, in an independent form or in combination with other data, to obtain a new vector view of the document. An example of such cumulative usage would be concatenation with a TF-IDF method vector. In this case, the vector representation of the document is compiled using the TF-IDF method. Compilation of the document vector can be carried out by performing the claimed method. Two vectors are concatenated (the operation of joining, gluing vectors). The result of the concatenation will be the document vector. Additionally, it is possible to carry out further algebraic transformations on the resulting document vector.

[0047]

[0048] In one particular embodiment of the method, fuzzy partitioning of the list of m-skip-n-grams into clusters is used. In this variant, each m-skip-n-gram corresponds to more than one cluster. Each m-skip-n-gram has its own correlation to a cluster depending on the distance to this cluster. In this case, both the coordinates of the cluster center and the coordinates of m-skip-n-grams from this and other clusters can be used to calculate this distance. In a special case of this clustering method, the m-skip-n-gram applies to all clusters. An example of an algorithm for implementing this particular clustering variant is C-Means.

[0049] In one particular implementation, each m-skip-n-gram belongs to several clusters. In this case, the m-skip-n-gram can refer to one or more clusters to the same extent.

[0050] In another particular implementation, each m-skip-n-gram has a weight characterizing its correlation to a given cluster. In contrast to the previous particular variants, in this case, the weight can characterize not only the proximity of the m-skip-n-gram to the cluster. When calculating this weight, for example, the density of clusters can be taken into account in order to correlate the m-skip-n-gram more to a less dense close cluster. The weight is a multiplier that is used for each m-skip-n-gram with each cluster. Weight can also be zero.

[0051] In yet another particular implementation, clustering a list of m-skip-n-grams by their vector representations is performed more than once. In this variant, clustering is performed several times for a different number of clusters by the same clustering algorithm. The resulting document vector will be a concatenation of the vectors for each of the clusterings (example in Fig. 5). In another case, different clustering algorithms are used to partition into the same number of clusters or into a different number of clusters. In another case, the same number of clusters and the same algorithm are used, but with different initial initializations, if the algorithm implies the possibility of different results with different initializations (K-Means, C-Means, Spectral clustering, Gaussian mixtures, and others). The number of clustering, the number of clusters in each clustering and the clustering algorithm is selected depending on the problem being solved. In a particular case, if the list of m-skip-n-grams is more than 10000, you can use a breakdown into clusters with sizes: 50, 100, 200, 300, 500, 700, 1000, 1500, 2000, 3000, 5000. separate lists, for example, for 0-skip-1-grams (single words), 0-skip-2-grams (bigrams of words), 0-skip-3-grams (trigrams of words). Each of these lists is separately clustered multiple times. The final document vector will be the concatenation of vectors for each of the clustering for each list.

[0052] In another particular implementation, weights are used for each m-skip-n-gram cluster in each clustering (if there is more than one). These weights characterize the significance of clusters for the vector representation of the document. The weights can also be zero. In this case, the number of occurrences of m-skip-n-grams in this cluster is reset to zero. This approach is useful when it is required to exclude from the vector of a cluster that contain stop words - words that do not carry a thematic meaning (i, k, y, o, at, on, and others). In particular cases, the weights for clusters can be calculated by machine learning methods and given as an expert estimate.

[0053] Also, the m-skip-n-gram can be expanded with new m-skip-n-grams. In this option, the method for calculating the m-skip-n-gram vector is selected. If the document encounters an m-skip-n-gram that is not in the list, then a vector representation (vector) is calculated for it. Further, this vector is related to the clusters that are obtained at the stage of clustering the list of m-skip-n-grams. In this case, the new m-skip-n-gram corresponds to at least one cluster. Further steps of the method are similar to the case if this m-skip-n-gram is present in the model for placing m-skip-n-grams in clusters.

[0054] The list of used m-skip-n-grams can be formed based on the occurrence of m-skip-n-grams in text data obtained from external data sources. Such a set of texts can be any external array of text data. This can be, for example, a set of analyzed texts on various topics, collected, for example, through news sites. Processing, classification and clustering of the collected information allows you to find semantic similarities, analogues, implement the ranking of search results or solve any other language processing problem. The application of the claimed solution does not limit the source and nature of the set of test data used.

[0055] An example of the implementation of such an option is the case of solving the problem of classifying text documents. Then the list of m-skip-n-grams is formed according to the initial set of documents. For example, all single words and bigrams that occur in these texts are taken. Or 200,000 most frequent bigrams. For these lists, their vector representations are calculated. Further, these vectors are fed to clustering, which is implemented by the described methods.

[0056] When implementing the present invention, all the described options can be used in any possible combination and combination. An example of such a particular variant is the case when all occurring single words, bigrams of words, trigrams of words are selected from the available data set. In each list, vectors of n-gram words are calculated. Further, each list of vectors is submitted for multiple clustering. In each clustering, for each n-gram, the weight of its correlation with the cluster is selected. Then we take each analyzed document. By the occurrence of n-grams, we obtain a vector for each clustering. The values in the clusters are multiplied by the weights of the clusters. Each such vector is normalized. All vectors are concatenated into a common vector. The description of the patent does not limit the combination of use of various approaches.

[0057] A distinctive feature of the invention is the ability to account for previously unknown m-skip-n-grams. Without limiting the invention, the following approach can be given to account for new m-skip-n-grams. For example, when passing through the textual representation of the document, when an m-skip-n-gram is found that is not in the used dictionary, the subsequent vector representation of this m-skip-n-gram is formed. After which the m-skip-n-gram refers to a certain cluster, the center of which is closest to it.

[0058] The claimed method (100) can be used in particular for searching and selecting relevant information, for example, news related to a given topic. Method (100) may be used as part of an automated cybersecurity management platform. News gathering is an important function for finding information potentially related to cybersecurity from various sources (news thematic sites, social networks, groups in instant messengers, etc.). News collection is carried out by various well-known mechanisms (RSS subscriptions, etc.).

[0059] The organization of interaction between the TIP and the N LP model is carried out via the API, by sending http get and http patch requests. In the json format, the body of the news is received from the TIP, then the news is preprocessed. The next step is classification, if the news is classified as “applicable”, clustering follows. The result obtained by news belonging to a class and a cluster is transmitted to TIP by sending an http patch request.

[0060] The NLP model performs the classification and clustering of news applicable to cybersecurity. To solve the classification problem, the method of vector representation of texts, disclosed in the description, is used. Further, the news that is of interest for cybersecurity purposes goes through the process of clustering into a limited set of groups (for example, news about the emergence of new vulnerabilities, the release of security updates, the emergence of new malware, etc.). For clustering, a vector representation of the text of the news, disclosed in the description, is used. Clustering occurs by calculating a certain distance between vector representations of news, if the resulting value is less than a given or adaptive threshold (for example, 0.5), the news belongs to one or more clusters.

[0061] In FIG. 6 shows a general view of a computing device (400) suitable for implementing the claimed solution. Device (400) may be, for example, a server or other type of computing device that can be used to implement the claimed technical solution. Including being part of a cloud computing platform.

[0062] In general, the computing device (400) contains one or more processors (401) connected by a common information exchange bus, memory means such as RAM (402) and ROM (403), input / output interfaces (404), input devices / output (405), and a device for networking (406).

[0063] The processor (401) (or multiple processors, multi-core processor) may be selected from a variety of devices currently widely used, such as Intel™, AMD™, Apple™, Samsung Exynos™, MediaTEK™, Qualcomm Snapdragon™, and etc. The processor (501) can also be a graphics processor such as Nvidia, AMD, Graphcore, etc.

[0064] RAM (402) is a random access memory and is designed to store machine-readable instructions executable by the processor (401) to perform the necessary data logical processing operations. The RAM (402) typically contains the executable instructions of the operating system and associated software components (applications, program modules, etc.).

[0065] A ROM (403) is one or more persistent storage devices such as a hard disk drive (HDD), a solid state drive (SSD), flash memory (EEPROM, NAND, etc.), optical storage media ( CD-R/RW, DVD-R/RW, BlueRay Disc, MD), etc.

[0066] Various types of I/O interfaces (404) are used to organize the operation of device components (400) and organize the operation of external connected devices. The choice of appropriate interfaces depends on the specific design of the computing device, which can be, but not limited to: PCI, AGP, PS/2, IrDa, Fire Wire, LPT, COM, SATA, IDE, Lightning, USB (2.0, 3.0, 3.1, micro , mini, type C), TRS/Audio jack (2.5, 3.5, 6.35), HDMI, DV1, VGA, Display Port, RJ45, RS232, etc.

[0067] To ensure user interaction with the computing device (400), various means (405) of I / O information are used, for example, a keyboard, a display (monitor), a touch screen, a touch pad, a joystick, a mouse, a light pen, a stylus, touch panel, trackball, speakers, microphone, augmented reality, optical sensors, tablet, indicator lights, projector, camera, biometric identification tools (retinal scanner, fingerprint scanner, voice recognition module), etc.

[0068] The networking means (406) enables data communication by the device (400) via an internal or external computer network, such as an Intranet, Internet, LAN, and the like. As one or more means (406) can be used, but not limited to: Ethernet card, GSM modem, GPRS modem, LTE modem, 5G modem, satellite communication module, NFC module, Bluetooth and / or BLE module, Wi-Fi module and others

[0069] Additionally, satellite navigation tools in the device (400) can also be used, for example, GPS, GLONASS, BeiDou, Galileo.

[0070] Thus, the claimed solution achieves the following advantages:

• Increase the preservation of semantic meaning by clustering m-skip-n-gram words, not just individual words as done in known patents.

• Increase the accuracy of text representation using latent topics through the implementation of multiple clustering of the same m-skip-n-gram words, in contrast to known patents, where it is not indicated anywhere that multiple or repeated clustering can be performed. In the approach of our patent, for example, it is possible to cluster by 100, 200, 357, 500, 500 (again by 500, but with a new initialization), 1000, 5000, 10000 clusters.

• Preserve different semantics of words by comparing words to several clusters, thereby preserving the ambiguity of terms, in contrast to known patents, where we assign each word to only one (closest) cluster.

• By using weights (a priori or a posteriori) for clusters, it is possible to reduce the significance or even exclude clusters with common words, thereby improving the quality of the analysis, in particular the calculation of the proximity of two texts, in contrast to known patents, where there are no cluster weights.

• Increase the accuracy of the analysis through the processing of new, previously unknown words.

• Expand the application of the resulting vector representation and the direct use of the resulting vector, without training classifiers, for the tasks of identifying semantic similarities, searching for analogues, ranking search results, and others.

[0071] The submitted application materials disclose preferred examples of the implementation of the technical solution and should not be construed as limiting other, particular examples of its implementation that do not go beyond the scope of the requested legal protection, which are obvious to specialists in the relevant field of technology.

Claims (30)

1. A computer-implemented method for obtaining a vector representation of an electronic document, which is performed using a processor and contains the following steps: - at least one model of placing m-skip-n-grams in clusters is formed, while the m-skip-n-gram represents at least a single word, and when forming the said model, the following is carried out: - definition of the list of used m-skip-n-grams; - conversion to vector representation of each m-skip-n-gram from the list; - clustering of m-skip-n-grams according to their vector representations; - processing at least one text document using the obtained m-skip-n-gram placement model, during which: - carry out the calculation of the occurrence of m-skip-n-grams in a text document; - determine the text document clusters based on the occurrence of m-skip-n-grams; - summarize the number of occurrences of m-skip-n-grams from each cluster; - form a vector representation of the text document based on the ordered sequence of sums of m-skip-n-grams. 2. The method according to claim 1, characterized in that fuzzy partitioning of the list of m-skip-n-grams into clusters is used. 3. The method according to claim 1, characterized in that each m-skip-n-gram belongs to several clusters. 4. The method according to p. 1, characterized in that each m-skip-n-gram has a weight characterizing its correlation to a given cluster. 5. The method according to claim 1, characterized in that the clustering of m-skip-n-grams according to their vector representations is performed more than once. 6. The method according to claim 1, characterized in that weights are additionally used for clusters of m-skip-n-grams, characterizing the significance of clusters for the vector representation of the document. 7. The method according to claim 1, characterized in that additionally, vector representations of m-skip-n-grams not included in the list are obtained and their correlation according to vector representations to at least one cluster. 8. The method according to claim 1, characterized in that the list or part of the list of used m-skip-n-grams is formed based on the occurrence of m-skip-n-grams in text data obtained from external data sources. 9. A system for obtaining a vector representation of an electronic document, comprising at least one processor and at least one memory storing machine-readable instructions that, when executed by the processor, implement the method according to any one of paragraphs. 1-8. 10. A computer-implemented method for searching for sources of information, performed using a processor and containing the steps at which: - forming at least one model for placing m-skip-n-grams in clusters on a given topic, while the m-skip-n-gram represents at least a single word, and when forming the said model, the following is carried out: - definition of the list of used m-skip-n-grams; - conversion to vector representation of each m-skip-n-gram from the list; - clustering of m-skip-n-grams according to their vector representations; - receive at least one source of information in the form of an electronic text document; - processing the received text document using the obtained m-skip-n-gram placement model, during which: - carry out the calculation of the occurrence of m-skip-n-grams in a text document; - determine the text document clusters based on the occurrence of m-skip-n-grams; - summarize the number of occurrences of m-skip-n-grams from each cluster; - form a vector representation of a text document on the basis of an ordered sequence of sums of m-skip-n-grams; - determine whether the vector representation of the document belongs to a given topic.

Images