RU2500024C2 - Method for automated language detection and (or) text document coding - Google Patents

Abstract

FIELD: information technologies.

SUBSTANCE: in the method of automated language detection and (or) text document coding, byte sequences are identified, and statistics of frequency of identified byte sequences is counted. Then, using the statistics, profiles of each language and (or) each coding are built, a search engine is built to extract sought-for byte sequences from the byte flow of the inspected document, and the built search engine and profiles of languages and (or) codes are saved into the memory. Byte sequences are found in electronic version of each inspected document with the help of the search engine, and statistics of frequency of found byte sequences is counted as the profile of the inspected document. The calculated profile of the inspected document is compared with profiles of languages and (or) codes to identify relevance of the language and (or) code to this inspected document.

EFFECT: expanded arsenal of technical facilities, making it possible to automatically detect language and coding of text according to previously collected statistics in any text documents.

3 cl

Description

FIELD OF THE INVENTION

The present invention relates to the automated determination of the language and / or encoding of a text document and can be used to develop new and improve existing systems for checking text documents.

State of the art

In the process of automatic analysis of text documents, for example, in the case of tracking documents passing through the company’s network for the presence of confidential information in them, the task of determining the language in which the document being checked is written, or the encoding of the binary representation of this document, is acute.

Currently, several methods are known for determining the language and / or encoding of a text document.

So, in RF patent No. 2251737 (publ. 10.05.2005), a method is described for automatically determining the language of a recognizable text with multilingual recognition, in which hypotheses about the language of a group of characters are formed from recognized individual characters of the text and these hypotheses are tested based on the list of used linguistic models, containing pre-compiled features of characters of all alleged languages. Something similar is also used in application for US patent No. 2009/0024385 (published on January 22, 2009), which describes a semantic grammar analyzer that performs sequential analysis of sentences with components in several languages, based on which a graph is used for further analysis. The effectiveness of these methods is not high enough due to the use of individual characters or groups of characters, as the text may contain quotes in another language.

The closest can be considered US patent No. 8041566 (publ. 18.10.2011), which discloses thematically specific models for formatting text and speech recognition. In this method, text is segmented, and each segment is assigned a model from a set of thematically specific models containing statistical information about the probability of a language model, text processing and formatting rules, for example, interpretation of commands for punctuation, highlighting parts of text, etc., as well characteristic vocabulary for this segment. The method has limited application, since it is primarily intended for formatting text.

Disclosure of invention

The purpose of the present invention is to provide such a method that would expand the arsenal of technical means and allow you to automatically determine the language and (or) the encoding of the text according to pre-set statistics in any text documents.

To solve this problem and achieve the indicated technical result, the present invention proposes a method for automatically determining the language and / or encoding of a text document, which consists in the following: select byte sequences included in electronic versions of reference documents, the texts of which are written in the corresponding language and ( or) encoded with the appropriate encoding; count the statistics of occurrence of selected byte sequences for reference documents whose texts are written in one language and (or) encoded in one encoding; build on the basis of the calculated statistics the profiles of each language and (or) each encoding in the form of a set of byte sequences indicating the weight of each byte sequence in a given language and (or) in this encoding; building a search engine to extract the desired byte sequences from the byte stream of the document being checked; save in memory the constructed search engine and profiles of languages and (or) encodings, collectively referred to as the ontology of this collection of languages and (or) encodings; find byte sequences in the electronic version of each document being checked using a search engine; counting the statistics of occurrence of found byte sequences based on the ontology as the profile of the document being checked; match the calculated profile of the checked document with the profiles of languages and (or) encodings to determine the relevance of the language and (or) the encoding of this checked document.

A feature of the method of the present invention is that for each language and / or for each encoding, the most characteristic byte sequences can be selected.

Another feature of the method of the present invention is that a search engine can be built on the basis of the Aho-Korasik algorithm for exact matching of sets.

DETAILED DESCRIPTION OF THE INVENTION

The present invention can be implemented in any computing system, for example, in a personal computer, on a server, etc. For the implementation of the invention, it is also necessary to have an appropriate database in which electronic files of text documents are stored.

A method for automatically determining the language and / or encoding of a text document is intended to automatically determine the language and / or encoding of a text document.

First, on the basis of specially selected collections of texts in various languages or already in the process of use (by supplementing the original collection of texts) in electronic versions of standard documents whose texts are written in the corresponding language and (or) encoded with the appropriate encoding, byte sequences are selected and statistics on the occurrence of selected byte sequences for reference documents whose texts are written in one language and (or) encoded in one encoding. Based on the calculated statistics, profiles of each language and (or) each encoding are constructed in the form of a set of byte sequences indicating the weight of each byte sequence in a given language and (or) in a given encoding.

It should be noted that when accumulating byte sequences for each language and (or) for each encoding, operators working at the workstation can select the most characteristic byte sequences. This operation is carried out for the so-called (in machine learning) "dimensional reduction". To do this, you can use various algorithms, in particular, an algorithm for calculating mutual occurrence (mutual information - see, for example, http://en.wikipedia.org/wiki/Mutual_information) of the byte sequence in the marked up documents and in the categories by which the documents are marked out. In this case, the categories are languages or encodings. For example, let the sequence “abv” be found in Ukrainian and Russian in 10 documents of the Ukrainian language and 15 documents of Russian. There are 50 documents in Russian, 30 in Ukrainian, respectively. Let there be 8 languages in all, in which documents are presented.

Based on these numbers, the mutual occurrence of the byte sequence (in this case, the sequence “abv”) in these documents is calculated.

The profile of each language and (or) each encoding is usually formed in the form of a vector of selected byte sequences characteristic of a given language and (or) this encoding, one vector per language.

In principle, different collections of documents (“training sets”) are used for languages and encodings, but the actions performed in both cases will be the same. As languages, not only natural languages can be used, but also programming languages. Encodings can be any: for example, it can be ASCII or any other, including specially designed.

Next, a search engine is built to extract the desired byte sequences from the byte stream of the document to be checked. This search engine can be built on the basis of the Aho-Korasik algorithm for exact matching of sets (see, for example, http://e-maxx.ru/algo/aho_corasick or http://aho-corasick.narod.ru). The constructed search engine and profiles of languages and (or) encodings, collectively referred to as the ontology of this collection of languages and (or) encodings, are stored in memory for possible further use when checking regular text documents.

Now, upon receipt of the input byte stream of the document being checked, byte sequences are found in the electronic version of each document being checked using the search engine and statistics on the occurrence of the found byte sequences based on the stored ontology are calculated. In other words, using the constructed and memorized search engine, the byte sequences highlighted in the reference documents are automatically detected in the document being checked and counts the statistics of occurrence. This statistic is the profile of the document being checked. Then, this profile (calculated statistics) of the checked document is compared with the stored profiles of languages and (or) encodings to determine the relevance of a particular language and (or) this or that encoding to this checked document. As the language and / or encoding determined as a result of these actions, choose, for example, the language or the encoding that have the highest degree of relevance.

Thus, the method of the present invention, expanding the arsenal of technical means, allows you to automatically determine the language and (or) the encoding of the text according to pre-collected statistics in any text documents.

Claims (3)

1. A method for automatically determining the language and (or) encoding of a text document, which consists in the fact that:
- select byte sequences included in electronic versions of standard documents, the texts of which are written in the corresponding language and (or) encoded with the corresponding encoding;
- calculate the statistics of occurrence of selected byte sequences for reference documents, the texts of which are written in one language and (or) encoded in one encoding;
- build on the basis of the calculated statistics the profiles of each language and (or) each encoding in the form of a set of byte sequences indicating the weight of each byte sequence in a given language and (or) in this encoding;
- build a search engine to extract the desired byte sequences from the byte stream of the document being checked;
- save in memory the constructed search engine and profiles of languages and (or) encodings, collectively referred to as the ontology of this collection of languages and (or) encodings;
- find byte sequences in the electronic version of each document being checked using the said search engine;
- count as the profile of the checked document the statistics of occurrence of found byte sequences based on the ontology;
- compare the calculated profile of the checked document with the mentioned profiles of languages and (or) encodings to determine the relevance of the language and (or) the encoding of this checked document. 2. The method according to claim 1, in which the most characteristic byte sequences are selected for each language and (or) for each encoding. 3. The method according to claim 1, wherein said search engine is built on the basis of the Aho-Korasik algorithm for exact matching of sets.