RU2473964C1 - Method of detecting identification features for different letter-symbol writing systems - Google Patents

Abstract

FIELD: information technology.

SUBSTANCE: unique three-byte reference information feature base address (RIFB) is calculated and the logic one value is recorded at said address. Representation of the reference information feature in form of a three-byte address enables, if necessary, to augment the RIFB without recalculating values of addresses and morphological coefficients of information features. Three-byte addresses for the information features do not depend on the symbol length of vocabulary features.

EFFECT: broader functional capabilities when using the method to detect identification features presented in different letter-symbol writing systems, high efficiency of using memory to store reference information features.

2 dwg, 7 tbl

Description

The invention relates to the field of computer science and computer technology and can be used to process information flows and detect specified standard information features in them, presented in various alphanumeric writing systems. The method can be used in information flow control devices for monitoring information traffic.

The known method implemented in the information processing device for information retrieval (RF patent No. 2096825, IPC ⁶ G06F 17/00, G06F 17/30, published on 11/20/1997). This method consists in preliminarily forming a base of reference information values to be detected in the information stream, storing them, storing the number of characters in the processed text fragment (TF), storing the number of characters in words (phrases), storing the number of numbers and special characters in TF, remember pre-selected combinations of characters corresponding to the structural features of TF, set the rules for selecting TF from the information stream. Next, an information stream is received, stored according to predefined rules of the next TF. Words and phrases are isolated from TF, for which they use previously memorized structural features. TFs are memorized, for which words and phrases are written in memory sequentially, similarly to the positions in the selected TFs. The memorized words and phrases are compared with the selected TF, for which they select the method of direct search from the memory of the word (phrases), determine the number and type of characters in the selected word for the presence of only numbers and (or) special characters, compare the number of characters with the reference value and store data comparisons. Remember the data on the number of repetitions of a given word in the TF (the number of identical words), remember the data on the number of matches of the symbol structure. The selected feature is compared with the reference one contained in the database of reference information features. In case of their coincidence, the sought feature is considered detected.

The disadvantages of this method are:

1) relatively low speed of information processing due to the use of sequential search algorithms;

2) significant costs of memory for storing reference information features.

The second drawback is due to the fact that with an increase in traffic intensity, the processing time of the necessary text unit (words, phrases, etc.) increases, resulting in an increase in the total processing time of the entire array of information attributes. The increase in memory and the need to increase the computing resource leads to unjustified economic costs.

To a large extent, the first drawback eliminates the method of processing information to detect identification signs in information flows (RF patent No. 2282889, IPC ⁶ G06F 17/30, published on 08.27.2006. Bull. No. 24). This method consists in preliminarily forming a base of reference information signs (BEIP) to be detected in the information stream, accepting the information stream, sequentially extracting and storing fragments of the received information stream, from which information signs are allocated according to established rules, and comparing them with the reference information signs from BEIP and according to the results of comparison fix the presence or absence of identification signs in each fragment of the information flow in the subject detection. For the formation of BEIP, a set of N _i reference information signs is selected, the symbols contained in them and distinguished from each other are distinguished. Then, the alphabet of characters (AC) is formed from the selected characters, the number S of the characters contained in it is calculated, assigned to the jth, where j = 1,2, ..., S, the character number n _j its position in the character alphabet and calculated for a given coefficient value filling K BEIP its volume N _k = N / K. After that, for the i-th, where i = 1, 2, ..., N, of the reference information attribute, calculate the number m _{i of} characters that form this attribute, and its morphological coefficient d _i , and also calculate using a hash function of a given form f ( d _i ) the address of the reference information attribute A _i = f (d _i ). Then, the i-th reference information attribute is stored in the BEIP at the position corresponding to its address A _i . To extract information signs from each fragment of the received information stream, a group of binary characters located between two spaces adjacent to each other is allocated in it, decode it to the type of information sign, and its morphological coefficient and address are calculated. After that, the selected and decoded information attribute is compared with the reference information attribute located at this address in BEIP.

For the i-th reference information feature, the morphological coefficient d _{i is} calculated by the formula:

where n _j is the position number of the j-th character in the alphabet of characters;

m _i - the number of characters forming the i-th sign;

S is the number of characters AC;

j = 1,2, ..., m _i is the position of the symbol in the i-th attribute.

As a hash function for calculating the address of the attribute A _i = f (d _i ) use a function of the form:

This method is the closest in technical essence and is selected as a prototype.

The disadvantage of this prototype method is its limited use, expressed in the possibility of detecting information signs represented from characters in one of the registers (only ascribed or lowercase characters) in one alphanumeric writing system. In addition, if it is necessary to add new information signs to the BEIP containing new additional characters that are not included in the existing AS, it is necessary to completely recalculate all the values of the morphological coefficients d _i , as well as a new calculation of the addresses A _i corresponding to each information attribute in the BEIP, when using hash functions f (d _i ).

The technical result of the implementation of the proposed method is to expand the functionality in the application of the method for detecting identification features expressed in various alphanumeric writing systems, as well as more efficient use of memory to store reference information features.

This result is achieved by the fact that the procedure for determining the morphological coefficient d _{i is} excluded from the prototype method, instead of it, the memory address A _i is determined by which the value of the logical unit level for the information attribute to be detected in the information stream is recorded. The remaining addresses of the allocated memory are recorded values of the level of logical zero. The memory address of the desired identification feature is a three-byte sequence of the form:

- 1-й байт адреса памяти,

- 2-й байт адреса памяти,

- 3-й байт адреса памяти.Where

- 1st byte of the memory address,

- 2nd byte of the memory address,

- 3rd byte of the memory address.

. Значение 2-го байта адреса

рассчитывают выражением:The value of the 1st byte of the address is brought into correspondence with the binary numeric value of the code sign of the 1st character of the word

. The value of the 2nd byte of the address

calculated by the expression:

where bin is a binary representation function of the value of the number represented in the decimal system;

m _i is the number of characters in the i-th identification attribute;

n _j is the code of the jth character in the alphabet of characters;

3 - the minimum odd prime number;

mod - base module function;

j = 1,2, ..., m _i is the position of the symbol in the i-th identification feature.

рассчитывают выражением:The value of the 3rd byte of the address

calculated by the expression:

The binary sequence A _i of three bytes determines the address of the allocated memory, in which for the standard information features necessary for selection, the values of the logical unit level are written to the memory at their calculated addresses, for the remaining information features that are not involved in the calculation of addresses, the values of the logical level zero. When monitoring the information flow, for each word extracted from the information flow, a three-byte address is determined, and if there is a logical unit at the given memory address, they decide on the presence of a reference information attribute in the information flow. When adding a new reference information feature, its three-byte address is determined, the logical unit level value is recorded at this memory address, and identification features are further detected in the information flows.

Thanks to the new set of essential features of the claimed method, more efficient use of memory for BEIP is achieved, and the need for additional recalculations of the content part of BEIP is eliminated when a new information sign appears in another alphanumeric writing system.

The analysis of the level of information processing technology has made it possible to establish that analogues that are characterized by a combination of features that are identical to all the features of a technical solution are not available in available sources of information, which indicates the compliance of the claimed method with the condition of patentability “novelty”.

The distinctive features introduced together: the presentation of information features of various alphanumeric writing systems in the form of unique three-byte BEIP addresses, as well as the placement of minimum lengths (1 bit of information: 1 or 0) characterizing the presence or absence of a given address at these addresses a reference sign, in analogues is not found. Therefore, the claimed method meets the criterion of "inventive step".

The claimed method is illustrated by drawings, which show:

figure 1 is a flowchart explaining a method for detecting identification features for various alphanumeric writing systems;

figure 2 is a graph of the dependence of the growth in the number of addresses on the number of reference information features according to the prototype method;

figure 3 is a summary table of dictionary features of the selection belonging to the Russian language, and three-byte addresses corresponding to them, presented in the decimal system of calculus;

figure 4 is a summary table of vocabulary signs of selection belonging to the English language, and three-byte addresses corresponding to them, presented in the decimal system of calculus;

figure 5 is a summary table of dictionary features of the selection belonging to the German language, and three-byte addresses corresponding to them, presented in the decimal system of calculus;

figure 6 is a summary table of vocabulary signs of selection belonging to the Italian language, and three-byte addresses corresponding to them, presented in the decimal system of calculus;

Fig.7 is a summary table of vocabulary signs of selection belonging to the Spanish language, and three-byte addresses corresponding to them, presented in the decimal system of calculus;

on Fig - summary table of vocabulary signs of selection belonging to the Lithuanian language, and three-byte addresses corresponding to them, presented in the decimal system of calculus;

figure 9 is a summary table of vocabulary signs of selection with the first character "i" of the languages of the Latin alphanumeric writing system and three-byte addresses corresponding to them, presented in the decimal system of calculus.

The claimed technical solution is achieved by eliminating from the prototype method a functional block for calculating the morphological coefficient of the selected word and unnecessary connections between 2 and other (1, 3, 4, 5) functional blocks. The figure 1 presents a flowchart explaining a method for detecting identification features for various alphanumeric writing systems. The following elements act as functional blocks of the block diagram of the prototype and the proposed method:

1 - functional block segmentation of the information flow according to the words;

2 - functional block for calculating the morphological coefficient of the selected word;

3 - a functional unit for calculating the address of the base of reference information features (BEIP);

4 - a base of reference information features;

5 - functional block comparison.

Functional block 2 is excluded from the method, so instead of the morphological coefficient d _i calculated by expression 1, the value of the logical unit level, which is a marker of the reference sign in the information stream, is entered into the BEIP memory at the address corresponding to a particular information attribute. From the proposed method also disappear information links between blocks 1, 3, 4, 5 and 2, represented in the figure by dashed lines.

Functional block 1 fully implements the actions described in the prototype method. In the functional unit 1, the information flow is segmented by words. At the input of the functional block 1, selected vocabulary signs are received at the BEIP filling stage, and during the control of the information schedule, the information flow. At the stage of BEIP formation according to the proposed method, only blocks 1, 3 and 4 function, and when monitoring information traffic - blocks 1, 3, 4 and 5. The information output of function block 1 is the information input of function block 3. To implement a new technical solution in the block 3, a function of calculating the BEIP address is different from the prototype method. Instead of an address sufficient to display N _k lines of BEIP according to the prototype method, for any words of length represented in various alphanumeric writing systems, in block 3 a three-byte address of the BIEC for the i-th reference information attribute is determined in accordance with expressions 3, 4 and 5. Three information outputs by the number of byte addresses are information inputs to block 4. In contrast to the prototype method, the formation of BEIP in the functional block 4 is carried out differently. Instead of recording the morphological coefficient at the calculated address A _i , the value of the level of the logical unit is recorded. The information output of function block 4 is the information input of block 5. Function block 5 carries out the comparison procedure only when monitoring information traffic. In function block 5, a comparison is made of the level value at the calculated BEIP address with the value of the logical unit level; if these values coincide, the output gives information about the presence of the necessary vocabulary sign in the current fragment of the information stream.

Consideration of the claimed method, it is advisable to carry out the example of actions implemented by the prototype method, and supplement with the necessary actions to obtain the claimed technical solution.

Let 39 identification features in 6 natural languages be selected as dictionary features for selecting informational messages N = 234 vocabulary signs, for 2 different letter-letter writing systems (Latin and Cyrillic spelling). As AS we use the extended ASCII code table, where the decimal place corresponds to the decimal value 32 or the binary value 00100000; lowercase letter "a" of the Latin writing system (English, German, Spanish, Italian and Lithuanian) corresponds to the decimal value 97 or the binary value 01100001; the lowercase letter “I” of the Cyrillic writing system corresponds to the decimal value 255 or the binary value 11111111. The composition of the AS contains a combination of different characters, sufficient to make any of N informational signs from them.

According to the prototype method, the number of lines of the base of reference features is defined as N _k = N / K, where K = 0.2, respectively, the minimum number of lines in the base of reference information features will be equal to:

The amount of memory V _BEIP required to store the reference information features in BEIP, the prototype method is calculated as:

where N _k is the number of lines in BEIP; d _max - the minimum number of bits required to store the morphological coefficient of the maximum bit length. So, for the ASCII vocabulary attribute “information”, the decimal notation of the codes that make up the word in lowercase is 232 237 244 238 240 236 224 246 232 255. The morphological coefficient calculated by the prototype method for the word “information” has the form 963701989565045000000000 in the decimal system, which corresponds in binary terms to the value 110001110111000100111010111000000101110000100100101001000000000 and is 63 bits of the required memory. Given the fact that the morphological coefficients of the prototype method have different lengths, and therefore require different amounts of memory to store their binary values, it is necessary to determine the information sign with the maximum value of the morphological coefficient. To record the morphological coefficient of a Russian word, not exceeding 12 characters in its structure and ending in "I", in the limit of 64 bits or 8 bytes of the required amount of memory is required. In addition, to display the addresses of 1170 BEIP lines, at least 12 bits are required for an address per line. Thus, the effective memory utilization factor η (M ₁ ) by the prototype method will be 9.5 bytes / attribute. The change in the number of information signs both in the direction of increase and decrease does not allow the use of BEIP calculations obtained earlier, because each BEIP address directly depends on a changing value of N _k , which in turn depends on the number of information signs N. A change in the number of information signs N in the direction of increase is proportional to the increase in lines by 5 times, because N _k = N / K, where K = 0.2 (according to the prototype method). The dependence of the growth in the number of lines N _k on the number of signs N is shown in Figure 2. With an increase in information signs from 1 to 10000, the address field should proportionally increase from 3 bits to 2 bytes per address.

In the proposed method, the address is fixed by 3 bytes and does not depend on the current number of information signs. For 39 identification features in 6 natural languages, i.e. N = 234 vocabulary signs, in two systems of alphanumeric writing systems (Latin and Cyrillic) the values of three-byte addresses and the corresponding information signs in Russian, English, German, Spanish, Italian and Lithuanian are presented in the tables of figures 3, 4, 5, 6, 7 and 8, respectively.

So, for the dictionary sign “information”, the address in the decimal system as a result of determining the code of the 1st character of the word and calculations by expressions 4 and 5 has the form: 232 13 159, which corresponds in binary terms to the value 11101000 00001101 10011111. For the Latin alphanumeric the sign system of writing for English, German, Spanish, Italian and Lithuanian, information signs are selected in one table of figure 9, starting with the symbol "i" that is common for different languages, corresponding to the value 105 in decimal in the ASCII code table th system of calculation, with the exception of German noun "Information", the initial symbol which begins with a capital sign "i", which corresponds to the value of 73 ASCII code table. The data presented in figure 9 show that the BEIP addresses are unique for words that have different symbolic spelling, although the same meaning in Russian. An exception is the word "importante" of the Spanish and Italian languages, which has the same spelling of characters in the Latin writing system, which does not affect the technical solution of the claimed method.

When a new sign necessary for highlighting in the information flow appears, no adjustment and recalculation of old addresses is required, but only a new address of the information sign is calculated and 1 bit of information corresponding to the level of a logical unit is written to this address. The effective memory utilization factor η (M ₂ ) according to the proposed method for any information attribute of various alphanumeric writing systems and any length (number of characters in the information attribute) is 3.125 bytes / attribute, because 3 bytes make up the address of the memory field, and the content at this address is represented by one bit of information, logical level 1 or 0.

By comparing the memory utilization coefficients according to the prototype method η (M ₁ ) and the proposed method η (M ₂ ), it is possible to note an increase in memory efficiency by 3.04 times.

The reduction in the prototype method of the function of calculating the morphological coefficient when changing the functions of determining the address of the reference information feature and filling the memory in BEIP can be implemented on the currently existing element base, for example, on any commercially available programmable logic integrated circuits (FPGAs).

Thus, from the considered essence of the proposed method, it follows that it ensures the detection in the information flows of the specified reference information features presented in various alphanumeric writing systems. This confirms the positive effect of the technical solution of the proposed method.

Claims (1)

A method for detecting identification features for various alphanumeric writing systems, which consists in the fact that to form a base of reference information features to be detected in the information stream, a combination of N _i , reference information features is selected, an alphabet of characters is formed, assigned to the jth, where j = 1, 2, ..., S, symbol alphabet n _j the value of its position in the symbol alphabet, for each i-th, where i = 1, 2, ..., N, the reference feature is calculated numerical value, it is appropriate, and counting using a hash function of a given type, the address of the reference information sign, then the i-th reference information sign is stored in the base of the reference information signs at the position corresponding to its address, and a group of binary characters is extracted from each fragment of the received information stream of information signs, located between two spaces adjacent to each other, decode it to the type of information sign, calculate the numerical value of the sign and address, compare the result the numerical value of the information sign with the numerical value of the reference information sign stored at a similar address in the database of reference information signs, characterized in that the alphabet of characters includes all possible characters of various alphanumeric writing systems and corresponds to one of the standard character encodings, i- The first identification character is represented in a single encoding for various alphanumeric writing systems in the form of a word from the consecutive numbers corresponding to the encoding word values of characters n _j , words convert from the first to the last character in a sequence of three bytes

where

- first byte, corresponds to the binary numeric value of the code sign of the first character of the word

,

- second byte, determined by the expression

, where bin is the binary representation function of the number in the decimal system, m _i is the number of characters in the i-th identification feature, 3 is the minimum odd prime number, mod is the base function of the module,

- the third byte is determined by the expression

, the binary sequence A _i , of three bytes determines the address of the allocated memory, in which for the standard information features required for selection, the values of the logical unit level are written to the memory at their calculated addresses, for the remaining information features not participating in the calculation of addresses, the values logical zero level, when monitoring the information flow for each word extracted from the information flow, a three-byte address is determined and if there is a level value logical at this memory address of the ith unit, they decide on the presence of a reference information feature in the information stream, when a new reference information feature is added, its three-byte address is determined, the logical unit level value is recorded at this memory address and further identification characteristics are detected in the information flows.

Images