RU2409850C1 - Address method of detecting identification features in information streams - Google Patents

Abstract

FIELD: information technology.

SUBSTANCE: to create a base of reference information features, a set of N_i reference information features is selected, symbols contained in said features and which are different from each other are selected and then used to form an alphabet of symbols, the number of symbols contained in the alphabet is calculated, the volume of the base of reference information features is calculated, for the i-th reference feature, the number mi of symbols which form it and its morphologic coefficient di are calculated, the address of the reference information feature A_i=f(d_i) is calculated using a hash function, the morphologic coefficient is cut and a secondary hash function is calculated therefrom, the obtained compact binary presentation of the information feature is recorded in the base of reference information features on the address A_i, to extract information features from each fragment of the received information stream, a group of binary characters lying between two spaces is selected and then decoded to the type of information feature, its morphologic coefficient and address are calculated, the obtained morphologic coefficient is compared with the morphologic coefficient stores on a similar address in the base of reference information features, presence or absence of identification features to be detected in each fragment of the information stream is determined.

EFFECT: reduction of the amount of memory for storing reference information features.

5 dwg

Description

The invention relates to the field of computer science and computer technology and can be used to process information flows and detect specified standard information signs in them. The method can be used in information flow control devices for monitoring information traffic.

A known method implemented in an information processing device for information retrieval (RF Patent No. 2096825, IPC ⁶ G06F 17/00, G06F 17/30, published on November 20, 1997). This method consists in preliminarily forming a base of reference information values to be detected in the information stream, storing them, storing the number of characters in the processed text fragment (TF), storing the number of characters in words (phrases), storing the number of numbers and special characters in TF, remember pre-selected combinations of characters corresponding to the structural features of TF, set the rules for selecting TF from the information stream. Next, an information stream is received, stored according to predefined rules of the next TF. Words and phrases are isolated from TF, for which they use previously memorized structural features. TFs are memorized, for which words and phrases are written in memory sequentially, similarly to the positions in the selected TFs. The memorized words and phrases are compared with the selected TF, for which they select the method of direct search from the memory of the word (phrases), determine the number and type of characters in the selected word for the presence of only numbers and (or) special characters, compare the number of characters with the reference value and store data comparisons. Remember the data on the number of repetitions of a given word in the TF (the number of identical words), remember the data on the number of matches of the symbol structure. The selected feature is compared with the reference one contained in the database of reference information features. In case of their coincidence, the sought feature is considered detected.

The disadvantages of this method are:

1) relatively low speed of information processing due to the use of sequential search algorithms;

2) significant costs of memory for storing reference information features.

The second drawback is due to the fact that with an increase in traffic intensity, the processing time of the necessary text unit (words, phrases, etc.) increases, as a result of which the total processing time of the entire array of information features increases. The increase in memory and the need to increase the computing resource lead to unjustified economic costs.

To a large extent, the first drawback eliminates the method of processing information to detect identification signs in information flows (RF Patent No. 2282889, IPC ⁶ G06F 17/30, published August 27, 2006, Bull. No. 24). This method is the closest in technical essence and is selected as a prototype.

The prototype method consists in preliminarily forming a base of reference information signs (BEIP) to be detected in the information stream, accepting the information stream, sequentially extracting and storing fragments of the received information stream, from which information signs are allocated according to established rules, and comparing them with the reference information signs from BEIP and according to the results of comparison record the presence or absence in each fragment of the information flow of identification Cove, to be detected. For the formation of BEIP, a set of N _i reference information signs is selected, the symbols contained in them and distinguished from each other are distinguished. Then, the alphabet of characters (AC) is formed from the selected characters, the number S of the characters contained in it is calculated, assigned to the jth, where j = 1, 2, ..., S, the character number n _j its position in the character alphabet and calculated for a given coefficient value filling K BEIP its volume N _k = N / K. After that, for the i-th, where i = 1, 2, ..., N, of the reference information attribute, calculate the number m _{i of} its constituent characters and its morphological coefficient d _i , and also calculate using a hash function of a given form f (d _i ) the address of the reference information attribute A _i = f (d _i ). Then, the i-th reference information attribute is stored in the BEIP at the position corresponding to its address A _i . To extract information signs from each fragment of the received information stream, a group of binary characters located between two spaces adjacent to each other is allocated in it, decode it to the type of information sign, and its morphological coefficient and address are calculated. After that, the selected and decoded information sign is compared with the reference information signs stored at this address in the BEIP.

For the i-th, where i = 1, 2, ..., N, of the reference information feature, its morphological coefficient d _{i is} calculated by the formula:

where n _j is the position number of the j-th character in the alphabet of characters;

m _i - the number of characters forming the i-th sign;

S is the number of characters AC;

j = 1, 2, ..., m _i is the position of the symbol in the i-th attribute.

As a hash function for calculating the address of the attribute A _i = f (d _i ), use a function of the form

The disadvantage of this method is the significant cost of memory for storing reference information features.

The technical result of the implementation of the proposed method is to reduce the amount of memory for storing reference information features.

, из которого осуществляют расчет вторичной хеш-функции с основанием модуля, равным N_k. Полученное компактное бинарное значение единого размера для любых по символьной длине словарных признаков записывают в БЭИП по адресу A_i=f(d_i). В дальнейшем при мониторинге информационного трафика фрагменты информационного потока декодируют к виду компактного бинарного значения признака, сравнивают с признаком в двоичной форме записи, хранящимся по адресу A_i=f(d_i) и по результатам сравнения фиксируют наличие или отсутствие в каждом фрагменте информационного потока идентификационных признаков, подлежащих выявлению.This technical result is achieved by the fact that the functional actions of the prototype method during the formation of BEIP additionally reduce the dimension of the morphological coefficient d _i presented in binary form by dividing by the binary constant R equal to the dimension of the product (N _k × 10), when this gives a new morphological coefficient of reduced dimension

, from which the secondary hash function is calculated with the base of the module equal to N _k . The obtained compact binary value of a uniform size for any dictionary characters according to the symbolic length is recorded in the BEIP at the address A _i = f (d _i ). Further, when monitoring information traffic, fragments of the information stream are decoded to the form of a compact binary value of the sign, compared with the sign in binary form stored at the address A _i = f (d _i ), and the presence or absence of identifying information in each fragment of the information stream is recorded by comparison signs to be identified.

Thanks to the new set of essential features of the claimed method, a reduction in the amount of memory required for storing the features in BEIP is achieved.

The analysis of the level of information processing technology has made it possible to establish that analogues that are characterized by a combination of features that are identical to all the features of a technical solution are not available in available sources of information, which indicates the compliance of the claimed method with the condition of patentability “novelty”.

в аналогах не встречаются. Следовательно, заявляемый способ соответствует критерию «изобретательский уровень».The distinctive features introduced in the aggregate: reduction of the morphological coefficient presented in binary form due to binary division by a binary constant R equal to the dimension of the product (N _k × 10), as well as binary calculation of the secondary hash function with the base of the module equal to N _k , in binary representation from the reduced morphological coefficient

in counterparts are not found. Therefore, the claimed method meets the criterion of "inventive step".

The claimed method is illustrated by drawings, which show:

figure 1 is a block diagram explaining the address method for detecting identification features in information flows;

figure 2 - table "alphabet characters" and their code values;

figure 3 is a summary table of dictionary features of the selection, the number of characters in that feature, addresses and morphological for dictionary features;

figure 4 is a summary table of vocabulary signs with their morphological coefficients calculated by the prototype method and the proposed method with the required memory values for storing binary sequences of these morphological coefficients;

, вторичных хеш-функций от

и их двоичное представление для БЭИП.figure 5 is a summary table of vocabulary signs, their addresses in decimal form for addressing in BEIP, abbreviated by the proposed method of morphological coefficients

secondary hash functions from

and their binary representation for BEIP.

The claimed technical solution is achieved by introducing additional functional blocks and connections between functional blocks into the prototype method. The figure 1 presents a block diagram explaining the address method for detecting identification signs in information flows. The following elements act as functional blocks of this block diagram:

1 - functional block segmentation of the information flow according to the words;

2 - functional block for calculating the morphological coefficient of the selected word;

3 - functional block for calculating the BEIP address;

4 - functional block reduction of the morphological coefficient;

5 is a functional block for calculating a compact record of a reference information feature;

6 - base of reference information features (BEIP);

7 is a functional block comparing informational features.

Functional blocks 1, 2, 3, 6, 7 fully implement the actions described in the prototype method, while the information outputs of functional block 2 are the prototype method of information inputs of functional blocks 6 and 7. To implement a new technical solution to the flowchart The prototype method additionally introduced functional blocks 4 and 5. In the proposed method, selected vocabulary attributes are received at the stage of filling the BEIP at the input of the functional unit 1, and when monitoring information traffic, an information stream. In the functional unit 1, the information flow is segmented by words. The information output of the functional block 1 is the information input of the functional block 2, where the morphological coefficient for the word selected in block 1 is calculated. The information outputs of function block 2 are the information inputs of function blocks 3 and 4. In function block 3, the BEIP address is calculated for the morphological coefficient presented from block 2. The information output of function block 3 is the information input of block 6. In function block 4, the morphological coefficient represented by functional block 2 is reduced. The information output of functional block 4 is the information input of functional block 5, where the reduced morphological coefficient is converted to a compact binary form limited volume sequences. The information output of function block 5 is the information input of block 6, into which the value of a binary sequence of a limited volume is written from function block 5 to the address calculated in function block 3. After filling in the BEIP, the system for address detection of identification signs in information streams is ready for use. The information stream through a sequence of functional transformations in functional blocks 1, 2, 4, 5 in the form of a binary sequence of limited volume gets from the functional block 5 to the input of the functional block 7. At the same time, the input of the functional block 7 from block 6 at the address calculated in the functional block 2, the reading is stored in the functional block 6 of the binary sequence of a limited volume. In the functional block 7, the comparison of both binary sequences is carried out, and the output is given the value of the presence or absence in the current fragment of the information stream of the selected dictionary features.

Consideration of the claimed method, it is advisable to carry out the example of actions implemented by the prototype method, and supplement with the necessary actions to obtain the claimed technical solution.

Let N = 100 dictionary features be selected as dictionary features for selecting informational messages. Vocabulary signs: "bank", "iron", "mask", "machine", "frame", "plane", "man", "1985 lift" - are taken from the prototype method. From the indicated N selected features, the characters contained in them and distinct from each other are distinguished and form the “alphabet of characters” (AC) with the assignment of a character serial number in the AC. We assume that the composition of all N features contains the symbols summarized in the table shown in figure 2. Each character from the AC corresponds to the code value of the position number n _j .

The composition of the AS contains a set of different characters, sufficient to compose from them any of the N pre-selected features.

Then, for a given value of the duty factor K BEIP, its volume N _k = N / K, i.e. the number of lines in the generated BEIP. By analogy with the prototype method K = 0.2, respectively, the number of lines in the database of reference information features will be equal to:

.

.

Further, for each 1st characteristic, its morphological coefficient d _{i is} calculated by the formula (1).

Further, taking into account the calculated value of the morphological coefficient, the address A _{i of} each 1st attribute is determined using the specified hash function (formula 2), i.e. determine the position of the reference characteristic in BEIP. Formula 2 can be simplified to the expression:

. Так как в морфологическом коэффициенте d_i уже содержится информация о значении адреса

, неэффективно использовать избыточную информацию для хранения в БЭИП. Сократить морфологический коэффициент d_i предлагается делением морфологического коэффициента на постоянную R, равную размерности произведения (N_k×10). При N_k=500 постоянная R представляется величиной 3-го порядка - 1000 в десятичном представлении или в двоичном представлении - 1111101000. При представлении морфологического коэффициента d_i в десятичной форме записи для получения сокращенного морфологического коэффициента

осуществляется деление d_i на постоянную R, далее дробная часть от деления отбрасывается, а оставляется только целая часть от деления. Так, для словарного признака "маска", занимающего 2-ю позицию словарного признака в таблице фигуры 3, морфологический коэффициент, рассчитываемый по формуле 1, составит в десятичной форме записи значение, равное 6961520. Для данного значения морфологического коэффициента адрес

, рассчитываемый по формуле 3, составит значение:In this case, the addressing in BEIP, unlike the addressing calculated by formula 2, will differ by a constant equal to 1, and will occupy the address area from 0 to 499. We designate the address obtained by formula 3 as

. Since the morphological coefficient d _i already contains information about the value of the address

, it is inefficient to use redundant information for storage in BEIP. It is proposed to reduce the morphological coefficient d _i by dividing the morphological coefficient by a constant R equal to the dimension of the product (N _k × 10). When N _k = 500, the constant R is represented by a third-order value - 1000 in decimal or in binary representation - 1111101000. When representing the morphological coefficient d _i in decimal notation to obtain a reduced morphological coefficient

d _i is divided by a constant R, then the fractional part of the division is discarded, and only the integer part of the division is left. So, for the dictionary attribute “mask”, which occupies the 2nd position of the dictionary attribute in the table of figure 3, the morphological coefficient calculated by formula 1 will be in decimal notation a value equal to 6961520. For a given value of the morphological coefficient, the address

calculated by the formula 3, will be the value:

.Since the address field from 0 to 499 is available in BEIP at N _k = 500, the last values of dimension N _k do not carry additional information, but are already contained in the address value

.

For 30 vocabulary signs, the calculations of addresses, morphological coefficients according to the prototype method and reduced morphological coefficients are summarized in table 3.

The calculated data of the required number of bits for storing the morphological coefficients in the BEIP of the corresponding vocabulary attributes is summarized in the table of figure 4. Thus, for the vocabulary tag “mask” in binary form, the morphological coefficient has the form 11010100011100101110000, which corresponds to 23 bits of the required memory size. For a reduced morphological coefficient, this value will be reduced to 13 bits and will be 1101100110001, which follows from the solution:

On average, reducing morphological coefficients to an abbreviated form increases the efficiency of using memory for storing reference information attributes by 10 bits for any length of the vocabulary attribute.

Given the fact that morphological coefficients have different lengths and, therefore, require different amounts of memory to store their binary values, it is necessary to bring the display of abbreviated morphological characters to a compact form. At the same time, the compact form of displaying abbreviated features should be of the same size for any dictionary characters with respect to the symbolic length. For this, it is proposed to calculate the secondary hash function of the reduced morphological coefficient by the formula:

The choice of the base of the module N _k for calculating the secondary hash function is based on the hash function calculation procedure already implemented in the prototype method by function block 2. The value of N _k also determines V _m - the necessary value of the amount of information in bits for storing a compact binary value of the attribute according to the formula:

In this case, the value of V _m is 8.965784, and when rounded to the nearest integer - 9 bits.

Binary representations of the reference information signs recorded in the BEIP at the addresses corresponding to the dictionary signs of the selection of information are presented in the table of figure 5.

The rationale for the positive effect of the proposed method is as follows. An indicator of the effectiveness of detection of identification features in information flows is the required amount of BEIP memory; therefore, a more efficient method is that requiring a smaller amount of memory for storing reference information signs in BEIP. For N vocabulary features presented in the table in figure 4, the conditional memory efficiency for storing reference information features η (M _i ) when implementing the M _i method is determined by the formula:

where N is the number of dictionary features;

- среднее количество бит, необходимых для хранения одного признака;

- the average number of bits required to store one attribute;

S _cp is the average number of characters in a dictionary feature.

The average number of characters for the dictionary features presented in the table in figure 3 is 6.4 characters per word with the number of dictionary features equal to 30.

For the prototype method, the average number of bits required to store one dictionary attribute, taking into account the data presented in the table of figure 4, will be a value equal to 29.7 bits / dictionary attribute. With this in mind, the conditional efficiency of using memory to store reference information features for the prototype method will be:

When abbreviated morphological coefficients are presented as reference information features, the conditional memory efficiency for storing reference information features will be 92.34375 bits / symbol.

When implementing the proposed method, the conditional efficiency of memory use for storing reference information features will be:

Thus, the use of memory for storing information signs of the proposed method in relation to the prototype method is 3.3 times more effective.

The practical implementation of the proposed method does not require large additional computational costs, since the calculation of a binary value modulo is a binary operation in the same way as the addition and subtraction operations are binary (Specific Mathematics. Basics of Computer Science / R. Graham, D. Knut, O. Patashnik Translated from English - 2nd ed., Rev. - M.: Mir; BINOM. Laboratory of Knowledge, 2006. - P.104).

Two additional actions introduced into the prototype method:

1) a binary division of the morphological coefficient calculated by the prototype method and presented in binary notation by the binary constant R equal to the dimension of the product (N _k × 10);

в двоичной форме записи,2) binary calculation of the secondary hash function with the base of the module equal to N _k in binary representation of the reduced morphological coefficient

in binary form,

it is possible to implement on the currently existing element base, for example, on any commercially available programmable logic integrated circuits (FPGA).

Thus, from the considered essence of the proposed method, it follows that it provides a reduction in the required amount of memory for storing reference information features. This confirms the positive effect of the implementation of the proposed method.

Claims (1)

An addressable method for detecting identification features in information flows, which consists in the fact that to form a base of reference information features to be detected in the information stream, a set of N _i reference information features is selected, the symbols contained in them and different from each other are selected, formed from them alphabet of characters, calculate the number S of the characters contained in it, assign the jth, where j = 1, 2, ..., S, the character number n _j its position in the character alphabet and calculate for a given value k the fill factor K of the base of reference information features its volume is N _k = N / K, after which for the i-th, where i = 1, 2, ..., N, of the reference feature, the number m _{i of the} symbols forming it and its morphological coefficient d _i are calculated and also using the hash function of the specified form f (d _i ), the address of the reference information sign A _i = f (d _i ) is calculated, then the i-th reference information sign is stored in the database of reference information signs at the position corresponding to its address A _i , and to highlight from each fragment of the received information the outflow of information signs, a group of binary characters is located in it between two adjacent spaces, decode it to the type of information sign, calculate its morphological coefficient and address, compare the morphological coefficient obtained during decoding with the morphological coefficient stored at a similar address in the reference database information features, characterized in that for the formation of information features additionally carry out a reduction in the morphological dimension coefficient d _i , presented in binary notation, dividing by a binary constant R equal to the dimension of the product (N _k × 10), from the morphological coefficient of reduced dimension

calculate the secondary hash function with the base of the module equal to N _k , and write the resulting compact binary value to the database of reference information signs at the address A _i = f (d _i ), when monitoring information traffic, fragments of the information stream lead to the form of a compact binary value of the attribute, compare with a sign in binary form stored at address A _i = f (d _i ) in the database of reference information signs, and the results of the comparison record the presence or absence of an identifier in each fragment of the information stream identification signs to be identified.

Images