RU77465U1 - DYNAMIC CONTENT FILTRATION SYSTEM OF ELECTRONIC DOCUMENTS - Google Patents

Abstract

The utility model relates to computer technology, in particular to a system of dynamic content filtering of electronic documents transmitted via HTTP (HyperText Transfer Protocol).

The technical result is to increase the completeness and accuracy of the content filtering of electronic documents.

The technical result is achieved by the fact that the system comprises a module for selecting reference addresses of sites in the server database, a module for identifying addresses of electronic documents in the access list, a module for identifying time cycles for selecting addresses from the access list, a module for generating signal control signals for selecting records in the server database, an access selection module to electronic documents.

Description

The utility model relates to computer technology, in particular to a system of dynamic content filtering of electronic documents transmitted via HTTP (HyperText Transfer Protocol).

The system implements a technological solution, the main task of which is to control user access to Internet resources, depending on the category of requested electronic documents and the adopted security policy. The system checks electronic documents requested by users for belonging to a specific category. In the case of a prohibited category, access to an electronic document is blocked.

An electronic document on the Internet means an information object accessible on a data transmission network, which can be obtained by means of an information network exchange of a user's terminal equipment with a server via HTTP and then reproduced using a user's terminal equipment in printed or other form.

Each electronic document on the World Wide Web has a URL (Uniform Resourse Locator) - an address that includes the type of resource and the location of the file on the server.

The general syntax is as follows:

scheme: //host.domain [: port] / path / filename.

Many electronic documents having the same root URL form a site or resource.

An electronic document in HTML format is usually called an electronic hypertext document (or simply a hypertext document).

The well-known Microsoft Internet Security and Acceleration (ISA) Server [l] system, which filters documents transmitted over high-level protocols (in particular, HTTP). Microsoft Internet Security and Acceleration (ISA) Server checks for compliance with the given conditions not only the network exchange packets (prohibits or allows the entire protocols), but also the contents of the documents.

Using this system, you can examine the contents of packages and decide on the validity of these contents. ISA Server provides the following filtering methods: by DNS name or specific URL, by keywords within electronic documents, and by file type.

To block access to unwanted electronic documents, you must specify a set of URLs as part of a firewall rule within an existing ISA Server configuration.

The main difference between the standard firewall policy rule and the filter rule is the type of destination. The destination in a firewall rule is a network entity (a single host or a group of IP addresses).

When deciding to create a filter, the administrator defines the set of URLs as the destination (addresses that the user does not have the right to access, the contents of which are undesirable) and sets a policy to prohibit all traffic.

An example of using such a rule is presented in the following table:

Field Value The rule To prohibit Protocol HTTP Sources Internal network (address range) Destination * .badsite.com

In the ruleset, special rules are assigned that block documents received via the HTTP and HTTPS protocols that contain an unwanted string or keyword. In this case, if ISA Server detects such a line in the HTTP response (within the byte range of the Web server), the page will be blocked.

The disadvantages of such a technical solution for filtering response lines are low completeness (it is impossible to judge the nature of a document by analyzing only a limited byte range) and accuracy (a decision on the nature of a document is made only by the presence of individual words).

Other technical solutions to the task [2], known as Cobian Orange Filter by Deerfield.com and EngiIPIP Content Filter by LogiSense, are designed exclusively for working with information delivered via HTTP channels and provide filtering using additional protocols.

Lists of access control are formed statically. Complete lists and the ability to correctly classify URLs are an essential attribute of an effective Web content filtering system.

Known systems that can be used to solve the problem (3, 4).

The first of the known systems contains data reception and storage units connected to control and data processing units, search and selection units connected to data storage and display units, the synchronizing inputs of which are connected to the outputs of the control unit (3).

A significant drawback of this system is the impossibility of solving the problem of updating the list of data stored in memory in the form of addresses of relevant documents at the same time as solving the problem of delivering the contents of these documents to users in real time.

Another system is known that contains data processing units, the information inputs of which are connected to the data reception and control units, and the outputs are connected to the first group of memory units, the central processor, the inputs of which are connected to the outputs of the memory units of the first group and data processing units, and the outputs are connected with the inputs of the memory blocks of the second group and data display blocks (4).

The last of the above technical solutions is closest to the described.

Its disadvantage lies in the low speed of the system, due to the fact that the data filtering procedure is implemented by searching for data across the entire database and their subsequent processing by the central processor, which inevitably leads to unreasonable waste of time.

The purpose of the invention is to increase the completeness and accuracy of the content filtering of electronic documents by dynamically updating static access lists containing the address identifiers of the requested electronic documents.

This goal is achieved by the fact that in a known system containing a module for selecting reference addresses of sites in the server database, the information and synchronization inputs of which are the first information and synchronization inputs of the system, respectively, while the first information input of the system is designed to receive requests for electronic documents from them network addresses, the first synchronizing input of the system is intended for receiving signals of entering codes of requests for electronic documents into the selection module site addresses in the server database, and the first information output of the site reference addresses selection module in the server database is intended to issue requests for electronic documents at their network addresses to the first information input of the database server, the module for identifying addresses of electronic documents in the access list, one information and the synchronizing inputs of which are the second information and synchronizing inputs of the system, respectively, while the second information input of the system is designed to receive and server database records, and the second synchronizing input of the system is designed to receive signals for entering server database records in the identification module of electronic document addresses in the access list, a signal generation module for controlling the selection of server database records, the first information input of which is connected to the second information output of the module selection of reference addresses of sites in the server database, a synchronization module for generating signals to control the selection of server database records is connected to the synchronizing output of the module for selecting the reference addresses of sites in the server database, the address output of the module for generating control signals for selecting records of the server database is the address output of the system intended for issuing the addresses of database records to the address input of the database server, and the synchronizing output of the module for generating control signals the selection of server database records is the first synchronizing system output intended for issuing control signals to the input of the first channel the database server, an identification module for the time cycles of selecting addresses from the access list was introduced, the information input of which is connected to the third information output for the module for selecting the reference addresses of sites in the server database, the synchronizing input of the module for identifying the time cycles of selecting addresses from the access list is connected to the synchronizing output the module for selecting reference addresses of sites in the server database, and the counting input of the module for identifying time cycles for selecting addresses from the access list of the connection it is connected with the clocking output of the module for identifying addresses of electronic documents in the access list, while one output of the module for identifying time cycles for selecting addresses from the access list is connected to the counting input of the module for generating signals for controlling the selection of records in the server database, and another output for the module for identifying time cycles for selecting the database from access list is the second synchronizing output of the system, designed to issue control signals to the input of the second channel of the database server interrupt, and mod For selection of access to electronic documents, one information input of which is connected to the first output of the module for selecting reference addresses of sites in the server database, another information input of the module of selection of access to electronic documents is connected to the information output of the module for identifying addresses of electronic documents in the access list, and the synchronization input the module for selecting access to electronic documents is connected to the synchronizing output of the module for identifying addresses of electronic documents in the access list, when m the information output of the selection module for access to electronic documents is the second information output of the system, intended for issuing codes of electronic addresses of documents on the Internet, the synchronizing output of the module for selecting access to electronic documents is the third synchronizing output of the system, for issuing synchronizing signals for transmitting codes of electronic addresses of documents to the Internet, and the signal output of the module for selecting access to electronic documents is a signal output the house of the system, designed to issue a signal prohibiting access to electronic documents of the network.

The invention is illustrated by drawings, where Fig. 1 is a structural diagram of a system, Fig. 2 is a structural diagram of a module for selecting reference addresses of sites in a server database, Fig. 3 is a structural diagram of a module for identifying addresses of electronic documents in an access list, in Fig. .4 is a block diagram of a module for identifying time cycles of accessing addresses from an access list, FIG. 5 is a block diagram of a module for generating control signals for fetching records of a server database, and FIG. 6 is a block diagram of an access selection module electronic documents 7 is a diagram of data flow in the system, and Figure 8 is a flowchart of the system.

The system (Fig. 1) comprises a module 1 for selecting reference addresses of sites in the server database, a module 2 for identifying addresses of electronic documents in the access list, a module 3 for identifying time cycles for selecting addresses from the access list, a module 4 for generating signals for controlling the selection of server database records, module 5 selection of access to electronic documents.

1 also shows the first 10 and second 11 information inputs of the system, the first 12 and second 13 synchronizing inputs of the system, the first 15 and second 16 information outputs of the system, address 17 system output, the first 18, second 19 and third 20 synchronizing outputs of the system, and signal 21 system output.

Module 1 (figure 2) selection of reference addresses of sites in the server database contains a register 25, a decoder 26, a memory node 27 made in the form of read-only memory, elements 28-30 And, and elements 31, 32 delay. The drawing also shows information 10 and synchronizing 12 inputs, as well as the first 36, second 37, third 38 and fourth 39 information and synchronizing 40 outputs.

Module 2 (figure 3) identifying the addresses of electronic documents in the access list contains a register 45, a comparator 46, a delay element 47. The drawing shows the first 48 and second 49 information and synchronizing 50 inputs. as well as information 51, clocking 52 and synchronizing 53 outputs.

Module 3 (Fig. 4) for identifying time cycles for accessing addresses from an access list contains a register 55, a counter 56, a comparator 57, delay elements 58, 59.

The drawing shows information 60, synchronizing 61 and counting 62 inputs, as well as the first 63 and second 64 outputs.

Module 4 (FIG. 5) of generating control signals for selecting records of the server database contains a counter 65, an OR element 66, a delay element 67. The drawing shows information 68, synchronizing 69 and counting 70 inputs, as well as address 17 and synchronizing 18 outputs.

Module 5 (Fig.6) selection of access to electronic documents contains a decoder 75, triggers 76, 77, elements 78, 79 And, a group of 80 And elements, element 91 delay. The drawing shows the first 84 and second 85 information, and synchronizing 86 inputs, as well as information 16, synchronizing 20 and signal 21 system outputs.

The system is based on the use of an automatic classifier of electronic documents to determine the subject of documents.

In this case, the classification of an electronic document means the following sequence of actions (Fig. 7):

- Transformation of the hypertext representation to a special data format: separation of the text of the natural language from the service HTML markup;

- analysis of the selected text of the natural language in order to highlight significant fragments; accounting for the weight of their occurrences in the text to build a model for representing the text of the document;

- Comparison of the constructed model for presenting the text with models of predefined classes of documents containing distinctive features and their meanings characterizing the subject of these classes;

- making a decision on the basis of a comparison on assigning a document to one or more classes, while making a decision on blocking the transmission of an HTTP request from a user to the server when the subject of the requested electronic document is defined as undesirable.

The general algorithm of the system is shown in Fig. 8 and consists in performing the following sequence of operations:

- input 10 of the system receives a request for access to the electronic document at its URL (steps 1 and 2 of the data flow diagram);

- the information on the document in the access list is checked (step 3). If the access list contains an entry about the class of the directly requested URL, then the result will be permission or deny access, if the document class is valid or undesirable, respectively (step 4). If the access list does not contain information directly about the requested URL, but the resource to which the document belongs is marked as undesirable, then access to the document is denied.

The following steps are performed only when the access list does not contain information about the document and about the resource to which the document belongs.

- the automatic classification mechanism starts: for this, the URL of the classified document is transmitted to the interface for receiving electronic documents (step 5). The interface for receiving electronic documents is provided by an external calling program or is implemented directly as a function of the automatic classification subsystem;

- the hypertext representation and URL of the document are transmitted further to directly classify the document (step 6);

- after determining the class of the document, the permission / prohibition of access in accordance with the specified class is returned through the filtering interface (step 7), and information is also placed in the access list (step 8);

- additionally, the classification mechanism of the resource to which the document belongs is activated (step 9). To carry out the resource classification procedure, a request is made for information about all known documents related to the resource (steps 10, 11). The result of the resource classification is also placed in the access list (step 12).

A resource is classified according to the following rule: a resource is undesirable if and only if the proportion of documents related to this resource and classified as undesirable exceeds the threshold value A, which is determined at the stage of system setup. In other cases, the resource is considered valid.

Thus, the system implements the following procedures:

- the documents and resources accessed by the user are placed on the access list;

- Once classified documents are not re-classified upon subsequent user requests.

The identifier of an electronic document is the following structure:

Field Type of Value siteID Integer CRC32 hash calculated from the part of the URL string defining the resource (site) address pathID Integer CRC32 hash calculated from the part of the URL string defining the path to the document relative to the address of the resource (site)

The resource identifier is similar to the identifier of the electronic document, but the second field pathID is always zero.

We illustrate this fact with the following example: let the document URL be http://www.site.com/foo/bar/doc.html.

Then siteID = CRC32 (http://www.site.com)

pathID = CRC32 (foo / bar / doc.html),

and the identifier of the electronic document is a pair (siteID, pathID), and the identifier of the corresponding resource to which the document belongs is (siteID, 0).

The data structure characterizing the thematic affiliation of the document is as follows:

Field Type of Value ID pair (siteID, pathID) Document id category transfer Permissible, undesirable, indefinite - depending on the classification result

To access the relevant electronic documents at the input 10 of the system receives the identifier of the electronic document in the form of a codogram having the following data structure:

The code The code siteID pathID

This codogram from the input 10 of the system is fed to the information input of module 1 and then to the information input of register 25, into which the indicated codes are entered by a synchronizing pulse from input 12.

From the output 33 of register 25, the entire code combination through the output 36 of module 1 is output to both the output 15 of the system and then to the information input of the admission database server, and to the input 85 of module 5.

From the output 34 of register 25 of module 1, the siteID code is fed to the information input of the decoder 26, which decrypts the incoming code and prepares the signal path from input 12, opening one of the elements 28-30 I. For definiteness, we assume that a high potential is received at one input of the element 30 I.

In parallel with this, the synchronizing pulse from the input 12 of the system is delayed by the element 31 of the module 1 for the duration of the operation of the register 25 and the decoder 26 and then polls the state of the elements 28-30 I.

Given the fact that only 30 AND element will be open on one input, then passing through this And element, the clock pulse arrives, first, at the read input of the corresponding fixed memory cell of the permanent storage device 27, where the reference address codes of the database server memory partitions are stored containing siteID site data.

The structure of the siteID site data code is as follows:

The code The code reference address of the first record of this total number of address data records in the database partition siteID siteID

The code of the reference address of the first record of this siteID is read from the memory of block 27 and through the output 37 of module 1 goes to the input 68 of module 4 and then to the information input of the counter 65, and the code of the total number of records of address data in the section of the SiteID database from the output 38 of module 1 is received to the information input 60 of module 3 and then to the information input of the register 55.

At the same time, the same read pulse from the output of element 31 is delayed by the delay element 32 for the duration of reading the contents of the fixed cell of the ROM 27 and then from the output 40 of module 1 is supplied to both the synchronizing input 61 of module 3 and the synchronizing input 69 of module 4.

The code from input 68 of module 4 by a synchronizing pulse from input 69 is entered into the counter 65, from the output of which this code is sent to the address output 17 of the system.

At the same time, the same synchronizing pulse passes through the OR element 66 and is delayed by entering the reference address code into the counter 65 by the element 67. Then this pulse is output to the system output 18 as a control signal for reading the contents of the server database memory cell to the address indicated at the output 17 systems.

The code of the total number of address data records in the siteID database section from the output 38 of module 1 is entered through the input 60 of module 3 into register 55 with the same clock pulse from the output 40 of module 1 through the input 61 of module 3. From the output of register 55 this code is sent to one comparator input 57.

Based on a signal from system output 18, the database server (not shown) goes to the subroutine for reading the contents of the database cell at the address indicated on output 17, issuing the read database record to the system’s information 11 input and entering its attributes into module 2 with a synchronizing pulse, coming from the server to the input 13 of the system.

As a result of this, from the entrance 49 to the register 45, the synchronizing pulse from the input 50 will be entered the attributes of the first record read from the database, having the following structure:

The code The code Access Tag Code - or permission to access an electronic document with a given address; siteID pathID - or deny access to an electronic document with a given address

The pathID code from the first read record from the database from the first output of the register 45 goes to one input of the comparator 46, the other information input of which from the output 39 of module 1 receives the pathID code of the requested electronic document.

The synchronizing pulse from the input 13 of the system is delayed by the delay element 47 for the duration of recording the database record in the register 45 of module 2 and is supplied to the synchronizing input of the comparator 46.

The comparator 46 compares the input codes by the synchronization signal, and if the pathID attributes being compared do not coincide, then the output 52 of module 2 generates a signal that goes to the input 62 of module 3 and then to the counting input of the counter 56, fixing the number of records read from the database server.

At this point in time, the counter 56 will record the fact of reading the first record from the server database. The output of the counter 56 is connected to one input of the comparator 57, comparing the total number of records of this site in the server database stored in the register 55 with the number of records in the counter 56.

The comparator 57 of module 3 compares the readings of the register 55 and the counter 56 by the clock pulse supplied to the clock input of the comparator 67 from the output of the delay element 58.

Since the first unit is fixed in the counter 56, its readings will be less than the readings of the register 55 and a pulse appears at the output of the module 3 63, which through the input 70 of the module 4 will go to the counting input of the counter 65, increasing the base address of the read cell by one.

In addition, the same pulse passes through the OR element 66 to the input of the delay element 67, where it is delayed by the end time of the counter 65, and then again issued through the system output 18 as a control signal for reading the next database record at the address generated at the output 17 system.

Based on a signal from system output 18, the database server again switches to the subroutine for reading the contents of the database cell at the address specified on output 17, issuing the read database record to the system information input 11 and entering its attributes into register 45 of module 2 with a synchronizing pulse from the server at the entrance 13.

This process of reading server database records will continue until the readings of counter 56 and register 55 are equal, which will indicate that there is no requested electronic document address in the server database. In this case, the pulse is generated at the output 64 of the module 3 and through the output 19 of the system is fed to the input of the second channel of the database server interrupt.

With the arrival of this signal, the server switches to a subroutine for automatic classification of the requested identifier of the electronic document, in accordance with which the data of the identifier of the electronic document from the output 15 of the system goes to the database server, where the following sequence of operations is performed:

- Transformation of the hypertext representation to a special data format: separation of the text of the natural language from the service HTML markup;

- analysis of the selected text of the natural language in order to highlight significant fragments; accounting for the weight of their occurrences in the text to build a model for representing the text of the document;

- Comparison of the constructed model for presenting the text with models of predefined classes of documents containing distinctive features and their meanings characterizing the subject of these classes;

- making a decision on the basis of a comparison about assigning a document to one or more classes, while making a decision to block the transmission of an HTTP request from a user to the server when the subject of the requested electronic document is defined as undesirable, as shown in Fig. 8.

If the compared pathID attributes at the inputs of the comparator 46 of module 2 coincide, then the output 53 of module 2 generates a signal fed to the input 86 of module 5 and then to one of the inputs of elements 78, 79 I, controlled by a decoder 75, to the input 84 of which output 51 of module 2 receives a code sign of access to this electronic document.

The decoder 75 decodes the code of the access flag, and if access to this electronic document is allowed, then the decoder 75 opens the And element 78 through the second input, through which the pulse from the input 86 passes and acts as a direct input to the trigger 76, setting it to a single state, so and on the reverse input of the trigger 77, confirming its initial state.

Turning to a single state, a flip-flop 76 with a direct output with high potential opens elements 80 And groups one input, to the other input 85 of which the output of module 36 receives the identifier of an electronic document, which through elements 80 And groups is output to system output 16.

In addition, the synchronizing pulse from the output of the And element 78 is delayed by the element 81 for the duration of the trigger 76 and the connection of the And elements of the group 80 and from the system output 20 is issued as a synchronization signal issuing an electronic identifier of an electronic document.

If, at the input of the decoder 75, a sign indicating the prohibition of access to the address of this electronic document is recorded, then element 79 And will be open at the second input, through which the synchronizing pulse from input 86 will set trigger 77 to a single state, and trigger 76 will be set to its original state at which the low potential from the direct output of the trigger 76 elements 80 And groups will be closed, thereby blocking access to the electronic document.

High potential from the direct output of the trigger 77 is issued to the output 21 of the system as a signal to deny access to this electronic document. Thus, the system blocks electronic documents that either belong to unwanted resources or are themselves classified as unwanted.

The use of the automatic classification method of electronic documents, with the help of which the analysis of the hypertext representation of the document is carried out, distinguishing features are distinguished, their values are compared and a decision is made whether the document belongs to the category of undesirable or permissible is a distinctive feature of the proposed dynamic content filtering system.

Thus, the introduction of new modules and new constructive connections made it possible to significantly increase the system performance by localizing the addresses of the records of the access list of the server database by the identifiers of electronic documents.

Sources of information taken into account when drawing up the description of the application:

1. Tumbes D. Web Content Filtering Using ISA Server / Open Systems: [Electronic Document] / (http://www.osp.ru/win2000/2006/08/3829260/).

2. Hill B. Web Content Filtering Solutions / Open Systems: [Electronic Document] / (http://www.osp.ru/win2000/2004/05/177073/).

3. US Patent No. 5136708 M. cl. G06F 15/16, 1992.

4. US Patent No. 5129083 M. cl. G06F 12/00, 15/40, 1992 (prototype).

Claims (1)

A system of dynamic content filtering of electronic documents, containing a module for selecting the reference addresses of sites in the server database, the information and synchronizing inputs of which are the first information and synchronizing inputs of the system, respectively, while the first information input of the system is designed to receive requests for electronic documents at their network addresses, the first the synchronizing input of the system is designed to receive signals of entering codes of requests for electronic documents into the selection module about Ornate addresses of sites in the server database, and the first information output of the module for selecting reference addresses of sites in the server database is designed to issue requests for electronic documents at their network addresses to the first information input of the database server, the module for identifying addresses of electronic documents in the access list, one information and the synchronizing inputs of which are the second information and synchronizing inputs of the system, respectively, while the second information input of the system is intended for volume of server database records, and the second synchronizing input of the system is designed to receive signals for entering server database records in the identification module of electronic document addresses in the access list, a signal generation module for generating server database records, the first information input of which is connected to the second information output of the module selection of reference addresses of sites in the server database, a synchronization module for generating signals to control the selection of records of the database server subkey n to the synchronizing output of the module for selecting the reference addresses of sites in the server database, the address output of the module for generating signals for controlling the selection of server database records is the address output of the system designed to provide addresses of database records to the address input of the database server, and the synchronizing output of the module for generating signals control the selection of server database records is the first synchronizing system output designed to issue control signals to the input of the first channel database server interruption, characterized in that the system comprises a module for identifying time cycles for selecting addresses from an access list, the information input of which is connected to a third information output for the output of a module for selecting reference addresses of sites in the server database, synchronizing the input of a module for identifying time cycles for selecting addresses from a list access is connected to the synchronizing output of the module for selecting the reference addresses of sites in the server database, and the counting input of the module for identifying time cycles of sampling addresses from the access list is connected to the clocking output of the electronic document address identification module in the access list, while one output of the identification module for temporary cycles of address selection from the access list is connected to the counting input of the signal generation module for generating control records for server database records, and another output of the temporary identification module cycles of selecting addresses from the access list is the second synchronizing output of the system, designed to issue control signals to the input of the second channel a database server, and a selection module for access to electronic documents, one information input of which is connected to the first output of the selection module of reference addresses of sites in the server database, another information input of the selection module of access to electronic documents is connected to the information output of the identification module of electronic documents addresses in access list, and the synchronizing input of the module for selecting access to electronic documents is connected to the synchronizing output of the module for identifying addresses of electronic documents access list, the information output of the module for selecting access to electronic documents is the second information output of the system, intended for issuing codes of electronic addresses of documents on the Internet, the synchronizing output of the module for selecting access to electronic documents is the third synchronizing output of the system, for issuing synchronizing signals for transmitting codes of electronic addresses of documents to the Internet, and the signal output of the module for selecting access to electronic dock omen is the system's signal output intended for issuing a signal to prohibit access to electronic network documents.