EA002016B1 - A method of searching for fragments with similar text and/or semantic contents in electronic documents stored on a data storage devices - Google Patents

Abstract

1. A method of searching for fragments with similar text and/or semantic content in electronic documents stored on data storage devices comprising: indexing each document which is saved in the archive, dividing said documents into fragments and forming subject groups containing one and more fragments, defining search parameters, conducting a search, ranking the list of document fragments received as a result of the search, wherein a set of unique information blocks appeared in the selected document fragment is defined as search parameters and said set is extended by preprocessing each of said unique information blocks, where a unique information block is an information block that appeared in the selected document fragment one or more times, where the preprocessing is an operation of obtaining from at least one unique information block one or more information blocks connected with said unique information block by a specified relation. 2. The method of claim 1, wherein an information block is a character string appeared in a document and limited by certain characters. 3. The method of claim 1, wherein a document fragment is any selected information block sequence in a document. 4. The method of claim 3, wherein a document fragment is a complete copy of a document. 5. The method of claim 3, wherein rules of dividing documents into fragments are specified. 6. The method of claim 5, wherein a document is divided into fragments using at least one rule. 7. The method of claim 5, wherein documents are automatically divided into fragments. 8. The method of claim 1, wherein a set of document fragments which are being searched for fragments with similar text and/or semantic content to the selected one is limited by indicating at least one rule which was used to divide documents into fragments and/or indicating a subject group. 9. The method of claim 1, wherein users form a subject group. 10. The method of claim 1, wherein a program module for processing documents is used to divide documents and their fragments into a set of information blocks. 11. The method of claim 1, wherein at least one operation of preprocessing is selected and users specify the order of preprocessing operations. 12. The method of claim 11, wherein the preprocessing consists of one logical operations of identity. 13. The method of claim 1, wherein after performing the operation of preprocessing, at least one initial block is removed or left in the query. 14. The method of claim 1, wherein all unique information blocks of the selected document fragment are included in a set of unique information blocks defined as search parameters. 15. The method of claim 1, a set of unique information blocks, which is defined as search parameters, is formed one or more functions. 16. The method of claim 15, a function of making a list of unique information blocks is used for the selected fragment. 17. The method of claim 15, wherein it is used a function of determining the number of occurrences of information blocks obtained after preprocessing unique information blocks in the text content of fragments of the selected subject group. 18. The method of claim 15, wherein it is used a function of determining the frequency of occurrences of unique information blocks in the text content of fragments of the selected subject group, where the frequency of occurrences is expressed in percents from the number of occurrences of the most frequently appeared unique information block in the text content of the selected subject group. 19. The method of claim 1, wherein a set of unique information blocks which is defined as search parameters includes a specified number of unique information blocks from the set which has been created according to user-defined rules. 20. The method of claim 1, wherein the search for document fragments is carried out on local and/or remote data storage devices. 21. The method of claim 20, wherein a remote device is any information resource or information retrieval system that operates in a computer network and provides in response to a search query a list of document fragments relevant to search parameters. 22. The method of claim 1, wherein the list of document fragments received as a result of search is ranked according to the number of information block groups appeared in found document fragments which combine information blocks obtained after preprocessing the unique information block. 23. The method of claim 22, wherein while ranking, in addition for document fragments with the coincident number of information block groups the number of occurrences of said information block groups in the text content of fragments of the selected subject group is determined. 24. The method of claim 1, wherein the fragments from the list of document fragments received as a result of search are displayed with highlighting differences between their text content and the text content of the selected fragment. 25. The method of claim 0, wherein a document which is being added to the archive is saved as a new version of previously saved document which has a specified similarity measure by text and/or semantic content with the document which is being added to the archive, and said new version is saved as a complete copy of the document which is being added to the archive or in the form of differences between the text content of the document which is added to the archive and the text content of said previously saved document. 26. The method of claim 0, wherein fragments which are saved on a data storage device are automatically classified. 27. The method of searching for electronic documents stored on data storage devices comprising indexing each document which is saved in the archive, defining parameters of searching for electronic documents which include forming the initial query containing two or more information blocks, specifying the maximal number of information blocks which may appear between said two or more information blocks in the desired document (the interval) and the order of alternation of said two or more information blocks in the desired document in the specified interval ranking the list of documents received as a result of search, said method further comprises: extending the initial query containing two or more information blocks by preprocessing one or more information blocks from the initial query, where the preprocessing of the initial query is an operation of obtaining from at least one unique information block one or more information blocks connected with said unique information block by a specified relation, searching for documents using any specified number of information blocks defined as search parameters while forming the initial query. 28. The method of claim 27, wherein an information block is a character string appeared in a document and limited by certain characters. 29. The method of claim 27, wherein a program module for processing documents is used to divide documents and their fragments into a set of information blocks. 30. The method of claim 27, wherein at least one operation of initial query preprocessing is selected and users specify the order of initial query preprocessing operations. 31. The method of claim 30, wherein the preprocessing of the initial query consists of one logical operations of identity. 32. The method of claim 27, wherein after performing the operation of preprocessing, at least one information block is removed or left in the query 33. The method of claim 27, wherein the search for documents is carried out on local and/or remote data storage devices. 34. The method of claim 33, wherein a remote device is any information resource or information retrieval system that operates in a computer network and provides in response to a search query a list of document fragments relevant to search parameters. 35. The method of claim 27, wherein the list of document fragments received as a result of search is ranked in accordance with the number of occurrences of relevant information block sequences in documents, the length of the interval in which said sequences fall and weighting coefficient assigned to information blocks. 36. The method of claim 35, wherein documents are displayed with highlighting the information block sequences which correspond to search parameters and the preprocessed query, and providing the navigation through the text content of documents using information block sequences within a whole list of documents received after search.

Description

The present invention relates to methods for retrieving information stored on local and remote data storage devices. In particular, the invention relates to methods for searching electronic devices and their fragments in data storage devices similar in text and / or semantic content to the selected fragment. The invention also relates to methods for searching documents for queries consisting of two or more words, extended semantic analogues and taking into account the interval between words and the order of their alternation in the query.

Description of the Related Art

Full-text index search methods for electronic documents stored on data storage devices are quite well known and are used in various information retrieval systems that operate both on local data storage devices and in distributed computer systems, including the Internet. Such systems allow you to search for documents of interest to the user stored in data archives, in accordance with the specified parameters.

One of the directions in the development of full-text search technologies is the development of systems allowing phrasal search of documents. Such systems provide for the formation of an initial query consisting of two or more words, and the determination of search parameters, including an indication of the interval between words and the sequence of their alternation in the desired documents. Moreover, existing systems allow you to search for documents containing in the specified interval any number of words from the original query. The list obtained as a result of a search operation includes documents that are relevant to the original query only by the formal attribute of the similarity of the vocabulary of the original and phrases included in the mentioned documents. Moreover, the most high-quality systems provide for expanding the query with morphological word forms, as well as synonyms from a dictionary predefined by the developers. Expanding queries with other semantic analogs, for example, through semantic thesauruses, as well as using several handlers when expanding a query at the same time, including user-created ones, is not provided. This negatively affects the performance of search operations.

The inclusion in the list of those documents that contain phrases similar to those specified not only in textual content, but also in semantic (the use of semantic thesauruses) is not provided, which reduces the effectiveness of search operations.

Existing systems designed for full-text phrase search do not solve all the problems associated with improving the efficiency of search operations. This is primarily due to the increase in the amount of information that any user working on a computer has to process. Unfortunately, information falling into an archive located on a data storage device from various sources often duplicates previously stored data. At the same time, lists of documents generated as a result of the usual full-text search may contain duplicates, for which the user is forced to spend time on reprocessing. Such a problem is especially relevant in the case when in a multi-user mode of operation several users simultaneously fill the archive with documents of the same subject.

Therefore, recently, systems designed to search for documents similar in text and / or semantic content have become very popular. Their popularity is due to the fact that they are designed to save the user from a number of problems, the main of which is associated with significant time costs for processing redundant (duplicate) information. Such systems allow you to optimize the process of filling the archive with documents by screening takes. In addition, the automatic classification of documents stored on a data storage device helps streamline the stored information archive.

Moreover, existing systems designed to search for documents similar in text and / or semantic content are not without drawbacks. The main one is that they do not allow you to use the entire set of words found in the text of the selected document to search. This is due to the fact that the development of a search query, consisting of a large number of words, requires considerable time and is therefore unacceptable to the user. The problem is further exacerbated if the original query is expanded with semantic analogues, for example, by using morphological analyzers or thesauruses. Therefore, existing systems designed to search for documents similar in textual content allow only a certain number of significant (key) words that are part of the document to be searched for to be used in the request. This approach to implementation significantly reduces the efficiency of performing search operations for documents similar in text content and does not allow to achieve the desired result.

Characteristics of analogues

As an analogue for the method for searching documents similar to textual and / or semantic content and their fragments, a method and device for searching for text using document signatures were selected [Patent I86029167 international class С06Р 017/00]. The method is designed to search for similar and identical fragments of documents stored in the database.

The method allows you to encode fragments of the texts of documents using a sequence of markers. At the same time, a marker signature is assigned to each fragment. The encoded fragment is compared with the same encoded fragments stored in the database. The comparison is carried out according to the sequences of markers (signatures) inherent in the fragments. In the event that fragments similar to the selected one (with identical signatures) are found in the database, documents containing fragments similar to the selected one are extracted from the database. After that, the selected fragment is compared with the documents found in the database by searching through successive lines of characters, or each word from the source fragment is compared with each word from the found documents.

The disadvantage of I86029167 invention is that it is intended to search for documents and their fragments only by formal signs of matching words included in their text content. Moreover, the invention I86029167 does not provide for the expansion of search queries by any analogues (morphological word forms, synonyms, etc.). Those. I86029167 searches for documents that are similar only in textual content, and does not allow you to find documents that are similar to the one selected in semantic content.

A further disadvantage of the invention I86029167 is that it lacks the possibility of choosing a methodology for generating a search query. That is, the search is carried out only using a specific manner of formed marker sequences. The ability to search for similar documents or fragments using other methods of generating a search query is not implemented, which limits the functionality of the invention to the search for plagiarism in various documents.

As an analogue for the method for searching documents and fragments similar in text and / or semantic content, the method of searching and retrieving documents using applications for automatic personalized search in the database was selected [Patent I85926812 international class С06Р 017/30]. The method allows you to search for documents of interest to the user on data storage devices. The method includes the following operations:

- determination of the set of words most often found in documents stored in the archive on the user device. This takes into account the number of occurrences of words in documents and their importance, determined by the location in the headings, etc.,

- forwarding the resulting set of words to a remote data storage device and searching on it for documents corresponding to said set of words. Formation of a multitude of documents matching the request,

- extracting from the archive stored on the remote device documents that have the highest degree of similarity with documents stored on the user device and display them to the user.

The disadvantage of the invention I85926812 is that it is intended to search for documents only by formal signs of matching words included in their text content. In this case, the invention I85926812 does not provide for the expansion of search queries by any analogues (morphological word forms, synonyms, etc.). Those. I85926812 searches for documents that are similar only in textual content, and does not allow you to find documents that are similar to the selected document in terms of semantic content.

A further disadvantage of the I85926812 invention is that it does not have the ability to select a methodology for generating a search query. That is, the search is carried out only using the most significant (keywords) words inherent in a given set of documents. The ability to search for similar documents using other methods of forming a search query is not implemented. For example, the invention does not provide the ability to search for words that occur in the text content of a given set of documents with a certain frequency, which limits the functionality of the invention.

As an analogue for the method of searching for documents by the initial query of two or more words, the method and device for indexing and searching for documents were selected [Patent ν поиска 9959085 international class С06Р 17/30]. The method includes the following operations:

- splitting a given document into elements (sequences) consisting of 3 words. Those. a document of N words is divided into N-2 elements, for example, a document of five words '01234' is divided into three sequences '012', '123' and '234',

- assignment to each element received for a given document of a certain rating,

- comparison of the ratings (all or some subset of them) obtained for a given document with the ratings in the index of the database of stored documents, the definition of GO of stored documents with the corresponding ratings and their search in the database,

- counting the number of elements present in the given and found documents to determine their identity.

The method involves the use of a device that ensures the execution of operations provided for by the invention of AO 9959085.

The disadvantage of AO 9959085 is that it is intended to search for documents only by formal signs of matching sequences included in word documents, which does not allow searching for documents that have semantic similarity.

The Internet search engine Ωοίρίηοη | \ y \ y \ u \ bs1r1pop.sot | was also selected as an analogue for the method of searching for documents by the initial query of two or more words. This search system allows a phrasal search of documents (patents) of interest to a user by a request consisting of two or more words, taking into account the distance between them and the order of their alternation in the desired document | 1Wp: // \ y \ y \ u.bs1r1iop.sot / 11s1r / 1apd11s1r |. In this case, the request is further expanded with synonyms for one or more words included in the request.

The disadvantage of the search engine Ωοίρίιίοη is that it allows you to expand the query only with synonyms from a dictionary predefined by the developers. Expanding the query with other semantic analogs, for example, through semantic thesauri, as well as the use of the specified sequence of handlers when expanding the query, is not provided in EslpOoi, which negatively affects the effectiveness of search operations. In addition, Ωοίρίιίοη does not provide for the possibility of the user creating their own thesauri.

The problem solved by the invention

The problem solved by the invention is to optimize the search on the data storage devices of documents and their fragments having similarities not only in text but also in semantic content, as well as in eliminating duplication of information stored in the archive. The problem solved by the invention is also to optimize the search for documents by requests of two or more words, taking into account the interval between them and the order of their alternation in the desired document. In this case, a phrasal search for documents is carried out not only by the formal sign of compliance with the request, but also taking into account semantic analogues. The problem is solved by

- using, as search parameters, documents similar in text and / or semantic content to the selected one as the whole set of words included in the selected document, and the specified number of words from the set formed according to certain rules, including those specified by the user,

- the use of various techniques for the formation of many words used as a search query depending on the goals of the search,

- determine the degree of similarity of documents and their automatic classification when entering the archive,

- expanding the source queries by adding semantic analogues to it for words included in the original query using pre-processing operations,

- the formation of lists of documents and their fragments corresponding to the search parameters and their ranking in accordance with relevance to the request received, including taking into account the preliminary processing,

- visualization in the displayed documents of the results corresponding to the search parameters and the request received after preliminary processing.

Brief Description of the Drawings

In FIG. 1 and 2 show the process of preprocessing a search query, which consists in using two types of handlers.

In FIG. Figure 3 shows the process of pre-processing a search query, consisting of one word using several complementary handlers.

In FIG. 4 shows a method for preprocessing a search query.

In FIG. 5 shows an algorithm for generating a list of documents for words included in a search query and combined by the Boolean operator 'OK'.

In FIG. 6 and 7 schematically show methods for breaking documents into fragments.

In FIG. Figure 8 shows the search algorithm on the data storage devices for fragments of documents similar in text and / or semantic content to the selected fragment.

In FIG. Figure 9 shows the search algorithm on the data storage devices for documents upon request from two or more source words, taking into account the interval between words and the order of their alternation in the desired phrases.

Key Points and Definitions

Information processing on a computer implemented in the present invention consists of operations, adding, calling, receiving, transmitting, comparing information, etc., which is often associated with manual operations performed by the operator. The operations described here are machine operations performed in conjunction with various input data provided by an operator or user who interacts with a computer.

The key to determining the unity of purpose of the present invention is the use of pre-processing of the search query originally formulated by the user.

Request pre-processing

As pre-processing operations, transformations are used according to various rules, according to which one or several words associated with the initial word with a given ratio are obtained from the original word. Such rules include the use of various dictionaries and thesauri: morphological, synonymous, semantic, bilingual, etc., as well as the use of various transformation functions. The conversion functions, for example, include: replacing lowercase characters with uppercase and vice versa, replacing Latin letters with Cyrillic and vice versa, etc. The rules used for pre-processing (dictionaries, thesauruses and conversion functions) can be either predefined or created by the user himself. The user forms his own rules in accordance with the criteria known to him. For example, for the convenience of searching for movies, the user will create his own thesaurus, according to which he expands the search query. That is, a lover of comedy films with the participation of specific actors will compile a thematic thesaurus, which will connect the word 'comedy' with the names of the actors 'Richard', 'Murphy', etc., which will allow him to further optimize the search for the necessary data. After working out the query 'comedy' expanded using such a thematic thesaurus, documents containing the words 'Richard' and 'Murphy' will also be presented in the resulting list. Thus, any number of thematic thesauruses are formed for their further use as handlers. The present invention provides for the selection of at least one request preprocessing operation, wherein the sequence of execution of the request preprocessing operations is specified by users. The request preprocessing mechanism provides for the use of two types of handlers: an expanding handler (RO) and a complementing handler (DO).

Each of the words used for preliminary processing of the request is included in a logical group formed for words that have similarities by a certain attribute. For example, the words 'green', 'green', 'green', etc. are included in one group of words, united by a morphological basis (single-root words). It should be noted that as logical groups for RO groups of words can be used, united by characters that are different from the morphological character. In a particular case, a logical group is represented in one word. Logical word groups are used when processing a request through an extension handler (RO). At the same time, each word is included in only one logical group of words that have similarities by a certain attribute (for example, only in one morphological group). PO is used to perform two types of conversion, namely

- from a word to a group, that is, a transformation that defines the identifier of a logical group of the desired type for each of the processed words;

- from a group to a word, that is, a transformation that identifies all words in a given group by the identifier of a logical group.

Various dictionaries and thesauruses are used as DOs: synonymous, semantic, bilingual, etc., as well as various conversion functions described above (replacing lowercase characters with uppercase and vice versa, etc.). DO is used to perform the conversion according to the following algorithm:

- from group to group, i.e., a transformation that identifies group identifiers corresponding to a specific logical group by the identifier of a particular logical group.

This approach to the implementation of the handlers optimizes the process of preliminary processing of the request. That is, the user does not have to manually establish connections between all the words included in the associated logical groups. Such processing will be carried out automatically.

Schematically, the entire process of preliminary processing of a request using handlers of two types (PO and DO) is shown in FIG. 1 in the form of a handler graph, in which the first vertex is a PO (word-to-group conversion), and all subsequent vertices are DO (group-to-group conversion). This process illustrates the use of an unlimited number (one or more) of DO at each of the stages of pre-processing. That is, the logical group obtained after conversion by PO is processed using DO-1/1 - DO-1 / Ν, the groups obtained after conversion by DO-1/1 are processed using DO-1/1 / 1 - DOL / 1 / Ν, groups obtained after conversion by DO-1 / Ν are processed using DO-1 / No. 1-DO-1 / M / S, etc.

In order to facilitate the perception of the described process of preliminary processing of a request, we give a specific example in which preliminary processing of a request consisting of one word is carried out using three complementary processors as shown in FIG. 2. Carry out the obtaining of thematic data from a database containing documents in various languages. Moreover, the data mentioned must correspond to the initial request, not only in the formal, but also in the semantic meaning. Those. the user searches for information of interest to him, stored not only in his native language (for example, Russian), but also that information that is stored in other languages (for example, English and French).

Thus, they perform a search in the archive, the data in which is stored in various languages. Moreover, to obtain the desired data, the original query is expanded using pre-processing operations. At the same time, they use the specified sequence of preliminary processing of the request, which consists in the phased use of an expanding morphological processor, a complementary synonymous processor, and two complementary processors represented by bilingual dictionaries for translation (Russian-English and Russian-French).

Use the following preprocessing algorithm. The user forms the initial request, which consists of one word and at the same time sets the type and number of handlers, as well as the sequence of their use. Since a thematic data search is carried out, at stage 1, as shown in Fig. 2, it is advisable to use a morphological transducer as a PO, which determines the identifier of a logical group for each of the words in the query (in this example, one word). After performing stage 1, a group will be obtained that includes words combined with the original word by a morphological trait (root words). The resulting group (first group) may include one or more words. Since the searched data must correspond to the initial query not only in formal, but also in semantic meaning, at stage 2 it is advisable to use a complementary processor that generates many groups associated with the first group with a synonymous sign. Thus, the dictionary of synonyms will be used as the first complementary processor DO-1, which will expand the original query with semantic analogues (synonyms). DO-1 converts from group to group and determines the group identifiers for the group obtained in stage 1 (the first group), in accordance with the conversion used. The words included in the group obtained in stage 2 constitute a synonymous series. One of the conditions for pre-processing the request for this example is the mandatory presence in the set of groups obtained in stage 2 of the first group. Since the words included in the first group are necessary for further information retrieval, and their exclusion from the further processing process will lead to a decrease in the effectiveness of the search operation. In other cases, the first group may be excluded from the plurality of groups obtained in step 2, and thereby not be further processed. Next, in step 3, each of the groups obtained in step 2 is processed using two complementary processors: DO-2, which uses the Russian-English dictionary and DO-3, which uses the Russian-French dictionary. As a result of performing step 3, many groups are formed associated with the groups obtained in step 2 by corresponding transformations.

All groups obtained in steps 1-3 are included in the final set of groups. Further, from the resulting final set of groups, a lot of words are formed, from which the final query is generated for searching the archive containing information in different languages (Russian, English, French), using semantic analogues for each of the mentioned languages. Those. The final stage of preprocessing involves the use of an expanding processor (RO) for conversion by type from group to word for all groups included in the final set. This transformation determines from the identifiers of logical groups all the words included in specific groups and forms from them the final set of words, including all the words inherent in the obtained groups. From these words, a final request is subsequently formed.

In the above example, the given sequence of processors includes one complementary processor of a specific type: one synonymous for Russian, one Russian-English for translation, and one Russian-French. This approach to the formation of a sequence of handlers is a special case. In the general case, if necessary, any number of complementary handlers of a particular type (several synonymic, several bilingual, etc.) is included in the given sequence in any order. Moreover, a variant is possible in which the complementary processor of the same type (for example, synonymous for the Russian language) will participate in the processing of the same group several times.

To simplify the perception of the process of preliminary processing of the request, we describe the above example in more detail. It is worth recalling that this example describes the preliminary processing of a request consisting of a single word. In the case when the request will consist of several words, the pre-processing algorithm will be applied for each of the words included in the original request. As an initial request for this example, as shown in FIG. 3, use the word 'Information'. Preliminary processing is carried out using RO, combining words into logical groups according to morphological characteristics. In addition, three complementary processors are used: synonymous for Russian, Russian-English for translation, and Russian-French. The transformations by PO are indicated in FIG. 3 by dashed lines, and conversions by DO in bold lines.

For the original word 'Information', as shown in FIG. 3, a logical group (LG-1), which includes words that have a common morphological characteristic with the original word, is determined by transforming the expanding processor from word to group. Many words for LG-1 will include the words 'Information' and 'Information' (words combined with the original word 'Information' by a morphological trait). Of course, a group may include a larger number of words, but the restriction on the number of words introduced in this example is associated with a simplification of the perception of the preprocessing process. It should be noted that the preliminary processing of the PO can consist in the operation of identity, which is used for those cases when the logical group consists of one word.

Next, preprocessing is performed using complementary handlers, including conversion from group to group. First, in accordance with the specified pre-processing parameters, the processing is carried out by the synonymous processor DO-1, which forms many groups associated with LG-1 by a synonymous basis. Such groups are LH-2 and LH-3. In addition, one of the conditions for preliminary processing of the request is the need for the presence of the LG-1 group in the set of groups to be further processed, as shown in FIG. 2. Thus, after processing the request by the DO-1 handler, we get many groups of LG-1, LG-2, and LG-3. Of course, the mentioned set may include a larger number of logical groups, but the restriction on the number of groups introduced in this example is associated with a simplification of the description of the preprocessing process and to facilitate its perception. Since the LG-2 and LG-3 groups were formed using a synonymous transformation, these groups will include words that make up a synonymic series with the word 'Information'. For example, a lot of words for LG-2 will include the words 'Details' and 'Information', and for LG-3 a lot of words will consist of 'Message' and 'Messages'. As can be seen from the description, the resulting sets of words are interconnected by a synonymous sign (processing DO-1), and the words in each of the sets are related by a morphological sign (processing RO).

This example provides for the subsequent processing of the received groups by the additional handlers DO-2 and DO-3. DO-2 is a Russian-English dictionary for translation, and DO-3 is a Russian-French dictionary. Processing by each of the mentioned handlers leads to new groups associated with the LG-1, LG-2 and LG-3 groups with the corresponding transformations. So, for LG1, through DO-2, a multitude of groups is formed, consisting of the group LG-4, which includes the words 1i1ogta1yui 'and' 1n1ogtaDopaT. Similarly, pre-processing of DO-2 for LG-2 leads to the formation of a set consisting of the group LG-6, which includes the words 'Ma1eDaT and' Ma1epa1z, and processing for LG-3 leads to the formation of a set consisting of the group LG-8, which includes the words 'Mezzade' and 'Mezzadez'. The connections between the sets of words included in LG-4 and LG-1, as well as for the connectives LG-6 - LG-2 and LG-8 - LG-3, are determined by the sign of the use of the Russian-English dictionary for translation.

In addition, the use of DO3 envisaged by the pre-processing algorithm for processing the LG-1, LG-2, and LG-3 groups leads to new sets of groups. For LG-1, the group LG-5 will be obtained, consisting of the words '1p1ogtaDop' and '1p1ogtaDoppeG, for LG-2, the group LG-7 will be obtained, the words' Repze1dtepG and 'Repze1dtep1z', and for LG-3, the group LG -9, consisting of the words 'Mezadeh' and 'Mezzadez'. The connections between the sets of words included in LG-5 and LG-1, as well as for the connectives LG-7 - LG -2 and LG-9 - LG-3, are determined by the sign of the use of the Russian-French dictionary for translation.

All logical groups formed during processing are included in the final set of groups, which after all the transformations will consist of: LG-1, LG-2, LG-3, LG-4, LG-5, LG-6, LG-7, LG-8 and LG9. The final stage of the preliminary processing of the request (after all the transformations using complementary handlers) includes a transformation for each of the logical groups included in the final set, consisting in processing all the logical groups mentioned by an expanding handler by type from group to word. This leads to the formation of many words for each of the logical groups. Moreover, as mentioned above, the sets of words obtained for each group will include words that have a common morphological characteristic.

The conversion by means of PO determines by the identifiers of logical groups all the words included in specific groups and forms from them the final set of words, including all the words inherent in the obtained groups. At the same time, redundant information is removed from the final set of words, i.e., duplicate words are deleted, such words in this example are the words 'Ιηίοηηαΐίοη', 'Meccade' and 'Meccadec,' repeated twice. Of the remaining words in the final set of words, a final query is subsequently formed by combining the words with the Boolean operator 'OK'.

Thus, for this example, the final request will take the form:

'Information OK Information OK Information OK Information OK Message OK Message OK ΙηίοπηηΙίοη OK Ιηίοπηηοοηηΐ OK 1pGogsha1uppe1 OK Ma1epa1 OK Ma1epa1k OK Mekkade OK Mekkadek OK Keηke ^ dteηΐ OK Keshegdeyteik'

It is worth paying attention to the fact that since the same word can be present in several languages, such as the word 'Meccade', which is equally used in both English and French, its processing of RO by converting from a word to a group may include The choice of type RO. That is, if for the considered example the word “Mekkade” will be the source in the request (instead of the word “Information”), a user will be faced with a dilemma when forming a sequence of handlers which type of PO should be used to process the word “Mekkade” with the purpose of the formation of LG-1 (English morphology or French). Moreover, the RO can be set by default, for example, when an English-speaking user works with the invention, the default RO of English morphology will be set. However, this does not deny the explicit choice of another type of RO, for example, RO of French morphology. It should be noted that the choice of the type of PO is carried out only during processing by conversion from a word to a group. In the case of the inverse transformation - from group to word, the determination of the desired type of PO is carried out automatically, since the type of PO will be determined by the identifier of a particular logical group.

The preprocessing algorithm is represented as a sequence of operations performed by the system in which the present invention operates, as shown in FIG. 4. At step 4, the initial data for preliminary processing are determined, which include the determination of the source word for preliminary processing, the number and sequence of use of additional handlers (DO), as well as the definition of the expanding handler (RO) for processing the source word. In a particular case, a PO is defined by default and is a morphological processor. At step 5, the initialization process is initialized, resulting in the formation of the final set of words (empty), the final set of groups (empty), the set of groups for processing for all DOs (empty), as well as an empty stack of processors (CO). The format of the stack of processors involves storing information about identifiers of DOs and information about identifiers of groups intended for processing by the said DOs. Next, at step 6, the acquisition is carried out according to the source word of the logical group specified in step 4, which includes this word. This operation includes processing by means of PO, and consists in converting from a word to a group.

At step 6, in accordance with the format of the stack of processors, information is entered into the CO, including identifiers of all DOs associated with the PO (as shown in Fig. 3, such will be DO-1), as well as identifiers of groups intended for processing the mentioned DO (as shown in Fig. 3, such a group for DO-1 will be LG-1).

In addition, the resulting group is recorded in the final set of groups. The total set of groups is understood to mean the entire set of groups obtained in the process of preliminary processing by complementary processors by means of transformations from group to group.

At step 7, the stack of handlers is checked. If the stack is empty, then at step 8, the resulting set of words is formed by conversion using the PO from the group to the word (the definition of the PO for each group at this stage is automatic). Those. from the final set of groups obtained at the previous stages, many words are obtained, from which possible duplicates are excluded. Then, at step 9, the operation of combining all the words included in the final set of words obtained at step 8 by the Boolean operator 'OK' is carried out.

If the CO is not empty, then at step 10 they turn to the last handler located on the stack (in this case, the handler is assigned the status of the current handler). At step 11, all the groups included in the set intended for processing by the current handler are processed using group-to-group conversion.

At step 11, in accordance with the format of the stack of processors, information is entered into the CO, including identifiers of all DOs associated with the current handler (as shown in Fig. 3, such handlers will be DO-2 and DO-3 associated with with the current DO-1 handler), as well as the identifiers of the groups intended for processing by specific DOs (as shown in Fig. 3, the stack will include information that the LG-1, LG-2, and LG-3 groups will be processed DO-2, as well as information that the mentioned groups will be processed DO-3).

In addition, the set of groups obtained after processing by the current DO is recorded in the final set of groups, excluding possible duplicates. Next, at step 12, a handler that has the status of the current one is deleted from the handler stack, and then they return to step 7.

The sequence of operations included in steps 7-12 is carried out until the stack of handlers is empty. Then proceed to step 8.

Since the example was given for a query consisting of one word, it is worth noting that in the case of preliminary processing of more complex queries consisting of several words, the results obtained after the preliminary processing of each of the words included in the original query are combined using Boolean operators. For example, the use of the operator 'в' in the case of searching by a group of words or the operator 'ΝΕΑΚ1 in the case of phrasal search, taking into account the sequence of words and the interval between them.

The extension of the search query through pre-processing contributes to a wider coverage of the information space used to search for the data of interest. At the same time, the final request is formed without much difficulty using both standard handlers (dictionaries and conversion functions), and user-created ones. Thus, pre-processing expands the search query with various semantic analogues, which contributes to the search for documents that are similar in textual content not only in formal but also in semantic meaning. This functionality is extremely useful in cases where documents of a similar subject matter are processed, but having different vocabulary, for example, containing the words 'information exchange' and 'data transfer'. In a conventional search (without using pre-processing), the mentioned phrases with visible semantic similarity will not be interpreted as having similarities, since there is no formal sign of similarity in vocabulary. In the case of using pre-processing using the synonym dictionary, both phrases will be defined as having similarities in meaning and, accordingly, when searching using pre-processing, documents containing any of the mentioned phrases will be found.

The number of handlers used in this invention is not limited, but it must be remembered that the use of a large number of them is not always advisable, since it can lead to the resulting lists containing redundant information.

Often used in the text of the description of the invention provisions and definitions

Information blocks are sequences of characters in a document that are limited to specific characters. Thus, words, sequences of characters representing dates (for example, 03/21/2001), sequences of numbers representing the amount of something, etc., separated by spaces, commas, periods, and other punctuation marks, are used as information blocks.

Software module for processing documents of a certain type - designed to break the selected fragment of the document into the original blocks of information (words). A particular module breaks up documents of a certain type (format) into blocks of information, i.e. It is a converter for extracting information blocks from the text content of documents of a certain format. Using this module allows you to compare documents of different formats with text content for similarity. For example, you can compare the text content of web pages with the text content of documents in Myugokoy \ Wogb format, etc. This module can be generated by the user. The invention also provides for the use of standard modules for processing documents of a certain type. Standard modules are used for breaking into source words documents of well-known formats (M1stoy OGyse, Aboye Asgoya, etc.). For example, a standard module (a converter developed by Myugokoy) will be used to split documents of the M1gokoy \ Wogb format and their fragments into source words.

Source words - a lot of blocks of information obtained after breaking a specific document by a software module for processing documents of a certain type.

Many unique words that make up a particular document. It is formed from the set of source words included in a particular document. For example, if a word appears several times in the mentioned document, it will be presented only once in a set of unique words. Thus, many unique words are an optimized set of source words.

The group of correspondence to the original (or unique) word - is a set of 17 words obtained in the process of preliminary processing of a specific source (or unique) word. The correspondence group includes all words associated with the original (or unique) word by the relations determined by the given pre-processing operations.

Correspondence group element - a word included in a specific correspondence group formed in the process of preliminary processing of the initial (or unique) word included in the search query.

Creating lists of documents for compliance groups

Common to the inventions included in this application is an algorithm for generating a list of documents (or fragments thereof) for a correspondence group associated with a specific word included in a search query. In the future, when describing the algorithm, for simplicity, only the term document will be used, since a full copy of the document is a special case of a fragment.

The algorithm is implemented as follows. Get an intermediate list of documents (PSD) for each element included in the group of correspondence associated with a particular word. The intermediate lists of documents for each of the elements of the correspondence group are saved. This is due to the fact that for several words included in the request, identical elements of correspondence groups can be obtained during the preliminary processing. For example, for the words 'information' and 'message' an identical element will be received - the word 'information'. Reprocessing this word in order to form an intermediate list for it when processing the words 'information' and 'message' is impractical. This is especially true in cases where the search query consists of words that are close in meaning, for example, synonyms. Having once formed an intermediate list for an element of the correspondence group, they are subsequently referred to.

The lists obtained for each element of the correspondence group (PSD1 - PSDt, where t is the number of elements included in the group) are combined into a common list of documents for a particular correspondence group. That is, for example, lists are combined for the words 'information', 'information' and 'message' that are in the same group. The mentioned general list of documents (SD) for the correspondence group associated with a specific word included in the search query is formed using the Boolean operator 'OK':

SD = 'PSD1' OK 'PSD2' OK 'PSDt'.

Thus, for a particular compliance group, a list of documents is formed in which at least one element of the said compliance group is found. For example, a list of documents for the word 'information' is formed. The correspondence group obtained for this word includes the words: 'information', 'information' and 'message'. The resulting list of documents will include documents containing at least one of the listed words included in the correspondence group for the word 'information'. For example, Document1 includes the words 'information' and 'information', Document2 includes the word 'information', and Document3 includes the word 'information'. Then the list of documents for the word 'information' will include all three documents, and each of the documents will be presented once (despite the presence in Document1 of two words that satisfy the request parameters).

Lists of documents received for compliance groups are retained. This is due to the fact that for several words included in the query, identical correspondence groups can be obtained. For example, for the words 'information' and 'message', an identical correspondence group can be obtained: 'information', 'message', 'information' and 'data'. Accordingly, re-forming the list for this group is impractical.

The above algorithm is used to form correspondence groups for each of the words included in the search query.

Formation of a list of documents for words combined in the query by the 'OK' operator

Common to the inventions included in the present application is also an algorithm for generating a list of documents for a variety of words combined in a search query by the Boolean operator 'OK'.

The format for the description of documents included in the final list includes the fields 'Methodology identifier', 'Document identifier', as well as the field for counting the number of correspondence groups contained in documents - 'Number of correspondence groups'. In the case when it comes to forming a list of fragments, the above description format will additionally include the 'Fragment identifier' field. Those. the format for describing fragments in the final list can be represented as:

Yu Methods II) Document Yu Fragment Number of Compliance Groups

If the fragment is a full copy of the document, then the fragment GO may coincide with the document GO. The formation of the final list (IS) of documents for words combined in the query by the Boolean operator 'OK' is as follows.

At step 13, as shown in FIG. 5, lists of documents generated for correspondence groups associated with words included in a search query are used as input parameters. At step 14, the list of documents generated for the correspondence group associated with the first word in the search query is entered into the final list of documents. In this case, the value of the field for counting the number of correspondence groups, for all documents entered in the IS, will take the value of unity. Next, at step 15, it is determined whether all the words included in the search query have been processed. If not, then at step 16 they turn to the correspondence group associated with the next word in the search query and assign it the current status. Then, at step 17, the final list of documents is generated using the rule:

IS = IS 'OK. 'CD of the current compliance group'

Those. at step 17, documents from the list obtained for a particular correspondence group associated with the word included in the search query are entered into the final list of documents. At step 17, the field value is also increased to calculate the number of correspondence groups for documents included in the information system. This operation is as follows. For example, a query consists of two words:

the first is 'information' and the second is 'necessary'. For each of these words, a correspondence group is obtained: the first group is 'information', 'information' and 'message'; the second group is 'necessary' and 'necessary'. There are four documents in the archive:

Document id Words in the text content of a document GO Document1 'necessary', 'information', 'necessary', 'information' GO Document2 'needed', 'information' GO DocumentZ 'information' GO Document4 'necessary'

Words from the matching group for the first word 'information' are included in only three documents. Therefore, the list of documents obtained for the compliance group associated with the word 'information' will include: Document1, Document2 and Document3. Accordingly, the final list after step 14 can be represented as follows:

Document id Number of Compliance Groups GO Document1 one GO Document2 one GO DocumentZ one

Words from the correspondence group for the second word 'necessary' are also not presented in all documents, but only in Document1, Document and Document4. Therefore, the final list of documents after step 17 for the group of compliance associated with the second word included in the search query will take the form:

GO Document2 2 GO DocumentZ one GO Document4 one

Since Document1 and Document2 include elements of correspondence groups associated with both the first and second words included in the search query, the value of the field for counting the number of correspondence groups found in these documents at step 17 will be two. The mentioned documents themselves are not re-entered in the final list. A new Document4 is also added to the final list, and the value of the field for counting the number of correspondence groups for it takes the value of unity.

Steps 16 and 17 are performed until at step 15 it is determined that all words from the search query have been processed. Then they proceed to step 18, where the final list of documents for the set of words combined in the search query by the Boolean operator 'OK' is considered to be generated.

The method of searching for fragments similar in text and / or semantic content in electronic documents stored on data storage devices

The method consists in indexing each document stored in the archive, dividing documents into fragments and forming topics from one or more fragments, determining search parameters, conducting a search and ranking the list of document fragments resulting from the search in accordance with certain parameters.

As search parameters, a plurality of unique information blocks included in a selected fragment of a document is determined and expanded by pre-processing each of said unique information blocks. Moreover, a unique information block is understood to mean a block of information that has occurred one or more times in a selected fragment of a document. As the preliminary processing, the operation of obtaining at least one unique block of information, one or more blocks of information associated with a unique block of information with a given ratio is used.

As a block of information using a sequence of characters in the document, limited to certain characters. As a document fragment, any selected sequence of information blocks included in the document is used, including a full copy of the document.

The document is divided into fragments using at least one established rule. Many fragments of documents, among which

Document id Number of Compliance Groups GO Document1 2

the lawsuit of fragments similar in text and / or semantic content to the selected one is limited to indicating at least one rule by which documents were divided into fragments and / or subject. The subject is a lot of fragments, united by a certain attribute. As subjects use, for example, the classifier section.

Splitting documents into fragments can be done automatically. To break fragments of documents into many blocks of information using a software module for processing documents.

At least one pre-processing operation is selected from a plurality of information blocks to form a search query, and the users specify the sequence of preliminary processing operations. Moreover, the invention as the preliminary processing of the request provides for the use of a single logical operation of identity. After the pre-processing operation is completed, the initial block of information is either removed from the resulting query or left in the resulting query.

The search for fragments of documents similar in text and / or semantic content to the selected one is carried out using either the whole set of unique information blocks included in the selected fragment of the document, or a certain number of them. The number of unique blocks of information defined as parameters for the search and the methodology for their selection are set by users. Many unique information blocks defined as search parameters are formed using one or more functions, which are used as

- the function of forming a list of unique information blocks for the selected fragment,

- a function for determining the number of occurrences in the text content of fragments of a selected topic of information blocks obtained in the process of preliminary processing of a unique information block,

- a function for determining the frequency of occurrence of unique information blocks in the text content of fragments of a selected topic, where the frequency of occurrence is calculated as a percentage of the number of occurrences of the unique information block most often used in the text content of the selected topic.

In addition, many unique blocks of information, defined as search parameters, are formed according to user-defined rules.

The search for fragments of documents corresponding to the search parameters is carried out on local and / or remote data storage devices. As remote devices, use any information resource or system designed to search for data, operating on a computer network and providing, in response to a search query, a list of document fragments satisfying the search parameters.

The obtained search results (lists of document fragments) display and rank them in accordance with certain parameters. The ranking of the list of fragments of documents similar in text and / or semantic content to the selected one is carried out in accordance with the number of groups of information blocks present in the found fragments of documents. Mentioned groups combine blocks of information obtained in the process of pre-processing a specific block of information. For those fragments of documents in which the number of groups of information blocks coincides, refinement sorting is performed. At the same time, for each fragment, the number of occurrences in the text content of fragments of the selected topic, all the groups of information blocks inherent in the fragment is counted. Fragments included in the list of fragments obtained as a result of the search are displayed with visualization of the differences between their text content and the text content of the selected fragment.

If a document is found in the archive that has a specified degree of similarity with the archived document, the archived document is saved as a new version of the document found in the archive. At the same time, the new version is saved as a full copy of the archived document or as differences between the text content of the archived document and the text content of the document found in the archive. Fragments of the archived documents are automatically classified.

Description of the invention

This method provides for the formation on the data storage device of an archive of documents and their fragments, in which all stored fragments of documents are indexed. A document fragment is any selected sequence of source words included in a document and consists of at least one source word. As a fragment of a document, for example, a sentence, paragraph, paragraph or section of a document, etc. is used. A special case of a document fragment is a full copy of the document. Indexing of documents recorded in the archive is carried out in accordance with established rules. Those. when a document is entered into the archive, the fragments are determined into which it will be divided and, in accordance with the selected technique for splitting the document, it is indexed for further index search of document fragments. In this case, regardless of whether the document will be divided into fragments or not, by default they index its full copy using a specific technique that is common for all documents. Thus, the document in the archive is stored in the form of at least one fragment, which is a full copy of it. Moreover, fragments representing complete copies of documents stored in the archive are indexed using a single technique.

The definition of a technique for splitting a document into fragments does not depend on the number of words included in a given sequence of words that make up the fragment. For example, in one case the document is divided into sentences, in the other case into paragraphs, and in the third case, the fragment is a complete document. The document can be simultaneously divided into different fragments, which are, for example, both paragraphs and sentences. Fragments may intersect each other. When defining a fragment, mark the area of the document, which is defined as a fragment. In FIG. 6 schematically shows a document, which is divided into five different fragments. In this case, Fragment1 and Fragment2 intersect, and Fragment5 overlaps Fragment3, Fragment4 and part Fragment2. When splitting a document, one technique was used (Method 1). This means that all the mentioned fragments of the document will be indexed using one technique.

In FIG. 7 schematically shows a document, which is divided into four fragments. Moreover, when determining Fragment 6-8, Method 1 was used, and when determining Fragment 8-9, Method 2 was used. Accordingly, in this case, fragments of one document will be indexed using various techniques. One and the same fragment of the document (Fragment8) can be defined and indexed using several techniques. The format of the document fragment stored in the archive can be represented as follows: _____________________________________________

II) Methods | II) Document | II) Fragment

This format allows you to uniquely identify any fragment stored in the archive when it is searched using any technique. The format of the document fragment stored in the archive may have a different look, for example, it can be expanded by introducing additional fields.

In the index search of document fragments, indicate the technique with which they will search for the mentioned fragments. Search for fragments of documents can be carried out using one or more techniques. For example, to search for fragments similar in textual and / or semantic content to Fragment8 using Methodology1, Fragments17 will be used. To search for fragments similar in textual and / or semantic content to Fragment8 using Methodology2, Fragment9 will be used. In the case of searching for fragments similar in textual and / or semantic content to Fragment8 using both methods, Fragments 1-7 and 9 will be involved. The invention provides for setting default parameters for using the technique. In this case, by default, the search is conducted according to the technique (or techniques) in accordance with which the selected fragment was formed. An indication of the technique allows you to limit the search to the desired document fragments. The search area can also be limited by indicating one or more topics, among which search for fragments similar in text and / or semantic content to the selected one.

The methodology for breaking documents into fragments is determined by users. At the same time, a mode is provided in which parameters are set for automatically breaking documents into fragments, for example, paragraphs, etc. To do this, form the rules in accordance with which the division of documents into fragments will be carried out. To split a document into sentences, such a rule would be to limit the sequence of words in the text to the corresponding punctuation mark, that is, a period. For determining paragraphs, for example, a marker for moving the cursor to a new line, etc. will be used. The invention provides for setting a mode in which self-learning of the system is carried out for further automatic splitting of documents into fragments in accordance with certain rules. This mode is provided for breaking into fragments of those documents whose format is an ordered sequence of information blocks. An example is the text of a patent application, which includes the mandatory sections: 'Description of the invention', '' claims ',' 'Abstract', etc. Having addressed several times a document of this type (which is a patent application) and dividing it into fragments using a specific technique, the user implicitly generates rules for further automatic splitting into fragments of all texts of patent applications. The system “remembers” the rules for splitting a document (by defining, for example, section headings as markers), and in the future, patent applications are split automatically without user intervention. The invention allows the search for fragments similar in text and / or semantic content to the one selected simultaneously for one or more selected fragments.

The invention provides for the formation of topics from one or more fragments. At the same time, a lot of fragments for the formation of a specific topic are determined by the user. A topic is, for example, a multitude of fragments tied to a classifier or to its specific section. One and the same document stored in the archive (and, accordingly, its fragment) can be tied to several topics.

The method is implemented as follows. At step 19, as shown in FIG. 8, the text of the selected fragment of the document is partitioned into source blocks of information (source words). The selected fragment of the document is that fragment for which a search is made for similar fragments of documents to the text and / or semantic content of the documents. To break the selected fragment of the document into source words, a software module for processing documents of a certain type is used. From the obtained set of source words, many unique words are generated that are inherent in the selected fragment of the document. Many unique words are an optimized set of source words. For example, if a word occurs several times in a selected fragment, it will be presented only once in a set of unique words.

Next, at step 20, each unique word is pre-processed from the set of unique words obtained at step 19 for the selected fragment. Thus, at step 20, matching groups are formed for each unique word. Pre-processing of unique words expands the search query with various semantic analogues, which helps to search for fragments of documents similar in text content not only in formal but also in semantic meaning.

At step 21, it is determined whether to search for fragments of documents that are similar in text and / or semantic content to the selected one, use all the unique words obtained at step 19. The list of unique words used for search can include either all unique words obtained at stage 19, or a certain number of them. If at step 21 it is determined that not all unique words will be used, but a certain number of them, then proceed to step 22. At this stage, it is determined whether user-defined rules will be used to determine the set of unique words. If yes, then proceed to step 23, which form a list of unique words for searching according to the rules established by the user. Such rules may be, for example, the formation of many unique words to search from the many unique words obtained in step 19 with the number of characters not exceeding a number specified by the user. To form many unique words for search, the user can establish more complex rules. For example, the definition to search for those words whose frequency of occurrence in the selected fragment is in a certain interval. In addition, the list of words to search can be specified by manually selecting words from the list of unique words obtained in step 19.

If it is determined at step 22 that user-defined rules will not be used to generate the set of unique words for searching, then proceed to step 24. At this stage, one or more functions provided by the present invention are selected to determine the set of unique words that will be used for searching. As functions by which the formation of many unique words for the search is carried out, use

- the function of forming a list of unique words for the selected fragment,

- a function of counting the number of occurrences of unique words in the text content of fragments of a selected topic,

- a function for determining the frequency of occurrence of unique words in the text content of fragments of a selected topic.

The use of the mentioned functions is relevant in those cases when a full copy of the document is used as the selected fragment. If a relatively small fragment is used as the selected fragment during the search and fragments having similarities are searched for in the archive, using the entire set of unique words from the selected fragment in the query does not create problems. If the request uses the entire set of unique words obtained for the selected full copy of a sufficiently large document, then the speed of forming the final list of fragments is reduced. Even more critical is the rate of the formation of the final list when searching for fragments on remote devices. Therefore, in some cases, it is much more convenient to use the following functions to form many unique words for search.

The function of forming a list of unique words for the selected fragment

This function (F1) is intended for the formation of many unique words inherent in the selected fragment.

The function of counting the number of occurrences of unique words in the selected subject

This function (Ф2) is implemented as follows. There is a unique word, and for it a correspondence group is formed, obtained during the preliminary processing. For each element of the correspondence group associated with a unique word, the number of its occurrences in the text content of fragments of the selected topic (or in the text content of all documents stored in the archive) is determined. Then summarize indicators of the number of occurrences of all elements of the correspondence groups that make up a particular group. For example, in the process of pre-processing the unique word 'information', a correspondence group is obtained that includes many elements: 'information', 'information' and 'message'. Then, for each element, the number of entries in the text content of the selected topic is counted: 'information' - ten, 'information' - five, 'message' - three. Thus, this function determines that the total number of entries in the selected topic of all the elements that make up the correspondence group obtained for the unique word 'information' is eighteen.

The function of determining the frequency of occurrence of unique words in the selected topic This function (Ф3) is implemented as follows. A unique word is taken and for it the frequency of occurrence of fragments of the selected topic in the text content is determined. The frequency of occurrence is calculated as a percentage of the number of occurrences of a unique word, most often used in the textual content of fragments of a selected topic. For example, a unique word 'information' is taken, for which it is necessary to determine the frequency of occurrence of fragments of the selected topic in the text content. It is determined that in the textual content of fragments of a selected topic, the unique word 'search' is most often found and the total number of occurrences of this word is 100 times. This indicator is considered 100%. Moreover, the number of entries in the selected topic of the unique word 'information' is 30, and this means that the frequency of its occurrences is 30%.

Manipulating the above functions allows the user to create various rules (logics) to determine the set of unique words used as search parameters. Here are three possible logics, each of which can be used depending on the purpose of the search operation.

First logic

This logic allows you to select a set number of words from a list of unique words obtained in step 19 and ordered in accordance with the total number of entries in the selected topic (or the entire archive) of all words obtained during the preliminary processing of a unique word.

This logic involves the use of the function of forming a list of unique words for the selected fragment and the function of counting the number of occurrences of unique words in the selected subject.

For the selected fragment, a list of unique words (F1) is determined, and for each of these unique words, the number of entries in the selected topic is calculated (using Ф2). Next, the list is sorted in ascending order of the number of occurrences of unique words in the selected subject. That is, in the first place in the list will be a unique word with the least number of occurrences (the rarest for the selected topic). After that, from an ordered list of unique words according to user-defined rules, a lot of words are formed for search. Such rules are the indication of the interval for choosing words. The interval for choosing words can be any. If the list includes fifty unique words and twenty words are selected, then you can set, for example, the following intervals: from the first to the twentieth word, from the eleventh to the thirtieth, from the thirty-first to the fiftieth, etc. When forming a set of unique words for searching using the first logic, you can specify a percentage indicator, for example, choose 30% of the rarest words from the list, etc.

The main objective of this logic is to select a set of the most rare or most often used unique words for the selected topic to search for.

Second logic

This logic allows you to select the words included in the selected subject (or the entire archive) with a given frequency from the set of unique words received for the selected fragment. This logic involves the use of the function of forming a list of unique words for the selected fragment and the function of determining the frequency of occurrence of unique words in the selected subject.

For a selected fragment, a lot of unique words (F1) are formed and for each of these unique words the frequency of its occurrence in the selected subject is determined (using F3). After that, from the whole set of unique words obtained for the selected fragment, the selection of unique words is carried out according to user-defined rules. As the rules indicate the interval of frequency of occurrence, in accordance with which form a list of unique words to search, or the number of words having the smallest or vice versa the highest rate of occurrence. For the example considered in the description of the function of determining the frequency of occurrences, in the case of setting an interval of 25-30%, the word 'information' will be selected for the search.

Using unique words with a given range of occurrence frequencies, unlike the first logic, allows a more flexible approach to the formation of a search query. Indication of the number of words (the first logic), even the most rarely included in the selected topic, does not always allow the formation of an optimal query. For example, many of the twenty most rare words identified may include words with a frequency of occurrence of 5% (first word) and 30% (twentieth word). Those. the gap between the frequency of occurrence of both words is too large. Using the second logic eliminates this problem.

Third logic

This logic allows you to create a set of unique words for search with the largest number of entries in the selected topic and at the same time having the least number of entries in the entire stored archive except for the selected topic. This logic involves the use of the function of forming a list of unique words for the selected fragment and the function of determining the frequency of occurrence of unique words in the selected subject.

Form a list of unique words for the selected fragment (F1). After that, for each of the obtained unique words, the frequency of its occurrence in the text content of fragments of the selected topic (F3) is determined. Next, they turn to each unique word from the resulting list in sequence and determine the frequency of its occurrence in the entire archive with the exception of the selected topic (F3). Thus, for each unique word, two indicators are determined: the frequency of entry into the selected topic (P1) and the frequency of entry into the entire archive except for the selected topic (P2). Then, for each unique word, the difference between the indicators Ch1 and Ch2 (P = Ch1-Ch2) is determined, after which the list of unique words is sorted in descending order of indicator P. From the generated list for the query, the specified number of unique words having the highest indicator R. is selected.

The advantage of this logic is that it allows you to search for words that are key to the topic of interest to the user. For example, the selected topic is a section of the classifier 'Investments'. Of course, the word 'Investments' and its word forms will appear quite often in the textual content of the chosen topic and, for example, have an indicator of Ch1 = 80%. A similar indicator (Ch1 = 80%) will have a unique word 'can', the use of which is not desirable in the request, since it is commonly used. The rate of occurrence of the word 'investment' in the entire archive, except for the selected subject, will be quite low, for example, Ch2 = 1%. What can not be said about the commonly used word 'can', the indicator of which is equal to, for example, 10%. Thus, the indicator P for the word 'investment' will be 80-1 = 79, and the indicator P for the word 'may' will be 80-10 = 70. The query will include the word 'investment', as having the highest indicator P. In the event of a possible equality of indicator P, a refinement sorting is carried out in accordance with the parameters defined by the user. For example, preference is given to a unique word that has a higher indicator of Ch1 or a word that has a lower indicator of Ch2.

Thus, this logic allows us to exclude common words from the query and form a query from a variety of unique words that most accurately characterize the text content of the selected topic. The use of this logic is relevant in cases where a large number of documents are stored in the archive that are tied to relevant topics. The use of this logic is not limited to the choice of a set number of unique words that have the largest index P. For search, unique words can be selected whose index P is in a given range.

It should be noted that the number of logics is not limited to the three logics described above, because the user can use other combinations of functions to create new logics.

After performing steps 23, 24, or if at step 21 it is determined that the whole set of unique words will be used for the search, they proceed to step 25, where a final list of unique words for participation in the search query is formed. After forming the final list of unique words for participating in the search query, they proceed to step 26. At this stage, the final list of documents for the set of words obtained at step 25 is formed. The list is generated using the algorithm for generating a list of documents (or fragments thereof) for many words combined in the search query by the boolean operator 'ΘΚ.'.

The search for fragments corresponding to the query parameters is carried out both on local and remote data storage devices. The invention allows the search for fragments of documents corresponding to the generated request simultaneously on local and remote devices. Moreover, any information resource or system designed to search for data and operating on a computer network is used as remote devices.

When searching for fragments on remote devices, an intermediate list of fragments obtained from user-specified remote devices is formed for each element of the correspondence group. In this case, the list of fragments received from remote devices can be a list of full copies of documents (links to them), as well as a list of fragments thereof. The splitting of full copies of documents received from remote devices into fragments is carried out according to formal rules, which can be set, for example, for documents received from specific devices. An example of a breakdown of documents representing patent applications has been described above. Similarly, documents of a different type can be broken up into fragments, for example, newsletter emails, etc. If it is not possible to split documents received from remote devices into fragments, then full copies of these documents are considered a fragment.

After the final list of fragments similar in text and / or semantic content to the selected one is formed, at step 26 it is ordered and displayed.

The list in step 26 is sorted according to the number of matching groups present in the fragments obtained. In this case, a specific correspondence group includes all words obtained in the process of preliminary processing of a unique word included in the selected fragment and included in the request. For example, a unique word is the word 'information'. For pre-processing, a morphological RO and two DOs are used - a dictionary of synonyms and a Russian-English dictionary for translation. Then, the correspondence group formed for the word 'information' will include, for example, elements: 'information', 'information', 'information', 'information', 'message', 'messages', 'ίπίοηηαΐίοη', etc. In the case of any number of occurrences of one or more of the mentioned elements in a fragment, it is considered that there is one correspondence group in it. Thus, the list is sorted in descending order of the number of matching groups included in the fragments. The list includes four fragments:

GO fragment Words in a fragment The number of groups in the fragment Number of entries groups in the subject GO Fragment! 'necessary', 'necessary', 'information', 'information', 'search', 'search' II) Group 1 / necessary, necessary / 7 GO Group2 / information, information / 10 GO GroupZ / search, search / 5

GO Fragment2 'information', 'information', 'search', 'search' GO Group2 / information, information / 10 GO GroupZ / search, search / 5 GO FragmentZ 'search', 'search', 'necessary' 'necessary' GO Group1 / necessary, necessary / 7 GO Group2 / information, information / 10 GO Fragment4 'information' GO Group2 / information, information / 10

Fragment1 is in the first place in the list, since it includes words from three correspondence groups. Next are two fragments with the same number of matching groups - Fragment2 and FragmentZ (two groups each). Closes the Fragment4 list with one matching group.

Then, a refinement sorting of the final list is carried out for those fragments in which the number of matching groups is the same: Fragment 2 (Group2, Group3) and FragmentZ (Group1, Group2). The refinement of the final list consists in determining the total number of occurrences of the correspondence groups inherent in the mentioned fragments in the text content of fragments of the selected subject. Priority is assigned to the fragment with the highest indicator of the number of occurrences of the correspondence groups in the selected topic. Since the number of occurrences of matching groups in the text content of fragments of the selected topic for Fragment2 is 15 (10 + 5), and for FragmentZ equal to 17 (7 + 10), the final form of the final list will be as follows: in the first place Fragment1, in the second FragmentZ, then Fragment2 , and closes the list Fragment4.

GO fragment The number of groups in the fragment The number of occurrences of groups in the subject GO Fragment! 3 22 GO FragmentZ 2 17 GO Fragment2 2 fifteen GO Fragment4 one 10

After the refinement sorting, Fragment2 and FragmentZ in the final list are swapped. In case of equality of indicators of the number of occurrences in the selected topic of the correspondence groups, for additional sorting, for example, weights assigned to the correspondence groups are used. In case of equality of all indicators, priority can be assigned by the date of the last access to the fragment, etc.

Next, at step 26, the final list of fragments is displayed to the user in accordance with the set parameters. As display parameters, for example, the minimum value of the degree of similarity index is used, at which a fragment included in the final list is displayed. Setting parameters for displaying the final list of fragments also includes limiting the number of displayed fragments. In this case, the final list is displayed, which contains a certain number of fragments that have the greatest degree of similarity with the selected one. Setting display options also includes a combined method. That is, as a display parameter, both an indicator of the degree of similarity and a limitation of the number of displayed fragments are simultaneously used.

The degree of similarity is determined as follows. Count the number of matching groups included in the set of unique words defined as search parameters. After that, the ratio of the number of matching groups in each of the fragments included in the final list to the number of matching groups in a set of unique words defined as search parameters is determined. The resulting indicator can be measured, for example, in percent. If there are ten matching groups in the set of unique words defined as search parameters, and eight groups in a particular fragment included in the final list, then the similarity coefficient for this fragment will be 8/10 = 0.8 or 80%. If, for example, a minimum indicator of 70% is set as display parameters, fragments included in the final list and having a degree of similarity with the selected fragment of less than 70% will not be displayed.

If the search for fragments having similarity with the selected one was carried out on remote devices, then after determining the degree of similarity, documents (fragments) that satisfy the specified display parameters are pumped out from remote devices. At the same time, a refinement sorting of the list of fragments received from remote devices is carried out. The refinement sorting is carried out on the basis of data obtained as a result of recounting the number of correspondence groups included in each mentioned fragment. The implementation of this operation is very important, because fragments received from remote devices may not match the search parameters due to poor-quality processing of the search query by remote devices. If the fragment received from the remote devices does not match the search parameters, the fragment is excluded from the final list. In addition, refinement sorting of the list of fragments received from the remote devices is carried out in accordance with the actual number of correspondence groups inherent in the obtained fragments.

Next, at step 27, fragments recorded in the archive are processed. Processing is carried out manually or automatically, in accordance with the established parameters. Setting parameters for the automatic processing of fragments recorded in the archive consists in determining the degree of similarity at which operations are performed on the above fragments: deletion, saving in the archive, auto-rubrication. That is, the user, for example, sets intervals for an index of the degree of similarity according to which fragments are processed.

For example, when searching on remote devices, documents were found that have a 100% indicator of the degree of similarity with the selected document stored in the archive. After the found documents are received (pumped out) from the remote devices, they are additionally checked to determine 100% similarity with the selected one. If after checking it is determined that the found and selected documents are really 100% identical, the received document is considered a take and deleted. If the similarity of the selected and received documents (less than 100%) is incomplete, the resulting document is displayed to the user, while visualizing all the differences between the text content of the received document and the selected one. For visualization of differences, use, for example, highlighting, highlighting, underlining, etc. When the received document is archived, it can be saved as a new version of the selected document. Moreover, the new version is saved either as a full copy of the received document, or in the form of differences in the text content of the received document from the text content of the selected (previously archived) document. Documents (fragments) that have a lower degree of similarity with the selected one, for example, in the range from 70 to 90%, can simply be stored in the archive.

The invention at step 27 provides for the automatic classification of fragments recorded in the archive of documents, which is implemented as follows. For the resulting fragment form many unique words. Each section of the classifier (subject) is also associated with many unique words, formed and updated as new fragments are tied to the subject. To determine the appropriate subject matter, to which it is advisable to bind the resulting fragment, a set of unique words inherent in the resulting fragment is compared with many unique words inherent in each subject. Based on the analysis, one or more topics are determined to which the resulting fragment can be attached. Binding is carried out both manually and in automatic mode, in accordance with the specified parameters. In automatic mode, an indicator of the degree of similarity is set, in accordance with which the expediency of linking the resulting fragment to a particular topic is determined. To determine the degree of similarity in automatic classification, not all unique words inherent to the topics can be used, but, for example, the rarest or vice versa, the most frequently used in the textual content of the topic.

The invention is illustrated by a simple implementation example.

As an example, let us consider the processing of information (news) received by mailing list and recorded in the archive. In the event that the user is subscribed to several newsletters, often the same news can completely or partially duplicate each other. In addition, the speed of receiving news in each newsletter is different, that is, identical news can come with a difference of several days. All this leads to the fact that the user has to re-view the same or identical information.

In this example, the present invention avoids re-viewing the news, as well as streamline the process of filling the archive with information coming from the mailing list. All incoming information is archived, and incoming documents are divided into fragments and indexed. In this case, the division of documents from a specific mailing into fragments occurs automatically in accordance with the rules established by the user. Setting rules is not a complicated procedure, since in each newsletter a fragment (news) is limited to a specific sequence of characters, for example, a specific heading. When determining these characters in a document, it automatically labels a specific area of the document, which is a fragment containing the news. In this example, a single technique is used to break documents into fragments (news) and index them. This means that the search for fragments similar in text and / or semantic content to the selected one will be performed by default among all fragments stored in the archive.

For each new fragment recorded in the archive, the archive is searched for fragments similar to it in textual and / or semantic content. For this purpose, each fragment is divided into source words, from which many unique words are formed for each fragment. At the same time, all stop words are eliminated, after which a list of unique words for the selected fragment is formed. For example, the news includes the text:

"The Moscow representative office of Atlant organized a press conference for journalists with the participation of the president of the company." The list of unique words for the selected fragment will include the words: “Moscow”, “representative office”, “company”, “Atlant”, “organized”, “journalists”, “press conference”, “participation” and “president”. Stop words 'for' and 's' are excluded from the list of unique words.

Each of the obtained unique words is pre-processed using a morphological processor. This allows you to search for news containing not only the mentioned unique words, but also their word forms. Those. fragments containing text of the form will be found: 'Atlant representative office opened in Moscow. The president of the company, who answered journalists' questions, took part in the organized press conference. ” Underlined are those words that are word forms of the corresponding unique words included in the selected fragment.

Thus, it is determined that the archive already contains news identical to the one selected. Moreover, the degree of similarity of the fragments is 100%, because in the selected and found fragments there are nine identical unique words (matching groups). Accordingly, the degree of similarity is 9/9 = 1 or 100%. In accordance with the parameters established for this example, the selected fragment (news) is entered into the archive, and its automatic classification is carried out. Automatic classification consists in linking the selected fragment to that section of the classifier (subject) to which the fragment found in the archive similar to it in text and / or semantic content is attached. All incoming news are handled in the same way.

Method for searching electronic documents stored on data storage devices

The method consists in indexing each document stored on the data storage device, determining search parameters, searching for documents on the data storage devices and ranking the results (lists of documents) in accordance with certain parameters. As the search parameters, two or more blocks of information are determined, the interval between them is indicated, as well as the order of their alternation in the desired document in the specified interval.

The generated search queries are expanded by pre-processing one or more blocks of information included in the original query, and a search is performed using a plurality of blocks of information generated by pre-processing. As the preliminary processing of the initial request, the operation of obtaining from at least one initial block of information, one or more blocks of information associated with the original block of information specified by the ratio is used. In this case, the search for documents is carried out using any given number of information blocks from among those defined as search parameters when generating the initial request.

As a block of information using a sequence of characters in the document, limited to certain characters. To split documents into many blocks of information using a software module for processing documents.

To extend the initial request, at least one pre-processing operation is selected, and the sequence of pre-processing operations is set by users. Moreover, the invention as the preliminary processing of the request provides for the use of a single logical operation of identity. After the pre-processing operation is completed, the initial block of information is either removed from the resulting query or left in the resulting query.

Documents matching the search parameters are searched on local and / or remote data storage devices. As remote devices, use any information resource or system designed to search for data, operating on a computer network and providing, in response to a search query, a list of documents that satisfy the search parameters.

The obtained search results (lists of documents) are ordered in accordance with the number of occurrences of documents of sequences of information blocks satisfying the search parameters and the length of the intervals in which the mentioned sequences are included. When arranging the list of documents, the weighting factors assigned to the information blocks are also taken into account.

The found fragments of documents are displayed with visualization of sequences of blocks of information corresponding to the search parameters and the request received after preliminary processing. The invention allows navigation through the text content of documents using the sequences of information blocks included in them within the entire list of documents obtained as a result of the search.

Description of the invention

This method involves forming a document archive on the data storage device in which all stored documents are indexed. To break documents into source blocks of information (source words), a software module for processing documents of a certain type is used. The use of document processing modules of a certain type for dividing documents into source words allows full-text search of documents of various formats with text content. The invention provides for the formation of software modules for processing documents of a certain type by users.

The method is implemented as follows. At step 28, as shown in FIG. 9, form an initial query consisting of at least two source words. In addition, at step 28, search parameters are determined, which are used as

- an indication of the maximum interval in which the source words should be in the search phrase in the document,

- an indication of the alternation of the mentioned source words in a given interval. The order of alternation of the source words can be specified either arbitrary or rigidly defined, depending on the user's preferences,

- determination of the minimum number of words from the number of source words included in the query for which the search operation will be performed. For example, the original query consists of four words, but they search for documents containing phrases of at least three of any words included in the original query. Accordingly, documents containing phrases with both four words and combinations of any three source words will be found,

- Definition of words that must be present in the searched phrase in the document without fail (required words). For example, for a query with the words 'necessary information', the word 'information' is mandatory for inclusion in the searched phrase. Thus, phrases that include either the words 'necessary' and 'information', or only the word 'information' will be defined as the ones searched. Phrases that include only one word 'necessary' (optional word) for this example will not be defined as searched

- indication of weights for the words included in the original query. Weighting factors determine the significance of words and will be used in the future to determine the relevance of the documents found to the original query when forming and organizing the final list of documents found.

At step 29, the source words to be pre-processed are determined and pre-processed.

Pre-processing expands the search query with various analogues, which helps to search for documents containing phrases that are relevant to the original query, not only in formal but also in semantic meaning. Thus, at step 29, correspondence groups are formed for each of the source words. For example, in the process of pre-processing the word 'information' using the synonym dictionary, a correspondence group is obtained that includes the words 'information', 'information' and 'message'. A special case of a preprocessing operation may be one logical identity operation. Moreover, for the word 'information' a group will be formed consisting of one identical word, i.e. the words 'information'. The identity operation is performed for those source words that were not determined at step 29 as being subject to preprocessing.

Next, at step 30, lists of documents are generated for the correspondence groups associated with the words included in the search query, after which they proceed to step 31. At step 31, it is determined whether the required words are included in the query.

If the required words are included in the query, they proceed to step 32, where the list of documents generated for the correspondence group associated with the first required word in the search query is entered into the final list of documents for required words (ISO). Next, at step 33, it is determined whether all the required words included in the search query have been processed. If not, then at step 34 they turn to the correspondence group associated with the next required word in the search query and assign it the current status. Then, at step 35, the final list of documents is generated using the rule: ISO = ISO 'ΛΝΏ' SD of the current compliance group '

Those. at step 35, documents are entered into the ISO from the list received for a particular compliance group associated with the required word included in the search query. Moreover, only those documents that are simultaneously presented in ISO and in the Board of Directors of the current compliance group are included in the final list.

Steps 34 and 35 are performed until at step 33 it is determined that all the required words from the search query have been processed. After that, the final list of documents for required words will be generated and in the field for counting the number of matching groups for each document in ISO will be written a value equal to the number of processed required words. The ISO formed at step 35 includes documents in which all required words are presented.

Next, proceed to step 36, which determines whether optional words are included in the request. If there are no optional words in the request, then at step 37 they form the final list of documents (IS), which includes the list of documents obtained for the required words (IS = ISO).

If optional words are present in the request, then at step 38 a list of documents for optional words is generated using the algorithm for generating a list of documents for words combined in the request by the 'OK' operator. Then, at step 39, on the basis of ISO and ISN, a final list of documents (IS) is formed. The final list will include all documents from ISO. At the same time, for documents of the final list, which are simultaneously presented in ISO and in the ISS, they count the number of compliance groups. Those. the calculation of the number of groups is carried out for those documents in which at the same time there are obligatory and optional words. Documents that contain only required words will be presented in the final list with an indicator of the field value for counting the number of correspondence groups obtained for them during the formation of the ISO.

When counting the number of conformity groups, the indicators of the field values are summarized to calculate the number of conformance groups of the document included in the ISO and the corresponding document from the ISN. For example, a query consists of the phrase 'search for necessary information', and two words - 'search' and 'information' are required. Document1 contains both required words, and therefore, when forming the ISO, the value of the field for counting the number of conformance groups Document1 will take two values (by the number of required words).

In addition to the two required words, Document1 also contains the optional word 'necessary' and therefore will be included in the IOS. Moreover, the value of the field for counting the number of correspondence groups for Document1 included in the ISN will be one (one optional word).

Accordingly, at step 39, after the formation of the IS, the value of the field for counting the number of correspondence groups in Document1 will be three (2 + 1). The obtained indicators of the values of the number of conformity groups inherent in the documents included in the IP are taken into account when organizing the final list.

If at step 31 it is determined that the request contains no required words, then they proceed to step 40, where they form a final list of documents (ISN) for the optional words included in the request (all words included in the request) using the algorithm for generating the list of documents for words combined in the query by the 'OK' operator. Then form the final list, which includes a list of documents obtained for optional words (IS = ISN).

After the final list of documents for words included in the initial query is generated at steps 37, 39 or 40, they proceed to step 41, where the final list of documents is organized, processed and displayed.

If the search for documents was carried out in a stored archive, then the final list includes documents that match the search parameters, and such a list does not need additional processing. Since the formation of the final list of documents received from the archive is accompanied by its optimization using additional indexes (checking intervals, word sequence, etc.). If the search for documents was carried out on remote devices, then the final list of documents is pumped from the remote devices and checked for compliance of the documents included in it with the search parameters. In this case, pumping out full copies of documents found on remote devices is carried out only once - after the final list of documents has been generated. Checking documents received from remote devices is very relevant, because the mentioned documents may not match the search parameters due to poor-quality processing of the search query by remote devices. In the event of a complete discrepancy received from the remote devices of the document with the search parameters, the said document is excluded from the final list. That is, those documents that do not contain a single phrase satisfying the search parameters are excluded from the final list.

For example, a search condition is to impose a restriction on the interval between the search words included in the initial query 'ba1a tapadet duDet' - no more than two (ΝΕΑΚ 2). In this case, preliminary processing of the request is taken into account, as a result of which the documents sought include the words 'ba1a'. 'tapadetepG and' duDet 'and' 4a1a ',' sop! go1 'and' duDet '. Thus, the following documents will be included in the final list of documents. A document1 that contains the phrase '... sottyssaOopkh duduet, 111egta1 sop! Go1 Boagb apb ba1a soogoG and a Document that contains the phrase' ... ipyryyeye 1aa4eg o £ 4a1a tapadete duDet tag! '. And although the texts from both documents contain words from the request, including those expanded by preliminary processing, the first document (Document1) does not correspond to the request parameters (the interval between the information blocks is no more than two). Since the phrase '... ш с д д д,,,,,,,,,, соп ег ег, 1 1 1 1 аг аг аг аг аг' '' '' '' не не не не не не не не не не не не не не не не не не не не не не не не не документ не не не не документ не документ не документ документ документ документ документ документ документ документ документ документ документ документ документ документ документ документ документ документ документ 111 111 111 111 111 111 111 111 111 111 111 111 111 111 111 111 111 111 111 111 111 111 111 111ет 111 Similarly, they check for compliance with the search parameters in the case of indicating a clear sequence of alternating words in the desired document. For example, if '4a1a, ta ^ det, duDet' or '4a1a, sogo1, duDet' is indicated as a clear sequence, then Document1 will be excluded from the final list by this criterion, since the given sequence of alternating the searched words is violated in it.

At step 41, the final list of documents received during phrasal search is sorted in accordance with the relevance of the documents with respect to the search query received taking into account the preliminary processing. This takes into account the number of searched sequences of processed words and the length of the intervals in which the mentioned sequences are included. In addition, they take into account the order of alternating words in the query, as well as additional parameters set by the user to search for documents containing phrases with the number of words Ν or less. The number of words is determined by the indicator of the number of matching groups inherent in the documents. When organizing lists, priority is given to those documents that include phrases containing all Ν words and then in descending order (Ν-1, Ν-2, etc.). If the number of words in the phrases present in the documents is equal (for example, Ν), priority is given to the document in which the words occur in the minimum interval. For example, if the final list is formed taking into account the interval between words ten and an arbitrary alternation of the words in the search interval, then both documents from the above example (Document1 and Document2) will correspond to the specified parameters. But at the same time, for Document1, the word spacing is five, and for Document2, the word spacing is two. Thus, Document2 will be placed first in the final list. When calculating the interval in this case, the opportunity to exclude the so-called stop words: prepositions, etc., was not used. However, the present invention provides the ability to exclude stop words from the sequences of the searched words when determining the interval between them.

If the minimum interval between the searched words is the same for several documents included in the final list, for them, the sequences of the searched words included in these documents are counted. In this case, the word sequences with the minimum matching interval are first counted and, if this indicator is equal, they proceed to counting the number of sequences having the interval of 'minimal matching interval +1'. If necessary, this operation is carried out until the interval value reaches the maximum value determined by the specified search parameters. That is, if the documents contain the same number of sequences of words with an interval of two, they go on to count the number of sequences with an interval of three, etc.

As additional parameters for organizing the final list, weights are used assigned to the searched words. In this case, priority will be given to the phrase having the highest total weight coefficient, which is formed, for example, as the sum of the weight coefficients of all the words included in the searched phrase. If all indicators are equal, the documents in the final list can be sorted in accordance with the settings. For example, priority can be assigned by the date of the last access to documents, etc.

The final list of documents may include documents that do not contain all the words defined as search parameters. Such a situation arises when both required and optional words are included in the initial query and the minimum number of words that must be present in the searched phrases is determined as search parameters. In this case, the final list may include documents that do not match the search parameters. For example, the original query includes seven words, three of which are identified as mandatory for inclusion in the search phrase. At the same time, the minimum number of words that must be present in the searched phrases, for example, five words, is specified as a search parameter. But the final list of documents includes documents containing phrases with any number of words in the range from three to seven (three required words and any number of optional words). Those. the final list includes documents containing phrases with the number of words three and four, which does not match the query parameters. Such documents at step 41 will be excluded from the final list of documents obtained by phrasal search.

Several phrases combined by the Boolean operators 'OK', 'ΑΝΏ', etc. can be included in the original query. In this case, the above phrase search algorithm is performed for each of the phrases included in the original query. After that, the final lists of documents obtained for each of the phrases are subjected to combination operations using the corresponding mentioned Boolean operators.

Next, at step 42, the documents of interest to the user from the final list are displayed with the visualization of all sequences of searched words found in them. When visualizing, query parameters are taken into account (the interval between words and the order of their alternation). As visualization, highlighting of search results, underlining, highlighting, etc. are used. Visualization of the search results immediately addresses the user to the desired place in the document. Moreover, the invention provides for the possibility of navigation through the document (by the sequences of the searched words visualized in the text of the document), and navigation is carried out within the entire list of documents.

The invention is illustrated by a simple implementation example.

There is an archive of documents received from diverse information sources. Moreover, documents of various formats are stored in the archive: web pages received from the Internet via links provided by the Internet by search engines in response to search queries, documents of the format М1сго§ой О £ ісе (\ Уугб. Ехсе1), obtained from the hard drives of computers functioning in the local network and other documents containing textual information. All documents stored in the archive are indexed.

The user is interested in obtaining a list of documents stored in the archive containing information about information technology, and more specifically about conferences dedicated to this topic. To obtain the list of documents of interest to the user, a request is made for the 'information technology conference' taking into account the interval between words and the order of alternating words in the request (phrasal search). As the search parameters, the interval between the words is set to four and an arbitrary order of alternating words in the searched documents. The setting of such an interval between words is caused by the fact that the desired phrase can be in the form of a 'conference on modern information technologies' and if a smaller interval is set, a document containing such a phrase will not be detected. All words in the original query are identified as required. The mentioned request is converted to a request of the form:

'(conference, information, technology) / YEAK (4)',

those. the stop word 'by' is excluded from it. In order to increase the effectiveness of the search, preliminary processing of the generated request is used, which consists in using two processors: a synonym dictionary and a Russian-English dictionary for translation. At the same time, all the words included in the request are subjected to preliminary processing. The sequence of pre-processing operations is set by the user and involves first of all the use of a semantic thesaurus, and then the Russian-English translation dictionary.

As a result of the preliminary processing operations, correspondence groups are formed for each of the words included in the initial request. Thus, for this case three groups of correspondence will be formed, for example: [conference, seminar, eGGsGspes. hettag]; [informational, ίηίοΓshayoi]; [technology, 1ec11po1ofekh |. Within each group, words are combined by the Boolean operator 'OH'. The final request will be as follows:

((conference OH seminar OH eGegeise OH zetstep), (information OH tGogtayop), (technology OH 1сппо1од1е8)) / НЕАЯ (4)

The final list of documents includes documents containing at least one of the following phrases (phrases): 'conference on modern information technologies' (Document1), 'seminar on information technologies' (Document2), 'tGogshaNop 1esNpo1o§1ekh sopGgepse' (Document3) or 'tGogshaNop 1es11po1oshche5 Hetta g' (Document4), etc. Moreover, the interval between the words included in the phrase should not exceed four.

The resulting list is sorted according to the search parameters and displayed to the user. At the same time, a document containing the phrase 'tGogtaiop 1ecNpo1od1ekh hittag' (Document4) will have a higher rating (place in the list) than a document containing the phrase 'conference on modern information technologies' (Document1). Since the spacing between words in Document4 is less than the spacing between words in Document1 (2 and 4, respectively). For documents containing phrases with the same spacing between words - Document3 and Document4, the number of phrases matching search queries and having an interval between two words is counted. At the same time, it is determined that Document3 contains one more phrase and thus is assigned a higher rating. The final form of the displayed list of documents will be as follows: in the first place is Document3, and then in order of priority are Document4, Document2 and Document1.

Next, the user accesses the document of interest to view it. When a document of interest to a user is displayed, the search results corresponding to the query received taking into account the preliminary processing are visualized. Thus, when accessing, for example, Document2, the entire sequence of words that matches the search parameters and the query received after preliminary processing is highlighted:

'seminar on information technology', although the word 'seminar' was not included in the original request: '(conference, information, technology) / ЕЕАК (4 ·).

The considered examples of the implementation of the inventions are just one of many possible examples that do not exclude other variants of implementation, and people with skills to work with the present inventions can make changes to them.

Other implementations or varieties of implementations of the inventions are within the scope of these inventions, which are defined by the patent claims.

Industrial applicability

The invention relates to methods for searching fragments similar in text and / or semantic content in electronic documents stored on data storage devices. The invention is also intended to search for documents by query, consisting of two or more words, extended semantic analogues, taking into account the interval between the words and the order of their alternation in the query.

The invention can function in various communication and computer networks, for example, local computer networks, the global Internet, etc. The invention can also be used in distributed computing systems where tasks are performed by remote computing devices that are integrated by a communication network, including in application programs running on various data storage devices.

The industrial applicability of the invention is due to the optimization of the search for documents (fragments) of similar subjects, including those having similarities not only in formal but also in semantic meaning and in eliminating duplication of information stored in the archive. Moreover, the effectiveness of search operations is determined by the ability to choose a method for generating a set of words for a search query, using certain rules, including those established by the user. The industrial applicability of the invention is also due to the optimization of the information processing by automatic classification of documents recorded in the archive.

This invention is applicable to systems related to the receipt, retrieval, processing and storage of information in computer systems, and improves the efficiency of working with information stored on data storage devices.

Advantages of the Invention

The present invention, in comparison with existing analogues, has a number of advantages that can significantly reduce the time required for obtaining and processing documents of interest to the user and their fragments. The invention allows you to search for documents (fragments) that are similar to the one selected for textual and / or semantic content using both the whole set of words included in the selected document (fragment) and their established number. At the same time, the efficiency of search operations is increased due to the possibility of choosing a method for generating a set of words for a search query, using certain rules, including those established by the user.

The invention determines the degree of similarity of the selected and the documents found in the archive or on the remote devices and automatically classifies the documents (fragments) when they are entered into the archive stored on the data storage device.

The invention also allows to optimize the search for documents by queries consisting of two or more words, taking into account the interval between words and the order of alternating words in the query, by expanding the aforementioned queries by preliminary processing the words included in it. This allows you to create a list of documents relevant to the request not only in formal but also in semantic meaning. Moreover, when displaying documents from the resulting list, they visualize the results corresponding to the search parameters and the request received after preliminary processing.

Claims (36)

CLAIM 1. The method of searching for fragments similar in text and / or semantic content in electronic documents stored on data storage devices, which consists in indexing each document stored in the archive, splitting said documents into fragments and forming topics from one or more fragments, determining search parameters, conducting a search, ranking the list of document fragments obtained as a result of the search, characterized in that as the search parameters, a plurality of the fragments included in the selected fragment are determined This is a document element of unique information blocks and is expanded by preliminary processing of each of the mentioned unique information blocks, where a unique information block is understood to mean an information block that occurs one or more times in a selected fragment of a document, where the operation of obtaining at least from one unique block of information, one or more blocks of information associated with a unique block of information with a given ratio. 2. The method according to claim 1, characterized in that the sequence of characters included in the document limited to certain characters is used as the information block. 3. The method according to claim 1, characterized in that as a fragment of the document use any selected sequence of information blocks included in the document. 4. The method according to claim 3, characterized in that a full copy of the document is used as a document fragment. 5. The method according to claim 3, characterized in that they establish the rules for splitting documents into fragments. 6. The method according to claim 5, characterized in that the document is divided into fragments using at least one rule. 7. The method according to claim 5, characterized in that the division of documents into fragments is carried out automatically. 8. The method according to claim 1, characterized in that the plurality of document fragments, among which search for fragments similar in text and / or semantic content to the selected one, is limited to indicating at least one rule by which documents were divided into fragments and / or subject matter. 9. The method according to claim 1, characterized in that the subjects are formed by users. 10. The method according to claim 1, characterized in that for the division of documents and document fragments into many blocks of information using a software module for processing documents. 11. The method according to claim 1, characterized in that at least one pre-processing operation is selected, wherein the sequence of pre-processing operations is set by users. 12. The method according to claim 11, characterized in that the preliminary processing consists of one logical operation of identity. 13. The method according to claim 1, characterized in that after the preliminary processing, at least one source block is removed from the request or left in the request. 14. The method according to claim 1, characterized in that the set of unique blocks of information defined as search parameters include all unique blocks of information included in the selected fragment of the document. 15. The method according to claim 1, characterized in that many unique blocks of information, defined as search parameters, are formed using one or more functions. 16. The method according to clause 15, characterized in that use the function of forming a list of unique information blocks for the selected fragment. 17. The method according to clause 15, characterized in that they use the function of determining the number of occurrences in the text content of fragments of the selected subject information blocks obtained in the process of pre-processing a unique block of information. 18. The method according to p. 15, characterized in that they use the function of determining the frequency of occurrence of unique blocks of information in the text content of fragments of the selected topic, where the frequency of occurrence is calculated as a percentage of the number of occurrences of the unique information block most often used in the text content of the selected topic. 19. The method according to claim 1, characterized in that the set of unique blocks of information defined as search parameters include a set number of unique blocks of information from the set generated by user-defined rules. 20. The method according to claim 1, characterized in that the search for fragments of documents is carried out on local and / or remote storage devices. 21. The method according to claim 20, characterized in that as the remote devices use any information resource or system designed to search for data, operating on a computer network and providing, in response to a search query, a list of document fragments that satisfy the search parameters. 22. The method according to claim 1, characterized in that the ranking of the resulting list of document fragments is carried out in accordance with the number of information block groups present in the found document fragments, combining information blocks obtained in the process of preliminary processing of a unique information block. 23. The method according to p. 22, characterized in that when ranking for those document fragments in which the number of groups of information blocks coincides, the number of occurrences in the text content of fragments of the selected subjects of the mentioned groups of information blocks is additionally determined. 24. The method according to claim 1, characterized in that they display the fragments included in the list of document fragments obtained as a result of the search, with visualization of the differences between their text content and the text content of the selected fragment. 25. The method according to claim 1, characterized in that the archived document is saved as a new version of a document previously stored in the archive that has a given degree of similarity in text and / or semantic content with the archived document, and wherein said new version is saved in the form of a full copy of the document entered into the archive or in the form of differences between the text content of the archived document and the text content of the aforementioned document stored earlier in the archive. 26. The method according to claim 1, characterized in that they automatically classify fragments stored on the document data storage device. 27. The method of searching for electronic documents stored on data storage devices, which consists in indexing each document stored in the archive, determining the search parameters for electronic documents, including generating an initial request from two or more blocks of information, indicating the maximum number of blocks of information that may be between the two two or more blocks of information in the search document (interval), as well as the alternation of the above two or more blocks of information in the search document in the specified interval, ranking the resulting list of documents, characterized in that the initial request formed from two or more blocks of information is expanded by pre-processing one or more blocks of information included in the original request, where the operation of obtaining is used as preliminary processing of the initial request, at least one block of information included in the original request, one or more blocks of information associated with the source block of information specified In this way, the search for documents is carried out using any given number of information blocks defined as search parameters when forming the initial request. 28. The method according to p. 27, characterized in that as a block of information used in the document a sequence of characters limited to certain characters. 29. The method according to item 27, wherein in order to split the documents into many blocks of information using a software module for processing documents. 30. The method according to item 27, wherein at least one operation of the preliminary processing of the initial request is selected, and the sequence of operations for the preliminary processing of the initial request is set by users. 31. The method according to p. 30, characterized in that the preliminary processing of the original request consists of one logical operation of identity. 32. The method according to item 27, wherein after the pre-processing operation, at least one block of information included in the original request is removed from the request or left in the request. 33. The method according to item 27, wherein the search for documents is carried out on local and / or remote data storage devices. 34. The method according to p. 33, characterized in that as the remote devices use any information resource or system designed to search for data, operating on a computer network and providing, in response to a search query, a list of documents satisfying the search parameters. 35. The method according to item 27, wherein the resulting list of documents is sorted in accordance with the number FIG. one FIG. 2 by entering document sequences of information blocks that satisfy the search parameters, the length of the intervals in which the mentioned sequences are included, and taking into account the weighting factors assigned to the information blocks. 36. The method according to clause 35, wherein the documents are displayed with visualization of sequences of information blocks corresponding to the search parameters and the query received after preliminary processing, and navigation is made through the text content of documents using the sequences of information blocks contained in them the limits of the entire list of documents obtained as a result of the search.