EA037156B1 - Method for template match searching in a text - Google Patents

Abstract

The invention relates to a method for template match searching in a text and ensures rapid detection of a set of concepts, entities and relations in natural language texts. The technical result is higher search rate, completeness and accuracy, ensuring a safe and language-independent searching. A text being searched is tokenized. Using the template description language, a set of templates is created as sequences, variations and repetitions of text tokens and occurrence of other templates. The set of templates is translated to search expressions with search indexes. Search of all template matches is carried out by a single sequential scanning of text tokens. Search expressions beginning with this token are retrieved for each token in search indexes, match candidates are created for corresponding templates and are introduced into the set of candidates. As screening the tokens, matches and mismatches are removed from the set of candidates.

Description

LIMITED LIABILITY COMPANY FORGET SOFTWARE (BY) (72) Inventor:

Surkov Dmitry Andreevich, Surkov Kirill Andreevich, Chetyrko Yuri Mikhailovich, Shimko Ivan Vladimirovich, Savenok Vladislav Alexandrovich (BY) (74) Representative:

Surkov D.A. (BY) (56) US-A1-20130080886 WO-A3-2008103398 US-B2-7054855 US-A1-20110295595 US-B2-8155946 RU-C2-2605077

037156 B1

037156 Bl (57) The invention relates to a method for searching the text for matches with patterns and provides a quick identification of a set of concepts, entities and relationships in natural language texts. The technical result is to increase the speed, completeness and accuracy of the search, to ensure the safety of the search and independence from the language. The text being searched is parsed into tokens. In the template description language, a set of templates is created in the form of sequences, variations and repetitions of text tokens and occurrences of other templates. A set of templates is translated into search expressions with search indexes. The search for all matches of all patterns is performed in one sequential scan of the text tokens. For each token in the search indexes, search expressions starting with that token are searched for, match candidates for the corresponding patterns are created, and they are entered into a set of candidates. As the tokens are scanned, matches and mismatches are removed from the set of candidates.

The technical field to which the invention relates

The invention relates to a method for searching for matches with patterns in the text and provides a quick identification of a set of concepts, entities and their relationships in natural language texts.

Prior art

Currently, many application software systems designed to automate various branches of human activity include tools for analyzing text in natural language. Sources of text data in such systems are databases, document repositories, publications on sites on the Internet, blog posts and forums, e-mail messages, social media messages, transcripts from call centers, messages of conversations in text communication programs, and others. systems. The task of analyzing text data in such systems is to identify concepts and entities in the text, as well as to determine the relationship between them. The analysis of textual data is often accompanied by the stage of enriching the textual data with meta information about the found concepts, entities, and relationships, collectively called tags. The process of enriching text data with tags is known as text tagging.

The task of analyzing text data is currently being solved in various ways, among which the most famous and used are full-text search, full-text indexing, machine learning methods, latent semantic analysis, analysis based on regular expressions, analysis based on formal grammars with or without regard to word morphology, as well as various combinations of the above methods.

An early approach to solving the problem of text analysis is full-text search - an automated documentary search for given combinations of keywords and expressions in the full text or in a significant part of the text of the document. In the process of full-text search, the following preparatory stages are usually present: parsing the text into words, removing noise words, highlighting word stems, converting words to dictionary form, performing statistical analysis. In this case, the morphology of the language of the analyzed text is often taken into account. For parsing text into words, approaches of varying degrees of complexity are used, among which the Unicode Standard Annex # 29 is known - it defines the most accurate rules for parsing text into sentences and words.

To significantly speed up the full-text search in the case of its application to large amounts of text data, full-text indexing is performed, i.e. create a search index. A search index is a data structure called a dictionary that stores all the words in a text along with information about where they appear in which documents. Full-text indexing is not an obligatory stage of full-text search, which can be performed by finding a substring in a string using well-known algorithms, however, it can significantly reduce the area in which the search is performed, and therefore significantly speed up the search compared to direct search in the text of source documents. When building a search index, so-called noise words are usually ignored - words that are contained in almost all documents. Such words often include prepositions, participles, interjections, particles, etc. Noise words are usually excluded from search queries.

To reduce the variability of word forms in a natural language, stemming is used - the process of finding the basis of a word for a given source word. Stemming eliminates the need to list word forms in queries and search rules, however, the ambiguity of words and imperfect stemming algorithms lead to an increase in the number of false matches.

Statistical analysis is often used to highlight keywords in the full-text indexing process. It involves counting the frequency of use of words in the text and highlighting the most frequently used words. A key component of this approach is the mathematical model that is used to calculate the frequency of use of words.

Full-text search is characterized by high performance and good scalability in a distributed computing environment. However, this approach is ineffective in solving complex problems of text analysis due to the low search accuracy and the lack of the ability to search for concepts, entities and their relationships, as it is focused on the search for keywords. The indicated drawback does not allow using full-text search for text tagging.

Currently, machine learning methods are widely used to analyze text in natural language. These methods are based on the classification of objects represented by descriptions in a specific feature space. There are two groups of machine learning methods: supervised learning and unsupervised learning. In the first group, the goal of training is to obtain rules that can be used to classify new objects based on those used for training. In the second group, there is no training sample, and the task is clustering - combining objects into groups based on a given measure of their similarity or difference. Machine learning is widely used to highlight keywords, identify entities and relationships, and recognize emotions, subject matter, and text language.

Among the machine learning methods, there are those that are focused on building a subject ontology with information about objects and their relationships. The objects are 1 037156 nouns, and the relations are the verbs with which the nouns are associated. There are known machine learning methods that reveal such semantic dependencies using mathematical models of neural networks.

Among the methods for analyzing text in natural language, latent semantic analysis is known, which solves the problem of finding the similarity of texts through the similarity of the meanings of words in the context of their use. The meanings of words are considered similar if the words are used in similar contexts.

The application of machine learning methods requires the following preparatory stages of text processing: parsing the text into lexemes, normalizing words, and morphological analysis. This is followed by the phase of grammatical analysis, during which machine learning methods are actively applied. Grammar analysis is divided into superficial and complete. Superficial analysis is able to reveal some semantic components in the text: nominal, verb and prepositional groups. A complete grammatical analysis presents the structure of the entire sentence in the form of a grammatical tree. In the course of full parsing, ambiguities often arise, the solution of which is a rather time-consuming task. In addition, from a practical point of view, there is no need for complete parsing to obtain an acceptable analysis result.

The main disadvantage of machine learning methods, which follows from the essence of this approach, is the impossibility of explaining the results obtained. This is due to the nature of mathematical models and the learning process of the classifier, which occurs through complex modifications of numerical coefficients based on the training sample. Another disadvantage is the complexity of training the system to recognize new entities and concepts, firstly, it requires a sufficiently large amount of new data for training; secondly, as a result of learning to recognize new concepts, the recognition accuracy of concepts added earlier may decrease. Finally, a significant disadvantage of machine learning methods is low performance. Both initial training and classification are time consuming, which makes it difficult to use this approach for quickly tagging text, in particular for repeatedly tagging a large number of documents in a database.

Currently, regular expressions are widely used for text analysis, which are a means of searching for patterns in text in both machine-oriented and natural languages. This approach is characterized by good explainability of results and relative freedom in specifying search patterns. The basic concepts of regular expressions include character literals, as well as their sequences, alternations, and sets. Most modern regex implementations additionally allow quantifiers to be specified, forward and backward lookups, and character class searches.

The main drawback of regular expressions is the lack of optimization for finding many patterns in a single pass through the text. As the number of templates increases, performance decreases linearly, since searching for each template requires a separate look at the text. Another disadvantage of this approach is working with text data at the level of characters, not words or tokens, as well as the impossibility of using links to other templates, including recursively defined templates, in templates, which reduces the speed, accuracy, and completeness of natural language text analysis. ... Another drawback of regular expressions is the existence of certain input data sets for individual templates, on which iterating over all matches leads to frequent backtracks in the text and a strong slowdown in the search, which is perceived by the user as an endless loop. A significant drawback of regular expressions is also the low readability of template descriptions - when writing an expression, there is a high probability of making an error, it is difficult to understand previously written templates.

More recent approaches to analyzing natural language text are based on the use of sets of formal grammars. The rules for defining formal grammars assume the use of one nonterminal symbol on the left side of an expression, and terminal and nonterminal symbols connected by some operators on the right side. Terminal symbols (terminals) include objects, words and symbols of a language that have a specific symbolic meaning. Nonterminal symbols (nonterminals) include objects that designate some entity of the language and do not have a specific symbolic meaning. Operators link terminals and nonterminals in sequence (chains), sets of alternatives (variations), repetitions, and other constructs.

A famous example of formal grammars is the Backus-Naur form. In this formal system, on the right side of the rule, it is allowed to indicate the sequence and variability of terminals and non-terminals. Also known are a number of modifications of the Backus-Naur form that allow you to describe repetitions and optional elements. In them, it is possible to specify both the possibility of skipping or repeating a separate part of the expression an arbitrary number of times, and the exact number of repetitions, indicating the range of values of the repeater.

Although the syntax of Backus-Naur grammars is easier to comprehend than the syntax of regular expressions, grammars of this kind are oriented towards parsing machine-oriented text in a root non-terminal and are poorly suited for the tasks of parsing and tagging natural language text.

A well-known example of the use of formal grammars for analyzing natural language texts is the JAPE template language and a system based on it. The JAPE language is a rule language for performing textual markup in the form of annotations attached to continuous fragments of processed text. Text analysis involves the following steps:

parsing text into lexemes: numbers, punctuation marks, symbols, spaces. The token type is saved to the corresponding annotation;

selection of named entities in accordance with their lists in a text file using the so-called gazetteer. Each list contains a separate set of entities: organizations, cities, names, and so on;

splitting the text into sentences. Sentence highlighting uses a list of gazetteer abbreviations to distinguish the end of a sentence from other uses of punctuation;

splitting text into sentences using regular expressions;

definition of parts of speech. A built-in set of rules is used, the identified parts of speech are saved in the annotation;

semantic annotation using the formulated rules in the JAPE language and annotations obtained as a result of the previous stages.

A JAPE grammar consists of a sequence of stages, each of which is a set of rules describing the actions performed for a certain text template. Rule sets form a cascade of annotation converters that run sequentially. The left side of the rule lists the template, and the right side lists the annotation transformation operators. Annotations matched in the pattern from the left side of the expression can be used on the right side by referencing the labels that are associated with the elements of the pattern. The left side of the rule is a template from annotations, in which you can use properties from other annotations, special properties containing the directly annotated text string and its length, as well as comparison operators and regular expressions for strings. Patterns can be combined in sequences and alternatives, as well as set repetitions of pattern elements. You can specify multiple combinations of template and annotation actions in one rule. To do this, templates are assigned unique labels that are used later on in the right side of the rules. Labeled templates can be nested and annotated hierarchically. On the left side of the rule, the use of so-called macros is allowed, i.e. wildcard elements that can be reused when writing rules. On the right side of the rule, it is permissible to use code in the Java programming language that allows you to implement complex logic of operations on annotations. This code is included in the grammar-generated Java program class.

The approach to text analysis based on the JAPE language is quite universal and extensible, but the price for this is the cumbersomeness of the resulting search patterns and the high complexity of the entire system. The most significant drawback is the focus on multi-pass sequential rule checking, which is confirmed by the use of regular expressions and Java program code, and as a result, a linear decrease in performance with an increase in the number of patterns, as well as insecurity in the sense of the possibility of too deep recursions and (conditionally) infinite looping of the search ...

The known approach for the analysis of text in natural language, taking into account the morphology. An example of the implementation of this approach is the language of lexical-syntactic templates LSPL and a system based on it. The LSPL language is a declarative language for the specification of lexical and grammatical properties of constructions highlighted in Russian texts. A template in this language is specified by indicating its name, followed by an equal sign, in the general case, a description of several alternative variants of the formalized language construction. The | character is used to separate alternatives. Different templates must have different names, and an arbitrary sequence of letters must be used as the name, with the first letter in uppercase. The sequence of elements in the template strictly corresponds to their location in the described construction. Elements of a template can be a word element, a string element, a template instance, a repetition of elements, and a set of alternatives. The first three types of elements belong to simple elements, and the rest - to complex ones, since they include other elements. Optional elements, repetitions, and alternatives are similar to operators in JAPE and augmented Backus-Naur form. A string element allows you to write a specific string of characters in a pattern, such as a specific word form, punctuation, or symbol. Regular expressions can be used on such strings. At the same time, it is not allowed to specify the morphological features of words in the elements of the lines. A word element corresponds to a single word, for which the following attributes are indicated:

part of speech: in this case, symbolic designations are used: N - noun, V - verb, A adjective, Pr - preposition, Pn - pronoun, etc .;

lexeme name - the initial form of a word, specifying the sets of all word forms of this word;

- 3 037156 morphological characteristics of the word: case, gender, number (singular or plural), etc.

Each part of speech corresponds to its own set of morphological characteristics, and some of them are fixed, for example, the gender of nouns, while others are changeable, for example, the case of a noun. LSPL allows the use of arbitrary words in templates, denoted by the letter W. Specific values of the changed characteristics can be written in angle brackets after the token name. For example, the following pattern specifies the adjective red in the feminine nominative:

A <(red), c = nom, g = fem>

here A is an adjective (from the English adjective), then in angle brackets the characteristics of the adjective, where red is the initial form of the word, c = nom is the designation of the nominative case (c from the English case, n from the English nominative), g = fem - designation of the feminine gender (g from the English gender gender, fern from the English feminine - feminine).

In the template, it is allowed to set the repetition of elements: in curly braces, elements are indicated that can occur in the text several times in a row. For repetitions, you can specify the maximum and minimum multiplier in angle brackets after the pattern. So, the entry {A} <1,3> denotes an adjective that can be repeated from one to three times. If the second value is not specified, only the minimum number of repetitions is considered committed. It should be noted that this approach to using the default value is not entirely obvious and requires mandatory clarification for the user. It is possible to specify optional (optional) elements by placing them in square brackets. Such notation is a shortened version of repetition from zero to one time: for example, the [not] element indicates the optional match of a string not with a piece of text. Optional elements and repetitions can consist of a set of alternatives, each of which, in turn, can be a sequence.

Matching conditions determine the grammatical agreement of individual elements of the template and are written in angle brackets after all matched elements. The condition of agreement is expressed in the form of equality of individual or all common morphological characteristics of the words being matched. For example, the following pattern describes a pair of words that are consistent in number and gender - a pronoun and a verb:

PV = Pn V <Pn.n = V.n, Pn.g = V.g>

Here Pn is a pronoun (from English pronoun), V is a verb (from English verb), then in angle brackets there follows a description of the conditions of agreement, where Pn.n = Vn (n from English number is a number) specifies the agreement of the number of the preposition and verb, a Pn.g = Vg - gender.

In such a record, compound names are used, formed from the name of the template element and the name of the matched feature, which are separated by a period. In the case when all the general morphological characteristics of two elements of the template must be agreed upon, the conditions for matching can be written using only the names of the matched elements, without listing the morphological characteristics.

In the case when the template contains several elements and several matching conditions, they can be written in turn one after another. The only limitation is that each matching condition in the template must not precede the writing of the matching elements themselves.

When describing a template, part of the morphological characteristics of words can be transferred to the template parameters, which are written after all its elements and conditions. In the parameters, you can specify only the morphological characteristics of the elements-words included in the template, which were not specified in the template itself. The following example shows the definition of the case and number of a noun using the template parameters:

ANp = [A] N1 N2 <c = nom> <A = N1> (Nl.c, Nl.n)

Here A is an optional adjective, N1 and N2 are nouns, where the restriction c = nom is specified for the noun N2 in angle brackets - the designation of the nominative case; further, in angle brackets A = N 1 sets the grammatical agreement of all common morphological characteristics of the adjective A and the noun N1, after which the morphological characteristics of the first noun are described in parentheses, allocated as parameters of the template ANp: c - case and n number. At the same time, the case of the noun N2 cannot be used as template parameters, since it has already been specified in the template itself.

For a template with parameters declared in this way, you can later create instances, fixing the required arguments, for use in other templates. For example, to create an instance of the given ANp template with a given nominative case and a singular number for the first noun, use the construction

ANp <c = nom, n = sing>

Here ANp is the name of the template for which the instance is being created, then in angle brackets c = nom is fixation as the first parameter of the nominative template, n = sing is fixation as the second parameter of the singular template.

To check for the presence of text words in a certain dictionary in templates, it is allowed to write dictionary conditions in the form of an appeal to some logical function that checks the entry into the dictionary.

- 4 037156

The function name in such expressions can be thought of as the name of the dictionary being searched. Like matching conditions, vocabulary conditions are written in angle brackets. For example, the record <Dict (A1)> means that the word element A1 must be included in the Dict dictionary.

A set of related templates defines a formal grammar augmented with additional terms. To recognize patterns, the procedure for imposing a template on the text is sequentially applied, which results in various overlay options. An overlay option is a found continuous piece of text that matches a template along with a set of specific meanings of the morphological characteristics of the words included in this segment. The values of the template elements are the specified text segments. In some cases, it is possible that the same template matches the same portion of the text in different ways, as well as the intersection of matches.

The language of lexico-syntactic templates LSPL contains rich tools for taking into account the morphological characteristics of words and their grammatical agreement. Unlike the JAPE language, the LSPL syntax is less cumbersome and relatively more understandable, but the pronounced linguistic focus of this approach negatively affects the ease of use and performance of the analysis. Orientation to the Russian language is not so much a disadvantage as a necessary measure, since taking into account morphology requires a close connection with a specific language. Pattern detection occurs sequentially, resulting in a linear performance degradation as the number of patterns to search increases. Although the use of regular expressions helps to increase the accuracy and completeness of the analysis, it reduces the safety of the approach in the sense of eliminating too long enumeration of variants of matches.

A deep analysis of the morphological features of a language is an important area of scientific research, but it does little to approach the solution of the problem of text tagging: grammatical agreement of words affects a small percentage of results, since the analyzed real texts in natural language are not compiled in order to complicate the operation of search algorithms. Discarding words that are nearby and match the pattern in their basics, but grammatically inconsistent words, does not significantly increase the quality of the search. Thus, to simplify the compilation of templates for languages with rich word formation, it is enough to use stemming, and in some cases just list the required word forms.

Another well-known method of text analysis is the approach based on Tomitaparser grammars. This approach is similar to the LSPL-based approach and uses keyword dictionaries and formal grammars that take into account morphological features of words to extract facts from natural language text. The Tomita parser grammar rules work with strings. One chain corresponds to one sentence in the text. Substrings are distinguished from the chain, which in turn are interpreted as facts separated by fields. For example, the rule

S -> Noun;

describes a noun. In this case, by default, the words are reduced to their initial form. In the above rule, Noun is the terminal. In the Tomita parser, terminals include the names of parts of speech, punctuation symbols and arithmetic signs, words in the initial form - lemmas, as well as some other special terminals, for example, the Word terminal, denoting any word. Terminals can only be on the right side of the rules. Nonterminals are written on the left side of the rules. In this example, S is a nonterminal. If a nonterminal occurs only on the left side of the rules, then it is considered the top of the grammar - the initial nonterminal. In this case, the initial nonterminal can also be specified explicitly. For use on the right side of the terminal and nonterminal sequence rules, they are separated by spaces. For example, a rule is defined as follows, highlighting an adjective and a noun following it:

S -> Adj Noun;

To impose restrictions on a terminal or nonterminal, the Tomita parser uses special attributes specified in angle brackets after the corresponding terminal or nonterminal. For example, the h-reg1 attribute is used to indicate that a word must begin with an uppercase letter. In addition to this attribute, there are attributes that indicate that the first letter of the word and at least one other letter of the word must be capitalized, or that the word must be fully capitalized. For this, the h-reg2 and l-reg attributes are used, respectively.

The repetition operator in the Tomita-parser grammar description language is denoted by the + sign and means that a terminal or a nonterminal can occur one or more times in a chain. The element is enclosed in parentheses to indicate optional. For example, following an adjective with several capitalized words, which may be preceded by a preposition, is described by the pattern

S -> Adj (Prep) Word <h-regl> +;

Here Adj is an adjective, Prep is a preposition, Word is any word, then h-reg1 in angle brackets specifies that the word Word should start with a capital letter, and the + symbol specifies that the word Word can be repeated one or more times.

In the Tomita-parser grammar description language, it is possible to use the alternative operator,

- 5 037156 denoted by the symbol |. For example, options for addressing a person are described by the template

FormOfAddress -> 'comrade' | 'mister' | "lord";

Tomita-parser, like the language of lexical-syntactic templates LSPL, has tools for analyzing morphology and indicating the grammatical agreement of words. Grammatical agreement is specified by special attributes, for example, the gnc-agr attribute (from the English gender, number, case agreement) is used to agree on gender, number, and case.

Each chain, defined by the Tomita-parser rule, has a main word, the grammatical features of which are inherited by the entire chain. By default, the first word in the string is set as the main word, but you can specify it manually using the rt attribute. This may be required due to the fact that word normalization is performed based on the given main word. In the case of normalization of the adjective and the noun associated with it, if the main word is not indicated by the noun, then as a result of normalization the adjective will be reduced to the masculine gender, which may not coincide with the gender of the noun.

Tomita parser allows not only to set restrictions on grammatical agreement, but also to separately indicate the required morphological characteristics of words. Morphological characteristics are set using attributes, for example, to search for a feminine word, he should set the gram attribute with the value of wives. In this attribute, you can specify other morphological characteristics of words, separated by commas, such as number, case, type of verb, form of adjectives, and so on.

To work at the character level using regular expressions, the Tomita parser contains the corresponding attributes wfrn, wff, and wfl. The syntax of regular expressions used in Tomitaparser is based on the syntax of those in the Perl language. For example, to define a date, you can use the template

Date -> AnyWord <wff = / [1-2]? [0-9] {1,3} \.? />

Some grammars require not only morphological, but also semantic information about words from the text. Such information can be specified using dictionaries. The dictionary consists of articles, each of which has a title and a set of words included in the article. Words can be listed both in the dictionary description file and in a separate text file by specifying its name in the dictionary description file. After the dictionary has been described to be used, it must be imported into the root dictionary. After that, the kwtype attribute can be used in the terminal attributes, the value of which determines the name of the dictionary, which should include the corresponding word.

The tomita parser extracts strings of text based on grammars in order to then interpret the strings as facts and obtain structured data. Fact types are described in a special format and contain the name of the fact and a description of the names and types of its constituent fields. For each field, it is indicated whether it must be filled in, the keywords optional and required are used for this. The grammar contains a description of how the highlighted string should be interpreted. To do this, use the interp keyword, after which the name of the fact and the name of the fact field, into which the corresponding string should fall, are written in parentheses.

Finding strings of text that match grammars in the Tomita parser is done using the GLR parser. The GLR parser is a modified version of the LR parser algorithm designed to parse non-deterministic and ambiguous grammars. The GLR parser, like the LR parser, is based on the use of a special data structure - an analysis table that contains the syntax of the parsed language. In this case, the LR parser analysis table allows only one state transition, determined by the state of the parser and the input terminal. To ensure work with non-deterministic and ambiguous grammars, the GLR parser table can contain several jumps. The occurrence of a conflict is solved by forking the parser stack into two or more parallel stacks, the last elements of which correspond to each of the possible transitions. Further, the input symbols are used to determine the subsequent transitions of all received branches. If for some of the branches on some terminal there is no transition, then it will be deleted as an erroneous one. At best, for a fully deterministic grammar, the GLR parser runs in linear time, at worst in cubic time. The above description is valid for one grammar, since the process of matching is reduced to folding the original string into the starting nonterminal of the grammar. Of all the possible variants of grammar matching, those that cover a larger area of the text are selected. After building the syntax tree, the Tomita parser creates facts in accordance with the given descriptions.

Tomita-parser has similar functionality to the language of lexical-syntactic templates LSPL, allowing you to specify the rules of grammatical agreement of words and the required morphological characteristics. Increased attention to morphology leads in this case to a similar complication of the syntax for writing templates. The tomita parser does a good job of identifying facts in the text, however, in the task of tagging text, the search, as a rule, is carried out by certain specific words of interest, and not by syntactic constructions such as, for example, verb or noun phrases. The use of the Tomita parser for the tagging task is complicated by its cumbersomeness and complexity

- 6 037156 template syntax, as well as low performance, due to the rich possibilities for working with morphology.

Summing up, we can state that the known approaches meet many, but not all, requirements for the means of identifying a set of concepts, entities and their relationships in natural language texts. What is actually required is a balanced pattern-based solution that is fast by searching the entire set of patterns in a single pass through the text tokens, does not degrade performance linearly as the number of patterns to search increases, while providing a powerful syntax for describing patterns in the form of formal grammars. including means of determining sequences (strings), variations (alternatives) and repetitions of text tokens and occurrences of other patterns, including recursive patterns, provides acceptable accuracy and completeness of the search, language independence, and also safety in the sense of avoiding too long searches of matches.

The essence of the invention

The technical result of the invention is to increase the speed of searching for matches with templates in the text due to preliminary parsing of the text into tokens and checking all matches of all templates in one sequential scan of the text tokens.

Another technical result of the invention is to increase the completeness and accuracy of searching in a text by templates due to the use of formal grammars as templates that allow sequences, variations and repetitions of text lexemes and occurrences of other templates, including grammars of unlimited nesting level, recursive grammars and parameterized grammars. ...

Another technical result of the invention is to ensure the independence of the search from the language of the text due to the rejection of the morphological analysis of the text and the use of variations in templates to take into account word forms.

Another technical result of the invention is to ensure the security of the search, which means avoiding the delay in the search for a long time, perceived by the user as a loop or endless search, by limiting the number of matches created during the search for candidates or by limiting the amount of computer resources consumed by candidates.

Another technical result of the invention is to increase the speed, completeness and accuracy of identifying a set of concepts, entities and their relationships in natural language texts in a limited time and regardless of the language.

In accordance with one aspect of the present invention, the listed technical results are achieved by performing the following steps: the text is pre-parsed into lexemes, which include at least words and word separators; in the template description language, a set of templates is created in which each template is a formal grammar consisting of at least sequences and (or) variations, and (or) repetitions of text tokens and (or) occurrences of other templates and admitting parameters intended for generalization of templates and (or) to refine search results; a set of templates is translated into search expression trees with search indexes, which allow for a given token or a given template name to quickly find all search expressions that directly or indirectly begin with a given token or pattern occurrence; create a variable set of candidates, each of which stores information about the coincidence of the text fragment with the corresponding search expression and serves to make a decision on the further coincidence or non-coincidence of the text fragment with the template; then the tokens of the text are sequentially scanned and for each token, at least the following actions are performed: in the search indexes, all templates starting with the current token are found, candidates are created to check for text matches with these templates, and they are added to the set of candidates; for each candidate from the set of candidates, a decision is made on complete match, non-match or incomplete match of the text with the corresponding template, depending on the value of the current token; for each template match, a decision is made on complete match, non-match or incomplete match of candidates having links to this template; to take into account various variants of the text coincidence with the checked templates, create and add to the set of candidates logical copies of those candidates for which different variants of matching with the text up to the current token inclusive are possible, and the logical copies of the candidates inherit a common state that corresponds to the already scanned text tokens, and differ in the part of the state that corresponds to the current token of the text; in the event that the number of candidates created in the search process and (or) the amount of computer resources consumed by candidates exceeds a certain specified limit, then, in order to avoid dragging out the search for a long time, the set of candidates is reduced by removing part or all of the candidates, which is equivalent to cutting off the text and beginning a new search with the current token.

List of drawings

FIG. 1 shows the architecture of the device;

in fig. 2 - an example of a template and the corresponding expression tree;

in fig. 3 is a block diagram explaining the general algorithm of the search performer's work;

- 7 037156 in Fig. 4 is a block diagram explaining the general decision-making algorithm for the candidate tree depending on the value of the current token;

in fig. 5 - a step-by-step change in the state of the search executor using the example of a pattern search with alternatives and exceptions of variation in a certain input sequence of tokens.

Information confirming the possibility of carrying out the invention

Information confirming the possibility of carrying out the invention is set forth below in the sections Device architecture, Template description language, Text processing rules, Parsing the input text into tokens, Representation of templates at the level of the search executor, Translation of the text description of templates, Data structures of candidates and match results, Algorithm of the executor search, General algorithm of decision making, Decision rules, Recommendations for testing, Performance tests, Conclusion.

Device architecture.

Part of the present invention is shown in FIG. 1 is a variant of the device architecture made by the authors to implement the invention. As shown in FIG. 1, a typical device includes a template description language translator 12, a text lexical analyzer 19, and a search engine 21.

The templates 11, written in the template description language, are loaded from a text file and fed to the input of the translator from the template description language 12, which parses the templates using the parser 13, and then creates the template package 15 using the template package generator 14. In the process its work translator 12 refers to the lexical analyzer of the text 19 to convert text literals, consisting of more than one token, into a sequence of tokens.

Template package 15 is translated descriptions of templates in the form of in-memory data structures and includes search expressions with search indexes 16 and a root search index 17.

Before performing the search, the lexical text analyzer 19 parses the input text 18 into tokens. The resulting parsing sequence of tokens 20 is passed along with the pattern packet 15 to the search engine 21.

The search engine 21 searches the text for all matches with patterns in one sequential scan of the text tokens 20. To implement this possibility, the search engine 21 creates and updates a data structure that contains a set of match candidates 22. For each token, the search engine 21 updates the set of candidates 22 based on decision rules 23 on complete match, non-match or incomplete match of the corresponding templates. Candidates for templates that completely match the text are converted to match results 24. Candidates for templates that do not match the text are removed. The candidates for overlapping text templates remain in the candidate set 22.

The embodiment of the invention made by the authors uses the methodology of object-oriented programming (OOP), which is done in order to better structure data and algorithms, as well as to simplify and speed up the implementation. However, the choice of a methodology and programming language with support for OOP is not required for the implementation of the invention. With the same result, a structured programming methodology or other methodology can be used, as well as any imperative programming language, including machine language (assembler).

Template description language.

Part of the present invention is one of the variants of the template description language, which allows declaratively describing the desired constructions in the text using formal grammars. The grammars are listed one after another in the text. Each grammar begins with a name (identifier), then an equal sign is written, followed by an expression consisting of text literals, token types, references to other grammars, and operators. The grammar is terminated with a semicolon. In general, an expression can contain the following elements:

text literals that are compared with or without case-sensitive comparisons (i.e., no uppercase or lowercase);

types of tokens: alphabetic, numeric or alphanumeric sequence of characters, punctuation mark, non-alphabetic character, whitespace or line feed character;

sequences with strict adherence of elements;

alternatives (variations), which are enumerations of elements that are allowed at a given point in the template, as well as elements that cannot be matched;

repetition of any element of the template the minimum and maximum number of times;

the scope of the template, which specifies the need to match one template within another;

the following of elements through words with an indication of the maximum and minimum permissible number of words between the elements;

links to other templates, i.e. occurrences of other patterns.

- 8 037156

An arbitrary combination of the above elements is permissible. Their syntax and examples in the template description language are discussed below.

Text literals.

A text literal is a sequence of Unicode characters enclosed in single or double quotes

Minsk

By default, text is compared in a case-insensitive manner. If you need to specify a text that is compared in a case-sensitive manner, then an exclamation mark is written after the quotation mark that ends the text. for example

Minsk!

When translating a template, each text literal is parsed into tokens using the same algorithm as the text in which the search is performed. As a result, the original text literal is converted into a sequence of tokens.

Types of tokens.

Token types are used to search for tokens without specifying their exact string representation. The following types of tokens are available:

Alpha - an alphabetic token (from the English alphabetic), a continuous sequence of alphabetic characters;

Num - digital token (from the English numeric), a continuous sequence of numbers;

AlphaNum - an alphanumeric token, a continuous sequence of letters and numbers starting with a letter (from the English alphanumeric);

NumAlpha - an alphanumeric token, a continuous sequence of letters and numbers, starting with a number (from the English numeric-alpha);

Punct - punctuation mark (from the English punctuation), includes all signs that perform the function of separating or highlighting semantic parts of the text, sentences, phrases, words, parts of a word, indicating the relationship between words, indicating the type of sentence, its emotional coloring, as well as some other functions;

Symbol - a non-alphabetic character, for example: &,%, $, #, _, {,}, etc .;

Space - space, tab or other separator without style;

NewLine - sign of a new line;

Start - sign of the beginning of the text;

End - a sign of the end of the text.

The token type is used by its identifier (name). All identifiers in the template description language are case sensitive.

Sequences.

Strict adherence of elements is specified by the sequence operator, where each element is a pattern. The + sign is used to combine elements into a sequence. For example, a .com domain name can be specified with the template + honeycomb

This notation is equivalent to the text literal .com. However, the sequence obtained by splitting a text literal into pieces without observing the rules for parsing into tokens is not equivalent to the original text literal. For example, replacing the string constant Minsk, consisting of one token, with a sequence of letters

4- П · £ п 4- 4- g 4-] ζ will lead to the fact that no match with the text Minsk will be found.

Variations.

The variation operator is used to list multiple elements that can be matched at a given point in the pattern. As with the sequence, each element of the variation is a pattern. The variation operator also has the ability to specify patterns that cannot be matched. Such elements of variation are called exceptions. Exceptions are capable of canceling coincidences of alternatives that began to coincide from the same position in the text, and at the same time they can cover a larger section of the text than the alternative itself. The variation is specified by listing all alternatives and exceptions in curly braces separated by commas. In this case, the exceptions are marked with a prefix ~. For example, a simplified version of the dot search pattern as a sentence separator can be written as follows:

-.net, -.com}

The above template will reveal a dot in the Sentence. Text, but will not reveal it in the Sentence.net text.

Repetitions.

The template description language allows you to specify the number of repetitions of any independent template element using the repetition operator. An independent element is understood as an element that is itself a template, i.e., for example, an exclusion element in a variation (the ~ sign and the following template) is not an independent element, but the template itself following the sign ~ is that. Repetition is specified using square brackets in front of the repeated element. Inside the square brackets, the minimum and maximum allowed number of repetitions are indicated, separated by the - sign. For example, repetition of element X from three to five times is specified by the pattern

[3-5] X

If the minimum and maximum permissible number of repetitions are the same, then it is permissible to specify the same number. For example, repetition of element X exactly three times is specified by the pattern

[3] X

If the minimum and maximum allowed number of repetitions are not specified explicitly, then they are assumed to be equal to one. The maximum limit can be omitted, which allows an arbitrarily large number of repetitions. In this case, a + sign is placed behind the minimum restriction. For example, repetition of element X one or more times is specified by the pattern

[1+] x

The absence of a maximum restriction in the record is interpreted by the search performer as a sufficiently large number, which at the same time should not be too large, so as not to significantly slow down the pattern checking. In the case of several overlapping matches, the largest sequence is selected, the match of which in the text began earlier.

The minimum number can be zero, then it is considered that the match of an element is optional, such elements are called optional (optional). In case an element does not occur at all or occurs once, you can use a question mark - an abbreviated form of writing repetition from zero to one time

Optional elements also affect the template they are included in. For example, a sequence of optional elements is itself considered optional, and a variation is also considered optional if at least one of its alternative elements is optional.

Standard templates.

There are predefined or standard templates in the template description language that are automatically available for use in any custom templates. Standard templates are used to shortly describe commonly used constructs. The following standard templates exist:

Any - any lexeme, excluding the signs of the beginning and end of the text, is equivalent to the variation {Alpha, Num, AlphaNum, NumAlpha, Space, Punct, Symbol, NewLine};

Word is a word equivalent to the variation {Alpha, Num, AlphaNum, NumAlpha};

Blanks - any number of spaces or line terminators, equivalent to the pattern [1+] {Space, NewLine};

WordBreaks - Any number of word separators such as spaces, punctuation, non-alphabetic and line terminators, equivalent to pattern

[1+] {Space, Punct, Symbol, NewLine}.

The template description language does not specify the implementation of standard templates at the search executor level. While all of them can be expressed in terms of the operators already described, a separate implementation is also acceptable for performance reasons.

Scope.

The scope operator is used to find a match of one pattern within another. This operator allows you to search for structures within certain parts of the text, for example, sentences or paragraphs. The scope operator is denoted by the @ symbol and includes both left and right sides. On the left side, the template that needs to be found is written, and in the right side, the template, inside which a match for the template from the left side should be found. One pattern is considered to be a match within another if the left border of the first match is no earlier in the text than the left border of the second match, and the right border is no later in the text than the right border of the second match. For example, to find a match for pattern X within pattern Y, use pattern

Here, instead of Y, it is allowed to use both the name of another declared template, and the template itself, declared in place. The scope operator has a restriction on specifying repetitions: for the scope template itself, only optional specifying is allowed. At the same time, there are no restrictions for templates on the left and right sides of the operator.

Follow through words and separators.

Search for certain words and constructions that are at some distance in the text from each other

- 10 037156 friend is a popular challenge. In the language for describing templates, there is a corresponding operator for following through words for this. In the word sequence operator, a word means a standard Word template. The distance between the left and right parts is specified in words and is written similarly to repetitions, but without square brackets. In the case of several overlapping matches, the result is a match of shorter length. This is due to the fact that when searching for related words, the smaller the distance between words, the more likely they are related and are the target of the search. The operator of following through words also allows specifying a pattern, the match of which between the left and right parts is not allowed. If it is necessary to prohibit several constructions, then a variation should be used. In general, the word-sequence operator is written as follows:

X .. Μ-Ν -Ζ .. Υ

Here, Y follows X at a distance from M to N words, and between X and Y there should not be a pattern Z. An equivalent to this construction can also be expressed through the already described operators and standard patterns

X + [M-N] (WordBreaks + {Word, ~ X, ~ Y, ~ Ζ}) +? {WordBreaks, ~ Y} + Υ

An important part of the above example is to include the X and Y patterns as variation exceptions. This ensures that the requirement to choose the shortest match from intersecting ones is fulfilled: neither X nor Y can be between X and Y. A special case of following through words is following through separators, i.e. following through zero words. The abbreviated form of such a construction is represented by the template

X. ... ... Υ which can also be expressed by an equivalent string

X +? {WordBreaks, ~ Y} + Y

The template description language does not specify a way to implement this operator at the level of the search executor: both an expression through other operators and a separate implementation are acceptable.

Consistent mention.

In some cases, you need to find some words in the text, similar to how it is done with a full-text search. To do this, the template description language uses the sequential reference operator, denoted by the & symbol. It is considered that patterns are consistently mentioned in the text if they occur in it at any distance and in any sequence. The sequential mention operator can be expressed through other operators, for example, the pattern

X & Υ can be replaced with the equivalent pattern {X .. 0+ .. Υ, Υ .. 0+ .. X]

The template description language does not specify a way to implement this operator at the level of the search executor; both the expression of the operator through other operators and a separate implementation are acceptable.

Priority of operations.

The following table lists the priorities of the operations in the template description language in descending order of priority.

Priority Name Designation one Eliminating a variation element 2 Reiteration [] 3 Variation {} four Consistent mention & five Sequence + 6 Follow through words and separators 7 Scope

If necessary, the priority of the operation can be specified explicitly using parentheses. For example, in the template

[1-5] (X + Υ) repetition refers to the entire chain. Without parentheses, repetition will only refer to pattern X.

Named templates and tags.

If you need to link to a template from another template or return it in search results, then it should be given a unique name. The template name is an identifier, i.e. starts with a letter or underscore followed by a sequence of letters, numbers, and signs

- 11 037156 underlines. Such a template is called named. The name, separated by an equal sign, is followed by the pattern itself. The definition of a named template ends with a semicolon. For example, in the template

Ζ = X;

named pattern Z is defined through pattern X. Match X means match Z. The identifier Z can be used to refer to a pattern in other patterns. In order for the template to be returned in search results, precede the template with the # sign. A template marked this way is called a target template or tag. For example, to define a Z tag through a string of X and Y, use the pattern #Z = X + Y;

Named templates that are not tags are used to create other templates, but they are not included in the search results.

Text processing rules.

Rules for parsing the input text into tokens.

Parsing the input text into tokens is implemented based on the following rules:

a) the beginning and end of the text should be selected as zero-length tokens of types Start and End, respectively;

b) a sequence of carriage return and linefeed characters must be allocated as a token of type NewLine;

c) a sequence of alphabetic characters must be selected as a token with the Alpha type;

d) a sequence of numeric characters should be allocated as a token with the Num type;

e) a sequence of numeric and alphabetic characters starting with a letter must be allocated as a token with the AlphaNum type;

f) a sequence of numeric and alphabetic characters starting with a digit must be allocated as a token of type NumAlpha;

g) a sequence of whitespace characters must be allocated as a token of type Space;

h) each punctuation mark must be highlighted as a separate token with the Punct type;

i) each non-alphabetic character that is not punctuation marks must be allocated as a separate token with the Symbol type.

Search pattern translation rules.

Translation of search templates from the template description language should be implemented taking into account the following rules:

a) the textual description of the templates should be translated into an expression tree used by the search engine;

b) an attempt to translate an incorrect text description of templates should lead to the generation of a program exception with information about the error;

c) the text describing templates for broadcasting must be loaded from a file or taken from a text string;

d) text literals consisting of more than one token must be converted into a sequence of tokens based on the same parsing rules that are used for the input text;

e) in the process of translation of the textual description of templates, some expressions in the expression tree can be replaced with other equivalent expressions in order to optimize the memory used and the search speed.

General search rules.

The search algorithm must be implemented taking into account the following rules:

a) the search for multiple patterns in one document should be performed in one scan of the sequence of tokens corresponding to the document;

b) if several overlapping matches of the same template are found, the result should be considered the largest match in length in lexemes, which began earlier in the text;

c) the maximum memory used during the search should be limited by the maximum number of simultaneously considered variants of matches;

d) search results should be presented in a hierarchical form, where for each element of the template is stored a piece of text with which a match was found, and similar results for its nested elements;

e) search results should be presented in the form of a list, where only matches of tags are saved, without detailing the nested elements;

f) Only tag matches should be returned as search results, i.e. matches of templates marked with #.

Search rules for text literals and token types.

Searching for text literals and token types must be implemented with the following rules in mind:

- 12 037156

a) the search for text literals should be performed in accordance with the specified case sensitivity;

b) the search for the type of token should be performed in accordance with the type of token assigned at the stage of parsing the input text into tokens, without additional analysis of the text of the token.

Sequence search rules.

Sequence search should be implemented taking into account the following rules:

a) any template can be specified as an element of the sequence;

b) the sequence must be considered matched when all of its mandatory elements have been matched;

c) when searching for optional elements of a sequence, both variants of matching the sequence should always be considered: with an optional element and without, even if a match for an optional element is found;

d) a sequence, all elements of which are optional, should be considered optional.

Variations search rules.

The search for variations should be implemented taking into account the following rules:

a) alternatively, any template can be specified;

b) any template can be specified as an exception;

c) when searching for a match of a variation, the options for matching each of its alternatives should be independently considered;

d) the match of the exclusion pattern should cancel the match of only those alternative patterns that coincided or began to match from the same initial position as the exclusion pattern;

e) a variation, at least one of the alternative elements of which is optional, should be considered optional.

Repetition search rules.

The search for repetitions should be implemented taking into account the following rules:

a) repetition can be specified without restriction for any template other than the scope template;

b) for the scope template, in addition to the default repetition value, only an indication of repetition from zero to one should be available, other values should be prohibited;

c) a pattern should be considered repeated if it matches several times in succession, and the next match begins with the token following the one that ends the previous repetition;

d) all variants of matching an element with a specified number of repetitions in the form of a range should be considered;

e) elements with a minimum number of repetitions equal to zero should be considered optional;

f) for elements that do not have a maximum number of repetitions specified, this number shall be set in such a way that the search time for matches does not exceed the interval waiting for a response from the calling system.

Search rules for scoped templates.

Searching for scoped templates should be implemented taking into account the following rules:

a) any pattern can be specified as the left operand;

b) both the template name and the template declared in place can be specified as the right operand;

c) a pattern should be considered as matched in the area of another pattern if the left border of the first pattern is no earlier in the text than the left border of the second pattern, and the right one is no later in the text than the right border of the second pattern.

Search rules for patterns following words and separators.

Searching for patterns following words and separators must be implemented taking into account the following rules:

a) any templates can be specified as operands;

b) when counting the number of words, a word should be understood as a digital, alphabetic or alphanumeric token, separators during counting must be ignored;

c) in the case of several overlapping variants of the match, the result should be a match with a smaller distance in words between the templates;

d) the match of the exclusion pattern should lead to the cancellation of the match of the pattern of following through the words;

e) in the operator of following through words and separators, a separator must be understood as a non-empty sequence of whitespace characters, punctuation marks, line feeds and other non-letter characters.

Search rules for sequentially mentioned patterns.

- 13 037156

The search for consistently mentioned patterns should be implemented taking into account the following rules:

a) any templates can be specified as operands;

b) sequentially mentioned patterns should be understood as patterns that occur in the text at any distance and in any order.

Rules for searching for links to templates.

The search for links to templates should be implemented taking into account the following rules:

a) in the case when a certain piece of text matches a template that is referenced by several other templates, the search for the template that is located by the link must be performed once;

b) a search for templates should be performed, the structure of links between which forms a left or right recursion;

c) when referring to a standard template, the expression corresponding to the standard template must be directly substituted at the link location.

Parsing the input text into tokens.

Part of this invention is one of the variants of the lexical text analyzer 19, which parses the input text into tokens. The input data for the lexical analyzer is a text string. No other data is required for work, since the rules for dividing into tokens are fixed. A text string in this case means a string object in a programming language, which is logically an array of characters in RAM. The decision to work with data only in RAM is due to the need for high performance of this stage of the search. Loading a text string into memory increases the likelihood of caching both parts of the string itself and the program code for accessing a symbol, which, with this approach, is a simple access by an index in an array. Thus, in this case, the trade-off between the cost of time and memory is resolved in favor of time. It is worth noting that the same decision was made in the search algorithm: identifying many patterns in the text in one scan of the text requires additional memory to store the state of all matches. It should be borne in mind that in languages with automatic memory management, an increase in memory consumption due to an increase in the number of objects created indirectly affects performance, since the overhead of garbage collection increases. In this regard, the output of the lexical analyzer is a special index containing the original text string and an array of structures encoding the type and boundaries of each token in the original text. The creation of a substring for each token from the source text is performed as needed. This structure drastically reduces the number of objects created in heap compared to simply creating a new substring for each token.

The lexical analyzer uses the Unicode Standard Annex # 29 word division rules with the following modifications:

a dot and an underscore separate an alphabetic or alphanumeric sequence into multiple tokens;

decimal separator divides a digital sequence into several tokens; the sequence of whitespace characters is not separated;

all punctuation marks and other non-alphabetic characters are distinguished by a separate token.

Word boundary properties are used to apply word boundary definition rules. These properties are defined for the entire range of Unicode characters, including those larger than two bytes. This solution uses UTF-16, which uses two bytes for one character. Characters beyond two bytes are ancient or rarely used languages and are therefore ignored.

During text processing, you need to determine the word boundary property for each character by searching some data structure that stores the word boundary property information for each double-byte Unicode character. To effectively solve this problem, a static two-dimensional array of word boundary properties is used. Addressing is performed on the high and low bytes of the character for which you want to get the word boundary property. The sequence of character properties has the following characteristics: the most common are continuous sequences of values of any and the letter of the alphabet. To reduce the size of the array for such sequences, the second addressing level does not use arrays with the same property values for each character, but references to empty marker arrays. Thus, when searching, you must first allocate the most significant byte of the desired character and use it for the first addressing level in the array. The resulting array reference must first be compared against the two marker arrays. In case of a match with one of them, the type of the symbol is determined. If no match is found, then it is necessary to select the least significant byte of the character and use it for addressing in the resulting array.

In addition to detecting boundaries, the lexical analyzer also determines the type of tokens. To identify the types of tokens for each new character, it is tracked to what type it belongs to, as well as what type the token has, taking into account the previously considered characters. Symbols are divided into letters, numbers, signs

- 14 037156 punctuation, carriage return and line feed. Other characters are considered non-alphabetic characters. This logical model is a finite state machine, where the state is the type of token, taking into account the previously considered symbols, and the transition between states is carried out by the input symbol. Resetting the automaton to its initial state, when the type of token is considered undefined, is performed at the boundaries of tokens. The main loop of the lexical analyzer, which scans the string, coordinates the actions of the token separator and the token classifier. The complete list of rules for determining the boundary of a token assumes a buffer of character properties with a size of four elements. In addition to the current character, look forward by two characters and backward by one are used. Moving to the next character involves defining a word boundary property for it and adding the property value to the buffer. At the beginning of text processing, a jump forward two characters is performed to initialize the buffer. Further, in the main loop, there is a movement forward along the line, during which the buffer moves and the characters are fed into the token classifier. After each character is processed, a check is made to see if there is a token border at the current position. In the case when the token has ended on a given character, its boundaries and type are added to the result. Also, artificial tokens of the beginning and end of the text of zero length are added to the corresponding positions of the result.

Representation of templates at the level of the search performer.

At the search performer level, each template is represented by a data structure that encodes a tree of all grammar elements that make up the template. This data structure is called the search expression tree and differs from the syntax tree resulting from the translation of the template in that it is adapted to efficiently implement the search. The search expression tree contains only the basic operators of the template language, such as sequence and variation, and does not contain derived operators expressible through basic operators, such as following through words and separators and sequential mention. The separation between the search expression tree and the template syntax tree provides the flexibility needed to modernize and improve the solution, optimize the internal data structures and algorithms of the search engine operator working at the search expression level, and maintains compatibility with existing clients working at the syntax tree level.

The search expression tree consists of elements (expressions) of the following type:

token (TokenExpression class);

template reference (ReferenceExpression class);

sequence (SequenceExpression class);

variation (VariationExpression class);

variation item (VariationItemExpression class);

scope (class EnclosureExpression);

pattern (PatternExpression class).

The repetition operator of the textual description language of templates is presented in the search engine not as a separate type of expression, but as an attribute of other expressions. This attribute is defined for the following expressions: token, template reference, sequence, variation, and scopes. This approach is optional and was chosen to reduce the number of dynamically created objects and to reduce the depth of the expression tree. The representation of the repetition operator as a separate type of search expression does not change the essence of the invention.

An expression of type pattern is the root of the search expression tree. The subordinate element of this expression can be an expression of any of the above types, except for the types template and variation element. Expressions with token type and template reference are leaf (terminal) nodes in the tree, i.e. cannot have subordinate elements; all other expressions are compound. In OOP terms, the terminal expression classes (TokenExpression and ReferenceExpression classes) derive from the TerminalExpression class. Composite expression classes (SequenceExpression, VariationExpression, VariationItemExpression, EnclosureExpression, and PatternExpression) can include subexpressions and are derived from CompoundExpression. The base class for the TerminalExpression and CompoundExpression classes, and indirectly for all expression classes, is the Expression class. All search expressions of type template are collected in a package of templates.

An example of a template and its corresponding expression tree is shown in Fig. 2. As is customary in computer science, the tree is drawn upside down: the root is at the top, and the terminal elements are at the bottom.

The search expression tree of each template is fully coherent, allowing not only to go down from the root to the terminal elements, but also to climb from the terminal elements to the head elements up the tree. This is important for the search engineer's work.

Each search expression contains a search index. A search index is a data structure (object) that allows, for a given input token or for a given matching pattern, to quickly obtain terminal elements (expressions with the token type and a reference to a template) of all subordinate expressions, with which a match or mismatch of the head expression can begin

- 15 037156 ny. The search index of the head expression is the result of merging the search indices of the subordinate expressions, and the way of merging depends on the type of the head expression. If the head expression is a variation, then its search index is a union, in the sense of combining sets, of search indices of all elements of the variation. If the head expression is a sequence, then its search index is a union, in the sense of combining sets, of the search indices of all the first optional elements with the search index of the first required element of the sequence. An element is considered optional if the minimum number of repetitions is zero. For a terminal expression (token or reference to a template), the search index consists, in fact, of one element and returns either one of this element or an empty set in the search results. It is obvious that the full or partial inclusion of the search indexes of subordinate elements in the search indexes of the head elements requires noticeable expenditures of RAM, therefore, it is advisable to carry out optimization in order to reduce duplication of the same data, in particular text strings, in memory.

The search index should provide different types of quick search for expressions: only tokens, only references to templates, any terminal expressions. Moreover, the search criteria can be values of types of tokens and text literals with and without regard to case, template identifiers, signs of considering exceptions in variations, i.e. terminal expressions that match or fail to match the search expression.

The search indices of all root search expressions of type pattern are merged, in the sense of combining sets, into the root search index. The search root is part of the template package.

In OOP terms, search expressions are objects of the corresponding classes and have the following main properties and procedures:

ParentExpression - reference to the head expression;

RepeatRange is a structure that stores the allowed number of repetitions;

IsRepeatable - Determines whether the repetition operator is valid for the given expression (overridden in derived classes);

IsOptional - Determines if the expression is optional;

GetOrCreateExpressionIndex - a procedure for getting the search index of an expression; if this is the first time it is accessed, the search index will be pre-created;

CreateExpressionIndex is a procedure, overridden in derived classes, that directly builds an index for the corresponding expression type.

After the complete creation of a package of templates with search expressions and search indexes, these data structures do not change anymore; during the search executor's work, access to them is performed only by reading.

Translation of text descriptions of templates.

The template description language proposed in this embodiment of the invention has a simple and understandable syntax, which is also convenient for automatic parsing. Translation of descriptions of templates from this language into search expressions is performed using well-known recursive algorithms using a machine or software stack.

In the considered embodiment of the invention, the translation of the textual description of the templates consists of two stages: the construction and optimization of the syntax tree based on the textual description of the templates and the generation of search expressions with search indices for the search performer.

The construction of the syntax tree is performed in a single scan of the tokens of the template language using a recursive algorithm and using the machine stack. After highlighting the next token, its textual representation and type are stored in the parser state. The types of tokens in this case differ from those in the lexical analyzer of the text, since in this case the analysis is not of a natural, but of a formal language. Identifiers, string and numeric literals, parentheses, curly brackets, square brackets, arithmetic signs, and other special characters that have a special meaning in the template description language are highlighted.

Parsing of formal grammars is performed by the bottom-up recursion method. The end result of parsing the text in the template description language is a template package object, represented as a syntax tree. Parsing begins with a call to the parse package, which in a loop calls the parse routine for the named template. The parser for the syntax of a named template, after determining the template name, parses the right side of the template body. This procedure calls the parsing procedure for the lowest-priority operator, the scope operator. Later, in the procedure for parsing the syntax of the scope operator, the procedure for parsing the previous operator in priority is called - the operator of following through words and separators, and so on. In places where any expression can be written, the procedure for parsing the main expression is called, in which the sequence of calls starts from the beginning, and the indication of the required operator precedence is processed using parentheses. This approach allows parsing to obtain the required operator priority. The resulting parsing syntax tree contains nodes that exactly correspond to the operators of the template description language.

The next step is to transform and optimize the syntax tree and link templates. Template linking refers to the notation of references to syntax trees of named templates at the locations of their references. Completing this step at the end allows you not to worry about the template writing order, use forward references to templates declared later, and create recursive templates. In this step, the following transformations and optimizations are performed:

replacement of sequence operators through words and separators and operators of sequential reference to equivalent constructions from basic operators;

lexical analysis of text literals and replacement with an equivalent sequence of tokens if the text literal consists of more than one token;

replacing sequences and variations consisting of one element with the element itself;

if a positive variation element is itself a variation, then its elements will be added directly to the parent variation.

Traversing and updating the syntax tree is accomplished using a visitor design technique, also called double dispatching. In the process of traversing the syntax tree using the visitor object, if a node is updated, all its parent nodes are recursively rebuilt. To satisfy the rules for parsing text literals, they are divided into two kinds: tokens and compound text literals. In the process of parsing the text, the parser considers all text literals to be composite, and when traversing the tree, each such literal is replaced with either one token or a sequence of tokens.

The generation of the search expression tree from the template syntax tree is also performed through tree traversal using the visitor design technique. The syntax tree traversal uses a stack of generated expressions. We recursively descend down to the terminal elements and create expressions for them, and then, ascending from the recursion, expressions are created for all parent elements. Expressions for template references and for the scope operator are created in a special way. In both cases, additional linking is required after all expressions have been created. In the case of the scope operator, an additional feature is the optimization of the search: if there is a template declared in place on the right side of the operator, then in order to perform the search, you need to create an anonymous template and place the expression written in place in its right side. A reference to the resulting anonymous template is used when constructing an expression for the scope operator. However, if the link to the template is immediately written on the right side of the operator, then you must use it immediately and not create an anonymous template.

At all stages of translation, in case of any errors, a software exception is created with information about the error. This approach is used to detect and handle syntax errors, input errors, and internal errors such as invalid parser or expression generator states.

The end result of template translation is a template package containing search expressions with their own search indexes, as well as a root search index.

Data structures of candidates and match results.

Checking the match of all templates from the template package is performed in one sequential scan of the text tokens. To implement this feature, the search engine stores and uses data on the validation status of search expressions that make up the templates at each step of the scan. A data structure that stores information about a complete or incomplete match of a text with a search expression, starting from a certain position in the text, and serves to make a decision on further matching or non-matching of a text fragment with a search expression, is called a match candidate (hereinafter simply a candidate). Like search expressions, match candidates form a candidate tree, where the root of the tree is the entire pattern match candidate, and the tree elements are match candidates for sequences, variation items, tokens, and other types of search expressions. The candidate tree of each template is fully connected, allowing not only to go down from the root down to the terminal elements, but also ascend from the terminal elements to the head elements up the tree, which is important for the search executor's work. The set of all candidates of all templates checked at each step of viewing text tokens is called a set of candidates and forms the state of the search executor.

As the text tokens are viewed, match candidates are created in the set of candidates, then used to decide whether the pattern matches or not matches the readable tokens of the text, and are ultimately removed from the set of candidates either because they do not match with the next token, or because candidates turn into results coincidences. Thus, the result of a match of a piece of text with a pattern is a pattern match candidate, by which the final decision on a match is made, and which contains comprehensive information explaining where and why the match occurred. Pattern matching results are used for decision making

- 17 037156 about the match of candidates of links to these templates. The results of matching target templates (tags) are returned to the client of the search performer, and at the request of the client, the results are returned either in full or in an abbreviated form. In the latter case, these can only be the positions of the initial and final tokens of the text corresponding to the match with the pattern. The candidate tree consists of elements (candidates) of the following type:

token (TokenCandidate class);

template reference (ReferenceCandidate class);

sequence (SequenceCandidate class):

a variation item (class VariationItemCandidate) is subdivided into an alternative candidate (class Inclusive VariationItemCandidate) and an exclusion candidate (class ExclusiveVariationItemCandidate);

scope (class EnclosureCandidate);

occurrence of a pattern (class PatternCandidate).

No additional candidate type is required to encode repetitions. Each repetition is presented as a separate candidate, in which the repetition sequence number is indicated. The candidate stores its repetition number, and the parent candidate stores references to candidates of all repetitions of its subordinate elements.

The candidate with the type occurrence of the pattern is the root of the candidate tree. Its subordinate elements can be candidates of other types, except for the type of template occurrence. The candidates with the token type and the template reference are leaf (terminal) elements in the tree, i.e. cannot have subordinate elements; all other candidates are composite. In OOP terms, the terminal candidate classes (TokenCandidate and ReferenceCandidate classes) are derived from the TerminalCandidate class. Composite candidate classes (the SequenceCandidate, VariationItemCandidate, EnclosureCandidate, and PatternCandidate classes) can include subordinate candidates and are derived from the CompoundCandidate class. The base class for the TerminalCandidate and CompoundCandidate classes, and indirectly for all candidate classes is the Candidate class.

In OOP terms, candidates are objects of the corresponding classes and have the following main properties:

ParentCandidate - link to the parent candidate;

RepeatNumber The number of the repetition;

StartTokenNumber The number of the token that started the match of the candidate.

Expression - a reference to the expression matching the candidate.

ChildRepeatsPerPosition - a two-dimensional list of subordinate candidates, where the position number of the subordinate candidate within the head is used for the first addressing level, and the repetition number for the second addressing level; the property is only defined for compound candidates.

The set of search executor candidates is arranged in a special way, it stores only the candidates of the last matched terminal elements (tokens and links to templates). Moreover, each variant of the pattern matching, starting from a certain position in the text, is represented by a completely separate tree of candidates. The tree of candidates for each variant of matching the text with the template is linked to the set of candidates by the last matched terminal element. If the link disappears as a result of removing the terminal element from the set of candidates, then this means that the entire candidate tree falls out of the set of candidates, and if this did not happen as a result of a complete match, then the candidate tree turns into garbage, which must be disposed of by the garbage collector. Garbage collection in modern programming platforms is automatic; for other programming platforms, the disposal of garbage candidates must be done manually.

Algorithm of the search performer's work.

A block diagram explaining the general operation of the search executor 21 is shown in FIG. 3. In accordance with the block diagram, the sequence of tokens of the text 20 is transferred to the main loop of the search executor 31-40, in which operations are performed for each token T, starting from 32 and ending with 39. The essence of these operations is that, depending on the specific the values of the token T make a decision on the coincidence, mismatch, or the need for further verification of the candidates to be checked in the set of candidates 22, and also create candidates for patterns, the coincidence of which is possible starting from the token T. To do this, in a loop 32-34 for each candidate C in the set of candidate tokens Step 33 is executed in which a decision is made on the candidate tree corresponding to Candidate C using the token T. The steps in Step 33 are described in more detail in FIG. 4. After the completion of the processing cycle of candidates 32-34, at step 35, new candidates are created and added to the set of candidates obtained on the basis of searching in the root search index for patterns starting with the token T. The creation of new candidates consists of two steps: creating new candidates tokens and the creation of new candidate links to templates. To create candidate lexemes, a call is made to the search index, which returns a list of terminal search expressions of tokens that match the lexeme T. For each obtained expression, a candidate tree is built. This is done as you go up the expression tree. If, after the creation of candidates, the candidate limit has not been exceeded, then the candidates are also created with links to templates. Candidates

- 18,037156 template references are also terminal candidates, but they are stored in the candidate set separately from the token candidates. This is due to the fact that in order to make decisions in such candidates, it is not the lexeme of the text that is needed, but a matched template, the link to which is presented by this candidate. The creation of a candidate tree for a link is similar to the creation of a candidate tree for a token: ascending up the expression tree. At step 36, it is checked if the token T is an end-of-text token, and if so, then step 37 is performed to remove all candidates from the set of candidates. If the token T is any other token, then step 37 is skipped. At step 38, it is checked whether the candidate limit has been exceeded, and if it is, then at step 39, an end-of-text token is artificially inserted into the token sequence, which is processed at the next iteration of loop 31-40. After exiting the main loop 31-40 in the data of the search artist, the results of matches 24 remain accumulated. These data must be deleted before searching in other text.

General decision-making algorithm.

A flowchart explaining the general decision-making algorithm for the candidate tree corresponding to the terminal candidate C, depending on the value of the current token T, is shown in FIG. 4. The input data of the 41 algorithm are a candidate C from a set of candidates and a lexeme T in a certain position of the text. Candidate C is a terminal candidate and can be either a token candidate or a template reference candidate.

Decision making is performed with respect to the candidate tree corresponding to candidate C. First, at step 42, it is checked whether candidate C is marked as deleted, and if it is, then all decision steps, starting from 43 and ending with 51, are skipped, and at step 52, the algorithm exits. decision making. If candidate C is not marked as deleted, then at step 43 it is checked whether the candidate tree completely matches.

In accordance with the selected OOP methodology, the decision-making procedures are procedures of the objects by which the candidates are presented, therefore, to make a decision at step 43, the lexeme T is passed to the decision-making procedure of the candidate C. calling the decision-making procedure of the lead candidate. Thus, to make a decision, the candidate of the token is first addressed, and the candidate of the token, if necessary, delegates the decision to its head element, and so on up to the root element of the tree - the candidate of the template.

If in step 43 the candidate tree completely matches, then in step 44 it is checked whether the corresponding template is the target, i.e. tag, and if so, then in step 45 the root candidate of this tree (template candidate) is added to the result set. If the matching template is not a tag, then step 45 is skipped. After step 44 and regardless of step 45, i. E. in the event that the candidate tree completely matches, and regardless of whether the corresponding template is a tag, a decision is made for link candidates to this template at step 48. The decision is made for link candidates present in the set of candidates and starting with that the same position in the text from which the candidate of the matched pattern begins. In fact, step 48 means recursively referring to the decision algorithm of FIG. four.

If at step 43 the candidate tree does not completely match, then at step 46 it is checked that the candidate tree continues to match, and if not, then at step 47 the candidate tree is marked for deletion, and at the next step 48 a recursive call to the acceptance algorithm is performed solutions for candidates to link to a matching non-matched template.

If at step 46 it turns out that the candidate tree continues to match, then at step 49 it is checked if there is more than one match. If there are several matching options, then at step 50, logical copies of the candidate tree are created for each match option, and the logical copies coincide in the part of the expression that precedes the token T, and differ in that part of the expression that corresponds to different options for matching the token T with the tokens in search expressions of the template. If there is only one match, then step 50 is skipped, no copies of the candidate tree are created, but the existing candidate tree is used. In the next step 51, for each match in the candidate tree, a candidate token T is added, which in turn is added to the set of candidates.

Decision making rules.

Decision rules for complete match, non-match, or overlap of candidates depend on the types of candidates.

A common part of the decision rules in all candidates is the analysis of the current repetition number. If the repetition has not yet reached the minimum permissible boundary, then it is necessary to check whether a new repetition of the pattern can begin to match from the next token of the text, and if so, then create a candidate for a new repetition. In the case when a new repetition match is impossible, the head candidate receives information about the mismatch of its subordinate element. This information is passed up the tree up to its root. In the case when the repetition has reached the maximum permissible limit, the decision on the match is made in the current candidate, which is immediately transmitted to its head candidate. The most difficult case is the match of the candidate with the repeat number

- 19 037156 rhenium, which is between the minimum and maximum border. When an item with a variable number of repetitions is matched, there are two options to consider: when this match was the last one, and when another match will be searched for. Based on the verification result, the above options can form different candidate trees, which ultimately can lead to different matches of the corresponding template. In order to take into account all the options, it is necessary to copy the candidate tree up to the root element - the template candidate. After copying, one copy of the candidate is guided by the decision-making rules when the maximum number of repetitions is reached, and the second - by the decision-making rules in the case when the repetition has not yet reached the minimum permissible limit.

An analysis of the position number of the matched subelement is added to the decision logic based on the repetition number in the sequence. A sequence is considered to be matched when its mandatory part matches. A mandatory part of a sequence means its subsequence from the first to the last required element. For example, in the sequence? A + B +? C + D +? E, the required part is the subsequence from B to D with an optional C in the middle.

In the event that optional elements follow the mandatory part, you must also check whether they can match. To do this, at the moment of coincidence of the mandatory part of the sequence, as in the case of the variable value of repetition, the candidate tree is copied. One of the instances tries to continue matching the optional part of the sequence, and the other instance is considered a match.

For variation, the decision-making rules also have their own characteristics. This is due to the logic of the work of exceptions: the coincidence of an exception must cancel the coincidence of all alternatives, the coincidence of which began from the same position in the text.

In this case, such a cancellation can occur both before the match of the candidate of the alternative, and after, after a certain number of tokens of the text. That being said, the main use case for forward preview assumes the second case. A more complex scenario is undo based on repetitions of elements. For example, if the exception coincides with the position of some non-first repetition of an element, then only those elements whose position in the text is greater than the position of the match to which the exception belongs is canceled. The variation coincidence state (hereinafter simply the variation state) is used to communicate between candidate exception and alternative trees. The variation state stores references to all candidates in a special structure that is convenient for searching by the position of the first matching token. The variation state is associated with the variation search expression object and the token position on which the first variation item candidate was created. All candidate items of the same variation, created during the processing of a single token, are bound by the same variation state. In the process of making a decision in the candidate, the state of the variation is consulted to obtain additional information and update the state.

Due to the fact that all patterns are detected in one scan of the text, checking for exceptions in variations must be done simultaneously with searching for matches of the elements following the variation. In this case, the matched candidates of alternatives, for which the exceptions are still being checked, are considered incompletely matched. The property of an inconclusive coincidence extends to the head candidates as well. So, for example, if all the elements of the sequence coincide, but at least one of them at any position coincides incompletely, then the entire sequence is considered to coincide inconclusively.

An example of how the described mechanism works is to search for the pattern # P = {A, ~ (A + B)} in the input sequence of tokens AAB. Hereinafter, in the examples, the letters of the Latin alphabet denote tokens of search expressions and the corresponding lexemes of the input text. The above pattern identifies a token A not followed by a token B.

The step-by-step change in the state of the search executor is shown in FIG. 5. For simplicity, FIG. 5 shows a list of lexemes without the lexeme of the beginning of the text, since it does not affect the search for the template in question. Blocks 110, 120, 130, 140 correspond to snapshots of the search executor's state and show the change in this state in the direction from 110 to 140 as the text tokens are processed: tokens A at position 0 (A ₀ ), tokens A at position 1 (A1), tokens B at position 2 (B ₂ ) and end of text tokens END. Each of the snapshots 110, 120, 130, 140 shows the states for the set of candidates 22 and for the match results 24. The processing of each token consists of two steps: making decisions for the candidates created during the processing of the previous token (hereinafter simply making decisions), and creating new candidates based on the root search index (hereinafter simply creating new candidates).

During the processing of the first token A _0, no candidates were previously created, so the decision step is skipped and the candidates are created based on the root index. By

- 20 037156 token A, two tokens are extracted from the root index, referring to two elements of the variation: alternative and exclusion. According to them, candidates 113 and 117, as well as a tree of candidates with the root candidate of the template 111, are created respectively. In this case, the elements of the variation 112 and 115 are associated with one state of the variation 114. The final state after this step is shown in snapshot 110.

When processing the second token A1 in the text, a decision is made on the set of candidates and a match is found in the candidate tree corresponding to the root candidate 111 of the template P in position 0. This is due to the coincidence of its subordinate element - the alternative of the variation 112. The alternative of the variation 112 coincides by itself , since its subordinate element 113 matches. But its final match occurs only after the check of all variation exceptions created at the same position in the text completes with a negative result. The order in which the candidates are processed is irrelevant in this case. In the case where processing starts from candidate 117, it is first determined that the exclusion of variation 115 does not match, the decision for alternative 112 will be deferred, since the subordinate candidate of alternative 113 itself has not yet been processed. After the subordinate candidate of alternative 113 has been processed, it will be checked that there are no variations associated with the alternative, and a final match decision can be made. Alternatively, when processing starts from candidate 113, variation alternative candidate 112 will be processed, eventually a non-final decision will be made, since variation exception checking has not been completed. The information about the non-final match will be transferred to the head candidate 111 - the candidate of the template P, but in this case, saving to the result will not occur yet. Saving the matched template candidate 111 to the result will occur only after its final match as a result of processing and making a decision for both candidate 113 and candidate 117. In the scenario under consideration, this will happen after the decision about the variation exclusion mismatch is made. The variation state accumulates information about the coincidence or non-coincidence of alternatives and exceptions that started from the same position, and allows you to make the final decision on the coincidence of related variation alternatives after the final decisions on all associated variation exceptions are made. In particular, the state of the variation 114 will accumulate information about the coincidence of the alternative 112 and will allow making the final decision on the coincidence of the variation, starting from position 0, after the final decision is made about the mismatch of the associated exception 115, starting from the same position 0. At the stage of creating new candidates for the second token A1 in the text, a structure is created similar to the one that was created when processing the first token A0. The differences will only be in the starting positions of the tokens that are stored in the candidates. In particular, by lexeme A, two lexemes are extracted from the root index, referring to two elements of the variation: alternative and exclusion. According to them, candidates 123 and 127, as well as a tree of candidates with the root candidate of the template 121, are created respectively. In this case, the elements of the variation 122 and 125 are associated with one state of the variation 124. The final state after the processing step of token A1 is shown in snapshot 120.

The processing of the next token B ₂ in the text will result in the creation of the candidate of the token 138, which is the second element of the sequence 126 in the exception 125. At the same time, a decision will be made about the non-final coincidence of alternative 122 and the entire candidate of template 121, starting from position 1. Candidates 123 and 127 will be removed from the set of candidates, and candidate 138 will be added to the set of candidates. The candidate tree corresponding to the root candidate 121 is not garbage at this step because it is reachable via candidate 138 that is present in the candidate set. The final state after the B2 token step is shown in Figure 130.

When processing the END token of the end of the text, a decision will be made on the final match of the sequence 126 and the exclusion of the variation 125, starting from position 1. As a result, the final decision on the state of the variation 124 will lead to the cancellation of the match of the entire template candidate 121. The final state after the step of processing the token END the end of the text is shown in snapshot 140.

As a result, discussed in FIG. The 5 steps will, as expected, reveal one match of the pattern P at position 0. A reference to the candidate 111 of the given match will be stored in the result. Thanks to the two-way connections between subordinates and head candidates, from a template candidate it is possible to obtain detailed information about the coincidence of all its subordinate elements up to the coincidence of terminals with specific tokens of the text. In particular, by the template candidate 111, you can find out that its match is due to the match of the alternative variation 112, which matched due to the match of the candidate 113 of the lexeme A with the text token at position 0.

In the above rules, decision-making is carried out directly on the basis of the lexemes of the text. The search for references to templates and templates in the scope is not based directly on tokens, but on the coincidence of other templates.

Template references, like tokens, are terminal elements, but candidates for them are created according to more complex rules. They can be created by other candidates in the decision-making process, or chosen from the root search index. For links, these two cases are handled differently. In the decision-making process, candidates are created for all links that

- 21 037156 may be in some position. This is due to the fact that it is impossible to determine in advance whether a link match in a given position is possible. Each link candidate, once created, is registered in the link checking state, where it is associated with the referenced template and the position at which it is expected to match.

After decisions have been made, new link candidates are created, which have already been matched. To do this, a list of candidate tags is used that matched during the processing of the given token. For optimization, not all tags are saved to such a list, but only those to which there are links. This information is stored in the tag expression during template translation. Matching links are selected from the search index and candidate trees are generated for them. Unlike constants, when a link candidate is created, a decision is immediately made based on the next token. In the process, there may be cases where a link match immediately results in a match for the entire pattern. Such a template, in turn, may also have links. A simple example of such a situation is the recursive definition of # P =? P +? A

The given pattern will match any number of A tokens. To search for such a pattern, the list of pattern candidates that matched during token processing must be replenished at the time of decision making for links created on the basis of the same list. To prevent loops, for each new candidate, a check is performed to see if the candidate was processed in the same pattern that started matching from the same position.

Link candidate matches that were not generated from already matched templates are detected through the link validation state. After matching a template to which links are possible, a search is performed for link candidates waiting for it to match at a given position. If such candidates are found, they are linked and then a decision is made based on the next token.

The logic of searching for patterns in the scope is similar to searching for links. However, in the case of links, the start position of the pattern match is strictly specified, and the scope implies the match of one pattern within the bounds of another.

Unlike the link candidate, when a template candidate is created in scope, it is not registered anywhere. It gets into the state only after its child element matches. This is due to the fact that only at the moment of the match of the child element does the position of the end of the match become known.

For each tag that is used as a scope, a list of candidates waiting to be matched is stored. This list is sorted by start and end token number. This allows, when a pattern is matched, to quickly search for candidates that match within the given pattern. After a pattern matches, a binary search finds the first candidate in the sorted list that can be affected by the match. Starting from this position, the list checks the boundaries of the match for each candidate and transfers the information to the head candidate if a match is found in the scope. Because the list is sorted, the first candidate found that is outside the scope of the matched template stops scanning the list. Matching candidates are removed from the list.

Testing recommendations.

A feature of the invention is the high complexity of the underlying algorithm. Therefore, it is recommended to implement the device using the test-driven development methodology. Test-driven development involves first writing a test that covers the desired functionality and then creating the required functionality to pass the generated test. The tests generated during this development process are typically unit tests and test the functions and procedures of the objects. Having unit tests makes it much easier to make changes to existing code, allowing you to accurately detect bugs due to new changes. However, it should be borne in mind that for this invention, writing unit tests is quite laborious. This is due to the large number of possible states that arise in the decision-making process. In addition to unit tests, it is recommended that you create integration tests that check in a black box that the actual search results match the expected results. To test the considered embodiment of the invention, more than three hundred automated tests were implemented, which are divided into the following groups:

lexical text analyzer tests;

template parser tests;

syntax tree translator tests;

expression tests for representing templates at the search performer level;

tests of candidates;

link checking status tests;

pattern validation status tests in the scope;

- 22 037156 tests of the search index.

The unit tests are designed according to the test-driven design principle during device implementation and test isolated functionality. In unit tests, imitation objects are actively used that implement the program environment specified in the test case. Modeling the external environment in candidate decision-making tests is the most time-consuming, but such tests allow the most accurate localization of implementation errors.

Integration tests are pattern search tests and are presented in two groups: tests, in which templates and tokens are created as program objects, and tests, in which templates and text are represented as strings. This approach is necessary to test the functions of the search executor without binding to the translator from the template description language. Also, manually creating search expressions makes debugging easier. For simplicity, a lexical analyzer is not used to programmatically create test tokens. Instead, tokens are created by hand and are letters of the Latin alphabet. Integration tests of the second group check the same situations that arise in the decision-making process, however, they also allow you to check the joint operation of the lexical text analyzer, the translator from the template description language, and the search performer on real data.

A description of the main test cases is given in the following table ._______

Name Template Lexemes Coincidences 1. Lexeme BUT; BUT BUT 2. Lexeme with repetition [2] A; AA AA 3. Lexeme with variable repetition [1-2] A; AA AA 4. Sequence A + B; AB AB 5. Sequence with repetition [2] (A + B); AB AB AB AB 6. Sequence with variable repetition [1-2] (A + B); AB AB ABAB 7. Sequence with repeating nested sequence [2-3] (A + B) + C; AB ABC AB ABC 8. Sequence starting with an optional element ? A + B AT AT 9. Sequence ending with an optional element A +? B BUT BUT 10. Sequence with an optional element in the middle A + 7A + B AB AB 11. Sequence with an optional element in the middle containing an exception A +? {B, ~ B} + B AB AB 12. Nested optional sequence C + (? A +? B) + D CABD CABD 13. Variation {A, B} AB A, B 14. Variation with repetition [2] {A, B} AB AB 15. Variation with variable repetition [1-2] {A, B} AB AB 16. Variation with an exception {A, CHA + B)} AAB BUT 17. Repetitive variation with exclusion [1-3] {C, D, ~ (C + D + B)} CCDB C, D 18. Nested exceptions {A, ~ {A + D, - {A + D + C}} ADC BUT 19. Even number of identical nested exceptions {A, ~ {A, ~ A}} BUT BUT

- 23 037156

20. An odd number of identical nested exceptions {L, ~ {A, CHA, ~ A}}} BUT - 21. Elimination on the second rep [1-2] {C, B, ~ {B + D}}} CBD WITH 22. Variation of optional elements in sequences C + {? A,? B} + D CD CD 23. Cancellation of several alternatives [1-3] {A, B, B + D, ~ (A + B + D)} AABD A, BD 24. Exception in the middle of a sequence {A, ~ (A + B + C + D)} + B AND ALL AB 25. Multiple exceptions of varying length {A, ~ (A + D), ~ (A + B + C)} ABCEAD - 26. Eliminate shorter length than alternative {A + B + C, ~ (A + D)} ABE - 27. Intersection of matches of different lengths [3] {A, B, ~ (B + A + B)} AVABAABB a ₂ va, abv 28. Link in the middle of the sequence # P1 = A + P2 + C; # P2 = B; ABC ABC (# P1), B (# P2) 29. Optional link in the middle of the sequence # P1 = A +? P2 + C; # P2 = B; AS AC (# P1) 30. Duplicate link # P1 = A + [2] P2 + C; # P2 = B; ABBC ABBC (# P1), Bi (# P2), B ₂ (# P2) 31. Link chain MF SP Ri Ri <SO O II II II - Ν P Ri Ri Ri 4k Chk Chk ABC ABC (# Р1), BC (# Р2), С (# РЗ) 32. Multiple links to one template and JV SP · Ri Ri CO + + + <<co II II II Ri Ri Ri Chk Chk Chk ABBC ABBC (# P1), ABB (# P2), BB (# RZ) 33. Variation of links # P1 = A + {P2, P3} + C; # P2 - B; # РЗ = B + C; ABCC ABCC (# Р1), В (# Р2), ВС (# РЗ) 34. Link in exclusion # P1 = {A, ~ P2}; # P2 = A + B; AAB A (# P1), AB (# P2) 35. Reference in variation with an exception # P1 = {P2, ~ (A + B + C)}; # P2 = A + B; ABAVS АоВ (# Р1), АоВ (# Р2), А ₂ В (# Р2) 36. Right recursion # P = A +? P AAA AAA

- 24 037156

37. Left recursion # P =? P +? A AAA AAA 38. Template inside # P1 = B @ P2; ABCV B (# P1), scope # P2 = A + B + C; ABC (# P2) 39. Pattern on the border # Р1 = А @ Р2; ABCV A (# P1), scope # P2 = A + B + C; ABC (# P2) 40. Template covering the entire scope # P1 = (A + B) @ P2; # P2 = A + B; AB AB (# P1), AB (# P2) 41. Optional template in scope # P1 = A +? B @ P2; # P2 = A + B + C; ABCAB AB (# P1), Az (# P2) 42. Nested scopes # P1 = А @ Р2 @ РЗ @ Р4; # P2 = B + A + B; # РЗ = С + В + А + В + С; # P4 = D + C + B + A + B + C DCBABCD A (# P1), BAB (# P2), SVAVS (# RZ), DCBABCD + D; (# P4) 43. Link in the middle # P1 = A + P2 + C; ABC ABC (# P1) sequences; one tag P2 = B; 44. Repetitive # P1 = A + [2] P2 + C; ABBC AVVS (# P1) link; one tag P2 = B; 45. Chain of links; one tag # P1 = A + P2; P2 = B + RZ; RZ = C; ABC ABC (# P1) 46. Several links to one template; one tag # P1 = A + P3 + C; P2 = A + P3; RZ = B + B; ABBC AVVS (# P1) 47. Variation of links; one tag # P1 = A + {P2, P3} + C; P2 = B; RZ = B + C; ABCC ABCC (# P1) 48. Link in exclusion; one tag # P1 = {A, ~ P2}; P2 = A + B; AAB A (# P1) 49. Link in variation with # P1 = {P2, ~ (A + B + C)}; AB ABC AoB (# P1) exception; one tag P2 = A + B; 50. Template within the scope; one # P1 = B @ P2; P2 = A + B + C; ABCV B (# P1) tag 51. Template on the border of the scope; one # Р1 = А @ Р2; P2 = A + B + C; ABCV A (# P1) tag 52. A template that covers the entire # P1 = (A + B) @ P2; AB (# P1) scope; one P2 = A + B; Ab tag 53. Optional template in scope # P1 = A +? B @ P2; ABCAB AB (# P1) actions; one tag P2 = A + B + C; # Р1 = А @ Р2 @ РЗ @ Р4; P2 = B + A + B 54. Nested areas RZ = C + B + A + B + C; DCBABCD A (# P1) actions; one tag P4 = D + C + B + A + B + C + D;

Performance tests.

Performance tests of the created embodiment were performed. The tests compared the search times for patterns and their equivalent regular expressions on the same set of texts. The following test bench environment was used:

OS: Windows 10x64;

software platform: .NET Core 2.1.104;

runtime environment: .NET Shared Framework Host 2.0.5;

processor: Intel Core i7-8700 3.20 GHz;

RAM: 64 GB.

As a set of texts, 19 text files with a total volume of 207,316 characters were used. The files were created from articles from news portals dedicated to business and finance. The tests used the standard regex implementation that ships with the .NET Core framework in use.

As part of the test, the search execution time was measured directly. Both text data and templates were preloaded into memory. This increased the accuracy of the comparison, excluded long-term disk accesses, the time of which could differ significantly from test to test. The creation of regular expression objects and the translation of templates also took place during the preparatory stage for testing. To reduce the impact of dynamic compilation of code used in .NET on

- 25 037156 during the preparation phase, several search iterations were also launched.

The comparison was made for three different classes of patterns: lexeme variations;

words at a distance;

complex patterns: phone, url, email, hash tag.

For the first two classes, templates were generated automatically based on company names and their stock labels (tickers). For each of the tests, 3383 templates were generated. In the examples below, A stands for the full name of a company, and B stands for its stock ticker.

Variation patterns of tokens reveal company mentions. In the template description language, they look like {A, B}

Equivalent regular expressions have the form (? I) \ bA \ b | \ bB \ b where the directive (? I) activates case-insensitive search mode, a \ b denotes a word boundary.

Remote word patterns reveal mentions of a company in the context of its stock ticker. A mention is considered to be in the text of the name of the company and its stock ticker at a distance of no more than five words. In the template description language, an expression for searching for such constructions is {A .. 0-5 .. B, B. ... 0-5 .. A};

Due to its cumbersomeness, a regular expression with a similar meaning is not given in its entirety. The regular expression (\ bA \ b) (?: [\ S ,.!? ()] + \ W +) {0, 5}? [\ S,. :; ! ? ()] + (\ bB \ b) where between A and B describes a construction of alphabetic or alphanumeric sequences separated by special or whitespace characters, repeated no more than five times.

Sophisticated templates are hand-written templates for identifying phone number, URL, email, and hash tag. The following definitions are used for the template description language:

#PhoneNumber =? + + {Num, (+ Num +)} + [2+] Space} + Num);

#Email = Word + [0+] {Word, +, -} + + Domain;

#Url = {http, https} +: // + Domain +? Path +? Query;

Domain = Word + [1 +] + Word + [0 +] {Word,, -});

Path = / + [0+] {Word, /,%};

Query =? + [0-1] (QueryParam + [0 +] (& + QueryParam));

QueryParam = Identifier + = + Identifier;

Identifier = {AlphaNum, Alpha, _} + [0 +] {Word, _};

#HashTag = # + Identifier;

In regular expression language, the phone number, email address, URL, and hashtag are described, respectively, by (? <= \ S) \ +? (\ D + | \ (\ d + \)) ([- \ s] \ d + ) {2,} (? = \ S)

[a-zA-Z0-9 _. + -] + @ ([\ w -] + {?: \. [\ w -] +) *) (https?: \ / \ /) ([\ w-] + {?: \. [\ w -] +) *) (\ / [\ w \ /% +] +)? (?: \? ((\ w + = \ w +) (?: & (\ w + = \ w +)) *))?

\ B (\ # [a-zA-Z] + \ b)

Both the number of test runs to calculate the mean and the number of dummy iterations were adjusted during the run of the test tests and the determination of the search time values for all three sets of templates. The running time of the pattern search tests and the test of searching for complex structures using regular expressions does not exceed a second, so ten dummy runs and twenty measured iterations were used to increase the accuracy. A regex search for tests with a large number of patterns takes more than ten seconds, so a dummy run: no iterations were performed, and three iterations were used to measure.

The measurement results and the rounded n-value showing the ratio of the regular expression search time to the pattern search time are shown in the following table.

- 26 037156

Test type Regular expression search time, seconds Pattern search time, s P Variations of constants 14 0.047 298 Following through words 15 0.2 75 Complex constructions 0.051 0.155 0.33

The test results turned out to be predictable: in the first two tests, when searching for 3383 constructs, the pattern search turned out to be much faster. This is due both to the ability to identify matches with many templates in one look at the text, and to work at the token level. In the test of searching for four complex structures, where the work is proceeding at a level close to the level of symbols, the pattern search was only about three times slower.

Conclusion.

Thus, it should be obvious to an ordinary specialist in this field that a device constructed in accordance with the proposed invention provides a quick search for matches with patterns in the text by pre-parsing the text into tokens and checking all matches of all patterns in one sequential scan of the text tokens. Achieved high speed, completeness and accuracy of the search, independence of the search from the language of the text and security, which means avoiding delaying the search for a long time, perceived by the user as a loop or endless search. The device constructed in accordance with the proposed invention allows you to quickly identify sets of concepts, entities and their relationships in natural language texts.

The invention is disclosed in some embodiments. It should be obvious that the disclosed device can be implemented with modifications without departing from the invention. The appended claims cover many variations and modifications that fall within the spirit of the present invention.

Claims (4)

1. A method of searching for matches with patterns in the text to identify a given set of concepts, entities and relationships in a text in a natural or machine-readable language, characterized by the fact that A) the text is pre-parsed into lexemes, which include at least words and word separators; B) in the template description language, create a set of templates corresponding to the given concepts, entities and relations, in which each template is a formal grammar consisting at least of sequences and (or) variations and (or) repetitions of text tokens, and (or) occurrences of other patterns; C) translate a set of templates into search expression trees with search indexes, which allow for a given token or a given template identifier to quickly find all search expressions that begin with a given token or a given template; D) for a given set of templates, a set of candidates is created, each of which stores information about matching text tokens with elements of the search expression tree for matching, and the matching order corresponds to the order of traversing the tree from leaves to the root, matching a sequence requires matching all its elements in a given order , the match of the variation requires the match of at least one of its elements, the match of the repetition requires the match of its element a specified number of times, the match of the text lexeme is performed, at least with or without capital and lowercase letters; E) then scan the lexemes of the text once sequentially and for each token perform at least the following actions: i) all patterns starting with the current token are searched in the search indices, candidates are created to check for text matches with these patterns, and they are added to the set of candidates; ii) for each candidate from the set of candidates, the current token of the text is compared with the next element or elements of the candidate search expression tree for a match; iii) if the next element of the search expression tree coincides with the current token of the text and is the last in the traversal order, for example, the root, then the candidate is considered to be completely matched and is transferred from the set of candidates to the set of results; if the next element of the search expression tree coincides with the current token of the text and is not the last in the traversal order, then the candidate is considered partially matched and is left in the set of candidates; if the next element of the search expression tree does not match the current token of the text, then the candidate is considered non-matching and is removed from the set of candidates; iv) to take into account different variants of text coincidence with the checked templates, create and add to the set of candidates logical copies of those candidates for which different variants of matching with the text are possible, and the logical copies of the candidates contain the same information about the coincidence of elements of the search expression tree with already scanned text tokens, and diversified - 27 037156 information about the match of an element or elements of the search expression tree with the current text token. 2. The method of searching for matches with patterns in the text according to claim 1, in which A) the templates being checked allow recursive definitions through the same templates and / or through other templates; B) the number of candidates created in the search process and (or) the amount of resources consumed by candidates and (or) the depth of recursion when enumerating matching options is limited; 3. The method for searching the text for matches with patterns according to claim 1, wherein the patterns support parameters for generalizing patterns and / or for refining search results. 4. The method for searching the text for matches with patterns according to claim 2, wherein the patterns support parameters for generalizing patterns and / or for refining search results.