RU2518946C1 - Method for automatic semantic indexing of natural language text - Google Patents

Abstract

FIELD: information technology.

SUBSTANCE: method for automatic semantic indexing of natural language text comprises segmenting the text into elementary first level units (words) and sentences; forming second level units (standardised word forms); calculating the frequency of occurrence of each first level unit for adjacent first level units and merging the sequence of words into third level units (stable word combinations); identifying in each sentence a semantically significant entity and an attribute thereof (fourth level units); identifying in each sentence semantically significant relationships between semantically significant entities and between semantically significant entities and attributes; determining the frequency of occurrence of second level and third level units; forming, for each semantically significant relationship, a plurality of triads (fifth level units); on the plurality of the formed triads, separately indexing all semantically significant entities linked by semantically significant relationships with their frequency of occurrence, all attributes with their frequency of occurrence and all formed triads.

EFFECT: high accuracy of indexing natural language texts.

6 cl, 2 dwg, 23 tbl

Description

FIELD OF THE INVENTION

The present invention relates to the field of information technology, and in particular to a method for automated semantic indexing of text in a natural language.

State of the art

Currently, various methods are known for automated indexing of texts in natural languages.

For example, in RF patent No. 2268488 (published on January 20, 2006), a method is described in which words, phrases, idioms, sentences are encoded for subsequent numerical processing. In the patent of the Russian Federation No. 2273879 (publ. 10.04.2006) a method for morphological and syntactic analysis of the text with subsequent indexation is disclosed. The EAPO patent No. 002016 (published on January 22, 2001) discloses a method in which unique blocks of information are determined in text fragments and used for subsequent processing and search. The method according to US patent No. 6871174 (publ. 22.03.2005) allows you to determine the similarity of texts by text fragments. In US patent No. 6189002 (publ. 13.02.2001) disclosed a method in which the text is divided into paragraphs and words, which are converted into vectors of ordered elements. Each element of the vector corresponds to a paragraph found by applying a given function to the number of occurrences in this paragraph of the word corresponding to this element. The text vector is considered as the semantic profile of the document. However, all these methods do not take into account the semantic ambiguity of words and expressions of the natural language.

Accounting for semantic ambiguity is carried out in many well-known methods. For example, in RF patent No. 2222048 (publ. 10.12.2004), in US patents No. 6871199 (publ. March 22, 2005), 7024407 (publ. 04.04.2006) and 7383169 (publ. 03.06.2008), in applications for US patent No. 2007/0005343 and 2007/0005344 (both publ. 04/04/2007), 2008/0097951 (publ. 04.24.2008), in Japanese applications laid out No. 05-128149 (publ. 05/25/1993), 06- 195374 (publ. 07/15/1994), 10-171806 (publ. 06/26/1998) and 2005-182438 (publ. 07/07/2005), in EPO application No. 0853286 (publ. 15.07.1998) methods are described that provide one or another way to eliminate the ambiguity of the words and / or expressions found in the texts. However, all these methods have only private application and do not affect the full semantic indexing of the text.

US Patent Application No. 2007/0073533 (published March 29, 2007) describes a method in which features such as a named entity, identity by reference, lexical article, semantic-structural relation, attributive attribute are distinguished from a segmented text from a segmented text. and meronymic information. Then, for each section of the text, its constituent structure is determined in the form of a list of constituents and their sequence. The functional structure is determined for each section of the text and, based on the functional structure, the predicate triples are found using linearization rules for transferring values. The constituent structure of the text section, the list of canonized predicate triples and the identified features are combined to form a canonized representation of the text section, from which the entire text index is formed. This method is limited due to the fact that only predicate triples are used to form the index, the remaining arguments of the extended predicate structure remain unused, which impairs the accuracy of semantic analysis.

Closest to the claimed group of inventions, a method for automated semantic indexing of text in a natural language is disclosed in the patent of the Russian Federation No. 2399959 (publ. 09/20/2010). In this method, text in digital form is segmented into elementary units of the first level (words); form for each elementary unit of the first level (words) an elementary unit of the second level (normalized word form); segment text in digital form into sentences corresponding to sections of indexed text; reveal in the text, in the process of linguistic analysis, elementary units of the third level (stable phrases); in the process of multi-stage semantic-syntactic analysis by referring to linguistic and heuristic rules preformed in the database in a predefined linguistic environment, elementary units of the fourth level (semantically significant object and its attribute) and semantically significant relationships between the identified semantically are revealed in each of the sentences significant objects, as well as between semantically significant objects and attributes; form within this text for each of the identified semantically significant relationships, a set of elementary units of the fifth level (triads); index on a set of formed triads all semantically significant objects connected by semantically significant relations, as well as attributes individually and all triads of the form “semantically significant object - semantically significant relation - semantically significant object”, as well as all triads of the form “semantically significant object - semantically significant” meaningful relation - attribute ”; save the generated triads and the resulting indexes in a database together with a link to the source text from which these triads are formed.

The disadvantage of this method is the lack of ranking of the formed elementary units of the second, third and fourth levels, which leads to an unreasonably large amount of calculations associated with the need to use the entire generated index for further processing.

Disclosure of invention

The purpose of the present invention is to expand the arsenal of methods for indexing texts in natural languages through the use of methods for their automated linguistic analysis and the subsequent use of its results for constructing semantic indexes.

Achieving this goal and obtaining the indicated technical result are provided in the present invention by means of a method for automated semantic indexing of text in a natural language, which consists in the following: represent indexed text in digital form for subsequent automatic and (or) automated processing; segment the text in digital form into elementary units of the first level, including at least words; Segment digitally into sentences according to graphematic rules; form on the basis of morphological analysis for each elementary unit of the first level, which is a word, an elementary unit of the second level, which includes a normalized word form, hereinafter referred to as the lemma; calculate the frequency of occurrence of each elementary unit of the first level for two or more adjacent units of the first level in this text and combine among the elementary units of the first level the sequence of words following each other in this text into elementary units of the third level, which are stable combinations of words, if, for every two or more consecutive words in a given text, the difference in the calculated frequencies of occurrence of these words for the first occurrence of a given sequence with s and a number of subsequent occurrences for each pair of sequences of words remain unchanged; identify, in the process of multi-stage semantic-syntactic analysis by referring to linguistic and heuristic rules pre-generated in the database in a predefined linguistic environment, in each of the generated sentences, a semantically significant object and its attribute, which are units of the fourth level; store in memory every semantically significant object and attribute; reveal, in the process of multi-stage semantic-syntactic analysis by referring to linguistic and heuristic rules pre-generated in the database in a predefined linguistic environment, in each of the generated sentences there are semantically significant relations between the identified units of the fourth level - semantically significant objects, as well as between semantically significant significant objects and attributes; assign to each semantically significant relation the corresponding type from the subject ontology stored in the database on the subject matter of the subject area to which the indexed text belongs; store in memory each semantically significant relation together with the type assigned to it; identify the frequency of occurrence of elementary units of the fourth level throughout the text; form within the given text for each of the identified semantically significant relations, linking both the corresponding semantically significant objects and the semantically significant object and its attribute, a multitude of triads that are elementary units of the fifth level; index on a set of formed triads individually all semantically meaningful relations related semantically significant objects with their frequencies of occurrence, all attributes with their frequencies of occurrence and all formed triads; they store in the database the formed elementary units of the second, third, fourth and fifth levels with their frequencies of occurrence, as well as the obtained indices, together with links to specific sentences of this text.

A feature of the method of the present invention is that for each unit of the fourth level they can fix the identity by reference between the corresponding semantically significant object, as well as the attribute, and the corresponding anaphoric link if it is in the indexed text, replacing each anaphoric link with the corresponding antecedent.

Another feature of the method of the present invention is that from these triads can form a semantic network so that the first elementary unit of the second or third level of the subsequent triad is associated with the same second elementary unit of the second or third level of the previous triad. Moreover, before storing the generated triads and obtained indices in the database, the iterative procedure renormalizes the frequencies of occurrence into the semantic weight of the elementary units of the second and third levels, which are the vertices of the semantic network, so that the elementary units of the second and third levels connected in networks with a large number of elementary units of the second and third levels with a high frequency of occurrence increase their semantic weight, while other elementary units of the second and third ovney lose it evenly.

Another feature of the method of the present invention is that they can rank the formed elementary units of the second and third levels by semantic weight by comparing their semantic weight with a predetermined threshold value.

Finally, another feature of the method of the present invention is that they can remove triads in which elementary units of the second and third levels have a semantic weight below the threshold.

Brief Description of the Drawings

The present invention is further explained by the description of a specific example of its implementation and the accompanying drawings.

Figure 1 shows the conditional block diagram explaining the claimed method.

Figure 2 shows the text processing scheme based on multi-stage semantic-syntactic analysis to identify semantically significant objects, attributes and semantically significant relationships.

DETAILED DESCRIPTION OF THE INVENTION

The method of the present invention can be implemented in almost any computing environment, for example, on a personal computer connected to external databases. The steps of the method are illustrated in FIG.

All further explanations are given as applied to the Russian language, which is one of the most highly inflected languages, although the proposed method is applicable to the semantic indexing of texts in any natural languages.

First of all, the text to be indexed must be submitted in electronic form for subsequent automated processing. This step in figure 1 is conventionally indicated by the reference number 1 and can be performed in any known manner, for example, by scanning the text and subsequent recognition using well-known means such as ABBYY FineReader. If the text is sent for indexing from an electronic network, for example, from the Internet, then the stage of its submission in electronic form is performed in advance, before this text is posted on the network.

Professionals should be clear that the operations of this and subsequent steps are carried out with storing intermediate results, for example, in random access memory (RAM).

The text converted into electronic form is sent for processing, during which the text is first segmented into elementary units of the first level, including at least words. In the mentioned patent of the Russian Federation No. 2399959, these elementary units of the first level are called tokens. A token can be any text object from the following set: words consisting of each sequence of letters and, possibly, hyphens; sequence of spaces; punctuation marks; numbers. Sometimes sequences of symbols such as А300, i150b, etc. are also included here. Tokens are always allocated according to fairly simple rules, for example, as in the aforementioned RF patent No. 2399959. In figure 1, this step is conventionally indicated by the reference position 2.

Following this, in step 3 (FIG. 1), the indexed text is digitally segmented into sentences corresponding to sections of the given text. Such segmentation is carried out according to graphematical rules. For example, the simplest rule for highlighting sentences is: “A sentence is a sequence of tokens, starting with a capital letter and ending with a period”.

Further, for each elementary unit of the first level (for each token), which is a word, on the basis of morphological analysis, the corresponding elementary unit of the second level is formed, which is a normalized word form, hereinafter referred to as the lemma. For example, for the word “go” the normalized word form will be “go”, for the word “beautiful” the normalized word form will be “beautiful”, and for the word “wall” the normalized word form will be “wall”. In addition, for each word form, the part of speech to which the given word belongs and its morphological characteristics are indicated. Naturally, for different parts of speech, these characteristics are different. For example, for nouns and adjectives it is a gender (masculine - feminine - average), number (singular - plural), case; for verbs, it is a form (perfect - imperfect), person, number (singular - plural), etc. Thus, for a given word, its normalized word form (lemma) + morphological characteristics, including part of speech, are its morph. One and the same word can have several morphs. For example, the word "glass" has two morphs - one for a noun of the middle gender and one for a past tense verb. This step is conventionally indicated in figure 1 by reference numeral 4.

The next step, conventionally indicated in FIG. 1 by reference numeral 5, is that for each of the mentioned first level elementary units, the frequency of occurrence is calculated in the said text. In other words, they determine how many times each word occurs in the processed text. This operation is carried out automatically, for example, by simply calculating the frequency of occurrence of each token, either as described in the patent of the Russian Federation No. 2164450 (publ. 05.20.2001), or in the aforementioned US patent No. 6189002. Simultaneously with the calculation of the frequency of occurrence, for every two or more words that follow in a given text, the differences in the calculated frequencies of occurrence of these words are found in the first occurrence of this sequence of words and in their subsequent occurrences. If these differences for the first occurrence of a given sequence of words and for several subsequent occurrences of them remain unchanged, such a sequence of words following each other in this text (i.e., elementary units of the second level) is combined into elementary units of the third level, which are stable collocations.

Further, in the next step, indicated in FIG. 1 by reference numeral 6, in order to identify semantically significant objects and attributes, a multi-stage semantic-syntactic analysis is performed. Such multistage semantic-syntactic analysis is performed by referring to the linguistic and heuristic rules generated in the database in a predetermined linguistic environment. Such an environment may be, for example, the linguistic environment mentioned in the aforementioned application for US patent No. 2007/0073533, or in the above patents of the Russian Federation No. 2242048 and the Russian Federation No. 2399959, or any other linguistic environment that defines the relevant rules that allow you to eliminate syntactic and semantic ambiguities of words and expressions of the real text. Linguistic and heuristic rules in the selected environment are referred to below as rules.

The identification of semantically significant objects and attributes, which are considered elementary units of the fourth level, is performed in the sentence on the set of elementary units of the first, second and (or) third levels.

For each semantically significant object and attribute, i.e. elementary units of the fourth level, with the type assigned to it, find the corresponding anaphoric link (if any). For example, in the sentence “Mechanics is a part of physics that studies the laws of mechanical motion and the causes that cause or change this movement”, the anaphoric reference to the word “mechanics” will be the pronoun “which”, while the word “mechanics” will be an antecedent for this anaphora, and also, the anaphoric reference to the word “mechanical” will be the pronoun “this,” while the word “mechanical” will be an antecedent for this anaphora. This step of finding the anaphoric link is conventionally indicated in FIG. 1 by the reference numeral 7. Each anaphoric link is replaced with its corresponding antecedent. After that, each identified semantically significant object and attribute is stored in the corresponding memory.

At the next stage, denoted by reference numeral 8 in FIG. 1, a multi-stage semantic-syntactic analysis is performed, with which, on the basis of elementary units of the first, second, third and fourth levels, semantically significant relations between semantically significant objects are found using the above rules, and between semantically significant objects and attributes.

At the step indicated by reference numeral 9 in FIG. 1, each semantically significant relation is assigned the corresponding type from the subject ontology stored in the database on the subject of the subject area to which the indexed text belongs. After that, each semantically significant relation is stored in the corresponding memory together with the type assigned to it and the morphological and semantic attributes found for it.

After that, at the stage indicated in FIG. 1 by reference numeral 10, the occurrence frequencies of semantically significant objects and attributes are identified throughout this text. This operation is performed in almost the same way as in step 4 for elementary units of the second level.

In the step indicated by reference numeral 11 in FIG. 1, stored semantically meaningful objects, as well as attributes, and semantically meaningful relationships are used to form triads. Moreover, within the limits of the text being indexed, for each of the identified semantically significant relations that connect certain semantically significant objects, many triads of two types are formed. Each of the many triads of the first type includes a semantically meaningful relationship and two semantically meaningful objects that are linked by this semantically meaningful relationship. Each of the many triads of the second type includes a semantically significant relation, one semantically significant object, as well as its attribute, which are associated with this semantically significant relation. If two semantically significant objects are denoted by O _i and O _j , and the semantically significant relation connecting them is denoted by R _ij , then each of the triads of the first type can be conditionally represented (depicted) as O _i → R _ij → O _j . Each of the triads of the second type can be represented as O _i → A _m , where A _m are the corresponding attributes. In these entries, the indices i, j, m are integers.

Then, in the step indicated in FIG. 1 by reference numeral 12, the text is indexed. At the same time, individually associated with semantically significant relations, all semantically significant objects with their frequencies of occurrence, all attributes with their frequencies of occurrence, and all formed triads are indexed individually on the set of formed triads.

To do this, on the set of formed triads, the entire semantically significant object and its attribute are individually indexed, with their frequencies of occurrence, and all triads of the form “semantically significant object - semantically significant relation - semantically significant object”, as well as all triads of the form “semantically significant object - semantically significant relation is an attribute. ” The indices generated in step 11 of the triad and the indices obtained in step 12, together with a link to specific sentences of the source text from which these triads are generated, are stored in the database (step 13 in FIG. 1).

For specialists, it is obvious that the storage devices mentioned at separate stages can in fact be both different devices, and one storage device of sufficient volume. Similarly, the individual databases mentioned at the respective stages can be not only physically separate databases, but also the only database. Moreover, said storage devices (memories) may be executed on the same single database, or be combined with one of said databases. Those skilled in the art will also understand that the methods claimed in the present invention are executed in an appropriate computing environment under the control of appropriate programs that are recorded on computer-readable media intended for direct participation in a computer.

A feature of the method of the present invention is that from these triads can form a semantic network so that the first semantically significant object of the subsequent triad is associated with the same second semantically significant object of the previous triad. Moreover, before storing the generated triads and obtained indices in the database, the iterative procedure performs renormalization of the frequencies of occurrence of semantically significant objects and attributes into the semantic weight of semantically significant objects and attributes that are the vertices of the semantic network, so that the semantically significant object and attribute connected in a network with a large number of semantically significant objects and attributes with a high frequency of occurrence, increase their semantic weight, while others semantically a significant object and its attribute uniformly lose it (step 14 in FIG. 1).

Another feature of the method of the present invention is that the generated semantically significant object and its attribute can be ranked by semantic weight by comparing their semantic weight with a predetermined threshold value (step 15 in FIG. 1).

Finally, another feature of the method of the present invention is that they can remove triads in which the semantically significant object and its attribute have a semantic weight below the threshold (step 16 in FIG. 1).

Example

To illustrate the implementation of the claimed method of automated semantic indexing of text in a natural language, consider the following example. Let there be some Russian-language text on the physics course presented on the Internet site http://www.kodges.ru/. Thus, we can assume that the conversion of texts into electronic form, indicated in FIG. 1 by reference numeral 1, has already been completed.

A typical example of such a text is the following fragment from the textbook of T. I. Trofimova “The Course of Physics”, Moscow, “Higher School”, 2001: “Mechanics is a part of physics that studies the laws of mechanical motion and the causes that cause or change this movement. Mechanical movement is a change over time in the relative position of bodies or their parts. ... "

In accordance with the claimed method of automated semantic indexing of text in a natural language, a previously created base of syntactic rules and dictionaries is used, within which text processing and construction of a semantic index will be carried out. Such databases are prepared by linguistic experts who, based on their experience and knowledge, determine the sequence and composition of the syntactic processing of the text that are specific to a particular language.

Linguistic experts preliminarily construct a variety of graphematical and syntactic rules that allow using the linguistic dictionaries previously constructed by expert linguists to automatically identify further specific information in the processed texts corresponding to semantically significant objects and attributes and semantically significant relationships that may have place in each pair of semantically significant objects or in each pair of semantically significant object and its attribute.

In addition to the specification of the subject area and the rules in accordance with the above methods, dictionaries of general and special vocabulary are used.

In accordance with the claimed method of automated semantic indexing of text in a natural language, the text is first segmented into elementary units — tokens (reference position 2 in FIG. 1) and morphological analysis of tokens representing words (reference position 3 in FIG. 1). As a result of this step, the source text is transformed into many tokens and morphs, which are presented in Table 1 and Table 2, respectively.

Then, after segmenting the text into tokens and morphological analysis of tokens - words, stable phrases are allocated (reference position 4 in FIG. 1). To do this, calculate the frequency of occurrence of words in sequences of two or more words in the text. Then, the differences in the frequencies of occurrence of words in the sequence are compared for the first occurrence of a given sequence of words and for several subsequent occurrences of them.

The frequency of occurrence of words at the first appearance of the sequence and at its subsequent appearance, as well as the difference of these frequencies are presented in Table 3.

As a result of this stage, the source text, in addition to elementary units of the first and second levels, is supplemented by many units of the third level - stable phrases. The phrases for our example are presented in Table 4.

After the above steps are completed, the processed text is fragmented into sentences (reference position 5 in FIG. 1). As a result of this step, the sets formed above are supplemented by the set of sentences presented in Table 5.

Thus, after all the above steps are completed, the processed text will be segmented into sentences, each of which is marked with sets of annotations of elementary units of the first, second and third levels.

Following this, in accordance with the claimed method, semantically significant objects and attributes (elementary units of the fourth level) are identified (reference position 6 in FIG. 1). It is made in each sentence on the set of elementary units of the first, second and (or) third levels by applying the aforementioned pre-formed set of linguistic and heuristic rules using the pre-formed corresponding linguistic dictionaries.

So, for example, in the sentence “Mechanics is a part of physics that studies the laws of mechanical motion and the causes that cause or change this motion” of the text in question using a variety of rules, the corresponding signal processing scheme is presented in Figure 2 (processing points 1-7) , and the dictionaries used in this rule are presented in Tables 6-16, the semantically significant object “mechanics” is highlighted. Other semantically significant objects and their attributes are distinguished using the same set of rules. As a result, a semantically significant object and its attribute are distinguished in the source text. The set of such semantically significant objects and attributes for the considered example are presented in Table 17.

Next, we construct simple syntactic groups corresponding to the attribute level of description (Tables 7 and 8): feature of an object / subject / action + object / subject / action, measure of the feature of an object / subject / action + object / subject / action.

Further in sentences of the text anaphoric links are revealed and revealed (if they are in the indexed text). To do this, within the entire text being processed, during the stage indicated in FIG. 1 by the reference numeral 7, pronouns are found that can be anaphoric references to the corresponding words, and for pronouns that really are, they fix the identity by reference between the corresponding semantically significant object and its anaphoric reference. So in the sentence “Mechanics is the part of physics that studies the laws of mechanical motion and the causes that cause or change this movement”, the pronoun “this” is replaced by the word “mechanical”: “Mechanics is the part of physics that studies the laws of mechanical motion and the causes that cause or changing mechanical motion. "

After that, the fragments are combined into simple sentences, including using subordinate unions (Table 10).

Then, the predicative minimum (including the main semantically significant objects) of the sentence is identified by comparing its structure with the dictionary of templates of minimal structural sentence schemes (Table 11). The result for our example is shown in Table 12.

Next, the construction of syntactic groups is carried out inside the received simple sentences, in which the predicate actants are the main words, with the help of syntactic rules that reveal the syntactic relations between words. The constructed groups are shown in Table 13.

The selection of the remaining members of a simple sentence (other semantically significant objects and attributes) and semantically significant relationships is carried out by a consistent comparison of the words of the sentence with the actual structure of the verb from the dictionary of valencies of verbs.

A fragment of the used dictionary of the valency of the verbs for the verbs “to appear” and “to study” is shown in Table 14. Variants in bold letters are suitable for the text of the example.

The filled valence nests for predicates of the example text are shown in Table 15.

Thus, a lot of semantically significant objects and attributes are revealed. For this example, they are summarized in Table 16.

After performing the previous steps on the set of selected elementary units of the first, second, third and fourth levels, using the above-mentioned rules, semantically significant relations between semantically significant objects are found (step 8 in FIG. 1). So, for example, in the sentence "Mechanics is a part of physics that studies the laws of mechanical motion and the causes that cause or change this motion" of the text in question using a variety of rules, the corresponding signal processing scheme is presented in Figure 2 (processing points 1-8) , and the dictionaries used in this rule are presented in Tables 6-15, the semantically significant relation “is” stands out. Other semantically significant relationships are distinguished using the same set of rules. Semantically significant relationships are assigned their type. As a result, semantically significant relationships are distinguished in the source text. Many such semantically significant relationships with the type assigned to them for the example in question are presented in Table 17.

Thus, after all the above processing steps have been completed, the source text will be marked out with a lot of annotations corresponding to semantically significant objects, their attributes and semantically significant relations between semantically significant objects, and also between semantically significant objects and attributes.

After this, at the stage indicated in FIG. 1 by reference numeral 9, the occurrence frequencies of semantically significant objects and attributes throughout this text are detected. This operation is performed in almost the same way as in step 4 for elementary units of the second level. A fragment of such a frequency dictionary for our example is presented in Table 18.

In the next step, indicated at 10 by reference numeral 10, triad formation is performed corresponding to stored semantically significant objects and semantically significant relationships. So for the source text containing the sentence “Mechanics is a part of physics that studies the laws of mechanical motion and the causes that cause or change this motion”, ..., “Mechanical motion is a change over time in the relative position of bodies or their parts”, ... a fragment of the set such triads for our example are presented in Table 19. In fact, the generated set of triads makes up the initial data for constructing a semantic index processed in the previous stages of the text.

In the step indicated by reference numeral 11 in FIG. 1, a semantic index is constructed as follows: first, from the set of triads obtained in the previous step, subsets of triads are formed, each of which corresponds to one semantically significant object with its attributes, and each obtained subset of triads is used as an input for one of the standard indexers, for example, the widely known Lucene freely distributed indexer, Yandex search engine indexer, Google indexer or any other indexer, from whose ode receives an index that is unique to a given subset of triads. A similar sequence of actions is performed for all subsets of triads of the form "semantically significant object - semantically significant relation - semantically significant object" and triads of the form "semantically significant object - semantically significant relation - attribute", receiving a set of corresponding unique indices, which together constitute the semantic index text.

The indices generated in step 10 of the triad and the indices obtained in step 11, together with a link to the source text from which these triads are generated, are stored in the database (step 16 in FIG. 1).

In accordance with the claimed method — before being stored in the database — a semantic network can be formed from the triads so that the first semantically significant object or attribute of the subsequent triad is associated with the same second semantically significant object or attribute of the previous triad (step 12 in FIG. 1) . An example of a fragment of such a semantic network is shown in Table 20.

In this case, before storing the generated triads and obtained indices in the database, the iterative procedure renormalizes the frequencies of occurrence of semantically significant objects into the semantic weight of semantically significant objects that are the vertices of the semantic network, so that the semantically significant object and its attribute are connected in networks with a large number of semantically significant objects and attributes with a high frequency of occurrence increase their semantic weight, while other semantically significant objects and their its attribute is evenly lost (step 13 in FIG. 1). An example of the renormalized into semantic weights of the numerical values of the weighting coefficients of the concepts of the semantic network is given in Table 21.

Next, the generated semantically significant object and its attribute are ranked by semantic weight by comparing their semantic weight with a predetermined threshold value (step 14 in FIG. 1). An example of stored ranked semantically significant objects and attributes is presented in Table 22.

Finally (Table 23), delete the semantically significant object and its attribute, which have a semantic weight below the threshold (the threshold, for example, is 50). This action corresponds to step 15 in figure 1.

It should be noted that the last three procedures (steps 13-15 in FIG. 1) are optional, but preferable, since they reduce the amount of computation and increase user convenience in visualizing the results of the analysis (semantic network).

Thus, the present invention provides a more accurate construction of indices of texts in natural languages by removing from it semantically little significant (in this text) objects and attributes (which have a semantic weight below the threshold). The main difference between this method and the known indexing methods is that the occurrence frequencies of semantically significant objects and attributes are calculated, followed by iterative renormalization to semantic weights. The combination of triads constructed from semantically significant objects and their attributes with the help of semantically significant relations into the semantic network, necessary for iterative renormalization of the frequencies of occurrence into semantic weight, provides associative navigation through documents and collections of documents, as well as high-precision and quick search for facts relevant to the user's information needs and documents, especially when applied to texts in highly inflected languages.

Table 1 Token text segmentation Token Position Start the end one Mechanics one one 8 2 part 2 12 16 3 physicists 3 eighteen 23 ... ... ... ... ...

table 2 Lemmas and morphs Position
nation Lemmas Morphs one Mechanics (mechanics; Named after J.R. Unit of Existence. Inanimate) 2 part (part; Nom. J.R. Unit of Existence. Inanimate.) 3 physics (physics; Rod. p. J.R. Unit of Existence. Inanimate.) (physics; Winn. p. J.R. Mn.Ch. Existence. Inanimate.) ...

Table 3 Frequencies of occurrence of the first and subsequent words of the sequence in the text, as well as differences in the frequencies of occurrence for different words in the sequence Repeating a sequence of words in a text Sequence words Frequency of occurrence Frequency difference one one vector one momentum one 0 2 vector 2 momentum 2 0 3 vector 3 momentum 3 0 2 one vector four speed one 3 2 vector 5 speed 2 3 3 vector 6 speed 3 3 ... ... ...

Table 4 Stable word combinations in the text Collocation one Pulse vector 2 Speed vector 3 Vector quantity ... ...

Table 5 Lots of text suggestions Text suggestions Level 1 Units Level 2 Units Level 3 Units one Mechanics is a part of physics that studies the laws of mechanical motion and the causes that cause or change this movement. Mechanics is a part of physics that studies the laws of mechanical motion and the causes that cause or change this movement. Mechanics is a part of physics that studies the laws of mechanical motion and the causes that cause or change this movement. mechanical movement ...

Table 6 Results of fragmentation of sentences after combining homogeneous series Offer No. Offer Segments Segment type one Mechanics are part of physics, DASH which studies the laws of mechanical motion and causes, Personal verb causing or altering this movement Participle 2 Mechanical movement is a change over time in the relative position of bodies or their parts DASH ...

Table 7 Description attribute level elements Offer Components Morphological features Object / Subject Noun pronoun Act Verb Item Feature Full adjective, ordinal, pronoun-adjective, consistent with the object / subject by gender, number and case Sign of action Adverb Attribute measure Adverb, adverbial numeral

Table 8 Syntactic groups corresponding to the attribute level of description Offer No. Syntax Group Elements Line items Syntax group one feature of the object + object 7 + 8 mechanical movement one feature of the object + object 14 + 15 this movement 2 feature of the object + object 16 + 17 mechanical movement ...

Table 9 Core Conferences Offer Number Antecedent Anaphora one part which is one mechanical of this 2 tel them ...

Table 10 Received simple sentences as a result of enlarging fragments and resolving anaphora Offer No. Simple sentences one Mechanics are part of physics. The part studies the laws of mechanical motion and the causes that cause or change mechanical motion 2 Mechanical movement is a change over time in the relative position of bodies or parts of bodies ...

Table 11 Minimum block diagrams of sentences (fragment) No. p / p Minimum structural diagrams Examples of offers one NlV (f) The Rooks Have Arrived. Things are done by people. 2 N1 Cop (f) Adj1 The night was quiet (quiet, quiet). N1 Cop (f) Adj5 The night is quiet (quiet). N1 Cop (i) Adj (f) The night was quieter than the day. 3 N1 Cop (f) N1 He (was) a student. N1 Cop (f) N5 He was a student. ...

Table 12 The predictive minimum of simple sentences that are part of the source text sentences Offer No. Simple sentences Minimal Structural Pattern Template Predictive minimum (Subject-Predicate) one Mechanics are a part of physics noun in the nominative case + copula ¹⁺ noun in the nominative case Mechanics - be part of The part studies the laws of mechanical motion and the causes that cause or change mechanical motion noun in the nominative case + conjugated form of the verb Part - to study 2 Mechanical movement is a change over time in the relative position of bodies or parts of bodies noun in the nominative case + copula + noun in the nominative case Movement is change ...

Table 13 Syntactic groups derived from source text using syntax rules Offer No. Offer snippets Syntax Groups Name of groups and rules one Mechanics is a part of physics that studies the laws of mechanical motion and the causes that cause or change this movement. part of physics; Genitive ² posture determination patterns of movement Genitive determination in postposition mechanical movement Object + Object Tag reasons causing or changing this movement Object + Object Tag 2 Mechanical movement is a change over time in the relative position of bodies or their parts mechanical movement Object + Object Tag over time Prepositional Group + Genitive Definition in Postposition change of location Genitive determination in postposition location of bodies Genitive determination in postposition arrangement of parts Genitive determination in postposition ... ¹ “Copula” in this case is a connection expressed using a word denoting an action (verb), for example, auxiliary words “to be” and “to have” ² “Genitive definition” means a definition showing ownership, possession, the relationship of a part to the whole, i.e. a definition that answers the question "whose?".

Table 14 The valency dictionary for the verbs "appear", "study" Verb Semanti
class Vale nests Morphological characteristics of actants Minimal Structural Pattern Template 1 * 7 1-N1 Nivf * 7 - N (in, on) 6, Adv to be Sentences reflecting the situation of being 1 ⁰ * 7 1 ⁰ - N3; NO Inf * 7 - N3 (in, on) 6, Adv 1 ⁰ * 7 1 ⁰ - N0 Vpl3 * 7 - N (in, on) 6, Adv to study Suggestions reflecting the situation of knowledge 12 1 - N1 Nivf 1 2 2 - N4, * N (o) 6 2 (x) - N3 1 ⁰ 2 1 ⁰ - N3; N0 Inf 1 ⁰ 2 (s) 2 - N4, * N (o) 6 2 (x) - N3 1 ⁰ 2 1 ⁰ - NO Vpl3 1 ⁰ 2 (s) 2 - N4, * N (o) 6 2 (x) - N3

Explanations for the table:

the digital indices in the “Valenced Nests” column indicate the necessary filling of certain valency-conditioned cells (1 - left-sided actant, or subject of action; 2, 3, 4, 5, 6, 7 - right-sided actants and, accordingly: object, addressee, instrument, initial, final, intermediate locatives);

an asterisk with a digital index in the “Valency” column indicates the optional filling of this valence-conditioned predicate cell;

1 ⁰ - superscript character "0" with a number indicates zero filling of the cell of the subject;

(o), (c) (on) - the alphabetic characters between N and the digital index denoting the case of a noun, indicate the pretext with which the filling of this cell is possible;

N is a noun;

N1 - the digital index at N denotes the case number in the paradigm in order (N1 N2 N3 N4 N5 N6);

Adj is an adjective;

Adv is an adverb;

Vf is the conjugated form of the verb;

Inf - indefinite form of the verb (infinitive);

Vpl3 - the plural form of the third person verb;

2 (sya) - the letter symbol with the number 2 symbol in the “Valency” column indicates that the filling of this valency cell (with the semantics of the object) occurs within the word and does not require additional formal expression.

Table 15 Fill valencies for example text predicates Offer No. Predicate 1. Subject 2. Object 3. Addressee 4. Tool 5-7. Locatives one to be Mechanics part - - physicists one to study part patterns, causes - - - 2 to be traffic change - - -

Table 16 Many semantically significant objects and attributes (fragment) Sentence Semantically significant objects Attributes one Mechanics is a part of physics that studies the laws of mechanical motion and the causes that cause or change this movement. mechanics, part, physics, regularity, mechanical motion, reason, mechanical motion provocative ...

Table 17 Relations between semantically significant objects, and between semantically significant objects and attributes Semantically significant object 1 Semantically significant object 2 Semantically significant relationship Type of semantically significant relationship Mechanics part there is whole part part physicists why why Mechanics regularity to study get information regularity mechanical movement why why Mechanics cause to study get information cause mechanical movement why why

Semantically significant object Attribute Semantically significant relationship Type of semantically significant relationship cause provocative which which cause changing which which

Table 18 The frequency of occurrence of semantically significant objects and attributes Semantically significant object or attribute Frequency of occurrence one Countdown 59 2 free 17 3 modifying 51 four launch four 5 rocket 16 ... one moment of inertia 17 2 plane twenty 3 kinetic energy 13 ... ... rocket 16 ...

Table 19 Many triads (fragment) Triads one mechanics - is - part 2 part - what - physics 3 mechanics - studies - regularity four regularity - of what - mechanical movement 5 mechanical movement - is - change ...

Table 20 Semantic network of triads (fragment) Main word Attitude Subordinate Word one Mechanics there is part 2 part why physics 3 Mechanics is studying regularity four regularity why mechanical movement 5 mechanical movement there is change ...

Table 21 The semantic weight of semantically significant objects and attributes Semantically significant object or attribute Frequency of occurrence one moment of inertia 97 2 plane 21 3 kinetic energy 96 ... ... rocket 12 ...

Table 22 Semantically significant object and its attribute (fragment), ranked by semantic weight Semantically significant object or attribute Frequency of occurrence one moment of inertia 97 2 kinetic energy 96 ... ... plane 21 ... rocket 12 ...

Table 23 Essential semantically significant objects and attributes in the text (fragment) Semantically significant object or attribute Frequency of occurrence one moment of inertia 97 2 kinetic energy 96 ...

Claims (6)

1. The method of automated semantic indexing of text in a natural language, which consists in the fact that:
- submit indexed text in digital form for subsequent automatic and (or) automated processing;
- segment the text in digital form into elementary units of the first level, including at least words;
- segment text in digital form into sentences according to graphematical rules;
- form for each elementary unit of the first level, which is a word, on the basis of morphological analysis, elementary units of the second level, including a normalized word form;
- calculate the frequency of occurrence of each elementary unit of the first level for two or more adjacent units of the first level in this text and combine among the mentioned elementary units of the first level sequences of words following one after another in this text into elementary units of the third level, which are stable combinations of words , if for every two or more consecutive words in a given text the difference is in the calculated frequencies of occurrence of these words for the first occurrence of a given sequence telnosti words and a number of subsequent occurrences for each pair of sequences of words remain unchanged;
- identify, in the process of multi-stage semantic-syntactic analysis by referring to linguistic and heuristic rules pre-generated in the database in a predefined linguistic environment, in each of the generated sentences, a semantically significant object and its attribute are units of the fourth level;
- store in memory each semantically significant object and attribute;
- identify, in the process of multistage semantic-syntactic analysis by referring to linguistic and heuristic rules pre-generated in the database in a predefined linguistic environment, in each of the generated sentences, semantically significant relationships between the identified units of the fourth level - semantically significant objects, as well as between semantically significant objects and attributes;
- assign to each semantically significant relation the corresponding type from the subject ontology stored in the database on the subject of the subject area to which the indexed text belongs; store in memory each semantically significant relation together with the type assigned to it;
- identify the frequency of occurrence of elementary units of the fourth level throughout the text;
- form within the given text for each of the identified semantically significant relations that connect both the corresponding semantically significant objects and the semantically significant object and its attribute, a lot of triads, which are elementary units of the fifth level;
- index on the set of formed triads individually all associated semantically significant relations semantically significant objects with their frequencies of occurrence, all attributes with their frequencies of occurrence and all formed triads;
- save in the database the formed elementary units of the second, third, fourth and fifth levels with their frequencies of occurrence, as well as the resulting indices, together with links to specific sentences of this text. 2. The method according to claim 1, in which for each unit of the fourth level the identity is fixed by reference between the corresponding semantically significant object, as well as the attribute and the corresponding anaphoric link, if any, in the indexed text, replacing each anaphoric link with the corresponding antecedent. 3. The method according to claim 1, in which a semantic network is formed from the said triads in such a way that the first elementary unit of the second or third level of the subsequent triad is associated with the same second elementary unit of the second or third level of the previous triad. 4. The method according to claim 3, in which, after the generated triads and the obtained indices are saved in the database, iteratively renormalizes the frequencies of occurrence into the semantic weight of the elementary units of the second and third levels, which are the vertices of the semantic network, so that the elementary units of the second and third levels connected in a network with a large number of elementary units of the second and third levels with a high frequency of occurrence increase their semantic weight, and other elementary units of the second and on the third level it evenly lose. 5. The method according to claim 1, in which the generated elementary units of the second and third levels are ranked by semantic weight by comparing their semantic weight with a predetermined threshold value. 6. The method according to claim 1, in which triads are deleted in which the elementary units of the second and third levels have a semantic weight below a threshold value.

Images