RU2643438C2 - Detection of linguistic ambiguity in a text - Google Patents

Abstract

FIELD: text processing.

SUBSTANCE: invention relates to a natural language text processing, in particular to defining the meaning of sentences in the text. In the method for detecting linguistic ambiguity a sentence is analyzed in order to determine syntactic links between its generalized components. On the basis of the syntactic links and the lexical-morphological structure of the sentence a graph of generalized components is built, which is analyzed in order to identify a plurality of the sentence syntactic structures. All syntactic structures are assigned with an estimate of the probability that the structure is the true hypothesis of a complete syntactic structure. Semantic structures corresponding to the syntactic structures are built. First and the second semantic structures are selected, each of which has estimates not lower than the threshold value, herewith the first semantic structure is different from the second semantic structure. Basing on the analysis of differences between the two semantic structures the semantic ambiguity in the sentence is identified.

EFFECT: technical result is providing the ability to automatically find and identify ambiguous phrases or statements in a document, which can be interpreted in several possible ways.

20 cl, 28 dwg

Description

BACKGROUND

[0001] Often when compiling documentation, in particular in two or more languages, one has to deal with phrases that are ambiguous in interpretation. As a result, these phrases may be misinterpreted. To avoid ambiguity at the present time, the verification of drafted documents is often completely carried out manually. In addition, there are many articles and manuals that contain rules and recommendations on how to write and draw up documents, including legal documents and legal acts, in order to avoid ambiguity in their interpretation. Misinterpretation of the documentation may have negative consequences. These articles and manuals usually contain a formalized set of rules that a specialist should follow. One of the best ways to check for ambiguity in a document is to independently check the document by several people. However, for a number of reasons, even such a check can be carried out formally, negligently or unprofessionally. In particular, the person checking the document may not have sufficient philological knowledge to identify ambiguous phrases and sentences. The task will prove even more time-consuming for a person who is not a native speaker, since the detection of ambiguity requires a deep knowledge of vocabulary, syntactic and morphological rules, exceptions, etc. At the same time, attracting highly qualified professionals - native speakers with a philological education can be quite a costly event, which may not always be available to a company or an individual.

SUMMARY OF THE INVENTION

[0002] The present description provides systems, computer-readable media, and methods for detecting language ambiguity in a text. The illustrated method includes the analysis of sentences from the first text (using one or more processors) with the aim of: determining syntactic relationships between the generalized components of the sentence; constructing a graph of generalized components of the sentence based on syntactic relationships and the lexical-morphological structure of the sentence; graph analysis to identify the many syntactic structures of the sentence; assignment of the assessment to all syntactic structures, where the assessment reflects the probability that the given syntactic structure is a valid hypothesis describing the complete syntactic structure of the sentence. Further, the method includes determining semantic structures corresponding to syntactic structures, and selecting a first semantic structure from semantic structures and a second semantic structure from semantic structures, where the first and second semantic structures have corresponding syntactic structures, each of which has estimates not lower than a threshold value, and where , the first semantic structure is different from the second semantic structure. Further, the method further includes detecting the semantic ambiguity of the sentence based on an analysis of the differences between the two semantic structures.

[0003] An illustrative system is provided that includes the analysis of sentences from the first text using one or more processors, with the goal of: determining syntactic relationships between the generalized components of a sentence; constructing a graph of generalized components of the sentence based on syntactic relationships and the lexical-morphological structure of the sentence; graph analysis to identify the many syntactic structures of the sentence; assignment of an assessment to all syntactic structures where the evaluation reflects the probability that the syntactic structure is the correct hypothesis describing the complete syntactic structure of the sentence. One or more processors is additionally configured to determine semantic structures corresponding to syntactic structures, and to select a first semantic structure from semantic structures and a second semantic structure from semantic structures, where the first and second semantic structures have corresponding syntactic structures, each of which has ratings not lower than threshold value, and where, the first semantic structure is different from the second semantic structure. Then, one or more processors are further configured to detect semantic ambiguity of a sentence based on an analysis of the differences between the first and second semantic structures.

[0004] Commands for analyzing sentences from the first text, using one or more processors, are stored on the non-volatile machine-readable storage medium described in the description, with the aim of: determining syntactic relationships between the generalized components of the sentence; constructing a graph of generalized components of the sentence based on syntactic relationships and the lexical-morphological structure of the sentence; graph analysis to identify the many syntactic structures of the sentence; assignment of the assessment to all syntactic structures, where the assessment reflects the probability that the syntactic structure is the correct hypothesis describing the complete syntactic structure of the sentence. The following commands are used to determine the semantic structures corresponding to syntactic structures, and to select the first semantic structure from semantic structures and the second semantic structure from semantic structures, where the first and second semantic structures have corresponding syntactic structures, each of which has ratings not lower than a threshold value, and where, the first semantic structure is different from the second semantic structure. Then, commands are used to detect the semantic ambiguity of the sentence based on the analysis of the differences between the first and second semantic structures.

BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The above and other features of the present disclosure will become more apparent from the following description and appended claims, taken in conjunction with the accompanying drawings. The presented illustrations show only a few embodiments in accordance with the disclosure of the invention and, therefore, should not be construed as limiting its scope. The invention will be disclosed with further specification and details by means of the accompanying drawings.

[0006] FIG. 1 illustrates a sequence of steps for performing in-depth analysis in accordance with an embodiment in the description.

[0007] FIG. 2 illustrates a sequence of structures constructed in the process of analyzing a proposal in accordance with an embodiment in the description.

[0008] FIG. 2A illustrates an example of a lexical-morphological structure for the English sentence "The child is smart, he’ll do well in life", in accordance with an embodiment in the description.

[0009] FIG. 3 illustrates a block diagram in accordance with an embodiment in the description.

[0010] FIG. 4 illustrates an example block diagram of semantic descriptions in accordance with an embodiment in the description.

[0011] FIG. 5 illustrates a block diagram of rough parsing in accordance with an embodiment in the description.

[0012] FIG. 6 illustrates an example of a graph of generalized components for the sentence "This child is smart, he’ll do well in life", in accordance with an example embodiment in the description.

[0013] FIG. 7 illustrates a block diagram of accurate parsing in accordance with an embodiment in the description.

[0014] FIG. 8 illustrates a block diagram of a syntax tree in accordance with an embodiment in the description.

[0015] FIG. 9A, 9B, 9B and 9D illustrate examples of syntactic and semantic structures for the sentence “The police shot rioters with guns” in accordance with an embodiment in the description.

[0016] FIG. 10 illustrates a block diagram of a language description in accordance with an embodiment in the description.

[0017] FIG. 11 illustrates a block diagram of a morphological description in accordance with an embodiment in the description.

[0018] FIG. 12 illustrates a block diagram of a semantic description in accordance with an embodiment in the description.

[0019] FIG. 13 illustrates a block diagram of a lexical description, in accordance with an embodiment in the description.

[0020] FIG. 14 illustrates a flowchart of an analysis of a document for the presence of sentences having several interpretations in accordance with an embodiment in the description.

[0021] FIG. 15A, 15B, 15B, and 15G illustrate examples (schemes) of a syntax tree obtained as a result of accurate parsing of the English phrase "The Russian history student" in accordance with an embodiment in the description.

[0022] FIG. 16 illustrates a flowchart for synthesizing translation using a universal semantic in accordance with an embodiment in the description.

[0023] FIG. 17 illustrates a flowchart of a sequence of steps for analyzing parallel documents for the presence of sentences having several interpretations in accordance with an embodiment in the description.

[0024] FIG. 18A and 18B illustrate examples of semantic trees obtained by analyzing a sentence in English “Chickens are ready for dinner” and a sentence in Russian “Chickens are ready to eat” in accordance with an embodiment in the description.

[0025] FIG. 19 illustrates an example of a semantic tree for a sentence in Russian “SOIL SHOULD BE COVERED WITH FERTILIZER BEFORE IT IS FREEZING” in accordance with an embodiment in the description.

[0026] FIG. 20 illustrates a hardware diagram in accordance with an embodiment in the description.

[0027] In the following detailed description, reference is made to the accompanying drawings. The same symbols in the drawings correspond to the same components, unless otherwise indicated. The application examples given in the detailed description, drawings and formulas are not the only possible ones. The invention may be applied or modified by other methods not described below, without violating the field or its essence. The various options described in the description of the invention and illustrated by the drawings can be arranged, replaced and grouped in a wide selection of different configurations, which are described in detail in the present description.

DESCRIPTION OF PREFERRED EMBODIMENTS

[0028] The application of various described implementation methods related to determining the meaning of sentences in the text, including through the use of semantic hierarchy.

[0029] The present invention includes systems, computer-readable media, and methods for detecting ambiguity in sentences in the text. According to the method presented in the present description, the user can obtain the results of an automatic check. For example, the results of this check can be presented in the form of visual and other signals indicating ambiguous sentences for which several semantic structures are constructed. The user has the opportunity to look at the discovered ambiguity and various ways of interpreting a sentence that has ambiguity. In addition, the user can examine in detail the semantic structures built for the sentence containing ambiguity, and manually check the results of the system according to the description provided.

[0030] For example, because of the ambiguity in the text, a situation may arise that the conditions in an already signed legal agreement may be interpreted by the parties to the agreement differently. The natural language analysis system of the present invention can automatically find and highlight ambiguous phrases or statements in a document that can be interpreted in two or more possible ways. Thus, the invention eliminates the risks associated with the human factor that may arise during the manual verification of agreements by lawyers of both parties. Moreover, this system can be used in machine translation (the user may be presented with several translation options for sentences with different interpretations) or to verify the correctness of the results of machine translation. Moreover, this invention can also be used to check parallel texts (text bodies) for the accuracy of their alignment. Such a function is especially useful when checking the manual alignment of parallel texts from external sources, translation databases, etc.

[0031] The present invention is intended to identify and analyze semantic ambiguity in texts (body of texts). The invention is based on the principle of parsing based on the exhaustive linguistic descriptions described in US patent 8078450. Since such an analysis is based on the use of language-independent semantic units, this invention also does not depend on the language and allows you to work with one or more natural languages.

US Pat. This technology can be used to identify the semantic ambiguity of the text. The method uses a wide range of linguistic descriptions and semantic mechanisms, both universal and related to a specific language, which allows you to reflect the diversity of the real language, without resorting to simplifications and artificial restrictions, as well as without the threat of uncontrolled growth of complexity. In addition, these analysis methods are based on the principles of holistic and targeted recognition. This means that hypotheses about the structure of a part of a sentence are verified as part of a test of the hypothesis about the structure of the whole sentence, which avoids the analysis of many anomalies and variations.

[0033] An in-depth analysis includes lexical, morphological, syntactic, and semantic analysis of sentences in a text corpus, as a result of which language-independent semantic structures are constructed in which each word of the text is associated with a corresponding semantic class. FIG. 1 illustrates a general outline of a depth analysis method. The text (105) is subjected to an exhaustive semantic-syntactic analysis (106) using linguistic descriptions of the source language and universal semantic descriptions, which allows us to analyze not only the surface syntactic structure, but also the deep semantic structure expressing the meaning of the statement contained in each sentence, as well as links between sentences or fragments of text. Linguistic descriptions may include lexical (101), morphological (102), syntactic (103), and semantic descriptions (104). Analysis (106) is a syntactic analysis implemented in the form of a two-stage algorithm (rough parsing and accurate parsing) using linguistic models and information of various levels to calculate probabilities and generate a variety of syntactic structures. FIG. 2 illustrates the sequence of structures built in the process of analyzing a proposal. FIG. 2 and FIG. 2A are described in more detail below.

[0034] Rough Parsing

[0035] FIG. 3 illustrates a block diagram of the step indicated by 106 in FIG. 1. At this point, a crude parser (322) is used to identify all potential syntactic relationships in the sentence. As a result, a graph of generalized components (360) is created based on the lexical-morphological analysis (120) of the lexical-morphological structure (350) performed by the analyzer (312) using surface models (410), depth models (512) and a lexical-semantic dictionary (314) ) The graph of generalized components (360) is an acyclic graph whose vertices are generalized (including all possible variants) lexical meanings of the sentence words, and arcs are superficial (syntactic) positions expressing different types of relations between connected lexical meanings. For each element of the lexico-morphological structure of the sentence, which may be the core of the components, all possible surface syntactic models are applied. Further, all possible components are included in the graph of generalized components (232). As a result of the analysis of all possible syntactic descriptions and structures for the initial sentence (302), based on the set of generalized components, a graph of generalized components (360) is constructed. The graph of generalized components (360) reflects all potential connections between the words of the original sentence (302) using the surface model. Since the number of parsing options can be large, the graph of generalized components (360) is redundant, i.e. can contain many options - both for choosing the lexical value (vertex), and for surface positions (graph arcs).

[0036] For each pair of “lexical meaning-grammatical meaning”, its surface model is initialized and other components in the surface positions (415) of the synthform (412) of its surface model (410) are attached to the right and left neighboring components. The syntax descriptions are illustrated in FIG. 4. If a suitable syntactic form is found in the surface model of the corresponding lexical meaning (410), the selected lexical meaning can serve as the core of a new component.

[0037] According to an embodiment, the graph of generalized components (360) is first constructed in the form of a tree (other structures can also be used) or another structure, starting from leaves to roots (bottom to top). The construction of additional components may occur from the bottom up by attaching the child components to the parent components by filling in the surface positions (415) of the parent components in order to cover all the initial lexical units of the original sentence (302).

[0038] According to an embodiment in the description, the root of the tree is the main part representing a special component corresponding to the various types of maximum units of text analysis (completed sentences, numbering, titles, etc.). The core of the main part is usually a predicate (predicate). In fact, the tree turns into a graph, since the components of lower levels (leaves) can be included in various components of the upper level (roots).

[0039] Some of the components built for the same components of the lexical-morphological structure can be subsequently generalized to create generalized components. The components are summarized on the basis of lexical (1312, Fig. 13) and grammatical meanings (414), for example, on the basis of parts of speech and their connections. Since there are many different syntactic links in the sentence, and the same word can be included in several components, the components are generalized using borders (links). As a result of rough syntactic analysis (130), a graph of generalized components (360) is constructed that describes the whole sentence.

[0040] FIG. 5 illustrates a rough parsing process (130) in accordance with one or more implementations of the invention. Rough parsing (130) usually involves pre-collecting components (510), building generalized components (520), filtering (170), building models of generalized components (540), matching processing (550), restoring ellipses (560), etc. .

[0041] According to an embodiment in the description, preliminary collection of components (510) at the stage of rough syntax analysis (130) is performed based on the lexical-morphological structure (350) of the analyzed sentence, including certain groups of words, words in brackets, quotation marks, etc. d. Only one word in a group (core of a component) can join a component or be attached to a component outside the group. A preliminary collection (510) is performed at the beginning of a rough syntactic analysis (130) before the construction of generalized components (520) and models of generalized components (530) in order to cover all the links in the whole sentence. In the process of rough syntax analysis (130), a huge number of components that can be built, as well as syntactic links between them, are processed. Some of the surface models (410) are selected to be sorted during the filtering process (570) before and after the construction of the components in order to significantly reduce the number of components that need to be analyzed. Therefore, at the initial stage of rough syntactic analysis (130), the most suitable surface models and synths are used, selected on the basis of a priori estimates. Coarse a priori estimates include lexical meanings, placeholder ratings, semantic description ratings, etc. Filtration (570) at the stage of rough syntactic analysis (130) is a filtration of a set of syntactic forms (412), which is performed before and during the construction of generalized components ( 520). Syntforms (412) and surface positions (415) are filtered in advance, and the components after they are already built. The filtering process (570) can significantly reduce the number of parsing options considered. It should be noted that the exclusion of unlikely variants of values, surface models and syntforms from the subsequent analysis can lead to the loss of an unlikely, but nonetheless possible meaning.

[0042] When all possible components are built, a generalization procedure is performed to build the generalized components (520). All possible homonyms and meanings for the elements of the original sentence, which can be represented by the same part of speech, are collected and generalized. All possible components constructed in this way are grouped into generalized components (522).

[0043] According to an embodiment, the generalized component (522) describes all components with all possible connections in the original sentence, which has dictionary forms as the main components, and various lexical meanings of this form of the word .. Next, models of the generalized components are constructed (530) and constructs many models (532) of generalized components that have generalized models of all generalized tokens. Models of generalized components of lexemes contain a generalized deep and generalized surface model. The generalized deep-seated token model includes a list of all deep-seated positions that have the same lexical meaning for each lexeme, as well as descriptions of all requirements for depth-placeholders. The generalized surface model contains information on synthforms (412), which can contain a lexeme, surface positions (415), diathesis (417) (connections of surface (415) and deep positions (1214)), as well as a description of the linear order (416).

[0044] Diathesis (417) is constructed at the stage of crude parsing (130). Each diathesis represents a correspondence between generalized surface and generalized deep models. A list of all possible semantic classes for all diathesis lexemes (417) is calculated for each surface position (415).

[0045] FIG. Figure 5 shows how information from the synth (412) syntactic (102) and semantic descriptions (104) is used to build models of generalized components (532). For example, dependent components can be attached to lexical meanings (1312). In this case, crude syntax analysis (130) is necessary in order to establish whether a possible component or dependent component can become a placeholder for the corresponding deep position of the semantic description (104) for the main component. Such a comparative analysis allows you to cut off early syntactic links.

[0046] Next, the construction of the graph of generalized components (540) is performed. The graph of generalized components (360) describes all possible syntactic structures of the sentence by collecting and linking the generalized components (522) with each other.

[0047] FIG. 6 shows an example of a graph of generalized components (600) for the sentence "This child is smart, he’ll do well in life". Components are presented in the form of rectangles. The core of each component is a token. The morphological paradigm of the core component (as a rule, this is part of speech) is expressed by grammes of parts of speech. In the diagram, the paradigm is located under the tokens and is indicated by angle brackets. Morphological paradigms (parts of the word description) may vary. The morphological description contains all the information about the inflection of one or more parts of speech. For example, the word "do" can be one of two parts of speech: a verb (<Verb>) or a noun (<Noun>). Such cases are indicated by the generalized morphological paradigm <Noun & Pronoun>. The graph shows two components for the word "do" (600).

In addition, the word "well" in the graph is also represented by two components. Since the abbreviation “he’ll” was used in the original sentence, the column contains two possible meanings: “will” and “shall”. The task of accurate parsing is to select from all possible components only those that will form the syntactic structure of the original sentence.

[0048] The bonds in column (600) represent the filled surface positions of the nucleus of the component. The position name is shown on the arrow of the graph. The component is formed by the core of the token, which may have outgoing named arrows that indicate surface positions (415) filled with daughter components together with daughter components as such. An incoming arrow indicates the attachment of this component to the surface position of another component. The graph (600) has many arrows (arcs) all the possible connections that can be established between the components of the proposal. Among them, there are connections that will be further rejected. Meaning of the rough methods of estimation mentioned above c Saved for each arrow indicating a filled depth position. As a rule, only superficial positions and relationships with a high rating score will be selected in the first place at the next step of the parsing.

[0049] It is possible that several branches may connect the same pairs of constituents. This means that for this pair of components there are several suitable surface models, and several surface positions of the parent components can be independently filled with these child components. For example, three surface positions - Idiomatic_Adverbial (610), Modifier Adverbial (620) and AdjunctTime (630) of the parent component "do <Verb>" (650) can be independently populated by the child component "well <Verb>" (640) in accordance with surface model of the "do <Verb>" component. Thus, "do <Verb>" (650) + "well <Verb>" form a new component with the core "do <Verb>", which is connected to another parent component - for example, #NormalSentence <Clause> (660) in surface position Verb (670) and with "child <Noun & Pronoun>" (680) in the surface position RelativClause_DirectFinite (690). The labeled element #NormalSentence <Clause> (660) is the "root" that applies to the entire sentence.

[0050] In FIG. Figure 5 shows how matching processing (550) is also performed on the graph of generalized components (360). Matching is a language phenomenon that is presented in sentences with numeral and (or) connective unions, such as [and], [or], [but], etc. A simple example of a coordination proposal is "John, Mary and Bill come home." In this case, only one of the daughter components is attached to the surface position of the parent component at the stage of constructing the graph of generalized components (540). If the component, which may be the parent, has a surface position filled for the agreed component, for all agreed components, an attempt is made to attach the child components to the parent, regardless of whether there is a connection between the agreed components. At the matching processing stage (550), a linear order is determined, as well as the possibility of multiple filling of surface positions. If possible, a preliminary form is created and attached, which refers to the common child component. In FIG. Figure 5 shows how the matching processor (582) or other algorithms can be adapted to perform matching processing (550) using descriptions (454) when constructing a graph of generalized components (540). The dispatcher algorithm (590) can be configured to build models of generalized components (540).

[0051] Building a graph of generalized components (540) may not be possible without restoring the ellipsis (560). Ellipsis is a phenomenon of language, expressed in the absence of the main component. The ellipse recovery process (560) is also necessary to restore missing components. In English, an example of an elliptical sentence would be "The President signed the agreement and the secretary [signed] the protocol". Coordination (550) and restoration of the ellipsis (560) are performed at the stage of each cycle of the dispatcher program (590) upon completion of the construction of the graph of generalized components (540), after which the construction can be continued (this step is indicated by arrow 542). If it is necessary to restore the ellipsis (560) and the results (130) of rough parsing (for example, unrelated components), all other components will not be processed. The ellipsis processing algorithm (580) can be adapted to restore the ellipsis (560).

[0052] Exact parsing

[0053] Exact parsing (140) is performed in order to construct a syntax tree of the original sentence, which is a tree of better syntax structure. The tree describes the best syntactic structure (370) of the original sentence, revealed on the basis of the analysis of the totality of estimates. Many syntax trees can be built. The most probable syntactic structure of the sentence (370) will be accepted as the best tree. Semantic analysis (150) is performed by a semantic analyzer (342) based on the best syntactic structure (370). On its basis, the semantic structure of the initial sentence is generated (380). FIG. 3 shows that the exact parser (332) or its analogues are intended to perform accurate parsing (140) and create the best syntax structure (370) based on the calculation of estimates using a priori estimates (336) from the graph of generalized components (360). A priori estimates (336) include estimates of lexical values (such as frequency or probability), estimates of syntactic constructions (idioms, phrases, etc.) for each element in a sentence, as well as estimates of the degree of consistency of the chosen syntactic construction and semantic description of deep Positions (1220). In addition to a priori estimates, statistical estimates obtained as a result of training the analyzer on large text corps can be used. Integral estimates are calculated and stored by the system.

[0054] In the next step, hypotheses are put forward about the general syntactic structure of the sentence. Each hypothesis is presented in the form of a tree, which in turn is a subgraph of the graph of generalized components (360), covering the entire sentence. After that, the above estimates are calculated for each syntax tree. In the course of performing exact parsing (140), hypotheses about the syntactic structure of sentences are verified by calculating various types of estimates. These estimates are calculated as the degree of consistency of the placeholder of the deep positions component to their grammatical and semantic descriptions, such as grammatical restrictions in synths (for example, grammatical values (414)) and semantic restrictions on filling in the deep positions (1214) in the deep model (1212). The degrees of freedom of lexical meanings (1312) of pragmatic descriptions (344) are also used. They represent absolute and (or) conditional probabilistic estimates of syntactic constructions (surface models (410)) and the degree of compatibility of their lexical meanings with other components.

[0055] Estimates for each type of hypothesis can be calculated based on rough a priori estimates obtained as a result of rough syntactic analysis (130). For example, a rough estimate is calculated for each generalized component in the column of generalized components (360), as a result of which rating estimates can be obtained. For various evaluations, syntax trees can be constructed. The calculated ratings are used to create hypotheses about the complete syntactic structure of the sentence. For this, the hypothesis with the highest rating is selected. The rating is calculated in the course of performing accurate parsing until a satisfactory result is obtained (the best syntax tree with the highest rating is built).

[0056] Next, hypotheses are generated and advanced that reflect the most likely syntactic structure of the whole sentence. Hypotheses derived from syntactic analysis (370) may have a higher or lower rating. The analysis is performed until a satisfactory result is obtained or the best syntax tree with the highest rating is built.

[0057] The best syntax tree is selected as the hypothesis of the syntactic structure with the highest rating, which is reflected in the column (360) of the generalized components. This syntax tree is considered the best (most probable) hypothesis about the syntactic structure of the original sentence (302). Then the proposal builds non-timber ties. After that, the syntax tree is transformed into a graph with the best syntactic structure (370), illustrating the best hypothesis about the syntactic structure of the original sentence. If non-wood links cannot be restored in the best syntactic structure, the analysis is repeated using the structure following in the rating.

[0058] If the exact parsing is unsuccessful or the most probable hypothesis cannot be found after the exact parsing, there is a return (334) from constructing the failed parsing structure at the stage of precise parsing (140) to the stage of conducting rough parsing (130). Moreover, in the process of reanalysis, all syntforms (and not just the best) are considered. If no best syntax tree is found or the system was unable to reconstruct non-wood links in all selected “best structures”, an additional rough syntax analysis is performed (130), which takes into account “bad” synths that have not been analyzed previously.

[0059] FIG. 7 illustrates in detail the precise parsing (140) that is performed to select a plurality of the best syntax structures (370), in accordance with the methods for implementing the invention. Accurate parsing (140) is performed from higher-level structures to lower-level structures (top-down). For example, analysis can be conducted from the top in a possible node of the graph of generalized components (360) to daughter components of a lower level.

[0060] Accurate parsing (140) can include various steps: the initial step, the step of creating a graph of the exact components (750), the step of creating syntax trees and differential selection of the best syntax structure (760), the step of restoring non-wood links (770), obtaining the best syntax structure etc. At the stage of preliminary analysis, the graph of generalized components (360) is analyzed in order to prepare data for accurate parsing (140).

[0061] In the process of accurate parsing (140), accurate components are constructed. Generalized components (522) are used to construct a graph of exact components (730), on the basis of which trees of exact components are then created. For each generalized component, all possible relationships and their child components are indexed and marked.

[0062] Generation of syntax trees (760) is performed in order to obtain the best syntax tree (720). At the stage of restoration of non-wood communications 770, the rules for establishing non-wood communications and information on the syntactic structure of previous sentences (375) are used to analyze syntax trees (720) and select the best syntactic structure (370). Each child generalized component can be included in one or more parent components in one or more fragments. Exact components are nodes of the graph (730), on the basis of which trees of exact components are built. If non-woody links cannot be restored in the best syntax structure (772), the analysis is repeated using the next structure in the rating (step 706).

[0063] The graph of exact components (730) is an intermediate representation between the graph of generalized components (360) and syntax trees. Unlike the syntax tree, the exact component graph (730) may have several alternative placeholders for each surface position. The exact components are arranged in the form of a graph so that each component can be included in several parent components at once in order to optimize further analysis to select the syntax tree. As a result, a compact structure of the intermediate graph is formed, which is convenient enough for calculating the structural rating.

[0064] During the recursive step of creating a graph of exact components (750), the exact components are built on a linear division graph (740) using the left and right connections of the core components. At step (750), various models of generalized components (732) can be analyzed. Exact Component Wizard (790) or other algorithms can be adapted to build a graph of exact components (750). For each component in the linear division graph (740), a path is constructed, many synths are determined, and for each of the synths, the linear order is checked and evaluated. As a result, an exact component is created for each synth, and the construction of the exact child components is launched recursively.

[0065] At the end of step 750, a graph of the exact components is constructed covering the entire sentence. If the stage of creating the graph of exact components 750 failed (730) (i.e., not all of the sentences were covered), a procedure is initiated with an attempt to cover the sentences with syntactically separate fragments (as described in the fragment granularity 710). Linear division graph (715) or another algorithm is responsible for detailing fragments.

[0066] FIG. 7 shows that if a graph of exact components (730), covering the entire sentence, was built successfully, one or more syntax trees can be constructed at the stage of creating 760 during accurate parsing 140. At the stage of generating syntax trees (760), one or several trees with a certain syntactic structure. Since the surface structure is fixed in a given component, corrections can be made in the ratings of the structural rating, including superimposed penalty syntforms, which may be complex or not consistent with the style or rating of the contact linear order, etc.

[0067] The graph of exact components (730) includes several alternatives corresponding to different fragmentations of the proposal and (or) different sets of surface positions. Thus, a graph is a set of possible trees (syntax, etc.), since each position can be occupied by several alternative placeholders. Top-rated aggregates are used as the exact components (wood) with the best rating. Thus, the exact components are the unambiguous (unambiguous) syntax tree with the best rating. At block 760, a search is made for alternatives for constructing trees with a fixed syntactic structure. Non-wood links in the tree created at this stage have not yet been established - they will be created at step 770 by the non-wood link generator (785). The result of this step is to obtain many syntax trees (720) with the highest ratings.

[0068] Syntactic trees are constructed based on a graph of the exact components, in descending order of their structural rating ratings. The lexical ratings at this stage cannot be fully used, since their deep semantic structure has not yet been determined. Unlike the original exact components, each syntax tree at the output has a fixed syntactic structure, and each exact component in it has its own placeholder for each surface position.

[0069] At block 760, the best syntax tree (720) can be generated recursively and transversally, based on the graph of the exact components (730). The best syntax subtrees are built for the best child exact components; the syntactic structure is built on the basis of a given exact component, and the child subtrees are attached to the generated syntactic structure. The best syntax tree (720) can be constructed, for example, by choosing the surface position with the best rating among the remaining surface positions of this component, or by creating a copy of the child component whose subtree has the best quality. This procedure is applied recursively to the child exact component.

[0070] Based on each exact component, a plurality of best syntax trees with a certain rating score can be generated. This rating can be calculated in advance and specified in the exact components. After the best trees are generated, a new component is created based on the previous exact component. This new component, in turn, generates syntax trees with the second highest ranking. Thus, based on the exact component, the best syntax tree can be obtained and built. For example, for each exact component at step 760, two types of ratings can be generated — the quality rating of the best syntax tree, which can be built on the basis of this exact component, and the quality rating of the second best tree. Then the syntax tree rating can be calculated based on the current exact component.

[0071] The syntax tree rating is calculated based on the following values: structural component rating, maximum rating for a set of lexical values; statistics on the maximum depth of subsidiary positions and the rating of subsidiary components. After the exact component is analyzed to calculate the rating of the syntax tree that can be created on its basis, the daughter components with the best ratings are analyzed in a surface position. The calculation of the rating for the second-rated syntax tree differs only in that for one of the child positions its second-highest component is selected. A syntax tree with a minimal difference in rating relative to the best syntax tree should be selected at block 760.

[0072] At block 760, a syntax tree with a fully defined syntax structure may also be constructed. In such a syntactic structure, the syntactic forms and child components that fill the surface positions are fully defined. The syntax tree (720), created on the basis of the best (most probable) hypothesis about the syntactic structure of the original sentence, is considered the best tree. If there are no syntactic trees with a satisfactory rating, or the exact parsing is not successful, the system returns (762) from creating syntax trees (720) to constructing a graph of generalized components (750).

[0073] FIG. 8 schematically illustrates a syntax tree according to an embodiment. In FIG. 8, the components are shown in the form of rectangles (e.g., 810, 820, 822, 824, 830 and 832), and the arrows indicate filled surface positions. The core of the component is a word with its morphological meaning (M-value) and semantic class (Semantic Class). The child components are connected by arrows, in which filled surface positions are indicated. Each component also has a syntactic value (S-value), expressed in the form of grammes of syntactic categories. Grams characterize the syntactic forms chosen for the component during the execution of precise parsing (140).

[0074] FIG. Figure 1 shows the stage of constructing a language-independent semantic structure (107) that conveys the meaning of the original sentence. This stage may also include the restoration of referential relations between offers. One example of a referential connection is anaphora - a language construct that can only be interpreted taking into account another, usually the previous, fragment of the text.

[0075] FIG. 2 illustrates a flowchart of a proposal analysis method according to the described embodiment of the invention. In FIG. 1 and FIG. 2 shows how the lexical-morphological structure (222) is determined at the stage of analysis (106) of the initial sentence (105). FIG. 2A illustrates an example of a lexical-morphological structure for the English sentence "The child is smart, he’ll do well in life". The lexical-morphological structure here is a data structure that stores many possible variants of the lexical / grammatical meaning pairs for each lexical unit (word) in a sentence. Each of the words may have several meanings, and, therefore, several variants of lexical and grammatical pairs. For example, "11" (2207) can mean "shall" (2112) and "will" (2114). Therefore, the diagram shows both of these options. The word "shall" can form the following types of lexical and grammatical pairs: <Verb, GTVerbModel, ZeroType, Present, Nonnegative, Composite_ ||>. The word "will" can form the following types of lexical and grammatical pairs: <Verb, GTVerbModel, ZeroType, Present, Nonnegative, Irregular, Composite_ ||>. Also in FIG. 2A shows six vocabulary variants for the six meanings of the word “smart” (2204). In FIG. 2A also provides alternatives for other vocabulary units in the sentence (2201, 2202, 2203, 2206, 2208, 2209, 2210 and 2211). Punctuation marks - commas, periods, etc. (2205 and 2212) - lexical and grammatical pairs are not assigned. The sentence parsing is implemented in the form of a two-stage algorithm (rough parsing and accurate parsing) using linguistic models and information of various levels to calculate probabilities and many syntactic structures.

[0076] As mentioned above, crude parsing is applied to the original sentence and includes, in particular, the generation of all the potential lexical meanings of the words forming the sentence or phrase, all the potential relationships between them and all potential components. All possible surface syntactic models are applied for each element of the lexical-morphological structure. Then all possible components are built and generalized so that all possible variants of the syntactic analysis of the sentence are presented. As a result, a graph of generalized components (232) is formed for subsequent accurate parsing. The graph of generalized components (232) includes all potential connections in the sentence. Rough parsing is followed by precise parsing on the graph of generalized components, as a result of which many syntax trees (242) describing the structure of the original sentence are "extracted" from it. The construction of the syntax tree (242) includes the lexical choice for the vertices of the graph and the choice of relations between them. A lot of a priori and statistical estimates of lexical options and relationships can be selected from the graph. A priori and statistical estimates can be used both for analysis of individual parts of the graph, and for the whole tree. At this stage, non-timber relationships are also tested and built. Further, from the obtained set of syntactic (or semantic) trees, only trees with a high integral rating and a dissimilar syntactic (or semantic structure) are selected. For example, based on the best syntactic structures, the best / most likely syntactic structures (246) and a universal, language-independent semantic structure (252) can be created. One of the possible ways to assess the similarity / difference of structures is described below.

[0077] In the described embodiment of the invention, the similarity between the i-th and j-th semantic structures can be measured as

[0078] In the described embodiment of the invention, the difference between the i-th and j-th semantic structures can be measured as

[0079]

[0080] The vector of variables (x _1i , ..., x _ni ) is used for the i-th semantic structure, and the vector of variables (x _1j , ..., x _nj ) is used for the j-th semantic structure. The coordinates of the vectors are the values of the parameters used to describe this semantic structure. For example, such variables as SemClass _i , .deepSlots _i , non - treeLinks, α, β, γ, etc., where SemClass are the semantic classes of the semantic structure; deepSlots - deep positions assigned to components in the semantic structure; non - treeLinks - non-tree links restored in the semantic structure; α, β, γ are the values of the weights assigned to the variables in this equation. For the described embodiment, for the sentence, a function can be calculated that determines the proximity measure (or distance) ρ (C _ki , C _lj ) between the semantic class C _k in the i-th and the class C _lj in the j-th semantic structures, where C (Structure _i ) = (C- _1i , ..., C _ki , ..., C _ni ) is a set of semantic classes for the i-th semantic structure, a (C (Structure _j ) = C _1j , ..., C _lj , ..., C _mj . ) - a set of semantic classes for the j-th semantic structure. The number of semantic classes n in the j-th semantic structure may not be equal to the number of semantic classes m in the j-th semantic structure.

[0081] For example, the similarities between Structure _i and Structure _j can be calculated using the following formula:

[0082] where | C (Structure _i ) | is the power of the set Structure _i (that is, the number of classes in the i-th structure), | С (Structure _i ) | is the power of the set Structure _j (i.e., the number of classes in the j-th structure), and the mathematical function acts as g. The above formula can be used to measure similarities between both semantic and syntactic structures.

[0083] In addition, the difference between the structures can be calculated by the formula:

[0084] FIG. 9A and 9B illustrate examples of syntactic structures (trees) 900a and 900b obtained as a result of accurate parsing of the English sentence "THE POLICE SHOT THE RIOTERS WITH GUNS". As a result of the analysis, the two most likely syntactic structures were identified. Structures 900a and 900b contain data about certain syntactic information such as lexical meanings, parts of speech, syntactic roles, grammatical meanings, syntactic relations (positions), syntactic models, types of non-woody links, etc. In view of the fact that the original sentence contains ambiguity (the noun “gun” can depend either on the noun “rioters” or on the verb “shot”), the syntactic structures 900a 900b differ both in structure and in their surface / depth positions. For example, there is a difference in the subordination of the word "gun". The nature of subordination is illustrated in the diagram by arrows 901 and 903. The content of the deep positions 902 and 904 for the lexical meaning "gun: ARTILLERY_AS_WEAPON" also differs. In deep position 903 it is “PartComplement_EntityLike”, and in position 904 it is “Instrument”. In other examples, differences are possible due to the choice of lexical meanings and semantic classes, which manifest themselves especially noticeably in cases of homonymy and homography.

[0085] The two-stage analysis approach is based on the principle of holistic and focused recognition. This means that hypotheses about the structure of a part of a sentence are verified through existing linguistic models within the whole sentence. With this approach, there is no need to analyze a lot of dead-end parsing options. In most cases, this approach can significantly reduce the load on the computing resources needed to analyze the proposal.

[0086] FIG. 9B and 9G illustrate a diagram of the semantic structures 900b and 900g obtained by analyzing the sentence "THE POLICE SHOT THE RIOTERS WITH GUNS". This structure does not depend on the language of the original sentence and contains all the information for establishing the transmitted meaning. Data structures 900v and 900g contain information about semantic classes, semantems (not shown in the figure), semantic relations (deep positions), non-wood relationships and other data sufficient to restore the meaning of the original sentence in any language.

[0087] The language-independent semantic structure of the sentence is an acyclic graph (tree supplemented by non-wood links), where each word of a particular language is replaced by universal (language-independent) semantic entities, also called semantic classes. The organization of semantic classes into a semantic hierarchy makes practical sense for this embodiment of the invention. The semantic hierarchy is organized in the form of a hierarchy of semantic classes, where each "child" semantic class and its "descendants" inherit a significant part of the properties of the parent and all previous semantic classes ("ancestors"). For example, the semantic class SUBSTANCE (substance) is a daughter class of a fairly wide class ENTITY (entity), and at the same time it is the "parent" for the semantic classes GAS (gas), LIQUID (liquid), METAL (metal), WOOD_MATERIAL (wood as material), etc. Each semantic class in the semantic hierarchy is tied to a deep semantic model. An in-depth model is a multitude of in-depth positions (e.g., types of semantic relationships in sentences). The deep positions reflect the semantic roles of child components (structural units of the sentence) in various sentences with objects of a particular semantic class as the core of the parent component, as well as possible semantic classes to fill in the positions. These deep positions express the semantic relationships between components, such as "agent" (agent), "addressee" (destination), "instrument" (instrument), "quantity" (quantity), etc. The child class inherits and adjusts the deep model of the parent class.

[0088] The semantic hierarchy is structured in such a way that more general concepts are at the upper levels of the hierarchy. For example, the following semantic classes can be applied to documents: PRINTED_MATTER (print), SCIENTIFIC_AND_LITERARY_WORK (scientific papers and literature), TEXT_AS_PART_OF_CREATIVE_WORK (creative texts). These classes can be descendants of the TEXT_OBJECTS_AND_DOCUMENTS class (text objects and documents). The PRINTED_MATTER class (print edition), in turn, is the parent class for the semantic class EDITION_AS_TEXT (edition as text), which contains the classes PERIODICAL (periodicals) and NONPERIODICAL (non-periodicals), where PERIODICAL (periodicals) is the parent class for the ISSUE classes (edition) , MAGAZINE (magazine), NEWSPAPER (newspaper), etc. The approach to dividing into classes may vary. Note that the basis of the described invention are concepts that are independent of a particular language.

[0089] FIG. 10 is a diagram illustrating the relationship of language descriptions (1010) according to the described embodiment of the invention. Linguistic descriptions (1010) are lexical (101), morphological (102), syntactic (103), and semantic descriptions (104). All linguistic descriptions (1010) are combined into a common concept. In FIG. 11 is a diagram illustrating in detail morphological descriptions (102). In FIG. 4, syntax descriptions are illustrated in detail (103). In FIG. 13 is a diagram illustrating in detail lexical descriptions (101). In FIG. 12 is a diagram illustrating semantic descriptions in detail (104).

[0090] A semantic hierarchy can be created at a time and then populated for each specific language. The semantic class in each individual language includes lexical meanings with corresponding models. Semantic descriptions (104) are language independent. Semantic descriptions (104) can contain descriptions of the deep components, as well as contain a semantic hierarchy, descriptions of deep positions, a system of semantems and pragmatic descriptions.

[0091] In the described embodiment, morphological (101), lexical (103), syntactic (102) and semantic descriptions (104) are interconnected, as shown in FIG. 10. Several superficial (syntactic) models due to semantems and pragmatic characteristics can have lexical meaning. Syntactic descriptions (102) and semantic descriptions (104) are also related. For example, the diathesis of syntactic descriptions (102) can be considered as an “interface” between language-dependent surface models and language-independent deep models of semantic description (104).

[0092] FIG. 4 gives an example of a syntax description (102). The syntax description (102) can include surface models (410), syntactic forms (412), surface positions (415), linear order descriptions (416), diathesis (417), grammatical values (414), control and matching descriptions ( 440), communicative descriptions (480), rules of analysis (460), rules for calculating semantems (462), rules for normalizing (464), descriptions of surface positions (420), descriptions of non-wood syntax (450), descriptions of ellipsis (452), descriptions of composition (454) and descriptions of referential and structural management (456). Syntactic descriptions (102) are used to construct possible syntactic structures of sentences in the source language, taking into account the word order, non-wood syntax (matching, ellipsis, etc.), referential control, and other phenomena.

[0093] FIG. 11 gives an example of a morphological description (101) according to an embodiment of the invention. The components of morphological descriptions (101) can be descriptions of inflections (1110), grammatical systems (1120) and derivations (1130). In the described embodiment, the grammar system (1120) includes grammatical categories such as “Part of speech”, “Case”, “Rod”, “Number”, “Person”, “Return”, “Time”, “View”, and also their meanings. Similar categories are called grammes.

[0094] FIG. 12 gives an example of a semantic description (104) according to an embodiment of the invention. While surface positions (420) reflect syntactic relations and ways of their implementation in a particular language, deep positions (1214) are the semantic roles of daughter (dependent) components in deep models (1212) of the semantic hierarchy (1210). Consequently, descriptions of surface positions (and surface models in general) will be unique to each specific language. Descriptions of deep models (1220) contain grammatical and semantic restrictions for placeholders of deep positions. Properties and limitations of deep positions (1214), as well as their placeholders in deep models (1212), are often similar and even identical for different languages.

[0095] The semantem system (1230) is a set of semantic categories. Semantems can reflect lexical (1234) and grammatical (1232) categories or attributes, as well as differential properties and stylistic, pragmatic and communicative characteristics. For example, the semantic category "DegreeOfComparison" (degree of comparison) can be used to describe the degrees of comparison expressed by different forms of adjectives, for example, "easy", "easier" and "easiest". Therefore, categories may include semantems such as Positive, ComparativeHigherDegree, and SuperlativeHighestDegree. Lexical semantems (1234) describe the specific properties of objects, such as being flat or being liquid, which act as constraints for placeholders of deep positions. The classification of grammatical (differential) semantems (1236) is used to express differential properties within a single semantic class. Pragmatic descriptions (1240) serve to fix its theme, style or genre in the process of analyzing a text, as well as assign appropriate characteristics to objects of the semantic hierarchy. Pragmatic descriptions (1240) can serve, for example, to describe topics such as economic policy, foreign policy, justice, legislation, trade, finance, etc.

[0096] FIG. 13 provides a diagram illustrating lexical descriptions (103) according to the described implementation of the invention. Lexical descriptions (103) include a lexical-semantic dictionary (1304), consisting of a series of lexical meanings (1312), forming, together with their semantic classes, a semantic hierarchy, where each lexical meaning (1312) is associated with a language-independent semantic category (or the “place” in the semantic hierarchy is 1302), the deep model (1212), the surface model (410), as well as the grammatical (1308) and semantic meaning (1310). The lexical meaning can combine various derivatives (for example, words, expressions, phrases) that express meaning using various parts of speech, word forms, cognate words, etc. In turn, the semantic class combines the lexical meanings of words and expressions that are similar in meaning in different languages .

[0097] In the process of analyzing the text, the system has to solve the problem of detecting ambiguity in the text.

[0098] Under the ambiguity refers to the identification of several different highly probable structures at one stage of the in-depth analysis of the text. Highly probable is understood as a structure with a high integrated rating. In addition, any other criteria for determining a highly probable structure can be used.

[0099] A step-by-step implementation of an in-depth analysis algorithm based on the use of semantic hierarchy (SI) is presented in the present description. The type of ambiguity is determined at the stage of its discovery. For example, syntactic ambiguity may mean the occurrence of several likely syntactic structures at the stage of parsing. Similarly, semantic ambiguity can mean the occurrence of several likely semantic structures at the stage of semantic analysis.

[00100] The structure that has a high integral estimate obtained on the basis of a lot of a priori and statistical estimates is taken as probable. A priori and statistical estimates can be used both for the analysis of individual parts of the graph at different stages, and for the whole tree. In the described implementation of the invention, a plurality of structures (for example, syntax trees) are constructed at the parsing stage. Of the many syntactic structures, the best are selected.

[00101] It may happen that a tree having a lower integral estimate can also correctly reflect the structure of the sentence being analyzed. However, it may be excluded from further analysis as less likely, due to various methods of calculating statistical estimates, it may have a lower integral estimate or other reasons. Such an opportunity is taken into account in the framework of the presented method for identifying ambiguity in natural language texts. Therefore, at different stages of the in-depth analysis of the text, differing structures with high integral estimates are extracted.

[00102] For example, at the stage of constructing syntactic and semantic structures for each corresponding structure in the text, integral estimates are calculated. Identification of several different syntactic (semantic) structures with the highest integral rating will mean the syntactic (semantic) ambiguity of the text. The value of Δ, reflecting the allowable difference between the integral estimates of the structures, can be set in advance, and based on it, individual structures can be distinguished. Conventionally, this rule can be written as inequality:

[00103] S _i is the integral estimate of the i-th semantic structure, S _j is the integral estimate of the j-th semantic structure, Δ is some predetermined (or selected) value within which the estimates of semantic structures may differ to be considered the most probable and be selected from the original set of structures.

[00104] In addition to a numerical value representing the evaluation of the structure (syntactic or semantic), other rules can be introduced according to which it is determined how different the semantic structures must be in order to be considered different. A sentence in which there is ambiguity can be visually highlighted in the text.

[00105] According to the described embodiment, ambiguity can be detected based on an analysis of the contents of the document. Suppose that the document is a legal or other legal act drawn up in natural language. However, the text in the document may not be limited to one single language, the document may include blocks of text in different languages. As a result of an in-depth analysis of all sentences in the text of the document, the system visually displays to the user words, phrases, sentences or paragraphs in which the ambiguity is recorded. Disambiguation will help the user prevent errors that may occur, for example, when translating a document into a foreign language due to an incorrect interpretation of the source text.

[00106] In addition, the present invention can be used in machine translation of a text. A detailed description of the various machine translation methods is given in the following US patent applications: 12/187 131 - "METHOD FOR TRANSLATING DOCUMENTS FROM ONE LANGUAGE USING AN OTHER TRANSLATION DATABASE, TERMINOLOGICAL DICTIONARY ONE IS FIRST TIME AND ONLY ONE WAYS FROM ONE-ONE-TIME AND ON-OFF SYSTEM; PROPOSALS FROM ONE LANGUAGE TO ANOTHER "(12/187 131 A METHOD FOR TRANSLATING DOCUMENTS FROM ONE LANGUAGE INTO ANOTHER USING A DATABASE OF TRANSLATIONS, A TERMINOLOGY DICTIONARY, A TRANSLATION DICTIONARY, AND A MACHINE TRANSLATION SYSTEM FORM / 13; TRANSLATING SENTENCES BETWEEN LANGUAGES).

[00107] Most machine translation systems offer one possible option for translating a sentence or phrase, providing the ability to view or change only translations of individual words. In the described embodiment of the invention, the user is granted access to alternative translations of the whole sentence (if its interpretation is ambiguous). Moreover, alternative translation options may reflect completely different meanings of the sentence and have different semantic and syntactic structures. This feature is especially useful for users who are not native speakers.

[00108] Thus, at the stage of translation synthesis, several translation options are generated based on the constructed semantic structures, each of which is available to the user as an alternative. FIG. 16 shows a flowchart illustrating a translation synthesis algorithm based on constructed semantic structures.

[00109] FIG. 14 illustrates a block diagram according to an embodiment of the invention. The system receives at the input a document or text (1410), in which it is supposed to find sentences or phrases containing ambiguity. The concept of "text" applies to any documents containing signs, including hieroglyphic ones. If the text is presented in an uneditable format, the document can be previously recognized using optical or intelligent recognition technology.

[00110] Then, an in-depth analysis of the text (1420) is carried out, including lexical-morphological, syntactic and semantic analysis. A detailed description of the process of in-depth analysis of the proposal is presented above. A detailed description of the in-depth analysis system is given in US Pat.

[00111] In the process of performing in-depth analysis 1420, a rough and then precise syntactic analysis is performed, graphs of generalized components are formed, on the basis of which syntactic and semantic trees are built. The syntax trees with the highest integral ratings are considered the best, or in other words, the most probable. Some trees obtained as a result of in-depth structural analysis, by their estimation, can only slightly differ from the most probable trees, but, nevertheless, have a different structure, other dependencies, non-wood connections, etc. Such additional structures may display additional (hidden, less frequency) the meaning of the sentence. Therefore, they are taken into account and processed in further analysis.

[00112] For example, syntax ambiguity can be analyzed in the text. Syntactic ambiguity can be detected at the parsing stage 1420. In other words, at the parsing stage, syntax trees with different structures, but high integral estimates, differing by some Δ value can be constructed.

[00113] At the stage of semantic analysis (1430) based on the best syntactic structures, language-independent semantic structures are constructed that convey the meaning of the original sentence. If at the stage of constructing syntactic structures several most probable trees are revealed, then several semantic structures are constructed. The evidence of semantic ambiguity in the text will be considered the presence of several probable semantic structures.

[00114] At block 1440, further analysis of semantic structures is performed. For example, the degree of similarity or difference between semantic structures (in accordance with the formulas described above) can be calculated. The identification of several different structures with high integral ratings in parallel sentences in parallel texts (when structures of similar positions in the text of sentences (1450) differ) serves as an indicator that ambiguity exists in the sentence (1460).

[00115] An example of syntactic ambiguity is provided in the following phrase: "THE RUSSIAN HISTORY STUDENT", which can be interpreted as "STUDENT STUDYING RUSSIAN HISTORY" or "RUSSIAN STUDENT STUDYING HISTORY". FIG. 15A and FIG. 15B illustrate examples of syntax trees 1500a and 15006 reflecting syntactic links in an English phrase. In FIG. 15A, the adjective "Russian" relating to the Lexical object "Russia: RUSSIA" with the deep position "Locative_AttributeGenitive" depends on the noun "history", while the noun "history" itself depends on the noun "student". According to a second interpretation, the syntax tree of which is illustrated in FIG. 15B, the adjective "Russian" and the noun "history" refer to the noun "student". Both syntax trees have high integral ratings, but different structure. This indicates that the original sentence contains syntactic ambiguity. Since the syntactic structures are different, the semantic structures built for each syntax tree separately will also differ. The semantic structures 1500c and 1500g are shown in FIG. 15B and FIG. 15G. According to the results of the synthesis synthesis based on semantic structures (block diagram in Fig. 14), the translation of the phrase into Russian according to the first syntactic structure will sound like “STUDENT LEARNING RUSSIAN HISTORY”, and into German “EIN STUDENT, DER DIE RUSSISCHE GESCHICHTE LERNT. " According to the second syntactic structure, the translation into Russian will sound like “RUSSIAN STUDENT STUDYING HISTORY”, and into German - “EIΝ RUSSISCHER STUDENT, DER DIE GESCHICHTE LERNT”.

[00116] Another example of syntactic ambiguity is the phrase “temporary and part-time employees”. Based on the first, most likely semantic structure, the translation may sound like “temporarily employed employees” and “employees who are temporary or employees who are part-time.” Based on the second semantic structure, the translation sounds like “Employees who are employed temporarily on a part-time basis.” (Employees, each of whom is both temporary and part-time). In this example, the noun “employee”, which may include the adjective (one of the homogeneous members), “temporary ". A similar kind of word-for-word connection combination may be the subject of an elliptical connection.

[00117] FIG. 16 illustrates a block diagram of a translation synthesis. According to the flowchart, translation is based on semantic structures (1600). Next, at step 1610, surface structures are constructed based on syntactic (102), lexical (103), and semantic (104) descriptions. After the formation of semantic structures, a morphological analysis (1620) is carried out on the basis of morphological (101) and lexical descriptions (103). As a result, the user is given the result of the transfer at step 1630.

[00118] Returning to an example sentence containing syntactic ambiguity: "THE POLICE SHOT THE RIOTERS WITH GUNS". FIG. 9A and FIG. 9B illustrate the two syntax trees with the highest integral ratings. In FIG. 9A shows that the noun “gun” of the lexical class (LC) “ARTILLERY_AS_WEAPON” depends on the noun “rioters” of the lexical class “RIOTERS”. In FIG. 9A, this relationship is reflected by arrow 901. FIG. 9B shows a second syntax tree, according to which the noun “gun” of the LC of the class “ARTILLERY_AS_WEAPON” depends on the verb “shoot” of the LC of “TO_KILL_BY_SHOOTING”. In FIG. 9B, this dependence is reflected by arrow 903. Based on syntactic trees, the corresponding semantic structures are constructed that are necessary at the stage of translation synthesis. They will also vary and, accordingly, convey different meanings of the original sentence. In FIG. Figure 16 shows how, at the stage of synthesis of translation into Russian, alternative translation options are formed. The first version of the translation, based on the first semantic structure, will sound like "POLICE SHOT TERRORISTS WITH WEAPONS". According to the second semantic structure, the translation will be different: "POLICE SHOOT TERRORISTS FROM WEAPONS". This example shows that semantic structures are not the same for different syntactic structures, therefore, the translation options that are synthesized based on them will also differ. Based on the analysis, it can be concluded that the original sentence - "THE POLICE SHOT THE RIOTERS WITH GUNS" - contains an ambiguity that can be identified using the described invention.

[00119] There are many examples of syntactic ambiguity. Another example is the following sentence: "THE YOUNG MEN AND WOMEN LEFT THE ROOM." This phrase can be interpreted as "THE (YOUNG MEN) AND WOMEN" or "THE YOUNG (MEN AND WOMEN)". Depending on which noun the adjective "young" refers to, semantic structures will differ, therefore, there will be different variants of translation into the target language. This phrase can also be the subject of non-referential connections, in particular, ellipsis. Phrases containing unions "or" or "and", and other ambiguities of this kind, are especially common in legal acts and legal documents. Identification and resolution of ambiguity in such documents is especially relevant.

[00120] Here are examples of the ambiguity that can be identified using the described method of implementing the invention. "NOT GAVE HER CAT FOOD." Based on the semantic structures of this example, the following versions of the translation into Russian can be formed: "HE GIVED HER CAT FOOD" and "HE GIVED HER FAT CAT". The sentence “NOT SAW JANE COMING TO THE BANK” has four variants of semantic structures with an equally high integral rating: “HE SAW JANE GOING TO THE BANK”; "HE SAW JANE GOING TO THE COAST"; "GOING TO THE BANK, HE SAW JANE"; "GOING TO THE COAST, HE SAW JANE."

[00121] In alternative implementations of the present invention, parallel texts in various languages, including translations of the same source document, may be input. For example, such documents may be legal acts, agreements and licenses in various languages. A situation may arise when the sentence or phrase in the text will be interpreted differently, as a result of which the original document and its translations into foreign languages will have different meanings.

[00122] In another embodiment, the system may receive parallel text corpuses and translation memories (from external sources) at the input. In this regard, before using the cases for analysis, for example, for training, it is necessary to check the quality of the case, namely, first of all, it is necessary to check the correctness of alignment, i.e. check how the sentences of one language correspond in meaning to the corresponding ones according to the alignment of the sentences of another language.

[00123] FIG. 17 shows a block diagram of an alternative method of application in which parallel text bodies, or parallel documents (1710), i.e. translations of one source document into various languages that need to be analyzed in order to identify ambiguity. In this case, it can be already aligned parallel texts or texts that have not been aligned.

[00124] At block 1720, correspondence between sentences and paragraphs in parallel texts is established. For this purpose, one of the existing alignment algorithms can be used. For example, one possible way to align text is described in US Patent Application 12/708 337, “METHODS AND SYSTEMS FOR ALIGNMENT OF PARALLEL TEXT CORPORA”.

[00125] After alignment, each of the parallel documents is processed by the system independently (1730). For each of the parallel documents, an in-depth analysis is carried out, including lexical-morphological, syntactic and semantic analysis of all sentences in the document, according to the algorithm described above.

[00126] Syntactic structures are constructed at the output, on the basis of which further language-independent semantic structures (1740) are formed for each sentence in all parallel texts (including texts in different languages). In addition, at the stage of semantic analysis, reference relationships are established. One example of a referential relationship may be anaphora. Anaphora resolution is one of the most laborious tasks of machine translation. However, even with manual translation of the text, some anaphoric connections may not be traced correctly, as a result of which the meaning of the sentence may be distorted. Thanks to the described method, it is possible to trace the anaphoric connections both in the original and in the translated text. The difference between the links in parallel texts will indicate that the sentence contains ambiguity, which was interpreted incorrectly during the translation.

[00127] Thus, after conducting lexical-morphological, syntactic and semantic analysis, for each sentence in each of the texts in different languages, language-independent semantic structures are constructed that reflect the original meaning. At step 1750 (Fig. 17), the semantic structures are compared for the corresponding sentences according to the results of alignment: a relationship is established between the semantic structure of the sentence in the text in language A with the corresponding sentence in the text in language B, etc.

[00128] The difference in semantic structures (1760) will mean the ambiguity of the original sentence. The presence of ambiguity can be visualized (1770) using one of the known methods. It is recommended to use the method of constructing the missing additional semantic structures identified in the analysis process. Also, based on the obtained semantic structures, a translation can be made.

[00129] In FIG. 18A and 18B illustrate ambiguity in interpretation by the example of a sentence in English: "Chickens are ready for dinner". Suppose that in the parallel document the following translation of this proposal into Russian was formed - "Hens are ready to eat." According to the described method for identifying ambiguity, this sentence will be carefully analyzed: an in-depth analysis is carried out, syntactic and semantic structures for each sentence are constructed. Thus, the resulting semantic structure (1800a) of the English sentence "Chickens are ready for dinner", presented in FIG. 18A. In this semantic structure, the noun "chickens" refers to the lexical class "CHICKEN_MEAT" with the deep position "Experiencer_Metaphoric". The semantic structure (1800b) of the Russian sentence “Chickens are ready to have lunch” is shown in FIG. 18B. In this semantic structure, the noun “chickens” refers to the lexical class “HEN_ROOSTER” with the deep position “Experiencer”. As a result of a parallel analysis of the texts, different semantic structures were revealed. Various semantic structures of parallel sentences indicate that there is ambiguity in interpretation in the original sentence.

[00130] Consider another example of ambiguity in the sentence "THE SOIL SHALL BE COVERED BY FERTILIZER BEFORE IT FREEZES". Suppose there are three sentences in three different languages. The first is the original sentence in English, containing ambiguity. Two other sentences are translations into Russian and German. If the ambiguity in the original English sentence was not recorded in the translation process (by a person or by a machine translation system), the resulting sentences will be different in meaning. The ambiguity in the original English sentence is that the pronoun "it" can refer to both the noun "soil" and the noun "fertilizer". For this reason, translations into the target languages, Russian or German, will differ in meaning, depending on which word the pronoun "it" refers to. Sentences of this kind can be understood in different ways by different translators, as a result of which the translations will be different. In such cases, they will be visualized as sentences containing ambiguity. For example, if during the translation the pronoun "it" was attributed to the noun "soil", the translation into Russian (according to the algorithm shown in Fig. 16) would be as follows: "FERTILIZER MUST BE FILLED IN SOIL BEFORE IT WILL BE FREEZED." But the pronoun "it" can also be attributed to the noun "fertilizer". In this case, the translation into German will result in: "DER BODEN SOLLTE MIT DUNGEMITTEL ABGEDECKT WERDEN, BEVOR ES FRIERT". According to the described invention, for each sentence in the Russian and German versions of the translation of the document, semantic structures are constructed that display various dependencies presented in the form of non-wood connections. An example of a semantic structure for the Russian translation variant (1900) is shown in Fig. 19. The structure reflects the dependence of the pronoun "she" (1901) on the noun "soil" of the lexical category "EARTH_AS_SUBSTANCE" (1902) in the form of a non-wood connection (1903). At the same time, the semantic structure of the German translation reflects the dependence of the pronoun "Es" on the noun "Düngemittel". Therefore, the above example demonstrates that the original sentence in English contains ambiguity, presented in the form of a referential connection - anaphora. This type of ambiguity is generated by asymmetry in different languages. For example, in English for the pronoun "it", which means inanimate objects, in Russian there are pronouns "he", "she", "it". The sentence is difficult to correctly translate out of context and with the ambiguous dependence of this pronoun on the antecedent. The described system helps to detect such ambiguities in the text and indicate them to the user. The ambiguity caused by incorrect punctuation in a sentence can be identified at the stage of parsing. Lexical ambiguity, which consists in the existence of a word with several lexical meanings, can also be revealed at the stage of conducting lexical analysis of the text.

[00131] In FIG. 20 is a hardware diagram (2000) that can be used to implement the present invention. The hardware (2000) must include at least one processor (2002) connected to the memory (2004). The word "processor" in the diagram (2002) can refer to one or more processors with one or more computing cores, a computing device, or any other CPU available on the market. The numbers 2004 indicate the device of random access memory (RAM), which is the main storage (2000), as well as additional levels of memory - cache, non-volatile, backup memory (for example, programmable or flash memory), ROM, etc. In addition, the designation of memory (2004) may also mean storage located in another part of the system (for example, the processor cache (2002) or other storage used as virtual memory, such as internal or external ROM (2010).

[00132] Hardware (2000) typically has a number of inputs and outputs for transmitting and receiving information from outside. As the user or operator interface of the hardware (2000), one or more user input devices (2006), such as a keyboard, mouse, imager, etc., as well as one or more output devices (liquid crystal display or other (2008), can be used ) and sound reproducing devices (speaker).

[00133] To obtain additional storage capacity, data storage devices (2010) are used, such as floppy disks or other removable disks, hard disks, direct access memory (DASD), optical drives (CDs, etc.), DVD drives, magnetic tape drives, etc. Hardware (2000) may also include a network connection interface (2012) - LAN, WAN, Wi-Fi, the Internet, etc. - to communicate with other computers on the network. In particular, you can use a local area network (LAN) or a wireless Wi-Fi network that is not connected to the world wide Internet. It should be noted that the hardware (2000) also includes various analog and digital interfaces for connecting the processor (2002) and other system components (2004, 2006, 2008, 2010 and 2012).

[00134] The hardware (2000) is running an Operating System (OS) (2014) that runs various applications, components, programs, objects, modules, etc., for the purpose of implementing the process described herein. The composition of the application software should include an application to identify the semantic ambiguity of the language.

A client dictionary, an application for automated translation and other installed applications for displaying test and graphic content (word processor, etc.) can also be included. In addition to this application, components, programs and other objects collectively indicated by the number 2016 in FIG. 20 may also be run on processors of other computers connected to hardware (2000) over a network (2012). In particular, the tasks and functions of a computer program can be distributed between computers in a distributed computing environment.

[00135] All routine operations for implementing implementations may be performed by the operating system or by individual applications, components, programs, objects, modules, or sequential instructions, collectively referred to as "computer programs". Typically, computer programs are a series of instructions executed at different times by different memory and data storage devices on a computer. After reading and following the instructions, the processors perform the operations necessary to run the elements of the described implementation. Several embodiments have been described in the context of fully functioning computers and computer systems. Industry experts will appreciate the ability to distribute certain modifications in the form of various software products on any type of information media. Examples of such media are both volatile and non-volatile memory devices such as floppy disks and other removable disks, hard disks, optical disks (e.g. CD-ROMs, DVDs, flash drives) and much more. Also, the software package can be downloaded over the Internet.

[00136] In the foregoing description, many specific details are set forth for illustrative purposes only. Those skilled in the art will appreciate that these specific details are merely examples. In other cases, structures and devices are shown only in block diagram form in order to avoid ambiguity in interpretation.

[00137] The references in this description to “one embodiment / implementation” or “embodiment / implementation” mean that a particular feature, structure, or characteristic described for an embodiment is a component of at least one embodiment. The use of the phrase “in one embodiment” in different fragments of the description does not mean that the descriptions refer to the same implementation variant or that these descriptions refer to different or alternative mutually exclusive variants of implementation. In addition, various descriptions of characteristics may relate to some implementation options, but not to relate to other implementation options. Various descriptions of requirements may relate to some implementation options and may not relate to other implementation options.

[00138] Some examples of embodiments have been described and shown in the accompanying illustrations. However, it must be understood that such implementations are merely examples, but not limitations of the described implementations, and that these implementations are not limited to the specific structures and devices shown and described, as those skilled in the art can create their own implementations based on the above materials. In the field of technology to which the present invention relates, rapid development and further advances are difficult to predict, therefore the described embodiments can be easily changed in the device and details due to the development of technology, while respecting the principles of the present described invention.

Claims (60)

1. A method for detecting language ambiguity, including: analysis, using one or more processors, of the sentence of the first text in order to determine the syntactic relationships between the generalized components of the sentence; building a graph of generalized components of the sentence based on syntactic relationships and the lexical-morphological structure of the sentence; graph analysis to identify many syntactic structures of sentences; assignment of estimates to each of the many syntactic structures describing the likelihood that the given syntactic structure is a valid hypothesis about the complete syntactic structure of the sentence; the construction of semantic structures corresponding to syntactic structures; identification of the first and second semantic structures from the set of semantic structures, where the first and second semantic structures are corresponding structures, each of which has ratings not lower than a threshold value, and where the first semantic structure is different from the second semantic structure; as well as establishing semantic ambiguity in a sentence based on an analysis of the differences between the two semantic structures. 2. The method of claim 1, wherein the semantic structures represent the meaning of the original sentence and are language independent. 3. The method according to p. 1, further comprising determining the differences between the first and second semantic structures by matching the first vector of values corresponding to the first semantic structure and the second vector of values corresponding to the second semantic structure, where each vector of values includes at least : one of the semantic classes of the corresponding semantic structure; the number of deep positions assigned by the generalized components in the semantic structure; the number of non-wood relationships restored in the corresponding semantic structure, and weight value. 4. The method according to p. 1, further comprising: alignment of the sentence of the first text and the sentence of the second text in order to establish the relationship between the sentences of the first and second text, where the second text is a translation of the first text into a foreign language; the construction of semantic structures corresponding to the sentence of the second text; comparing the semantic structures corresponding to the sentence from the first text and the semantic structures of the second text in order to determine the difference between the semantic structures of the first text and the semantic structures of the second text, and identifying the ambiguity in a sentence from the first text by determining the difference between the semantic structure corresponding to the sentence of the first text and the semantic structure of the sentence from the second text. 5. The method according to claim 1, further comprising visualizing the sentence using the user interface, where the visualization of the sentence arises based on the detection of semantic ambiguity of the text in the sentence. 6. The method according to claim 5, in which the visualization of the sentence includes the selection of at least one word, sentence, phrase or paragraph in which the ambiguity is detected. 7. The method according to p. 1, further comprising: synthesis of the translation of a sentence and its alternative translation taking into account semantic ambiguity; as well as output translation of the proposal and its alternative translation using the user interface. 8. The method according to claim 1, in which the graph is an acyclic graph, the vertices of which are summarized in such a way that each of them contains alternative lexical meanings of words in a sentence, and the arcs depict the relationship between lexical meanings. 9. The method according to p. 1, in which the graph represents the proposal in full. 10. A language ambiguity detection system, including: one or more processors configured to: analysis of the sentence of the first text in order to determine the syntactic relationships between the generalized components of the sentence; building a graph of the components of the sentence based on syntactic relationships and the lexical and morphological structure of the sentence; graph analysis to identify many syntactic structures of sentences; assignment of estimates to each of the many syntactic structures describing the likelihood that the given syntactic structure is a valid hypothesis about the complete syntactic structure of the sentence; the construction of semantic structures corresponding to syntactic structures; identification of the first and second semantic structures from the set of semantic structures, where the first and second semantic structures are corresponding structures, each of which has an estimate not lower than a threshold value, and where the first semantic structure is different from the second semantic structure; as well as establishment of semantic ambiguity of a sentence based on the analysis of differences between two semantic structures. 11. The system of claim 10, wherein the semantic structures represent the meaning of the original sentence and are language independent. 12. The system of claim 10, wherein one or more processors are configured to determine differences between the first and second semantic structures by matching the first vector of values corresponding to the first semantic structure and the second vector of values corresponding to the second semantic structure, where each vector of values includes in itself, at least: one of the semantic classes of the corresponding semantic structure; the number of deep positions assigned by the generalized components in the semantic structure; the number of non-wood links restored in the corresponding semantic structure, as well as the weight value. 13. The system of claim 10, wherein one or more processors are configured to: alignment of the sentence of the first text and the sentence of the second text in order to establish the relationship between the sentences of the first and second text, where the second text is a translation of the first text into a foreign language; the construction of semantic structures corresponding to the sentence of the second text; comparing the semantic structures corresponding to the sentence from the first text and the semantic structures of the second text in order to determine the difference between the semantic structures of the first text and the semantic structures of the second text; as well as identifying the ambiguity in a sentence from the first text by determining the difference between the semantic structure corresponding to the sentence of the first text and the semantic structure of the sentence from the second text. 14. The system of claim 10, wherein one or more processors are configured to render the sentence using a user interface, where the sentence visualization arises based on the detection of semantic ambiguity of the text in the sentence. 15. The system of claim 10, wherein one or more processors are configured to: synthesis of the translation of a sentence and its alternative translation taking into account semantic ambiguity; as well as output translation of the proposal and its alternative translation using the user interface. 16. The system of claim 10, wherein the graph is an acyclic graph, the vertices of which are summarized in such a way that each of them contains alternative lexical meanings of words in a sentence, and arcs depict relations between lexical meanings. 17. Non-volatile computer-readable storage medium containing the following commands: teams to analyze the sentence of the first text in order to determine the syntactic relationships between the generalized components of the sentence; teams for constructing a graph of generalized components of a sentence based on syntactic relationships and the lexical-morphological structure of the sentence; graph analysis teams to identify multiple syntactic sentence structures; instructions for assigning an assessment to a plurality of syntactic structures describing the probability that the syntactic structure confirms the hypothesis of the complete syntactic structure of the sentence; commands for constructing semantic structures corresponding to syntactic structures; teams to identify the first and second semantic structures from the set of semantic structures, where the first and second semantic structures are corresponding structures, each of which has ratings not lower than a threshold value, and where the first semantic structure is different from the second semantic structure; as well as Teams for establishing semantic ambiguity of sentences based on the analysis of differences between the two semantic structures. 18. Non-volatile machine-readable storage medium according to claim 17, containing: teams to determine the differences between the first and second semantic structures by comparing the first vector of values corresponding to the first semantic structure and the second vector of values corresponding to the second semantic structure, where each vector of values includes at least: one of the semantic classes of the corresponding semantic structure ; the number of deep positions assigned by the generalized components in the semantic structure; the number of non-wood links restored in the corresponding semantic structure, as well as the weight value. 19. A non-volatile computer-readable storage medium according to claim 18, comprising: teams to align the sentences of the first text and the sentences of the second text in order to establish the relationship between the sentences of the first and second text, where the second text is a translation of the first into a foreign language; teams for the construction of semantic structures corresponding to the sentence of the second text; commands for comparing the semantic structures corresponding to the sentence from the first text and the semantic structures of the second text in order to determine the difference between the semantic structures of the first text and the semantic structures of the second text; as well as teams to identify the ambiguity in the sentence from the first text by determining the difference between the semantic structure corresponding to the sentence of the first text and the semantic structure of the sentence from the second text. 20. A non-volatile computer-readable storage medium according to claim 18, comprising: teams for the synthesis of the translation of the sentence and its alternative translation, taking into account semantic ambiguity; as well as commands for the withdrawal of the translation of the proposal and its alternative translation using the user interface.

Images