RU2769427C1 - Method for automated analysis of text and selection of relevant recommendations to improve readability thereof - Google Patents

Abstract

FIELD: physics.

SUBSTANCE: invention relates to a method for automated analysis of text and selection of relevant recommendations to improve its readability. Method comprises obtaining source text; detecting stop words in a source text in accordance with given rules for determining stop words through morphological, lexical, semantic and syntactic analysis of the source text based on application of fuzzy search, search by initial forms of words, search by grammatical features and search by punctuation; evaluating the readability of the source text based on calculating the ratio of the number of all words in the source text and the number of stop words detected in the source text; displaying relevant recommendations for improving the readability of the text, including at least an indication of the rule for determining stop words, according to which words in the source text were determined as stop words.

EFFECT: high accuracy of generating recommendations to improve readability of text.

5 cl, 7 tbl

Description

The technical field to which the invention belongs

The invention relates to the field of digital data processing using electrical devices, in particular to natural language data processing, namely, natural language analysis, semantic, morphological and lexical.

Glossary

In order to ensure the sufficiency of the disclosure of the invention and to enable information retrieval in relation to the claimed technical solution, below is a list of terms used in the description of the claimed invention.

A precise search is a method of information retrieval in which an exact comparison of information with a given search pattern is performed.

Fuzzy search is a search for information in which information is compared to a given search pattern or a value close to it.

Linguistic token is an object that describes an element of a natural language sentence in terms of the presence of morphological and other linguistic characteristics, including, but not limited to, the characteristic of a part of speech, case, gender, number, person, tense and other characteristics, and also contains the initial form of the word.

Regular expressions are a formal language for searching and manipulating substrings in text based on the use of metacharacters. For the search, a pattern string is used, consisting of symbols and metacharacters and specifying the search rule.

Superregular expressions (linguistic regular expressions) are a formal language for describing chains of linguistic tokens, as well as a mechanism for searching for these chains based on the use of a language for describing the characteristics of individual linguistic tokens, as well as a language for specifying chains of linguistic tokens.

API (API) - an application programming interface (a set of classes, procedures, functions, structures or constants) with which one computer system can interact with another system, as well as a way to use these elements using a protocol.

Affixes are morphemes that are attached to the root of a word and serve to form words.

Word forms are a chain of phonemes with the features of a word, formed from one lexeme.

Stop words are fragments of text that, with a high degree of probability, do not carry meaning for the reader regarding the purpose of a particular text, or are subject to reformulation or addition through analysis of the source text.

Stop words, as a rule, include words and phrases from the following conditional categories: advertising, newspaper and household stamps, clichés, set expressions, stationery, imprecise wording, biased assessments, qualitative adjectives, phrases with verbal nouns, excessive generalizations, pleonasms, vague wording, redundant time indications, misused borrowings, obscene expressions and euphemisms.

Stop words also include a number of unjustified syntactic turns that make it difficult to understand the text, in particular repetitions, passive voice, introductory constructions, modality and other turns, the elimination of which will not affect the meaning of the sentence, but will make the text easier to read. Stop words include any words or phrases that can be removed from the text without losing their meaning.

Stop words also include generalizing and evaluative concepts that need to be disclosed in more detail in order for the reader to understand the author's thought.

Editor - a person who writes, checks and corrects (edits) the text. In the process of editing the text, the editor works through stop words: removes them, reveals concepts hidden by stop words, or deliberately leaves stop words in those places where their use is justified, and deletion will lead to a loss of meaning. After such processing, the text becomes clearer, shorter, more informative and easier to read - the readability of the text increases. Thanks to this, the reader spends less time reading the text and quickly learns the meaning of what is written.

Readability of a text is a measure of the readability of a written text, determined by the analysis of a number of factors, including syntactic complexity, vocabulary, expressiveness of a topic, coherence of topics, etc. ( https://psychology_dictionary.academic.ru/9253 ). In the English-language literature, the term “readability” is used for this concept. To improve the readability of text, editors use editing techniques to help clean up the text from stop words and fill it with useful information.

A formant is a morpheme that attaches to a root and serves to form words.

Inflection is a complex of grammatical categories expressed in inflection, a set of morphemes that carry out inflection.

Inflection - changing words according to their grammatical forms.

State of the art

In 1920, a book by Professor William Strunk Jr. was published in the USA. "Elements of style". Strunk gave rules on how to write clearly and avoid common mistakes:

“In order for the text to be energetic, it must be capacious. There should be no unnecessary words in a sentence, no unnecessary sentences in a paragraph, just as there should be no unnecessary strokes in a picture, and unnecessary parts in a mechanism. This does not mean that the author should keep each sentence as short as possible, avoid details, or describe his subject only in general terms. This means that every word must carry a meaning.

In 1972, the book of the editor and translator Nora Gal "The Word Living and Dead" was published in the USSR. Gal also advises to avoid clericalism, use verbs instead of verbal nouns, change the official tone to a simple one, use Russian words instead of borrowed ones, write to the point and quickly come to the point.

So, over the past century, editing techniques have been gradually formed that helped to clear the text of unnecessary words and fill the text with useful information.

With the development of computer technology, systems and methods of automated text analysis and processing began to appear, designed to increase the readability of the processed text.

The claimed technical solution produces a semantic, lexical, morphological and syntactic analysis of the text. At the same time, semantic analysis should be understood as analysis in order to determine the meaning of a certain part of the text, lexical analysis - analysis of lexemes in order to search for words, morphological analysis - analysis of parts of speech, syntactic analysis - analysis of punctuation marks.

During the patent search, documents were found that define the state of the art and are not considered particularly relevant in relation to the claimed invention, namely:

“Method for automated analysis of text documents” (patent for invention No. 474870 RU, Application: 2011146888/08, 11/18/2011, Patent holder: Natalia Kaspersky Innovation Center Limited Liability Company (RU));

"Automatic extraction of named entities from the text" (patent for invention No. 2665239 RU, Application: 2014101126 dated 01/15/2014, Patent holder: Abi Production Limited Liability Company (RU));

"Extraction of entities from texts in natural language" (patent for invention No. 2626555 RU, Application: 2015151699, 02.12.2015, Patentee: Abi Production Limited Liability Company (RU));

“Sentiment analysis at the level of aspects using machine learning methods” (patent for invention No. RU 2 657 173, Application: 2016131180, 07/28/2016, Patent holder: Abi Production Limited Liability Company (RU));

"Method of Sentiment Analysis of Text Data" (patent for invention No. 2571373 RU, Application: 2014112242/08, 03/31/2014, Patent holder: Abi InfoPoisk Limited Liability Company (RU));

“Suggestion of relevant terms during text input” (patent for invention No. 2589727 RU, Convention priority: 01.11.2010 US 61/408,699, Patent holder: Koninkleike Philips Electronics n.v. (NL));

"Readability evaluation method, readability evaluation device and readability evaluation program" , Publication date 11/22/2012, Patent grant number 5733617, Patent issue date 04/24/2015);

"Text normalization method, device and equipment and storage medium" (CN110765733, IFLYTEK CO., LTD., Office of China, Application No. 201911017291.4, Filing Date 10/24/2019, Publication Number 110765733, Publication date 02/07/2020);

"System and method for enhancing comprehension and readability of text" (GB2514725, QUILLSOFT LTD, United Kingdom Office, Application Number 201416621, Filing Date 02/22/2013, Publication Number 2514725, Publication Date 05.11 .2014).

Also, technical solutions containing individual features or their equivalents used in the claimed invention were discovered and investigated.

“A method for detecting insignificant lexical units in a text message and a computer” (patent for invention No. 2580424RU, Application: 2014147903/08, November 28, 2014, Patentee: Yandex Limited Liability Company) This method refers to systems for processing an incoming email message intended for the user mail. The technical result consists in providing the possibility of identifying insignificant lexical units in the text of an e-mail message. This result is achieved by parsing the e-mail message to determine the lexical item as a candidate for non-meaningful lexical items; carry out the first and second checks of the candidate for insignificant lexical units by comparing with insignificant lexical units from the first and from the second database of lexical units, where the first database is formed as a result of parsing previous e-mail messages intended for the user, and the second database is formed as a result of parsing previous e-mail messages intended for a group of users from a plurality of users. In response to a positive result of any of the first test and the second test, the candidate for the non-significant lexical items is determined as the non-significant lexical item.

The above method differs from the claimed invention and has a different purpose, but in general, this method to some extent allows you to identify insignificant lexical units in the text, like the claimed invention. Unlike the claimed technical solution, this method uses only semantic text analysis and does not use lexical and morphological analysis. Also, this method is intended to achieve a different technical result.

"Method for displaying degree of difficulty of readability of text in word processor, involves comparing numerical readability subscripts in overview of text, so that readability assessments of passages are compared with each other" readability indexes in a text review so that readability scores of passages are compared with each other), (DE102010027146, Ramps Ullrich, German Office, Application number 102010027146, Filing date 07/09/2010, Publication number 102010027146, Publication date 01/12/2012) includes a determination of the degree of difficulty reading text in a word processor by an application based on quantitative linguistic units, such as letters, syllables, and words in text in narrow passages. Indexes of numerical readability are simultaneously and separately presented for individual passages of the analyzed text as similar closed language units. Indexes are presented in the text review so that the readability scores of individual passages are compared with each other.

The above method, like the claimed invention, allows you to automatically evaluate the readability of the text. However, unlike the claimed invention, this method evaluates the readability of the text only in terms of the semantic complexity of the text, and is not intended for further text editing.

Computer system featured in "Detecting document text that is hard to read", (US08990224, Google Inc., United States Office, Application Number 13674320, Filing Date 11/12/2012, Publication Number 08990224, Publication Date 03/24 .2015) is configured to determine the parts of the text extracted from the corresponding group of documents; provides the ability to process a specific part of the text using a set of filters, where the specific part of the text can correspond to a specific document, and where each of the filters can generate an appropriate score based on the processing of a specific part of the text; calculate a readability score based on the corresponding scores generated by the filters; determine that the readability score meets the threshold score; and generate or select a new portion of text for a particular document based on the determination that the readability score meets the threshold score. This technical solution, in contrast to the claimed invention, is intended solely to analyze the headings of the text and support the user in correcting them to improve their compliance with the content and simplify perception.

Apparatus and method for improving line-to-line word positioning of text for easier reading (US6766495, International Business Machines Corporation, Office of the United States of America, Number applications 09406188, Filing Date 09/27/1999, Publication Number 6766495, Publication Date 07/20/2004) improve the readability of text in a computer system by changing the arrangement of one or more words to eliminate potential problems in readability that can be identified by examining the text. When a potential problem is identified, word-for-word text positioning can be adjusted to compress one or more lines and/or expand one or more lines to move one or more words to another line. For example, if two adjacent lines start with the same word, the first line can be compressed so that the first word of the second line is moved to the end of the first line. Alternatively, the first line may be expanded so that the last word of the first line is moved to the first word of the second line. By selectively changing the arrangement of words, you can significantly improve the readability of the text.

The above technical solution, like the claimed invention, allows you to analyze the text for the presence of repetitions, however, unlike the claimed invention, the functionality of this technical solution is limited to detecting repetitions and does not analyze the text for other opportunities to improve the readability of the text.

Natural language processing system "Readability awareness in natural language processing systems" (Ensuring readability in natural language processing systems), (US20170193093 - International Business Machines Corporation, Office of the United States of America, Application number 15162641, Filing date 05/24/2016, Publication number 20170193093, Publication date 07/06/2017, Patent grant number 09875300, Patent issue date 01/23/2018) and its developing technology "Readability awareness in natural language processing systems" (Providing readability in natural language processing systems), (US20190179840, International Business Machines Corporation, Office United States of America, Application number 16274663, Filing date 02/13/2019, Publication number 20190179840, Publication date 06/13/2019, Patent grant number 10534803, Patent date 01/14/2020) are designed to determine the readability level of the text based on the readability level indicator. Unlike the claimed invention, these systems calculate the level of readability depending on the presence of grammatical and spelling errors, jargon terms in the text and do not take into account other morphological, lexical and syntactic criteria.

You can also highlight a number of technical solutions aimed at analyzing text in natural language: "Multi-stage recognition of named entities in texts in natural language based on morphological and semantic features (patent for invention No. 2619193 RU, Application: 2016124139, 06/17/2016, Patent holder: with limited liability "Abi InfoPoisk" (RU)), "Identification of phrases in texts in natural language" (patent for invention No. 2618374 RU, Application: 2015147536, 05.11.2015, Patentee: Limited Liability Company "Abi InfoPoisk" (RU) ), “The text summarization method and the device and machine-readable information carrier used for its implementation” (patent for invention No. RU 2 635 213, Application: 2016138082, 09/26/2016, Patent holder: Samsung Electronics CO., LTD. (KR)), “ Detection of linguistic ambiguity in the text ", (patent for invention No. RU 2 643 438, Application: 2013157757, 12/25/2013, Patent holder: Limited response company property of Abi Production (RU)).

The above technical solutions, as well as the claimed invention, allow one way or another to analyze the text, however, the principle and operation of the analysis differ significantly from that implemented in the present invention.

As a prototype of the claimed invention, one can consider the technical solution disclosed in the publication US20160306787 “Computer processes for analyzing and suggesting improvements for text readability” (“Computer processes for analyzing and suggesting improvements for text readability”) (Copyright holder: Wordrake Holdings, LLC, Office of the United States of America, Application number 15191418, Filing date 06/23/2016, Publication number 20160306787, Publication date 10/20/2016, Patent grant number 09953026).

The technical solution presented in US20160306787 describes a computer process for analyzing and improving the readability of documents. The readability of a document is improved by using rules and appropriate logic to automatically detect various types of recording problems and make and/or suggest changes to fix such problems. Many of the rules aim to create more concise wording of the sentences being analyzed, for example by eliminating unnecessary words, rearranging words and phrases, and introducing various other types of editing. Proposed changes can be communicated, for example, through a word processing platform, by changing the appearance of the text to indicate how the text will look with (or with and without) editing.

The main disadvantage of the technical solution presented in the publication US20160306787 is that this system is intended for the analysis of the English language and cannot be effectively used for the analysis of Slavic languages, including the Russian language. This is due to differences between the languages themselves and their word-formation rules, which in turn implies the use of different approaches to the analysis of texts in these languages.

English belongs to analytical languages with an agglutinative structure, in which grammatical meanings are expressed mainly with the help of function words. Formants in English, as a rule, do not form indivisible structures and do not change under the influence of other formants. In this regard, text processing in natural English is possible by using a fixed dictionary containing specific word forms, as provided in the above technical solution US20160306787.

The Russian language belongs to synthetic languages with an inflectional structure, where inflection dominates with the help of inflections - formants that combine several meanings at once. In this regard, systems for processing text in natural Russian should not only use a fixed dictionary, but also take into account the morphology of words and their possible endings. Given that the above technical solution US20160306787 provides for the use of a fixed dictionary, we can conclude that the use of this technical solution for processing text in Russian cannot be complete, because this system is based solely on the clear search algorithm and will not take into account all possible word forms formed with the help of inflections, and, as a result, will not respond to these word forms.

In the claimed invention, this technical problem is solved by using the use of a fuzzy search, when words are indicated with possible options (for example, "my|his|her|...", which corresponds to possessive pronouns in different genders, persons and cases), with fuzzy endings (e.g. "\bjary\w{1,2} +impressions\w{1,3}\b", which corresponds to the phrases "bright impressions" or "bright impressions"). Also, the claimed invention uses a search mechanism for the initial forms of words (for example, "<%at%><><%dawn|sunrise|exodus%>", which corresponds to "at dawn", "at the end", etc.) and the mechanism search by grammatical features (for example, “<deepr>” corresponds to gerunds, and “(<A|S, im><>){5,10}” specifies a chain of 5–10 adjectives or nouns in the nominative case.). These technical features make it possible to fully analyze the text in Russian, when the text analysis system takes into account not only specific word forms recorded in the dictionary, but also all their possible variations.

Also, the claimed invention in comparison with the prototype has a number of other technical differences. Unlike the prototype, the claimed invention is made with the possibility of automatic evaluation of text quality, the possibility of optimizing text processing to automatically evaluate the quality of text after editing, and also with the possibility of providing an API for connecting third-party systems (for example, content management systems (Content Management System, CMS ) and other information systems.Thus, the proposed technical solution differs significantly from the technical solution presented in the prototype.

The technical problem to be solved by the present invention is to create a method, the use of which will allow the editor to increase the speed and efficiency of text editing in order to improve its readability.

The technical result of the present invention is to enable automated selection of relevant recommendations to improve the readability of the text.

The specified technical result when using the claimed invention is achieved through automated morphological, lexical and syntactic analysis of the source text, performed taking into account the characteristics of Slavic languages as synthetic languages with an inflectional system, and aimed at identifying stop words in the analyzed text, assessing the readability of the text, selecting and displaying relevant recommendations for improving the readability of the text, as well as re-analyzing user-edited sentences and recalculating the readability score of the text after editing.

Disclosure of invention

Method for automated text analysis and selection of relevant recommendations to improve its readability, performed on a computer, including

- getting the original text;

- identifying stop words in the source text in accordance with the specified rules for determining stop words by conducting morphological, lexical, semantic and syntactic analysis of the source text based on the use of fuzzy search, search by initial forms of words, search by grammatical features and search by punctuation;

- assessment of the readability of the source text based on the calculation of the ratio of the number of all words in the source text and the number of stop words identified in the source text;

- displaying relevant recommendations for improving the readability of the text, including at least an indication of the rule for determining stop words, according to which words in the source text were determined to be stop words.

Obtaining the source text when implementing the method claimed for registration can be carried out by entering the source text by the user, copying or loading the text from a file. Also, the source text can be obtained from other computer systems through the API.

Such APIs may have a number of predefined restrictions depending on the predefined access parameters of a particular user. So, depending on the access parameters for a particular user, the allowable amount of the source text and / or the available set of rules can be set in accordance with which the analysis of the source text and / or the content of recommendations displayed to the user to improve the readability of the text and / or the available number of requests in a certain time period.

Identification of stop words in the source text is carried out in accordance with the specified rules for determining stop words by conducting morphological, lexical, semantic and syntactic analysis of the source text based on the use of fuzzy search, search by initial forms of words, search by grammatical features and search by punctuation;

The rules for determining stop words are set using the method of regular and superregular expressions.

The method for defining stopword rules using regular expressions is to construct a regular expression pattern (using a dialect provided by the Python standard library) that defines one or more strings of characters.

Examples:

Sample Explanation Example strings that match the pattern \boften\b "\b" means a word boundary.
The template defines the string "often" framed by word boundaries, i.e. fragments where alphabetic and non-alphabetic characters meet, or sentence boundaries. often \bvarious+kinds\b "+" (space, plus) specifies a sequence of one or more spaces. different kind
different kind
different kind \ba little(hot|hard)\b "(go|zhko)" matches the string "go" or "zhko" Little
a little \bactual\w{1,4}\b "\w" matches any literal character.
{1,4} specifies the character to be repeated 1 to 4 times. topical
up-to-date
relevant

The method of specifying the rules for determining stop words using superregular expressions is to compose a template in the superregular expression language created in the framework of this invention. These templates define a set of chains of linguistic tokens obtained from the text being checked.

This language has four component components: linguistic tokens, gramme set pattern definition language, linguistic token pattern definition language, token pattern chain pattern definition language.

1. Linguistic tokens

When preparing the text to be checked, it is divided into words and interword chains of characters, each word is matched with the initial form and a set of grammatical features (grammes). In case of ambiguity, several combinations of initial forms and grammes can be associated with a word.

Examples:

original word initial form Grammemes friend friend S, husband, od, dat, ed wow wow INTJ analysis analysis S, husband, neod, vin, ed analysis S, husband, neod, im, ed

2. Language for defining gramme sets

The gramme set definition language is used to construct templates against which the gramme sets obtained after splitting a sentence into lexical tokens are checked. The language allows you to find tokens with certain grammatical characteristics.

Language grammar in EBNF format:

pattern = and_expr | or_expr "," and_expr

or_expr = not_expr | not_expr OR or_expr

not_expr = "~" simple_expr | simple_expr

simple_expr = string

string = letter | letter string

letter = "A" | ... | "Z" | "a" | ... "z" | "A" | ... | "I" | "-" | "_"

In accordance with the grammar, the priority of operations is defined as follows: "AND", "OR", "NOT".

Examples:

Sample Explanation Examples of sets that match the pattern S The set must contain the gramme "S" (noun) S, husband, od, dat, ed
("friend") S, Wednesday The set must contain the gramme "S" and the gramme "environment" S, medium, neod, vin, ed ("word") S|A The set must contain the gramme "S" or the gramme "A" S, husband, neod, vin, ed (“example”)

A, vin, ed, full, husband, neod ("approximate") ~V The set must not contain the gramme "V" INTJ ("wow") S|A, genus, ~ unit The set must contain the gramme "S" or "A", as well as the gramme "wine", but must not contain the gramme "ed" S, medium, neod, genus, mn ("words")

3. Language for defining linguistic tokens

The linguistic token definition language is used to compile templates against which linguistic tokens are compared, obtained after linguistic analysis of the text being checked.

Each template consists of three parts:

1. A subpattern that the set of grammes of the token must match.

2. A subpattern that the initial form of the token must match.

3. A subpattern that a specific string of characters from the text being checked must match.

However, each part is optional.

A linguistic token is considered to match a pattern in the language token definition language if and only if all of the specified subpatterns are matched.

The token language template is composed according to the following grammar:

pattern = subpatterns | "!" subpatterns

subpatterns = grammar_pattern | grammar_pattern "%" subpatterns_2
| "%" subpatterns_2

subpatterns_2 = lexeme_pattern | lexeme_pattern "%" form_pattern
| "%" form_patten

lexeme_pattern = regex

word_pattern = regex

At the same time, grammar_pattern is a pattern in the grammar set definition language, and regex is the regular expression language used in the Python language.

In case of ambiguity, if a given word form has several sets of grammes and/or several initial forms, the linguistic token is considered to match the template in accordance with the presence of the sign "!" at the beginning of the template: if the sign "!" is specified, then the token is considered to match the pattern if all parsing options match it. If a "!" is not specified, then one of the options is sufficient.

Examples:

Sample Explanation Examples of text snippets that match the pattern A|S, im A subpattern for grammes was specified, but there are no subpatterns for initial form and specific form word %dawn|sunrise|exodus% A subpattern for the initial form is specified, but there are no templates for grammes and a specific form. dawn
sunrise
outcome %%! Specified subpattern for a specific form of interword character string ! %%[^,]+ A subpattern was specified for a character string that does not contain a comma. example

4. Language for defining chains of tokens (superregular expressions)

The token pattern chain pattern definition language is used to construct patterns against which a chain of one or more linguistic tokens, obtained as a result of parsing the text being checked, is checked against.

The language is defined by the following grammar

superegex = part

| part superegex

| superegex "|" superegex

part = predicate

| repeatable

| repeatable "+"

| repeatable "*"

| repeatable "?"

| repeatable "{" number comma number "}"

| repeatable "{" comma number "}"

| repeatable "{" number comma "}"

| repeatable "{" number "+" "}"

| repeatable "{" number "}"

| repeatable "{" "}"

| repeatable "{" comma "}"

| "^"

| "$"

repeatable = "(" superegex ")" | predicate

number=digit | digital number

digit = "0" | "1" | "2" | "3" | "4" | "5" | "6" | "7" | "8" | "nine"

comma="," | ";"

predicate="<"token_pattern">"

token_pattern in this context is the linguistic token pattern described above.

Templates allow you to set sequences of linguistic tokens indicating the optional occurrence of a token, repetition of tokens in a certain amount, indicating variants of tokens, indicating the location of the token (beginning or end of a sentence).

Syntax Meaning <...> One token. <...> <> <...> <> <...> A sequence of five tokens.

<…> can specify certain characteristics of words, and <> between words matches any token and thus can match punctuation or spaces between words. repetitions <...>{n, m} Repeat the token n to m times. <...>{n,}
<...>{n+} Repeat the token from n times. <...>{,n} Repeat the token up to n times. <...>{n} Repeat the token exactly n times. <...>?
<...>{0,1} Optionality of token entry <...>+
<...>{1, } Repeat token 1 or more times <...>*
<...>{0, } Repeat token 0 or more times grouping ( <...> <...> <...> ) Patterns can be grouped using parentheses Alternatives <...> <...> | <...> <...> If the character "|" specified outside parentheses between different patterns, the token chain is considered to match the pattern if it matches the left or right side. <...> (<...> | <...>) <...> If "|" specified inside parentheses, it specifies alternatives for the template fragment. Beginning and the end ^... Corresponds to the beginning of the text (sentence). … $ Corresponds to the end of the text (sentence).

Examples

Sample Explanation <CONJ> Matches a chain of one union. ^<CONJ> Matches a string of one conjunction at the beginning of a sentence. ^<V, inf> Matches a string of infinitive verbs at the beginning of a sentence. (<S><>){5,} Matches a string of five or more nouns in a row, separated by arbitrary interword characters. <%presence|absence%><><genus> Matches a string consisting of a word whose initial form is "presence" or "absence", followed by an arbitrary string of characters, followed by a word in the genitive <V%s?do%><>+<create> Matches a verb whose initial form is "make" or "do" followed by a string of one or more arbitrary tokens followed by a word in the instrumental case.

The rules are divided into categories:

postcard stamp

Personal pronoun

Possessive pronoun

Amplifier

Generalization

Uncertainty

Biased assessment

Chancery

Modern journalistic stamp

Information style stamp

household stamp

Journalese

Pleonasm

Time parasite

Corporate stamp

Advertising stamp

Say it in Russian

Political correctness or euphemism

Compound predicate

Possibly a bad subject

introductory construction

Complex syntax

Superfluous submission

Phrase with verbal noun

Phrase with modal verb

indefinite

Prepositions

Weak verb

Chain of words in common case

Possibly a syntax issue

hard to read

Suspicion of a complex subconstruction

Suspicion of parceling

Heavy opening construction

Stylistic features

Featureism

Minor Syntax

obscene expression

Euphemism

The source text is divided into sentences, each of which is searched for stop words. The search for stop words in the source text is carried out by applying fuzzy search, search by the initial forms of words, search by grammatical features and search by punctuation.

When conducting a fuzzy search for stop words in the source text, stop words are identified taking into account possible variants (word forms). So, for example, if the rules for determining stop words set the inflectional pronouns “my|his|her|...” as a stop word, which corresponds to possessive pronouns in different genders, persons and cases), as a result of a fuzzy search, these pronouns can be identified regardless of their specific word form. with indistinct endings (for example, "\bjary\w{1,2} +impressions\w{1,3}\b", which corresponds to the phrases "bright impressions" or "bright impressions").

Conducting a fuzzy search allows using the claimed method to provide the possibility of automated selection of relevant recommendations to improve the readability of text in synthetic languages with an inflectional system, including languages of the Slavic language group, for example, in Russian.

When searching for the initial forms of words, stop words can be found by their initial forms. For example, if the stop word rules set the stop word to the newspaper stamp "at dawn", "at sunrise" and "at the end", such words can be identified by searching their initial forms using the rule "<%on%> <><%dawn|sunrise|exodus%>”, as a result of which the lexemes “at dawn”, “at sunrise”, “at the end” will be identified as stop words in the source text, regardless of their specific word form, depending on noun case.

When searching by grammatical features, stop words in the source text are determined in accordance with their grammatical feature. So, for example, if the rules for determining stop words set a sequence of three consecutive nouns as stop words, then for the search by grammatical features, it is established that “<n>” matches nouns, and the rule “(<A|S, them ><>){3,…}” defines a sequence of three nouns and adjectives in the nominative case, which makes it possible to search the source text for a sequence of three nouns in the nominative case.

When searching for punctuation, speech structures that impair readability are identified using the analysis of punctuation marks. For example, to highlight fragments from interrogative sentences ending in the words “what?” / “what?” / “what”, a rule can be set:

<% what %> <% \? %>

All of the above search methods can be combined. For example, we can set one rule that will determine:

1. Word noun or adjective

2. The word is not in the nominative case

3. The original form of the word begins with "den"

4. A specific word form ends with "m".

This rule can be presented in the form:

<n|adj, ~im % den\w+ % \w+m>

Using this rule will make it possible to identify the word "day" in the text, whose initial form is "day".

Also, when using the proposed method, the rules can be set for several consecutive words. For example, we can set a rule that will look for the word "day" if it is preceded by the word "but" and not followed by the word "this". In this case, the rule can be represented as:

< % but % > <> <n|adj, ~im % den\w+ % \w+m> <> < % (?! this) % >

As a result of the morphological and semantic analysis of the source text, stop words are identified in the source text.

Next, the readability of the text is evaluated and the readability of the text is displayed to the user. To assess the readability of the text, the number of words in the source text and the number of identified stop words are determined. The text readability score is calculated by the ratio of the number of all words in the source text to the number of those words in the source text that are defined as stop words.

In the event that the claimed method is used for further automated publication of text through computer systems, such systems may provide for a ban on the publication of text whose readability score is less than the established allowable readability score of the text allowed for publication.

Further, relevant recommendations for improving the readability of the text are displayed.

Recommendations for improving the readability of the text are displayed for each detected stop word and depend on the rule according to which the word in the source text was determined to be a stop word. A recommendation to improve readability may contain a description in natural language of the rule by which this word was determined as a stop word, an indication of the category of rules of the identified stop word, examples of editing stop words of the specified category in the "before-after" format, hyperlinks to the publication, containing more detailed recommendations on editing stop words from the specified category, as well as other recommendations that will be useful for the editor to work out the stop word in order to increase the readability and informativeness of the text.

The claimed method may additionally include displaying a text readability score to a computer user based on the analysis of the source text.

The readability score of the text is used by the user to decide whether it is worth further editing the source text, or the text already has a sufficiently high readability score, which means it will be qualified by the reader as readable.

The claimed method, when assessing the readability of the text, can additionally take into account the categories of rules, according to which the stop words were identified in the source text, when assessing the readability of the source text.

In this case, to assess the readability of the text, the number of words in the source text is determined and the sum of the weights of the rules for determining stop words is determined. Each rule is assigned a conditional weight expressed as a number. Depending on the categories and the number of stop words found in the source text, as well as the ratio of the number of words in the source text and the number of those words in the source text that are defined as stop words, the text readability score is calculated.

For example, the readability of a text can be evaluated on a ten-point scale using the formula:

— количество баллов по шкале от 0 до 1 (при этом значение может выходить за границы диапазона).

- the number of points on a scale from 0 to 1 (in this case, the value may go beyond the limits of the range).

— итоговое количество баллов по шкале от 0 до 10

- the final number of points on a scale from 0 to 10

where the weights variable corresponds to the sum of the weights of the rules for determining stop words, the words variable corresponds to the number of words in the source text, the penalties variable corresponds to the sum of the estimated penalties for the found stop words, and the score indicator is the final indicator of the text readability on a ten-point scale.

For example, the following paragraph is parsed:

Who doesn't know back pain? This is certainly one of the main problems of modern man. Doctors from all over the world are puzzling over the question of how to save us from this scourge.

In such a text, when using the proposed method, the following stop words will be detected:

Fragment Category The weight Fine Who doesn't know back pain? syntax rules fifty 0 undoubtedly amplifier 100 0 one of the main indefinite 100 0 modern man advertising stamp 100 0 racking their brains journalese 100 0 misfortune journalese 100 0 us personal pronoun 0 0 from prepositions thirty 0 from all over the world generalization 100 0 Total 680 0

In 28 words of the source text, 9 stop words (words, phrases, constructions) from different categories were found. Thus:

Taking into account the weights of the rules, the score on a ten-point scale is calculated as 4.3, which means that the readability and informativeness of the text is low.

The claimed method may additionally include visual selection of stop words in the source text, determined as a result of the analysis of the source text. Visual highlighting is done so that the user can quickly determine which words of the source text are defined as stop words. For example, visual highlighting of stop words can be done by underlining them and/or highlighting the identified stop words with a color that differs from the color of other words in the source text that are not defined as stop words.

The claimed method may additionally provide the possibility of editing the source text by the computer user and analysis and selection of recommendations to improve the readability of the edited text.

In this case, the user edits the source text in accordance with the recommendations for improving the readability of the text, then the text is reanalyzed. In order to optimize the load on the executing computer, reanalysis can be carried out not for the entire text, but only for the edited part. In this case, those proposals are selected that have not been analyzed before, in particular, modified and added proposals are selected. Further, the analysis of sentences edited by the user is carried out in the same way as the analysis of the original text was carried out.

Then, the readability of the text is evaluated, taking into account the sentences edited by the user, and the readability rating of the edited text is displayed to the user. Based on the readability score of the edited text, the user may decide to continue editing the text or to end editing the text.

An exemplary embodiment of the invention

The method claimed for registration as an invention can be implemented in an automated computer-implemented decision support system for the editor (hereinafter referred to as DSS).

In this case, the system will work as follows.

The editor on the computer monitor is shown the user interface - the web interface of the system or its interface in third-party applications or plug-ins with which the system is integrated. The user interface contains two fields. The first field is for entering and displaying the source text. The second field is designed to display DSS recommendations corresponding to the stop word selected by the editor in the source text.

At the first stage of the method, the editor enters the source text into the appropriate field of the DSS. Also, the text can be received by the DSS through the API directly from other computer systems. As a result of the stage execution, the DSS receives the source text for further analysis.

At the second stage of the implementation of the method, the internal DSS controller performs morphological and semantic analysis of the text, interacting with the rule base and the cache of compiled rules contained in the system or external data storage. For text analysis, a linguistic analyzer module, a regular expression search engine module, a linguistic regular expression search engine module, and a module of optimized search functions are used. At the same time, DSS can be connected to other modules aimed at solving user problems.

If necessary, the DSS highlights stop words in the source text with underlining and color. The way text is visually highlighted may depend on the category of rules by which the stop word was defined.

At the fourth stage of the implementation of the method, the DSS performs the function of evaluation and calculates the readability of the text on a ten-point scale. The score may be reflected to the editor.

At the fourth stage of the implementation of the DSS method, the editor displays recommendations for improving the readability of the text. Demonstration of recommendations for the development of stop words can be implemented as follows: the editor points the mouse or caret cursor over the stop word selected by the DSS, after which the DSS displays to the editor the category of this stop word, advice on how to eliminate it, and can also display examples of eliminating similar stops -words in the “before-before” format, and/or a hyperlink to a publication containing more detailed recommendations on working out stop words from the specified category. Such a publication with more detailed recommendations can be an article, an audio or video file, an interactive training simulator.

Next, the editor edits the text, and the DSS can cache the text and analyze only the edited or new sentences without re-analyzing sentences that have not been changed by the user. As a result, the DSS recalculates the readability score of the text on a ten-point scale and displays the readability score of the edited text.

Claims (9)

1. A method for automated text analysis and selection of relevant recommendations to improve its readability, performed on a computer, including: getting the original text; identifying stop words in the source text in accordance with the specified rules for determining stop words by conducting morphological, lexical, semantic and syntactic analysis of the source text based on the use of fuzzy search, search by initial forms of words, search by grammatical features and search by punctuation; assessment of the readability of the source text based on the calculation of the ratio of the number of all words in the source text and the number of stop words identified in the source text; displaying relevant recommendations for improving the readability of the text, including at least an indication of the stop word definition rule, according to which words in the source text were determined to be stop words. 2. The method according to claim 1, additionally including displaying a text readability score to a computer user based on the analysis of the source text. 3. The method according to claim 1, in which the assessment of the readability of the source text based on the calculation of the ratio of the number of all words in the source text and the number of stop words detected in the source text is carried out taking into account the categories of rules, according to which the stop words were detected in the source text. text. 4. The method according to claim 1, additionally including visual selection of stop words in the source text, determined as a result of the analysis of the source text. 5. The method according to claim 1, additionally providing the possibility of editing the source text by the computer user and carrying out subsequent automated analysis of the source text sentences edited by the computer user, assessing the readability of the edited text and selecting relevant recommendations to improve the readability of the edited text.