RU2718978C1 - Automated legal advice system control method - Google Patents

Abstract

FIELD: computer equipment.

SUBSTANCE: method of managing an automated system of legal consultations consists in forming a database of arbitrary questions, selecting questions of one area of law, for which the number of directions of responses is determined in the form of categories of questions, text questions are vectorized into binary vectors of questions, questions are classified by categories on the basis of proximity of vectors, so that classes are formed for branches of law, and when a user question is entered into a system, a user question is parsed vocabulary with determining the lexical structure of the user question, determining, based on the generated classes for branches of the right, the provision of the regulatory legal act contained in the knowledge database, relevant to user question according to normative legal acts vectors, generating a lexical response structure to a user, corresponding to the lexical structure of the user question and the position of the regulatory act relevant to the user's response, and providing the user with a lexical response structure in the form of a text message.

EFFECT: high probability of generating a correct answer to a user question, reducing the effect of unskilled experts on system operation results.

10 cl, 5 dwg, 1 tbl

Description

FIELD OF TECHNOLOGY

The invention relates to computer technology and, in particular, to automated expert systems and can be used as an automated system of answers to questions of a legal nature, for example, in identifying the possibilities of making claims, establishing facts of offenses, as well as on issues of committing actions that do not contradict the law in difficult domestic situations.

BACKGROUND

Known systems use feedback to form critical methods, based on the results of the reaction of end users. At the same time, method errors are eliminated, as a rule, as the database of questions and answers expands, and as answers are collected, assessments of the correctness of the expert decisions made based on the results of user surveys are collected. A disadvantage of the known systems is that when applied to legal practice issues, the systems do not provide the right decisions or the formation of the right answers for reasons of an objective and subjective nature.

One of the objective reasons is the fact that for some legal conflicts there are no correct answers and a decision on a question is formed in a subjective way, for example, by a judge based on the subjective concept of justice. The subjective reason may be the interaction of the user of the expert system and the department making the decision on the object of interest. For example, information provided inappropriately and beyond the established deadline may not be taken into consideration, which may lead to negative feedback from the user of the expert system on the whole issue, despite the fact that the disadvantage of the recommendation was that there was no indication of the importance of observing the deadline form proper documents. In addition, in the process of enforcement, interpretations of regulatory documents may change.

The prior art is a reference legal system for storing and searching data (see RU 2223537, 02.10.2004), containing a search type selection unit, a query generator, one output of which is connected to the search unit, the input-output of which is connected via databases to the corresponding buses system data, a display unit and a controller for managing data search, characterized in that a search condition selection unit is introduced into the system, the input of which is connected to the output of the search type selection unit, the first output is connected to the input I request a second output coupled to an input of the attribute filter, a first input-output of which is connected to the second input-output of the request, the second input-output - with the block of memory attribute filter, a first input-output request generator coupled to the memory block requests.

The disadvantages of this solution include, at least, the lack of the possibility of forming answers to the question posed by the user and ensuring the high adequacy of answers, which consists in understanding the question posed by the user and forming the answer corresponding to the question posed.

The prior art system is known for collecting legal information and analyzing it to identify problematic issues and deficiencies in legal regulation (see RU115095, 04/20/2012), which contains databases, a unit for entering documents into these databases, a central server and computerized user tools, characterized in that the databases are formed in a block that includes at least the Legal Acts database, the Legal Advice database, the Judicial Practice database, the Thesis and Research database, the Reviews, articles pu database “newsletters”, the base “Foreign Legislation”, the base “Regional Legislation”, the base “Legislation of the State”, containing a selection of documents related to the topics of the listed databases and having identification signs for their detection and presentation on a monitor in electronic format, a block for entering these documents into a database executed with the function of translating a document on paper or machine-readable medium into electronic format with the assignment of identification features and distribution according to the specified databases , a document loading unit from a database unit selected according to a common criterion for matching identification features, which is connected to a central server configured to connect to computerized means of users for displaying downloaded documents on their screens and which is configured to provide a search apparatus for generating a request in a request processing unit associated with a central server, a user registration unit configured to store data about users with their access codes to the block with the databases and the scope of rights for this access, which is connected with the central server and with the request processing block for allocating the authorized access channel for the user after identification when connecting to the central server, the block for collecting offers sent in the mode free access from computerized means of users, performed with the function of registering an offer and identifying the sender and transmitting this proposal to the block for grouping proposals thematic spirit or substantive destination and transmitting the grouped packages these proposals analytical processing unit for comparing the reflected questions in sentences with the legal provisions regulating them with documents from the database unit data to generate output.

The disadvantages of this solution include, at least, the lack of the possibility of forming answers to the question posed by the user and ensuring the high adequacy of answers, which consists in understanding the question posed by the user and forming the answer corresponding to the question posed.

Also, a method and system are known from the prior art (see US 20100063925 A1, March 11, 2010) for providing certain types of legal services to licensed and practicing lawyers to clients and potential clients of a law firm, and a system implementing this method contains a first computing device for transmitting the first set of information requested by the client; a second computing device for transmitting a set of legal advisory information, the second device being remote from the first device; an intermediate server configured to remotely work with said first device and second device; as well as at least one database associated with said intermediate server, the database being capable of storing information requested by the client and information of legal advice, the method comprising inputting an answer by a lawyer to a client's question; storing such a response in the database and an interface for receiving the specified response by the client using software installed on the user's computing device.

The disadvantages of this solution are at least the low probability of forming the correct answer to the question posed by the user, the high impact of experts, including unskilled experts, on the results of the system, as well as the low adequacy of the answers, which consists in understanding the question posed by the user and formation of the answer corresponding to the question posed.

Thus, in the prior art there is a need for an automated expert system related to solving legal issues with the maximum probability of making the right decisions and issuing recommendations with the maximum completeness, sufficient enough for the user to either solve the issue further on his own or evaluate the difficulty of solving the issue and entrusted his decision to a qualified professional. Also, in the prior art there is a need for a method of managing an automated legal advice system.

ESSENCE

The proposed invention solves the problem, as well as ensures the achievement of a technical result, which consists in increasing the probability of forming the correct answer to the question posed by the user, updating law enforcement practice, reducing the impact of unskilled experts on the results of the system, as well as the adequacy of the answers. Here, adequacy is understood as a correct understanding of the question posed by the user and the formation of the answer corresponding to the question posed.

According to one implementation option, a method for managing an automated system of legal advice containing a knowledge database that contains a set of norms and rules of existing legal documents, a module for preprocessing and classification of questions, performed by forming a vector representation of the essence of the questions, and a module for generating answers from the knowledge database to the lexical representation corresponding to the lexical construction of the question, while forming a database of arbitrary questions, choosing they ask questions of one area of law, for which the number of directions of answers is determined in the form of categories of questions, vectorize the text of the questions into binary question vectors by presenting unique words in the form of a table containing unique words indicating the presence or absence of unique words in the text of each question, and values of the table forms a vector, questions are classified into categories based on the proximity of vectors by grouping questions using the norms and rules of equal documents stored in the knowledge database and cluster analysis, during which clusters are formed, the number of which is equal to the groups formed by experts for each branch of law, the number of which, together with question vectors and a dictionary of unique words and phrases, is processed using cluster analysis, so what the classes for the branches of law form and when a user's question enters the system, they perform lexical parsing of the user's question with the definition of the lexical structure of the question the user, on the basis of the generated classes for the branches of law, determine the position of the normative legal act in the knowledge database relevant to the user's question on the vectors of normative legal acts, form the lexical structure of the answer to the user corresponding to the lexical structure of the user's question and the position of the normative legal act the user's response and provide the user with the lexical structure of the response in the form of a text message.

In one of the private embodiments, user questions are converted into a format suitable for automated analysis by means of automated processing of questions with the formation of a vector representation for each question.

In one of the private implementation options, the grouping of questions according to the proximity of the vectors and the grouping of questions by the directions of answers is determined, and if the grouping of questions by the proximity of the vectors and the grouping of questions by the directions of answers do not match, the parameters for converting user questions into a format suitable for automated analysis according to the grouping of questions on the basis of proximity of vectors and the grouping of questions by direction of answers, the parameters of ducation user issues in a format suitable for automated analysis, to convert the user issues in a format suitable for automated analysis, and the provisions of legal acts with the formation of the vectors of normative legal acts.

In one particular embodiment, the converted user questions into a format suitable for automated analysis are presented in the form of query arguments and descriptions of answers, presented in the form of function values.

In one of the private options for implementation, questions are grouped according to the proximity of the vectors and questions are grouped in the directions of the answers.

In one of the private implementation options when processing the text of questions: remove irrelevant characters from the text of the questions, including at least punctuation marks, HTML markup elements; perform tokenization of texts, dividing sentences into separate characters that are tokens; perform concatenation of words with prepositions and particles to preserve the emotional coloring of the texts; remove irrelevant words that do not carry a semantic load, including pronouns and interrogative words and words found in more than 80% of questions; perform lemmatization of tokens by reducing tokens to a dictionary form; perform stemming by finding the word base for a given source word, if the dictionary form is not found.

In one particular embodiment, the dictionary of unique words consists of dictionary forms obtained in the process of lemmatizing tokens and / or stamming.

In one of the private embodiments for each of the phrases, the presence of unique words and phrases in the question phrase is determined and a tabular line is formed in which the number of cells corresponds to the number of words and phrases in the dictionary, so that each cell corresponds to one of the dictionary words and phrases, and the value entered in the cell corresponds to the presence or absence of the corresponding word or phrase in the phrase corresponding to the position or number of the cell, where the set of values entered in the table cells is a vector and is used to analyze the corresponding phrase.

In one of the private options for implementation, the k-means algorithm is used for clustering, sorting the questions marked out by experts into clusters, where each of the questions refers to one cluster located at the smallest distance from the question vector.

In one particular embodiment, lexical parsing is performed using at least one semantic parser, the parsers searching for strings of characters predefined in dictionaries, where strings of characters are words or phrases combined by a common property in the source text of a user's question so that they create classes of legal problems that are identified by the algorithm and have a common legislative justification that determines whether the issue belongs to the cluster.

BRIEF DESCRIPTION OF GRAPHIC MATERIALS

FIG. 1 illustrates a configuration diagram and operation of the proposed system that implements the proposed method;

FIG. 2 illustrates the operation scheme of a semantic parser;

FIG. 3 illustrates an exemplary embodiment of the present invention;

FIG. 4 illustrates a simplified example of a hardware implementation of the proposed invention;

FIG. 5 illustrates an example of a computing system suitable for implementing elements of the proposed invention.

DESCRIPTION OF EMBODIMENTS OF THE INVENTION

To solve the problem and achieve a technical result, a method is proposed that is implemented by a legal expert system, the configuration and operation diagram of which is shown in FIG. 1.

In the process of managing the system, a database of tagged data (a database of arbitrary questions) 128 is generated using data from a database of unallocated information (data) 115, as described in the framework of the present invention.

The unallocated information database 115 contains user questions and consultant recommendations formulated in accordance with the questions posed, where questions and recommendations are presented in natural language terms. As a database of unallocated information 115, a database can be used, created, in particular, prepared, based on the results of communication of users 111 with consultants - individuals. Further, the database of unallocated information 115 can also be replenished with dialogs with the system, in particular, user questions (222, FIG. 2) containing the source text (232, FIG. 2), and generated by the system, in particular, the module for generating responses, answers ( 282, FIG. 2).

The database of marked-up data 128 is formed by the module for preprocessing and classification of questions from user questions (converted to a format suitable for automated analysis), in particular, queries presented in the form of arguments and response descriptions presented in the form of function values.

The system shown in FIG. 1 also contains a knowledge database 147, which contains a set of norms and rules of existing legal documents, presented in the form of arguments for descriptions of answers.

The legal expert system also contains:

- means for marking user questions 124, presented in natural language, ensuring the separation of user questions from separate topics and grouping topics into several independent groups based on similarity. Markers for user questions 124 are implemented in the assessment block that implements assessment functions;

- implementing markup validation cross-validation algorithms 126, made with a choice, for each separate subject of the question, the most likely group for each of the proposed ones;

- means for grouping marked-up data that are used (and, in a particular case, which are part of it), at least with a module for preprocessing and classification of questions, performing classification according to the topics of questions according to formal signs of similarity, where a specific answer to the question is used for a specific user question;

- dialogue manager (dialog box, means of (automated) dialogue with the user) 137, executed by the user 111 receiving the question in natural language, performing semantic analysis of the question text, taking into account the topics defined earlier, and forming the answer to the user question, with the final system response (using the module for generating responses) is based on user refinements received in the course of interaction with the dialog manager 137. System questions are formed, initially, in the format, both sintering the utmost brevity of the answer, and then, in a format understandable to the user based on the results of the previous clarification, and the subject is the name of a separate group of questions, for example, “Section of the property of spouses”. Also, the dialog manager 137 can convert questions (including parts of questions, for example, phrases) into a format suitable for automated analysis by means of the described system, as well as extract the subject of the question from the transformed question,

The module for preprocessing and classification of questions uses (in a particular case, contains) a means for marking user questions 124 and a means for (cross-) validating markup 126.

In additional embodiments of the invention, for the formation of the answer, clarifications are required, a presentation in the description of the situation corresponding to the question is used, and an additional system of refinements is used that forms the questions for the user. Questions are formed, initially, in a format that provides the utmost conciseness of the answer, and then, in a format that the user understands based on the results of the previous answer.

In another further embodiment of the invention,

- for systematization, duplication of the sorting of concepts is used, for which a human verification system is used in parallel with the systematization machine.

- for each of the systematizers, a quality parameter is set, in a particular case, by voting, and the grouping results are accepted taking into account the probability of a correct answer,

- the grouping is carried out both directly and indirectly, where for the grouping by the direct way the expert formulates the correct answer, and for an indirect assessment, the quality of the proposed answer corresponding to one of the topics is determined. In the particular case for each expert, the probability of the quality of the assessment is determined by comparing his answers with the answers of other experts and with the results of the expert system. The quality of the assessment of a particular expert is determined taking into account influencing factors, for example, time of day or topic.

After the initial markup of questions by experts, an automated system is trained.

To train an automated system with a module for preprocessing and classifying questions, the topics (or phrases) contained in user questions are converted into a formalized form, in particular, vectorized texts of user questions are used, and the module for preprocessing and classification of questions is used to vectorize the text of the questions. A formalized view refers to the presentation of topics set forth in questions in the form of vectors, that is, strings of a certain length. An additional topic parameter may be a description of the lexical construction of the question. The description of the lexical construction of the question is used for the automated formulation of the answer to the question, taking into account the grammatical features of the question, subject area and object of discussion. For example, for the question: “in what time is“ men's shoes ”of inadequate quality to be returned?”, The answer will contain the term, in particular, the phrase, “men's shoes”. For a question with a term, in particular with the word “boots”, the answer will contain a term, in particular with the word “boots”.

In the process of managing the system, questions of one area of law are selected, for which the number of directions of answers (categories) is determined. For subjects of questions, a (preliminary) grouping of data is performed, which is then used to form an automated decision-making system. The initial choice of topics is determined by the area of law to which the questions belong, for example, “copyright” or “consumer protection.” Grouping is carried out taking into account the possibility of errors caused, for example, by the possibility of ambiguous interpretation of the issue, or by the carelessness of the specialist performing the sorting.

Additionally, when grouping, specialists are presented with possible answers to questions related to a phrase or topic in order to assess the relevance of an answer to a posed question or phrase from a question.

At the same time, the answers also indicate the group to which the question belongs, for example, the correct answer, obviously belonging to a certain answer group, indicates the group to which the question belongs, an incorrect answer indicates that the question does not belong to the answer group.

The markup of the database of unallocated data 115 is carried out by performers, in particular, specialists (assessors, for example, legal assessors) 122 using means for marking user questions 124 and the methodology described above (and further in the description), which allows at least speeding up the markup data to solve the problems of automation of the further process of consulting users.

The methodology is implemented in the assessment block, which provides the formation of a database of tasks for specialists and requirements for completing tasks that include instructions (121). For the formed tasks, the presentation of tasks to assessors is provided, as well as the collection of the results of the tasks and the processing of the results.

To perform data markup, in particular, to simplify data markup:

- a category tree is created (where the categories are, for example: Civil Code, Part 4 of the Civil Code, patents, utility model patents, civil law, consumer protection, return of goods, replacement of goods, etc.). So, for example, the category may be “Family Law”, and the subcategories “Section of the property of spouses”, “Marriage”, “Determination of the child’s place of residence”, “Movement of the child”, “Custody”;

- performers (assessors) using means of marking user questions 124, i.e. in the particular case, the performers are connected to the assessor’s functional, in particular, markup means 124 are used;

- tasks for marking are formed, in particular, data files are loaded, marking rules (from existing ones) are selected, executors for specific tasks are appointed. The markup rules are a list of categories that are possible for marking, and for each category an instruction is developed that indicates all the conditions (rules) for assigning the issue to this category, as well as a list of exceptions to these rules. So, for example, the data in the task can be categorized into the following categories: “Poor-quality goods”, “Poor-quality technically complex goods”, “Imposed service”, “Denial of service”, etc. An instruction is created for the “Low-quality product” category that contains the conditions for classifying the issue into this category, for example: a low-quality product is a product from Part 1 of Article 18 of the Law of the Russian Federation “On the Protection of Consumer Rights” with the exception of technically complex goods; if the user (client) asks how soon the money will be returned to him in case of rejection of defective goods; if the client asks what to do, if defective goods are repaired for a long time, etc. The number of conditions varies depending on the category and complexity of the questions in this category.

- Other conditions for completing the task are determined: for example, how many times each object is shown to different performers for marking.

Further, through the personal account (of the performer), using logins and passwords, performers get access to tasks, performers are shown objects for marking, performers mark up data, for example, by indicating categories from the category tree.

Tagged data is stored in the database of tagged data 128, in particular, a local database, for example, implemented using MySQL, where category labels indicated by assessors are indicated, as well as information that allows you to take into account the individual level of effectiveness of assessors (using the cross-validation algorithm markup of assessors-lawyers, which allows you to automatically calculate the probabilities of each response of the assessor and cut off erroneous answers, as described in the framework of the present invention), affecting and markup results. Thus, category labels can be category names, category identifiers, etc., which are stored in the database of marked up questions 128.

In the next step, cross-validation of the markup of different assessors is performed by means of cross-validation of the markup 126.

When implementing the method, the probability of a correct solution is determined for each action, and incorrect answers identified using the proposed method, as well as answers that differ from obviously correct answers, are excluded from the subsequent consideration, but are taken into account when determining the probability of receiving an incorrect answer from the corresponding specialist.

It should also be noted that pre-selected questions of users from the database of asked questions of legal advice services 113, i.e. the database 115 stores information contained in the questions of users asked (and stored) on various resources, in exchange systems (text, instant, etc.) messages, etc. So, for example, for marking up, some of the questions can be selected from the total number of questions asked by users, for example, out of 3,000,000 asked questions, for marking up, 2,000,000 questions can be selected, each of which may be related to several branches of law.

Further, part of the questions selected for marking, for example, 300,000 questions, are marked out by specialists 122 using the marking tools of user questions 124, other questions can be marked out automatically using the conversion of questions into a format suitable for automated analysis, and the formation of vectors for each topic of the question .

The markup of data, in particular, the classification of questions into categories, including on the basis of proximity of vectors, including on the basis of specified rules, includes the joint use of an expert grouping of questions and results of cluster analysis methods. To carry out the classification of questions into categories, the system is trained to classify questions into (selected) categories. Thus, algorithms can be trained to classify questions into selected categories. So, words found in similar contexts tend to have a close meaning, for example, according to the distribution hypothesis in linguistics. A word can be represented as a vector, the elements of which correspond to the number of occurrences in a certain context. The proximity of vectors determines semantic similarity. The expert grouping of questions is carried out on the basis of analysis by a system of a representative sample from a database of unallocated information 115. The aforementioned representative sample is a sample of a finite volume that has all the properties of the initial population that are significant from the point of view of the research tasks.

Specialists (legal experts) 122 group issues based on the legislative framework and legal cases (cases) that arise from users. Further, using a clustering algorithm using the k-means clustering method (k-means method), clusters (machine clusters) containing texts of questions are formed, the number of which was predetermined based on the expert grouping. Thus, according to the results of marking up questions and topics of questions by assessors, the number of sets or clusters to which marked-up questions belong is set. In the simplest case, for the (first) branch of law, for example, to “protect consumer rights”, the number of such clusters is 44, which is one of the parameters passed to the clustering algorithm (k-means algorithm), so for each article ( for each norm of the corresponding branch of law) an appropriate cluster is formed. In addition, the basis is taken for the correspondence of the marked questions to one of the clusters. Further, the mentioned parameter is passed to the k-means algorithm as the parameter k, which is the number of clusters. Using the number of clusters equal to the number of norms of the relevant branch of law is the most convenient option, since changes in legislation, in most cases, lead to clarifications in the generated answers and do not require re-clustering of questions.

The selection for the allocation of machine clusters uses the same representative selection from the database of unallocated information 115 as in the expert grouping.

The sample is pre-processed, in particular, during processing (including normalization) of the text for clustering (including automated clustering), in the process of which natural language processing technologies are used (from the English - Natural Language Processing, NLP), including operations indicated below:

1. Clearing questions from irrelevant characters, for example, punctuation marks, html elements (HTML markup elements), etc., in particular, the removal of irrelevant characters;

2. Tokenization of texts - separation of sentences into separate characters (tokens);

3. Concatenation of words with certain prepositions and particles to maintain the correct meaning (presence / absence), as well as to preserve the emotional coloring of the texts. So, for example, when processing the text “We DO NOT agree with the results of the examination,” if you do not concatenate the particle “DO” with the word “agree,” when deleting irrelevant words, you get the following text “We agree with the results of the examination”, which has the opposite meaning original text;

4. Removing irrelevant words that do not carry a semantic load, in particular, which can occur in any context - pronouns, interrogative words, and those that were found in more than 80% of questions (in particular, observations) from the analyzed sample;

5. Token lemmatization - reduction of tokens to the (basic) dictionary form, and where the dictionary form is not found, stemming is performed, that is finding the basis of the word for a given source word.

In the particular case of the implementation for each of the topics (that is, for compound phrases of questions having different semantic contents) of questions of users by means of the system, (separate) normalization is performed, that is, conversion (phrases) into a format suitable for automated analysis.

In the particular case of the implementation, natural language processing technologies are used that implement phrase analysis to ensure that the phrase belongs to a specific topic, as well as highlight key elements from phrases that provide the subsequent synthesis of response phrases, in terms of generating competent text.

Question texts are converted to binary vectors, i.e. they are vectorized using a dictionary of unique terms (a dictionary of unique words and phrases), in particular, a presentation of the data is selected - vectorization of the text of the questions into binary vectors (vector of questions, vector of observations). A dictionary of unique terms is constructed by choosing, in particular, the formation of terms from a database of marked up data 128 (the source database). Question vectors are transferred to the clustering algorithm (k-means), based on the results of which a list of machine clusters is formed. Thus, a dictionary of unique terms is formed, consisting of dictionary forms used at the lemmatization stage of tokens, as well as dictionary forms obtained as a result of stemming. For each of the phrases, the presence of unique terms in the phrase is determined and a tabular line is formed in which the number of cells corresponds to the number of dictionary terms, each cell corresponds to one of the terms from the dictionary, and the value entered in the cell corresponds to the presence or absence of the corresponding term in the phrase, corresponding to the position or cell number. The set of values entered in the table cells is called a vector and is used to analyze the corresponding phrase. Thus, for normalized vectors, using the k-means algorithm, labeled questions are sorted into clusters, where each of the questions belongs to one cluster located at the smallest distance from the question vector. At the first step of the k-means algorithm, the data are randomly divided into clusters, and for each of the clusters the center of mass is calculated in accordance with the values of the data vectors included in the cluster. In the next steps, for each vector, the nearest center of mass is found, the vectors are redistributed into clusters, the next steps of the process occur iteratively until the next step gives the vectors redistribution relative to the previous one.

Centers of mass or clusters are subsequently used to determine whether unplaced questions are related to relevant topics.

With the usual initial mismatch between machine and expert clusters of about 40%, new groups are formed for mismatched clusters based on the constructed semantic parsers, and the parsers (242, FIG. 2) search for specific chains of characters predefined in the dictionaries (252, FIG. 2) . Character chains are words or phrases that are combined by a common property in the source text of a question of a user 111 asking a question, for example, on the website 153. For example, a common property of a group of questions may be “low-quality technically complex products”. Using the identifier of the class of problems (292, FIG. 2) of a multilayer neural network, such classes of legal problems can be created that, on the one hand, can identify the algorithm with high quality, on the other hand, have a general legislative justification, in particular, defining belonging to a cluster. In the particular case, 33 such classes were obtained in the consumer protection industry.

For the formation of an automated system of answers to questions or legal advice, the system is trained, namely, when training the system, the relevance of normative acts is checked. So, for example, the normative acts contained in the knowledge base 147 are compared with the normative acts contained in the reference (regularly updated) current database (for example, the database contained on a remote server, the Internet, etc.) and the results comparisons, the data in the database 147 is updated. For training the system, a multilayer neural network is used, the architecture of which is developed taking into account the specifics of the processed data. Also, in the process of developing the network architecture, Monte Carlo methods were used and numerous Monte Carlo experiments were carried out, which allow you to choose the best network parameters for training.

In the future, normalization algorithms that use neural networks are used to normalize the topics of other branches of law, which can significantly save time and resources for preparing the system for use, the correctness of the formation of vectors for other branches of law is verified by specialists 122 in an accelerated mode, mainly by determining the area of the corresponding cluster .

At the final stage of machine learning, a model is trained, which subsequently allows you to predict the category of the question asked by the user.

When processing text for clustering, NLP operations are also used:

- cleaning questions from irrelevant characters, for example, punctuation marks, html elements, etc .;

- tokenization of texts - separation of sentences into separate characters (tokens);

- concatenation of words with certain prepositions and particles to preserve the emotional coloring of the texts;

- removal of irrelevant words that do not carry a semantic load - pronouns, interrogative words and those that were found in more than 80% of observations from the analyzed sample;

- lemmatization of tokens - reducing them to a dictionary form, and where the dictionary form is not found, stemming is performed.

In the particular case, the decision trees are formed by the gradient boosting method, which determine the format of the answer to the question in accordance with the subject vectors, and the content of the answers is determined in accordance with the cluster to which the subject belongs. Among other things, the stylistic features of the issue are taken into account.

Texts of questions are converted into binary vectors (vectorized). Conversion to vectors is performed using a dictionary of unique terms extracted from a selection of questions from the source database. Question vectors are passed to the k-means algorithm, the result of which is a grouping of questions into clusters.

As mentioned above, on the topic of consumer protection, the proposed methodology formed 33 classes of 44 clusters.

When marking up questions by experts 122, the input quality of work is determined for each specialist, which is expressed in the proportion of correctly marked questions.

Table 1 shows an approximate variant of marking up questions by specialists.

Table 1.

Input quality of work Specialist Question 1 Question 2 Question 3 0.65 Specialist 1 A B B 0.91 Specialist 2 A IN A 0.8 Specialist 3 B G B

So, for example, for Question 1, three specialists marked out Question 1. The probability of the correct answer for Question 1 is calculated using the following formula:

0.661.

0.661.

Thus, the most likely answer to Question 1 is A with a probability of 66%. This approach allows to identify errors of specialists and, in particular, to reduce the cost of markup by 20%, while the quality of the markup has increased by 40%, which improves the quality of the classification algorithms.

The value of the input quality of work of each specialist varies with each question marked out, i.e. may increase or decrease, depending on whether the specialist noted the most likely answer or not.

This stage allows you to create a database of marked up data 128 with labels of classes of problems in several areas of law.

An important step is the preprocessing of text data using the text processing methods 133 described in the present invention, which include lemmatization, stemming, stop word removal, lowercase text, tokenization, and model 135 architecture, which includes a multilayer neural network. In addition to cleaning the text from noise and vectorization, the proposed solution uses other original methods of text processing.

 The preprocessing and classification module for questions brings the text to a view that is understandable by machine learning algorithms. Algorithms accept tensors of different ranks with numerical values as input, so the text is converted into a vector format with numerical values, where each individual value is the index of a word from the dictionary.

In a particular case, the module for preprocessing and classification of questions, or at least one part of it, in particular, the part that processes the source text, is implemented as a class of a programming language, inside which the class methods convert the source text into numerical vectors. Class methods use NLP algorithms - tokenization, lemmatization, and stemming, as described in more detail above. So, the source text of the question (232), converted into a vector of numbers by the module for preprocessing and classification of questions, goes to the first level of the trained model, which determines whether the received question relates to the target branch of law. This is a binary model that returns two outcomes - “0” or “1”. If the model returns 1, then the question is passed to the next model, which identifies the legal problem, i.e. multiclass classification is carried out.

The result of the multiclass classification model is a list of class labels sorted in descending order of probability. The first element of this list is the most probable according to model calculations. In the process of training the model, it tracks the places where it is mistaken - which categories of legal problems it can be difficult to differentiate correctly. To avoid such errors, an algorithm is used that identifies pairs of categories similar to the model and in this case allows you to clarify with the client what exactly his problem is related to. After clarifying the client, the robot, implemented by software and being part of the dialog manager 137, selects the final class label, which is used to select the topic and form the answer. Such an approach, in particular, makes it possible to increase the accuracy of the model by 15%.

When implementing the system, dialogue manager 137 is used, where the robot communicates with the client and asks context-sensitive questions, using the implemented architecture of the system with context-sensitive connections, generates an answer prepared for a specific user for the user. Dialogue manager 137 has an architecture with context-sensitive relationships and models of semantic processing (semantic analysis) of text, and dialog manager 137 is part of the module for generating responses from the knowledge database into a lexical representation corresponding to the lexical construction of the question. In a particular case, the said robot is an application capable of interacting with system modules, including simulating human actions, for example, interacting with system interfaces and database data and performing certain operations in accordance with specified data processing algorithms.

The class label defined by the model at the second level is passed as a parameter to the dialog manager 137. The dialog module 137 of the dialog manager 137 is connected to the class. The dialog manager 137 implements a tree structure through which the dialogue between the robot and the user is performed. The question at each subsequent step depends on the user's response to the previous question. The semantic analyzer extracts the necessary entities from the text of the client’s question and uses them as an answer to at least part of the questions by the response generation module. Due to this, the time of interaction between the client and the robot is reduced and the speed of answering the question increases by about 30%.

The responses of the mentioned robot presented to users by the response generation module are generated dynamically by the system - after the dialogue with the client (user), the response generation module refers to the knowledge base data 147, which is updated and complies with real legislation. So, with changes in laws, the knowledge base 147 is updated and, consequently, the responses of the robot, i.e. the response generation module generates responses based on the updated (and processed) information contained in the knowledge database 147.

In FIG. 3 shows an exemplary embodiment of the present invention.

In step 310, a tagged data database (a random questions database) 128 is generated using data from a non-tagged information (data) database 115, as described in the framework of the present invention. The Unallocated Information Database 115 contains user questions and consultant recommendations in natural language terms. Data in the database of arbitrary questions can be added from existing databases in which the questions of users asked on various Internet resources, in various applications (applications for a computer, smartphone, etc.), etc. are saved, and also saved answers of specialists (consultants) to such questions of users. Also, after processing by the system, user questions asked by users using the dialog manager 137, to which the described system generates answers using the response generation module, can be added to the markup data database 128.

Formation of the database of marked-up data 128, including marking up the database of unallocated data 115, is carried out by means of the system and specialists using the module for preprocessing questions, using at least the means for marking up user questions 124.

In step 315, questions of one area of law are selected for which the number of directions of answers (categories) is determined, as described in the framework of the present invention, and experts are presented with possible answers to questions related to the phrase or topic in order to assess the relevance of the answer to the question or phrase from question, and the answers also indicate the group to which the question belongs.

Specialists using the data processing tools provided by the system, as well as the system tools, select separate topics of questions from users' questions and group topics into several independent groups based on similarity, moreover, for topics of questions data is sorted to eliminate errors made by specialists. The means for marking user questions 124 is carried out and allows specialists to isolate separate questions from user questions and group topics into several independent groups based on similarity.

Means of (cross-) validation markup 126 allows you to make the choice for each separate topic of the question the most likely group for each of the proposed.

In step 320, the text of the user questions is vectorized as described in the framework of the present invention.

The module for classifying questions converts topics contained in user questions into a formalized form, in particular, it carries out vectorization of texts of user questions. Texts of user questions are converted into binary vectors using a dictionary of unique words and phrases (terms). So, the vectorization of texts (user questions) is carried out by presenting unique words and phrases in the form of a table containing unique terms (words and phrases) indicating the presence or absence of unique words and phrases in the text of each of the questions, and the set of table values forms a vector. For each of the phrases, the presence of unique terms in the question is determined and a tabular line is formed in which the number of cells corresponds to the number of dictionary terms, so that each cell corresponds to one of the dictionary terms, and the value entered in the cell corresponds to the presence or absence of the corresponding term in the phrase corresponding to the position or cell number, where the set of values entered in the table cells is a vector and is used to analyze the corresponding phrase, including for classifications and questions as described in the framework of the present invention.

When processing texts of questions, in a particular case, both when creating a database of arbitrary questions and when processing questions asked by users during a user’s dialogue with the dialogue manager using a computing device, questions, irrelevant characters are deleted from the text, including at least punctuation marks, HTML markup elements, etc. Also, tokenization of texts is carried out with the separation of parts of the text, for example, sentences into separate parts, in particular, symbols that are tokens, and the words are concatenated with prepositions and particles to preserve the emotional coloring of the texts. Also, the removal of irrelevant words that do not carry a semantic load, including pronouns and interrogative words and words, is carried out, as described in the framework of the present invention. Also, lemmatization of tokens, i.e. the tokens are reduced to the dictionary form and stemming is performed by finding the word base for a given source word if the dictionary form is not found.

In step 325, the questions are classified into categories based on the proximity of the vectors based on the given rules. The means for grouping marked-up data, in particular, which are part of the module for classifying questions, carry out and allow specialists to classify according to topics of questions according to formal signs of similarity. So, at least, the classification is carried out by grouping questions using the norms and rules of existing legal documents from the knowledge database, and cluster analysis, during which clusters are formed, the number of which is equal to the groups formed by experts for each branch of law, the number of which together with the vectors questions and a dictionary of unique words, processed using cluster analysis, so that they form classes for branches of law. When clustering the system, the k-means algorithm is used, sorting the questions marked by experts into clusters, where each of the questions belongs to one cluster located at the smallest distance from the question vector.

Also, in particular, in the intermediate step, questions are converted to a format suitable for automated analysis when a user's question enters the system. With the above-mentioned processing of questions (texts of user questions), irrelevant characters are removed from the text of the questions, texts are tokenized, words are concatenated with prepositions and particles to preserve the emotional coloring of the texts, irrelevant words that do not carry a semantic load are deleted, lemmatization of tokens or stamming are performed. In the particular case of implementation for each of the topics, that is, for components (for example, phrases, phrases, etc. of questions having different semantic contents) of user questions, the phrase can be converted into a format suitable for automated data processing, including analysis number. In the particular case, the conversion of questions into a format suitable for automated analysis is carried out through automated processing of questions with the formation of a vector representation for each question.

In step 335, the lexical parsing of the user's question is carried out with the definition of the lexical structure of the user's question. Lexical parsing is performed using at least one semantic parser, and parsers search for strings of characters predefined in dictionaries, where strings of characters are words or phrases combined by a common property in the source text of a user's question, so that they create classes of legal problems that are identified by the algorithm and have a common legislative justification that determines whether the issue belongs to the cluster.

In step 340, the provisions of the normative legal act contained in the knowledge database are determined that are relevant to the user's question on the vectors of normative legal acts. So, for example, a comparison of the provisions of regulatory acts from a knowledge database and a user's question can be carried out, in particular, a comparison of (pre) vectorized provisions of regulatory legal acts and vectorized questions of users is carried out. A grouping of questions by the proximity of vectors and a grouping of questions by direction of answers can also be carried out.

In step 345, the lexical structure of the answer to the user is formed, corresponding to the lexical structure of the user's question and the provision of the legal act relevant to the user's answer, and the system processes and takes into account the grammatical features of the question, subject area and object of discussion, in particular, the question.

Dialogue manager 137 using automated means of communication with the user, in particular, a software robot, conducts a dialogue in text format with the user, asking context-sensitive questions, using the implemented system architecture with context-sensitive connections, and generates an answer prepared according to the user's question. The responses are generated dynamically by the system, in particular, including during the dialogue with the user (client), the response generation module requests the information necessary for processing the user's question and for generating the response, at least from the updated knowledge base (database) 147, as well as from the database of marked-up data 128, information in which is automatically adjusted by the system and specialists, including on the basis of user questions and system responses.

At step 350, the user is presented with the lexical structure of the response in the form of a message, for example, text. Such a message can be presented to the (asking question) user on a website, in an application, etc. displayed on the screen of a computing device.

Also, in a particular case, in the process of processing user questions, including when forming an answer to a user’s question, the grouping of questions according to the proximity of the vectors and the grouping of questions according to the directions of answers and if the grouping of questions according to the proximity of the vectors and the grouping of questions by areas of answers are changing the parameters for converting user questions into a format suitable for automated analysis, and subject to Billing questions by the proximity of vectors and grouping questions by answer directions, the parameters of converting user questions into a format suitable for automated analysis are used to convert user questions into a format suitable for automated analysis, and the provisions of regulatory acts with the formation of regulatory acts vectors .

In FIG. 4 shows a simplified example of a hardware implementation of the proposed invention. A computer network can be a geographically distributed set of nodes connected by communication lines and segments for transferring data between end nodes, such as personal computers, workstations, or peripheral devices, such as printers or scanners. Many types of networks are available, from local area networks (LAN) to wide area networks (WAN). As shown in FIG. 4, an example computer network 900 may comprise a plurality of network devices, such as routers, switches, computers, and the like, connected by communication lines. For example, multiple network devices may connect one or more user devices 910 (or “client devices” 910) that may be used by a user, such as computers, smartphones, tablets, etc. Communication channels connecting various network devices may be wired lines or common media (e.g., wireless lines) where certain devices can communicate with other devices based on distance, signal strength, current operating status, location, etc. Communication channels can connect various network devices in any possible configuration. One or more servers 920 (eg, host servers, web servers, databases, etc.) may communicate with a network 900 and thus with a plurality of client devices 910. Those skilled in the art will understand that in a computer network, any number and arrangement of nodes, devices, communication lines, etc. can be used, and the illustration shown in FIG. 4 is a simplified example of a hardware implementation of a system.

In FIG. 5 shows an example of a computing system suitable for implementing elements of the proposed invention. As shown in FIG. 5, the computing system comprises a computing device 1000 that can be used as a user or server device, in memory 1040 of which the operating system 1042, computer programs 1044, and data structures 1045 are stored. The memory 1040 of device 1000 is connected via bus 1050 to power supply 1060, at least at least one processor 1020 and at least one network interface 1010, transmitting data to the network 900 (FIG. 4) and from the network 900 (FIG. 4).

In conclusion, it should be noted that the information provided in the description are examples that do not limit the scope of the present invention defined by the claims. One skilled in the art will recognize that other embodiments of the present invention may exist consistent with the spirit and scope of the present invention.

Claims (26)

1. A way to manage an automated legal advice system containing a knowledge database that contains a set of norms and rules of existing legal documents, a preprocessing and classification module for questions, which is a vector representation of the essence of the questions and a module for generating responses from the knowledge database to the lexical representation corresponding to the lexical building a question, while form a database of arbitrary questions, choose questions of one area of law, for which they determine the number of directions of answers in the form of categories of questions, vectorize the question texts into binary question vectors by presenting unique words in the form of a table containing unique words indicating the presence or absence of unique words in the text of each of the questions, and the set of table values forms a vector, questions are classified into categories based on proximity of vectors by grouping questions using the norms and rules of existing legal documents stored in the knowledge database, and cluster analysis, during which clusters are formed, the number of which is equal to the groups formed by experts for each branch of law, the number of which together with question vectors and a dictionary of unique words and phrases is processed using cluster analysis, so that classes are formed for branches of law and when a user question enters the system, the user question is lexically parsed with the lexical structure of the user question defined,  on the basis of the generated classes for the branches of law, determine the position of the regulatory legal act contained in the database of knowledge relevant to the user's question on the vectors of regulatory legal acts, form the lexical structure of the response to the user, corresponding to the lexical structure of the user's question and the provision of the normative legal act relevant to the user's answer and provide the user with a lexical response structure in the form of a text message. 2. The method according to claim 1, in which users' questions are converted into a format suitable for automated analysis by means of automated processing of questions with the formation of a vector representation for each question. 3. The method according to claim 2, in which the grouping of questions according to the proximity of the vectors and the grouping of questions according to the directions of the answers is determined and if the grouping of the questions according to the proximity of the vectors and the grouping of the questions along the directions of the answers do not change the parameters for converting user questions into a format suitable for automated analysis, and if the grouping of questions according to the proximity of the vectors and the grouping of questions by the direction of the answers are used, the conversion HAVO users in a format suitable for automated analysis, to convert the user issues in a format suitable for automated analysis, and the provisions of legal acts with the formation of the vectors of normative legal acts. 4. The method according to claim 2, in which the transformed user questions into a format suitable for automated analysis are presented in the form of query arguments and descriptions of answers presented in the form of function values. 5. The method according to claim 1, in which also: - group questions according to the proximity of vectors,  - group questions according to directions of answers. 6. The method according to claim 1, in which when processing the texts of the questions: - remove irrelevant characters from the question text, including at least punctuation marks, HTML markup elements; - perform tokenization of texts, dividing sentences into separate characters that are tokens; - perform the concatenation of words with prepositions and particles to preserve the emotional coloring of the texts; - remove irrelevant words that do not carry a semantic load, including pronouns, and interrogative words, and words that occur in more than 80% of the questions; - perform lemmatization of tokens, bringing the tokens to a dictionary form; - perform stemming by finding the word base for a given source word, if the dictionary form is not found. 7. The method according to claim 6, in which the dictionary of unique words consists of dictionary forms obtained in the process of lemmatization of tokens and / or stemming. 8. The method according to claim 1, in which for each of the phrases the presence in the question phrase of unique words and phrases is determined and a table row is formed in which the number of cells corresponds to the number of words and phrases of the dictionary, so that each of the cells corresponds to one of the dictionary words and phrases, and the value entered in the cell corresponds to the presence or absence of the corresponding word or phrase in the phrase corresponding to the position or number of the cell, where the set of values entered in the table cells is a vector set and used for analysis of the relevant phrase. 9. The method according to claim 1, in which the clustering uses the k-means algorithm, sorting the questions marked out by experts into clusters, where each of the questions refers to one cluster located at the smallest distance from the question vector. 10. The method according to claim 1, in which lexical parsing is carried out using at least one semantic parser, the parsers searching for strings of characters predefined in dictionaries, where strings of characters are words or phrases united by a common property in the source text of the user's question, so that they create classes of legal problems that are identified by the algorithm and have a common legislative justification that determines whether the question belongs to the cluster.

Images