CN118070783A - A text intelligent proofreading method, system and device based on large language model - Google Patents

Abstract

The present invention discloses a text intelligent proofreading method, system and device based on a large language model, and relates to the technical field of text detection. The method comprises: obtaining input text, segmenting the text; converting the segmentation result into a vector using a vectorization model, searching the conversion result using a vector database, and obtaining relevant factual text; extracting entities from the input text; traversing and combining entities in pairs, and constructing relevant prompts based on the input text and the factual text, then calling the large language model to judge the entity relationship, constructing triples, and then traversing and combining the triples in turn according to the rule matching method to construct multi-tuples; for the multi-tuples, calling the large language model to construct questions, and obtaining a question set; locating wrong entities and sentences based on the factual text and the question set, and calling the large language model to correct errors, and displaying the corresponding real information to the user. The present invention can solve the text quality problem well.

Description

A text intelligent proofreading method, system and device based on large language model

Technical Field

The present invention relates to the field of text detection technology, and in particular to a method, system and device for intelligent text proofreading based on a large language model.

Background technique

With the development of Internet technology and the improvement of social informatization, the amount of text data has shown explosive growth. With its rich information content and wide application fields, text data has become an important form for people to acquire knowledge, express opinions and transmit information. However, due to various reasons, such as personal language level, cognitive ability, and limitations of input devices, people may make various errors when writing and editing texts, such as word fraud, word errors, and grammatical errors. These errors not only affect the quality of the text, but also hinder people's accurate understanding and effective use of text information. Faced with a large number of erroneous texts, traditional manual proofreading methods cannot meet the growing proofreading needs, and the workload is huge, time-consuming, labor-intensive, and inefficient.

Therefore, it is of great theoretical significance and practical value to study and design a computer system that can automatically detect and correct text errors. With the development of artificial intelligence and deep learning, text error detection is a technology based on machine learning, which is mainly used for deep analysis and error correction of text. In modern society, large-scale text information processing has become a demand, and many of these text information often have various grammatical and spelling errors. This requires a powerful text error detection technology to improve the quality and accuracy of text information. In essence, text error detection can be regarded as a natural language processing (NLP) problem, which uses machine learning and deep learning techniques to predict, detect and correct errors in different application scenarios. The specific technical background includes but is not limited to machine learning (such as decision trees, random forests, logistic regression, support vector machines, etc.), deep learning (such as neural networks, long short-term memory networks-LSTM, convolutional neural networks-CNN, variational autoencoders-VAE, etc.), and various technologies of natural language processing (including but not limited to language models, word segmentation technology, word sense disambiguation, syntactic analysis, etc.).

In recent years, with the prosperity and development of deep learning and machine learning technologies, text error detection technology has also made significant progress. For example, Transformer models (such as BERT, GPT, etc.) perform well in text error detection. What's more, some models are already able to consider the context of sentences and perform fine-grained detection and correction, such as the pre-trained model BERT. However, despite the progress, text error detection still faces challenges such as diverse error types, lack of high-quality annotated data, and detection of domain-specific errors. In general, in the face of increasingly complex and massive text information, text error detection is still an issue worthy of in-depth study.

Summary of the invention

In view of the needs and shortcomings of current technological development, the present invention provides a text intelligent proofreading method, system and device based on a large language model to improve the quality of text and enhance the communication effect of information, while reducing the workload of manual proofreading and improving the efficiency of text processing.

In the first aspect, the present invention provides a method for intelligent text proofreading based on a large language model, and the technical solution adopted to solve the above technical problems is as follows:

A text intelligent proofreading method based on a large language model comprises the following steps:

Get the user input text and segment the user input text;

The segmentation results are converted into vectors using a vectorization model, and the converted results are searched using a vector database to obtain relevant factual texts;

Perform entity extraction on the user input text to obtain an entity set;

The entities in the entity set are traversed and combined in pairs, and relevant prompts are constructed based on the user input text and the factual text. The large language model is then called to determine the entity relationship and construct triples. The triples are then traversed and combined in sequence according to the rule matching method to construct multi-tuples.

For the multi-tuple, call the large language model to construct questions and obtain a set of questions;

Build prompts based on factual text and question sets, use prompts to locate incorrect entities and sentences, call a large language model to correct errors, and display the corresponding real information to users.

Optionally, use NLTK tool to segment the user input text, obtain paragraph-level text and sentence-level text in turn, and then normalize the sentence-level text.

Optionally, a named entity recognition model is used to extract entities from the user input text to obtain an entity set.

Further optionally, a named entity recognition model is used to extract entities from the user input text, and a useless entity library is used to filter useless entities to obtain an entity set.

In the second aspect, the present invention is a text intelligent proofreading system based on a large language model, and the technical solution adopted to solve the above technical problems is as follows:

A text intelligent proofreading system based on a large language model, comprising:

The text preprocessing module is used to obtain user input text and segment and normalize the user input text;

The text retrieval module is used to convert sentence-level text into vectors using a vectorization model, and search the vector database to retrieve relevant factual text;

The entity extraction module is used to extract entities from the user input text to obtain an entity set;

The entity relationship construction module is used to traverse and combine entities in the entity set in pairs, then build relevant prompts based on the user input text and factual text, and call the large language model to determine the entity relationship and build triples. Then, the triples are traversed and combined in turn according to the rule matching method to build multi-tuples;

The question generation module is used to construct questions for multiple groups by calling the large language model to obtain a question set;

The error detection and correction module is used to build prompts based on factual text and question sets, use prompts to locate erroneous entities and sentences, call the large language model to correct errors, and display the corresponding real information to users.

Optionally, the text preprocessing module involved uses the NLTK tool to segment the user input text, obtain paragraph-level text and sentence-level text in turn, and then normalizes the sentence-level text.

Optionally, the entity extraction module involved uses a named entity recognition model to perform entity extraction on the user input text to obtain an entity set.

Further optionally, the entity extraction module involved uses a named entity recognition model to perform entity extraction on the user input text and the factual text, and then uses a useless entity library to filter useless entities to obtain an entity set.

In a third aspect, a computer device of the present invention adopts the following technical solution to solve the above technical problem:

A computing device comprises a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method described in the first aspect is implemented.

The text intelligent proofreading method, system and device based on a large language model of the present invention have the following beneficial effects compared with the prior art:

(1) The present invention can solve the text quality problem well, and does not require manual collection of a large amount of data. It has the advantages of being simple and efficient, easy to maintain, and having a wide range of application scenarios;

(2) The present invention realizes an efficient and low-cost error detection and correction method, providing users with a convenient and accurate text error detection service. At the same time, it also has good scalability and can be applied to different fields and scenarios. It has the advantages of low training cost, simple maintenance, and high accuracy.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a module connection diagram of the second embodiment of the present invention.

Detailed ways

In order to make the technical solution, the technical problem solved and the technical effect of the present invention more clearly understood, the technical solution of the present invention is clearly and completely described below in conjunction with specific embodiments.

Embodiment 1:

Referring to FIG. 1 , this embodiment proposes a text intelligent proofreading method based on a large language model, which includes the following steps:

S1. Obtain user input text, use NLTK tool to segment the user input text, obtain paragraph-level text and sentence-level text in turn, and then normalize the sentence-level text.

S2. Use the vectorization model to convert the normalized sentence-level text into a vector, and use the vector database to search the conversion results to obtain relevant factual text.

S3, use the named entity recognition model NER to extract entities from the user input text, use the useless entity library to filter out useless entities, and obtain an entity set;

S4, traverse and combine entities in the entity set in pairs, and build relevant prompts based on the user input text and the factual text, then call the large language model LLM to determine the entity relationship and build triples, and then traverse and combine the triples in turn according to the rule matching method to build five-tuples;

For the five-tuple, call the large language model LLM to construct questions and obtain a question set;

Prompts are built based on factual text and question sets, and incorrect entities and sentences are located using the prompts. The large language model (LLM) is called to correct errors and the corresponding real information is displayed to users.

Embodiment 2:

Referring to FIG. 1 , this embodiment proposes a text intelligent proofreading system based on a large language model, which includes:

The text preprocessing module is used to obtain user input text, segment the user input text using the NLTK tool, obtain paragraph-level text and sentence-level text in turn, and then normalize the sentence-level text;

The text retrieval module is used to convert the normalized sentence-level text into vectors using a vectorization model, and search the vector database to retrieve relevant factual texts;

The entity extraction module is used to extract entities from user input text using the named entity recognition model NER, and to filter useless entities using the useless entity library to obtain an entity set;

The entity relationship construction module is used to traverse and combine entities in the entity set in pairs, then build relevant prompts based on the user input text and factual text, and call the large language model LLM to determine the entity relationship and build triples. Then, the triples are traversed and combined in sequence according to the rule matching method to build five-tuples;

The question generation module is used to construct questions for the five-tuple by calling the large language model LLM to obtain a question set;

The error detection and correction module is used to build prompts based on factual text and question sets, use prompts to locate erroneous entities and sentences, call the large language model LLM to correct errors, and display the corresponding real information to users.

In a third aspect, an embodiment of the present specification provides a computing device, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method in the first embodiment is implemented.

It is understandable that the explanation, specific implementation, beneficial effects, examples, etc. of the relevant contents in the computing device provided in the embodiment of the present invention can be found in the corresponding parts of the method provided in the first aspect, and will not be repeated here.

Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

Those skilled in the art should be aware that in one or more of the above examples, the functions described in the present invention can be implemented by hardware, software, widgets, or any combination thereof. When implemented by software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solution of the present invention should be included in the scope of protection of the present invention.

Claims (9)

1. A text intelligent correction method based on a large language model is characterized by comprising the following steps:

acquiring a user input text, and segmenting the user input text;

converting the segmentation result into a vector by using a vectorization model, and searching the conversion result by using a vector database to obtain a related fact text;

Entity extraction is carried out on the text input by the user, so that an entity set is obtained;

Performing two-by-two traversal combination on the entities in the entity set, constructing related prompts based on the text input by the user and the fact text, then calling a large language model to judge entity relation, constructing triples, and then sequentially traversing the triples according to a rule matching mode to construct multiple groups;

aiming at the multiple groups, calling a large language model to construct a problem to obtain a problem set;

And constructing prompts based on the fact text and the problem set, positioning error entities and sentences by utilizing the prompts, calling a large language model to correct errors, and displaying corresponding real information to a user.

2. The intelligent text proofreading method based on a large language model according to claim 1, wherein a NLTK tool is used for segmenting user input text, paragraph level text and sentence level text are obtained in sequence, and then normalization processing is carried out on the sentence level text.

3. The intelligent text proofreading method based on a large language model according to claim 1, wherein a named entity recognition model is used to perform entity extraction on a text input by a user to obtain an entity set.

4. A method for intelligent proofreading of text based on a large language model according to claim 3, wherein named entity recognition models are used to perform entity extraction on the text input by the user, and useless entities are filtered by using a useless entity library to obtain an entity set.

5. A large language model based text intelligent collation system comprising:

the text preprocessing module is used for acquiring a text input by a user and carrying out segmentation and standardization processing on the text input by the user;

The text retrieval module is used for converting sentence-level texts into vectors by using the vectorization model, and retrieving a vector database to retrieve related fact texts;

The entity extraction module is used for extracting the entity of the text input by the user to obtain an entity set;

The entity relation construction module is used for carrying out pairwise traversal combination on the entities in the entity set, then constructing related prompts based on the user input text and the fact text, calling a large language model to judge the entity relation, constructing triples, and then sequentially traversing the triples according to a rule matching mode to construct a multi-tuple;

The problem generation module is used for calling a large language model to construct a problem aiming at the multiple groups to obtain a problem set;

and the error detection and correction module is used for constructing prompts based on the fact text and the problem set, positioning error entities and sentences by utilizing the prompts, calling the large language model to correct errors, and displaying corresponding real information to a user.

6. The intelligent text collation system based on large language model according to claim 5, wherein the text preprocessing module uses NLTK tool to cut user input text, sequentially obtain paragraph level text, sentence level text, and then normalize sentence level text.

7. The intelligent text collation system based on large language model according to claim 5, wherein the entity extraction module uses named entity recognition model to perform entity extraction on the text input by the user to obtain entity set.

8. The intelligent text collation system based on large language model according to claim 7, wherein the entity extraction module uses named entity recognition model to extract the entities of user input text and fact text, and then uses useless entity library to filter useless entities to obtain entity set.

9. A computing device comprising a memory having executable code stored therein and a processor that, when executing the executable code, performs the method of any of claims 1-4.