CN117171328A - Text question-answering processing method and device, electronic equipment and storage medium - Google Patents
Text question-answering processing method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN117171328A CN117171328A CN202311236897.3A CN202311236897A CN117171328A CN 117171328 A CN117171328 A CN 117171328A CN 202311236897 A CN202311236897 A CN 202311236897A CN 117171328 A CN117171328 A CN 117171328A
- Authority
- CN
- China
- Prior art keywords
- text
- long
- question
- information
- texts
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003860 storage Methods 0.000 title claims abstract description 25
- 238000003672 processing method Methods 0.000 title abstract description 20
- 239000013598 vector Substances 0.000 claims abstract description 62
- 238000012545 processing Methods 0.000 claims abstract description 52
- 238000000034 method Methods 0.000 claims abstract description 42
- 238000003058 natural language processing Methods 0.000 claims description 20
- 230000011218 segmentation Effects 0.000 claims description 18
- 238000004590 computer program Methods 0.000 claims description 12
- 238000004364 calculation method Methods 0.000 claims description 6
- 230000004044 response Effects 0.000 claims description 5
- 238000012163 sequencing technique Methods 0.000 claims description 4
- 238000005520 cutting process Methods 0.000 claims description 3
- 238000005516 engineering process Methods 0.000 abstract description 6
- 238000010801 machine learning Methods 0.000 description 9
- 238000010586 diagram Methods 0.000 description 8
- 230000008569 process Effects 0.000 description 6
- 238000013473 artificial intelligence Methods 0.000 description 5
- 230000006870 function Effects 0.000 description 5
- 230000003287 optical effect Effects 0.000 description 5
- 238000004891 communication Methods 0.000 description 3
- 238000009826 distribution Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 238000013459 approach Methods 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000018109 developmental process Effects 0.000 description 2
- 239000000284 extract Substances 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 239000004973 liquid crystal related substance Substances 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 238000005096 rolling process Methods 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 238000012549 training Methods 0.000 description 2
- 238000013519 translation Methods 0.000 description 2
- 230000006978 adaptation Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000013475 authorization Methods 0.000 description 1
- 238000013136 deep learning model Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 229920001971 elastomer Polymers 0.000 description 1
- 239000000806 elastomer Substances 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 239000012634 fragment Substances 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application provides a text question-answering processing method, a text question-answering processing device, electronic equipment and a storage medium, wherein the method comprises the following steps: responding to a problem solving request of a user for a long text, and carrying out vectorization processing on a problem of the user to obtain a problem vector; and carrying out similarity retrieval on text contents of a plurality of long texts according to the problem vectors to determine at least one target text paragraph related to the problem in the plurality of long texts, wherein the text vector of each long text is determined according to element information, text paragraph and global information of the long text, and the global information of each long text at least comprises: the combined information of the text paragraph of the long text and the summary information of the long text; and determining an answer corresponding to the question according to the target text paragraph related to the question. A knowledge base question-answering processing scheme suitable for long texts is provided to solve the problem that the related technology cannot give accurate answers due to lack of global information when processing the questions related to the long texts.
Description
Technical Field
The present application relates to artificial intelligence technologies, and in particular, to a text question-answering processing method, apparatus, electronic device, and storage medium.
Background
In recent years, with the development of deep learning and natural language processing techniques, large models have been able to achieve satisfactory performance over many tasks, including text generation, question-and-answer machine translation, and the like. These large models are typically pre-trained on large amounts of text data, learn the semantic and grammatical rules of the text, and then fine-tune on specific tasks to suit specific task requirements.
However, large models tend to see only a portion of the text when processing long text due to input length limitations, which is particularly evident when processing vertical domain knowledge base questions and answers. For example, when a user asks a question, especially a question that involves global, a large model may not give an accurate answer because it cannot see global information.
Disclosure of Invention
The application provides a text question-answering processing method, a device, electronic equipment and a storage medium, which are used for solving the problem that the related technology cannot give an accurate answer due to lack of global information when processing the problem related to a long text, and realizing the technical effect of providing a knowledge base question-answering processing scheme suitable for the long text and improving the accuracy of the answer.
In one aspect, the present application provides a text question-answering processing method, where the method includes:
responding to a problem solving request of a user for a long text, and carrying out vectorization processing on the problem of the user to obtain a problem vector;
and searching the similarity of the text vectors of each of the long texts according to the problem vector to determine at least one target text paragraph related to the problem in the long texts, wherein the text vector of each long text is determined according to element information, text paragraph and global information of the long text, and the global information of each long text at least comprises: the combination information of the text paragraph of the long text and the summary information of the long text;
and determining an answer corresponding to the question according to the target text paragraph related to the question.
An alternative embodiment, determining an answer corresponding to the question according to a target text paragraph related to the question, includes:
according to the target text paragraphs related to the problems, correspondingly generating new question-answer guide sentences aiming at the problems in a natural language processing model;
and determining an answer corresponding to the question based on the new question-answer guide statement by adopting the natural language processing model.
In an alternative embodiment, if there are a plurality of text paragraphs related to the question, determining at least one target text paragraph related to the question from among a plurality of long texts includes:
sequencing a plurality of text paragraphs related to the problems to obtain sequencing results;
and selecting at least one target text paragraph related to the problem according to the sorting result.
In an alternative embodiment, before the user's question is vectorized in response to the user's question answering request for the long text, the method further includes:
acquiring summary information and element information corresponding to each of a plurality of long texts;
the method comprises the steps of respectively carrying out segmentation treatment on a plurality of long texts to obtain a plurality of text paragraphs corresponding to the long texts;
combining each text paragraph in the long texts with summary information of the long texts to obtain global information corresponding to each long text;
and calculating the global information, the element information and the text paragraphs corresponding to the long texts to obtain the text vectors of the long texts.
An alternative embodiment, obtaining summary information and element information of long text includes:
extracting summary information in the limited length of the long text by adopting an initial question-answer guide sentence of a natural language processing model;
extracting at least one of the following information of the long text: title, author, distribution time, text length, and topic classification to obtain the element information.
An optional implementation manner, the processing of splitting the long text to obtain a plurality of text paragraphs corresponding to the long text respectively includes:
determining the text segmentation length according to the processing calculation force of the natural language processing model;
and according to the text segmentation length, respectively carrying out segmentation processing on the long texts to obtain a plurality of text paragraphs corresponding to the long texts.
In another aspect, the present application provides a text question-answering processing apparatus, including:
the vectorization processing module is used for responding to a problem solving request of a user for a long text, vectorizing the problem of the user and obtaining a problem vector;
the retrieval module is used for carrying out similarity retrieval on each text vector of a plurality of long texts according to the problem vector so as to determine at least one target text paragraph related to the problem in the long texts, wherein the text vector is determined according to element information, text paragraphs and global information corresponding to the long texts, and the global information is obtained by combining the text paragraphs and summary information;
And the determining module is used for determining an answer corresponding to the question according to the target text paragraph related to the question.
In another aspect, the present application provides an electronic device, including: a processor and a memory connected with the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method as described in any one of the above.
In another aspect, the application provides a computer-readable storage medium having stored therein computer-executable instructions which, when executed by a processor, are adapted to carry out a method as any one of the above.
In another aspect, the application provides a computer program product comprising a computer program which, when executed by a processor, implements any of the methods described above.
According to the text question-answering processing method, the text question-answering processing device, the electronic equipment and the storage medium, the problem of the user is vectorized by responding to the problem-answering request of the user for the long text, so that a problem vector is obtained; and carrying out similarity retrieval on text contents of a plurality of long texts according to the problem vectors to determine at least one target text paragraph related to the problem in the plurality of long texts, wherein the text vector of each long text is determined according to element information, text paragraph and global information of the long text, and the global information of each long text at least comprises: the combined information of the text paragraph of the long text and the summary information of the long text; and determining an answer corresponding to the question according to the target text paragraph related to the question.
By adopting the scheme of the embodiment of the application, the problem that the related technology cannot give an accurate answer due to lack of global information when processing the problem related to the long text can be solved, and the technical effect of improving the accuracy of the answer is achieved by providing the knowledge base question-answer processing scheme suitable for the long text.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and together with the description, serve to explain the principles of the application.
FIG. 1 is a schematic flow chart of a text question-answering processing method provided by an embodiment of the application;
FIG. 2 is a schematic flow chart of an alternative text question-answering processing method according to an embodiment of the present application;
FIG. 3 is a schematic flow chart of an alternative text question-answering processing method according to an embodiment of the present application;
FIG. 4 is a schematic flow chart of an alternative text question-answering processing method according to an embodiment of the present application;
FIG. 5 is a schematic flow chart of an alternative text question-answering processing method according to an embodiment of the present application;
fig. 6 is a block diagram of a text question-answering processing device according to an embodiment of the present application;
fig. 7 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Specific embodiments of the present application have been shown by way of the above drawings and will be described in more detail below. The drawings and the written description are not intended to limit the scope of the inventive concepts in any way, but rather to illustrate the inventive concepts to those skilled in the art by reference to the specific embodiments.
Detailed Description
Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers in different drawings refer to the same or similar elements, unless otherwise indicated. The implementations described in the following exemplary examples do not represent all implementations consistent with the application. Rather, they are merely examples of apparatus and methods consistent with aspects of the application as detailed in the accompanying claims.
First, the terms involved in the present application will be explained:
large Model (Large Model): in the field of machine learning and artificial intelligence, large models are often referred to as models with a very large number of parameters. These models typically require a significant amount of computing resources and data to train, but they also provide better performance and more accurate predictions.
Question and answer guide statement (promt): in natural language processing, prompt is an input statement that is used to direct a model to generate a particular type of output. For example, in a question-answering system, the question is a prompt.
Embedding vector (Embedding): in machine learning, ebadd is a method of converting discrete variables (such as words or questions) into continuous vectors. Such vectors are typically capable of capturing semantic information of the variables.
Vector retrieval engine (Vector Search Engine): a vector search engine is a system that can perform search based on the similarity between vectors. In the present invention, it is used to retrieve the text passage most relevant to the question.
Open source search engine (elastomer): a distributed full text search engine is provided having an HTTP web interface and documents in the modeless JSON format. The elastic search is designed to be used in cloud computing, can achieve real-time searching, is stable, reliable and quick, and is convenient to install and use.
GPT, collectively referred to as generating Pre-Trained Transformer (Generative Pre-training transducer model), is an Internet-based, data-capable, training, text-generated deep learning model.
In recent years, with the development of deep learning and natural language processing techniques, large models such as GPT-3 and the like have been able to achieve satisfactory performance over many tasks, including text generation, question-and-answer machine translation and the like. These large models are typically pre-trained on large amounts of text data, learn the semantic and grammatical rules of the text, and then fine-tune on specific tasks to suit specific task requirements.
However, large models tend to see only a portion of the text when processing long text due to input length limitations, which is particularly evident when processing vertical domain knowledge base questions and answers. For example, when a user asks questions related to a global situation, a large model may not give an accurate answer because it cannot see the global information.
For example, currently, there are solutions mainly by splitting a knowledge base into a number of small paragraphs, then searching for the corresponding paragraphs using a search technique, and then letting the large model question and answer according to the corresponding paragraphs. However, this approach has the disadvantage that, since the large model can only see a partial paragraph, an accurate answer may not be given when answering a question that involves global.
The main disadvantages of the prior art are:
1. local viewing angle: because of the input length limitations of large models, existing solutions typically require splitting a long text into multiple paragraphs, and then letting the large model question and answer based on these paragraphs. The problem with this approach is that the large model only sees part of the paragraphs when answering the question, and therefore may not answer questions that involve global. For example, when a user asks "how many chapters are the article? What is the summary of the article? "when a large model may not give an accurate answer because it cannot see global information.
2. Information loss: some important information may be lost in splitting a long text into multiple paragraphs. For example, structural information (e.g., chapter titles, sub-titles, etc.), contextual information (e.g., relationships between previous and subsequent paragraphs, etc.) of an article, etc., which may not be available to the large model after splitting.
Aiming at the defects, the embodiment of the application aims to provide a novel text question-answering processing method, which can enable a large model to see global information and give accurate answers within the length limit when processing long texts. Specifically, the embodiment of the application extracts the summary information and the element information of the long text and cuts the text paragraph, thereby effectively solving the problem of large model input length limitation. In the question-answering flow, the vector retrieval engine is used for carrying out similarity retrieval on the vector of the question, then the most relevant text paragraph is obtained, and then the question-answering is carried out through the large model.
The application provides a text question-answering processing method, which aims to solve the technical problems in the prior art. The following describes the technical scheme of the present application and how the technical scheme of the present application solves the above technical problems in detail with specific embodiments. The following embodiments may be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments. Embodiments of the present application will be described below with reference to the accompanying drawings.
Fig. 1 is a flow chart of a text question-answering processing method provided by an embodiment of the present application, as shown in fig. 1, the method includes:
s101, responding to a problem solving request of a user for a long text, and carrying out vectorization processing on the problem of the user to obtain a problem vector;
s102, performing similarity retrieval on respective text vectors of a plurality of long texts according to the problem vectors to determine at least one target text paragraph related to the problem in the long texts, wherein the text vector of each long text is determined according to element information, text paragraph and global information of the long text, and the global information of each long text at least comprises: the combination information of the text paragraph of the long text and the summary information of the long text;
S103, determining answers corresponding to the questions according to the target text paragraphs related to the questions.
Optionally, the embodiment of the application can be applied to the fields of machine learning and artificial intelligence, for example, the application scenario of realizing question-answer processing of long text by using a natural language processing model (such as a large model) and a vector retrieval engine technology.
Optionally, the text vector of each long text is determined according to element information, text paragraph and global information of the long text, and the global information of each long text at least includes: and combining the text paragraphs of the long text with the summary information of the long text.
Alternatively, the global information corresponding to each of the long texts may be specifically understood as global information and overall information obtained by combining text paragraphs of each of the long texts and summary information of the long texts.
Firstly, a problem posed by a user is subjected to problem vectorization processing, for example, assuming that the problem posed by the user is q, vectorizing the q by using a large model, and the purpose of this step is to convert the problem of the user into a problem vector, and the vector can usually capture semantic information of variables so as to facilitate subsequent similarity calculation.
Secondly, in the embodiment of the application, according to the problem vector, in the text retrieval library, similarity retrieval is performed on text contents of a plurality of long texts, namely, the similarity retrieval can be performed on the problem vector ebedding of the problem q by using a vector retrieval engine so as to determine text paragraphs, such as 5 text paragraphs, which are most relevant to the problem proposed by the user.
Finally, in the question answering link of the embodiment of the application, the semantic understanding capability of the large model can be utilized, and the answer corresponding to the question can be generated according to the searched text paragraph.
In an alternative embodiment, the following is a specific interaction example:
for example, suppose a user presents a problem on the front-end interface of a front-end device: what is machine learning? ".
First, the front-end device, after receiving the user's question, sends the question to the back-end server for processing.
Secondly, after receiving the problem, the back-end server first uses a large model to calculate a problem vector corresponding to the problem, such as embedding vector embedding.
Further, the backend server sends the computed email to a distributed full text search engine, for example, an open source search engine, and performs similarity search on the email to obtain the first M (top M) text paragraphs returned by the email and most relevant to the problem. In embodiments of the present application, it may be assumed that the text paragraphs returned are definitions, histories, and applications related to machine learning.
Still further, the backend server reconstructs a new question-answer guide sentence prompt from the retrieved text paragraphs, for example: the following paragraphs come from some text about machine learning: 'machine learning is an artificial intelligence method that enables the system to automatically learn and refine experiences … …'. Please answer questions based on these information: what is machine learning? '".
Thereafter, the back-end server generates an answer to the question based on the new question-answer guide statement using the large model, e.g., the answer to the question may be, but is not limited to,: "machine learning is an artificial intelligence method that enables the system to automatically learn and refine experiences … …". And, the back-end server returns the generated answer to the front-end.
Finally, the front-end interface displays the answer to the user.
According to the embodiment of the application, the segmented paragraphs of each long text and the summary information of the long text are combined into one piece of information, so that the text paragraphs and the global information of the long text are obtained, and the large model can simultaneously consider the global and the global information of the long text when processing each paragraph.
According to the text question-answering processing method, the text question-answering processing device, the electronic equipment and the storage medium, the problem of the user is vectorized by responding to the problem-answering request of the user for the long text, so that a problem vector is obtained; and searching the text content of the long texts according to the problem vectors to determine at least one target text paragraph related to the problem in the long texts, wherein the text vector of each long text is determined according to the element information, the text paragraph and the global information of the long text, and the global information of each long text at least comprises: the combination information of the text paragraph of the long text and the summary information of the long text; and determining an answer corresponding to the question according to the target text paragraph related to the question.
By adopting the scheme of the embodiment of the application, the problem that the related technology cannot give an accurate answer due to lack of global information when processing the problem related to the long text can be solved, and the technical effect of improving the accuracy of the answer is achieved by providing the knowledge base question-answer processing scheme suitable for the long text.
Fig. 2 is a schematic flow chart of an alternative text question-answering processing method according to an embodiment of the present application, as shown in fig. 2, for determining an answer corresponding to the question according to a target text paragraph related to the question, including:
s201, correspondingly generating a new question-answer guide sentence aiming at the problem in a natural language processing model according to a target text paragraph related to the problem;
s202, determining answers corresponding to the questions based on the new question-answer guidance sentences by adopting the natural language processing model.
In one example, for example, a question and answer guide sentence prompt of a large model may be reconstructed according to the retrieved 5 or 7-item label text paragraphs, for example: the following paragraphs come from some long text fragments, which may not be relevant to q-correlation. Please get the answer to the q question from the following M texts: paragraph @. If there is no answer within the paragraph, the answer is not known. Further, the large model may determine answers to questions based on the new question-answer guidance sentences.
According to the embodiment of the application, the similarity retrieval is carried out on the empdding of the question by using the vector retrieval engine, then a new question-answering guide sentence prompt is reconstructed according to the retrieved text paragraph, and the answer corresponding to the question of the user is determined, so that the text paragraph most relevant to the question can be considered by the large model when the question is answered, and the accuracy of the question answer is further improved.
Fig. 3 is a schematic flow chart of an alternative text question-answering processing method according to an embodiment of the present application, where, as shown in fig. 3, if there are a plurality of text paragraphs related to the question, determining at least one target text paragraph related to the question in a plurality of long texts includes:
s301, sorting a plurality of text paragraphs related to the problems to obtain a sorting result;
s302, selecting at least one target text paragraph related to the problem according to the sorting result.
Optionally, in the case that it is determined that the text paragraphs related to the problem are multiple, the text paragraphs related to the problem may be ranked, so as to obtain a ranking result; then, at least one (one, two or more) question-related target text passage is selected according to the sorting result, for example, from high to low according to the relevance.
Through the embodiment, when the answer corresponding to the question is determined by the large model, the text paragraph most relevant to the question can be considered, and the accuracy of the answer of the question is further improved.
Fig. 4 is a schematic flow chart of an alternative text question-answering processing method provided by the embodiment of the present application, as shown in fig. 4, before a user's question is vectorized to obtain a question vector in response to a question-answering request for a long text, the method further includes:
s401, obtaining summary information and element information corresponding to each of the long texts.
S402, respectively carrying out segmentation processing on the long texts to obtain a plurality of text paragraphs corresponding to the long texts.
S403, combining each text paragraph in the long texts with the summary information of the long texts to obtain global information corresponding to each long text.
S404, calculating global information, element information and a plurality of text paragraphs corresponding to the long texts to obtain respective text vectors of the long texts.
According to the embodiment of the application, the long text is segmented with the rolling window, and then the segmented paragraphs of the long text and the summary information of the long text are combined into one piece of information, so that the global information of the text paragraphs and the long text is obtained, and the global and overall information of the long text can be considered simultaneously when the large model processes each paragraph.
An alternative embodiment, obtaining summary information and element information of long text includes:
and extracting summary information within the limited length of the long text by adopting an initial question-answer guide sentence of a natural language processing model.
Extracting at least one of the following information of the long text: title, author, distribution time, text length, and topic classification to obtain the element information.
In the embodiment of the application, before the problem of the user is vectorized in response to the problem solving request of the user for the long text to obtain the problem vector, a data processing flow is executed: specifically, the summary information of each long text may be extracted, for example, each long text d_i may be traversed, and the summary information of the long text is extracted using the initial question-answer guide sentence prompt of the large model, for example, "please help me summarize the above text, which requires to contain the information that can be contained as thoroughly as possible, and the length is limited to be within" K ", and is defined as s_i. Where d_i represents each long text, s_i represents the corresponding summary information, and K is a configurable (customizable) length parameter for limiting the length of the summary information of the long text.
In addition, the embodiment of the application can also extract the element information of each long text, such as title, author, release time, text length, theme classification and the like.
By adopting the embodiment of the application, the initial question-answer guide statement prompt of the large model is used for traversing each long text, and the summary information and the element information corresponding to each long text are extracted. In this way, the main information of the long text can be summarized and compressed into a short summary information, so that the large model can process more long text.
Fig. 5 is a schematic flow chart of an alternative text question-answering processing method according to an embodiment of the present application, where, as shown in fig. 5, a plurality of long texts are respectively segmented to obtain a plurality of text paragraphs corresponding to the long texts, and global information obtained by combining each text paragraph with the summary information includes:
s501, determining the text segmentation length according to the processing calculation force of the natural language processing model.
S502, according to the text segmentation length, respectively carrying out segmentation processing on a plurality of long texts to obtain a plurality of text paragraphs corresponding to the long texts.
Optionally, in the embodiment of the present application, the long text is split with a rolling window, and the specific splitting length can be defined according to the capability and calculation force of the large model.
In an alternative embodiment, at least more than one text paragraph p_ij is split for each long text d_i, where p_ij represents the j-th text paragraph of the i-th long text. And combining each text paragraph obtained by cutting each long text with the summary information to form a piece of global information. For example, the global information may be, but is not limited to,: this is a piece of long text from the xx author that is published at xx time (and other element information): p_ij (paragraph), the long text mainly describes: s_i (summary information).
Then, the embodiment of the application cuts the global information and the element information of the long text to obtain text paragraphs, and the text paragraphs are used as text contents to calculate the vector empedding completely and then stored in a vector database for retrieval, namely a text retrieval library. By adopting the embodiment of the application, the vector embedding can be used for calculating the similarity between texts.
Based on the above embodiments, it can be seen that the advantages of the embodiments of the present application can be mainly, but not limited to, the following aspects:
1. The ability of large models to handle long text is improved: by summarizing and segmenting the long text, the large model can process each paragraph while considering global information of the long text, so that the problem that the input length of the large model is limited is solved, and the capability of the large model for processing the long text is improved.
2. The accuracy of question answering is improved: similarity retrieval is carried out on the empdding of the questions by using a vector retrieval engine, and then the template is reconstructed according to the retrieved text paragraphs, so that the text paragraphs most relevant to the questions can be considered when the large model answers the questions, and the accuracy of the questions is improved.
3. The cost is reduced: compared with a newly trained global information extraction model or a fine tuning large model, the scheme of the application does not need to retrain the large model again, thereby greatly reducing the cost.
In conclusion, the method effectively solves the problem of processing long text by a large model, improves the accuracy of question answering, reduces the cost and has high practical value.
It should be noted that, the user information (including but not limited to user equipment information, user personal information, etc.) and the data (including but not limited to data for analysis, stored data, presented data, etc.) related to the present application are information and data authorized by the user or fully authorized by each party, and the collection, use and processing of the related data need to comply with the related laws and regulations and standards of the related country and region, and provide corresponding operation entries for the user to select authorization or rejection.
According to one or more embodiments of the present application, there is provided a text question-answering processing apparatus, and fig. 6 is a block diagram of the structure of the text question-answering processing apparatus provided by the embodiment of the present application, as shown in fig. 6, the apparatus includes:
the vectorization processing module 601 is configured to perform vectorization processing on a problem of a user in response to a problem solving request of the user for a long text, so as to obtain a problem vector;
a retrieving module 602, configured to perform similarity retrieval on respective text vectors of the long texts according to the question vector to determine at least one target text passage related to the question in the long texts, where the text vector of each long text is determined according to element information, text passage and global information of the long text, and the global information of each long text at least includes: the combination information of the text paragraph of the long text and the summary information of the long text;
a determining module 603, configured to determine an answer corresponding to the question according to the target text paragraph related to the question.
According to one or more embodiments of the present application, the determining module includes:
a generating unit, configured to correspondingly generate a new question-answer guide sentence for the question in the natural language processing model according to a target text paragraph related to the question;
And the first determining unit is used for determining an answer corresponding to the question based on the new question-answer guide statement by adopting the natural language processing model.
In an alternative embodiment, if there are a plurality of text paragraphs related to the above problem, the search module includes:
the ordering unit is used for ordering a plurality of text paragraphs related to the problems to obtain an ordering result;
and the selecting unit is used for selecting at least one target text paragraph related to the problem according to the sorting result.
In an alternative embodiment, the apparatus further comprises:
the information acquisition module is used for acquiring summary information and element information corresponding to each of the long texts;
the segmentation module is used for respectively carrying out segmentation processing on the long texts to obtain a plurality of text paragraphs corresponding to the long texts;
the combination module is used for combining each text paragraph in the long texts with the summary information of the long texts to obtain global information corresponding to each long text;
and the storage module is used for calculating the global information, the element information and the text paragraphs corresponding to the long texts to obtain the text vectors of the long texts.
An alternative embodiment, an information acquisition module, comprising:
a first extraction unit for extracting summary information within a limited length of the long text by using an initial question-answer guide sentence of a natural language processing model;
a second extracting unit for extracting at least one of the following information of the long text: title, author, distribution time, text length, and topic classification to obtain the element information.
An alternative embodiment, a segmentation module, comprising:
the second determining unit is used for determining the text segmentation length according to the processing calculation force of the natural language processing model;
and the segmentation unit is used for respectively carrying out segmentation processing on the long texts according to the text segmentation length to obtain a plurality of text paragraphs corresponding to the long texts.
In an exemplary embodiment, an embodiment of the present application further provides an electronic device, including: a processor and a memory connected with the processor;
the memory stores computer-executable instructions;
the processor executes the computer-executable instructions stored in the memory to implement the method as described in any one of the above.
In an exemplary embodiment, an embodiment of the application further provides a computer-readable storage medium having stored therein computer-executable instructions that, when executed by a processor, are configured to implement a method as any one of the above.
In an exemplary embodiment, the application also provides a computer program product comprising a computer program which, when executed by a processor, implements any of the methods described above.
In order to achieve the above embodiment, the embodiment of the present application further provides an electronic device. Referring to fig. 7, there is shown a schematic structural diagram of an electronic device 700 suitable for use in implementing an embodiment of the present application, where the electronic device 700 may be a terminal device or a server. The terminal device may include, but is not limited to, a mobile terminal such as a mobile phone, a notebook computer, a digital broadcast receiver, a messaging device, a game console, a medical device, an exercise device, a personal digital assistant (Personal Digital Assistant, PDA for short), a tablet computer (Portable Android Device, PAD for short), a portable multimedia player (Portable Media Player, PMP for short), an in-vehicle terminal (e.g., in-vehicle navigation terminal), and the like, and a fixed terminal such as a digital TV, a desktop computer, and the like. The electronic device shown in fig. 7 is only an example and should not be construed as limiting the functionality and scope of use of the embodiments of the application.
As shown in fig. 7, the electronic apparatus 700 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 701 that may perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 702 or a program loaded from a storage device 708 into a random access Memory (Random Access Memory, RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 are also stored. The processing device 701, the ROM 702, and the RAM 703 are connected to each other through a bus 704. An input/output (I/O) interface 705 is also connected to bus 704.
In general, the following devices may be connected to the I/O interface 705: input devices 706 including, for example, a touch screen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, and the like; an output device 707 including, for example, a liquid crystal display (Liquid Crystal Display, LCD for short), a speaker, a vibrator, and the like; storage 708 including, for example, magnetic tape, hard disk, etc.; and a communication device 709. The communication means 709 may allow the electronic device 700 to communicate wirelessly or by wire with other devices to exchange data. While fig. 7 shows an electronic device 700 having various means, it is to be understood that not all of the illustrated means are required to be implemented or provided. More or fewer devices may be implemented or provided instead.
In particular, according to embodiments of the present application, the processes described above with reference to flowcharts may be implemented as computer software programs. For example, embodiments of the present application include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method shown in the flowcharts. In such an embodiment, the computer program may be downloaded and installed from a network via communication device 709, or installed from storage 708, or installed from ROM 702. When being executed by the processing means 701, performs the above-described functions defined in the method of the embodiment of the present application.
The computer readable medium of the present application may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor apparatus or device, or a combination of any of the foregoing. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution apparatus or device. In the present application, however, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with the computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution apparatus or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, fiber optic cables, RF (radio frequency), and the like, or any suitable combination of the foregoing.
The computer readable medium may be contained in the electronic device; or may exist alone without being incorporated into the electronic device.
The computer-readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to perform the methods shown in the above-described embodiments.
Computer program code for carrying out operations of the present application may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, smalltalk, C ++ and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a local area network (Local Area Network, LAN for short) or a wide area network (Wide Area Network, WAN for short), or it may be connected to an external computer (e.g., connected via the internet using an internet service provider).
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units involved in the embodiments of the present application may be implemented in software or in hardware. The name of the unit does not in any way constitute a limitation of the unit itself, for example the first acquisition unit may also be described as "unit acquiring at least two internet protocol addresses".
The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: a Field Programmable Gate Array (FPGA), an Application Specific Integrated Circuit (ASIC), an Application Specific Standard Product (ASSP), a system on a chip (SOC), a Complex Programmable Logic Device (CPLD), and the like.
In the context of the present application, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution apparatus or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor devices or apparatuses, or any suitable combination of the above. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. This application is intended to cover any variations, uses, or adaptations of the application following, in general, the principles of the application and including such departures from the present disclosure as come within known or customary practice within the art to which the application pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the application being indicated by the following claims.
It is to be understood that the application is not limited to the precise arrangements and instrumentalities shown in the drawings, which have been described above, and that various modifications and changes may be effected without departing from the scope thereof. The scope of the application is limited only by the appended claims.
Claims (10)
1. A method for processing a text question and answer, the method comprising:
responding to a problem solving request of a user for a long text, and carrying out vectorization processing on a problem of the user to obtain a problem vector;
and searching the similarity of the text vectors of each long text according to the problem vector to determine at least one target text paragraph related to the problem in the long text, wherein the text vector of each long text is determined according to element information, text paragraph and global information of the long text, and the global information of each long text at least comprises: the combined information of the text paragraph of the long text and the summary information of the long text;
And determining an answer corresponding to the question according to the target text paragraph related to the question.
2. The method of claim 1, wherein determining an answer to the question based on the passage of the target text associated with the question comprises:
according to the target text paragraphs related to the questions, correspondingly generating new question-answer guide sentences aiming at the questions in a natural language processing model;
and determining an answer corresponding to the question based on the new question-answer guide statement by adopting the natural language processing model.
3. The method of claim 1, wherein if there are a plurality of text paragraphs associated with the question, determining a target text paragraph associated with the question for at least one of the plurality of long texts comprises:
sequencing a plurality of text paragraphs related to the problems to obtain sequencing results;
and selecting at least one target text paragraph related to the problem according to the sorting result.
4. A method according to any one of claims 1 to 3, wherein before vectorising a question of a user in response to a user's question-answer request for long text, the method further comprises:
Acquiring summary information and element information corresponding to each of a plurality of long texts;
the method comprises the steps of respectively carrying out segmentation treatment on a plurality of long texts to obtain a plurality of text paragraphs corresponding to the long texts;
combining each text paragraph in the long texts with summary information of the long texts to obtain global information corresponding to each long text;
and calculating global information, element information and a plurality of text paragraphs corresponding to each of the long texts to obtain respective text vectors of the long texts.
5. The method of claim 4, wherein obtaining summary information and element information of the long text comprises:
extracting summary information within the limited length of the long text by adopting an initial question-answer guide sentence of a natural language processing model;
extracting at least one of the following information of the long text: title, author, release time, text length, topic classification to obtain the element information.
6. The method of claim 4, wherein the step of performing segmentation processing on the long text to obtain a plurality of text paragraphs corresponding to the long text, respectively, includes:
Determining the text segmentation length according to the processing calculation force of the natural language processing model;
and respectively cutting the long texts according to the text cutting lengths to obtain a plurality of text paragraphs corresponding to the long texts.
7. A text question-answering apparatus, the apparatus comprising:
the vectorization processing module is used for responding to a problem solving request of a user for a long text, vectorizing the problem of the user and obtaining a problem vector;
the retrieval module is configured to perform similarity retrieval on respective text vectors of the long texts according to the problem vector, so as to determine at least one target text paragraph related to the problem in the long texts, where the text vector of each long text is determined according to element information, text paragraph and global information of the long text, and the global information of each long text at least includes: the combined information of the text paragraph of the long text and the summary information of the long text;
and the determining module is used for determining an answer corresponding to the question according to the target text paragraph related to the question.
8. An electronic device, comprising: a processor, and a memory coupled to the processor;
the memory stores computer-executable instructions; the processor executes computer-executable instructions stored in the memory to implement the method of any one of claims 1 to 6.
9. A computer readable storage medium having stored therein computer executable instructions which when executed by a processor are adapted to carry out the method of any one of claims 1 to 6.
10. A computer program product comprising a computer program which, when executed by a processor, implements the method of any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311236897.3A CN117171328A (en) | 2023-09-22 | 2023-09-22 | Text question-answering processing method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311236897.3A CN117171328A (en) | 2023-09-22 | 2023-09-22 | Text question-answering processing method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117171328A true CN117171328A (en) | 2023-12-05 |
Family
ID=88936093
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311236897.3A Pending CN117171328A (en) | 2023-09-22 | 2023-09-22 | Text question-answering processing method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117171328A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117688138A (en) * | 2024-02-02 | 2024-03-12 | 中船凌久高科(武汉)有限公司 | Long text similarity comparison method based on paragraph division |
-
2023
- 2023-09-22 CN CN202311236897.3A patent/CN117171328A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117688138A (en) * | 2024-02-02 | 2024-03-12 | 中船凌久高科(武汉)有限公司 | Long text similarity comparison method based on paragraph division |
| CN117688138B (en) * | 2024-02-02 | 2024-04-09 | 中船凌久高科(武汉)有限公司 | Long text similarity comparison method based on paragraph division |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107679039B (en) | Method and device for determining statement intention | |
| CN111428010B (en) | Man-machine intelligent question-answering method and device | |
| CN111898643A (en) | Semantic matching method and device | |
| EP3832475A1 (en) | Sentence processing method and system and electronic device | |
| CN117171328A (en) | Text question-answering processing method and device, electronic equipment and storage medium | |
| CN111597800A (en) | Method, device, equipment and storage medium for obtaining synonyms | |
| CN116882372A (en) | Text generation method, device, electronic equipment and storage medium | |
| US20240079002A1 (en) | Minutes of meeting processing method and apparatus, device, and medium | |
| CN112182255A (en) | Method and apparatus for storing media files and for retrieving media files | |
| CN111428011B (en) | Word recommendation method, device, equipment and storage medium | |
| CN111444321B (en) | Question answering method, device, electronic equipment and storage medium | |
| CN111078849A (en) | Method and apparatus for outputting information | |
| CN114120166A (en) | Video question and answer method and device, electronic equipment and storage medium | |
| CN109657046B (en) | Content analysis processing method and device, electronic equipment and storage medium | |
| CN112231444A (en) | Processing method and device for corpus data combining RPA and AI and electronic equipment | |
| CN117349515A (en) | Search processing method, electronic device and storage medium | |
| CN111815274A (en) | Information processing method and device and electronic equipment | |
| CN111488450A (en) | Method and device for generating keyword library and electronic equipment | |
| CN110765357A (en) | Method, device and equipment for searching online document and storage medium | |
| CN106959945B (en) | Method and device for generating short titles for news based on artificial intelligence | |
| CN110502630B (en) | Information processing method and device | |
| CN112651231B (en) | Spoken language information processing method and device and electronic equipment | |
| CN112148751B (en) | Method and device for querying data | |
| CN114020896A (en) | Intelligent question and answer method, system, electronic equipment and storage medium | |
| CN109857838B (en) | Method and apparatus for generating information |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |