CN117575022A - Intelligent document question-answering method, device, equipment, medium and program product - Google Patents
Intelligent document question-answering method, device, equipment, medium and program product Download PDFInfo
- Publication number
- CN117575022A CN117575022A CN202311585275.1A CN202311585275A CN117575022A CN 117575022 A CN117575022 A CN 117575022A CN 202311585275 A CN202311585275 A CN 202311585275A CN 117575022 A CN117575022 A CN 117575022A
- Authority
- CN
- China
- Prior art keywords
- answer
- question
- document
- batch
- determining
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 78
- 238000011156 evaluation Methods 0.000 claims abstract description 77
- 125000004122 cyclic group Chemical group 0.000 claims abstract description 14
- 238000004422 calculation algorithm Methods 0.000 claims description 37
- 238000004590 computer program Methods 0.000 claims description 20
- 238000012896 Statistical algorithm Methods 0.000 claims description 16
- 238000012986 modification Methods 0.000 claims description 7
- 230000004048 modification Effects 0.000 claims description 7
- 238000013473 artificial intelligence Methods 0.000 abstract description 3
- 238000005516 engineering process Methods 0.000 abstract description 2
- 238000010586 diagram Methods 0.000 description 14
- 238000012549 training Methods 0.000 description 13
- 230000015654 memory Effects 0.000 description 10
- 230000006870 function Effects 0.000 description 9
- 238000004891 communication Methods 0.000 description 8
- 238000012545 processing Methods 0.000 description 8
- 230000008713 feedback mechanism Effects 0.000 description 7
- 230000008569 process Effects 0.000 description 6
- 238000004458 analytical method Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 238000012854 evaluation process Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 239000012634 fragment Substances 0.000 description 2
- 230000014509 gene expression Effects 0.000 description 2
- 238000007619 statistical method Methods 0.000 description 2
- 239000013598 vector Substances 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 238000013475 authorization Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000002457 bidirectional effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000004880 explosion Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 238000004806 packaging method and process Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000013077 scoring method Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000000758 substrate Substances 0.000 description 1
- 239000013589 supplement Substances 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N5/00—Computing arrangements using knowledge-based models
- G06N5/04—Inference or reasoning models
- G06N5/041—Abduction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/33—Querying
- G06F16/332—Query formulation
- G06F16/3329—Natural language query formulation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/22—Matching criteria, e.g. proximity measures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/205—Parsing
- G06F40/216—Parsing using statistical methods
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Artificial Intelligence (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Mathematical Physics (AREA)
- Evolutionary Computation (AREA)
- Human Computer Interaction (AREA)
- General Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- Databases & Information Systems (AREA)
- Probability & Statistics with Applications (AREA)
- Health & Medical Sciences (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Software Systems (AREA)
- Life Sciences & Earth Sciences (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Electrically Operated Instructional Devices (AREA)
Abstract
The disclosure provides an intelligent document question-answering method which can be applied to the field of artificial intelligence and the field of financial technology. The method comprises the following steps: determining the professional field; determining a pre-trained document question-answer large model according to the professional field; executing intelligent document questions and answers by adopting a document question and answer large model; the document question-answer large model is obtained by circularly adjusting parameters; the cyclic parameter adjustment comprises batch question and answer and evaluation feedback on the batch question and answer result; and the evaluation feedback of the batch question and answer results comprises traceability evaluation, answer evaluation and model acceptance. The disclosure also provides an intelligent document question answering device, equipment, a storage medium and a program product.
Description
Technical Field
The present disclosure relates to the field of artificial intelligence and the field of finance, and in particular, to an intelligent document question-answering method, apparatus, device, medium, and program product.
Background
In this age of information explosion, an ability to efficiently acquire document information is required. With the continued development of artificial intelligence technology, document question-answering systems have become one of the important means to promote this capability. In recent years, the capability of a large-scale language model represented by a GPT-3 model is continuously improved, and a new opportunity is brought for intelligent document questions and answers.
For a large-scale language model (hereinafter referred to as a document question-answer large model) used in the document question-answer field, the prior art is often focused on the natural language generation capability and the model iteration upgrading capability, and the research strength for evaluating the natural language generation result is insufficient.
The general field document question-answering large model can tolerate some answer errors, and the general field document question-answering large model can answer related contents. In contrast, people have higher requirements on a large document question-answer model in the professional field, such as higher accuracy, and the like, and all the parameters need to be adjusted to cope with specific business scenes. Moreover, different fields have different tolerance capabilities for the evaluation dimension of the document question-answering large model.
Disclosure of Invention
In view of the above, the present disclosure provides an intelligent document question-answering method, apparatus, device, medium, and program product that improve performance such as accuracy of intelligent document question-answering, for at least partially solving the above technical problems.
According to a first aspect of the present disclosure, there is provided an intelligent document question-answering method, including: determining the professional field; determining a pre-trained document question-answer large model according to the professional field; executing intelligent document questions and answers by adopting a document question and answer large model; the document question-answer large model is obtained by circularly adjusting parameters; the cyclic parameter adjustment comprises batch question and answer and evaluation feedback on the batch question and answer result; and the evaluation feedback of the batch question and answer results comprises traceability evaluation, answer evaluation and model acceptance.
According to an embodiment of the present disclosure, a document question-answer large model is derived from a cyclic call, comprising: initializing and configuring a document question-answer large model according to the professional field; carrying out batch questioning on the document question-answering large model which is initialized and configured to obtain a batch question-answering result; according to the batch question and answer results, carrying out configuration adjustment on the document question and answer big model; and repeating the batch questioning and configuration adjustment steps until the parameter adjustment turns are finished.
According to an embodiment of the present disclosure, performing configuration adjustment on a document question-answer large model according to a batch question-answer result includes: determining a tracing list, an original text list, an answer list and a correct answer list; according to the traceability list and the original text list, determining the traceability accuracy and/or the traceability recall rate of the batch question and answer results; determining at least one of answer recall rate, answer redundancy and answer stability of the batch question and answer results according to the answer list and the correct answer list; and determining the model acceptance according to the tracing accuracy and/or the tracing recall rate and at least one of the answer recall rate, the answer redundancy and the answer stability.
According to an embodiment of the present disclosure, determining the model acceptance comprises: determining the tracing accuracy and/or tracing recall rate, and the score of at least one of answer recall rate, answer redundancy and answer stability; determining the tracing accuracy and/or tracing recall rate and the weight of at least one of answer recall rate, answer redundancy and answer stability; normalizing the trace-source accuracy and/or trace-source recall rate and the score of at least one of answer recall rate, answer redundancy and answer stability to obtain a normalized score; and weighting according to the normalized score and the corresponding weight to obtain the model acceptance.
According to an embodiment of the present disclosure, determining the traceability accuracy and/or traceability recall, and the score of at least one of answer recall, answer redundancy, and answer stability comprises: determining the tracing accuracy, the answer recall rate and the answer stability by adopting a similarity algorithm; determining the source tracing recall rate and answer redundancy by adopting a statistical algorithm; the similarity algorithm at least comprises a cosine similarity algorithm, a Jaccard similarity algorithm and an edit distance similarity algorithm; and the statistical algorithm at least comprises a TF-IDF statistical algorithm and an N-gram statistical algorithm.
According to an embodiment of the present disclosure, according to the batch question-answer result, performing configuration adjustment on the document question-answer large model further includes: determining a custom index of a batch question and answer result according to one or more of the traceability list, the original text list, the answer list and the correct answer list; the custom index at least comprises a traceability stability index of a batch of question and answer results.
According to an embodiment of the present disclosure, repeating the batch questioning and configuration adjustment steps until the completion of the tuning pass includes: after each batch of question and answer results are obtained, judging whether the parameter adjustment turns are finished; under the condition that the model acceptance is greater than an acceptance threshold, determining that the parameter adjustment turn is finished; or determining that the parameter adjustment turn is finished under the condition that the running time of the document question-answer large model is larger than a time threshold value; or under the condition that a system stop instruction is received, determining that the parameter adjusting round is finished.
According to an embodiment of the present disclosure, according to the batch question-answer result, performing configuration adjustment on the document question-answer large model further includes: and displaying the historical full-quantity parameter, the current full-quantity parameter and the current modification parameter.
A second aspect of the present disclosure provides an intelligent document question-answering apparatus, including: the first determining module is used for determining the professional field; the second determining module is used for determining a pre-trained document question-answer model according to the professional field; the execution module is used for executing intelligent document questions and answers by adopting a document question and answer big model; the document question-answer large model is obtained by circularly adjusting parameters; the cyclic parameter adjustment comprises batch question and answer and evaluation feedback on the batch question and answer result; and the evaluation feedback of the batch question and answer results comprises traceability evaluation, answer evaluation and model acceptance.
A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the method of any of the embodiments described above.
A fourth aspect of the present disclosure also provides a computer readable storage medium having stored thereon executable instructions which, when executed by a processor, cause the processor to perform the method of any of the embodiments described above.
A fifth aspect of the present disclosure also provides a computer program product comprising a computer program which, when executed by a processor, implements the method of any of the embodiments described above.
Compared with the prior art, the intelligent document question-answering method, the intelligent document question-answering device, the electronic equipment, the storage medium and the program product have at least the following beneficial effects:
(1) According to the intelligent document question-answering method, the document intelligent question-answering is carried out by adopting the document question-answering large model with the automatic circulation parameter adjustment, so that the workload of manual parameter adjustment is reduced, the question-answering result is fed back through traceability evaluation and answer evaluation, and the accuracy and stability of document question-answering are improved. And the model parameter adjusting effect is characterized by the model acceptance, so that the method is visual and convenient.
(2) According to the intelligent document question-answering method, the document question-answering large model is initialized and configured and the parameters are adjusted circularly according to different professional fields, so that the method can be applied to intelligent document question-answering in a plurality of different fields, and the scene adaptability of the document question-answering large model is improved.
(3) According to the intelligent document question-answering method, the parameter adjusting effect of the document question-answering large model is evaluated by adopting a plurality of evaluation dimensions such as the tracing accuracy, the tracing recall, the answer redundancy, the answer stability and the like, the evaluation dimensions are rich, the scene adaptability is strong, and the accuracy of the question-answering result and the robustness of the document question-answering large model are improved.
Drawings
The foregoing and other objects, features and advantages of the disclosure will be more apparent from the following description of embodiments of the disclosure with reference to the accompanying drawings, in which:
FIG. 1 schematically illustrates an application scenario diagram of an intelligent document question-answering method, apparatus, device, medium and program product according to embodiments of the present disclosure;
FIG. 2 schematically illustrates a flow chart of an intelligent document question-answering method according to an embodiment of the present disclosure;
FIG. 3A schematically illustrates a flow chart of a method of circularly tuning a document question-answer large model in accordance with an embodiment of the disclosure; FIG. 3B schematically illustrates a flow chart of a method of circularly tuning a document question-answer large model in accordance with another embodiment of the present disclosure;
FIG. 4 schematically illustrates a flow diagram of a method for configuration adjustment of a document question-answer large model in accordance with an embodiment of the disclosure;
FIG. 5 schematically illustrates a method flow diagram for determining model acceptance according to an embodiment of the disclosure;
FIG. 6A schematically illustrates a flow chart of a method of determining a scoring index score according to an embodiment of the present disclosure; FIG. 6B schematically illustrates a flow chart of a question-answer result automatic evaluation feedback mechanism according to an embodiment of the disclosure;
FIG. 7 schematically illustrates a flow diagram of a method for configuration adjustment of a document question-answer large model in accordance with another embodiment of the present disclosure;
FIG. 8 schematically illustrates a flow chart of a method of determining a turn of a call, according to an embodiment of the disclosure;
FIG. 9 schematically illustrates a flow diagram of a method of making configuration adjustments to a document question-answer large model in accordance with yet another embodiment of the present disclosure;
FIG. 10 schematically illustrates a block diagram of a configuration of an intelligent document questioning and answering apparatus according to an embodiment of the present disclosure; and
fig. 11 schematically illustrates a block diagram of an electronic device adapted to implement the intelligent document question-answering method according to an embodiment of the present disclosure.
Detailed Description
Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood that the description is only exemplary and is not intended to limit the scope of the present disclosure. In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the present disclosure. It may be evident, however, that one or more embodiments may be practiced without these specific details. In addition, in the following description, descriptions of well-known structures and techniques are omitted so as not to unnecessarily obscure the concepts of the present disclosure.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The terms "comprises," "comprising," and/or the like, as used herein, specify the presence of stated features, steps, operations, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, or components.
All terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art unless otherwise defined. It should be noted that the terms used herein should be construed to have meanings consistent with the context of the present specification and should not be construed in an idealized or overly formal manner.
Where expressions like at least one of "A, B and C, etc. are used, the expressions should generally be interpreted in accordance with the meaning as commonly understood by those skilled in the art (e.g.," a system having at least one of A, B and C "shall include, but not be limited to, a system having a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.).
The embodiment of the disclosure provides an intelligent document question-answering method, device, equipment, medium and program product, which can be used in the financial field or other fields. It should be noted that the method, apparatus, device, medium and program product for question-answering of an intelligent document of the present disclosure may be used in the financial field, and may also be used in any field other than the financial field, and the application fields of the method, apparatus, device, medium and program product for question-answering of an intelligent document of the present disclosure are not limited.
In the technical scheme of the invention, the related user information (including but not limited to user personal information, user image information, user equipment information, such as position information and the like) and data (including but not limited to data for analysis, stored data, displayed data and the like) are information and data authorized by a user or fully authorized by all parties, and the processing of the related data such as collection, storage, use, processing, transmission, provision, disclosure, application and the like are all conducted according to the related laws and regulations and standards of related countries and regions, necessary security measures are adopted, no prejudice to the public welfare is provided, and corresponding operation inlets are provided for the user to select authorization or rejection.
The embodiment of the disclosure provides an intelligent document question-answering method, which comprises the following steps: determining the professional field; determining a pre-trained document question-answer large model according to the professional field; executing intelligent document questions and answers by adopting a document question and answer large model; the document question-answer large model is obtained by circularly adjusting parameters; the cyclic parameter adjustment comprises batch question and answer and evaluation feedback on the batch question and answer result; and the evaluation feedback of the batch question and answer results comprises traceability evaluation, answer evaluation and model acceptance. By adopting the document question-answering large model of the automatic cyclic call, the intelligent question-answering of the document is carried out, the workload of manual call is reduced, the question-answering result is fed back through traceability evaluation and answer evaluation, and the accuracy and stability of the document question-answering are also improved. And the model parameter adjusting effect is characterized by the model acceptance, so that the method is visual and convenient.
Fig. 1 schematically illustrates an application scenario diagram of an intelligent document question-answering method, apparatus, device, medium and program product according to an embodiment of the present disclosure.
As shown in fig. 1, an application scenario 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used as a medium to provide communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.
The user may interact with the server 105 via the network 104 using the terminal devices 101, 102, 103 to receive or send messages or the like. Various communication client applications, such as model management class applications, shopping class applications, web browser applications, search class applications, instant messaging tools, mailbox clients, social platform software, etc. (by way of example only) may be installed on the terminal devices 101, 102, 103.
The terminal devices 101, 102, 103 may be a variety of electronic devices having a display screen and supporting web browsing, including but not limited to smartphones, tablets, laptop and desktop computers, and the like.
The server 105 may be a server providing various services, such as a background management server (by way of example only) providing support for websites browsed by users using the terminal devices 101, 102, 103. The background management server may analyze and process the received data such as the user request, and feed back the processing result (e.g., the web page, information, or data obtained or generated according to the user request) to the terminal device. In particular, server 105 may be a server that provides model training and applications.
It should be noted that, the method for asking and answering an intelligent document provided by the embodiment of the present disclosure may be generally executed by the server 105. Accordingly, the intelligent document question-answering apparatus provided by the embodiments of the present disclosure may be generally provided in the server 105. The intelligent document question-answering method provided by the embodiments of the present disclosure may also be performed by a server or a server cluster that is different from the server 105 and is capable of communicating with the terminal devices 101, 102, 103 and/or the server 105. Accordingly, the intelligent document answering apparatus provided by the embodiments of the present disclosure may also be provided in a server or a server cluster that is different from the server 105 and is capable of communicating with the terminal devices 101, 102, 103 and/or the server 105.
It should be understood that the number of terminal devices, networks and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
First, the related terms of the embodiments of the present disclosure are explained as follows:
document question-answering big model: a large-scale language training model is used for searching answers related to user questions from various unstructured document data. The original document is generally unstructured data, and in the large model processing process, the original document is generally converted into structured knowledge, the structured knowledge is stored in a knowledge base, query reasoning is carried out, an accurate answer for solving the problem is found and fed back to a user, and the user is assisted in solving different types of problems.
Document question-answering system: the document question-answering system refers to a complete set of system for searching answers related to user questions from large-scale document data. The system will receive the text entered question Q and, given the document set D, automatically extract the information and generate the corresponding answer a. Document question-answering systems are clearly distinguished from general question-answering systems. The question-answering system is typically a multi-round dialog system, and the document-question-answering system is more like a sub-question of a multi-round dialog, considering only the current question, and answering according to a set of documents given by this question.
Evaluation of question and answer results: the answers generated by the document questions and answers are good or bad, and the results are scored from different dimensions by using some evaluation indexes, so that the quality of the text generated by the model is evaluated.
The intelligent document question-answering method of the disclosed embodiment will be described in detail below with reference to the scenario described in fig. 1 through fig. 2 to 9.
Fig. 2 schematically illustrates a flowchart of an intelligent document question-answering method according to an embodiment of the present disclosure.
As shown in fig. 2, the intelligent document question-answering method of the embodiment includes, for example, operations S210 to S230, and the intelligent document question-answering method can be executed by a computer program on corresponding computer hardware.
In operation S210, a professional field is determined.
In operation S220, a pre-trained document question-answer large model is determined according to the professional field.
In operation S230, an intelligent document question-answering is performed using the document question-answering large model. The document question-answer large model is obtained by circularly adjusting parameters. The cyclic tuning comprises batch questioning and answering and evaluation feedback on the results of the batch questioning and answering. And the evaluation feedback of the batch question and answer results comprises traceability evaluation, answer evaluation and model acceptance.
For example, many fields have the need to use documents to make intelligent questions and answers, so that the system does not set options, and the user can fill in the field names by himself in a purely self-defined filling form. Such as the professional domain may be the intellectual encyclopedia, legal, boring, financial domain, etc. Various document question-answering models exist at present, and can provide services for different clients, such as open-source chatGLM-6B, T5 (Text-to-Text Transfer Transformers), BART (Bidirectional and Auto-Regressive Transformer) and other models. The premise of the present disclosure is, for example, that a document question-answer large model has been deployed locally. The user can select a target document question-answer large model at the step, and the question-answer query interface provided by the model is filled in a request sending window of the system and stored.
For example, an intelligent document question-answering system is intended to provide question-answering services to users in the financial domain. The system adopts a document question-answering large model in the financial field, and optimizes the performance of the model through cyclic parameter adjustment. First, it was determined that the area of expertise for this system was "finance". Then, according to the field, a pre-trained document question-answer large model of a financial field is selected. The model is trained on a large number of financial documents and question-answer data sets, and can provide accurate answers to questions in the financial field. This document question and answer large model is then employed to perform intelligent document questions and answers. The user may enter a question about the financial market, investment policy, risk management, or other relevant area, and the model looks for relevant information from a large number of financial documents and generates an answer.
The document question-answering large model in the financial field is subjected to cyclic parameter adjustment, answers can be generated by the model according to a plurality of different questions in a batch question-answering mode, and the answers are evaluated and fed back. For example, answers to the model may be evaluated with specific indicators or criteria in the financial domain. If the answer of the model is not accurate enough or does not meet the actual requirements, the problem can be analyzed through traceability evaluation. For example, models have some misunderstandings or bias in understanding certain terms of art or specific financial contexts. Meanwhile, the answers can be evaluated, and whether the answers generated by the analysis model meet the actual requirements of the financial field or not is analyzed. Finally, the acceptability of the model can also be evaluated to see if the user is satisfied with the answer generated by the model. Through the cyclic parameter adjustment, the large document question-answer model in the financial field can be continuously optimized, the performance and accuracy of the large document question-answer model are improved, and intelligent question-answer service can be better provided for users in the financial field. Meanwhile, the application of the model can be expanded to other related fields, and more comprehensive and professional knowledge solution service can be provided for the user.
FIG. 3A schematically illustrates a flow chart of a method of circularly tuning a document question-answer large model in accordance with an embodiment of the disclosure. FIG. 3B schematically illustrates a flow chart of a method of circularly tuning a document question-answer large model in accordance with another embodiment of the present disclosure.
According to an embodiment of the present disclosure, as shown in fig. 3A and 3B, the document question-answering large model is cyclically referred, for example, through operations S231 to S234.
In operation S231, the document question-answer large model is initialized according to the professional field.
In operation S232, batch questioning is performed on the document question-answering large model of the initialization configuration, and a batch question-answering result is obtained.
In operation S233, a document question-answer large model is configured and adjusted according to the batch question-answer result.
In operation S234, the batch questioning and configuration adjustment steps are repeated until the parameter tuning round is completed.
For example, in this step, the path and filename of the document question-answer large model configuration file are filled in on the system interface. The system supports reading txt, json, config format configuration files, and displaying key-value information in the configuration files on an interface.
In the interface related to the step, the following parameters are opened for filling or checking, and the method specifically comprises the following steps:
Default init-value for each configuration item: default value initialization is carried out by default of the configuration file, and after clicking the lowest save, the latest value is written back to the configuration file.
Minimum step: for each value, a designation of "minimum step" is allowed, defaulting to 0.1.
Whether two ends adjust two-ends-adjustment: allowing the user to set whether to increase or decrease step size starting from both ends of the specified parameter, the user can select "yes" or "no" and default to "yes".
Running wheel number batch: the number of rounds of parameter adjustment was designated and defaults to 10.
Whether or not to call the number-by-number item by item: whether one parameter runs the batch round number is not finished, and then the next parameter runs the batch round number can be selected as yes or no, and the default is yes. The parameter is related to the batch parameter. In a configuration file with the parameter number of n, the total running wheel number of the whole system is sum-batch, and the value is calculated as follows:
when number-by-number=yes, sum-batch=batch;
when number-by-number=no, sum-batch=batch;
examples: the parameters of the user initialization configuration file and the interface settings are as follows:
TABLE 1 initialization configuration parameters
Then after the tuning process is finished, the values of the reference source scores are respectively subjected to 0.5, 0.55, 0.45, 0.6 and 0.4, the values of the reference source reference score threshold values are subjected to 7, 8, 6, 9 and 5 after 5 rounds of adjustment of the values of the reference source scores are finished, and the total parameter tuning process is subjected to 10 rounds.
For example, as shown in FIG. 3B, after initial configuration of the model parameters, a large model is run. In the system interface corresponding to the step, a script path corresponding to the restarting large model service needs to be filled in, and parameter suffixes possibly attached to the script are filled in the parameter area. After the parameter configuration/adjustment is completed, a restarting button is clicked, and the document question-answering model is restarted to run under the new parameter condition. In order to improve the generation efficiency of the question and answer results, the system integrates the function of batch question and answer. In this step, the user may choose to upload a local txt file containing the batch of questions, or may choose to paste the batch of questions into text boxes in the interface. Clicking the question button to start batch generation of inquiry results of the document questions and answers. After the query is completed, 2 files, such as an answer list of the query result and a traceability list of the query result, are generated, and the click can be directly downloaded. And the original text list and the correct answer list corresponding to the batch question-answer process can be obtained, so that automatic evaluation feedback of question-answer results is realized, and the accuracy of generating answers by the model is further verified. The step needs to input an answer list, a traceability list, an original text list and a correct answer list of a large document question-answer model under the parameter configuration, and output a final output model overall acceptability and all traceability evaluation processes and all answer evaluation processes.
FIG. 4 schematically illustrates a flow chart of a method of configuration adjustment of a document question-answer large model in accordance with an embodiment of the disclosure.
According to an embodiment of the present disclosure, as shown in fig. 4, a document question-answering large model is configured and adjusted according to a batch question-answering result, for example, through operations S4331 to S4334.
In operation S4331, a traceability list, an original list, an answer list, and a correct answer list are determined.
In operation S4332, the tracing accuracy and/or the tracing recall of the batch question-answer result is determined according to the tracing list and the original text list.
In operation S4333, at least one of answer recall rate, answer redundancy, and answer stability of the batch question-answer result is determined according to the answer list and the correct answer list.
In operation S4334, a model acceptance is determined based on the traceability accuracy and/or the traceability recall and at least one of answer recall, answer redundancy, and answer stability.
For example, the question and answer result automatic evaluation feedback mechanism requires 4 input data files: a traceability list, an answer list, an original text list and a correct answer list. All the tracing fragments related to each answer are listed in the tracing list, and each tracing fragment is from the document original document. The answer list records the following information: questions, answers containing a traceability sequence number. The two data files are transferred into the system processing flow corresponding to the feedback mechanism, and the operation of the whole feedback subsystem depends on the two file streams. The traceability list and the answer list in the question and answer result can be derived into corresponding evaluation dimensions, and the system is embedded with 5 indexes, namely traceability accuracy and traceability recall rate based on the traceability list, answer recall rate, answer redundancy and answer stability based on the answer list.
Fig. 5 schematically illustrates a flow chart of a method of determining pattern acceptance according to an embodiment of the disclosure.
According to an embodiment of the present disclosure, as shown in fig. 5, the model acceptance is determined, for example, by operations S5341 to S5344.
In operation S5341, a trace-source accuracy and/or trace-source recall, and a score of at least one of answer recall, answer redundancy, and answer stability are determined.
In operation S5342, a traceability accuracy and/or traceability recall and a weight of at least one of answer recall, answer redundancy, and answer stability are determined.
In operation S5343, the trace-source accuracy and/or trace-source recall and the score of at least one of answer recall, answer redundancy, and answer stability are normalized to obtain a normalized score. And
In operation S5344, the model acceptance is weighted according to the normalized score and the corresponding weight.
For example, a partial similarity algorithm and a statistical method are integrated in the system, and a scoring method or algorithm is assigned to each evaluation dimension to realize automatic evaluation. The specific flow is as follows: the question and answer result evaluation dimension is selected, and an existing algorithm or a newly added custom algorithm is selected.
Taking the relevant question and answer result evaluation dimension corresponding to the answer list as an example:
when the existing algorithm is selected, the system automatically transmits the correct answer list and the answer list generated by the document question-answer large model into the selected algorithm respectively, and the system automatically calculates the corresponding score. When a new custom algorithm is selected, a user needs to fill in a service address corresponding to the algorithm, the system subsequently and automatically transmits a correct answer list and an answer list generated by a document question-answer large model into the selected algorithm respectively, and the system automatically calculates a corresponding score.
For example, score normalization, controls scoring results for each algorithm and method to the same level. Definition of traceability list derivationScore A corresponding to the evaluation dimension of (2) i The answer list derived evaluation dimension corresponds to a score of S j . For example, a similarity algorithm scores the answer redundancy as 89 points (100 points full), and the normalized value range of the answer redundancy score is [0,1 ]]The answer redundancy score that is finally output to the next step is actually 0.89 minutes.
For example, to unify all metrics, to get the overall acceptability of the document question-answer large model, a weight needs to be set for each evaluation dimension in order to calculate the score with subsequent weighting item by item. After each question and answer result evaluates the display box of the dimension, an input text box is arranged for inputting the weight of the corresponding dimension. By adjusting the weights of different dimensions, the evaluation feedback of the question and answer results can be flexibly adjusted, and the question and answer accuracy of the large model is further improved. Defining the weight corresponding to the evaluation dimension derived from the traceability list as W ai The answer list derived evaluation dimension corresponds to a weight of W sj 。
Further, according to the score A i Score S j Weight W ai And weight is W sj The weighted score G corresponding to the evaluation index derived from the traceability list can be obtained ai Weighted score G corresponding to the evaluation index derived from the traceability list sj . For example, the number of the cells to be processed,
the trace source list item evaluation dimension weighting score is as follows:
G ai ＝W ai ×A i (1)
the answer list item evaluation dimension weighting score is:
G sj ＝W sj ×S j (2)
and according to the tracing list single item evaluation dimension weighted score and the answer list single item evaluation dimension weighted score, solving the weighted sum of all indexes, wherein the value range of the result is [0,1], and the result can be equivalent to the acceptance degree T of the model.
From the time of obtaining the batch question and answer results to the time of calculating the acceptability of the model corresponding to the results, the question and answer results automatic evaluation feedback mechanism completes one operation. And then, feeding back the model acceptance and all the tracing and answer evaluation process data to a parameter adjustment turn judgment step in a parameter adjustment system of the professional field document question-answering large model.
Fig. 6A schematically illustrates a flow chart of a method of determining a scoring index score according to an embodiment of the disclosure. Fig. 6B schematically illustrates a flow chart of a question-answer result automatic evaluation feedback mechanism according to an embodiment of the disclosure.
According to an embodiment of the present disclosure, as shown in fig. 6A, scores of evaluation indexes such as a trace-source accuracy rate, a trace-source recall rate, an answer redundancy, and an answer stability are determined, for example, through operations S6411 to S6412.
In operation S6411, a similarity algorithm is used to determine the tracing accuracy, answer recall, and answer stability.
In operation S6412, a statistical algorithm is used to determine the trace-source recall and answer redundancy. The similarity algorithm at least comprises a cosine similarity algorithm, a Jaccard similarity algorithm and an edit distance similarity algorithm. And the statistical algorithm at least comprises a TF-IDF statistical algorithm and an N-gram statistical algorithm.
For example, the system provides a plurality of embedded full text similarity algorithms, ROUGE (Recall-Oriented Understudy for Gisting Evaluation) abstract evaluation methods and traditional statistical methods, and also provides a custom method to be added as a grading method supplement for the evaluation dimension of the new question and answer result. As shown in fig. 6B, a cosine similarity algorithm may be used to determine the traceability accuracy, an answer recall is used to determine the answer recall, and an edit distance similarity algorithm is used to determine the answer stability. And determining the traceability recall rate by adopting a TF-IDF (word frequency inverse document frequency) statistical algorithm, and determining the answer redundancy by adopting an N-gram (continuous subsequence with the length of N) statistical algorithm. The scores of the relevant evaluation dimensions are calculated respectively by adopting a similarity algorithm and a statistical algorithm, and an algorithm is supported to be appointed for each dimension, so that a document question-answer large model adapting to different fields can be trained conveniently.
It will be appreciated that other similarity algorithms may be used to implement the calculations described above, in addition to the similarity algorithms and statistical algorithms described above. Such as the longest common subsequence (Longest Common Subsequence) algorithm, simHash algorithm, N-Gram algorithm, levenshtein algorithm, etc. Word embedding/Word vectors (Word vectors), LSA/LSI (Latent Semantic Analysis/Latent Semantic Indexing), topic Models (Topic Models), HMM (Hidden Markov Model), CRF (Conditional Random Field), etc.
FIG. 7 schematically illustrates a flow chart of a method of configuration adjustment of a document question-answer large model in accordance with another embodiment of the present disclosure.
According to an embodiment of the present disclosure, as shown in fig. 7, for example, the document question-answer large model may also be configured and adjusted according to the batch question-answer result through operation S7331.
In operation S7331, a custom index of the batch question and answer result is determined according to one or more of the traceability list, the original text list, the answer list and the correct answer list. The custom index at least comprises a traceability stability index of a batch of question and answer results.
For example, custom evaluation dimensions can be added, such as setting a traceability stability index based on a traceability list. The function of custom indicators has two benefits: firstly, the evaluation dimension can be further enriched, so that the evaluation of the question and answer result is more accurate. Secondly, different large models have different parameters, different evaluation indexes can reflect adjustment results of different parameters, and the system can only integrate general evaluation indexes and can not integrate specific evaluation indexes, so that the custom new addition of the evaluation indexes can improve the robustness of evaluation dimensions.
Fig. 8 schematically illustrates a flow chart of a method of determining a turn of a call, according to an embodiment of the disclosure.
According to an embodiment of the present disclosure, as shown in fig. 8, it is determined whether the tuning is ended, for example, through operations S8341 to S8342.
In operation S8341, after each acquisition of the batch question-answer result, it is determined whether the parameter adjustment round is ended.
In operation S8342, in the case where the model acceptance is greater than the acceptance threshold, it is determined that the tone turn ends. Or determining that the parameter tuning turn is finished under the condition that the running time of the document question-answering large model is larger than a time threshold value. Or under the condition that a system stop instruction is received, determining that the parameter adjustment turn is finished.
For example, in this step, for determining the selector of the iteration round, an early end condition of the tone training round may be set in the system, and the system provides 3 options for early end of the tone training, which are specifically as follows:
stopping after exceeding the threshold value: and when the acceptance threshold of the document question-answer large model exceeds the threshold filled by the user, ending the parameter tuning training turn.
Stopping after the exceeding time: when the machine running the large model has the limit of the service time, the option can be selected to be used, the overtime time is filled, and after the time reaches the set time threshold, the parameter adjustment training round is finished.
Direct stop: the system provides a direct stop button, and after clicking, the user can stop the parameter adjustment training round in advance.
If the user does not select to use the function of stopping in advance, the system defaults to perform the next round of parameter adjustment training (the parameter adjustment round judging step returns to the document question-answer large model parameter adjusting step, and corresponding steps are sequentially and repeatedly executed) until all training rounds related to the document question-answer large model parameter adjusting step are finished. At least three parameter adjustment ending conditions are provided, and the training flexibility of the document question-answer large model is improved.
For example, whether the parameter adjustment training round or the normal training round is finished in advance, a group of optimal parameter configurations are directly displayed in the interface after the parameter adjustment training round is finished, and the user can click the [ write configuration file ] button to write the optimal information back to the configuration file of the document question-answering large model. The other parameter configurations and the detail dimension/summary scores are provided in a file form, and a user is required to click a button (download), so that the parameter tuning information containing scoring results in full is downloaded.
FIG. 9 schematically illustrates a flow chart of a method of configuration adjustment of a document question-answer large model in accordance with yet another embodiment of the disclosure.
According to an embodiment of the present disclosure, as shown in fig. 9, for example, the document question-answer large model may also be configured and adjusted according to the batch question-answer result through operation S9331.
In operation S9331, the historical full-scale parameter, the current full-scale parameter, and the current modification parameter are presented.
For example, to show the parameters that have been debugged in detail, the user is prompted for historical parameters, after each modification of the parameters is completed and clicking is performed, the latest value is transferred into the historical parameter display area in addition to the configuration file being written back. The display information comprises key-value information of the previous full-quantity parameter, key-value information of the current full-quantity parameter and key-value information of the current modification parameter. The invention provides a feedback mechanism for automatically evaluating the question and answer result, and provides a document question and answer model parameter adjusting method in the financial field for users based on the feedback mechanism, the whole parameter adjusting flow is packaged into a complete system, the parameter adjusting workload of the users on the document question and answer model is reduced, the users can more conveniently perform parameter configuration of the document question and answer model aiming at specific scenes in the financial field, and the document question and answer model can better serve specific scenes.
It should be noted that the above system of the present disclosure may operate in a common desktop environment, and the document question-answering large model docked in the system needs to be deployed in a linux environment with GPU cards.
Based on the intelligent document question-answering method, the disclosure also provides an intelligent document question-answering device. The intelligent document answering machine will be described in detail with reference to fig. 10.
Fig. 10 schematically shows a block diagram of the intelligent document answering apparatus according to the embodiment of the present disclosure.
As shown in fig. 10, the intelligent document question-answering apparatus 1000 of this embodiment includes, for example: a first determination module 1010, a second determination module 1020, and an execution module 1030.
The first determining module 1010 is configured to determine a domain of expertise. In an embodiment, the first determining module 1010 may be configured to perform the operation S210 described above, which is not described herein.
The second determining module 1020 is configured to determine a pre-trained document question-answer model according to the professional field. In an embodiment, the second determining module 1020 may be configured to perform the operation S220 described above, which is not described herein.
The execution module 1030 is configured to execute the intelligent document question-answering using the document question-answering big model. The document question-answer large model is obtained by circularly adjusting parameters. The cyclic tuning comprises batch questioning and answering and evaluation feedback on the results of the batch questioning and answering. And the evaluation feedback of the batch question and answer results comprises traceability evaluation, answer evaluation and model acceptance. In an embodiment, the execution module 1030 may be configured to execute the operation S230 described above, which is not described herein.
Any of the plurality of modules of the first determination module 1010, the second determination module 1020, and the execution module 1030 may be combined in one module to be implemented, or any of the plurality of modules may be split into a plurality of modules, according to embodiments of the present disclosure. Alternatively, at least some of the functionality of one or more of the modules may be combined with at least some of the functionality of other modules and implemented in one module. According to embodiments of the present disclosure, at least one of the first determination module 1010, the second determination module 1020, and the execution module 1030 may be implemented at least in part as hardware circuitry, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system on a chip, a system on a substrate, a system on a package, an Application Specific Integrated Circuit (ASIC), or may be implemented in hardware or firmware in any other reasonable way of integrating or packaging circuitry, or in any one of or a suitable combination of any of the three. Alternatively, at least one of the first determination module 1010, the second determination module 1020, and the execution module 1030 may be at least partially implemented as a computer program module which, when executed, may perform the corresponding functions.
Fig. 11 schematically illustrates a block diagram of an electronic device adapted to implement the intelligent document question-answering method according to an embodiment of the present disclosure.
As shown in fig. 11, an electronic device 1100 according to an embodiment of the present disclosure includes a processor 1101 that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 1102 or a program loaded from a storage section 1108 into a Random Access Memory (RAM) 1103. The processor 1101 may include, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor and/or an associated chipset and/or a special purpose microprocessor (e.g., an Application Specific Integrated Circuit (ASIC)), or the like. The processor 1101 may also include on-board memory for caching purposes. The processor 1101 may comprise a single processing unit or a plurality of processing units for performing the different actions of the method flow according to embodiments of the present disclosure.
In the RAM 1103, various programs and data necessary for the operation of the electronic device 1100 are stored. The processor 1101, ROM 1102, and RAM 1103 are connected to each other by a bus 1104. The processor 1101 performs various operations of the method flow according to the embodiments of the present disclosure by executing programs in the ROM 1102 and/or the RAM 1103. Note that the program may be stored in one or more memories other than the ROM 1102 and the RAM 1103. The processor 1101 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in the one or more memories.
According to an embodiment of the disclosure, the electronic device 1100 may also include an input/output (I/O) interface 1105, the input/output (I/O) interface 1105 also being connected to the bus 1104. The electronic device 1100 may also include one or more of the following components connected to the I/O interface 1105: an input section 1106 including a keyboard, a mouse, and the like; an output portion 1107 including a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, a speaker, and the like; a storage section 1108 including a hard disk or the like; and a communication section 1109 including a network interface card such as a LAN card, a modem, and the like. The communication section 1109 performs communication processing via a network such as the internet. The drive 1110 is also connected to the I/O interface 1105 as needed. Removable media 1111, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like, is installed as needed in drive 1110, so that a computer program read therefrom is installed as needed in storage section 1108.
The present disclosure also provides a computer-readable storage medium that may be embodied in the apparatus/device/system described in the above embodiments; or may exist alone without being assembled into the apparatus/device/system. The computer-readable storage medium described above carries one or more programs that, when executed, implement the intelligent document question-answering method according to the embodiments of the present disclosure.
According to embodiments of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, which may include, for example, but is not limited to: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this disclosure, a computer-readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. For example, according to embodiments of the present disclosure, the computer-readable storage medium may include ROM 1102 and/or RAM1103 described above and/or one or more memories other than ROM 1102 and RAM 1103.
Embodiments of the present disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowcharts. The program code, when executed in a computer system, causes the computer system to implement the intelligent document question-answering method provided by embodiments of the present disclosure.
The above-described functions defined in the system/apparatus of the embodiments of the present disclosure are performed when the computer program is executed by the processor 1101. The systems, apparatus, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the disclosure.
In one embodiment, the computer program may be based on a tangible storage medium such as an optical storage device, a magnetic storage device, or the like. In another embodiment, the computer program can also be transmitted, distributed over a network medium in the form of signals, downloaded and installed via the communication portion 1109, and/or installed from the removable media 1111. The computer program may include program code that may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the foregoing.
In such an embodiment, the computer program can be downloaded and installed from a network via the communication portion 1109, and/or installed from the removable medium 11. The above-described functions defined in the system of the embodiments of the present disclosure are performed when the computer program is executed by the processor 1101. The systems, devices, apparatus, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the disclosure.
According to embodiments of the present disclosure, program code for performing computer programs provided by embodiments of the present disclosure may be written in any combination of one or more programming languages, and in particular, such computer programs may be implemented in high-level procedural and/or object-oriented programming languages, and/or assembly/machine languages. Programming languages include, but are not limited to, such as Java, c++, python, "C" or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, partly on a remote computing device, or entirely on the remote computing device or server. In the case of remote computing devices, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., connected via the Internet using an Internet service provider).
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
Those skilled in the art will appreciate that the features recited in the various embodiments of the disclosure and/or in the claims may be provided in a variety of combinations and/or combinations, even if such combinations or combinations are not explicitly recited in the disclosure. In particular, the features recited in the various embodiments of the present disclosure and/or the claims may be variously combined and/or combined without departing from the spirit and teachings of the present disclosure. All such combinations and/or combinations fall within the scope of the present disclosure.
The embodiments of the present disclosure are described above. However, these examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the embodiments cannot be used advantageously in combination. The scope of the disclosure is defined by the appended claims and equivalents thereof. Various alternatives and modifications can be made by those skilled in the art without departing from the scope of the disclosure, and such alternatives and modifications are intended to fall within the scope of the disclosure.
Claims (12)
1. An intelligent document question-answering method, which is characterized by comprising the following steps:
determining the professional field;
determining a pre-trained document question-answer large model according to the professional field;
Executing intelligent document questions and answers by adopting the document question and answer large model;
the document question-answer large model is obtained by circularly adjusting parameters;
the cyclic call parameters comprise batch questions and answers and evaluation feedback on the results of the batch questions and answers; and
the evaluation feedback of the batch question and answer results comprises traceability evaluation, answer evaluation and model acceptance.
2. The method of claim 1, wherein the document question-answer large model is derived from a cyclic call comprising:
initializing and configuring the document question-answer large model according to the professional field;
carrying out batch questioning on the document question-answering large model which is initialized and configured to obtain a batch question-answering result;
according to the batch question and answer results, carrying out configuration adjustment on the document question and answer large model;
and repeating the batch questioning and configuration adjustment steps until the parameter adjustment turns are finished.
3. The method of claim 2, wherein the configuring the document question-answer large model according to the batch question-answer results comprises:
determining a tracing list, an original text list, an answer list and a correct answer list;
determining the tracing accuracy and/or tracing recall rate of the batch question and answer results according to the tracing list and the original text list;
Determining at least one of answer recall rate, answer redundancy and answer stability of the batch question and answer results according to the answer list and the correct answer list;
and determining the model acceptance according to the tracing accuracy and/or the tracing recall rate and at least one of the answer recall rate, the answer redundancy and the answer stability.
4. The method of claim 3, wherein said determining the model acceptance based on the traceability accuracy and/or traceability recall and at least one of the answer recall, answer redundancy, and answer stability comprises:
determining the tracing accuracy rate and/or tracing recall rate and the score of at least one of answer recall rate, answer redundancy and answer stability;
determining the tracing accuracy rate and/or tracing recall rate and the weight of at least one of answer recall rate, answer redundancy and answer stability;
normalizing the trace-source accuracy and/or trace-source recall rate and the score of at least one of the answer recall rate, the answer redundancy and the answer stability to obtain a normalized score; and
And weighting according to the normalized score and the corresponding weight to obtain the model acceptance.
5. The method of claim 4, wherein the determining the trace-to-source accuracy and/or trace-to-source recall, and the score for at least one of answer recall, answer redundancy, and answer stability comprises:
determining the tracing accuracy, the answer recall rate and the answer stability by adopting a similarity algorithm;
determining the source tracing recall rate and the answer redundancy by adopting a statistical algorithm;
the similarity algorithm at least comprises a cosine similarity algorithm, a Jaccard similarity algorithm and an edit distance similarity algorithm; and
the statistical algorithm at least comprises a TF-IDF statistical algorithm and an N-gram statistical algorithm.
6. The method of claim 3, wherein said configuring the document question-answer large model according to the batch question-answer results further comprises:
determining a custom index of the batch question and answer result according to one or more of the traceability list, the original text list, the answer list and the correct answer list;
the custom index at least comprises a traceability stability index of the batch question and answer results.
7. The method of claim 2, wherein repeating the batch questioning and configuration adjustment steps until the completion of the tuning pass comprises:
after each time of obtaining the batch question and answer results, judging whether the parameter adjustment turns are finished;
determining that the parameter adjustment turn is finished under the condition that the acceptance degree of the model is larger than an acceptance degree threshold value; or alternatively
Determining that the parameter adjustment turn is finished under the condition that the running time of the document question-answer large model is larger than a time threshold; or alternatively
And under the condition that a system stop instruction is received, determining that the parameter adjustment turn is finished.
8. The method of claim 2, wherein the configuring the document question-answer large model according to the batch question-answer results further comprises:
and displaying the historical full-quantity parameter, the current full-quantity parameter and the current modification parameter.
9. An intelligent document question-answering device, characterized by comprising:
the first determining module is used for determining the professional field;
the second determining module is used for determining a pre-trained document question-answer large model according to the professional field; and
the execution module is used for executing intelligent document questions and answers by adopting the document question and answer big model;
The document question-answer large model is obtained by circularly adjusting parameters;
the cyclic call parameters comprise batch questions and answers and evaluation feedback on the results of the batch questions and answers; and
the evaluation feedback of the batch question and answer results comprises traceability evaluation, answer evaluation and model acceptance.
10. An electronic device, comprising:
one or more processors;
storage means for storing one or more programs,
wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the method of any of claims 1-8.
11. A computer readable storage medium having stored thereon executable instructions which, when executed by a processor, cause the processor to perform the method according to any of claims 1-8.
12. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311585275.1A CN117575022A (en) | 2023-11-24 | 2023-11-24 | Intelligent document question-answering method, device, equipment, medium and program product |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311585275.1A CN117575022A (en) | 2023-11-24 | 2023-11-24 | Intelligent document question-answering method, device, equipment, medium and program product |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117575022A true CN117575022A (en) | 2024-02-20 |
Family
ID=89863929
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311585275.1A Pending CN117575022A (en) | 2023-11-24 | 2023-11-24 | Intelligent document question-answering method, device, equipment, medium and program product |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117575022A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117909451A (en) * | 2024-03-18 | 2024-04-19 | 中国电子技术标准化研究院 | Question and answer result tracing method, device, equipment, medium and program product |
| CN118332093A (en) * | 2024-06-12 | 2024-07-12 | 河北比特聚客科技有限公司 | Intelligent customer service question-answering system and method based on large model processing |
-
2023
- 2023-11-24 CN CN202311585275.1A patent/CN117575022A/en active Pending
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117909451A (en) * | 2024-03-18 | 2024-04-19 | 中国电子技术标准化研究院 | Question and answer result tracing method, device, equipment, medium and program product |
| CN117909451B (en) * | 2024-03-18 | 2024-06-28 | 中国电子技术标准化研究院 | Question and answer result tracing method, device, equipment, medium and program product |
| CN118332093A (en) * | 2024-06-12 | 2024-07-12 | 河北比特聚客科技有限公司 | Intelligent customer service question-answering system and method based on large model processing |
| CN118332093B (en) * | 2024-06-12 | 2024-09-24 | 河北比特聚客科技有限公司 | Intelligent customer service question-answering system and method based on large model processing |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11966389B2 (en) | Natural language to structured query generation via paraphrasing | |
| US20240020538A1 (en) | Systems and methods for real-time search based generative artificial intelligence | |
| US11100052B2 (en) | System for classification based on user actions | |
| US11163936B2 (en) | Interactive virtual conversation interface systems and methods | |
| CN117575022A (en) | Intelligent document question-answering method, device, equipment, medium and program product | |
| CN111194442A (en) | Ranking documents based on semantic richness of the documents | |
| US11423232B2 (en) | Systems and methods for deploying computerized conversational agents | |
| US11379527B2 (en) | Sibling search queries | |
| CN118132305A (en) | Log analysis method, device, equipment, medium and program product | |
| KR102682244B1 (en) | Method for learning machine-learning model with structured ESG data using ESG auxiliary tool and service server for generating automatically completed ESG documents with the machine-learning model | |
| US11688393B2 (en) | Machine learning to propose actions in response to natural language questions | |
| CN118228015A (en) | Data processing method, apparatus, device, medium, and program product | |
| CN117952584A (en) | Information recommendation method and device, electronic equipment and storage medium | |
| US20220335227A1 (en) | Machine translation guided by reference documents | |
| EP4466630A1 (en) | Generating output sequences with inline evidence using language model neural networks | |
| US20210406294A1 (en) | Relevance approximation of passage evidence | |
| CN114580533A (en) | Method, apparatus, device, medium, and program product for training feature extraction model | |
| CN118093821A (en) | Work order processing method, apparatus, device, medium, and program product | |
| US20250005026A1 (en) | Methods and systems for search and ranking of code snippets using machine learning models | |
| US11074420B2 (en) | Self-learning input-driven brokering of language-translation engines | |
| US20220067539A1 (en) | Knowledge induction using corpus expansion | |
| CN118779427A (en) | Text determination method, device, electronic device and medium | |
| CN118673159A (en) | Method, apparatus, device, medium and program product for determining answers to questions | |
| CN118193732A (en) | Question labeling method, device, electronic equipment and storage medium | |
| CN118673122A (en) | Customer service question answering method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |