WO2022160442A1

WO2022160442A1 - Answer generation method and apparatus, electronic device, and readable storage medium

Info

Publication number: WO2022160442A1
Application number: PCT/CN2021/082863
Authority: WO
Inventors: 李雷来; 王健宗; 瞿晓阳
Original assignee: 平安科技（深圳）有限公司
Priority date: 2021-01-28
Filing date: 2021-03-25
Publication date: 2022-08-04
Also published as: CN112800178A

Abstract

The present application relates to intelligent decision making. Disclosed is an answer generation method, comprising: establishing a computer cluster, and controlling the computer cluster to execute first distributed training on an initial question answering model on the basis of a first corpus to obtain a first question answering model; controlling the computer cluster to execute second distributed training on the first question answering model on the basis of a second corpus to obtain a second question answering model; obtaining a target question carried by the answer generation request, obtaining a target text corresponding to the target question from a third database, and inputting the target text and the target question into the second question answering model to obtain the probability that each word in the target text is an answer starting vocabulary to the target question and the probability that each word in the target text is an answer ending vocabulary to the target question; and on the basis of the probability of the answer starting vocabulary and the probability of the answer ending vocabulary, determining a target answer corresponding to the target question. The present application further provides an answer generation apparatus, an electronic device, and a readable storage medium. According to the present application, answer generation accuracy is improved.

Description

Answer generation method, apparatus, electronic device and readable storage medium

This application claims the priority of the Chinese patent application with the application number CN202110124138.2 and the title of "Answer Generation Method, Device, Electronic Device and Readable Storage Medium" filed with the Chinese Patent Office on January 28, 2021, the entire contents of which are Incorporated herein by reference.

technical field

The present application relates to the field of intelligent decision-making, and in particular, to an answer generation method, apparatus, electronic device, and readable storage medium.

Background technique

With the advent of the information age, the application of intelligent question and answer in people's lives is becoming more and more extensive. For example, a company has developed a new product. When a user consults the product, the intelligent question and answer model can be used from the product manual. Intelligently extract the answer corresponding to the user's question and feed it back to the user.

The inventor realizes that in order to obtain an intelligent question answering model with good performance, a large amount of corpus is usually required to train a question answering model with a huge amount of parameters. However, a lot of computing resources need to be invested in the training process, and the training process is relatively long, due to cost and time constraints. , people usually reduce the corpus and model structure, but this method makes the performance of the trained intelligent question answering model not high enough, and the accuracy of the answers matched by the model is not high. Therefore, an answer generation method is urgently needed to improve the accuracy of answer generation.

SUMMARY OF THE INVENTION

The answer generation methods provided in this application include:

establishing a computer cluster, obtaining the first corpus from the first database, and controlling the computer cluster to perform the first distributed training on the initial question-answer model based on the first corpus to obtain the first question-answer model;

Obtain a second corpus from the second database, and control the computer cluster to perform second distributed training on the first question answering model based on the second corpus to obtain a second question answering model;

Parse the user's request based on the answer sent by the client, obtain the target question carried by the request, perform word segmentation and entity recognition processing on the target question, obtain the entity recognition result, and obtain the entity recognition result from the third database. the target text;

Input the target text and target question into the second question answering model, and obtain the probability that each word in the target text is the starting word of the answer of the target question and the probability of the ending word of the answer, based on the starting word of the answer The probability of the word and the probability of the answer end word determine the target answer corresponding to the target question.

The present application also provides an answer generation device, the device comprising:

The first training module is used to establish a computer cluster, obtain the first corpus from the first database, and control the computer cluster to perform the first distributed training on the initial question-answer model based on the first corpus to obtain the first question-and-answer model;

a second training module, configured to obtain a second corpus from a second database, and control the computer cluster to perform second distributed training on the first question answering model based on the second corpus to obtain a second question answering model;

The entity recognition module is used to parse the user's request based on the answer generated by the client, obtain the target question carried by the request, perform word segmentation and entity recognition processing on the target question, obtain the entity recognition result, and obtain and match from the third database. the target text matched by the entity recognition result;

An answer determination module, for inputting the target text and the target question into the second question answering model, to obtain the probability that each word in the target text is the starting word of the answer to the target question and the probability of the ending word of the answer, The target answer corresponding to the target question is determined based on the probability of the start word of the answer and the probability of the end word of the answer.

The present application also provides an electronic device, the electronic device comprising:

at least one processor; and,

a memory communicatively coupled to the at least one processor; wherein,

The memory stores an answer generation program executable by the at least one processor, the answer generation program being executed by the at least one processor to enable the at least one processor to perform the following steps:

Parse the user's request based on the answer generated by the client, obtain the target question carried by the request, perform word segmentation and entity recognition processing on the target question, obtain an entity recognition result, and obtain a match with the entity recognition result from a third database the target text;

The present application also provides a computer-readable storage medium, where an answer generation program is stored on the computer-readable storage medium, and the answer generation program can be executed by one or more processors to realize the following steps:

Description of drawings

1 is a schematic flowchart of an answer generation method provided by an embodiment of the present application;

FIG. 2 is a schematic block diagram of an answer generating apparatus provided by an embodiment of the present application;

3 is a schematic structural diagram of an electronic device for implementing an answer generation method provided by an embodiment of the present application;

The realization, functional characteristics and advantages of the purpose of the present application will be further described with reference to the accompanying drawings in conjunction with the embodiments.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application will be described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, but not to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

It should be noted that the descriptions involving "first", "second", etc. in this application are only for the purpose of description, and should not be construed as indicating or implying their relative importance or implying the number of indicated technical features . Thus, a feature delimited with "first", "second" may expressly or implicitly include at least one of that feature. In addition, the technical solutions between the various embodiments can be combined with each other, but must be based on the realization by those of ordinary skill in the art. When the combination of technical solutions is contradictory or cannot be realized, it should be considered that the combination of such technical solutions does not exist. , is not within the scope of protection claimed in this application.

The present application provides an answer generation method. Referring to FIG. 1 , it is a schematic flowchart of an answer generation method provided by an embodiment of the present application. The method may be performed by an electronic device, which may be implemented by software and/or hardware.

In this embodiment, the answer generation method includes:

S1. Establish a computer cluster, obtain a first corpus from a first database, and control the computer cluster to perform a first distributed training on an initial question-answer model based on the first corpus to obtain a first question-and-answer model.

In this embodiment, by aggregating multiple computers to establish a computer cluster, and using virtualization technology (for example, using a docker container for virtualization processing) to configure the environment required for training for the computing nodes in the computer cluster, distributed parallel training can be realized, Effectively improve training efficiency.

In order to obtain a question and answer model with good performance, the amount of data obtained in the first corpus is relatively large. The first corpus includes public data obtained from Baidu Know, Baidu Encyclopedia and other channels, as well as data in specific fields crawled by crawler programs, such as Data in healthcare, finance and sports.

The controlling the computer cluster to perform the first distributed training on the initial question answering model based on the first corpus includes steps A11-A13:

A11. Set the maximum iteration round of the first distributed training, perform masking and labeling processing on the first corpus, and obtain a third corpus with a label;

In this embodiment, performing masking and tagging processing on the first corpus to obtain a third corpus with tags, including steps B11-B14:

B11. Extract a first preset number of first texts from the first corpus, randomly adjust the order of sentences in the first text, take the adjusted text as a first sample, and use the first text as a first sample. The sentence order of the first sample is taken as the label of the first sample, and the set of the first sample carrying the label is taken as the first sample set;

B12. Extract a second preset number of second texts from the first corpus, randomly mask a third preset number of words in the second text, and use the masked text as a second sample, which will be masked The words of the second sample are used as the label of the second sample, and the set of the second samples carrying the label is used as the second sample set;

B13. Extract a fourth preset number of third texts and a fifth preset number of fourth texts from the first corpus, and randomly use the sixth preset number of sentences in the fourth text from other texts The extracted sentences are replaced to obtain the replaced text, the label of the replaced text is set to a first numerical value (for example, 0), and the label of the third text is set to a second numerical value (for example, 1), Taking the set of the replaced text and the third text with the label as the third sample set;

B14. Use the set of the first sample set, the second sample set and the third sample set as the third corpus.

A12. Acquire hardware resource information of each computing node in the computer cluster, split the third corpus into multiple sub-corpora based on the hardware resource information, and distribute the multiple sub-corpora to the respective computing nodes to for each computing node to train the initial question answering model based on the sub-corpus;

In this embodiment, the third corpus may be distributed to each computing node according to the quantity of a certain hardware resource (for example, a graphics card), or the total hardware resource of each computing node may be calculated according to the quantity of each hardware resource and its corresponding weight Score, distribute the third corpus to each computing node based on the total score.

In this embodiment, the initial question answering model is a bert model, and the structure is a 12-layer superimposed transformer structure.

A13. Receive the model gradients fed back by the respective computing nodes, update the model parameters of the initial question answering model based on the model gradients, and send the updated model parameters to the respective computing nodes for the computing nodes to use based on the The updated model parameters update the initial question-and-answer model, and continue training based on the updated initial question-and-answer model, and when the iteration reaches the maximum number of iteration rounds, the first distributed training ends.

Optionally, the calculation formula of the model gradient is:

Among them, g _ti is the model gradient corresponding to the i-th computing node in the computer cluster in the t-th iteration, and T _ti is the total number of training samples in the sub-corpus of the i-th computing node in the computer cluster in the t-round iteration , x _t(ij) is the j-th sample participating in training in the sub-corpus of the i-th computing node in the computer cluster in the t-th iteration, and s _t(ij) is the i-th calculation in the computer cluster in the t-round iteration The label of the jth sample participating in the training in the sub-corpus of the node, and l() is the output of the initial question answering model.

The updating of the model parameters of the initial question answering model based on the model gradient includes steps C11-C13:

C11. Calculate the average value of the gradient of the model, substitute the average value into the convergence parameter calculation formula to calculate the convergence parameter of the initial question and answer model, and substitute the average value into the update parameter calculation formula to calculate the initial question answer model. update parameters;

The calculation formula of the convergence parameter is:

Among them, m _t is the convergence parameter in the t-th iteration, α is the first equilibrium hyperparameter with a fixed value (usually 0.9), m _t-1 is the convergence parameter in the t-1 iteration, g _t is the average value of the model gradient at the t-th iteration, and α _t is the second balanced hyperparameter learned at the t-th iteration.

The update parameter calculation formula is:

Among them, v _t is the update parameter in the t-th iteration, β is the third equilibrium hyperparameter with a fixed value (usually 0.9), v _t-1 is the update parameter in the t-1 iteration, g _t is the mean value of the model gradient at the t-th iteration, and β _t is the fourth balanced hyperparameter learned at the t-th iteration.

The purpose of introducing the convergence parameter in this embodiment is to speed up the convergence of the model at the position/dimension where the gradient changes are small, and the purpose of introducing the update parameter is to obtain better model parameters through training.

C12. Substitute the convergence parameters and update parameters into the scaling ratio calculation formula to calculate the scaling ratio of the initial question answering model, and substitute the initial model parameters and initial learning rate of the initial question answering model into the learning rate calculation formula to calculate the initial question answering model. The new learning rate for the question answering model;

The scaling ratio calculation formula is:

Among them, r _t is the scaling rate at the t-th iteration, m _t is the convergence parameter at the t-th iteration, v _t is the update parameter at the t-th iteration, and ∈ is the fifth equilibrium hyperparameter with a fixed value.

The scaling ratio determined by the convergence parameter and the update parameter is used to adjust the scaling ratio of the model parameters.

The learning rate calculation formula is:

U _ti =u _t *min{max(‖p _ti ‖,γ),δ}

Among them, _ut is the learning rate of the initial question answering model in the t-th iteration, U _ti is the new learning rate of the i-th layer parameters of the initial question-answer model in the t-th iteration, and p _ti is the initial question-answer model in the t-th iteration. i-layer model parameters, γ is the sixth equilibrium hyperparameter whose value has been fixed, and δ is the seventh equilibrium hyperparameter whose value has been fixed.

In this embodiment, the learning rate is automatically adjusted with the parameters of the current layer of the model through the above learning rate calculation formula, so that better model parameters can be obtained by training, and the answer matched by the final question answering model is more accurate.

C13. Substitute the initial model parameters, the scaling rate and the new learning rate into a model parameter calculation formula to calculate new model parameters of the initial question answering model.

The model parameter calculation formula is:

Among them, p _(t+1)i is the model parameter of the i-th layer of the initial question answering model in the t+1 round of iteration, p _ti is the model parameter of the i-th layer of the initial question-answer model in the t-th iteration, and U _ti is the t-th layer The new learning rate of the i-th layer parameters of the initial question answering model during the iteration, r _t is the scaling rate at the t-th iteration, and λ is the eighth equilibrium hyperparameter whose value has been fixed.

The sending of the updated model parameters to each computing node includes steps D11-D13:

D11, performing quantization and compression processing on the updated model parameters to obtain compressed model parameters;

Due to the large number of parameters in each layer of the model and the huge number of overall parameters, even under the network bandwidth of 100M, it takes a long time to send the updated model parameters to each computing node. The model parameters are quantized and compressed to reduce the amount of transmitted data and improve transmission efficiency.

The quantization process uses float16 quantization. For a floating point number, 4-byte identifiers are required under normal circumstances, and 2-byte (16-bit) identifiers are used to reduce the accuracy of the model in the case of large-scale training. However, The network transfer speed can be doubled.

The compression processing adopts the method of sparse compression and storage to change the sparse matrix into a dense matrix. Most of the parameters of the model exist in the form of 2-dimensional or 3-dimensional matrices, especially after a small number of iterations, the matrix is often a sparse matrix. A matrix, that is, there are a large number of 0 elements in the matrix, can be processed by sparse row storage or column storage to eliminate the 0 elements in the matrix.

D12. Calculate the standard information digest value of the compressed model parameter;

D13. Obtain the public key corresponding to each computing node from the fourth database, encrypt the compressed model parameters and the standard information digest value using the public key, obtain ciphertext data, and distribute the ciphertext data to the corresponding computing node.

In this embodiment, the encryption public key corresponding to each computing node is stored in the fourth database, and the encryption private key is stored by each computing node. By encrypting and transmitting the updated model parameters, the security of the updated model parameters is ensured At the same time, the ciphertext of the standard information digest value of the updated model parameters is also transmitted. After each computing node decrypts the ciphertext data with its own private key, it calculates the information digest value of the decrypted data, and compares the calculated information digest value. Whether it is consistent with the standard information digest value to ensure that the ciphertext data has not been tampered with, further ensuring the security of the updated model parameters.

S2, obtaining a second corpus from a second database, and controlling the computer cluster to perform second distributed training on the first question-answering model based on the second corpus to obtain a second question-answering model;

The process of the second distributed training is basically the same as that of the first distributed training, and only the training samples and training tasks are different. The training samples are the text extracted from the second corpus and the questions set based on the extracted text. The label of is the pre-set answer for the question, the training task is to predict the answer to the question from the extracted text, and the training objective is that the similarity between the predicted answer and the answer in the label is greater than a preset threshold.

S3. Parse the user's request for generating an answer based on the answer sent by the client, obtain the target question carried by the request, perform word segmentation and entity recognition processing on the target question, obtain an entity recognition result, and obtain and identify the entity from a third database. The target text that the result matches.

For example, the target question is "how long is the claim period for health insurance", and the word sequence obtained after performing word segmentation on the target question is {health insurance, yes, claim period, yes, how long}, and the entity recognition model is used to perform entity recognition on the word sequence. Identify, identify and obtain the entity name "health insurance", the text corresponding to each entity name (for example, the instructions corresponding to each type of insurance) is pre-stored in the third database, and the instructions corresponding to health insurance in the third database are used as the target corresponding to the target question text.

S4. Input the target text and the target question into the second question answering model, and obtain the probability that each word in the target text is the starting word of the answer of the target question and the probability of the ending word of the answer, based on the answer The probability of the start word and the probability of the answer end word determine the target answer corresponding to the target question.

As can be seen from the above embodiments, the answer generation method proposed by the present application, first, establishes a computer cluster, obtains the first corpus from the first database, controls the computer cluster to perform the first distributed training on the initial question answering model based on the first corpus, and obtains the first corpus. a question and answer model; then, obtain a second corpus from the second database, and control the computer cluster to perform second distributed training on the first question and answer model based on the second corpus to obtain a second question and answer model; then, obtain the target carried by the answer generation request Then, perform word segmentation and entity recognition processing on the target question, obtain the entity recognition result, and obtain the target text matching the entity recognition result from the third database; finally, input the target text and the target question into the second question answering model, and get the target text in the target text. Each word is the probability of the start word of the answer and the probability of the end word of the answer, and the target answer corresponding to the target question is determined based on the probability of the start word of the answer and the probability of the end word of the answer. This solution implements the first distributed training and the second distributed training through the computer cluster, and realizes the training of the question and answer model with a huge amount of parameters by using massive corpus in a relatively short period of time. Because the corpus and model structure are not reduced, the training is guaranteed. The resulting high performance of the second question answering model and the high accuracy of the answers generated by the second question answering model. Therefore, the present application improves the answer generation accuracy.

As shown in FIG. 2 , it is a schematic block diagram of an answer generating apparatus according to an embodiment of the present application.

The answer generating apparatus 100 described in this application can be installed in an electronic device. According to the implemented functions, the answer generation apparatus 100 may include a first training module 110 , a second training module 120 , an entity recognition module 130 and an answer determination module 140 . The modules described in this application may also be referred to as units, which refer to a series of computer program segments that can be executed by the processor of an electronic device and can perform fixed functions, and are stored in the memory of the electronic device.

In this embodiment, the functions of each module/unit are as follows:

The first training module 110 is configured to establish a computer cluster, obtain a first corpus from a first database, and control the computer cluster to perform a first distributed training on an initial question-answer model based on the first corpus to obtain a first question-answer model.

The controlling the computer cluster to perform the first distributed training on the initial question answering model based on the first corpus includes steps A21-A23:

A21. Set the maximum iteration round of the first distributed training, perform masking and labeling processing on the first corpus, and obtain a third corpus with a label;

In this embodiment, performing masking and tagging processing on the first corpus to obtain a third corpus with tags, including steps B21-B24:

B21. Extract a first preset number of first texts from the first corpus, randomly adjust the order of sentences in the first text, take the adjusted text as a first sample, and use the first text as a first sample. The sentence order of the first sample is taken as the label of the first sample, and the set of the first sample carrying the label is taken as the first sample set;

B22. Extract a second preset number of second texts from the first corpus, randomly mask a third preset number of words in the second text, and use the masked text as a second sample, which will be masked The words of the second sample are used as the label of the second sample, and the set of the second samples carrying the label is used as the second sample set;

B23. Extract a fourth preset number of third texts and a fifth preset number of fourth texts from the first corpus, and randomly use the sixth preset number of sentences in the fourth text from other texts The extracted sentences are replaced to obtain the replaced text, the label of the replaced text is set to a first numerical value (for example, 0), and the label of the third text is set to a second numerical value (for example, 1), Taking the set of the replaced text and the third text with the label as the third sample set;

B24. Use the set of the first sample set, the second sample set and the third sample set as the third corpus.

A22. Obtain hardware resource information of each computing node in the computer cluster, split the third corpus into multiple sub-corpora based on the hardware resource information, and distribute the multiple sub-corpora to the respective computing nodes to for each computing node to train the initial question answering model based on the sub-corpus;

A23. Receive the model gradients fed back by the respective computing nodes, update the model parameters of the initial question answering model based on the model gradients, and send the updated model parameters to the respective computing nodes for the computing nodes to use based on The updated model parameters update the initial question-and-answer model, and continue training based on the updated initial question-and-answer model, and when the iteration reaches the maximum number of iteration rounds, the first distributed training ends.

Optionally, the calculation formula of the model gradient is:

The updating of the model parameters of the initial question answering model based on the model gradient includes steps C21-C23:

C21. Calculate the average value of the gradient of the model, substitute the average value into the convergence parameter calculation formula to calculate the convergence parameter of the initial question-and-answer model, and substitute the average value into the update parameter calculation formula to calculate the value of the initial question-and-answer model. update parameters;

The calculation formula of the convergence parameter is:

The update parameter calculation formula is:

C22. Substitute the convergence parameters and update parameters into the scaling rate calculation formula to calculate the scaling rate of the initial question answering model, and substitute the initial model parameters and initial learning rate based on the initial question answering model into the learning rate calculation formula to calculate the The new learning rate for the initial question answering model;

The scaling ratio calculation formula is:

The learning rate calculation formula is:

U _ti =u _t *min{max(‖p _ti ‖,γ),δ}

C23. Substitute the initial model parameters, the scaling rate and the new learning rate into a model parameter calculation formula to calculate new model parameters of the initial question answering model.

The model parameter calculation formula is:

Among them, p _(t+1)i is the model parameter of the i-th layer of the initial question answering model in the t+1 round of iteration, p _ti is the model parameter of the i-th layer of the initial question-answer model in the t-th iteration, and U _ti is the t-th layer The new learning rate of the i-th layer parameters of the initial question answering model during the iteration, r _t is the scaling rate during the t-th iteration, and λ is the eighth equilibrium hyperparameter whose value has been fixed.

Sending the updated model parameters to each computing node includes steps D21-D23:

D21, performing quantization and compression processing on the updated model parameters to obtain compressed model parameters;

The compression process adopts the method of sparse compression and storage, and changes the sparse matrix into a dense matrix. Most of the parameters of the model exist in the form of 2-dimensional or 3-dimensional matrices, especially after a small number of iterations, the matrix is often a sparse matrix. A matrix, that is, there are a large number of 0 elements in the matrix, can be processed by sparse row storage or column storage to eliminate the 0 elements in the matrix.

D22. Calculate the standard information digest value of the compressed model parameter;

D23. Obtain the public key corresponding to each computing node from the fourth database, encrypt the compressed model parameters and the standard information digest value by using the public key, obtain ciphertext data, and distribute the ciphertext data to the corresponding computing node.

In this embodiment, the encryption public key corresponding to each computing node is stored in the fourth database, and the encryption private key is stored by each computing node itself. By encrypting and transmitting the updated model parameters, the security of the updated model parameters is ensured At the same time, the ciphertext of the standard information digest value of the updated model parameters is also transmitted. After each computing node decrypts the ciphertext data with its own private key, it calculates the information digest value of the decrypted data, and compares the calculated information digest value. It is consistent with the standard information digest value to ensure that the ciphertext data has not been tampered with, which further ensures the security of the updated model parameters.

The second training module 120 is configured to obtain a second corpus from a second database, and control the computer cluster to perform second distributed training on the first question-answering model based on the second corpus to obtain a second question-answer model;

The entity identification module 130 is used to parse the user based on the answer generation request sent by the client, obtain the target question carried by the request, perform word segmentation and entity identification processing on the target question, obtain the entity identification result, and obtain from the third database target text that matches the entity recognition result.

The answer determination module 140 is configured to input the target text and the target question into the second question answering model, and obtain the probability that each word in the target text is the starting word of the answer of the target question and the probability of the ending word of the answer , and the target answer corresponding to the target question is determined based on the probability of the initial word of the answer and the probability of the end word of the answer.

As shown in FIG. 3 , it is a schematic structural diagram of an electronic device for implementing an answer generation method provided by an embodiment of the present application.

The electronic device 1 is a device that can automatically perform numerical calculation and/or information processing according to pre-set or stored instructions. The electronic device 1 may be a computer, a single network server, a server group composed of multiple network servers, or a cloud composed of a large number of hosts or network servers based on cloud computing, where cloud computing is a type of distributed computing, A super virtual computer consisting of a collection of loosely coupled computers.

In this embodiment, the electronic device 1 includes, but is not limited to, a memory 11, a processor 12, and a network interface 13 that can be communicatively connected to each other through a system bus, and the memory 11 stores an answer generation program 10, the answer generation program 10 is executable by the processor 12 . FIG. 3 only shows the electronic device 1 having the components 11-13 and the answer generating program 10. Those skilled in the art can understand that the structure shown in FIG. 3 does not constitute a limitation on the electronic device 1, and may include a Fewer or more components are shown, or some components are combined, or a different arrangement of components.

The memory 11 includes a memory and at least one type of readable storage medium. The memory provides a cache for the operation of the electronic device 1; the readable storage medium can be, for example, flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM) ), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), programmable read only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. non-volatile storage media. In some embodiments, the readable storage medium may be an internal storage unit of the electronic device 1, such as a hard disk of the electronic device 1; in other embodiments, the non-volatile storage medium may also be an external storage unit of the electronic device 1 A storage device, such as a pluggable hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash memory card (Flash Card), etc. equipped on the electronic device 1. In this embodiment, the readable storage medium of the memory 11 is generally used to store the operating system and various application software installed in the electronic device 1 , for example, to store the code of the answer generation program 10 in an embodiment of the present application. In addition, the memory 11 can also be used to temporarily store various types of data that have been output or will be output.

In some embodiments, the processor 12 may be a central processing unit (Central Processing Unit, CPU), a controller, a microcontroller, a microprocessor, or other data processing chips. The processor 12 is generally used to control the overall operation of the electronic device 1, such as performing control and processing related to data interaction or communication with other devices. In this embodiment, the processor 12 is configured to run the program code or process data stored in the memory 11, for example, run the answer generation program 10 and the like.

The network interface 13 may include a wireless network interface or a wired network interface, and the network interface 13 is used to establish a communication connection between the electronic device 1 and a client (not shown in the figure).

Optionally, the electronic device 1 may further include a user interface, and the user interface may include a display (Display), an input unit such as a keyboard (Keyboard), and an optional user interface may also include a standard wired interface and a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode, organic light-emitting diode) touch device, and the like. The display may also be appropriately called a display screen or a display unit, which is used for displaying information processed in the electronic device 1 and for displaying a visualized user interface.

It should be understood that the embodiments are only used for illustration, and are not limited by this structure in the scope of the patent application.

The answer generation program 10 stored in the memory 11 in the electronic device 1 is a combination of multiple instructions, and when running in the processor 12, can realize:

Specifically, for the specific implementation method of the above-mentioned answer generating program 10 by the processor 12, reference may be made to the description of the relevant steps in the corresponding embodiment of FIG. 1 , which is not repeated here.

Further, if the modules/units integrated in the electronic device 1 are implemented in the form of software functional units and sold or used as independent products, they may be stored in a computer-readable storage medium. The computer-readable medium may be non-volatile or non-volatile. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, removable hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory) .

An answer generation program 10 is stored on the computer-readable storage medium, and the answer generation program 10 can be executed by one or more processors to realize the following steps:

In the several embodiments provided in this application, it should be understood that the disclosed apparatus, apparatus and method may be implemented in other manners. For example, the apparatus embodiments described above are only illustrative. For example, the division of the modules is only a logical function division, and there may be other division manners in actual implementation.

The modules described as separate components may or may not be physically separated, and components shown as modules may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional module in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware, or can be implemented in the form of hardware plus software function modules.

It will be apparent to those skilled in the art that the present application is not limited to the details of the above-described exemplary embodiments, but that the present application can be implemented in other specific forms without departing from the spirit or essential characteristics of the present application.

Accordingly, the embodiments are to be regarded in all respects as illustrative and not restrictive, and the scope of the application is to be defined by the appended claims rather than the foregoing description, which is therefore intended to fall within the scope of the claims. All changes within the meaning and scope of the equivalents of , are included in this application. Any reference signs in the claims shall not be construed as limiting the involved claim.

The blockchain referred to in this application is a new application mode of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanism, and encryption algorithm. Blockchain, essentially a decentralized database, is a series of data blocks associated with cryptographic methods. Each data block contains a batch of network transaction information to verify its Validity of information (anti-counterfeiting) and generation of the next block. The blockchain can include the underlying platform of the blockchain, the platform product service layer, and the application service layer.

Furthermore, it is clear that the word "comprising" does not exclude other units or steps and the singular does not exclude the plural. Several units or means recited in the system claims can also be realized by one unit or means by means of software or hardware. Second-class terms are used to denote names and do not denote any particular order.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application rather than limitations. Although the present application has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present application can be Modifications or equivalent substitutions can be made without departing from the spirit and scope of the technical solutions of the present application.

Claims

An answer generation method, wherein the method comprises:

establishing a computer cluster, obtaining the first corpus from the first database, and controlling the computer cluster to perform the first distributed training on the initial question-answer model based on the first corpus to obtain the first question-answer model;

Obtain a second corpus from the second database, and control the computer cluster to perform second distributed training on the first question answering model based on the second corpus to obtain a second question answering model;

Parse the user's request based on the answer generated by the client, obtain the target question carried by the request, perform word segmentation and entity recognition processing on the target question, obtain an entity recognition result, and obtain a match with the entity recognition result from a third database the target text;

Input the target text and target question into the second question answering model, and obtain the probability that each word in the target text is the starting word of the answer of the target question and the probability of the ending word of the answer, based on the starting word of the answer The probability of the word and the probability of the answer end word determine the target answer corresponding to the target question.
The answer generation method according to claim 1, wherein the controlling the computer cluster to perform the first distributed training on the initial question answering model based on the first corpus comprises:

Setting the maximum iteration round of the first distributed training, performing masking and labeling processing on the first corpus, and obtaining a third corpus with a label;

Obtain hardware resource information of each computing node in the computer cluster, split the third corpus into multiple sub-corpora based on the hardware resource information, and distribute the multiple sub-corpora to the respective computing nodes for all each computing node trains the initial question answering model based on the sub-corpus;

Receive the model gradients fed back by the respective computing nodes, update the model parameters of the initial question-and-answer model based on the model gradients, and send the updated model parameters to the respective computing nodes for the respective computing nodes to use based on the The initial question answering model is updated with the updated model parameters, and the training is continued based on the updated initial question answering model. When the iteration reaches the maximum number of iteration rounds, the first distributed training ends.
The answer generation method according to claim 2, wherein the updating the model parameters of the initial question answering model based on the model gradient comprises:

Calculate the average value of the gradient of the model, substitute the average value into the convergence parameter calculation formula to calculate the convergence parameter of the initial question-and-answer model, and substitute the average value into the update parameter calculation formula to calculate the update parameter of the initial question-and-answer model ;

Substitute the convergence parameters and update parameters into the scaling rate calculation formula to calculate the scaling rate of the initial question answering model, and substitute the initial model parameters and initial learning rate of the initial question answering model into the learning rate calculation formula to calculate the initial question answering model. The new learning rate of ;

Substituting the initial model parameters, the scaling rate and the new learning rate into a model parameter calculation formula to calculate new model parameters of the initial question answering model.
The answer generation method according to claim 2, wherein, performing masking and labeling processing on the first corpus to obtain a third corpus with tags, comprising:

Extract a first preset number of first texts from the first corpus, randomly adjust the order of sentences in the first text, take the adjusted text as a first sample, and use the sentences in the first text as a first sample. The sequence is used as the label of the first sample, and the set of the first samples carrying the label is used as the first sample set;

Extracting a second preset number of second texts from the first corpus, randomly masking a third preset number of words in the second text, using the masked text as a second sample, and masking the masked words As the label of the second sample, the set of the second samples carrying the label is used as the second sample set;

A fourth preset number of third texts and a fifth preset number of fourth texts are extracted from the first corpus, and the sixth preset number of sentences in the fourth text are randomly selected from the sentences extracted from other texts. Sentence replacement, to obtain the replaced text, the label of the replaced text is set to the first numerical value, the label of the third text is set to the second numerical value, the replaced text and the third text that carry the label are set The set of , as the third sample set;

A set of the first sample set, the second sample set and the third sample set is used as the third corpus.
The answer generation method according to claim 2, wherein the sending the updated model parameters to each computing node comprises:

Quantizing and compressing the updated model parameters to obtain compressed model parameters;

calculating standard information digest values of the compressed model parameters;

Obtain the public key corresponding to each computing node from the fourth database, use the public key to encrypt the compressed model parameters and the standard information digest value, obtain ciphertext data, and distribute the ciphertext data to the corresponding calculate node.
The answer generation method according to claim 2, wherein the calculation formula of the model gradient is:

Among them, g ti is the model gradient corresponding to the i-th computing node in the computer cluster in the t-th iteration, and T ti is the total number of training samples in the sub-corpus of the i-th computing node in the computer cluster in the t-round iteration , x t(ij) is the j-th sample participating in training in the sub-corpus of the i-th computing node in the computer cluster in the t-th iteration, and s t(ij) is the i-th calculation in the computer cluster in the t-round iteration The label of the jth sample participating in the training in the sub-corpus of the node, and l() is the output of the initial question answering model.
An answer generating apparatus, wherein the apparatus comprises:

The first training module is used to establish a computer cluster, obtain the first corpus from the first database, and control the computer cluster to perform the first distributed training on the initial question-answer model based on the first corpus to obtain the first question-and-answer model;

a second training module, configured to obtain a second corpus from a second database, and control the computer cluster to perform second distributed training on the first question answering model based on the second corpus to obtain a second question answering model;

The entity recognition module is used to parse the user's request based on the answer generated by the client, obtain the target question carried by the request, perform word segmentation and entity recognition processing on the target question, obtain the entity recognition result, and obtain and match from the third database. the target text matched by the entity recognition result;

An answer determination module, for inputting the target text and the target question into the second question answering model, to obtain the probability that each word in the target text is the starting word of the answer to the target question and the probability of the ending word of the answer, The target answer corresponding to the target question is determined based on the probability of the start word of the answer and the probability of the end word of the answer.
The answer generating apparatus according to claim 7, wherein the controlling the computer cluster to perform the first distributed training on the initial question answering model based on the first corpus comprises:

Setting the maximum iteration round of the first distributed training, performing masking and labeling processing on the first corpus, and obtaining a third corpus with a label;

Obtain hardware resource information of each computing node in the computer cluster, split the third corpus into multiple sub-corpora based on the hardware resource information, and distribute the multiple sub-corpora to the respective computing nodes for all each computing node trains the initial question answering model based on the sub-corpus;

Receive the model gradients fed back by each of the computing nodes, update the model parameters of the initial question answering model based on the model gradients, and send the updated model parameters to the respective computing nodes for the computing nodes to use based on the The initial question answering model is updated with the updated model parameters, and the training is continued based on the updated initial question answering model. When the iteration reaches the maximum number of iteration rounds, the first distributed training ends.
An electronic device, wherein the electronic device comprises:

at least one processor; and,

a memory communicatively coupled to the at least one processor; wherein,

The memory stores an answer generation program executable by the at least one processor, the answer generation program being executed by the at least one processor to enable the at least one processor to perform the following steps:

A computer cluster is established, the first corpus is obtained from the first database, and the computer cluster is controlled to perform the first distributed training on the initial question and answer model based on the first corpus to obtain the first question and answer model;

Obtain a second corpus from the second database, and control the computer cluster to perform second distributed training on the first question answering model based on the second corpus to obtain a second question answering model;

Parse the user's request based on the answer generated by the client, obtain the target question carried by the request, perform word segmentation and entity recognition processing on the target question, obtain an entity recognition result, and obtain a match with the entity recognition result from a third database the target text;

Input the target text and target question into the second question answering model, and obtain the probability that each word in the target text is the starting word of the answer of the target question and the probability of the ending word of the answer, based on the starting word of the answer The probability of the word and the probability of the answer end word determine the target answer corresponding to the target question.
The electronic device of claim 9, wherein the controlling the computer cluster to perform a first distributed training on an initial question answering model based on the first corpus comprises:

Setting the maximum iteration round of the first distributed training, performing masking and labeling processing on the first corpus, and obtaining a third corpus with a label;

Obtain hardware resource information of each computing node in the computer cluster, split the third corpus into multiple sub-corpora based on the hardware resource information, and distribute the multiple sub-corpora to the respective computing nodes for all each computing node trains the initial question answering model based on the sub-corpus;

Receive the model gradients fed back by the respective computing nodes, update the model parameters of the initial question-and-answer model based on the model gradients, and send the updated model parameters to the respective computing nodes for the respective computing nodes to use based on the The initial question answering model is updated with the updated model parameters, and the training is continued based on the updated initial question answering model. When the iteration reaches the maximum number of iteration rounds, the first distributed training ends.
The electronic device according to claim 10, wherein the updating the model parameters of the initial question answering model based on the model gradient comprises:

Calculate the average value of the gradient of the model, substitute the average value into the convergence parameter calculation formula to calculate the convergence parameter of the initial question-and-answer model, and substitute the average value into the update parameter calculation formula to calculate the update parameter of the initial question-and-answer model ;

Substitute the convergence parameters and update parameters into the scaling rate calculation formula to calculate the scaling rate of the initial question answering model, and substitute the initial model parameters and initial learning rate of the initial question answering model into the learning rate calculation formula to calculate the initial question answering model. The new learning rate of ;

Substituting the initial model parameters, the scaling rate and the new learning rate into a model parameter calculation formula to calculate new model parameters of the initial question answering model.
The electronic device according to claim 10, wherein the masking and tagging processing is performed on the first corpus to obtain a third corpus with a tag, comprising:

Extract a first preset number of first texts from the first corpus, randomly adjust the order of sentences in the first text, take the adjusted text as a first sample, and use the sentences in the first text as a first sample. The sequence is used as the label of the first sample, and the set of the first samples carrying the label is used as the first sample set;

Extracting a second preset number of second texts from the first corpus, randomly masking a third preset number of words in the second text, using the masked text as a second sample, and masking the masked words As the label of the second sample, the set of the second samples carrying the label is used as the second sample set;

A fourth preset number of third texts and a fifth preset number of fourth texts are extracted from the first corpus, and the sixth preset number of sentences in the fourth text are randomly selected from the sentences extracted from other texts. Sentence replacement, to obtain the replaced text, the label of the replaced text is set to the first numerical value, the label of the third text is set to the second numerical value, the replaced text and the third text that carry the label are set The set of , as the third sample set;

A set of the first sample set, the second sample set and the third sample set is used as the third corpus.
The electronic device according to claim 10, wherein the sending the updated model parameters to each computing node comprises:

Quantizing and compressing the updated model parameters to obtain compressed model parameters;

calculating standard information digest values of the compressed model parameters;

Obtain the public key corresponding to each computing node from the fourth database, use the public key to encrypt the compressed model parameters and the standard information digest value, obtain ciphertext data, and distribute the ciphertext data to the corresponding calculate node.
The electronic device according to claim 10, wherein the calculation formula of the model gradient is:

Among them, g ti is the model gradient corresponding to the i-th computing node in the computer cluster in the t-th iteration, and T ti is the total number of training samples in the sub-corpus of the i-th computing node in the computer cluster in the t-round iteration , x t(ij) is the j-th sample participating in training in the sub-corpus of the i-th computing node in the computer cluster in the t-th iteration, and s t(ij) is the i-th calculation in the computer cluster in the t-round iteration The label of the jth sample participating in the training in the sub-corpus of the node, and l() is the output of the initial question answering model.
A computer-readable storage medium, wherein an answer generation program is stored on the computer-readable storage medium, and the answer generation program can be executed by one or more processors to realize the following steps:

establishing a computer cluster, obtaining the first corpus from the first database, and controlling the computer cluster to perform the first distributed training on the initial question-answer model based on the first corpus to obtain the first question-answer model;

Obtain a second corpus from the second database, and control the computer cluster to perform second distributed training on the first question answering model based on the second corpus to obtain a second question answering model;

Parse the user's request based on the answer generated by the client, obtain the target question carried by the request, perform word segmentation and entity recognition processing on the target question, obtain an entity recognition result, and obtain a match with the entity recognition result from a third database the target text;

Input the target text and target question into the second question answering model, and obtain the probability that each word in the target text is the starting word of the answer of the target question and the probability of the ending word of the answer, based on the starting word of the answer The probability of the word and the probability of the answer end word determine the target answer corresponding to the target question.
The computer-readable storage medium of claim 15, wherein the controlling the computer cluster to perform a first distributed training on an initial question answering model based on the first corpus comprises:

Setting the maximum iteration round of the first distributed training, performing masking and labeling processing on the first corpus, and obtaining a third corpus with a label;

Obtain hardware resource information of each computing node in the computer cluster, split the third corpus into multiple sub-corpora based on the hardware resource information, and distribute the multiple sub-corpora to the respective computing nodes for all each computing node trains the initial question answering model based on the sub-corpus;

Receive the model gradients fed back by the respective computing nodes, update the model parameters of the initial question-and-answer model based on the model gradients, and send the updated model parameters to the respective computing nodes for the respective computing nodes to use based on the The initial question answering model is updated with the updated model parameters, and the training is continued based on the updated initial question answering model. When the iteration reaches the maximum number of iteration rounds, the first distributed training ends.
The computer-readable storage medium of claim 16, wherein the updating the model parameters of the initial question answering model based on the model gradient comprises:

Calculate the average value of the gradient of the model, substitute the average value into the convergence parameter calculation formula to calculate the convergence parameter of the initial question-and-answer model, and substitute the average value into the update parameter calculation formula to calculate the update parameter of the initial question-and-answer model ;

Substitute the convergence parameters and update parameters into the scaling ratio calculation formula to calculate the scaling ratio of the initial question answering model, and substitute the initial model parameters and initial learning rate of the initial question answering model into the learning rate calculation formula to calculate the initial question answering model. The new learning rate of ;

Substitute the initial model parameters, the scaling rate and the new learning rate into the model parameter calculation formula to calculate the new model parameters of the initial question answering model.
The computer-readable storage medium according to claim 16, wherein the masking and tagging processing is performed on the first corpus to obtain a third corpus carrying a tag, comprising:

Extract a first preset number of first texts from the first corpus, randomly adjust the order of sentences in the first text, take the adjusted text as a first sample, and use the sentences in the first text as a first sample. The sequence is used as the label of the first sample, and the set of the first samples carrying the label is used as the first sample set;

Extracting a second preset number of second texts from the first corpus, randomly masking a third preset number of words in the second text, using the masked text as a second sample, and masking the masked words As the label of the second sample, the set of the second samples carrying the label is used as the second sample set;

A fourth preset number of third texts and a fifth preset number of fourth texts are extracted from the first corpus, and the sixth preset number of sentences in the fourth text are randomly selected from the sentences extracted from other texts. Sentence replacement, to obtain the replaced text, the label of the replaced text is set to the first numerical value, the label of the third text is set to the second numerical value, the replaced text and the third text that carry the label are set The set of , as the third sample set;

A set of the first sample set, the second sample set and the third sample set is used as the third corpus.
The computer-readable storage medium of claim 16, wherein the sending the updated model parameters to each computing node comprises:

Quantizing and compressing the updated model parameters to obtain compressed model parameters;

calculating standard information digest values of the compressed model parameters;

Obtain the public key corresponding to each computing node from the fourth database, use the public key to encrypt the compressed model parameters and the standard information digest value, obtain ciphertext data, and distribute the ciphertext data to the corresponding calculate node.
The computer-readable storage medium of claim 16, wherein the calculation formula of the model gradient is:

Among them, g ti is the model gradient corresponding to the i-th computing node in the computer cluster in the t-th iteration, and T ti is the total number of training samples in the sub-corpus of the i-th computing node in the computer cluster in the t-round iteration , x t(ij) is the j-th sample participating in training in the sub-corpus of the i-th computing node in the computer cluster in the t-th iteration, and s t(ij) is the i-th calculation in the computer cluster in the t-round iteration The label of the jth sample participating in the training in the sub-corpus of the node, and l() is the output of the initial question answering model.