CN110704597A

CN110704597A - Dialogue system reliability verification method, model generation method and device

Info

Publication number: CN110704597A
Application number: CN201910933174.6A
Authority: CN
Inventors: 王玉昕; 郑祺星
Original assignee: Beijing Kingsoft Internet Security Software Co Ltd
Current assignee: Beijing Kingsoft Internet Security Software Co Ltd
Priority date: 2019-09-29
Filing date: 2019-09-29
Publication date: 2020-01-17
Anticipated expiration: 2039-09-29
Also published as: CN110704597B

Abstract

The application provides a method for verifying reliability of a dialog system, a method and a device for generating a model, and belongs to the technical field of computer application. The reliability verification method of the dialogue system comprises the following steps: acquiring a first question-answer pair to be verified, wherein the first question-answer pair comprises a first question and a first answer; acquiring at least one reference question-answer pair from the corpus according to the similarity between the first question-answer pair and each second question-answer pair in the corpus; determining a first feature vector corresponding to the first question-answer pair according to a first vector corresponding to the first question-answer pair and each second vector corresponding to each reference question-answer; and coding and decoding the first characteristic vector by using a pre-generated check model to generate the matching degree between the first question sentence and the first answer sentence in the first question-answer pair. Therefore, by the method for verifying the reliability of the dialogue system, the authenticity of the reliability verification of the dialogue system is improved, the verification efficiency is improved, and the labor cost is saved.

Description

Dialogue system reliability verification method, model generation method and device

Technical Field

The present application relates to the field of computer application technologies, and in particular, to a method for verifying reliability of a dialog system, a method for generating a model, and an apparatus for generating a model.

Background

With the rapid development of artificial intelligence technology, various intelligent dialogue systems are becoming popular, such as service interaction robots. In practical application, how to judge whether the dialog system really answers the question of the user meets the user requirement so as to further improve the reliability of the dialog system is a problem to be solved urgently.

In the related art, the dialog data of the user and the dialog system are usually checked manually to determine the reliability of the man-machine dialog. However, because the dialogue data generated by the dialogue system is huge, the manual check is adopted, the authenticity cannot be guaranteed, the efficiency is low, and the labor cost is increased.

Disclosure of Invention

The method and the device for verifying the reliability of the dialogue system are used for solving the problems that in the related art, the reliability of the dialogue data of the dialogue system is verified in a manual mode, and due to the fact that the dialogue data generated by the dialogue system are huge, authenticity cannot be guaranteed, efficiency is low, and labor cost is improved.

An embodiment of an aspect of the present application provides a method for checking reliability of a dialog system, including: acquiring a first question-answer pair to be verified, wherein the first question-answer pair comprises a first question sentence and a first answer sentence; according to the similarity between the first question-answer pair and each second question-answer pair in the corpus, at least one reference question-answer pair is obtained from the corpus, and each second question-answer pair comprises a second question and a second answer sentence; determining a first feature vector corresponding to the first question-answer pair according to a first vector corresponding to the first question-answer pair and each second vector corresponding to each reference question-answer pair; and coding and decoding the first characteristic vector by using a pre-generated check model to generate the matching degree between the first question sentence and the first answer sentence in the first question-answer pair.

Optionally, in a possible implementation form of the embodiment of the first aspect, the obtaining, according to a similarity between the first question-answer pair and each second question-answer pair in a corpus, at least one reference question-answer pair from the corpus includes:

according to the similarity between the first question-answer pair and each second question-answer pair, obtaining a second question-answer pair with the highest similarity with the first question-answer pair from the corpus;

or,

and according to the similarity between the first question-answer pair and each second question-answer pair, acquiring the second question-answer pair with the highest similarity with the first question-answer pair and the nth second question-answer pair in the first M second question-answer pairs with the similarity with the first question-answer pair larger than a threshold value from the corpus, wherein M is a positive integer, and n is a positive integer larger than 1.

Optionally, in another possible implementation form of the embodiment of the first aspect, the obtaining, according to the similarity between the first question-answer pair and each second question-answer pair in the corpus, at least one reference question-answer pair from the corpus includes:

obtaining each suspected reference question-answer pair according to the similarity between the first question and each second question, and determining at least one reference question-answer pair according to the similarity between the first question and the second question in each suspected reference question-answer pair;

or,

and obtaining each suspected reference question-answer pair according to the similarity between the first answer sentence and each second answer sentence, and determining at least one reference question-answer pair according to the similarity between the first question sentence and the second question sentence in each suspected reference question-answer pair.

Optionally, in yet another possible implementation form of the embodiment of the first aspect, after the determining the first feature vector corresponding to the first question-answer pair, the method further includes:

expanding the first feature vector according to the similarity between the first question-answer pair and the at least one reference question-answer pair, the matching degree between a second question and a second question in each reference question-answer pair, the intention of the first question-answer pair, the intention of each reference question-answer pair, the field to which the first question-answer pair belongs, and/or the field to which each reference question-answer pair belongs to generate a second feature vector;

the encoding and decoding process of the first feature vector includes:

and carrying out encoding and decoding processing on the second feature vector.

Optionally, in another possible implementation form of the embodiment of the first aspect, the corpus set includes a first question-answer pair set in which matching degrees of the second question and the second answer are greater than or equal to a threshold, and a second question-answer pair set in which matching degrees of the second question and the second answer are less than the threshold;

before the encoding and decoding process of the second feature vector, the method further includes:

and updating the second characteristic vector according to the question-answer pair set to which each reference question-answer pair belongs.

Optionally, in another possible implementation form of the embodiment of the first aspect, before the determining the first feature vector corresponding to the first question-answer pair, the method further includes:

determining a vector corresponding to each second question sentence in the first question sentence and each reference question-answer pair respectively by adopting a first preset method;

and determining vectors corresponding to the first answer sentence and each second answer sentence in each reference question-answer pair by adopting a second preset method.

determining an initiator and a responder which respectively correspond to the first question-answer pair and each reference question-answer pair;

determining vectors corresponding to the first question and the first answer sentence respectively according to the initiator and the responder corresponding to the first question-answer pair;

and determining vectors corresponding to the second question sentence and the second answer sentence in each reference question-answer pair respectively according to the initiator and the responder corresponding to each reference question-answer pair.

In another aspect of the present application, a method for generating a reliability verification model of a dialog system includes: extracting training question-answer pairs from the corpus set, wherein each second question-answer pair in the corpus set comprises a second question and a second answer sentence, and a first matching degree exists between a target second question and the target second answer sentence in the training question-answer pairs; acquiring at least one reference question-answer pair according to the similarity between the training question-answer pair and other second question-answer pairs in the corpus; determining a third feature vector corresponding to the training question-answer pair according to the third vector corresponding to the training question-answer pair and each fourth vector corresponding to each reference question-answer pair; encoding and decoding the third feature vector by using a preset model, and determining a second matching degree corresponding to the training question-answer pair; and adjusting the preset model parameters according to the difference value between the first matching degree and the second matching degree to generate a verification model.

Optionally, in a possible implementation form of the embodiment of the second aspect, the obtaining, according to the similarity between the training question-answer pair and each of the other second question-answer pairs in the corpus, at least one reference question-answer pair from the corpus includes:

according to the similarity between the training question-answer pair and other second question-answer pairs, obtaining a second question-answer pair with the highest similarity with the training question-answer pair from the corpus set;

or,

and according to the similarity between the training question-answer pair and other second question-answer pairs, acquiring a second question-answer pair with the highest similarity with the training question-answer pair and a Kth second question-answer pair in the first L second question-answer pairs with the similarity with the training question-answer pair larger than a threshold value from the corpus set, wherein L is a positive integer, and K is a positive integer larger than 1.

Optionally, in another possible implementation form of the embodiment of the second aspect, the obtaining, according to the similarity between the training question-answer pair and each of the other second question-answer pairs, at least one reference question-answer pair from the corpus includes:

obtaining each suspected reference question-answer pair according to the similarity between the target second question and the second question in each other second question-answer pair, and determining at least one reference question-answer pair according to the similarity between the target first question and the second question in each suspected reference question-answer pair;

or,

and obtaining each suspected reference question-answer pair according to the similarity between the target second question and the second question in each other second question-answer pair, and determining at least one reference question-answer pair according to the similarity between the second target question and the second question in each suspected reference question-answer pair.

Optionally, in yet another possible implementation form of an embodiment of the second aspect, after the determining the third feature vector corresponding to the training question-answer pair, the method further includes:

expanding the third feature vector according to the similarity between the training question-answer pair and the at least one reference question-answer pair, the matching degree between a second question and a second answer in each reference question-answer pair, the intention of the training question-answer pair, the intention of each reference question-answer pair, the field to which the training question-answer pair belongs, and/or the field to which each reference question-answer pair belongs to, and generating a fourth feature vector;

the encoding and decoding processing of the third feature vector by using a preset model includes:

and performing encoding and decoding processing on the fourth feature vector by using a preset model.

Optionally, in another possible implementation form of the embodiment of the second aspect, the corpus set includes a first question-answer pair set in which a matching degree between the second question and the second answer sentence is greater than or equal to a threshold, and a second question-answer pair set in which a matching degree between the second question and the second answer sentence is less than a threshold;

before the encoding and decoding process of the fourth feature vector, the method further includes:

and updating the fourth feature vector according to the question-answer pair set to which each reference question-answer pair belongs.

Optionally, in another possible implementation form of the embodiment of the second aspect, before the determining the third feature vector corresponding to the training question-answer pair, the method further includes:

determining vectors corresponding to the target second question and each second question in each reference question-answer pair by adopting a first preset method;

and determining the vectors corresponding to the target answer sentence and each second answer sentence in each reference question-answer pair by adopting a second preset method.

determining an initiator and a responder which respectively correspond to the training question-answer pairs and each reference question-answer pair;

determining vectors corresponding to the target second question sentence and the target second answer sentence respectively according to the initiator and the responder corresponding to the training question-answer pair;

and determining vectors corresponding to the second question sentence and the second answer sentence in each reference question-answer pair respectively according to the initiator and the responder corresponding to each reference question-answer pair. In another aspect of the present application, an apparatus for verifying reliability of a dialog system includes: the system comprises a first acquisition module, a second acquisition module and a verification module, wherein the first acquisition module is used for acquiring a first question-answer pair to be verified, and the first question-answer pair comprises a first question sentence and a first answer sentence; a second obtaining module, configured to obtain at least one reference question-answer pair from the corpus according to a similarity between the first question-answer pair and each second question-answer pair in the corpus, where each second question-answer pair includes a second question sentence and a second answer sentence; a first determining module, configured to determine a first feature vector corresponding to the first question-answer pair according to a first vector corresponding to the first question-answer pair and each second vector corresponding to each reference question-answer pair; and the first generation module is used for performing coding and decoding processing on the first feature vector by using a pre-generated check model to generate the matching degree between the first question sentence and the first answer sentence in the first question-answer pair.

Optionally, in a possible implementation form of the embodiment of the third aspect, the second obtaining module is specifically configured to:

or,

Optionally, in another possible implementation form of the embodiment of the third aspect, the second obtaining module is further configured to:

or,

and obtaining each suspected reference question-answer pair according to the similarity between the first question and each second question, and determining at least one reference question-answer pair according to the similarity between the first question and each second question in each suspected reference question-answer pair.

Optionally, in another possible implementation form of the embodiment of the third aspect, the apparatus further includes:

a second generating module, configured to expand the first feature vector according to a similarity between the first question-answer pair and the at least one reference question-answer pair, a matching degree between a second question and a second question in each reference question-answer pair, an intention of the first question-answer pair, an intention of each reference question-answer pair, a domain to which the first question-answer pair belongs, and/or a domain to which each reference question-answer pair belongs, and generate a second feature vector;

the first generation module is specifically configured to:

and carrying out encoding and decoding processing on the second feature vector.

Optionally, in a further possible implementation form of the embodiment of the third aspect, the corpus set includes a first question-answer pair set in which a matching degree between the second question and the second answer sentence is greater than or equal to a threshold, and a second question-answer pair set in which a matching degree between the second question and the second answer sentence is less than a threshold; the device, still include:

and the updating module is used for updating the second characteristic vector according to the question-answer pair set to which each reference question-answer pair belongs.

a second determining module, configured to determine, by using a first preset method, a vector corresponding to each second question in the first question and each reference question-answer pair;

and the third determining module is used for determining the vectors corresponding to the first answer sentences and each second answer sentence in each reference question-answer pair by adopting a second preset method.

a fourth determining module, configured to determine an initiator and a responder that correspond to the first question-answer pair and each reference question-answer pair, respectively;

a fifth determining module, configured to determine, according to an initiator and a responder that correspond to the first question-answer pair, vectors corresponding to the first question sentence and the first answer sentence, respectively;

and the sixth determining module is used for determining vectors corresponding to the second question sentences and the second answer sentences in each reference question-answer pair respectively according to the initiator and the responder corresponding to each reference question-answer pair.

the extraction module is used for extracting a first training question-answer pair from the corpus set, and a first matching degree exists between a second question sentence and a second answer sentence in the first training question-answer pair;

a third obtaining module, configured to obtain at least one second training question-and-answer pair according to a similarity between the first training question-and-answer pair and each of the other second question-and-answer pairs in the corpus;

a seventh determining module, configured to determine a third feature vector corresponding to the first training question-answer pair according to a third vector corresponding to the first training question-answer pair and each fourth vector corresponding to each second training question-answer pair;

and the adjusting module is used for inputting the third feature vector into a preset model and adjusting the preset model parameter so as to enable the difference between the second matching degree output by the generated verification model and the first matching degree to be within a preset range.

In another aspect of the present application, an apparatus for generating a dialogue system reliability verification model includes: the extraction module is used for extracting training question-answer pairs from the corpus set, each second question-answer pair in the corpus set comprises a second question sentence and a second answer sentence, and a first matching degree exists between a target second question sentence and the target second answer sentence in the training question-answer pair; an obtaining module, configured to obtain at least one reference question-answer pair according to a similarity between the first training question-answer pair and each of the other second question-answer pairs in the corpus; the first determining module is used for determining a third feature vector corresponding to the training question-answer pair according to a third vector corresponding to the training question-answer pair and each fourth vector corresponding to each reference question-answer pair; a second determining module, configured to perform encoding and decoding processing on the third feature vector by using a preset model, and determine a second matching degree corresponding to the training question-answer pair; and the adjusting module is used for adjusting the preset model parameters according to the difference value between the first matching degree and the second matching degree so as to generate a verification model.

Optionally, in a possible implementation form of the embodiment of the fourth aspect, the obtaining module is specifically configured to:

or,

Optionally, in another possible implementation form of the embodiment of the fourth aspect, the obtaining module is further configured to:

or,

Optionally, in yet another possible implementation form of the embodiment of the fourth aspect, the apparatus further includes:

an extension module, configured to extend the third feature vector according to a similarity between the training question-answer pair and the at least one reference question-answer pair, a matching degree between a second question and a second question in each reference question-answer pair, an intention of the training question-answer pair, an intention of each reference question-answer pair, a domain to which the training question-answer pair belongs, and/or a domain to which each reference question-answer pair belongs, and generate a fourth feature vector;

the second determining module is specifically configured to:

Optionally, in a further possible implementation form of the embodiment of the fourth aspect, the corpus set includes a first question-answer pair set in which a matching degree between the second question and the second answer sentence is greater than or equal to a threshold, and a second question-answer pair set in which a matching degree between the second question and the second answer sentence is less than a threshold; the device, still include:

and the updating module is used for updating the fourth feature vector according to the question-answer pair set to which each reference question-answer pair belongs.

Optionally, in another possible implementation form of the embodiment of the fourth aspect, the apparatus further includes:

a third determining module, configured to determine, by using a first preset method, a vector corresponding to each second question in the target second question and each reference question-answer pair;

and the fourth determining module is used for determining the vectors corresponding to the target answer sentence and each second answer sentence in each reference question-answer pair by adopting a second preset method.

a fifth determining module, configured to determine an initiator and a responder that correspond to the training question-answer pair and each reference question-answer pair respectively;

a sixth determining module, configured to determine, according to the initiator and the responder that correspond to the training question-answer pair, vectors corresponding to the target second question sentence and the target second answer sentence, respectively;

and the seventh determining module is used for determining vectors corresponding to the second question sentences and the second answer sentences in each reference question-answer pair respectively according to the initiator and the responder corresponding to each reference question-answer pair.

An embodiment of another aspect of the present application provides an electronic device, which includes: memory, processor and computer program stored on the memory and executable on the processor, characterized in that the processor implements a dialog system reliability check method or a dialog system reliability check model generation method as described above when executing the program.

In another aspect, the present application provides a computer-readable storage medium, on which a computer program is stored, where the computer program is executed by a processor to implement the dialog system reliability check method or the dialog system reliability check model generation method as described above.

In another aspect of the present application, a computer program is provided, which is executed by a processor to implement the dialog system reliability check method according to the embodiment of the present application.

The dialog system reliability verification method, the dialog system reliability model generation device, the electronic device, the computer-readable storage medium and the computer program, which are provided by the embodiment of the application, can obtain a first question-answer pair to be verified, obtain at least one reference question-answer pair from a corpus according to the similarity between the first question-answer pair and each second question-answer pair in the corpus, determine a first feature vector corresponding to the first question-answer pair according to a first vector corresponding to the first question-answer pair and each second vector corresponding to each reference question-answer, and further perform coding and decoding processing on the first feature vector by using a pre-generated verification model to generate the matching degree between a first question and a first sentence in the first question-answer pair. Therefore, the corpus is concentrated to be similar to the first question-answer pair, the reference question-answer pair with the matching degree is used as a reference, and the matching degree between the first question sentence and the first answer sentence in the first question-answer pair is determined by utilizing the pre-generated verification model, so that the reliability of the dialogue system is automatically verified, the authenticity of the reliability verification of the dialogue system is improved, the verification efficiency is improved, and the labor cost is saved.

Additional aspects and advantages of the present application will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the present application.

Drawings

The foregoing and/or additional aspects and advantages of the present application will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings of which:

fig. 1 is a schematic flowchart illustrating a method for verifying reliability of a dialog system according to an embodiment of the present application;

fig. 2 is a schematic flowchart of another session system reliability checking method according to an embodiment of the present application;

fig. 3 is a schematic flowchart of a dialog system reliability verification model generation method according to an embodiment of the present application;

fig. 4 is a schematic flowchart of another dialog system reliability verification model generation method according to an embodiment of the present application;

fig. 5 is a schematic structural diagram of a session system reliability checking apparatus according to an embodiment of the present application;

fig. 6 is a schematic structural diagram of another session system reliability checking apparatus according to an embodiment of the present application;

fig. 7 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed Description

Reference will now be made in detail to the embodiments of the present application, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to the like or similar elements throughout. The embodiments described below with reference to the drawings are exemplary and intended to be used for explaining the present application and should not be construed as limiting the present application.

The embodiment of the application aims at the problems that in the related art, the reliability of dialogue data of a dialogue system is verified in an artificial mode, and because the dialogue data generated by the dialogue system is huge, the authenticity cannot be guaranteed, the efficiency is low, and the labor cost is improved, and provides a method for verifying the reliability of the dialogue system.

The dialog system reliability verification method provided by the embodiment of the application can obtain a first question-answer pair to be verified, obtain at least one reference question-answer pair from a corpus according to the similarity between the first question-answer pair and each second question-answer pair in the corpus, determine a first feature vector corresponding to the first question-answer pair according to a first vector corresponding to the first question-answer pair and each second vector corresponding to each reference question-answer, and further perform coding and decoding processing on the first feature vector by using a pre-generated verification model to generate the matching degree between a first question and a first answer sentence in the first question-answer pair. Therefore, the corpus is concentrated to be similar to the first question-answer pair, the reference question-answer pair with the matching degree is used as a reference, and the matching degree between the first question sentence and the first answer sentence in the first question-answer pair is determined by utilizing the pre-generated verification model, so that the reliability of the dialogue system is automatically verified, the authenticity of the reliability verification of the dialogue system is improved, the verification efficiency is improved, and the labor cost is saved.

The following describes in detail a dialog system reliability check method, apparatus, electronic device, storage medium, and computer program provided by the present application with reference to the accompanying drawings.

Fig. 1 is a schematic flowchart of a method for verifying reliability of a dialog system according to an embodiment of the present application.

As shown in fig. 1, the method for verifying the reliability of the dialog system includes the following steps:

step 101, a first question-answer pair to be verified is obtained, wherein the first question-answer pair comprises a first question sentence and a first answer sentence.

The first question-answer pair to be verified refers to a question-answer pair which needs to be verified to match the first question sentence and the first answer sentence.

It should be noted that the dialogue system reliability verification method according to the embodiment of the present application may be executed by the dialogue system reliability verification apparatus according to the embodiment of the present application. The dialogue system reliability verification device of the embodiment of the application can be configured in any electronic equipment.

Preferably, the dialog system reliability verification apparatus in the embodiment of the present application may be configured in a server of the dialog system, so as to obtain, in real time, dialog data that is reported by the dialog system and generated in a working process of the dialog system, that is, a first question-answer pair to be verified.

And step 102, acquiring at least one reference question-answer pair from the corpus according to the similarity between the first question-answer pair and each second question-answer pair in the corpus, wherein each second question-answer pair comprises a second question and a second answer sentence.

It should be noted that the corpus may include a large number of second question-answer pairs, and the matching degree between the second question sentences included in the second question-answer pairs and the second answer sentences is known. As a possible implementation, the second question-answer pair may include a second question sentence and a second answer sentence that are highly matched positive samples.

The similarity between the first question-answer pair and each second question-answer pair in the corpus may be a cosine similarity between a vector corresponding to the first question-answer pair and a vector corresponding to each second question-answer pair.

As a possible implementation manner, a vector corresponding to the first question and answer pair and a vector corresponding to the first answer sentence included in the first question and answer pair may be determined by a preset text vectorization method, and then the first vector corresponding to the first question and answer pair is determined according to the vector corresponding to the first question and the vector corresponding to the first answer sentence. For example, the vector corresponding to the first question sentence and the mean value of the vectors corresponding to the first answer sentence may be determined as the first vector corresponding to the first question-answer pair; or, the vector corresponding to the first question sentence is spliced with the vector corresponding to the first answer sentence to serve as the first vector corresponding to the first question-answer pair, and so on, which is not limited in the embodiment of the present application.

Correspondingly, the second vectors corresponding to the second question-answer pairs can be determined in the same manner. Then, the cosine similarity between the first vector and the second vector corresponding to each second question-answer pair can be calculated, the cosine similarity between the first vector and each second vector is determined as the similarity between the first question-answer pair and each second question-answer pair, and one or more reference question-answer pairs are obtained from the corpus according to the similarity between the first question-answer pair and each second question-answer pair. For example, one or more second question-answer pairs having the greatest similarity to the first question-answer pair may be determined as reference question-answer pairs.

Furthermore, when the reference question-answer pairs are obtained from the corpus, a threshold value which needs to be met by the reference question-answer pairs can be preset, and then the reference question-answer pairs are determined according to the relationship between the similarity between the first question-answer pairs and each second question-answer pair and the threshold value. That is, in a possible implementation form of the embodiment of the present application, the step 102 may further include:

according to the similarity between the first question-answer pair and each second question-answer pair, obtaining a second question-answer pair with the highest similarity with the first question-answer pair from the corpus set;

or,

As a possible implementation manner, the number of the reference question-answer pairs may be 1, and at this time, one second question-answer pair with the highest similarity to the first question-answer pair in the corpus may be determined as the reference question-answer pair according to the similarity between the first question-answer pair and each second question-answer pair.

As a possible implementation manner, the number of the reference question-answer pairs may also be multiple, and at this time, a second question-answer pair with the highest similarity to the first question-answer pair in the corpus may be determined as the first reference question-answer pair; and then, according to a threshold value which needs to be met by the reference question-answer pair, determining that the similarity between the reference question-answer pair and the first question-answer pair is greater than the first M second question-answer pairs before the threshold value, and further obtaining the reference question-answer pair with the similarity between the first question-answer pair and the first M second question-answer pairs at the middle level from the corpus set, namely the nth second question-answer pair in the M second question-answer pairs as the reference question-answer pair.

Optionally, there may be a plurality of values of n, so that a plurality of reference question-answer pairs may be obtained from the corpus. For example, there are 10 second reference question-answer pairs whose similarity to the first question-answer pair is greater than the threshold, that is, M has a value of 10, and n has a value of 4, 5, or 6, so that 4 reference question-answer pairs including the second question-answer pair with the highest similarity to the first question-answer pair can be obtained from the corpus.

It should be noted that the above examples are only illustrative and should not be construed as limiting the present application. In actual use, the specific value of the threshold and the specific value of n may be determined according to actual needs, which is not limited in the embodiments of the present application.

Further, when the reference question-answer pairs are obtained from the corpus, the similarity between the first question sentences and the second question sentences and the similarity between the first answer sentences and the second answer sentences can be respectively calculated, and the reference question-answer pairs are determined according to the similarity between the first question sentences and the second answer sentences. That is, in a possible implementation form of the embodiment of the present application, the step 102 may include:

obtaining each suspected reference question-answer pair according to the similarity between the first question and each second question, and determining at least one suspected reference question-answer pair according to the similarity between the first question and the second question in each suspected reference question-answer pair;

or,

and obtaining each suspected reference question-answer pair according to the similarity between the first question and each second question, and determining at least one reference question-answer pair according to the similarity between the first question and the second question in each suspected reference question-answer pair.

As a possible implementation manner, the reference question-answer pair may be determined by the similarity between the first question-answer pair and the second question-answer pair, or may be determined by the similarity between the first question and each second question, and the similarity between the first question and each second question.

Optionally, a text vectorization method may be used to determine vectors corresponding to the first question and the first answer sentence included in the first question-answer pair and vectors corresponding to the second question and the second answer sentence included in each second question-answer pair, and calculate a cosine similarity between the vector corresponding to the first question and the vectors corresponding to the second question as a similarity between the first question and the second question, and further determine, according to the similarity between the first question and each second question, a plurality of second question pairs with the highest similarity between the first question and the second question (for example, a plurality of second question sentences whose similarity between the first question and the second question is greater than a threshold value), and determine the second question-answer pairs corresponding to the plurality of determined second question pairs as suspected question-answer pairs.

Then, the similarity between the vector corresponding to the first answer sentence and the second answer sentences included in each suspected reference question-answer pair is calculated, one or more second answer sentences with the highest similarity between the vector corresponding to the first answer sentence and the second answer sentences are determined (for example, one or more second answer sentences with the similarity between the vector corresponding to the first answer sentence and the first answer sentence being greater than a threshold value), and the suspected reference question-answer pairs corresponding to the determined one or more second answer sentences are determined as reference question-answer pairs.

Optionally, the cosine similarity between the vector corresponding to the first answering sentence and the vectors corresponding to the second answering sentences may be calculated first, and the cosine similarity is used as the similarity between the first answering sentence and the second answering sentences, and then according to the similarity between the first answering sentence and each second answering sentence, a plurality of second answering sentences with the highest similarity to the first answering sentence (for example, a plurality of second answering sentences with the similarity to the first answering sentence larger than a threshold value) are determined, and the second question-answer pairs corresponding to the determined plurality of second answering sentences are determined as suspected reference question-answer pairs.

Then, the similarity between the vector corresponding to the first question and the second question included in each suspected reference question-answer pair is calculated, one or more second question sentences with the highest similarity with the first question (for example, one or more second question sentences with the similarity with the first question larger than a threshold value) are determined, and further the suspected reference question-answer pairs corresponding to the determined one or more second question sentences are determined as reference question-answer pairs.

Step 103, determining a first feature vector corresponding to the first question-answer pair according to the first vector corresponding to the first question-answer pair and each second vector corresponding to each reference question-answer.

The first vector corresponding to the first question-answer pair may be determined according to a vector corresponding to the first question and a vector corresponding to the first answer sentence included in the first question-answer pair, for example, the first vector may be an average value of the vector corresponding to the first question and the vector corresponding to the first answer sentence, or a vector obtained by splicing the vector corresponding to the first question and the vector corresponding to the first answer sentence, and the like.

Correspondingly, the second vector corresponding to the reference question-answer pair may be determined according to the vector corresponding to the second question sentence included in the reference question-answer pair and the vector corresponding to the second answer sentence, for example, the second vector may be an average value of the vector corresponding to the second question sentence and the vector corresponding to the second answer sentence, or a vector obtained by splicing the vector corresponding to the second question sentence and the vector corresponding to the second answer sentence, and so on. It should be noted that the manner of determining the second vector corresponding to the reference answer pair is the same as the manner of determining the first vector corresponding to the first answer pair.

As a possible implementation manner, the first feature vector corresponding to the first question-answer pair may be a vector formed by splicing the first vector corresponding to the first question-answer pair and the second vector corresponding to each reference question-answer pair.

For example, the first vector corresponding to the first question-answer pair is a, there are 2 reference vectors obtained from the corpus, and the reference question-answers areThe second vector corresponding to 1 is B₁The second vector corresponding to the reference question-answer pair 2 is B₂If the first feature vector corresponding to the first question-answer pair is T ═ a, B₁,B₂]。

Further, when determining a first vector corresponding to the first question-answer pair and a second vector corresponding to the second question-answer pair, a text vectorization method used for calculating the vectors corresponding to the first question and the second question may be different from a text vectorization method used for calculating the vectors corresponding to the first question and the second question. That is, in a possible implementation form of the embodiment of the present application, before the step 103, the method may further include:

determining vectors corresponding to the first question and each second question in each reference question-answer pair by adopting a first preset method;

The first preset method and the second preset method refer to two different text vectorization methods. For example, the term frequency-inverse text frequency (TF-IDF) algorithm, the Latent Semantic retrieval (LSI) algorithm, the Linear Discriminant Analysis (LDA) algorithm, the doc2bow, the doc2vec algorithm, and the like.

It should be noted that, in a dialog system, the sources of the question and the answer sentence are usually different, so that there is a certain difference in characteristics such as the length, complexity, and spoken language degree of the question and the answer sentence, therefore, when determining the first vector corresponding to the first question-answer pair and the second vector corresponding to each reference question-answer pair, different text vectorization methods may be used to determine the vectors corresponding to the first question and each second question and the vectors corresponding to the first answer and each second answer sentence according to the characteristics of the question and the question sentence, so that the first vector and the second vector may express semantic information of the first question-answer pair and the reference question-answer pair more accurately.

For example, if the dialog system is an intelligent interactive robot, the first question-answer pair includes a first question sentence and the second question-answer pair includes a second question sentence, which are issued by the user, and the first question-answer pair includes a first question sentence and the second question-answer pair includes a second question sentence, which are issued by the robot, and the question sentence issued by the user is generally long, high in complexity and high in spoken language; the answer sentences sent by the robot are usually simpler and have low spoken language degree, so that different text vectorization methods can be selected according to the respective characteristics of the question sentences and the answer sentences to determine the vectors corresponding to the question sentences and the answer sentences respectively.

As a possible implementation, the first predetermined method may be a doc2vec algorithm, and the second predetermined method may be a TF-IDF algorithm. Because the doc2vec algorithm has higher accuracy and is suitable for text vectorization of a more complex sentence structure, the question sentences in the question-answer pairs are usually initiated by users, and the sentence results are more complex, vectors corresponding to the first question sentences and the second question sentences can be determined by adopting the doc2vec algorithm with higher accuracy so as to ensure the accuracy of semantic representation of the first question sentences and the second question sentences; because the algorithm complexity of the TF-IDF algorithm is lower, the TF-IDF algorithm is more suitable for text vectorization of short sentences, the answer sentences in question-answer pairs are usually initiated by a dialogue system, and the sentence structure is simpler, the TF-IDF algorithm can be adopted to determine the vectors corresponding to the first answer sentences and the second answer sentences, so that the semantic representation accuracy of the first answer sentences and the second answer sentences can be ensured, and the algorithm complexity can be reduced.

Further, in some scenarios, the dialog system may also ask a question to the user to determine its next operation according to the answer of the user, that is, the question in the question-answer pair may be initiated by the dialog system, and the answer in the question-answer pair may be initiated by the user, so that the text vectorization methods corresponding to the question and the answer respectively may be determined according to the initiator and the responder of the question-answer pair. That is, in a possible implementation form of the embodiment of the present application, before the step 103, the method may further include:

determining vectors corresponding to a first question and a first answer sentence respectively according to an initiator and a responder corresponding to the first question-answer pair;

The initiator corresponding to the question-answer pair refers to the party sending out the question sentence corresponding to the question-answer pair; the responder of the question-answer pair refers to the party who sends out the answer sentence corresponding to the question-answer pair. For example, if a first question sentence included in the first question-answer pair is sent by the dialog system and the first answer sentence is sent by the user, the initiator corresponding to the first question-answer pair is the dialog system and the responder is the user; if the first question and answer included in the first question and answer pair is sent by the user and the first answer is sent by the dialogue system, the initiator corresponding to the first question and answer pair is the user, and the responder is the dialogue system.

As a possible implementation manner, a text vectorization method may be adopted for the sentences sent by the dialog system, and another text vectorization method may be adopted for the sentences sent by the user, so that the text vectorization methods respectively corresponding to the first question and the first answer included in the first question-answer pair and the second question and the second answer included in the each reference question-answer pair may be determined according to the initiator and the responder respectively corresponding to the first question-answer pair and each reference question-answer pair, and further the vectors respectively corresponding to the first question and the first answer may be determined according to the text vectorization methods respectively corresponding to the determined first question and the determined first answer; and determining vectors respectively corresponding to the second question sentence and the second answer sentence in each reference question-answer pair according to the text vectorization method respectively corresponding to the second question sentence and the second answer sentence in each reference question-answer pair.

For example, a text vectorization method corresponding to a statement sent by a dialog system is preset as a TF-IDF algorithm, a text vectorization method corresponding to a statement sent by a user is preset as a doc2vec algorithm, if it is determined that an initiator corresponding to a first question-answer pair is the dialog system and a responder is the user, the text vectorization algorithm corresponding to a first question sentence in the first question-answer pair can be determined as the TF-IDF algorithm, and the text vectorization algorithm corresponding to the first answer sentence is the doc2vec algorithm, so that a vector corresponding to the first question sentence can be determined according to the TF-IDF algorithm, and a vector corresponding to the first answer sentence can be determined according to the doc2vec algorithm; if the initiator corresponding to one reference question-answer pair is determined as a user and the responder is a dialogue system, the text vectorization algorithm corresponding to the second question sentence in the reference question-answer pair can be determined as doc2vec algorithm, and the text vectorization algorithm corresponding to the second answer sentence is TF-IDF algorithm, so that the vector corresponding to the second question sentence can be determined according to the doc2vec algorithm, and the vector corresponding to the second answer sentence can be determined according to the TF-IDF algorithm.

And 104, coding and decoding the first feature vector by using a pre-generated check model to generate the matching degree between the first question sentence and the first answer sentence in the first question-answer pair.

In the embodiment of the application, after the first feature vector corresponding to the first question-answer pair to be verified is determined, the first feature vector is input into a pre-generated verification model, so that the pre-generated verification model is utilized to perform coding and decoding processing on the first feature vector, and the matching degree between the first question sentence and the first answer sentence in the first question-answer pair is generated according to the coding and decoding processing result and output.

It can be understood that the first feature vectors corresponding to the first question-answer pair include a first vector corresponding to the first question-answer pair and a second vector corresponding to each reference question-answer pair, where the first question-answer pair has a higher similarity with each reference question-answer pair, and the matching degree between the second question and the second question in each reference question-answer pair is known, so that the pre-generated verification model can obtain the relationship between the matching degree between the first question and the first answer in the first question-answer pair and the matching degree between the second question and the second answer in each reference question-answer pair through analysis of the first feature vectors, thereby generating the matching degree between the first question and the first answer in the first question-answer pair.

The calibration model generated in advance may be generated by training. When the verification model is trained, the feature vectors included in the training question-answer degrees in the training data and the matching degrees between the question sentences included in the training question-answer pairs and the answer sentences in the training question-answer pairs are input into the training model, so that the finally generated verification model can identify the relation between the feature vectors corresponding to the question-answer pairs and the matching degrees between the question sentences included in the training question-answer pairs and the answer sentences, and the matching degrees between the first question sentences in the first question-answer pairs and the first answer sentences in the first question-answer pairs can be generated according to the first feature vectors corresponding to the first question-answer pairs to be verified.

In a possible implementation form of the embodiment of the present application, the first feature vector corresponding to the first question-answer pair may be further expanded according to the intention of the first question-answer pair, the field to which the first question-answer pair belongs, the intention corresponding to the reference question-answer pair, the field to which the first question-answer pair belongs, the matching degree between the second question sentence and the second question sentence, and the like, so as to further improve the accuracy of the reliability check of the dialog system.

The dialog system reliability check method provided in the embodiment of the present application is further described below with reference to fig. 2.

Fig. 2 is a schematic flowchart of another session system reliability checking method according to an embodiment of the present application.

As shown in fig. 2, the method for verifying the reliability of the dialog system includes the following steps:

step 201, a first question-answer pair to be verified is obtained, wherein the first question-answer pair includes a first question sentence and a first answer sentence.

Step 202, according to the similarity between the first question-answer pair and each second question-answer pair in the corpus, at least one reference question-answer pair is obtained from the corpus, and each second question-answer pair comprises a second question and a second answer sentence.

Step 203, determining a first feature vector corresponding to the first question-answer pair according to the first vector corresponding to the first question-answer pair and each second vector corresponding to each reference question-answer.

The detailed implementation process and principle of the steps 201-203 can refer to the detailed description of the above embodiments, and are not described herein again.

And 204, expanding the first feature vector according to the similarity between the first question-answer pair and at least one reference question-answer pair, the matching degree between a second question and a second question in each reference question-answer pair, the intention of the first question-answer pair, the intention of each reference question-answer pair, the field of the first question-answer pair and/or the field of each reference question-answer pair, and generating a second feature vector.

The intention and the field of the first question-answer pair can be obtained by performing semantic analysis on the first question-answer pair; the intention of each reference question-answer pair and the field thereof can be pre-stored in the corpus and directly obtained from the corpus.

In the embodiment of the present application, to further improve the accuracy of the verification, the intention and the field to which the first question-answer pair belongs, the intention and the field to which each reference question-answer pair belongs may be further obtained, and a vector corresponding to the intention and the field to which each reference question-answer pair belongs may be determined, and then the first feature vector corresponding to the first question-answer pair may be expanded to generate the second feature vector corresponding to the first question-answer pair according to the similarity between the first question-answer pair and each reference question-answer pair, the matching degree between the second question sentence and the second question-answer in each reference question-answer pair, the vector corresponding to the intention of the first question-answer pair, the vector corresponding to the intention of each reference question-answer pair, the vector corresponding to the field to which each reference question-answer pair belongs.

For example, the similarity between the first feature vector corresponding to the first question-answer pair and each reference question-answer pair, the matching degree between the second question and the second question in each reference question-answer pair, the vector corresponding to the intention of the first question-answer pair, the vector corresponding to the intention of each reference question-answer pair, the vector corresponding to the domain to which the first question-answer pair belongs, and the vector corresponding to the domain to which each reference question-answer pair belongs may be spliced to generate the second feature vector.

And step 205, performing coding and decoding processing on the second feature vector by using a pre-generated check model to generate a matching degree between the first question sentence and the first answer sentence in the first question-answer pair.

In the embodiment of the present application, after the first feature vector corresponding to the first question-answer pair is expanded to generate the second feature vector corresponding to the first question-answer pair, the second feature vector may be encoded and decoded by using a pre-generated verification model, and a matching degree between the first question sentence and the first answer sentence in the first question-answer pair is generated and output, so that the matching degree between the generated first question sentence and the first answer sentence is more accurate, and the accuracy of the reliability verification of the dialog system is further improved.

Further, the corpus set may include a first question-answer pair set in which the matching degree between the second question and the second answer is greater than or equal to a threshold, and a second question-answer pair set in which the matching degree between the second question and the second answer is less than the threshold, and at this time, the second feature vector needs to be updated according to the question-answer set to which each reference question-answer pair belongs. That is, in a possible implementation form of the embodiment of the present application, before the step 205, the method may further include:

It should be noted that the corpus may include a first question-answer pair set (i.e. a positive sample set with a higher matching degree between the second question and the second answer sentence) whose matching degree between the second question and the second answer sentence is greater than or equal to a threshold value, a second question-answer pair set (i.e. a negative sample set with a low matching degree between the second question and the second answer sentence) with a matching degree between the second question and the second answer sentence smaller than a threshold value, so that in order to distinguish positive examples from negative examples in the second feature vector corresponding to the first question-answer pair, the corresponding dimension of each parameter corresponding to the positive sample in the second feature vector and the dimension of each parameter corresponding to the negative sample in the second feature vector can be preset, and determining the dimensionality of each parameter corresponding to each reference question-answer pair in the second characteristic vector according to the question-answer set of each reference question-answer pair, and further updating the second characteristic vector.

For example, assume that the dimension in the second feature vector of the reference answer pair in the first answer set is the middle dimensions, and the dimension in the second feature vector of the reference answer pair in the second answer set is the end dimensions. The second feature vector corresponding to the first question-answer pair is T₂＝[A₁,B₃,B₄]Wherein A is₁A vector composed of a first vector corresponding to the first question-answer pair, a vector corresponding to parameters such as the intention, the field to which the first question-answer pair belongs, and the like, B₃A vector composed of a second vector corresponding to the reference question-answer pair 1 and vectors corresponding to parameters such as the intention, the field to which the second question-answer pair belongs, the matching degree between the second question sentence and the second answer sentence, and the like, B₄A vector formed by a second vector corresponding to the reference question-answer pair 2 and vectors corresponding to parameters such as the intention, the field to which the second question-answer pair belongs, the matching degree between the second question-answer and the second answer, and the like, wherein the reference question-answer pair 1 belongs to a second question-answer set, the reference question-answer pair 2 belongs to a first question-answer set, and the updated second characteristic vector is T₃＝[A₁,B₄,B₃]。

The method for verifying reliability of a dialog system provided in the embodiment of the present application may include expanding a first feature vector to generate a second feature vector according to a similarity between a first question-answer pair and at least one reference question-answer pair, a matching degree between a second question and a second question in each reference question-answer pair, an intention of the first question-answer pair, an intention of each reference question-answer pair, a field to which the first question-answer pair belongs, and/or a field to which each reference question-answer pair belongs, and then encoding and decoding the second feature vector by using a pre-generated verification model to generate a matching degree between the first question and the first answer in the first question-answer pair. Therefore, the corpus is concentrated to be similar to the first question-answer pair and the reference question-answer pair with the matching degree is used as the reference, and the pre-generated verification model is used for determining the matching degree between the first question sentence and the first answer sentence in the first question-answer pair, so that the reliability of the dialogue system is automatically verified, the verification efficiency is improved, the labor cost is saved, and the accuracy of the reliability verification of the dialogue system is further improved.

In order to implement the above embodiment, the present application further provides a method for generating a reliability check model of a dialog system.

Fig. 3 is a schematic flowchart of a dialog system reliability verification model generation method according to an embodiment of the present application.

As shown in fig. 3, the dialog system reliability verification model generation method includes the following steps:

step 301, extracting training question-answer pairs from the corpus, wherein each second question-answer pair in the corpus comprises a second question and a second answer, and a first matching degree exists between a target second question and a target second answer in the training question-answer pairs.

In the embodiment of the application, the question-answer pair with the matching degree between the second question and the second answer sentence known in the corpus can be used as a training question-answer pair, the verification model is trained, and the parameters of the verification model are optimized, so that the final verification result is more accurate. Namely, the training question-answer pairs with the first matching degree between the target second question sentences and the target second answer sentences can be extracted from the corpus set.

Step 302, at least one reference question-answer pair is obtained according to the similarity between the training question-answer pair and each other second question-answer pair in the corpus.

In the embodiment of the present application, after the training question-answer pair is obtained from the corpus set, at least one reference question-answer pair may be obtained according to the similarity between the training question-answer pair and each of the other second question-answer pairs.

Furthermore, when the reference question-answer pairs are obtained from the corpus, a threshold value which needs to be met by the reference question-answer pairs can be preset, and then the reference question-answer pairs are determined according to the relationship between the similarity between the training question-answer pairs and each second question-answer pair and the threshold value. That is, in a possible implementation form of the embodiment of the present application, the step 102 may further include:

or,

Further, when the reference question-answer pairs are obtained from the corpus, the similarity between the target second question sentences and each second question sentence and the similarity between the target second answer sentences and each second answer sentence can be respectively calculated, and then the reference question-answer pairs are determined according to the similarities between the target second question sentences and each second answer sentence. That is, in a possible implementation form of the embodiment of the present application, the step 102 may include:

or,

It should be noted that the manner of obtaining at least one reference question-answer pair from the corpus is the same as the manner of extracting at least one reference question-answer pair according to the similarity between the first question-answer pair to be verified and each second question-answer pair in the corpus, and details are not repeated here.

Step 303, determining a third feature vector corresponding to the training question-answer pair according to the third vector corresponding to the training question-answer pair and each fourth vector corresponding to each reference question-answer pair.

In the embodiment of the application, after the training question-answer pair and at least one reference question-answer pair are obtained from the corpus set, the third vector corresponding to the training question-answer pair and each fourth vector corresponding to each reference question-answer pair can be determined according to a preset text vectorization method, and then the third vector corresponding to the training question-answer pair and each fourth vector corresponding to each reference question-answer pair are spliced to generate the third feature vector corresponding to the training question-answer pair.

Further, when determining the third vector corresponding to the training question-answer pair and the fourth vector corresponding to the reference question-answer pair, the text vectorization method used for calculating the vector corresponding to the target second question and the second question may be different from the text vectorization method used for calculating the vector corresponding to the target second question and the second question. That is, in a possible implementation form of the embodiment of the present application, before the step 303, the method may further include:

Further, in some scenarios, the dialog system may also ask a question to the user to determine its next operation according to the answer of the user, that is, the question in the question-answer pair may be initiated by the dialog system, and the answer in the question-answer pair may be initiated by the user, so that the text vectorization methods corresponding to the question and the answer respectively may be determined according to the initiator and the responder of the question-answer pair. That is, in a possible implementation form of the embodiment of the present application, before the step 303, the method may further include:

it should be noted that the manner of determining the third vector corresponding to the training question-answer pair is the same as the manner of determining the first vector corresponding to the first question-answer pair; determining the mode of each fourth vector corresponding to each reference answer pair in the training process, wherein the mode is the same as the mode of determining each second vector corresponding to each reference answer pair in the prediction process; the way of determining the third feature vector corresponding to the training question-answer pair is the same as the way of determining the first feature vector corresponding to the first question-answer pair, and details are not repeated here.

And 304, encoding and decoding the third feature vector by using a preset model, and determining a second matching degree corresponding to the training question-answer pair.

In the embodiment of the application, after the third feature vector corresponding to the training question-answer pair is determined, the third feature vector may be input into a preset model, so as to perform encoding and decoding processing on the third feature vector by using the preset model, and generate and output a second matching degree between the target second question sentence and the target second answer sentence in the training question-answer pair according to the encoding and decoding processing result.

Step 305, adjusting preset model parameters according to a difference value between the first matching degree and the second matching degree to generate a verification model.

In the embodiment of the present application, after the third feature vector corresponding to the training question-answer pair is determined, the third feature vector may be input into a preset model, so that the preset model generates and outputs the second matching degree between the target second question sentence and the target second question sentence in the training question-answer pair, and the preset model parameter is adjusted according to the difference between the second matching degree and the first matching degree, thereby generating the verification model.

Specifically, if the difference between the second matching degree output by the preset model and the first matching degree is within a preset range, the preset model parameter does not need to be adjusted; if the difference between the second matching degree output by the preset model and the first matching degree is not in the preset range, the preset model parameters need to be adjusted until the difference between the second matching degree and the first matching degree is in the preset range.

The method for generating the reliability check model of the dialog system, provided by the embodiment of the application, may extract training question-answer pairs from a corpus set, where a first matching degree exists between a target second question and a target second question in the training question-answer pairs, and obtain at least one reference question-answer pair according to a similarity between the training question-answer pair and each of other second question-answer pairs in the corpus set, and then determine a third feature vector corresponding to the training question-answer pair according to a third vector corresponding to the training question-answer pair and each fourth vector corresponding to each reference question-answer pair, and further perform encoding and decoding processing on the third feature vector by using a preset model, determine a second matching degree corresponding to the training question-answer pair, and adjust preset model parameters according to a difference between the first matching degree and the second matching degree to generate the check model. Therefore, the verification model is generated by adjusting the preset model parameters, so that the reliability of the dialogue system is automatically verified, the authenticity of the reliability verification of the dialogue system is improved, the verification efficiency is improved, and the labor cost is saved.

In a possible implementation form of the embodiment of the application, the intention of the training question-answer pair, the field to which the training question-answer pair belongs, the intention corresponding to the reference question-answer pair, the field to which the training question-answer pair belongs, and the matching degree between the second question sentence and the second answer sentence may be used as training data to train, so as to further improve the accuracy of the generated verification model.

Fig. 4 is a schematic flowchart of another dialog system reliability verification model generation method according to an embodiment of the present application.

As shown in fig. 4, the dialog system reliability verification model generation method includes the following steps:

step 401, extracting training question-answer pairs from the corpus, where each second question-answer pair in the corpus includes a second question and a second answer, and a first matching degree exists between a target second question and a target second answer in the training question-answer pairs.

And step 402, acquiring at least one reference question-answer pair according to the similarity between the training question-answer pair and other second question-answer pairs in the corpus.

Step 403, determining a third feature vector corresponding to the training question-answer pair according to the third vector corresponding to the training question-answer pair and each fourth vector corresponding to each reference question-answer pair.

The detailed implementation process and principle of the steps 401-403 may refer to the detailed description of the above embodiments, and are not described herein again.

Step 404, expanding the third feature vector according to the similarity between the training question-answer pair and at least one reference question-answer pair, the matching degree between the second question and the second question in each reference question-answer pair, the intention of the training question-answer pair, the intention of each reference question-answer pair, the field to which the training question-answer pair belongs, and/or the field to which each reference question-answer pair belongs, and generating a fourth feature vector.

The intention and the field of the training question-answer pair and the intention and the field of each reference question-answer pair can be pre-stored in a corpus and directly obtained from the corpus.

In the embodiment of the application, in order to further improve the prediction accuracy of the generated verification model, the intention and the field of the training question-answer pair, the intention and the field of each reference question-answer pair can be further obtained, and the corresponding vector is determined, and further according to the similarity of the training question-answer pair and each reference question-answer pair, the matching degree between the second question sentence and the second answer sentence in each reference question-answer pair, the vector corresponding to the intention of the training question-answer pair, the vector corresponding to the intention of each reference question-answer pair, the vector corresponding to the field to which the training question-answer pair belongs and the vector corresponding to the field to which each reference question-answer pair belongs, expanding the third feature vector corresponding to the training question-answer pair to generate a fourth feature vector corresponding to the training question-answer pair, and the fourth feature vector is used as training data to improve the reliability and accuracy of the verification model generated by training.

For example, the similarity between the third feature vector corresponding to the training question-answer pair and each of the training question-answer pairs, the matching degree between the second question and the second question in each of the reference question-answer pairs, the vector corresponding to the intention of the training question-answer pair, the vector corresponding to the intention of each of the reference question-answer pairs, the vector corresponding to the domain to which the training question-answer pair belongs, and the vector corresponding to the domain to which each of the reference question-answer pairs belongs may be spliced to generate the fourth feature vector.

And 405, performing encoding and decoding processing on the fourth feature vector by using a preset model, and determining a second matching degree corresponding to the training question-answer pair.

In the embodiment of the application, after the third feature vector corresponding to the training question-answer pair is expanded to generate the fourth feature vector corresponding to the training question-answer pair, the fourth feature vector may be encoded and decoded by using a preset model, and the matching degree between the target second question sentence and the target second answer sentence in the training question-answer pair is generated and output, so that the matching degree between the generated target second question sentence and the target second answer sentence is more accurate, and the accuracy of the reliability check model of the dialog system is further improved.

Further, the corpus set may include a first question-answer pair set in which the matching degree between the second question and the second answer is greater than or equal to a threshold, and a second question-answer pair set in which the matching degree between the second question and the second answer is less than the threshold, and at this time, the fourth feature vector needs to be updated according to the question-answer set to which each reference question-answer pair belongs. That is, in a possible implementation form of the embodiment of the present application, before the step 405, the method may further include:

It should be noted that the manner of updating the fourth feature vector corresponding to the training question-answer sentence is the same as the manner of updating the second feature vector corresponding to the first question-answer sentence, and the specific process may refer to the detailed description of the above embodiment, which is not described herein again.

Step 406, adjusting the preset model parameter according to the difference between the first matching degree and the second matching degree to generate the verification model.

The detailed implementation process and principle of the step 406 may refer to the detailed description of the above embodiments, and are not described herein again.

The method for generating the reliability check model of the dialog system according to the embodiment of the application may include expanding a third feature vector according to a similarity between a training question-answer pair and at least one reference question-answer pair, a matching degree between a second question sentence and a second answer sentence in each reference question-answer pair, an intention of the training question-answer pair, an intention of each reference question-answer pair, a field to which the training question-answer pair belongs, and/or a field to which each reference question-answer pair belongs to, generating a fourth feature vector, encoding and decoding the fourth feature vector by using a preset model, determining a second matching degree corresponding to the training question-answer pair, and adjusting preset model parameters according to a difference between the first matching degree and the second matching degree to generate the check model. Therefore, the accuracy of the verification model is further improved by expanding the content of the training data, so that the reliability of the dialogue system is automatically verified, the verification efficiency is not improved, the labor cost is saved, and the accuracy of the reliability verification of the dialogue system is further improved.

In order to implement the above embodiment, the present application further provides a device for verifying reliability of a dialog system.

Fig. 5 is a schematic structural diagram of a dialog system reliability checking apparatus according to an embodiment of the present application.

As shown in fig. 5, the dialogue system reliability verification apparatus 50 includes:

a first obtaining module 51, configured to obtain a first question-answer pair to be verified, where the question-answer pair includes a first question and a first answer;

a second obtaining module 52, configured to obtain at least one reference question-answer pair from the corpus according to a similarity between the first question-answer pair and each second question-answer pair in the corpus, where each second question-answer pair includes a second question sentence and a second answer sentence;

a first determining module 53, configured to determine a first feature vector corresponding to the first question-answer pair according to a first vector corresponding to the first question-answer pair and each second vector corresponding to each reference question-answer pair;

a first generating module 54, configured to perform encoding and decoding processing on the first feature vector by using a pre-generated check model, and generate a matching degree between a first question sentence and a first answer sentence in the first question-answer pair.

In practical use, the dialog system reliability check device provided in the embodiment of the present application may be configured in any electronic device to execute the dialog system reliability check method.

The dialog system reliability verification device provided by the embodiment of the application can obtain a first question-answer pair to be verified, obtain at least one reference question-answer pair from a corpus according to the similarity between the first question-answer pair and each second question-answer pair in the corpus, determine a first feature vector corresponding to the first question-answer pair according to a first vector corresponding to the first question-answer pair and each second vector corresponding to each reference question-answer, and perform coding and decoding processing on the first feature vector by using a pre-generated verification model to generate the matching degree between a first question and a first answer sentence in the first question-answer pair. Therefore, the corpus is concentrated to be similar to the first question-answer pair, the reference question-answer pair with the matching degree is used as a reference, and the matching degree between the first question sentence and the first answer sentence in the first question-answer pair is determined by utilizing the pre-generated verification model, so that the reliability of the dialogue system is automatically verified, the authenticity of the reliability verification of the dialogue system is improved, the verification efficiency is improved, and the labor cost is saved.

In a possible implementation form of the present application, the second obtaining module 52 is specifically configured to:

or,

Further, in another possible implementation form of the present application, the second obtaining module 52 is further configured to:

or,

In a possible implementation form of the present application, the above-mentioned telephone system reliability checking apparatus 50 further includes:

accordingly, the first generating module 54 is specifically configured to:

and carrying out encoding and decoding processing on the second feature vector.

Further, in another possible implementation form of the present application, the corpus set includes a first question-answer pair set in which a matching degree between a second question and a second answer sentence is greater than or equal to a threshold, and a second question-answer pair set in which a matching degree between the second question and the second answer sentence is less than a threshold; accordingly, the above-mentioned speech system reliability verifying apparatus 50 further includes:

Further, in another possible implementation form of the present application, the above-mentioned system reliability verifying apparatus 50 further includes:

Further, in another possible implementation form of the present application, the above-mentioned session system reliability checking apparatus 50 further includes:

It should be noted that the above explanation of the embodiment of the dialog system reliability check method shown in fig. 1 and fig. 2 is also applicable to the dialog system reliability check device 50 of this embodiment, and will not be repeated here.

The dialog system reliability verification apparatus provided in the embodiment of the present application may generate the matching degree between the first question and the first answer pair in the first question-answer pair by expanding the first feature vector according to the similarity between the first question-answer pair and at least one reference question-answer pair, the matching degree between the second question and the second answer in each reference question-answer pair, the intention of the first question-answer pair, the intention of each reference question-answer pair, the domain to which the first question-answer pair belongs, and/or the domain to which each reference question-answer pair belongs, and generating the second feature vector, and then performing encoding and decoding processing on the second feature vector by using a pre-generated verification model. Therefore, the corpus is concentrated to be similar to the first question-answer pair and the reference question-answer pair with the matching degree is used as the reference, and the pre-generated verification model is used for determining the matching degree between the first question sentence and the first answer sentence in the first question-answer pair, so that the reliability of the dialogue system is automatically verified, the verification efficiency is improved, the labor cost is saved, and the accuracy of the reliability verification of the dialogue system is further improved.

In order to implement the above embodiments, the present application further provides a dialog system reliability verification model generation apparatus.

Fig. 6 is a schematic structural diagram of a dialog system reliability verification model generation apparatus according to an embodiment of the present application.

As shown in fig. 6, the dialogue system reliability check model generation apparatus 60 includes:

an extraction module 61, configured to extract training question-answer pairs from the corpus set, where each second question-answer pair in the corpus set includes a second question and a second answer, and a first matching degree exists between a target second question and a target second answer in the training question-answer pair;

an obtaining module 62, configured to obtain at least one reference question-answer pair according to a similarity between the first training question-answer pair and each of the other second question-answer pairs in the corpus;

a first determining module 63, configured to determine a third feature vector corresponding to the training question-answer pair according to the third vector corresponding to the training question-answer pair and each fourth vector corresponding to each reference question-answer pair;

a second determining module 64, configured to perform encoding and decoding processing on the third feature vector by using a preset model, and determine a second matching degree corresponding to the training question-answer pair;

an adjusting module 65, configured to adjust the preset model parameter according to a difference between the first matching degree and the second matching degree, so as to generate a verification model.

In practical use, the dialogue system reliability verification model generation apparatus provided in the embodiment of the present application may be configured in any electronic device to execute the aforementioned dialogue system reliability verification model generation method.

The dialog system reliability verification model generation device provided in the embodiment of the application may extract training question-answer pairs from a corpus set, where a first matching degree exists between a target second question and a target second question in the training question-answer pairs, and obtain at least one reference question-answer pair according to a similarity between the training question-answer pair and each of other second question-answer pairs in the corpus set, and then determine a third feature vector corresponding to the training question-answer pair according to a third vector corresponding to the training question-answer pair and each fourth vector corresponding to each reference question-answer pair, and further perform encoding and decoding processing on the third feature vector by using a preset model, determine a second matching degree corresponding to the training question-answer pair, and adjust preset model parameters according to a difference between the first matching degree and the second matching degree to generate a verification model. Therefore, the verification model is generated by adjusting the preset model parameters, so that the reliability of the dialogue system is automatically verified, the authenticity of the reliability verification of the dialogue system is improved, the verification efficiency is improved, and the labor cost is saved.

In a possible implementation form of the present application, the obtaining module 62 is specifically configured to:

or,

Further, in another possible implementation form of the present application, the obtaining module 62 is further configured to:

or,

In one possible implementation form of the present application, the above dialog system reliability verifying apparatus 60 further includes:

correspondingly, the second determining module 64 is specifically configured to:

Further, in another possible implementation form of the present application, the corpus set includes a first question-answer pair set in which a matching degree between a second question and a second answer sentence is greater than or equal to a threshold, and a second question-answer pair set in which a matching degree between the second question and the second answer sentence is less than a threshold;

accordingly, the above dialog system reliability verification device 60 further includes:

Further, in another possible implementation form of the present application, the above dialog system reliability check device 60 further includes:

It should be noted that the foregoing explanation on the embodiment of the dialog system reliability verification method shown in fig. 1 and fig. 2 and the foregoing explanation on the embodiment of the dialog system reliability verification model generation method shown in fig. 3 and fig. 4 are also applicable to the dialog system reliability verification model generation apparatus 60 of this embodiment, and are not repeated here.

The dialog system reliability verification model generation apparatus provided in the embodiment of the present application may perform, according to a similarity between a training question-answer pair and at least one reference question-answer pair, a matching degree between a second question and a second answer in each reference question-answer pair, an intention of the training question-answer pair, an intention of each reference question-answer pair, a field to which the training question-answer pair belongs, and/or a field to which each reference question-answer pair belongs, expand a third feature vector to generate a fourth feature vector, perform encoding and decoding processing on the fourth feature vector by using a preset model to determine a second matching degree corresponding to the training question-answer pair, and adjust a preset model parameter according to a difference between the first matching degree and the second matching degree to generate a verification model. Therefore, the accuracy of the verification model is further improved by expanding the content of the training data, so that the reliability of the dialogue system is automatically verified, the verification efficiency is not improved, the labor cost is saved, and the accuracy of the reliability verification of the dialogue system is further improved.

In order to implement the above embodiments, the present application further provides an electronic device.

Fig. 7 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.

As shown in fig. 7, the electronic device 200 includes:

a memory 210 and a processor 220, a bus 230 connecting different components (including the memory 210 and the processor 220), wherein the memory 210 stores a computer program, and when the processor 220 executes the program, the dialog system reliability check method or the dialog system reliability check model generation method according to the embodiment of the present application is implemented.

Bus 230 represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example, such architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, micro-channel architecture (MAC) bus, enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

Electronic device 200 typically includes a variety of electronic device readable media. Such media may be any available media that is accessible by electronic device 200 and includes both volatile and nonvolatile media, removable and non-removable media.

Memory 210 may also include computer system readable media in the form of volatile memory, such as Random Access Memory (RAM)240 and/or cache memory 250. The electronic device 200 may further include other removable/non-removable, volatile/nonvolatile computer system storage media. By way of example only, storage system 260 may be used to read from and write to non-removable, nonvolatile magnetic media (not shown in FIG. 7, commonly referred to as a "hard drive"). Although not shown in FIG. 7, a magnetic disk drive for reading from and writing to a removable, nonvolatile magnetic disk (e.g., a "floppy disk") and an optical disk drive for reading from or writing to a removable, nonvolatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 230 by one or more data media interfaces. Memory 210 may include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the application.

A program/utility 280 having a set (at least one) of program modules 270, including but not limited to an operating system, one or more application programs, other program modules, and program data, each of which or some combination thereof may comprise an implementation of a network environment, may be stored in, for example, the memory 210. The program modules 270 generally perform the functions and/or methodologies of the embodiments described herein.

Electronic device 200 may also communicate with one or more external devices 290 (e.g., keyboard, pointing device, display 291, etc.), with one or more devices that enable a user to interact with electronic device 200, and/or with any devices (e.g., network card, modem, etc.) that enable electronic device 200 to communicate with one or more other computing devices. Such communication may occur via input/output (I/O) interfaces 292. Also, the electronic device 200 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network such as the Internet) via the network adapter 293. As shown, the network adapter 293 communicates with the other modules of the electronic device 200 via the bus 230. It should be appreciated that although not shown in the figures, other hardware and/or software modules may be used in conjunction with the electronic device 200, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, among others.

The processor 220 executes various functional applications and data processing by executing programs stored in the memory 210.

It should be noted that, for the implementation process and the technical principle of the electronic device of this embodiment, reference is made to the foregoing explanation on the dialog system reliability verification method or the dialog system reliability verification model generation method according to this embodiment, and details are not described here again.

The electronic device provided in this application embodiment may execute the foregoing method for verifying reliability of a dialog system or method for generating a reliability verification model of a dialog system, obtain a first question-answer pair to be verified, obtain at least one reference question-answer pair from a corpus according to a similarity between the first question-answer pair and each second question-answer pair in the corpus, determine a first feature vector corresponding to the first question-answer pair according to a first vector corresponding to the first question-answer pair and each second vector corresponding to each reference question-answer, and perform encoding and decoding processing on the first feature vector by using a pre-generated verification model to generate a matching degree between a first question and a first answer sentence in the first question-answer pair. Therefore, the corpus is concentrated to be similar to the first question-answer pair, the reference question-answer pair with the matching degree is used as a reference, and the matching degree between the first question sentence and the first answer sentence in the first question-answer pair is determined by utilizing the pre-generated verification model, so that the reliability of the dialogue system is automatically verified, the authenticity of the reliability verification of the dialogue system is improved, the verification efficiency is improved, and the labor cost is saved.

In order to implement the above embodiments, the present application also proposes a computer-readable storage medium.

The computer readable storage medium stores thereon a computer program, and the computer program is executed by a processor to implement the dialog system reliability check method or the dialog system reliability check model generation method according to the embodiment of the present application.

In order to implement the foregoing embodiments, a further embodiment of the present application provides a computer program, which when executed by a processor, implements the dialog system reliability check method or the dialog system reliability check model generation method described in the embodiments of the present application.

In an alternative implementation, the embodiments may be implemented in any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the consumer electronic device, partly on the consumer electronic device, as a stand-alone software package, partly on the consumer electronic device and partly on a remote electronic device, or entirely on the remote electronic device or server. In the case of remote electronic devices, the remote electronic devices may be connected to the consumer electronic device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external electronic device (e.g., through the internet using an internet service provider).

Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention following, in general, the principles of the application and including such departures from the present disclosure as come within known or customary practice within the art to which the invention pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the application being indicated by the following claims.

It will be understood that the present application is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the application is limited only by the appended claims.

Claims

1. A dialog system reliability verification method, comprising:

acquiring a first question-answer pair to be verified, wherein the first question-answer pair comprises a first question sentence and a first answer sentence;

according to the similarity between the first question-answer pair and each second question-answer pair in the corpus, at least one reference question-answer pair is obtained from the corpus, and each second question-answer pair comprises a second question and a second answer sentence;

determining a first feature vector corresponding to the first question-answer pair according to a first vector corresponding to the first question-answer pair and each second vector corresponding to each reference question-answer pair;

and coding and decoding the first characteristic vector by using a pre-generated check model to generate the matching degree between the first question sentence and the first answer sentence in the first question-answer pair.

2. The method of claim 1, wherein the obtaining at least one reference question-answer pair from the corpus based on a similarity between the first question-answer pair and each second question-answer pair in the corpus comprises:

or,

3. The method of claim 1, wherein the obtaining at least one reference question-answer pair from the corpus based on a similarity between the first question-answer pair and each second question-answer pair in the corpus comprises:

or,

4. The method of claim 1, wherein after determining the first feature vector corresponding to the first question-answer pair, further comprising:

the encoding and decoding process of the first feature vector includes:

and carrying out encoding and decoding processing on the second feature vector.

5. The method of claim 4, wherein the corpus comprises a first set of question-answer pairs for which a second question-answer degree is greater than or equal to a threshold value, and a second set of question-answer pairs for which a second question-answer degree is less than a threshold value;

6. A dialogue system reliability check model generation method is characterized by comprising the following steps:

extracting training question-answer pairs from the corpus set, wherein each second question-answer pair in the corpus set comprises a second question and a second answer sentence, and a first matching degree exists between a target second question and the target second answer sentence in the training question-answer pairs;

acquiring at least one reference question-answer pair according to the similarity between the training question-answer pair and other second question-answer pairs in the corpus;

determining a third feature vector corresponding to the training question-answer pair according to the third vector corresponding to the training question-answer pair and each fourth vector corresponding to each reference question-answer pair;

encoding and decoding the third feature vector by using a preset model, and determining a second matching degree corresponding to the training question-answer pair;

and adjusting the preset model parameters according to the difference value between the first matching degree and the second matching degree to generate a verification model.

7. A dialog system reliability verification apparatus, comprising:

the system comprises a first acquisition module, a second acquisition module and a verification module, wherein the first acquisition module is used for acquiring a first question-answer pair to be verified, and the first question-answer pair comprises a first question sentence and a first answer sentence;

a second obtaining module, configured to obtain at least one reference question-answer pair from the corpus according to a similarity between the first question-answer pair and each second question-answer pair in the corpus, where each second question-answer pair includes a second question sentence and a second answer sentence;

a first determining module, configured to determine a first feature vector corresponding to the first question-answer pair according to a first vector corresponding to the first question-answer pair and each second vector corresponding to each reference question-answer pair;

and the first generation module is used for performing coding and decoding processing on the first feature vector by using a pre-generated check model to generate the matching degree between the first question sentence and the first answer sentence in the first question-answer pair.

8. A dialogue system reliability verification model generation apparatus, comprising:

the extraction module is used for extracting training question-answer pairs from the corpus set, each second question-answer pair in the corpus set comprises a second question sentence and a second answer sentence, and a first matching degree exists between a target second question sentence and the target second answer sentence in the training question-answer pair;

an obtaining module, configured to obtain at least one reference question-answer pair according to a similarity between the first training question-answer pair and each of the other second question-answer pairs in the corpus;

the first determining module is used for determining a third feature vector corresponding to the training question-answer pair according to a third vector corresponding to the training question-answer pair and each fourth vector corresponding to each reference question-answer pair;

a second determining module, configured to perform encoding and decoding processing on the third feature vector by using a preset model, and determine a second matching degree corresponding to the training question-answer pair;

and the adjusting module is used for adjusting the preset model parameters according to the difference value between the first matching degree and the second matching degree so as to generate a verification model.

9. An electronic device, comprising: memory, processor and program stored on the memory and executable on the processor, characterized in that the processor implements the dialog system reliability check method according to any of claims 1 to 5 or the dialog system reliability check model generation method according to claim 6 when executing the program.

10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, implements the dialogue system reliability check method according to any one of claims 1 to 5 or the dialogue system reliability check model generation method according to claim 6.