WO2022126987A1

WO2022126987A1 - Test method and apparatus for question-and-answer intention classification model, device and medium

Info

Publication number: WO2022126987A1
Application number: PCT/CN2021/091718
Authority: WO
Inventors: 宫雪
Original assignee: 平安科技（深圳）有限公司
Priority date: 2020-12-15
Filing date: 2021-04-30
Publication date: 2022-06-23
Also published as: CN112541739A; CN112541739B

Abstract

The present application relates to the technical field of artificial intelligence, and discloses a test method and apparatus for a question-and-answer intention classification model, a device and a medium. The method comprises: inputting a test sample subset corresponding to each product identifier into a corresponding question-and-answer intention classification model to be tested for intention prediction to obtain an intention prediction result set corresponding to each product identifier; accurately determining the intention prediction of each test sample according to the intention prediction result sets corresponding to each product identifier, question intention calibration data of test questions of the test sample subsets corresponding to each product identifier, and whether the test questions are intended for data calibration so as to obtain an intention prediction accurate result set corresponding to each product identifier; and generating a report according to the test sample subsets intention prediction accurate result set corresponding to each product identifier to obtain a target question-and-answer intention classification model test report. The method avoids the problem of manual calculation being time consuming and inaccurate.

Description

Test method, device, equipment and medium for question answering intent classification model

This application claims the priority of the Chinese patent application with the application number 2020114798351 and the invention titled "Testing method, device, equipment and medium for the question-and-answer intent classification model" filed with the China Patent Office on December 15, 2020, the entire content of which is approved by Reference is incorporated in this application.

technical field

The present application relates to the field of artificial intelligence technology, and in particular, to a testing method, apparatus, device and medium for a question-and-answer intent classification model.

Background technique

The model test of the classification model needs to be based on sample data. When the number of sample data is small, some testers will use tools such as Excel to perform manual calculation and statistics. The inventor realizes that when the number of sample data is large, manual calculation takes a long time and does not precise. Moreover, the model is continuously iteratively optimized, resulting in a very large number of calculations, which further increases the workload of the calculation.

technical problem

The purpose is to solve the technical problem that manual calculation is time-consuming and inaccurate after the classification model of the prior art is trained and the model is tested by manual calculation and statistics through Excel.

technical solutions

The main purpose of this application is to provide a test method, device, equipment and medium for a question-and-answer intent classification model, which aims to solve the problem that the prior art classification model is trained by manual calculation and statistics through Excel to perform model testing, resulting in manual calculation consumption. Long and inaccurate technical issues.

In order to achieve the above purpose of the invention, the present application proposes a method for testing a question-and-answer intent classification model, the method comprising:

acquiring a test sample set, where the test sample set includes a plurality of test samples, and the test samples include: product identification, test question sample data, test question sentence intent identification data, and test question intent identification data;

Divide the plurality of test samples by using the product identifiers to obtain a subset of test samples corresponding to each of the product identifiers;

Inputting the test sample subset corresponding to each product identifier into the corresponding question-and-answer intent classification model to be tested to perform intent prediction, and obtaining the respective intent prediction result set corresponding to each of the product identifiers;

Carry out each test according to the intent prediction result set corresponding to each of the product identifiers, the test question question intent determination data and the test question intent determination data of the test sample subsets corresponding to each of the product identifiers respectively. Accurately judge the intention prediction of the test sample, and obtain a set of accurate intention prediction results corresponding to each of the product identifiers;

The report is generated according to the respective corresponding test sample subsets of the product identifiers and the accurate intent prediction result set, to obtain the target question answering intent classification model test report. The target question answering intent classification model test report includes: the respective product identifiers corresponding to The precision data, recall data and total number of positive samples for each intent value of .

The present application also proposes a test device for a question-and-answer intent classification model, the device comprising:

A test sample acquisition module, configured to acquire a test sample set, the test sample set includes a plurality of test samples, and the test samples include: product identification, test question sample data, test question question sentence intent identification data, and test question intent identification data;

a test sample dividing module, configured to divide the plurality of test samples by using the product identifier, and obtain a test sample subset corresponding to each of the product identifiers;

an intent prediction module, configured to respectively input the test sample subset corresponding to each product identifier into the corresponding question-and-answer intent classification model to be tested to perform intent prediction, and obtain an intent prediction result set corresponding to each of the product identifiers;

Intent prediction accurate judgment module, used for each of the product identifiers corresponding to the intent prediction result set, the test question question sentence intent determination data and the test question of the test sample subset corresponding to each of the product identifiers respectively Whether the intention identification data is used to accurately judge the intention prediction of each of the test samples, and obtain a set of accurate intention prediction results corresponding to each of the product identifications;

The report generation module is configured to generate a report according to the respective corresponding test sample subsets and the accurate intent prediction result set of each of the product identifiers, and obtain a test report of the target question answering intent classification model, where the target question answering intent classification model test report includes: each Accuracy data, recall data and total number of positive samples of each intent value corresponding to the product identifiers.

The present application also proposes a computer device, including a memory and a processor, the memory stores a computer program, and the processor implements the following method steps when executing the computer program:

The present application also proposes a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the following method steps are implemented:

beneficial effect

A test method, device, equipment and medium for a question-and-answer intent classification model of the present application, by acquiring a test sample set, the test sample set includes a plurality of test samples, and the test samples include: product identification, test question sample data, and test question questions Intention-specific data and whether the test questions are intention-specific data; use product identifiers to divide multiple test samples to obtain test sample subsets corresponding to each product identifier; input the test sample subsets corresponding to each product identifier into their corresponding The question-and-answer intent classification model to be tested performs intent prediction, and obtains the corresponding intent prediction result set for each product identifier; according to the respective intent prediction result set corresponding to each product identifier and the test sample subset corresponding to each product identifier, the test question asks. Sentence intent rating data and whether the test question is intent rating data. Accurately judge the intent prediction of each test sample, and obtain an accurate result set of intent prediction corresponding to each product identifier; according to each product identifier, the corresponding test sample subset and intent prediction are accurate The result set is used for report generation, and the target question answering intent classification model test report is obtained. The target question answering intent classification model test report includes: accuracy data, recall data and total number of positive samples of each intent value corresponding to each product identifier, thus realizing the adoption of the test. The sample set is used to test the question-and-answer intent classification model to be tested and automatically generate the target question-and-answer intent classification model test report, which avoids manual model testing, avoids the problem of time-consuming and inaccurate manual calculation, and improves the accuracy of the question-and-answer intent classification model.

Description of drawings

1 is a schematic flowchart of a method for testing a question-and-answer intent classification model according to an embodiment of the present application;

FIG. 2 is a schematic block diagram of a structure of a testing device for a question-and-answer intent classification model according to an embodiment of the present application;

FIG. 3 is a schematic structural block diagram of a computer device according to an embodiment of the present application.

The realization, functional features and advantages of the present application will be further described with reference to the accompanying drawings in conjunction with the embodiments.

Embodiments of the present invention

In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application will be described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, but not to limit the present application.

In order to solve the technical problem that manual calculation is time-consuming and inaccurate after the classification model is trained in the prior art, the model is tested by means of manual calculation and statistics in Excel. The method is applied in the field of artificial intelligence technology. The test method for the question-answer intent classification model uses a test sample set to test the question-and-answer intent classification model to be tested and automatically generates a test report of the target question-answer intent classification model, which avoids manual model testing and avoids the problem of long and inaccurate manual calculation. , which improves the accuracy of the question answering intent classification model.

Referring to FIG. 1 , an embodiment of the present application provides a method for testing a question-and-answer intent classification model. The method includes:

S1: Obtain a test sample set, where the test sample set includes a plurality of test samples, and the test samples include: product identification, test question sample data, test question question sentence intent identification data, and test question intent identification data;

S2: dividing the plurality of test samples by using the product identifiers to obtain a test sample subset corresponding to each of the product identifiers;

S3: Input the test sample subset corresponding to each product identifier into the corresponding question-and-answer intent classification model to be tested to perform intent prediction, and obtain an intent prediction result set corresponding to each of the product identifiers;

S4: Perform each step according to the intent prediction result set corresponding to each of the product identifiers, the test question question sentence intent determination data and the test question intent determination data of the test sample subsets corresponding to each of the product identifiers respectively. Accurately judge the intention prediction of each of the test samples, and obtain a set of accurate intention prediction results corresponding to each of the product identifiers;

S5: Generate a report according to the respective test sample subsets corresponding to each of the product identifiers and a set of accurate intent prediction results, and obtain a target question answering intent classification model test report, where the target question answering intent classification model test report includes: each of the product identifiers Accuracy data, recall data, and total number of positive samples corresponding to each intent value.

In this embodiment, a test sample set is obtained, and the test sample set includes a plurality of test samples, and the test samples include: product identification, test question sample data, test question sentence intent identification data, and test question intent identification data; the product identification to many Divide the test samples to obtain the corresponding test sample subsets for each product identifier; input the test sample subsets corresponding to each product identifier into the corresponding question-and-answer intent classification model to be tested for intent prediction, and obtain the respective product identifiers. Corresponding intent prediction result set; carry out each test sample according to the respective intent prediction result set corresponding to each product identifier, the test question question intent rating data and the test question intent rating data corresponding to each product identifier respective test sample subset. According to each product identifier corresponding to the corresponding test sample subset and intent prediction accurate result set, the report is generated, and the target question answering intent classification model test report is obtained, the target The question and answer intent classification model test report includes: accuracy data, recall rate data and total number of positive samples of each intent value corresponding to each product identifier, so that the question and answer intent classification model to be tested is tested using the test sample set and the target question and answer is automatically generated. The intent classification model test report avoids manual model testing, avoids the problem of time-consuming and inaccurate manual calculations, and improves the accuracy of the question answering intent classification model.

For S1, the test sample set input by the user may be obtained, or the test sample set sent by the third-party application system.

It can be understood that, in the test sample, product identification, test question sample data, test question question intent identification data, and test question intent identification data are in one-to-one correspondence.

Optionally, the test sample further includes a sample identification. The sample identifier may be an identifier that uniquely identifies a test sample, such as a sample name, a sample ID, or the like.

The product identifier may be an identifier that uniquely identifies a product, such as a product name, a product ID, or the like.

The test question sample data refers to the text data of the questions raised by the user. Each test question sample data corresponds to the text data of a question posed by a user in a round of dialogue.

The test question question sentence intent calibration data refers to the question sentence intent calibration data corresponding to the test question sample data. Question intent includes multiple intent values. For example, the intent value of the question intent under the product identifier aries includes: the previous application failed, there is a problem with the credit report, and how do I know my phone number, which is not specifically limited in this example. For another example, when the sample data of the test question is "I haven't done it last month, and I don't know if I can pass it now", the test question means the specified data as "The previous application failed", and when the sample data of the test question is "Product A1" Yeah, product A2 has been tried without success", the test question question means the specified data is "the previous application failed", when the test question sample data is "the company G1 applied for a credit yesterday and said it is not qualified", The test question question means the specified data is "Failed to apply before", when the test question sample data is "That credit report is not very good", the test question question means the specified data is "There is a problem with the credit report", when the test question sample data is "The credit report is not good" When the test question is "I haven't done credit investigation", the test question question means the specified data is "There is a problem with the credit investigation", and when the test question sample data is "Where did you get the phone number", the test question question means the specified data is "How do you know me?" telephone", which will not be specific in this example.

Whether the test question is intended to be calibrated data refers to the calibration data of whether the test question sample data is intended or not. Whether the intent includes two intent values, the two intent values are yes and no. For example, when the sample data of the test question is "true", whether the test question intends to define the data as "yes" is not specifically limited in this example.

Optionally, the step of obtaining the test sample set includes:

S11: Obtain a model test request, where the model test request carries the storage address of the Excel file and the name of the Excel file;

Wherein, the model test request may be sent by the user, or may be actively triggered by the program file of the present application.

A model test request refers to a request to test the question-and-answer intent classification model to be tested.

S12: Obtain the Excel file according to the storage address of the Excel file and the name of the Excel file carried by the model test request, and obtain the target Excel file;

Wherein, under the directory of the storage address of the Excel file, a file with the same file name as the Excel file name is obtained, and the obtained file is used as the target Excel file.

S13: Read data from the target Excel file to obtain the test sample set.

From the target Excel file, the data is read row by row starting from the first row, and each row of data is taken as a test sample; all the test samples are taken as the test sample set.

It can be understood that the table headers in the target Excel file include but are not limited to: sample identification, product identification, test question sample data, test question question intent identification data, and test question intent identification data.

For S2, the test samples with the same product identification are put into a subset, and the subset is taken as the test sample subset corresponding to the product identification. That is, each product identifier corresponds to a test sample subset, and all test samples in each test sample subset have the same product identifier.

For S3, each test sample in the test sample subset corresponding to each product identifier is sequentially input into the question-and-answer intent classification model to be tested corresponding to the product identifier to perform intent prediction, and the corresponding product identifier is obtained. The intent prediction result of the test sample subset, and all the obtained intent prediction results are taken as the intent prediction result set corresponding to the product identifier corresponding to the test sample subset. That is, each product identifier corresponds to an intent prediction result set. By making the product identification of the test sample for testing the question-and-answer intent classification model to be tested be the same as the product identification corresponding to the question-and-answer intent classification model to be tested, it is beneficial to improve the accuracy of the test.

Understandably, each test sample corresponds to an intent prediction result. The intent prediction result has only one value, and the intent prediction result is: question intent or intent.

The question and answer intent classification model to be tested, that is, the question and answer intent classification model that has been trained and needs further testing.

The question and answer intent classification model is a model that predicts the intent of a question sentence and whether it is intended for text data.

For S4, the intent prediction of each test sample is accurately judged on the intent prediction result set corresponding to the same product identifier, the test question question intent determination data and the test question intent determination data of the test sample subset, An accurate result set of intent prediction corresponding to the product identifier is obtained. That is, each product identifier corresponds to a set of accurate results for intent prediction.

For example, there are 3 test samples S1 in the test sample subset of the product identifier C1 (the test question is intended to indicate that the data is empty, and the test question is to indicate that the data is yes), S2 (the test question is intended to be defined as the data) SF2, whether the test question means the specified data is empty), S3 (the test question means the specified data is SF2, whether the test question means the specified data is empty), the intent prediction corresponding to the test sample S1 in the intent prediction result set of the product identifier C1 The result is question intent SF1, the intent prediction result corresponding to test sample S2 is question intent SF2, and the intent prediction result corresponding to test sample S3 is question intent SF1, then the accurate result of intent prediction corresponding to test sample S1 is wrong (test question The question intent is defined data is empty, the test question is intended to define the data is yes, the intent prediction result is question intent SF1, the test question question intent is not the same as the intent prediction result), the intent prediction corresponding to the test sample S2 is accurate result It is correct (the test question means the intended data is SF2, the test question means that the intended data is empty, the intent prediction result is the question intent SF2, the test question is the same as the intent prediction result), the corresponding test sample S3 If the intent prediction is accurate, the result is an error (the test question asks the intent-specific data to be SF2, the test question does the intent-specific data to be empty, the intent prediction result is the question intent SF1, and the test-question intent-specific data is not the same as the intent prediction result), This example is not specifically limited.

For S5, perform statistical calculation of each intent value according to the test sample subset corresponding to the same product identifier and the intent prediction accurate result set, and obtain the accuracy data, recall rate data and positive samples of each intent value corresponding to the product identifier The total number; according to the accuracy rate data, recall rate data and the total number of positive samples corresponding to all the product identifiers, the report is generated according to the preset report generation rule, and the test report of the target question answering intent classification model is obtained.

Preset report generation rules include but are not limited to: report templates.

A positive sample is the number of test samples in which the calibration data (that is, the test question is intended to be calibrated and the test question is intended to be calibrated) is the same as the intended value to be calculated.

For example, when calculating the accuracy data, recall data and the total number of positive samples of the intent value Y1, the positive samples refer to the test samples whose calibration data is Y1, and the test samples whose calibration data is not Y1 are negative samples, which are not specifically limited in this example. .

The accuracy rate refers to how many of the judgments are correct, that is, the positive samples are judged as positive, and the negative samples are judged as negative; there is a total of TP (the number of positive samples that are predicted to be positive) )+FN (the number of positive samples predicted to be negative)+FP (the number of negative samples predicted to be positive)+TN (the number of negative samples predicted to be negative), so the accuracy rate: Acc=(TP+TN) /(TP+TN+FN+FP).

The recall rate is relative to the sample, that is, how many positive samples in the sample are predicted correctly, there are TP, all positive samples have two directions, one is judged to be positive, the other is wrong The judgment is negative, so there are a total of TP+FN, so the recall rate R=TP/(TP+FN).

In one embodiment, the test sample subset corresponding to each product identifier is input into the corresponding question-and-answer intent classification model to be tested to perform intent prediction, and a set of intent prediction results corresponding to each product identifier is obtained. steps, including:

S31: Extract a test sample subset from the test sample subsets corresponding to each of the product identifiers by using the product identifiers to be predicted, and obtain a target test sample subset, where the product identifiers to be predicted are among the respective product identifiers any of the;

S32: Search from the model library to be tested according to the product identifier to be predicted, to obtain the target question-and-answer intent classification model to be tested;

S33: Input each of the test samples in the target test sample subset into the target question-and-answer intent classification model to be tested to perform intent prediction, and obtain the intent prediction result set corresponding to the product identifier to be predicted;

S34: Repeat the step of using the product identifiers to be predicted to extract the test sample subsets from the test sample subsets corresponding to the respective product identifiers to obtain the target test sample subsets, until all the product identifiers corresponding to the product identifiers are determined. The intent prediction result set.

In this embodiment, the intent prediction result set corresponding to each of the product identifiers is determined, which provides a basis for the subsequent determination of the accuracy and recall rate of the question-and-answer intent classification model to be tested.

For 31, use any one of the product identifiers as the product identifier to be predicted; search the product identifier to be predicted in the test sample subset corresponding to each of the product identifiers, and use the product identifier to be predicted in each of the product identifiers The test sample subset corresponding to the product identifier found in the test sample subset corresponding to each product identifier is used as the target test sample subset.

In step 32, the product identifier to be predicted is searched from the model library to be tested, and the question answer intent classification model to be tested corresponding to the product identifier found in the model library to be tested is used as the target question answer intent classification model to be tested.

The model library to be tested includes: a correspondence table between product identifiers and model identifiers, and data of the question-and-answer intent classification model to be tested. The product identification and model identification correspondence table includes: product identification, model identification, and each product identification corresponds to a model identification.

The model identifier may be an identifier that uniquely identifies a question-answer intent classification model to be tested, such as a model name, a model ID, or the like.

In step 33, input each test sample in the target test sample subset into the target question-and-answer intent classification model to be tested to perform intent prediction, and obtain a plurality of the intent prediction results corresponding to the product identifier to be predicted ; Take all the intent prediction results corresponding to the product identifier to be predicted as the intent prediction result set corresponding to the product identifier to be predicted. That is to say, the target question-and-answer intent classification model to be tested only performs intent prediction on one test sample at a time.

For step 34, steps S31 to S34 are repeatedly performed until the intent prediction result sets corresponding to all the product identifiers are determined.

In one embodiment, the above-mentioned intent prediction result set corresponding to each of the product identifiers, the test question question sentence intent determination data and the test question whether the test sample subset corresponding to each of the product identifiers are respectively The intention identification data is used to accurately judge the intention prediction of each of the test samples, and the steps of obtaining an accurate intention prediction result set corresponding to each of the product identifications include:

S41 : respectively process the test question question sentence intent determination data and the test question intent determination data of each of the test samples in the test sample set according to the intent priority, to obtain a test sample set after the intent priority processing;

S42: Extracting the intention prediction results in turn from the intention prediction result set corresponding to each product identifier, respectively, to obtain the target intention prediction result;

S43: When the target intention prediction result is whether the target intention is intended or not, extract whether the test question is intended to determine the data from the test sample set after the intention priority processing according to the target intention prediction result, and obtain the test question to be determined Whether the target intent prediction result is the same as whether the target intent prediction result and the to-be-determined test question are intended to delineate the data, determine whether the intent prediction result corresponding to the target intent prediction result is correct, otherwise determine the target The accurate result of the intent prediction corresponding to the intent prediction result is an error;

S44: When the target intent prediction result is a question sentence intent, extract the test question question sentence intent determination data from the test sample set after the intent priority processing according to the target intent prediction result, to obtain the test to be judged Question question intent identification data, when the target intent prediction result and the test question question intent identification data to be judged are the same, determine that the target intent prediction result corresponds to the intent prediction accurate result is correct, otherwise determine The accurate result of the intention prediction corresponding to the target intention prediction result is an error;

S45: Repeat the step of sequentially extracting the intent prediction results from the intent prediction result set corresponding to each product identifier to obtain the target intent prediction result, until the intent of all the intent prediction results is determined predict accurate results;

S46: Determine a set of accurate intention prediction results corresponding to each of the product identifiers according to all the accurate intention prediction results.

This embodiment realizes the accurate judgment of the intention prediction of each test sample, which provides a basis for the subsequent judgment of the accuracy and recall rate of the question-and-answer intention classification model to be tested; and the test samples are processed according to the intention priority. When the intent priority is satisfied, it is ensured that the calibration data of each test sample has a unique intent value, which is conducive to improving the accuracy of model testing and making the optimization of the model conform to the intent priority.

For step 41 , the test question question intent rating data and the test question intent rating data of the same test sample are processed according to intent priority, and a processed test sample with intent priority corresponding to the test sample is obtained after processing. Therefore, when the test question of the same test sample asks whether the intent calibration data and the test question whether the intent calibration data exists, the calibration data with the highest intent priority is determined according to the intent priority as the test sample after the intent priority processing. Calibration data. That is to say, the calibration data of the test sample after intent priority processing has only one intent value.

For step 42, extract an intent prediction result from the intent prediction result set corresponding to each product identifier according to a preset extraction rule, and use the extracted intent prediction result as a target intent prediction result. The preset extraction rules include but are not limited to: extracting in sequence according to the sequence of the sample identifiers.

For 43, when the target intention prediction result is whether it is intention or not, it means that it is necessary to compare it with the test question whether it is intended or not; the sample identifier of the test sample corresponding to the target intention prediction result is in the intention priority Extract the test samples after the intention priority processing from the processed test sample set, extract whether the test question is intentional or not from the extracted test sample after the intention priority processing, and determine whether the extracted test question is Intention calibration data as the test question to be judged is the intention calibration data; when the target intention prediction result and the to-be-judged test question whether the intention calibration data are the same, it means that the target intention prediction result is correct, then determine The accurate result of the intention prediction corresponding to the target intention prediction result is correct; when the target intention prediction result and the test question to be judged whether the intention specification data are different, it means that the target intention prediction result is wrong , at this time, it is determined that the accurate result of the intention prediction corresponding to the target intention prediction result is an error.

For 44, when the target intent prediction result is the question intent, it means that it needs to be compared with the test question question intent identification data; the sample identifier of the test sample corresponding to the target intent prediction result is in the intent Extracting the test sample after the intention priority processing from the test sample set after the priority processing, extracting the test question question sentence intention determination data from the extracted test sample after the intention priority processing, and using the extracted The test question question intent determination data is used as the test question intent determination data to be judged; when the target intent prediction result is the same as the test question intent determination data to be determined, it means the target intent prediction result To be correct, at this time, it is determined that the accurate result of the intent prediction corresponding to the target intent prediction result is correct; when the target intent prediction result and the to-be-determined question sentence intent determination data are different, it means that the target intent prediction result is not the same. If the target intention prediction result is wrong, at this time, it is determined that the intention prediction accurate result corresponding to the target intention prediction result is wrong.

For step 45, steps S42 to S45 are repeatedly executed until the accurate results of the intention prediction of all the intention prediction results are determined.

For 46, the accurate results are predicted according to all the intentions as a set of accurate prediction results of the intention corresponding to each of the product identifiers.

In one embodiment, the above-mentioned processing of the test question question intention identification data and the test question intention identification data of each of the test samples in the test sample set are carried out according to the intention priority, and the intention priority processing is obtained. The steps to test the sample collection include:

S411 : respectively comparing the test question question intent rating data and the test question intent rating data of each of the test samples in the test sample set;

S412: Delete the test question question intent calibration data of the test sample when both the test question question intent calibration data and the test question intent calibration data of the test sample exist. Processing, get the test sample after intent priority processing;

S413: Determine the set of test samples after the intention priority processing according to all the test samples processed by the intention priority.

In this embodiment, the test samples are processed according to the intent priority, and when the intent priority is satisfied, it is ensured that the calibration data of each test sample has a unique intent value, which is beneficial to improve the accuracy of the model test and make the The optimization of the model conforms to the intent priority.

For 411 , the test question question intent rating data and the test question intent rating data of the same test sample are compared each time.

For step 412, when there are both the test question question intention calibration data and the test question intention calibration data of the test sample, it means that the test sample has two calibration data, because the intention is given priority The level is whether the intent is higher than the OA intent. At this time, delete processing is performed on the test question question intent determination data of the test sample, so as to retain the intent determination data corresponding to the test question with high intent priority. , the test sample with only one calibration data after deletion processing is regarded as the test sample after intent priority processing.

For 413, use all the test samples after the intention priority processing as the set of test samples after the intention priority processing.

In one embodiment, the above-mentioned steps of generating a report according to the respective corresponding test sample subsets and intent prediction accurate result sets of the product identifiers to obtain the test report of the target question answering intent classification model include:

S51: Use the target product identifier to extract data from the test sample subset and the intent prediction accurate result set corresponding to each of the product identifiers, and obtain the test sample subset to be calculated and the intent prediction accurate result set to be calculated. The product identification is any of the respective said product identifications;

S52: Calculate the accuracy rate and recall rate of each intent value according to the test sample subset to be calculated and the intent prediction accurate result set to be calculated, and obtain each of the intent values corresponding to the target product identifier. the precision data, the recall data, and the total number of positive samples;

S53: Repeatedly executing the use of target product identifiers to extract data from the respective corresponding test sample subsets and intent prediction accurate result sets to obtain the test sample subsets to be calculated and the intent prediction accurate result sets to be calculated , the target product identification is any one of the product identifications, until the accuracy data, the recall data and the positive samples of each of the intention values corresponding to all the product identifications are determined total;

S54: Generate a report according to the accuracy rate data, the recall rate data, and the total number of positive samples of the respective intent values corresponding to the respective product identifiers, to obtain the target question answering intent classification model test report.

This embodiment automatically generates reports according to the respective test sample subsets corresponding to each of the product identifiers and the accurate result set of intention prediction, which avoids manual model testing, avoids the problem of time-consuming and inaccurate manual calculation, and improves the classification of question-and-answer intentions. accuracy of the model.

For 51, extract any product identification from each of the product identifications as the target product identification; search the target product identification in the test sample subset corresponding to each of the product identifications, and use the corresponding product identification in each of the product identifications. The test sample subset corresponding to the product identifier found in the test sample subset is taken as the test sample subset to be calculated; the target product identifier is searched in the respective corresponding intent prediction accurate result sets of the product identifiers, and the target product identifier will be searched in each The set of accurate intention prediction results corresponding to each of the product identifiers is used as the set of accurate intention prediction results to be calculated.

At step 52 , extracting intent values according to the subset of test samples to be calculated and the set of accurate intent prediction results to be calculated, to obtain a target intent value set, wherein each intent value in the target intent value set is unique .

Extracting the intent values from the target intent value set in turn to obtain the intent value to be calculated; according to the test sample subset to be calculated and the intent prediction accurate result set to be calculated, the accuracy of the intent value to be calculated is performed. rate calculation and recall rate calculation, and obtain the accuracy rate data of the intent value to be calculated corresponding to the target product identifier, the recall rate data of the intent value to be calculated, and the total value of the intent value to be calculated. Describe the total number of positive samples; repeat the steps of sequentially extracting intent values from the target intent value set to obtain the intent value to be calculated, until the accuracy data, the recall data and the total number of positive samples.

For step 53, step S51 to step S53 are repeatedly performed until the accuracy rate data, the recall rate data, and the total number of positive samples of each of the intention values corresponding to all the product identifiers are determined.

For step 54, generate a report according to the preset report generation rule according to the accuracy rate data, the recall rate data, and the total number of positive samples of the respective intent values corresponding to the respective product identifiers, and use the generated report as The target question answering intent classification model test report.

In one embodiment, according to the test sample subset to be calculated and the intention prediction accurate result set to be calculated, the accuracy calculation and recall calculation of each intention value are performed, and each target product identifier corresponding to the target product identifier is obtained. The steps of the accuracy rate data, the recall rate data and the total number of positive samples of the intent value include:

S521: Calculate the total number of test samples according to the subset of test samples to be calculated, to obtain the total number of test samples corresponding to the target product identifier;

S522: Calculate the number of positive sample correct predictions for each of the intent values according to the test sample subset to be calculated and the to-be-calculated set of accurate intent prediction results, to obtain each of the intents corresponding to the target product identifiers The number of correct predictions for positive samples of the value;

S523: Calculate the number of negative sample correct predictions for each of the intent values according to the subset of test samples to be calculated and the set of accurate intent prediction results to be calculated, to obtain each of the intents corresponding to the target product identifiers The number of correct predictions for negative samples of the value;

S524: Perform an accuracy calculation according to the total number of test samples corresponding to the target product identifier, the number of positive sample correct predictions of each of the intent values, and the correct number of negative samples of each of the intent values, to obtain the corresponding target product identifier. the accuracy data for each of the intent values;

S525: Calculate the total number of the test samples for each of the intent values according to the subset of test samples to be calculated, to obtain the total number of positive samples for each of the intent values corresponding to the target product identifier;

S526: Calculate the recall rate according to the total number of positive samples of each of the intent values corresponding to the target product identifiers and the correct predicted number of positive samples of each of the intent values, to obtain each of the intents corresponding to the target product identifiers value of the recall data.

This embodiment realizes the automatic calculation of the accuracy rate and recall rate of each intent value according to the subset of test samples to be calculated and the set of accurate results of intent prediction to be calculated, which provides a basis for subsequent report generation.

For 521, perform a total number calculation on the test samples in the to-be-calculated test sample subset to obtain the total number of test samples corresponding to the target product identifier.

For 522, extract the intent value according to the test sample subset to be calculated and the intent prediction accurate result set to be calculated to obtain the intent value set to be deduplicated; perform deduplication processing on the intent value set to be deduplicated , obtain the target intent value set; extract the intent value from the target intent value set, and obtain the intent value to be calculated; perform the to-be-calculated test sample subset and the to-be-calculated intent prediction accurate result set. The calculation of the number of correct predictions of positive samples of the intent value, to obtain the number of correct predictions of positive samples of the intent value to be calculated corresponding to the target product identifier; repeated execution to extract the intent value from the target intent value set to obtain the intent value to be calculated until the positive sample correct prediction number of each of the intention values corresponding to the target product identifier is determined.

The number of correct predictions for positive samples means that the specified data is the intent value to be calculated, and the intent prediction result is also the intent value to be calculated.

For 523, extract the intent value from the target intent value set to obtain the intent value to be calculated; extract the intent value from the target intent value set to obtain the intent value to be calculated; according to the test sample subset to be calculated and The set of accurate results of intent prediction to be calculated is performed to calculate the correct number of negative samples of the intent value to be calculated, and the correct number of negative samples of the intent value to be calculated corresponding to the target product identifier is obtained; The step of extracting the intent value from the value set, and obtaining the intent value to be calculated, until the correct prediction number of negative samples of each of the intent values corresponding to the target product identifier is determined.

The number of correct predictions for negative samples means that the specified data is not the intent value to be calculated, and the intent prediction result is not the intent value to be calculated.

In step 524, the intent value is extracted from the target intent value set, and the intent value to be calculated is obtained; the positive sample correct prediction number and the negative sample correct prediction number of the intent value to be calculated corresponding to the target product identifier are added, Obtain the total number of correct predictions of the intention values to be calculated corresponding to the target product identification; divide the total number of correct predictions of the intention values to be calculated corresponding to the target product identification by the total number of test samples corresponding to the target product identification to obtain the total number of correct predictions. the accuracy data of the intent value to be calculated corresponding to the target product identifier; repeat the steps of extracting the intent value from the target intent value set to obtain the intent value to be calculated, until the target product identifier corresponding to each the accuracy data for the intent value.

For 525, extract the intent value from the target intent value set to obtain the intent value to be calculated; perform the calculation of the total number of the test samples corresponding to the intent value to be calculated on the subset of test samples to be calculated to obtain the The total number of positive samples of the intent value to be calculated corresponding to the target product identifier, repeat the steps of extracting the intent value from the target intent value set to obtain the intent value to be calculated, until it is determined that the target product identifier corresponds to the total number of positive samples for each of the intent values.

For 526, extract the intent value from the target intent value set to obtain the intent value to be calculated; divide the positive sample correct prediction number of the intent value to be calculated corresponding to the target product identifier by the to-be-calculated intent value corresponding to the target product identifier The total number of positive samples of the calculated intent values is obtained to obtain the recall rate data of the intent values to be calculated corresponding to the target product identifiers; repeating the extraction of intent values from the target intent value set is performed to obtain the to-be-calculated intent values. The step of intent value is until the recall rate data of each of the intent values corresponding to the target product identifier is determined.

In an embodiment, the above-mentioned report is generated according to the accuracy data, the recall data and the total number of positive samples of each of the intention values corresponding to each of the product identifiers, to obtain the target question answering intention classification Steps for model test reporting, including:

S61: Generate an Excel document according to the accuracy rate data, the recall rate data, and the total number of positive samples of the respective intent values corresponding to the respective product identifiers, to obtain the target question answering intent classification model test report;

S62: Obtain a report download request, where the report download request carries download mode data;

S63: Send the target question answering intent classification model test report according to the download method data.

This embodiment realizes the generation of the test report of the target question answering intent classification model in the Excel document format, thereby facilitating the secondary processing of the data and satisfying the personalized needs of the user.

For step 61, according to the accuracy data, the recall data, and the total number of positive samples of each of the intention values corresponding to each of the product identifiers, generate an Excel document according to a preset chart rule, and obtain the target question and answer Intent classification model test report;

For 62, the report download request sent by the user is obtained.

The report download request is a request to download the test report of the target question answering intent classification model.

The download method data includes but is not limited to: sending to a preset mailbox, sending it to a third-party software system according to a preset transmission method, and storing it in a local folder according to a preset path.

For 63, when the download mode data is to be sent to the preset mailbox, send the target question answering intent classification model test report to the preset mailbox; when the download mode data is to be sent to the third-party software system by the preset transmission mode, send the The target question answering intent classification model test report is sent to the third-party software system in a preset transmission mode; when the download mode data is stored in a local folder according to a preset path, the target question answering intent classification model test report is stored in the preset. Set the local folder corresponding to the path.

Referring to FIG. 2, the present application also proposes a test device for a question-and-answer intent classification model, the device includes:

The test sample acquisition module 100 is configured to acquire a test sample set, the test sample set includes a plurality of test samples, and the test samples include: product identification, test question sample data, test question question sentence intent identification data, and test question intent calibration data;

A test sample dividing module 200, configured to divide the plurality of test samples by using the product identifiers to obtain a test sample subset corresponding to each of the product identifiers;

The intent prediction module 300 is configured to respectively input the test sample subset corresponding to each product identifier into the corresponding question-and-answer intent classification model to be tested to perform intent prediction, and obtain an intent prediction result set corresponding to each of the product identifiers;

The intention prediction accuracy judgment module 400 is configured to use the test question question sentence intention identification data and the test according to the intention prediction result set corresponding to each of the product identifiers and the test sample subset corresponding to each of the product identifiers respectively. Whether the question is intended to identify the data to accurately judge the intent prediction of each of the test samples, and obtain an accurate result set of intent prediction corresponding to each of the product identifiers;

The report generation module 500 is configured to generate a report according to the respective test sample subsets corresponding to each of the product identifiers and a set of accurate intention prediction results to obtain a test report of the target question answering intent classification model, where the target question answering intent classification model test report includes: Accuracy data, recall data, and total number of positive samples of each intent value corresponding to each of the product identifiers.

Referring to FIG. 3 , an embodiment of the present application further provides a computer device. The computer device may be a server, and its internal structure may be as shown in FIG. 3 . The computer device includes a processor, memory, a network interface, and a database connected by a system bus. Among them, the processor of the computer design is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium, an internal memory. The nonvolatile storage medium stores an operating system, a computer program, and a database. The memory provides an environment for the execution of the operating system and computer programs in the non-volatile storage medium. The database of the computer equipment is used for storing data such as the testing method of the question-answering intent classification model. The network interface of the computer device is used to communicate with an external terminal through a network connection. The computer program, when executed by a processor, implements a method for testing a question answering intent classification model. The test method for the question-answer intent classification model includes: acquiring a test sample set, the test sample set including a plurality of test samples, the test samples including: product identification, test question sample data, test question question sentence intent identification data and Whether the test question is intended to denote data; use the product identifier to divide the multiple test samples to obtain a test sample subset corresponding to each product identifier; separate the test sample subset corresponding to each product identifier Input the respective corresponding question-and-answer intent classification models to be tested to perform intent prediction, and obtain a set of intent prediction results corresponding to each of the product identifiers; The corresponding test sample sub-set of the test question question is intended to determine the data and whether the test question is intended to determine the data to accurately determine the intention prediction of each of the test samples, and obtain the corresponding intention prediction of each of the product identifiers is accurate. Result set; generate a report according to the corresponding test sample subsets of each of the product identifiers and the accurate result set of intention prediction, and obtain a test report of the target question answering intent classification model, and the target question answering intent classification model test report includes: each of the products Identify the precision data, recall data, and total number of positive samples corresponding to each intent value.

An embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, a method for testing a question-and-answer intent classification model is implemented, including the steps of: acquiring a test sample set, the The test sample set includes a plurality of test samples, and the test samples include: product identification, test question sample data, test question sentence intent identification data, and test question intent identification data; using the product identification to identify the plurality of test samples Divide and obtain the respective test sample subsets corresponding to each of the product identifiers; respectively input the test sample subsets corresponding to each of the product identifiers into the corresponding question-and-answer intent classification model to be tested to perform intent prediction, and obtain each of the said product identifiers. The intent prediction result set corresponding to each product identifier; the intent prediction result set corresponding to each product identifier, the test question question sentence intent determination data and the test sample subset corresponding to each product identifier respectively. Whether the test question intends to identify the data to accurately judge the intent prediction of each of the test samples, and obtain an accurate result set of intent prediction corresponding to each of the product identifiers; according to the respective corresponding test sample subsets and intent predictions of each of the product identifiers A report is generated on the accurate result set, and a target question answering intent classification model test report is obtained. The target question answering intent classification model test report includes: accuracy data, recall data and total number of positive samples of each intent value corresponding to each of the product identifiers.

The test method of the above-mentioned question and answer intent classification model is obtained by obtaining a test sample set, the test sample set includes a plurality of test samples, and the test samples include: product identification, test question sample data, test question question sentence intent identification data, and test question intention Calibration data; use product identifiers to divide multiple test samples to obtain test sample subsets corresponding to each product identifier; respectively input the test sample subsets corresponding to each product identifier into the corresponding question-and-answer intent classification model to be tested. Intent prediction, obtain the corresponding set of intention prediction results for each product identifier; according to the corresponding intent prediction result set of each product identifier and the test sample subset corresponding to each product identifier The intent prediction data is used to accurately judge the intent prediction of each test sample, and the corresponding intent prediction accurate result set corresponding to each product identifier is obtained; the report is generated according to the corresponding test sample subset and intent prediction accurate result set corresponding to each product identifier, and the target is obtained. Question and answer intent classification model test report, the target question answer intent classification model test report includes: accuracy data, recall rate data and total number of positive samples of each intent value corresponding to each product identifier, thus realizing the question and answer intent classification to be tested by using the test sample set The model is tested and the target question answering intent classification model test report is automatically generated, which avoids manual model testing, avoids the problem of time-consuming and inaccurate manual calculation, and improves the accuracy of the question answering intent classification model.

The computer storage medium can be non-volatile or volatile.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage In the medium, when the computer program is executed, it may include the processes of the above-mentioned method embodiments. Wherein, any reference to memory, storage, database or other medium provided in this application and used in the embodiments may include non-volatile and/or volatile memory. Nonvolatile memory may include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double-rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous Link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

It should be noted that, herein, the terms "comprising", "comprising" or any other variation thereof are intended to encompass non-exclusive inclusion, such that a process, device, article or method comprising a series of elements includes not only those elements, It also includes other elements not expressly listed or inherent to such a process, apparatus, article or method. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in the process, apparatus, article, or method that includes the element.

The above are only the preferred embodiments of the present application, and are not intended to limit the scope of the patent of the present application. Any equivalent structure or equivalent process transformation made by using the contents of the description and drawings of the present application, or directly or indirectly applied to other related The technical field is similarly included in the scope of patent protection of this application.

Claims

A method for testing a question-and-answer intent classification model, wherein the method includes:

acquiring a test sample set, where the test sample set includes a plurality of test samples, and the test samples include: product identification, test question sample data, test question sentence intent identification data, and test question intent identification data;

Divide the plurality of test samples by using the product identifiers to obtain a subset of test samples corresponding to each of the product identifiers;

Inputting the test sample subset corresponding to each product identifier into the corresponding question-and-answer intent classification model to be tested to perform intent prediction, and obtaining the respective intent prediction result set corresponding to each of the product identifiers;

Carry out each test according to the intent prediction result set corresponding to each of the product identifiers, the test question question intent determination data and the test question intent determination data of the test sample subsets corresponding to each of the product identifiers respectively. Accurately judge the intention prediction of the test sample, and obtain a set of accurate intention prediction results corresponding to each of the product identifiers;

The report is generated according to the respective corresponding test sample subsets of the product identifiers and the accurate intent prediction result set, to obtain the target question answering intent classification model test report. The target question answering intent classification model test report includes: the respective product identifiers corresponding to The precision data, recall data and total number of positive samples for each intent value of .
The method for testing a question-and-answer intent classification model according to claim 1 , wherein the test sample subset corresponding to each product identifier is input into the corresponding question-and-answer intent classification model to be tested to perform intent prediction, and obtain each The steps of the respective corresponding intent prediction result sets of the product identifiers include:

The test sample subset is extracted from the test sample subset corresponding to each of the product identifiers by using the product identifier to be predicted, and the target test sample subset is obtained, and the product identifier to be predicted is any one of the product identifiers. One;

Search from the model library to be tested according to the product identifier to be predicted, and obtain the question-answer intent classification model of the target to be tested;

Inputting each of the test samples in the target test sample subset into the target question-and-answer intent classification model to be tested for intent prediction, and obtaining the intent prediction result set corresponding to the product identifier to be predicted;

Repeat the steps of using the product identifiers to be predicted to extract the test sample subsets from the test sample subsets corresponding to the respective product identifiers, and obtain the target test sample subsets, until it is determined that all the product identifiers corresponding to the test sample subsets are determined. A collection of intent prediction results.
The method for testing a question-and-answer intent classification model according to claim 1, wherein the test is based on the respective intent prediction result sets corresponding to each of the product identifiers, and the test sample subsets corresponding to each of the product identifiers. The steps of performing an accurate judgment on the intention prediction of each of the test samples, and obtaining the set of accurate intention prediction results corresponding to each of the product identifiers, include:

respectively process the test question question sentence intent rating data and the test question intent rating data of each of the test samples in the test sample set according to the intent priority, and obtain the test sample set after the intent priority processing;

Extracting intention prediction results from the intention prediction result set corresponding to each product identifier in turn, to obtain a target intention prediction result;

When the target intention prediction result is whether the target intention is intended or not, extract the test question intention determination data from the test sample set after the intention priority processing according to the target intention prediction result, and obtain whether the test question to be judged is intended or not Calibration data, when the target intention prediction result and the test question to be judged whether the intention calibration data is the same, determine that the target intention prediction result corresponding to the target intention prediction result is correct, otherwise determine the target intention prediction The accurate result of the intention prediction corresponding to the result is an error;

When the target intent prediction result is the question intent, extract the test question question intent determination data from the test sample set after the intent priority processing according to the target intent prediction result, and obtain the test question question to be determined. Sentence intent determination data, when the target intent prediction result and the to-be-determined question sentence intent determination data are the same, it is determined that the intent prediction accurate result corresponding to the target intent prediction result is correct, otherwise it is determined that the intent prediction result is correct. The accurate result of the intention prediction corresponding to the target intention prediction result is an error;

Repeat the step of sequentially extracting the intent prediction results from the intent prediction result set corresponding to each product identifier to obtain the target intent prediction result, until it is determined that the intent predictions of all the intent prediction results are accurate result;

According to all the accurate intention prediction results, a set of accurate intention prediction results corresponding to each of the product identifiers is determined.
The method for testing a question-and-answer intent classification model according to claim 3, wherein the test question question sentence intent rating data and the test question intent rating data of each of the test samples in the test sample set are determined by intent The steps of processing the priority and obtaining the set of test samples after the priority processing of the intent include:

respectively comparing the test question question intent rating data and the test question intent rating data of each of the test samples in the test sample set;

When both the test question question intent calibration data and the test question intent calibration data of the test sample exist, delete the test question question intent calibration data of the test sample, Get the test sample after intent priority processing;

According to all the test samples processed by the intent priority, the set of test samples processed by the intent priority is determined.
The method for testing a question-and-answer intent classification model according to claim 1, wherein the report is generated according to the respective corresponding test sample subsets and the intent prediction accurate result set of each of the product identifiers, to obtain a test report of the target question-and-answer intent classification model steps, including:

The target product identifier is used to extract data from the test sample subset and the intent prediction accurate result set corresponding to each of the product identifiers to obtain the test sample subset to be calculated and the intent prediction accurate result set to be calculated. The target product identifier is any of the respective said product identifications;

Accuracy calculation and recall calculation of each intent value are performed according to the test sample subset to be calculated and the intent prediction accurate result set to be calculated, to obtain the said intent value corresponding to the target product identifier. precision data, the recall data and the total number of positive samples;

Repeatedly executing the use of target product identifiers to extract data from the respective test sample subsets and intent prediction accurate result sets corresponding to each of the product identifiers, to obtain the test sample subsets to be calculated and the intent prediction accurate result sets to be calculated, so The target product identification is any one of the product identifications, until the accuracy data, the recall data and the total number of positive samples of each of the intention values corresponding to all the product identifications are determined;

Generate a report according to the accuracy data, the recall data and the total number of positive samples of each of the intention values corresponding to each of the product identifiers, to obtain the target question answering intention classification model test report.
The method for testing a question-and-answer intent classification model according to claim 5, wherein the accuracy calculation and recall of each intent value are performed according to the to-be-calculated test sample subset and the to-be-calculated intent prediction accurate result set The steps of obtaining the accuracy rate data, the recall rate data and the total number of positive samples of each of the intention values corresponding to the target product identifier include:

Calculate the total number of test samples according to the subset of test samples to be calculated, to obtain the total number of test samples corresponding to the target product identifier;

According to the test sample subset to be calculated and the accurate intention prediction result set to be calculated, the positive sample correct prediction number of each of the intention values is calculated, and the number of each of the intention values corresponding to the target product identifier is obtained. Number of correct predictions for positive samples;

According to the test sample subset to be calculated and the accurate intention prediction result set to be calculated, the correct prediction number of negative samples of each of the intention values is calculated, and the number of each of the intention values corresponding to the target product identifier is obtained. The number of correct predictions for negative samples;

Accuracy calculation is performed according to the total number of test samples corresponding to the target product identifier, the number of positive samples correctly predicted for each of the intent values, and the number of negative samples correctly predicted for each of the intent values, to obtain the respective data corresponding to the target product identifier. the accuracy data of the intended value;

Calculate the total number of the respective test samples for each of the intent values according to the subset of test samples to be calculated, to obtain the total number of positive samples for each of the intent values corresponding to the target product identifier;

Calculate the recall rate according to the total number of positive samples of each of the intention values corresponding to the target product identifier and the correct predicted number of positive samples of each of the intention values, to obtain the total number of the intention values corresponding to the target product identifier. the recall data.
The method for testing a question-and-answer intent classification model according to claim 5, wherein the accuracy data, the recall data and the positive samples of the respective intent values corresponding to the respective product identifiers The total number of reports is generated, and the steps of obtaining the test report of the target question answering intent classification model include:

According to the accuracy rate data, the recall rate data and the total number of positive samples of the respective intent values corresponding to the respective product identifiers, an Excel document is generated to obtain the target question answering intent classification model test report;

obtaining a report download request, where the report download request carries download mode data;

Send the target question answering intent classification model test report according to the download method data.
A test device for a question-and-answer intent classification model, wherein the device includes:

A test sample acquisition module, configured to acquire a test sample set, the test sample set includes a plurality of test samples, and the test samples include: product identification, test question sample data, test question question sentence intent identification data, and test question intent identification data;

a test sample dividing module, configured to divide the plurality of test samples by using the product identifier, and obtain a test sample subset corresponding to each of the product identifiers;

an intent prediction module, configured to respectively input the test sample subset corresponding to each product identifier into the corresponding question-and-answer intent classification model to be tested to perform intent prediction, and obtain an intent prediction result set corresponding to each of the product identifiers;

Intent prediction accurate judgment module, used for each of the product identifiers corresponding to the intent prediction result set, the test question question sentence intent determination data and the test question of the test sample subset corresponding to each of the product identifiers respectively Whether the intention identification data is used to accurately judge the intention prediction of each of the test samples, and obtain a set of accurate intention prediction results corresponding to each of the product identifications;

The report generation module is configured to generate a report according to the respective corresponding test sample subsets and the accurate intent prediction result set of each of the product identifiers, and obtain a test report of the target question answering intent classification model, where the target question answering intent classification model test report includes: each Accuracy data, recall data and total number of positive samples of each intent value corresponding to the product identifiers.
A computer device includes a memory and a processor, wherein the memory stores a computer program, wherein the processor implements the following method steps when executing the computer program:

acquiring a test sample set, where the test sample set includes a plurality of test samples, and the test samples include: product identification, test question sample data, test question sentence intent identification data, and test question intent identification data;

Divide the plurality of test samples by using the product identifiers to obtain a subset of test samples corresponding to each of the product identifiers;

Inputting the test sample subset corresponding to each product identifier into the corresponding question-and-answer intent classification model to be tested to perform intent prediction, and obtaining the respective intent prediction result set corresponding to each of the product identifiers;

Carry out each test according to the intent prediction result set corresponding to each of the product identifiers, the test question question intent determination data and the test question intent determination data of the test sample subsets corresponding to each of the product identifiers respectively. Accurately judge the intention prediction of the test sample, and obtain a set of accurate intention prediction results corresponding to each of the product identifiers;

The report is generated according to the respective corresponding test sample subsets of the product identifiers and the accurate intent prediction result set, to obtain the target question answering intent classification model test report. The target question answering intent classification model test report includes: the respective product identifiers corresponding to The precision data, recall data and total number of positive samples for each intent value of .
The computer device according to claim 9, wherein the test sample subset corresponding to each product identifier is input into the corresponding question-and-answer intent classification model to be tested to perform intent prediction, and each product identifier is obtained. The steps of the corresponding intent prediction result set include:

The test sample subset is extracted from the test sample subset corresponding to each of the product identifiers by using the product identifier to be predicted, and the target test sample subset is obtained, and the product identifier to be predicted is any one of the product identifiers. One;

Search from the model library to be tested according to the product identifier to be predicted, to obtain the target question-and-answer intent classification model to be tested;

Inputting each of the test samples in the target test sample subset into the target question-and-answer intent classification model to be tested for intent prediction, and obtaining the intent prediction result set corresponding to the product identifier to be predicted;

Repeat the steps of using the product identifiers to be predicted to extract the test sample subsets from the test sample subsets corresponding to the respective product identifiers, and obtain the target test sample subsets, until it is determined that all the product identifiers corresponding to the test sample subsets are determined. A collection of intent prediction results.
The computer device according to claim 9, wherein the intent prediction result set of each of the product identifiers corresponding to each of the product identifiers and the test question question sentence intent identifiers of the respective sub-sets of test samples corresponding to each of the product identifiers Whether the data and the test question are intended to designate the data to accurately judge the intention prediction of each of the test samples, and obtain the corresponding set of accurate intention prediction results for each of the product identifiers, the steps include:

respectively process the test question question sentence intent rating data and the test question intent rating data of each of the test samples in the test sample set according to the intent priority, and obtain the test sample set after the intent priority processing;

The intention prediction results are sequentially extracted from the intention prediction result set corresponding to each of the product identifiers, and the target intention prediction results are obtained;

When the target intention prediction result is whether the target intention is intended or not, extract the test question intention determination data from the test sample set after the intention priority processing according to the target intention prediction result, and obtain whether the test question to be judged is intended or not Calibration data, when the target intention prediction result and the test question to be judged whether the intention calibration data is the same, determine that the target intention prediction result corresponding to the target intention prediction result is correct, otherwise determine the target intention prediction The accurate result of the intention prediction corresponding to the result is an error;

When the target intent prediction result is the question intent, extract the test question question intent determination data from the test sample set after the intent priority processing according to the target intent prediction result, and obtain the test question question to be determined. Sentence intent determination data, when the target intent prediction result and the to-be-determined question sentence intent determination data are the same, it is determined that the intent prediction accurate result corresponding to the target intent prediction result is correct, otherwise it is determined that the intent prediction result is correct. The accurate result of the intention prediction corresponding to the target intention prediction result is an error;

Repeat the step of sequentially extracting the intent prediction results from the intent prediction result set corresponding to each product identifier to obtain the target intent prediction result, until it is determined that the intent predictions of all the intent prediction results are accurate result;

According to all the accurate intention prediction results, a set of accurate intention prediction results corresponding to each of the product identifiers is determined.
The computer device according to claim 11 , wherein, the test question question sentence intent determination data and the test question intent determination data of each of the test samples in the test sample set are processed according to intent priority, The steps to obtain a set of test samples after intent priority processing include:

respectively comparing the test question question sentence intent rating data and the test question intent rating data of each of the test samples in the test sample set;

When both the test question question intent calibration data and the test question intent calibration data of the test sample exist, delete the test question question intent calibration data of the test sample, Get the test sample after intent priority processing;

According to all the test samples processed by the intent priority, the set of test samples processed by the intent priority is determined.
The computer device according to claim 9, wherein the step of generating a report according to the respective corresponding test sample subsets and the intent prediction accurate result set according to each of the product identifiers, and obtaining the test report of the target question answering intent classification model, comprises:

The target product identifier is used to extract data from the test sample subset and the intent prediction accurate result set corresponding to each of the product identifiers to obtain the test sample subset to be calculated and the intent prediction accurate result set to be calculated. The target product identifier is any of the respective said product identifications;

Accuracy calculation and recall calculation of each intent value are performed according to the test sample subset to be calculated and the intent prediction accurate result set to be calculated, to obtain the said intent value corresponding to the target product identifier. precision data, the recall data and the total number of positive samples;

Repeatedly executing the use of target product identifiers to extract data from the test sample subsets and intent prediction accurate result sets corresponding to each of the product identifiers to obtain the test sample subsets to be calculated and the intent prediction accurate result sets to be calculated. The target product identification is any one of the product identifications, until the accuracy data, the recall data and the total number of positive samples of each of the intention values corresponding to all the product identifications are determined;

The report is generated according to the accuracy data, the recall data and the total number of positive samples of the respective intent values corresponding to the respective product identifiers to obtain the target question answering intent classification model test report.
The computer device according to claim 13, wherein the accuracy calculation and recall calculation of each intention value are performed according to the test sample subset to be calculated and the intention prediction accurate result set to be calculated to obtain the obtained The steps of the accuracy rate data, the recall rate data and the total number of positive samples of each of the intention values corresponding to the target product identifiers include:

Calculate the total number of test samples according to the subset of test samples to be calculated, to obtain the total number of test samples corresponding to the target product identifier;

According to the test sample subset to be calculated and the accurate intention prediction result set to be calculated, the positive sample correct prediction number of each of the intention values is calculated, and the number of each of the intention values corresponding to the target product identifier is obtained. Number of correct predictions for positive samples;

According to the test sample subset to be calculated and the accurate intention prediction result set to be calculated, calculate the number of negative sample correct predictions of each of the intention values, and obtain the target product identifier corresponding to each of the intention values. Number of correct predictions for negative samples;

Accuracy calculation is performed according to the total number of test samples corresponding to the target product identifier, the number of positive samples correctly predicted for each of the intent values, and the number of correct negative samples for each of the intent values, to obtain each the accuracy data of the intended value;

Calculate the total number of the respective test samples for each of the intent values according to the subset of test samples to be calculated, to obtain the total number of positive samples for each of the intent values corresponding to the target product identifier;

Calculate the recall rate according to the total number of positive samples of each of the intention values corresponding to the target product identifier and the correct predicted number of positive samples of each of the intention values, to obtain the total number of the intention values corresponding to the target product identifier. the recall data.
A computer-readable storage medium on which a computer program is stored, wherein when the computer program is executed by a processor, the following method steps are implemented:

acquiring a test sample set, where the test sample set includes a plurality of test samples, and the test samples include: product identification, test question sample data, test question sentence intent identification data, and test question intent identification data;

Dividing the plurality of test samples by using the product identifiers to obtain a subset of test samples corresponding to each of the product identifiers;

Inputting the test sample subset corresponding to each product identifier into the corresponding question-and-answer intent classification model to be tested to perform intent prediction, and obtaining the respective intent prediction result set corresponding to each of the product identifiers;

Carry out each test according to the intent prediction result set corresponding to each of the product identifiers, the test question question intent determination data and the test question intent determination data of the test sample subsets corresponding to each of the product identifiers respectively. Accurately judge the intention prediction of the test sample, and obtain a set of accurate intention prediction results corresponding to each of the product identifiers;

The report is generated according to the corresponding test sample subsets and the accurate intent prediction result set for each of the product identifiers, to obtain a target question answering intent classification model test report, where the target question answering intent classification model test report includes: each of the product identifiers corresponds to The precision data, recall data, and total number of positive samples for each intent value of .
The computer-readable storage medium according to claim 15, wherein the test sample subset corresponding to each of the product identifiers is respectively input into the corresponding question-and-answer intent classification model to be tested to perform intent prediction, to obtain each of the The steps of each corresponding set of intent prediction results for the product identifiers include:

The test sample subset is extracted from the test sample subset corresponding to each of the product identifiers by using the product identifier to be predicted, and the target test sample subset is obtained, and the product identifier to be predicted is any one of the product identifiers. One;

Search from the model library to be tested according to the product identifier to be predicted, to obtain the target question-and-answer intent classification model to be tested;

Inputting each of the test samples in the target test sample subset into the target question-and-answer intent classification model to be tested for intent prediction, and obtaining the intent prediction result set corresponding to the product identifier to be predicted;

Repeat the steps of using the product identifiers to be predicted to extract the test sample subsets from the test sample subsets corresponding to the respective product identifiers, and obtain the target test sample subsets, until it is determined that all the product identifiers corresponding to the test sample subsets are determined. A collection of intent prediction results.
The computer-readable storage medium according to claim 15, wherein the test questions are asked according to the set of intention prediction results corresponding to each of the product identifiers and the test sample subsets corresponding to each of the product identifiers respectively. The steps of performing an accurate judgment on the intention prediction of each of the test samples and obtaining the set of accurate intention prediction results corresponding to each of the product identifiers, including:

respectively processing the test question question sentence intent rating data and the test question intent rating data of each of the test samples in the test sample set according to the intent priority, to obtain a test sample set after the intent priority processing;

Extracting intention prediction results from the intention prediction result set corresponding to each product identifier in turn, to obtain a target intention prediction result;

When the target intention prediction result is whether the target intention is intended, extract the test question intention determination data from the test sample set after the intention priority processing according to the target intention prediction result, and obtain whether the test question to be judged is intended or not Calibration data, when the target intention prediction result and the test question to be judged whether the intention calibration data are the same, determine that the target intention prediction result corresponding to the target intention prediction result is correct, otherwise, determine the target intention prediction The accurate result of the intention prediction corresponding to the result is an error;

When the target intent prediction result is the question sentence intent, extract the test question question sentence intent determination data from the test sample set after the intent priority processing according to the target intent prediction result, and obtain the test question question to be determined. Sentence intent determination data, when the target intent prediction result is the same as the to-be-determined question sentence intent determination data, it is determined that the intent prediction accurate result corresponding to the target intent prediction result is correct, otherwise it is determined that the intent prediction result is correct. The accurate result of the intention prediction corresponding to the target intention prediction result is an error;

Repeat the step of sequentially extracting the intent prediction results from the intent prediction result set corresponding to each product identifier to obtain the target intent prediction result, until it is determined that the intent predictions of all the intent prediction results are accurate result;

According to all the accurate intention prediction results, a set of accurate intention prediction results corresponding to each of the product identifiers is determined.
18. The computer-readable storage medium of claim 17, wherein the test question question sentence intent determination data and the test question intent determination data for each of the test samples in the test sample set, respectively, are prioritized by intent The steps of processing to obtain a set of test samples after intent priority processing include:

respectively comparing the test question question sentence intent rating data and the test question intent rating data of each of the test samples in the test sample set;

When both the test question question intent calibration data and the test question intent calibration data of the test sample exist, delete the test question question intent calibration data of the test sample, Get the test sample after intent priority processing;

According to all the test samples processed by the intent priority, the set of test samples processed by the intent priority is determined.
The computer-readable storage medium according to claim 15, wherein the step of generating a report according to the respective corresponding test sample subsets and the intent prediction accurate result set of each of the product identifiers to obtain a test report of the target question answering intent classification model ,include:

The target product identifier is used to extract data from the test sample subset and the intent prediction accurate result set corresponding to each of the product identifiers to obtain the test sample subset to be calculated and the intent prediction accurate result set to be calculated. The target product identifier is any of the respective said product identifications;

Accuracy calculation and recall calculation of each intent value are performed according to the test sample subset to be calculated and the intent prediction accurate result set to be calculated, to obtain the said intent value corresponding to the target product identifier. precision data, the recall data and the total number of positive samples;

Repeatedly executing the use of target product identifiers to extract data from the test sample subsets and intent prediction accurate result sets corresponding to each of the product identifiers to obtain the test sample subsets to be calculated and the intent prediction accurate result sets to be calculated. The step in which the target product identification is any one of the product identifications, until the accuracy data, the recall data and the total number of positive samples of each of the intention values corresponding to all the product identifications are determined;

A report is generated according to the accuracy rate data, the recall rate data and the total number of positive samples of the respective intent values corresponding to the respective product identifiers, to obtain the target question answering intent classification model test report.
The computer-readable storage medium according to claim 19, wherein the accuracy calculation and recall calculation of each intention value are performed according to the test sample subset to be calculated and the intention prediction accurate result set to be calculated. , the steps of obtaining the accuracy rate data, the recall rate data and the total number of positive samples of each of the intention values corresponding to the target product identifiers include:

Calculate the total number of test samples according to the subset of test samples to be calculated, to obtain the total number of test samples corresponding to the target product identifier;

According to the test sample subset to be calculated and the accurate intention prediction result set to be calculated, the positive sample correct prediction number of each of the intention values is calculated, and the number of each of the intention values corresponding to the target product identifier is obtained. Number of correct predictions for positive samples;

According to the test sample subset to be calculated and the accurate intention prediction result set to be calculated, the correct prediction number of negative samples of each of the intention values is calculated, and the number of each of the intention values corresponding to the target product identifier is obtained. Number of correct predictions for negative samples;

Accuracy calculation is performed according to the total number of test samples corresponding to the target product identifier, the number of positive samples correctly predicted for each of the intent values, and the number of negative samples correctly predicted for each of the intent values, to obtain the respective data corresponding to the target product identifier. the accuracy data of the intended value;

Calculate the total number of the respective test samples for each of the intent values according to the subset of test samples to be calculated, to obtain the total number of positive samples for each of the intent values corresponding to the target product identifier;

The recall rate is calculated according to the total number of positive samples of each of the intent values corresponding to the target product identifier, and the number of correct predictions of positive samples of each of the intent values, to obtain the total number of the intent values corresponding to the target product identifier. the recall data.