WO2020000779A1 - 质量评价模型获取方法及装置、计算机设备与存储介质 - Google Patents

质量评价模型获取方法及装置、计算机设备与存储介质 Download PDF

Info

Publication number
WO2020000779A1
WO2020000779A1 PCT/CN2018/110089 CN2018110089W WO2020000779A1 WO 2020000779 A1 WO2020000779 A1 WO 2020000779A1 CN 2018110089 W CN2018110089 W CN 2018110089W WO 2020000779 A1 WO2020000779 A1 WO 2020000779A1
Authority
WO
WIPO (PCT)
Prior art keywords
question
quality evaluation
sample
answer data
feature
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2018/110089
Other languages
English (en)
French (fr)
Inventor
顾宝宝
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Ping An Technology Shenzhen Co Ltd
Original Assignee
Ping An Technology Shenzhen Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Ping An Technology Shenzhen Co Ltd filed Critical Ping An Technology Shenzhen Co Ltd
Publication of WO2020000779A1 publication Critical patent/WO2020000779A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/30—Semantic analysis
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
    • G06Q10/063—Operations research, analysis or management
    • G06Q10/0639—Performance analysis of employees; Performance analysis of enterprise or organisation operations
    • G06Q10/06395—Quality analysis or management
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00—Commerce
    • G06Q30/02—Marketing; Price estimation or determination; Fundraising
    • G06Q30/0281—Customer communication at a business location, e.g. providing product or service information, consulting

Definitions

  • the present application relates to the field of computer technology, and in particular, to a method and device for acquiring a quality evaluation model, computer equipment, and a storage medium.
  • a scene in which a responder or a response server responds to a question asked by a user, and whether the reply question accurately solves the user's question is generally As the performance appraisal standard of the responder or the quality evaluation standard of the response server.
  • the quality evaluation of Q & A data based on manual implementation is relatively complicated. Therefore, in the prior art, scoring rules are generally set in advance, and the complex rule configuration process is used. In this way, when the quality evaluation of the Q & A data of the responder or the response server is required In this case, the quality evaluation score needs to be finally determined according to a preset scoring rule.
  • embodiments of the present application provide a method and device for acquiring a quality evaluation model, a computer device, and a storage medium, which are used to solve the problem of low accuracy of scores caused by quality evaluation through preset configuration rules in the prior art. problem.
  • an embodiment of the present application provides a method for acquiring a quality evaluation model, including:
  • sample question and answer data process the sample question and answer data using a slot filling method to obtain a sample quality evaluation score corresponding to the sample question and answer data; use the sample question and answer data as an input of an initial quality evaluation model, and use the sample quality
  • the evaluation score is used as an output of the initial quality evaluation model, and the initial quality evaluation model is trained by using a neural network mechanism to obtain a trained target quality evaluation model.
  • any optional implementation manner further provide an implementation manner, wherein the sample Q & A data is processed by using a slot filling method to obtain a sample quality evaluation score corresponding to the sample Q & A data, including: :
  • any optional implementation manner further provide an implementation manner, and determining the sample quality evaluation score based on the valid question answering characteristics includes:
  • the number of valid question answering features is obtained as the sample quality evaluation score.
  • any optional implementation manner further provide an implementation manner, and determining the sample quality evaluation score based on the valid question answering characteristics includes:
  • a ratio between the first number and the second number is obtained as the sample quality evaluation score.
  • any optional implementation manner further provide an implementation manner, and determining the sample quality evaluation score based on the valid question answering characteristics includes:
  • a sum of products of each questioning feature is obtained as the sample quality evaluation score.
  • the question-and-answer data to be evaluated is input into the target quality evaluation model to obtain a quality evaluation score of the question-and-answer data to be evaluated.
  • the question answering quality of the respondent corresponding to the quality assessment score of the at least one question answering data is evaluated to obtain the question answering quality score of the respondent.
  • the sample quality evaluation scores corresponding to the sample question and answer data are obtained based on the slot filling method. Therefore, the sample question and answer data is used as the input of the initial quality evaluation model to obtain the sample quality.
  • the evaluation score is used as the output of the initial quality evaluation model.
  • the target quality evaluation model can be obtained by training the initial quality evaluation model using a neural network mechanism.
  • the target quality evaluation model obtained based on neural network training in the embodiment of the present application is more close to the scoring requirements of question and answer data and has higher flexibility Therefore, as long as the cardinality of the sample question and answer data is sufficiently large, the quality evaluation score of the question and answer data can have a high accuracy rate. Therefore, the technical solution provided in the embodiment of the present application can solve the problem that the accuracy of the score is low due to the quality evaluation through the preset configuration rules in the prior art.
  • an embodiment of the present application provides a device for acquiring a quality evaluation model, including: an acquisition unit for acquiring sample question and answer data; and a processing unit for processing the sample question and answer data by using a slot filling method to obtain A sample quality evaluation score corresponding to the sample question and answer data; a training unit configured to use the sample question and answer data as an input of an initial quality evaluation model, and use the sample quality evaluation score as an output of the initial quality evaluation model, using a neural network Mechanism to train the quality evaluation model to obtain a trained target quality evaluation model.
  • an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes all
  • the computer program implements the method for acquiring a quality evaluation model according to any one of the first aspects.
  • an embodiment of the present application provides a computer non-volatile readable storage medium, including: computer-executable instructions, used to execute the computer-executable instructions as described in any one of the first aspects when the computer-executable instructions are executed.
  • the sample quality evaluation scores corresponding to the sample question and answer data are obtained based on the slot filling method. Therefore, the sample question and answer data is used as the input of the initial quality evaluation model to obtain the sample quality.
  • the evaluation score is used as the output of the initial quality evaluation model.
  • the target quality evaluation model can be obtained by training the initial quality evaluation model using a neural network mechanism. Therefore, compared with the quality evaluation scores obtained by configuring complicated scoring rules in the prior art, the target quality evaluation model obtained based on neural network training in the embodiment of the present application is more close to the scoring requirements of question and answer data and has higher flexibility.
  • the technical solution provided in the embodiment of the present application can solve the problem that the accuracy of the score is low due to the quality evaluation through the preset configuration rules in the prior art.
  • FIG. 1 is a schematic flowchart of Embodiment 1 of a method for acquiring a quality evaluation model according to an embodiment of the present application;
  • FIG. 2 is a schematic flowchart of a second embodiment of a method for acquiring a quality evaluation model according to an embodiment of the present application
  • FIG. 3 is a schematic flowchart of a third embodiment of a method for acquiring a quality evaluation model according to an embodiment of the present application
  • FIG. 4 is a functional block diagram of a quality evaluation model acquisition device provided by an embodiment of the present application.
  • FIG. 5 is a functional block diagram of a computer device according to an embodiment of the present application.
  • first, second, third, etc. may be used to describe numbers and the like in the embodiments of the present application, these numbers should not be limited to these terms. These terms are only used to distinguish numbers from each other.
  • the first number may also be referred to as the second number, and similarly, the second number may also be referred to as the first number.
  • the word “if” as used herein can be interpreted as “at” or “when” or “responding to determination” or “responding to detection”.
  • the phrases “if determined” or “if detected (the stated condition or event)” can be interpreted as “when determined” or “responded to the determination” or “when detected (the stated condition or event) ) “Or” in response to a test (statement or event stated) ".
  • the configuration rules are complex and the accuracy rate is low.
  • the embodiments of the present application provide the following solution: learn sample question-and-answer data through a neural network to obtain more flexibility. 3. A quality evaluation model that is closer to the Q & A quality evaluation method, so that the quality evaluation model is used to achieve the quality evaluation of the Q & A data.
  • an embodiment of the present application provides a method for acquiring a quality evaluation model. Please refer to FIG. 1, the method may include the following steps:
  • S104 Process the sample question and answer data by using a slot filling method to obtain a sample quality evaluation score corresponding to the sample question and answer data.
  • the sample question and answer data is used as the input of the initial quality evaluation model, the sample quality evaluation score is used as the output of the initial quality evaluation model, and the initial quality evaluation model is trained by using a neural network mechanism to obtain a trained target quality evaluation model.
  • the sample question and answer data involved in the embodiment of the present application includes: response data of a respondent or response data of a response server, and may further include question data of a user.
  • the expression form of question and answer data is generally expressed as sentences.
  • the application scenario of the question and answer data is not particularly limited in the embodiments of the present application.
  • the question and answer data may be question and answer data between a shopper and a sales customer service in a shopping platform; or, for example, in another feasible implementation scenario, the question and answer data may also be an enterprise Website question and answer data for business inquiries.
  • the sample question and answer data may be derived from the active input of an operator, or may be obtained automatically when the solution is executed.
  • the solution there can be multiple ways of automatic acquisition. For example, when this solution is executed, all historical question-and-answer data that is stored is automatically retrieved as sample question-and-answer data. Or, for example, considering that the historical question and answer data that is too long may be inaccurate and affect the processing result, the historical question and answer data within a certain period of time before the current time may also be automatically retrieved as the sample question and answer data.
  • the historical question and answer data may be stored in itself, or in another storage space, such as a cloud server, or in a third-party storage space.
  • a storage space other than itself it can be implemented by sending a data query request and receiving historical question and answer data based on the data query request feedback from the storage space; or, if you have the right to call the data, You can go directly to the third-party storage space to call these historical question and answer data.
  • the application concept of the embodiment of the present application is to perform deep learning based on sample question and answer data to obtain a target quality evaluation model.
  • the input of the target quality assessment model is the question and answer data of the responder, and the output is the quality evaluation score of the question and answer data. It can be understood that the larger the number of sample question and answer data, the more accurate and accurate the target quality evaluation model is.
  • a sample quality evaluation score corresponding to the sample question and answer data can be obtained by using a slot filling method.
  • the slot filling method refers to using question characteristics and response characteristics in the question and answer data as slots respectively, and scoring is achieved by matching whether the slots between the two match.
  • the question and answer features can be obtained through semantic recognition.
  • S1042 Perform semantic recognition on the sample question and answer data to obtain at least one question feature and at least one response feature of the sample question and answer data.
  • the slot filling method is used to match the characteristics of each question with the characteristics of each reply to obtain an effective question and answer feature.
  • the semantic recognition involved in the process of S1042 is to model the context structure of the current problem, analyze the user's context information when presenting the current problem through machine learning, and analyze the sentence composition of the current problem. Identify the missing sentence components of the current question, then perform context analysis and semantic analysis based on the context, expand and understand the semantic information of the current question, and then retrieve answers from the system that match the semantics of the current question.
  • the embodiment of the present application provides a specific implementation of achievable semantic recognition.
  • the sample Q & A data is constructed and analyzed, and the constituents of the sample Q & A data are obtained through a sentence component analysis model, and then the sample Q & A data is determined What is the default component in the sample composition, get the component composition of the sample question and answer data and the default component record. Then, through the analysis of people's spoken habits, it is found that the default problem components usually exist in the context of the dialogue. Then, by analyzing the components of the context and combining the sentence components of the consultation question, the user's semantic recognition is jointly conducted. The basic components of the sentence are complete. Therefore, by taking the constituents of the sample question and answer data as input, recombining the questions, and generating a sentence expression that the system can explain, at least one question feature and at least one response feature of the sample question and answer data are obtained.
  • the matching of questioning characteristics and response characteristics is semantic matching. And, it can be preset to determine whether the two match. And, when presetting, one questioning feature can be preset to correspond to multiple answering features, and one replying feature can also correspond to multiple questioning features.
  • the implementation of determining the sample quality evaluation score may include but is not limited to the following:
  • the number of valid question answering features is obtained as the sample quality evaluation score.
  • the number of valid question-answering features is used as the sample quality evaluation score. Then, when the questioning features match the answering features, it is recorded as 1; when they do not match, it is recorded as 0. In this way, each question is The response scores of the characteristics are summed to obtain the sample quality evaluation score.
  • a sample question and answer data is semantically identified to obtain 5 question characteristics and 5 response characteristics, and then one-to-one matching is performed based on the semantics between the two. When there are response characteristics that match the question characteristics, the count is increased by one. Otherwise, it does not count. In this way, if three of the five question characteristics can be matched in the response characteristics, then the quality evaluation score of the sample question and answer data can be obtained as 3.
  • the second method is to obtain the first number of valid question-answering features and the second number of total questioning features, and then obtain the ratio between the first number and the second number as the sample quality evaluation score. .
  • the proportion of the effective question-and-answer features in the total questioning characteristics is used as the sample quality evaluation score. Then, after obtaining the sample quality evaluation score, it is necessary to perform normalization processing based on the number of total questioning characteristics.
  • the product of each questioning feature and its corresponding weighting is obtained according to the preset weighting of each questioning feature, wherein the weight of the valid question-answering feature is greater than the weights of the remaining questioning features, and then the product of each questioning feature is obtained The sum is used as the sample quality evaluation score.
  • This solution is based on the importance of questioning features, setting different weights for each questioning feature, and correcting the weights to obtain sample quality evaluation scores.
  • the weights of the five questioning features are 0.1, 0.3, 0.3, 0.1, 0.2, and the first questioning feature, the third questioning feature and the fifth questioning feature are matched. Then, according to the weight correction, the sample quality is obtained.
  • a neural network is a computing system that mimics the structure of biological brain synapses-neurons for data processing. It consists of multi-layered computing nodes and connections between layers. Each node simulates a neuron and performs a specific operation, such as an activation function. The connections between the nodes simulate synapses. The weighted value of the connections from the input of the previous node represents the synaptic weight.
  • Neural networks have powerful nonlinear and adaptive information processing capabilities. The neuron in the neural network processes the accumulated value from the connected input as its output after processing it with an activation function.
  • artificial neural networks include dozens of network models such as perceptrons, Hopfield networks, and Boltzmann machines, which can implement a variety of functions. In pattern recognition, Complex control, signal processing, and optimization are all used.
  • the sample quality evaluation score corresponding to each sample question and answer data can be obtained.
  • the sample question and answer data is used as the input of the initial quality evaluation model, and the sample quality evaluation score is used as the initial quality evaluation model.
  • the target quality evaluation model can be obtained.
  • a target quality evaluation model trained based on sample question and answer data can be obtained. Based on this, when performing quality assessment on the question and answer data to be evaluated, as shown in FIG. 3, the following steps can also be included:
  • the quality evaluation of the question and answer data to be evaluated can be conveniently implemented, and the target quality evaluation model obtained based on this training response method is closer to the question and answer than the quality evaluation score obtained by configuring complex scoring rules in the prior art.
  • Data scoring needs have high flexibility; therefore, as long as the cardinality of the sample question and answer data is large enough, the quality evaluation score of the question and answer data can have a high accuracy rate.
  • a responder may include a responder or a response server, and there may be multiple question and answer data, then based on the quality evaluation score of these question and answer data, the respondent may be further The overall response quality was evaluated.
  • the method may further include the following steps:
  • the overall question and answer quality of the respondent is evaluated according to the quality evaluation score of at least one question and answer data of the responder, and the overall question and answer quality score of the respondent is obtained. This can further simplify the quality evaluation process for the respondent, further simplify the manual operation, and free up manpower.
  • an embodiment of the present application further provides a computer non-volatile readable storage medium, including: computer executable instructions, which are used to perform the following steps when the computer executable instructions are executed:
  • sample Q & A data use the slot filling method to process the sample Q & A data to obtain sample quality evaluation scores corresponding to the sample Q & A data; use the sample Q & A data as input to the initial quality evaluation model, and use the sample quality evaluation score as the
  • the output uses the neural network mechanism to train the initial quality evaluation model to obtain the trained target quality evaluation model.
  • the computer-executable instructions when executed, it is further configured to perform the following steps: perform semantic recognition on the sample question and answer data to obtain at least one question feature and at least one response feature of the sample question and answer data; and use slot filling to The questioning characteristics are matched with the characteristics of each reply to obtain the valid question and answer characteristics; according to the valid question and answer characteristics, the sample quality evaluation score is determined.
  • the computer-executable instructions when executed, it is further configured to perform the following steps: obtaining the number of valid question answering features as a sample quality evaluation score.
  • the following steps are further performed: obtaining a first number of valid question-answering features and a second number of total questioning features; obtaining a ratio between the first number and the second number, to As the sample quality evaluation score.
  • the sample quality evaluation scores corresponding to the sample question and answer data are obtained based on the slot filling method. Therefore, the sample question and answer data is used as the input of the initial quality evaluation model to obtain the sample quality.
  • the evaluation score is used as the output of the initial quality evaluation model.
  • the target quality evaluation model can be obtained by training the initial quality evaluation model using a neural network mechanism. Therefore, compared with the quality evaluation scores obtained by configuring complicated scoring rules in the prior art, the target quality evaluation model obtained based on neural network training in the embodiment of the present application is more close to the scoring requirements of question and answer data and has higher flexibility.
  • the technical solution provided in the embodiment of the present application can solve the problem that the accuracy of the score is low due to the quality evaluation through the preset configuration rules in the prior art.
  • the embodiment of the present application further provides an apparatus embodiment for implementing each step and method in the foregoing method embodiment.
  • an embodiment of the present application further provides a device for acquiring a quality evaluation model.
  • the device 400 for acquiring a quality evaluation model may include:
  • An obtaining unit 41 configured to obtain sample question and answer data
  • the processing unit 42 is configured to process the sample question and answer data by using a slot filling method to obtain a sample quality evaluation score corresponding to the sample question and answer data;
  • a training unit 43 is configured to use the sample question and answer data as an input of the initial quality evaluation model, use the sample quality evaluation score as the output of the initial quality evaluation model, and use a neural network mechanism to train the quality evaluation model to obtain a target quality evaluation after training model.
  • the processing unit 42 is specifically configured to:
  • the slot filling method is used to match the characteristics of each question with the characteristics of each reply to obtain an effective question and answer feature
  • the processing unit 42 may be specifically used to perform the step of determining the sample quality evaluation score based on the valid question answering characteristics:
  • the processing unit 42 when the processing unit 42 executes the step of determining the sample quality evaluation score based on the valid question and answer characteristics, it may be specifically used to:
  • the ratio between the first number and the second number is obtained as a sample quality evaluation score.
  • the processing unit 42 when the processing unit 42 executes the step of determining the sample quality evaluation score based on the valid question and answer characteristics, it may be specifically used to:
  • the sum of the products of each questioning feature is obtained as the sample quality evaluation score.
  • the quality evaluation model obtaining device 400 may further include:
  • An evaluation unit (not shown in FIG. 4) is configured to input the question-and-answer data to be evaluated into a target quality evaluation model, and obtain a quality assessment score of the question-and-answer data.
  • the evaluation unit may be further used for:
  • the question answering quality of the respondent corresponding to the quality assessment score of the at least one question answering data is evaluated to obtain the question answering quality score of the respondent.
  • the computer device 500 includes: a memory 51, a processor 52, and a computer program stored in the memory 51 and executable on the processor 52.
  • the processor 52 implements the following steps when executing a computer program:
  • sample question and answer data use the slot filling method to process the sample question and answer data to obtain sample quality evaluation scores corresponding to the sample question and answer data; use the sample question and answer data as the input of the initial quality evaluation model, and use the sample quality evaluation score as the initial quality evaluation model.
  • the output uses the neural network mechanism to train the initial quality evaluation model to obtain the trained target quality evaluation model.
  • the processor when the processor executes the computer program, the processor further implements the following steps: performing semantic recognition on the sample question and answer data to obtain at least one question feature and at least one answer feature of the sample question and answer data; and using a slot filling method, each question feature and each The response characteristics are matched to obtain valid question and answer characteristics; according to the valid question and answer characteristics, the sample quality evaluation score is determined.
  • the processor executes the computer program, the following steps are further implemented: obtaining the number of valid question answering features as a sample quality evaluation score.
  • the processor executes the computer program, the following steps are further implemented: obtaining a first number of valid question-answering features and a second number of total questioning features; obtaining a ratio between the first number and the second number as a sample quality evaluation fraction.
  • the sample quality evaluation scores corresponding to the sample question and answer data are obtained based on the slot filling method. Therefore, the sample question and answer data is used as the input of the initial quality evaluation model to obtain the sample quality.
  • the evaluation score is used as the output of the initial quality evaluation model.
  • the target quality evaluation model can be obtained by training the initial quality evaluation model using a neural network mechanism. Therefore, compared with the quality evaluation scores obtained by configuring complicated scoring rules in the prior art, the target quality evaluation model obtained based on neural network training in the embodiment of the present application is more close to the scoring requirements of question and answer data and has higher flexibility.
  • the technical solution provided in the embodiment of the present application can solve the problem that the accuracy of the score is low due to the quality evaluation through the preset configuration rules in the prior art.
  • the disclosed systems, devices, and methods may be implemented in other ways.
  • the device embodiments described above are only schematic.
  • the division of the unit is only a logical function division.
  • multiple units or components may be combined.
  • the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or units, which may be electrical, mechanical or other forms.
  • the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the objective of the solution of this embodiment.
  • each functional unit in each embodiment of the present application may be integrated into one processing unit, or each of the units may exist separately physically, or two or more units may be integrated into one unit.
  • the above integrated unit may be implemented in the form of hardware, or in the form of hardware plus software functional units.
  • the above integrated unit implemented in the form of a software functional unit may be stored in a computer-readable storage medium.
  • the above software functional unit is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device) or a processor to execute the methods described in the embodiments of the present application. Some steps.
  • the foregoing storage media include: U disks, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks or compact discs and other media that can store program codes .

Landscapes

  • Business, Economics & Management (AREA)
  • Engineering & Computer Science (AREA)
  • Human Resources & Organizations (AREA)
  • Development Economics (AREA)
  • Theoretical Computer Science (AREA)
  • Strategic Management (AREA)
  • Physics & Mathematics (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Economics (AREA)
  • General Physics & Mathematics (AREA)
  • Game Theory and Decision Science (AREA)
  • General Business, Economics & Management (AREA)
  • Marketing (AREA)
  • Finance (AREA)
  • Educational Administration (AREA)
  • Accounting & Taxation (AREA)
  • General Engineering & Computer Science (AREA)
  • Computational Linguistics (AREA)
  • General Health & Medical Sciences (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Artificial Intelligence (AREA)
  • Health & Medical Sciences (AREA)
  • Operations Research (AREA)
  • Quality & Reliability (AREA)
  • Tourism & Hospitality (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)
  • Electrically Operated Instructional Devices (AREA)

Abstract

本申请实施例提供了一种质量评价模型获取方法及装置、计算机设备与存储介质。本申请实施例中，通过获取样本问答数据，然后，利用槽位填充方式处理所述样本问答数据，得到与所述样本问答数据对应的样本质量评价分数，从而，将所述样本问答数据作为初始质量评价模型的输入，将所述样本质量评价分数作为所述初始质量评价模型的输出，利用神经网络机制，对所述初始质量评价模型进行训练，得到训练后的目标质量评价模型。因此，本申请实施例提供的技术方案能够解决现有技术中通过预设的配置规则进行质量评价导致分数的准确率较低的问题。

Description

质量评价模型获取方法及装置、计算机设备与存储介质
本申请要求于2018年06月28日提交中国专利局、申请号为201810690936.X、申请名称为“质量评价模型获取方法及装置、计算机设备与存储介质”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及计算机技术领域，尤其涉及一种质量评价模型获取方法及装置、计算机设备与存储介质。
背景技术
目前，在通过网络实现的问答系统中，如购物网站提供的购物问答系统，涉及应答人员或应答服务器针对用户提问的问题进行回复的场景，并且，回复的问题是否准确解决用户的提问，一般可作为应答人员的绩效考核标准或应答服务器的质量评价标准。
基于人工实现对问答数据的质量评价比较复杂，因此，现有技术中一般是通过预先设置好打分规则，通过复杂的规则配置过程，如此，当需要对应答人员或应答服务器的问答数据进行质量评价时，需要根据预设的打分规则，来最终确定其质量评价分数。
但是，现有的质量评价方法存在配置规则复杂，以及，由于预先配置好的打分规则灵活性较差，质量评价分数的准确率较低。
申请内容
有鉴于此，本申请实施例提供了一种质量评价模型获取方法及装置、计算机设备与存储介质，用以解决现有技术中通过预设的配置规则进行质量评价导致分数的准确率较低的问题。
第一方面，本申请实施例提供了一种质量评价模型获取方法，包括：
获取样本问答数据；利用槽位填充方式处理所述样本问答数据，得到与所述样本问答数据对应的样本质量评价分数；将所述样本问答数据作为初始质量评价模型的输入，将所述样本质量评价分数作为所述初始质量评价模型的输出，利用神经网络机制，对所述初始质量评价模型进行训练，得到训练后的目标质量评价模型。
如上所述的方面和任一可选的实现方式，进一步提供一种实现方式，所述利用槽位填充方式处理所述样本问答数据，得到与所述样本问答数据对应的样本质量评价分数，包括：
对所述样本问答数据进行语义识别，得到所述样本问答数据的至少一个提问特征与至少一个答复特征；
利用所述槽位填充方式，对各提问特征与各答复特征进行匹配，得到有效问答特征；
根据所述有效问答特征，确定所述样本质量评价分数。
如上所述的方面和任一可选的实现方式，进一步提供一种实现方式，所述根据所述有效问答特征，确定所述样本质量评价分数，包括：
获取所述有效问答特征的数目，以作为所述样本质量评价分数。
如上所述的方面和任一可选的实现方式，进一步提供一种实现方式，所述根据所述有效问答特征，确定所述样本质量评价分数，包括：
获取所述有效问答特征的第一数目与总提问特征的第二数目；
获取所述第一数目与所述第二数目之间的比例，以作为所述样本质量评价分数。
如上所述的方面和任一可选的实现方式，进一步提供一种实现方式，所述根据所述有效问答特征，确定所述样本质量评价分数，包括：
根据预设的各提问特征的权重，获取各提问特征与各自对应的 权重之积，其中，所述有效问答特征的权重大于其余提问特征的权重；
获取各提问特征的积之和，以作为所述样本质量评价分数。
如上所述的方面和任一可选的实现方式，进一步提供一种实现方式，所述方法还包括：
将待评价问答数据输入所述目标质量评价模型，得到所述待评价问答数据的质量评价分数。
如上所述的方面和任一可选的实现方式，进一步提供一种实现方式，所述方法还包括：
根据至少一个问答数据的质量评价分数，对所述至少一个问答数据的质量评价分数对应的应答人员的问答质量进行评价，得到所述应答人员的问答质量评分。
上述技术方案中的一个技术方案具有如下有益效果：
本申请实施例中，获取到样本问答数据后，会基于槽位填充方式得到这些样本问答数据对应的样本质量评价分数，从而，将样本问答数据作为初始质量评价模型的输入，将得到的样本质量评价分数作为初始质量评价模型的输出，利用神经网络机制对初始质量评价模型进行训练，即可得到目标质量评价模型。由此，相较于现有技术中通过配置复杂的打分规则得到的质量评价分数，本申请实施例中基于神经网络训练得到的目标质量评价模型更加贴近问答数据的评分需求，具有较高的灵活性；由此，只要样本问答数据的基数足够大，那么，得到的问答数据的质量评价分数就能够具备较高的准确率。因此，本申请实施例所提供的技术方案能够解决现有技术中通过预设的配置规则进行质量评价导致分数的准确率较低的问题。
第二方面，本申请实施例提供了一种质量评价模型获取装置，包括：获取单元，用于获取样本问答数据；处理单元，用于利用槽位填充方式处理所述样本问答数据，得到与所述样本问答数据对应的样本质量评价分数；训练单元，用于将所述样本问答数据作为初始质量评价模型的输入，将所述样本质量评价分数作为所述初始质量评 价模型的输出，利用神经网络机制，对所述质量评价模型进行训练，得到训练后的目标质量评价模型。
第三方面，本申请实施例提供了一种计算机设备，包括存储器、处理器以及存储在所述存储器中并可在所述处理器上运行的计算机程序，其特征在于，所述处理器执行所述计算机程序时实现如第一方面任一项所述的质量评价模型获取方法。
第四方面，本申请实施例提供了一种计算机非易失性可读存储介质，包括：计算机可执行指令，当所述计算机可执行指令被运行时用以执行如第一方面任一项所述的质量评价模型获取方法。
上述技术方案中的一个技术方案具有如下有益效果：
本申请实施例中，获取到样本问答数据后，会基于槽位填充方式得到这些样本问答数据对应的样本质量评价分数，从而，将样本问答数据作为初始质量评价模型的输入，将得到的样本质量评价分数作为初始质量评价模型的输出，利用神经网络机制对初始质量评价模型进行训练，即可得到目标质量评价模型。由此，相较于现有技术中通过配置复杂的打分规则得到的质量评价分数，本申请实施例中基于神经网络训练得到的目标质量评价模型更加贴近问答数据的评分需求，具有较高的灵活性；由此，只要样本问答数据的基数足够大，那么，得到的问答数据的质量评价分数就能够具备较高的准确率。因此，本申请实施例所提供的技术方案能够解决现有技术中通过预设的配置规则进行质量评价导致分数的准确率较低的问题。
附图说明
为了更清楚地说明本申请实施例的技术方案，下面将对实施例中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本申请的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动性的前提下，还可以根据这些附图获得其它的附图。
图1是本申请实施例所提供的质量评价模型获取方法的实施例 一的流程示意图；
图2是本申请实施例所提供的质量评价模型获取方法的实施例二的流程示意图；
图3是本申请实施例所提供的质量评价模型获取方法的实施例三的流程示意图；
图4是本申请实施例所提供的质量评价模型获取装置的功能方块图；
图5是本申请实施例所提供的计算机设备的功能方块图。
具体实施方式
为了更好的理解本申请的技术方案，下面结合附图对本申请实施例进行详细描述。
应当明确，所描述的实施例仅仅是本申请一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域普通技术人员在没有作出创造性劳动前提下所获得的所有其它实施例，都属于本申请保护的范围。
在本申请实施例中使用的术语是仅仅出于描述特定实施例的目的，而非旨在限制本申请。在本申请实施例和所附权利要求书中所使用的单数形式的“一种”、“所述”和“该”也旨在包括多数形式，除非上下文清楚地表示其他含义。
应当理解，本文中使用的术语“和/或”仅仅是一种描述关联对象的关联关系，表示可以存在三种关系，例如，A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B这三种情况。另外，本文中字符“/”，一般表示前后关联对象是一种“或”的关系。
应当理解，尽管在本申请实施例中可能采用术语第一、第二、第三等来描述数目等，但这些数目不应限于这些术语。这些术语仅用来将数目彼此区分开。例如，在不脱离本申请实施例范围的情况下，第一数目也可以被称为第二数目，类似地，第二数目也可以被称为第一数目。
取决于语境，如在此所使用的词语“如果”可以被解释成为“在……时”或“当……时”或“响应于确定”或“响应于检测”。类似地，取决于语境，短语“如果确定”或“如果检测(陈述的条件或事件)”可以被解释成为“当确定时”或“响应于确定”或“当检测(陈述的条件或事件)时”或“响应于检测(陈述的条件或事件)”。
针对现有技术的针对问答数据的质量评价方式存在配置规则复杂且准去率较低的问题，本申请实施例提供了如下解决思路：通过神经网络对样本问答数据进行学习，得到灵活性更高、更贴近问答质量评估方式的质量评价模型，从而，利用该质量评价模型来实现对问答数据的质量评价。
在该思路的引导下，本方案实施例提供了以下可行的实施方案。
实施例一
第一方面，本申请实施例给出一种质量评价模型获取方法。请参考图1，该方法可以包括以下步骤：
S102，获取样本问答数据。
S104，利用槽位填充方式处理样本问答数据，得到与样本问答数据对应的样本质量评价分数。
S106，将样本问答数据作为初始质量评价模型的输入，将样本质量评价分数作为初始质量评价模型的输出，利用神经网络机制，对初始质量评价模型进行训练，得到训练后的目标质量评价模型。
本申请实施例所涉及的样本问答数据包括：应答人员的应答数据或者应答服务器的应答数据，此外，还可以包括用户的提问数据。
其中，问答数据的表现形式一般表现为语句。在其他的具体应用场景中，还可以有其他表现形式，本申请实施例对此无特别限定。
以及，本申请实施例对于问答数据的应用场景无特别限定。例如，在一个可行的实现场景中，该问答数据可以为购物平台中购物者与销售客服之间的问答数据；或者，又例如，在另一个可行的实现场景中，该问答数据还可以为企业网站针对业务咨询的问答数据。
本申请实施例中在具体实现S102步骤时，样本问答数据可以来 源于操作人员的主动输入，或者，还可以在执行本方案时自动获取。其中，自动获取的方式可以有多种，例如，当本方案执行时，自动调取存储的所有历史问答数据，以作为样本问答数据。或者，又例如，考虑到过久的历史问答数据可能不准确，影响处理结果，因此，也可以自动调取与当前时刻之前的某段时间段内的历史问答数据，以作为样本问答数据。
其中，历史问答数据可以存储于自身，或者，也可以存储于其他存储空间，如存储于云服务器，或者如存储于第三方的存储空间。当历史问答数据存储于除自身之外的其他存储空间时，可以通过发送数据查询请求，并接收存储空间基于该数据查询请求反馈的历史问答数据的方式实现；或者，若具备数据调用权限的话，则可以直接去第三方存储空间调用这些历史问答数据。
本申请实施例的申请构思是基于样本问答数据进行深度学习，得到目标质量评价模型，该目标质量评价模型的输入为应答方的问答数据，输出为这些问答数据的质量评价分数。可以理解的是，样本问答数据的数目越多，该目标质量评价模型越准确且越贴近真实情况。
基于此，在执行S104时，可通过槽位填充方式来获取与样本问答数据对应的样本质量评价分数。
其中，槽位填充的方式是指将问答数据中的提问特征与答复特征分别作为槽位，通过匹配二者之间的槽位是否匹配来实现评分。在具体实现时，提问特征与答复特征可以通过语义识别的方式来获取。
具体的，针对每个样本问答数据，执行S104时可以参考图2，具体包括如下步骤：
S1042，对样本问答数据进行语义识别，得到样本问答数据的至少一个提问特征与至少一个答复特征。
S1044，利用槽位填充方式，对各提问特征与各答复特征进行匹配，得到有效问答特征。
S1046，根据有效问答特征，确定样本质量评价分数。
其中，S1042过程中所涉及的语义识别是通过对当前问题产生的上下文进行对话结构建模，通过机器学习方法分析出用户在提出当前问题时的上下文信息，通过对当前问题的句子组成结构分析，识别出当前问题所缺失的句子组成成份，再结合上下文进行主题分析和语义分析，对当前问题的语义信息进行扩展和理解，然后从系统检索出符合当前问题语义的答案。
为了便于理解，本申请实施例给出一种可实现的语义识别的具体实现方式：首先，对样本问答数据进行构造分析，通过句子成分分析模型得到样本问答数据的组成成分，然后判断样本问答数据中缺省的成分是什么，得到样本问答数据的成分组成与缺省成分记录。之后，通过人的口语习惯分析，发现缺省的问题成分通常会存在于对话的上下文中，则通过对上下文的成分进行分析，并结合咨询问题的句子成分，共同进行用户的语义识别，最终将句子的基础组成成分补充完整。从而，通过对样本问答数据的组成成分做为输入，将问题进行重新组合，并生成为系统能进行解释的句子表达方式，得到样本问答数据的至少一个提问特征与至少一个答复特征。
在图2所示方案中，S1044中，针对每个提问特征，当该提问特征能够在答复特征中得到匹配，则确定该提问特征为有效问答特征。
其中，提问特征与答复特征的匹配为语义匹配。以及，判定二者是否匹配可以提前预设。以及，在预设时，可以预设一个提问特征对应于多个答复特征，一个答复特征也可以对应于多个提问特征。
基于S1044得到的有效问答特征，确定样本质量评价分数的实现方式可以包括但不限于以下几种：
第一种，获取所述有效问答特征的数目，以作为所述样本质量评价分数。该方案是将有效问答特征的数目作为样本质量评价分数，那么，当提问特征与答复特征相匹配时，则记为1，当二者不匹配时，则记为0；如此，将每个提问特征的答复特征的匹配分数进行求和，就可以得到样本质量评价分数。
例如，一个样本问答数据经语义识别，得到5个提问特征和5个答复特征，然后基于二者之间的语义进行一一匹配，当存在与提问特征相匹配的答复特征时，则计数加一，反之，则不计数。如此，若5个提问特征中有3个能够在答复特征中得到匹配，那么，则可以得到该样本问答数据的质量评价分数为3。
第二种，获取所述有效问答特征的第一数目与总提问特征的第二数目，然后，获取所述第一数目与所述第二数目之间的比例，以作为所述样本质量评价分数。该方案是将有效问答特征在总提问特征中所占的比例，作为样本质量评价分数，那么，在得到样本质量评价分数后，还需要根据总提问特征的数目做归一化处理。
此时，仍以上述举例进行说明。此时，可以获取3个得到匹配的提问特征在总的提问特征中所占的比例，得到样本质量的评价分数，此时，为3/5＝0.6，也就是，该样本问答数据的质量评价分数为0.6。
第三种，根据预设的各提问特征的权重，获取各提问特征与各自对应的权重之积，其中，所述有效问答特征的权重大于其余提问特征的权重，然后，获取各提问特征的积之和，以作为所述样本质量评价分数。该方案是根据提问特征的重要程度，为每个提问特征设置不同的权重，通过权重的校正，得到样本质量评价分数。
此时，仍以上述举例进行说明。5个提问特征的权重分别为0.1、0.3、0.3、0.1、0.2，且第一个提问特征、第三个提问特征与第五个提问特征得到匹配，那么，根据权重的校正，得到其样本质量评价分数为：1*0.1+1*0.3+1*0.2＝0.6。
需要说明的是，以上三种实现方式可以单独使用，或者，也可以以至少两种实现方式的组合方式使用，本申请实施例对此无特别限定。
具体的，神经网络是一种模仿生物大脑突触-神经元结构进行数据处理的计算系统，由分为多层的计算节点和层间的连接组成。每个节点模拟一个神经元，执行某个特定运算，例如激活函数，节点之 间的连接模拟神经突触，连接对于来自上一层节点输入的加权值代表了突触权重。神经网络具有强大的非线性、自适应信息处理能力。神经网络中的神经元将来自连接输入的累加值用激活函数处理后作为自身的输出。对应于不同的网络拓扑结构、神经元模型和学习规则，人工神经网络又包括感知器、Hopfield网络、波耳兹曼机等数十种网络模型，可以实现多种多样的功能，在模式识别、复杂控制、信号处理和最优化等方面都有应用。
那么，基于以上步骤，可以得到每个样本问答数据对应的样本质量评价分数，基于此，S106在实现时，将样本问答数据作为初始质量评价模型的输入，将样本质量评价分数作为初始质量评价模型的输出，利用神经网络机制，对初始质量评价模型进行学习，即可得到目标质量评价模型。
基于上述方法，可以得到基于样本问答数据训练后的目标质量评价模型，基于此，在对待评价问答数据进行质量评价时，如图3所示，还可以包括如下步骤：
S108，将待评价问答数据输入目标质量评价模型，得到待评价问答数据的质量评价分数。
如此，能够便捷的实现对待评价问答数据的质量评价，且相较于现有技术中通过配置复杂的打分规则得到的质量评价分数，基于这种训练应答的方式得到的目标质量评价模型更加贴近问答数据的评分需求，具有较高的灵活性；由此，只要样本问答数据的基数足够大，那么，得到的问答数据的质量评价分数就能够具备较高的准确率。
进一步的，在另一具体应用过过程中，考虑到一个应答方，可以包括应答人员或应答服务器，可能存在多个问答数据，那么，基于这些问答数据的质量评价分数，还可以进一步对应答方的整体应答质量进行评价。
此时，如图3所示，该方法还可以包括如下步骤：
S109，根据至少一个问答数据的质量评价分数，对至少一个问 答数据的质量评价分数对应的应答人员的问答质量进行评价，得到应答人员的问答质量评分。
也就是说，根据应答方的至少一个问答数据的质量评价分数，对应答方的整体问答质量进行评价，得到应答人员的整体问答质量评分。这也能够进一步简化对应答方的质量评价过程，进一步简化人工操作，解放人力。
基于上述质量评价模型获取方法，本申请实施例还提供了一种计算机非易失性可读存储介质，包括：计算机可执行指令，当所述计算机可执行指令被运行时用以执行以下步骤：
获取样本问答数据；利用槽位填充方式处理样本问答数据，得到与样本问答数据对应的样本质量评价分数；将样本问答数据作为初始质量评价模型的输入，将样本质量评价分数作为初始质量评价模型的输出，利用神经网络机制，对初始质量评价模型进行训练，得到训练后的目标质量评价模型。
可选地，当计算机可执行指令被运行时还用以执行以下步骤：对样本问答数据进行语义识别，得到样本问答数据的至少一个提问特征与至少一个答复特征；利用槽位填充方式，对各提问特征与各答复特征进行匹配，得到有效问答特征；根据有效问答特征，确定样本质量评价分数。
可选地，当计算机可执行指令被运行时还用以执行以下步骤：获取有效问答特征的数目，以作为样本质量评价分数。
可选地，当计算机可执行指令被运行时还用以执行以下步骤：获取有效问答特征的第一数目与总提问特征的第二数目；获取第一数目与第二数目之间的比例，以作为样本质量评价分数。
本申请实施例的技术方案具有以下有益效果：
本申请实施例中，获取到样本问答数据后，会基于槽位填充方式得到这些样本问答数据对应的样本质量评价分数，从而，将样本问答数据作为初始质量评价模型的输入，将得到的样本质量评价分数作为初始质量评价模型的输出，利用神经网络机制对初始质量评价 模型进行训练，即可得到目标质量评价模型。由此，相较于现有技术中通过配置复杂的打分规则得到的质量评价分数，本申请实施例中基于神经网络训练得到的目标质量评价模型更加贴近问答数据的评分需求，具有较高的灵活性；由此，只要样本问答数据的基数足够大，那么，得到的问答数据的质量评价分数就能够具备较高的准确率。因此，本申请实施例所提供的技术方案能够解决现有技术中通过预设的配置规则进行质量评价导致分数的准确率较低的问题。
实施例二
基于上述实施例二所提供的质量评价模型获取方法，本申请实施例进一步给出实现上述方法实施例中各步骤及方法的装置实施例。
一方面，本申请实施例还提供了一种质量评价模型获取装置，具体的，可以参考图4，该质量评价模型获取装置400可以包括：
获取单元41，用于获取样本问答数据；
处理单元42，用于利用槽位填充方式处理样本问答数据，得到与样本问答数据对应的样本质量评价分数；
训练单元43，用于将样本问答数据作为初始质量评价模型的输入，将样本质量评价分数作为初始质量评价模型的输出，利用神经网络机制，对质量评价模型进行训练，得到训练后的目标质量评价模型。
其中，处理单元42具体用于：
对样本问答数据进行语义识别，得到样本问答数据的至少一个提问特征与至少一个答复特征；
利用槽位填充方式，对各提问特征与各答复特征进行匹配，得到有效问答特征；
根据有效问答特征，确定样本质量评价分数。
在一个实现场景中，处理单元42在执行根据有效问答特征，确定样本质量评价分数这一步骤时，可具体用于：
获取有效问答特征的数目，以作为样本质量评价分数。
在另一个实现场景中，处理单元42在执行根据有效问答特征，确定样本质量评价分数这一步骤时，可具体用于：
获取有效问答特征的第一数目与总提问特征的第二数目；
获取第一数目与第二数目之间的比例，以作为样本质量评价分数。
在另一个实现场景中，处理单元42在执行根据有效问答特征，确定样本质量评价分数这一步骤时，可具体用于：
根据预设的各提问特征的权重，获取各提问特征与各自对应的权重之积，其中，有效问答特征的权重大于其余提问特征的权重；
获取各提问特征的积之和，以作为样本质量评价分数。
本申请实施例中，该质量评价模型获取装置400还可以包括：
评价单元(图4未示出)，用于将待评价问答数据输入目标质量评价模型，得到待评价问答数据的质量评价分数。
在另一实现场景中，该评价单元，还可进一步用于：
根据至少一个问答数据的质量评价分数，对至少一个问答数据的质量评价分数对应的应答人员的问答质量进行评价，得到应答人员的问答质量评分。
另一方面，本申请实施例还提供了一种计算机设备，请参考图5，该计算机设备500包括：存储器51、处理器52以及存储在存储器51中并可在处理器52上运行的计算机程序，处理器52执行计算机程序时实现以下步骤：
获取样本问答数据；利用槽位填充方式处理样本问答数据，得到与样本问答数据对应的样本质量评价分数；将样本问答数据作为初始质量评价模型的输入，将样本质量评价分数作为初始质量评价模型的输出，利用神经网络机制，对初始质量评价模型进行训练，得到训练后的目标质量评价模型。
可选地，处理器执行计算机程序时还实现以下步骤：对样本问答数据进行语义识别，得到样本问答数据的至少一个提问特征与至少一个答复特征；利用槽位填充方式，对各提问特征与各答复特征进行匹配，得到有效 问答特征；根据有效问答特征，确定样本质量评价分数。
可选地，处理器执行计算机程序时还实现以下步骤：获取有效问答特征的数目，以作为样本质量评价分数。
可选地，处理器执行计算机程序时还实现以下步骤：获取有效问答特征的第一数目与总提问特征的第二数目；获取第一数目与第二数目之间的比例，以作为样本质量评价分数。
由于本实施例中的各单元能够执行实施例一所示的方法，本实施例未详细描述的部分，可参考对实施例一的相关说明。
本申请实施例的技术方案具有以下有益效果：
本申请实施例中，获取到样本问答数据后，会基于槽位填充方式得到这些样本问答数据对应的样本质量评价分数，从而，将样本问答数据作为初始质量评价模型的输入，将得到的样本质量评价分数作为初始质量评价模型的输出，利用神经网络机制对初始质量评价模型进行训练，即可得到目标质量评价模型。由此，相较于现有技术中通过配置复杂的打分规则得到的质量评价分数，本申请实施例中基于神经网络训练得到的目标质量评价模型更加贴近问答数据的评分需求，具有较高的灵活性；由此，只要样本问答数据的基数足够大，那么，得到的问答数据的质量评价分数就能够具备较高的准确率。因此，本申请实施例所提供的技术方案能够解决现有技术中通过预设的配置规则进行质量评价导致分数的准确率较低的问题。
所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，上述描述的系统，装置和单元的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。
在本申请所提供的几个实施例中，应该理解到，所揭露的系统，装置和方法，可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，所述单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如，多个单元或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执 行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性，机械或其它的形式。
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外，在本申请各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现，也可以采用硬件加软件功能单元的形式实现。
上述以软件功能单元的形式实现的集成的单元，可以存储在一个计算机可读取存储介质中。上述软件功能单元存储在一个存储介质中，包括若干指令用以使得一台计算机装置(可以是个人计算机，服务器，或者网络装置等)或处理器(Processor)执行本申请各个实施例所述方法的部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(Read-Only Memory，ROM)、随机存取存储器(Random Access Memory，RAM)、磁碟或者光盘等各种可以存储程序代码的介质。
以上所述仅为本申请的较佳实施例而已，并不用以限制本申请，凡在本申请的精神和原则之内，所做的任何修改、等同替换、改进等，均应包含在本申请保护的范围之内。

Claims (20)

  1. 一种质量评价模型获取方法，其特征在于，包括：
    获取样本问答数据；
    利用槽位填充方式处理所述样本问答数据，得到与所述样本问答数据对应的样本质量评价分数；
    将所述样本问答数据作为初始质量评价模型的输入，将所述样本质量评价分数作为所述初始质量评价模型的输出，利用神经网络机制，对所述初始质量评价模型进行训练，得到训练后的目标质量评价模型。
  2. 根据权利要求1所述的方法，其特征在于，所述利用槽位填充方式处理所述样本问答数据，得到与所述样本问答数据对应的样本质量评价分数，包括：
    对所述样本问答数据进行语义识别，得到所述样本问答数据的至少一个提问特征与至少一个答复特征；
    利用所述槽位填充方式，对各提问特征与各答复特征进行匹配，得到有效问答特征；
    根据所述有效问答特征，确定所述样本质量评价分数。
  3. 根据权利要求2所述的方法，其特征在于，所述根据所述有效问答特征，确定所述样本质量评价分数，包括：
    获取所述有效问答特征的数目，以作为所述样本质量评价分数。
  4. 根据权利要求2所述的方法，其特征在于，所述根据所述有效问答特征，确定所述样本质量评价分数，包括：
    获取所述有效问答特征的第一数目与总提问特征的第二数目；
    获取所述第一数目与所述第二数目之间的比例，以作为所述样本质量评价分数。
  5. 根据权利要求2所述的方法，其特征在于，所述根据所述有效问答特征，确定所述样本质量评价分数，包括：
    根据预设的各提问特征的权重，获取各提问特征与各自对应的权重之积，其中，所述有效问答特征的权重大于其余提问特征的权重；
    获取各提问特征的积之和，以作为所述样本质量评价分数。
  6. 根据权利要求1所述的方法，其特征在于，所述方法还包括：
    将待评价问答数据输入所述目标质量评价模型，得到所述待评价问答数据的质量评价分数。
  7. 根据权利要求6所述的方法，其特征在于，所述方法还包括：
    根据至少一个问答数据的质量评价分数，对所述至少一个问答数据的质量评价分数对应的应答人员的问答质量进行评价，得到所述应答人员的问答质量评分。
  8. 一种质量评价模型获取装置，其特征在于，包括：
    获取单元，用于获取样本问答数据；
    处理单元，用于利用槽位填充方式处理所述样本问答数据，得到与所述样本问答数据对应的样本质量评价分数；
    训练单元，用于将所述样本问答数据作为初始质量评价模型的输入，将所述样本质量评价分数作为所述初始质量评价模型的输出，利用神经网络机制，对所述质量评价模型进行训练，得到训练后的目标质量评价模型。
  9. 根据权利要求8所述的装置，其特征在于，所述处理单元用于：
    对所述样本问答数据进行语义识别，得到所述样本问答数据的至少一个提问特征与至少一个答复特征；
    利用槽位填充方式，对各提问特征与各答复特征进行匹配，得到有效问答特征；
    根据所述有效问答特征，确定所述样本质量评价分数。
  10. 根据权利要求9所述的装置，其特征在于，所述处理单元用于：
    获取所述有效问答特征的数目，以作为所述样本质量评价分数。
  11. 根据权利要求9所述的装置，其特征在于，所述处理单元用于：
    获取所述有效问答特征的第一数目与总提问特征的第二数目；
    获取所述第一数目与所述第二数目之间的比例，以作为所述样本质量评价分数。
  12. 根据权利要求9所述的装置，其特征在于，所述处理单元用于：
    根据预设的各提问特征的权重，获取各提问特征与各自对应的权重之积，其中，所述有效问答特征的权重大于其余提问特征的权重；
    获取各提问特征的积之和，以作为所述样本质量评价分数。
  13. 一种计算机设备，包括存储器、处理器以及存储在所述存储器中并可在所述处理器上运行的计算机程序，其特征在于，所述处理器执行所述计算机程序时实现以下步骤：
    获取样本问答数据；
    利用槽位填充方式处理所述样本问答数据，得到与所述样本问答数据对应的样本质量评价分数；
    将所述样本问答数据作为初始质量评价模型的输入，将所述样本质量评价分数作为所述初始质量评价模型的输出，利用神经网络机制，对所述初始质量评价模型进行训练，得到训练后的目标质量评价模型。
  14. 根据权利要求13所述的计算机设备，其特征在于，所述处理器执行所述计算机程序时还实现以下步骤：
    对所述样本问答数据进行语义识别，得到所述样本问答数据的至少一个提问特征与至少一个答复特征；
    利用所述槽位填充方式，对各提问特征与各答复特征进行匹配，得到有效问答特征；
    根据所述有效问答特征，确定所述样本质量评价分数。
  15. 根据权利要求14所述的计算机设备，其特征在于，所述处理器执行所述计算机程序时还实现以下步骤：
    获取所述有效问答特征的数目，以作为所述样本质量评价分数。
  16. 根据权利要求14所述的计算机设备，其特征在于，所述处理器执行所述计算机程序时还实现以下步骤：
    获取所述有效问答特征的第一数目与总提问特征的第二数目；
    获取所述第一数目与所述第二数目之间的比例，以作为所述样本质量评价分数。
  17. 一种计算机非易失性可读存储介质，存储有计算机可执行指令，其特征在于，当所述计算机可执行指令被运行时用以执行以下步骤：
    获取样本问答数据；
    利用槽位填充方式处理所述样本问答数据，得到与所述样本问答数据对应的样本质量评价分数；
    将所述样本问答数据作为初始质量评价模型的输入，将所述样本质量评价分数作为所述初始质量评价模型的输出，利用神经网络机制，对所述初始质量评价模型进行训练，得到训练后的目标质量评价模型。
  18. 根据权利要求17所述的计算机非易失性可读存储介质，其特征在于，当所述计算机可执行指令被运行时用以执行以下步骤：
    对所述样本问答数据进行语义识别，得到所述样本问答数据的至少一个提问特征与至少一个答复特征；
    利用所述槽位填充方式，对各提问特征与各答复特征进行匹配，得到有效问答特征；
    根据所述有效问答特征，确定所述样本质量评价分数。
  19. 根据权利要求18所述的计算机非易失性可读存储介质，其特征在于，当所述计算机可执行指令被运行时还用以执行以下步骤：
    获取所述有效问答特征的数目，以作为所述样本质量评价分数。
  20. 根据权利要求18所述的计算机非易失性可读存储介质，其特征在于，当所述计算机可执行指令被运行时还用以执行以下步骤：
    获取所述有效问答特征的第一数目与总提问特征的第二数目；
    获取所述第一数目与所述第二数目之间的比例，以作为所述样本质量评价分数。
PCT/CN2018/110089 2018-06-28 2018-10-12 质量评价模型获取方法及装置、计算机设备与存储介质 Ceased WO2020000779A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201810690936.X 2018-06-28
CN201810690936.XA CN109241519B (zh) 2018-06-28 2018-06-28 质量评价模型获取方法及装置、计算机设备与存储介质

Publications (1)

Publication Number Publication Date
WO2020000779A1 true WO2020000779A1 (zh) 2020-01-02

Family

ID=65072155

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2018/110089 Ceased WO2020000779A1 (zh) 2018-06-28 2018-10-12 质量评价模型获取方法及装置、计算机设备与存储介质

Country Status (2)

Country Link
CN (1) CN109241519B (zh)
WO (1) WO2020000779A1 (zh)

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111079894A (zh) * 2019-11-08 2020-04-28 万翼科技有限公司 Bp神经网络模型构建、商家评价方法与装置
CN111428032A (zh) * 2020-03-20 2020-07-17 北京小米松果电子有限公司 内容质量评价方法及装置、电子设备、存储介质
CN112766785A (zh) * 2021-01-28 2021-05-07 中国人寿保险股份有限公司上海数据中心 用于保险数据的质量评价方法、系统、设备及存储介质
CN115017415A (zh) * 2022-06-21 2022-09-06 拉扎斯网络科技（上海）有限公司 实体对象端评价展示方法、装置、电子设备和存储介质
CN116663679A (zh) * 2023-07-25 2023-08-29 南栖仙策(南京)高新技术有限公司 一种语言模型训练方法、装置、设备和存储介质
CN117272011A (zh) * 2023-09-28 2023-12-22 北京百度网讯科技有限公司 模型测评方法、装置、电子设备和存储介质
CN117556005A (zh) * 2023-11-09 2024-02-13 北京百度网讯科技有限公司 质量评估模型的训练方法、多轮对话质量评估方法和装置
CN120067603A (zh) * 2025-04-29 2025-05-30 山东神力索具有限公司 基于神经网络的安全钩热处理工艺性能评估方法及装置

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110472113B (zh) * 2019-02-26 2022-08-26 杭州蓦然认知科技有限公司 一种智能交互引擎优化方法、装置、设备
CN110675017A (zh) * 2019-08-13 2020-01-10 平安科技（深圳）有限公司 基于人工智能的绩效评价方法和装置
CN112685547A (zh) * 2020-12-29 2021-04-20 平安普惠企业管理有限公司 话术模板评估方法、装置、电子设备及存储介质
CN117763128B (zh) * 2024-01-18 2024-06-04 杭州阿里云飞天信息技术有限公司 人机交互的数据处理方法、服务器、存储介质和程序产品

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106066851A (zh) * 2016-06-06 2016-11-02 清华大学 一种考虑评价指标的神经网络训练方法及装置
CN107832294A (zh) * 2017-11-06 2018-03-23 广州杰赛科技股份有限公司 客服质量评价方法和装置
CN107885726A (zh) * 2017-11-06 2018-04-06 广州杰赛科技股份有限公司 客服质量评价方法和装置

Family Cites Families (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101470700A (zh) * 2007-12-28 2009-07-01 日电（中国）有限公司 文本模板生成器、文本生成设备、文本检验设备及其方法
CN101739867B (zh) * 2008-11-19 2012-03-28 中国科学院自动化研究所 运用计算机对口语翻译质量进行评分的方法
CN101520802A (zh) * 2009-04-13 2009-09-02 腾讯科技（深圳）有限公司 一种问答对的质量评价方法和系统
US10235358B2 (en) * 2013-02-21 2019-03-19 Microsoft Technology Licensing, Llc Exploiting structured content for unsupervised natural language semantic parsing
US10867597B2 (en) * 2013-09-02 2020-12-15 Microsoft Technology Licensing, Llc Assignment of semantic labels to a sequence of words using neural network architectures
CN106909573A (zh) * 2015-12-23 2017-06-30 北京奇虎科技有限公司 一种评价问答对质量的方法和装置
US11783173B2 (en) * 2016-06-23 2023-10-10 Microsoft Technology Licensing, Llc Multi-domain joint semantic frame parsing
CN106156003B (zh) * 2016-06-30 2018-08-28 北京大学 一种问答系统中的问句理解方法
CN107256267B (zh) * 2017-06-19 2020-07-24 北京百度网讯科技有限公司 查询方法和装置
CN107452405B (zh) * 2017-08-16 2021-04-09 北京易真学思教育科技有限公司 一种根据语音内容进行数据评价的方法及装置
CN108090127B (zh) * 2017-11-15 2021-02-12 北京百度网讯科技有限公司 建立问答文本评价模型与评价问答文本的方法、装置
CN107808004B (zh) * 2017-11-15 2021-02-26 北京百度网讯科技有限公司 模型训练方法和系统、服务器、存储介质
CN108170792B (zh) * 2017-12-27 2021-12-28 北京百度网讯科技有限公司 基于人工智能的问答引导方法、装置和计算机设备
CN108108449A (zh) * 2017-12-27 2018-06-01 哈尔滨福满科技有限责任公司 一种面向医疗领域的基于多源异构数据问答系统及该系统的实现方法

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106066851A (zh) * 2016-06-06 2016-11-02 清华大学 一种考虑评价指标的神经网络训练方法及装置
CN107832294A (zh) * 2017-11-06 2018-03-23 广州杰赛科技股份有限公司 客服质量评价方法和装置
CN107885726A (zh) * 2017-11-06 2018-04-06 广州杰赛科技股份有限公司 客服质量评价方法和装置

Cited By (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111079894A (zh) * 2019-11-08 2020-04-28 万翼科技有限公司 Bp神经网络模型构建、商家评价方法与装置
CN111428032A (zh) * 2020-03-20 2020-07-17 北京小米松果电子有限公司 内容质量评价方法及装置、电子设备、存储介质
CN111428032B (zh) * 2020-03-20 2024-03-29 北京小米松果电子有限公司 内容质量评价方法及装置、电子设备、存储介质
CN112766785A (zh) * 2021-01-28 2021-05-07 中国人寿保险股份有限公司上海数据中心 用于保险数据的质量评价方法、系统、设备及存储介质
CN112766785B (zh) * 2021-01-28 2024-03-15 中国人寿保险股份有限公司上海数据中心 用于保险数据的质量评价方法、系统、设备及存储介质
CN115017415A (zh) * 2022-06-21 2022-09-06 拉扎斯网络科技（上海）有限公司 实体对象端评价展示方法、装置、电子设备和存储介质
CN116663679A (zh) * 2023-07-25 2023-08-29 南栖仙策(南京)高新技术有限公司 一种语言模型训练方法、装置、设备和存储介质
CN117272011A (zh) * 2023-09-28 2023-12-22 北京百度网讯科技有限公司 模型测评方法、装置、电子设备和存储介质
CN117556005A (zh) * 2023-11-09 2024-02-13 北京百度网讯科技有限公司 质量评估模型的训练方法、多轮对话质量评估方法和装置
CN120067603A (zh) * 2025-04-29 2025-05-30 山东神力索具有限公司 基于神经网络的安全钩热处理工艺性能评估方法及装置

Also Published As

Publication number Publication date
CN109241519B (zh) 2022-08-12
CN109241519A (zh) 2019-01-18

Similar Documents

Publication Publication Date Title
WO2020000779A1 (zh) 质量评价模型获取方法及装置、计算机设备与存储介质
CN110580516B (zh) 一种基于智能机器人的交互方法及装置
CN109783704A (zh) 人机混合的应答方法、系统、装置
CN111276148A (zh) 基于卷积神经网络的回访方法、系统及存储介质
US12206630B2 (en) Virtual-assistant-based resolution of user inquiries via failure-triggered document presentation
CN112182186A (zh) 智能客服的运行方法、装置以及系统
CN112925888A (zh) 一种问答响应及小样本文本匹配模型的训练方法和装置
CA3147634A1 (en) Method and apparatus for analyzing sales conversation based on voice recognition
CN108632137A (zh) 应答模型训练方法、智能聊天方法、装置、设备及介质
JP2019192246A (ja) 自然言語質問回答システム用のトレーニングデータを提供する方法および装置
CN113778871A (zh) Mock测试方法、装置、设备及存储介质
US11995523B2 (en) Systems and methods for determining training parameters for dialog generation
WO2022007723A1 (zh) 语句类型识别方法、装置、电子设备和存储介质
CN113205281A (zh) 一种基于场景模拟的人员能力评价方法及相关设备
CN113918687A (zh) 话术辅助训练系统、方法、计算机装置和存储介质
CN117573946A (zh) 对话样本生成方法、聊天对话大模型训练方法及相关装置
CN107562911A (zh) 多轮交互概率模型训练方法及自动应答方法
WO2021135322A1 (zh) 一种自动出题方法、装置及系统
Zhang et al. Digital transformation (DX) for skill learners: The design methodology and implementation of educational chatbot using knowledge connection and emotional expression
CN117745488A (zh) 客服培训方法及设备
CN119443802A (zh) 一种员工流失风险预测方法、装置及设备
CN117422198A (zh) 基于人工智能的学生成长评价系统及方法
CN110069783A (zh) 一种答题内容评测方法及装置
WO2020024452A1 (zh) 基于深度学习的应答方法、装置及可读存储介质
CN113011447B (zh) 一种机器人自主学习的方法、装置及机器人

Legal Events

Date Code Title Description
NENP Non-entry into the national phase

Ref country code: DE

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18924070

Country of ref document: EP

Kind code of ref document: A1

122 Ep: pct application non-entry in european phase

Ref document number: 18924070

Country of ref document: EP

Kind code of ref document: A1