WO2020024699A1 - 客服机器人的自检方法、电子设备和计算机可读存储介质 - Google Patents

客服机器人的自检方法、电子设备和计算机可读存储介质 Download PDF

Info

Publication number
WO2020024699A1
WO2020024699A1 PCT/CN2019/089964 CN2019089964W WO2020024699A1 WO 2020024699 A1 WO2020024699 A1 WO 2020024699A1 CN 2019089964 W CN2019089964 W CN 2019089964W WO 2020024699 A1 WO2020024699 A1 WO 2020024699A1
Authority
WO
WIPO (PCT)
Prior art keywords
dialogue
matching score
score
robot
trigger
Prior art date
Application number
PCT/CN2019/089964
Other languages
English (en)
French (fr)
Inventor
徐易楠
刘云峰
吴悦
胡晓
汶林丁
Original Assignee
深圳追一科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 深圳追一科技有限公司 filed Critical 深圳追一科技有限公司
Publication of WO2020024699A1 publication Critical patent/WO2020024699A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00—Commerce
    • G06Q30/01—Customer relationship services
    • G06Q30/015—Providing customer assistance, e.g. assisting a customer within a business location or via helpdesk
    • G06Q30/016—After-sales
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
    • G06Q10/063—Operations research, analysis or management
    • G06Q10/0639—Performance analysis of employees; Performance analysis of enterprise or organisation operations

Definitions

  • the present application relates to the technical field of customer service robots, and in particular, to a self-test method for a customer service robot, an electronic device, and a nonvolatile computer-readable storage medium.
  • Customer service is a major way for companies to obtain user feedback and resolve user product questions.
  • the traditional customer service business is mainly handled by professional human customer service personnel, so that the company's investment in customer service will increase rapidly with the increase in customer service business volume, which can not be ignored.
  • a more common solution is to introduce an intelligent customer service robot.
  • the method is to analyze the user's high-frequency and clear-cut hot questions, and abstract them into several types of standard questions (Frequently Asked Questions, FAQs).
  • FAQs Frequently Asked Questions
  • For each FAQ a professional business person configures standard answers;
  • For specific problems use technical means to analyze whether the problem is reduced to any existing FAQ, and if successful, return the pre-configured answer to the user to achieve the effect of efficiently solving the user's questions.
  • the introduction of customer service robots can significantly reduce the amount of manual customer service and save a lot of customer service costs.
  • the customer service robot application does have obvious advantages in customer service work. First, it improves user perception and provides unified and intelligent self-service support for online customer service and new media customer service, which reduces the difficulty and complexity of user problems. Service efficiency, shortening the time limit for consulting and processing, offloading the pressure of traditional manual customer service, and saving service costs; Third, quickly collect user demand and behavior data to support iterative product optimization.
  • customer service robots have the above advantages, any technology will face some problems. For example, in some cases, the intelligent customer service robot will return an answer that does not meet the expectations. We can call it a badcase. If we want to maintain a good user experience for the customer service robot, we must quickly find these badcases and fix them in time.
  • the main way to evaluate the intelligent customer service is to randomly check a part of the online response at a fixed time (which can be called a case), and manually analyze to obtain the robot response accuracy rate.
  • this evaluation method can cover a part of badcases, badcase analysis cannot be performed in real time.
  • all user questions cannot be checked, which is not conducive to optimizing the experience of intelligent customer service.
  • a self-test method for a customer service robot, an electronic device, and a non-volatile computer-readable storage medium are provided.
  • a self-checking method for a customer service robot includes:
  • An electronic device includes a memory and a processor.
  • the memory stores computer-readable instructions.
  • the processor causes the processor to perform the following operations:
  • One or more non-volatile computer-readable storage media containing computer-executable instructions when the computer-executable instructions are executed by one or more processors, cause the processors to perform the following operations:
  • FIG. 1 is an application environment diagram of a self-check method of a customer service robot in one or more embodiments.
  • FIG. 2 is a flowchart of a self-checking method of a customer service robot in one or more embodiments.
  • FIG. 3 is a frame diagram of a self-check system of a customer service robot in one or more embodiments.
  • FIG. 4 is a structural block diagram of a self-checking device of a customer service robot in one or more embodiments.
  • FIG. 5 is a schematic diagram of an internal structure of an electronic device provided in one or more embodiments.
  • the self-checking method of the customer service robot provided in this application can be applied to the application environment shown in FIG. 1.
  • the application environment includes an electronic device 102.
  • the electronic device 102 can obtain the user question and its corresponding robot response; process the user question and the robot response to obtain a matching score; and determine whether to trigger an error warning based on the matching score.
  • the electronic device 102 is various personal computers, notebook computers, smart phones, tablet computers, portable wearable devices, and the like.
  • the electronic device 102 may also be a server, and the server may be implemented by an independent server or a server cluster composed of multiple servers.
  • FIG. 2 is a flowchart of a self-checking method of a customer service robot in one or more embodiments. As shown in Figure 2, the method includes:
  • Operation 201 Obtain a user question and a corresponding robot response.
  • the user question is a question sentence input by the user
  • the robot reply is the content that the customer service robot responds to the user question.
  • the user question and the robot response are processed to obtain a matching score.
  • the matching score is the basis for evaluating the correlation between the household question and the robot response, and is used for self-evaluation to find out badcases.
  • Operation 203 Determine whether to trigger an error warning according to the matching score.
  • the main responsibility of the intelligent customer service robot is to provide customers with professional problem-solving services.
  • the focus of their attention is on how much problem-solving ability is within the business scope. Therefore, if the intelligent customer service robot can find the badcase error response and feedback to the customer service personnel, it can greatly improve the user experience of the customer service robot.
  • the self-inspection method for customer service robots proposed in this application can reduce the labor cost of the robots' response to quality inspection; it can expand the coverage of the inspection robot's response; it can also improve the real-time nature of the robot's response to the quality inspection and detect robot problems in time. Since the error response can be found in time and feedback to the customer service staff, the user experience of the customer service robot can be greatly improved.
  • a user question and a corresponding robot reply are used as a conversation
  • the process of processing the user question and the robot response includes:
  • the dialogue information contained in a dialogue is input into the algorithm model; the algorithm model automatically outputs the matching score of the current dialogue according to the dialogue information.
  • operation 202 includes a process of processing a user question and a robot response, including:
  • the algorithm model calculates the matching score of the current round of dialogue based on the matching score of each conversation.
  • the matching score can be a continuous score of 1-5.
  • a total of n dialogues are included, and the robot inputs the user questions and robot responses included in each dialogue into the self-check system to obtain the corresponding check result score.
  • the final user dialogue result score can be obtained by a weighted average of the check result scores of each user dialogue or other algorithms.
  • the solution of the present application is not limited to the above-mentioned expression form using digital scores, and can also be described using discrete states, such as "very qualified”, “relatively qualified”, “general”, “unqualified”, “very unqualified”, and the like.
  • This application is not limited to the degree of classification and refinement of the matching degree, and a coarser or finer-grained matching degree division manner may be designed according to specific business requirements.
  • the figure shows a framework diagram of a self-check system of a customer service robot.
  • the algorithm model is a model trained by using a deep learning algorithm to analyze the degree of association between the user's question and the robot's reply, and determine a matching degree score according to the degree of association.
  • the algorithm model is a model trained by using a search technology algorithm.
  • operation 203 The process of determining whether to trigger an error warning according to the matching score includes:
  • operation 203 determining whether to trigger an error warning according to the matching score, includes: reading the environmental parameter and the matching score of the previous round of dialogue; and according to the environmental parameter, the matching score of the current round of dialogue and the previous round of dialogue Calculate the comprehensive score based on the matching score of the round of dialogue; determine whether to trigger an error warning based on the comprehensive score.
  • the environmental parameters include: the current number of conversation rounds, and / or the confidence level of the robot customer service response.
  • the self-checking method for a customer service robot further includes: when an error alarm is triggered, a warning message is sent through a preset alarm mode.
  • the main idea of this solution is to add a result check module to the customer service robot, and for each service, it can give a check result of a matching degree between the user question and the robot answer. This can give maintenance personnel a real-time feedback, which can timely fix some user experience problems caused by incorrect answers, and optimize the customer experience.
  • the result checking module can be obtained by using a machine learning algorithm, a search technical solution, or other algorithms according to the manually labeled data.
  • this solution uses a specific training algorithm (deep learning algorithm) for illustration, but the implementation of this solution is not limited to this form.
  • the input of the model includes, but is not limited to, current user input, robot reply, etc., and may also include the current number of interaction rounds, and the confidence of the robot reply.
  • the algorithm flow in this example is as follows:
  • the inspection score of the result inspection module can be obtained by using relevant labeled training algorithms (such as deep learning) through manually labeled score data;
  • the corresponding early warning information can be sent to the relevant personnel, so as to perform timely repair.
  • the above data can also be gradually added, and the effect can be gradually and iteratively optimized.
  • the method of the present application has the following beneficial effects: the effect evaluation of the robot answer is obtained in real time, reducing the human resources required for the result evaluation and the lag caused by it; it can quickly iterate the robot to improve the customer service experience; and improve the inspection of the customer service robot answer Coverage; to avoid errors in inspection results due to the negligence of inspectors.
  • FIG. 4 is a structural block diagram of a self-checking device of a customer service robot in one or more embodiments. As shown in Figure 4, the device includes:
  • the obtaining module 401 is configured to obtain a user question and a corresponding robot response.
  • the processing module 402 is configured to process the user question and the robot response to obtain a matching score.
  • a determining module 403 is configured to determine whether to trigger an error warning according to the matching score.
  • a self-checking device for a customer service robot provided further includes an early warning module 404, and the early warning module 404 is configured to issue a warning message through a preset warning mode when an error warning is triggered.
  • the processing module 402 may be further configured to input the dialogue information contained in a dialogue into an algorithm model; and automatically output the matching score of the current dialogue according to the dialogue information through the algorithm model.
  • the processing module 402 may be further configured to record the matching score of each conversation; and an algorithm model is used to calculate the matching score of the current round of dialogue according to the matching score of each conversation.
  • the determining module 403 may be further configured to read the matching score of the previous round of dialogue; calculate the comprehensive score based on the matching score of the current round of dialogue and the matching score of the previous round of dialogue; Whether to trigger error warning.
  • the determination module 403 may be further configured to read the environmental parameter and the matching score of the previous round of dialogue; and calculate the synthesis based on the environmental parameter, the matching score of the current round of dialogue, and the matching score of the previous round of dialogue. Grading; judging whether to trigger an error warning based on the comprehensive scoring.
  • An embodiment of the present application further provides an electronic device including a memory and a processor.
  • the memory stores computer-readable instructions.
  • the processor is caused to execute the service robot provided by the foregoing embodiment. Operation of the self-test method.
  • FIG. 5 is a schematic diagram of an internal structure of an electronic device according to an embodiment.
  • the electronic device includes a processor, a memory, a network interface, a display screen, and an input device connected through a system bus.
  • the processor of the electronic device is used to provide computing and control capabilities.
  • the memory of the electronic device includes a non-volatile storage medium and an internal memory.
  • the non-volatile storage medium stores an operating system and a computer program.
  • the internal memory provides an environment for running an operating system and computer programs in a non-volatile storage medium.
  • the network interface of the electronic device is used to communicate with an external terminal through a network connection.
  • the computer program is executed by a processor to implement a self-checking method for a customer service robot.
  • the display screen of the electronic device may be a liquid crystal display screen or an electronic ink display screen.
  • the input device of the electronic device may be a touch layer covered on the display screen, or a button, a trackball, or a touchpad provided on the electronic device casing. , Or an external keyboard, trackpad, or mouse.
  • the self-checking device of the customer service robot provided in this application may be implemented in the form of a computer program, and the computer program may be run on an electronic device as shown in FIG. 5.
  • Each program module constituting the self-testing device of the customer service robot can be stored in the memory of the electronic device.
  • the computer program constituted by each program module causes the processor to perform the operations in the self-check method of the customer service robot of each embodiment of the present application described in this specification.
  • Each module in the self-checking device of the above-mentioned customer service robot can be realized in whole or in part by software, hardware, and a combination thereof.
  • the above-mentioned modules may be embedded in the hardware or independent of the processor in the electronic device, or may be stored in the memory of the electronic device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.
  • a module may be, but is not limited to, a process running on a processor, a processor, an object, executable code, a thread of execution, a program, and / or a computer.
  • a module may be, but is not limited to, a process running on a processor, a processor, an object, executable code, a thread of execution, a program, and / or a computer.
  • the application running on the server and the server can be modules.
  • One or more modules can reside within a process and / or thread of execution and a module can be localized on one computer and / or distributed between two or more computers.
  • Any process or method description in a flowchart or otherwise described herein can be understood as representing a module, fragment, or portion of code that includes one or more executable instructions for implementing the operation of a particular logical function or process
  • the scope of the preferred embodiments of this application includes additional implementations in which the functions may be performed out of the order shown or discussed, including performing the functions in a substantially simultaneous manner or in the reverse order according to the functions involved, which should It is understood by those skilled in the art to which the embodiments of the present application pertain.
  • each part of the application may be implemented by hardware, software, firmware, or a combination thereof.
  • multiple operations or methods may be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system.
  • a suitable instruction execution system For example, if implemented in hardware, as in another embodiment, it may be implemented using any one or a combination of the following techniques known in the art: Discrete logic circuits, application-specific integrated circuits with suitable combinational logic gate circuits, programmable gate arrays (PGA), field programmable gate arrays (FPGA), etc.
  • each functional unit in each embodiment of the present application may be integrated into one processing module, or each unit may exist separately physically, or two or more units may be integrated into one module.
  • the above integrated modules may be implemented in the form of hardware or software functional modules. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a non-volatile computer-readable storage medium.
  • the non-volatile computer-readable storage medium mentioned above may be a read-only memory, a magnetic disk, or an optical disk.

Landscapes

  • Business, Economics & Management (AREA)
  • Human Resources & Organizations (AREA)
  • Engineering & Computer Science (AREA)
  • Strategic Management (AREA)
  • Development Economics (AREA)
  • Economics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Marketing (AREA)
  • Educational Administration (AREA)
  • General Business, Economics & Management (AREA)
  • Physics & Mathematics (AREA)
  • Tourism & Hospitality (AREA)
  • Quality & Reliability (AREA)
  • Operations Research (AREA)
  • Game Theory and Decision Science (AREA)
  • Accounting & Taxation (AREA)
  • Finance (AREA)
  • Manipulator (AREA)
  • Electrically Operated Instructional Devices (AREA)

Abstract

一种客服机器人的自检方法，包括：获取用户问句及其对应的机器人回复；对所述用户问句和所述机器人回复进行处理，得出一个匹配度评分；根据所述匹配度评分确定是否触发错误预警。

Description

客服机器人的自检方法、电子设备和计算机可读存储介质
相关申请的交叉引用
本申请要求于2018年07月30日提交中国专利局、申请号为201810855366.5、发明名称为“客服机器人的自检方法和装置”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及客服机器人技术领域，具体涉及一种客服机器人的自检方法、电子设备和非易失性计算机可读存储介质。
背景技术
客服是企业获得用户反馈意见、解决用户产品疑问的一个主要途径。传统的客服业务主要由专业的人工客服人员来处理，使得企业在客服方面的投入会随着客服业务量的增加而高速增长，成为不可忽视的支出。
针对这一问题，目前比较普遍的方案是引入智能客服机器人。其做法首先是对用户高频、意图明确的热门问题进行分析，抽象成若干类标准问句(Frequently Asked Questions，简称FAQ)，对每一个FAQ由专业的业务人员配置好标准答案；然后针对用户的具体问题，采用技术手段分析该问题是否规约到任何一个已有FAQ，若成功则将预先配置好的答案返回给用户，达到高效地解决用户疑问的效果。客服机器人的引入可以显著降低人工客服量，节约大量客服成本。
客服机器人应用在客服工作中确实有着显而易见的优势：一是提高用户感知，为企业在线客服、新媒体客服等提供统一智能的自助服务支撑，降低用户问题得到解决的难度和复杂度；二是提升服务效率，缩短咨询处理时限，分流传统人工客服压力，节省服务成本；三是快速收集用户诉求和行为数据， 支撑产品迭代优化。
虽说客服机器人有着以上种种的优势，但是任何技术都会面临一些问题。比如智能客服机器人在某些情况下会返回不符合预期的答案，我们可称之为badcase(错误回复)；要维持客服机器人具有较好的用户体验，就要快速地发现这些badcase并及时修复。
相关技术中，评价智能客服的主要方式为，在固定时间抽检一部分线上回复(可以称为case)，人工分析得到机器人回复准确率。这种评价方式虽然能够覆盖一部分badcase，但是不能实时进行badcase分析，同时由于人力资源的限制，也不能对所有用户问句进行检查，不利于优化智能客服的体验。
发明内容
根据本申请的各种实施例，提供一种客服机器人的自检方法、电子设备和非易失性计算机可读存储介质。
一种客服机器人的自检方法，包括：
获取用户问句及其对应的机器人回复；
对所述用户问句和所述机器人回复进行处理，得出一个匹配度评分；及
根据所述匹配度评分确定是否触发错误预警。一种电子设备，包括存储器及处理器，所述存储器中储存有计算机可读指令，所述指令被所述处理器执行时，使得所述处理器执行如下操作：
获取用户问句及其对应的机器人回复；
对所述用户问句和所述机器人回复进行处理，得出一个匹配度评分；及
根据所述匹配度评分确定是否触发错误预警。
一个或多个包含计算机可执行指令的非易失性计算机可读存储介质，当所述计算机可执行指令被一个或多个处理器执行时，使得所述处理器执行如下操作：
获取用户问句及其对应的机器人回复；
对所述用户问句和所述机器人回复进行处理，得出一个匹配度评分；及
根据所述匹配度评分确定是否触发错误预警。
本发明的一个或多个实施例的细节在下面的附图和描述中提出。本发明的其它特征、目的和优点将从说明书、附图以及权利要求书变得明显。
附图说明
为了更清楚地说明本申请实施例或现有技术中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本申请的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
为了更好地描述和说明本申请公开的实施例和/或示例，可以参考一副或者多副附图。用于描述附图的附加细节或示例不应当被认为是对所公开的发明、目前描述的实施例和/或示例以及目前理解的这些发明的最佳模式中的任何一者的范围的限制。
图1为一个或多个实施例中客服机器人的自检方法的应用环境图。
图2为一个或多个实施例中客服机器人的自检方法的流程图。
图3为一个或多个实施例中客服机器人的自检系统的框架图。
图4为一个或多个实施例中一种客服机器人的自检装置的结构框图。
图5为一个或多个实施例中提供的电子设备的内部结构示意图。
具体实施方式
这里将详细地对示例性实施例进行说明，其示例表示在附图中。下面的描述涉及附图时，除非另有表示，不同附图中的相同数字表示相同或相似的要素。以下示例性实施例中所描述的实施方式并不代表与本申请相一致的所有实施方式。相反，它们仅是与如所附权利要求书中所详述的、本申请的一些方面相一致的装置和方法的例子。
本申请提供的客服机器人的自检方法，可以应用于如图1所示的应用环境中。该应用环境包括电子设备102。电子设备102可以获取用户问句及其 对应的机器人回复；对用户问句和机器人回复进行处理，得出一个匹配度评分；根据匹配度评分确定是否触发错误预警。其中，电子设备102是各种个人计算机、笔记本电脑、智能手机、平板电脑和便携式可穿戴设备等。可选地，电子设备102也可以是服务器，服务器可以用独立的服务器或者是多个服务器组成的服务器集群来实现。
图2为一个或多个实施例中客服机器人的自检方法的流程图。如图2所示，该方法包括：
操作201，获取用户问句及其对应的机器人回复。
其中，用户问句是由用户输入的问题语句，机器人回复是客服机器人针对用户问句所回复的内容。
操作202，对用户问句和机器人回复进行处理，得出一个匹配度评分。
匹配度评分是评价户问句和机器人回复之间关联性高低的依据，用以自我评价，从而找出错误回复(badcase)。
操作203，根据匹配度评分确定是否触发错误预警。
在找出错误回复之后，需要及时发出预警进行反馈，通知技术人员及时修复。
智能客服机器人的主要职责是为客户提供专业的解决问题的服务，其关注的焦点是聚焦在业务范围内，有多大的解决问题的能力。所以智能客服机器人若能自我发现badcase错误回复并反馈给客服人员，就能够极大提高客服机器人的使用体验。
本申请提出的客服机器人自检方法，能够减少机器人回复质量检查所耗费的人力成本；能扩大检查机器人回复的覆盖范围；还能提高机器人回答质量检查的实时性，能及时发现机器人问题。由于能够及时发现错误回复并反馈给客服人员，因而能够极大提高客服机器人的使用体验。
一些实施例中，将一个用户问句及其对应的一个机器人回复作为一次对话；
则操作202中，对用户问句和机器人回复进行处理的过程，包括：
将一次对话所包含的对话信息输入算法模型；算法模型根据对话信息自动输出当前一次对话的匹配度评分。
在一些实施例中，将一个用户与客服机器人之间的多次对话作为一轮对话；则操作202，对用户问句和机器人回复进行处理的过程，包括：
记录每一次对话的匹配度评分；算法模型根据每一次对话的匹配度评分计算出当前一轮对话的匹配度评分。
为便于理解，匹配度评分可以采用1-5分的连续分值。对于每轮用户对话，一共包含n次对话，机器人将每次对话包含的用户问句以及机器人回复输入至自检系统中，从而得到该次对应的检查结果评分。对于每轮用户对话，最终的用户对话的结果评分可以由每次用户对话的检查结果评分的加权平均或其他算法得到。
本申请的方案不限于上述采用数字分值的表现形式，也可使用离散状态进行描述，比如“非常合格”，“比较合格”，“一般”，“不合格”，“非常不合格”等。本申请也不限于匹配度的分级细化程度，可根据具体业务需求设计更粗粒度或更细粒度的匹配度划分方式。
如图3所示，图中示出了客服机器人的自检系统的框架图。
一些实施例中，算法模型为采用深度学习算法训练得到的模型，用于分析用户问句和机器人回复之间的关联程度，并根据关联程度确定一个匹配度评分。
一些实施例中，算法模型为采用搜索技术算法训练得到的模型。
在一些实施例中，操作203，根据匹配度评分确定是否触发错误预警的过程，包括：
读取上一轮对话的匹配度评分；根据当前一轮对话的匹配度评分和上一轮对话的匹配度评分计算综合评分；根据综合评分判断是否触发错误预警。
在一些实施例中，操作203，根据匹配度评分确定是否触发错误预警，包括：读取环境参数和上一轮对话的匹配度评分；根据环境参数、当前一轮对话的匹配度评分和上一轮对话的匹配度评分计算综合评分；根据综合评分 判断是否触发错误预警。
在一些实施例中，环境参数包括：当前的对话轮数，和/或，机器人客服的回复置信度。
在一些实施例中，提供的客服机器人的自检方法还包括：当触发错误报警时，通过预设的报警方式发出预警信息。
本方案的主要思想，是为客服机器人增加结果检查模块，对于每次服务均能给出用户问句与机器人回答的一个匹配程度的检查结果。这样能给维护人员一个实时的反馈，从而能及时修复一些错误回答导致的用户体验的问题，优化客户体验。
其中，结果检查模块可根据人工标注数据采用机器学习算法、搜索技术方案或其他算法训练得到。为方便理解，本方案采用某一具体的训练算法(深度学习算法)进行说明，但是本方案的实现并不限于此种形式。在结果检查模块中，模型的输入包括但不限于当前用户输入、机器人回复等，还可包括当前交互轮数、机器人的回复的置信度等。本例中的算法流程如下：
1、首先对于结果检查模块的检查得分可通过人工标注的得分数据使用相关训练算法(如深度学习)训练得到；
2、然后使用用户问句及其对应的机器人回复作为输入进入结果检查模块，得到该机器人回答的一个结果匹配度打分；
3、根据该打分结果及前轮对话的打分结果进行处理，可根据模型算法或规则匹配决定是否触发错误回答预警系统；
4、当触发预警信号时，则可发出对应的预警信息给相关人员，从而进行及时的修复。
在客服机器人服务过程中，也可逐渐补充上述数据，逐步迭代优化效果。
本申请的方法具备以下有益效果：实时获得机器人回答的效果评价，降低结果评价所需要的人力资源以及其带来的滞后性；可快速迭代机器人，提升用户客服体验；提高了客服机器人回答的检查的覆盖率；避免了由于检查人员的疏忽造成检查结果的误差。
图4为一个或多个实施例中一种客服机器人的自检装置的结构框图。如图4所示，该装置包括：
获取模块401，用于获取用户问句及其对应的机器人回复。
处理模块402，用于对用户问句和机器人回复进行处理，得出一个匹配度评分。
确定模块403，用于根据匹配度评分确定是否触发错误预警。
在一个实施例中，提供的一种客服机器人的自检装置还包括预警模块404，预警模块404用于当触发错误报警时，通过预设的报警方式发出预警信息。
在一个实施例中，处理模块402还可以用于将一次对话所包含的对话信息输入算法模型；通过算法模型根据对话信息自动输出当前一次对话的匹配度评分。
在一个实施例中，处理模块402还可以用于记录每一次对话的匹配度评分；通过算法模型根据每一次对话的匹配度评分计算出当前一轮对话的匹配度评分。
在一个实施例中，确定模块403还可以用于读取上一轮对话的匹配度评分；根据当前一轮对话的匹配度评分和上一轮对话的匹配度评分计算综合评分；根据综合评分判断是否触发错误预警。
在一个实施例中，确定模块403还可以用于读取环境参数和上一轮对话的匹配度评分；根据环境参数、当前一轮对话的匹配度评分和上一轮对话的匹配度评分计算综合评分；根据综合评分判断是否触发错误预警。
本申请实施例还提供一种电子设备，包括存储器和处理器，存储器中存有计算机可读指令，当计算机可读指令被处理执行时，使处理器执行如上述实施例所提供的客服机器人的自检方法的操作。
图5为一个实施例中电子设备的内部结构示意图。如图5所示，该电子设备包括通过系统总线连接的处理器、存储器、网络接口、显示屏和输入装置。其中，该电子设备的处理器用于提供计算和控制能力。该电子设备的存 储器包括非易失性存储介质、内存储器。该非易失性存储介质存储有操作系统和计算机程序。该内存储器为非易失性存储介质中的操作系统和计算机程序的运行提供环境。该电子设备的网络接口用于与外部的终端通过网络连接通信。该计算机程序被处理器执行时以实现一种客服机器人的自检方法。该电子设备的显示屏可以是液晶显示屏或者电子墨水显示屏，该电子设备的输入装置可以是显示屏上覆盖的触摸层，也可以是电子设备外壳上设置的按键、轨迹球或触控板，还可以是外接的键盘、触控板或鼠标等。
在一个实施例中，本申请提供的客服机器人的自检装置可以实现为一种计算机程序的形式，计算机程序可在如图5所示的电子设备上运行。电子设备的存储器中可存储组成该客服机器人的自检装置的各个程序模块。各个程序模块构成的计算机程序使得处理器执行本说明书中描述的本申请各个实施例的客服机器人的自检方法中的操作。
可以理解的是，上述各实施例中相同或相似部分可以相互参考，在一些实施例中未详细说明的内容可以参见其他实施例中相同或相似的内容。
上述客服机器人的自检装置中的各个模块可全部或部分通过软件、硬件及其组合来实现。上述各模块可以硬件形式内嵌于或独立于电子设备中的处理器中，也可以以软件形式存储于电子设备中的存储器中，以便于处理器调用执行以上各个模块对应的操作。
如在本申请中所使用的，术语“装置”、“模块”等旨在表示计算机相关的实体，它可以是硬件、硬件和软件的组合、软件、或者执行中的软件。例如，模块可以是但不限于是，在处理器上运行的进程、处理器、对象、可执行码、执行的线程、程序和/或计算机。作为说明，运行在服务器上的应用程序和服务器都可以是模块。一个或多个模块可以驻留在进程和/或执行的线程中，并且模块可以位于一个计算机内和/或分布在两个或更多的计算机之间。
需要说明的是，在本申请的描述中，术语“第一”、“第二”等仅用于描述目的，而不能理解为指示或暗示相对重要性。此外，在本申请的描述中， 除非另有说明，“多个”的含义是指至少两个。
流程图中或在此以其他方式描述的任何过程或方法描述可以被理解为，表示包括一个或更多个用于实现特定逻辑功能或过程的操作的可执行指令的代码的模块、片段或部分，并且本申请的优选实施方式的范围包括另外的实现，其中可以不按所示出或讨论的顺序，包括根据所涉及的功能按基本同时的方式或按相反的顺序，来执行功能，这应被本申请的实施例所属技术领域的技术人员所理解。
应当理解，本申请的各部分可以用硬件、软件、固件或它们的组合来实现。在上述实施方式中，多个操作或方法可以用存储在存储器中且由合适的指令执行系统执行的软件或固件来实现。例如，如果用硬件来实现，和在另一实施方式中一样，可用本领域公知的下列技术中的任一项或他们的组合来实现：具有用于对数据信号实现逻辑功能的逻辑门电路的离散逻辑电路，具有合适的组合逻辑门电路的专用集成电路，可编程门阵列(PGA)，现场可编程门阵列(FPGA)等。
本技术领域的普通技术人员可以理解实现上述实施例方法携带的全部或部分操作是可以通过程序来指令相关的硬件完成，的程序可以存储于一种计算机可读存储介质中，该程序被处理器执行时，可以实现上述实施例中所提供的客服机器人的自检方法。。
此外，在本申请各个实施例中的各功能单元可以集成在一个处理模块中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个模块中。上述集成的模块既可以采用硬件的形式实现，也可以采用软件功能模块的形式实现。集成的模块如果以软件功能模块的形式实现并作为独立的产品销售或使用时，也可以存储在一个非易失性计算机可读取存储介质中。
上述提到的非易失性计算机可读存储介质可以是只读存储器，磁盘或光盘等。
在本说明书的描述中，参考术语“一个实施例”、“一些实施例”、“示例”、“具体示例”、或“一些示例”等的描述意指结合该实施例或示例描 述的具体特征、结构、材料或者特点包含于本申请的至少一个实施例或示例中。在本说明书中，对上述术语的示意性表述不一定指的是相同的实施例或示例。而且，描述的具体特征、结构、材料或者特点可以在任何的一个或多个实施例或示例中以合适的方式结合。尽管上面已经示出和描述了本申请的实施例，可以理解的是，上述实施例是示例性的，不能理解为对本申请的限制，本领域的普通技术人员在本申请的范围内可以对上述实施例进行变化、修改、替换和变型。

Claims (25)

  1. 一种客服机器人的自检方法，包括：
    获取用户问句及其对应的机器人回复；
    对所述用户问句和所述机器人回复进行处理，得出一个匹配度评分；及
    根据所述匹配度评分确定是否触发错误预警。
  2. 根据权利要求1所述的方法，其特征在于，将一个用户问句及其对应的一个机器人回复作为一次对话；
    所述对所述用户问句和所述机器人回复进行处理，包括：
    将一次对话所包含的对话信息输入算法模型；及
    通过所述算法模型根据所述对话信息自动输出当前一次对话的匹配度评分。
  3. 根据权利要求2所述的方法，其特征在于，将一个用户与客服机器人之间的多次对话作为一轮对话；
    所述对所述用户问句和所述机器人回复进行处理，包括：
    记录每一次对话的匹配度评分；及
    通过所述算法模型根据每一次对话的匹配度评分计算出当前一轮对话的匹配度评分。
  4. 根据权利要求2或3所述的方法，其特征在于，所述算法模型为采用深度学习算法训练得到的模型，用于分析用户问句和机器人回复之间的关联程度，并根据关联程度确定一个匹配度评分。
  5. 根据权利要求2或3所述的方法，其特征在于，所述算法模型为采用搜索技术算法训练得到的模型。
  6. 根据权利要求3所述的方法，其特征在于，所述根据所述匹配度评分确定是否触发错误预警，包括：
    读取上一轮对话的匹配度评分；
    根据当前一轮对话的匹配度评分和上一轮对话的匹配度评分计算综合评分；及
    根据综合评分判断是否触发错误预警。
  7. 根据权利要求3所述的方法，其特征在于，所述根据所述匹配度评分确定是否触发错误预警，包括：
    读取环境参数和上一轮对话的匹配度评分；
    根据所述环境参数、当前一轮对话的匹配度评分和上一轮对话的匹配度评分计算综合评分；及
    根据综合评分判断是否触发错误预警。
  8. 根据权利要求7所述的方法，其特征在于，所述环境参数包括：当前的对话轮数，和/或，机器人客服的回复置信度。
  9. 根据权利要求6至8中任一项所述的方法，其特征在于，还包括：
    当触发错误报警时，通过预设的报警方式发出预警信息。
  10. 一种电子设备，包括存储器及处理器，所述存储器中储存有计算机可读指令，所述指令被所述处理器执行时，使得所述处理器执行如下操作：
    获取用户问句及其对应的机器人回复；
    对所述用户问句和所述机器人回复进行处理，得出一个匹配度评分；及
    根据所述匹配度评分确定是否触发错误预警。
  11. 根据权利要求10所述的电子设备，其特征在于，所述处理器执行所述对所述用户问句和所述机器人回复进行处理时，还执行如下操作：
    将一次对话所包含的对话信息输入算法模型，其中，一次对话为一个用户问句及其对应的一个机器人回复；及
    通过所述算法模型根据所述对话信息自动输出当前一次对话的匹配度评分。
  12. 根据权利要求11所述的电子设备，其特征在于，所述处理器执行所述对所述用户问句和所述机器人回复进行处理时，还执行如下操作：
    记录每一次对话的匹配度评分；及
    通过所述算法模型根据每一次对话的匹配度评分计算出当前一轮对话的匹配度评分，其中，一轮对话为一个用户与客服机器人之间的多次对话。
  13. 根据权利要求11或12所述的电子设备，其特征在于，所述处理器还执行：通过所述算法模型分析用户问句和机器人回复之间的关联程度，并根据关联程度确定一个匹配度评分，其中，所述算法模型为采用深度学习算法训练得到的模型。
  14. 根据权利要求11或12所述的电子设备，其特征在于，所述处理器还执行：通过采用搜索技术算法训练得到的算法模型分析用户问句和机器人回复之间的关联程度，并根据关联程度确定一个匹配度评分。
  15. 根据权利要求12所述的电子设备，其特征在于，所述处理器执行所述根据所述匹配度评分确定是否触发错误预警时，还执行如下操作：
    读取上一轮对话的匹配度评分；
    根据当前一轮对话的匹配度评分和上一轮对话的匹配度评分计算综合评分；及
    根据综合评分判断是否触发错误预警。
  16. 根据权利要求12所述的电子设备，其特征在于，所述处理器执行所述根据所述匹配度评分确定是否触发错误预警时，还执行如下操作：
    读取环境参数和上一轮对话的匹配度评分，其中，所述环境参数包括：当前的对话轮数，和/或，机器人客服的回复置信度；
    根据所述环境参数、当前一轮对话的匹配度评分和上一轮对话的匹配度评分计算综合评分；及
    根据综合评分判断是否触发错误预警。
  17. 根据权利要求15或16所述的电子设备，其特征在于，所述处理器还执行如下操作：
    当触发错误报警时，通过预设的报警方式发出预警信息。
  18. 一个或多个包含计算机可执行指令的非易失性计算机可读存储介质，当所述计算机可执行指令被一个或多个处理器执行时，使得所述处理器执行如下操作：
    获取用户问句及其对应的机器人回复；
    对所述用户问句和所述机器人回复进行处理，得出一个匹配度评分；及
    根据所述匹配度评分确定是否触发错误预警。
  19. 根据权利要求18所述的非易失性计算机可读存储介质，其特征在于，所述处理器执行所述对所述用户问句和所述机器人回复进行处理时，还执行如下操作：
    将一次对话所包含的对话信息输入算法模型，其中，一次对话为一个用户问句及其对应的一个机器人回复；及
    通过所述算法模型根据所述对话信息自动输出当前一次对话的匹配度评分。
  20. 根据权利要求19所述的非易失性计算机可读存储介质，其特征在于，所述处理器执行所述对所述用户问句和所述机器人回复进行处理时，还执行如下操作：
    记录每一次对话的匹配度评分；及
    通过所述算法模型根据每一次对话的匹配度评分计算出当前一轮对话的匹配度评分，其中，一轮对话为一个用户与客服机器人之间的多次对话。
  21. 根据权利要求19或20所述的非易失性计算机可读存储介质，其特征在于，所述处理器还执行：通过所述算法模型分析用户问句和机器人回复之间的关联程度，并根据关联程度确定一个匹配度评分，其中，所述算法模型为采用深度学习算法训练得到的模型。
  22. 根据权利要求19或20所述的计算机可读存储介质，其特征在于，所述处理器还执行：通过采用搜索技术算法训练得到的算法模型分析用户问句和机器人回复之间的关联程度，并根据关联程度确定一个匹配度评分。
  23. 根据权利要求20所述的非易失性计算机可读存储介质，其特征在于，所述处理器执行所述根据所述匹配度评分确定是否触发错误预警时，还执行如下操作：
    读取上一轮对话的匹配度评分；
    根据当前一轮对话的匹配度评分和上一轮对话的匹配度评分计算综合评 分；及
    根据综合评分判断是否触发错误预警。
  24. 根据权利要求20所述的非易失性计算机可读存储介质，其特征在于，所述处理器执行所述根据所述匹配度评分确定是否触发错误预警时，还执行如下操作：
    读取环境参数和上一轮对话的匹配度评分，其中，所述环境参数包括：当前的对话轮数，和/或，机器人客服的回复置信度；
    根据所述环境参数、当前一轮对话的匹配度评分和上一轮对话的匹配度评分计算综合评分；及
    根据综合评分判断是否触发错误预警。
  25. 根据权利要求23或24所述的非易失性计算机可读存储介质，其特征在于，所述处理器还执行如下操作：
    当触发错误报警时，通过预设的报警方式发出预警信息。
PCT/CN2019/089964 2018-07-30 2019-06-04 客服机器人的自检方法、电子设备和计算机可读存储介质 WO2020024699A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201810855366.5A CN109086400A (zh) 2018-07-30 2018-07-30 客服机器人的自检方法和装置
CN201810855366.5 2018-07-30

Publications (1)

Publication Number Publication Date
WO2020024699A1 true WO2020024699A1 (zh) 2020-02-06

Family

ID=64830992

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/089964 WO2020024699A1 (zh) 2018-07-30 2019-06-04 客服机器人的自检方法、电子设备和计算机可读存储介质

Country Status (2)

Country Link
CN (1) CN109086400A (zh)
WO (1) WO2020024699A1 (zh)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109086400A (zh) * 2018-07-30 2018-12-25 深圳追科技有限公司 客服机器人的自检方法和装置

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105591882A (zh) * 2015-12-10 2016-05-18 北京中科汇联科技股份有限公司 一种智能机器人与人混合客服的方法及系统
CN107220353A (zh) * 2017-06-01 2017-09-29 深圳追科技有限公司 一种智能客服机器人满意度自动评价方法及系统
US20170364979A1 (en) * 2010-07-23 2017-12-21 Ebay Inc. Instant messaging robot to provide product information
CN109086400A (zh) * 2018-07-30 2018-12-25 深圳追科技有限公司 客服机器人的自检方法和装置

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2016167424A1 (ko) * 2015-04-16 2016-10-20 주식회사 플런티코리아 답변 추천 장치, 자동 문장 완성 시스템 및 방법
CN105471712A (zh) * 2015-11-25 2016-04-06 深圳狗尾草智能科技有限公司 一种机器人答复系统及其答复方法
CN105630917A (zh) * 2015-12-22 2016-06-01 成都小多科技有限公司 智能应答方法及装置
CN105938484B (zh) * 2016-04-12 2020-06-23 华南师范大学 基于用户反馈知识库的机器人交互方法和系统
CN107305578A (zh) * 2016-04-25 2017-10-31 北京京东尚科信息技术有限公司 人机智能问答方法和装置
CN107239538B (zh) * 2017-06-01 2021-01-19 深圳追一科技有限公司 具有自学习功能的平行客服机器人系统及其自学习方法

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170364979A1 (en) * 2010-07-23 2017-12-21 Ebay Inc. Instant messaging robot to provide product information
CN105591882A (zh) * 2015-12-10 2016-05-18 北京中科汇联科技股份有限公司 一种智能机器人与人混合客服的方法及系统
CN107220353A (zh) * 2017-06-01 2017-09-29 深圳追科技有限公司 一种智能客服机器人满意度自动评价方法及系统
CN109086400A (zh) * 2018-07-30 2018-12-25 深圳追科技有限公司 客服机器人的自检方法和装置

Also Published As

Publication number Publication date
CN109086400A (zh) 2018-12-25

Similar Documents

Publication Publication Date Title
Willis et al. Validation of economic and health outcomes simulation model of type 2 diabetes mellitus (ECHO-T2DM)
Bédubourg et al. Evaluation and comparison of statistical methods for early temporal detection of outbreaks: A simulation-based study
WO2021027422A1 (zh) 基于数据收集的接口服务功能监测方法及系统
US11205046B2 (en) Topic monitoring for early warning with extended keyword similarity
TW201734841A (zh) 分布式環境下監督學習算法的基準測試方法和裝置
US20160019484A1 (en) System and method for managing resources of a project
US10459835B1 (en) System and method for controlling quality of performance of digital applications
CN110286938B (zh) 用于输出针对用户的评价信息的方法和装置
CA3064412A1 (en) Systems and methods for automatic identification and recommendation of techniques and experts
US20230112576A1 (en) Techniques for data processing predictions
Xiong et al. Factors that affect information and communication technology adoption by small businesses in China
US20180239496A1 (en) Clustering and analysis of commands in user interfaces
Licorish et al. On satisfying the Android OS community: User feedback still central to developers' portfolios
WO2020024699A1 (zh) 客服机器人的自检方法、电子设备和计算机可读存储介质
WO2021027552A1 (zh) 对话机器人配置方法、装置、计算机设备和存储介质
Hjorth-Hansen et al. Real-time automatic quantification of left ventricular function by hand-held ultrasound devices in patients with suspected heart failure: a feasibility study of a diagnostic test with data from general practitioners, nurses and cardiologists
Shatz Assumption-checking rather than (just) testing: The importance of visualization and effect size in statistical diagnostics
Gutiérrez et al. Analysing the energy impact of different optimisations for machine learning models
US20160012366A1 (en) System and method for optimizing project operations, resources and associated processes of an organization
US11551817B2 (en) Assessing unreliability of clinical risk prediction
CN113645357B (zh) 通话质检方法、装置、计算机设备和计算机可读存储介质
US20220414262A1 (en) Rule-based anonymization of datasets
Rudolf et al. Order indication solicitation to assess clinical laboratory test utilization: D-dimer order patterns as an illustrative case
Brown et al. The Relationship Between Perceived Uncontrollable Mortality Risk and Health Effort: Replication, Secondary Analysis, and Mini Meta-analysis
KR20180118606A (ko) 애플리케이션 프로그램 사용자 인터페이스 자동화 테스트 방법, 전자 기기, 시스템 및 저장 매체

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19844195

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 09/06/2021)

122 Ep: pct application non-entry in european phase

Ref document number: 19844195

Country of ref document: EP

Kind code of ref document: A1