CN113407694B - Method, device and related equipment for detecting ambiguity of customer service robot knowledge base - Google Patents

Method, device and related equipment for detecting ambiguity of customer service robot knowledge base Download PDF

Info

Publication number
CN113407694B
CN113407694B CN202110693227.9A CN202110693227A CN113407694B CN 113407694 B CN113407694 B CN 113407694B CN 202110693227 A CN202110693227 A CN 202110693227A CN 113407694 B CN113407694 B CN 113407694B
Authority
CN
China
Prior art keywords
category
deep learning
learning model
ambiguity
knowledge base
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202110693227.9A
Other languages
Chinese (zh)
Other versions
CN113407694A (en
Inventor
潘晟锋
刘云峰
吴悦
胡晓
汶林丁
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shenzhen Zhuiyi Technology Co Ltd
Original Assignee
Shenzhen Zhuiyi Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shenzhen Zhuiyi Technology Co Ltd filed Critical Shenzhen Zhuiyi Technology Co Ltd
Priority to CN202110693227.9A priority Critical patent/CN113407694B/en
Publication of CN113407694A publication Critical patent/CN113407694A/en
Application granted granted Critical
Publication of CN113407694B publication Critical patent/CN113407694B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/33—Querying
    • G06F16/332—Query formulation
    • G06F16/3329—Natural language query formulation or dialogue systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/35—Clustering; Classification
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/22—Matching criteria, e.g. proximity measures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/20—Natural language analysis
    • G06F40/205—Parsing
    • G06F40/211—Syntactic parsing, e.g. based on context-free grammar [CFG] or unification grammars
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/30—Semantic analysis
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00—Commerce
    • G06Q30/02—Marketing; Price estimation or determination; Fundraising
    • G06Q30/0281—Customer communication at a business location, e.g. providing product or service information, consulting

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Computational Linguistics (AREA)
  • Health & Medical Sciences (AREA)
  • Mathematical Physics (AREA)
  • General Health & Medical Sciences (AREA)
  • Evolutionary Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Business, Economics & Management (AREA)
  • Molecular Biology (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Biophysics (AREA)
  • Software Systems (AREA)
  • Biomedical Technology (AREA)
  • Accounting & Taxation (AREA)
  • Development Economics (AREA)
  • Databases & Information Systems (AREA)
  • Finance (AREA)
  • Computing Systems (AREA)
  • Strategic Management (AREA)
  • Game Theory and Decision Science (AREA)
  • Marketing (AREA)
  • General Business, Economics & Management (AREA)
  • Economics (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Human Computer Interaction (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

The application relates to a customer service robot knowledge base ambiguity detection method, a device and related equipment, comprising the following steps: constructing a knowledge base, wherein the knowledge base is divided according to FAQs, each FAQ is provided with at least one similar question, and each FAQ is of a category; dividing the knowledge base into a test set and a training set of a deep learning model; training a deep learning model on a training set, and performing ambiguity detection by using the learned deep learning model; updating the knowledge base according to the ambiguity detection result; repeating the steps until the learning effect is not improved. According to the method, the knowledge base is updated according to the ambiguity detection result, the training steps are repeated until the learning effect reaches the expected standard, the ambiguity of the knowledge base can be found and corrected manually, the disambiguated knowledge base is obtained, data are extracted from the disambiguated knowledge base to serve as a training set and a testing set of the deep learning model, and the learning effect of the deep learning model is further improved.

Description

Method, device and related equipment for detecting ambiguity of customer service robot knowledge base
The present application claims the filing of the divisional application of the chinese patent office, application number 201810801678.8, and the name of the "disambiguation detection method for customer service robot knowledge base" at 2018, month 07 and day 19.
Technical Field
The application relates to the technical field of artificial intelligence, in particular to a method and a device for detecting ambiguity of a customer service robot knowledge base and related equipment.
Background
With the increase of internet users, service pressure of customer service departments of enterprises is continuously increasing. Since most of the questions encountered by the user are recurring, these recurring questions can often be answered with a fixed template. In order to reduce the labor cost of the customer service center, robot customer service can be introduced, a program is used for judging the problem type of a user, if the problem belongs to FAQ (Frequently Asked Questions, frequently asked problems), a standard answer is directly given, and otherwise, the manual service is switched to perform special intervention.
In the related art, a customer service robot recognizes user intention by using a machine learning technology, and converts the intention recognition into question classification. Each FAQ corresponds to a category, and each category has more than one sentence of similar questions. All FAQs and corresponding similar questions form the robot's knowledge base.
The effect of the machine learning model often depends on training data selected from a knowledge base, and particularly the labeling accuracy of the training data can have a great influence on the model effect. However, due to limitations of time and labor, a great deal of ambiguity exists in the knowledge base, such as that questions correspond to wrong categories, category and category semantics coincide, and the like, and the ambiguity can lead the model to learn wrong knowledge, so that the accuracy of the model is negatively affected.
Disclosure of Invention
In order to overcome the problems existing in the related art to at least a certain extent, the application provides a method, a device and related equipment for detecting ambiguity of a customer service robot knowledge base.
In a first aspect, the present application provides a method for detecting ambiguity in a knowledge base of a customer service robot, including:
constructing a knowledge base, wherein the knowledge base is divided according to FAQs, each FAQ is provided with at least one similar question, and each FAQ is of a category;
dividing the knowledge base into a test set and a training set of a deep learning model;
training a deep learning model on a training set, and performing ambiguity detection by using the learned deep learning model;
updating the knowledge base according to the ambiguity detection result;
repeating the steps until the learning effect is not improved, and obtaining the disambiguated knowledge base.
Further, the dividing the knowledge base into a test set and a training set of the deep learning model includes: randomly extracting a preset number of similar questions corresponding to each FAQ as test data of the corresponding FAQ category, and taking the rest similar questions as training data of the corresponding FAQ category. All kinds of test data constitute a test set, and all kinds of training data constitute a training set.
Further, the deep learning model includes: feature extractor, shallow classifier, training the deep learning model on the training set, including:
inputting questions in the training set FAQ as input parts to the deep learning model;
converting the question in the input part into a feature vector by using a feature extractor in the deep learning model;
calculating a prediction result according to the characteristic vector by using a shallow classifier in the deep learning model, wherein the prediction result is a category corresponding to a question in an input part;
optimizing a training model by using an optimizer, and minimizing the average difference between the actual category of question marks in the training set and the prediction result of the deep learning model;
and evaluating the trained model by using the test set, and calculating the consistency ratio of the model prediction result and the actual category marked by the question in the test set to be used as the evaluation of the model learning effect.
Further, the ambiguity detection includes: category ambiguity detection, annotation error detection and annotation ambiguity detection, wherein the ambiguity detection by using the learned deep learning model comprises the following steps:
detecting ambiguity by using a feature extractor in the deep learning model;
and detecting ambiguity by using a shallow classifier in the deep learning model.
Further, the detecting ambiguity by using the feature extractor in the deep learning model includes:
converting similar questions in a data set into feature vectors by using a feature extractor in the deep learning model, wherein the data set comprises a training set or/and a testing set;
combining the feature vectors corresponding to the questions into question feature vector pairs (x, y), wherein the questions corresponding to the feature vectors x and the questions corresponding to the feature vectors y are respectively from different categories;
calculating vector similarity cos (x, y) of each set of question feature vector pairs, the
Figure BDA0003126963990000031
And sorting all question feature vector pairs according to the vector similarity from high to low, selecting the question feature vector pair with the top vector similarity, and judging whether ambiguity exists according to the question feature vector pair with the top vector similarity.
Further, the determining whether there is ambiguity according to the question feature vector pair with the top rank of the vector similarity includes:
judging whether a labeling ambiguity or a labeling error exists or not: extracting a first preset number of question feature vector pairs with top similarity ranking, and manually checking whether corresponding question pairs have mislabeling and mislabeling;
judging whether category ambiguity exists: and counting the repeated occurrence times of the corresponding category pairs for the first preset number of question feature vector pairs, sorting the corresponding category pairs from high to low according to the occurrence times, and taking the second preset number of category pairs to manually check whether category ambiguity exists.
Further, the detecting ambiguity by using the deep learning model shallow classifier includes:
counting the classification results of the deep learning model and forming a confusion matrix, wherein each row i of the confusion matrix corresponds to the labeled category, each column j corresponds to the category predicted by the deep learning model, and the element x is the element x ij Is the number of questions marked as category i and the model predicts as category j, element x ji The number of questions marked as category j and predicted as category i by the model;
calculating the number of samples marked as class i in the data set, wherein the number of samples of the class i is
Figure BDA0003126963990000041
Wherein k is any class;
calculating the number of samples marked as a class j in the data set, wherein the number of samples of the class j is
Figure BDA0003126963990000042
Wherein k is any class;
calculating the proportion P of samples marked as class i in the data set predicted as class j by the deep learning model ij Ratio P to prediction of samples labeled Category j to Category i ji The P is ij And P ji The calculation formulas are respectively as follows:
Figure BDA0003126963990000043
the category i and the category j belong to different categories, and the data set comprises a training set or/and a testing set;
calculating the confusion degree of the category pair (category i, category j), wherein the confusion degree is P ij And P ji Is the harmonic mean S of (2) ij The said
Figure BDA0003126963990000044
And judging whether the category i and the category j have ambiguity according to the confusion degree.
Further, the determining whether the category i and the category j have ambiguity according to the confusion degree includes:
sorting the calculated confusion;
and extracting a third preset number of categories with the top confusion degree rank, and manually detecting whether category ambiguity exists.
Further, the detecting ambiguity by using the deep learning model shallow classifier further includes: and (3) finding out data with inconsistent actual categories marked in the data set and categories predicted by the deep learning model, and manually checking whether marking errors exist or not, wherein the data set comprises a training set or/and a testing set.
Further, the updating the knowledge base according to the ambiguity detection result includes:
manually rewriting and manually remarking the detected ambiguous question, and deleting the original annotation;
and (3) recombining and distributing similar questions of the detected ambiguity categories, and deleting the original ambiguity categories.
In a second aspect, the present application provides a customer service robot knowledge base ambiguity detection apparatus, including:
the construction module is used for constructing a knowledge base, the knowledge base is divided according to FAQs, each FAQ is provided with at least one similar question sentence, and each FAQ is of a category;
the division module is used for dividing the knowledge base into a test set and a training set of the deep learning model;
the training module is used for training a deep learning model on the training set and carrying out ambiguity detection by utilizing the learned deep learning model; the ambiguity detection includes: category ambiguity detection, annotation error detection and annotation ambiguity detection, said learning using said deep learning modelLine ambiguity detection, comprising: detecting ambiguity using a shallow classifier in a deep learning model, comprising: counting the classification results of the deep learning model and forming a confusion matrix, wherein each row i of the confusion matrix corresponds to the labeled category, each column j corresponds to the category predicted by the deep learning model, and the element x is the element x ij Is the number of questions marked as category i and the model predicts as category j, element x ji The number of questions marked as category j and predicted as category i by the model; calculating the number of samples marked as class i in the data set, wherein the number of samples of the class i is
Figure BDA0003126963990000051
Wherein k is any class; calculating the number of samples marked as class j in the dataset, wherein the number of samples of the class j is +.>
Figure BDA0003126963990000052
Wherein k is any class; calculating the proportion P of samples marked as class i in the data set predicted as class j by the deep learning model ij Ratio P to prediction of samples labeled Category j to Category i ji The P is ij And P ji The calculation formulas are respectively as follows: />
Figure BDA0003126963990000053
The category i and the category j belong to different categories, and the data set comprises a training set or/and a testing set; calculating the confusion degree of the category pair (category i, category j), wherein the confusion degree is P ij And P ji Is the harmonic mean S of (2) ij Said->
Figure BDA0003126963990000054
Judging whether ambiguity exists between the category i and the category j according to the confusion degree;
the updating module is used for updating the knowledge base according to the ambiguity detection result;
and the repeating module is used for repeating the steps until the learning effect is not improved any more, and obtaining a disambiguated knowledge base.
In a third aspect, the present application provides an electronic device, comprising:
at least one memory for storing a program;
at least one processor for loading the program to perform the customer service robot knowledge base ambiguity detection method of any one of the first aspects.
In a fourth aspect, the present application provides a computer-readable storage medium having stored therein a program executable by a processor, comprising:
the processor executable program when executed by a processor is for performing the customer service robot knowledge base ambiguity detection method of any one of the first aspects.
The technical scheme provided by the embodiment of the application can comprise the following beneficial effects:
according to the method, the knowledge base is updated according to the ambiguity detection result, the training steps are repeated until the learning effect reaches the expected standard, the ambiguity of the knowledge base can be found and corrected manually, the disambiguated knowledge base is obtained, data are extracted from the disambiguated knowledge base to serve as a training set and a testing set of the deep learning model, and the learning effect of the deep learning model is further improved.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and together with the description, serve to explain the principles of the application.
Fig. 1 is a flowchart of a method for detecting ambiguity in a knowledge base of a customer service robot according to an embodiment of the present application.
Fig. 2 is a flowchart of a method for detecting ambiguity of a customer service robot knowledge base according to another embodiment of the present application.
Detailed Description
The present invention will be described in detail with reference to the accompanying drawings and examples.
Fig. 1 is a flowchart of a method for detecting ambiguity in a knowledge base of a customer service robot according to an embodiment of the present application.
As shown in fig. 1, the method of the present embodiment includes:
s11: constructing a knowledge base, dividing the knowledge base according to FAQs, wherein each FAQ is provided with an indefinite number of similar questions, and each FAQ is a category.
The knowledge base is developed on the basis of large-scale knowledge processing and is applied to technical industries of large-scale knowledge processing, natural language understanding, knowledge management, automatic question-answering systems, reasoning and the like, and intelligent customer service not only provides fine-grained knowledge management technology for enterprises, but also establishes a quick and effective technical means based on natural language for communication between the enterprises and massive users. Taking a customer service robot knowledge base of an e-commerce enterprise as an example, the knowledge base includes a plurality of FAQs, such as a "refund process" and a "refund process". Taking the "return flow" as an example, the FAQ may include the following similar questions: "what did i buy yesterday return? "how should me operate to return goods? ".
S12: the knowledge base is divided into a test set and a training set of the deep learning model.
And selecting N FAQs needing to detect ambiguity from the knowledge base as N categories. For each FAQ, randomly extracting a preset number of similar questions as test data of the category, and taking the rest similar questions as training data of the category. All kinds of test data constitute a test set, and all kinds of training data constitute a training set.
For example, the knowledge base includes 10 FAQs, each FAQ includes 20 similar questions, a preset amount, for example, 3 similar questions, are randomly extracted from each category of the knowledge base as a test set of the deep learning model, then 30 similar questions are included in the test set, and the remaining 170 similar questions are included in a training set of the deep learning model.
It should be noted that, the number of categories included in the knowledge base and the number of similar questions included in each category are not limited to the examples in the embodiments, and are not repeated here.
S13: training a deep learning model on the training set, and performing ambiguity detection by using the learned deep learning model.
The deep learning model includes: feature extractor, shallow classifier.
The ambiguity detection includes: category ambiguity detection, annotation error detection and annotation ambiguity detection;
the ambiguity includes:
category ambiguity: that is, the meanings of the two categories are very similar, for example, category 1 is an "order problem", category 2 is a "change cancel problem of product", the semantics of category 1 and the semantics of category 2 are coincident, because category 1 can basically cover category 2;
disambiguation: i.e. questions may be given the same time scale as classes, for example: category 1 is "return problem of product", category 2 is "price problem of product", if the question is "this thing is too expensive, i want to return", there is a disambiguation of this sentence, because the question contains the meanings of both categories;
labeling errors: the question corresponds to the wrong category, for example, category 1 is "return problem of product", category 2 is "price problem of product", and if the question is "I do not want to" but is labeled as category 2, a labeling error will occur.
The ambiguity detection is for a test set or/and training set.
The performing ambiguity detection by using the learned deep learning model comprises:
detecting ambiguity by using a feature extractor in the deep learning model;
detecting ambiguity by using a shallow classifier in the deep learning model;
the detecting ambiguity using the feature extractor in the deep learning model includes:
converting similar questions in a data set into feature vectors by using a feature extractor in the deep learning model, wherein the data set comprises a training set or/and a testing set;
combining the feature vectors corresponding to the questions into question feature vector pairs (x, y), wherein the questions corresponding to the feature vectors x and the questions corresponding to the feature vectors y are respectively from different categories;
calculating vector similarity cos (x, y) of each set of question feature vector pairs, the
Figure BDA0003126963990000081
And sorting all question feature vector pairs according to the vector similarity from high to low, selecting the question feature vector pair with the top vector similarity, and judging whether ambiguity exists according to the question feature vector pair with the top vector similarity.
The judging whether ambiguity exists according to the question feature vector pair with the top rank of the vector similarity comprises the following steps:
judging whether a labeling ambiguity or a labeling error exists or not: extracting first preset number of question feature vector pairs with the top similarity ranking of 30, and manually checking whether corresponding question pairs have mislabeling and mislabeling;
judging whether category ambiguity exists: and counting the repeated occurrence times of the corresponding category pairs for the first preset number of question feature vector pairs, sorting the corresponding category pairs from high to low according to the occurrence times, taking a second preset number of category pairs, for example 20, and manually checking whether category ambiguity exists.
The detecting ambiguity using the deep learning model shallow classifier includes:
counting the classification results of the deep learning model and forming a confusion matrix, wherein each row i of the confusion matrix corresponds to the labeled category, each column j corresponds to the category predicted by the deep learning model, and the element x is the element x ij Is the number of questions marked as category i and the model predicts as category j, element x ji The number of questions marked as category j and predicted as category i by the model;
calculating the number of samples marked as class i in the data set, wherein the number of samples of the class i is
Figure BDA0003126963990000091
Wherein k is any of the classes；
Calculating the number of samples marked as a class j in the data set, wherein the number of samples of the class j is
Figure BDA0003126963990000092
Wherein k is any class;
calculating the proportion P of samples marked as class i in the data set predicted as class j by the deep learning model ij Ratio P to prediction of samples labeled Category j to Category i ji The P is ij And P ji The calculation formulas are respectively as follows:
Figure BDA0003126963990000093
the category i and the category j belong to different categories, and the data set comprises a training set or/and a testing set;
calculating the confusion degree of the category pair (category i, category j), wherein the confusion degree is P ij And P ji Is the harmonic mean S of (2) ij The said
Figure BDA0003126963990000094
And judging whether the category i and the category j have ambiguity according to the confusion degree.
Judging whether the category i and the category j have ambiguity according to the confusion degree comprises the following steps:
sorting the calculated confusion;
extracting a third preset number of category pairs with the top ranking of 5 confusion degrees, and manually detecting whether category ambiguity exists.
The method for detecting ambiguity by using the deep learning model shallow classifier further comprises the following steps: and (3) finding out data with inconsistent actual categories marked in the data set and categories predicted by the deep learning model, and manually checking whether marking errors exist or not, wherein the data set comprises a training set or/and a testing set.
S14: updating the knowledge base according to the ambiguity detection result, including:
manually rewriting and manually remarking the detected ambiguous question, and deleting the original annotation;
and (3) recombining and distributing similar questions of the detected ambiguity categories, and deleting the original ambiguity categories.
S15: repeating the steps until the learning effect is not improved, and obtaining the disambiguated knowledge base.
The learning effect is the consistency rate of the model prediction result and the actual category marked by the question in the test set, and the consistency rate is, for example, the prediction accuracy rate, that is, the number of questions with consistent prediction result divided by the total number of questions. The learning effect is not improved any more, for example, the prediction accuracy is improved by less than 0.5%.
When the model learning effect is not improved any more, the model performance degradation caused by knowledge base ambiguity is eliminated, and the knowledge base can be used for training the model and deployed into a production environment for use.
In this embodiment, the knowledge base is updated according to the ambiguity detection result, and the training steps are repeated until the learning effect reaches the expected standard, so that the knowledge base ambiguity can be found and corrected manually, a disambiguated knowledge base is obtained, and data is extracted from the disambiguated knowledge base as a training set and a testing set of the deep learning model, so that the learning effect of the deep learning model is further improved.
Fig. 2 is a flowchart of a method for detecting ambiguity of a customer service robot knowledge base according to another embodiment of the present application.
As shown in fig. 2, the training of the deep learning model on the training set includes:
the deep learning concept is derived from the research of an artificial neural network and comprises a multi-layer perceptron with multiple hidden layers. Deep learning forms more abstract high-level representation attribute categories or features by combining low-level features to discover distributed feature representations of data. The deep learning model includes a feature extractor and a shallow classifier.
S21: inputting questions in the training set as input parts to the deep learning model;
s22: converting the question in the input part into a feature vector by using a feature extractor in the deep learning model;
the feature extractor is, for example, a recurrent neural network. The model reads in each word in the question sequentially and outputs a feature vector of a fixed dimension. It should be noted that the feature extractor is not limited to the cyclic neural network illustrated, and any method that can convert a question into a feature vector with a fixed dimension may be used as the feature extractor.
S23: calculating a prediction result according to the characteristic vector by using a shallow classifier in the deep learning model, wherein the prediction result is a category corresponding to a question in an input part;
the shallow classifier is, for example, a linear classifier. The classifier reads in a feature vector with fixed dimension, calculates the linear combination of vector elements to obtain the scoring of each category, and takes the category with the highest scoring as the prediction result. It should be noted that the shallow classifier is not limited to the linear classifier, and any method that can convert the feature vector with a fixed dimension into the score of each class may be used as the shallow classifier.
S24: optimizing a training model by using an optimizer, and minimizing the average difference between the actual category of question marks in the training set and the prediction result of the deep learning model;
the average difference is for example a loss function. The loss function is, for example, cross entropy.
The optimizer is, for example, a gradient descent method. The Gradient Descent is one of the iterative methods, and when solving model parameters of a machine learning algorithm, i.e. unconstrained optimization problems, gradient Descent (Gradient device) is one of the most commonly adopted methods. When the minimum value of the loss function is solved, the minimum loss function and corresponding model parameter values can be obtained through one-step iterative solution by a gradient descent method.
S25: and evaluating the trained model by using a test set, and calculating the consistency rate of the model prediction result and the actual category marked by the questions in the test set, wherein the consistency rate is used for evaluating the model learning effect, and is the prediction accuracy rate, i.e. the number of questions with consistent prediction result divided by the total number of questions.
In this embodiment, the deep learning model is used to train the FAQ in the training set, and the optimizer is used to continuously optimize the model during the training process, so as to continuously iterate and improve the learning effect of the deep learning model, and continuously improve the ambiguity detection accuracy.
An embodiment of the present application provides a customer service robot knowledge base ambiguity detection device, including:
the construction module is used for constructing a knowledge base, the knowledge base is divided according to FAQs, each FAQ is provided with at least one similar question sentence, and each FAQ is of a category;
the division module is used for dividing the knowledge base into a test set and a training set of the deep learning model;
the training module is used for training a deep learning model on the training set and carrying out ambiguity detection by utilizing the learned deep learning model; the ambiguity detection includes: category ambiguity detection, annotation error detection and annotation ambiguity detection, wherein the ambiguity detection by using the learned deep learning model comprises the following steps: detecting ambiguity using a shallow classifier in a deep learning model, comprising: counting the classification results of the deep learning model and forming a confusion matrix, wherein each row i of the confusion matrix corresponds to the labeled category, each column j corresponds to the category predicted by the deep learning model, and the element x is the element x ij Is the number of questions marked as category i and the model predicts as category j, element x ji The number of questions marked as category j and predicted as category i by the model; calculating the number of samples marked as class i in the data set, wherein the number of samples of the class i is
Figure BDA0003126963990000121
Wherein k is any class; calculating the number of samples marked as class j in the dataset, wherein the number of samples of the class j is +.>
Figure BDA0003126963990000122
Wherein k is any class; calculating the proportion P of samples marked as class i in the data set predicted as class j by the deep learning model ij Ratio to predict samples labeled category j to category iExample P ji The P is ij And P ji The calculation formulas are respectively as follows: />
Figure BDA0003126963990000123
The category i and the category j belong to different categories, and the data set comprises a training set or/and a testing set; calculating the confusion degree of the category pair (category i, category j), wherein the confusion degree is P ij And P ji Is the harmonic mean S of (2) ij Said->
Figure BDA0003126963990000124
Judging whether ambiguity exists between the category i and the category j according to the confusion degree;
the updating module is used for updating the knowledge base according to the ambiguity detection result;
and the repeating module is used for repeating the steps until the learning effect is not improved any more, and obtaining a disambiguated knowledge base.
In some embodiments, further comprising:
the random extraction module is used for dividing the knowledge base into a test set and a training set of the deep learning model, and comprises the following steps: randomly extracting a preset number of similar questions corresponding to each FAQ as test data of the corresponding FAQ category, and taking the rest similar questions as training data of the corresponding FAQ category; all kinds of test data constitute a test set, and all kinds of training data constitute a training set.
The sorting module is used for sorting the calculated confusion, extracting the category with the highest ranking of the third preset number of the confusion, and manually detecting whether category ambiguity exists.
The labeling module is used for detecting ambiguity by using the deep learning model shallow classifier, and further comprises: and (3) finding out data with inconsistent actual categories marked in the data set and categories predicted by the deep learning model, and manually checking whether marking errors exist or not, wherein the data set comprises a training set or/and a testing set.
One embodiment of the present application provides an electronic device, including:
at least one memory for storing a program;
at least one processor for loading the program to perform the customer service robot knowledge base ambiguity detection method as described in the above embodiments.
One embodiment of the present application provides a computer-readable storage medium having stored therein a program executable by a processor, comprising:
the processor executable program when executed by the processor is for performing the customer service robot knowledge base ambiguity detection method as described in the above embodiments.
It is to be understood that the same or similar parts in the above embodiments may be referred to each other, and that in some embodiments, the same or similar parts in other embodiments may be referred to.
It should be noted that in the description of the present application, the terms "first," "second," and the like are used for descriptive purposes only and are not to be construed as indicating or implying relative importance. Furthermore, in the description of the present application, unless otherwise indicated, the meaning of "plurality" means at least two.
Any process or method descriptions in flow charts or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps of the process, and further implementations are included within the scope of the preferred embodiment of the present application in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art of the embodiments of the present application.
It is to be understood that portions of the present application may be implemented in hardware, software, firmware, or a combination thereof. In the above-described embodiments, the various steps or methods may be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, may be implemented using any one or combination of the following techniques, as is well known in the art: discrete logic circuits having logic gates for implementing logic functions on data signals, application specific integrated circuits having suitable combinational logic gates, programmable Gate Arrays (PGAs), field Programmable Gate Arrays (FPGAs), and the like.
Those of ordinary skill in the art will appreciate that all or a portion of the steps carried out in the method of the above-described embodiments may be implemented by a program to instruct related hardware, where the program may be stored in a computer readable storage medium, and where the program, when executed, includes one or a combination of the steps of the method embodiments.
In addition, each functional unit in each embodiment of the present application may be integrated in one processing module, or each unit may exist alone physically, or two or more units may be integrated in one module. The integrated modules may be implemented in hardware or in software functional modules. The integrated modules may also be stored in a computer readable storage medium if implemented in the form of software functional modules and sold or used as a stand-alone product.
The above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, or the like.
In the description of the present specification, a description referring to terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present application. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiments or examples. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
Although embodiments of the present application have been shown and described above, it will be understood that the above embodiments are illustrative and not to be construed as limiting the application, and that variations, modifications, alternatives, and variations may be made to the above embodiments by one of ordinary skill in the art within the scope of the application.
It should be noted that the present invention is not limited to the above-mentioned preferred embodiments, and those skilled in the art can obtain other products in various forms without departing from the scope of the present invention, however, any changes in shape or structure of the present invention, and all technical solutions that are the same or similar to the present application, fall within the scope of the present invention.

Claims (8)

1. The method for detecting the ambiguity of the customer service robot knowledge base is characterized by comprising the following steps of:
constructing a knowledge base, wherein the knowledge base is divided according to FAQs, each FAQ is provided with at least one similar question, and each FAQ is of a category;
dividing the knowledge base into a test set and a training set of a deep learning model;
training a deep learning model on a training set, and performing ambiguity detection by using the learned deep learning model; the ambiguity detection includes: category ambiguity detection, annotation error detection and annotation ambiguity detection, wherein the ambiguity detection by using the learned deep learning model comprises the following steps: detecting ambiguity using a shallow classifier in a deep learning model, comprising:
counting the classification results of the deep learning model and forming a confusion matrix, wherein each row i of the confusion matrix corresponds to the labeled category, each column j corresponds to the category predicted by the deep learning model, and elements
Figure QLYQS_1
Is marked as category i, and model predicts as the number of questions of category j, element +.>
Figure QLYQS_2
The number of questions marked as category j and predicted as category i by the model;
calculating the number of samples marked as class i in the data set, wherein the number of samples of the class i is
Figure QLYQS_3
Wherein k is any class;
computing a data setThe number of samples marked as category j in the list is that
Figure QLYQS_4
Wherein k is any class;
calculating the proportion of samples marked as class i in the dataset predicted by the deep learning model as class j
Figure QLYQS_5
Proportion to predicting the sample labeled class j to class i +.>
Figure QLYQS_6
Said->
Figure QLYQS_7
And->
Figure QLYQS_8
The calculation formulas are respectively as follows: />
Figure QLYQS_9
、
Figure QLYQS_10
The category i and the category j belong to different categories, and the data set comprises a training set or/and a testing set;
calculating the confusion degree of the category pair (category i, category j), wherein the confusion degree is that
Figure QLYQS_11
And->
Figure QLYQS_12
Is>
Figure QLYQS_13
The said
Figure QLYQS_14
；
Judging whether ambiguity exists between the category i and the category j according to the confusion degree; wherein the deep learning model includes: feature extractor, shallow classifier, training the deep learning model on the training set, including:
inputting questions in the training set as input parts to the deep learning model;
converting the question in the input part into a feature vector by using a feature extractor in the deep learning model;
calculating a prediction result according to the characteristic vector by using a shallow classifier in the deep learning model, wherein the prediction result is a category corresponding to a question in an input part;
optimizing a training model by using an optimizer, and minimizing the average difference between the actual category of question marks in the training set and the prediction result of the deep learning model;
and evaluating the trained model by using the test set, and calculating the consistency ratio of the model prediction result and the actual category marked by the question in the test set to be used as the evaluation of the model learning effect.
Updating the knowledge base according to the ambiguity detection result;
repeating the steps until the learning effect is not improved, and obtaining the disambiguated knowledge base.
2. The method of claim 1, wherein the partitioning the knowledge base into a test set and a training set of deep learning models comprises: randomly extracting a preset number of similar questions corresponding to each FAQ as test data of the corresponding FAQ category, and taking the rest similar questions as training data of the corresponding FAQ category; all kinds of test data constitute a test set, and all kinds of training data constitute a training set.
3. The method of claim 1, wherein determining whether there is ambiguity in category i, category j based on the confusion, comprises:
sorting the calculated confusion;
and extracting a third preset number of categories with the top confusion degree rank, and manually detecting whether category ambiguity exists.
4. The method of claim 1, wherein the detecting ambiguity using a deep learning model shallow classifier further comprises: and (3) finding out data with inconsistent actual categories marked in the data set and categories predicted by the deep learning model, and manually checking whether marking errors exist or not, wherein the data set comprises a training set or/and a testing set.
5. The method of claim 1, wherein updating the knowledge base based on the ambiguity detection results comprises:
manually rewriting and manually remarking the detected ambiguous question, and deleting the original annotation;
and (3) recombining and distributing similar questions of the detected ambiguity categories, and deleting the original ambiguity categories.
6. The utility model provides a customer service robot knowledge base ambiguity detection device which characterized in that includes:
the construction module is used for constructing a knowledge base, the knowledge base is divided according to FAQs, each FAQ is provided with at least one similar question sentence, and each FAQ is of a category;
the division module is used for dividing the knowledge base into a test set and a training set of the deep learning model;
the training module is used for training a deep learning model on the training set and carrying out ambiguity detection by utilizing the learned deep learning model; the ambiguity detection includes: category ambiguity detection, annotation error detection and annotation ambiguity detection, wherein the ambiguity detection by using the learned deep learning model comprises the following steps: detecting ambiguity using a shallow classifier in a deep learning model, comprising: counting the classification results of the deep learning model and forming a confusion matrix, wherein each row i of the confusion matrix corresponds to the labeled category, each column j corresponds to the category predicted by the deep learning model, and elements
Figure QLYQS_17
Is marked as category i, and model predicts as the number of questions of category j, element +.>
Figure QLYQS_19
The number of questions marked as category j and predicted as category i by the model; calculating the number of samples marked as class i in the dataset, wherein the number of samples of the class i is +.>
Figure QLYQS_21
Wherein k is any class; calculating the number of samples marked as class j in the dataset, wherein the number of samples of the class j is +.>
Figure QLYQS_16
Wherein k is any class; calculating the proportion of samples marked as class i in the dataset predicted by the deep learning model as class j>
Figure QLYQS_24
Proportion to predicting the sample labeled class j to class i +.>
Figure QLYQS_26
Said->
Figure QLYQS_28
And->
Figure QLYQS_15
The calculation formulas are respectively as follows: />
Figure QLYQS_20
、/>
Figure QLYQS_22
The category i and the category j belong to different categories, and the data set comprises a training set or/and a testing set; calculating the confusion of class pairs (class i, class j), said confusion being +.>
Figure QLYQS_23
And->
Figure QLYQS_18
Is>
Figure QLYQS_25
Said->
Figure QLYQS_27
The method comprises the steps of carrying out a first treatment on the surface of the Judging whether ambiguity exists between the category i and the category j according to the confusion degree; wherein the deep learning model includes: feature extractor, shallow classifier, training the deep learning model on the training set, including: />
Inputting questions in the training set as input parts to the deep learning model;
converting the question in the input part into a feature vector by using a feature extractor in the deep learning model;
calculating a prediction result according to the characteristic vector by using a shallow classifier in the deep learning model, wherein the prediction result is a category corresponding to a question in an input part;
optimizing a training model by using an optimizer, and minimizing the average difference between the actual category of question marks in the training set and the prediction result of the deep learning model;
and evaluating the trained model by using the test set, and calculating the consistency ratio of the model prediction result and the actual category marked by the question in the test set to be used as the evaluation of the model learning effect.
The updating module is used for updating the knowledge base according to the ambiguity detection result;
and the repeating module is used for repeating the steps until the learning effect is not improved any more, and obtaining a disambiguated knowledge base.
7. An electronic device, comprising:
at least one memory for storing a program;
at least one processor for loading the program to perform the customer service robot knowledge base ambiguity detection method of any one of claims 1-5.
8. A computer-readable storage medium having stored therein a program executable by a processor, comprising:
the processor executable program when executed by a processor is for performing the customer service robot knowledge base ambiguity detection method of any one of claims 1-5.
CN202110693227.9A 2018-07-19 2018-07-19 Method, device and related equipment for detecting ambiguity of customer service robot knowledge base Active CN113407694B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110693227.9A CN113407694B (en) 2018-07-19 2018-07-19 Method, device and related equipment for detecting ambiguity of customer service robot knowledge base

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202110693227.9A CN113407694B (en) 2018-07-19 2018-07-19 Method, device and related equipment for detecting ambiguity of customer service robot knowledge base
CN201810801678.8A CN109101579B (en) 2018-07-19 2018-07-19 Customer service robot knowledge base ambiguity detection method

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
CN201810801678.8A Division CN109101579B (en) 2018-07-09 2018-07-19 Customer service robot knowledge base ambiguity detection method

Publications (2)

Publication Number Publication Date
CN113407694A CN113407694A (en) 2021-09-17
CN113407694B true CN113407694B (en) 2023-06-02

Family

ID=64846947

Family Applications (2)

Application Number Title Priority Date Filing Date
CN201810801678.8A Active CN109101579B (en) 2018-07-09 2018-07-19 Customer service robot knowledge base ambiguity detection method
CN202110693227.9A Active CN113407694B (en) 2018-07-19 2018-07-19 Method, device and related equipment for detecting ambiguity of customer service robot knowledge base

Family Applications Before (1)

Application Number Title Priority Date Filing Date
CN201810801678.8A Active CN109101579B (en) 2018-07-09 2018-07-19 Customer service robot knowledge base ambiguity detection method

Country Status (1)

Country Link
CN (2) CN109101579B (en)

Families Citing this family (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2020010930A1 (en) * 2018-07-09 2020-01-16 深圳追一科技有限公司 Method for detecting ambiguity of customer service robot knowledge base, storage medium, and computer device
CN109902285B (en) * 2019-01-08 2023-09-22 平安科技（深圳）有限公司 Corpus classification method, corpus classification device, computer equipment and storage medium
CN112818127A (en) * 2019-11-15 2021-05-18 北京中关村科金技术有限公司 Method, device and medium for detecting corpus conflict in knowledge base
CN111209404B (en) * 2020-04-17 2020-12-22 浙江百应科技有限公司 Method for generating similar question sentences based on deep learning assistance
CN111625636B (en) * 2020-05-28 2023-08-04 深圳追一科技有限公司 Method, device, equipment and medium for rejecting man-machine conversation
CN112257443B (en) * 2020-09-30 2024-04-02 华泰证券股份有限公司 MRC-based company entity disambiguation method combined with knowledge base
CN112559723B (en) * 2020-12-28 2024-05-28 广东国粒教育技术有限公司 FAQ search type question-answering construction method and system based on deep learning
CN112699226B (en) * 2020-12-29 2024-09-20 江苏苏宁云计算有限公司 Method and system for semantic confusion detection
CN117114103A (en) * 2023-10-20 2023-11-24 国家电网有限公司 Corpus reconstruction method and device

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2014025135A1 (en) * 2012-08-10 2014-02-13 에스케이텔레콤 주식회사 Method for detecting grammatical errors, error detecting apparatus for same, and computer-readable recording medium having the method recorded thereon

Family Cites Families (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7392185B2 (en) * 1999-11-12 2008-06-24 Phoenix Solutions, Inc. Speech based learning/training system using semantic decoding
CN1979638A (en) * 2005-12-02 2007-06-13 中国科学院自动化研究所 Method for correcting error of voice identification result
CN101373532A (en) * 2008-07-10 2009-02-25 昆明理工大学 FAQ Chinese request-answering system implementing method in tourism field
CN103617157B (en) * 2013-12-10 2016-08-17 东北师范大学 Based on semantic Text similarity computing method
CN104268134B (en) * 2014-09-28 2017-04-19 苏州大学 Subjective and objective classifier building method and system
WO2016051228A1 (en) * 2014-09-30 2016-04-07 Umm-Al-Qura University A method and system for an accurate and energy efficient vehicle lane detection
US10509814B2 (en) * 2014-12-19 2019-12-17 Universidad Nacional De Educacion A Distancia (Uned) System and method for the indexing and retrieval of semantically annotated data using an ontology-based information retrieval model
CN104657463B (en) * 2015-02-10 2018-04-27 乐娟 Question Classification method and device applied to automatically request-answering system
CN105512209B (en) * 2015-11-28 2018-06-19 大连理工大学 The biomedical event trigger word recognition methods that a kind of feature based learns automatically
CN107102989B (en) * 2017-05-24 2020-09-29 南京大学 Entity disambiguation method based on word vector and convolutional neural network
CN107292338A (en) * 2017-06-14 2017-10-24 大连海事大学 A kind of feature selection approach based on sample characteristics Distribution value degree of aliasing
CN107301246A (en) * 2017-07-14 2017-10-27 河北工业大学 Chinese Text Categorization based on ultra-deep convolutional neural networks structural model
CN107993724B (en) * 2017-11-09 2020-11-13 易保互联医疗信息科技（北京）有限公司 Medical intelligent question and answer data processing method and device
CN107977356B (en) * 2017-11-21 2019-10-25 新疆科大讯飞信息科技有限责任公司 Method and device for correcting recognized text
CN108227932B (en) * 2018-01-26 2020-06-23 上海智臻智能网络科技股份有限公司 Interaction intention determination method and device, computer equipment and storage medium

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2014025135A1 (en) * 2012-08-10 2014-02-13 에스케이텔레콤 주식회사 Method for detecting grammatical errors, error detecting apparatus for same, and computer-readable recording medium having the method recorded thereon

Also Published As

Publication number Publication date
CN109101579B (en) 2021-11-23
CN113407694A (en) 2021-09-17
CN109101579A (en) 2018-12-28

Similar Documents

Publication Publication Date Title
CN113407694B (en) Method, device and related equipment for detecting ambiguity of customer service robot knowledge base
US11036811B2 (en) Categorical data transformation and clustering for machine learning using data repository systems
Ahmed et al. Clarify of the random forest algorithm in an educational field
CN110442516B (en) Information processing method, apparatus, and computer-readable storage medium
US12086548B2 (en) Event extraction from documents with co-reference
US20220100772A1 (en) Context-sensitive linking of entities to private databases
CN110019822B (en) Few-sample relation classification method and system
US20220100967A1 (en) Lifecycle management for customized natural language processing
CN110275928B (en) Iterative entity relation extraction method
CN110134777A (en) Problem De-weight method, device, electronic equipment and computer readable storage medium
CN114936158A (en) Software defect positioning method based on graph convolution neural network
US11531927B2 (en) Categorical data transformation and clustering for machine learning using natural language processing
CN112420125A (en) Molecular attribute prediction method and device, intelligent equipment and terminal
EP4222635A1 (en) Lifecycle management for customized natural language processing
CN113705159A (en) Merchant name labeling method, device, equipment and storage medium
CN113505590A (en) Method and system for identifying key data elements in database based on entity disambiguation
WO2024098282A1 (en) Geometric problem-solving method and apparatus, and device and storage medium
CN111339258A (en) University computer basic exercise recommendation method based on knowledge graph
Amirian et al. Data science and analytics
US20230162518A1 (en) Systems for Generating Indications of Relationships between Electronic Documents
CN113010687B (en) Exercise label prediction method and device, storage medium and computer equipment
CN109657710A (en) Data screening method, apparatus, server and storage medium
Barella et al. Simulating complexity measures on imbalanced datasets
CN113627522A (en) Image classification method, device and equipment based on relational network and storage medium
WO2020010930A1 (en) Method for detecting ambiguity of customer service robot knowledge base, storage medium, and computer device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant