CN117076654B - Abnormality detection method and device for dialogue system - Google Patents
Abnormality detection method and device for dialogue system Download PDFInfo
- Publication number
- CN117076654B CN117076654B CN202311344618.5A CN202311344618A CN117076654B CN 117076654 B CN117076654 B CN 117076654B CN 202311344618 A CN202311344618 A CN 202311344618A CN 117076654 B CN117076654 B CN 117076654B
- Authority
- CN
- China
- Prior art keywords
- intention
- detected
- user
- sample
- proportion
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 34
- 230000005856 abnormality Effects 0.000 title claims description 26
- 230000002159 abnormal effect Effects 0.000 claims abstract description 24
- 238000000034 method Methods 0.000 claims abstract description 16
- 230000008569 process Effects 0.000 abstract description 5
- 238000011156 evaluation Methods 0.000 abstract description 2
- 238000002372 labelling Methods 0.000 description 5
- 230000004048 modification Effects 0.000 description 5
- 238000012986 modification Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/33—Querying
- G06F16/332—Query formulation
- G06F16/3329—Natural language query formulation or dialogue systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/08—Logistics, e.g. warehousing, loading or distribution; Inventory or stock management
- G06Q10/083—Shipping
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Business, Economics & Management (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- Artificial Intelligence (AREA)
- Economics (AREA)
- Mathematical Physics (AREA)
- General Engineering & Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Human Computer Interaction (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Databases & Information Systems (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Computational Linguistics (AREA)
- Development Economics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Entrepreneurship & Innovation (AREA)
- Human Resources & Organizations (AREA)
- Marketing (AREA)
- Operations Research (AREA)
- Quality & Reliability (AREA)
- Strategic Management (AREA)
- Tourism & Hospitality (AREA)
- General Business, Economics & Management (AREA)
- Machine Translation (AREA)
Abstract
The invention relates to an anomaly detection method and device for a dialogue system, belonging to the technical field of dialogue system evaluation, wherein the method comprises the following steps: predicting probability values of different user intentions of the input calling text, taking the calling text corresponding to the probability value meeting the threshold condition of the probability value as a sample to be detected, acquiring the corresponding recognition intention and the preset intention to be selected, acquiring a reply preset template corresponding to the sample to be detected based on the intention to be selected, sending the reply preset template to a user terminal, receiving answer information of the user, extracting the true intention of the user based on the answer information, comparing the true intention of the user with the recognition intention corresponding to the corresponding sample to be detected, and judging whether the intention recognition model is abnormal or not according to a comparison result. The method and the device provided by the invention can detect whether the model training/replying has abnormal conditions, and solve the problem that the robot replying speaking operation is irrelevant in the human-computer conversation process, so that the user experience is poor.
Description
Technical Field
The present invention relates to the field of dialog system evaluation technologies, and in particular, to a method and an apparatus for detecting abnormalities in a dialog system.
Background
For anomaly detection in a dialogue system, the prior art mainly has the following problems: firstly, in abnormal data detection, an abnormal data sample is usually determined based on a mode of manually recalling abnormal data and manually marking, more abnormal data samples to be detected are screened based on a training model mode, the labor cost is high, the operation experience and subjective ideas of staff are relied, and the accuracy is low; secondly, an abnormal detection model trained based on a labeling sample and an online model of a dialogue system are fractured, separate training is needed, and because the distribution of abnormal data is often smaller than that of normal data, the sample is unbalanced, and the accuracy of the abnormal detection model is often not guaranteed; in addition, the anomaly detection model trained by using the history sample is effective only for the newly added intention, and the model result is difficult to change by the operations of data enhancement, diversity generalization and the like of the history sample, so that the true intention cannot be confirmed.
Disclosure of Invention
The invention aims to provide an abnormality detection method and device for a dialogue system, which are used for solving the defects in the prior art.
The invention provides an abnormality detection method for a dialogue system, which comprises the following steps:
acquiring dialogue text data in the industry field, marking user intention on the dialogue text data to form a training sample, training the intention recognition model through the training sample, and deploying the trained intention recognition model into a dialogue system;
user intention statistics is carried out on dialogue corpus texts generated in a dialogue system with an intention recognition model deployed, a plurality of intention branches are obtained, and the number of dialogue corpus texts corresponding to each intention branch and the percentage of the number of dialogue corpus texts in the total number are obtained;
when intention branches with the total number of the dialogue corpus texts being smaller than a percentage threshold exist, collecting a plurality of calling texts which are newly input into a dialogue system provided with an intention recognition model, predicting probability values of different user intentions corresponding to the calling texts through the intention recognition model, taking the user intention with the largest probability value as the recognition intention corresponding to the corresponding calling text, and taking the calling text corresponding to the condition of meeting the probability value threshold as a sample to be detected;
obtaining a to-be-selected intention corresponding to a to-be-detected sample, obtaining a reply preset template corresponding to the to-be-detected sample through a dialogue system deployed with an intention recognition model based on the to-be-selected intention, sending the reply preset template to a user terminal, receiving answer information of a user, and extracting the real intention of the user based on the answer information;
comparing the real intention of the user with the recognition intention corresponding to the corresponding sample to be detected, and judging whether the intention recognition model is abnormal or not according to the comparison result.
In the above-described aspect, the plurality of intention branches includes: delivery express, sending express, getting the piece from the upper door, modifying the address of the express, getting the piece from the upper door, and the like.
In the above scheme, the probability value threshold includes a first probability value threshold and a second probability value threshold, and the second probability value threshold is greater than the first probability value threshold.
In the above-mentioned scheme, taking the calling text corresponding to the probability value threshold condition as the sample to be detected includes: and taking the calling text corresponding to the probability value which is larger than the first probability value threshold and smaller than the second probability value threshold as a sample to be detected.
In the above scheme, the obtaining the candidate intention corresponding to the sample to be detected includes:
arranging probability values of samples to be detected corresponding to different user intentions from large to small;
and taking the user intention corresponding to the probability value arranged at the second position as the candidate intention corresponding to the sample to be detected.
In the above-described scheme, extracting the real intention of the user based on the answer information includes:
judging the answer information of the user, wherein the answer of the user is judged to be affirmative or negative;
when the answer of the user is judged to be affirmative, taking the candidate intention corresponding to the sample to be detected as the real intention of the user;
and when the answer of the user is judged to be negative, taking the identification intention corresponding to the sample to be detected as the real intention of the user.
In the above-mentioned scheme, comparing the real intention of the user with the recognition intention corresponding to the corresponding sample to be detected, and judging whether the intention recognition model is abnormal according to the comparison result includes:
when the real intention of the user is consistent with the recognition intention corresponding to the corresponding sample to be detected, storing the corresponding sample to be detected in a first sample set;
when the real intention of the user is inconsistent with the recognition intention corresponding to the corresponding sample to be detected, storing the corresponding sample to be detected in a second sample set;
the method comprises the steps of obtaining the proportion of the number of samples to be detected in a first sample set to the number of all samples to be detected, taking the proportion as a first proportion, or obtaining the proportion of the number of samples to be detected in a second sample set to the number of all samples to be detected, and taking the proportion as a second proportion;
when the first proportion is larger than or equal to the proportion threshold value or the second proportion is smaller than the proportion threshold value, judging that the intention recognition model is normal;
and when the first proportion is smaller than the proportion threshold value or the second proportion is larger than or equal to the proportion threshold value, judging that the model is abnormal.
In the above scheme, the ratio threshold is 80%.
The abnormality detection device for a dialogue system provided by the invention adopts the abnormality detection method for the dialogue system to detect the abnormality of the dialogue system, and comprises the following steps:
the model deployment module is used for acquiring dialogue text data in the industry field, marking user intention on the dialogue text data to form a training sample, training the intention recognition model through the training sample, and deploying the trained intention recognition model into a dialogue system;
the preliminary judgment module is used for carrying out user intention statistics on dialogue corpus texts generated in the dialogue system with the intention recognition model deployed, obtaining a plurality of intention branches, and obtaining the quantity of dialogue corpus texts corresponding to the intention branches and the percentage of the quantity of dialogue corpus texts to the total quantity.
In the above-mentioned scheme, the abnormality detection device for a dialogue system provided by the present invention further includes:
the system comprises a sample acquisition module to be detected, a recognition module and a detection module, wherein the sample acquisition module to be detected is used for acquiring a plurality of calling texts newly input into a dialogue system provided with an intention recognition model when intention branches with the number of dialogue corpus texts accounting for the total number of the intention branches smaller than a percentage threshold exist, predicting probability values of different user intentions corresponding to the calling texts through the intention recognition model, taking the user intention with the largest probability value as the recognition intention corresponding to the corresponding calling text, and taking the calling text corresponding to the condition meeting the probability value threshold as a sample to be detected;
the true intention acquisition module is used for acquiring the intention to be selected corresponding to the sample to be detected, acquiring a reply preset template corresponding to the sample to be detected based on a dialogue system in which an intention recognition model is deployed on the basis of the intention to be selected, sending the reply preset template to the user terminal, receiving answer information of the user, and extracting the true intention of the user based on the answer information;
the abnormality judging module is used for comparing the real intention of the user with the recognition intention corresponding to the corresponding sample to be detected and judging whether the intention recognition model is abnormal or not according to the comparison result.
The embodiment of the invention has the following advantages:
according to the anomaly detection method and device for the dialogue system, the probability values corresponding to different user intentions are predicted for the input calling text based on the intention recognition model in the dialogue system, the calling text corresponding to the condition of the threshold value of the probability value is used as a sample to be detected, the intention to be selected corresponding to the sample to be detected is obtained, the reply preset template corresponding to the sample to be detected is obtained based on the intention to be selected, the reply preset template is sent to the user terminal, answer information of the user is received, the true intention of the user is extracted based on the answer information, whether the model training/reply has an anomaly condition or not can be detected, the problem that the robot reply is irrelevant in the process of human-computer dialogue, and poor user experience is caused is solved.
Drawings
FIG. 1 is a step diagram of an anomaly detection method for a dialog system of the present invention;
fig. 2 is a schematic diagram of the composition of an abnormality detection apparatus for a dialogue system according to the present invention.
Detailed Description
It should be noted that, in the case of no conflict, the embodiments and features in the embodiments may be combined with each other. The invention will be described in detail below with reference to the drawings in connection with embodiments.
The dialogue system provided by the invention can provide services for the called user in the busy, unreachable and other scenes of the user, and can provide dialogue services in various scenes such as living things (such as express, takeaway and network about cars), real estate and transportation things (such as real estate, rent and urge payment), financial things (such as financial, credit card), acquaintance about (such as about eating, about shopping) and the like in the man-machine dialogue process by performing semantic understanding, intention recognition and dialogue generation based on artificial intelligence and completing text-to-speech conversion through ASR (Automatic Speech Recognition) and TTS (Text To Speech) so as to complete interaction dialogue with the calling user.
As shown in fig. 1, the present invention provides an anomaly detection method for a dialogue system, including:
step S1: and acquiring dialogue text data in the industry field, marking user intention on the dialogue text data to form a training sample, training the intention recognition model through the training sample, and deploying the trained intention recognition model into a dialogue system.
Specifically, in the embodiment of the invention, dialogue text data in an express scene is obtained, and user intention labeling is performed on the dialogue text data to form a training sample, wherein the user intention labeling can be performed in a mode of manual labeling, chatgpt large model labeling or open source data sorting, and the like, the intention recognition model is trained through the training sample, wherein the intention recognition model adopts a BERT pre-training language model and is deployed into a dialogue system, and for the voice content of a calling user, the intention recognition model can recognize which intention belongs to, such as delivery express, pickup, modification of express addresses and pickup of 5 intention branches.
In an embodiment of the present invention, for example, in a dialog system, a caller proposes: "you buy express delivery to", the intention recognition model deployed in the system needs to first determine that the intention corresponding to the sentence is delivery express delivery, then enter the intention branch, and return to the corresponding reply template: "please ask which express company to press? ", continuing to develop the dialogue, and the complete dialogue flow for delivering the express intention is shown in table 1:
TABLE 1 delivery express intent dialog flow sheet
Step S2: user intention statistics is carried out on dialogue corpus texts generated in a dialogue system with an intention recognition model deployed, a plurality of intention branches are obtained, and the number of dialogue corpus texts corresponding to each intention branch and the percentage of the number of dialogue corpus texts to the total number are obtained.
Specifically, in order to detect whether a reply call in a dialogue system is normal or not, statistics needs to be performed on user intention identified by the dialogue system, in the embodiment of the invention, statistics is performed on user intention with respect to dialogue corpus texts generated by 10W online dialogue systems, and the number of 5 intention branches entering delivery express, express delivery, pick-up, address modification express delivery and delivery time inquiry are respectively counted, wherein the percentage of the number of dialogue corpus texts corresponding to each intention branch to the total number is respectively: 30%, 20%, 25%, 24%, 1%, for the inquiry delivery time intent, only 1% of the conversations are identified as the intent, which is less than the percentage threshold of 10%, in order to further determine whether the conversational system has an identification abnormality, such as misidentifying the inquiry delivery time intent as other intentions, resulting in a robot reply that does not conform to the conversational context, affecting the user experience, resulting in user complaints, requiring further determination.
Step S3: when intention branches with the number of the dialogue corpus texts accounting for the total number less than a percentage threshold exist, collecting a plurality of calling texts newly input into a dialogue system provided with an intention recognition model, predicting probability values of different user intentions corresponding to the calling texts through the intention recognition model, taking the user intention with the largest probability value as the recognition intention corresponding to the corresponding calling text, and taking the calling text corresponding to the condition of meeting the probability value threshold as a sample to be detected.
Specifically, the probability value threshold includes a first probability value threshold and a second probability value threshold, the second probability value threshold is greater than the first probability value threshold, the first probability value threshold is 0.2, and the second probability value threshold is 0.7.
Specifically, taking the calling text corresponding to the probability value threshold condition as a sample to be detected includes: and taking the calling text corresponding to the probability value which is larger than the first probability value threshold and smaller than the second probability value threshold as a sample to be detected.
In one embodiment of the present invention, for a caller text, the intention recognition model outputs probability values of different user intentions corresponding to each caller text when predicting intention, and the user intention with the largest probability value is taken as the recognition intention corresponding to the corresponding caller text, for example, the intention recognition model outputs probability values of different user intentions corresponding to each caller text when predicting intention as shown in table 2:
table 2 probability value tables for each caller text corresponding to different user intentions
Determining the recognition intentions corresponding to the calling text 1, the calling text 2, the calling text 3, the calling text 4 and the calling text 5 by judging the intentions corresponding to the maximum probability value respectively as follows: the method comprises the steps of delivering an express, sending the express, getting a piece on the door, modifying an express address, getting a piece on the door, setting a first probability value threshold to be 0.2 for obtaining a sample to be detected, setting a second probability value threshold to be 0.7, obtaining a calling text corresponding to a probability value smaller than 0.7 and larger than 0.2, wherein the calling text corresponding to a probability value smaller than 0.7 and larger than 0.2 is called a calling text 2 and a calling text 5, accordingly, the calling text 2 and the calling text 5 are samples to be detected, the probability values of different user intentions corresponding to the calling text 2 and the calling text 5 are arranged according to the size, the user intentions corresponding to the probability values arranged at the second position are selected intentions corresponding to the calling text 2 and the calling text 5, the selected intentions of the calling text 2 are query delivery time, and the selected intentions of the calling text 5 are modified express addresses.
Step S4: obtaining a to-be-selected intention corresponding to a to-be-detected sample, obtaining a reply preset template corresponding to the to-be-detected sample through a dialogue system deployed with an intention recognition model based on the to-be-selected intention, sending the reply preset template to a user terminal, receiving answer information of a user, and extracting the real intention of the user based on the answer information.
Specifically, the obtaining the candidate intention corresponding to the sample to be detected includes:
arranging probability values of samples to be detected corresponding to different user intentions from large to small;
and taking the user intention corresponding to the probability value arranged at the second position as the candidate intention corresponding to the sample to be detected.
Specifically, the reply preset template may be "please ask you to be in question { is the candidate intention name? "or other template, for caller text 2, dialog system replies to the preset template: "do you ask you to ask the time of delivery express? ", for caller text 5, the robot replies to the preset template: "do you ask you to modify express addresses in question? ".
Specifically, extracting the real intention of the user based on the answer information includes:
judging the answer information of the user, wherein the answer of the user is judged to be affirmative or negative;
when the answer of the user is judged to be affirmative, taking the candidate intention corresponding to the sample to be detected as the real intention of the user;
and when the answer of the user is judged to be negative, taking the identification intention corresponding to the sample to be detected as the real intention of the user.
For example, in the above-described embodiment, when the answer to the caller text 2 is determined to be affirmative, the actual intention of the user is the inquiry delivery time, and when the answer to the user is determined to be negative, the actual intention of the user is the posting of the express; and for the calling text 5, when the answer of the user is judged to be affirmative, the real intention of the user is to modify the express address, and when the answer of the user is judged to be negative, the real intention of the user is to get the express mail.
Step S5: comparing the real intention of the user with the recognition intention corresponding to the corresponding sample to be detected, and judging whether the intention recognition model is abnormal or not according to the comparison result.
Specifically, step S5 includes:
when the real intention of the user is consistent with the recognition intention corresponding to the corresponding sample to be detected, storing the corresponding sample to be detected in a first sample set;
when the real intention of the user is inconsistent with the recognition intention corresponding to the corresponding sample to be detected, storing the corresponding sample to be detected in a second sample set;
the method comprises the steps of obtaining the proportion of the number of samples to be detected in a first sample set to the number of all samples to be detected, taking the proportion as a first proportion, or obtaining the proportion of the number of samples to be detected in a second sample set to the number of all samples to be detected, and taking the proportion as a second proportion;
when the first proportion is larger than or equal to the proportion threshold value or the second proportion is smaller than the proportion threshold value, judging that the intention recognition model is normal;
and when the first proportion is smaller than the proportion threshold value or the second proportion is larger than or equal to the proportion threshold value, judging that the model is abnormal.
For example, in the above-described embodiment, for the calling text 2, when it is determined that the real intention of the user is to inquire about the delivery time, it is stored in the second sample set, and when it is determined that the real intention of the user is to send an express, it is stored in the first sample set; and for the calling text 5, when the real intention of the user is judged to be the modification express address, the real intention of the user is stored in the second sample set, and when the real intention of the user is judged to be the getting-on piece, the real intention of the user is stored in the first sample set.
If in the above embodiment, when the calling text 2 and the calling text 5 are both stored in the first sample set, the corresponding first proportion is 100%, greater than the proportion threshold 80%, and the second proportion is 0%, less than the proportion threshold 80%, so as to determine that the intention recognition model is normal; if in the above embodiment, when both the calling text 2 and the calling text 5 are stored in the second sample set, the corresponding second proportion is 100%, greater than the proportion threshold 80%, and the first proportion is 0%, less than the proportion threshold 80%, so as to determine that the model is intended to be identified as abnormal; if one of the calling text 2 and the calling text 5 is stored in the first sample set and the other is stored in the second sample set in the above embodiment, the corresponding first proportion and the second proportion are both 50%, and the first proportion is smaller than the proportion threshold value 80%, so as to determine that the model is intended to be identified as abnormal.
As shown in fig. 2, the present invention provides an abnormality detection apparatus for a dialogue system, which performs abnormality detection of the dialogue system using the abnormality detection method for a dialogue system as described above, comprising:
the model deployment module is used for acquiring dialogue text data in the industry field, marking user intention on the dialogue text data to form a training sample, training the intention recognition model through the training sample, and deploying the trained intention recognition model into a dialogue system;
the preliminary judgment module is used for carrying out user intention statistics on dialogue corpus texts generated in the dialogue system with the intention recognition model deployed, obtaining a plurality of intention branches, and obtaining the quantity of dialogue corpus texts corresponding to the intention branches and the percentage of the quantity of dialogue corpus texts in the total quantity;
the system comprises a sample acquisition module to be detected, a recognition module and a detection module, wherein the sample acquisition module to be detected is used for acquiring a plurality of calling texts newly input into a dialogue system provided with an intention recognition model when intention branches with the number of dialogue corpus texts accounting for the total number of the intention branches smaller than a percentage threshold exist, predicting probability values of different user intentions corresponding to the calling texts through the intention recognition model, taking the user intention with the largest probability value as the recognition intention corresponding to the corresponding calling text, and taking the calling text corresponding to the condition meeting the probability value threshold as a sample to be detected;
the true intention acquisition module is used for acquiring the intention to be selected corresponding to the sample to be detected, acquiring a reply preset template corresponding to the sample to be detected through a dialogue system deployed with an intention recognition model based on the intention to be selected, sending the reply preset template to the user terminal, receiving answer information of the user, and extracting the true intention of the user based on the answer information;
the abnormality judging module is used for comparing the real intention of the user with the recognition intention corresponding to the corresponding sample to be detected and judging whether the intention recognition model is abnormal or not according to the comparison result.
It should be noted that the foregoing detailed description is exemplary and is intended to provide further explanation of the invention. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.
It is noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments in accordance with the present application. As used herein, the singular is intended to include the plural unless the context clearly indicates otherwise. Furthermore, it will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, steps, operations, devices, components, and/or groups thereof.
It should be noted that the terms "first," "second," and the like in the description and claims of the present application and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the terms so used are interchangeable under appropriate circumstances such that the embodiments of the application described herein are capable of operation in sequences other than those illustrated or otherwise described herein.
Furthermore, the terms "comprise" and "have," as well as any variations thereof, are intended to cover a non-exclusive inclusion. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those elements but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
Spatially relative terms, such as "above … …," "above … …," "upper surface at … …," "above," and the like, may be used herein for ease of description to describe one device or feature's spatial location relative to another device or feature as illustrated in the figures. It will be understood that the spatially relative terms are intended to encompass different orientations in use or operation in addition to the orientation depicted in the figures. For example, if the device in the figures is turned over, elements described as "above" or "over" other devices or structures would then be oriented "below" or "beneath" the other devices or structures. Thus, the exemplary term "above … …" may include both orientations of "above … …" and "below … …". The device may also be positioned in other different ways, such as rotated 90 degrees or at other orientations, and the spatially relative descriptors used herein interpreted accordingly.
In the above detailed description, reference is made to the accompanying drawings, which form a part hereof. In the drawings, like numerals typically identify like components unless context indicates otherwise. The illustrated embodiments described in the detailed description, drawings, and claims are not meant to be limiting. Other embodiments may be utilized, and other changes may be made, without departing from the spirit or scope of the subject matter presented herein.
The above description is only of the preferred embodiments of the present invention and is not intended to limit the present invention, but various modifications and variations can be made to the present invention by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (9)
1. An anomaly detection method for a dialog system, the method comprising:
acquiring dialogue text data in the industry field, marking user intention on the dialogue text data to form a training sample, training the intention recognition model through the training sample, and deploying the trained intention recognition model into a dialogue system;
user intention statistics is carried out on dialogue corpus texts generated in a dialogue system with an intention recognition model deployed, a plurality of intention branches are obtained, and the number of dialogue corpus texts corresponding to each intention branch and the percentage of the number of dialogue corpus texts in the total number are obtained;
when intention branches with the total number of the dialogue corpus texts being smaller than a percentage threshold exist, collecting a plurality of calling texts which are newly input into a dialogue system provided with an intention recognition model, predicting probability values of different user intentions corresponding to the calling texts through the intention recognition model, taking the user intention with the largest probability value as the recognition intention corresponding to the corresponding calling text, and taking the calling text corresponding to the condition of meeting the probability value threshold as a sample to be detected;
obtaining a to-be-selected intention corresponding to a to-be-detected sample, obtaining a reply preset template corresponding to the to-be-detected sample through a dialogue system deployed with an intention recognition model based on the to-be-selected intention, sending the reply preset template to a user terminal, receiving answer information of a user, and extracting the real intention of the user based on the answer information;
comparing the real intention of the user with the recognition intention corresponding to the corresponding sample to be detected, and judging whether the intention recognition model is abnormal or not according to the comparison result;
comparing the real intention of the user with the recognition intention corresponding to the corresponding sample to be detected, and judging whether the intention recognition model is abnormal according to the comparison result comprises the following steps:
when the real intention of the user is consistent with the recognition intention corresponding to the corresponding sample to be detected, storing the corresponding sample to be detected in a first sample set;
when the real intention of the user is inconsistent with the recognition intention corresponding to the corresponding sample to be detected, storing the corresponding sample to be detected in a second sample set;
the method comprises the steps of obtaining the proportion of the number of samples to be detected in a first sample set to the number of all samples to be detected, taking the proportion as a first proportion, or obtaining the proportion of the number of samples to be detected in a second sample set to the number of all samples to be detected, and taking the proportion as a second proportion;
when the first proportion is larger than or equal to the proportion threshold value or the second proportion is smaller than the proportion threshold value, judging that the intention recognition model is normal;
and when the first proportion is smaller than the proportion threshold value or the second proportion is larger than or equal to the proportion threshold value, judging that the model is abnormal.
2. The abnormality detection method for a dialogue system according to claim 1, characterized in that the plurality of intention branches includes: delivery express, sending express, getting the piece from the upper door, modifying the address of the express and getting the piece from the upper door.
3. The anomaly detection method for a dialog system of claim 1, wherein the probability value threshold comprises a first probability value threshold and a second probability value threshold, the second probability value threshold being greater than the first probability value threshold.
4. The abnormality detection method for a dialogue system according to claim 3, wherein taking, as a sample to be detected, a calling text corresponding when a probability value threshold condition is satisfied includes: and taking the calling text corresponding to the probability value which is larger than the first probability value threshold and smaller than the second probability value threshold as a sample to be detected.
5. The abnormality detection method for a dialogue system according to claim 1, wherein obtaining a candidate intention corresponding to a sample to be detected includes:
arranging probability values of samples to be detected corresponding to different user intentions from large to small;
and taking the user intention corresponding to the probability value arranged at the second position as the candidate intention corresponding to the sample to be detected.
6. The abnormality detection method for a dialogue system according to claim 1, characterized in that extracting a user's real intention based on answer information includes:
judging the answer information of the user, wherein the answer of the user is judged to be affirmative or negative;
when the answer of the user is judged to be affirmative, taking the candidate intention corresponding to the sample to be detected as the real intention of the user;
and when the answer of the user is judged to be negative, taking the identification intention corresponding to the sample to be detected as the real intention of the user.
7. The abnormality detection method for a dialogue system according to claim 1, characterized in that the proportion threshold value is 80%.
8. An abnormality detection apparatus for a dialogue system, which performs abnormality detection of the dialogue system using the abnormality detection method for a dialogue system according to any one of claims 1 to 7, characterized by comprising:
the model deployment module is used for acquiring dialogue text data in the industry field, marking user intention on the dialogue text data to form a training sample, training the intention recognition model through the training sample, and deploying the trained intention recognition model into a dialogue system;
the preliminary judgment module is used for carrying out user intention statistics on dialogue corpus texts generated in the dialogue system with the intention recognition model deployed, obtaining a plurality of intention branches, and obtaining the quantity of dialogue corpus texts corresponding to the intention branches and the percentage of the quantity of dialogue corpus texts to the total quantity.
9. The abnormality detection apparatus for a dialogue system according to claim 8, characterized by further comprising:
the system comprises a sample acquisition module to be detected, a recognition module and a detection module, wherein the sample acquisition module to be detected is used for acquiring a plurality of calling texts newly input into a dialogue system provided with an intention recognition model when intention branches with the number of dialogue corpus texts accounting for the total number of the intention branches smaller than a percentage threshold exist, predicting probability values of different user intentions corresponding to the calling texts through the intention recognition model, taking the user intention with the largest probability value as the recognition intention corresponding to the corresponding calling text, and taking the calling text corresponding to the condition meeting the probability value threshold as a sample to be detected;
the true intention acquisition module is used for acquiring the intention to be selected corresponding to the sample to be detected, acquiring a reply preset template corresponding to the sample to be detected through a dialogue system deployed with an intention recognition model based on the intention to be selected, sending the reply preset template to the user terminal, receiving answer information of the user, and extracting the true intention of the user based on the answer information;
the abnormality judging module is used for comparing the real intention of the user with the recognition intention corresponding to the corresponding sample to be detected and judging whether the intention recognition model is abnormal or not according to the comparison result;
comparing the real intention of the user with the recognition intention corresponding to the corresponding sample to be detected, and judging whether the intention recognition model is abnormal according to the comparison result comprises the following steps:
when the real intention of the user is consistent with the recognition intention corresponding to the corresponding sample to be detected, storing the corresponding sample to be detected in a first sample set;
when the real intention of the user is inconsistent with the recognition intention corresponding to the corresponding sample to be detected, storing the corresponding sample to be detected in a second sample set;
the method comprises the steps of obtaining the proportion of the number of samples to be detected in a first sample set to the number of all samples to be detected, taking the proportion as a first proportion, or obtaining the proportion of the number of samples to be detected in a second sample set to the number of all samples to be detected, and taking the proportion as a second proportion;
when the first proportion is larger than or equal to the proportion threshold value or the second proportion is smaller than the proportion threshold value, judging that the intention recognition model is normal;
and when the first proportion is smaller than the proportion threshold value or the second proportion is larger than or equal to the proportion threshold value, judging that the model is abnormal.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311344618.5A CN117076654B (en) | 2023-10-18 | 2023-10-18 | Abnormality detection method and device for dialogue system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311344618.5A CN117076654B (en) | 2023-10-18 | 2023-10-18 | Abnormality detection method and device for dialogue system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117076654A CN117076654A (en) | 2023-11-17 |
| CN117076654B true CN117076654B (en) | 2024-02-27 |
Family
ID=88704746
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311344618.5A Active CN117076654B (en) | 2023-10-18 | 2023-10-18 | Abnormality detection method and device for dialogue system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117076654B (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111104495A (en) * | 2019-11-19 | 2020-05-05 | 深圳追一科技有限公司 | Information interaction method, device, equipment and storage medium based on intention recognition |
| CN111538820A (en) * | 2020-04-10 | 2020-08-14 | 出门问问信息科技有限公司 | Exception reply processing device and computer readable storage medium |
| CN113569581A (en) * | 2021-08-26 | 2021-10-29 | 中国联合网络通信集团有限公司 | Intention recognition method, device, equipment and storage medium |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10741176B2 (en) * | 2018-01-31 | 2020-08-11 | International Business Machines Corporation | Customizing responses to users in automated dialogue systems |
-
2023
- 2023-10-18 CN CN202311344618.5A patent/CN117076654B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111104495A (en) * | 2019-11-19 | 2020-05-05 | 深圳追一科技有限公司 | Information interaction method, device, equipment and storage medium based on intention recognition |
| CN111538820A (en) * | 2020-04-10 | 2020-08-14 | 出门问问信息科技有限公司 | Exception reply processing device and computer readable storage medium |
| CN113569581A (en) * | 2021-08-26 | 2021-10-29 | 中国联合网络通信集团有限公司 | Intention recognition method, device, equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117076654A (en) | 2023-11-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107886951B (en) | Voice detection method, device and equipment | |
| CN111917924A (en) | Customer service voice quality inspection method and related equipment | |
| CN109326305B (en) | Method and system for batch testing of speech recognition and text synthesis | |
| CN103474065A (en) | Method for determining and recognizing voice intentions based on automatic classification technology | |
| CN101662549A (en) | Customer evaluation system and customer evaluation method based on voice | |
| CN105931642B (en) | Voice recognition method, device and system | |
| CN110390946A (en) | A kind of audio signal processing method, device, electronic equipment and storage medium | |
| CN107885723B (en) | Conversation role distinguishing method and system | |
| CN111445928A (en) | Voice quality inspection method, device, equipment and storage medium | |
| CN105827787B (en) | number marking method and device | |
| CN104183238A (en) | Old people voiceprint recognition method based on questioning and answering | |
| CN114399379A (en) | Artificial intelligence-based collection behavior recognition method, device, equipment and medium | |
| CN110019285A (en) | A kind of alert identification allocating method and electronic equipment | |
| CN111556207A (en) | Method, device, computer equipment and storage medium for identifying crank calls | |
| CN112235468A (en) | Audio processing method and system for voice customer service evaluation | |
| CN117076654B (en) | Abnormality detection method and device for dialogue system | |
| CN110880318A (en) | Voice recognition method and device | |
| CN116167674A (en) | Logistics complaint list input method, device, equipment and storage medium | |
| CN107645613A (en) | The method and apparatus of service diverting search | |
| CN110990570A (en) | Mail drop extraction method based on deep learning | |
| CN110728145A (en) | Method for establishing natural language understanding model based on recording conversation | |
| CN116915903A (en) | Call voice processing method, device, server and readable storage medium | |
| CN113191711A (en) | Express delivery sending strategy determining method, device, equipment and storage medium | |
| CN114155845A (en) | Service determination method and device, electronic equipment and storage medium | |
| CN110428184B (en) | Backlog distribution method, backlog distribution device, backlog distribution equipment and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |