CN111177388A

CN111177388A - Processing method and computer equipment

Info

Publication number: CN111177388A
Application number: CN201911391846.1A
Authority: CN
Inventors: 董孝政; 邹丹
Original assignee: Lenovo Beijing Ltd
Current assignee: Lenovo Beijing Ltd
Priority date: 2019-12-30
Filing date: 2019-12-30
Publication date: 2020-05-19
Anticipated expiration: 2039-12-30
Also published as: CN111177388B

Abstract

The application relates to a processing method and computer equipment, the method is based on the relation among the model class labels to divide the class label system in advance, and construct a plurality of classification models according to the multi-label system obtained by division, wherein, each class label included in the same label system satisfies the mutual exclusion relation, correspondingly, for the classification model corresponding to the label system constructed, the influence of each class label satisfying the mutual exclusion condition on the model processing result of the class corresponding to the other party is not generated, thus, when the classification model is used for classifying the target problem, the model can be ensured to give a more accurate prediction result aiming at each class label in the label category of the model, and finally, the answer is recalled through synthesizing the more accurate model prediction result of the multi-model, the method can achieve the purpose of high coverage of the real answer, can more accurately cover the real answer, and improves the accuracy of answer recall.

Description

Processing method and computer equipment

Technical Field

The application belongs to the field of artificial intelligence, and particularly relates to a processing method and computer equipment.

Background

With the rapid development of computer technology and artificial intelligence technology, more and more customer service robots are currently available, in order to help users to obtain answers better and faster, the customer service robots recall topN (usually N ═ 10, and N is a natural number) answers from an answer database and push the answers to the users, so that the users can determine needed answers from the answers.

At present, a customer service robot generally identifies a user intention category based on a pre-trained model and pushes a top N answer, however, the accuracy of the model is difficult to achieve an ideal accuracy, which may cause a low accuracy of answer recall, and it is difficult to cover a real answer (an answer required by a user) in the pushed top N answer with high accuracy, so how to enable the top N answer to more accurately cover the real answer and improve the accuracy of answer recall becomes a problem to be solved by the customer service robot.

Disclosure of Invention

In view of this, the present application provides a processing method and a computer device, which are used to solve the above problems of the customer service robot, so that the top N answer can more accurately cover the real answer, and the accuracy of answer recall is improved.

Therefore, the application discloses the following technical scheme:

a method of processing, comprising:

acquiring a target problem;

classifying the target problem by utilizing a plurality of pre-constructed classification models; the different classification models correspond to different label systems, and the label systems corresponding to the classification models comprise a plurality of class labels meeting the mutual exclusion condition; the mutex condition can be used to indicate: in the process of model classification, different types of labels meeting the mutual exclusion condition cannot influence the model processing result of the corresponding type of the opposite side beyond a preset influence limit value;

obtaining classification processing results of the target problems by the classification models;

and determining at least one target answer matched with the target question based on the classification processing result.

The above method, preferably, further comprises: pre-constructing the plurality of classification models;

the construction process of the plurality of classification models comprises the following steps:

determining a plurality of category labels to be classified;

determining the incidence relation among different category labels in the plurality of category labels to be divided;

if the association relation among the different types of tags is the cross relation which does not meet the mutual exclusion condition, dividing the different types of tags into different tag systems respectively;

if the association relation among the different types of tags is the mutual exclusion relation meeting the mutual exclusion condition, dividing the different types of tags into the same or different tag systems;

respectively constructing different classification models aiming at different label systems obtained by division; wherein, each class label that the classification model corresponds includes: and the classification model corresponds to each class label provided by the label system.

Preferably, the determining the association relationship between different category labels in the plurality of category labels to be classified includes:

constructing a first model corresponding to the plurality of category labels to be classified;

removing a first class label from the plurality of class labels to be classified, and constructing a second model corresponding to the plurality of class labels to be classified which are not removed;

determining a second class label from the plurality of class labels to be classified which are not removed; the second category label is: the classification performance in the second model is improved compared with the classification performance in the first model, and the improvement amplitude meets the class label of an amplitude condition;

determining the association relationship between the second category label and the first category label as: a cross-relation that does not satisfy the mutual exclusion condition;

determining the association relationship between the labels of the plurality of non-rejected labels of the to-be-classified categories except the label of the second category and the label of the first category as follows: and meeting the mutual exclusion relation of the mutual exclusion condition.

determining similarity among different category labels in the plurality of category labels to be divided;

determining the incidence relation between different types of labels with the similarity reaching the similarity threshold as follows: a cross-relation that does not satisfy the mutual exclusion condition;

determining the association relationship between different types of labels with similarity not reaching the similarity threshold as follows: and meeting the mutual exclusion relation of the mutual exclusion condition.

The method preferably, wherein the dividing of the different types of tags into the corresponding tag systems includes:

traversing each class label, and dividing the currently traversed class label into label systems with the time length of the label not allocated to reach the time length condition until each class label is divided into the corresponding label system;

determining whether all the class labels which do not meet the mutual exclusion condition are divided in the same label system;

if yes, separating different types of labels which do not meet the mutual exclusion condition in the same label system into different label systems.

The above method, preferably, the determining at least one target answer matching the target question based on the classification processing result includes:

determining category labels of various categories to which the target problem belongs based on classification processing results of various classification models;

determining at least one target class label from the class labels of the various classes;

and determining at least one answer corresponding to the at least one target category label as the at least one target answer.

A computer device, comprising:

a memory for storing at least one set of instructions;

a processor for invoking and executing the set of instructions in the memory, by executing the set of instructions:

acquiring a target problem;

The computer device is preferably configured such that the processor is further configured to: pre-constructing the plurality of classification models;

the process of the processor building the plurality of classification models comprises:

determining a plurality of category labels to be classified;

Preferably, in the computer device, the determining, by the processor, an association relationship between different category labels in the plurality of category labels to be classified includes:

constructing a first model corresponding to the plurality of category labels to be classified; removing a first class label from the plurality of class labels to be classified, and constructing a second model corresponding to the plurality of class labels to be classified which are not removed; determining a second class label from the plurality of class labels to be classified which are not removed; the second category label is: the classification performance in the second model is improved compared with the classification performance in the first model, and the improvement amplitude meets the class label of an amplitude condition; determining the association relationship between the second category label and the first category label as: a cross-relation that does not satisfy the mutual exclusion condition; determining the association relationship between the labels of the plurality of non-rejected labels of the to-be-classified categories except the label of the second category and the label of the first category as follows: a mutual exclusion relationship satisfying the mutual exclusion condition;

alternatively, the first and second electrodes may be,

determining similarity among different category labels in the plurality of category labels to be divided; determining the incidence relation between different types of labels with the similarity reaching the similarity threshold as follows: a cross-relation that does not satisfy the mutual exclusion condition; determining the association relationship between different types of labels with similarity not reaching the similarity threshold as follows: and meeting the mutual exclusion relation of the mutual exclusion condition.

The computer device preferably, wherein the processor determines at least one target answer matching the target question based on the classification processing result, and includes:

It can be known from the above solutions that the processing method and the computer device provided in the present application perform system division on category labels in advance based on the relationship between model category labels, and construct a plurality of classification models according to the multi-label system obtained by division, wherein each category label included in the same label system satisfies the mutual exclusion relationship, and accordingly, for the constructed classification model corresponding to the label system, the influence that each category label satisfying the mutual exclusion condition does not generate the influence exceeding the predetermined influence limit on the model processing result of the category corresponding to the counterpart cannot be generated between the category labels, so that when the classification model is used to classify the target problem, it can be ensured that the model can give a more accurate prediction result for each category label corresponding to the model in the category of the model itself, and finally answer is recalled by synthesizing the more accurate model prediction result of the multi-model, the method can achieve the purpose of high coverage of the real answer, can more accurately cover the real answer, and improves the accuracy of answer recall.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, it is obvious that the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.

FIG. 1 is a schematic flow chart of a processing method provided by an embodiment of the present application;

FIG. 2 is another schematic flow chart diagram of a processing method provided in an embodiment of the present application;

FIG. 3 is a schematic flow chart of a processing method provided by an embodiment of the present application;

FIG. 4 is a schematic flow chart of building a plurality of classification models provided by an embodiment of the present application;

FIG. 5 is a logic diagram for constructing a plurality of classification models based on a plurality of label systems and for performing answer recall using the plurality of classification models constructed according to an embodiment of the present application;

FIG. 6 is a schematic flow chart illustrating a process of determining an association relationship between different tags according to an embodiment of the present application;

FIG. 7 is a flowchart illustrating the systematic division of different category labels according to an embodiment of the present disclosure;

FIG. 8 is a matrix of associations between a plurality of different tags as provided by embodiments of the present application;

FIG. 9 is a diagram of another matrix of associations between a plurality of different tags as provided by embodiments of the present application;

fig. 10 is a schematic structural diagram of a computer device provided in an embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

The current customer service robot generally identifies the user intention category based on a pre-trained classification model, and then pushes the top N answer according to the identified intention category. The inventor has found that there may be different associations between different category labels in the classification task, and the following two examples are provided:

example one:

the labels are entertainment and game, the meanings of the two labels are crossed, the boundary between the two labels is not clear, and the two labels are difficult to distinguish in some cases;

the labels of 'science and technology' and 'history' have large semantic gaps, and are easy to distinguish;

example two:

the labels of 'external network login mailbox is not available' and 'internal network login mailbox is not available', and the two labels are different from each other in the internal network and the external network, so that the models are difficult to distinguish.

For different category labels with less distinct boundaries and difficult distinction, the present application considers that such different category labels have a cross relationship, and the different category labels with the cross relationship have a great influence on the model classification result of the category corresponding to the counterpart during the model classification process, for example, if the model corresponds to a plurality of labels such as "entertainment", "game", "science", "history", "military", "politics", and the like, since the semantic cross exists between "entertainment" and "game", when the true label of the target problem is one of the two labels, the two labels will have a great influence on the model classification result of the category corresponding to the counterpart, for example, once the confidence level of "entertainment" is predicted by the model is higher (e.g. 80%), the confidence level of "game" is predicted to be greatly reduced (not more than 20%), in this case, because semantic intersection exists between the two, the prediction results influence each other, once model prediction is wrong, the real intention of the user question is covered or omitted, and accordingly, the answer really needed by the user is difficult to be covered in the finally recalled answer.

For different labels which have larger semantic gaps and are easy to distinguish, the incidence relation between the labels is defined as the mutual exclusion relation, the different labels meeting the mutual exclusion relation do not have larger influence on the model classification result of the class corresponding to the opposite side correspondingly in the model classification process, for example, for the label history with larger deviation from entertainment and game, the corresponding confidence coefficient is lower, such as 5 percent and 3 percent, and the like, and no matter whether the model accurately predicts the target problem between entertainment and game, the prediction result of the history label can not be greatly influenced.

Based on the characteristics, in order to improve the answer recall accuracy of the customer service robot and enable the recalled top N answer to more accurately cover the real answer, the following technical concept is provided in the application: the method comprises the steps of carrying out system division on class labels based on the relation among model class labels, separating the class labels which have larger mutual influence and have a cross relation into different label systems, correspondingly enabling the class labels which are divided in each system to be the class labels which have smaller mutual influence and have a mutual exclusion relation, further constructing a plurality of classification models according to the multi-label system obtained by division, and carrying out question intention identification and answer recall based on the plurality of constructed classification models.

Based on the technical concept, the present application discloses a processing method and a computer device, which will be described in detail below by specific embodiments.

In an alternative embodiment of the present application, a processing method is disclosed, which may be applied to, but not limited to, a portable terminal device such as a smart phone, a personal digital assistant, a tablet computer, or the like, or may also be applied to a portable computer (such as a notebook), a desktop computer or a large and medium sized computer, a backend server, or the like in a general/special purpose computing or configuration environment.

Referring to the flow chart of the processing method shown in fig. 1, in the present embodiment, the processing method includes:

and step 101, acquiring a target problem.

More typically, the scheme of the application can be applied to, but not limited to, an intelligent question-answering scene based on a customer service robot (intelligent robot), and/or a pre-training scene of a customer service robot processing model.

In this step, the target question may be obtained by obtaining a question to be solved input by a natural person user or an electronic device, for example, obtaining a question to be solved input by a natural person user through voice or text input and the like, "how do what is a case when a mailbox is logged in? "and the like.

And 102, classifying the target problem by using a plurality of pre-constructed classification models.

After the target problem is obtained, the target problem is respectively input into a plurality of classification models trained in advance, and each classification model in the classification models classifies the target problem.

In the embodiment of the application, when the target problem is classified, a plurality of classification models are adopted, each classification model corresponds to one tag system, and the tag systems corresponding to the classification models comprise a plurality of class tags meeting the mutual exclusion condition.

The mutual exclusion condition is a condition that can be used to indicate the following meaning: in the process of model classification, different types of labels meeting the mutual exclusion condition cannot influence the model processing result of the corresponding type of the opposite side beyond the preset influence limit value.

For example, for the "entertainment" and the "game" in the above example one, when the tag system is constructed, since the two tags have a cross relationship and do not satisfy the mutual exclusion condition, the two tags are divided into different tag systems, so that different tags in the same tag system are tags that can satisfy the mutual exclusion condition and do not have the cross relationship. Thus, for the above-mentioned "entertainment", "game", "science and technology", "history", "military", "politics", etc. tags, when dividing the tags, they can be divided into at least two tag systems, such as: the label system I: entertainment, science, history, military; and (2) label system II: games, politics; similarly, for example two, it is also at least divided into two label systems, such as: and (3) label system III: the external network can not log in the mailbox and the mobile phone can not be started; and a label system IV: and the intranet does not log in the mailbox, refreshes the mobile phone, installs the mobile phone software and the like.

And respectively constructing classification models based on different label systems obtained by division, such as a classification model 1 corresponding to the third label system, a classification model 2 corresponding to the fourth label system and the like. After the target problem is obtained, the target problem is respectively input into different classification models constructed based on different label systems, and the target problem is respectively classified and processed by the different classification models.

It should be noted that, although the description in this embodiment describes that the labels divided in the label system are in the form of labels with actual semantics, such as "entertainment", "science", etc., in an actual implementation, characters such as "0", "1", "2", etc. may be used as the labels, and each label corresponds to a corresponding intention category.

And 103, obtaining the classification processing result of the target problem by the plurality of classification models.

After the target problem is respectively input into the plurality of classification models, the prediction results (namely classification processing results) of the plurality of classification models on the target problem are correspondingly obtained, and the given prediction results can be at least used for indicating the class label of the target problem.

One possible scenario is that, when outputting the prediction result, the classification model specifically gives the confidence result that the target problem belongs to each corresponding class label, for example, the classification model 1 gives the prediction result: label 1-confidence 1, label 2-confidence 2; the prediction results given by classification model 2: for the situation, the class label of the class to which the target problem belongs can be determined according to different confidence degrees of the target problem corresponding to different labels subsequently; another possibility is that the classification model gives an exact class label of the class to which the target problem belongs, as the prediction result given by classification model 1: label 2, prediction results given by classification model 2: and a label 5.

And 104, determining at least one target answer matched with the target question based on the classification processing result.

After the target problem is classified, the classification models output a plurality of prediction results, that is, a plurality of classification results, and classify the target problem and provide the prediction results, so as to identify an intention category corresponding to the target problem, for example, "how did not get a mailbox for the target problem? After classifying the mobile phone, the intention type of the mobile phone can be further identified as one of the categories of "no mailbox is logged in to the external network", "no mailbox is logged in to the internal network", "the mobile phone cannot be started", "mobile phone is booted", and "mobile phone installation software" according to the prediction result of the model.

After identifying a plurality of intention categories of the target question based on a plurality of prediction results of a plurality of classification models, at least one corresponding target answer, namely TOP N answer, is further matched from the answer database and fed back to the user, wherein N is a set numerical value, usually N < 10, and N is a natural number.

It can be known from the above solutions that, in the processing method of this embodiment, the class labels are classified in advance based on the relationship between the class labels of the models, and a plurality of classification models are constructed according to the multi-label system obtained by the classification, wherein each class label included in the same label system satisfies the mutual exclusion relationship, and accordingly, for the constructed classification model corresponding to the label system, the influence exceeding the predetermined influence limit value is not generated on the model processing result of the class corresponding to the other party among the class labels satisfying the mutual exclusion condition, so that when the classification model is used to classify the target problem, it can be ensured that the model can give a more accurate prediction result for each class label corresponding to the model in the category of the label itself, and finally, the answer can be recalled by synthesizing the more accurate model prediction result of the multi-model, so as to achieve the purpose of high coverage of the real answer, the real answer can be covered more accurately, and the accuracy of answer recall is improved.

In an alternative embodiment of the present application, referring to fig. 2, the step 104 can be further implemented by the following processing procedure:

step 1041, determining a category label of each category to which the target problem belongs based on a classification processing result of each classification model.

Specifically, if the prediction result output by the classification model is in a tag-confidence form, the class tag with the highest corresponding confidence can be determined from the prediction result, and the class tag is used as the class tag of the class to which the target problem belongs; if the prediction result output by the classification model is an exact class label of the class to which the target problem belongs, the label output by the classification model can be directly read as the class label of the class to which the target problem belongs. The category label of the category to which the target question belongs represents the intention category corresponding to the target question.

Step 1042, determining at least one target class label from the class labels of the classes.

After a plurality of category labels of a plurality of categories to which the target question belongs (one category to which the target question belongs is determined for each classification model) are determined based on the prediction results of the plurality of classification models, a part or all of the labels may be determined therefrom as target category labels required when answer recall is performed, that is, a part or all of intention categories may be determined from a plurality of intention categories determined based on the plurality of models as target intention categories required when answer recall is performed.

Step 1043, determining at least one answer corresponding to the at least one target category label as the at least one target answer.

Preferably, in order to cover the real answers with the highest probability, all the intention categories identified based on the plurality of classification models may be combined, and then at least one target answer corresponding to each intention category of all the intention categories is matched from the answer database, and the target answers corresponding to all the intention categories form a TOP N answer and are fed back to the user together.

For example, assume that the intent classes identified based on classification model 1 are: and class 2, if the intention class identified based on the classification model 2 is class 5, the classes 2 and 5 are both used as the intention classes of the target question, and are not screened from the intention classes, and finally, the provided TOP N answers correspondingly comprise answers corresponding to the classes 2 and 5.

In this embodiment, the labels with the cross relationship are separately divided into different label systems, that is, the labels in the same label system satisfy the mutual exclusion relationship, on one hand, the classification model constructed for each label system can be enabled not to affect each other during the classification processing, so that the model can give a more accurate prediction result for each class label in the label category of the model, and finally, the coverage of the real answer can be obviously improved by synthesizing a plurality of more accurate prediction results of a plurality of classification models and recalling the answer, thereby achieving the purpose of high coverage of the real answer.

Or, in another aspect, for the case that the real category to which the target question belongs is one of two or more category labels having a cross relationship, based on the embodiment, by integrating a plurality of more accurate prediction results of a plurality of classification models, the two or more category labels having a cross relationship and being difficult to distinguish are all included or identified as the intention category to which the target question belongs, so that answers corresponding to the plurality of category labels having a cross relationship are all put into the generated TOP N answer, and the real answer is not recalled back into the TOP N answer due to a model identification error (mistakenly identified as one non-real category of the plurality of category labels having a cross relationship).

The following examples are given.

Assuming that the intention category of the target problem is identified as "no mailbox for external network login" based on the prediction result given by the classification model 1, and the intention category of the target problem is identified as "no mailbox for internal network login" based on the prediction result given by the classification model 2, the target problem "no mailbox for external network login? The intention identification is that the external network does not log on the mailbox and the internal network does not log on the mailbox, and then the target answers corresponding to each intention category in the two intention categories can be matched from the answer database and fed back to the user.

The answers obtained by the user include answers corresponding to the intention category of 'no mailbox for external network login', and answers corresponding to the intention category of 'no mailbox for internal network login', so that the TOP N answers finally recalled can be covered no matter which real answer is, and the user can further select the answer required by the real answer according to the actual requirement. Compared with the prior art that labels are not classified systematically, the intention category of the target question is identified as one of 'no mailbox for external network login', 'no mailbox for internal network login', 'mobile phone flashing' … …, and then TOP N answer recall is carried out, the coverage degree of the real answer is obviously improved, the real answer can be more accurately covered, and the answer recall accuracy is improved.

In processing the target question to realize answer recall by using the processing method of the embodiment of the present application, a plurality of classification models constructed in advance are required to be used as a basis, and thus, in an alternative embodiment of the present application, referring to fig. 3, before the target question is obtained in step 101, the processing method may further include the following processing:

step 101', a plurality of classification models are constructed in advance.

The following describes the process of constructing the plurality of classification models, and as shown in fig. 4, the process of constructing the plurality of classification models can be implemented by:

step 401, determining a plurality of category labels to be divided.

The classification task includes a plurality of classification tags to be classified, that is, a plurality of classification tags required by an actual classification task.

In the implementation, a plurality of category labels to be divided can be determined according to the actual task requirements. For example, optionally, a plurality of category labels to be classified may be determined as: "entertainment", "gaming", "science", "history", "military", "politics"; or, determining a plurality of category labels to be divided as: the method comprises the following steps of not logging in a mailbox through an external network, not starting a mobile phone, not logging in a mailbox through an internal network, flashing a mobile phone, installing mobile phone software and the like.

Step 402, determining the incidence relation between different category labels in a plurality of category labels to be divided.

In the embodiment of the application, the association relationship between different types of tags is divided into a mutual exclusion relationship which meets a mutual exclusion condition and a cross relationship which does not meet the mutual exclusion condition. For the definition of the cross relationship and the mutual exclusion relationship, the above description may be referred to.

Therefore, in this embodiment, the association relationship between the different types of tags is determined, that is, it is determined whether the different types of tags satisfy the mutual exclusion relationship of the mutual exclusion condition or do not satisfy the cross relationship of the mutual exclusion condition.

As an alternative implementation, it may be determined whether the different types of tags are in a mutual exclusion relationship or a cross relationship according to the similarity between the different types of tags.

In this implementation, the similarity between different category labels in the plurality of category labels to be classified may be specifically determined; and determining the incidence relation between different types of tags with the similarity reaching the similarity threshold as the cross relation which does not meet the mutual exclusion condition, and determining the incidence relation between different types of tags with the similarity not reaching the similarity threshold as the mutual exclusion relation which meets the mutual exclusion condition.

That is, in this implementation, the mutex condition is specifically set to: the similarity does not reach a similarity threshold.

Optionally, the similarity between the category labels may be a semantic similarity, and in a specific implementation, exemplarily, different category labels may be vectorized to obtain a label vector of the category label, and a cosine distance method, an euclidean distance method, and the like may be further adopted to determine the semantic similarity between the different label vectors.

In addition, the embodiment of the application also provides a model-based actual measurement method for actually testing and verifying the influence of different types of tags on each other in the model classification process, and further determining whether the relationship between the different types of tags is a mutual exclusion relationship or a cross relationship based on the influence condition of the influence on each other. This implementation will be described in detail in the following embodiments, with particular reference to the statements below.

And 403, if the association relationship among the different types of tags is the cross relationship which does not meet the mutual exclusion condition, dividing the different types of tags into different tag systems respectively.

And step 404, if the association relationship among the different types of tags is the mutual exclusion relationship meeting the mutual exclusion condition, dividing the different types of tags into the same or different tag systems.

Then, system division is carried out on the category labels further based on the determined association relationship among the different category labels, and at least the different labels of which the association relationship is the cross relationship which does not meet the mutual exclusion condition are separately divided into different label systems, so that the situation that the category labels with the cross relationship have influence on the classification result of the corresponding category of the other party when in the same label system, and the classification accuracy of the model is reduced is avoided; the different types of tags whose association relations are mutual exclusion relations satisfying the mutual exclusion condition are not limited, and may be divided into the same tag system or different tag systems.

In specific implementation, different tags with mutual exclusion relationship can be distributed in corresponding different or same tag systems based on the principle of balancing the number of tags in different tag systems.

Step 405, respectively constructing different classification models for different label systems obtained by division; wherein, each class label that the classification model corresponds includes: and the classification model corresponds to each class label provided by the label system.

After different label systems are obtained through label division, a plurality of groups of different data samples (one label corresponds to one group of data samples) corresponding to a plurality of different types of labels in the label system can be collected for each label system, and then a classification model corresponding to the label system is trained based on the collected data samples.

A plurality of different classification models are available corresponding to a plurality of different label systems.

When a plurality of classification models are constructed for a plurality of label systems, different classification models may be constructed based on the same or different algorithms, for example, if two label systems exist, two classification models may be constructed based on a CNN (Convolutional neural networks) or based on the same conventional machine learning method, or one classification model may be constructed based on a CNN, and another classification model is constructed based on the conventional machine learning method, and the like, which is not limited in this embodiment.

In addition, optionally, in a specific implementation, the number of label systems may be pre-specified according to task requirements, referring to a processing logic schematic diagram shown in fig. 5, where the processing logic schematic diagram performs model training in an early stage (off-line) and performs intent classification by using a model in a later stage (on-line) and implements answer recall, assuming that the customer service robot is required by the current task to recall TOP 3 (at this time, n is 3 in fig. 5) answers to the user question, in an implementation, the number of label systems may be specified as 3, and each to-be-classified label corresponding to the task is specifically divided into 3 label systems, so that 3 classification models are trained correspondingly corresponding to the 3 label systems, and subsequently, after the models are put into application, for the target question, 3 intent classes can be identified by using the 3 classification models, and then 3 target answers corresponding to each intent class are recalled from the answer library, i.e., TOP 3 answer, is ultimately fed back to the user.

In the embodiment, different types of labels with cross relationship are separately divided into different label systems, so that a plurality of labels in the same label system have mutual exclusion relationship, and further a plurality of corresponding classification models are constructed aiming at a plurality of label systems, so that each classification model can be ensured to be classified, the different category labels of the model do not have great influence on the model processing result of the category corresponding to the opposite side, the classification accuracy of each model is ensured, the misjudgment of the model is not caused, the intention categories identified by each model are finally integrated and the answer is recalled, the answer corresponding to the different category labels with the cross relationship (the real label of the problem is assumed to be one of a plurality of labels with the cross relationship) can be brought into the TOP N answer, therefore, the real answer of the target question can be covered to a higher degree, and the accuracy of the intelligent robot in answer recall is improved.

Next, a description is given of an implementation process for determining an association relationship between different tags based on a model actual measurement method.

Referring to fig. 6, in this embodiment, the association relationship between different types of tags may be determined through the following model actual measurement processing procedure:

step 601, constructing a first model corresponding to a plurality of category labels to be divided.

Specifically, a corresponding data set may be first extracted and divided into a first data set and a second data set, wherein the first data set is used for model training and the second data set is used for comparison testing of model performance.

The first data set may then be used to construct a first model corresponding to the category labels to be classified, using a suitable algorithm, such as CCN or some conventional machine learning algorithm, depending on the task requirements.

For example, assume that the plurality of category labels to be classified are: and if the mailbox is not logged in by the external network, the mobile phone cannot be started, the mailbox is not logged in by the internal network, the mobile phone is refreshed and the mobile phone software is installed, a first model corresponding to the 5 category labels can be constructed by utilizing the first data set.

Step 602, one first class label is removed from the plurality of class labels to be classified, and a second model corresponding to the plurality of class labels to be classified which are not removed is constructed.

A certain category label can be removed from a plurality of category labels to be divided based on a set strategy, wherein the removed category label is called a first category label, for example, a first label in the labels is selected in sequence, that is, the first label is "no mailbox is logged in to the external network", and is removed (of course, other modes, such as randomly selecting a certain label and removing the certain label, and the like, can also be adopted).

Then, based on the data samples corresponding to the plurality of to-be-classified labels which are not removed in the first data set, a second model corresponding to the plurality of to-be-classified labels which are not removed is constructed, for example, a second model corresponding to other 4 labels which are left after the "external network login is not in a mailbox" is constructed.

Step 603, determining a second class label from the plurality of class labels to be classified which are not removed; the second category label is: and the classification performance in the second model is improved compared with the classification performance in the first model, and the class label of which the improvement amplitude meets the amplitude condition is improved.

Compared with the first model, the second model defaults to the rejected class label, such as the above-mentioned "no mail box is logged in to the extranet", after the first model and the second model are obtained, the two models can be respectively subjected to performance tests by using the second data set, so as to determine the classification accuracy of each model corresponding to each class label.

For example, assume that after performing a performance test on the first model corresponding to the above 5 labels by using the second data set, the classification accuracy of each class label is known as:

the external network logs in the mailbox: x 1%;

the mobile phone can not be started: x 2%;

the intranet logs in the mailbox without logging in: x 3%;

the mobile phone is refreshed: x 4%;

mobile phone software installation: x 5%.

After the second data set is used for carrying out performance test on the second model corresponding to the 4 labels, the classification accuracy of each class label is obtained as

The mobile phone can not be started: y 2%;

the intranet logs in the mailbox without logging in: y 3%;

the mobile phone is refreshed: y 4%;

mobile phone software installation: y 5%.

The two models will have a tag difference, so the classification performance corresponding to the same class tag will usually vary, and as described above, x 2% is usually different from y 2%, x 3% is usually different from y 3%. By comparing the performance variation range of the same type of labels (such as 'intranet login no mail box') in the two models, the influence of the removed labels, such as 'extranet login no mail box', on the model processing result of the labels in the model classification process (such as 'intranet login no mail box') can be determined.

Through experimental observation, it can be known that, in the classification performance of the first model and the second model, the classification performance of the label "no mail box is logged in by an intranet" is improved, the improvement range is large, and the classification performance of other categories (the mobile phone cannot be started, the mobile phone is refreshed, and the mobile phone software is installed) is small in change. Therefore, it can be determined that the removed label 'external network can not log in mailbox' can generate great influence on the model processing result of the label 'internal network can not log in mailbox'.

In this embodiment, the class label, in which the classification performance in the second model is improved compared with the classification performance in the first model and the improvement amplitude satisfies the amplitude condition, is determined as the second class label.

Wherein the amplitude condition may be: the lifting amplitude reaches a set amplitude threshold value, such as an amplitude threshold value of 15% or 20%, and the like.

Step 604, determining that the association relationship between the second category label and the first category label is: and the cross relation of the mutual exclusion condition is not satisfied.

In addition, in this embodiment, the determined second category label whose classification performance is improved and whose improvement amplitude satisfies the amplitude condition is further determined as having a cross relationship with the removed first category label, that is, the relationship between the first category label and the second category label is a cross relationship that does not satisfy the mutual exclusion condition.

For example, according to the comparison test of performance, when the magnitude of the classification accuracy improvement of the label "intranet login no-mailbox" in the second model reaches the magnitude threshold, the label "intranet login no-mailbox" and the removed label "extranet login no-mailbox" may be determined to have a cross relationship.

Step 605, determining that the correlation between the labels of the plurality of non-removed to-be-classified labels, except the second class label, and the first class label is: and meeting the mutual exclusion relation of the mutual exclusion condition.

Correspondingly, if the labels except the second label in the plurality of labels to be classified which are not removed do not meet the amplitude condition, determining the relationship between the labels and the first label to be removed as follows: and meeting the mutual exclusion relation of the mutual exclusion condition.

In practical implementation, after a certain category of tags are removed and the association relationship between the remaining non-removed category of tags and the removed tags is determined, a next round of processing may be started, that is, another category of tags is removed from the category of tags to be divided (for example, the 5 tags to be divided), for example, "the mobile phone cannot be turned on" is removed from the 5 tags, and the remaining 4 tags are obtained: the method comprises the following steps of 'no mail box login on an external network', 'no mail box login on an internal network', 'mobile phone flashing' and 'mobile phone software installation'.

Then, based on the remaining 4 labels in the current round, the processing procedure is circulated, so that the incidence relation between the label 'the mobile phone cannot be started up' which is removed in the current round and the remaining 4 labels is obtained; in the specific implementation, the above processing procedure may be repeated for a plurality of times until the association relationship between any two different tags in each category tag to be divided is determined, and of course, in the specific implementation, only a small number of cycles may be executed in combination with various considerations such as the actual performance requirement of the model and the processing complexity requirement, that is, the association relationship between some (but not all) of the different tags in each category tag to be divided is determined, which can also achieve the purpose of improving the answer recall accuracy of the customer service robot.

In this embodiment, based on a model actual measurement method, through actually observing the influence of the class labels on the classification performance of the model, the incidence relation between different labels is determined, and it can be more accurately determined whether the different class labels are in a mutual exclusion relation or a cross relation, so as to provide support for accurate division of the label system, further improve the coverage of a plurality of classification models corresponding to a plurality of label systems to the real intention, and correspondingly further improve the answer recall accuracy of the customer service robot based on the plurality of models.

The following provides a preferred implementation procedure for performing systematic division on different types of tags in the processing method of the present application. As shown in fig. 7, the classification of different types of tags in the to-be-classified tags into corresponding tag systems can be realized through the following processing procedures:

step 701, traversing each class label, and dividing the currently traversed class label into label systems with the time length of the unassigned label reaching the time length condition until each class label is divided into the corresponding label systems.

The duration condition may be, but is not limited to: the length of time that the label is not allocated is longest, or the length of time that the label is not allocated reaches a set length threshold value, and the like.

Next, a process of traversing each category label and assigning it to a corresponding label hierarchy will be described, taking as an example that the time length condition is set such that the time length for which no label is assigned is the longest.

As an optional implementation manner, a plurality of label systems with empty information in a specified number may be first constructed, a label system with the longest duration to which a label is not allocated at a certain time is determined based on a timing means, and then each traversed category label is allocated to the label system with the longest duration to which a label is not allocated at a corresponding time until each label is allocated to the corresponding label system; if a plurality of label systems with the same time length without labels distributed exist when traversing to a certain type of labels, the labels are selected from the label systems and put into the label systems.

As another optional implementation manner, label allocation may be performed according to a depth-first manner to implement label hierarchy partitioning, and the processing procedure is as follows: and in the t step, the searched labels are put into the t% n label systems, and when one label is searched, t +1 correspondingly puts the labels into the (t + 1)% n label systems until all the labels are divided into the corresponding label systems, wherein n is the number of the specified label systems.

Through t% n (or (t + 1)% n, etc.), the label system with the longest time length of not being allocated with the label when traversing to the category label can be indirectly determined, and the traversed label is put into the label system.

Taking the category labels to be divided to include four labels a, b, c, and d, and specifying two label systems as an example, in this embodiment, an association relationship matrix is constructed for each label in advance according to the association relationship between the labels, as shown in fig. 8, where 0 represents that the two category labels are in a mutual exclusion relationship, 1 represents that the two category labels are in a cross relationship, and 2 represents that the category labels are in a relationship with itself.

Based on the depth-first manner described above, the class labels with a value of 2 in the matrix may be traversed first, where initially t is 0, when searching for the class label with the corresponding association relationship value of 2, a is first searched, and according to t% n, it may be placed in label system 0, and then t is 1, b is continuously searched, and according to t% n, it is correspondingly placed in label system 1 … until the last label d is placed in label system 1, at this time, the labels in label system 0 include a and c, and the labels in label system 1 include b and d.

Step 702, determining whether all the category labels which do not meet the mutual exclusion condition are divided in the same label system.

After each traversed label is distributed to the corresponding label system based on the mode, it is difficult to ensure that different labels in the same label system have mutual exclusion relation.

Therefore, whether the labels which do not meet the mutual exclusion condition, namely have the cross relationship, are divided in the same label system can be further determined. For example, one can continue to traverse the category labels c, d in FIG. 8 whose corresponding association value is 1 to determine if they are classified in the same label hierarchy.

And 703, if yes, separating different types of tags which do not meet the mutual exclusion condition in the same tag system into different tag systems.

If it is determined that the different category labels which do not satisfy the mutual exclusion condition, that is, have the cross relationship, are divided into different label systems, the system division requirements of the scheme of the application on the category labels are satisfied, and the division process of the category labels can be finished. For example, for the association relationship between the labels shown in fig. 8, the labels c and d having the cross relationship are divided into different label systems, and the classification process of the category label can be finished correspondingly when the system classification requirement of the category label is satisfied.

On the contrary, if it is determined that the different types of labels having the cross relationship are classified into the same label system, the labels need to be separated into different label systems, for example, assuming that the relationship among a, b, c, and d is changed to the label relationship shown in fig. 9, for the labels a and c, since the two have the cross relationship, the labels a and c need to be separated, specifically, the labels a or c can be selected to be classified into the label system 1. It should be noted that after a and c are separated into different label systems, the number of labels in the label system 0 is small, and based on the principle of label number balance in each label system, label adjustment may also be performed on the label system, for example, the labels b or d in the label system 1 may be divided into the label system 0, so that the number of labels in each label system is balanced.

In this embodiment, by dividing the class labels in the direct timing manner or the depth-first manner, not only different class labels having a cross relationship can be divided into different label systems, but also the number of labels in each label system can be ensured to be relatively balanced, and the problem that the model volumes of different trained models are not balanced enough and the model performances (such as the classification speeds) are not uniform correspondingly due to excessive or insufficient number of labels in a certain/some label systems can be avoided.

Corresponding to the processing method, the application also discloses a computer device, which can be, but is not limited to, a portable terminal device such as a smart phone, a personal digital assistant, a tablet computer and the like, or can also be a portable computer (such as a notebook), a desktop computer or a large and medium-sized computer, a background server and the like in a general/special computing or configuration environment.

Referring to fig. 10, a schematic diagram of a computer device is shown, the computer device at least comprising:

a memory 1001 for storing at least one set of instructions;

a processor 1002, configured to call and execute the set of instructions in the memory, and by executing the set of instructions, perform the following operations:

acquiring a target problem;

The processor 1002 may obtain the target question, which may be a question to be solved input by a natural person user or an electronic device, for example, obtain a question to be solved input by a natural person user through voice or text entry, for example, "how is a mailbox logged in? "and the like.

It can be known from the above solutions that, in the computer device of this embodiment, the class labels are classified in advance based on the relationship between the class labels of the models, and a plurality of classification models are constructed according to the multi-label system obtained by the classification, wherein each class label included in the same label system satisfies the mutual exclusion relationship, and accordingly, for the constructed classification model corresponding to the label system, the influence that the processing result of the model corresponding to the other party does not exceed the predetermined influence limit value is not generated between the class labels satisfying the mutual exclusion condition, so that when the classification model is used to classify the target problem, it can be ensured that the model can provide a more accurate prediction result for each class label corresponding to the model in the label category of the model, and finally, the answer is recalled by synthesizing the more accurate model prediction result of the multi-model, so as to achieve the purpose of high coverage of the real answer, the real answer can be covered more accurately, and the accuracy of answer recall is improved.

In an alternative embodiment of the present application, the processor 1002 may further determine at least one target answer matching the target question through the following processing procedure:

determining category labels of various categories to which the target problem belongs based on classification processing results of various classification models; determining at least one target class label from the class labels of the various classes; and determining at least one answer corresponding to the at least one target category label as the at least one target answer.

The following examples are given.

When the processor 1002 of the computer device is used to process the target question to realize answer recall, it is necessary to base on a plurality of classification models which are constructed in advance, and therefore, in an alternative embodiment of the present application, the processor 1002 is further configured to perform the following processes before acquiring the target question:

a plurality of classification models are constructed in advance.

The following describes a process of constructing the plurality of classification models by the processor 1002, wherein the processor 1002 may implement the construction of the plurality of classification models by the following processes:

determining a plurality of category labels to be classified; determining the incidence relation among different category labels in a plurality of category labels to be divided; if the association relation among the different types of tags is the cross relation which does not meet the mutual exclusion condition, dividing the different types of tags into different tag systems respectively; if the association relation among the different types of tags is the mutual exclusion relation meeting the mutual exclusion condition, dividing the different types of tags into the same or different tag systems; respectively constructing different classification models aiming at different label systems obtained by division; wherein, each class label that the classification model corresponds includes: and the classification model corresponds to each class label provided by the label system.

When a plurality of classification models are constructed for a plurality of label systems, different classification models may be constructed based on the same or different algorithms, for example, if two label systems exist, two classification models may be constructed based on the CNN or based on the same conventional machine learning method, or one classification model may be constructed based on the CNN, and another classification model is constructed based on the conventional machine learning method, and the like, which is not limited in this embodiment.

Next, a description will be given of an implementation process in which the processor 1002 of the computer device determines the association between different tags based on a model measurement method.

The processor 1002 may specifically determine the association relationship between different types of tags through the following model actual measurement processing procedure:

constructing a first model corresponding to the plurality of category labels to be classified; removing a first class label from the plurality of class labels to be classified, and constructing a second model corresponding to the plurality of class labels to be classified which are not removed; determining a second class label from the plurality of class labels to be classified which are not removed; the second category label is: the classification performance in the second model is improved compared with the classification performance in the first model, and the improvement amplitude meets the class label of an amplitude condition; determining the association relationship between the second category label and the first category label as: a cross-relation that does not satisfy the mutual exclusion condition; determining the relationship between the labels of other labels except the second class label in the plurality of labels of classes to be classified which are not removed and the first class label as follows: and meeting the mutual exclusion relation of the mutual exclusion condition.

the external network logs in the mailbox: x 1%;

the mobile phone can not be started: x 2%;

the intranet logs in the mailbox without logging in: x 3%;

the mobile phone is refreshed: x 4%;

mobile phone software installation: x 5%.

The mobile phone can not be started: y 2%;

the intranet logs in the mailbox without logging in: y 3%;

the mobile phone is refreshed: y 4%;

mobile phone software installation: y 5%.

One preferred implementation of the hierarchical partitioning of different classes of tags by processor 1002 in the computer device of the present application is provided below. The processor 1002 may implement the following processing procedures to divide different types of tags in the to-be-divided type tags into corresponding tag systems:

traversing each class label, and dividing the currently traversed class label into label systems with the time length of the label not allocated to reach the time length condition until each class label is divided into the corresponding label system; determining whether all the class labels which do not meet the mutual exclusion condition are divided in the same label system; if yes, separating different types of labels which do not meet the mutual exclusion condition in the same label system into different label systems.

It should be noted that, in the present specification, the embodiments are all described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments may be referred to each other.

For convenience of description, the above system or apparatus is described as being divided into various modules or units by function, respectively. Of course, the functionality of the units may be implemented in one or more software and/or hardware when implementing the present application.

From the above description of the embodiments, it is clear to those skilled in the art that the present application can be implemented by software plus necessary general hardware platform. Based on such understanding, the technical solutions of the present application may be essentially or partially implemented in the form of a software product, which may be stored in a storage medium, such as a ROM/RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method according to the embodiments or some parts of the embodiments of the present application.

Finally, it is further noted that, herein, relational terms such as first, second, third, fourth, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.

The foregoing is only a preferred embodiment of the present application and it should be noted that those skilled in the art can make several improvements and modifications without departing from the principle of the present application, and these improvements and modifications should also be considered as the protection scope of the present application.

Claims

1. A method of processing, comprising:

acquiring a target problem;

2. The method of claim 1, further comprising: pre-constructing the plurality of classification models;

determining a plurality of category labels to be classified;

3. The method of claim 2, the determining an association between different ones of the plurality of category labels to be partitioned, comprising:

4. The method of claim 2, the determining an association between different ones of the plurality of category labels to be partitioned, comprising:

5. The method of claim 3 or 4, wherein the partitioning of different categories of tags into respective tag hierarchies comprises:

6. The method of claim 1, the determining at least one target answer that matches the target question based on the classification processing result, comprising:

7. A computer device, comprising:

a memory for storing at least one set of instructions;

acquiring a target problem;

8. The computer device of claim 7, the processor further to: pre-constructing the plurality of classification models;

determining a plurality of category labels to be classified;

9. The computer device of claim 8, the processor determining an association between different ones of the plurality of category labels to be classified, comprising:

alternatively, the first and second electrodes may be,

10. The computer device of claim 7, the processor determining at least one target answer matching the target question based on the classification processing result, comprising: