CN111309715A

CN111309715A - Call scene identification method and device

Info

Publication number: CN111309715A
Application number: CN202010042390.4A
Authority: CN
Inventors: 付荑曼
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2020-01-15
Filing date: 2020-01-15
Publication date: 2020-06-19
Anticipated expiration: 2040-01-15
Also published as: CN111309715B

Abstract

The application relates to a method and a device for identifying a call scene, wherein the method comprises the following steps: acquiring a marked call set and an unmarked call set belonging to a first scene from a call set; determining a reliable negative sample which does not belong to a first scene from the unlabeled call set according to a first classifier obtained based on the call features of the labeled call set and the unlabeled call set; performing classification training according to the call characteristics of the marked call set and the reliable negative sample to obtain a first scene recognition model; a first subset of calls belonging to a first scenario and a second subset of calls not belonging to the first scenario are identified from the set of calls by a first scenario identification model. The scheme provided by the application can realize the recognition of the call scene.

Description

Call scene identification method and device

Technical Field

The present application relates to the field of computer technologies, and in particular, to a method and an apparatus for recognizing a call scenario, a computer-readable storage medium, and a computer device.

Background

With the rapid development of internet technology and intelligent terminals, various instant messaging tools have become popular and become a common communication mode among users. The user can establish a call group through the instant messaging tool, and multi-person communication among group members can be realized through the call group.

The scenes that a majority of users initiate multi-person calls in the call group can be roughly divided into a working scene, a chatting scene and other scenes, and the number of the initiated multi-person calls belonging to different call scenes can provide guidance for the functional design of the instant messaging tool. For example, besides supporting multi-person calls, the call group can provide functions of red envelope giving, expression giving, file giving, business card giving and the like, and the number of multi-person calls in different call scenes can provide reference for the development sequence of the functions. However, how to identify a scenario of a multi-person call initiated in a call group does not exist in the prior art.

Disclosure of Invention

Based on this, it is necessary to provide a call scenario identification method, apparatus, computer-readable storage medium, and computer device for solving the problem that a scenario of a multi-person call initiated in a call group cannot be identified in the prior art.

A method for recognizing call scenes comprises the following steps:

acquiring a marked call set and an unmarked call set belonging to a first scene from a call set;

determining a reliable negative sample which does not belong to a first scene from the unmarked call set according to a first classifier obtained based on the call features of the marked call set and the unmarked call set;

performing classification training according to the call characteristics of the labeled call set and the reliable negative sample to obtain a first scene recognition model;

a first subset of calls belonging to a first scenario and a second subset of calls not belonging to the first scenario are identified from the set of calls by the first scenario identification model.

An apparatus for identifying a call scenario, the apparatus comprising:

the acquisition module is used for acquiring a marked call set and an unmarked call set which belong to a first scene from the call set;

the reliable negative sample determining module is used for determining a reliable negative sample which does not belong to a first scene from the unlabeled call set according to a first classifier obtained based on the call features of the labeled call set and the unlabeled call set;

the training module is used for carrying out classification training according to the call characteristics of the labeling call set and the reliable negative sample to obtain a first scene recognition model;

and the identification module is used for identifying a first call subset belonging to a first scene and a second call subset not belonging to the first scene from the call set through the first scene identification model.

A computer-readable storage medium, storing a computer program which, when executed by a processor, causes the processor to perform the steps of a method of identifying a call scenario.

A computer device comprising a memory and a processor, the memory storing a computer program which, when executed by the processor, causes the processor to perform the steps of a method of identifying a call scenario.

The method, the device, the computer readable storage medium and the computer equipment for recognizing the call scenes are characterized in that after a small number of marked call sets belonging to a first scene are determined from a call set, an unlabelled call set is formed according to the calls left in the call set, then training is carried out according to the call characteristics of each call in the current marked call set and the unlabelled call set to obtain a first classifier, a reliable negative sample which does not belong to the first scene is determined from the unlabelled call set according to the first classifier, then classification training is carried out according to the call characteristics of each call in the marked call set and the reliable negative sample to obtain a first scene recognition model which can recognize whether the call scene is the first scene, so that a first call subset belonging to the first scene and a second call subset not belonging to the first scene can be recognized from the call set through the first scene recognition model, the recognition of the call scene is realized.

Drawings

FIG. 1 is a diagram of an exemplary embodiment of a call scenario recognition method;

FIG. 2 is a block diagram of a method for identifying a call scenario in one embodiment;

FIG. 3 is a schematic diagram of feature engineering set up for a call in one embodiment;

FIG. 4 is a flowchart illustrating a method for identifying a call scenario in one embodiment;

FIG. 5 is a diagram illustrating a logarithmic transformation of feature data in one embodiment;

FIG. 6 is a flow diagram illustrating the determination of reliable negative examples that do not belong to the first scenario in one embodiment;

FIG. 7 is a flow diagram illustrating the process of identifying calls belonging to a first scenario in an exemplary embodiment;

FIG. 8 is a flow diagram that illustrates the identification of a call belonging to a second scenario from a call set, in one embodiment;

FIG. 9 is a flow diagram illustrating the determination of reliable negative examples that do not belong to the second scenario in one embodiment;

FIG. 10 is a flow diagram illustrating the identification of a call belonging to a second scenario in an exemplary embodiment;

FIG. 11 is a general flow diagram illustrating a call scenario for identifying individual calls in a call set, according to one embodiment;

FIG. 12 is a block diagram showing the structure of a call scene recognition apparatus according to an embodiment;

FIG. 13 is a block diagram of a computer device in one embodiment.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.

Fig. 1 is an application environment diagram of a call scenario recognition method in one embodiment. Referring to fig. 1, the method for identifying a call scenario is applied to a system for identifying a call scenario. The call scene recognition system includes a terminal 110 and a server 120. The terminal 110 and the server 120 are connected through a network. The terminal 110 may specifically be a desktop terminal or a mobile terminal, and the mobile terminal may specifically be at least one of a mobile phone, a tablet computer, a notebook computer, and the like. The server 120 may be implemented as a stand-alone server or a server cluster composed of a plurality of servers.

In one embodiment, the terminal 110 runs an instant messaging client, and a user can establish a talk group through the instant messaging client and initiate a multi-user instant messaging session in the talk group. The talk group may be at least one of a discussion group, a multi-person video talk group, and a multi-person audio talk group. The server 120 may provide support for an instant messaging service provided by an instant messaging client, and the server 120 may further record feature data corresponding to a multi-person call initiated by a user on the terminal 110, and obtain a call feature corresponding to each call according to the feature data.

In one embodiment, the server may obtain calls of a large number of users within a period of time as a call set, count feature data of each call in the call set within the period of time, and obtain call features corresponding to each call in the call set according to the feature data. The aforementioned period of time may be one week, one month, three months, etc., and may be selected according to specific needs. Further, the server can also acquire a marked call set and an unmarked call set belonging to the first scene from the call set; determining a reliable negative sample which does not belong to a first scene from the unlabeled call set according to a first classifier obtained based on the call features of the labeled call set and the unlabeled call set; performing classification training according to the call characteristics of the marked call set and the reliable negative sample to obtain a first scene recognition model; a first subset of calls belonging to a first scenario and a second subset of calls not belonging to the first scenario are identified from the set of calls by a first scenario identification model.

The call scene is a use scene of an initiated multi-person call, and can be divided into a work scene, a chat scene and other scenes. The work scenario is generally a usage scenario of a multi-person call initiated for efficient communication purposes, such as a work conference, an online course, an online discussion, an online training, and so on. The chat scene is generally a use scene of a multi-person call initiated for emotional communication, such as family communication, friend chat, friend-friend making, interactive entertainment, and the like. Other scenarios are usage scenarios of multi-person calls other than the two call scenarios, in which the activity of the call group in which the initiated multi-person call is located is low, or the call group in which the initiated multi-person call is located is a temporary call group, and may be defined as an accidentally initiated multi-person call. According to the method for identifying the call scene in the embodiment of the application, each call in the call set can be identified as belonging to one of a working scene, a chatting scene or other scenes.

The server can identify the call scene of each call according to the call characteristics of each call in the obtained call set. Of course, in some embodiments, the terminal may also obtain a marked call set and an unmarked call set belonging to the first scenario from the call set; determining a reliable negative sample which does not belong to a first scene from the unlabeled call set according to a first classifier obtained based on the call features of the labeled call set and the unlabeled call set; performing classification training according to the call characteristics of the marked call set and the reliable negative sample to obtain a first scene recognition model; a first subset of calls belonging to a first scenario and a second subset of calls not belonging to the first scenario are identified from the set of calls by a first scenario identification model.

Fig. 2 is a schematic diagram of a method for identifying a call scenario in an embodiment. Referring to fig. 3, the method mainly comprises four steps, wherein the first step is feature selection, namely, a feature project is set up, and feature data of each call in a call set is obtained, wherein the feature data comprises call duration features, call user portrait features, features of a call group where the call is located, and other behavior features during the call; the second step is data cleaning, namely preprocessing the characteristic data of each call in the call set to obtain call characteristics, including performing principal component analysis on the characteristic data and extracting principal component characteristics, and also including performing logarithmic transformation on the characteristic data to obtain uniformly distributed characteristic data; the third step is model training, which is used for selecting proper positive samples and negative samples from the call set, determining reliable negative samples and diffusing the positive samples in the selection process, and then performing model iterative training according to the determined positive samples and negative samples to obtain a call scene identification model for identifying call scenes; and the fourth step is total prediction, namely predicting and verifying all calls in the call set by using a trained call scene recognition model so as to classify the call scenes of all calls in the call set.

As shown in fig. 3, which is a schematic diagram of a feature engineering set up for multi-person conversation in an embodiment, the feature engineering takes multi-person conversation initiated each time as a statistical object, and divides characteristic data of conversation into four categories, including conversation duration characteristics, conversation user portrait characteristics, characteristics of a conversation group where the conversation is located, and other behavior characteristics during the conversation.

The call duration characteristics are characteristics related to call duration of a multi-person call initiated within a period of time, and include whether the call is a workday, a call hour, a call number, a call longest duration, a call duration variance, a call duration mean, a call duration sum, a video longest duration, a video duration median, a video duration mean, an audio longest duration, an audio duration median, an audio duration mean, an audio duration sum, and the like. It will be appreciated that since the call duration characteristic is statistically derived from the durations of a plurality of calls initiated over a period of time, the data in the dimension of the call duration characteristic is the same for each call initiated over the period of time.

The call user profile feature is a profile feature of a user participating in a multi-person call initiated at the current time, and includes the number of male users participating in the multi-person call at the current time, the male user ratio, the number of users using the iOS system, the user ratio using the iOS system, the number of users using WiFi, the user ratio using WiFi, the number of users having location information of a first-line city, the number of users having location information of a second-line city, the number of users having location information of a third-line city, the user ratio of a first-line city, the user ratio of a second-line city, the user ratio of a third-line city, the maximum age of a participating call member, the minimum age of a participating call member, the average age of a participating call member, and the. It will be appreciated that the call user profile feature is a feature that is personalized to each call, with the data for each call differing in the dimension of the call user profile feature.

The call group in which the call is located is characterized by feature data of the call group in which a multi-person call initiated in a period of time belongs in the period of time, and mainly comprises the number of text messages sent in the call group in the period of time, the number of picture messages sent, the number of audio messages sent, the number of business card messages sent, the number of red packets sent, the number of text messages sent in working days, the number of picture messages sent in working days, the number of audio messages sent in working days, the number of business card messages sent in working days, the number of text messages sent in working time, the number of picture messages sent in working time, the number of audio messages sent in working time, the number of business card messages sent in working time, the number of video messages sent in working time and the like. It can be understood that, since the feature of the talk group in which the talk is located describes the feature of the talk group in which the talk is located, the feature data of the talk initiated in the same talk group in the feature dimension of the talk group is the same in this period of time.

Other behavior characteristics during the call are characteristics counted according to the user behavior in the current call process, and include the number of text messages sent during the call, the number of picture messages sent during the call, the number of audio messages sent during the call, the number of business card messages sent during the call, the number of video messages sent during the call, the minimum click or click of clicking during the call, the mute click or click of clicking during the call, the invitation of a new member to join the call during the call, and the like. It will be appreciated that the other behavioral characteristics during the call are personalized characteristics for each call, and that the data for each call in the dimensions of the other behavioral characteristics during the call are different.

From the established feature engineering, the feature data corresponding to each call specifically includes feature data on a plurality of feature dimensions, including feature data with commonality and personalized feature data, and the feature dimensions can be divided into discrete features and continuous features according to different values of the features. In order to better represent each call by using the feature data, the computer device may perform preprocessing on the feature data of each call, obtain call features of each call, and implement recognition of a call scene based on the call features.

As shown in fig. 4, in one embodiment, a method for identifying a call scenario is provided. The embodiment is mainly illustrated by applying the method to a computer device (such as the terminal 110 or the server 120 in fig. 1). Referring to fig. 4, the method for identifying a call scenario specifically includes the following steps:

s402, a marked call set and an unmarked call set belonging to a first scene are obtained from the call set.

The call set is a set formed by a plurality of calls, and the computer equipment can obtain the call set according to a plurality of calls initiated in the instant messaging client within a period of time by a user and obtain the call characteristics corresponding to each call in the call set. The computer device needs to obtain a scene recognition model for recognizing a call scene, and needs to obtain a positive sample and a negative sample for training the scene recognition model, so that the computer device can divide a call set into a marked call set and an unmarked call set belonging to a first scene.

Specifically, the computer device may obtain calls belonging to the first scene from the call set, and obtain a labeled call set belonging to the first scene after labeling the calls as belonging to the first scene, where the labeled call set is an initial positive sample used for training the first scene recognition model, and the initial positive sample is less in number, and is directly obtained from the call set, and is not necessarily completely accurate, and may perform diffusion and correction in a subsequent iteration process. The negative sample used for training the first scene recognition model in the call set is missing, and the call which does not belong to the first scene cannot be determined from the call set, so that the computer device can divide the call which is not divided into the labeled call set in the call set into the unlabeled call set, wherein the unlabeled call set may include the call which belongs to the first scene, and may also include the call which does not belong to the first scene.

In one embodiment, the step of the computer device obtaining the marked call set marked as the first scene from the call set comprises: acquiring a first preset word corresponding to a first scene; acquiring a group name of a call group to which each call in a call set belongs; screening out a call group with the group name matched with a first preset word; and obtaining a marked call set belonging to the first scene according to the call initiated in the screened call group.

In this embodiment, the marked call set belonging to the first scenario may be determined according to the group name of the call group where the call is initiated. Specifically, the computer device may set a first preset word corresponding to a first scene, for example, the first scene is a work scene, and the first preset word may be at least one of words such as "work", "company", "meeting", and the like. For each call in the call set, the computer device may obtain a group name of a call group to which each call belongs, screen out a call group whose group name matches a set first preset word from the call groups, and mark a call initiated in the screened call group as a call belonging to a first scene, thereby obtaining a marked call set belonging to the first scene. It can be seen that the marked call set belonging to the first scenario may include multiple calls initiated in the same call group.

S404, according to a first classifier obtained based on the call features of the marked call set and the unmarked call set, determining a reliable negative sample which does not belong to the first scene from the unmarked call set.

As mentioned earlier, the call feature is generated from feature data generated by the user based on various operations performed by the instant messaging client when initiating a multi-person call, the feature data including feature data in various feature dimensions. The computer equipment can acquire the feature data corresponding to each call in the call set according to the set feature dimensions, and preprocess the feature data to acquire the call features corresponding to each call. And the computer equipment trains a second classifier based on the call characteristics of all calls in the marked call set and the call characteristics of all calls in the unmarked call set to obtain a first classifier, and then adopts the trained first classifier to identify according to the call characteristics of all calls in the unmarked call set, so as to determine a reliable negative sample which does not belong to the first scene. The second classifier may be a naive bayes classifier or a classifier based on XGBoost (eXtreme Gradient Boosting model).

In one embodiment, the step of the computer device pre-processing the feature data comprises: carrying out normality inspection on the feature data under the continuous feature dimension; and when the test result indicates that the feature data do not conform to normal distribution, carrying out logarithmic transformation on the feature data under the feature dimensionality to obtain the transformed feature data.

The continuous characteristic dimension refers to a value of the characteristic being a continuous value, for example, a value of "the number of text messages sent during a call" may be any value, and the characteristic dimension is a continuous characteristic dimension. Specifically, the computer device may obtain feature data of each call in the call set in the continuous feature dimension, perform a normality test on the feature data, and when a test result indicates that the feature data does not conform to a normal distribution or that the data has a severe inclination, the computer device may perform a logarithmic transformation on the feature data of each call in the continuous feature dimension to obtain transformed feature data.

In one embodiment, the computer device may perform a logarithmic transformation on the feature data according to the following formula:

x' ═ ln (1+ θ X), where X is^’The feature data after transformation is represented, X represents the feature data before transformation, and θ is a transformation parameter and is usually set to 1.

Fig. 5 is a schematic diagram of transformed feature data obtained by performing logarithmic transformation on data in feature dimensions, such as a call duration mean value calltime _ mean, a number of text messages msg _ text, and a video duration mean value video _ mean, in one embodiment. It can be seen that after logarithmic transformation, the data distribution is more biased to the normal distribution, and the influence of the inclination of the data distribution is reduced.

In this embodiment, by performing logarithmic transformation on the feature data, the distribution of the data can be made to approach a normal distribution and to be unrelated to the average value of the data, and the size relationship of the data itself is not changed, and the values of the originally densely distributed intervals are dispersed as much as possible, and the values of the originally dispersed intervals are aggregated as much as possible.

In one embodiment, the step of the computer device pre-processing the feature data comprises: acquiring feature data of various feature dimensions corresponding to each call in a call set; and extracting the principal component from the feature data to obtain the call feature of each call.

Specifically, principal component analysis extracts principal components from more feature dimensions through orthogonal transformation, obtains data with fewer dimensions, can eliminate mutual influence of data under different feature dimensions, can express original data as much as possible by using shorter data, and achieves dimension reduction of feature data.

In one embodiment, extracting the principal component from the feature data to obtain the call feature of each call includes: performing decentralized processing on the feature data of various feature dimensions, and calculating a covariance matrix; calculating eigenvalues of the covariance matrix and eigenvectors corresponding to the eigenvalues; obtaining a principal component matrix according to the eigenvector corresponding to the selected eigenvalue; and performing linear transformation on the characteristic data according to the principal component matrix to obtain the call characteristics of each call.

For example, the call set includes n calls, each call includes features in m feature dimensions, each feature has its own value, and there are m kinds of features in total, and data in each feature dimension is n-dimensional, that is, the original feature data is an m × n matrix a. The feature data in the i-th feature dimension is Xi ═ (Z1, Z2, …, Zn), where i is 1, 2, …, m, and the feature data in the feature dimension is subjected to a decentralization process to obtain Xi ═ Z1- μ, Z2- μ, …, Zn- μ,

then according to each feature dimension, centralizing processingThe subsequent characteristic data obtain an m × n matrix B, a covariance matrix is calculated according to the matrix B to obtain an m × m matrix C, then the eigenvalue λ of the covariance matrix C and the corresponding eigenvector P are calculated by utilizing matrix knowledge, m eigenvalues are totally obtained, the m eigenvalues are sorted according to the sequence from large to small, the first k largest eigenvalues are selected, a principal component matrix P is obtained according to the corresponding k eigenvectors, P is a k × m matrix, and finally the original characteristic data A is projected onto the principal component matrix P to obtain a k × n matrix Y, wherein the matrix Y comprises new k dimensional data after dimension reduction of each call and can be used as the call characteristics of each call.

In one embodiment, the computer device may perform a principal component analysis after performing a logarithmic transformation on the feature data in the continuous feature dimension.

In one embodiment, the step S404 of determining a reliable negative example not belonging to the first scenario from the unlabeled call set according to a first classifier obtained based on call features of the labeled call set and the unlabeled call set includes: selecting a spy sample from a marked call set belonging to a first scene; after spy samples are removed from the marked call set, training positive samples are obtained; after the spy sample is added to the call set which is not marked, a training negative sample is obtained; obtaining a first classifier according to the call characteristics of the training positive sample and the training negative sample; obtaining the classification probability of each call in the call set which is not marked through a first classifier; determining a probability threshold value for judging the call as not belonging to the first scene according to the classification probability of the spy samples; and taking the calls of which the classification probability corresponding to the call sets which are not marked is smaller than the probability threshold as reliable negative samples which do not belong to the first scene.

Specifically, the computer device may set the call set as a, the marked call set belonging to the first scenario as P, the unmarked call set as U, the reliable negative sample not belonging to the first scenario as RN, and the initial RN as an empty set. Then, randomly selecting a part of the sample P as a spy sample S according to a preset proportion (for example, 15%), and after the spy sample S is removed from the marked call set P, obtaining a training positive sample PS, wherein the PS is P-S, and the label of each call in the PS is 1; after the spy sample S is added to the unlabeled call set U, a training negative sample US is obtained, wherein the label of each call in the US is-1, and the US is U + S. Then, according to the call characteristics of each call in the PS and the US, a classification training is performed to obtain a first classifier g1, and the prediction is performed on each call in the U by the first classifier g1 to obtain a classification probability p1(d) of each call d in the U. And finally, sorting the classification probabilities of all calls in the spy sample S, taking the classification probability corresponding to the call ranked later (for example, ranked 10% later) as a probability threshold value threshold not belonging to the first scene, and adding d to RN when p1(d) < threshold, thereby obtaining a reliable negative sample not belonging to the first scene in U.

In one embodiment, the method further comprises: and after reliable negative samples which do not belong to the first scene are removed from the unlabeled call set in an iterative execution mode, obtaining an updated unlabeled call set, and determining the reliable negative samples which do not belong to the first scene from the updated unlabeled call set according to the call characteristics of the labeled call set and the updated unlabeled call set until the iteration times meet the preset number.

In this embodiment, in order to obtain more reliable negative samples that do not belong to the first scenario from the unlabeled call set, the computer device may repeat the iteration until the number of iterations satisfies a preset number or it is impossible to obtain more reliable negative samples from the unlabeled call set, for example, the iteration may be performed for 15 times. The computer equipment can remove the reliable negative samples which are determined at the previous time and do not belong to the first scene from the original unlabelled call set U to obtain an updated unlabelled call set, then carries out classification training according to the call features of all calls in the updated unlabelled call set and the labeled call set P, and determines the reliable negative samples which do not belong to the first scene from the updated unlabelled call set again through a classifier obtained through training.

Fig. 6 is a schematic flow chart of determining reliable negative examples that do not belong to the first scenario from the call set in one embodiment. Referring to fig. 6, first, a marked call set P belonging to a first scenario is obtained from the call sets, and other calls in the call sets are added to an unmarked call set U, for example, the first scenario is a working scenario, and a call whose group name of a call group in the call set includes "work" may be added to the marked call set P. And then randomly selecting calls with a preset proportion from the U as a spy sample S, and marking P-S as PS and as 1, and marking U + S as US and as-1. And then, performing classification training by using the call characteristics of each call in the PS and the US to obtain a first classifier, and obtaining the classification probability of each call through the first classifier. And finally, determining a probability threshold value serving as a negative sample according to the classification probability of each call in the spy sample S, and adding the call of which the classification probability in the U is smaller than the probability threshold value as a reliable negative sample to the RN. And repeating the process, and retraining the classifier according to the remaining unlabeled samples and the labeled call set P in the U until the iteration times reach a preset number, and dividing the original call set into the labeled call set P belonging to the first scene, the reliable negative sample RN not belonging to the first scene and the remaining unlabeled call set U at the moment.

S406, performing classification training according to the call features of the marked call set and the reliable negative sample to obtain a first scene recognition model.

Specifically, the first scene recognition model is a model having the capability of recognizing whether or not a call belongs to the first scene based on call characteristics of the call. The first scene recognition model is a classification model, and the recognition result of the call is that the call belongs to one of the first scene and the call does not belong to the first scene. After the computer device obtains the marked call set belonging to the first scene and the reliable negative sample not belonging to the first scene, classification training can be performed based on call characteristics of all calls in the set to obtain a first scene recognition model.

In one embodiment, step S406, performing classification training according to the call features of the labeled call set and the reliable negative example, to obtain a first scene recognition model, including: dividing a training set and a verification set according to the marked call set and the reliable negative sample; training a second classifier according to the call characteristics of the training set to obtain a first scene recognition model; and verifying the obtained first scene recognition model according to the call characteristics of the verification set.

Specifically, when performing classification training according to a labeled call set belonging to a first scene and a reliable negative sample not belonging to the first scene, the computer device may divide the labeled call set and the reliable negative sample into a training set and a verification set, for example, 7:3, according to a preset ratio, and then train the second classifier using call features corresponding to calls belonging to the first scene and calls not belonging to the first scene in the training set to obtain a first scene recognition model. The second classifier may be a naive bayes classifier or an XGBoost based classifier.

The computer device can also verify the accuracy of the first scene recognition model obtained by training by using the call characteristics corresponding to the calls belonging to the first scene and the calls not belonging to the first scene in the verification set. When the verification fails, after step S408, dividing the call set into a new marked call set and an unmarked call set belonging to the first scenario according to the first call subset may be performed iteratively; determining a reliable negative sample which does not belong to a first scene from the unlabeled call set according to a first classifier obtained based on the call features of the labeled call set and the unlabeled call set; performing classification training according to the call characteristics of the marked call set and the reliable negative sample to obtain a first scene recognition model; and identifying a first call subset belonging to the first scene and a second call subset not belonging to the first scene from the call set through the first scene identification model until the first scene identification model obtained after the iteration passes the verification or the iteration number meets the preset number.

S408, a first conversation subset belonging to the first scene and a second conversation subset not belonging to the first scene are identified from the conversation set through the first scene identification model.

Specifically, after obtaining the first scene recognition model, the computer device may perform full prediction on the call set by using the obtained first scene recognition model, that is, predict the call scene to which each call in the original call set belongs, thereby determining that the first call subset belonging to the first scene in the call set does not belong to the second call subset of the first scene.

In one embodiment, identifying a first subset of calls belonging to a first scenario and a second subset of calls not belonging to the first scenario from a set of calls by a first scenario identification model comprises: acquiring call characteristics of each call in a call set; inputting call characteristics of each call into a first scene recognition model; and obtaining classification probability corresponding to each call according to the call characteristics through the first scene recognition model, and dividing the call set into a first call subset belonging to the first scene and a second call subset not belonging to the first scene according to the classification probability.

The classification probability may be used to reflect the possibility that the call belongs to the first scenario, and a higher classification probability may indicate that the call belongs to the first scenario with a higher possibility, and conversely, the call belongs to the first scenario with a lower possibility. Therefore, the computer device can identify the call scene of the call according to the classification probability obtained by the first scene identification model, divide the call with the classification probability greater than the preset threshold into a first call subset belonging to the first scene, and divide the call with the classification probability less than the preset threshold into a second call subset not belonging to the first scene.

In one embodiment, after step S408, the method further comprises: the iteration execution divides the call set into a new marked call set and an unmarked call set which belong to a first scene according to the first call subset; determining a reliable negative sample which does not belong to a first scene from the unlabeled call set according to a first classifier obtained based on the call features of the labeled call set and the unlabeled call set; performing classification training according to the call characteristics of the marked call set and the reliable negative sample to obtain a first scene recognition model; and identifying a first call subset belonging to the first scene and a second call subset not belonging to the first scene from the call set through a first scene identification model until an iteration stop condition is met.

In this embodiment, since the number of calls in the initial marked call set belonging to the first scenario obtained from the call set is small, after the call set is divided into the first call subset belonging to the first scenario and the second call subset not belonging to the first scenario by the first scenario recognition model, the computer device may use the first call subset as a new marked call set belonging to the first scenario, that is, modify and diffuse the initial marked call set belonging to the first scenario according to the prediction result of the full prediction performed by the first scenario recognition model, so as to obtain a more accurate and more reliable positive samples, and thus the call set is divided into a new call set belonging to the first scenario and an unmarked call set, and based on this, the above steps S402-S408 are iteratively performed until a new first scenario recognition model is obtained, and carrying out full prediction again by using the new first scene recognition model until the iteration number reaches a preset number or the prediction result of the first scene recognition model obtained by the last iteration is in accordance with the expectation. It can be understood that each iteration process realizes the correction and diffusion of the marked call set belonging to the first scenario.

The computer device may verify the recognition rate of the first scene recognition model obtained for the first time or during subsequent iterations. The verification method can include the following steps: the first method is to verify by using a verification set reserved during training of the first scene recognition model, namely, verify the obtained first scene recognition model by using call features of a call set and a reliable negative sample marked in the verification set, and if the recognition rate of the verification set does not meet a preset condition, continue iteration if the currently obtained first scene recognition model does not pass verification. The second is that the recognition rate of calls initiated in a call group whose group name includes the first preset word may be counted according to the full prediction result, for example, the recognition rate of calls initiated in a call group whose group name includes "training" may be counted, and the recognition rate of calls initiated in a call group whose group name includes "shipping" may be counted, and when the recognition rates of these calls do not satisfy the preset threshold, the first scene recognition model obtained currently does not meet the expectation, and the iteration may be continued. And thirdly, randomly extracting N calls, manually determining call scenes to which the N calls belong according to group names of call groups in which the N calls belong, and when the recognition rate of the N calls in the total prediction result does not meet a preset threshold value, obtaining a first scene recognition model at present, wherein the first scene recognition model does not meet the expectation, and the iteration can be continued.

The method for identifying the call scene determines a small number of marked call sets belonging to the first scene from the call sets, forming an unlabelled call set according to the remaining calls in the call set, then training according to the call characteristics of each call in the current labeled call set and the unlabelled call set to obtain a first classifier, determining reliable negative samples which do not belong to the first scene from the unlabeled call set according to the first classifier, performing classification training according to call characteristics of each call in the labeled call set and the reliable negative samples to obtain a first scene identification model capable of identifying whether the call scene is the first scene, therefore, a first call subset belonging to the first scene and a second call subset not belonging to the first scene can be identified from the call set through the first scene identification model, and the call scene is identified.

Fig. 7 is a flowchart illustrating a process of identifying a call belonging to a first scenario from a call set in a specific embodiment. Referring to fig. 7, the method specifically includes the following steps:

s702, acquiring a marked call set and an unmarked call set belonging to a first scene from a call set;

s704, selecting a spy sample from a marked call set belonging to a first scene;

s706, after spy samples are removed from the marked call set, training positive samples are obtained;

s708, after the spy sample is added to the call set which is not marked, a training negative sample is obtained;

s710, training according to the call characteristics of the training positive sample and the training negative sample to obtain a first classifier;

s712, obtaining the classification probability of each call in the call set without being labeled through the first classifier;

s714, determining a probability threshold value for judging the call as not belonging to the first scene according to the classification probability of the spy samples;

and S716, taking the calls with the classification probability smaller than the probability threshold value corresponding to the call sets without labels as reliable negative samples which do not belong to the first scene.

In order to obtain more reliable negative examples, after reliable negative examples not belonging to the first scene are removed from the unlabeled call set, an updated unlabeled call set is obtained, and S704 to S716 are iteratively performed according to the labeled call set belonging to the first scene and the updated unlabeled call set. For example, it may be iterated 15 times.

S718, dividing a training set and a verification set according to the marked call set and the reliable negative sample;

s720, training a second classifier according to the call characteristics of the training set to obtain a first scene recognition model;

s722, acquiring call characteristics of each call in the call set;

s724, inputting the call characteristics of all calls into a first scene recognition model;

and S726, obtaining classification probability corresponding to each call according to the call characteristics through the first scene recognition model, and dividing the call set into a first call subset belonging to the first scene and a second call subset not belonging to the first scene according to the classification probability.

And S728, verifying the recognition rate of the first scene recognition model.

And when the verification fails, dividing the call set into a new marked call set and an unmarked call set belonging to the first scene according to the first call subset, and iteratively executing S702 to S728. When the verification is passed, the recognition rate of the first scene recognition model is proved to be in accordance with the expectation, and at the moment, the recognition of calls belonging to the first scene and calls not belonging to the first scene from the call set is also realized.

In this embodiment, after a small number of marked call sets belonging to a first scene are determined from the call sets, an unlabeled call set is formed according to the remaining calls in the call sets, then training is performed according to the call features of each call in the current marked call set and the unlabeled call set to obtain a first classifier, a reliable negative sample not belonging to the first scene is determined from the unlabeled call set according to the first classifier, the process can be iterated for many times to obtain more reliable negative samples, then a verification set and a training set are divided according to the marked call set and the reliable negative samples, classification training is performed by using the call features of each call in the training set to obtain a first scene recognition model capable of recognizing whether the call scene is the first scene, so that the call set can be subjected to full prediction by the first scene recognition model to obtain a full prediction result, and verifying the recognition rate of the first scene recognition model according to the total prediction result or the verification set, repeating the iteration process after the initial marked call set is diffused and corrected according to the first call subset recognized to belong to the first scene when the verification fails, until the first scene recognition model obtained after the iteration is in line with the expectation, and recognizing the call belonging to the first scene from the call set.

In one embodiment, as shown in fig. 8, after identifying a first subset of calls belonging to a first scenario and a second subset of calls not belonging to the first scenario from the call set, the method for identifying a call scenario further includes a step of identifying calls belonging to a second scenario and calls not belonging to the second scenario from the remaining second subset of calls, and specifically includes the following steps S802-S808:

s802, a marked call set and an unmarked call set belonging to a second scene are obtained from the second call subset.

Specifically, the computer device needs to obtain a scene recognition model for recognizing a call scene belonging to the second scene, and needs to obtain a positive sample and a negative sample for training the second scene recognition model, so that the computer device can divide the second call subset recognized from the call set into a labeled call set and an unlabeled call set belonging to the second scene. The computer device may first obtain calls belonging to the second scene from the second call subset, and obtain a labeled call set belonging to the second scene after labeling the calls as belonging to the second scene, where the labeled call set is an initial positive sample used for training the second scene recognition model, and the initial positive sample is less in number, and is directly obtained from the second call subset, and is not necessarily completely accurate, and may be diffused and corrected in a subsequent iteration process. The negative samples used for training the second scene recognition model in the second call subset are missing, and calls not belonging to the second scene cannot be determined from the call set, so that the computer device can divide calls not divided into the labeled call set in the second call subset into the unlabeled call set, and the unlabeled call set may include calls belonging to the second scene or calls not belonging to the second scene.

In one embodiment, the step of obtaining the marked call set marked as the second scenario from the second call subset comprises: acquiring a second preset word corresponding to a second scene; acquiring the group name of a call group to which each call belongs in the second call subset; screening out a call group with the group name matched with a second preset word; and obtaining a marked call set belonging to the second scene according to the call initiated in the screened call group.

In this embodiment, the marked call set belonging to the second scenario may be determined according to the group name of the call group where the call is initiated. Specifically, the computer device may set a second preset word corresponding to a second scene, for example, the second scene is a chatting scene, and the second preset word may be at least one word of words such as "bagua", "chatting", and the like. For each call in the second call subset, the computer device may obtain a group name of a call group to which each call belongs, screen out a call group whose group name matches a set second preset word from the call groups, and label a call initiated in the screened call group as a call belonging to a second scene, thereby obtaining a labeled call set belonging to the second scene. It can be seen that the marked call set belonging to the second scenario may include multiple calls initiated in the same call group.

S804, according to a third classifier obtained based on the call features of the marked call set and the unmarked call set, reliable negative samples which do not belong to the second scene are determined from the unmarked call set.

Specifically, the computer device may obtain feature data corresponding to each call in the second call subset according to the set feature dimension, and perform preprocessing on the feature data to obtain call features corresponding to each call. And the computer equipment trains a second classifier based on the call characteristics of all calls in the marked call set and the call characteristics of all calls in the unmarked call set belonging to the second scene to obtain a third classifier, and then adopts the trained third classifier to perform recognition according to the call characteristics of all calls in the unmarked call set, so as to determine a reliable negative sample which does not belong to the second scene.

In one embodiment, determining reliable negative examples that do not belong to the second scenario from the unlabeled call set according to a third classifier obtained based on call features of the labeled call set and the unlabeled call set includes: selecting a spy sample from a marked call set belonging to a second scene; after spy samples are removed from the marked call set, training positive samples are obtained; after the spy sample is added to the call set which is not marked, a training negative sample is obtained; obtaining a third classifier according to the call characteristics of the training positive sample and the training negative sample; obtaining the classification probability of each call in the call set which is not marked through a third classifier; determining a probability threshold value for judging the call as not belonging to the second scene according to the classification probability of the spy samples; and taking the calls with the classification probability smaller than the probability threshold value corresponding to the calls in the call set which are not marked as the reliable negative samples which do not belong to the second scene.

Specifically, the computer device may set the second call subset to B, where the marked call set belonging to the second scenario is P, the unmarked call set is U, the reliable negative sample not belonging to the second scenario is RN, and the initial RN is set to be an empty set. Then, randomly selecting a part of the sample P as a spy sample S according to a preset proportion (for example, 15%), and after the spy sample S is removed from the marked call set P, obtaining a training positive sample PS, wherein the PS is P-S, and the label of each call in the PS is 1; after the spy sample S is added to the unlabeled call set U, a training negative sample US is obtained, wherein the label of each call in the US is-1, and the US is U + S. Then, according to the call characteristics of each call in the PS and the US, a classification training is performed to obtain a third classifier g3, and the prediction is performed on each call in the U by the third classifier g3 to obtain a classification probability p1(d) of each call d in the U. And finally, sorting the classification probabilities of all calls in the spy sample S, taking the classification probability corresponding to the call ranked later (for example, ranked 10% later) as a probability threshold value threshold not belonging to the second scene, and adding d to RN when p1(d) < threshold, thereby obtaining a reliable negative sample not belonging to the second scene in U.

In one embodiment, the method further comprises: and after iteratively removing the reliable negative samples which do not belong to the second scene from the second call subset, obtaining an updated unlabelled call set, and determining the reliable negative samples which do not belong to the second scene from the updated unlabelled call set according to the call characteristics of the labeled call set and the updated unlabelled call set until the iteration number meets the preset number.

In this embodiment, in order to obtain more reliable negative samples that do not belong to the second scenario from the unlabeled call set, the computer device may repeat the iteration until the number of iterations satisfies a preset number or it is impossible to obtain more reliable negative samples from the unlabeled call set, for example, the iteration may be performed for 15 times. The computer device can remove the reliable negative samples which are determined at the previous time and do not belong to the second scene from the original unlabeled call set U to obtain an updated unlabeled call set, then perform classification training according to call features of all calls in the updated unlabeled call set and the labeled call set P, and determine the reliable negative samples which do not belong to the second scene from the updated unlabeled call set again through a classifier obtained through training.

Fig. 9 is a schematic flow diagram illustrating the process of determining reliable negative examples that do not belong to the second scenario from the second communication subset in one embodiment. Referring to fig. 9, first, a marked call set P belonging to a second scenario is obtained from a second call subset, and other calls in the second call subset are added to an unmarked call set U, for example, the second scenario is a chat scenario, and calls whose group names of call groups in the second call subset include "chat" may be added to the marked call set P. And then randomly selecting calls with a preset proportion from the U as a spy sample S, and marking P-S as PS and as 1, and marking U + S as US and as-1. And then, performing classification training by using the call characteristics of each call in the PS and the US to obtain a third classifier, and obtaining the classification probability of each call through the third classifier. And finally, determining a probability threshold value serving as a negative sample according to the classification probability of each call in the spy sample S, and adding the call of which the classification probability in the U is smaller than the probability threshold value as a reliable negative sample to the RN. And repeating the process, and retraining the classifier according to the remaining unlabeled samples and the labeled call set P in the U until the iteration times reach a preset number, and at the moment, dividing the second call subset into the labeled call set P belonging to the second scene, the reliable negative sample RN not belonging to the second scene and the remaining unlabeled call set U.

And S806, performing classification training according to the call characteristics of the marked call set and the reliable negative sample to obtain a second scene recognition model.

Specifically, the second scene recognition model is a model having the capability of recognizing whether or not a call belongs to the second scene based on the call characteristics of the call. The second scene recognition model is a classification model, and the recognition result of the call is that the call belongs to one of the second scene and the call does not belong to the second scene. After the computer device obtains the marked call set belonging to the second scene and the reliable negative sample not belonging to the second scene, classification training can be performed based on call characteristics of all calls in the set to obtain a second scene recognition model.

In one embodiment, step S806, training a fourth classifier according to the call features of the labeled call set and the reliable negative examples, and obtaining a second scene recognition model includes: dividing a training set and a verification set according to the marked call set and the reliable negative sample; training the fourth classifier according to the call characteristics of the training set to obtain a second scene recognition model; and verifying the second scene recognition model according to the verification set.

Specifically, when performing the classification training according to the labeled call set belonging to the second scene and the reliable negative sample not belonging to the second scene, the computer device may divide the labeled call set and the reliable negative sample into a training set and a verification set, for example, 7:3, according to a preset ratio, and then perform the classification training on the fourth classifier by using call features corresponding to the calls belonging to the second scene and the calls not belonging to the second scene in the training set, so as to obtain the second scene recognition model.

The computer device can also verify the accuracy of the trained second scene recognition model by using the call characteristics corresponding to the calls belonging to the second scene and the calls not belonging to the second scene in the verification set. When the verification fails, after step S808, iteratively performing dividing the second call subset into a new marked call set and an unmarked call set belonging to the second scenario according to the third call subset; determining a reliable negative sample which does not belong to a second scene from the unlabeled call set according to a third classifier obtained based on the call features of the labeled call set and the unlabeled call set; performing classification training according to the call characteristics of the marked call set and the reliable negative sample to obtain a second scene recognition model; and identifying a third communication subset belonging to the second scene and a fourth communication subset not belonging to the second scene from the second communication subset through the second scene identification model until the second scene identification model obtained after the current iteration passes verification or the iteration number meets the preset number.

And S808, identifying a third communication subset belonging to the second scene and a fourth communication subset not belonging to the second scene from the second communication subset through the second scene identification model.

Specifically, after the computer device obtains the second scene recognition model, the obtained second scene recognition model may be used to perform full prediction on the second call subset, that is, to predict call scenes to which each call in the second call subset belongs, so as to determine a third call subset belonging to the second scene and a fourth call subset not belonging to the second scene in the second call subset.

In one embodiment, identifying, by the second scene identification model, from the second subset of calls, a third subset of calls belonging to the second scene and a fourth subset of calls not belonging to the second scene includes: acquiring the call characteristics of each call in the second call subset; inputting the call characteristics of each call into a second scene recognition model; and obtaining classification probabilities corresponding to all calls through the second scene recognition model according to the call characteristics, and dividing the second call subset into a third call subset belonging to the second scene and a fourth call subset not belonging to the second scene according to the classification probabilities.

The classification probability may be used to reflect the possibility that the call belongs to the second scenario, and a higher classification probability may indicate that the call belongs to the second scenario with a higher possibility, whereas a lower possibility belongs to the second scenario. Therefore, the computer device can identify the call scene of the calls in the second call subset according to the classification probability obtained by the second scene identification model, divide the calls with the classification probability greater than the preset threshold into the third call subset belonging to the second scene, and divide the calls with the classification probability less than the preset threshold into the fourth call subset not belonging to the second scene.

In one embodiment, after step S808, the method further includes: the iteration execution divides the second call subset into a new marked call set and an unmarked call set which belong to the second scene according to the third call subset; determining a reliable negative sample which does not belong to a second scene from the unlabeled call set according to a third classifier obtained based on the call features of the labeled call set and the unlabeled call set; performing classification training according to the call characteristics of the marked call set and the reliable negative sample to obtain a second scene recognition model; and identifying a third communication subset belonging to the second scene and a fourth communication subset not belonging to the second scene from the second communication subset through the second scene identification model until an iteration stop condition is met.

In this embodiment, since the number of calls in the initial marked call set belonging to the second scenario obtained from the second call subset is small, after the second call subset is divided into the third call subset belonging to the second scenario and the fourth call subset not belonging to the second scenario by the second scenario identification model, the computer device may use the third call subset as a new marked call set belonging to the second scenario, that is, modify and diffuse the initial marked call set belonging to the second scenario according to the prediction result of the full prediction performed by the second scenario identification model, so as to obtain a more accurate and more reliable positive samples, so that the second call subset is divided into a new call set belonging to the second scenario and a non-marked call set, and based on this, iteratively perform the above steps S802-S808 until a new second scenario identification model is obtained, and carrying out full prediction again by using the new second scene recognition model until the iteration number reaches a preset number or the prediction result of the second scene recognition model obtained by the last iteration is in accordance with the expectation. It can be understood that each iteration process realizes the correction and diffusion of the marked call set belonging to the second scenario.

The computer device may verify the recognition rate of the second scene recognition model obtained for the first time or during subsequent iterations. The verification method can include the following steps: the first method is to verify the second scene recognition model by using a verification set reserved during training of the second scene recognition model, namely, verify the obtained second scene recognition model by using call characteristics of a call set and a reliable negative sample marked in the verification set, and if the recognition rate of the verification set does not meet a preset condition, continue iteration if the currently obtained second scene recognition model does not pass the verification. Secondly, the recognition rate of the calls initiated in the call group with the group name including the second preset word can be counted according to the full prediction result, for example, the recognition rate of the calls initiated in the call group with the group name including "chat" can be counted, the recognition rate of the calls initiated in the call group with the group name including "eight diagrams" can also be counted, and when the recognition rates of the calls do not meet the preset threshold value, the currently obtained second scene recognition model is not in accordance with the expectation, and the iteration can be continued. And thirdly, randomly extracting N calls, manually determining call scenes to which the N calls belong according to group names of call groups in which the N calls belong, and when the recognition rate of the N calls in the total prediction result does not meet a preset threshold value, obtaining a second scene recognition model at present, wherein the second scene recognition model does not meet the expectation, and the iteration can be continued.

The method for identifying the call scene comprises the steps of determining a small number of marked call sets belonging to a second scene from a second call subset, forming an unmarked call set according to the remaining calls in the second call subset, training according to the call characteristics of the calls in the current marked call set and the unmarked call set to obtain a third classifier, determining a reliable negative sample not belonging to the second scene from the unmarked call set according to the third classifier, performing classification training according to the call characteristics of the calls in the marked call set and the reliable negative sample to obtain a second scene identification model capable of identifying whether the call scene is the second scene, identifying a third call subset belonging to the second scene and a fourth call subset not belonging to the second scene from the second call subset through the second scene identification model, and identifying the calls belonging to the first scene from the original call set, the calls belonging to the second scenario continue to be identified from the remaining calls.

Fig. 10 is a flow chart illustrating a process of identifying a call belonging to a second scenario from a second subset of calls in a specific embodiment. Referring to fig. 10, the method specifically includes the following steps:

s1002, acquiring a marked call set and an unmarked call set belonging to a second scene from a second call subset;

s1004, selecting a spy sample from the marked call set belonging to the second scene;

s1006, after a spy sample is removed from the marked call set, a training positive sample is obtained;

s1008, after the spy sample is added to the call set which is not marked, a training negative sample is obtained;

s1010, training according to the call characteristics of the training positive sample and the training negative sample to obtain a third classifier;

s1012, obtaining the classification probability of each call in the call set which is not marked through a third classifier;

s1014, determining a probability threshold value for judging the call as not belonging to the second scene according to the classification probability of the spy sample;

and S1016, taking the calls of which the classification probability corresponding to the call sets which are not marked is smaller than the probability threshold as reliable negative samples which do not belong to the second scene.

In order to obtain more reliable negative samples, after reliable negative samples not belonging to the second scene are removed from the unlabeled call set, an updated unlabeled call set is obtained, and S1004 to S1016 are iteratively executed according to the labeled call set belonging to the second scene and the updated unlabeled call set. For example, it may be iterated 15 times.

S1018, dividing a training set and a verification set according to the marked call set and the reliable negative sample;

s1020, training a fourth classifier according to the call characteristics of the training set to obtain a second scene recognition model;

s1022, obtaining the call characteristics of each call in the second call subset;

s1024, inputting the call characteristics of each call into a second scene recognition model;

and S1026, obtaining classification probabilities corresponding to all calls through the second scene recognition model according to the call characteristics, and dividing the second call subset into a third call subset belonging to the second scene and a fourth call subset not belonging to the second scene according to the classification probabilities.

S1028, verifying the recognition rate of the second scene recognition model.

And when the verification fails, dividing the second call subset into a new marked call set and an unmarked call set belonging to the second scene according to the third call subset, and iteratively executing S1002 to S1028. When the verification is passed, the recognition rate of the second scene recognition model is in accordance with the expectation, and at the moment, the recognition of calls belonging to the second scene and calls not belonging to the second scene from the second call subset is also realized.

In this embodiment, after a small number of marked call sets belonging to the second scene are determined from the second call subset, an unlabeled call set is formed according to the remaining calls in the second call subset, then training is performed according to the call features of each call in the current marked call set and the unlabeled call set to obtain a third classifier, a reliable negative sample not belonging to the second scene is determined from the unlabeled call set according to the third classifier, the process can be iterated for multiple times to obtain more reliable negative samples, then a verification set and a training set are divided according to the marked call set and the reliable negative sample, classification training is performed by using the call features of each call in the training set to obtain a second scene recognition model capable of recognizing whether the call scene is the second scene, so that the call set can be fully predicted by the second scene recognition model, and obtaining a total prediction result, verifying the recognition rate of the second scene recognition model according to the total prediction result or the verification set, repeating the iteration process until the second scene recognition model obtained after the iteration meets the expectation after the initial marked call set is diffused and corrected according to the third call subset recognized to belong to the second scene when the verification fails, and recognizing the calls belonging to the second scene from the second call subset.

Fig. 11 is a general flow diagram for identifying call scenarios for each call in a call set according to an embodiment. Referring to fig. 11, the method specifically includes the following steps:

determining a marked call set P1 and an unmarked call set U1 belonging to a first scene from the call set A; performing classification training according to the marked call set P1 and the unmarked call set U1 to obtain a first classifier g 1; determining a reliable negative sample RN1 which does not belong to the first scene from U1 according to g1, wherein the process of obtaining the reliable negative sample RN1 can be iterated for multiple times; performing classification training by using the marked call set P1 and a reliable negative sample RN1 which does not belong to the first scene to obtain a second classifier g2, namely a first scene recognition model; and performing full prediction on the call set A by using the first scene recognition model g2, recognizing a first call subset belonging to the first scene and a second call subset B not belonging to the first scene from the call set A, and performing multiple iterations of the previous steps after correcting and diffusing the marked call set P1 according to the first call subset. Thus, the call belonging to the first scene is identified from the call set A.

The identification may also continue for a second subset of calls B that do not belong to the first scenario.

Determining a marked call set P2 and an unmarked call set U2 belonging to a second scene from the second call subset B; performing classification training according to the marked call set P2 and the unmarked call set U2 to obtain a third classifier g 3; determining a reliable negative sample RN2 which does not belong to the second scene from U2 according to g2, wherein the process of obtaining the reliable negative sample RN2 can be iterated for multiple times; performing classification training by using the marked call set P2 and the reliable negative sample RN2 which does not belong to the second scene to obtain a fourth classifier g4, namely a second scene recognition model; and performing full prediction on the second call subset B by using the second scene recognition model g4, recognizing a third call subset belonging to the second scene and a fourth call subset C not belonging to the second scene from the second call subset B, and performing multiple iterations of the previous steps after correcting and diffusing the marked call set P2 according to the third call subset. Thus, the identification of calls belonging to the second scenario from the original call set a is achieved.

The calls divided into the fourth call subset C do not belong to the first scenario or the second scenario, and the call scenarios can be directly divided into other types, that is, calls initiated occasionally, which have fewer use scenarios.

It should be noted that, when the first scene is a working scene and the second scene is a chat scene, after the first scene identification model and the second scene identification model are obtained, the computer device may identify a usage scene of any multi-person call initiated in the call group, specifically, identify whether the usage scene belongs to the working scene through the first scene identification model, identify whether the usage scene belongs to the chat scene through the second scene, and if the usage scene does not belong to the working scene or the chat scene, may determine that the usage scene of the initiated multi-person call is another scene, which is a multi-person call initiated by accident.

It should be understood that, although the steps in the flowcharts of fig. 7 and 11 are shown in sequence as indicated by the arrows, the steps are not necessarily performed in sequence as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least some of the steps in fig. 7 and 11 may include multiple sub-steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of performing the sub-steps or stages is not necessarily sequential, but may be performed alternately or alternately with other steps or at least some of the sub-steps or stages of other steps.

In one embodiment, as shown in fig. 12, there is provided a call scenario recognition apparatus 1200, which may be implemented by software, hardware, or a combination thereof as all or part of a computer device. The apparatus includes an acquisition module 1202, a reliable negative example determination module 1204, a training module 1206, and an identification module 1208, wherein:

an obtaining module 1202, configured to obtain a marked call set and an unmarked call set belonging to a first scene from a call set;

a reliable negative sample determining module 1204, configured to determine, according to a first classifier obtained based on call features of the labeled call set and the unlabeled call set, a reliable negative sample that does not belong to the first scene from the unlabeled call set;

the training module 1206 is used for performing classification training according to the call characteristics of the marked call set and the reliable negative sample to obtain a first scene recognition model;

an identifying module 1208, configured to identify, from the call set, a first call subset belonging to the first scenario and a second call subset not belonging to the first scenario through the first scenario identification model.

In one embodiment, the obtaining module 1202 is further configured to obtain a first preset word corresponding to a first scene; acquiring a group name of a call group to which each call in a call set belongs; screening out a call group with the group name matched with a first preset word; and obtaining a marked call set belonging to the first scene according to the call initiated in the screened call group.

In one embodiment, the device further comprises a feature data preprocessing module, configured to acquire feature data of various feature dimensions corresponding to each call in the call set; carrying out normality inspection on the feature data under the continuous feature dimension; and when the test result indicates that the feature data do not conform to normal distribution, carrying out logarithmic transformation on the feature data under the feature dimensionality to obtain the transformed feature data.

In one embodiment, the device further comprises a feature data preprocessing module, configured to acquire feature data of various feature dimensions corresponding to each call in the call set; and extracting the principal component from the feature data to obtain the call feature of each call.

In one embodiment, the feature data preprocessing module is further configured to calculate a covariance matrix after performing decentralized processing on feature data of each feature dimension; calculating eigenvalues of the covariance matrix and eigenvectors corresponding to the eigenvalues; obtaining a principal component matrix according to the eigenvector corresponding to the selected eigenvalue; and performing linear transformation on the characteristic data according to the principal component matrix to obtain the call characteristics of each call.

In one embodiment, the reliable negative example determining module 1204 is further configured to select a spy example from the callout collection belonging to the first scenario; after spy samples are removed from the marked call set, training positive samples are obtained; after the spy sample is added to the call set which is not marked, a training negative sample is obtained; obtaining a first classifier according to the call characteristics of the training positive sample and the training negative sample; obtaining the classification probability of each call in the call set which is not marked through a first classifier; determining a probability threshold value for judging the call as not belonging to the first scene according to the classification probability of the spy samples; and taking the calls of which the classification probability corresponding to the call sets which are not marked is smaller than the probability threshold as reliable negative samples which do not belong to the first scene.

In an embodiment, the reliable negative sample determining module 1204 is further configured to iteratively perform the step of removing the reliable negative sample that does not belong to the first scene from the unlabeled call set, obtaining an updated unlabeled call set, and determining the reliable negative sample that does not belong to the first scene from the updated unlabeled call set according to the call features of the labeled call set and the updated unlabeled call set until the iteration number satisfies the preset number.

In one embodiment, the training module 1206 is further configured to partition a training set and a validation set according to the labeled call set and the reliable negative samples; training a second classifier according to the call characteristics of the training set to obtain a first scene recognition model; wherein the verification set is used for verifying the first scene recognition model.

In one embodiment, the identifying module 1208 is further configured to obtain call characteristics of each call in the call set; inputting call characteristics of each call into a first scene recognition model; and obtaining classification probability corresponding to each call according to the call characteristics through the first scene recognition model, and dividing the call set into a first call subset belonging to the first scene and a second call subset not belonging to the first scene according to the classification probability.

In one embodiment, the apparatus is further configured to iteratively perform the dividing of the call set into a new marked call set and an unmarked call set belonging to the first scenario according to the first call subset; determining a reliable negative sample which does not belong to a first scene from the unlabeled call set according to a first classifier obtained based on the call features of the labeled call set and the unlabeled call set; performing classification training according to the call characteristics of the marked call set and the reliable negative sample to obtain a first scene recognition model; and identifying a first call subset belonging to the first scene and a second call subset not belonging to the first scene from the call set through a first scene identification model until an iteration stop condition is met.

In one embodiment, the apparatus further includes a second scene recognition module, where the second scene recognition module specifically includes an obtaining unit, a reliable negative sample determination unit, a training unit, and a recognition unit.

The acquiring unit is used for acquiring a marked call set and an unmarked call set belonging to a second scene from the second call subset; determining a reliable negative sample which does not belong to a second scene from the unlabeled call set according to a third classifier obtained based on the call features of the labeled call set and the unlabeled call set; performing classification training according to the call characteristics of the marked call set and the reliable negative sample to obtain a second scene recognition model; a third subset of calls belonging to the second scenario and a fourth subset of calls not belonging to the second scenario are identified from the second subset of calls by the second scenario identification model.

In one embodiment, the obtaining unit is further configured to obtain a second preset word corresponding to a second scene; acquiring the group name of a call group to which each call belongs in the second call subset; screening out a call group with the group name matched with a second preset word; and obtaining a marked call set belonging to the second scene according to the call initiated in the screened call group.

In one embodiment, the reliable negative sample determination unit is further configured to select a spy sample from the callout collection belonging to the second scenario; after spy samples are removed from the marked call set, training positive samples are obtained; after the spy sample is added to the call set which is not marked, a training negative sample is obtained; obtaining a third classifier according to the call characteristics of the training positive sample and the training negative sample; obtaining the classification probability of each call in the call set which is not marked through a third classifier; determining a probability threshold value for judging the call as not belonging to the second scene according to the classification probability of the spy samples; and taking the calls with the classification probability smaller than the probability threshold value corresponding to the calls in the call set which are not marked as the reliable negative samples which do not belong to the second scene.

In an embodiment, the reliable negative sample determining unit is further configured to iteratively perform a step of removing reliable negative samples that do not belong to the second scene from the unlabeled call set, obtain an updated unlabeled call set, and determine reliable negative samples that do not belong to the second scene from the updated unlabeled call set according to call features of the labeled call set and the updated unlabeled call set until the iteration number satisfies a preset number.

In one embodiment, the training unit is further configured to partition a training set and a verification set according to the marked call set and the reliable negative samples; training a fourth classifier according to the call characteristics of the training set to obtain a second scene recognition model; wherein the verification set is used for verifying the second scene recognition model.

In one embodiment, the identifying unit is further configured to obtain call characteristics of each call in the second subset of calls; inputting the call characteristics of each call into a second scene recognition model; and obtaining classification probabilities corresponding to all calls through the second scene recognition model according to the call characteristics, and dividing the second call subset into a third call subset belonging to the second scene and a fourth call subset not belonging to the second scene according to the classification probabilities.

In one embodiment, the second scenario identification module is further configured to iteratively perform the dividing of the second call subset into a new marked call set and an unmarked call set belonging to the second scenario according to the third call subset; determining a reliable negative sample which does not belong to a second scene from the unlabeled call set according to a third classifier obtained based on the call features of the labeled call set and the unlabeled call set; performing classification training according to the call characteristics of the marked call set and the reliable negative sample to obtain a second scene recognition model; and identifying a third communication subset belonging to the second scene and a fourth communication subset not belonging to the second scene from the second communication subset through the second scene identification model until an iteration stop condition is met.

The device for identifying call scenes determines a small number of marked call sets belonging to a first scene from the call sets, forming an unlabelled call set according to the remaining calls in the call set, then training according to the call characteristics of each call in the current labeled call set and the unlabelled call set to obtain a first classifier, determining reliable negative samples which do not belong to the first scene from the unlabeled call set according to the first classifier, performing classification training according to call characteristics of each call in the labeled call set and the reliable negative samples to obtain a first scene identification model capable of identifying whether the call scene is the first scene, therefore, a first call subset belonging to the first scene and a second call subset not belonging to the first scene can be identified from the call set through the first scene identification model, and the call scene is identified. Further, a third subset of calls belonging to the second scenario and a fourth subset of calls not belonging to the second scenario may also be identified from the second subset of calls in the same manner as described above.

FIG. 13 is a diagram illustrating an internal structure of a computer device in one embodiment. The computer device may specifically be the terminal 110 or the server 120 in fig. 1. As shown in fig. 13, the computer device includes a processor, a memory, a network interface connected by a system bus. Wherein the memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium of the computer device stores an operating system and may also store a computer program, which, when executed by the processor, causes the processor to implement the recognition method of the call scenario. The internal memory may also store a computer program, and the computer program, when executed by the processor, may cause the processor to perform the method for recognizing a call scenario.

Those skilled in the art will appreciate that the architecture shown in fig. 13 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.

In one embodiment, the recognition apparatus 1200 for call scenario provided in the present application may be implemented in the form of a computer program, and the computer program may be run on a computer device as shown in fig. 13. The memory of the computer device may store various program modules that make up the recognition means of the call scenario, such as the acquisition module 1202, the reliable negative example determination module 1204, the training module 1206, and the recognition module 1208 shown in fig. 12. The computer program constituted by the respective program modules causes the processor to execute the steps in the call scene recognition method according to the respective embodiments of the present application described in the present specification.

For example, the computer device shown in fig. 13 may execute step S402 through the obtaining module 1202 in the recognition apparatus of the call scenario shown in fig. 12. The computer device may perform step S404 by the reliable negative examples determination module 1204. The computer device may perform step S406 through the training module 1206. The computer device may perform step S408 through the identification module 1208.

In one embodiment, a computer device is provided, which includes a memory and a processor, the memory storing a computer program, and the computer program, when executed by the processor, causes the processor to execute the steps of the above-mentioned recognition method for call scenarios. Here, the steps of the identification method of the call scenario may be steps in the identification method of the call scenario of the above embodiments.

In one embodiment, a computer-readable storage medium is provided, which stores a computer program, which, when executed by a processor, causes the processor to perform the steps of the above-described call scenario identification method. Here, the steps of the identification method of the call scenario may be steps in the identification method of the call scenario of the above embodiments.

It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, which can be stored in a non-volatile computer-readable storage medium, and can include the processes of the embodiments of the methods described above when the program is executed. Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory, among others. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.

The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the present application. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.

Claims

1. A method for recognizing call scenes comprises the following steps:

2. The method of claim 1, wherein the step of obtaining the marked call set marked as the first scenario from the call sets comprises:

acquiring a first preset word corresponding to the first scene;

acquiring a group name of a call group to which each call in the call set belongs;

screening out a call group with a group name matched with the first preset word;

and obtaining a marked call set belonging to the first scene according to the call initiated in the screened call group.

3. The method of claim 1, further comprising:

acquiring feature data of various feature dimensions corresponding to each call in the call set;

carrying out normality inspection on the feature data under the continuous feature dimension;

when the test result indicates that the characteristic data does not conform to the normal distribution, then

And carrying out logarithmic transformation on the feature data under the feature dimension to obtain transformed feature data.

4. The method of claim 1, further comprising:

acquiring feature data of various feature dimensions corresponding to each call in a call set;

and extracting principal components from the feature data to obtain the call features of the calls.

5. The method according to claim 4, wherein the extracting a principal component from the feature data to obtain the call feature of each call comprises:

performing decentralized processing on the feature data of various feature dimensions, and calculating a covariance matrix;

calculating eigenvalues of the covariance matrix and eigenvectors corresponding to the eigenvalues;

obtaining a principal component matrix according to the eigenvector corresponding to the selected eigenvalue;

and carrying out linear transformation on the characteristic data according to the principal component matrix to obtain the call characteristics of each call.

6. The method of claim 1, wherein the determining reliable negative examples that do not belong to a first scenario from the unlabeled call set according to a first classifier obtained based on call features of the labeled call set and the unlabeled call set comprises:

selecting a spy sample from the marked call set belonging to the first scene;

after spy samples are removed from the marked call set, training positive samples are obtained;

adding a spy sample into the unlabeled call set to obtain a training negative sample;

obtaining a first classifier according to the call characteristics of the training positive sample and the training negative sample;

obtaining the classification probability of each call in the call set which is not marked through a first classifier;

determining a probability threshold value for judging the call as not belonging to the first scene according to the classification probability of the spy samples;

and taking the calls of which the classification probability corresponding to the call sets not labeled is smaller than the probability threshold value as reliable negative samples which do not belong to the first scene.

7. The method of claim 6, further comprising:

and after reliable negative samples which do not belong to the first scene are removed from the unmarked call set in an iterative execution mode, obtaining an updated unmarked call set, and determining the reliable negative samples which do not belong to the first scene from the updated unmarked call set according to the call characteristics of the marked call set and the updated unmarked call set until the iteration times meet the preset number.

8. The method of claim 1, wherein performing classification training according to the call features of the labeled call set and the reliable negative examples to obtain a first scene recognition model comprises:

dividing a training set and a verification set according to the marked call set and the reliable negative sample;

training a second classifier according to the call characteristics of the training set to obtain a first scene recognition model;

wherein the verification set is used to verify the first scene recognition model.

9. The method of claim 1, wherein the identifying, from the call set through the first scenario identification model, a first subset of calls belonging to a first scenario and a second subset of calls not belonging to the first scenario comprises:

acquiring the call characteristics of each call in the call set;

inputting call characteristics of each call to the first scene recognition model;

and obtaining classification probability corresponding to each call according to the call characteristics through the first scene recognition model, and dividing the call set into a first call subset belonging to a first scene and a second call subset not belonging to the first scene according to the classification probability.

10. The method of claim 1, further comprising:

the iterative execution divides the call set into a new marked call set and an unmarked call set which belong to a first scene according to the first call subset; determining a reliable negative sample which does not belong to a first scene from the unmarked call set according to a first classifier obtained based on the call features of the marked call set and the unmarked call set; performing classification training according to the call characteristics of the labeled call set and the reliable negative sample to obtain a first scene recognition model; and identifying a first call subset belonging to a first scene and a second call subset not belonging to the first scene from the call set through the first scene identification model until an iteration stop condition is met.

11. The method of any one of claims 1 to 10, further comprising:

acquiring a marked call set and an unmarked call set belonging to a second scene from a second call subset;

determining a reliable negative sample which does not belong to a second scene from the unmarked call set according to a third classifier obtained based on the call features of the marked call set and the unmarked call set;

performing classification training according to the call characteristics of the labeled call set and the reliable negative sample to obtain a second scene recognition model;

identifying, by the second scene identification model, a third subset of calls belonging to the second scene and a fourth subset of calls not belonging to the second scene from the second subset of calls.

12. The method of claim 11, wherein the step of obtaining the marked call set marked as the second scenario from the second call subset comprises:

acquiring a second preset word corresponding to the second scene;

acquiring the group name of a call group to which each call belongs in the second call subset;

screening out a call group with the group name matched with the second preset word;

and obtaining a marked call set belonging to the second scene according to the call initiated in the screened call group.

13. The method of claim 11, wherein the determining reliable negative examples that do not belong to a second scenario from the unlabeled call set according to a third classifier obtained based on call features of the labeled call set and the unlabeled call set comprises:

selecting a spy sample from the marked call set belonging to the second scene;

obtaining a third classifier according to the call characteristics of the training positive sample and the training negative sample;

obtaining the classification probability of each call in the call set which is not marked through a third classifier;

determining a probability threshold value for judging the call as not belonging to the second scene according to the classification probability of the spy samples;

and taking the calls of which the classification probability corresponding to the call sets not labeled is smaller than the probability threshold value as reliable negative samples which do not belong to a second scene.

14. The method of claim 11, wherein the identifying, by the second scene recognition model, a third subset of calls belonging to the second scene and a fourth subset of calls not belonging to the second scene from the second subset of calls comprises:

acquiring the call characteristics of each call in the second call subset;

inputting call characteristics of each call to the second scene recognition model;

and obtaining classification probability corresponding to each call according to the call characteristics through the second scene recognition model, and dividing the second call subset into a third call subset belonging to the second scene and a fourth call subset not belonging to the second scene according to the classification probability.

15. An apparatus for identifying a call scenario, the apparatus comprising: