CN111340053A - Order classification method, classification system, computer device and readable storage medium - Google Patents

Abstract

Embodiments of the present disclosure provide an order classification method, a classification system, a computer device, and a readable storage medium. The order classification method includes: acquiring abnormal orders, extracting characteristic information of abnormal orders; inputting characteristic information of abnormal orders into a preset classification model, and classifying responsible parties for abnormal orders. By adopting the embodiments of the present disclosure, the responsible party of an abnormal order can be accurately determined, so as to take intervention measures to reduce the abnormal situation of the order and improve the success rate of the order.

Description

Order classification method, classification system, computer device and readable storage medium

technical field

The embodiments of the present disclosure relate to the field of computer technologies, and in particular, to an order classification method, a classification system, a computer device, and a readable storage medium.

Background technique

In the projects controlled by the car-hailing platform for drivers and users, for an abnormal order such as cancellation, complaint or bad review, the responsible party for the abnormality needs to be determined. The responsible parties mainly include the driver, the user or the car-hailing platform. It is easy to obtain the order, but the labeling of the responsible party is very difficult, that is, it is very difficult to obtain the label of the training sample, and the number of obtained is also very limited.

In the related art, some samples are marked manually and according to the feedback of drivers or users, including positive samples and negative samples, but the samples marked by this marking method are very limited, and there is no guarantee that they are free from the overall data set. Therefore, directly using this part of the sample for training will cause the problem of inconsistency between the training sample and the actual estimated sample. It may perform well on the training set, but the actual prediction effect is not good, that is, overfitting. a kind of.

SUMMARY OF THE INVENTION

The embodiments of the present disclosure are intended to solve at least one of the technical problems existing in the prior art or related technologies

To this end, an aspect of the embodiments of the present disclosure is to propose an order classification method.

Another aspect of the embodiments of the present disclosure is to provide an order classification system.

Yet another aspect of the embodiments of the present disclosure is to provide a computer device.

Yet another aspect of the embodiments of the present disclosure is to provide a computer-readable storage medium.

In view of this, according to an aspect of the embodiments of the present disclosure, an order classification method is proposed, including: acquiring abnormal orders, extracting feature information of the abnormal orders; inputting the feature information of the abnormal orders into a preset classification model, and classifying the abnormal orders Classification of responsible parties.

The order classification method provided by the embodiment of the present disclosure acquires an abnormal order with abnormal conditions such as cancellation, complaint, or bad review, and extracts one or more characteristic information related to the abnormal condition of the abnormal order. Further, a preset classification model is used to classify the responsible party of the abnormal order according to one or more characteristic information, that is, to distinguish whether the responsible party of the abnormal order is a driver, a user or an online car-hailing platform. By adopting the embodiments of the present disclosure, the responsible party of an abnormal order can be accurately determined, so as to take intervention measures to reduce the abnormal situation of the order and improve the success rate of the order.

The above-mentioned order classification method according to the embodiment of the present disclosure may further have the following technical features:

In the above technical solution, preferably, it further includes: collecting a plurality of sample abnormal orders, and dividing the plurality of sample abnormal orders into a first-type sample abnormal order and a second-type abnormal sample order; according to each first-type sample abnormal order to obtain the labeling information of the responsible party for each abnormal sample order of the first type; train the labeling information of the responsible party to establish a preset classification model; input the feature information of the second-type abnormal sample order into the preset classification model, and analyze the Classify the responsible party for the second-type abnormal sample orders and obtain the responsible party confidence for each second-type abnormal sample order; iterate the first-type abnormal sample orders and the second-type abnormal sample orders whose responsible party confidence is greater than the first threshold Train until the preset classification model reaches the preset convergence condition.

In this technical solution, PU-learning (Learning from Positive and UnlabeledExample) is a semi-supervised binary classification model, which is trained by labeled positive samples and a large number of unlabeled samples. A binary classifier, different from ordinary binary classification problems, the scale of P in the PU problem is usually quite small, and it is difficult to expand the set of positive samples; and the scale of U is usually large, such as unidentified web page resources in web page classification It can be obtained from the network very cheaply and conveniently. The purpose of introducing U is to reduce the preparatory workload of manual classification, and at the same time improve the accuracy, so as to achieve the effect of automatic classification as much as possible. In the embodiment of the present disclosure, the idea of PU-learning is introduced, and a large number of abnormal sample orders collected are divided into the first type of abnormal sample orders (labeled samples) and the second type of abnormal sample orders (unlabeled samples), Obtain the characteristic information of the first type of sample abnormal orders, label the responsible party according to the characteristic information, and initially establish a preset classification model by training the labeling information. Further, use the preset classification model to estimate the unlabeled second-type abnormal sample orders, that is, input the feature information of the second-type abnormal sample orders into the preset classification model to obtain the confidence of the responsible party (responsible party probability), for example, a certain The probability that the responsible party for the second type of abnormal sample order is the driver, the responsible party is the user, or the responsible party is the online car-hailing platform (there is only one responsible party for an abnormal sample order). Add the second type of abnormal sample orders with the confidence of the responsible party greater than the first threshold to the model training, continue to train the model together with the first type of abnormal sample orders, and iterate continuously according to this method until the preset classification model reaches the preset value. Convergence condition. The embodiment of the present disclosure solves the classification problem in unlabeled machine learning by using very limited labeled samples and the method of iteratively labeling samples based on PU-learning, and obtains an accurate classification model of abnormal orders.

In any of the above technical solutions, preferably, the preset convergence conditions include that the sum of the second type of abnormal sample orders with the confidence of the responsible party greater than the first threshold and the number of the first type of abnormal sample orders is greater than the second threshold, the first type of abnormal sample orders The accuracy of sample abnormal orders is greater than the third threshold and the recall rate of the first type of sample abnormal orders is greater than the fourth threshold.

In this technical solution, the preset convergence conditions include that the scale of the labeled samples is large enough (the sum of the second type of abnormal sample orders with the confidence of the responsible party greater than the first threshold and the number of the first type of abnormal sample orders is greater than the second threshold), And for the part of the samples with obvious labels (the first type of sample abnormal orders), the accuracy rate is greater than the third threshold and its recall rate is greater than the fourth threshold, and the accuracy rate is the sample of the first type of sample abnormal orders to determine the responsible party. The ratio to the total sample, the recall rate is the ratio of the sample of a responsible party to the samples of all responsible parties in the first type of sample abnormal order (that is, the sample of the responsible party is the user and the responsible party is the user, driver or online car-hailing platform). The ratio of all samples), when the preset convergence condition is reached, a classification model of accurate abnormal orders is established.

In any of the above-mentioned technical solutions, preferably, acquiring the labeling information of the responsible party for each abnormal order of the first type of sample specifically includes: receiving the first responsible party of the abnormal order of the first type of sample obtained according to the first preset order information labeling information; and/or labeling the abnormal orders of the first type of samples with second responsible party labeling information according to the second preset order information; wherein the responsible party labeling information includes the first responsible party labeling information and the second responsible party labeling information.

In this technical solution, positive samples are provided for establishing a preset classification model by marking the responsible party of each abnormal order of the first type of samples. When labeling the responsible party of each first-class sample abnormal order, two methods can be included. One is for the labeling team to manually label the first type of sample abnormal orders according to the first preset order information, where the first preset order information is non-obvious information related to the abnormal situation, for example, it needs to ask the driver or user by phone The other is that the system automatically marks the first type of sample abnormal orders with second preset order information, where the second preset order information includes obvious feedback from drivers, users or online car-hailing platform customer service information, such as complaints or bad reviews on the order.

In any of the above technical solutions, preferably, the first preset order information includes one or a combination of the following: order track information, historical user order cancellation information, driver historical order cancellation information, user historical negative comments or complaint information, The driver's historical negative review or complaint information, the user's call information with the driver, the user's call information with the online car-hailing platform, the driver's call information with the online car-hailing platform, the online car-hailing platform's telephone return visit information to the user, the online car-hailing platform's communication The driver's telephone return visit information; the second preset order information includes one or a combination of the following: user's negative comment or complaint information, and driver's negative comment or complaint information.

In this technical solution, the preset order information is also the feature information of each order, and the first preset order information and the second preset order information include but are not limited to the above information. The preset order information can accurately mark the responsible party of the sample abnormal order.

In any of the above technical solutions, preferably, the responsible party for the abnormal order includes any one of the following: a user, a driver or an online car-hailing platform.

In this technical solution, abnormal orders are classified into abnormal orders with the responsible party being the user, abnormal orders with the responsible party being the driver, or abnormal orders with the responsible party being the car-hailing platform, so as to facilitate intervention measures. This will reduce order exceptions, thereby ensuring the quality of service for users and protecting the interests of drivers and online car-hailing platforms.

According to another aspect of the embodiments of the present disclosure, an order classification system is proposed, including: a feature extraction unit for acquiring abnormal orders and extracting feature information of the abnormal orders; a classification unit for inputting the feature information of abnormal orders into a The preset classification model is used to classify the responsible party for abnormal orders.

The order classification system provided by the embodiment of the present disclosure acquires an abnormal order with abnormal conditions such as cancellation, complaint, or bad review, and extracts one or more characteristic information related to the abnormal condition of the abnormal order. Further, a preset classification model is used to classify the responsible party of the abnormal order according to one or more characteristic information, that is, to distinguish whether the responsible party of the abnormal order is a driver, a user or an online car-hailing platform. By adopting the embodiments of the present disclosure, the responsible party of an abnormal order can be accurately determined, so as to take intervention measures to reduce the abnormal situation of the order and improve the success rate of the order.

The above-mentioned order classification system according to the embodiment of the present disclosure may further have the following technical features:

In the above technical solution, preferably, it further includes: a model building unit, configured to collect multiple sample abnormal orders, and divide the multiple sample abnormal orders into the first type of abnormal sample orders and the second type of abnormal sample orders; The characteristic information of the abnormal orders of the first type of sample is obtained, and the labeling information of the responsible party for each abnormal order of the first type of sample is obtained; the labeling information of the responsible party is trained to establish a preset classification model; the characteristic information of the abnormal sample orders of the second type is input into the pre-defined Set up a classification model to classify the responsible party for the second type of abnormal sample orders and obtain the responsible party confidence for each second type of abnormal sample order; Anomaly-like sample orders are iteratively trained until the preset classification model reaches the preset convergence condition.

In this technical solution, PU-learning (Learning from Positive and UnlabeledExample) is a semi-supervised binary classification model, which is trained by labeled positive samples and a large number of unlabeled samples. A binary classifier, different from ordinary binary classification problems, the scale of P in the PU problem is usually quite small, and it is difficult to expand the set of positive samples; and the scale of U is usually large, such as unidentified web page resources in web page classification It can be obtained from the network very cheaply and conveniently. The purpose of introducing U is to reduce the preparatory workload of manual classification, and at the same time improve the accuracy, so as to achieve the effect of automatic classification as much as possible. In the embodiment of the present disclosure, the idea of PU-learning is introduced, and a large number of abnormal sample orders collected are divided into the first type of abnormal sample orders (labeled samples) and the second type of abnormal sample orders (unlabeled samples), Obtain the characteristic information of the first type of sample abnormal orders, label the responsible party according to the characteristic information, and initially establish a preset classification model by training the labeling information. Further, use the preset classification model to estimate the unlabeled second-type abnormal sample orders, that is, input the feature information of the second-type abnormal sample orders into the preset classification model to obtain the confidence of the responsible party (responsible party probability), for example, a certain The probability that the responsible party for the second type of abnormal sample order is the driver, the responsible party is the user, or the responsible party is the online car-hailing platform (there is only one responsible party for an abnormal sample order). Add the second type of abnormal sample orders with the confidence of the responsible party greater than the first threshold to the model training, continue to train the model together with the first type of abnormal sample orders, and iterate continuously according to this method until the preset classification model reaches the preset value. Convergence condition. The embodiment of the present disclosure solves the classification problem in unlabeled machine learning by using very limited labeled samples and the method of iteratively labeling samples based on PU-learning, and obtains an accurate classification model of abnormal orders.

In any of the above technical solutions, preferably, the preset convergence conditions include that the sum of the second type of abnormal sample orders with the confidence of the responsible party greater than the first threshold and the number of the first type of abnormal sample orders is greater than the second threshold, the first type of abnormal sample orders The accuracy of sample abnormal orders is greater than the third threshold and the recall rate of the first type of sample abnormal orders is greater than the fourth threshold.

In this technical solution, the preset convergence conditions include that the scale of the labeled samples is large enough (the sum of the second type of abnormal sample orders with the confidence of the responsible party greater than the first threshold and the number of the first type of abnormal sample orders is greater than the second threshold), And for the part of the samples with obvious labels (the first type of sample abnormal orders), the accuracy rate is greater than the third threshold and its recall rate is greater than the fourth threshold, and the accuracy rate is the sample of the first type of sample abnormal orders to determine the responsible party. The ratio to the total sample, the recall rate is the ratio of the sample of a responsible party to the samples of all responsible parties in the first type of sample abnormal order (that is, the sample of the responsible party is the user and the responsible party is the user, driver or online car-hailing platform). The ratio of all samples), when the preset convergence condition is reached, a classification model of accurate abnormal orders is established.

In any of the above technical solutions, preferably, the model building unit is specifically configured to receive the first responsible party annotation information of the first type of sample abnormal order obtained according to the first preset order information; and/or according to the second preset order information; Order information, marking the abnormal orders of the first type of samples with the second responsible party marking information; wherein the responsible party marking information includes the first responsible party marking information and the second responsible party marking information.

In this technical solution, positive samples are provided for establishing a preset classification model by marking the responsible party of each abnormal order of the first type of samples. When labeling the responsible party of each first-class sample abnormal order, two methods can be included. One is for the labeling team to manually label the first type of sample abnormal orders according to the first preset order information, where the first preset order information is non-obvious information related to the abnormal situation, for example, it needs to ask the driver or user by phone The other is that the system automatically marks the first type of sample abnormal orders with second preset order information, where the second preset order information includes obvious feedback from drivers, users or online car-hailing platform customer service information, such as complaints or bad reviews on the order.

In any of the above technical solutions, preferably, the first preset order information includes one or a combination of the following: order track information, historical user order cancellation information, driver historical order cancellation information, user historical negative comments or complaint information, The driver's historical negative review or complaint information, the user's call information with the driver, the user's call information with the online car-hailing platform, the driver's call information with the online car-hailing platform, the online car-hailing platform's telephone return visit information to the user, the online car-hailing platform's communication The driver's telephone return visit information; the second preset order information includes one or a combination of the following: user's negative comment or complaint information, and driver's negative comment or complaint information.

In this technical solution, the preset order information is also the feature information of each order, and the first preset order information and the second preset order information include but are not limited to the above information. The preset order information can accurately mark the responsible party of the sample abnormal order.

In any of the above technical solutions, preferably, the responsible party for the abnormal order includes any one of the following: a user, a driver or an online car-hailing platform.

In this technical solution, abnormal orders are classified into abnormal orders with the responsible party being the user, abnormal orders with the responsible party being the driver, or abnormal orders with the responsible party being the car-hailing platform, so as to facilitate intervention measures. This will reduce order exceptions, thereby ensuring the quality of service for users and protecting the interests of drivers and online car-hailing platforms.

According to yet another aspect of the embodiments of the present disclosure, a computer device is proposed, including a memory, a processor, and a computer program stored in the memory and running on the processor, where the processor implements any of the foregoing techniques when executing the computer program The steps of the order classification method for the scenario.

In the computer device provided by the embodiment of the present disclosure, when the processor executes the computer program, the steps of the order classification method of any of the above technical solutions are implemented, so the computer device includes all the beneficial effects of the order classification method of any of the above technical solutions.

According to yet another aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, implements the steps of the order classification method according to any of the foregoing technical solutions.

In the computer-readable storage medium provided by the embodiments of the present disclosure, when the computer program is executed by the processor, the steps of the order classification method according to any of the foregoing technical solutions are implemented, so the computer-readable storage medium includes the order classification method of any of the foregoing technical solutions. all beneficial effects.

Additional aspects and advantages of embodiments of the present disclosure will become apparent in the description section that follows, or learned by practice of embodiments of the present disclosure.

Description of drawings

The above and/or additional aspects and advantages of embodiments of the present disclosure will become apparent and readily understood from the following description of the embodiments in conjunction with the accompanying drawings, wherein:

FIG. 1 shows a schematic flowchart of an order classification method according to an embodiment of the present disclosure;

FIG. 2 shows a schematic flowchart of an order classification method according to another embodiment of the present disclosure;

FIG. 3 shows a schematic diagram of an order classification method according to a specific embodiment of an embodiment of the present disclosure;

FIG. 4 shows a schematic diagram of an order classification system according to an embodiment of the present disclosure;

FIG. 5 shows a schematic diagram of an order classification system according to another embodiment of the embodiment of the present disclosure;

FIG. 6 shows a schematic diagram of a computer device of one embodiment of an embodiment of the present disclosure.

Detailed ways

In order to more clearly understand the above objects, features and advantages of the embodiments of the present disclosure, the embodiments of the present disclosure will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments may be combined with each other in the case of no conflict.

Many specific details are set forth in the following description to facilitate a full understanding of the embodiments of the present disclosure. However, the embodiments of the present disclosure may also be implemented in other ways different from those described herein. Therefore, the protection scope of the embodiments of the present disclosure does not Not to be limited by the specific embodiments disclosed below.

In the embodiment of the first aspect of the embodiment of the present disclosure, an order classification method is proposed. FIG. 1 shows a schematic flowchart of the order classification method according to an embodiment of the embodiment of the present disclosure. Among them, the method includes:

Step 102, obtaining an abnormal order, and extracting feature information of the abnormal order;

In step 104, the characteristic information of the abnormal order is input into the preset classification model, and the responsible party is classified for the abnormal order.

The order classification method provided by the embodiment of the present disclosure acquires an abnormal order with abnormal conditions such as cancellation, complaint, or bad review, and extracts one or more characteristic information related to the abnormal condition of the abnormal order. Further, a preset classification model is used to classify the responsible party of the abnormal order according to one or more characteristic information, that is, to distinguish whether the responsible party of the abnormal order is a driver, a user or an online car-hailing platform. By adopting the embodiments of the present disclosure, the responsible party of an abnormal order can be accurately determined, so as to take intervention measures to reduce the abnormal situation of the order and improve the success rate of the order.

Among them, the feature information includes order track information, user history canceled order information, driver history cancel order information, user history bad review or complaint information, driver history bad review or complaint information, user and driver call information, user and online appointment Car platform call information, driver and online car-hailing platform call information, online car-hailing platform’s phone call back information to users, online car-hailing platform’s driver’s phone call back information, order user negative comment or complaint information, order driver bad comment or Complaint information, etc.

For example, obtain an abnormal order with cancellation exception, extract the order trajectory information (feature information) of the abnormal order, analyze the order trajectory information through a preset classification model, and determine that when the driver goes to the user's location to pick up the user, the distance from the user If the order is canceled by the user at a distance of one kilometer, and the scheduled pick-up time is not reached at this time, that is, the driver has not timed out, the responsible party for judging the cancellation of the order is the user. Therefore, the abnormal order is classified as the responsible party. User's order. Another example is to obtain an abnormal order with a user complaint, extract the user complaint information (feature information) of the abnormal order, and the complaint information is that the driver has a bad attitude, then it is judged that the responsible party for the abnormal order is the driver. The abnormal order is classified as an order in which the responsible party is the driver. For another example, obtain an abnormal order with an abnormal cancellation condition, and extract the cancellation information (feature information) of the abnormal order. The cancellation information is that the online car-hailing platform has not assigned a vehicle to the user for a long time after the user placed the order, so the user cancels the order, then It is determined that the responsible party for the abnormal order is the online car-hailing platform. Therefore, the abnormal order is classified as an order in which the responsible party is the online car-hailing platform.

FIG. 2 shows a schematic flowchart of an order classification method according to another embodiment of the present disclosure. Among them, the method includes:

Step 202, collecting multiple sample abnormal orders, and dividing the multiple sample abnormal orders into a first type of sample abnormal order and a second type of abnormal sample order;

Step 204, according to the characteristic information of each first-type sample abnormal order, obtain the responsible party labeling information of each first-type sample abnormal order; train the responsible party labeling information, and establish a preset classification model;

Step 206: Input the feature information of the second-type abnormal sample order into a preset classification model, classify the responsible party for the second-type abnormal sample order, and obtain the responsible party confidence for each second-type abnormal sample order;

Step 208, performing iterative training on the first type of abnormal sample orders and the second type of abnormal sample orders with the confidence of the responsible party greater than the first threshold, until the preset classification model reaches the preset convergence condition;

Step 210, obtaining an abnormal order, and extracting feature information of the abnormal order;

Step 212: Input the feature information of the abnormal order into a preset classification model, and classify the responsible party for the abnormal order.

In this embodiment, PU-learning (Learning from Positive and Unlabeled Example) is a semi-supervised binary classification model that is trained by labeled positive samples and a large number of unlabeled samples Different from ordinary binary classification problems, the scale of P in the PU problem is usually quite small, and it is difficult to expand the set of positive samples; and the scale of U is usually large, such as unidentified webpages in webpage classification Resources can be obtained from the network very cheaply and conveniently. The purpose of introducing U is to reduce the preparatory workload of manual classification, and at the same time improve the accuracy, so as to achieve the effect of automatic classification as much as possible. In the embodiment of the present disclosure, the idea of PU-learning is introduced, and a large number of abnormal sample orders collected are divided into the first type of abnormal sample orders (labeled samples) and the second type of abnormal sample orders (unlabeled samples), Obtain the characteristic information of the first type of sample abnormal orders, label the responsible party according to the characteristic information, and initially establish a preset classification model by training the labeling information. Further, use the preset classification model to estimate the unlabeled second-type abnormal sample orders, that is, input the feature information of the second-type abnormal sample orders into the preset classification model to obtain the confidence of the responsible party (responsible party probability), for example, a certain The probability that the responsible party for the second type of abnormal sample order is the driver, the responsible party is the user, or the responsible party is the online car-hailing platform (there is only one responsible party for an abnormal sample order). Add the second type of abnormal sample orders with the confidence of the responsible party greater than the first threshold to the model training, continue to train the model together with the first type of abnormal sample orders, and iterate continuously according to this method until the preset classification model reaches the preset value. Convergence condition. The embodiment of the present disclosure solves the classification problem in unlabeled machine learning by using very limited labeled samples and the method of iteratively labeling samples based on PU-learning, and obtains an accurate classification model of abnormal orders.

Preferably, the preset convergence conditions include that the sum of the second type of abnormal sample orders with the confidence of the responsible party greater than the first threshold and the number of the first type of abnormal sample orders is greater than the second threshold, and the accuracy rate of the first type of abnormal sample orders is greater than the third The recall rate of the three thresholds and the abnormal orders of the first type of samples is greater than the fourth threshold.

In this embodiment, the preset convergence condition includes that the scale of the labeled samples is large enough (the sum of the second-type abnormal sample orders whose confidence level of the responsible party is greater than the first threshold and the number of the first-type abnormal sample orders is greater than the second threshold), And for the part of the samples with obvious labels (the first type of sample abnormal orders), the accuracy rate is greater than the third threshold and its recall rate is greater than the fourth threshold, and the accuracy rate is the sample of the first type of sample abnormal orders to determine the responsible party. The ratio to the total sample, the recall rate is the ratio of the sample of a responsible party to the samples of all responsible parties in the first type of sample abnormal order (that is, the sample of the responsible party is the user and the responsible party is the user, driver or online car-hailing platform). The ratio of all samples), when the preset convergence condition is reached, a classification model of accurate abnormal orders is established.

Preferably, in step 204, acquiring the labeling information of the responsible party for each abnormal order of the first type of sample specifically includes: receiving the labeling information of the first responsible party of the abnormal order of the first type of sample obtained according to the first preset order information; and /Or according to the second preset order information, mark the abnormal order of the first type of samples with the second responsible party marking information; wherein the responsible party marking information includes the first responsible party marking information and the second responsible party marking information.

In this embodiment, positive samples are provided for establishing the preset classification model by marking the responsible party of each abnormal order of the first type of samples. When labeling the responsible party of each first-class sample abnormal order, two methods can be included. One is for the labeling team to manually label the first type of sample abnormal orders according to the first preset order information, where the first preset order information is non-obvious information related to the abnormal situation, for example, it needs to ask the driver or user by phone The other is that the system automatically marks the first type of sample abnormal orders with second preset order information, where the second preset order information includes obvious feedback from drivers, users or online car-hailing platform customer service information, such as complaints or bad reviews on the order.

Preferably, the first preset order information includes one or a combination of the following: order track information, historical user order cancellation information, driver historical order cancellation information, user historical negative review or complaint information, and driver historical negative review or complaint information , user and driver call information, user and online car-hailing platform call information, driver and online car-hailing platform call information, online car-hailing platform to user telephone return information, online car-hailing platform to driver telephone return information; second The preset order information includes one or a combination of the following: user negative comment or complaint information, driver bad comment or complaint information.

In this embodiment, the preset order information is also the feature information of each order, and the first preset order information and the second preset order information include but are not limited to the above information. The preset order information can accurately mark the responsible party of the sample abnormal order. For example, when it is determined from the order track information that the driver is going to pick up the user at the user's location, the order is cancelled by the user at a distance of one kilometer from the user's location, and the scheduled pick-up time is not reached at this time, that is, the driver has not timed out. The responsible party for the cancellation of the order is the user. It should be noted that the negative review or complaint information included in the first preset order information is historical information, and the negative review or complaint information included in the second preset order information is the information in this abnormal order.

Preferably, the responsible party for the abnormal order includes any one of the following: a user, a driver or an online car-hailing platform.

In this embodiment, abnormal orders are classified into abnormal orders in which the responsible party is the user, abnormal orders in which the responsible party is the driver, or abnormal orders in which the responsible party is the online car-hailing platform, so as to carry out intervention measures This will reduce order exceptions, thereby ensuring the quality of service for users and protecting the interests of drivers and online car-hailing platforms.

FIG. 3 shows a schematic diagram of an order classification method according to a specific embodiment of the embodiment of the present disclosure. Among them, the method includes:

Step 302, pull all the abnormal orders, and obtain all the features in the abnormal orders that are required when training the model;

Step 304, manual labeling and feedback information labeling, randomly select some abnormal orders (thousand-level), and label the team according to the order track information, driver and user history information (historical cancellation of orders, complaints or negative comments), call information, and necessary Manually mark abnormal orders based on information such as driver and user telephone return visit information. In addition, some abnormal orders have obvious direct feedback information from drivers, users or online car-hailing customer service, such as complaints or negative comments, etc., use the feedback information to mark, and mark the labels of some abnormal orders according to the above two methods (ie The responsible party that marked the abnormal order), this part of the abnormal order is the abnormal order with the label;

Step 306 , train a model, and train a model by using the labeled abnormal orders as samples.

Step 308: Use the model to estimate unlabeled abnormal order samples, obtain abnormal order samples with relatively high confidence, and iteratively train until the scale of the training model samples is large enough and the model has the best effect. Estimate those unlabeled abnormal order samples, and then select some abnormal order samples with relatively high confidence from these unlabeled abnormal order samples to add to the training of the model, and iteratively train according to this method until convergence. The condition for the model to converge is that the scale of the abnormal order samples for training the model is large enough, and the model has a high enough accuracy and recall rate for the abnormal orders with obvious labels (that is, the model has the best effect);

Step 310, output the model.

An embodiment of the second aspect of the embodiment of the present disclosure proposes an order classification system, and FIG. 4 shows a schematic diagram of an order classification system 40 according to an embodiment of the embodiment of the present disclosure. Wherein, the system 40 includes:

A feature extraction unit 402, configured to acquire abnormal orders and extract feature information of abnormal orders;

The classification unit 404 is configured to input the characteristic information of the abnormal order into a preset classification model, and classify the responsible party for the abnormal order.

The order classification system 40 provided by the embodiment of the present disclosure acquires an abnormal order with abnormal conditions such as cancellation, complaint, or bad review, and extracts one or more characteristic information related to the abnormal condition of the abnormal order. Further, a preset classification model is used to classify the responsible party of the abnormal order according to one or more characteristic information, that is, to distinguish whether the responsible party of the abnormal order is a driver, a user or an online car-hailing platform. By adopting the embodiments of the present disclosure, the responsible party of an abnormal order can be accurately determined, so as to take intervention measures to reduce the abnormal situation of the order and improve the success rate of the order.

Among them, the feature information includes order track information, user history canceled order information, driver history cancel order information, user history bad review or complaint information, driver history bad review or complaint information, user and driver call information, user and online appointment Car platform call information, driver and online car-hailing platform call information, online car-hailing platform’s phone call back information to users, online car-hailing platform’s driver’s phone call back information, order user negative comment or complaint information, order driver bad comment or Complaint information, etc.

For example, obtain an abnormal order with cancellation exception, extract the order trajectory information (feature information) of the abnormal order, analyze the order trajectory information through a preset classification model, and determine that when the driver goes to the user's location to pick up the user, the distance from the user If the order is canceled by the user at a distance of one kilometer, and the scheduled pick-up time is not reached at this time, that is, the driver has not timed out, the responsible party for judging the cancellation of the order is the user. Therefore, the abnormal order is classified as the responsible party. User's order. Another example is to obtain an abnormal order with a user complaint, extract the user complaint information (feature information) of the abnormal order, and the complaint information is that the driver has a bad attitude, then it is judged that the responsible party for the abnormal order is the driver. The abnormal order is classified as an order in which the responsible party is the driver. For another example, obtain an abnormal order with an abnormal cancellation condition, and extract the cancellation information (feature information) of the abnormal order. The cancellation information is that the online car-hailing platform has not assigned a vehicle to the user for a long time after the user placed the order, so the user cancels the order, then It is determined that the responsible party for the abnormal order is the online car-hailing platform. Therefore, the abnormal order is classified as an order in which the responsible party is the online car-hailing platform.

FIG. 5 shows a schematic diagram of an order classification system 50 according to another embodiment of the present disclosure. Wherein, the system 50 includes:

The model building unit 502 is configured to collect a plurality of sample abnormal orders, and divide the plurality of sample abnormal orders into a first type of sample abnormal order and a second type of abnormal sample order; according to the characteristic information of each first type of sample abnormal order, obtain Labeling information of the responsible party for each abnormal sample order of the first type; training the responsible party to label the information to establish a preset classification model; inputting the characteristic information of the abnormal sample order of the second type into the preset classification model, Perform responsible party classification and obtain the responsible party confidence of each second-type abnormal sample order; iteratively train the first-type abnormal sample orders and the second-type abnormal sample orders whose responsible party confidence is greater than the first threshold until the preset The classification model reaches the preset convergence condition;

A feature extraction unit 504, configured to acquire abnormal orders and extract feature information of abnormal orders;

The classification unit 506 is configured to input the characteristic information of the abnormal order into the preset classification model, and classify the responsible party for the abnormal order.

In this embodiment, PU-learning (Learning from Positive and Unlabeled Example) is a semi-supervised binary classification model that is trained by labeled positive samples and a large number of unlabeled samples Different from ordinary binary classification problems, the scale of P in the PU problem is usually quite small, and it is difficult to expand the set of positive samples; and the scale of U is usually large, such as unidentified webpages in webpage classification Resources can be obtained from the network very cheaply and conveniently. The purpose of introducing U is to reduce the preparatory workload of manual classification, and at the same time improve the accuracy, so as to achieve the effect of automatic classification as much as possible. In the embodiment of the present disclosure, the idea of PU-learning is introduced, and a large number of abnormal sample orders collected are divided into the first type of abnormal sample orders (labeled samples) and the second type of abnormal sample orders (unlabeled samples), Obtain the characteristic information of the first type of sample abnormal orders, label the responsible party according to the characteristic information, and initially establish a preset classification model by training the labeling information. Further, use the preset classification model to estimate the unlabeled second-type abnormal sample orders, that is, input the feature information of the second-type abnormal sample orders into the preset classification model to obtain the confidence of the responsible party (responsible party probability), for example, a certain The probability that the responsible party for the second type of abnormal sample order is the driver, the responsible party is the user, or the responsible party is the online car-hailing platform (there is only one responsible party for an abnormal sample order). Add the second type of abnormal sample orders with the confidence of the responsible party greater than the first threshold to the model training, continue to train the model together with the first type of abnormal sample orders, and iterate continuously according to this method until the preset classification model reaches the preset value. Convergence condition. The embodiment of the present disclosure solves the classification problem in unlabeled machine learning by using very limited labeled samples and the method of iteratively labeling samples based on PU-learning, and obtains an accurate classification model of abnormal orders.

Preferably, the preset convergence conditions include that the sum of the second type of abnormal sample orders with the confidence of the responsible party greater than the first threshold and the number of the first type of abnormal sample orders is greater than the second threshold, and the accuracy rate of the first type of abnormal sample orders is greater than the third The recall rate of the three thresholds and the abnormal orders of the first type of samples is greater than the fourth threshold.

In this embodiment, the preset convergence condition includes that the scale of the labeled samples is large enough (the sum of the second-type abnormal sample orders whose confidence level of the responsible party is greater than the first threshold and the number of the first-type abnormal sample orders is greater than the second threshold), And for the part of the samples with obvious labels (the first type of sample abnormal orders), the accuracy rate is greater than the third threshold and its recall rate is greater than the fourth threshold, and the accuracy rate is the sample of the first type of sample abnormal orders to determine the responsible party. The ratio to the total sample, the recall rate is the ratio of the sample of a responsible party to the samples of all responsible parties in the first type of sample abnormal order (that is, the sample of the responsible party is the user and the responsible party is the user, driver or online car-hailing platform). The ratio of all samples), when the preset convergence condition is reached, a classification model of accurate abnormal orders is established.

Preferably, the model building unit 502 is specifically configured to receive the first responsible party marking information of the first type of sample abnormal orders obtained according to the first preset order information; and/or according to the second preset order information, for the first type of abnormal orders. The sample abnormal order is marked with the marked information of the second responsible party; wherein the marked information of the responsible party includes the marked information of the first responsible party and the marked information of the second responsible party.

In this embodiment, positive samples are provided for establishing the preset classification model by marking the responsible party of each abnormal order of the first type of samples. When labeling the responsible party of each first-class sample abnormal order, two methods can be included. One is for the labeling team to manually label the first type of sample abnormal orders according to the first preset order information, where the first preset order information is non-obvious information related to the abnormal situation, for example, it needs to ask the driver or user by phone The other is that the system automatically marks the first type of sample abnormal orders with second preset order information, where the second preset order information includes obvious feedback from drivers, users or online car-hailing platform customer service information, such as complaints or bad reviews on the order.

Preferably, the first preset order information includes one or a combination of the following: order track information, historical user order cancellation information, driver historical order cancellation information, user historical negative review or complaint information, and driver historical negative review or complaint information , user and driver call information, user and online car-hailing platform call information, driver and online car-hailing platform call information, online car-hailing platform to user telephone return information, online car-hailing platform to driver telephone return information; second The preset order information includes one or a combination of the following: user negative comment or complaint information, driver bad comment or complaint information.

In this embodiment, the preset order information is also the feature information of each order, and the first preset order information and the second preset order information include but are not limited to the above information. The preset order information can accurately mark the responsible party of the sample abnormal order. For example, when it is determined from the order track information that the driver is going to pick up the user at the user's location, the order is cancelled by the user at a distance of one kilometer from the user's location, and the scheduled pick-up time is not reached at this time, that is, the driver has not timed out. The responsible party for the cancellation of the order is the user. It should be noted that the negative review or complaint information included in the first preset order information is historical information, and the negative review or complaint information included in the second preset order information is the information in this abnormal order.

Preferably, the responsible party for the abnormal order includes any one of the following: a user, a driver or an online car-hailing platform.

In this embodiment, abnormal orders are classified into abnormal orders in which the responsible party is the user, abnormal orders in which the responsible party is the driver, or abnormal orders in which the responsible party is the online car-hailing platform, so as to carry out intervention measures This will reduce order exceptions, thereby ensuring the quality of service for users and protecting the interests of drivers and online car-hailing platforms.

In an embodiment of the third aspect of the embodiment of the present disclosure, a computer device is provided, and FIG. 6 shows a schematic diagram of a computer device 60 according to an embodiment of the embodiment of the present disclosure. Wherein, the computer equipment 60 includes:

A memory 602, a processor 604 and a computer program stored on the memory 602 and executable on the processor 604, when the processor 604 executes the computer program, implements the steps of the order classification method according to any of the above embodiments.

In the computer device 60 provided by the embodiment of the present disclosure, when the processor 604 executes the computer program, the steps of the order classification method in any of the above embodiments are implemented, so the computer device includes all the beneficial effects of the order classification method in any of the above embodiments.

Embodiments of the fourth aspect of the embodiments of the present disclosure provide a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, implements the steps of the order classification method according to any of the foregoing embodiments.

In the computer-readable storage medium provided by the embodiments of the present disclosure, when the computer program is executed by the processor, the steps of the order classification method in any of the foregoing embodiments are implemented. Therefore, the computer-readable storage medium includes the order classification method in any of the foregoing embodiments. all beneficial effects.

In the description of this specification, the terms "first" and "second" are only used for the purpose of description, and should not be construed as indicating or implying relative importance, unless otherwise explicitly specified and limited; the terms "connection", " "Installation" and "fixing" should be understood in a broad sense. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be directly connected or indirectly connected through an intermediate medium. Those of ordinary skill in the art can understand the specific meanings of the above terms in the embodiments of the present disclosure according to specific situations.

In the description of this specification, the description of the terms "one embodiment", "some embodiments", "specific embodiment", etc. means that a particular feature, structure, material or characteristic described in connection with the embodiment or example is included in the present disclosure In at least one embodiment or example of an embodiment. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or instance. Furthermore, the particular features, structures, materials or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

The above descriptions are only preferred embodiments of the embodiments of the present disclosure, and are not intended to limit the embodiments of the present disclosure. For those skilled in the art, various modifications and changes may be made to the embodiments of the present disclosure. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the embodiments of the present disclosure should be included within the protection scope of the embodiments of the present disclosure.

Claims (14)

1. An order classification method, comprising:

acquiring an abnormal order and extracting the characteristic information of the abnormal order;

and inputting the characteristic information of the abnormal order into a preset classification model, and performing responsibility party classification on the abnormal order.

2. The order classification method according to claim 1, further comprising:

collecting a plurality of sample abnormal orders, and dividing the sample abnormal orders into a first type of sample abnormal orders and a second type of abnormal sample orders;

acquiring the responsibility party marking information of each first-class sample abnormal order according to the characteristic information of each first-class sample abnormal order;

training the responsibility party labeling information, and establishing the preset classification model;

inputting the characteristic information of the second-class abnormal sample orders into the preset classification model, performing responsibility party classification on the second-class abnormal sample orders and obtaining responsibility party confidence of each second-class abnormal sample order;

and performing iterative training on the first class of abnormal sample orders and the second class of abnormal sample orders with the responsibility side confidence coefficient larger than a first threshold value until the preset classification model reaches a preset convergence condition.

3. The order classification method according to claim 2,

the preset convergence conditions comprise that the sum of the numbers of the second type abnormal sample orders with the responsibility party confidence degree larger than the first threshold value and the first type abnormal sample orders is larger than a second threshold value, the accuracy rate of the first type abnormal sample orders is larger than a third threshold value, and the recall rate of the first type abnormal sample orders is larger than a fourth threshold value.

4. The order classification method according to claim 2, wherein the obtaining of the liability party labeling information of each abnormal order of the first type includes:

receiving first responsibility party marking information of the first type sample abnormal order acquired according to first preset order information; and/or

According to second preset order information, second responsibility party marking information is marked on the first type sample abnormal order;

the responsibility party marking information comprises the first responsibility party marking information and the second responsibility party marking information.

5. The order classification method according to claim 4,

the first preset order information comprises one or a combination of the following information: order track information, user historical order cancellation information, driver historical order cancellation information, user historical poor evaluation or complaint information, driver historical poor evaluation or complaint information, user and driver communication information, user and network car booking platform communication information, driver and network car booking platform communication information, network car booking platform call return information to users, and network car booking platform call return information to drivers;

the second preset order information comprises one or a combination of the following information: user bad comment or complaint information, driver bad comment or complaint information.

6. The order classification method according to any one of claims 1 to 5,

the responsible party of the abnormal order comprises any one of the following: a user, a driver, or a network appointment platform.

7. An order classification system, comprising:

the characteristic extraction unit is used for acquiring an abnormal order and extracting the characteristic information of the abnormal order;

and the classification unit is used for inputting the characteristic information of the abnormal order into a preset classification model and classifying the abnormal order by responsible parties.

8. The order classification system of claim 7, further comprising:

the model establishing unit is used for acquiring a plurality of sample abnormal orders and dividing the sample abnormal orders into a first type of sample abnormal orders and a second type of sample abnormal orders; acquiring the responsibility party marking information of each first-class sample abnormal order according to the characteristic information of each first-class sample abnormal order; training the responsibility party labeling information, and establishing the preset classification model; inputting the characteristic information of the second-class abnormal sample orders into the preset classification model, performing responsibility party classification on the second-class abnormal sample orders and obtaining responsibility party confidence of each second-class abnormal sample order; and performing iterative training on the first class of abnormal sample orders and the second class of abnormal sample orders with the responsibility side confidence coefficient larger than a first threshold value until the preset classification model reaches a preset convergence condition.

9. The order classification system of claim 8,

the preset convergence conditions comprise that the sum of the numbers of the second type abnormal sample orders with the responsibility party confidence degree larger than the first threshold value and the first type abnormal sample orders is larger than a second threshold value, the accuracy rate of the first type abnormal sample orders is larger than a third threshold value, and the recall rate of the first type abnormal sample orders is larger than a fourth threshold value.

10. The order classification system of claim 8,

the model establishing unit is specifically used for receiving first responsibility party marking information of the first type sample abnormal order acquired according to first preset order information; and/or marking second responsibility party marking information for the first type sample abnormal order according to second preset order information;

the responsibility party marking information comprises the first responsibility party marking information and the second responsibility party marking information.

11. The order classification system of claim 10,

the first preset order information comprises one or a combination of the following information: order track information, user historical order cancellation information, driver historical order cancellation information, user historical poor evaluation or complaint information, driver historical poor evaluation or complaint information, user and driver communication information, user and network car booking platform communication information, driver and network car booking platform communication information, network car booking platform call return information to users, and network car booking platform call return information to drivers;

the second preset order information comprises one or a combination of the following information: user bad comment or complaint information, driver bad comment or complaint information.

12. The order classification system according to any one of claims 7 to 11,

the responsible party of the abnormal order comprises any one of the following: a user, a driver, or a network appointment platform.

13. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor when executing the computer program realizes the steps of the order classification method according to any of claims 1 to 7.

14. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the order classification method according to any one of claims 1 to 7.

Images