CN116701624A

CN116701624A - Data processing method, device and equipment

Info

Publication number: CN116701624A
Application number: CN202310608640.XA
Authority: CN
Inventors: 林鑫; 陈帅
Original assignee: Alipay Hangzhou Information Technology Co Ltd
Current assignee: Alipay Hangzhou Information Technology Co Ltd
Priority date: 2023-05-26
Filing date: 2023-05-26
Publication date: 2023-09-05

Abstract

The embodiment of the specification provides a data processing method, a device and equipment, wherein the method comprises the following steps: acquiring a first text data sample for training a first model, acquiring a type label corresponding to the first text data sample, training the first model based on the first text data sample and the type label corresponding to the first text data sample, acquiring a plurality of candidate rules obtained by rule learning processing of the first text data sample by the first model under the condition that the first model meets the preset convergence condition, acquiring a prediction label output by inputting the first text data sample into the first model, determining classification accuracy corresponding to the candidate rules based on the prediction label and the type label corresponding to the first text data sample, and determining a target rule corresponding to a negative sample in the first text data sample in the candidate rules based on the classification accuracy corresponding to the candidate rules.

Description

Data processing method, device and equipment

Technical Field

The present disclosure relates to the field of data processing technologies, and in particular, to a data processing method, apparatus, and device.

Background

With the rapid development of computer technology, the types and the number of application services provided by enterprises for users are also increasing, and accordingly, the data volume of user data is increasing, and the data structure is becoming complex, which results in higher complexity of detecting anomalies of user data or service data waiting for detecting data.

In the case of performing anomaly detection, since the data to be detected contains more noise feature data and redundant feature data, which results in poor detection efficiency and detection accuracy of anomaly detection, a solution capable of improving the detection efficiency and detection accuracy of anomaly detection of data is required.

Disclosure of Invention

An object of the embodiments of the present disclosure is to provide a data processing method, apparatus, and device, so as to provide a solution capable of improving detection efficiency and detection accuracy of anomaly detection on data.

In order to achieve the above technical solution, the embodiments of the present specification are implemented as follows:

in a first aspect, embodiments of the present disclosure provide a data processing method, including: acquiring a first text data sample for training a first model and a type label corresponding to the first text data sample, wherein the first model is constructed based on a preset rule learning algorithm and is used for generating a model of a rule corresponding to a negative sample in the text data sample; training the first model based on the first text data sample and a type label corresponding to the first text data sample, and acquiring a plurality of candidate rules obtained by the first model through rule learning processing on the first text data sample under the condition that the first model meets a preset convergence condition; acquiring a prediction tag which is output by inputting the first text data sample into the first model, and determining the classification accuracy corresponding to the candidate rule based on the prediction tag and the type tag corresponding to the first text data sample; and determining a target rule corresponding to the negative sample in the first text data sample in the candidate rule based on the classification accuracy corresponding to the candidate rule, wherein the target rule is used for performing anomaly detection processing on the data to be detected.

In a second aspect, embodiments of the present disclosure provide a data processing method, including: acquiring target text data to be detected and a first prediction tag corresponding to the target text data; training a first model based on the target text data and a first prediction tag corresponding to the target text data book, and acquiring a plurality of candidate rules obtained by performing rule learning processing on the target text data by the first model under the condition that the first model meets a preset convergence condition, wherein the first model is constructed based on a preset rule learning algorithm and is used for generating a model of a rule corresponding to a negative sample in a text data sample; acquiring a second prediction tag which is output by inputting the target text data into the first model, and determining the classification accuracy corresponding to the candidate rule based on the second prediction tag and the first prediction tag corresponding to the target text data; determining a target rule corresponding to a negative sample in the target text data in the candidate rule based on the classification accuracy corresponding to the candidate rule; and carrying out anomaly detection processing on the target text data based on the target rule to obtain a second anomaly detection result, and determining the anomaly detection result corresponding to the target text data based on the second anomaly detection result and a first prediction tag corresponding to the target text data.

In a third aspect, embodiments of the present specification provide a data processing apparatus, the apparatus comprising: the system comprises a sample acquisition module, a first model generation module and a model generation module, wherein the sample acquisition module is used for acquiring a first text data sample for training a first model and a type label corresponding to the first text data sample, the first model is constructed based on a preset rule learning algorithm and is used for generating a rule corresponding to a negative sample in the text data sample; the rule acquisition module is used for training the first model based on the first text data sample and the type label corresponding to the first text data sample, and acquiring a plurality of candidate rules obtained by the rule learning processing of the first text data sample by the first model under the condition that the first model meets the preset convergence condition; the label acquisition module is used for acquiring a prediction label which is output by inputting the first text data sample into the first model, and determining the classification accuracy corresponding to the candidate rule based on the prediction label and the type label corresponding to the first text data sample; the rule determining module is used for determining a target rule corresponding to the negative sample in the first text data sample in the candidate rule based on the classification accuracy corresponding to the candidate rule, wherein the target rule is used for carrying out abnormality detection processing on the data to be detected.

In a fourth aspect, embodiments of the present specification provide a data processing apparatus, the apparatus comprising: the data acquisition module is used for acquiring target text data to be detected; the first detection module is used for carrying out abnormality detection processing on the target text data based on the pre-trained second model to obtain a first abnormality detection result; the feature determining module is used for acquiring marginal contribution of feature data contained in the target text data based on the trained third model and determining first feature data in the feature data contained in the target text data based on the marginal contribution of the feature data contained in the target text data; the second detection module is used for carrying out abnormality detection processing on the target text data based on the target rule to obtain a second abnormality detection result; and the result determining module is used for determining an abnormality detection result aiming at the target text data based on the first characteristic data, the first abnormality detection result and the second abnormality detection result in the characteristic data contained in the target text data.

In a fifth aspect, embodiments of the present specification provide a data processing apparatus, the data processing apparatus including: a processor; and a memory arranged to store computer executable instructions that, when executed, cause the processor to: acquiring a first text data sample for training a first model and a type label corresponding to the first text data sample, wherein the first model is constructed based on a preset rule learning algorithm and is used for generating a model of a rule corresponding to a negative sample in the text data sample; training the first model based on the first text data sample and a type label corresponding to the first text data sample, and acquiring a plurality of candidate rules obtained by the first model through rule learning processing on the first text data sample under the condition that the first model meets a preset convergence condition; acquiring a prediction tag which is output by inputting the first text data sample into the first model, and determining the classification accuracy corresponding to the candidate rule based on the prediction tag and the type tag corresponding to the first text data sample; and determining a target rule corresponding to the negative sample in the first text data sample in the candidate rule based on the classification accuracy corresponding to the candidate rule, wherein the target rule is used for performing anomaly detection processing on the data to be detected.

In a sixth aspect, embodiments of the present specification provide a data processing apparatus, including: a processor; and a memory arranged to store computer executable instructions that, when executed, cause the processor to: acquiring target text data to be detected and a first prediction tag corresponding to the target text data; training a first model based on the target text data and a first prediction tag corresponding to the target text data book, and acquiring a plurality of candidate rules obtained by performing rule learning processing on the target text data by the first model under the condition that the first model meets a preset convergence condition, wherein the first model is constructed based on a preset rule learning algorithm and is used for generating a model of a rule corresponding to a negative sample in a text data sample; acquiring a second prediction tag which is output by inputting the target text data into the first model, and determining the classification accuracy corresponding to the candidate rule based on the second prediction tag and the first prediction tag corresponding to the target text data; determining a target rule corresponding to a negative sample in the target text data in the candidate rule based on the classification accuracy corresponding to the candidate rule; and carrying out anomaly detection processing on the target text data based on the target rule to obtain a second anomaly detection result, and determining the anomaly detection result corresponding to the target text data based on the second anomaly detection result and a first prediction tag corresponding to the target text data.

In a seventh aspect, embodiments of the present disclosure provide a storage medium for storing computer-executable instructions that, when executed, implement the following: acquiring a first text data sample for training a first model and a type label corresponding to the first text data sample, wherein the first model is constructed based on a preset rule learning algorithm and is used for generating a model of a rule corresponding to a negative sample in the text data sample; training the first model based on the first text data sample and a type label corresponding to the first text data sample, and acquiring a plurality of candidate rules obtained by the first model through rule learning processing on the first text data sample under the condition that the first model meets a preset convergence condition; acquiring a prediction tag which is output by inputting the first text data sample into the first model, and determining the classification accuracy corresponding to the candidate rule based on the prediction tag and the type tag corresponding to the first text data sample; and determining a target rule corresponding to the negative sample in the first text data sample in the candidate rule based on the classification accuracy corresponding to the candidate rule, wherein the target rule is used for performing anomaly detection processing on the data to be detected.

In an eighth aspect, the present description provides a storage medium for storing computer-executable instructions that when executed implement the following: acquiring target text data to be detected and a first prediction tag corresponding to the target text data; training a first model based on the target text data and a first prediction tag corresponding to the target text data book, and acquiring a plurality of candidate rules obtained by performing rule learning processing on the target text data by the first model under the condition that the first model meets a preset convergence condition, wherein the first model is constructed based on a preset rule learning algorithm and is used for generating a model of a rule corresponding to a negative sample in a text data sample; acquiring a second prediction tag which is output by inputting the target text data into the first model, and determining the classification accuracy corresponding to the candidate rule based on the second prediction tag and the first prediction tag corresponding to the target text data; determining a target rule corresponding to a negative sample in the target text data in the candidate rule based on the classification accuracy corresponding to the candidate rule; and carrying out anomaly detection processing on the target text data based on the target rule to obtain a second anomaly detection result, and determining the anomaly detection result corresponding to the target text data based on the second anomaly detection result and a first prediction tag corresponding to the target text data.

Drawings

In order to more clearly illustrate the embodiments of the present description or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described below, it being obvious that the drawings in the following description are only some of the embodiments described in the present description, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a schematic diagram of a data processing system of the present specification;

FIG. 2A is a flow chart of an embodiment of a data processing method of the present disclosure;

FIG. 2B is a schematic diagram illustrating a data processing method according to the present disclosure;

FIG. 3 is a schematic diagram illustrating a processing procedure of another data processing method according to the present disclosure;

FIG. 4 is a schematic diagram of a model training process of the present disclosure;

FIG. 5A is a flowchart illustrating an embodiment of a data processing method according to the present disclosure;

FIG. 5B is a schematic diagram illustrating a data processing method according to the present disclosure;

FIG. 6 is a schematic diagram illustrating a processing procedure of another data processing method according to the present disclosure;

FIG. 7 is a schematic diagram of an embodiment of a data processing apparatus according to the present disclosure;

FIG. 8 is a schematic diagram of an embodiment of a data processing apparatus according to the present disclosure;

fig. 9 is a schematic diagram of a data processing apparatus of the present specification.

Detailed Description

The embodiment of the specification provides a data processing method, a device and equipment.

In order to make the technical solutions in the present specification better understood by those skilled in the art, the technical solutions in the embodiments of the present specification will be clearly and completely described below with reference to the drawings in the embodiments of the present specification, and it is obvious that the described embodiments are only some embodiments of the present specification, not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are intended to be within the scope of the present disclosure.

The technical scheme of the specification can be applied to a data processing system, as shown in fig. 1, the data processing system can be provided with terminal equipment and a server, wherein the server can be an independent server or a server cluster formed by a plurality of servers, and the terminal equipment can be equipment such as a personal computer and the like or mobile terminal equipment such as a mobile phone, a tablet personal computer and the like.

The data processing system may include n terminal devices and m servers, where n and m are positive integers greater than or equal to 1, where the terminal devices may be configured to collect data samples, for example, the terminal devices may obtain corresponding data samples for different anomaly detection scenarios, for example, for a data anomaly detection scenario of the question-answering system, the terminal devices may collect feedback information of a user-needle session as the data samples, for a data anomaly detection scenario of a preset service, the terminal devices may collect service data corresponding to the preset service (such as data required for executing the preset service) as the data samples, and so on.

The terminal device may send the collected data samples to any server in the data processing system, and the server may preprocess the received data samples, and store the preprocessed data samples as text data samples. Among other things, the preprocessing operations may include text conversion preprocessing (i.e., converting audio data into text data, etc.), text format conversion processing (e.g., converting english text into chinese text, etc.), and the like.

The server can perform anomaly detection, anomaly attribution and policy recommendation based on the preprocessed data, wherein the anomaly detection refers to a process of discovering an anomaly value by using a technical means. Outliers, also known as outliers, refer to data points that are significantly different from, or do not conform to, the expected normal pattern represented by most of the data points. Outliers are generally characterized by two characteristics: few and different. That is, the outliers have a small duty cycle in the overall sample, and the outliers are different from most samples. The anomaly detection can be applied to the scenes of fraud detection, account theft, fault investigation and the like. The abnormality attribution means that the server can judge that each abnormal point is judged as the cause of the abnormality after detecting the abnormal point by the abnormality detection technique. The policy recommendation means that the server can summarize the commonalities of a large number of abnormal points, and extract the summarized commonalities into policies to be recommended to a service processor, so that the purpose of covering the abnormal points by using a plurality of policies is achieved, and further deep cognition of the commonalities of the abnormal points is achieved.

In addition, the terminal device can also send the collected data samples to the corresponding servers based on the application scenes corresponding to the data samples. For example, assuming that the server 1 and the server 2 are used for processing data anomaly detection of the question-answering system and the server 3 and the server 4 are used for processing data anomaly detection of the preset service in the data processing system, the terminal device may send the collected data samples corresponding to the question-answering system to the server 1 and the server 2 and send the collected data samples corresponding to the preset service to the server 3 and the server 4.

In this way, the server, upon receiving the training instruction for the first model, can train the first model with the text data sample corresponding to the first model as the first text data sample.

In addition, there may be a central server (e.g., server 1) in the data processing system, where the central server is configured to train the first model to be trained based on the first text data samples sent by the other servers (e.g., server 2 and server 3) when the model training period is reached, and determine a target rule corresponding to the server, where the central server may return the target rule to the corresponding server. Thus, other servers in the data processing system can provide business services for users without interruption, and perform abnormality detection processing on business data.

Because noise may exist in the text data sample acquired by the server, that is, the acquired first text data sample may contain noise feature data and redundant feature data, rule learning processing can be performed on the first text data sample through a first model constructed based on a preset rule learning algorithm to obtain a plurality of candidate rules, and then a target rule corresponding to a negative sample in the first text data sample is determined through classification accuracy corresponding to the candidate rules.

The data processing method in the following embodiments can be implemented based on the above-described data processing system configuration.

Example 1

As shown in fig. 2A and fig. 2B, the embodiment of the present disclosure provides a data processing method, where an execution body of the method may be a server, and the server may be an independent server or may be a server cluster formed by a plurality of servers. The method specifically comprises the following steps:

in S202, a first text data sample for training a first model and a type tag corresponding to the first text data sample are acquired.

The first model may be a model constructed based on a preset rule learning algorithm and used for generating a rule corresponding to a negative sample in the text data samples, the first text data samples may be any text data sample to be detected, for example, the first text data samples may be any user data and/or preset service data acquired in a model training period, etc., and type labels corresponding to the first text data samples may be used for representing abnormal conditions of the first text data samples, for example, the type labels corresponding to the first text data samples may be high risk, medium risk, low risk, no risk, etc.

In implementation, with the rapid development of computer technology, the types and the number of application services provided by enterprises for users are also increasing, and accordingly, the data volume of user data is increasing, and the data structure is becoming complex, which results in higher complexity of anomaly detection on user data or service data waiting for detection data. In the case of performing anomaly detection, since the data to be detected contains more noise feature data and redundant feature data, which results in poor detection efficiency and detection accuracy of anomaly detection, a solution capable of improving the detection efficiency and detection accuracy of anomaly detection of data is required. For this reason, the embodiments of the present specification provide a technical solution that can solve the above-mentioned problems, and specifically, reference may be made to the following.

The server may obtain a first text data sample for training the first model when the model training period is reached, for example, the server may receive service data corresponding to a preset service sent by the terminal device and/or other servers in the data processing system when the model training period is reached, perform preprocessing on the received service data, and determine the text data obtained by the preprocessing as the first text data sample for training the first model.

In S204, training the first model based on the first text data sample and the type label corresponding to the first text data sample, and obtaining a plurality of candidate rules obtained by performing rule learning processing on the first text data sample by the first model when the first model meets a preset convergence condition.

The candidate rule may be used to classify the first text data sample according to feature data contained in the first text data sample, that is, the candidate rule may be used to determine a prediction label corresponding to the first text data sample, and the prediction label corresponding to the first text data sample may be used to determine a positive sample and a negative sample in the first text data sample.

In an implementation, the server may input the first text data sample into the first model to obtain a prediction tag corresponding to the first text data sample, the server may determine a loss value corresponding to the first model based on the type tag corresponding to the first text data sample and the prediction tag corresponding to the first text data sample, determine whether the first model converges based on the loss value, and if the first model does not converge, the server may continue training the first model until the first model converges based on the first text data sample and the type tag corresponding to the first text data sample, to obtain the trained first model.

In addition, the above method for determining whether the first model meets the preset convergence condition is an optional and implementable determination method, and in the actual application scenario, there may be a plurality of different determination methods, and the determination method may be different according to the actual application scenario, which is not specifically limited in the embodiment of the present disclosure.

The server may obtain a plurality of candidate rules for classifying the first text data sample during the training of the first model. Taking the first text data sample as service data corresponding to the resource transfer service as an example, the type label corresponding to the first text data sample may include risk and risk-free, and the candidate rule may be a prediction label for determining the first text data sample by the resource transfer number and the resource transfer time in the first text data sample, where the candidate rule 1 is: if the number of resource transfers is greater than 300, the predictive label corresponding to the first text data sample may be risky, otherwise, the predictive label corresponding to the first text data sample is risky, and candidate rule 2 is: if the resource transfer time is approximately 3 days, the prediction label of the first text data sample may be risk-free, otherwise, the prediction label corresponding to the first text data sample is risk-free, and the candidate rule 3 is: if the resource transfer time is approximately 1 day and the number of resource transfers is greater than 500, the predictive label of the first text data sample may be risky, otherwise, the predictive label corresponding to the first text data sample is risky. When the predictive label corresponding to the first text data sample is at risk, the first text data sample can be a negative sample, and when the predictive label corresponding to the first text data sample is no risk, the first text data sample can be a positive sample.

In S206, a prediction tag output by inputting the first text data sample into the first model is acquired, and a classification accuracy corresponding to the candidate rule is determined based on the prediction tag and a type tag corresponding to the first text data sample.

In implementation, the server may obtain a first text data sample corresponding to each candidate rule, and determine a classification accuracy corresponding to each candidate rule based on a prediction tag and a type tag corresponding to the first text data sample corresponding to each candidate rule.

The classification accuracy may be determined by the data amount of the first text data sample corresponding to the candidate rule and the data amount of the first text data sample with the same predictive label and the same type label in the first text data sample, or, if the type label and the predictive label corresponding to the first text data sample are determined by a probability value between 0 and 1 (i.e. if the probability value is greater than 0.5 and less than 1, the type label and/or the predictive label may be label 1, and if the probability value is greater than 0 and not greater than 0.5, the type label and/or the predictive label may be label 2), the server may further determine the classification accuracy based on a difference value between the probability values corresponding to the labels, for example, an average value (or a sum value, a maximum value, etc.) of the difference value between the probability value of each first text data sample corresponding to the candidate rule and the probability value of the predictive label may be determined as the classification accuracy corresponding to the candidate rule.

The method for determining the classification accuracy corresponding to the candidate rule is an optional and implementable method, and in an actual application scene, there may be a plurality of different determining methods, and may be different according to different actual application scenes, which is not specifically limited in the embodiment of the present disclosure.

In S208, a target rule of the candidate rules corresponding to the negative sample in the first text data sample is determined based on the classification accuracy corresponding to the candidate rule.

The target rule may be used to perform anomaly detection processing on the data to be detected.

In an implementation, the server may determine a sub-rule for determining a negative sample from among candidate rules having the greatest classification accuracy among the candidate rules as a target rule corresponding to the negative sample of the first text data sample.

For example, taking a first text data sample as service data corresponding to a resource transfer service as an example, a candidate rule with the highest classification accuracy among candidate rules is: if the number of resource transfers is greater than 300, the predictive label corresponding to the first text data sample may be risky, otherwise, the predictive label corresponding to the first text data sample is risky. The server may determine the target rule as: if the number of resource transfers is greater than 300, the prediction tag corresponding to the first text data sample may be risky, that is, the target rule determined by the server may be used to detect whether the data to be detected has an abnormality, if the number of resource transfers in the data to be detected is greater than 300, the server may determine that the data to be detected has an abnormality.

The above method for determining the target rule is an optional and implementable determining method, and in an actual application scenario, there may be a plurality of different determining methods, for example, the server may determine a sub-rule for determining a negative sample in candidate rules with a classification accuracy greater than a preset classification threshold value, as a target rule corresponding to the negative sample of the first text data sample, and the like, which may be different according to the actual application scenario, and this embodiment of the present disclosure is not limited specifically.

After determining the target rule, the server may send the target rule to other servers and/or terminal devices in the data processing system, so that the other servers and/or terminal devices in the data processing system may perform anomaly detection on the data to be detected through the target rule.

The embodiment of the specification provides a data processing method, which is used for acquiring a first text data sample for training a first model and a type label corresponding to the first text data sample, wherein the first model is constructed based on a preset rule learning algorithm and is used for generating a model of a rule corresponding to a negative sample in the text data sample, training the first model based on the first text data sample and the type label corresponding to the first text data sample, acquiring a plurality of candidate rules obtained by the rule learning processing of the first text data sample by the first model when the first model meets a preset convergence condition, acquiring a prediction label output by inputting the first text data sample into the first model, determining classification accuracy corresponding to the candidate rules based on the prediction label and the type label corresponding to the first text data sample, determining a target rule corresponding to the negative sample in the first text data sample in the candidate rules based on the classification accuracy corresponding to the candidate rules, and performing abnormality detection processing on data to be detected by the target rules. Therefore, the target rule is determined through the classification accuracy corresponding to the candidate rule, so that the identification accuracy of the negative sample in the text data sample can be improved through the target rule, and the coverage of the negative sample in the text data sample can be realized through the target rule, so that the detection efficiency and the detection accuracy of the abnormal detection of the data can be improved through the abnormal detection processing of the data to be detected through the target rule.

Example two

As shown in fig. 3, the embodiment of the present disclosure provides a data processing method, where an execution body of the method may be a server, where the server may be an independent server or may be a server cluster formed by a plurality of servers. The method specifically comprises the following steps:

in S202, a first text data sample for training a first model is acquired.

In S302, based on the second model trained in advance, the abnormality detection processing is performed on the first text data sample, and an abnormality detection result of the first text data sample is obtained.

The second model is an unsupervised learning model constructed based on a preset machine learning algorithm, for example, the second model may be a tree model (such as an iferst model), a density-based model (such as a DBSACN-based model, a LOF-based model), a linear algorithm-based model (such as an OCSVM-based or PCA-based model), a neural network algorithm-based model (such as an Auto Encoder-based or DAGMM-based model), a statistical distribution algorithm-based model (such as a gaussian distribution algorithm-based or HBOS-based model), and the like.

In implementation, when abnormality detection is performed, feature construction is required, that is, related staff can construct feature data according to own experience and professional knowledge in related fields, and the constructed feature set can be used as an alternative pool for subsequent feature screening. Because the feature set is constructed manually, useless features, noise features, redundant features and the like may exist in the feature set inevitably, and further screening of the feature data is needed to ensure the effect of subsequent anomaly detection.

The server may input the first text data sample into a pre-trained second model to obtain an abnormality detection result of the first text data sample, where the abnormality detection result of the first text data sample may include a prediction type corresponding to the first text data sample obtained by classifying the first text data sample based on the pre-trained second model.

In S304, training the third model based on the first text data sample, to obtain a trained third model, and determining a marginal contribution degree of the feature data included in the first text data sample based on the trained third model.

The third model may be a tree model that determines a marginal contribution of the feature data included in the text data sample based on model parameters corresponding to the feature data included in the text data sample.

In an implementation, the server may obtain a first marginal contribution of the feature data included in the first text data sample under different feature sets based on the trained third model, and determine the marginal contribution of the feature data included in the first text data sample based on the first marginal contribution.

Taking the third model as an XGB tree model as an example, the XGB tree model can be trained through the first text data sample, and further the shape interpretability of the trained XGB tree model is used for attributing the abnormal points. The shape value can be used to solve the contribution and revenue distribution problems of the collaborative game. In multi-person collaboration, the contributions to the members are not the same because the contributions from the individual members are not the same, and therefore, the ideal distribution is such that the contributions are equal to the gains. The main ideas in the interpretability are: the shape value of a feature data can be obtained by calculating the marginal contribution of the feature data when it is added to the model and then weighting and summing the marginal contributions of the feature data in the case of different feature sequences.

For example, an input formula may be entered

Obtaining marginal contribution degree of characteristic data contained in the first text data sample, wherein,for the marginal contribution degree of the ith feature data in the first text data sample, M is the number of feature data in the first text data sample, N is the set containing all feature data in the first text data sample, S is the set with the ith feature data removed from N, S is the number of elements contained in the set S, f _x (S.u.i.) is the predicted value, f, output by the third model when only S.u.i.. Feature data is present in the first text data sample _x (S) is the predicted value, f, outputted by the third model when there is only S feature data in the first text data sample _x (S { i }) and f _x The difference value of the step (S) is the first marginal contribution degree of the ith feature data under the subset S, and the marginal contribution degree of the ith feature data can be obtained by adding the first marginal contribution degree of the ith feature data under the subset S.

In S306, target feature data among the feature data included in the first text data sample is determined based on the marginal contribution of the feature data included in the first text data sample.

In implementations, the feature data may be ranked based on a marginal contribution of the feature data, and the target feature data may be determined based on the ranked feature data. For example, the feature data with the marginal contribution degree greater than the preset contribution degree threshold may be determined as target feature data, or the first n feature data (i.e., n feature data with greater marginal contribution degrees) in the sorted feature data may be determined as target feature data, where n may be a positive integer determined based on the service scenario corresponding to the first text data sample, or the like.

In S308, a type tag corresponding to the first text data sample is determined based on the abnormality detection result of the first text data sample and the target feature data.

The first model may be a model constructed based on a preset rule learning algorithm for generating rules corresponding to negative samples in the text data samples.

In implementation, the target feature data may be used to characterize the importance of the feature data, and the anomaly detection result of the first text data sample may be explained by the target feature data, so as to assist in determining a type tag corresponding to the first text data sample.

Further, the preset rule learning algorithm may include a sequential overlay algorithm. The sequential coverage algorithm may be to delete the training data samples covered by a rule every time a rule is learned on the training set, then compose a new training set with the remaining training data samples, and repeat the above procedure. For example, as shown in fig. 4, the training data may include a plurality of data samples, when the model learns rule 1, the data sample corresponding to rule 1 may be deleted, and model training may be performed based on the deleted training data, after learning rule 2, the data sample corresponding to rule 2 may be deleted, and model training may be performed based on the deleted training data until the model converges.

In S310, a first text data sample is sampled to obtain a first training set, and a first model is trained based on the first training set to obtain a first candidate rule corresponding to the first training set.

In an implementation, taking a sequential coverage algorithm as an example, the server may sample the first text data sample to obtain a first training set, and train the first model based on the first training set to obtain a first candidate rule corresponding to the first training set.

In S312, a second training set is determined based on the first text data samples other than the first training set among the first text data samples.

In an implementation, the server may sample the first text data samples except for the first training set in the first text data samples to obtain the second training set.

In practical applications, the processing manner of S312 may be varied, and the following provides an alternative implementation manner, which may be specifically referred to the following steps one to three:

step one, determining the classification accuracy corresponding to a first candidate rule based on a prediction label corresponding to a first training set output by a first model and a type label corresponding to the first training set.

And step two, screening the first training set based on the classification accuracy corresponding to the first candidate rule to obtain a screened first training set.

And thirdly, determining a second training set based on the first text data samples except the screened first training set in the first text data samples.

In implementation, through screening processing of the first training set, the training efficiency and accuracy of the subsequent model can be improved.

In S314, the first model is trained based on the second training set, resulting in a second candidate rule corresponding to the second training set.

In S316, a plurality of candidate rules are determined based on the first candidate rule and the second candidate rule.

In an implementation, the server may further determine a third training set based on the first text data samples except the first training set and the second training set, train the first model based on the third training set to obtain a third candidate rule corresponding to the third training set, and continue sampling and training the remaining first text data samples until sampling of all the first text data samples is completed, so as to obtain a plurality of candidate rules.

In S318, target text data to be detected is acquired.

The target text data may include service data corresponding to the execution target service, for example, assuming that the target service is a resource transfer service, the target text data may include data such as interaction data, resource transfer time, and resource transfer number between the user and the resource transfer object.

In implementation, when the user triggers the execution of the target service through the terminal device, the terminal device may collect service data corresponding to the execution of the target service and send the collected service data to the corresponding server. The server can preprocess the received service data to obtain target text data to be detected.

Or when the server detects that the target service is abnormal in operation, the server can acquire service data corresponding to the target service, and can preprocess the service data to obtain target text data to be detected.

In S320, based on the second model trained in advance, the abnormality detection processing is performed on the target text data, and the first abnormality detection result is obtained.

In implementation, the server may input the target text data sample into a pre-trained second model, so as to classify the target text data sample through the second model, and obtain a first anomaly detection result corresponding to the target text data.

In S322, based on the trained third model, a marginal contribution of feature data included in the target text data is obtained, and based on the marginal contribution of feature data included in the target text data, first feature data in the feature data included in the target text data is determined.

In implementation, the marginal contribution degree of the feature data included in the target text data and the determination process of the first feature data may refer to the processing procedures of S304 to S306, which are not described herein.

In S324, the target text data is subjected to abnormality detection processing based on the target rule, and a second abnormality detection result is obtained.

In S326, an abnormality detection result for the target text data is determined based on the first feature data, the first abnormality detection result, and the second abnormality detection result in the feature data included in the target text data.

In the implementation, since the target rule is obtained by refining the commonality of the negative sample in the first text data sample, and the first feature data is attributed to the abnormality determination of the target text data, the abnormality detection result for the target text data can be accurately determined by the first feature data, the first abnormality detection result, and the second abnormality detection result in the feature data included in the target text data.

The embodiment of the specification provides a data processing method, obtain a first text data sample for training a first model, and a type label corresponding to the first text data sample, the first model is constructed based on a preset rule learning algorithm, and is used for generating a model of a rule corresponding to a negative sample in the text data sample, training the first model based on the first text data sample and the type label corresponding to the first text data sample, and under the condition that the first model meets a preset convergence condition, obtain a plurality of candidate rules obtained by performing rule learning processing on the first text data sample by the first model, obtain a prediction label output by inputting the first text data sample into the first model, determine classification accuracy corresponding to the candidate rules based on the prediction label and the type label corresponding to the first text data sample, determine a target rule corresponding to the negative sample in the first text data sample in the candidate rules based on the classification accuracy corresponding to the candidate rules, and the target rule is used for performing abnormality detection processing on data to be detected. Therefore, the target rule is determined through the classification accuracy corresponding to the candidate rule, so that the identification accuracy of the negative sample in the text data sample can be improved through the target rule, and the coverage of the negative sample in the text data sample can be realized through the target rule, so that the detection efficiency and the detection accuracy of the abnormal detection of the data can be improved through the abnormal detection processing of the data to be detected through the target rule.

Example III

As shown in fig. 5A and fig. 5B, the embodiment of the present disclosure provides a data processing method, where an execution body of the method may be a server, and the server may be an independent server or a server cluster formed by a plurality of servers. The method specifically comprises the following steps:

in S502, target text data to be detected and a first prediction tag corresponding to the target text data are acquired.

The target text data may include service data corresponding to the execution target service, for example, assuming that the target service is a resource transfer service, the target text data may include data such as interaction data, resource transfer time, and resource transfer number between the user and the resource transfer object, and the first prediction label corresponding to the target text data may be used to represent an abnormal situation of the target text data, for example, the first prediction label corresponding to the target text data may be high risk, medium risk, low risk, no risk, and the like.

In S504, training the first model based on the target text data and the first prediction tag corresponding to the target text data book, and obtaining a plurality of candidate rules obtained by performing rule learning processing on the target text data by the first model when the first model meets a preset convergence condition.

In S506, a second prediction tag, which is output by inputting the target text data into the first model, is acquired, and a classification accuracy corresponding to the candidate rule is determined based on the second prediction tag and the first prediction tag corresponding to the target text data.

In S508, a target rule corresponding to the negative sample in the target text data in the candidate rule is determined based on the classification accuracy corresponding to the candidate rule.

The specific processing procedures of S504 to S508 can be referred to the relevant contents of S204 to S208 in the first embodiment, and are not described herein.

In S510, based on the target rule, performing an anomaly detection process on the target text data to obtain a second anomaly detection result, and determining an anomaly detection result corresponding to the target text data based on the second anomaly detection result and the first prediction tag corresponding to the target text data.

In implementation, the target rule is determined by the classification accuracy corresponding to the candidate rule, so that the recognition accuracy of the negative sample in the target text data can be improved through the target rule, and the coverage of the negative sample in the target text data can be realized through the target rule, namely, the server can determine the abnormal detection result corresponding to the target text data based on the second abnormal detection result and the first prediction label corresponding to the target text data, and the abnormal detection accuracy of the target text data is improved.

The embodiment of the specification provides a data processing method, which comprises the steps of obtaining target text data to be detected, obtaining a first prediction label corresponding to the target text data, training a first model based on the target text data and the first prediction label corresponding to the target text data, obtaining a plurality of candidate rules obtained by carrying out rule learning processing on the target text data by the first model under the condition that the first model meets a preset convergence condition, wherein the first model is constructed based on a preset rule learning algorithm and is used for generating a rule corresponding to a negative sample in a text data sample, obtaining a second prediction label which is output by inputting the target text data into the first model, determining a classification accuracy corresponding to the candidate rules based on the second prediction label and the first prediction label corresponding to the target text data, determining a target rule corresponding to the negative sample in the target text data in the candidate rules based on the classification accuracy corresponding to the candidate rules, carrying out abnormality detection processing on the target text data based on the target rules, obtaining a second abnormality detection result, and determining an abnormality detection result corresponding to the target text data based on the second abnormality detection result and the first prediction label corresponding to the target text data. Therefore, the target rule is determined through the classification accuracy corresponding to the candidate rule, so that the recognition accuracy of the negative sample in the text data sample can be improved through the target rule, and the coverage of the negative sample in the text data sample can be realized through the target rule, so that the detection efficiency and the detection accuracy of the anomaly detection of the data can be improved based on the second anomaly detection result obtained by performing anomaly detection processing on the target text data through the target rule and combining the first predictive label of the target text data.

Example IV

As shown in fig. 6, the embodiment of the present disclosure provides a data processing method, where an execution body of the method may be a server, where the server may be an independent server or may be a server cluster formed by a plurality of servers. The method specifically comprises the following steps:

in S502, target text data to be detected is acquired.

In S602, based on the second model trained in advance, abnormality detection processing is performed on the target text data, so as to obtain a first prediction tag corresponding to the target text data.

The second model may be an unsupervised learning model constructed based on a preset machine learning algorithm, for example, the second model may be a tree model (such as an ifest model), a density-based model (such as a DBSACN-based model, a LOF-based model), a linear algorithm-based model (such as an OCSVM-based or PCA-based model), a neural network algorithm-based model (such as an Auto Encoder-based or DAGMM-based model), a statistical distribution algorithm-based model (such as a gaussian distribution algorithm-based or HBOS-based model), and the like.

In implementation, the server may input the target text data into the pre-trained second model to obtain a first anomaly detection result corresponding to the target text data, and the server may determine the first prediction tag corresponding to the target text data based on the first anomaly detection result.

Or, the server may further train the third model based on the target text data to obtain a trained third model, and determine a marginal contribution degree of the feature data included in the target text data based on the trained third model. And determining target feature data in the feature data contained in the target text data based on the marginal contribution degree of the feature data contained in the target text data.

The server may determine a first predictive tag corresponding to the target text data based on the first abnormality detection result of the target text data and the target feature data.

The determination process of the marginal contribution degree of the feature data included in the target text data, the target feature data, and the first anomaly detection result of the target text data may refer to the processing procedures of S302 to S308 in the second embodiment, which are not described herein.

Example five

The data processing method provided in the embodiment of the present disclosure is based on the same concept, and the embodiment of the present disclosure further provides a data processing device, as shown in fig. 7.

The data processing apparatus includes: a sample acquisition module 701, a rule acquisition module 702, a tag acquisition module 703, and a rule determination module 704, wherein:

the sample obtaining module 701 is configured to obtain a first text data sample for training a first model, and a type tag corresponding to the first text data sample, where the first model is constructed based on a preset rule learning algorithm, and is used to generate a model of a rule corresponding to a negative sample in the text data sample;

the rule obtaining module 702 is configured to train the first model based on the first text data sample and a type tag corresponding to the first text data sample, and obtain a plurality of candidate rules obtained by performing rule learning processing on the first text data sample by the first model when the first model meets a preset convergence condition;

a tag obtaining module 703, configured to obtain a prediction tag output by inputting the first text data sample into the first model, and determine a classification accuracy corresponding to the candidate rule based on the prediction tag and a type tag corresponding to the first text data sample;

The rule determining module 704 is configured to determine, based on the classification accuracy corresponding to the candidate rule, a target rule corresponding to a negative sample in the first text data sample in the candidate rule, where the target rule is used to perform anomaly detection processing on data to be detected.

In this embodiment of the present disclosure, the preset rule learning algorithm includes a sequential coverage algorithm, and the rule obtaining module 702 is configured to:

sampling the first text data sample to obtain a first training set, and training the first model based on the first training set to obtain a first candidate rule corresponding to the first training set;

determining a second training set based on a first text data sample of the first text data samples other than the first training set;

training the first model based on the second training set to obtain a second candidate rule corresponding to the second training set;

the plurality of candidate rules is determined based on the first candidate rule and the second candidate rule.

In the embodiment of the present disclosure, the rule obtaining module 702 is configured to:

determining the classification accuracy corresponding to the first candidate rule based on the prediction label corresponding to the first training set and the type label corresponding to the first training set output by the first model;

Screening the first training set based on the classification accuracy corresponding to the first candidate rule to obtain a screened first training set;

and determining the second training set based on the first text data samples except the screened first training set in the first text data samples.

In the embodiment of the present disclosure, the sample acquiring module 701 is configured to:

performing anomaly detection processing on the first text data sample based on a pre-trained second model to obtain an anomaly detection result of the first text data sample, wherein the second model is an unsupervised learning model constructed based on a preset machine learning algorithm;

training a third model based on the first text data sample to obtain a trained third model, determining the marginal contribution of the feature data contained in the first text data sample based on the trained third model, wherein the third model is a tree model for determining the marginal contribution of the feature data contained in the text data sample based on model parameters corresponding to the feature data contained in the text data sample;

determining target feature data in the feature data contained in the first text data sample based on marginal contribution degree of the feature data contained in the first text data sample;

And determining a type tag corresponding to the first text data sample based on an abnormality detection result of the first text data sample and the target feature data.

and based on the trained third model, acquiring first marginal contribution degrees of the characteristic data contained in the first text data sample under different characteristic sets, and determining the marginal contribution degrees of the characteristic data contained in the first text data sample based on the first marginal contribution degrees.

In an embodiment of the present disclosure, the apparatus further includes:

the data acquisition module is used for acquiring target text data to be detected;

the first detection module is used for carrying out abnormality detection processing on the target text data based on the pre-trained second model to obtain a first abnormality detection result;

the feature determining module is used for acquiring marginal contribution of feature data contained in the target text data based on the trained third model and determining first feature data in the feature data contained in the target text data based on the marginal contribution of the feature data contained in the target text data;

The second detection module is used for carrying out abnormality detection processing on the target text data based on the target rule to obtain a second abnormality detection result;

and the result determining module is used for determining an abnormality detection result aiming at the target text data based on the first characteristic data, the first abnormality detection result and the second abnormality detection result in the characteristic data contained in the target text data.

The embodiment of the specification provides a data processing device, which is used for acquiring a first text data sample for training a first model and a type label corresponding to the first text data sample, wherein the first model is constructed based on a preset rule learning algorithm and is used for generating a model of a rule corresponding to a negative sample in the text data sample, training the first model based on the first text data sample and the type label corresponding to the first text data sample, acquiring a plurality of candidate rules obtained by the rule learning processing of the first text data sample by the first model when the first model meets a preset convergence condition, acquiring a prediction label output by inputting the first text data sample into the first model, determining classification accuracy corresponding to the candidate rules based on the prediction label and the type label corresponding to the first text data sample, determining a target rule corresponding to the negative sample in the first text data sample in the candidate rules based on the classification accuracy corresponding to the candidate rules, and performing abnormality detection processing on data to be detected by the target rules. Therefore, the target rule is determined through the classification accuracy corresponding to the candidate rule, so that the identification accuracy of the negative sample in the text data sample can be improved through the target rule, and the coverage of the negative sample in the text data sample can be realized through the target rule, so that the detection efficiency and the detection accuracy of the abnormal detection of the data can be improved through the abnormal detection processing of the data to be detected through the target rule.

Example six

The data processing method provided in the embodiment of the present disclosure is based on the same concept, and the embodiment of the present disclosure further provides a data processing device, as shown in fig. 8.

The data processing apparatus includes: a data acquisition module 801, a rule acquisition module 802, a tag acquisition module 803, a rule determination module 804, and an anomaly detection module 805, wherein:

the data acquisition module 801 is configured to acquire target text data to be detected and a first prediction tag corresponding to the target text data;

a rule obtaining module 802, configured to train a first model based on the target text data and a first prediction tag corresponding to the target text data book, and obtain a plurality of candidate rules obtained by performing rule learning processing on the target text data by the first model when the first model meets a preset convergence condition, where the first model is a model constructed based on a preset rule learning algorithm and is used to generate a rule corresponding to a negative sample in a text data sample;

a tag obtaining module 803, configured to obtain a second prediction tag that is output by inputting the target text data into the first model, and determine a classification accuracy corresponding to the candidate rule based on the second prediction tag and a first prediction tag corresponding to the target text data;

A rule determining module 804, configured to determine, based on the classification accuracy corresponding to the candidate rule, a target rule corresponding to the negative sample in the target text data in the candidate rule;

the anomaly detection module 805 is configured to perform anomaly detection processing on the target text data based on the target rule to obtain a second anomaly detection result, and determine an anomaly detection result corresponding to the target text data based on the second anomaly detection result and a first prediction tag corresponding to the target text data.

In this embodiment of the present disclosure, the first prediction label corresponding to the target text data is a label obtained by performing anomaly detection processing on the target text data based on a pre-trained second model, where the second model is an unsupervised learning model constructed based on a preset machine learning algorithm.

The embodiment of the specification provides a data processing device, which is used for acquiring target text data to be detected and a first prediction label corresponding to the target text data, training a first model based on the target text data and the first prediction label corresponding to the target text data book, acquiring a plurality of candidate rules obtained by performing rule learning processing on the target text data by the first model under the condition that the first model meets a preset convergence condition, wherein the first model is constructed based on a preset rule learning algorithm and is used for generating a model of a rule corresponding to a negative sample in a text data sample, acquiring a second prediction label which is output by inputting the target text data into the first model, determining a classification accuracy corresponding to the candidate rules based on the second prediction label and the first prediction label corresponding to the target text data, determining a target rule corresponding to the negative sample in the target text data in the candidate rules based on the classification accuracy corresponding to the candidate rules, performing abnormality detection processing on the target text data based on the target rules to obtain a second abnormality detection result, and determining an abnormality detection result corresponding to the target text data based on the second abnormality detection result and the first prediction label corresponding to the target text data. Therefore, the target rule is determined through the classification accuracy corresponding to the candidate rule, so that the recognition accuracy of the negative sample in the text data sample can be improved through the target rule, and the coverage of the negative sample in the text data sample can be realized through the target rule, so that the detection efficiency and the detection accuracy of the anomaly detection of the data can be improved based on the second anomaly detection result obtained by performing anomaly detection processing on the target text data through the target rule and combining the first predictive label of the target text data.

Example seven

Based on the same idea, the embodiment of the present disclosure further provides a data processing apparatus, as shown in fig. 9.

The data processing apparatus may vary widely in configuration or performance, may include one or more processors 901 and memory 902, and may store one or more storage applications or data in memory 902. Wherein the memory 902 may be transient storage or persistent storage. The application programs stored in the memory 902 may include one or more modules (not shown) each of which may include a series of computer executable instructions for use in a data processing apparatus. Still further, the processor 901 may be arranged to communicate with a memory 902 and execute a series of computer executable instructions in the memory 902 on a data processing device. The data processing device may also include one or more power supplies 903, one or more wired or wireless network interfaces 904, one or more input output interfaces 905, and one or more keyboards 906.

In particular, in this embodiment, the data processing apparatus includes a memory, and one or more programs, wherein the one or more programs are stored in the memory, and the one or more programs may include one or more modules, and each module may include a series of computer-executable instructions for the data processing apparatus, and the one or more programs configured to be executed by the one or more processors comprise instructions for:

Acquiring a first text data sample for training a first model and a type label corresponding to the first text data sample, wherein the first model is constructed based on a preset rule learning algorithm and is used for generating a model of a rule corresponding to a negative sample in the text data sample;

training the first model based on the first text data sample and a type label corresponding to the first text data sample, and acquiring a plurality of candidate rules obtained by the first model through rule learning processing on the first text data sample under the condition that the first model meets a preset convergence condition;

acquiring a prediction tag which is output by inputting the first text data sample into the first model, and determining the classification accuracy corresponding to the candidate rule based on the prediction tag and the type tag corresponding to the first text data sample;

and determining a target rule corresponding to the negative sample in the first text data sample in the candidate rule based on the classification accuracy corresponding to the candidate rule, wherein the target rule is used for performing anomaly detection processing on the data to be detected.

In addition, in particular in the present embodiment, the data processing apparatus includes a memory, and one or more programs, wherein the one or more programs are stored in the memory, and the one or more programs may include one or more modules, and each module may include a series of computer executable instructions for the data processing apparatus, and the one or more programs configured to be executed by the one or more processors include instructions for:

Acquiring target text data to be detected and a first prediction tag corresponding to the target text data;

training a first model based on the target text data and a first prediction tag corresponding to the target text data book, and acquiring a plurality of candidate rules obtained by performing rule learning processing on the target text data by the first model under the condition that the first model meets a preset convergence condition, wherein the first model is constructed based on a preset rule learning algorithm and is used for generating a model of a rule corresponding to a negative sample in a text data sample;

acquiring a second prediction tag which is output by inputting the target text data into the first model, and determining the classification accuracy corresponding to the candidate rule based on the second prediction tag and the first prediction tag corresponding to the target text data;

determining a target rule corresponding to a negative sample in the target text data in the candidate rule based on the classification accuracy corresponding to the candidate rule;

and carrying out anomaly detection processing on the target text data based on the target rule to obtain a second anomaly detection result, and determining the anomaly detection result corresponding to the target text data based on the second anomaly detection result and a first prediction tag corresponding to the target text data.

In this specification, each embodiment is described in a progressive manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for data processing apparatus embodiments, the description is relatively simple, as it is substantially similar to method embodiments, with reference to the description of method embodiments in part.

The embodiment of the specification provides data processing equipment, because the target rule is determined through the classification accuracy corresponding to the candidate rule, the identification accuracy of the negative sample in the text data sample can be improved through the target rule, and the coverage of the negative sample in the text data sample can be realized through the target rule, so that the detection efficiency and the detection accuracy of the abnormal detection of the data can be improved through the abnormal detection processing of the data to be detected through the target rule.

Example eight

The embodiments of the present disclosure further provide a computer readable storage medium, where a computer program is stored, where the computer program when executed by a processor implements each process of the embodiments of the data processing method, and the same technical effects can be achieved, and for avoiding repetition, a detailed description is omitted herein. Wherein the computer readable storage medium is selected from Read-Only Memory (ROM), random access Memory (Random Access Memory, RAM), magnetic disk or optical disk.

The embodiment of the specification provides a computer readable storage medium, because the target rule is determined by the classification accuracy corresponding to the candidate rule, the recognition accuracy of the negative sample in the text data sample can be improved by the target rule, and the coverage of the negative sample in the text data sample can be realized by the target rule, so that the detection efficiency and the detection accuracy of the abnormal detection of the data can be improved by performing the abnormal detection processing on the data to be detected by the target rule

The foregoing describes specific embodiments of the present disclosure. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims can be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

In the 90 s of the 20 th century, improvements to one technology could clearly be distinguished as improvements in hardware (e.g., improvements to circuit structures such as diodes, transistors, switches, etc.) or software (improvements to the process flow). However, with the development of technology, many improvements of the current method flows can be regarded as direct improvements of hardware circuit structures. Designers almost always obtain corresponding hardware circuit structures by programming improved method flows into hardware circuits. Therefore, an improvement of a method flow cannot be said to be realized by a hardware entity module. For example, a programmable logic device (Programmable Logic Device, PLD) (e.g., field programmable gate array (Field Programmable Gate Array, FPGA)) is an integrated circuit whose logic function is determined by the programming of the device by a user. A designer programs to "integrate" a digital system onto a PLD without requiring the chip manufacturer to design and fabricate application-specific integrated circuit chips. Moreover, nowadays, instead of manually manufacturing integrated circuit chips, such programming is mostly implemented by using "logic compiler" software, which is similar to the software compiler used in program development and writing, and the original code before the compiling is also written in a specific programming language, which is called hardware description language (HardwareDescription Language, HDL), but not just one of the hdds, but a plurality of kinds, such as ABEL (AdvancedBoolean Expression Language), AHDL (Altera Hardware Description Language), confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware DescriptionLanguage), lava, lola, myHDL, PALASM, RHDL (Ruby Hardware Description Language), etc., VHDL (Very-High-Speed Integrated Circuit Hardware DescriptionLanguage) and Verilog are currently most commonly used. It will also be apparent to those skilled in the art that a hardware circuit implementing the logic method flow can be readily obtained by merely slightly programming the method flow into an integrated circuit using several of the hardware description languages described above.

The controller may be implemented in any suitable manner, for example, the controller may take the form of, for example, a microprocessor or processor and a computer readable medium storing computer readable program code (e.g., software or firmware) executable by the (micro) processor, logic gates, switches, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable logic controllers, and embedded microcontrollers, examples of which include, but are not limited to, the following microcontrollers: ARC625D, atmel AT91SAM, microchip PIC18F26K20, and Silicone Labs C8051F320, the memory controller may also be implemented as part of the control logic of the memory. Those skilled in the art will also appreciate that, in addition to implementing the controller in a pure computer readable program code, it is well possible to implement the same functionality by logically programming the method steps such that the controller is in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers, etc. Such a controller may thus be regarded as a kind of hardware component, and means for performing various functions included therein may also be regarded as structures within the hardware component. Or even means for achieving the various functions may be regarded as either software modules implementing the methods or structures within hardware components.

The system, apparatus, module or unit set forth in the above embodiments may be implemented in particular by a computer chip or entity, or by a product having a certain function. One typical implementation is a computer. In particular, the computer may be, for example, a personal computer, a laptop computer, a cellular telephone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

For convenience of description, the above devices are described as being functionally divided into various units, respectively. Of course, the functionality of the units may be implemented in one or more software and/or hardware when implementing one or more embodiments of the present description.

It will be appreciated by those skilled in the art that embodiments of the present description may be provided as a method, system, or computer program product. Accordingly, one or more embodiments of the present description may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Moreover, one or more embodiments of the present description can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

Embodiments of the present description are described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the specification. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

In one typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of computer-readable media.

Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer-readable media, as defined herein, does not include transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.

It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises the element.

One or more embodiments of the present specification may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. One or more embodiments of the present description may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.

In this specification, each embodiment is described in a progressive manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for system embodiments, since they are substantially similar to method embodiments, the description is relatively simple, as relevant to see a section of the description of method embodiments.

The foregoing is merely exemplary of the present disclosure and is not intended to limit the disclosure. Various modifications and alterations to this specification will become apparent to those skilled in the art. Any modifications, equivalent substitutions, improvements, or the like, which are within the spirit and principles of the present description, are intended to be included within the scope of the claims of the present description.

Claims

1. A data processing method, comprising:

2. The method according to claim 1, wherein the preset rule learning algorithm includes a sequential overlay algorithm, the training the first model based on the first text data sample and a type tag corresponding to the first text data sample, and obtaining a plurality of candidate rules obtained by performing rule learning processing on the first text data sample by the first model if the first model meets a preset convergence condition, including:

3. The method of claim 2, the determining a second training set based on the first text data samples of the first text data samples other than the first training set, comprising:

4. The method of claim 1, wherein the obtaining a type tag corresponding to the first text data sample comprises:

5. The method of claim 4, the determining, based on the trained third model, a marginal contribution of feature data contained by the first text data sample, comprising:

6. The method of claim 5, the method further comprising:

acquiring target text data to be detected;

performing anomaly detection processing on the target text data based on the pre-trained second model to obtain a first anomaly detection result;

acquiring marginal contribution degree of the feature data contained in the target text data based on the trained third model, and determining first feature data in the feature data contained in the target text data based on the marginal contribution degree of the feature data contained in the target text data;

Performing anomaly detection processing on the target text data based on the target rule to obtain a second anomaly detection result;

an abnormality detection result for the target text data is determined based on first feature data, the first abnormality detection result, and the second abnormality detection result among feature data included in the target text data.

7. A data processing method, comprising:

8. The method of claim 7, wherein the first prediction label corresponding to the target text data is a label obtained by performing anomaly detection processing on the target text data based on a pre-trained second model, and the second model is an unsupervised learning model constructed based on a preset machine learning algorithm.

9. A data processing apparatus comprising:

the system comprises a sample acquisition module, a first model generation module and a model generation module, wherein the sample acquisition module is used for acquiring a first text data sample for training a first model and a type label corresponding to the first text data sample, the first model is constructed based on a preset rule learning algorithm and is used for generating a rule corresponding to a negative sample in the text data sample;

the rule acquisition module is used for training the first model based on the first text data sample and the type label corresponding to the first text data sample, and acquiring a plurality of candidate rules obtained by the rule learning processing of the first text data sample by the first model under the condition that the first model meets the preset convergence condition;

The label acquisition module is used for acquiring a prediction label which is output by inputting the first text data sample into the first model, and determining the classification accuracy corresponding to the candidate rule based on the prediction label and the type label corresponding to the first text data sample;

the rule determining module is used for determining a target rule corresponding to the negative sample in the first text data sample in the candidate rule based on the classification accuracy corresponding to the candidate rule, wherein the target rule is used for carrying out abnormality detection processing on the data to be detected.

10. A data processing apparatus comprising:

the data acquisition module is used for acquiring target text data to be detected and a first prediction tag corresponding to the target text data;

the rule acquisition module is used for training a first model based on the target text data and a first prediction label corresponding to the target text data book, and acquiring a plurality of candidate rules obtained by performing rule learning processing on the target text data by the first model under the condition that the first model meets a preset convergence condition, wherein the first model is constructed based on a preset rule learning algorithm and is used for generating a model of a rule corresponding to a negative sample in a text data sample;

The label acquisition module is used for acquiring a second prediction label which is output by inputting the target text data into the first model, and determining the classification accuracy corresponding to the candidate rule based on the second prediction label and the first prediction label corresponding to the target text data;

the rule determining module is used for determining a target rule corresponding to the negative sample in the target text data in the candidate rule based on the classification accuracy corresponding to the candidate rule;

the anomaly detection module is used for carrying out anomaly detection processing on the target text data based on the target rule to obtain a second anomaly detection result, and determining the anomaly detection result corresponding to the target text data based on the second anomaly detection result and a first prediction tag corresponding to the target text data.

11. A data processing apparatus, the data processing apparatus comprising:

a processor; and

a memory arranged to store computer executable instructions that, when executed, cause the processor to:

12. A data processing apparatus, the data processing apparatus comprising:

a processor; and