CN116861039A - Data processing method and device, electronic equipment and storage medium - Google Patents

Abstract

The application provides a data processing method, a data processing device, electronic equipment and a storage medium, relates to the technical field of communication, and is used for solving the problems of high total resistance data acquisition cost, high time delay and inaccurate total resistance type judgment. The method comprises the following steps: acquiring a service data set at the current moment; obtaining a total resistance data set according to the service data set at the current moment; the total resistance data set comprises business data with business volume of 0 or business success rate lower than a first threshold value; and determining a target total resistance data set at the current moment according to the total resistance data set and the historical total resistance data set before the current moment.

Description

Data processing method and device, electronic equipment and storage medium

Technical Field

The present application relates to the field of communications technologies, and in particular, to a data processing method, a data processing device, an electronic device, and a storage medium.

Background

The situation that the service volume is 0 or the success rate is low is called total resistance, namely total resistance; the service data in which the service volume is 0 or the service success rate is low is called full resistance data. The processing of the full-resistance data is related to the reporting of the service condition, so that higher timeliness and accuracy are required.

At present, the method for processing the total resistance data mainly comprises the following steps: using a general distributed data processing engine (Spark) to read all business data in the previous period (for example, 1 day, 2 days, or one week, etc.), and then checking all resistance data from the business data; or using a manual real-time observation mode to view the total resistance data.

However, the existing full-resistance data processing method has the problems of high full-resistance data acquisition cost, high time delay and inaccurate full-resistance type judgment, and the timeliness and the accuracy of reporting the service condition are difficult to ensure.

Disclosure of Invention

The application provides a data processing method, a device, electronic equipment and a storage medium, relates to the technical field of communication, and can solve the problems that the existing full-resistance data processing method is high in full-resistance data acquisition cost, high in time delay and inaccurate in full-resistance type judgment, and timeliness and accuracy of service condition reporting are difficult to guarantee.

In a first aspect, the present application provides a data processing method, including: acquiring a service data set at the current moment; obtaining a total resistance data set according to the service data set at the current moment; the total resistance data set comprises business data with business volume of 0 or business success rate lower than a first threshold value; and determining a target total resistance data set at the current moment according to the total resistance data set and the historical total resistance data set before the current moment.

The technical scheme provided by the application has at least the following beneficial effects: by analyzing and filtering the service data set at the current moment, the target total resistance data set at the current moment is determined, and compared with the existing total resistance data processing method, the total resistance data can be obtained in real time, the time delay and the cost are reduced, and therefore timeliness of reporting the service condition is guaranteed.

In one possible implementation, the service data in the service data set includes a traffic volume and a service status; the traffic status includes success or failure; obtaining a total resistance data set according to the service data set at the current moment, wherein the total resistance data set comprises: and if the service data with the service success rate lower than the first threshold exists in the service data set at the current moment, the service data is used as the total resistance data of the first total resistance type, and the total resistance data set is obtained.

Based on the possible implementation manner, the total resistance data of the first total resistance type is determined by dividing the total resistance type of the total group data, so that a total resistance data set is obtained, and compared with the existing total resistance data processing method, the method can accurately judge the first total resistance type, improves the accuracy of judging the total resistance type, and further ensures the accuracy of reporting the service condition.

In another possible implementation manner, the method further includes: and if the service data with the service volume of 0 and the service data lasting more than the first time exists in the service data set at the current moment, adding the service data into the total resistance data set as the total resistance data of the second total resistance type.

Based on the possible implementation manner, the total resistance data of the second total resistance type is determined by dividing the total resistance type of the whole group of data, so that a total resistance data set is obtained, and compared with the existing total resistance data processing method, the second total resistance type can be accurately judged, the accuracy of total resistance type judgment is improved, and the accuracy of reporting of service conditions is ensured.

In yet another possible implementation manner, the method further includes: if the service success rate of the service data set at the current moment is higher than the first threshold value and lower than the second threshold value and the service data with the service success rate being longer than the first time is continued, the service data is used as the total resistance data of the third total resistance type to be added into the total resistance data set; wherein the second threshold is greater than the first threshold.

Based on the possible implementation manner, the total resistance data of the third total resistance type is determined by dividing the total resistance type of the whole group of data, so that a total resistance data set is obtained, and compared with the existing total resistance data processing method, the third total resistance type can be accurately judged, the accuracy of judging the total resistance type is improved, and the accuracy of reporting the service condition is ensured.

In yet another possible implementation manner, the method further includes: sequentially sequencing the target total resistance data in the target total resistance data set according to the acquisition time to obtain a sequencing result; under the condition that the target total resistance data in the target total resistance data set are all the total resistance data of the first type, obtaining the time length of occurrence of the total resistance event of the first type according to the difference between the acquisition time moments of the last total resistance data and the first total resistance data in the sequencing result; the first type is any one of a first total resistance type, a second total resistance type and a third total resistance type; determining a quantized score according to the duration of the occurrence of the first type of total resistance event; the duration is inversely proportional to the quantization score.

In yet another possible implementation manner, the method further includes: the visualization shows the target total resistance dataset and the quantized score.

In another possible implementation manner, the service data set at the current time is stored in the Kafka cluster, and the obtaining the service data set at the current time includes: acquiring a current offset of data in a current consumption Kafka cluster; reading data of a target offset from the Kafka cluster based on the offset as a service data set at the current moment; the target offset is equal to the current offset plus 1.

In a second aspect, the present application provides a data processing apparatus, the apparatus comprising: an acquisition module and a processing module.

The acquisition module is used for acquiring the service data set at the current moment.

The processing module is used for obtaining a total resistance data set according to the service data set at the current moment; and determining a target total resistance data set at the current moment according to the total resistance data set and the historical total resistance data set before the current moment.

Optionally, the processing module is specifically configured to, if the service data set at the current moment has service data with a service success rate lower than a first threshold, use the service data as the total resistance data of the first total resistance type, and obtain a total resistance data set.

Optionally, the processing module is specifically configured to, if the service data set at the current moment has a service volume of 0 and lasts for more than the first time, add the service data as full-resistance data of the second full-resistance type to the full-resistance data set.

Optionally, the processing module is further configured to, if the service success rate in the service data set at the current moment is higher than the first threshold and lower than the second threshold and continues for the service data above the first time, add the service data as the total resistance data of the third total resistance type to the total resistance data set.

Optionally, the processing module is further configured to sequentially sort the target total resistance data in the target total resistance data set according to the acquisition time, so as to obtain a sorting result; under the condition that the target total resistance data in the target total resistance data set are all the total resistance data of the first type, obtaining the time length of occurrence of the total resistance event of the first type according to the difference between the acquisition time moments of the last total resistance data and the first total resistance data in the sequencing result; the first type is any one of a first total resistance type, a second total resistance type and a third total resistance type; determining a quantized score according to the duration of the occurrence of the first type of total resistance event; the duration is inversely proportional to the quantization score.

Optionally, the processing module is further configured to graphically display the target total resistance dataset and the quantization score.

Optionally, the processing module is specifically configured to obtain a current offset of data in the current consumption Kafka cluster; and reading the data of the target offset from the Kafka cluster based on the offset as a service data set at the current moment.

In a third aspect, the present application provides an electronic device comprising: a processor and a memory for storing processor-executable instructions; wherein the processor is configured to perform a data processing method as in the first aspect and any one of its possible implementations.

In a fourth aspect, the present application provides a computer readable storage medium, which when executed by a processor of a server, enables the server to perform a method as provided by the first aspect and any one of its possible implementations; alternatively, the instructions in the computer-readable storage medium, when executed by a processor of the client, enable the client to perform the method as provided by the first aspect and any one of its possible implementations.

The advantages of the second to fourth aspects described above may be referred to in the first aspect, and are not described here.

Drawings

In order to more clearly illustrate the embodiments of the application or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a schematic diagram of a data processing system according to an embodiment of the present application;

FIG. 2 is a schematic flow chart of a data processing method according to an embodiment of the present application;

FIG. 3 is a schematic flow chart of another data processing method according to an embodiment of the present application;

FIG. 4 is a schematic flow chart of a data processing method according to an embodiment of the present application;

FIG. 5 is a schematic flow chart of a data processing method according to an embodiment of the present application;

FIG. 6 is a schematic flow chart of a data processing method according to an embodiment of the present application;

FIG. 7 is a schematic flow chart of a data processing method according to an embodiment of the present application;

FIG. 8 is a schematic flow chart of a data processing method according to an embodiment of the present application;

FIG. 9 is a schematic flow chart of a data processing method according to an embodiment of the present application;

FIG. 10 is a schematic diagram of a data processing apparatus according to an embodiment of the present application;

fig. 11 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed Description

The following description of the embodiments of the present application will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present application, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.

It should be noted that, in the embodiments of the present application, words such as "exemplary" or "such as" are used to mean serving as an example, instance, or illustration. Any embodiment or design described herein as "exemplary" or "for example" is not necessarily to be construed as preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "such as" is intended to present related concepts in a concrete fashion.

In addition, in the description of the embodiments of the present application, unless otherwise indicated, "/" means or, for example, a/B may mean a or B. "and/or" herein is merely an association relationship describing an association object, and means that three relationships may exist, for example, a and/or B may mean: a exists alone, A and B exist together, and B exists alone. In addition, in the description of the embodiments of the present application, "plurality" means two or more than two.

In order to clearly describe the technical solution of the embodiments of the present application, in the embodiments of the present application, the terms "first", "second", etc. are used to distinguish the same item or similar items having substantially the same function and effect, and those skilled in the art will understand that the terms "first", "second", etc. are not limited in number and execution order.

First, technical terms related to the present application will be described:

1. the Hadoop distributed file system (Hadoop Distributed File System, HDFS) is a core component of a Hadoop big data system, and mainly solves the problem of storing massive big data files.

2. Spark is a general distributed data processing engine, is a fast computing engine specially designed for large-scale data processing, is an open-source cluster computing environment similar to Hadoop, but enables a memory distributed data set, and can provide interactive inquiry and optimize iterative workload.

3. Spark Streaming data processing (Spark Streaming), which is one of the core components of Spark, provides expandable, high-throughput and fault-tolerant stream computing capability for Spark; spark Streaming can integrate a variety of input data sources, such as Kafka, flume, HDFS, even a common TCP socket.

4. Redis, an open source written using ANSIC, contains various data structures, supports the network, is based on memory, the key value pair storage database of optional persistence, is one of the most popular NoSQL databases.

5. Kafka, a high throughput distributed publish-subscribe messaging system that handles all action flow data for consumers in websites; kafka may unify on-line and off-line message processing and provide real-time messages through a parallel loading mechanism with Hadoop.

The situation that the service volume is 0 or the success rate is low is called total resistance, namely total resistance; the service data in which the service volume is 0 or the service success rate is low is called full resistance data. The processing of the full-resistance data is related to the reporting of the service condition, so that higher timeliness and accuracy are required.

At present, the method for processing the total resistance data mainly comprises the following steps: using a general distributed data processing engine (Spark) to read all business data in the previous period (for example, 1 day, 2 days, or one week, etc.), and then checking all resistance data from the business data; or using a manual real-time observation mode to view the total resistance data.

However, the existing full-resistance data processing method has the problems of high full-resistance data acquisition cost, high time delay and inaccurate full-resistance type judgment, and the timeliness and the accuracy of reporting the service condition are difficult to ensure.

Based on the above, the embodiment of the application provides a data processing method, a device, equipment and a storage medium, which can acquire full-resistance data in real time by analyzing and filtering a service data set at the current moment, thereby reducing time delay and cost and ensuring timeliness and accuracy of reporting service conditions.

For easy understanding, the data processing method provided by the present application is specifically described below with reference to the accompanying drawings.

FIG. 1 is a schematic diagram of a data processing system according to an embodiment of the present application. As shown in FIG. 1, the data processing system includes a ticket file storage device 100, a Kafka cluster 200, a Spark Streaming300, a Redis cluster 400, an HDFS file system 500, and a display device 600.

The ticket file storage device 100 may be a locally accessed data storage medium such as a blu-ray disc, a high density digital video disc (Digital Video Disc, DVD), a compact disc-read Only (Compact Disc Read-Only Memory, CD-ROM), flash Memory, or other suitable digital storage medium for storing encoded video data.

The ticket file storage device 100 is used for collecting and storing ticket data (or service data) generated when a service occurs.

The Kafka cluster 200 is a high throughput distributed publish-subscribe messaging system, and reference is specifically made to the related art, and will not be repeated here.

The Kafka cluster 200 is used to collect the service data set at the current time.

Spark Streaming300 is one of the core components of Spark.

The Spark Streaming300 is used for monitoring and processing the service data set in Kafka in real time, and screening out the full-resistance data, and the specific process can be described in the following method embodiments, which are not repeated here.

In some embodiments Spark Streaming may also be used to process the traffic data set in Kafka into a desired format (e.g., (time, dimension information), number of traffic, number of successful traffic/total number of traffic).

Redis cluster 400 is a key-value pair storage database, and specific reference to the related art is omitted here.

The dis cluster 400 is used for receiving data written by the Spark Streaming300, and a specific process may be described in the following method embodiments, which are not repeated herein. The HDFS file system 500 is an easily scalable distributed file system, and reference may be made to the related art for details, which are not described herein.

The HDFS file system 500 is configured to store and provide historical full-block data to the Spark Streaming300, and the specific process may be described in the following method embodiments, which are not repeated herein. The data display device 600 may be a display device having a display function such as a light emitting diode (Light Emitting Diode, LED)/liquid crystal (liquid crystal display, LCD) display, an LED/LCD television, a mobile phone, a notebook computer, an integrated machine, or a tablet computer.

The display device 600 may be used to visually present a full resistance dataset.

The execution subject of the data processing method provided by the embodiment of the application can be a data processing device. Alternatively, the data processing apparatus may be an electronic device with a computing function in Spark Streaming300 in fig. 1; alternatively, the data processing device may be a processor in the electronic apparatus; still alternatively, the data processing apparatus may be an Application (APP) having a data processing function installed in the electronic device; alternatively, the data processing device may be a functional module or the like having a data processing function in the electronic apparatus. The embodiments of the present application are not limited in this regard.

For simplicity of description, the following description will generally take an electronic device as an example to describe a data processing method provided by an embodiment of the present application.

The following describes a data processing method provided by an embodiment of the present application with reference to the accompanying drawings.

Fig. 2 is a schematic flow chart of a data processing method according to an embodiment of the present application. As shown in fig. 2, the data processing method includes:

s101, the electronic equipment acquires a service data set at the current moment.

The service data set can comprise call ticket data, and the service data in the service data set comprises service volume and service state; the traffic status includes success or failure.

Optionally, the service data may further include: calling number, called number, time of ticket, province of ticket attribution, sp number and service direction.

Optionally, the electronic device may further convert the service data in the service data set into a binary encrypted file after acquiring the service data set at the current time.

The specific process of S101 may be described in S1011 to S1012 of fig. 3 as follows, and will not be described here.

S102, the electronic equipment obtains a total resistance data set according to the service data set at the current moment.

Wherein the total resistance data set comprises service data with a service volume of 0 or a service success rate lower than a first threshold value.

For example, taking the first threshold as 20%, the total resistance data set includes traffic data with traffic volume of 0 or traffic success rate lower than 20%.

The specific process of S102 may be described in S201 to S401 in fig. 4 to 6 as follows, and will not be described here.

And S103, the electronic equipment determines a target total resistance data set at the current moment according to the total resistance data set and the historical total resistance data set before the current moment.

For example, the electronic device may read a historical total resistance dataset stored in the HDFS file system within a preset period (e.g., 10 minutes) before the current time, and combine the total resistance dataset obtained according to the service dataset at the current time to synthesize a target total resistance dataset at the current time.

In the data processing method provided by the embodiment of the application, the electronic equipment can analyze and filter the service data set at the current moment to determine the target total resistance data set at the current moment, and compared with the existing total resistance data processing method, the total resistance data can be acquired in real time, the time delay and the cost are reduced, and the timeliness of reporting the service condition is ensured.

In some possible embodiments, the service data set at the current time is stored in the Kafka cluster, in which case fig. 3 is another flow chart of the data processing method provided in the embodiment of the present application. As shown in fig. 3, S101 may specifically include S1011 to S1012.

S1011, the electronic equipment acquires the current offset of the data in the current consumption Kafka cluster.

S1012, reading data of a target offset from the Kafka cluster based on the current offset as a service data set at the current moment.

Wherein the target offset is equal to the current offset plus 1.

Taking the current offset as 5 as an example, the target offset is equal to the current offset plus 1 is equal to 6, the electronic device reads the data from 6 and later as the service data set at the current time.

In some possible embodiments, as described above, the traffic data in the traffic data set includes traffic volume and traffic status; the traffic status includes success or failure. In this case, fig. 4 is a schematic flow chart of a data processing method according to an embodiment of the present application. As shown in fig. 4, the data processing method includes:

s201, if service data with the service success rate lower than a first threshold exists in the service data set at the current moment, the service data is used as the total resistance data of the first total resistance type, and the total resistance data set is obtained.

Optionally, the electronic device may filter the service data having the service success rate lower than the first threshold in the service data set at the current time to obtain a full-resistance data set.

For example, taking the first threshold value as 20% as an example, when the electronic device determines that the service success rate (the success rate is the service number that is counted successfully according to the dimension divided by the total service number) at a certain time point is 0, it may be determined that the service success rate 0 at the time point is lower than the first threshold value by 20%, so that the service data is used as the total resistance data of the first total resistance type, and a total resistance data set is obtained.

In some possible embodiments, the service data in the service data set includes a traffic volume and a service status; the traffic status includes success or failure. In this case, fig. 5 is a schematic flow chart of a data processing method according to an embodiment of the present application. As shown in fig. 5, the method further includes:

and S301, if the service data with the service volume of 0 and the duration of more than the first time exists in the service data set at the current moment, adding the service data into the full-resistance data set as the full-resistance data of the second full-resistance type.

Optionally, the electronic device may filter the service data set having the service volume of 0 and lasting for more than the first time to obtain the full-resistance data set.

For example, taking the first duration of 10 minutes as an example, when the electronic device determines that the service data set at the current moment has service data with the service volume of 0 and lasting for more than 10 minutes, the service data can be determined to be the second type of total resistance data. If the total resistance data set does not exist currently, taking the second type of total resistance data as the total resistance data set; if the total resistance data set exists currently, adding the second type of total resistance data into the existing total resistance data set.

In some possible embodiments, the service data in the service data set includes a traffic volume and a service status; the traffic status includes success or failure. In this case, fig. 6 is a schematic flow chart of a data processing method according to an embodiment of the present application. As shown in fig. 6, the method further includes:

s401, if the service success rate of the service data set at the current moment is higher than the first threshold and lower than the second threshold, and the service data with the service success rate being longer than the first time is continued, the service data is added into the total resistance data set as the total resistance data of the third total resistance type.

Wherein the second threshold is greater than the first threshold.

Optionally, the electronic device may filter the service data set with the service success rate higher than the first threshold and lower than the second threshold and lasting for more than the first time to obtain the full-resistance data set.

For example, taking the first threshold value as 0, the second threshold value as 20, and the first duration as 10 minutes as an example, when the electronic device determines that the service data set at the current moment has service data with a service success rate greater than 0 and less than 20% and lasting for more than 10 minutes, the electronic device can determine that the service data is all-resistive data of a third type. If the total resistance data set does not exist currently, taking the total resistance data of the third type as the total resistance data set; if the total resistance data set exists currently, adding the third type of total resistance data into the existing total resistance data set.

In some possible embodiments, after S103, the electronic device may further determine a quantized score of the traffic according to the target total resistance dataset. In this case, fig. 7 is a schematic flow chart of a data processing method according to an embodiment of the present application. As shown in fig. 7, the method further includes:

s501, sequentially sorting the target total resistance data in the target total resistance data set according to the acquisition time to obtain a sorting result.

Alternatively, as described above, the total resistance data set may include total resistance data of the first total resistance type, total resistance data of the second total resistance type, and total resistance data of the third total resistance type. In this case, the electronic device may define a corresponding variable stack (Scala stack) for each of the target total resistance data of the total resistance type as a result of the ordering of the target total resistance data of the corresponding total resistance type.

Optionally, the electronic device may also initialize attribute data (or metadata) for each of the variable stacks. Wherein the attribute data of each variable stack may include a plurality of dimensions, time, and total resistance types.

For example, the electronic device may traverse the target total resistance dataset and determine whether the target total resistance data is present in the corresponding variable stack. If the target total resistance data does not exist, the electronic equipment writes the target total resistance data into the corresponding variable stack, and records the writing time; if so, the electronic device writes the target total resistance data into the corresponding variable stack and determines the total resistance type of the target total resistance data (as described in S503 below), updating the attribute data of the corresponding variable stack.

Taking the example that the total resistance type in the target total resistance data set is the first total resistance type as an example, assuming that the total resistance data of the first total resistance type does not exist in the corresponding variable stack, the electronic device writes the target total resistance data into the stack, and records the writing time.

Taking the example that the total resistance type in the target total resistance data set is the second total resistance type as an example, assuming that the total resistance data of the second total resistance type exists in the corresponding variable stack, the electronic device writes the target total resistance data in the stack, records the writing time, and determines the total resistance type of the target total resistance data (as described in S503 below), and updates the attribute data of the corresponding variable stack.

S502, obtaining the time length of the first type of total resistance event according to the difference between the acquisition time of the last total resistance data and the acquisition time of the first total resistance data in the sequencing result under the condition that the target total resistance data in the target total resistance data set are all the first type of total resistance data.

Wherein the first type is any one of a first total resistance type, a second total resistance type, and a third total resistance type.

Taking the first type as the first total resistance type as an example, it is assumed that target total resistance data of the first total resistance type already exists in the variable stack, and the target total resistance data of the first total resistance type in the sequencing result is: the total resistance data 1 (09:20:06), the total resistance data 2 (09:20:36), the total resistance data 3 (09:20:56) and the total resistance data 4 (09:21:16) can determine that the last total resistance data is obtained at 09:21:16, the first total resistance data is obtained at 09:20:06, and the electronic device can determine that the duration of the first type of total resistance event is the difference between the two times 09:21:16-09:20:06=00:01:10, namely 1 minute and 10 seconds.

S503, determining a quantized score according to the duration of the first type of total resistance event.

Wherein the duration of the occurrence of the first type of all-resistive event is inversely proportional to the quantitative score.

Optionally, a quantization score deduction rule may be preset in the electronic device, a quantization score is determined, and the target total resistance data is processed according to the preset rule.

Illustratively, the pre-set quantization score deduction rule in the electronic device may be: the first total resistance type deduction rule is as follows: if the duration of the first total resistance event is less than or equal to 30 minutes, not deducting the quantitative score, and updating the total resistance data of the first total resistance type in the corresponding variable stack; if the time length of the first total resistance type total resistance event is longer than 30 minutes, the deduction basis is 10, the quantized score is deducted according to a calculation method of deduction value=10+ (the time length of the first total resistance type total resistance event-30)/5, and the total resistance data of the first total resistance type in the corresponding variable stack is updated. The second full-resistance type of deduction rule is: if the duration is less than 10 minutes, not deducting the quantitative score, and directly writing the full resistance data corresponding to the second full resistance type into Redis; if the duration reaches 10 minutes, the service early warning is performed, the quantitative score is not deducted, and the corresponding total resistance data of the second total resistance type in the variable stack is updated. The third total resistance type of the deduction rule is as follows: if the duration is less than 10 minutes, not deducting the quantitative score, and directly writing the total resistance data corresponding to the third total resistance type into Redis; if the duration reaches 10 minutes as the service early warning, the quantitative score is not deducted, and the total resistance data of the third total resistance type in the corresponding variable stack is updated.

For example, assuming that the first type is a first total resistance type, total resistance data of the first total resistance type already exists in the corresponding variable stack, and the aggregate time granularity is 1 minute, so that target total resistance data of the first total resistance type in the sequencing result is sequentially: for example, the electronic device may determine that the last total resistance data acquisition time is 09:21:16, and the first total resistance data acquisition time is 09:20:06, and may determine that the duration of the first type of total resistance event is the difference between two times 09:21:16-09:20:06=00:01:10, i.e. 1 minute 10 seconds, greater than 1 minute less than 30 minutes, without deducting the quantization score. The electronic equipment updates the corresponding total resistance data of the first total resistance type in the variable stack, and records that the total resistance type is the first total resistance type.

For example, assuming that the first type is a first total resistance type, total resistance data of the first total resistance type already exists in the corresponding variable stack, and the aggregate time granularity is 1 minute, so that target total resistance data of the first total resistance type in the sequencing result is sequentially: for example, the electronic device may determine that the last total resistance data is obtained at a time of 09:55:06 and the first total resistance data is obtained at a time of 09:20:06, and may determine that the duration of the first type of total resistance event is a difference between two times of 09:55:06-09:20:06=00:35:00, that is, 35 minutes, greater than 30 minutes, and deduct the quantization score according to a deduction score=10+ (35-30)/5=11 minutes. The electronic equipment updates the corresponding total resistance data of the first total resistance type in the variable stack, and records that the total resistance type is the first total resistance type.

For example, assuming that the first type is the second total resistance type, total resistance data of the second total resistance type already exists in the corresponding variable stack, and the aggregate time granularity is 1 minute, so that target total resistance data of the second total resistance type in the sequencing result is sequentially: for example, the electronic device may determine that the last total resistance data is obtained at a time of 11:38:28 and the first total resistance data is obtained at a time of 11:30:08, and may determine that the duration of the first type of total resistance event is a difference between two times of 11:38:28-11:30:08=00:08:20, that is, 8 minutes for 20 seconds, less than 10 minutes, and not deducted the quantization score. And the electronic equipment directly writes the full resistance data corresponding to the second full resistance type into the Redis.

For example, assuming that the first type is the second total resistance type, total resistance data of the second total resistance type already exists in the corresponding variable stack, so that target total resistance data of the second total resistance type in the sequencing result are sequentially: for example, the electronic device may determine that the last total resistance data acquisition time is 11:38:28, and the first total resistance data acquisition time is 11:20:08, and may determine that the duration of the first type of total resistance event is the difference between the two times 11:38:28-11:20:08=00:18:20, that is, 18 minutes 20 seconds, greater than 10 minutes, is a service early warning, and does not deduct the quantization score. And the electronic equipment updates the corresponding total resistance data of the second total resistance type in the variable stack, and records that the total resistance type is the second total resistance type.

For example, assuming that the first type is a third total resistance type, total resistance data of the third total resistance type already exists in the corresponding variable stack, and the aggregate time granularity is 1 minute, so that target total resistance data of the third total resistance type in the sequencing result is sequentially: for example, the electronic device may determine that the last total resistance data is obtained at a time of 14:41:16 and the first total resistance data is obtained at a time of 14:35:16, and may determine that the duration of the first type of total resistance event is a difference between two times of 14:41:16-14:35:16=00:06:00, i.e. 6 minutes, less than 10 minutes, without deducting the quantization score. And the electronic equipment directly writes the full resistance data corresponding to the third full resistance type into the Redis.

For example, assuming that the first type is a third total resistance type, total resistance data of the third total resistance type already exists in the corresponding variable stack, so that target total resistance data of the third total resistance type in the sequencing result are sequentially: for example, the electronic device may determine that the last total resistance data acquisition time is 14:41:16, and the first total resistance data acquisition time is 14:30:16, and may determine that the duration of the first type of total resistance event is the difference between the two times of 14:41:16-14:30:16=00:11:00, that is, 11 minutes, greater than 10 minutes, is a service early warning, and is not deducted in the score. And the electronic equipment updates the corresponding total resistance data of the third total resistance type in the variable stack, and records that the total resistance type is the third total resistance type.

In some possible embodiments, after S503, the electronic device may store both the target total resistance data set and the quantized score in the Redis cluster, and perform a visual display on the target total resistance data set and the quantized score through a display device. In this case, fig. 8 is a schematic flow chart of a data processing method according to an embodiment of the present application. As shown in fig. 8, the method further includes:

s601, the target total resistance data set is displayed in a visual mode, and the score is quantized.

For example, the target total resistance data set and the quantized score in the Redis cluster can be read through a WebScokt interface, and then the target total resistance data set and the quantized score are visually displayed by a display device by using a web application framework (such as Vue. Js).

Based on the understanding of the foregoing embodiments, fig. 9 is a schematic flow chart of a data processing method according to an embodiment of the present application. As shown in fig. 9, the data processing method may include:

s701, obtaining Kafka ticket data.

The specific process of S701 may be described with reference to S101 in fig. 1, and will not be described herein.

S702, filtering a multi-dimensional real-time total resistance data set.

The electronic device filters the Kafka call ticket data to obtain a multi-dimensional real-time total resistance data set, and the specific process of S702 may be described with reference to S102 in fig. 1, which is not described herein.

S703, merging the historical total resistance data in the HDFS.

The specific process of S703 may be described with reference to S103 in fig. 1, and will not be described herein.

S704, determining a total resistance data set at the current moment.

The electronic device obtains the total resistance data at the current moment, combines the historical total resistance data stored in the HDFS in a preset period (for example, 10 minutes), and generates the target total resistance data set by combining at the current moment, and the specific process of S704 may be described with reference to S103 in fig. 1, which is not repeated herein.

S705, judging whether the number of the total resistance data at the current moment is larger than i.

Wherein, i is added with 1 each time, judging whether the number of the total resistance data at the current moment is larger than i; if yes, judging the type of the total resistance data and the quantized score (S706); if not, ending the flow.

S706, judging the type of the total resistance data and the quantization score.

S706 may specifically include S7061 to S7067.

S7061, it is determined whether the previous time period is full resistance and the current time is also full resistance.

Judging whether the previous time period and the current time period are all resistance, if so, determining the type of the all resistance at the current moment; if not, determining the total resistance type at the current moment.

The specific process of S7061 may be described with reference to S201 to S401 in fig. 4 to 6, and will not be described here.

S7062, judging the total resistance type.

The electronic equipment determines the total resistance type at the current moment, and if the total resistance type at the current moment is 1, S7063 is executed; if the total resistance type at the present time is 2or3 (illustrated as 2or3 in fig. 9), S7064 is executed.

The specific process of S7062 may be described with reference to S502 in fig. 7, and will not be described here again.

S7063, judging whether the total resistance time is longer than 30 minutes.

The specific process of S7063 for the electronic device to determine whether the total resistance duration is greater than 30 minutes may be described with reference to S503 in fig. 7, which is not described herein.

S70631, writing the full resistance data of the time period into Redis and deducting.

The specific process of S70631 may be described with reference to S503 in fig. 7, and will not be described herein.

S70632, the full resistance data writing Redis of the time period is not deducted.

The specific process of S70632 may be described with reference to S503 in fig. 7, and will not be described herein.

S7064, judging whether the total resistance time is longer than 30 minutes.

The specific process of S7064 for the electronic device to determine whether the total resistance duration is greater than 30 minutes may be described with reference to S503 in fig. 7, which is not described herein.

S70641, the full resistance data writing Redis of the time period is not deducted.

The specific process of S70641 may be described with reference to S503 in fig. 7, and will not be described herein.

S7065, judging the total resistance type.

The electronic equipment determines the total resistance type at the current moment, and if the total resistance type at the current moment is 1, S7066 is executed; if the total resistance type at the present time is 2or3 (illustrated as 2or3 in fig. 9), S7067 is executed.

The specific process of S7065 may be described with reference to S502 in fig. 7, and will not be described here again.

S7066, the full-resistance data write Redis for this period is not deducted.

The specific process of S7066 may be described with reference to S503 in fig. 7, and will not be described here again.

S7067, writing HDFS.

All of the total resistance data is written into the HDFS.

In an exemplary embodiment, the embodiment of the present application further provides a data processing apparatus, and fig. 10 is a schematic structural diagram of the data processing apparatus provided in the embodiment of the present application. As shown in fig. 10, the apparatus includes:

an acquisition module 1001 and a processing module 1002.

The acquiring module 1001 is configured to acquire a service data set at a current time.

The processing module 1002 is configured to obtain a total resistance data set according to a service data set at a current moment; and determining a target total resistance data set at the current moment according to the total resistance data set and the historical total resistance data set before the current moment.

In some possible embodiments, the processing module 1002 is specifically configured to, if there is service data with a service success rate lower than a first threshold in the service data set at the current time, use the service data as the total resistance data of the first total resistance type, and obtain a total resistance data set.

In some possible embodiments, the processing module 1002 is specifically configured to add the service data as the second total resistance type total resistance data to the total resistance data set if the service data set at the current time has a service volume of 0 and lasts for more than the first time.

In some possible embodiments, the processing module 1002 is further configured to add the service data as the third total resistance type total resistance data to the total resistance data set if the service success rate in the service data set at the current time is higher than the first threshold, lower than the second threshold, and continues for the service data above the first time.

In some possible embodiments, the processing module 1002 is further configured to sequentially sort the target total resistance data in the target total resistance data set according to the obtaining time, so as to obtain a sorting result; under the condition that the target total resistance data in the target total resistance data set are all the total resistance data of the first type, obtaining the time length of occurrence of the total resistance event of the first type according to the difference between the acquisition time moments of the last total resistance data and the first total resistance data in the sequencing result; the first type is any one of a first total resistance type, a second total resistance type and a third total resistance type; determining a quantized score according to the duration of the occurrence of the first type of total resistance event; the duration is inversely proportional to the quantization score.

In some possible embodiments, the processing module 1002 is further configured to graphically display the target total resistance dataset and the quantization score.

In some possible embodiments, the processing module 1002 is specifically configured to obtain a current offset of data in the current consumption Kafka cluster; and reading the data of the target offset from the Kafka cluster based on the offset as a service data set at the current moment.

In an exemplary embodiment, the present application also provides a computer program product, which when run on a computer causes the computer to perform the above-mentioned related method steps to implement the data processing method in the above-mentioned embodiments.

In an exemplary embodiment, the embodiment of the application further provides electronic equipment. Fig. 11 is a schematic structural diagram of an electronic device according to an embodiment of the present application. As shown in fig. 11, the electronic device may include: a processor 1101 and a memory 1102; memory 1102 stores instructions executable by processor 1101; the processor 1101 is configured to execute instructions, which, when executed, cause the electronic device to implement the method as described in the method embodiments described previously.

In an exemplary embodiment, embodiments of the application also provide a computer-readable storage medium having stored thereon computer program instructions; the computer program instructions, when executed by an electronic device, cause the electronic device to implement the method as described in the previous embodiments. The computer readable storage medium may be a non-transitory computer readable storage medium, which may be, for example, ROM, random Access Memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.

The foregoing is merely illustrative of specific embodiments of the present application, and the scope of the present application is not limited thereto, but any changes or substitutions within the technical scope of the present application should be covered by the scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims (10)

1. A method of data processing, the method comprising:

acquiring a service data set at the current moment;

obtaining a total resistance data set according to the service data set at the current moment; the total resistance data set comprises business data with business volume of 0 or business success rate lower than a first threshold value;

and determining a target total resistance data set at the current moment according to the total resistance data set and the historical total resistance data set before the current moment.

2. The method of claim 1, wherein the service data in the service data set comprises traffic volume and service status; the service status includes success or failure; the obtaining the total resistance data set according to the service data set at the current moment comprises the following steps:

and if the service data with the service success rate lower than the first threshold exists in the service data set at the current moment, the service data is used as the total resistance data of the first total resistance type, and the total resistance data set is obtained.

3. The method according to claim 2, wherein the method further comprises:

and if the service data with the service volume of 0 and the service data lasting for more than the first time exists in the service data set at the current moment, adding the service data into the total resistance data set as the total resistance data of the second total resistance type.

4. A method according to claim 3, characterized in that the method further comprises:

if the service success rate of the service data set at the current moment is higher than the first threshold and lower than the second threshold and the service data with the service data length longer than the first time is continued, the service data is used as the total resistance data of the third total resistance type to be added into the total resistance data set; the second threshold is greater than the first threshold.

5. The method according to any one of claims 2-4, further comprising:

sequentially sequencing the target total resistance data in the target total resistance data set according to the acquisition time to obtain a sequencing result;

under the condition that the target total resistance data in the target total resistance data set are all the total resistance data of the first type, obtaining the time length of occurrence of the total resistance event of the first type according to the difference between the acquisition time moments of the last total resistance data and the first total resistance data in the sequencing result; the first type is any one of a first total resistance type, a second total resistance type and a third total resistance type;

Determining a quantized score according to the duration of the first type of total resistance event; the duration is inversely proportional to the quantization score.

6. The method of claim 5, wherein the method further comprises:

the target total resistance dataset and the quantized score are visually shown.

7. The method according to any one of claims 1-4, wherein the service data set at the current time is stored in a Kafka cluster, and the obtaining the service data set at the current time includes:

acquiring a current offset of data in a current consumption Kafka cluster;

reading data of a target offset from the Kafka cluster based on the offset as a service data set of the current moment; the target offset is equal to the current offset plus 1.

8. A data processing apparatus, the apparatus comprising: the device comprises an acquisition module and a processing module;

the acquisition module is used for acquiring a service data set at the current moment;

the processing module is used for obtaining a total resistance data set according to the service data set at the current moment; and determining a target total resistance data set at the current moment according to the total resistance data set and the historical total resistance data set before the current moment.

9. An electronic device, the electronic device comprising: a processor and a memory;

the memory stores instructions executable by the processor;

the processor is configured to, when executing the instructions, cause the electronic device to implement the method of any one of claims 1-7.

10. A computer readable storage medium having stored therein software instructions which, when executed in an electronic device, cause the electronic device to implement the method of any of claims 1 to 7.